Document Sample

Greedy Algorithms Greed is good. (Some of the time) Outline Elements of greedy algorithm Greedy choice property Optimal substructures Minimum spanning tree Kruskal’s algorithm Prim’s algorithm Huffman code Activity selection Chapter 3: Greedy Algorithms www.eusman.com Introduction Concepts Choosing the best possible choice at each step. This decision leads to the best over all solution. Greedy algorithms do not always yield optimal solutions. Chapter 3: Greedy Algorithms www.eusman.com Elements of Greedy Algorithms Greedy-choice property A globally optimal solution is derived from a locally optimal (greedy) choice. When choices are considered, the choice that looks best in the current problem is chosen, without considering results from subproblems. Chapter 3: Greedy Algorithms www.eusman.com Optimal substructures A problem has optimal substructure if an optimal solution to the problem is composed of optimal solutions to subproblems. This property is important for both greedy algorithms and dynamic programming. Chapter 3: Greedy Algorithms www.eusman.com Greedy Algorithm v.s. Dynamic Programming Greedy algorithm Dynamic programming The best choice is made at A choice is made at each each step and after that the step. subproblem is solved. The choice made by a The choice made at each greedy algorithm may step usually depends on the solutions to subproblems. depend on choices so far, but it cannot depend on any future choices or on the solutions to subproblems. A greedy strategy usually Dynamic-programming progresses in a top-down problems are often solved in fashion, making one greedy a bottom-up manner. choice after another, reducing each given problem instance to a smaller one. Chapter 3: Greedy Algorithms www.eusman.com Steps in Design Greedy Algorithms Determine the optimal substructure of the problem. Develop a recursive solution. Prove that at any stage of the recursion, one of the optimal choices is the greedy choice. Thus, it is always safe to make the greedy choice. Show that all but one of the subproblems induced by having made the greedy choice are empty. Develop a recursive algorithm that implements the greedy strategy. Convert the recursive algorithm to an iterative algorithm. Chapter 3: Greedy Algorithms www.eusman.com Shortcuts Design Form the optimization problem so that, after a choice is made and there is only one subproblem left to be solved. Prove that there is always an optimal solution to the original problem that makes the greedy choice, so that the greedy choice is always safe. Demonstrate that, having made the greedy choice, what remains is a subproblem with the property that if we combine an optimal solution to the subproblem with the greedy choice we have made, we arrive at an optimal solution to the original problem. Chapter 3: Greedy Algorithms www.eusman.com Minimum Spanning Tree Definitions Let G = (V, E) be an undirected graph. T is a minimum spanning tree of G if T ⊆ E is an acyclic subset that connects all of the vertices and whose total weight w(T ) = w(u, v) is minimized. (u,v)∈T Let A be a subset of some minimum spanning tree. An edge (u, v) is a safe edge for A if A {(u, v)} is also a subset of a minimum spanning tree. Chapter 3: Greedy Algorithms www.eusman.com Definitions Let G = (V, E) be an undirected graph. A cut (S, V − S) of G is a partition of V. An edge (u, v) E crosses the cut (S, V − S) if one of its endpoints is in S and the other is in V − S. A cut respects a set A of edges if no edge in A crosses the cut. An edge is a light edge crossing a cut if its weight is the minimum of any edge crossing the cut. Chapter 3: Greedy Algorithms www.eusman.com Theorem Let G = (V, E) be a connected, S undirected graph with a real-valued weight function w defined on E, A u A be a subset of E that is included in x light edge some minimum spanning tree for G, (S, V − S) be any cut of G that respects A. v y If (u, v) is a light edge crossing (S, V − S), then edge (u, v) is safe for V-S A. Chapter 3: Greedy Algorithms www.eusman.com Proof Let T be a minimum spanning tree S that includes A. If (u,v) is in T, then (u,v) is safe for A. A u x If (u,v) is not in T, then T contains an light edge edge (x,y) crossing S and V-S. T Since both (u,v) and (x,y) cross S and v V-S, there is a cycle for edges in T y (u,v). Another spanning tree T’ can be V-S created from T. Chapter 3: Greedy Algorithms www.eusman.com Proof S S A A u u x x light edge light edge T T’ v v y y V-S V-S Remove ’edge (x,y)create T. T (u,v) -> from cycle Add edgeis created -> cutathe cycle Chapter 3: Greedy Algorithms www.eusman.com Proof Thus, T’ is a spanning tree that includes A. Next, we need to show that T’ is a minimum spanning tree. From the construction of T’ , T’ = T-{(x,y)} {(u,v)}. Thus, w(T’ ) = w(T) - w(x,y) + w(u,v). Since (u,v) is a light edge cross S and V-S, w(u,v) w(x,y). Thus, w(T’ ) w(T). Since T is a minimum spanning tree, w(T’ ) = w(T). Then, T’ is also a minimum spanning tree . As a result, (u, v) is safe for A. Chapter 3: Greedy Algorithms www.eusman.com Corollary Let G = (V, E) be a connected, undirected graph with a real- valued weight function w defined on E, A be a subset of E that is included in some minimum spanning tree for G, and C = (VC, EC) be a connected component (tree) in the forest GA = (V, A). If (u, v) is a light edge connecting C to another tree in GA, then (u, v) is safe for A. Proof (VC, V−VC) respects A, and (u, v) is a light edge for this cut. Therefore, (u, v) is safe for A. Chapter 3: Greedy Algorithms www.eusman.com Kruskal’s Algorithm Concept Build a forest of minimum spanning trees. Repeatedly connect the trees to create a subset of a minimum spanning tree, until all nodes are covered. In connecting two trees, choose the edge of the least weight. Let C1 and C2 denote the two trees that are connected by (u, v). Since (u, v) must be a light edge connecting C1 to some other tree, we need to prove that (u, v) is a safe edge for C1. Kruskal’s algorithm is a greedy algorithm, because at each step it adds to the forest an edge of least possible weight. Chapter 3: Greedy Algorithms www.eusman.com Example of Kruskal’s Algorithm 12 8 2 9 1 9 5 3 11 8 7 3 5 1 10 Chapter 3: Greedy Algorithms www.eusman.com Kruskal’s Algorithm MST-KRUSKAL(G,w) A= for each vertex v in V [G] do MAKE-SET(v) sort the edges of E in nondecreasing order by weight w for each edge (u, v) in E, taken in nondecreasing order by weight do if FIND-SET(u) FIND-SET(v) then A = A {(u, v)} UNION(u, v) return A Chapter 3: Greedy Algorithms www.eusman.com Kruskal’s Algorithm: Complexity MST-KRUSKAL(G,w) A= for each vertex v in V [G] O(v) do MAKE-SET(v) O(e lg e) sort the edges of E in nondecreasing order by weight w for each edge (u, v) in E, taken in nondecreasing order by weight do if FIND-SET(u) FIND-SET(v) then A = A {(u, v)} UNION(u, v) O((v+e) lg v) return A e v ; O(e lg v) 2 Chapter 3: Greedy Algorithms www.eusman.com Prim’s Algorithm Prim’s algorithm has the property that the edges in the set A always form a single tree. The tree starts from an arbitrary root vertex r. Grow the tree until it spans all the vertices in V. At each step, a light edge is added to the tree A that connects A to an isolated vertex of GA = (V, A). Chapter 3: Greedy Algorithms www.eusman.com Example of Prim’s Algorithm 12 8 2 9 1 9 5 3 11 8 7 3 5 1 10 Chapter 3: Greedy Algorithms www.eusman.com Prim’s Algorithm PRIM(G,w, r) for each u in V[G] do key[u] = ∞ π[u] = NIL key[r] = 0 Q = V[G] while Q do u = EXTRACT-MIN(Q) for each v in Adj[u] do if v in Q and w(u, v) < key[v] then π[v] = u key[v] = w(u, v) Chapter 3: Greedy Algorithms www.eusman.com Execution of Prim’s Algorithm 12 8 0 9 3 12 8 2 9 1 9 5 2 3 1 3 11 8 7 3 5 5 1 7 1 10 5 10 Chapter 3: Greedy Algorithms www.eusman.com Prim’s Algorithm:Complexity PRIM(G,w, r) for each u in V[G] Also build min-heap do key[u] = ∞ O(v) π[u] = NIL key[r] = 0 Q = V[G] O(e lg v) while Q do u = EXTRACT-MIN(Q) O(lg v) O(v lg v) for each v in Adj[u] do if v in Q and w(u, v) < key[v] then π[v] = u key[v] = w(u, v) O(lg v) Chapter 3: Greedy Algorithms www.eusman.com Huffman Code Problem Find an optimal prefix code representing a set of characters in a file, where each character has a frequency of occurrences. Prefix code Codes in which no codeword is also a prefix of some other codeword. {0, 110, 101, 111, 1000, 1001} is a prefix code. {0, 110, 101, 111, 1010, 0111} is not prefix code Optimality The code yields a file with the minimum number of bits. Chapter 3: Greedy Algorithms www.eusman.com Creating Huffman Code: Example 100 1 64 0 1 0 34 30 1 0 1 0 15 0 1 ก: 36 ข: 17 ค: 17 ง: 15 จ: 10 ฉ: 5 0 100 101 110 1110 1111 Chapter 3: Greedy Algorithms www.eusman.com Optimal Code An optimal code for a file is always represented by a full binary tree, in which every nonleaf node has two children. If C is the alphabet from which the characters are drawn and all character frequencies are positive, then the tree for an optimal prefix code has exactly |C| leaves, one for each letter of the alphabet, and exactly |C|−1 internal nodes. Chapter 3: Greedy Algorithms www.eusman.com Full Binary Trees for Prefix Code Tree for 1 letter Tree for 2 letters A Tree for 4 letters 1 1 2 B Tree for 3 letters C A B 4 3 1 2 3 A C 1 2 4 B A 3 A 1 B 1 2 2 3 Chapter 3: Greedy Algorithms www.eusman.com Full Binary Trees for Prefix Code Tree for 4 letters Tree for 5 letters B B C A C A 4 3 1 2 D 3 1 2 5 4 C D 4 B 5 B 3 A C A 1 2 4 3 1 2 Chapter 3: Greedy Algorithms www.eusman.com Creating Huffman Code: Example กขคงจฉ: 100 1 ขคงจฉ: 64 0 0 1 ขค: 34 งจฉ: 30 1 0 1 0 จฉ: 15 0 1 ก: 36 ข: 17 ค: 17 ง: 15 จ: 10 ฉ: 5 0 100 101 110 1110 1111 Chapter 3: Greedy Algorithms www.eusman.com Theorem A full binary tree for an optimal prefix code for C letters has exactly C leaves, and exactly C−1 internal nodes. Proof by induction. Basis: C=1. If there is one letter, the binary tree requires only 1 leaf node, and 0 internal node. Induction hypotheses: For C< n, the full binary tree for an optimal prefix code for C letters has exactly C leaves, and exactly C−1 internal nodes. Chapter 3: Greedy Algorithms www.eusman.com Theorem Induction Step: Let T be a full binary tree for an optimal prefix code for C+1 letters. To create a full binary tree or optimal prefix code, we can take a full binary tree for an optimal prefix code for C letters, and add another leaf node L by either adding a new root node R and put L and the old root of T as its children, or replacing a leaf node N of T by an internal node and put L and N as its children. In either case, the number of leaf nodes of T is C+1 and the number of internal nodes is C. Chapter 3: Greedy Algorithms www.eusman.com Creating Huffman Code: Algorithms HUFFMAN(C) Use min-heap for Q n = |C| O(n) to build min-heap Q=C for i = 1 to n − 1 O(n lg n) do allocate a new node z z.left = x = EXTRACT-MIN(Q) z.right = y = EXTRACT-MIN(Q) z.f =x.f + y.f O(lg n) INSERT(Q, z) ►Return the root of the tree. return EXTRACT-MIN(Q) Chapter 3: Greedy Algorithms www.eusman.com Greedy-choice property of Huffman Code Let C be an alphabet in which each character c C has frequency f [c], and x and y be two characters in C having the lowest frequencies. Then, there exists an optimal prefix code for C in which the codewords for x and y have the same length and differ only in the last bit. Chapter 3: Greedy Algorithms www.eusman.com Proof The idea of the proof is to : take the tree T representing an arbitrary optimal prefix code modify it to make a tree representing another optimal prefix code such that the characters x and y appear as sibling leaves of maximum depth in the new tree. If we can do this, then their codewords will have the same length and differ only in the last bit. Chapter 3: Greedy Algorithms www.eusman.com Proof Let a and b be two characters that are sibling leaves of maximum depth in T . Assume that f [a] ≤ f [b] and f [x] ≤ f [y]. Since f [x] and f [y] are the two lowest leaf frequencies, in order, and f [a] and f [b] are two arbitrary frequencies, in order, we have f [x] ≤ f [a] and f [y] ≤ f [b]. Chapter 3: Greedy Algorithms www.eusman.com Proof Exchange the positions of a and x in T to produce T’ . B(T) − B(T’ ) = f(c) dT (c) − f(c)dT’ (c) cC cC = f [x] dT (x) + f [a] dT (a) − f [x] dT’ (x) − f [a] dT’ (a) = f [x] dT (x) + f [a] dT (a) − f [x] dT (a) − f [a] dT (x) = ( f [a] − f [x])( dT (a) − dT (x)) ≥ 0 T T’ x a y y a b x b Chapter 3: Greedy Algorithms www.eusman.com Proof Then, exchange the positions of b and y in T’ to produce T’’. T’ a Similarly, it does not increase the cost, and so B(T’ ) − B(T’’ ) 0. y Therefore, B(T’’ ) ≤ B(T). x b Since T is optimal, B(T) ≤ B(T’’ ). Then, B(T’’ ) = B(T). T’’ Thus, T’’ is an optimal tree in which a x and y appear as sibling leaves of maximum depth. b x y Chapter 3: Greedy Algorithms www.eusman.com Optimal-substructure Property Let C be a given alphabet with frequency f [c] defined for each character c C, x and y be two characters in C with minimum frequency, C’ be the alphabet C with characters x, y removed and (new) character z added, so that C’ = C − {x, y} {z}; define f for C’ as for C, except that f [z] = f [x] + f [y], and T be any tree representing an optimal prefix code for the alphabet C ’. Then the tree T , obtained from T ’ by replacing the leaf node for z with an internal node having x and y as children, represents an optimal prefix code for the alphabet C. Chapter 3: Greedy Algorithms www.eusman.com Proof Show that the cost B(T) can be expressed in terms of the cost B(T’) by considering the component costs. For each c C − {x, y}, we have dT (c) = dT ’ (c). f [c] dT (c) = f [c] dT ’ (c). Since dT (x) = dT (y) = dT ’(z) + 1, we have f [x]dT (x) + f [y]dT (y) = ( f [x] + f [y])( dT’ (z) + 1) = f [z] dT ’ (z) + ( f [x] + f [y]) Thus, B(T) = B(T ’) + f [x] + f [y]. That is, B(T ’) = B(T) − f [x] − f [y] . Chapter 3: Greedy Algorithms www.eusman.com Proof We now prove by contradiction. Suppose T does not represent an optimal prefix code for C. Then there exists a tree T ’’ such that B(T ’’) < B(T). Without loss of generality, T ’’ has x and y as siblings. Chapter 3: Greedy Algorithms www.eusman.com Proof Let T ’’’ be the tree T ’’ with the common parent of x and y replaced by a leaf z with frequency f [z] = f [x] + f [y]. Then, B(T ’’’) = B(T ’’) − f [x] − f [y] < B(T) − f [x] − f [y] = B(T ’). We reach a contradiction to the assumption that T ’ represents an optimal prefix code for C’. Thus, T must represent an optimal prefix code for the alphabet C. Chapter 3: Greedy Algorithms www.eusman.com Interval Scheduling Problem definition Let S be {a1, a2, . . . , an} of n proposed activities that wish to use the same resource. Only one activity can use the resource at a time. Each activity ai has a start time si and a finish time fi, where 0 ≤ si < fi < ∞. If selected, activity ai takes place during the half-open time interval [si , fi ). Activities ai and aj are compatible if the intervals [si , fi ) and [s j , f j ) do not overlap . The activity-selection problem is to select a largest subset of mutually compatible activities. Chapter 3: Greedy Algorithms www.eusman.com Example S A1 A2 A3 Chapter 3: Greedy Algorithms www.eusman.com Subproblems Si j = {ak S : fi ≤ sk < fk ≤ s j } the subset of activities in S that can start after activity ai finishes and finish before activity aj starts. Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞. Then S = S0,n+1, and 0 ≤ i, j ≤ n + 1. Chapter 3: Greedy Algorithms www.eusman.com Optimal Solution Let Ai j be an optimal solution to Si j . Let c[i, j ] be the number of activities in a maximum-size subset of mutually compatible activities in Si j. c[i, j ] = 0 for i ≥ j (i.e. Si j = ) ak c[i, j ] = c[i, k] + c[k, j ] + 1 if ak be an activity in Ai j. Then, c[i, j ] = 0 if Si j = max {c[i, k] + c[k, j ] + 1} if Si j i<k<j akSi j Chapter 3: Greedy Algorithms www.eusman.com Theorem Consider any nonempty subproblem Si j , and let am be the activity in Si j with the earliest finish time: fm = min { fk : ak ∈ Si j } . Then, Activity am is used in some maximum-size subset of mutually compatible activities of Si j . The subproblem Sim is empty, so that choosing am leaves the subproblem Smj as the only one that may be nonempty. Chapter 3: Greedy Algorithms www.eusman.com Proof: First part Let Ai j be a maximum-size subset of mutually compatible activities of Si j , and Ai j is sorted in monotonically increasing order of finish time. Let ak be the first activity in Ai j. If ak = am, am is used in some maximum-size subset of mutually compatible activities of Si j. If ak am, we construct the subset A’i j = Ai j − {ak} {am}. S Aij ak A’ij am Chapter 3: Greedy Algorithms www.eusman.com Proof: First part The activities in A’i j are disjoint, since the activities in Aij are, ak is the first activity in Ai j to finish, and fm ≤ fk . Noting that A’i j has the same number of activities as Ai j , we see that A’i j is a maximum-size subset of mutually compatible activities of Si j that includes am. S Aij ak A’ij am Chapter 3: Greedy Algorithms www.eusman.com Proof: Second part Let Sim be nonempty. Then, there is an activity ak such that fi ≤ sk <fk ≤ sm< fm. Then, ak is also in Si j and it has an earlier finish time than am, which contradicts our choice of am. We conclude that Sim is empty. ai am S ak aj Chapter 3: Greedy Algorithms www.eusman.com Greedy Solution Ai j = {ak} Akj where ak is the activity which finishes earliest among activities in Si j. Chapter 3: Greedy Algorithms www.eusman.com Recursive Algorithms ACTIVITY-SELECTOR(s, f, i, n) m=i+1 while m ≤ n and s[m] < f [i] ► Find the first activity in Si,n+1. do m= m+1 if m ≤ n then return {am} ACTIVITY-SELECTOR(s, f,m, n) else return Chapter 3: Greedy Algorithms www.eusman.com Iterative Algorithm GREEDY-ACTIVITY-SELECTOR(s, f ) n = length[s] A = {a1} i=1 for m = 2 to n do if s[m ] ≥ f[i] then A = A {am} i=m return A Chapter 3: Greedy Algorithms www.eusman.com

DOCUMENT INFO

Shared By:

Categories:

Tags:

Stats:

views: | 5 |

posted: | 8/18/2012 |

language: | English |

pages: | 57 |

OTHER DOCS BY ewghwehws

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.