Docstoc

Greedy Algorithms - eusman.com

Document Sample
Greedy Algorithms - eusman.com Powered By Docstoc
					Greedy Algorithms
     Greed is good.
   (Some of the time)
Outline
   Elements of greedy algorithm
       Greedy choice property
       Optimal substructures
   Minimum spanning tree
       Kruskal’s algorithm
       Prim’s algorithm
 Huffman code
 Activity selection




                        Chapter 3: Greedy Algorithms www.eusman.com
Introduction
   Concepts
       Choosing the best possible choice at each step.
       This decision leads to the best over all solution.

   Greedy algorithms do not always yield
    optimal solutions.




                         Chapter 3: Greedy Algorithms www.eusman.com
   Elements of
Greedy Algorithms
Greedy-choice property
   A globally optimal solution is derived from a
    locally optimal (greedy) choice.

   When choices are considered, the choice that
    looks best in the current problem is chosen,
    without considering results from
    subproblems.




                    Chapter 3: Greedy Algorithms www.eusman.com
Optimal substructures
   A problem has optimal substructure if an
    optimal solution to the problem is composed
    of optimal solutions to subproblems.

   This property is important for both greedy
    algorithms and dynamic programming.




                    Chapter 3: Greedy Algorithms www.eusman.com
Greedy Algorithm v.s. Dynamic Programming
Greedy algorithm                    Dynamic programming
 The best choice is made at         A choice is made at each
  each step and after that the        step.
  subproblem is solved.
 The choice made by a                 The choice made at each
  greedy algorithm may                  step usually depends on the
                                        solutions to subproblems.
  depend on choices so far,
  but it cannot depend on any
  future choices or on the
  solutions to subproblems.
 A greedy strategy usually            Dynamic-programming
  progresses in a top-down              problems are often solved in
  fashion, making one greedy            a bottom-up manner.
  choice after another,
  reducing each given problem
  instance to a smaller one.

                       Chapter 3: Greedy Algorithms www.eusman.com
Steps in Design Greedy Algorithms
   Determine the optimal substructure of the problem.
   Develop a recursive solution.
   Prove that at any stage of the recursion, one of the
    optimal choices is the greedy choice. Thus, it is
    always safe to make the greedy choice.
   Show that all but one of the subproblems induced by
    having made the greedy choice are empty.
   Develop a recursive algorithm that implements the
    greedy strategy.
   Convert the recursive algorithm to an iterative
    algorithm.
                       Chapter 3: Greedy Algorithms www.eusman.com
Shortcuts Design
   Form the optimization problem so that, after a
    choice is made and there is only one subproblem left
    to be solved.
   Prove that there is always an optimal solution to the
    original problem that makes the greedy choice, so
    that the greedy choice is always safe.

   Demonstrate that, having made the greedy choice,
    what remains is a subproblem with the property that
    if we combine an optimal solution to the subproblem
    with the greedy choice we have made, we arrive at
    an optimal solution to the original problem.


                      Chapter 3: Greedy Algorithms www.eusman.com
Minimum Spanning Tree
Definitions
   Let G = (V, E) be an undirected graph.
    T is a minimum spanning tree of G if T ⊆ E is an
    acyclic subset that connects all of the vertices and whose
    total weight w(T ) =  w(u, v) is minimized.
                        (u,v)∈T



   Let A be a subset of some minimum spanning tree.
    An edge (u, v) is a safe edge for A if A  {(u, v)} is also
    a subset of a minimum spanning tree.



                         Chapter 3: Greedy Algorithms www.eusman.com
Definitions
   Let G = (V, E) be an undirected graph.
    A cut (S, V − S) of G is a partition of V.
    An edge (u, v)  E crosses the cut (S, V − S) if one of its
    endpoints is in S and the other is in V − S.
    A cut respects a set A of edges if no edge in A crosses the
    cut.
    An edge is a light edge crossing a cut if its weight is the
    minimum of any edge crossing the cut.




                        Chapter 3: Greedy Algorithms www.eusman.com
Theorem
Let G = (V, E) be a connected,                                          S
  undirected graph with a real-valued
  weight function w defined on E,                                                     A
                                                                u
  A be a subset of E that is included in                                          x




                                                       light edge
  some minimum spanning tree for G,
  (S, V − S) be any cut of G that
  respects A.
                                                                    v
                                                                                  y
  If (u, v) is a light edge crossing (S,
  V − S), then edge (u, v) is safe for                                      V-S
  A.
                       Chapter 3: Greedy Algorithms www.eusman.com
Proof
  Let T be a minimum spanning tree                                   S
  that includes A.
If (u,v) is in T, then (u,v) is safe for A.                                    A
                                                             u
                                                                           x
If (u,v) is not in T, then T contains an




                                                    light edge
   edge (x,y) crossing S and V-S.
                                                                               T
Since both (u,v) and (x,y) cross S and
                                                                 v
  V-S, there is a cycle for edges in T                                     y
   (u,v).
Another spanning tree T’ can be                                      V-S
 created from T.
                         Chapter 3: Greedy Algorithms www.eusman.com
Proof
                           S                                             S


                                         A                                         A
                     u                                               u
                                     x                                         x




                                                        light edge
        light edge




                                         T                                         T’
                     v                                               v
                                    y                                          y


                          V-S                                            V-S

                         Remove ’edge (x,y)create T.
                               T (u,v) -> from cycle
                         Add edgeis created -> cutathe cycle
                                Chapter 3: Greedy Algorithms www.eusman.com
Proof
Thus, T’ is a spanning tree that includes A.

Next, we need to show that T’ is a minimum spanning tree.
From the construction of T’ , T’ = T-{(x,y)}  {(u,v)}.
Thus, w(T’ ) = w(T) - w(x,y) + w(u,v).
Since (u,v) is a light edge cross S and V-S, w(u,v)  w(x,y).
Thus, w(T’ )  w(T).
Since T is a minimum spanning tree, w(T’ ) = w(T).
Then, T’ is also a minimum spanning tree .
As a result, (u, v) is safe for A.
                       Chapter 3: Greedy Algorithms www.eusman.com
Corollary
Let G = (V, E) be a connected, undirected graph with a real-
   valued weight function w defined on E,
   A be a subset of E that is included in some minimum
   spanning tree for G, and
   C = (VC, EC) be a connected component (tree) in the
   forest GA = (V, A).
If (u, v) is a light edge connecting C to another tree in GA,
   then (u, v) is safe for A.
Proof
(VC, V−VC) respects A, and (u, v) is a light edge for this cut.
Therefore, (u, v) is safe for A.
                       Chapter 3: Greedy Algorithms www.eusman.com
Kruskal’s Algorithm
   Concept
       Build a forest of minimum spanning trees.
       Repeatedly connect the trees to create a subset of a
        minimum spanning tree, until all nodes are covered.
       In connecting two trees, choose the edge of the least
        weight.

   Let C1 and C2 denote the two trees that are
    connected by (u, v). Since (u, v) must be a light
    edge connecting C1 to some other tree, we need to
    prove that (u, v) is a safe edge for C1.
   Kruskal’s algorithm is a greedy algorithm, because
    at each step it adds to the forest an edge of least
    possible weight.

                           Chapter 3: Greedy Algorithms www.eusman.com
Example of Kruskal’s Algorithm

                12                          8


            2        9                1            9

    5                       3                                11

        8            7              3               5



                1                           10




                     Chapter 3: Greedy Algorithms www.eusman.com
Kruskal’s Algorithm
MST-KRUSKAL(G,w)
 A=
 for each vertex v in V [G]
     do     MAKE-SET(v)
 sort the edges of E in nondecreasing order by weight w
 for each edge (u, v) in E, taken in nondecreasing order
 by weight
      do if FIND-SET(u)  FIND-SET(v)
            then A = A  {(u, v)}
                   UNION(u, v)
 return A
                      Chapter 3: Greedy Algorithms www.eusman.com
Kruskal’s Algorithm: Complexity
MST-KRUSKAL(G,w)
 A=
 for each vertex v in V [G] O(v)
     do     MAKE-SET(v)                        O(e lg e)
 sort the edges of E in nondecreasing order by weight w
 for each edge (u, v) in E, taken in nondecreasing order
 by weight
      do if FIND-SET(u)  FIND-SET(v)
            then A = A  {(u, v)}
                   UNION(u, v)         O((v+e) lg v)
 return A
                                   e  v ; O(e lg v)
                                     2
                      Chapter 3: Greedy Algorithms www.eusman.com
Prim’s Algorithm
   Prim’s algorithm has the property that the edges in the
    set A always form a single tree.
 The tree starts from an arbitrary root vertex r.
 Grow the tree until it spans all the vertices in V.
       At each step, a light edge is added to the tree A that connects A
        to an isolated vertex of GA = (V, A).




                             Chapter 3: Greedy Algorithms www.eusman.com
Example of Prim’s Algorithm

                12                          8


            2        9                1            9

    5                       3                                11

        8            7              3               5



                1                           10




                     Chapter 3: Greedy Algorithms www.eusman.com
Prim’s Algorithm
PRIM(G,w, r)
  for each u in V[G]
   do key[u] = ∞
       π[u] = NIL
  key[r] = 0
  Q = V[G]
  while Q  
      do      u = EXTRACT-MIN(Q)
              for each v in Adj[u]
                do if v in Q and w(u, v) < key[v]
                       then π[v] = u
                             key[v] = w(u, v)

                         Chapter 3: Greedy Algorithms www.eusman.com
Execution of Prim’s Algorithm

                 12                          8
     0                       9
                             3
                             12                             8

             2        9                1            9

    5            2           3               1
                                             3                11

         8            7              3               5


     5           1            7
                              1              10            5
                                                           10



                      Chapter 3: Greedy Algorithms www.eusman.com
Prim’s Algorithm:Complexity
PRIM(G,w, r)
  for each u in V[G] Also build min-heap
   do key[u] = ∞
                      O(v)
       π[u] = NIL
  key[r] = 0
  Q = V[G]                                        O(e                lg v)
  while Q  
      do      u = EXTRACT-MIN(Q) O(lg v) O(v                         lg v)
              for each v in Adj[u]
                do if v in Q and w(u, v) < key[v]
                       then π[v] = u
                             key[v] = w(u, v) O(lg v)

                       Chapter 3: Greedy Algorithms www.eusman.com
Huffman Code
Problem
   Find an optimal prefix code representing a
    set of characters in a file, where each
    character has a frequency of occurrences.
   Prefix code
       Codes in which no codeword is also a prefix of
        some other codeword.
       {0, 110, 101, 111, 1000, 1001} is a prefix code.
       {0, 110, 101, 111, 1010, 0111} is not prefix code
   Optimality
       The code yields a file with the minimum number
        of bits.

                       Chapter 3: Greedy Algorithms www.eusman.com
Creating Huffman Code: Example
               100
                             1

                                   64
         0                                     1
                     0

                 34                                   30
                                                             1

             0         1                      0             15

                                                        0           1

 ก: 36   ข: 17        ค: 17            ง: 15         จ: 10 ฉ: 5
  0      100           101              110          1110        1111


                      Chapter 3: Greedy Algorithms www.eusman.com
Optimal Code
   An optimal code for a file is always
    represented by a full binary tree, in which
    every nonleaf node has two children.

   If C is the alphabet from which the
    characters are drawn and all character
    frequencies are positive, then the tree for an
    optimal prefix code has exactly |C| leaves,
    one for each letter of the alphabet, and
    exactly |C|−1 internal nodes.


                    Chapter 3: Greedy Algorithms www.eusman.com
Full Binary Trees for Prefix Code
Tree for 1 letter           Tree for 2 letters
                                    A                        Tree for 4 letters
        1
                               1          2
                                                                     B
   Tree for 3 letters
                                                         C                   A
            B                                      4             3       1           2
      3         A                                            C
            1           2
                                                     4               B
            A                                                3           A
    1           B                                                    1           2
            2       3

                                   Chapter 3: Greedy Algorithms www.eusman.com
Full Binary Trees for Prefix Code
     Tree for 4 letters                                 Tree for 5 letters

             B                                                    B
     C               A                                  C                    A
 4       3       1        2                       D           3       1          2
                                             5           4

          C
                                                          D
     4           B
                                                   5              B
             3       A
                                                        C                    A
                 1        2
                                                   4          3       1          2

                              Chapter 3: Greedy Algorithms www.eusman.com
Creating Huffman Code: Example
               กขคงจฉ: 100

                             1


                          ขคงจฉ: 64
           0       0                        1

               ขค: 34                           งจฉ: 30
                                                          1

           0        1                      0          จฉ: 15
                                                     0           1

 ก: 36   ข: 17     ค: 17            ง: 15         จ: 10 ฉ: 5
  0      100        101              110           1110          1111
                   Chapter 3: Greedy Algorithms www.eusman.com
Theorem
 A full binary tree for an optimal prefix code for C letters
  has exactly C leaves, and exactly C−1 internal nodes.
Proof by induction.
Basis:
  C=1. If there is one letter, the binary tree requires only 1
  leaf node, and 0 internal node.
Induction hypotheses:
  For C< n, the full binary tree for an optimal prefix code
  for C letters has exactly C leaves, and exactly C−1
  internal nodes.

                       Chapter 3: Greedy Algorithms www.eusman.com
Theorem
   Induction Step:
    Let T be a full binary tree for an optimal prefix code for
    C+1 letters.
    To create a full binary tree or optimal prefix code, we
    can take a full binary tree for an optimal prefix code for
    C letters, and add another leaf node L by either
       adding a new root node R and put L and the old root of T as its
        children, or
       replacing a leaf node N of T by an internal node and put L and
        N as its children.
    In either case, the number of leaf nodes of T is C+1 and
    the number of internal nodes is C.

                            Chapter 3: Greedy Algorithms www.eusman.com
Creating Huffman Code: Algorithms
HUFFMAN(C) Use min-heap for Q
 n = |C|        O(n) to build min-heap
 Q=C
 for i = 1 to n − 1                                  O(n lg n)
  do allocate a new node z
     z.left = x = EXTRACT-MIN(Q)
     z.right = y = EXTRACT-MIN(Q)
     z.f =x.f + y.f                                   O(lg n)
     INSERT(Q, z)
 ►Return the root of the tree.
 return EXTRACT-MIN(Q)
                     Chapter 3: Greedy Algorithms www.eusman.com
Greedy-choice property of Huffman Code
Let C be an alphabet in which each character c  C has
  frequency f [c], and
  x and y be two characters in C having the lowest
  frequencies.
 Then, there exists an optimal prefix code for C in which
  the codewords for x and y have the same length and
  differ only in the last bit.




                      Chapter 3: Greedy Algorithms www.eusman.com
Proof
   The idea of the proof is to :
       take the tree T representing an arbitrary optimal prefix code
       modify it to make a tree representing another optimal prefix
        code such that the characters x and y appear as sibling leaves of
        maximum depth in the new tree.
   If we can do this, then their codewords will have the
    same length and differ only in the last bit.




                             Chapter 3: Greedy Algorithms www.eusman.com
Proof
Let a and b be two characters that are sibling leaves of
  maximum depth in T .

Assume that f [a] ≤ f [b] and f [x] ≤ f [y].

Since f [x] and f [y] are the two lowest leaf frequencies, in
  order, and f [a] and f [b] are two arbitrary frequencies, in
  order, we have f [x] ≤ f [a] and f [y] ≤ f [b].




                        Chapter 3: Greedy Algorithms www.eusman.com
Proof
Exchange the positions of a and x in T to produce T’ .
   B(T) − B(T’ ) =  f(c) dT (c) −  f(c)dT’ (c)
                     cC                         cC

= f [x] dT (x) + f [a] dT (a) − f [x] dT’ (x) − f [a] dT’ (a)
= f [x] dT (x) + f [a] dT (a) − f [x] dT (a) − f [a] dT (x)
= ( f [a] − f [x])( dT (a) − dT (x)) ≥ 0

   T                                           T’
                         x                                                 a

        y                                            y
              a      b                                        x        b
                             Chapter 3: Greedy Algorithms www.eusman.com
Proof
 Then, exchange the positions of b
 and y in T’ to produce T’’.                       T’
                                                                       a
 Similarly, it does not increase the
 cost, and so B(T’ ) − B(T’’ )  0.     y
 Therefore, B(T’’ ) ≤ B(T).                                  x     b
 Since T is optimal, B(T) ≤ B(T’’ ).
 Then, B(T’’ ) = B(T).
                                       T’’
 Thus, T’’ is an optimal tree in which                                 a
 x and y appear as sibling leaves of
 maximum depth.                         b
                                                             x     y
                     Chapter 3: Greedy Algorithms www.eusman.com
Optimal-substructure Property
Let C be a given alphabet with frequency f [c] defined for
  each character c  C,
  x and y be two characters in C with minimum frequency,
  C’ be the alphabet C with characters x, y removed and
  (new) character z added, so that C’ = C − {x, y}  {z};
  define f for C’ as for C, except that f [z] = f [x] + f [y], and
  T be any tree representing an optimal prefix code for the
  alphabet C ’.
 Then the tree T , obtained from T ’ by replacing the leaf
  node for z with an internal node having x and y as
  children, represents an optimal prefix code for the
  alphabet C.
                         Chapter 3: Greedy Algorithms www.eusman.com
Proof
Show that the cost B(T) can be expressed in terms of the
  cost B(T’) by considering the component costs.
For each c  C − {x, y}, we have
      dT (c) = dT ’ (c).
      f [c] dT (c) = f [c] dT ’ (c).
Since dT (x) = dT (y) = dT ’(z) + 1, we have
  f [x]dT (x) + f [y]dT (y) = ( f [x] + f [y])( dT’ (z) + 1)
                            = f [z] dT ’ (z) + ( f [x] + f [y])
Thus, B(T) = B(T ’) + f [x] + f [y].
That is, B(T ’) = B(T) − f [x] − f [y] .

                                Chapter 3: Greedy Algorithms www.eusman.com
Proof
We now prove by contradiction.

Suppose T does not represent an optimal prefix code for C.

Then there exists a tree T ’’ such that B(T ’’) < B(T).

Without loss of generality, T ’’ has x and y as siblings.




                        Chapter 3: Greedy Algorithms www.eusman.com
Proof
 Let T ’’’ be the tree T ’’ with the common parent of x and
 y replaced by a leaf z with frequency f [z] = f [x] + f [y].

 Then, B(T ’’’) = B(T ’’) − f [x] − f [y]
                < B(T) − f [x] − f [y]                    =     B(T ’).

 We reach a contradiction to the assumption that T ’
 represents an optimal prefix code for C’.

 Thus, T must represent an optimal prefix code for the
 alphabet C.

                      Chapter 3: Greedy Algorithms www.eusman.com
Interval Scheduling
Problem definition
 Let S be {a1, a2, . . . , an} of n proposed activities that
  wish to use the same resource.
 Only one activity can use the resource at a time.
 Each activity ai has a start time si and a finish time fi,
  where 0 ≤ si < fi < ∞.
 If selected, activity ai takes place during the half-open
  time interval [si , fi ).
 Activities ai and aj are compatible if the intervals [si , fi )
  and [s j , f j ) do not overlap .
 The activity-selection problem is to select a largest
  subset of mutually compatible activities.
                         Chapter 3: Greedy Algorithms www.eusman.com
Example



S



A1
A2
A3


          Chapter 3: Greedy Algorithms www.eusman.com
Subproblems
   Si j = {ak  S : fi ≤ sk < fk ≤ s j }
       the subset of activities in S that can start after activity ai
        finishes and finish before activity aj starts.




   Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞.
   Then S = S0,n+1, and 0 ≤ i, j ≤ n + 1.
                              Chapter 3: Greedy Algorithms www.eusman.com
Optimal Solution
 Let Ai j be an optimal solution to Si j .
 Let c[i, j ] be the number of activities in a maximum-size
  subset of mutually compatible activities in Si j.
       c[i, j ] = 0 for i ≥ j (i.e. Si j = )
                                               ak
   c[i, j ] = c[i, k] + c[k, j ] + 1          if ak be an activity in Ai j.
Then,
 c[i, j ] =  0                                                    if Si j = 
             max {c[i, k] + c[k, j ] + 1}                          if Si j  
                 i<k<j
                          akSi j
                                Chapter 3: Greedy Algorithms www.eusman.com
Theorem
 Consider any nonempty subproblem Si j , and let am be
  the activity in Si j with the earliest finish time:
             fm = min { fk : ak ∈ Si j } .
Then,
 Activity am is used in some maximum-size subset of
  mutually compatible activities of Si j .
 The subproblem Sim is empty, so that choosing am leaves
  the subproblem Smj as the only one that may be
  nonempty.


                      Chapter 3: Greedy Algorithms www.eusman.com
Proof: First part
 Let Ai j be a maximum-size subset of mutually
  compatible activities of Si j , and Ai j is sorted in
  monotonically increasing order of finish time.
 Let ak be the first activity in Ai j.
 If ak = am, am is used in some maximum-size subset of
  mutually compatible activities of Si j.
 If ak  am, we construct the subset A’i j = Ai j − {ak} 
  {am}.
           S

           Aij ak
           A’ij     am

                         Chapter 3: Greedy Algorithms www.eusman.com
Proof: First part
 The activities in A’i j are disjoint, since the activities in Aij
  are, ak is the first activity in Ai j to finish, and fm ≤ fk .
 Noting that A’i j has the same number of activities as Ai j ,
  we see that A’i j is a maximum-size subset of mutually
  compatible activities of Si j that includes am.

           S

           Aij ak
           A’ij      am



                          Chapter 3: Greedy Algorithms www.eusman.com
Proof: Second part
Let Sim be nonempty.
 Then, there is an activity ak such that fi ≤ sk <fk ≤ sm< fm.
 Then, ak is also in Si j and it has an earlier finish time than
  am, which contradicts our choice of am.
 We conclude that Sim is empty.

         ai

                       am
              S
                  ak
                                                                          aj

                            Chapter 3: Greedy Algorithms www.eusman.com
Greedy Solution


 Ai j = {ak}  Akj

 where ak is the activity which finishes earliest
 among activities in Si j.




                     Chapter 3: Greedy Algorithms www.eusman.com
Recursive Algorithms
ACTIVITY-SELECTOR(s, f, i, n)
  m=i+1
  while m ≤ n and s[m] < f [i]
     ► Find the first activity in Si,n+1.
     do     m= m+1
            if m ≤ n
               then return {am} 
                               ACTIVITY-SELECTOR(s, f,m, n)
               else return 

                       Chapter 3: Greedy Algorithms www.eusman.com
Iterative Algorithm
GREEDY-ACTIVITY-SELECTOR(s, f )
  n = length[s]
  A = {a1}
  i=1
  for m = 2 to n
   do if s[m ] ≥ f[i]
       then A = A  {am}
             i=m
  return A

                    Chapter 3: Greedy Algorithms www.eusman.com

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:8/18/2012
language:English
pages:57