# Greedy Algorithms - eusman.com

Document Sample

```					Greedy Algorithms
Greed is good.
(Some of the time)
Outline
   Elements of greedy algorithm
   Greedy choice property
   Optimal substructures
   Minimum spanning tree
   Kruskal’s algorithm
   Prim’s algorithm
 Huffman code
 Activity selection

Chapter 3: Greedy Algorithms www.eusman.com
Introduction
   Concepts
   Choosing the best possible choice at each step.
   This decision leads to the best over all solution.

   Greedy algorithms do not always yield
optimal solutions.

Chapter 3: Greedy Algorithms www.eusman.com
Elements of
Greedy Algorithms
Greedy-choice property
   A globally optimal solution is derived from a
locally optimal (greedy) choice.

   When choices are considered, the choice that
looks best in the current problem is chosen,
without considering results from
subproblems.

Chapter 3: Greedy Algorithms www.eusman.com
Optimal substructures
   A problem has optimal substructure if an
optimal solution to the problem is composed
of optimal solutions to subproblems.

   This property is important for both greedy
algorithms and dynamic programming.

Chapter 3: Greedy Algorithms www.eusman.com
Greedy Algorithm v.s. Dynamic Programming
Greedy algorithm                    Dynamic programming
 The best choice is made at         A choice is made at each
each step and after that the        step.
subproblem is solved.
 The choice made by a                 The choice made at each
greedy algorithm may                  step usually depends on the
solutions to subproblems.
depend on choices so far,
but it cannot depend on any
future choices or on the
solutions to subproblems.
 A greedy strategy usually            Dynamic-programming
progresses in a top-down              problems are often solved in
fashion, making one greedy            a bottom-up manner.
choice after another,
reducing each given problem
instance to a smaller one.

Chapter 3: Greedy Algorithms www.eusman.com
Steps in Design Greedy Algorithms
   Determine the optimal substructure of the problem.
   Develop a recursive solution.
   Prove that at any stage of the recursion, one of the
optimal choices is the greedy choice. Thus, it is
always safe to make the greedy choice.
   Show that all but one of the subproblems induced by
having made the greedy choice are empty.
   Develop a recursive algorithm that implements the
greedy strategy.
   Convert the recursive algorithm to an iterative
algorithm.
Chapter 3: Greedy Algorithms www.eusman.com
Shortcuts Design
   Form the optimization problem so that, after a
choice is made and there is only one subproblem left
to be solved.
   Prove that there is always an optimal solution to the
original problem that makes the greedy choice, so
that the greedy choice is always safe.

   Demonstrate that, having made the greedy choice,
what remains is a subproblem with the property that
if we combine an optimal solution to the subproblem
with the greedy choice we have made, we arrive at
an optimal solution to the original problem.

Chapter 3: Greedy Algorithms www.eusman.com
Minimum Spanning Tree
Definitions
   Let G = (V, E) be an undirected graph.
T is a minimum spanning tree of G if T ⊆ E is an
acyclic subset that connects all of the vertices and whose
total weight w(T ) =  w(u, v) is minimized.
(u,v)∈T

   Let A be a subset of some minimum spanning tree.
An edge (u, v) is a safe edge for A if A  {(u, v)} is also
a subset of a minimum spanning tree.

Chapter 3: Greedy Algorithms www.eusman.com
Definitions
   Let G = (V, E) be an undirected graph.
A cut (S, V − S) of G is a partition of V.
An edge (u, v)  E crosses the cut (S, V − S) if one of its
endpoints is in S and the other is in V − S.
A cut respects a set A of edges if no edge in A crosses the
cut.
An edge is a light edge crossing a cut if its weight is the
minimum of any edge crossing the cut.

Chapter 3: Greedy Algorithms www.eusman.com
Theorem
Let G = (V, E) be a connected,                                          S
undirected graph with a real-valued
weight function w defined on E,                                                     A
u
A be a subset of E that is included in                                          x

light edge
some minimum spanning tree for G,
(S, V − S) be any cut of G that
respects A.
v
y
If (u, v) is a light edge crossing (S,
V − S), then edge (u, v) is safe for                                      V-S
A.
Chapter 3: Greedy Algorithms www.eusman.com
Proof
Let T be a minimum spanning tree                                   S
that includes A.
If (u,v) is in T, then (u,v) is safe for A.                                    A
u
x
If (u,v) is not in T, then T contains an

light edge
edge (x,y) crossing S and V-S.
T
Since both (u,v) and (x,y) cross S and
v
V-S, there is a cycle for edges in T                                     y
 (u,v).
Another spanning tree T’ can be                                      V-S
created from T.
Chapter 3: Greedy Algorithms www.eusman.com
Proof
S                                             S

A                                         A
u                                               u
x                                         x

light edge
light edge

T                                         T’
v                                               v
y                                          y

V-S                                            V-S

Remove ’edge (x,y)create T.
T (u,v) -> from cycle
Add edgeis created -> cutathe cycle
Chapter 3: Greedy Algorithms www.eusman.com
Proof
Thus, T’ is a spanning tree that includes A.

Next, we need to show that T’ is a minimum spanning tree.
From the construction of T’ , T’ = T-{(x,y)}  {(u,v)}.
Thus, w(T’ ) = w(T) - w(x,y) + w(u,v).
Since (u,v) is a light edge cross S and V-S, w(u,v)  w(x,y).
Thus, w(T’ )  w(T).
Since T is a minimum spanning tree, w(T’ ) = w(T).
Then, T’ is also a minimum spanning tree .
As a result, (u, v) is safe for A.
Chapter 3: Greedy Algorithms www.eusman.com
Corollary
Let G = (V, E) be a connected, undirected graph with a real-
valued weight function w defined on E,
A be a subset of E that is included in some minimum
spanning tree for G, and
C = (VC, EC) be a connected component (tree) in the
forest GA = (V, A).
If (u, v) is a light edge connecting C to another tree in GA,
then (u, v) is safe for A.
Proof
(VC, V−VC) respects A, and (u, v) is a light edge for this cut.
Therefore, (u, v) is safe for A.
Chapter 3: Greedy Algorithms www.eusman.com
Kruskal’s Algorithm
   Concept
   Build a forest of minimum spanning trees.
   Repeatedly connect the trees to create a subset of a
minimum spanning tree, until all nodes are covered.
   In connecting two trees, choose the edge of the least
weight.

   Let C1 and C2 denote the two trees that are
connected by (u, v). Since (u, v) must be a light
edge connecting C1 to some other tree, we need to
prove that (u, v) is a safe edge for C1.
   Kruskal’s algorithm is a greedy algorithm, because
at each step it adds to the forest an edge of least
possible weight.

Chapter 3: Greedy Algorithms www.eusman.com
Example of Kruskal’s Algorithm

12                          8

2        9                1            9

5                       3                                11

8            7              3               5

1                           10

Chapter 3: Greedy Algorithms www.eusman.com
Kruskal’s Algorithm
MST-KRUSKAL(G,w)
A=
for each vertex v in V [G]
do     MAKE-SET(v)
sort the edges of E in nondecreasing order by weight w
for each edge (u, v) in E, taken in nondecreasing order
by weight
do if FIND-SET(u)  FIND-SET(v)
then A = A  {(u, v)}
UNION(u, v)
return A
Chapter 3: Greedy Algorithms www.eusman.com
Kruskal’s Algorithm: Complexity
MST-KRUSKAL(G,w)
A=
for each vertex v in V [G] O(v)
do     MAKE-SET(v)                        O(e lg e)
sort the edges of E in nondecreasing order by weight w
for each edge (u, v) in E, taken in nondecreasing order
by weight
do if FIND-SET(u)  FIND-SET(v)
then A = A  {(u, v)}
UNION(u, v)         O((v+e) lg v)
return A
e  v ; O(e lg v)
2
Chapter 3: Greedy Algorithms www.eusman.com
Prim’s Algorithm
   Prim’s algorithm has the property that the edges in the
set A always form a single tree.
 The tree starts from an arbitrary root vertex r.
 Grow the tree until it spans all the vertices in V.
   At each step, a light edge is added to the tree A that connects A
to an isolated vertex of GA = (V, A).

Chapter 3: Greedy Algorithms www.eusman.com
Example of Prim’s Algorithm

12                          8

2        9                1            9

5                       3                                11

8            7              3               5

1                           10

Chapter 3: Greedy Algorithms www.eusman.com
Prim’s Algorithm
PRIM(G,w, r)
for each u in V[G]
do key[u] = ∞
π[u] = NIL
key[r] = 0
Q = V[G]
while Q  
do      u = EXTRACT-MIN(Q)
for each v in Adj[u]
do if v in Q and w(u, v) < key[v]
then π[v] = u
key[v] = w(u, v)

Chapter 3: Greedy Algorithms www.eusman.com
Execution of Prim’s Algorithm

12                          8
0                       9
3
12                             8

2        9                1            9

5            2           3               1
3                11

8            7              3               5

5           1            7
1              10            5
10

Chapter 3: Greedy Algorithms www.eusman.com
Prim’s Algorithm:Complexity
PRIM(G,w, r)
for each u in V[G] Also build min-heap
do key[u] = ∞
O(v)
π[u] = NIL
key[r] = 0
Q = V[G]                                        O(e                lg v)
while Q  
do      u = EXTRACT-MIN(Q) O(lg v) O(v                         lg v)
for each v in Adj[u]
do if v in Q and w(u, v) < key[v]
then π[v] = u
key[v] = w(u, v) O(lg v)

Chapter 3: Greedy Algorithms www.eusman.com
Huffman Code
Problem
   Find an optimal prefix code representing a
set of characters in a file, where each
character has a frequency of occurrences.
   Prefix code
   Codes in which no codeword is also a prefix of
some other codeword.
   {0, 110, 101, 111, 1000, 1001} is a prefix code.
   {0, 110, 101, 111, 1010, 0111} is not prefix code
   Optimality
   The code yields a file with the minimum number
of bits.

Chapter 3: Greedy Algorithms www.eusman.com
Creating Huffman Code: Example
100
1

64
0                                     1
0

34                                   30
1

0         1                      0             15

0           1

ก: 36   ข: 17        ค: 17            ง: 15         จ: 10 ฉ: 5
0      100           101              110          1110        1111

Chapter 3: Greedy Algorithms www.eusman.com
Optimal Code
   An optimal code for a file is always
represented by a full binary tree, in which
every nonleaf node has two children.

   If C is the alphabet from which the
characters are drawn and all character
frequencies are positive, then the tree for an
optimal prefix code has exactly |C| leaves,
one for each letter of the alphabet, and
exactly |C|−1 internal nodes.

Chapter 3: Greedy Algorithms www.eusman.com
Full Binary Trees for Prefix Code
Tree for 1 letter           Tree for 2 letters
A                        Tree for 4 letters
1
1          2
B
Tree for 3 letters
C                   A
B                                      4             3       1           2
3         A                                            C
1           2
4               B
A                                                3           A
1           B                                                    1           2
2       3

Chapter 3: Greedy Algorithms www.eusman.com
Full Binary Trees for Prefix Code
Tree for 4 letters                                 Tree for 5 letters

B                                                    B
C               A                                  C                    A
4       3       1        2                       D           3       1          2
5           4

C
D
4           B
5              B
3       A
C                    A
1        2
4          3       1          2

Chapter 3: Greedy Algorithms www.eusman.com
Creating Huffman Code: Example
กขคงจฉ: 100

1

ขคงจฉ: 64
0       0                        1

ขค: 34                           งจฉ: 30
1

0        1                      0          จฉ: 15
0           1

ก: 36   ข: 17     ค: 17            ง: 15         จ: 10 ฉ: 5
0      100        101              110           1110          1111
Chapter 3: Greedy Algorithms www.eusman.com
Theorem
 A full binary tree for an optimal prefix code for C letters
has exactly C leaves, and exactly C−1 internal nodes.
Proof by induction.
Basis:
C=1. If there is one letter, the binary tree requires only 1
leaf node, and 0 internal node.
Induction hypotheses:
For C< n, the full binary tree for an optimal prefix code
for C letters has exactly C leaves, and exactly C−1
internal nodes.

Chapter 3: Greedy Algorithms www.eusman.com
Theorem
   Induction Step:
Let T be a full binary tree for an optimal prefix code for
C+1 letters.
To create a full binary tree or optimal prefix code, we
can take a full binary tree for an optimal prefix code for
C letters, and add another leaf node L by either
   adding a new root node R and put L and the old root of T as its
children, or
   replacing a leaf node N of T by an internal node and put L and
N as its children.
In either case, the number of leaf nodes of T is C+1 and
the number of internal nodes is C.

Chapter 3: Greedy Algorithms www.eusman.com
Creating Huffman Code: Algorithms
HUFFMAN(C) Use min-heap for Q
n = |C|        O(n) to build min-heap
Q=C
for i = 1 to n − 1                                  O(n lg n)
do allocate a new node z
z.left = x = EXTRACT-MIN(Q)
z.right = y = EXTRACT-MIN(Q)
z.f =x.f + y.f                                   O(lg n)
INSERT(Q, z)
►Return the root of the tree.
return EXTRACT-MIN(Q)
Chapter 3: Greedy Algorithms www.eusman.com
Greedy-choice property of Huffman Code
Let C be an alphabet in which each character c  C has
frequency f [c], and
x and y be two characters in C having the lowest
frequencies.
 Then, there exists an optimal prefix code for C in which
the codewords for x and y have the same length and
differ only in the last bit.

Chapter 3: Greedy Algorithms www.eusman.com
Proof
   The idea of the proof is to :
   take the tree T representing an arbitrary optimal prefix code
   modify it to make a tree representing another optimal prefix
code such that the characters x and y appear as sibling leaves of
maximum depth in the new tree.
   If we can do this, then their codewords will have the
same length and differ only in the last bit.

Chapter 3: Greedy Algorithms www.eusman.com
Proof
Let a and b be two characters that are sibling leaves of
maximum depth in T .

Assume that f [a] ≤ f [b] and f [x] ≤ f [y].

Since f [x] and f [y] are the two lowest leaf frequencies, in
order, and f [a] and f [b] are two arbitrary frequencies, in
order, we have f [x] ≤ f [a] and f [y] ≤ f [b].

Chapter 3: Greedy Algorithms www.eusman.com
Proof
Exchange the positions of a and x in T to produce T’ .
B(T) − B(T’ ) =  f(c) dT (c) −  f(c)dT’ (c)
cC                         cC

= f [x] dT (x) + f [a] dT (a) − f [x] dT’ (x) − f [a] dT’ (a)
= f [x] dT (x) + f [a] dT (a) − f [x] dT (a) − f [a] dT (x)
= ( f [a] − f [x])( dT (a) − dT (x)) ≥ 0

T                                           T’
x                                                 a

y                                            y
a      b                                        x        b
Chapter 3: Greedy Algorithms www.eusman.com
Proof
Then, exchange the positions of b
and y in T’ to produce T’’.                       T’
a
Similarly, it does not increase the
cost, and so B(T’ ) − B(T’’ )  0.     y
Therefore, B(T’’ ) ≤ B(T).                                  x     b
Since T is optimal, B(T) ≤ B(T’’ ).
Then, B(T’’ ) = B(T).
T’’
Thus, T’’ is an optimal tree in which                                 a
x and y appear as sibling leaves of
maximum depth.                         b
x     y
Chapter 3: Greedy Algorithms www.eusman.com
Optimal-substructure Property
Let C be a given alphabet with frequency f [c] defined for
each character c  C,
x and y be two characters in C with minimum frequency,
C’ be the alphabet C with characters x, y removed and
(new) character z added, so that C’ = C − {x, y}  {z};
define f for C’ as for C, except that f [z] = f [x] + f [y], and
T be any tree representing an optimal prefix code for the
alphabet C ’.
 Then the tree T , obtained from T ’ by replacing the leaf
node for z with an internal node having x and y as
children, represents an optimal prefix code for the
alphabet C.
Chapter 3: Greedy Algorithms www.eusman.com
Proof
Show that the cost B(T) can be expressed in terms of the
cost B(T’) by considering the component costs.
For each c  C − {x, y}, we have
   dT (c) = dT ’ (c).
   f [c] dT (c) = f [c] dT ’ (c).
Since dT (x) = dT (y) = dT ’(z) + 1, we have
f [x]dT (x) + f [y]dT (y) = ( f [x] + f [y])( dT’ (z) + 1)
= f [z] dT ’ (z) + ( f [x] + f [y])
Thus, B(T) = B(T ’) + f [x] + f [y].
That is, B(T ’) = B(T) − f [x] − f [y] .

Chapter 3: Greedy Algorithms www.eusman.com
Proof
We now prove by contradiction.

Suppose T does not represent an optimal prefix code for C.

Then there exists a tree T ’’ such that B(T ’’) < B(T).

Without loss of generality, T ’’ has x and y as siblings.

Chapter 3: Greedy Algorithms www.eusman.com
Proof
Let T ’’’ be the tree T ’’ with the common parent of x and
y replaced by a leaf z with frequency f [z] = f [x] + f [y].

Then, B(T ’’’) = B(T ’’) − f [x] − f [y]
< B(T) − f [x] − f [y]                    =     B(T ’).

We reach a contradiction to the assumption that T ’
represents an optimal prefix code for C’.

Thus, T must represent an optimal prefix code for the
alphabet C.

Chapter 3: Greedy Algorithms www.eusman.com
Interval Scheduling
Problem definition
 Let S be {a1, a2, . . . , an} of n proposed activities that
wish to use the same resource.
 Only one activity can use the resource at a time.
 Each activity ai has a start time si and a finish time fi,
where 0 ≤ si < fi < ∞.
 If selected, activity ai takes place during the half-open
time interval [si , fi ).
 Activities ai and aj are compatible if the intervals [si , fi )
and [s j , f j ) do not overlap .
 The activity-selection problem is to select a largest
subset of mutually compatible activities.
Chapter 3: Greedy Algorithms www.eusman.com
Example

S

A1
A2
A3

Chapter 3: Greedy Algorithms www.eusman.com
Subproblems
   Si j = {ak  S : fi ≤ sk < fk ≤ s j }
   the subset of activities in S that can start after activity ai
finishes and finish before activity aj starts.

   Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞.
   Then S = S0,n+1, and 0 ≤ i, j ≤ n + 1.
Chapter 3: Greedy Algorithms www.eusman.com
Optimal Solution
 Let Ai j be an optimal solution to Si j .
 Let c[i, j ] be the number of activities in a maximum-size
subset of mutually compatible activities in Si j.
   c[i, j ] = 0 for i ≥ j (i.e. Si j = )
ak
   c[i, j ] = c[i, k] + c[k, j ] + 1          if ak be an activity in Ai j.
Then,
 c[i, j ] =  0                                                    if Si j = 
max {c[i, k] + c[k, j ] + 1}                          if Si j  
    i<k<j
akSi j
Chapter 3: Greedy Algorithms www.eusman.com
Theorem
 Consider any nonempty subproblem Si j , and let am be
the activity in Si j with the earliest finish time:
fm = min { fk : ak ∈ Si j } .
Then,
 Activity am is used in some maximum-size subset of
mutually compatible activities of Si j .
 The subproblem Sim is empty, so that choosing am leaves
the subproblem Smj as the only one that may be
nonempty.

Chapter 3: Greedy Algorithms www.eusman.com
Proof: First part
 Let Ai j be a maximum-size subset of mutually
compatible activities of Si j , and Ai j is sorted in
monotonically increasing order of finish time.
 Let ak be the first activity in Ai j.
 If ak = am, am is used in some maximum-size subset of
mutually compatible activities of Si j.
 If ak  am, we construct the subset A’i j = Ai j − {ak} 
{am}.
S

Aij ak
A’ij     am

Chapter 3: Greedy Algorithms www.eusman.com
Proof: First part
 The activities in A’i j are disjoint, since the activities in Aij
are, ak is the first activity in Ai j to finish, and fm ≤ fk .
 Noting that A’i j has the same number of activities as Ai j ,
we see that A’i j is a maximum-size subset of mutually
compatible activities of Si j that includes am.

S

Aij ak
A’ij      am

Chapter 3: Greedy Algorithms www.eusman.com
Proof: Second part
Let Sim be nonempty.
 Then, there is an activity ak such that fi ≤ sk <fk ≤ sm< fm.
 Then, ak is also in Si j and it has an earlier finish time than
am, which contradicts our choice of am.
 We conclude that Sim is empty.

ai

am
S
ak
aj

Chapter 3: Greedy Algorithms www.eusman.com
Greedy Solution

Ai j = {ak}  Akj

where ak is the activity which finishes earliest
among activities in Si j.

Chapter 3: Greedy Algorithms www.eusman.com
Recursive Algorithms
ACTIVITY-SELECTOR(s, f, i, n)
m=i+1
while m ≤ n and s[m] < f [i]
► Find the first activity in Si,n+1.
do     m= m+1
if m ≤ n
then return {am} 
ACTIVITY-SELECTOR(s, f,m, n)
else return 

Chapter 3: Greedy Algorithms www.eusman.com
Iterative Algorithm
GREEDY-ACTIVITY-SELECTOR(s, f )
n = length[s]
A = {a1}
i=1
for m = 2 to n
do if s[m ] ≥ f[i]
then A = A  {am}
i=m
return A

Chapter 3: Greedy Algorithms www.eusman.com

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 5 posted: 8/18/2012 language: English pages: 57