Greedy Algorithms
COSC 3101N J. Elder
Greedy Algorithms
Every two year old knows the greedy algorithm.
In order to get what you want,
just start grabbing what looks best.
Surprisingly, many important and
practical computational problems
can be solved this way.
COSC 3101N J. Elder
Optimization Problems
Ingredients:
•Instances: The possible inputs to the problem.
•Solutions for Instance: Each instance has an exponentially large
set of solutions.
•Cost of Solution: Each solution has an easy-to-compute cost or
value.
Specification
•Preconditions: The input is one instance.
•Postconditions: A valid solution with optimal cost. (minimum
or maximum)
COSC 3101N J. Elder
Example 1: Making Change
Problem: Find the minimum # of quarters, dimes,
nickels, and pennies that total to a given amount.
COSC 3101N J. Elder
The Brute Force Algorithm
Try every solution!
Exponential Time,
because exponential number
of possible solutions
COSC 3101N J. Elder
The Greedy Choice
Commit to the object that looks the ``best''
Must prove that this locally greedy choice
does not have negative global consequences.
COSC 3101N J. Elder
Making Change Example
Instances: A set of objects
and a relationship between them.
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Solutions for Instance:
A subset of the coins that total the amount.
Some subsets are not allowed
because some objects conflict
COSC 3101N J. Elder
Making Change Example
Instances: A set of objects and a relationship between them.
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Solutions for Instance: A subset of the coins that total the amount.
Cost of Solution: The number of objects in solution or the sum of
the costs of objects = 14
Goal: Find an optimal non-conflicting solution.
COSC 3101N J. Elder
Making Change Example
Instances: A set of objects
and a relationship between them.
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Greedy Choice:
Start by grabbing quarters until exceeds amount,
then dimes, then nickels, then pennies.
Does this lead to an optimal # of coins?
Cost of Solution: 7
COSC 3101N J. Elder
Hard Making Change Example
Problem: Find the minimum # of
4, 3, and 1 cent coins to make up 6 cents.
Greedy Choice: Start by grabbing a 4 coin.
Consequences:
4+1+1 = 6 mistake
3+3=6 better
Greedy Algorithm does not work!
COSC 3101N J. Elder
When Does It Work?
Greedy Algorithms: Easy to understand and to code, but
do they work?
For most optimization problems,
all greedy algorithms tried do not work (i.e. yield sub-
optimal solutions)
But a few problems can be solved optimally by a greedy
algorithm.
The proof that they work, however, is subtle.
As with all iterative algorithms,
we use loop invariants.
COSC 3101N J. Elder
Designing an Algorithm
Define Problem Define Loop Define Measure of
Invariants Progress
79 km
to school
Define Step Define Exit Condition Maintain Loop Inv
Exit
Exit
Make Progress Initial Conditions Ending
Exit
79 km 75 km km 0 km Exit Exit
COSC 3101N J. Elder
Define Step
The algorithm chooses the “best” object
from amongst those not considered so far
and either commits to it or rejects it.
Make Progress
Exit
79 km 75 km Another object considered
Exit Condition
Exit When all objects have been considered
COSC 3101N J. Elder
Loop Invariant
We have not gone wrong.
There is at least one optimal solution
consistent with the choices made so far.
Establishing Loop Invariant
codeA
Initially no choices have been made and
hence all optimal solutions are consistent
with these choices.
COSC 3101N J. Elder
Exit Maintaining Loop Invariant
¬
codeB
$ opt sol consistent with choices made.
Let optSLI denote one.
codeB Commits or rejects the next best object.
? $ opt sol consistent with all choices.
Proof massages optSLI into optSours and proves it
•is a valid solution
•is consistent both with previous and new choices.
• is optimal
COSC 3101N J. Elder
Three Players
optSLI
Algorithm: Prover: Fairy God Mother:
commits or Proves LI is Holds the hypothetical
rejects next maintained. opt sol optSLI.
best object Emphasizes his
Emphasizes alg
actions not part and prover do not
of algorithm know optSLI.
COSC 3101N
She gives no feedback.
J. Elder
Massaging optSLI into optSours
Amount = 92¢ ¢ ¢
25¢ 25¢ 25¢ 25¢ 25¢ 25 25¢ 25¢ 25¢ 25¢
¢ ¢ ¢
10¢ 10¢ 10¢ 10¢ 10 ¢ 10¢ 10 10 10 10
5 ¢ 5 ¢ 5 ¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1
¢ 1¢ 1¢ 1¢ 1¢
optSLI
hold optSLI witnessing that
IIhold optSourswitnessing that
there is an opt sol consistent
with previous choices.
with previous & new choices.
I commit to keeping
have committed to
another 25¢
these coins.
I instruct how to massage
optSLI into optSours so that it
is consistent with previous &
COSC 3101N new choice. J. Elder
As Time Goes On
Amount = 92¢ ¢ ¢
25¢ 25¢ 25¢ 25¢ 25¢ 25 25¢ 25¢ 25¢ 25¢
¢ ¢ ¢
10¢ 10¢ 10¢ 10¢ 10 ¢ 10¢ 10 10 10 10
5 ¢ 5 ¢ 5 ¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1
¢ 1¢ 1¢ 1¢ 1¢
optSLI
I always hold an opt sol optSLI
but which one keeps changing.
I keep making more
choices.
I know that her optSLI
is consistent with these choices.
Hence, I know more and more of optSLI
COSC 3101N In the end, I know it all. J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
I will now instruct how to
massage optSLI into optSours so
that it is consistent with
previous & new choice.
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
If it happens to be the case
that what you hold is
consistent with this new
choice that was made, then we
are done.
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
The Algorithm has
92¢-50¢ = 42¢ 25¢ unchosen.
Fairy God Mother must also
have 25¢ that I don’t know about.
There are different cases.
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Replace With
•A different 25¢ •Alg’s 25¢
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Replace With
•A different 25¢ •Alg’s 25¢
•3×10¢ •Alg’s 25¢ + 5¢
Oops, this is not actually optimal,
But we must consider all cases.
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Replace With
•A different 25¢ •Alg’s 25¢
•3×10¢ •Alg’s 25¢ + 5¢
•2×10¢ + 1×5¢ •Alg’s 25¢
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Replace With
•A different 25¢ •Alg’s 25¢
•3×10¢ •Alg’s 25¢ + 5¢
•2×10¢ + 1×5¢ •Alg’s 25¢
•1×10¢ + 3×5¢ •Alg’s 25¢
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
Replace With
•A different 25¢ •Alg’s 25¢
•3×10¢ •Alg’s 25¢ + 5¢
•2×10¢ + 1×5¢ •Alg’s 25¢
•1×10¢ + 3×5¢ •Alg’s 25¢
•?? + 5×1¢ •Alg’s 25¢
COSC 3101N J. Elder
Massaging optSLI into optSours
Done
optSLI
optSours
She now has something
we must prove that it is
what we want.
COSC 3101N J. Elder
Massaging optSLI into optSours
optSours is valid optSours
optSLI was valid and we
introduced no new conflicts.
Total remains unchanged.
Replace With
•A different 25¢ •Alg’s 25¢
•3×10¢ •Alg’s 25¢ + 5¢
•2×10¢ + 1×5¢ •Alg’s 25¢
•1×10¢ + 3×5¢ •Alg’s 25¢
COSC 3101N
•?? + 5×1¢ J. Elder
Massaging optSLI into optSours
optSours is consistent optSours
optSLI was consistent with
previous choices and we made
it consistent with new.
Amount = 92¢ ¢ ¢
25¢ 25¢ 25¢ 25¢ 25¢ 25 25¢ 25¢ 25¢ 25¢
¢ ¢ ¢
10¢ 10¢ 10¢ 10¢ 10 ¢ 10¢ 10 10 10 10
5 ¢ 5 ¢ 5 ¢ 5 ¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1
¢ 1¢ 1¢ 1¢ 1¢
COSC 3101N J. Elder
Massaging optSLI into optSours
optSours is optimal
We do not even know the optSours
cost of an optimal solution.
optSLI was optimal and
optSours cost (# of coins) is not bigger.
Replace With
•A different 25¢ •Alg’s 25¢
•3×10¢ •Alg’s 25¢ + 5¢
•2×10¢ + 1×5¢ •Alg’s 25¢
•1×10¢ + 3×5¢ •Alg’s 25¢
COSC 3101N
•?? + 5×1¢ J. Elder
Massaging optSLI into optSours Case 1
optSours is valid optSours
optSours is consistent
optSours is optimal
optSours
Exit Maintaining Loop Invariant
¬
COSC 3101N codeB J. Elder
Massaging optSLI into optSours
Amount = 92¢ Case 2
¢ ¢
25¢ 25¢ 25¢ 25¢ 25¢ 25 25¢ 25¢ 25¢ 25¢
¢ ¢ ¢
10¢ 10¢ 10¢ 10¢ 10 ¢ 10¢ 10 10 10 10
5¢ 5 ¢ 5 ¢ 5¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1
¢ 1¢ 1¢ 1¢ 1¢
I hold optSLI witnessing that optSLI
there is an opt sol consistent
with previous choices.
I reject the next 25¢
I must make sure that what the Fairy
God Mother has is consistent with this
new choice.
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 92¢
25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ 25¢ optSLI
10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢ 10¢
5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢
The Algorithm has
92¢-75¢ = 17¢
¬
codeB
COSC 3101N J. Elder
As Time Goes On
Amount = 92¢ ¢ ¢
25¢ 25¢ 25¢ 25¢ 25¢ 25 25¢ 25¢ 25¢ 25¢
¢ ¢ ¢
10¢ 10¢ 10¢ 10¢ 10 ¢ 10¢ 10 10 10 10
5 ¢ 5 ¢ 5 ¢ 5 ¢ 5¢ 5¢ 5¢ 5¢ 5¢ 5¢
1¢ 1¢ 1¢ 1¢ 1¢ 1
¢ 1¢ 1¢ 1¢ 1¢
optSLI
I always hold an opt sol optSLI
but which one keeps changing.
I keep making more
choices.
I know that her optSLI
is consistent with these choices.
Hence, I know more and more of optSLI
COSC 3101N In the end, I know it all J. Elder
Clean up loose ends
Exit
codeC
Alg commit to or reject each object.
Has yielded a solution S.
$ opt sol consistent with these choices.
S must be optimal.
codeC Alg returns S .
COSC 3101N J. Elder
Making Change Example
Problem: Find the minimum # of quarters, dimes,
nickels, and pennies that total to a given amount.
Greedy Choice: Start by grabbing quarters
until exceeds amount, then dimes, then
nickels, then pennies.
Does this lead to an optimal # of coins?
Yes
COSC 3101N J. Elder
Hard Making Change Example
Problem: Find the minimum # of
4, 3, and 1 cent coins to make up 6 cents.
Greedy Choice: Start by grabbing a 4 coin.
COSC 3101N J. Elder
Massaging optSLI into optSours
Amount = 6¢ ¢
4 ¢ 4¢ 4¢ 4¢ 4¢ 4¢ 4¢ 4¢ 4¢ 4
¢
3 ¢ 3¢ 3¢ 3¢ 3 ¢ 3¢ 3¢ 3¢ 3¢ 3 optSLI
¢
1 ¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1¢ 1
I hold optSLI.
I commit to keeping a 4¢
I will now instruct how to
massage optSLI into optSours so
that it is consistent with
previous & new choice. Oops!
COSC 3101N J. Elder
Hard Making Change Example
Problem: Find the minimum # of
4, 3, and 1 cent coins to make up 6 cents.
Greedy Choice: Start by grabbing a 4 coin.
Consequences:
4+1+1 = 6 mistake
3+3=6 better
Greedy Algorithm does not work!
COSC 3101N J. Elder
Running Time
Greedy algorithms are very fast because they
take only a small amount of time per object in
the instance.
COSC 3101N J. Elder
Example 2: The Job/Event
Scheduling Problem
Ingredients:
•Instances: Events with starting and finishing times
,,… ,>.
•Solutions: A set of events that do not overlap.
•Value of Solution: The number of events scheduled.
•Goal: Given a set of events, schedule as many as
possible.
COSC 3101N J. Elder
Possible Criteria for Defining “Best”
Optimal
Greedy Criteria: The Shortest Event
Motivation: Does not book the room
for a long period of time.
Schedule first
Optimal
Counter Example
COSC 3101N J. Elder
Possible Criteria for Defining “Best”
Optimal
Greedy Criteria: The Earliest Starting Time
Motivation: Common scheduling algorithm.
Schedule first
Optimal
Counter Example
COSC 3101N J. Elder
Possible Criteria for Defining “Best”
Optimal
Greedy Criteria:
Conflicting with the Fewest Other Events
Motivation: So many can still be scheduled.
Schedule first
Optimal
Counter Example
COSC 3101N J. Elder
Possible Criteria for Defining “Best”
Greedy Criteria: Earliest Finishing Time
Motivation: Schedule the event that will
free up your room for someone
else as soon as possible. Works!
COSC 3101N J. Elder
The Greedy Algorithm
COSC 3101N J. Elder
Massaging optSLI into optSours
optSLI
Start by adding new event i.
Delete events conflicting with job i.
COSC 3101N J. Elder
Massaging optSLI into optSours
optSLI
optSours is valid
optSLI was valid and we
removed any new conflicts.
COSC 3101N J. Elder
Massaging optSLI into optSours
optSours is consistent with our choices. optSLI
optSLI was consistent with our prior choices.
We added event i.
Events in Commit don’t conflict with event i
and hence were not deleted.
COSC 3101N J. Elder
Massaging optSLI into optSours
optSours is optimal optSLI
optSLI was optimal.
If we delete at most one event
then optSours is optimal too.
COSC 3101N J. Elder
Massaging optSLI into optSours
j
j’
Deleted at most one event j optSLI
i
Exit Maintaining Loop Invariant
¬
COSC 3101N codeB J. Elder
Massaging optSLI into optSours Case 2
I hold optSLI witnessing that
optSLI
there is an opt sol consistent
with previous choices.
I reject next event i.
Event i conflicts with committed to
events so can’t be in optSLI either.
COSC 3101N J. Elder
Massaging optSLI into optSours
optSLI
Exit Maintaining Loop Invariant
¬
codeB
COSC 3101N J. Elder
Clean up loose ends
Exit
codeC
Alg commit to or reject each event.
Has a solution S.
$ opt sol consistent with these choices.
S must be optimal.
codeC Alg returns optS .
COSC 3101N J. Elder
Running Time
Greedy algorithms are very fast because they
take only a small amount of time per object in
the instance.
Checking whether next event i conflicts with
previously committed events requires
only comparing it with the last such event.
COSC 3101N J. Elder
Recursive Back Tracking
COSC 3101N J. Elder
Optimization Problems
• An important and practical class of computational
problems.
• For most of these, the best known algorithm runs in
exponential time.
• Industry would pay dearly to have faster algorithms.
• Heuristics
• Some have quick Greedy or Dynamic Programming
algorithms
• For the rest, Recursive Back Tracking is the best option.
COSC 3101N J. Elder
Example: Search Graph For Best Path
We use it because it nicely demonstrates
the concepts in a graphical way.
COSC 3101N J. Elder
Search Graph For Best Path
An instance (input) consists of .
G is a weighted directed layered graph
s source node
t sink node
COSC 3101N J. Elder
Optimization Problems
Ingredients:
•Instances: The possible inputs to the problem.
Eg.
•Solutions for Instance: Each instance has an
exponentially large set of solutions.
Eg. Paths from s to t
•Cost of Solution: Each solution has an easy to
compute cost or value.
Eg. Weight of path
COSC 3101N J. Elder
Optimization Problems
Specification of an Optimization Problem
•Preconditions: The input is one instance.
•Postconditions:
The output is one of the valid solutions for this
instance with optimal cost.
(minimum or maximum)
Eg. A minimum weight path
•The solution might not be unique.
COSC 3101N J. Elder
Recursive Back Tracking
Suppose I am searching a maze
and come to a fork.
? ?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Suppose, I am searching a maze
and come to a fork.
Which way should I go?
? ?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
? ?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
? ?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
?
? ? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
?
? ?
COSC 3101N J. Elder
Recursive Back Tracking
Idea: Try all options and
keep the best!
?
? ?
COSC 3101N J. Elder
Help From Friends (Recursion)
Too much for me to keep track of.
? ?
? ? ?
COSC 3101N J. Elder
Help From Friends (Recursion)
Too much for me to keep track of.
For each option, a friend
gets the best answer for this
option.
COSC 3101N J. Elder
Help From Friends (Recursion)
I collect the info and determine
which option leads to best
answer.
This is my answer!
COSC 3101N J. Elder
Help From Friends (Recursion)
Don't worry about your what your
friend does
but he will have his own forks to
deal with.
COSC 3101N J. Elder
Best of Best
Suppose I am searching
for the best animal.
COSC 3101N J. Elder
Recursive Back Tracking
Depth-first search traversal of classification tree.
Some friend looks at each animal.
COSC 3101N J. Elder
Recursive Back Tracking
Time?
COSC 3101N J. Elder
Recursive Back Tracking
Time? Same as the brute force algorithm that
simply compares the animals.
COSC 3101N J. Elder
Speeding Up the Time
But sometimes we can exploit the structure to
speed up the algorithm.
COSC 3101N J. Elder
Speeding Up the Time
Sometimes entire an branch can be pruned off.
•Perhaps because these solutions are not valid
or not highly valued.
COSC 3101N J. Elder
Speeding Up the Time
Sometimes entire an branch can be pruned off.
•Or because there is at least one optimal
solution elsewhere in the tree.
COSC 3101N J. Elder
Speeding Up the Time
A Greedy algorithm prunes off all branches
except the one that looks best.
COSC 3101N J. Elder
Speeding Up the Time
Memoization
Definition: An algorithmic technique which saves (memoizes) a computed answer
for later reuse, rather than recomputing the answer.
•Memo functions were invented by Professor Donald Michie of Edinburgh
University. The idea was further developed by Robin Popplestone in his Pop2
language long before it was ever worked into LISP.
•This same principle is found at the hardware level in computer architectures which
use a cache to store recently accessed memory locations.
•["'Memo' functions: and machine learning", Donald Michie, Nature, 218, 19-22,
1968].
COSC 3101N J. Elder
Speeding Up the Time
Memoization in Optimization:
•Remembers the solutions for the subinstances
•So that if ever a sufficiently similar subinstance
needs to be solved, the same answer can be used.
•This effectively prunes off this later branch of the
classification tree.
COSC 3101N J. Elder
Fibonacci Example
Time?
COSC 3101N J. Elder
Fibonacci Example
Time:
Exponential
Waste time
redoing work
COSC 3101N J. Elder
Memoization
COSC 3101N J. Elder
Speeding Up the Time
Dynamic Programming
takes this idea one step further.
•The algorithm determines the set of subinstance
that need to be solved.
•The algorithm no longer traverses recursively
through the classification tree,
•No longer keeps track of which friends are
waiting for answers from which friends
•But simply solves each of the required
subinstance in smallest to largest order.
COSC 3101N J. Elder
Dynamic Programming
First determine the complete set of subinstances
{100, 99, 98,…, 0}
Compute them in an order
Smallest to largest
such that no friend must wait.
1
0
COSC 3101N J. Elder
Dynamic Programming
Fill out a table containing
an optimal solution for each subinstance.
0 1 1 2 3 5 2.19×1020 3.54×1020
0, 1, 2, 3, 4, 5, …. 99, 100
1
0
COSC 3101N J. Elder
Dynamic Programming
COSC 3101N J. Elder
The (General) 0-1 Knapsack Problem
0-1 knapsack problem:
• n items.
• Item i is worth $vi , weighs wi pounds.
• Find a most valuable subset of items with total weight ≤ W.
• Have to either take an item or not take it - can’t take part of
it.
Is there a greedy solution to this problem?
COSC 3101N J. Elder
What are good greedy local choices?
• Select most valuable object?
• Select smallest object?
• Select object most valuable by weight?
COSC 3101N J. Elder
Some example problem instances
Let W Capacity of knapsack = 10kg
Problem Instance 1: Problem Instance 2: Problem Instance 3:
v1 $60, w1 6kg v1 $60, w1 10kg v1 $60, w1 6kg
v2 $50, w2 5kg v2 $50, w2 9kg v2 $40, w2 5kg
v3 $50, w3 5kg v3 $40, w3 5kg
• Select most valuable object?
• Select smallest object? All Fail!
• Select object most valuable by weight?
COSC 3101N J. Elder
Simplified 0-1 Knapsack Problem
• The general 0-1 knapsack problem cannot
be solved by a greedy algorithm.
• What if we make the problem simpler:
Suppose vi wi
• Can this simplified knapsack problem be
solved by a greedy algorithm?
• No!
COSC 3101N J. Elder
Some example problem instances
Let W Capacity of knapsack = 10kg
Problem Instance 1: Problem Instance 2:
v1 w1 6 v1 w1 10
v2 w2 5 v2 w2 9
v3 w3 5
• Select largest (most valuable) object?
Both Fail!
• Select smallest object?
COSC 3101N J. Elder
Approximate Greedy Solution
• For the simplified knapsack problem:
• the greedy solution (taking the most valuable object
first) isn’t that bad:
Vˆ 1 V , where
2
ˆ
V Total value of items selected by greedy algorithm
V Total value of items selected by optimal algorithm
COSC 3101N J. Elder
Approximate Greedy Solution
ˆ 1
Claim:V V
2
Proof:
Let C capacity of knapsack.
ˆ
Let S subset of items selected by greedy algorithm.
ˆ
1. Suppose V V claim is true
ˆ
2. SupposeV V
ˆ 1
Claim:V C
2
Proof by contradiction:
ˆ 1 1 ˆ ˆ
Suppose V C s C V s S
2 2
s Cs S ˆ
(otherwise s would have been selected first by greedy algorithm)
ˆ
V V contradiction ˆ 1
Thus V V
2
COSC 3101N J. Elder
Fractional Knapsack Problem
•Like the 0-1 knapsack problem, but can take
fraction of an item.
•Can this be solved with a greedy algorithm?
•Yes!
•To solve the fractional problem, rank items by
value/weight: vi/wi .
•Let vi / wi ≥ vi+1/ wi+1 for all i .
COSC 3101N J. Elder
Fractional Knapsack Problem
FRACTIONAL-KNAPSACK(v,w,W)
load ← 0
i←1
while load < W and i ≤ n
do if wi ≤ W - load
then take all of item i
else take (W - load)/ wi of item i
add what was taken to load
i ←i + 1
Time: O(n lg n) to sort, O(n) thereafter.
COSC 3101N J. Elder