Recursive Back Tracking
&
Dynamic Programming
Jeff Edmonds York University
Lecture 7
COSC 31011
Techniques
Optimization Problems
A Sequence of Decisions
The Little Bird & Friend
Optimal Substructure
Memoization
Set of Sub-Instances
Tracing Dyn. Prog. Alg
Reversing
Code
Speeding Up Running Time
Multiple Opt Solutions
Review
Question for Little Bird
Review & Don'ts
Problems
Best Path
Printing Neatly
Longest Common Subsequence
Knapsack Problem
The Event Scheduling Problem
Parsing
Satisfiability
2
Dynamic Programming
• A hard topic.
• I try to provide a unified way to think of it
and a fixed set of steps to follow.
• Even if you don’t get the details of the
algorithm correct, at least get the right structure.
• I provide analogies (little bird) to make it
hopefully more fun & easier to follow.
3
Optimization Problems
• An important and practical class of computational
problems.
• For most of these, the best known algorithm runs in
exponential time.
• Industry would pay dearly to have faster algorithms.
• Heuristics
• Some have quick Greedy or Dynamic Programming
algorithms
• For the rest, Recursive Back Tracking is the best option.
4
Optimization Problems
Ingredients:
•Instances: The possible inputs to the problem.
•Solutions for Instance: Each instance has an
exponentially large set of solutions.
•Cost of Solution: Each solution has an easy to
compute cost or value.
5
Optimization Problems
Specification of an Optimization Problem
•<preCond>: The input is one instance.
•<postCond>:
The output is one of the valid solutions for this
instance with optimal cost.
(minimum or maximum)
•The solution might not be unique.
•Be clear about these ingredients!
6
Search Graph For Best Path
We use it because it nicely demonstrates
the concepts in a graphical way.
7
Search Graph For Best Path
An instance (input) consists of <G,s,t>.
G is a weighted directed layered graph
s source node
t sink node
4
2
2
5
3
5
4
2
3
3
2
1
8
Search Graph For Best Path
An instance (input) consists of <G,s,t>.
A solution for an instance is a path from s to t.
The cost of a solution is the sum of the weights.
The goal is to find a path with minimum total cost.
2+6+3+7=18
4+2+1+5=12
9
Brute Force Algorithm
Try all paths, return the best.
But there may be an exponential number of paths!
10
An Algorithm As
A Sequence of Decisions
I ask a question about the solution.
“Which edge should we take first?”
Some how I decide <s,v3>.
My friend asks the next question.
“Which edge do we take second?”
Some how he decides <v3,v5>.
His friend asks the next question.
“Which edge do we take third?”
Some how he decided <v5,v8>.
11
An Algorithm As
A Sequence of Decisions
I ask a question about the solution.
“Which edge should we take first?”
How do I decide?
The greedy algorithm?
Taking the best first edge.
Does not work!
12
Local vs Global Considerations
• We are able to make local observations and choices.
– Eg. Which edge out of s is cheapest?
• But it is hard to see the global consequences
– Which path is the overall cheapest?
• Sometimes a local initial sacrifice can globally
lead to a better overall solution.
13
An Algorithm As
A Sequence of Decisions
I ask a question about the solution.
“Which edge should we take first?”
How do I decide?
•In reality we will try
all possible first edges.
•But let's skip this part
by pretending that we have
a little bird to answer this
little question.
14
"Little Bird" Abstraction
(It is up to you whether or not to use it)
Recall: Non-deterministic Finite Automata
Non-deterministic Turing Machine
0
These have a higher power to tell them
which way to go.
The little bird is a little higher power,
answering a little question about an
optimal solution.
15
Little Bird & Friend Alg
I ask a question about the solution.
“Which edge should we take first?”
The bird answers <s,v1>.
My friend asks the next question.
“Which edge do we take second?”
The bird answers <v1,v4>.
But we don’t want to
worry about how our friend
solves his problem.
16
Sub-Instance for Friend
Our instance is <G,s,t>: Find best path from s to t.
Our friend is recursion
•i.e. he is a smaller version of ourselves
•we can trust him to give us a correct answer
•as long as we give him
•a smaller instance
•of the same problem.
•What sub-instance do
we give him?
17
Little Bird & Friend Alg
The bird answers <s,v1>.
If I trust the little bird, I take step along edge <s,v1>
and ask my friend,
“Which is the best path from v1 to t?”
Friend answers <v1,v6,t>
with weight 10.
To get my solution
I tack on the bird’s edge
making the path <s,v1,v6,t>
with weight 10+3=13.
18
Faulty Bird
But what if we do not have a bird that we trust?
This work is not wasted, because we have found
•the best solution to our instance
from amongst those
consistent with this bird' s answer.
•i.e. the best path from s to t
from amongst those
starting with <s,v1>.
Define optS<I,k> to be:
the optimum solution
for instance I
consistent with the kth bird' s answer.
19
Faulty Bird
But what if we do not have a bird that we trust?
This work is not wasted, because we have found
•the best solution to our instance
from amongst those
consistent with this bird' s answer.
•i.e. the best path from s to t
from amongst those
starting with <s,v1>.
In reality we will try
all possible first edges,
giving …..
20
Faulty Bird
…the best path from amongst
those starting with <s,v1>.
21
Faulty Bird
… and the best path from amongst
those starting with <s,v2>.
22
Faulty Bird
… and the best path from amongst
those starting with <s,v3>.
23
Faulty Bird
… and the best path from amongst
those starting with <s,v4>.
24
Faulty Bird
At least one of these four paths
must be an over all best path.
I give the best of the best
as the best path.
25
Bird/Friend - Best of the Best
Consider our instance I.
Consider the set of solutions
A sequence of question to
a little bird about a solution
forms a tree of possible
answers.
26
Bird/Friend - Best of the Best
Consider our instance I.
Consider the set of solutions
k
But we only care about
the first bird answer.
The answers classifies
the possible solutions.
k
Solutions consistent
with the kth bird' s answer.
27
Bird/Friend - Best of the Best
Consider our instance I.
Consider the set of solutions
Define optS<I,k> to be:
the optimum solution
for instance I
consistent with the kth bird' s answer.
Do this for each k.
optS<I,k>
k
Solutions consistent
with the kth bird' s answer.
28
Bird/Friend - Best of the Best
Consider our instance I.
Consider the set of solutions
optS[I]
optS<I,k>
Define optS<I,k> to be:
the optimum solution
kmax k
for instance I
consistent with the kth bird' s answer.
Do this for each k.
Let kmax be the bird' s answer
giving the best optS<I,k>.
optS[I] = optS<I,k max> = Bestk optS<I,k >
29
Bird/Friend - Best of the Best
Constructing optS<I,k> :
the optimum solution
for instance I
consistent with the kth bird' s answer.
Given my instance I.
I ask my little bird for
an answer k.
I ask my friend for his solution.
I combine them.
30
Recursive
backtracking
code always has this
same basic structure.
31
Be clear what are
•the instances
•it’s solution
•the cost of a sol.
32
Loop through the
bird answers.
Be clear which is
the current one
being tried.
33
Give the bird
& friend algorithm
as a comment.
(Unless it is in
an earlier question.)
34
What is the bird
asked?
What does she
answer?
35
Get help from friend
Be clear what
sub-instance
you give him.
Store the solution
& cost
he gives you.
36
How do you form
your solution from
the friend’s and from
the bird’s?
37
How do you form
your cost from
the friend’s and from
the bird’s?
38
optSolk
is a best solution
for our instance
from amongst
those consistent
with the bird's
kth answer.
Take the best
of the best
39
Return the solution
and cost for the
original instance.
40
Base Cases:
Instances that are too
small to have smaller
instances to give
to friends.
What are these?
What are their
solutions
and costs?
41
Optimal Substructure
In order to be able to design a recursive backtracking
algorithm for a computational problem,
the problem needs to have a recursive structure,
i.e. for a path from s to t to be optimal,
the sub-path from vi to t must optimal.
If  shorter from vi to t.
  shorter to s to t.
42
Optimal Substructure
In order to be able to design a recursive backtracking
algorithm for a computational problem,
the problem needs to have an optimal substructure,
i.e. for a path from s to t to be optimal,
the sub-path from vi to t must optimal.
And finding such a sub-path is a
sub-instance of the same
computational problem.
43
Optimal Substructure
• Optimal substructure means that
– Every optimal solution to a problem contains...
– ...optimal solutions to subproblems
• Optimal substructure does not mean that
– If you have optimal solutions to all subproblems...
– ...then you can combine any of them to get an optimal solution
to a larger problem.
• Example: In Canadian coinage,
– The optimal solution to 7¢ is 5¢ + 1¢ + 1¢, and
– The optimal solution to 6¢ is 5¢ + 1¢, but
– The optimal solution to 13¢ is not 5¢ + 1¢ + 1¢ + 5¢ + 1¢
• But there is some way of dividing up 13¢ into subsets
with optimal solutions (say, 11¢ + 2¢) that will give an
optimal solution for 13¢
– Hence, the making change problem exhibits optimal
substructure.
44
Optimal Substructure
Don’t all problems have this
optimal substructure property?
45
Optimal Substructure
Longest simple path
B
1
1
3
2
4
C
D
A
• Consider the
following graph:
• The longest simple path (path not containing a
cycle) from A to D is A B C D
• However, the subpath A B is not the longest
simple path from A to B (A C B is longer)
• The principle of optimality is not satisfied for
this problem
• Hence, the longest simple path problem cannot
be solved by a dynamic programming
approach
NP-Complete
46
Same as Brute Force Algorithm
I try each edge out of s.
A friend tries each edge
out of these.
A friend tries each edge
out of these.
Time?
Same as the brute force algorithm
that tries each path.
47
Same as Brute Force Algorithm
But there may be an exponential number of paths!
48
Speeding Up the Time
Why do all this work with birds & friends?
•How else would you iterate through all paths?
•But sometimes we can exploit the structure
to speed up the algorithm.
49
Speeding Up the Time
Sometimes entire an branch can be pruned off.
•Perhaps because these solutions are not valid
or not highly valued.
•Or because there is at least one optimal
solution elsewhere in the tree.
•A Greedy algorithm prunes off all branches
except the one that looks best.
50
Speeding Up the Time
Memoization
•Remembers the solutions for the sub-instances
so that if ever it needs to be solved again,
the answer can be used.
•This effectively prunes off this later branch of the
classification tree.
51
Exponential Time
Redoing Work
“Which is the best path from v7 to t?”
How many friends solve this sub-instance?
52
Exponential Time
Redoing Work
“Which is the best path from s to t?”
53
Exponential Time
Redoing Work
“Which is the best path from v1 to t?”
54
Exponential Time
Redoing Work
“Which is the best path from v4 to t?”
55
Exponential Time
Redoing Work
“Which is the best path from v7 to t?”
There’s one.
56
Exponential Time
Redoing Work
“Which is the best path from s to t?”
57
Exponential Time
Redoing Work
“Which is the best path from v3 to t?”
58
Exponential Time
Redoing Work
“Which is the best path from v5 to t?”
59
Exponential Time
Redoing Work
“Which is the best path from v7 to t?”
There’s another.
60
Exponential Time
Redoing Work
“Which is the best path from v7 to t?”
How many friends solve this sub-instance?
Once for each
path to v7
Waste time
redoing work
Save time by only
doing once.
61
Depth First
Search
Drop
bread crumbs
and don’t revisit.
But we need
shortest path
62
Dynamic Programming
Having many friends
solving this same sub-instance
is a waste of time.
We allocate one friend to the job.
63
Dynamic Programming
It is my job to learn
and remember
the optSol to my sub-Instance
i.e. the best path from v7 to t
64
Dynamic Programming
When I need to find
the best path from v4 to t
I will ask you for
the best path from v7 to t
I will find my best path
and tell you.
65
Dynamic Programming
When I need to find
the best path from v2 to t
I will ask you for
the best path from v7 to t
I remember my best path
and will tell you.
66
Dynamic Programming
When I need to find
the best path from v5 to t
I will ask you for
the best path from v7 to t
I remember my best path
and will tell you.
67
Dynamic Programming
Avoid waiting.
When I need to find
the best path from v2 to t
I will ask you for
the best path from v7 to t
I will find my best path
and tell you.
But I hate to wait for you.
Recursion has a lot of overhead
Why don’t you go first?
68
Dynamic Programming
Before anyone asks me,
I will find my best path
and remember it.
69
Set of Sub-Instances
But what sub-instance need to be solved
and in which order?
Given an instance I,
Imagine running the
recursive algorithm on it.
Determine the complete
set of sub-Instances ever
given to you, your friends,
their friends, …
70
Set of Sub-Instances
Guess the complete set S of sub-Instances.
“Best path from v7 to t?”
“Best path from v21 to t?”
v21
Yes
No
v21 is not a part of our
original instance.
71
Set of Sub-Instances
Guess the complete set S of sub-Instances.
“Best path from v7 to t?”
“Best path from v21 to t?”
“Best path from v3 to v7?”
Yes
No
No
All paths considered
end in t.
72
Set of Sub-Instances
Guess the complete set S of sub-Instances.
“Best path from v7 to t?”
“Best path from v21 to t?”
“Best path from v3 to v7?”
Yes
No
No
All paths considered
end in t.
73
Set of Sub-Instances
Guess the complete set S of sub-Instances.
“Best path from v7 to t?”
Yes
“Best path from v21 to t?” No
“Best path from v3 to v7?” No
 i “Best path from vi to t?” Yes
74
Set of Sub-Instances
Guess the complete set S of sub-Instances is
 i “Best path from vi to t?”
Assign one friend
to each sub-Instance.
75
Set of Sub-Instances
Guess the complete set S of sub-Instances is
 i “Best path from vi to t?”
The set S of sub-Instances needs to:
•include our given I
76
Set of Sub-Instances
Guess the complete set S of sub-Instances is
 i “Best path from vi to t?”
The set S of sub-Instances needs to:
•include our given I
•closed under “friend” operation
Integers closed under addition
 x,y  I  x+y  I
 sub-Instance  S 
subsub-Instance  S
77
Set of Sub-Instances
Guess the complete set S of sub-Instances is
 i “Best path from vi to t?”
The set S of sub-Instances needs to:
•include our given I
•closed under “friend” operation
•each sub-Instance needs to be
asked of some friend, friend, …
78
Set of Sub-Instances
Guess the complete set S of sub-Instances is
 i “Best path from vi to t?”
The set S of sub-Instances needs to:
•include our given I
•closed under “friend” operation
•each sub-Instance needs to be
asked of some friend, friend, …
A fine set of sub-instances!
79
Order to complete
The complete set S of sub-Instances is
 i “Best path from vi to t?”
In what order should they go?
•in an order such that
no friend must wait.
•from “smallest” to “largest”
For this problem,
the order relies on
the graph being “leveled.”
80
Order to complete
The complete set S of sub-Instances is
 i “Best path from vi to t?”
In what order should they go?
•in an order such that
no friend must wait.
•from “smallest” to “largest”
First
Base Case easy
Last
Instance to be solved.
81
Dynamic Programming
"Which is the best path from t to t?"
Base Case
easy
82
Dynamic Programming
"Which is the best path from v8 to t?"
easy
83
Dynamic Programming
"Which is the best path from v7 to t?"
easy
84
Dynamic Programming
"Which is the best path from v6 to t?"
easy
85
Dynamic Programming
"Which is the best path from v5 to t?"
Harder
86
Dynamic Programming
"Which is the best path from v5 to t?"
Little bird
suggests first
edge <v5,v7>
Friend gives best
path <v7,t>.
87
Dynamic Programming
"Which is the best path from v5 to t?"
Little bird
suggests first
edge <v5,v8>
Friend gives best
path <v8,t>.
88
Dynamic Programming
"Which is the best path from v5 to t?"
Take best of best
89
Dynamic Programming
"Which is the best path from v4 to t?"
90
Dynamic Programming
"Which is the best path from v4 to t?"
Little bird
suggests first
edge <v4,v6>
Friend gives best
path <v7,t>.
91
Dynamic Programming
"Which is the best path from v4 to t?"
Little bird
suggests first
edge <v4,t>
Friend gives best
path <t,t>.
92
Dynamic Programming
"Which is the best path from v4 to t?"
Little bird
suggests first
edge <v4,v7>
Friend gives best
path <v7,t>.
93
Dynamic Programming
"Which is the best path from v4 to t?"
Take best of best
94
Dynamic Programming
"Which is the best path from v3 to t?"
95
Dynamic Programming
"Which is the best path from v3 to t?"
Little bird
suggests first
edge <v3,v5>
Friend gives best
path <v5,t>.
96
Dynamic Programming
"Which is the best path from v3 to t?"
Little bird
suggests first
edge <v3,v8>
Friend gives best
path <v8,t>.
97
Dynamic Programming
"Which is the best path from v3 to t?"
Take best of best
98
Dynamic Programming
"Which is the best path from v2 to t?"
99
Dynamic Programming
"Which is the best path from v2 to t?"
Little bird
suggests first
edge <v2,v4>
Friend gives best
path <v4,t>.
100
Dynamic Programming
"Which is the best path from v2 to t?"
Little bird
suggests first
edge <v2,v7>
Friend gives best
path <v7,t>.
101
Dynamic Programming
"Which is the best path from v2 to t?"
Take best of best
102
Dynamic Programming
"Which is the best path from v1 to t?"
103
Dynamic Programming
"Which is the best path from v1 to t?"
Little bird
suggests first
edge <v1,v3>
Friend gives best
path <v3,t>.
104
Dynamic Programming
"Which is the best path from v1 to t?"
Little bird
suggests first
edge <v1,v4>
Friend gives best
path <v4,t>.
105
Dynamic Programming
"Which is the best path from v1 to t?"
Little bird
suggests first
edge <v1,v5>
Friend gives best
path <v5,t>.
106
Dynamic Programming
"Which is the best path from v1 to t?"
Take best of best
107
Dynamic Programming
"Which is the best path from s to t?"
Original Problem
108
Dynamic Programming
"Which is the best path from s to t?"
Little bird
suggests first
edge <s,v1>
Friend gives best
path <v1,t>.
109
Dynamic Programming
"Which is the best path from s to t?"
Little bird
suggests first
edge <s,v2>
Friend gives best
path <v2,t>.
110
Dynamic Programming
"Which is the best path from s to t?"
Little bird
suggests first
edge <s,v3>
Friend gives best
path <v3,t>.
111
Dynamic Programming
"Which is the best path from s to t?"
Little bird
suggests first
edge <s,v4>
Friend gives best
path <v4,t>.
112
Dynamic Programming
"Which is the best path from s to t?"
Take best of best
DONE
113
Dynamic Programming
Construct a table
•for storing an optimal solution & cost
•for each sub-instance.
Map
 i “Best path from vi to t?”
Sub-Instances
Indexes
 i ϵ [n], i.e. for each node vi
Cell of table
“Which is the best path from vi to t?”
t,
v , v , vi , v , ….,
s
114
Dynamic Programming
Fill out a table containing
an optimal solution for each sub-instance.
“Which is the best path from vi to t?”
t, v8, v7, v6, v5, ….,
Base case
s
Original
115
Communication
Friend k gives friend i a best path from vk to t.
Recursive BackTracking
i <optSubSol,optSubCost>
= LeveledGraph(<G,vk,t>)
k return(optSolmin,optCostmin) ,
i
k
Dynamic Programming
k optSol[k] = optSolmin
i
optSubSol = optSol[k]
optSolk = <vi,vk> + optSol[k]
116
Dynamic
Programming
code always has this
same basic structure.
117
Be clear what are
•the instances
•it’s solution
•the cost of
a solution.
118
Dynamic Programs
do not recurse making
the instance smaller
and smaller.
Instead, it up front
determines the set S
of all sub-instances
that ever need
to be solved.
Be clear what
sub-instances are.
119
Be clear what
sub-instances are.
How are they indexed?
Tables indexed by
these sub-instances
store an optimal
solution and it’s cost.
120
The set S
of sub-instances
are solved from
smallest to largest
so that no body waits.
Base Cases:
Instances that are too
small to have smaller
instances to give
to friends.
They get solved first
and their solutions
stored.
121
Then we iterate
through the remaining
sub-instances.
From smallest
to largest.
Each gets solved
and their solutions
stored.
122
Consider yourself
to be a friend
working on one
of these.
Be clear which
sub-instance is yours.
Solve this as you
did before.
123
Loop through the
bird answers.
Be clear which is
the current one
being tried.
124
Give the bird
& friend algorithm
as a comment.
(Unless it is in
an earlier question.)
125
What is the bird
asked?
What does she
answer?
126
k
k
k
k
Get help from friend
Be clear what
sub-instance
you give your friend.
127
k
k
k
k
Get help from friend
Instead of recursing,
we simply look
in the table
for the solution.
Because his instance
is smaller, he has
already solved it and
stored sol in the table.
128
How do you form
your solution from
the friend’s and from
the bird’s?
129
How do you form
your cost from
the friend’s and from
the bird’s? 130
optSol<i,k>
is a best solution for
our instance subI[i]
from amongst
those consistent
with the bird's
kth answer.
Take the best
of the best
131
Store the solution to
our instance subI[i]
in the table.
132
Base Cases:
Instances that are too
small to have smaller
instances to give
to friends.
Is this code correct?
133
Dynamic Programs
do not recurse making
the instance smaller
and smaller.
Hence, lets not worry
about our instance I
being a base case.
134
But there is a table
of subinstances
that must be solved.
Some of these will be
base cases
and their solutions
must be stored
in the table.
135
t=
n
n
n
n
But there is a table
of subinstances
that must be solved.
Some of these will be
base cases
and their solutions
must be stored
in the table.
136
But there is a table
of subinstances
that must be solved.
Then we solve
the rest.
137
s=
0
0
n
0
Return the solution
and cost for the
original instance.138
Order Feels Backwards
Path from s to t.
Path from t to t.
139
An Esthetically Better
Path from s to s.
Path from s to t.
140
Reversing
141
Reversing
Determine the complete set of sub-instances
“Which is the best path from s to vi?”  i
142
Reversing
Fill out a table containing
an optimal solution for each sub-instance.
“Which is the best path from s to vi?”
s, v1, v2, v3, v4, ….,
Base case
t
Original
143
144
Running Time
Time =
# of Sub-Instances
× # of Bird Answers
× q(1) ?
= n× d
145
Communication Time
optSolk = <optSol[k],vi>
Friend k gives best path from s to vk
to friend i, who adds the edge <vk,vi>.
Time = q(1)
k
?
Size of path = q(n).
Time = q(n)
i
146
Running Time
Time =
# of Sub-Instances
× # of Bird Answers
× size of solution
= q(n × d × n)
Store path costs,
not paths
Space =
# of Sub-Instances
× q(1)
= q(n)
147
Store Path Costs, not Paths
"What is cost of the best path from s to v7?"
148
Store Path Costs, not Paths
"What is cost of the best path from s to v7?"
8
Little bird suggests last
edge <v4,v7> with weight 2.
Friend gives cost 8
of best path <s,v4>.
Best cost via <v4,v7> is 8+2=10.
149
Store Path Costs, not Paths
"What is cost of the best path from s to v7?"
2
Little bird suggests last
edge <v2,v7> with weight 7.
Friend gives cost 2
of best path <s,v2>.
Best cost via <v2,v7> is 2+7=9.
150
Store Path Costs, not Paths
"What is cost of the best path from s to v7?"
6
Little bird suggests last
edge <v5,v7> with weight 5.
Friend gives cost 6
of best path <s,v5>.
Best cost via <v5,v7> is 6+5=11.
151
Store Path Costs, not Paths
"What is cost of the best path from s to v7?"
2
9
Take best of best: 10, 9, 11
We also learn
the wise little bird’s advice.
We will store this in the table too.
152
Running Time
birdsAdvice[i] = kmin
153
Leave these lines
as comments
for extra clarity
for the reader
154
Find Optimal Path
Previous algorithm gives:
• Cost of the best path
from s to vi,  i.
• Bird’s advice of
last edge to vi.
We run the bird-friend
algorithm again,
but with a reliable bird.
155
Find Optimal Path
The bird gives that
the last edge
of the best path from
s to t is <v8,t>.
156
Find Optimal Path
The bird gives that
the last edge
of the best path from
s to v8 is <v5,v8>.
157
Find Optimal Path
The bird gives that
the last edge
of the best path from
s to v5 is <v3,v5>.
158
Find Optimal Path
The bird gives that
the last edge
of the best path from
s to v3 is <s,v3>.
159
Find Optimal Path
Done!
160
Find Optimal Path
This could be done iteratively.
As an exercise, design it.
161
Multiple Optimal Solutions
I ask the bird:
“Which is the last edge?”
She could give either answer.
By giving this edge she says
“There exists an optimal solution
consistent with this answer.”
Similar to greedy proof.
6
162
Multiple Optimal Solutions
I ask the bird:
“Which is the last edge?”
We try all the bird answers.
When we try this bird answer,
we find this best solution.
When we try this bird answer,
we find this best solution.
When we take best of best,
we choose between them.
6
163
Review
Designing Recursive Back Tracking Algorithm
•What are instances, solutions, and costs?
•Given an instance I,
•What question do you ask the little bird?
•Given a bird answer k  [K],
•What instance sub-Instance do your give your friend?
•Assume he gives you optSubSol for subI.
•How do you produce an optSol for I from
•the bird’s k and
•the friend’s optSubSol?
•How do you determine the cost of optSol from
•the bird’s k and
•the cost of the friend’s optSubSol?
•Try all bird’s answers and take best of best.
164
Review Recursive Back Tracking Algorithm
Dynamic Programming Algorithm
•Given an instance I,
•Imagine running the recursive alg on it.
• Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
• Build a table indexed by these sub-Instances
•Fill in the table in order so that nobody waits.
•the cost of its optimal solution
•advice given by the bird
•Run the recursive alg with bird’s advice to find
the solution to your instance.
165
The Question For the Little Bird
Purpose of Little Bird:
•An abstraction from which it is
easier to focus on the difficult issues.
•Her answers give us a list of things to try.
•Temporarily trusting the bird,
helps us focus on the remaining question
helping us formulate sub-instance for friend.
•Coming up with which question is one of the
main creative steps.
•Hint: Ask about a local property
•There are only so many question that you
might ask so just try them all.
166
The Question For the Little Bird
An instance: Graph, s, and t
A solution: a path
t
s
I ask the bird:
“What is the first edge in the path?”
The Dynamic Programming reverses the recursive
backtracking algorithm.
Hence, to end up with a “forward order”,
we first reverse the recursive backtracking algorithm.167
The Question For the Little Bird
An instance: Graph, s, and t
A solution: a path
t
s
I ask the bird:
“What is the last edge in the path?”
The Dynamic Programming reverses the recursive
backtracking algorithm.
Hence, to end up with a “forward order”,
we first reverse the recursive backtracking algorithm.168
The Question For the Little Bird
An instance: Graph, s, and t
A solution: a path
t
s
I ask the bird:
“What is the last edge in the path?”
A good question for the bird leaves you
with a good recursive sub-instance to ask your friend.
“What is the rest of the path?”
169
The Question For the Little Bird
An instance: Graph, s, and t
A solution: a path
t
s
I ask the bird:
“What is the last edge in the path?”
Giving a good follow up question for your friend to
ask the bird.
“What is the second last edge in the path?”
170
The Question For the Little Bird
• You can only ask the bird a little question.
– Together with your question, you provide the little
bird with a list A1, A2, …, AK of possible answers.
– The little bird answers, k  [1..K].
– For an efficient algorithm, K must be small.
•Eg. “What is best last edge?”
– K = number of edges into node t.
t
s
K
171
The Question For the Little Bird
• You can only ask the bird a little question.
– Together with your question, you provide the little
bird with a list A1, A2, …, AK of possible answers.
– The little bird answers, k  [1..K].
– For an efficient algorithm, K must be small.
•Eg. “What is an optimal solution?”
– K = # of solutions.
Trying all is the Brute Force algorithm.
172
The Question For the Little Bird
An instance: Graph, s, and t
A solution: a path
t
s
I ask the bird:
“How many edges are in the path?”
Bad Question:
•it is not a local property
•How does this help us solve the problem?
•What is a good follow up question for the
friend to ask?
173
The Question For the Little Bird
An instance: ???
A solution: a sequence of objects
Z=abcd
I ask the bird:
“What is the last object in the sequence?”
# of answers K = # of possible last objects.
I ask my friend:
“What is the rest of the solution?”
174
The Question For the Little Bird
An instance: a sequence of objects
X == aa ss bb ee ff cc hh dd aa
X
A solution: a subset of these objects
Z=abcd
I ask the bird:
“What is the last object in the sequence?”
# of answers K = # of possible last objects.
Is there a smaller question that we could ask?
175
The Question For the Little Bird
An instance: a sequence of objects
X=asbefchda
A solution: a subset of these objects
Z=abcd
I ask the bird:
“Is the last object of the instance included
in the optimal solution?”
# of answers K = 2, Yes or No
176
The Question For the Little Bird
An instance: ???
A solution: a binary tree of objects
38
25
17
51
31
4 21
28 35
I ask the bird:
“What object is at the root?”
I ask my friend:
42
40
63
49
55
71
I ask a second friend:
“What is the left sub-tree?”“What is the right sub-tree?”
Previous problems had one friend given a bird ans.177
The Question For the Little Bird
An instance: ???
A solution: a binary tree of objects
38
25
17
51
31
42
63
4 21
28 35 40 49
55 71
I ask the bird:
“What object is at a leaf?”
Bad Question:
•How does this help us solve the problem?
•What is a good follow up question for the
friend to ask?
178
Printing Neatly
179
Printing Neatly
An instance: text to print neatly & # chars per line
“Love life man while there as we be”, 11
A solution: # of words to put on each line.
The cost: a measure of how neat,
few blanks on the end of each line.
3
small punishment
8
8
216 big punishment
27
259
11
The goal is to to print it as “neatly” as possible.
Love.life..
man.while..
there......
as.we.be...
2
3
2
3
6
3
3
=
=
=
=
180
Brute Force Algorithm
Try all ways to print, return the best.
But there may be an exponential number of ways to!
love.life..
man........
love.......
life.man...
love.......
life.man...
love.life..
man........
181
Bird & Friend Algorithm
An instance:“Love life man while there as we be”, 11
I ask the bird:
“How many words on the last line?”
She may answer 3.
I ask the friend:
“Which is the best way to print
the remaining n-3 words?”
I combine
bird’s and
friend’s answers.
182
Bird & Friend Algorithm
An instance:“Love life man while there as we be”, 11
Even if the bird was wrong, this work is not wasted.
This is best way to print from
amongst those ending in 3 words.
We try the bird
answers 12words,
34
5
and take best of best.
183
Same as Brute Force Algorithm
I try each # words on last line.
A friend tries # on next.
A friend tries # on next.
Time?
Same as the brute force algorithm
that tries each path.
184
Memoization
Assign one friend to each sub-instance.
“Which is the best path from vi to t?”  i
185
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
“Love life man while there as we be”, 11
•Imagine running the recursive algorithm on
it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their friends,
…
186
Set of Sub-Instances
Guess the complete set of sub-Instances.
“Love life man while there as we be”, 11
“Love life man while there”, 11
“Hi there”, 81
“man while”, 11
Yes
No
No
This may appear on a line,
but it will never be a sub-Instance for a friend.
187
Set of Sub-Instances
The set of sub-Instances is the set of prefixes.
“Love life man while there as we be”, 11
“Love life man while there as we”, 11
“Love life man while there as”, 11
“Love life man while there”, 11
“Love life man while”, 11
“Love life man”, 11
“Love life”, 11
“Love”, 11 The set S of sub-Instances needs to:
“”, 11
•include our given I
•closed under “friend” operation
 sub-Instance  S 
188
Set of Sub-Instances
The set of sub-Instances is the set of prefixes.
“Love life man while there as we be”, 11
“Love life man while there as we”, 11
“Love life man while there as”, 11
“Love life man while there”, 11
“Love life man while”, 11
“Love life man”, 11
“Love life”, 11
“Love”, 11 The set S of sub-Instances needs to:
“”, 11
•include our given I
•closed under “friend” operation
The bird
 sub-Instance  S 
answers 12words,
subsub-Instance  i
34
5
189
Set of Sub-Instances
The set of sub-Instances is the set of prefixes.
“Love life man while there as we be”, 11
“Love life man while there as we”, 11
“Love life man while there as”, 11
“Love life man while there”, 11
“Love life man while”, 11
“Love life man”, 11
“Love life”, 11
“Love”, 11 The set S of sub-Instances needs to:
“”, 11
•include our given I
•closed under “friend” operation
•each sub-Instance needs to be
asked of some friend, friend, …
190
Set of Sub-Instances
The set of sub-Instances is the set of prefixes.
“Love life man while there as we be”, 11
“Love life man while there as we”, 11
“Love life man while there as”, 11
“Love life man while there”, 11
“Love life man while”, 11
The bird
answers 1.
The set S of sub-Instances needs to:
•include our given I
•closed under “friend” operation
•each sub-Instance needs to be
asked of some friend, friend, …
A fine set of sub-instances! 191
Set of Sub-Instances
The set of sub-Instances is the set of prefixes.
“Love life man while there as we be”, 11
“Love life man while there as we”, 11
“Love life man while there as”, 11
“Love life man while there”, 11
“Love life man while”, 11
“Love life man”, 11
“Love life”, 11
“Love”, 11
In what order should they go?
“”, 11
•in an order such that
no friend must wait.
•from “smallest” to “largest”
First Base Case easy
Last
Instance to be solved.
192
The Table
Construct a table
•for storing the cost of opt sol and bird’s advice.
•for each sub-instance.
Map
The set of prefixes of words.
Sub-Instances
Indexes
 i ϵ [n], i.e. for each word.
Cell of table
“Which is the best printing of first i words?”
i
193
Dynamic Programming
Fill out a table containing
an optimal solution for each sub-instance.
“Which is the best printing of first i words?”
Base case
Original
194
“Love life man while there as we be”, 11
The 5th sub-instance is
“Love life man while there”, 11
5 words
with 4, 4, 3, 5,
5 letters.
195
“Love life man while there as we be”, 11
The 5th sub-instance is
Love.life.. “Love life man while there”, 11
man.while.. Its solution is
there...... with 2,2,1 words on each line.
The bird’s advice is 1 word on last.
Solution’s cost is 23 + 23 +63 = 232 196
“Love life man while there as we be”, 11
Assume the table is filled in so far.
We will work to fill in the last line
197
“Love life man while there as we be”, 11
Love.life..
man.while..
there.as.we
be.........
198
“Love life man while there as we be”, 11
Love.life..
man.while..
there.as...
we.be......
199
“Love life man while there as we be”, 11
Love.life..
man.while..
there......
as.we.be...
200
“Love life man while there as we be”, 11
there.as.we.be
201
“Love life man while there as we be”, 11
Tried all bird
answers.
Choose best of the best.
202
“Love life man while there as we be”, 11
Choose best of the best.
203
Dynamic
Programming
code always has this
same basic structure.
Amusingly,
when formatting
this code, I had to fight
with line breaks to get
the height/width ratio
Printing Neatly.
204
Be clear what are
•the instances
•it’s solution
•the cost of
a solution.
205
Dynamic Programs
do not recurse making
the instance smaller
and smaller.
Instead, it up front
determines the set S
of all sub-instances
that ever need
to be solved.
Be clear what
sub-instances are.
206
Be clear what
sub-instances are.
How are they indexed?
Tables indexed by
these sub-instances
store an optimal
solution and it’s cost.
207
The set S
of sub-instances
are solved from
smallest to largest
so that no body waits.
Base Cases:
Instances that are too
small to have smaller
instances to give
to friends.
They get solved first
and their solutions
stored.
208
Then we iterate
through the remaining
sub-instances.
From smallest
to largest.
Each gets solved
and their solutions
stored.
Actually, we store the
bird’s advice instead
of the solution.
209
Consider yourself
to be a friend
working on one
of these.
Be clear which
sub-instance is yours.
Solve this as you
did before.
210
Loop through the
bird answers.
Be clear which is
the current one
being tried.
211
Give the bird
& friend algorithm
as a comment.
(Unless it is in
an earlier question.)
212
What is the bird
asked?
What does she
answer?
213
i-k
i-k
i-k
Get help from friend
Be clear what
sub-instance
you give your friend.
214
i-k
i-k
i-k
Instead of recursing,
we simply look
in the table
for the solution.
Because his instance
is smaller, he has
already solved it and
stored sol in the table.
215
How do you form
your solution from
the friend’s and from
the bird’s?
216
How do you form
your cost from
the friend’s and from
the bird’s? 217
optSol<i,k>
is a best solution for
our instance subI[i]
from amongst
those consistent
with the bird's
kth answer.
Take the best
of the best
218
Store the solution to
our instance subI[i]
in the table.
Actually, we store the
bird’s advice instead
of the solution.
219
Base Cases:
Instances that are too
small to have smaller
instances to give
to friends.
Is this code correct?
220
Dynamic Programs
do not recurse making
the instance smaller
and smaller.
Hence, lets not worry
about our instance I
being a base case.
221
But there is a table
of subinstances
that must be solved.
Some of these will be
base cases
and their solutions
must be stored
in the table.
222
But there is a table
of subinstances
that must be solved.
Some of these will be
base cases
and their solutions
must be stored
in the table.
223
But there is a table
of subinstances
that must be solved.
Then we solve
the rest.
224
Return the solution
and cost for the
original instance.
But actually,
we don’t have
the solution.
We must rerun it,
this time with advice
from the bird. 225
Time =
# of Sub-Instances
× # of Bird Answers
= q(n × n)
Space =
# of Sub-Instances
= q(n)
226
Find Optimal Path
Previous algorithm gives cost and bird’s advice.
We run the bird-friend algorithm again,
but with a reliable bird.
227
“Love life man while there as we be”, 11
Love.life..
man.while..
there……
as we be…
<2 ,2 ,1,3>
228
Longest Common Subsequence problem
X=asbefchda
Y=rtwabgjcktfd
229
Longest Common Subsequence problem
An instance: Two strings
X
X = a s b e ft c h d a
Y
Y=r tw a b g jc k tf d
A solution: A common subsequence.
Z=abcd
The cost: The length of Z.
The goal is to find a longest common subsequence.
230
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers one of :
•Last of X is not included
•Last of Y is not included
•Last of X is included
•Last of Y is included
•Neither are included
231
•Both are included
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X is not included
I ask my friend:
The instance:
X=asbetchd
Y=rtwabgjcktfd
232
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X is not included
I combine
My friend answers:
Z=abcd
The instance:
X=asbetchd
Y=rtwabgjcktfd
bird’s and
friend’s answers
and give the same Z.
233
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of Y is not included
I ask my friend:
The instance:
X=asbetchda
Y=rtwabgjcktf
234
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
I combine
•Last of Y is not included
bird’s and
friend’s answers
My friend answers:
the
same
Z.
and
give
Z=abc
The instance:
XX == aa ss bb ee tt cc hh dd aa
w aa bb gg jj cc kk tt ff
YY == rr tt w
Not as good as last
but we need to try.235
Bird & Friend Algorithm
An instance:
X=asbetchd
Y=rtwabgjckdfd
Last chars equal
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X and last of Y are both included
I ask my friend:
The instance:
X=asbetch
Y=rtwabgjckdf
236
Bird & Friend Algorithm
An instance:
X=asbetchd
Y=rtwabgjckdfd
Last chars equal
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X and last of Y are both included
I combine
My friend answers:
bird’s and
Z=abc
The instance:
friend’s answers
X=asbetch
and give
Y=rtwabgjckdf
Zd = abcd.
237
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
Last chars not equal
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X and last of Y are both included
I politely tell her that she is wrong.
238
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X is included
I ask my friend:
The instance:
X=asbetchd
Y=rtwabgjcktfd
239
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X is included
My friend answers:
Z=abcd
The instance:
Wrong
X=asbetchd
Y=rtwabgjcktfd
I combine
bird’s and
friend’s answers
and give
240
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X is included
I ask my friend:
The instance:
X=asbetchd
Y=rtw
241
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers:
•Last of X is included
I combine
My friend answers:
Z=t
The instance:
X=asbetchd
Y=rtw
bird’s and
friend’s answers
and give
Za = ta.
242
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
Last chars not equal
I ask the bird:
“Is the last character of either X or Y included in Z?”
Can we eliminate
She answers one of :
•Last of X is not included some of her answers?
•Last of Y is not included
Given any optSol
•Last of X is included ?
she needs to have
•Last of Y is included ?
a valid answer.
•Neither are included ?
243
•Both are included
Bird & Friend Algorithm
An instance:
X=asbetchda
Y=rtwabgjcktfd
Last chars not equal
I ask the bird:
“Is the last character of either X or Y included in Z?”
She answers one of :
•Last of X is not included
•Last of Y is not included
•Last of X is included
•Last of Y is included
# of answers K = 3
•Neither are included
244
•Both are included
Same as Brute Force Algorithm
I try each of 3 bird ans.
My friends try 3
His friends try 3
Time?
Same as the brute force algorithm
that tries each solution.
245
Memorization
Assign one friend to each sub-instance.
“Which is the best path from vi to t?”  i
246
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
X=asbetchda
Y=rtwabgjcktfd
•Imagine running the recursive alg on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their
friends…
Is this a sub-Instance?
X’ = a s b e t c
Yes
Y’ = r t w a b g j c k
247
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
X=asbetchda
Y=rtwabgjcktfd
•Imagine running the recursive alg on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their
friends, …
Is this a sub-Instance?
No
X’ = b e t
Y’ = a b g j c k
248
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
X=asbetchda
Y=rtwabgjcktfd
•Imagine running the recursive alg on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their
friends, …
Is this a sub-Instance?
Yes
X’ = x1,…xi
Y’ = y1,…,yj  i  [0..|X|]  j  [0..|Y|]
|X| × |Y| of these.
249
Set of Sub-Instances
Guess the complete set S of sub-Instances.
Xi = x1,…xi
Yj = y1,…,yj
 i  [0..|X|]  j  [0..|Y|]
The set S of sub-Instances needs to:
•include our given I
Yes: i = |X| & j = |Y|
250
Set of Sub-Instances
Guess the complete set S of sub-Instances.
Xi = x1,…xi
Yj = y1,…,yj
 i  [0..|X|]  j  [0..|Y|]
The set S of sub-Instances needs to:
•include our given I
•closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
Xi-1 = x1,…xi-1 Yj = y1,…,yj
Xi = x1,…xi
 S  Xi = x1,…xi Yj-1 = y1,…,yj-1
Yj = y1,…,yj
Xi-1 = x1,…xi-1 Yj-1 = y1,…,yj-1
S
251
Set of Sub-Instances
Guess the complete set S of sub-Instances.
Xi = x1,…xi
Yj = y1,…,yj
 i  [0..|X|]  j  [0..|Y|]
The set S of sub-Instances needs to:
•include our given I
•closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
•each sub-Instance needs to be
asked of some friend, friend, …
We showed this.
This is a fine set of sub-Instances!252
The Table
Construct a table
•for storing the cost of opt sol and bird’s advice.
•for each sub-instance.
Map
Xi = x1,…xi
Sub-Instances
Yj = y1,…,yj
 i  [0..|X|]  j  [0..|Y|]
Indexes
Cell of table
“LCS of x1,…xi and y1,…,yj ?”
j
i
253
The Table
254
Table
Y
Original instance
I = <X,Y>
X
255
Table
Yj
Xi
j=
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Optimal Solution
= Longest Common
i=
Subsequence
Cost = length of LCS.
256
Table
Yj
Xi
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Optimal Solution
= Longest Common
Subsequence
Bird’s Advice
•delete xi
257
Table
Yj
Xi
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Optimal Solution
= Longest Common
Subsequence
Bird’s Advice
•delete xi
•take both xi and yj
258
Table
Yj
Xi
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Optimal Solution
= Longest Common
Subsequence
Bird’s Advice
•delete xi
•delete yj
•take both xi and yj
259
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•delete xi
5
Friend’s sub-Instance
Our cost
= friend’s cost
260
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•delete yj
5
Friend’s sub-Instance
Our cost
= friend’s cost
261
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•take both xi and yj
6
Friend’s sub-Instance
Our cost
= friend’s cost
+1
262
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•Take best of best
6
263
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•delete xi
4
Friend’s sub-Instance
Our cost
= friend’s cost
264
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•delete yj
3
Friend’s sub-Instance
Our cost
= friend’s cost
265
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•take both xi and yj
-
Sorry bird is
wrong.
Our cost
= -
266
Fill in Box
Yj
sub-Instancei,j =
Xi = x1,…xi
Yj = y1,…,yj
Xi
Fill in box
•Try all bird’s ans.
•Take best of best
4
267
Fill in Box
268
Fill in Box
269
Order to Fill in Table
Order to fill table:
•so that nobody waits
This guy waits for
270
Order to Fill in Table
(later)
271
Base Cases
Base Cases:
•general algorithm
does not work
•This guy’s
friends are
272
Base Cases
Base Cases:
•general algorithm
does not work
273
Base Cases
274
With Advice
275
With Advice
Done276
Knapsack Problem
Get as much value
as you can
into the knapsack
277
Knapsack Problem
Ingredients:
•Instances: The volume V of the knapsack.
The volume and price of n objects
<<v1,p1>,<v2,p2>,… ,<vn,pn>>.
•Solutions: A set of objects that fit in the knapsack.
•i.e. i  S vi  V
•Cost of Solution: The total value of objects in set.
•i.e. i  S pi
•Goal: Get as much value as you can
into the knapsack.
278
Greedy Algorithm
Greedy Criteria: Most valuable pi
v=4,p=4
v=7,p=5 v=4,p=4
v=4,p=4
V=8
Greedy give 5
v=4,p=4
V=8
Optimal gives 8
279
Greedy Algorithm
Greedy Criteria: Most dense in value
pi
vi
v=7,p=5
V=8
V=7
Greedy give 4
v=4,p=4
v=4,p=4
v=7,p=5
V=7
Optimal gives 5
280
Greedy Algorithm
¾ of
Greedy Criteria: Most dense in value
pi
vi
If fractional solutions are allowed.
Works
Often an Integer
solution is MUCH
v=4,p=4
harder to find.
v=4,p=4
v=7,p=5
V=7
Greedy give 4 + ¾ × 4 = 7 = Optimal gives 5
V=7
281
Bird & Friend Algorithm
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
V=9
v=7,p=5
v=4,p=4
v=4,p=4
A solution:
<<v5,p5>,<v9,p9>,...........,<v82,p82>>.
I ask the bird:
“What is the last object to take?”
# of answers K = n
282
Bird & Friend Algorithm
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
V=9
v=7,p=5
v=4,p=4
v=4,p=4
A solution:
<<v5,p5>,<v9,p9>,...........,<v82,p82>>.
I ask the bird:
“Do we keep the last object?”
# of answers K = 2 Yes & No
283
Bird & Friend Algorithm
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
V=9
v=7,p=5
v=4,p=4
v=4,p=4
Bird says, Yes keep the last object.
Trust her and put it into your knapsack.
I ask my friend:
To fill the rest of the knapsack.
But what instance do I give him?
284
Bird & Friend Algorithm
His instance:
<V-vn:<v1,p1>,<v2,p2>,.........<vn-1,pn-1>,<vn,pn>>.
V=9-4
v=7,p=5
v=4,p=4
v=4,p=4
His solution:
<<v5,p5>,<v9,p9>,...........,<v82,p82>>.
My solution:
<<v5,p5>,<v9,p9>,...........,<v82,p82>,<vn,pn>>
My cost: same + pn
285
Bird & Friend Algorithm
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
V=9
v=7,p=5
v=4,p=4
v=4,p=4
If we trust the bird and friend,
this is valid and optimal.
My solution:
<<v5,p5>,<v9,p9>,...........,<v82,p82>,<vn,pn>>
My cost: same +pn
286
Bird & Friend Algorithm
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
V=9
v=7,p=5
v=4,p=4
v=4,p=4
Bird says, No do not keep the last object.
Trust her and delete it.
I ask my friend:
To fill the knapsack with the rest.
What instance do I give him?
287
Bird & Friend Algorithm
His instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
V=9
v=7,p=5
v=4,p=4
v=4,p=4
His solution:
<<v5,p5>,<v9,p9>,...........,<v82,p82>>.
My solution: same
My cost: same
If we trust the bird and friend,
this is valid and optimal.
288
Same as Brute Force Algorithm
I try each of 2 bird ans.
My friends tries 2
His friends tries 2
Time?
Same as the brute force algorithm
that tries each solution.
289
Memoization
Assign one friend to each sub-instance.
“Which is the best path from vi to t?”  i
290
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
<V:<v1,p1>,<v2,p2>,<v3,p3>,<v4,p4>,<v5,p5>,<v6,p6>>.
•Imagine running the recursive algorithm on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
Is this a sub-Instance?
<V:<v1,p1>,<v2,p2>,<v3,p3>>.
Yes, if the bird keeps saying “No”.
291
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
<V:<v1,p1>,<v2,p2>,<v3,p3>,<v4,p4>,<v5,p5>,<v6,p6>>.
•Imagine running the recursive algorithm on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
Is this a sub-Instance?
<V:<v1,p1>,<v2,p2>,<v3,p3>,<v4,p4>,<v5,p5>,<v6,p6>>.
No, the set of objects is always a prefix
of the original set.
292
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
<V:<v1,p1>,<v2,p2>,<v3,p3>,<v4,p4>,<v5,p5>,<v6,p6>>.
•Imagine running the recursive algorithm on it.
•Determine the complete set of sub-Instances ever
given to you, your friends, their friends, …
Quite possibly, if V’  V.
It is easier to solve than to
Is this a sub-Instance? determine if it is a sub-instance.
<V’:<v1,p1>,<v2,p2>,<v3,p3>>.
293
Set of Sub-Instances
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
Guess the complete set S of sub-Instances.
 V’  [0..V]
<V’:<v1,p1>,<v2,p2>,......,<vi,pi>>.
 i  [0..n]
The set S of sub-Instances needs to:
•include our given I
Yes: V’=V & i = n
•closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
<V’:<v1,p1>,<v2,p2>,......,<vi,pi>>  S
No <V’ :<v1,p1>,<v2,p2>,...,<vi-1,pi-1>>  S
Yes <V’-vi:<v1,p1>,<v2,p2>,...,<vi-1,pi-1>>
294
The Table
Construct a table
•for storing the cost of opt sol and bird’s advice.
•for each sub-instance.
Map
<V’:<v1,p1>,<v2,p2>,......,<vi,pi>>.
Sub-Instances
Indexes
Cell of table
 V’  [0..V]
 i  [0..n]
“Which of first i objects
to put in a knapsack of size v’?”
v’
i
295
The Table
The complete set S of sub-Instances.
<V’:<v1,p1>,<v2,p2>,......,<vi,pi>>.
OptSol Cost
0
&
1
Bird’s Advice
2
for this
sub-Instance
No i-1
Yes
i
Our cost? n
0 1 2
V’-vi
 V’  [0..V]
 i  [0..n]
V’
V
same + pi
same
Take best
of best.
296
The Table
The complete set S of sub-Instances.
<V’:<v1,p1>,<v2,p2>,......,<vi,pi>>.
OptSol Cost
0
&
1
Bird’s Advice
2
for this
sub-Instance
i-1
Order to fill
i
so nobody
waits?
n
0 1 2
V’-vi
 V’  [0..V]
 i  [0..n]
V’
V
297
The Code
298
299
300
Running Time
My instance:
<V:<v1,p1>,<v2,p2>,..........................,<vn,pn>>.
The complete set S of sub-Instances is  V’  [0..V]
<V’:<v1,p1>,<v2,p2>,......,<vi,pi>>.
 i  [0..n]
No
Yes
Running time
= ( # of sub-instances × # bird answers )
= (
Vn
×
2
)
= ( 2#bits in V × n )
Polynomial?
Exponential in “size” in instance!
301
The Knapsack Problem
•Dynamic Programming
Running time = ( V × n )
= ( 2#bits in V × n )
•Poly time if size of knapsack is small
•Exponential time if size is an arbitrary integer.
302
The Knapsack Problem
•Dynamic Programming
Running time = ( V × n )
= ( 2#bits in V × n )
•NP-Complete
If there is a poly-time algorithm
for the Knapsack Problem
For EVERY optimization problem
there is a poly-time algorithm.
303
The Knapsack Problem
•Dynamic Programming
Running time = ( V × n )
= ( 2#bits in V × n )
•NP-Complete
Likely there is not a poly-time algorithm
for the Knapsack Problem.
Likely there is not a poly-time algorithm
for EVERY optimization problem.
304
The Knapsack Problem
•Dynamic Programming
Running time = ( V × n )
= ( 2#bits in V × n )
•NP-Complete
•Approximate Algorithm
•In poly-time, solution can be found
that is (1+) as good as optimal.
done
305
The Job/Event Scheduling Problem
Schedule as
many events
in your room
as possible
306
The Job/Event Scheduling Problem
Ingredients:
•Instances: Events with starting and finishing times
<<s1,f1>,<s2,f2>,… ,<sn,fn>>.
•Solutions: A set of events that do not overlap.
•Cost of Solution: The number of events scheduled.
•Goal: Given a set of events, schedule as many as
possible.
307
Greedy Algorithm
Greedy Criteria: Earliest Finishing Time
Motivation: Schedule the event which will
free up your room for someone
else as soon as possible.
308
Weighted Event Scheduling
Ingredients:
•Instances: Events with starting and finishing times
and weights
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
•Solutions: A set of events that do not overlap.
•Cost of Solution: Total weight of events scheduled.
•Goal: Given a set of events, schedule max weight
309
Greedy Algorithm
1
100
1
Greedy Criteria: Earliest Finishing Time
Motivation: Schedule the event which will
free up your room for someone
else as soon as possible.
310
Bird & Friend Algorithm
An instance:
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
A solution:
<<s5,f5,w5>,<s9,f9,w9>,… ,<s82,f82,w82>>.
I ask the bird:
“What is the last event to take?”
# of answers K = n
311
Bird & Friend Algorithm
An instance:
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
A solution:
<<s5,f5,w5>,<s9,f9,w9>,… ,<s82,f82,w82>>.
I ask the bird:
“Do we keep the last event?”
# of answers K = 2 Yes & No
312
Bird & Friend Algorithm
An instance:
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
I ask the bird:
“Do we keep the last event?”
She answers: No
I ask my friend:
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
His solution:
<<s5,f5,w5>,<s9,f9,w9>,… ,<s82,f82,w82>>.
My solution: same
My cost: same
313
Bird & Friend Algorithm
An instance:
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
I ask the bird:
“Do we keep the last event?”
She answers: Yes
Carefull
I ask my friend:
<<s1,f1,w1>,<s2,f2,w2>,… ,<sn,fn,wn>>.
His solution:
<<s5,f5,w5>,<s9,f9,w9>,… ,<s82,f82,w82>>.
My solution: same + <sn,fn,wn>. My cost:same +w
314n
Bird & Friend Algorithm
last event
Bird answers:
“Yes keep the last event.”
Give the rest to my friend.
No this solution
is not valid!
Here is my best subsolution.
I add to his subsolution the bird’s answer.
315
Bird & Friend Algorithm
Bird answers:
No we trust the
“Yes keep the last event.”
bird!
Then I should politely tell the bird
she is wrong
316
Bird & Friend Algorithm
Bird answers:
“Yes keep the last event.”
No we trust the
bird!
You only tell her she is wrong
if you really know.
Eg k words don’t fit on the last line
The bear does not fit into the knapsack
317
Bird & Friend Algorithm
Bird answers:
“Yes keep the last event.”
No we trust the
bird!
Your friend could have just as easily
given you this subsolution that does not
conflict with the bird’s answer.
318
Bird & Friend Algorithm
Bird answers:
“Yes keep the last event.”
No we trust the
bird!
Or maybe he needs to make a sacrifice
when finding his answer in order that the
overall solution is the best.
319
Bird & Friend Algorithm
Bird answers:
“Yes keep the last event.”
No we trust the
bird!
Or goal now is to find the best solution to
our instance that is consistent with the
bird’s answer.
Then we will take the best of best.
320
Bird & Friend Algorithm
Bird answers:
“Yes keep the last event.”
No we trust the
bird!
Dude! It is your job to give me the right
subinstance so that I give you a
subsolution that does not conflict with the
bird
321
Bird & Friend Algorithm
My instance:
last event
Bird answers:
“Yes keep the last event.”
Cant keep any events that overlap with it.
322
Bird & Friend Algorithm
My instance:
last event
Bird answers:
“Yes keep the last event.”
I ask my friend:
323
Bird & Friend Algorithm
His instance:
His solution
Bird answers:
“Yes keep the last event.”
I ask my friend:
324
Bird & Friend Algorithm
My instance:
My solution:
Bird answers:
“Yes keep the last event.”
I ask my friend:
My solution: same + <sn,fn,wn>. Valid?
Yes
325
Bird & Friend Algorithm
My instance:
My solution:
Bird answers:
“Yes keep the last event.”
I ask my friend:
My solution: same + <sn,fn,wn>. My cost: same +wn
326
Same as Brute Force Algorithm
I try each of 2 bird ans.
My friends tries 2
His friends tries 2
Time?
Same as the brute force algorithm
that tries each solution.
327
Memorization
Assign one friend to each sub-instance.
“Which is the best path from vi to t?”  i
328
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
•Imagine running the recursive algorithm on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
329
Every subset of {1,…,9}
is a possible sub-Instance.
I.e. could be an exponential
number of them.
Hence, running time
is exponential.
Greedy algorithm sorted
jobs by finishing time.
Let us do that too.
330
Each sub-Instance is prefix.
I.e. only n of them.
Hence, running time
is polynomial!
331
Set of Sub-Instances
My instance:
<<s1,f1,w1>,<s2,f2,w2>,................… ,<sn,fn,wn>>.
Guess the complete set S of sub-Instances.
<<s1,f1,w1>,<s2,f2,w2>,… ,<si,fi,wi>>  i  [0..n]
The set S of sub-Instances needs to:
•include our given I
Yes: i = n
•closed under “friend” operation
 sub-Instance  S  subsub-Instance  S ?
•each sub-Instance needs to be
asked of some friend, friend… Only n sub-Instances
Good enough. 332
Set of Sub-Instances
Show closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
sub-Instance =
last event
<<s1,f1,w1>,<s2,f2,w2>,................................,<si,fi,wi>>
Events sorted by finishing time.
333
Set of Sub-Instances
Show closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
sub-Instance =
last event
<<s1,f1,w1>,<s2,f2,w2>,................................,<si,fi,wi>>
Bird answers:
“Yes keep the last event.”
Delete overlapping events for friend.
334
Set of Sub-Instances
Show closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
subsub-Instance =
<<s1,f1,w1>,<s2,f2,w2>,.....,<sj,fj,wj>>
Bird answers:
“Yes keep the last event.”
Delete overlapping events for friend.
335
Set of Sub-Instances
Show closed under “friend” operation
 sub-Instance  S  subsub-Instance  S
subsub-Instance =
typical kept job
typical deleted job
<<s1,f1,w1>,<s2,f2,w2>,.....,<sj,fj,wj>>
Event j is kept
fj  si
 set of kept jobs is a prefix of events.
 subsub-Instance  S
336
Set of Sub-Instances
My instance:
<<s1,f1,w1>,<s2,f2,w2>,................… ,<sn,fn,wn>>.
The complete set S of sub-Instances is
<<s1,f1,w1>,<s2,f2,w2>,… ,<si,fi,wi>>  i  [0..n]
Table:
0, 1,
Base case
2,
3, 4, ….
n
Original
337
Set of Sub-Instances
My instance:
<<s1,f1,w1>,<s2,f2,w2>,................… ,<sn,fn,wn>>.
The complete set S of sub-Instances is
<<s1,f1,w1>,<s2,f2,w2>,… ,<si,fi,wi>>  i  [0..n]
Running time
= # of sub-instances × # bird answers
=
n
×
2
But to find your friend’s “yes” sub-instance
you must know how many events overlap
with your last event. This takes time:
O(logn) using binary search
Done
338
for a total of O(nlogn) time.
Parsing
Input: s=6*8+((2+42)*(5+12)+987*7*123+15*54)
Output:
339
Parsing
Recursive Alg:
•GetExp calls GetTerm
•GetTerm calls GetFact
•GetFact calls GetExp
340
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
B  TA
 BC
b
e
C  CB
 AC
c
d
Context Free Grammar
(Not look ahead one)
For ease, we will assume every non-terminal
either goes to two non-terminals
or to one terminal
341
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
start non-terminal = T
string to parse
= a1a2a3 ..... an
= baeaadbda
B  TA
 BC
b
e
T
A
C
C
A
A
Output: A parsing
C  CB
 AC
c
d
A
C B
A C
B
T
C A
T
A B
b a e a a d b b d a342
B
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
B  TA
 BC
b
e
C  CB
 AC
c
d
T
C
A
Recursive Algorithm:
C
GetT does not know
B T
A
whether to call
A
B
C
A
A
C
GetA, GetC, or GetT.
A C
B T
A B
b a e a a d b b d a343
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
Ask Little Bird:
•For first rule
Ask Friend
•Parse left
Ask Another Friend
•Parse right.
B  TA
 BC
b
e
C  CB
 AC
c
d
T
A
C
C
A
A
A
C B
A C
B
T
C A
T
A B
b a e a a d b b d a344
B
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
Ask Little Bird:
•For first rule
Instance to give Friend
B  TA
 BC
b
e
C  CB
 AC
c
d
T
C
A
•?
b a e a a d b b d a345
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
B  TA
 BC
b
e
C  CB
 AC
c
d
T
Ask Little Bird:
A
C
•For first rule
C
A
Want from Friend:
•Left sub-parse tree.
A
A C B
Instance to give him:
T
A C
B
•C  baeaadb
A B
b a e a a d b b d a346
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
B  TA
 BC
b
e
C  CB
 AC
c
d
T
Ask Little Bird:
A
C
•For first rule
C
B T
A
How can we know split?
•Ask the Bird!
C A
A
A C B
A C
B T
A B
b a e a a d b b d a347
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
B  TA
 BC
b
e
C  CB
 AC
c
d
T
Ask Little Bird:
A
C
•For first rule
# of ans K = mT = # of rules for T.
•For the split.
# of ans K = n = # chars in string.
Total # of ans K = mT × n.
b a e a a d b b d a348
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
B  TA
 BC
b
e
C  CB
 AC
c
d
T
Ask left friend:
•Instance: C  baeaadb
•Solution: Left parsing
A
C
C
A
A
A
C B
A C
T
A B
b a e a a d b b d a349
B
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
Ask right friend:
•Instance: A  bda
•Solution: Right parsing
B  TA
 BC
b
e
C  CB
 AC
c
d
T
C
A
B
T
C A
b a e a a d b b d a350
Parsing
T  AB
 CA
 TT
A  AA
 BT
a
Input: T  a1a2a3 ..... an
B  TA
 BC
b
e
C  CB
 AC
c
d
T
A
Combine:
C
•Instance:
C
T
B
A
•Bird’s Answer
•Left Friend’s Answer A
C A
A C B
•Right Friend’s Answer
A C
B T
A B
b a e a a d b b d a351
352
Same as Brute Force Algorithm
I try each of 2 bird ans.
My friends tries 2
His friends tries 2
Time?
Same as the brute force algorithm
that tries each solution.
353
Memoization
Assign one friend to each sub-instance.
“Which is the best path from vi to t?”  i
354
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
•Imagine running the recursive algorithm on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
355
Set of Sub-Instances
Determine the complete set of sub-Instances.
My instance I:
T  a1a2a3 ..... an
T
gives:
A
C
My left sub-Instance.
A
C
gives:
His right sub-Instance.
b a e a a d b b d a356
Set of Sub-Instances
Determine the complete set of sub-Instances.
My instance I:
T  a1a2a3 ..... an
sub-Instances:
T’  aiai+1 ..... aj
•  non-terminals T’
•  i,j  [1,n]
T’= C
# of sub-Instances = # of non-terminals × n2
a1...ai-1 aiai+1...aj aj+1...an
a d b
357
The Table
Construct a table
•for storing the cost of opt sol and bird’s advice.
•for each sub-instance.
Map
Sub-Instances
T’  aiai+1 ..... aj
T’  i,j  [1,n]
Indexes
T’
i
Cell of table
j
358
359
Running Time
Running time
= ( # of sub-instances
× # bird answers )
= ( # of non-terminals × n2 × # of rules · n )
sub-Instances:
T’  aiai+1 ..... aj  non-terminals T’
gives: First rule and split
&  i,j  [1,n]
Done
360
Find a Satisfying Assignment
An instance (input) consists of a circuit:
c = (x3 or x5 or x6) and (x2 or x5 or x7) and (x3 or x4)
true
true
true
true
true
false true
true
A solution is an assignment of the variables.
x1 = 0, x2 = 1, x3 = 0, x4 = 0, x5 = 1, x6 = 0, x7 = 1
The cost of a solution is
• 1 if the assignment satisfies the circuit.
• 0 if not.
The goal is to find satisfying assignment.
361
Find a Satisfying Assignment
Instance:
c = (x3 or x5 or x6) and (x2 or x5 or x7) and (x3 or x4)
Ask the little bird
Value of x1 in an optimal solution
or even better
Value of x3 in an optimal solution
We will have to try both x3 = 0 and x3 = 1.
For now, suppose she answered x3 = 0.
362
Find a Satisfying Assignment
Instance:
c = (x3 or x5 or x6) and (x2 or x5 or x7) and (x3 or x4)
true
false
true
Commit to x3 = 0 and simplify
Sub-Instance:
c=
(x2 or x5 or x7) and x4
Friend gives Sub-Solution:
x1 = 0, x2 = 1,
x4 = 0, x5 = 1, x6 = 0, x7 = 1
Our Solution:
x1 = 0, x2 = 1, x3 = 0, x4 = 0, x5 = 1, x6 = 0, x7 = 1
363
Speeding Up the Time
x3
x2
x1
0
1
0
0
1
1
0
x1
x1 x2
0
1
x2
1
1
0
1
0
In the end, some friend looks
at each of the 2n assignments,
364
Memoization
Assign one friend to each sub-instance.
“Which is the best path from vi to t?”  i
365
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
•Imagine running the recursive algorithm on it.
•Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
366
Set of Sub-Instances
Determine the complete set of sub-Instances.
Given an instance I,
c = (x1 or y1) and (x2 or y2) and (x3 or y3) and (x4 or y4)
Is this a sub-Instance?
c = (x1)
and (x3)
Commit to
y1=0,
y2=1,
y3=0,
y4=1,
and simplify
Yes
True for any subset of the xi.
 could be an exponential # of different sub-Instances.
 running time is exponential.
367
Speeding Up the Time
x3
x2
x1
0
1
0
0
1
1
0
x1
x1 x2
0
1
x2
1
1
0
1
0
In the end, some friend looks
at each of the 2n assignments,
368
Speeding Up the Time
x3
x2
x1
0
1
0
0
1
1
0
x1
x1 x2
0
1
x2
1
1
0
1
0
But sometimes we can prune off branches.
369
Find a Satisfying Assignment
Instance:
c = (x2 or x5 or x7) and x3
x3 is forced to x3 = 0
x3
x2
x1
0
1
0
0
1
1
0
x1
x1 x2
0
1
x2
1
1
0
1
0
370
Find a Satisfying Assignment
Instance:
c = (x2 or x5 or x7) and x3 and x3
This is trivially unsatisfiable
because x3 can’t be both 0 and 1.
x3
x2
x1
0
1
0
0
1
1
0
x1
x1 x2
0
1
x2
1
1
0
1
0
371
372
373
Review
Designing Recursive Back Tracking Algorithm
•What are instances, solutions, and costs?
•Given an instance I,
•What question do you ask the little bird?
•Given a bird answer k  [K],
•What sub-Instance do your give your friend?
•Assume he gives you optSubSol for sub-Instance.
•How do you produce an optSol for I from
•the bird’s k and
•the friend’s optSubSol?
•How do you determine the cost of optSol from
•the bird’s k and
•the cost of the friend’s optSubSol?
•Try all bird’s answers and take best of best.
374
Review Recursive Back Tracking Algorithm
Dynamic Programming Algorithm
•Given an instance I,
•Imagine running the recursive alg on it.
• Determine the complete set of sub-Instances
ever given to you, your friends, their friends, …
• Build a table indexed by these sub-Instances
•Fill in the table in order so that nobody waits.
•the cost of its optimal solution
•advice given by the bird
•Run the recursive algorithm with bird’s advice to
find the solution to your instance.
375
Optimization Problems
• Don’t mix up the following
–
–
–
–
–
What is an instance
What are the objects in an instance
What is a solution
What are the objects in a solution
What is the cost of a solution
• Greedy algorithm
– What does the algorithm do & know
– What does the Prover do & know
– What does the Fairy God Mother do & know
• Recursive Backtracking / Dynamic Programming
– What does the algorithm do & know
– What does the little bird do & know
– What does the friend do & know
376
Dynamic Programming
Don’ts
•Yes, the code has a basic structure that you should learn.
•But don’t copy other code verbatim
•Don’t say if(ai = cj)
(i.e. Longest Common Subsequence)
when our problem does not have cj
377
Dynamic Programming
Don’ts
•When looping over the sub-instances
•be clear what the set of sub-instances are
•which is currently being solved,
i.e. which instance is cost(i,j)?
•If you know that the set of sub-instances are the
prefixes of the input, i.e. <a1,a2, …, ai>,
then don’t have a two dimensional table.
Table[1..n,1..n].
•Don’t loop over i and loop over j if j never gets
mentioned again.
378
Dynamic Programming
Don’ts
•When trying all bird answers
•be clear what the set of bird answers are,
•which is currently being tried,
•& what it says about the solution being looked for.
•When getting help from your friend,
•be clear what the sub-instance is that you are giving him
•How do you use the current instance and the bird' s
answer to form his sub-instance?
•Don’t simply say cost(i-1,j-1)
379
Dynamic Programming
Don’ts
•Think about what the base cases should be.
•Don’t make an instance a base cases
if they can be solved using the general method.
•% is used to start a comment.
Don’t put it in front of code.
380
The Question For the Little Bird
•Eg. The Best Binary Search Tree problem,
–“Which key is at the root of the tree?”
38
25
51
17
4
31
21
28
42
35
40
63
49
55
71
• If a solution is a binary tree of objects,
–“What object is at the root of the tree?”
381
Matrix Multiplication
382
383
384
385
386
All Pairs Shortest Paths
387
388
389
390
391
392
end
393
Dynamic Programming
Construct a table
•for storing an optimal solution & cost
•for each sub-instance.
Map
 i “Best path from vi to t?”
Sub-Instances
 i ϵ [n], i.e. for each node vi
Index
Cell of table
“Which is the best path from vi to t?”
t,
v , v , vi , v , ….,
s
394