CS 3343: Analysis of
Algorithms
Review for final
7/24/2016
1
Final Exam
• Closed book exam
• Coverage: the whole semester
• Cheat sheet: you are allowed one lettersize sheet, both sides
• Monday, May 6, 10:30am – 1:00pm
• Basic calculator (no graphing) allowed
7/24/2016
2
Final Exam: Study Tips
• Study tips:
– Study each lecture
– Study the homework and homework solutions
– Study the midterm exams
• Re-make your previous cheat sheets
7/24/2016
3
Topics covered (1)
By reversed chronological order:
• Graph algorithms
– Representations
– MST (Prim’s, Kruskal’s)
– Shortest path (Dijkstra’s)
– Running time analysis with different
implementations
7/24/2016
4
Topics covered (2)
• Dynamic programming
–
–
–
–
–
LCS
Restaurant location problem
Shortest path problem on a grid
Other problems
How to define recurrence solution, and use dynamic
programming to solve it
• Greedy algorithm
– Unit-profit restaurant location problem
– Fractional knapsack problem
– Prim’s, Kruskal’s, and Dijkstra’s are also examples of greedy
algorithms
– How to prove that certain greedy choices are optimal
7/24/2016
5
Topics covered (3)
• Hash tables
– Division-based hash functions
– Multiplication-based hash functions
– A universal hash function family
• Binary heap and priority queue
– Heapify, buildheap, insert, exatractMax,
changeKey
– Running time
7/24/2016
6
Topics covered (4)
• Sorting algorithms
–
–
–
–
–
–
–
–
Insertion sort
Merge sort
Quick sort
Heap sort
Linear time sorting: counting sort, radix sort
Stability of sorting algorithms
Worst-case and expected running time analysis
Memory requirement of sorting algorithms
• Order statistics
– Rand-Select
– Worst-case Linear-time selection
– Running time analysis
7/24/2016
7
Topics covered (5)
• Analysis
– Order of growth
– Asymptotic notation, basic definition
• Limit method
• L’ Hopital’s rule
• Stirling’s formula
– Best case, worst case, average case
• Analyzing non-recursive algorithms
– Arithmetic series
– Geometric series
• Analyzing recursive algorithms
– Defining recurrence
– Solving recurrence
• Recursion tree (iteration) method
• Substitution method
• Master theorem
7/24/2016
8
Review for finals
• In chronological order
• Only the more important concepts
– Very likely to appear in your final
• Does not mean to be exclusive
7/24/2016
9
Asymptotic notations
•
•
•
•
•
•
O: Big-Oh
Ω: Big-Omega
Θ: Theta
o: Small-oh
ω: Small-omega
Intuitively:
O is like 
o is like <
7/24/2016
 is like 
 is like >
 is like =
10
Big-Oh
• Math:
– O(g(n)) = {f(n):  positive constants c and n0
such that 0 ≤ f(n) ≤ cg(n)  n>n0}
– Or: lim n→∞ g(n)/f(n) > 0 (if the limit exists.)
• Engineering:
– g(n) grows at least as faster as f(n)
– g(n) is an asymptotic upper bound of f(n)
• Intuitively it is like f(n) ≤ g(n)
7/24/2016
11
Big-Oh
• Claim: f(n) = 3n2 + 10n + 5  O(n2)
• Proof:
3n2 + 10n + 5  10n2 + 10n + 10
 3 x 10 n2 for n  1
Therefore,
• Let c = 30 and n0 = 1
• f(n)  c n2,  n ≥ n0
7/24/2016
12
Big-Omega
• Math:
– Ω(g(n)) = {f(n):  positive constants c and n0
such that 0 ≤ cg(n) ≤ f(n)  n>n0}
– Or: lim n→∞ f(n)/g(n) > 0 (if the limit exists.)
• Engineering:
– f(n) grows at least as faster as g(n)
– g(n) is an asymptotic lower bound of f(n)
• Intuitively it is like g(n) ≤ f(n)
7/24/2016
13
Big-Omega
• f(n) = n2 / 10 = Ω(n)
• Proof: f(n) = n2 / 10, g(n) = n
– g(n) = n ≤ n2 / 10 = f(n) when n ≥ 10
– Therefore, c = 1 and n0 = 10
7/24/2016
14
Theta
• Math:
– Θ(g(n)) = {f(n):  positive constants c1, c2, and n0
such that c1 g(n)  f(n)  c2 g(n)  n  n0  n>n0}
– Or: lim n→∞ f(n)/g(n) = c > 0 and c < ∞
– Or: f(n) = O(g(n)) and f(n) = Ω(g(n))
• Engineering:
– f(n) grows in the same order as g(n)
– g(n) is an asymptotic tight bound of f(n)
• Intuitively it is like f(n) = g(n)
• Θ(1) means constant time.
7/24/2016
15
Theta
• Claim: f(n) = 2n2 + n = Θ (n2)
• Proof:
– We just need to find the three constants c1, c2,
and n0 such that
– c1n2 ≤ 2n2+n ≤ c2n2 for all n > n0
– A simple solution is c1 = 2, c2 = 3, and n0 = 1
7/24/2016
16
Using limits to compare orders of
growth
f(n)  o(g(n))
• lim f(n) / g(n) =
n→∞
0
c >0
∞
f(n)  O(g(n))
f(n)  Θ (g(n))
f(n)  Ω(g(n))
f(n)  ω (g(n))
7/24/2016
17
• Compare 2n and 3n
• lim 2n / 3n = lim(2/3)n = 0
n→∞
n→∞
• Therefore, 2n  o(3n), and 3n  ω(2n)
7/24/2016
18
L’ Hopital’s rule
lim f(n) / g(n) = lim f(n)’ / g(n)’
n→∞
7/24/2016
n→∞
If both lim f(n) and
lim g(n) goes to ∞
19
• Compare n0.5 and log n
• lim n0.5 / log n = ?
n→∞
•
•
•
•
7/24/2016
(n0.5)’ = 0.5 n-0.5
(log n)’ = 1 / n
lim (n-0.5 / 1/n) = lim(n0.5) = ∞
Therefore, log n  o(n0.5)
20
Stirling’s formula
n
n
n 1 / 2  n
n!  2 n    2 n
e
e
n! 
7/24/2016
(constant)
n
n 1/ 2  n
e
21
• Compare 2n and n!
n
n!
c nn
 n 
lim n  lim n n  lim c n    
n  2
n  2 e
n 
 2e 
n
• Therefore, 2n = o(n!)
7/24/2016
22
More advanced dominance ranking
7/24/2016
23
General plan for analyzing time
efficiency of a non-recursive algorithm
• Decide parameter (input size)
• Identify most executed line (basic operation)
• worst-case = average-case?
• T(n) = i ti
• T(n) = Θ (f(n))
7/24/2016
24
Analysis of insertion Sort
Statement
InsertionSort(A, n) {
for j = 2 to n {
key = A[j]
i = j - 1;
while (i > 0) and (A[i] > key) {
A[i+1] = A[i]
i = i - 1
}
A[i+1] = key
}
}
7/24/2016
cost time__
c1
c2
c3
c4
c5
c6
0
c7
0
n
(n-1)
(n-1)
S
(S-(n-1))
(S-(n-1))
(n-1)
25
Best case
Inner loop stops when A[i] <= key, or i = 0
i j
1
sorted
Key
• Array already sorted
n
S  1  n  (n)
j 1
7/24/2016
26
Worst case
Inner loop stops when A[i] <= key
i j
1
sorted
Key
• Array originally in reverse order
n(n  1)
2
S   j  1  2  ...  n 
 (n )
2
j 1
n
7/24/2016
27
Average case
Inner loop stops when A[i] <= key
i j
1
sorted
Key
• Array in random order
j 1 n
n(n  1)
E(S )     j 
 (n 2 )
2 j 1
4
j 1 2
n
7/24/2016
28
Find the order of growth for sums
n
T (n)   j  (n )
2
n
T ( n )   log( j )  ?
j 1
j 1
n
T (n)   2 j  ?
j 1
n
n
T (n)   j  ?
j 1 2
...
• How to find out the actual order of growth?
– Remember some formulas
– Learn how to guess and prove
7/24/2016
29
Arithmetic series
• An arithmetic series is a sequence of numbers
such that the difference of any two successive
members of the sequence is a constant.
e.g.: 1, 2, 3, 4, 5
or 10, 12, 14, 16, 18, 20
• In general:
a j  a j 1  d
Or:
7/24/2016
a j  a1  ( j  1)d
Recursive definition
Closed form, or explicit formula
30
Sum of arithmetic series
If a1, a2, …, an is an arithmetic series, then
n( a1  an )
ai 

2
i 1
n
7/24/2016
31
Geometric series
• A geometric series is a sequence of numbers
such that the ratio between any two successive
members of the sequence is a constant.
e.g.: 1, 2, 4, 8, 16, 32
or 10, 20, 40, 80, 160
or 1, ½, ¼, 1/8, 1/16
• In general:
a j  ra j 1
j 1
Or:
7/24/2016
a j  r a0
Recursive definition
Closed form, or explicit formula
32
Sum of geometric series
(1  r n 1 ) /(1  r )
n
 n 1
i
r  ( r  1) /( r  1)

i 0

n 1

if r < 1
if r > 1
if r = 1
n 1
2
1
i
n 1
n 1
2


2

1

2

2 1
i 0
n
n
n
1
1
i
1
lim n  i  lim n  ( 2 ) 
2
1  12
i 0 2
i 0
n
n
1
0
0
lim n  i  lim n   1 2    1 2   2  1  1
i 1 2
i 0
7/24/2016
33
Important formulas
3
n
i 2   ( n 3 )

3
i 1
n
n
1  n  (n)
i 1
n
i 
i 1
n(n  1)
 (n 2 )
2
 1  (1) ( r  1)
r 


n

(
r
) ( r  1)
r

1
i 0

n
i
r
n 1
n k 1
k 1
i



(
n
)

k 1
i 1
n
k
n
i
n 1
n
i
2

(
n

1
)
2

2


(
n
2
)

i 1
n
1
 (lg n )

i 1 i
n
 lg i  (n lg n)
i 1
Remember them, or remember where to find them!
7/24/2016
34
Sum manipulation rules
 (a  b )   a   b
 ca  c a
i
i
i
i
n
i
i i
i i
i i
x
n
a  a  a
i m
i
i m
i
i  x 1
i
Example:
n
n
n
i 1
i 1
i 1
i
i
n 1
(
4
i

2
)

4
i

2

2
n
(
n

1
)

2
2

 
n
n
n
1
 n i  n

i
i 1 2
i 1 2
7/24/2016
35
Recursive algorithms
• General idea:
– Divide a large problem into smaller ones
• By a constant ratio
• By a constant or some variable
– Solve each smaller one recursively or
explicitly
– Combine the solutions of smaller ones to form
a solution for the original problem
Divide and Conquer
7/24/2016
36
How to analyze the time-efficiency
of a recursive algorithm?
• Express the running time on input of size n
as a function of the running time on
smaller problems
7/24/2016
37
Analyzing merge sort
T(n)
MERGE-SORT A[1 . . n]
Θ(1)
1. If n = 1, done.
2T(n/2) 2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
f(n)
3. “Merge” the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) ,
but it turns out not to matter asymptotically.
7/24/2016
38
Analyzing merge sort
1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Merge two sorted subarrays
T(n) = 2 T(n/2) + f(n) +Θ(1)
# subproblems
subproblem size
7/24/2016
1.
What is the time for the base case?
2.
What is f(n)?
3.
What is the growth order of T(n)?
Work dividing and
Combining
Constant
39
Solving recurrence
• Running time of many algorithms can be
expressed in one of the following two
recursive forms
T ( n )  aT ( n  b)  f ( n )
or
T ( n )  aT ( n / b)  f ( n )
Challenge: how to solve the recurrence to get a closed
form, e.g. T(n) = Θ (n2) or T(n) = Θ(nlgn), or at least some
bound such as T(n) = O(n2)?
7/24/2016
40
Solving recurrence
1. Recurrence tree (iteration) method
- Good for guessing an answer
2. Substitution method
- Generic method, rigid, but may be hard
3. Master method
- Easy to learn, useful in limited cases only
- Some tricks may help in other cases
7/24/2016
41
The master method
The master method applies to recurrences of
the form
T(n) = a T(n/b) + f (n) ,
where a  1, b > 1, and f is asymptotically
positive.
1. Divide the problem into a subproblems, each of size n/b
2. Conquer the subproblems by solving them recursively.
3. Combine subproblem solutions
Divide + combine takes f(n) time.
7/24/2016
42
Master theorem
T(n) = a T(n/b) + f (n)
Key: compare f(n) with nlogba
CASE 1: f (n) = O(nlogba – e)  T(n) = (nlogba) .
CASE 2: f (n) = (nlogba)  T(n) = (nlogba log n)
.
CASE 3: f (n) = (nlogba + e) and a f (n/b)  c f (n)
 T(n) = ( f (n)) .
e.g.: merge sort: T(n) = 2 T(n/2) + Θ(n)
a = 2, b = 2  nlogba = n
 CASE 2  T(n) = Θ(n log n) .
7/24/2016
43
Case 1
Compare f (n) with nlogba:
f (n) = O(nlogba – e) for some constant e > 0.
: f (n) grows polynomially slower than nlogba (by
an ne factor).
Solution: T(n) = (nlogba) i.e., aT(n/b) dominates
e.g. T(n) = 2T(n/2) + 1
T(n) = 4 T(n/2) + n
T(n) = 2T(n/2) + log n
T(n) = 8T(n/2) + n2
7/24/2016
44
Case 3
Compare f (n) with nlogba:
f (n) =  (nlogba + e) for some constant e > 0.
: f (n) grows polynomially faster than nlogba (by an
ne factor).
Solution: T(n) = (f(n)) i.e., f(n) dominates
e.g. T(n) = T(n/2) + n
T(n) = 2 T(n/2) + n2
T(n) = 4T(n/2) + n3
T(n) = 8T(n/2) + n4
7/24/2016
45
Case 2
Compare f (n) with nlogba:
f (n) =  (nlogba).
: f (n) and nlogba grow at similar rate.
Solution: T(n) = (nlogba log n)
e.g. T(n) = T(n/2) + 1
T(n) = 2 T(n/2) + n
T(n) = 4T(n/2) + n2
T(n) = 8T(n/2) + n3
7/24/2016
46
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
7/24/2016
47
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
T(n)
7/24/2016
48
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
T(n/2)
7/24/2016
T(n/2)
49
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn/2
T(n/4)
7/24/2016
T(n/4)
dn/2
T(n/4)
T(n/4)
50
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn/2
dn/2
dn/4
dn/4
dn/4
dn/4
(1)
7/24/2016
51
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn/2
dn/2
h = log n dn/4
dn/4
dn/4
dn/4
(1)
7/24/2016
52
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn
dn/2
dn/2
h = log n dn/4
dn/4
dn/4
dn/4
(1)
7/24/2016
53
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn
dn/2
dn/2
h = log n dn/4
dn/4
dn/4
dn
dn/4
(1)
7/24/2016
54
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn
dn/2
dn/2
dn/4
dn/4
dn/4
dn
…
h = log n dn/4
dn
(1)
7/24/2016
55
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn
dn/2
dn/2
dn/4
dn/4
dn/4
dn
…
h = log n dn/4
dn
(1)
7/24/2016
#leaves = n
(n)
56
Recursion tree
Solve T(n) = 2T(n/2) + dn, where d > 0 is constant.
dn
dn
dn/2
dn/2
dn/4
dn/4
dn/4
dn
…
h = log n dn/4
dn
(1)
#leaves = n
(n)
Total (n log n)
7/24/2016
57
Substitution method
The most general method to solve a recurrence
(prove O and  separately):
1. Guess the form of the solution:
(e.g. using recursion trees, or expansion)
2. Verify by induction (inductive step).
7/24/2016
58
Proof by substitution
• Recurrence: T(n) = 2T(n/2) + n.
• Guess: T(n) = O(n log n). (eg. by recurrence tree
method)
• To prove, have to show T(n) ≤ c n log n for
some c > 0 and for all n > n0
• Proof by induction: assume it is true for T(n/2),
prove that it is also true for T(n). This means:
• Fact: T(n) = 2T(n/2) + n
• Assumption: T(n/2)≤ cn/2 log (n/2)
• Need to Prove: T(n)≤ c n log (n)
7/24/2016
59
Proof
• Fact: T(n) = 2T(n/2) + n
• Assumption: T(n/2)≤ cn/2 log (n/2)
• Need to Prove: T(n)≤ c n log (n)
• Proof:
Substitute T(n/2) into the recurrence function
=> T(n) = 2 T(n/2) + n ≤ cn log (n/2) + n
=> T(n) ≤ c n log n - c n + n
=> T(n) ≤ c n log n (if we choose c ≥ 1).
7/24/2016
60
Proof by substitution
• Recurrence: T(n) = 2T(n/2) + n.
• Guess: T(n) = Ω(n log n).
• To prove, have to show T(n) ≥ c n log n for
some c > 0 and for all n > n0
• Proof by induction: assume it is true for T(n/2),
prove that it is also true for T(n). This means:
• Fact: T(n) = 2T(n/2) + n
• Assumption:T(n/2) ≥ cn/2 log (n/2)
• Need to Prove: T(n) ≥ c n log (n)
7/24/2016
61
Proof
• Fact: T(n) = 2T(n/2) + n
• Assumption: T(n/2) ≥ cn/2 log (n/2)
• Need to Prove: T(n) ≥ c n log (n)
• Proof:
Substitute T(n/2) into the recurrence function
=> T(n) = 2 T(n/2) + n ≥ cn log (n/2) + n
=> T(n) ≥ c n log n - c n + n
=> T(n) ≥ c n log n (if we choose c ≤ 1).
7/24/2016
62
Quick sort
Quicksort an n-element array:
1. Divide: Partition the array into two subarrays
around a pivot x such that elements in lower
subarray  x  elements in upper subarray.
x
x
≥x
2. Conquer: Recursively sort the two subarrays.
3. Combine: Trivial.
Key: Linear-time partitioning subroutine.
7/24/2016
63
Partition
• All the action takes place in the
partition() function
– Rearranges the subarray in place
– End result: two subarrays
• All values in first subarray  all values in second
– Returns the index of the “pivot” element
separating the two subarrays
p
x
7/24/2016
r
q
x
≥x
64
Partition Code
Partition(A, p, r)
x = A[p];
// pivot is the first element
i = p;
j = r + 1;
while (TRUE) {
repeat
i++;
until A[i] > x or i >= j;
repeat
What is the running time of
j--;
until A[j] < x or j < i;
partition()?
if (i < j)
Swap (A[i], A[j]);
else
break;
partition() runs in O(n) time
}
swap (A[p], A[j]);
return j;
7/24/2016
65
p
x=6
6
r
10
5
8
13
3
2
11
i
6
j
10
5
8
13
3
i
6
2
2
5
8
13
3
2
5
8
13
5
2
5
p
7/24/2016
3
2
5
3
10 11
j
3
13
i
6
10 11
j
i
6
11
j
i
6
2
8
10 11
j
3
13
j
i
q
6
13
8
10 11
r
8
10 11
66
6
3
7/24/2016
10
2
5
8
11
3
2
5
6
11
8
13
10 13
2
3
5
6
10
8
11
13
2
3
5
6
8
10
11
13
2
3
5
6
8
10
11
13
67
Quicksort Runtimes
• Best case runtime Tbest(n)  O(n log n)
• Worst case runtime Tworst(n)  O(n2)
• Worse than mergesort? Why is it called
quicksort then?
• Its average runtime Tavg(n)  O(n log n )
• Better even, the expected runtime of
randomized quicksort is O(n log n)
7/24/2016
68
Randomized quicksort
• Randomly choose an element as pivot
– Every time need to do a partition, throw a die to
decide which element to use as the pivot
– Each element has 1/n probability to be selected
Partition(A, p, r)
d = random();
// a random number between 0 and 1
index = p + floor((r-p+1) * d); // p<=index<=r
swap(A[p], A[index]);
x = A[p];
i = p;
j = r + 1;
while (TRUE) {
…
}
7/24/2016
69
Running time of randomized
quicksort
T(n) =
T(0) + T(n–1) + dn
T(1) + T(n–2) + dn
M
T(n–1) + T(0) + dn
if 0 : n–1 split,
if 1 : n–2 split,
if n–1 : 0 split,
• The expected running time is an average
of all cases
Expectation
1 n 1
T ( n )   T ( k )  T ( n  k  1)   n  (n log n )
n k 0
7/24/2016
70
Heaps
• In practice, heaps are usually
implemented as arrays:
16
14
10
8
2
7
4
3
1
16 14 10 8
7/24/2016
9
7
9
3
2
4
1
71
Heaps
• To represent a complete binary tree as an array:
–
–
–
–
–
The root node is A[1]
Node i is A[i]
The parent of node i is A[i/2] (note: integer divide)
The left child of node i is A[2i]
The right child of node i is A[2i + 1]
16
14
A = 16 14 10 8
7
9
3
2
4
8
1 =
2
7/24/2016
10
7
4
9
3
1
72
The Heap Property
• Heaps also satisfy the heap property:
A[Parent(i)]  A[i]
for all nodes i > 1
– In other words, the value of a node is at most the
value of its parent
– The value of a node should be greater than or equal
to both its left and right children
• And all of its descendents
– Where is the largest element in a heap stored?
7/24/2016
73
Heap Operations: Heapify()
Heapify(A, i)
{ // precondition: subtrees rooted at l and r are heaps
l = Left(i); r = Right(i);
if (l <= heap_size(A) && A[l] > A[i])
largest = l;
Among A[l], A[i], A[r],
else
which one is largest?
largest = i;
if (r <= heap_size(A) && A[r] > A[largest])
largest = r;
if (largest != i) {
Swap(A, i, largest);
If violation, fix it.
Heapify(A, largest);
}
} // postcondition: subtree rooted at i is a heap
7/24/2016
74
Heapify() Example
16
4
10
14
2
7
8
3
1
A = 16 4 10 14 7
7/24/2016
9
9
3
2
8
1
75
Heapify() Example
16
4
10
14
2
7
8
3
1
A = 16 4 10 14 7
7/24/2016
9
9
3
2
8
1
76
Heapify() Example
16
4
10
14
2
7
8
3
1
A = 16 4 10 14 7
7/24/2016
9
9
3
2
8
1
77
Heapify() Example
16
14
10
4
2
7
8
3
1
A = 16 14 10 4
7/24/2016
9
7
9
3
2
8
1
78
Heapify() Example
16
14
10
4
2
7
8
3
1
A = 16 14 10 4
7/24/2016
9
7
9
3
2
8
1
79
Heapify() Example
16
14
10
8
2
7
4
3
1
A = 16 14 10 8
7/24/2016
9
7
9
3
2
4
1
80
Heapify() Example
16
14
10
8
2
7
4
3
1
A = 16 14 10 8
7/24/2016
9
7
9
3
2
4
1
81
Analyzing Heapify(): Formal
• T(n)  T(2n/3) + (1)
• By case 2 of the Master Theorem,
T(n) = O(lg n)
• Thus, Heapify() takes logarithmic time
7/24/2016
82
Heap Operations: BuildHeap()
• We can build a heap in a bottom-up manner by
running Heapify() on successive subarrays
– Fact: for array of length n, all elements in range
A[n/2 + 1 .. n] are heaps (Why?)
– So:
• Walk backwards through the array from n/2 to 1, calling
Heapify() on each node.
• Order of processing guarantees that the children of node i
are heaps when i is processed
7/24/2016
83
BuildHeap()
// given an unsorted array A, make A a heap
BuildHeap(A)
{
heap_size(A) = length(A);
for (i = length[A]/2 downto 1)
Heapify(A, i);
}
7/24/2016
84
BuildHeap() Example
• Work through example
A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}
4
1
3
2
14
7/24/2016
16
8
9
10
7
85
4
1
3
2
14
7/24/2016
16
8
9
10
7
86
4
1
3
14
2
7/24/2016
16
8
9
10
7
87
4
1
10
14
2
7/24/2016
16
8
9
3
7
88
4
16
10
14
2
7/24/2016
7
8
9
3
1
89
16
14
10
8
2
7/24/2016
7
4
9
3
1
90
Analyzing BuildHeap(): Tight
• To Heapify() a subtree takes O(h) time
where h is the height of the subtree
– h = O(lg m), m = # nodes in subtree
– The height of most subtrees is small
• Fact: an n-element heap has at most
n/2h+1 nodes of height h
• CLR 7.3 uses this fact to prove that
BuildHeap() takes O(n) time
7/24/2016
91
Heapsort Example
• Work through example
A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}
4
1
3
2
14
16
8
A=
7/24/2016
9
10
7
4
1
3
2 16 9 10 14 8
7
92
Heapsort Example
• First: build a heap
16
14
10
8
2
7
4
3
1
A = 16 14 10 8
7/24/2016
9
7
9
3
2
4
1
93
Heapsort Example
• Swap last and first
1
14
10
8
2
7
4
A=
7/24/2016
9
3
16
1 14 10 8
7
9
3
2
4 16
94
Heapsort Example
• Last element sorted
1
14
10
8
2
7
4
A=
7/24/2016
9
3
16
1 14 10 8
7
9
3
2
4 16
95
Heapsort Example
• Restore heap on remaining unsorted
elements
14
8
10
4
2
7
1
16
A = 14 8 10 4
7/24/2016
9
3
Heapify
7
9
3
2
1 16
96
Heapsort Example
• Repeat: swap new last and first
1
8
10
4
2
7
14
A=
7/24/2016
9
3
16
1
8 10 4
7
9
3
2 14 16
97
Heapsort Example
• Restore heap
10
8
9
4
2
7
14
3
16
A = 10 8
7/24/2016
1
9
4
7
1
3
2 14 16
98
Heapsort Example
• Repeat
9
8
3
4
10
7
14
A=
7/24/2016
1
2
16
9
8
3
4
7
1
2 10 14 16
99
Heapsort Example
• Repeat
8
7
3
4
10
2
14
A=
7/24/2016
1
9
16
8
7
3
4
2
1
9 10 14 16
100
Heapsort Example
• Repeat
1
2
3
4
10
7
14
A=
7/24/2016
8
9
16
1
2
3
4
7
8
9 10 14 16
101
Analyzing Heapsort
• The call to BuildHeap() takes O(n) time
• Each of the n - 1 calls to Heapify()
takes O(lg n) time
• Thus the total time taken by HeapSort()
= O(n) + (n - 1) O(lg n)
= O(n) + O(n lg n)
= O(n lg n)
7/24/2016
102
HeapExtractMax Example
16
14
10
8
2
7
4
3
1
A = 16 14 10 8
7/24/2016
9
7
9
3
2
4
1
103
HeapExtractMax Example
Swap first and last, then remove last
1
14
10
8
2
7
4
A=
7/24/2016
9
3
16
1 14 10 8
7
9
3
2
4
16
104
HeapExtractMax Example
Heapify
14
8
10
4
2
7
1
3
16
A = 14 8 10 4
7/24/2016
9
7
9
3
2
1
16
105
HeapChangeKey Example
Increase key
16
14
10
8
2
7
4
3
1
A = 16 14 10 8
7/24/2016
9
7
9
3
2
4
1
106
HeapChangeKey Example
Increase key
16
14
10
15
2
7
4
3
1
A = 16 14 10 15 7
7/24/2016
9
9
3
2
4
1
107
HeapChangeKey Example
Increase key
16
15
10
14
2
7
4
3
1
A = 16 15 10 14 7
7/24/2016
9
9
3
2
4
1
108
HeapInsert Example
HeapInsert(A, 17)
16
14
10
8
2
7
4
3
1
A = 16 14 10 8
7/24/2016
9
7
9
3
2
4
1
109
HeapInsert Example
HeapInsert(A, 17)
16
14
10
8
2
7
4
1
9
3
-∞
-∞ makes it a valid heap
A = 16 14 10 8
7/24/2016
7
9
3
2
4
1
-∞
110
HeapInsert Example
HeapInsert(A, 17)
16
14
10
8
2
7
4
1
9
3
17
Now call changeKey
A = 16 14 10 8
7/24/2016
7
9
3
2
4
1
17
111
HeapInsert Example
HeapInsert(A, 17)
17
16
10
8
2
14
4
1
9
7
A = 17 16 10 8 14 9
7/24/2016
3
3
2
4
1
7
112
• Heapify: Θ(log n)
• BuildHeap: Θ(n)
• HeapSort: Θ(nlog n)
•
•
•
•
7/24/2016
HeapMaximum: Θ(1)
HeapExtractMax: Θ(log n)
HeapChangeKey: Θ(log n)
HeapInsert: Θ(log n)
113
Counting sort
1. for i  1 to k
Initialize
do C[i]  0
Count
2. for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
Compute running sum
3.for i  2 to k
do C[i]  C[i] + C[i–1]
⊳ C[i] = |{key  i}|
4.for j  n downto 1
Re-arrange
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
7/24/2016
114
Counting sort
A:
1
2
3
4
5
4
1
3
4
3
B:
3. for i  2 to k
do C[i]  C[i] + C[i–1]
7/24/2016
1
2
3
4
C:
1
0
2
2
C':
1
1
3
5
⊳ C[i] = |{key  i}|
115
Loop 4: re-arrange
A:
B:
1
2
3
4
5
4
1
3
4
3
3
1
2
3
4
C:
1
1
3
5
C':
1
1
3
5
4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
7/24/2016
116
Analysis
(k)
(n)
(k)
(n)
1. for i  1 to k
do C[i]  0
2.for j  1 to n
do C[A[ j]]  C[A[ j]] + 1
3.for i  2 to k
do C[i]  C[i] + C[i–1]
4.for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
(n + k)
7/24/2016
117
Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.
A:
4
1
3
4
3
B:
1
3
3
4
4
Why this is important?
What other algorithms have this property?
7/24/2016
118
Radix sort
• Similar to sorting the address books
• Treat each digit as a key
• Start from the least significant bit
Most significant
Least significant
198099109123518183599
340199540380128115295
384700101594539614696
382408360201039258538
614386507628681328936
7/24/2016
119
Time complexity
• Sort each of the d digits by counting sort
• Total cost: d (n + k)
– k = 10
– Total cost: Θ(dn)
• Partition the d digits into groups of 3
– Total cost: (n+103)d/3
• We work with binaries rather than decimals
–
–
–
–
–
Partition a binary number into groups of r bits
Total cost: (n+2r)d/r
Choose r = log n
Total cost: dn / log n
Compare with dn log n
• Catch: faster than quicksort only when n is very large
7/24/2016
120
Randomized selection algorithm
RAND-SELECT(A, p, q, i) ⊳ ith smallest of A[ p . .
q]
if p = q & i > 1 then error!
r  RAND-PARTITION(A, p, q)
kr–p+1
⊳ k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT( A, p, r – 1, i )
else return RAND-SELECT(A, r + 1, q, i – k )
k
 A[r]
 A[r]
p
r
q
7/24/2016
121
Example
Select the i = 6th smallest:
7 10
pivot
Partition:
3 2
5
8
11
3
2
13
k=4
5 7 11
8
10 13
i=6
Select the 6 – 4 = 2nd smallest recursively.
7/24/2016
122
Complete example: select the 6th smallest element.
7
i=6
k=4
3
10
2
5
5
8
7
11
3
2
11
8
10
8
8
10
13
10 13
i=6–4=2
k=3
Note: here we always
used first element as
pivot to do the
partition (instead of
rand-partition).
11
13
i=2<k
k=2
i=2=k
7/24/2016
10
123
Intuition for analysis
(All our analyses today assume that all elements
are distinct.)
Lucky:
T(n) = T(9n/10) + (n)
nlog10 / 9 1  n0  1
CASE 3
= (n)
Unlucky:
arithmetic series
T(n) = T(n – 1) + (n)
= (n2)
Worse than sorting!
7/24/2016
124
Running time of randomized selection
T(n) ≤
T(max(0, n–1)) + n
T(max(1, n–2)) + n
M
T(max(n–1, 0)) + n
if 0 : n–1 split,
if 1 : n–2 split,
if n–1 : 0 split,
• For upper bound, assume ith element always falls in
larger side of partition
• The expected running time is an average of all cases
Expectation
7/24/2016
1 n 1
T (n )   T max( k , n  k  1)   n  (n )
n k 0
125
Worst-case linear-time selection
SELECT(i, n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (i–k)th
smallest element in the upper part
7/24/2016
Same as
RANDSELECT
126
Developing the recurrence
T(n)
SELECT(i, n)
(n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (i–k)th
smallest element in the upper part
T(n/5)
(n)
T(7n/10
+3)
7/24/2016
127
Solving the recurrence
1 
7



T (n )  T  n   T  n  3  n
5 
 10

Assumption: T(k)  ck for all k < n
T ( n )  c( n 5)  c(7n 10  3)  n
 cn 5  3cn / 4  n if n ≥ 60
 19cn / 20  n
 cn  ( cn / 20  n )
 cn if c ≥ 20 and n ≥ 60
7/24/2016
128
Hash tables
U
(universe of keys)
|U| >> K
&
|U| >> m
k2
0
h(k1)
h(k4)
k1
k4
K
(actual
keys)
T
k5
collision
k3
h(k2) = h(k5)
h(k3)
m-1
• Problem: collision
7/24/2016
129
Chaining
• Chaining puts elements that hash to the
same slot in a linked list:
U
(universe of keys)
k6
k2
k1
k4 ——
k5
k2
——
——
——
k1
k4
K
(actual
k7
keys)
T
——
k5
k7 ——
——
k8
k3
k3 ——
k8
k6 ——
——
7/24/2016
130
Hashing with Chaining
• Chained-Hash-Insert (T, x)
– Insert x at the head of list T[h(key[x])].
– Worst-case complexity – O(1).
• Chained-Hash-Delete (T, x)
– Delete x from the list T[h(key[x])].
– Worst-case complexity – proportional to length of list with
singly-linked lists. O(1) with doubly-linked lists.
• Chained-Hash-Search (T, k)
– Search an element with key k in list T[h(k)].
– Worst-case complexity – proportional to length of list.
7/24/2016
131
Analysis of Chaining
• Assume simple uniform hashing: each key in
table is equally likely to be hashed to any slot
• Given n keys and m slots in the table, the
load factor  = n/m = average # keys per slot
• Average cost of an unsuccessful search for a
key is (1+) (Theorem 11.1)
• Average cost of a successful search is
(2 + /2) = (1 + ) (Theorem 11.2)
• If the number of keys n is proportional to the
number of slots in the table,  = n/m = O(1)
– The expected cost of searching is constant if  is
constant
7/24/2016
132
Hash Functions:
The Division Method
• h(k) = k mod m
– In words: hash k into a table with m slots using the slot given by
the remainder of k divided by m
– Example: m = 31 and k = 78 => h(k) = 16.
• Advantage: fast
• Disadvantage: value of m is critical
– Bad if keys bear relation to m
– Or if hash does not depend on all bits of k
• Pick m = prime number not too close to power of 2 (or 10)
7/24/2016
133
Hash Functions:
The Multiplication Method
• For a constant A, 0 < A < 1:
• h(k) = m (kA mod 1) =  m (kA - kA) 
Fractional part of kA
•
•
•
•
•
•
7/24/2016
Advantage: Value of m is not critical
Disadvantage: relatively slower
Choose m = 2P, for easier implementation
Choose A not too close to 0 or 1
Knuth: Good choice for A = (5 - 1)/2
Example: m = 1024, k = 123, A  0.6180339887…
h(k) = 1024(123 · 0.6180339887 mod 1)
= 1024 · 0.018169...  = 18.
134
A Universal Hash Function
• Choose a prime number p that is larger than all possible
keys
• Choose table size m ≥ n
• Randomly choose two integers a, b, such that
1  a  p -1, and 0  b  p -1
• ha,b(k) = ((ak+b) mod p) mod m
• Example: p = 17, m = 6
h3,4 (8) = ((3*8 + 4) % 17) % 6 = 11 % 6 = 5
• With a random pair of parameters a, b, the chance of a
collision between x and y is at most 1/m
• Expected search time for any input is (1)
7/24/2016
135
Elements of dynamic programming
• Optimal sub-structures
– Optimal solutions to the original problem
contains optimal solutions to sub-problems
• Overlapping sub-problems
– Some sub-problems appear in many solutions
7/24/2016
136
Two steps to dynamic programming
• Formulate the solution as a recurrence
relation of solutions to subproblems.
• Specify an order to solve the subproblems
so you always have what you need.
7/24/2016
137
Optimal subpaths
• Claim: if a path startgoal is optimal, any sub-path,
startx, or xgoal, or xy, where x, y is on the optimal
path, is also the shortest.
• Proof by contradiction
– If the subpath between x and y is not the shortest, we can
replace it with the shorter one, which will reduce the total length
of the new path => the optimal path from start to goal is not the
shortest => contradiction!
– Hence, the subpath xy must be the shortest among all paths
from x to y
b
start
a
x
y
b’
7/24/2016
a + b + c is shortest
c
goal
b’ < b
a + b’ + c < a + b + c
138
Dynamic programming illustration
S
0
3
5
3
9
3
5
3
2
6
2
6
13
2
4
17
9
4
5
11
17
11
6
14
2
17
13
13
3
17
2
18
15
3
3
16
4
3
1
3
15
3
7
3
2
2
9
3
6
1
8
13
3
3
2
3
1
3
3
7
12
20
3
2
20
G
F(i-1, j) + dist(i-1, j, i, j)
F(i, j) = min
F(i, j-1) + dist(i, j-1, i, j)
7/24/2016
139
Trace back
0
3
5
9
3
5
3
2
6
2
6
13
2
4
17
9
4
8
11
17
11
6
14
5
17
2
13
2
13
2
17
3
18
16
4
3
1
3
15
3
7
3
15
3
9
3
2
13
3
3
6
1
1
3
2
3
12
3
3
7
7/24/2016
3
20
3
2
20
140
Longest Common Subsequence
• Given two sequences x[1 . . m] and y[1 . . n], find a
longest subsequence common to them both.
“a” not “the”
x: A
B
C
B
D
A
y: B
D
C
A
B
A
B
BCBA =
LCS(x, y)
functional notation,
but not a function
7/24/2016
141
Optimal substructure
• Notice that the LCS problem has optimal substructure:
parts of the final solution are solutions of subproblems.
– If z = LCS(x, y), then any prefix of z is an LCS of a prefix of
x and a prefix of y.
i
m
x
z
n
y
j
• Subproblems: “find LCS of pairs of prefixes of x and y”
7/24/2016
142
Finding length of LCS
m
x
n
y
• Let c[i, j] be the length of LCS(x[1..i], y[1..j])
=> c[m, n] is the length of LCS(x, y)
• If x[m] = y[n]
c[m, n] = c[m-1, n-1] + 1
• If x[m] != y[n]
c[m, n] = max { c[m-1, n], c[m, n-1] }
7/24/2016
143
DP Algorithm
• Key: find out the correct order to solve the sub-problems
• Total number of sub-problems: m * n
c[i, j] =
c[i–1, j–1] + 1
max{c[i–1, j], c[i, j–1]}
0
j
if x[i] = y[j],
otherwise.
n
0
i
C(i, j)
m
7/24/2016
144
LCS Example (0)
j
i
0
X[i]
1
A
2
B
3
C
4
B
0
Y[j]
1
B
2
D
3
C
4
A
ABCB
BDCAB
5
B
X = ABCB; m = |X| = 4
Y = BDCAB; n = |Y| = 5
Allocate array c[5,6]
7/24/2016
145
LCS Example (1)
j
i
0
Y[j]
1
B
2
D
3
C
4
A
B
0
0
0
0
0
0
X[i]
0
1
A
0
2
B
0
3
C
0
4
B
0
for i = 1 to m
for j = 1 to n
7/24/2016
ABCB
BDCAB
5
c[i,0] = 0
c[0,j] = 0
146
LCS Example (2)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
0
0
0
0
X[i]
0
0
1
A
0
0
2
B
0
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
147
LCS Example (3)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
0
0
X[i]
0
0
0
0
1
A
0
0
0
0
2
B
0
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
148
LCS Example (4)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
0
X[i]
0
0
0
0
0
1
A
0
0
0
0
1
2
B
0
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
149
LCS Example (5)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
150
LCS Example (6)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
1
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
151
LCS Example (7)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
1
1
1
1
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
152
LCS Example (8)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
1
1
1
1
2
3
C
0
4
B
0
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
153
LCS Example (14)
j
i
ABCB
BDCAB
5
0
Y[j]
1
B
2
D
3
C
4
A
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
1
1
1
1
2
3
C
0
1
1
2
2
2
4
B
0
1
1
2
2
3
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
7/24/2016
154
LCS Algorithm Running Time
• LCS algorithm calculates the values of each
entry of the array c[m,n]
• So what is the running time?
O(m*n)
since each c[i,j] is calculated in
constant time, and there are m*n
elements in the array
7/24/2016
155
How to find actual LCS
• The algorithm just found the length of LCS, but not LCS itself.
• How to find the actual LCS?
• For each c[i,j] we know how it was acquired:
if x[i]  y[ j ],
c[i  1, j  1]  1
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise
• A match happens only when the first equation is taken
• So we can start from c[m,n] and go backwards, remember
x[i] whenever c[i,j] = c[i-1, j-1]+1.
7/24/2016
2
2
2
3
For example, here
c[i,j] = c[i-1,j-1] +1 = 2+1=3
156
Finding LCS
j
0
Y[j]
1
B
2
D
3
C
4
A
5
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
1
1
1
1
2
3
C
0
1
1
2
2
2
4
B
0
1
1
2
2
3
i
Time for trace back: O(m+n).
7/24/2016
157
Finding LCS (2)
j
0
Y[j]
1
B
2
D
3
C
4
A
5
B
0
X[i]
0
0
0
0
0
0
1
A
0
0
0
0
1
1
2
B
0
1
1
1
1
2
3
C
0
1
1
2
2
2
4
B
0
1
1
2
2
3
i
7/24/2016
LCS (reversed order): B C B
LCS (straight order):
B C B
(this string turned out to be a palindrome)
158
LCS as a longest path problem
B
D
C
7/24/2016
1
1
1
C
B
B
1
A
B
A
1
1
159
LCS as a longest path problem
B
A
D
C
0
0
0
0
0
0
0
0
1
1
0
1
1
B
0
0
0
1
1
1
1
2
2
2
2
1
1
B
C
1
1
B
0
7/24/2016
A
1
1
1
1
1
2
3
160
Restaurant location problem 1
• You work in the fast food business
• Your company plans to open up new restaurants in
Texas along I-35
• Towns along the highway called t1, t2, …, tn
• Restaurants at ti has estimated annual profit pi
• No two restaurants can be located within 10 miles of
each other due to some regulation
• Your boss wants to maximize the total profit
• You want a big bonus
7/24/2016
10 mile
161
A DP algorithm
• Suppose you’ve already found the optimal solution
• It will either include tn or not include tn
• Case 1: tn not included in optimal solution
– Best solution same as best solution for t1 , …, tn-1
• Case 2: tn included in optimal solution
– Best solution is pn + best solution for t1 , …, tj , where j < n is the
largest index so that dist(tj, tn) ≥ 10
7/24/2016
162
Recurrence formulation
• Let S(i) be the total profit of the optimal solution when the
first i towns are considered (not necessarily selected)
– S(n) is the optimal solution to the complete problem
S(n) = max
S(n-1)
j < n & dist (tj, tn) ≥ 10
S(j) + pn
Generalize
S(i) = max
S(i-1)
j < i & dist (tj, ti) ≥ 10
S(j) + pi
Number of sub-problems: n. Boundary condition: S(0) = 0.
Dependency:
7/24/2016
S
j
i-1 i
163
Example
Distance (mi)
dummy 100
0
Profit (100k)
S(i)
5
2 2
6
7
6
3
6
3
6
7 9 8
3
6
7 9 9
10
10
7
4
3
12
2
4
12
5
12 12
14
26
26
Optimal: 26
S(i) = max
S(i-1)
S(j) + pi
j < i & dist (tj, ti) ≥ 10
• Natural greedy 1: 6 + 3 + 4 + 12 = 25
• Natural greedy 2: 12 + 9 + 3 = 24
7/24/2016
164
Complexity
• Time: (nk), where k is the maximum
number of towns that are within 10 miles
to the left of any town
– In the worst case, (n2)
– Can be improved to (n) with some
preprocessing tricks
• Memory: Θ(n)
7/24/2016
165
Knapsack problem
• Each item has a value and a weight
• Objective: maximize value
• Constraint: knapsack has a weight limitation
Three versions:
0-1 knapsack problem: take
each item or leave it
Fractional knapsack problem:
items are divisible
Unbounded knapsack problem:
unlimited supplies of each item.
Which one is easiest to solve?
We study the 0-1 problem today.
7/24/2016
166
Formal definition (0-1 problem)
• Knapsack has weight limit W
• Items labeled 1, 2, …, n (arbitrarily)
• Items have weights w1, w2, …, wn
– Assume all weights are integers
– For practical reason, only consider wi < W
• Items have values v1, v2, …, vn
• Objective: find a subset of items, S, such that
iS wi  W and iS vi is maximal among all such
(feasible) subsets
7/24/2016
167
A DP algorithm
• Suppose you’ve find the optimal solution S
• Case 1: item n is included
• Case 2: item n is not included
wn
Total weight limit:
W
Find an optimal solution using items
1, 2, …, n-1 with weight limit W - wn
7/24/2016
wn
Total weight limit:
W
Find an optimal solution using items
1, 2, …, n-1 with weight limit W
168
Recursive formulation
• Let V[i, w] be the optimal total value when items 1, 2, …, i
are considered for a knapsack with weight limit w
=> V[n, W] is the optimal solution
V[n-1, W-wn] + vn
V[n, W] = max
V[n-1, W]
Generalize
V[i, w] =
V[i-1, w-wi] + vi
item i is taken
V[i-1, w]
item i not taken
max
V[i-1, w] if wi > w
item i not taken
Boundary condition: V[i, 0] = 0, V[0, w] = 0. Number of sub-problems = ?
7/24/2016
169
Example
• n = 6 (# of items)
• W = 10 (weight limit)
• Items (weight, value):
2
4
3
5
2
6
7/24/2016
2
3
3
6
4
9
170
w
0
1
2
3
4
5
6
7
8
9
10
0
0
0
0
0
0
0
0
0
0
i
wi
vi
0
1
2
2
0
2
4
3
0
3
3
3
0
4
5
6
0
5
2
4
0
6
6
9
0
wi
V[i-1, w-wi]
V[i-1, w]
V[i, w]
V[i-1, w-wi] + vi item i is taken
max
V[i, w] =
V[i-1, w]
V[i-1, w] if wi > w
7/24/2016
item i not taken
item i not taken
171
w
0
1
2
3
4
5
6
7
8
9
10
i
wi
vi
0
0
0
0
0
0
0
0
0
0
0
1
2
2
0
0
2
2
2
2
2
2
2
2
2
2
4
3
0
0
2
2
3
3
5
5
5
5
5
3
3
3
0
0
2
3
3
5
5
6
6
8
8
4
5
6
0
0
2
3
3
6
6
8
9
9
11
5
2
4
0
0
4
4
6
7
7
10 10 12 13
6
6
9
0
0
4
4
6
7
9
10 13 13 15
V[i-1, w-wi] + vi item i is taken
max
V[i, w] =
7/24/2016
V[i-1, w]
V[i-1, w] if wi > w
item i not taken
item i not taken
172
w
0
1
2
3
4
5
6
7
8
9
10
i
wi
vi
0
0
0
0
0
0
0
0
0
0
0
1
2
2
0
0
2
2
2
2
2
2
2
2
2
2
4
3
0
0
2
2
3
3
5
5
5
5
5
3
3
3
0
0
2
3
3
5
5
6
6
8
8
4
5
6
0
0
2
3
3
6
6
8
9
9
11
5
2
4
0
0
4
4
6
7
7
10 10 12 13
6
6
9
0
0
4
4
6
7
9
10 13 13 15
Optimal value: 15
Item: 6, 5, 1
Weight: 6 + 2 + 2 = 10
Value: 9 + 4 + 2 = 15
7/24/2016
173
Time complexity
• Θ (nW)
• Polynomial?
– Pseudo-polynomial
– Works well if W is small
• Consider following items (weight, value):
(10, 5), (15, 6), (20, 5), (18, 6)
• Weight limit 35
– Optimal solution: item 2, 4 (value = 12). Iterate: 2^4 = 16 subsets
– Dynamic programming: fill up a 4 x 35 = 140 table entries
• What’s the problem?
– Many entries are unused: no such weight combination
– Top-down may be better
7/24/2016
174
Longest increasing subsequence
• Given a sequence of numbers
125329493568
• Find a longest subsequence that is nondecreasing
– E.g. 1 2 5 9
– It has to be a subsequence of the original list
– It has to in sorted order
=> It is a subsequence of the sorted list
Original list: 1 2 5 3 2 9 4 9 3 5 6 8
LCS:
Sorted:
122334556899
7/24/2016
1234568
175
Events scheduling problem
e3
e1
e2
e6
e4
e5
e8
e7
e9
Time
•
•
•
A list of events to schedule (or shows to see)
– ei has start time si and finishing time fi
– Indexed such that fi < fj if i < j
Each event has a value vi
Schedule to make the largest value
– You can attend only one event at any time
•
Very similar to the new restaurant location problem
– Sort events according to their finish time
– Consider: if the last event is included or not
7/24/2016
176
Events scheduling problem
e3
e1
e2
e6
e4
s8 e8 f8
e7
e5
s7
e9
f7
s9
f9
Time
• V(i) is the optimal value that can be achieved
when the first i events are considered
V(n-1)
• V(n) = max {
V(j) + vn
en not selected
en selected
j < n and fj < sn
7/24/2016
177
Coin change problem
• Given some denomination of coins (e.g., 2, 5, 7,
10), decide if it is possible to make change for a
value (e.g, 13), or minimize the number of coins
• Version 1: Unlimited number of coins for each
denomination
– Unbounded knapsack problem
• Version 2: Use each denomination at most once
– 0-1 Knapsack problem
7/24/2016
178
Use DP algorithm to solve new
problems
• Directly map a new problem to a known problem
• Modify an algorithm for a similar task
• Design your own
– Think about the problem recursively
– Optimal solution to a larger problem can be computed
from the optimal solution of one or more subproblems
– These sub-problems can be solved in certain
manageable order
– Works nicely for naturally ordered data such as
strings, trees, some special graphs
– Trickier for general graphs
• The text book has some very good exercises.
7/24/2016
179
Unit-profit restaurant location
problem
• Now the objective is to maximize the
number of new restaurants (subject to the
distance constraint)
– In other words, we assume that each
restaurant makes the same profit, no matter
where it is opened
10 mile
7/24/2016
180
A DP Algorithm
• Exactly as before, but pi = 1 for all i
S(i-1)
S(i) = max
S(j) + pi
j < i & dist (tj, ti) ≥ 10
S(i-1)
S(i) = max
S(j) + 1
7/24/2016
j < i & dist (tj, ti) ≥ 10
181
Greedy algorithm for restaurant
location problem
select t1
d = 0;
for (i = 2 to n)
d = d + dist(ti, ti-1);
if (d >= min_dist)
select ti
d = 0;
end
end
5
d
7/24/2016
0
2 2
5 7 9
6
6
15
0
3
6
6
9
10
15
0
7
10
0
7
182
Complexity
• Time: Θ(n)
• Memory:
– Θ(n) to store the input
– Θ(1) for greedy selection
7/24/2016
183
Optimal substructure
• Claim 1: if A = [m1, m2, …, mk] is the optimal solution to the
restaurant location problem for a set of towns [t1, …, tn]
– m1 < m2 < … < mk are indices of the selected towns
– Then B = [m2, m3, …, mk] is the optimal solution to the sub-problem
[tj, …, tn], where tj is the first town that are at least 10 miles to the
right of tm1
• Proof by contradiction: suppose B is not the optimal
solution to the sub-problem, which means there is a better
solution B’ to the sub-problem
– A’ = mi || B’ gives a better solution than A = mi || B => A is not
optimal => contradiction => B is optimal
A
A’
7/24/2016
m1
m1
m2
B
mk
B’ (imaginary)
184
Greedy choice property
• Claim 2: for the uniform-profit restaurant location
problem, there is an optimal solution that chooses t1
• Proof by contradiction: suppose that no optimal solution
can be obtained by choosing t1
– Say the first town chosen by the optimal solution S is ti, i > 1
– Replace ti with t1 will not violate the distance constraint, and the
total profit remains the same => S’ is an optimal solution
– Contradiction
– Therefore claim 2 is valid
S
S’
7/24/2016
185
Fractional knapsack problem
• Each item has a value and a weight
• Objective: maximize value
• Constraint: knapsack has a weight limitation
0-1 knapsack problem: take
each item or leave it
Fractional knapsack problem:
items are divisible
Unbounded knapsack problem:
unlimited supplies of each item.
Which one is easiest to solve?
We can solve the fractional knapsack
problem using greedy algorithm
7/24/2016
186
Greedy algorithm for fractional
knapsack problem
• Compute value/weight ratio for each item
• Sort items by their value/weight ratio into
decreasing order
– Call the remaining item with the highest ratio the most
valuable item (MVI)
• Iteratively:
– If the weight limit can not be reached by adding MVI
• Select MVI
– Otherwise select MVI partially until weight limit
7/24/2016
187
Example
item
Weight
(LB)
Value
($)
$ / LB
1
2
2
1
2
4
3
0.75
3
3
3
1
4
5
6
1.2
5
2
4
2
6
6
9
1.5
7/24/2016
• Weight limit: 10
188
Example
item
Weight
(LB)
Value
($)
$ / LB
5
2
4
2
6
6
9
1.5
4
5
6
1.2
1
2
2
1
3
3
3
1
2
4
3
0.75
7/24/2016
• Weight limit: 10
• Take item 5
– 2 LB, $4
• Take item 6
– 8 LB, $13
• Take 2 LB of item 4
– 10 LB, 15.4
189
Why is greedy algorithm for
fractional knapsack problem valid?
• Claim: the optimal solution must contain the MVI as
much as possible (either up to the weight limit or until
MVI is exhausted)
• Proof by contradiction: suppose that the optimal solution
does not use all available MVI (i.e., there is still w (w <
W) pounds of MVI left while we choose other items)
– We can replace w pounds of less valuable items by MVI
– The total weight is the same, but with value higher than the
“optimal”
– Contradiction
w
w
w
7/24/2016
w
190
Graphs
• A graph G = (V, E)
– V = set of vertices
– E = set of edges = subset of V  V
– Thus |E| = O(|V|2)
1
Vertices: {1, 2, 3, 4}
Edges: {(1, 2), (2, 3), (1, 3), (4, 3)}
2
4
3
7/24/2016
191
Graphs: Adjacency Matrix
• Example:
1
2
4
3
A
1
2
3
4
1
0
1
1
0
2
0
0
1
0
3
0
0
0
0
4
0
0
1
0
How much storage does the adjacency matrix require?
A: O(V2)
7/24/2016
192
Graphs: Adjacency List
• Adjacency list: for each vertex v  V, store a list
of vertices adjacent to v
• Example:
–
–
–
–
Adj[1] = {2,3}
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
1
2
• Variation: can also keep
a list of edges coming into vertex
7/24/2016
4
3
193
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
194
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
195
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
196
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
197
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
198
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
199
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
200
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
201
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
202
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
203
Kruskal’s algorithm: example
c-d:
3
b-f:
5
b-a:
7
b-d:
8
f-g:
9
d-e:
10
a-f:
12
b-c:
14
e-h:
15
7/24/2016
6
6
f-e:
a
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
204
Time complexity
• Depending on implementation
• Pseudocode:
sort all edges according to weights
Θ(m log m)
T = {}. tree(v) = v for all v.
= Θ(m log n)
m edges
for each edge (u, v)
Avg time spent per edge
if tree(u) != tree(v)
Naïve: Θ (n)
T = T U (u, v);
Better: Θ (log n)
union (tree(u), tree(v))
using set union
Overall time complexity
Naïve: Θ(nm)
Better implementation: Θ(m log n)
7/24/2016
205
Prim’s algorithm: example
a
6
5
b
14
3
7/24/2016
b
∞ ∞
f
7
8
c
a
12
e
9
15
g
h
10
d
c
d
e
f
g
h
∞
∞
∞
∞
∞
∞
206
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
ChangeKey
7/24/2016
c
b
a
d
e
f
g
h
0
∞
∞
∞
∞
∞
∞
∞
207
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
ExctractMin
7/24/2016
h
b
a
d
e
f
g
∞
∞
∞
∞
∞
∞
∞
208
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
ChangeKey
7/24/2016
d
b
a
h
e
f
g
3
14 ∞
∞
∞
∞
∞
209
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
ExctractMin
b
7/24/2016
g
a
h
e
f
14 ∞
∞
∞
∞
∞
210
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
d
9
15
g
h
10
Changekey
7/24/2016
b
e
a
h
g
f
8
10 ∞
∞
∞
∞
211
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
d
9
15
g
h
10
ExtractMin
e
7/24/2016
f
10 ∞
a
h
g
∞
∞
∞
212
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
Changekey
7/24/2016
f
e
a
h
g
5
10 6
∞
∞
213
Prim’s algorithm: example
a
6
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
ExtractMin
7/24/2016
12
a
e
g
h
6
10 ∞
∞
214
Prim’s algorithm: example
a
6
12
5
b
14
f
7
8
c
3
e
9
15
g
h
10
d
Changekey
7/24/2016
a
e
g
h
6
7
9
∞
215
Prim’s algorithm: example
a
6
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
ExtractMin
7/24/2016
e
h
g
7
∞
9
216
Prim’s algorithm: example
a
6
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
ExtractMin
7/24/2016
g
h
9
∞
217
Prim’s algorithm: example
a
6
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
Changekey
7/24/2016
g
h
9
15
218
Prim’s algorithm: example
a
6
12
5
b
14
7
8
c
3
f
e
d
9
15
g
h
10
ExtractMin
h
7/24/2016
15
219
Prim’s algorithm: example
a
6
5
b
14
3
f
7
8
c
7/24/2016
12
e
d
9
15
g
h
10
220
Complete Prim’s Algorithm
MST-Prim(G, w, r)
Q = V[G];
for each u  Q
Overall running time: Θ(m log n)
key[u] = ;
key[r] = 0;
Cost per
ChangeKey
T = {};
n vertices
while (Q not empty)
u = ExtractMin(Q);
Θ(n) times
for each v  Adj[u]
if (v  Q and w(u,v) < key[v])
T = T U (u, v);
ChangeKey(v, w(u,v));
Θ(n2) times?
How often is ExtractMin() called?
How often is ChangeKey() called?
7/24/2016
Θ(m) times
221
Summary
• Kruskal’s algorithm
– Θ(m log n)
– Possibly Θ(m + n log n) with counting sort
• Prim’s algorithm
– With priority queue : Θ(m log n)
• Assume graph represented by adj list
– With distance array : Θ(n^2)
• Adj list or adj matrix
– For sparse graphs priority queue wins
– For dense graphs distance array may be better
7/24/2016
222
14
b
9
14
h
f
7
1
6
7
9
3
5
5
e
d
6
8
0
a
i
4
7
2
1
c
7
g
7/24/2016
a
b
c
d
e
f
g
h
i
∞
14
7
5
0
∞
∞
∞
∞
Dijkstra’s algorithm
223
14 11
b
9
14
h
f
7
1
6
7
9
3
5
5
e
d
6
8
0
a
i
4
11
7
2
1
c
7
g
a
7/24/2016
b
c
d
e
f
g
h
i
11 11
7
5
0
∞
∞
∞
∞
Dijkstra’s algorithm
224
14 11
b
9
14
h
f
7
1
6
7
9
3
5
a
1
11
9
i
5
e
d
6
8
0
4
7
2
c
7
g
7/24/2016
a
b
c
d
e
f
g
h
i
9
11
7
5
0
∞
∞
∞
∞
Dijkstra’s algorithm
225
14 11
b
9
14
12
h
f
7
1
6
7
9
3
5
17
a
1
11
9
i
5
e
d
6
8
0
4
7
2
c
7
g
7/24/2016
a
b
c
d
e
f
g
h
i
9
11
7
5
0
12
∞
∞
17
Dijkstra’s algorithm
226
9
20
14 11
b
14
12
h
f
7
1
6
7
9
3
5
17
a
1
11
9
i
5
e
d
6
8
0
4
7
2
c
7
g
7/24/2016
a
b
c
d
e
f
g
h
9
11
7
5
0
12
∞
20 17
Dijkstra’s algorithm
i
227
9
20
19
14 11
b
14
12
h
f
7
1
6
7
9
3
5
17
a
1
11
9
i
5
e
d
6
8
0
4
7
2
c
7
g
7/24/2016
a
b
c
d
e
f
g
h
9
11
7
5
0
12
∞
19 17
Dijkstra’s algorithm
i
228
9
20
19
18
14 11
b
14
12
h
f
7
1
6
7
9
3
5
17
a
1
11
9
i
4
7/24/2016
e
7
2
c
g
5
d
6
8
0
7
18
a
b
c
d
e
9
11
7
5
0
f
g
h
i
12 18 18 17
Dijkstra’s algorithm
229
9
20
19
18
14 11
b
14
12
h
f
7
1
6
7
9
3
5
17
a
1
11
9
i
4
7/24/2016
e
7
2
c
g
5
d
6
8
0
7
18
a
b
c
d
e
9
11
7
5
0
f
g
h
i
12 18 18 17
Dijkstra’s algorithm
230
9
20
19
18
14 11
b
14
12
h
f
7
1
6
7
9
3
5
17
a
1
11
9
i
4
7/24/2016
e
7
2
c
g
5
d
6
8
0
7
18
a
b
c
d
e
9
11
7
5
0
f
g
h
i
12 18 18 17
Dijkstra’s algorithm
231
Prim’s Algorithm
MST-Prim(G, w, r)
Q = V[G];
for each u  Q
Overall running time: Θ(m log n)
key[u] = ;
key[r] = 0;
Cost per
ChangeKey
T = {};
while (Q not empty)
u = ExtractMin(Q);
for each v  Adj[u]
if (v  Q and w(u,v) < key[v])
T = T U (u, v);
ChangeKey(v, w(u,v));
7/24/2016
232
Dijkstra’s Algorithm
Dijkstra(G, w, r)
Q = V[G];
for each u  Q
Overall running time: Θ(m log n)
key[u] = ;
key[r] = 0;
Cost per
ChangeKey
T = {};
while (Q not empty)
u = ExtractMin(Q);
for each v  Adj[u]
if (v  Q and key[u]+w(u,v) < key[v])
T = T U (u, v);
ChangeKey(v, key[u]+w(u,v));
Running time of Dijkstra’s algorithm is the same as Prim’s algorithm
7/24/2016
233
Good luck with your final!
7/24/2016
234