1.1 Algorithms
z Algorithm:
Any well-defined computation
procedure that takes some value, or set of
values, as input and produces some value, or
set of values, as output.
z Or: tool for solving well specific computational
problem.
z Example: Sorting problem
z Input: A sequence of n numbers < a1 , a2 ,..., a n >
' '
'
z Output: A permutation < a1 , a2 ,..., an >
of the input sequence such that a1' ≤ a'2 ≤... ≤ a.'n
1. The Role of the Algorithms in
Computer
Jen-Yeu Chen
陳震宇
2
What kind of problem can be solved by
algorithm?
z An instance of a problem consists of all
inputs needed to compute a solution to the
problem.
z An algorithm is said to be correct if for every
input instance, it halts with the correct output.
z A correct algorithm solves the given
computational problem. An incorrect algorithm
might not halt at all on some input instance, or
it might halt with other than the desired answer.
3
z The Human Genome Project
z The Internet Applications
z Electronic Commerce with Public-key
cryptography and digital signatures
z Manufacturing and other commercial
settings
4
2.1 Insertion sort
„ Example:
2. Getting started
Sorting problem
Input: A sequence of n numbers < a1 , a2 ,..., an >
Output: A permutation < a1' , a2' ,..., a'n >of the
input sequence such that a1′ ≤ a2′ ≤ ... ≤ an′.
Jen-Yeu Chen
The number that we wish to sort are known as
the keys.
陳震宇
2
Pseudocode
The operation of Insertion-Sort
Insertion sort
Insertion-sort(A)
1 for j ←2 to length[A]
2 do key←A[j]
3
∗Insert A[j] into the sorted sequence A[1..j-1]
4
i ←j-1
5
while i>0 and A[i]>key
6
do A[i+1] ←A[i]
7
i ←i-1
8
A[i +1] ←key
3
4
2.2 Analyzing algorithms
Analyzing an algorithm has come to mean
predicting the resources that the algorithm
requires.
„ Sorted in place :
The numbers are rearranged within the array
A, with at most a constant number of them
sorted outside the array at any time.
Resources: memory, communication, bandwidth,
logic gate, time.
Assumption:one processor, RAM
(We shall have occasion to investigate models for
parallel computers and digital hardware.)
„ Loop invariant :
At the start of each iteration of the for loop of
line 1-8, the subarray A[1..j-1] consists of the
elements originally in A[1..j-1] but in sorted
order.
5
2.2 Analyzing algorithms
6
Insertion-sort(A)
cost times
c1
n
1 for j ←2 to length[A]
c2 n −1
2
do key←A[j]
3
∗Insert A[j] into the
0
sorted sequence A[1..j -1]
c
4 n −1
4
i ←j -1
n
c5 j∑=2 t j
5
while i>0 and A[i]>key
n
c6 j∑=2 ( t j − 1)
6
do A[i +1] ←A[i]
n
c7 j∑=2 ( t j − 1)
7
i ←i -1
c8 n −1
8
A[i +1] ←key
tj
: the number of times the while loop test in
line 5 is executed for the value of j.
The best notion for input size depends on
the problem being studied..
The running time of an algorithm on a
particular input is the number of primitive
operations or “steps” executed. It is
convenient to define the notion of step so
that it is as machine-independent as possible
7
8
the running time
the running time
n
t j = j for j = 2,3,…,n : quadratic function on
n.
T ( n ) = c1 n + c 2 ( n − 1 ) + c 4 ( n − 1 ) + c5 ∑ t j +
j=2
n
n
j=2
j=2
c6 ∑ ( t j − 1 ) + c7 ∑ ( t j − 1 ) + c8 ( n − 1 )
„
T (n) = c1n + c2 (n − 1) + c4 (n − 1) + c5 (
n(n + 1)
− 1) +
2
n(n − 1)
n(n − 1)
) + c8 (n − 1)
) + c7 (
2
2
c +c +c
c −c −c
= ( 5 6 7 )n 2 − (c1 + c2 + c4 + 5 6 7 + c8 )n
2
2
− (c2 + c4 + c5 + c8 )
c6 (
t j = 1 for j = 2,3,…,n : Linear function on n
T ( n ) = c1n + c2 ( n − 1 ) + c4 ( n − 1 ) + c5 ( n − 1 ) + c8 ( n − 1 )
= ( c1 + c2 + c4 + c5 + c8 )n − ( c2 + c4 + c5 + c8 )
9
Worst-case and average-case
analysis
10
Order of growth
„ Usually, we concentrate on finding only on
the worst-case running time
„ Reason:
It is an upper bound on the running time
The worst case occurs fair often
The average case is often as bad as the
worst case. For example, the insertion sort.
Again, quadratic function.
11
„ In
some particular cases, we shall be
interested in average-case, or expect
running time of an algorithm.
„ It
is the rate of growth, or order of
growth, of the running time that really
interests us.
12
2.3 Designing algorithms
Merge(A,p,q,r)
„ There are many ways to design algorithms:
„ Incremental approach: insertion sort
„ Divide-and-conquer: merge sort
recursive:
„ divide
„ conquer
„ combine
13
1 n1 ← q – p + 1
2 n2 ← r – q
3 create array L[ 1 .. n1 + 1 ] and R[ 1 ..n2 + 1 ]
4 for i ← 1 to n1
5
do L[ i ] ←A[ p + i – 1 ]
6 for j ← 1 to n2
7
do R[ i ] ←A[ q + j ]
8 L[n1 + 1] ← ∞
9 R[n2 + 1] ← ∞
14
Example of Merge Sort
Merge(A,p,q,r)
10 i ←1
11 j ←1
12 for k ← p to r
13
do if L[ i ] ≤ R[ j ]
14
then A[ k ] ← L[ i ]
15
i←i+1
16
else A[ k ] ← R[ j ]
17
j ←j+1
15
16
MERGE-SORT(A,p,r)
1 if p < r
2
then q←⎣(p+r)/2⎦
3
MERGE-SORT(A,p,q)
4
MERGE-SORT(A,q+1,r)
5
MERGE(A,p,q,r)
18
Analysis of merge sort
„ Analyzing divide-and-conquer algorithms
Θ(1 )
if n ≤ c
⎧
T( n ) = ⎨
⎩aT ( n / b ) + D( n ) + c( n ) otherwise
See Chapter 4
„ Analysis of merge sort
Θ(1 )
if
⎧
T( n ) = ⎨
⎩2T ( n / 2 ) + Θ( n ) if
n =1
n >1
T ( n ) = Θ( n log n )
19
20
Analysis of merge sort
c
if n = 1
⎧
T ( n) = ⎨
⎩2T (n / 2) + cn if n > 1
where the constant c represents the time
require to solve problems of size 1 as well
as the time per array element of the
divide and combine steps.
21
Outperforms insertion sort!
23
22
Computer Theory Lab.
3.1 Asymptotic notation
Θ( g (n)) = { f (n) | ∃ c1 , c2 , n0 s.t. 0 ≤ c1 g (n) ≤ f (n) ≤ c2 g (n)
3.Growth of Functions
for all n ≥ n0 }
f (n) = Θ( g (n))
⇒ g(n) is an asymptotic tight bound for f(n).
„
``=’’ abuse
Chapter 3
P.2
Computer Theory Lab.
Computer Theory Lab.
Example:
„
n2 n2
n2
if n>7.
≤
− 3n ≤
14 2
2
6 n 3 ≠ Θ( n 2 )
The definition of required every member of
be asymptotically nonnegative.
c2g(n)
f(n)
f ( n ) = an2 + bn + c , a, b, c constants, a>0.
f ( n ) = Θ( g( n ))
⇒ f(n)=Θ(n2 ).
c1g(n)
„
n
Chapter 3
n0
In general,
d
p ( n) = ∑ i = 0 ai ni where ai are constant w ith ad > 0.
Then P(n) = Θ(n d ).
P.3
Chapter 3
P.4
Computer Theory Lab.
Computer Theory Lab.
asymptotic upper bound
asymptotic lower bound
O( g (n)) = { f (n) | ∃c, n0 s.t. 0 ≤ f (n) ≤ cg (n) ∀n ≥ n0 }
Ω( g (n)) = { f (n) | ∃c, n0 s.t. 0 ≤ cg (n) ≤ f (n) ∀n ≥ n0 }
cg(n)
f(n)
f(n)
f ( n ) = O( g( n ))
f ( n ) = Ω( g( n ))
cg(n)
n
n
n0
n0
Chapter 3
P.5
Chapter 3
P.6
Computer Theory Lab.
Computer Theory Lab.
Theorem 3.1.
„
For any two functions f(n) and g(n),
f ( n ) = Θ( g( n )) if and only if f ( n ) = O( g( n ))
and f ( n ) = Ω( g( n )) .
„ o( g( n )) = { f ( n )| ∀c , ∃n0 ∀n > n0 , 0 ≤ f ( n ) ≤ cg( n )}
„
f ( n ) = o( g( n )) ⇔ limn→∞
f ( n)
=0
g( n )
„ ω ( g( n )) = { f ( n )| ∀c , ∃n0 ∀n > n0 , 0 ≤ cg( n ) ≤ f ( n )}
„
Chapter 3
P.7
Chapter 3
f ( n ) = ω ( g( n )) ⇔ limn→∞
f ( n)
=∞
g( n )
P.8
Computer Theory Lab.
„
Computer Theory Lab.
Transitivity
„
f ( n ) = Θ( g( n )) ∧ g( n ) = Θ( h( n )) ⇒ f ( n ) = Θ( h( n ))
f ( n ) = O( g( n )) ∧ g( n ) = O( h( n )) ⇒ f ( n ) = O( h( n ))
Transpose symmetry
f ( n ) = O( g( n )) ⇔ g( n ) = Ω( f ( n ))
f ( n ) = o( g( n )) ⇔ g( n ) = ω ( f ( n ))
f ( n ) = Ω( g( n )) ∧ g( n ) = Ω( h( n )) ⇒ f ( n ) = Ω( h( n ))
f ( n ) = o( g( n )) ∧ g( n ) = o( h( n )) ⇒ f ( n ) = o( h( n ))
f ( n ) = O( g( n )) ≈ a ≤ b
f ( n ) = Ω( g( n )) ≈ a ≥ b
f ( n ) = Θ( g( n )) ≈ a = b
f ( n ) = o( g( n )) ≈ a < b
f ( n ) = ω ( g( n )) ≈ a > b
f ( n ) = ω ( g( n )) ∧ g( n ) = ω ( h( n )) ⇒ f ( n ) = ω ( h( n ))
„
Reflexivity
f ( n ) = Θ( f ( n ))
f ( n ) = O( f ( n ))
f ( n ) = Ω( f ( n ))
„
Symmetry
f ( n ) = Θ( g( n )) ⇔ g( n ) = Θ( f ( n ))
Chapter 3
P.9
Chapter 3
P.10
Computer Theory Lab.
Computer Theory Lab.
2.2 Standard notations and common
functions
Trichotomy
„
a < b, a = b, or a > b.
„
e.g., n , n1+ sin n
n0
„
Monotonicity:
„
n2
„
„
„
Chapter 3
P.11
Chapter 3
A function f is monotonically increasing if m ≤ n
implies f(m) ≤ f(n).
A function f is monotonically decreasing if m ≤ n
implies f(m) ≥ f(n).
A function f is strictly increasing if m < n implies
f(m) < f(n).
A function f is strictly decreasing if m > n implies
f(m) > f(n).
P.12
Computer Theory Lab.
Computer Theory Lab.
Floor and ceiling
Modular arithmetic
x −1 < ⎣x⎦ ≤ x ≤ ⎡x⎤ < x + 1
„
⎡n / 2⎤ + ⎣n / 2⎦ = n
⎡ ⎡ n / a ⎤ / b⎤ = ⎡ n / ab⎤
⎣ ⎣ n / a ⎦ / b⎦ = ⎣ n / ab⎦
⎡a / b ⎤ ≤ (a + (b − 1)) / b
⎣a / b⎦ ≥ (a − (b − 1)) / b
Chapter 3
P.13
For any integer a and any positive integer n,
the value a mod n is the remainder (or
residue) of the quotient a/n :
a mod n =a -⎣a/n⎦n.
„
If(a mod n) = (b mod n). We write a ≡ b (mod
n) and say that a is equivalent to b, modulo n.
„
We write a ≢ b (mod n) if a is not equivalent
to b modulo n.
Chapter 3
P.14
Computer Theory Lab.
Computer Theory Lab.
Polynomials v.s. Exponentials
Logarithms
d
„
Polynomials: P( n ) = ∑ a i n i
i =0
„
„
ln(1 + x ) = x −
A function is polynomial bounded if f (n) = n
Exponentials:
„
n b = o( a n )
O (1)
.
x
≤ ln(1 + x ) < x
1+ x
( a > 1)
Any positive exponential function grows faster
than any polynomial.
xi
i = 0 i!
A function f(n) is polylogarithmically bounded
O (1)
if f (n) = log n
b
a
„ log n = o( n ) for any constant a > 0.
„ Any positive polynomial function grows faster
than any polylogarithmic function.
„
∞
ex = ∑
1+ x ≤ ex ≤ 1+ x +
Chapter 3
x2
if | x |< 1
2
x
lim n →∞ (1 + ) n = e x
n
P.15
x 2 x 3 x 4 x5
+
−
+
−... if | x| < 1
2
3
4
5
Chapter 3
P.16
Computer Theory Lab.
Computer Theory Lab.
Factorials
„
Function iteration
Stirling’s approximation
n
1
n!= 2πn ( ) n (1 + Θ( ))
e
n ! = o( n n )
n ! = ω( 2 n )
log( n !) = Θ( n log n )
n
ifｉ= 0,
⎧
f (i ) ( n) = ⎨
(i −1)
(n)) ifｉ> 0.
⎩f(f
n
n
n!=
⎛n⎞
2π n ⎜ ⎟ eα n
⎝ e ⎠
For example, if f (n) = 2n , then f (i ) (n) = 2i n
where
1
1
< αn <
12n + 1
12n
Chapter 3
P.17
Chapter 3
Computer Theory Lab.
P.18
Computer Theory Lab.
The iterative logarithm function
if i = 0
⎧n
(
i
−
1
)
⎪⎪lg(lg n) if i > 0 and lg (i −1) n > 0
(i )
lg (n) = ⎨
lg(i −1 ) n ≤ 0
⎪undefined if i > 0(i −and
⎪⎩
or lg 1 ) n is undefined.
„
lg* ( n ) = min{ i ≥ 0 |lg( i ) ≤ 1 }
lg* 2 = 1
lg* 4 = 2
lg* 16 = 3
lg* 65536 = 4
lg* 265536 = 5
Chapter 3
P.19
Since the number of atoms in the
observable universe is estimated to be
80
65536
about 10 , which is much less than 2 ,
we rarely encounter a value of n such
that lg* n > 5 .
Chapter 3
P.20
Computer Theory Lab.
Fibonacci numbers
F0 = 0
F1 = 1
Fi = Fi −1 + Fi − 2
)
φi −φ i
Fi =
5
1+ 5
= 1.61803...
2
) 1− 5
φ=
= −0.61803...
2
φ=
Chapter 3
P.21
Computer Theory Lab.
Recurrences -Substitution method
 Recursion-tree method
 Master method

4.Recurrences
T (n)  aT (n / b)  f (n)
Chapter 4
P.2
Computer Theory Lab.
Computer Theory Lab.
4.1 The substitution method
: Mathematical induction
Technicalities

Chapter 4
We neglect certain technical details
when we state and solve recurrences.
A good example of a detail that is often
glossed over is the assumption of
integer arguments to functions.
Boundary conditions is ignored. Omit
floors, ceilings.
P.3

The substitution method for solving
recurrence entails two steps:
1. Guess the form of the solution.
2. Use mathematical induction to find the
constants and show that the solution
works.
Chapter 4
P.4
Computer Theory Lab.
Computer Theory Lab.
Example
T ( n )  2T (  n / 2  )  n

T (1 )  1
n
T ( n )  2( c  n / 2  log  n / 2  )  n  cn log  n
2
 cn log n  cn log 2  n  cn log n (if c  1.)
(We may omit the initial condition later.)
Initial condition 1  T (1)  cn log 1  0()
Guess T (n)  O(n log n)
However,
4  T ( 2 )  cn log 2 ( if c  4 )
Assume T ( n / 2  )  c n / 2  logn / 2 
Chapter 4
P.5
Chapter 4
P.6
Computer Theory Lab.
Computer Theory Lab.
Subtleties

Making a good guess
T ( n )  T ( n / 2 )  T ( n / 2 )  1
T ( n )  2T (  n / 2   17 )  n

Guess T ( n )  O( n )
 Assume T ( n )  cn
We guess T ( n )  O( n log n )
T ( n )  c  n / 2   c  n / 2   1  cn  1  cn
Making guess provides loose upper
bound and lower bound. Then improve
the gap.
Chapter 4

However, assume T ( n )  cn  b
T ( n )  ( cn / 2  b )  ( cn / 2  b )  1
 cn  2 b  1  cn  b ( Choose b  1 )
P.7
Chapter 4
P.8
Computer Theory Lab.
Computer Theory Lab.
n
Show that the solution to T(n) = 2T(2  + 17) + n is O(n lg n)
anlg(n) 21/a+ (34a-2b)lg(n) - 2c
Solution:
assume a > 0, b > 0, c > 0 and T(n) ≦an lg n – blg n - c
n
n≧ 2 +17，n≧34
n
n≧ (2 +17) 21/a，∵21/2≦1.5∴n ≧12
n
n
n
T(n) ≦2[(2 + 17)lg(2 +17) - blg(2 + 17)-c]+n
34a-2b≦ -b，b ≧34a
n
n
≦ (an + 34a)lg(2 +17) - 2blg(2 +17) - 2c + n
c > 0，-c > -2c
 T(n) ≦anlgn - blgn - c，T(n) ≦anlgn
n
n
≦anlg(2 +17) + anlg21/a + (34a-2b)lg(2 +17) - 2c
 T(n) = O(nlgn)
≦anlg(n) 21/a+ (34a-2b)lg(n) - 2c
Chapter 4
P.9
Chapter 4
P.10
Computer Theory Lab.
Computer Theory Lab.
Avoiding pitfalls
Changing variables
T ( n )  2T (  n / 2  )  n

T (1 )  1
 Assume T ( n )  O( n )

T ( n )  2T ( n )  lg n
Let m  lg n .
T ( 2 m )  2T ( 2 m / 2 )  m
Hence T ( n )  cn
T (n)  2(c n / 2)  n  cn  n  O(n)
Then S ( m )  2 S ( m / 2 )  m .
(Since c is a constant)

Chapter 4
 S ( m )  O( m lg m )
(WRONG!) You cannot find such a c.
 T ( n )  T ( 2 m )  S ( m )  O( m lg m )
 O(lg n lg lg n )
P.11
Chapter 4
P.12
Computer Theory Lab.
Computer Theory Lab.
4.2 the Recursion-tree method
T (n)  3T ( n / 4)  (n2 )
Chapter 4
P.13
Chapter 4
P.14
Computer Theory Lab.
Computer Theory Lab.
The cost of the entire tree
2
3
3
3
T (n)  cn  cn 2    cn 2  ...   
16
 16 
 16 
2

log 4 n 1

i 0
i

3 2
log 3
  cn   n
16
 
4
log 4 n 1
2
cn  (n
log 4 3
T ( n) 
)
  16  cn 2  nlog 3 
log 4 n 1
i 0
i
3
 
i
4


 3
 
    cn 2   nlog 3
i  0  16 
1

cn 2   nlog 3
1  (3 / 16)
16
 cn 2  (nlog 3 )
13

4

(3 / 16)log n  1 2

cn  (nlog 3 ).
(3 / 16)  1
4
4
4

4
 O(n 2 )
Chapter 4
P.15
Chapter 4
P.16
Computer Theory Lab.
Computer Theory Lab.
substitution method
T (n)  T (n / 3)  T (2n / 3)  cn
We want to Show that T(n) ≤ dn2 for some constant
d > 0. using the same constant c > 0 as before,
we have
T (n)  3T ( n / 4)  cn 2
 3d n / 42  cn 2
 3d (n / 4) 2  cn 2
3
 dn 2  cn 2
16
 dn 2 ,
Where the last step holds as long as d (16/13)c.
Chapter 4
P.17
Chapter 4
P.18
Computer Theory Lab.
Computer Theory Lab.
substitution method
4.3 The master method
Theorem 4.1 ((M
Maasstteerr tthheeoorreem
m))
T (n)  T (n / 3)  T (2n / 3)  cn
Let a  1 and b  1 be constants, let f ( n ) be a function, and T ( n ) be defined on
 d (n / 3) lg( n / 3)  d (2n / 3) lg( 2n / 3)  cn
 (d (n / 3) lg n  d (n / 3) lg 3)  (d (2n / 3) lg n  d (2n / 3) lg(3 / 2))  cn
the nonnegative integers by the recurrence
T ( n )  aT ( n / b )  f ( n )
 dn lg n  d ((n / 3) lg 3  (2n / 3) lg(3 / 2))  cn
 dn lg n  d ((n / 3) lg 3  (2n / 3) lg 3  (2n / 3) lg 2  cn
where we interpret n / b mean either  n / b or  n / b .
 dn lg n  dn(lg 3  2 / 3)  cn
1. If f ( n )  O( n
 dn lg n,
2. If f (n)  (n
3. If f (n)  (n
As long as d  c/lg3 – (2/3)).
logb a  
log b a
) for some constant   0 , then T ( n )  ( n logb a ) .
) then T ( n )  ( n logb a log n ) .
log b a 
) for some constant   0 and if af ( n / b )  cf ( n ) for
some constant c  1 and all sufficiently large n, then T ( n )  ( f ( n )) .
Proof. (In section 4.4 by recursive tree)
Chapter 4
P.19
Chapter 4
P.20
Computer Theory Lab.
Computer Theory Lab.
 T ( n )  9T ( n / 3 )  n
a  9, b  3, f ( n )  n
n log3 9  n2 ,
 T ( n )  3T ( n / 4 )  n log n
f ( n )  O( n log3 9 1 )
a  3 , b  4 , f ( n )  n log n
Case 1  T ( n )  ( n2 )
n log4 3  n0 .793 ,
f ( n )  O( n log4 3   )
 T( n )  T( 2n / 3 )  1
Case
a  1, b  3 / 2 , f ( n )  1
3
Check
n log3 / 2 1  n0  1  f ( n ),
af(n/b)=3(
Case
Chapter 4
2  T ( n )  (log n )
3
for c= , and sufficiently large n
4
 T ( n )  ( n log n )
P.21
Computer Theory Lab.
The master method does not apply to the
recurrence T (n)  2T (n / 2)  n lg n,
even though it has the proper form: a = 2,
log a
b=2, f(n)= n lgn, and n b  n. It might
seem that case 3 should apply, since f(n)=
n lgn is asymptotically larger than nlog b a  n.
 The problem is that it is not polynomially
larger.

Chapter 4
n
n
3n
) log ( ) 
log n=cf(n)
4
4
4
P.23
Chapter 4
P.22
7.1 Description of quicksort
P.2

Divide

Conquer

Combine
7.QUICKSORT
LANDIS
Lab of Advanced Networks and Distributed Intelligent
Systems
Chapter 7
Partition(A, p, r)
P.3
QUICKSORT(A,p,r)
1 if p  r
2 then q  PARTITION ( A, p , r )
3 QUICKSORT(A,p,q)
4 QUICKSORT(A,q+1,r)
Chapter 7
P.4
1
2
3
4
5
6
7
8
x  A[r]
i p–1
for j  p to r -1
do if A[j] ≤ x
then i  i + 1
exchange A[i]  A[j]
exchange A[i +1]  A[r]
return i +1
Chapter 7
The operation of Partition on a sample array
P.5
P.6
At the beginning of each iteration of the loop of lines
3-6, for any array index k,
1. if p ≤ k ≤ i, then A[k] ≤ x.
2. if i + 1 ≤ k≤ j -1, then A[k] > x.
3. if k = r, then A[k] = x.
Chapter 7
Chapter 7
Two cases for one iteration of procedure Partition
7.2 Performance of quicksort
P.8
P.7
Worst-case partition:
T ( n )  T ( n  1 )  ( n )
n
n
k 1
k 1
  ( k )  (  k )  ( n2 )
Best-case partition:
Complexity:
Partition on A[p…r] is (n)
where n = r –p +1
T ( n )  2T ( n / 2 )  ( n )
 T ( n )  ( n log n )
Chapter 7
Chapter 7
Intuition for the average case T(n) = (n logn)
P.9
P.10
Balanced partition T ( n )  ( n log n )
T (n)  T (9n / 10)  T (n / 10)  (n)
 T (n)  (n log n)
Chapter 7
Chapter 7
7.4 Analysis of quicksort
7.3 Randomized versions of partition
RANDOMIZED_PARTITION(A,p,r)
1 i  RANDOM ( p , r )
2 exchange A[ p ]  A[ i ]
3 return PARTITION(A,p,r)
P.1
2
P.11
7.4.1 Worst-case analysis
T (n)  max (T (q)  T (n  q  1))  (n)
0 q  n 1
guess T ( n )  cn2
T ( n)  max (cq 2  c( n  q  1) 2 )  ( n)
0  q  n 1
 c max ( q 2  ( n  q  1) 2 )  ( n)
RANDOMIZED_QUICKSORT(A,p,r)
0  q  n 1
 cn 2  2c( n  1)  ( n)
1 if p  r
2 then
q  RANDOMIZED _ PARTITION ( A, p , r )
3 RANDOMIZED_QUICKSORT(A,p,q)
 cn 2
pick the constant c large enough so that the 2 c( n  1 ) term dominates the
( n ) term.
 T ( n )  ( n2 )
4 RANDOMIZED_QUICKSORT(A,q+1,r)
Chapter 7
Chapter 7
7.4-2
7.4.2 Expected running time
P.1
3
P.1
4
Running time and comparsions
 Lemma 7.1

Show that q  (n  q) achieves a maximum over
2
2
q  1,2,....., n  1
when q  1 or q  n  1
ans: 先令
f ( q)  q  (n  q)
一次微分:
f ' ( q )  2 q  2( n  q )  4 q  2 n
2
令
f ' ( q)  0

二次微分:
f '' ( q)  4
(開口向上)
因為 1  q  n  1 所以
 Let X be the number of comparisons performed in line
4 of partition over the entire execution of Quicksort on
an n-element array. Then the running rime of Quicksort
is O(n+X)
2
4 q  2n  0

0
q
n/2
n
n
(極小值)
2
f (1)  f ( n  1)  1  ( n  1) 2
(相對極大值)
Chapter 7
Chapter 7
we define
X ij  I {zi is compared to zj},
Pr{zi is compared to zj} = Pr{zi or zj is first pivot chosen from Zij}
= Pr{zi is first pivot chosen from Zij}
+ Pr{zj is first pivot chosen from Zij}
n 1 n
X    X ij .
1
1

j  i 1 j  i 1
2

j  i 1

i 1 j i 1
n 1 n

E[ X ]  E    X ij 
 i 1 j i 1 
n 1 n
 
   E X ij
i 1 j  i 1
n 1 n
2
.
i 1 j i 1 j  i  1
 E[ X ]   
n 1 n
   Pr {zi is compared to zj}
i 1 j  i 1
15
16
another analysis
n 1 n
2
E[ X ]   
i 1 j  i 1 j  i  1
P.1
8
n 1n  i
2
i 1 k 1 k  1
T( n ) 
n 1 n 2
T (1 )  1
 
n 1
1
( T (1 )  T ( n  1 )   ( T ( q )  T ( n  q ))  ( n )
n
q 1

1

  ( T (1 )  T ( n  1 ))  O( n )
n

T ( n  1 )  O( n2 )
 
i 1 k 1 k
n 1
T( n ) 
  O(lg n)
n 1
1
( T (1 )  T ( n  1 )   ( T ( q )  T ( n  q ))  ( n )
n
q 1
i 1
 O(n lg n)
17

1 n 1
(  T ( q )  T ( n  q ))  ( n )
n q 1

2 n 1
(  T ( k ))  ( n )
n k 1
Chapter 7
2a 1 2
n 2 2b(n  1)
( n log n  ) 
 ( n )
n 2
8
n
an
 an log n   2b  (n)
4
an
 an log n  b  ((n)  b  )
4
 an log n  b
guess T (n)  an log n  b
T( n ) 

T ( n) 
2 n 1
(  ak log k  b )  ( n )
n k 1
2 a n 1
2b
( n  1 )  ( n )
 k log k 
n k 1
n
n 1
We will prove  k log k 
k 1
n2 log n n2

2
8
Choose a large enough so that
an
 ( n )  b .
4
 T ( n )  O( n log n ).
Chapter 7
P.19
Chapter 7
P.20
n 1
 n / 2  1
k 1
k 1
 k log k 

k log k 
n 1

k  n / 2 
k log k
 n / 2  1
n 1
k 1
k  n / 2 
 (log n  1)  k  log n
n 1
 n / 2  1
k 1
k 1
 log n  k 

k
k
n(n  1) log n 1 n
n
 (  1)
2
2 2
2
2
2
n log n n


2
8
if n  2.

Chapter 7
Another approach: Using
1 2
1 2
 x ln xdx  x ln x  x
2
4
P.21
Computer Theory Lab.
◼
15.Dynamic Programming
Dynamic programming is typically
applied to optimization problems. In
such problem there can be many
solutions. Each solution has a value,
and we wish to find a solution with the
optimal value.
Chapter 15
Computer Theory Lab.
P.2
Computer Theory Lab.
15.1 Assembly-line scheduling
The development of a dynamic programming
algorithm can be broken into a sequence of
four steps:
1. Characterize the structure of an optimal
solution.
2. Recursively define the value of an optimal
solution.
3. Compute the value of an optimal solution in a
bottom up fashion.
4. Construct an optimal solution from computed
information.
Chapter 15
P.3
An automobile chassis enters each assembly
line, has parts added to it at a number of
stations, and a finished auto exits at the end
of the line.
Each assembly line has n stations, numbered
j = 1, 2,...,n. We denote the jth station on
line j ( where i is 1 or 2) by Si,j. The jth
station on line 1 (S1,j) performs the same
function as the jth station on line 2 (S2,j).
◼
Chapter 15
P.4
Computer Theory Lab.
Computer Theory Lab.
The stations were built at different times and
with different technologies, however, so that
the time required at each station varies, even
between stations at the same position on the
two different lines. We denote the assembly
time required at station Si,j by ai,j.
As the coming figure shows, a chassis enters
station 1 of one of the assembly lines, and it
progresses from each station to the next.
There is also an entry time ei for the chassis
to enter assembly line i and an exit time xi
for the completed auto to exit assembly line i.
Chapter 15
a manufacturing problem
to find the fast way through a factory
P.5
Chapter 15
Computer Theory Lab.
Computer Theory Lab.
Normally, once a chassis enters an assembly
line, it passes through that line only. The time
to go from one station to the next within the
same assembly line is negligible.
Occasionally a special rush order comes in,
and the customer wants the automobile to be
manufactured as quickly as possible.
For the rush orders, the chassis still passes
through the n stations in order, but the
factory manager may switch the partiallycompleted auto from one assembly line to
the other after any station.
Chapter 15
P.7
P.6
The time to transfer a chassis away from
assembly line i after having gone through
station Sij is ti,j, where i = 1, 2 and j = 1, 2, ...,
n-1 (since after the nth station, assembly is
complete). The problem is to determine which
stations to choose from line 1 and which to
choose from line 2 in order to minimize the
total time through the factory for one auto.
Chapter 15
P.8
Computer Theory Lab.
Computer Theory Lab.
An instance of the assembly-line
problem with costs
Step1 The structure of the fastest way
through the factory
◼
the fast way through station S1,j is either
the fastest way through Station S1,j-1 and then directly
through station S1,j, or
◼ the fastest way through station S2,j-1, a transfer from line 2
to line 1, and then through station S1,j.
◼
◼
Using symmetric reasoning, the fastest way through
station S2,j is either
the fastest way through station S2,j-1 and then directly
through Station S2,j, or
◼ the fastest way through station S1,j-1, a transfer from line 1
to line 2, and then through Station S2,j.
◼
Chapter 15
P.9
Chapter 15
P.10
Computer Theory Lab.
Computer Theory Lab.
Step 2: A recursive solution
step 3: computing the fastest times
Let ri(j)be the number of references made to fi[j] in a
recursive algorithm.
r1(n)=r2(n)=1
r1(j) = r2(j)=r1(j+1)+r2(j+1)
n
◼ The total number of references to all fi[j] values is (2 ).
◼
f = min( f1[n ] + x1 , f 2 [n ] + x2 )
*
e1 + a1,1
if j = 1,
f1[ j ] = 
min( f1[ j − 1] + a1, j , f 2 [ j − 1] + t 2, j −1 + a1, j ) if j  2
e2 + a2,1
if j = 1,
f2[ j] = 
min( f 2 [ j − 1] + a2, j , f1[ j − 1] + t1, j −1 + a2, j ) if j  2
Chapter 15
◼
P.11
Chapter 15
We can do much better if we compute the fi[j] values in
different order from the recursive way. Observe that for j 
2, each value of fi[j] depends only on the values of f1[j-1]
and f2[j-1].
P.12
Computer Theory Lab.
Computer Theory Lab.
FASTEST-WAY procedure
FASTEST-WAY(a, t, e, x, n)
1 f1[1]  e1 + a1,1
2 f2[1]  e2 + a2,1
3 for j  2 to n
4
do if f1[j-1] + a1,j  f2[j-1] + t2,j-1 +a1,j
5
then f1[j]  f1[j-1] + a1,j
6
l1[j]  1
7
else f1[j]  f2[j-1] + t2,j-1 +a1,j
8
l1[j]  2
9
if f2[j-1] + a2, j  f1[j-1] + t1,j-1 +a2,j
10
then f2[j]  f2[j – 1] + a2,j
11
l2[j]  2
12
else f2[j]  f1[j – 1] + t1,j-1 + a2,j
13
l2[j]  1
14 if f1[n] + x1  f2[n] + x2
15
then f* = f1[n] + x1
16
l* = 1
17
else f* = f2[n] + x2
18
l* = 2
Chapter 15
P.13
Chapter 15
P.14
Computer Theory Lab.
Computer Theory Lab.
step 4: constructing the fastest way
through the factory
PRINT-STATIONS(l, n)
1 i  l*
2 print “line” i “,station” n
3 for j  n downto 2
4
do i  li[j]
5
print “line” i “,station” j – 1
Chapter 15
15.2 Matrix-chain multiplication
◼
output
line 1, station 6
line 2, station 5
line 2, station 4
line 1, station 3
line 2, station 2
line 1, station 1
P.15
Chapter 15
A product of matrices is fully
parenthesized if it is either a single
matrix, or a product of two fully
parenthesized matrix product,
surrounded by parentheses.
P.16
Computer Theory Lab.
Computer Theory Lab.
MATRIX MULTIPLY
How to compute A1 A2 ... An where Ai
is a matrix for every i.
◼ Example: A1 A2 A3 A4
MATRIX MULTIPLY(A,B)
1 if columns[A]  column[B]
2 then error “incompatible dimensions”
3 else for i  1 to rows[A]
4 do for j  1 to columns[B]
5 do c [ i , j ]  0
6 for k  1 to columns[A]
7 do c [ i , j ]  c [ i , j ] + A[ i , k ] B [ k , j ]
8 return C
◼
( A1( A2 ( A3 A4 ))) ( A1(( A2 A3 ) A4 ))
(( A1 A2 )( A3 A4 )) (( A1( A2 A3 )) A4 )
((( A1 A2 ) A3 ) A4 )
Chapter 15
P.17
Chapter 15
P.18
Computer Theory Lab.
Computer Theory Lab.
Complexity:
◼
Chapter 15
Example:
Let A be a p  q matrix, and B be a
q  r matrix. Then the complexity is
pqr.
◼
P.19
Chapter 15
A1 is a 10  100 matrix, A2 is a 100  5
matrix, and A3 is a 5  50 matrix. Then
(( A1 A2 ) A3 ) takes 10  100  5 + 10  5  50 = 7500
time. However ( A1( A2 A3 )) takes
100  5  50 + 10  100  50 = 75000 time.
P.20
Computer Theory Lab.
Computer Theory Lab.
The matrix-chain multiplication
problem:
◼
Counting the number of
parenthesizations:
Given a chain  A1 , A2 ,..., An  of n matrices,
where for i=0,1,…,n, matrix Ai has
dimension pi-1pi, fully parenthesize the
product A1 A2 ... An in a way that
minimizes the number of scalar
multiplications.
Chapter 15
P.21
1
if n = 1

n −1
P( n ) = 
 P( k ) p( n − k ) if n  2

k =1
◼
=
1  2 n
4n
=
(
)
 
n +1  n 
n3 / 2
Chapter 15
P.22
Computer Theory Lab.
Computer Theory Lab.
Step 1: The structure of an
optimal parenthesization
Step 2: A recursive solution
Define m[i, j]= minimum number of
scalar multiplications needed to
compute the matrix Ai.. j = Ai Ai +1 .. A j
◼ goal m[1, n]
◼ m[ i , j ] =
◼
Optimal
(( A1 A2 ... Ak )( Ak +1 Ak + 2 ... An ))
0
i= j


min ik  j {m[i, k ] + m[k + 1, j ] + pi −1 pk p j } i  j
Combine
Chapter 15
P( n ) = C( n −1) [Catalan number]
P.23
Chapter 15
P.24
Computer Theory Lab.
MATRIX_CHAIN_ORDER
Step 3: Computing the optimal
costs
MATRIX_CHAIN_ORDER(p)
1 n  length[p] –1
2 for i  1 to n
3
do m[i, i]  0
4 for l  2 to n
5
do for i  1 to n – l + 1
6
do j  i + l – 1
7
m[i, j]  
8
for k  i to j – 1
9
do q  m[i, k] + m[k+1, j]+ pi-1pkpj
10
if q < m[i, j]
11
then m[i, j]  q
12
s[i, j]  k
13 return m and s
Complexity: O( n3 )
Chapter 15
Computer Theory Lab.
Computer Theory Lab.
the m and s table computed by
MATRIX-CHAIN-ORDER for n=6
Example:
A1 30  35
A2 35  15
A3 15  5
A4 5  10
A5 10  20
A6 20  25
Chapter 15
P.26
= p0  p1
= p1  p2
= p2  p3
= p3  p4
= p4  p5
= p5  p6
P.27
Chapter 15
P.28
Computer Theory Lab.
Step 4: Constructing an optimal
solution
m[2,5]=
min{
m[2,2]+m[3,5]+p1p2p5=0+2500+351520=13000,
m[2,3]+m[4,5]+p1p3p5=2625+1000+35520=7125,
m[2,4]+m[5,5]+p1p4p5=4375+0+351020=11374
}
=7125
Chapter 15
P.29
Computer Theory Lab.
MATRIX_CHAIN_MULTIPLY
16.3 Elements of dynamic
programming
MATRIX_CHAIN_MULTIPLY(A, s, i, j)
1 if j > i
2 then x  MCM ( A, s, i , s[ i , j ])
3 y  MCM ( A, s , s [ i , j ] + 1, j )
4 return MATRIX-MULTIPLY(X,Y)
5 else return Ai
◼
Chapter 15
example: (( A1( A2 A3 ))(( A4 A5 ) A6 ))
P.31
Computer Theory Lab.
Computer Theory Lab.
Optimal substructure:
◼
◼
1. You show that a solution to the problem consists of
making a choice, Making this choice leaves one or
more subproblems to be solved.
2. You suppose that for a given problem, you are given
the choice that leads to an optimal solution.
3. Given this choice, you determine which subproblems
ensue and how to best characterize the resulting
space of subproblems.
4. You show that the solutions to the subproblems used
within the optimal solution to the problem must
themselves be optimal by using a “cut-and-paste”
technique.
We say that a problem exhibits optimal
substructure if an optimal solution to
the problem contains within its optimal
solution to subproblems.
Example: Matrix-multiplication problem
Chapter 15
P.33
Chapter 15
P.34
Computer Theory Lab.
Computer Theory Lab.
Subtleties
Optimal substructure varies across problem
domains in two ways:
◼
1. how many subproblems are used in an
optimal solutiion to the original problem, and
One should be careful not to assume that optimal
substructure applies when it does not. consider the
following two problems in which we are given a
directed graph G = (V, E) and vertices u, v  V.
◼ Unweighted shortest path:
◼
2. how many choices we have in determining
which subproblem(s) to use in an optimal
solution.
Chapter 15
◼
Unweighted longest simple path:
◼
P.35
Chapter 15
Find a path from u to v consisting of the fewest edges.
Good for Dynamic programming.
Find a simple path from u to v consisting of the most
edges. Not good for Dynamic programming.
P.36
Computer Theory Lab.
RECURSIVE_MATRIX_CHAIN
RECURSIVE_MATRIX_CHAIN(p, i, j)
1 if i = j
2
then return 0
3 m[i, j]  
4 for k  i to j – 1
5
do q  RMC(p,i,k) + RMC(p,k+1,j) + pi-1pkpj
6
if q < m[i, j]
7
then m[i, j]  q
8 return m[i, j]
Overlapping subproblems:
example:
MAXTRIX_CHAIN_ORDER
Chapter 15
P.38
Computer Theory Lab.
Computer Theory Lab.
The recursion tree for the computation
of RECURSUVE-MATRIX-CHAIN(P, 1, 4)
T (1)  1

T (n)  1 + n−1(T (k ) + T (n − k ) + 1) for n  1

k =1
n −1
T( n )  2  T( i ) + n
i =1
◼
Chapter 15
P.39
Chapter 15
We can prove that T(n) =(2n) using
substitution method.
P.40
Computer Theory Lab.
Computer Theory Lab.
MEMORIZED_MATRIX_CHAIN
T (1)  1 = 2 0
n −1
MEMORIZED_MATRIX_CHAIN(p)
1 n  length[p] –1
2 for i  1 to n
3
do for j  1 to n
4
do m[i, j]  
5 return LC (p,1,n)
n−2
T (n)  2  2i −1 + n =2  2i + n
i =1
= 2(2
n −1
i =0
− 1) + n = (2 − 2) + n  2 n−1
n
Solution:
1. bottom up
2. memorization (memorize the natural, but
inefficient)
Chapter 15
P.41
Chapter 15
P.42
Computer Theory Lab.
Computer Theory Lab.
LOOKUP_CHAIN
16.4 Longest Common
Subsequence
LOOKUP_CHAIN(p, i, j)
1 if m[i, j] < 
2
then return m[i, j]
3 if i = j
4
then m[i, j]  0
5
else for k  i to j – 1
6
do q  LC(p, i, k) +LC(p, k+1, j)+pi-1pkpj
7
if q < m[i, j]
8
then m[i, j]  q
9 return m[i, j]
X = < A, B, C, B, D, A, B >
Y = < B, D, C, A, B, A >
◼ < B, C, A > is a common subsequence
of both X and Y.
◼ < B, C, B, A > or < B, C, A, B > is the
longest common subsequence of X and
Y.
Time Complexity: O(n3 )
Chapter 15
P.43
Chapter 15
P.44
Computer Theory Lab.
Computer Theory Lab.
Longest-common-subsequence
problem:
◼
◼
Theorem 16.1. (Optimal
substructure of LCS)
We are given two sequences X =
<x1,x2,...,xm> and Y = <y1,y2,...,yn> and
wish to find a maximum length common
subsequence of X and Y.
We Define Xi = < x1,x2,...,xi >.
Chapter 15
P.45
Let X = <x1,x2,...,xm> and Y = <y1,y2,...,yn> be the
sequences, and let Z = <z1,z2,...,zk> be any LCS of
X and Y.
1. If xm = yn
then zk = xm = yn and Zk-1 is an LCS of Xm-1 and Yn-1.
2. If xm  yn
then zk  xm implies Z is an LCS of Xm-1 and Y.
3. If xm  yn
then zk  yn implies Z is an LCS of X and Yn-1.
◼
Chapter 15
P.46
Computer Theory Lab.
Computer Theory Lab.
A recursive solution to
subproblem
◼
Computing the length of an LCS
LCS_LENGTH(X,Y)
1 m  length[X]
2 n  length[Y]
3 for i  1 to m
4
do c[i, 0]  0
5 for j  1 to n
6
do c[0, j]  0
7 for i  1 to m
8
do for j  1 to n
Define c [i, j] is the length of the LCS of
Xi and Yj .
0
if i=0 or j=0


c[i, j ] = 
c[i − 1, j − 1] + 1
if i,j>0 and xi=y j
max{c[i, j − 1], c[i − 1, j ]} if i,j>0 and x  y
i
j

Chapter 15
P.47
Chapter 15
P.48
Computer Theory Lab.
Computer Theory Lab.
9
do if xi = yj
10
then c[i, j]  c[i-1, j-1]+1
11
b[i, j]  “”
12
else if c[i–1, j]  c[i, j-1]
13
then c[i, j]  c[i-1, j]
14
b[i, j]  “”
15
else c[i, j]  c[i, j-1]
16
b[i, j]  “”
17 return c and b
Chapter 15
P.49
Complexity: O(mn)
Chapter 15
Computer Theory Lab.
Computer Theory Lab.
PRINT_LCS
15.5 Optimal Binary search trees
PRINT_LCS(b, X, c, j )
1 if i = 0 or j = 0
Complexity: O(m+n)
2
then return
3 if b[i, j] = “”
4
then PRINT_LCS(b, X, i-1, j-1)
5
print xi
6 else if b[i, j] = “”
7
then PRINT_LCS(b, X, i-1, j)
8 then PRINT_LCS(b, X, i, j-1)
Chapter 15
P.50
P.51
cost:2.80
Chapter 15
cost:2.75
optimal!!
P.52
Computer Theory Lab.
Computer Theory Lab.
expected cost
n
n
i =1
i =0
 pi +  qi = 1
◼
the expected cost of a search in T is
n
n
E[search cost in T] =  (depth T (ki ) + 1)  pi +  (depth T (d i ) + 1)  qi
i =1
i =0
n
n
i =1
i =0
◼
= 1 +  depth T (ki )  pi +  depth T (d i )  qi
Chapter 15
P.53
For a given set of probabilities, our goal is to
construct a binary search tree whose expected
search is smallest. We call such a tree an
optimal binary search tree.
Chapter 15
P.54
Computer Theory Lab.
Computer Theory Lab.
Step 1: The structure of an optimal binary
search tree
Consider any subtree of a binary search tree. It
must contain keys in a contiguous range ki, ...,kj,
for some 1  i  j  n. In addition, a subtree that
contains keys ki, ..., kj must also have as its leaves
the dummy keys di-1, ..., dj.
◼ If an optimal binary search tree T has a subtree T'
containing keys ki, ..., kj, then this subtree T' must
be optimal as well for the subproblem with keys
ki, ..., kj and dummy keys di-1, ..., dj.
◼
Chapter 15
P.55
Step 2: A recursive solution
j
j
l =i
l =i −1
w(i, j ) =  pl +  ql
qi−1
if j = i - 1,

e[i, j ] = 
min{e[i, r − 1] + e[r + 1, j ] + w(i, j )} if i  j.
 ir j
Chapter 15
P.56
Computer Theory Lab.
Computer Theory Lab.
Step 3:computing the expected search
cost of an optimal binary search tree
OPTIMAL-BST(p,q,n)
1
2
3
4
5
6
7
8
Chapter 15
9
for r  i to j
10
do t  e[i, r –1]+e[r +1, j]+w[i, j]
11
if t < e[i, j]
12
then e[i, j]  t
13
root[i, j]  r
14
return e and root
3
◼ the OPTIMAL-BST procedure takes (n ), just
like MATRIX-CHAIN-ORDER
for i  1 to n + 1
do e[i, i – 1]  qi-1
w[i, i – 1]  qi-1
for l  1 to n
do for i  1 to n – l + 1
do j  i + l – 1
e[i, j]  
w[i, j]  w[i, j – 1] + pj+qj
P.57
Chapter 15
P.58
Computer Theory Lab.
Computer Theory Lab.
The table e[i,j], w[i,j], and root[i,j] computer by
OPTIMAL-BST on the key distribution.
◼
Knuth has shown that there are always roots
of optimal subtrees such that root[i, j –1] 
root[i+1, j] for all 1  i  j  n. We can use this
fact to modify Optimal-BST procedure to run
in (n2) time.
Chapter 15
P.59
Chapter 15
P.60
Computer Theory Lab.
16.1 An activity-selection
problem
Suppose we have a set S = {a1, a2, ..., an} of n
proposed activities that with to use a
resource. Each activity ai has a start time si
and a finish time fi, where 0  si < fi < .
 If selected, activity ai take place during the
half-open time interval [si, fi). Activities ai and
aj are compatible if the intervals [si, fi) and
[sj, fj) do not overlap (i.e., ai and aj are
compatible if si  fj or sj  fi).

16.Greedy algorithms
Chapter 16
Computer Theory Lab.


Example:
Chapter 16
1
2
3
4
5
6
7
8
9
10 11
si
1
3
0
5
3
fi
4
5
6
7
8
5
6
8
8
2
9
10 11 12 13 14
Computer Theory Lab.
We shall solve this problem in several steps. We
start by formulating a dynamic programming
solution to this program in which we combine
optimal solutions to two subproblems to form an
optimal solution to the original problem.
 We shall then observe that we need only
consider one choice – the greedy choice – and
that when we make the greedy choice, one of
the subproblems is guaranteed to be empty, so
that only one nonempty subproblem remains.
The activity-selection problem is to select a
maximum-size subset of mutually compatible
activities.
i
P.2
12
P.3
Chapter 16
P.4
Computer Theory Lab.
Computer Theory Lab.
The optimal substructure of the
activity-selection problem
A recursive solution
Sij={akS : fi  sk < fk  sj},
So that Sij is the subset of activities in S
that can start after activity ai finishes
and finish before activity aj starts.
 f0=0 and sn+1 = . Then S = S0,n+1, and
the ranges for i and j are given by 0  i,
j  n+1.
 Aij = Aik {ak} Akj

Chapter 16
c[i, j ]  c[i, k ]  c[k , j ]  1
0
if Sij  0

c[i, j ]  
{c[i, k ]  c[k , j ]  1} if Sij  0
max
ik  j
P.5
Chapter 16
P.6
Computer Theory Lab.
Computer Theory Lab.
Converting a dynamic-programming
solution to a greedy solution

Chapter 16
A recursive greedy algorithm
Theorem 16.1
RECURSIVE-ACTIVITY-SELECTOR(s, f, i, j)
Consider any nonempty subproblem Sij, and let am
be the activity in Sij with the earliest finish time:
fm = min { fk : ak  Sij}.
then
1. Activity am is used in some maximum-size subset
of mutually compatible activities of Sij.
2. The subproblem Sim is empty, so that choosing am
leaves the subproblem Smj as the only one that
may be nonempty.
1 m  i +1
2 while m < j and sm < fi
Find the first activity in Sij
3
do m  m + 1
4 if m < j
5
then return {am}RAS(s, f, m, j)
6
else return 0
P.7
Chapter 16
P.8
Computer Theory Lab.
Computer Theory Lab.
The operation of RECURSIVE-ACTIVITY-SELECTOR
on the 11 activities given earlier
Chapter 16
P.9
Chapter 16
Computer Theory Lab.
Computer Theory Lab.
16.2 Elements of the greedy
strategy
An iterative greedy algorithm
GREEDY-ACTIVITY-SELECTOR(s, f)
1
2
3
4
5
6
7
8
Chapter 16
P.10
1. Determine the optimal substructure of the
problem.
2. Develop a recursive solution.
3. Prove that at any stage of the recursion, one
of the optimal choices is the greedy choice.
Thus, it is always safe to make the greedy
choice.
n  length[s]
a  {a1}
i1
for m  2 to n
do if sm  fi
then A  A  {am}
im
return A
P.11
Chapter 16
P.12
Computer Theory Lab.
Computer Theory Lab.
Designing a greedy algorithm
4. Show that all but one of the subproblems
induced by having make the greedy choice are
empty.
5. Develop a recursive algorithm that implements
the greedy strategy.
6. Convert the recursive algorithm to an iterative
algorithm.
 Alternatively, we could have fashioned our
optimal substructure with a greedy choice in
mind.
Chapter 16
P.13
1. Cast the optimization problem as one in which we
make a choice and are left with one subproblem to
solve.
2. Prove that there is always an optimal solution to the
original problem that makes the greedy choice, so
that the greedy choice is always safe.
3. Demonstrate that, having made the greedy choice ,
what remains is a subproblem with the property that
if we combine an optimal solution to the subproblem
with the greedy choice we have made, we arrive at
an optimal solution to the original problem.
Chapter 16
P.14
Computer Theory Lab.
Computer Theory Lab.
Greedy versus dynamic programming


Chapter 16
Greedy-choice property: A global
optimal solution can be achieved by
making a local optimal (optimal) choice.
0-1 knapsack problem
 fractional knapsack problem
 (Exercise 16.2.2)

Optimal substructure: An optimal
solution to the problem within its
optimal solution to subproblem.
P.15
Chapter 16
P.16
Computer Theory Lab.
Computer Theory Lab.
16.3 Huffman codes
The greedy strategy does not work for the 0-1 knapsack

Chapter 16
P.17
a
b
c
d
e
f
Frequency (in hundred)
45
13
12
16
9
5
Fixed length codeword
000
001 010 011 100
101
Variable length codeword
0
101 100 111 1101
1100
Prefix code: no codeword is also a prefix
of some other codeword.
Chapter 16
P.18
Computer Theory Lab.
Computer Theory Lab.
Tree correspond to the coding schemes



Can be shown that the optimal data
compression achievable by a character
code can always be achieved with
prefix codes.
Simple encoding and decoding.
An optimal code for a file is always
represented by a binary tree.
B(T )   f (c)dT (c)
cC
Chapter 16
P.19
Chapter 16
cost of tree T
P.20
Computer Theory Lab.
Computer Theory Lab.
The steps of Huffman’s algorithm
Constructing a Huffman code
HUFFMAN( C )
Chapter 16
1 n  |C|
2 QC
3 for i  1 to n – 1
4
do allocate a new node z
5
left[z]  x  EXTRACT-MIN(Q)
6
right[z]  y  EXTRACT-MIN(Q)
7
f[z]  f[x] + f[y]
8
INSERT(Q,Z)
9 return EXTRACT-MIN(Q)
Complexity: O(n log n)
P.21
Chapter 16
P.22
Computer Theory Lab.
Computer Theory Lab.
Correction of Huffman’s algorithm

Lemma 16.2.
Let C be an alphabet in which c  C has
frequency f [c]. Let x and y be the two
characters in C having the lowest
frequencies. Then there exists an optimal
prefix code in C in which the codeword for x
and y having the same length and differ
only in the last bit.
Chapter 16
P.23
Chapter 16
P.24
Computer Theory Lab.
Computer Theory Lab.
Proof.
B(T )  B(T ' )   f (c)d (c)   f (c)d (c)
cT
T
cT
T
...  0
B(T )  B(T " )  0
Chapter 16
P.25
Chapter 16
P.26
Computer Theory Lab.
Computer Theory Lab.
Lemma 16.3

Lemma 16.3 (cont)
Let T' be any tree representing an optimal
prefix code for the alphabet C'. Then the tree
T, obtained from T' by replacing the leaf node
for z with an internal node having x and y as
children, represents an optimal prefix code
for the alphabet C.
Let C be a given alphabet with frequency f[c]
defined for each character c  C. Let x and y be
two characters in C with minimum frequency.
Let C’ be the alphabet C with characters x, y
removed and character z added, so that C’ = C {x, y}{z}; define f for C' as for C, except that
f[z] = f[x] + f[y].

Proof
B(T) = B(T') + f[x] + f[y]
Chapter 16
P.27
Chapter 16
P.28
Computer Theory Lab.
Theorem 16.4

Chapter 16
Procedure HUFFMAN produces an
optimal prefix code.
P.29
22.1 Representations of graphs
• adjacency-matrix representation (dense)
• adjacency-list representation (sparse)
22.Elementary Graph
Algorithms
1
2
3
4
5
1 0
2 1
1
1 0 0 1
0 1 1 1


3 0 1 0 1 0 
4 0 1 1 0 1 


5 1 1 0 1 0 


2
3
5
4
1
2
5
2
1
5
3
2
4
4
2
5
3
5
4
1
2
3
4
Chapter 22
Weighted graphs:
1 0
2
3
4
1
2
3
5
6
4
Chapter 22
5
22.2 Breadth-first search
w: E  R
1
0

0
0

0

0
2
3
4
5
BFS(G, s)
1 for each vertex u V ( G )  { s }
6
2 do color [ u ]  white
1 0 1 0 0
0 0 0 1 0

0 0 0 1 1
1 0 0 0 0 
0 0 1 0 0

0 0 0 0 1
1
2
2
5
3
6
4
2
5
4
6
6
P.2
4
6
5
10 do u  head [ Q ]
11 for each v  adj [ u ]
3 d[ u]  
12 do if color [ v ]  white
4  [ u ]  NIL
13 then color [ v ]  gray
5 color [ s ]  gray
14 d [ v ]  d [ u ] 1
6 d[ s]  0
15  [ v ]  u
7  [ s ]  NIL
16 ENQUEUE(Q, v)
8 Q  { s}
17 DEQUEUE(Q)
9 while Q  
18 color [ u ]  BLACK
Analysis: O(V+E )
P.4
Chapter 22
P.5
The operation of BFS
Chapter 22
The operation of BFS
P.6
The operation of BFS
Chapter 22
P.7
Shortest paths
 ( s, v ): shortest path from s to v
Lemma 22.1.
Let G = (V, E) be a directed or undirected graph,
and let s V be an arbitrary vertex.
Then for any edge
( u, v )  E ,  ( s, v )   ( s, u ) 1.
Chapter 22
P.8
Chapter 22
P.9
Lemma 22.2. Let G = (V, E) be a directed or undirected graph, and
suppose that BFS is run on G from a given source s V .
Then upon
termination, for each vertex v V , the value d[v] computed by BFS
Corollary 22.4. Suppose vertices vi and vj are enqueued during the
satisfies d [ v ]   ( s, v ) .
execution of BFS, and that vi is enqueued before vj is enqueued.
Proof. (Induction on the number of times a vertex is placed in the
Then d [v ]  d [v ] at the time that vj is enqueued.
i
queue)
proof
j
Immediate form Lemma 22.3 and the property that each
vertex receives a finite d value at most once during the course of
Lemma 22.3. Suppose that during the execution of BFS on a graph
BFS.
G = (V, E), the queue Q contains the vertices  v1 , v2 ,..., vr  , where
v1 is the head of Q and vr is the tail.
Then d [ vr ]  d [ v1 ]  1
We can now prove that breadth-first search correctly finds
and d [ vi ]  d [ vi 1 ] for i=1,2,…,r-1.
shortest-path distances.
Proof. (induction on the number of queue operations)
Chapter 22
P.10
Chapter 22
P.11
Theorem 22.5 (Correctness of BFS)
For a graph G = (V, E) with source s, we define the predecessor subgraph
Let G = (V, E) be a directed or undirected graph, and suppose
of G as G  ( V , E ) where V  { v V | [ v ]  NIL }  { s } , and
that BFS is run on G from a given source s V . Then, during
its execution, BFS discovers every vertex v V that is reachable
from the source, and upon termination d [ v ]   ( s, v ) for all v V .
Chapter 22
E  {(  [ v ], v )  E | v V  { s }} . The edges in E are called tree
edges.
Moreover, for any v  s that is reachable from s, one of the
shortest paths from s to v is the shortest path from s to  [ v ]
Lemma 22.6. When applied to a directed or undirected graph G = (V, E),
followed by the edge (  [ v ], v ) .
procedure BFS constructs 
Proof.
G  ( V , E ) is a breadth-first tree.
By induction.
P.12
Chapter 22
so that the predecessor subgraph
P.13
22.3 Depth-First Search
PRINT_PATH(G, s, v )
1 if v = s
2
then print s
3
else if π[v]=NIL
4
then print “no path from” s “to” v
“exist”
5
else PRINT-PATH(G, s, π[v])
6
print v
DFS(G)
DFS-VISIT(u)
1 for each vertex u V [ G ]
1 color [ u ]  gray
2 do color [ u ]  white
2 d [ u ]  time  time 1
3  [ v ]  NIL
3 for each v  adj [ u ]
4 time  0
4 do if color [ u ]  white
5 for each vertex u V [ G ]
5 then  [ v ]  u
6 do if color [ u ]  white
6 DFS-VISIT(v)
7 then DFS-VISIT(u)
7 color [ u ]  black
8
Chapter 22
P.14
Chapter 22
f [ u ]  time  time 1
P.15
The progress of DFS
predecessor subgraph:
depth-first forest,
depth-first tree
Time stamps: d(u) discovered
f(u) finished
Complexity: O(V+E)
Chapter 22
P.16
Chapter 22
P.17
The progress of DFS
Chapter 22
The progress of DFS
P.18
Chapter 22
P.19
Properties of depth-first search
The progress of DFS
Theorem 22.6. (Parenthesis theorem)
In any depth-first search of a (directed or undirected) graph G = (V, E)
for any two vertices u and v, exactly one of the following conditions
holds:

the intervals [ d ( u ), f ( u )] and [ d ( v ), f ( v )] are entirely disjoint.

the interval [ d ( u ), f ( u )] is contained entirely within the interval
[ d ( v ), f ( v )] , and u is a descendant of v in the depth-first tree, or

the interval [ d ( v ), f ( v )] is contained entirely within the interval
[ d ( u ), f ( u )] , and v is a descendant of u in the depth-first tree.
Chapter 22
P.20
Chapter 22
P.21
Property of DFS
Corollary 22.8. (Nesting of descendants’ interval)
Vertex v is a proper descendant of a vertex u in the
depth-first forest for a (directed or undirected) graph G if
and only if d ( u )  d ( v )  f ( v )  f ( u ).
Chapter 22
P.22
Chapter 22
P.23
22.4 Topological sort
Theorem 22.9 (white path theorem)
In a depth-first forest of a (directed or undirected) graph G = (V,
E), vertex v is a descendant of vertex u if and only if at time d[u]
A topological sort of a dag G = (V, E) is a linear ordering of all its
that the search discover u, vertex can be reached from u along a
vertices such that if G contains an edge (u, v), then u appears before
path consisting entirely of white vertices.
v in the ordering.
Classification of edges:
Tree edges
Back edges
Forward edges
Cross edges
Theorem 22.10.
Chapter 22
In a depth-first search of an undirected graph
G, every edge of G is either a tree edge or a back edge.
P.24
Chapter 22
P.25
TOPOLOGICAL_SORT(G)
1 call DFS(G) to compute finishing time f(v) for each
vertex v.
2 as each vertex is finished, insert it onto the front of
a
link list.
3 return the link list of vertices
Chapter 22
P.26
Chapter 22
P.27
Lemma 22.11. A directed graph G is acyclic if and only if
a depth first search of G yields no back edge.
Theorem 22.12. TOPOLOGICAL_SORT(G) produces a
topological sort of a directed acyclic graph G.
Chapter 22
P.28
Chapter 22
P.29
22.5 Strongly connected components
STRONGLY_CONNECTED_COMPONENT(G)
1 call DFS(G) to compute finishing time f[u] for
Lemma 22.13. Let C and C’ be distinct strongly connected
each vertex u
components in directed graph G = (V, E), let u, v  C, let u’,
v’  C’, and suppose that there is a path u ~> u’ in G. Then
there cannot also be a path v’ ~> v in G.
2 compute G T
3 call DFS(G), but in the main loop of DFS,
the order of decreasing
consider the vertices in
f[u].
Lemma 22.14. Let C and C’ be distinct strongly connected
4 Output the vertices of each tree in the depth- first forest of step 3 as
components in directed graph G = (V, E). Suppose that there
is an edge(u, v)  E, where u  C and v  C’. Then f(C) >
f(C’).
a separate strongly connected components
Complexity: O(V + E)
Chapter 22
P.30
Chapter 22
Corollary 22.15. Let C and C’ be distinct strongly
connected components in directed graph G = (V, E).
Suppose that there is an edge (u, v)  ET, where u  C and v
 C’. Then f(C) < f(C’).
Theorem
22.16.
Strongly-connected-component(G)
correctly computes the strongly connected components of a
directed graph G.
Chapter 22
P.32
P.31
Let G=(V,E) be a connected, undirected graph.
For each edge
(u , v)  E , we have a weight w(u,v) specifying the cost to connect
u and v. We wish to find an acyclic subset T  E that connects
23.Minimum
spanning tree
all of the vertices and whose total weight w(T )  
( u ,v )T
w(u, v) is
minimized. Since T is acyclic and connects all of the vertices, it
must form a tree, which we call a spanning tree.
problem of determine
the tree T the
minimum spanning
tree problem.
8
b
We call the
7
c
d
9
4
2
11
a
i
14
4
6
e
7
8
10
h
g
1
2
f
P.2
Chapter 23
23.1 Growing a minimum spanning tree
A cut (S,V-S) of an undirected graph G = (V,E) is a partition of V. We
GENERIC-MST(G, w)
say that an edge (u , v)  E crosses the cut (S,V-S) if one of its
1 A
endpoints is in S and the other is in V-S. We say a cut respects the set
2 while A does not form a spanning tree
A of edges if no edge in A crosses the cut.
3 do find an edge (u, v) that is safe for A
crossing a cut if its weight is the minimum of any edge crossing the cut.
4 A  A  {( u, v )}
Note that there can be more than one light edge crossing a cut in case
5 return A
Chapter 23
An edge is a light edge
of ties.
P.3
Chapter 23
P.4
Theorem 23.1.
8
b
7
c
d
4
i
14
4
6
11
b
h
g
1
2
f
i
V-S
d
c
2
crossing (S, V-S).
f
Then, edge (u, v) is
v
safe for A.
V-S
P.6
Chapter 23
P.5
Chapter 23
y
respects A, and let (u, v) be a light edge
4
10
P
u
for G, let (S, V-S) be any cut of G that
2
14
e
x
Let A be a subset of E that is
included in some minimum spanning tree
g
7
9
S
E.
1
6
8
10
real-valued weight function w defined on
7
4
e
7
8
a
9
connected undirected graph with a
h
8
2
11
a
S
Let G = (V, E) be a
23.2 The algorithms of Kruskal and Prim
Corollary 23.2.
MST_KRUSKAL(G, w)
Let G = (V, E) be a connected, undirected graph with
a real-valued weighted function w defined on E.
1 A
Let A be a subset of E
2 for each vertex v V [ G ]
that is induced in some minimum spanning tree for G, and let C be a
3 do MAKE-SET(v)
connected component (tree) in the forest G A  ( V , A ) . If (u,v) is a light
4 sort the edge of E by nondecreasing weight w
5 for each edge ( u, v )  E , in order by
edge connecting C to some other component G A , then (u, v) is safe for
nondecreasing weight
A.
6 do if FIND_SET(u)FIND_SET(v)
7 then A  A  {( u, v )}
8 UNION(u, v)
9 return A
Chapter 23
P.7
Chapter 23
Complexity O(E log E)
P.8
8
b
7
c
d
4
i
14
4
6
e
7
8
g
i
14
4
9
6
g
i
14
4
6
8
g
8
7
c
d
4
i
14
4
6
e
7
8
9
g
1
2
d
f
i
14
4
6
g
1
2
7
c
f
d
i
14
4
6
g
7
c
d
i
14
4
6
e
7
8
9
2
11
a
f
2
1
8
f
2
4
10
h
e
10
g
b
9
9
14
4
6
h
e
7
8
8
2
11
a
10
h
8
b
i
7
1
4
e
7
8
9
2
11
a
10
h
7
c
4
2
11
a
8
b
d
2
11
a
f
2
7
c
4
10
h
8
b
9
e
7
1
b
d
2
11
a
f
2
1
7
c
4
10
h
8
b
e
7
8
f
2
1
d
2
11
a
10
h
7
c
4
2
11
a
8
b
9
10
h
g
f
2
1
Continue ……
Chapter 23
P.9
P.10
Chapter 23
8
b
Prim’s algorithm
i
g
8
b
i
6
10
h
g
6 while Q  
8
b
d
4
Complexity:
O(V log V + E log V), or
O(E + V log V)
P.11
i
6
10
h
g
1
2
f
d
i
14
4
6
e
7
10
g
8
f
2
7
c
d
4
i
14
4
6
e
7
8
9
2
11
a
9
2
11
b
9
e
7
8
Chapter 23
14
4
7
c
h
2
11
a
f
2
1
7
c
8
8
f
2
1
e
4
a
9
10
g
b
9
14
4
6
h
e
7
8
7 do u  ECTRACT _ MIN ( Q )
14
4
i
7
8
2
11
a
5  [ r ]  NIL
Chapter 23
d
d
2
11
a
1
7
c
7
c
4
f
2
4
4 key [ r ]  0
11 key [ v ]  w( u, v )
e
8
b
9
10
1
3 do key [ u ]  
10 then  [ u ]  u
6
h
2 for each u  Q
14
4
7
8
1 Q V[G]
9 do if v  Q and w( u, v )  key [ v ]
d
2
11
a
MST_PRIM(G, w, r)
8 for each v  Adj [ u ]
7
c
4
10
h
g
1
2
f
P.12
8
b
7
c
d
4
11
a
i
14
4
6
e
7
8
10
h
g
8
f
2
1
b
7
c
d
4
i
14
4
6
e
7
8
10
h
g
8
f
2
1
b
7
c
d
4
i
e
10
h
g
1
Chapter 23
14
4
6
7
8
9
2
11
a
9
2
11
a
9
2
2
f
P.13
In a shortest-paths problem, we are given a weighted, directed graph
G=(V,E), with weight function w : E  R mapping edges to
real-valued weights. The weight of path
p  v0 , v1 ,..., vk  is the
sum of the weights of its constituent edges:
24.Single-Source Shortest Paths
k
w( P )   w( vi 1 , vi ) .
i 1
P.2
Chapter 24
We define the shortest-path weight from u to v by
Variants:
 Single-destination shortest path problem
P
 ( u, v )  min{ w( P )| u v } if there is a path from u to v and
~
 ( u, v )   if otherwise.
 Single pair shortest path problem
A shortest path from vertex u to vertex v is then defined as
any path p with weight w( p )   (u , v) .
 All-pair shortest paths problem
Chapter 24
P.3
Chapter 24
P.4
Negative-weight edge
Optimal substructure of a shortest path
Lemma 24.1. (Subpaths of shortest paths are shortest paths)
Given a weighted directed graph G = (V, E) with weight function
a
w: E  R , let P  v1 , v2 ,..., v k  be a shortest path from v1 to v k
6
5
5
11
2
e
3

-6
 g
8
i
2

d
-3
S
to v j .
h
4
c
0
be the subpath from vi to v j . Then Pij is a shortest path from vi
P.5
b
-1
-4
3
and, for any i and j such that 1  i  j  k , let Pij  vi , vi 1 ,..., v j 
Chapter 24
3

-8
3
7


f
j
P.6
Chapter 24
Shortest path tree is not unique
u
Representing shortest paths:
predecessor subgraph
v
3
G  ( V , E )
3
4
0
1
2
11
2
s
Shortest-path tree:
2
5
1. V is the set of vertices reachable from s in G.
x
2. G forms a rooted tree with root s.
5
3
6
3
path from s to v.
11
6
u
3. for all v V the unique simple path from s to v is a shortest
y
v
u
9
3
v
0
3
4
1
2
11
2
1
2
11
2
2
5
x
Chapter 24
0
s
2
5
P.7
9
6
4
s
Chapter 24
9
6
5
6
11
y
x
5
6
11
y
P.8
Relaxation
INITIALIZE-SINGLE-SOURCE(G, s)
u
5
1 for each vertex v V ( G )
2
v
9
u
5
v
6
2
2 do d [ v ]  
Relax(u,v)
3  [ v ]  NIL
4 d[ s]  0
u
5
2
v
7
Relax(u,v)
u
5
2
v
6
RELAX(u, v, w)
1 if d [ v ]  d [ u ]  w( u, v )
Dijkstra
2 then d [ v ]  d [ u ]  w( u, v )
Order of relaxation
3 [v]  u
Bellman Ford
Chapter 24
P.9
Chapter 24
Properties of relaxation
Properties of relaxation (cont)
Triangle inequality (Lemma 24.10)
Path-relaxation property (Lemma 24.15)
For any edge (u, v) E, we have δ(s, v) ≤ δ(s, u) + (u, v)
P.10
If p = v0, v1, …, vk is a shortest path from s = v0 to vk, and the edges
of p are relaxed in the order (v0, v1), (v1, v2), …, (vk-1, vk), then d[vk] =
δ(s, vk). This property holds regardless of any other relaxation steps
that occur, even if they are intermixed with relaxations of the edges of
p.
Upper-bound property (Lemma24.11)
We always have d[v]  δ(s, v) for all vertices v V, and once d[v]
achieves the value δ(s, v), it never changes.
No-path property (Corollary 24.12)
If there is no path from s to v, then we always have d[v] = δ(s, v) = .
Convergence property (Lemma 24.14)
Predecessor-subgraph property (Lemma 24.17)
once d[v] = δ(s, v) for all v  V, the predecessor subgraph is a
shortest-paths tree rooted at s.
If s ~> u  v is a shortest path in G for some u, v  V, and if d[u] =
δ(s, u) at any time prior to relaxing edge(u, v), then d[v] = δ(s, v) at
all times afterward.
Chapter 24
P.11
Chapter 24
P.12
BELLMAN-FORD(G, w, s)
24.1 The Bellman-Ford Algorithm
 Edge weight can be negative but no negative cycles.
(This
condition can be checked.)
1
INITIALIZE-SINGLE-SOURCE(G, s)
2
for i  1 to |V(G)| - 1
3
do for each edge (u, v)  E[G]
4
do RELAX(u, v, w)
for each edge (u, v)  E[G]
5
6
do if d[v] > d[u] + w(u, v)
7
then return FALSE
8
P.13
Chapter 24
6
u
5
v

-2

-3
8
0

6
x
y
Lemma 24.2. Let G = (V, E) be a weighted, directed graph with
source s and weight function w: E  R , and assume that G contains
5
v
u
5
v

2
-2

6
x
6
7
y
u
5
2
-2
8

9
Then at the
iterations of BELLMAN-FORD, we have d [ v ]   ( s, v ) for all
2
x
vertices that are reachable from s.

Proof.
y
Let
v
be
a
vertex
reachable
from
s,
and
let
v
P  v0 , v1 ,..., v k  be a shortest path from s to v, where v0  s and

vk  v .
-3
The path P is simple, so k |V |1 .
We can prove by
induction for i  0 ,1,..., k that d [ vi ]   ( s, vi ) .
7
-4
s
no negative weight cycles that reachable from s.
7
-4
s

9
-3
8
0
7
-4

7

9
-2
2
0

y
-3
P.14
Chapter 24
7
u
s
7
-3
-4
6
8
0
v

Complexity: O (VE )
2
7

9
x
5
-2
s
2
7
u
6
8
0
7
-4
s
2

x
Chapter 24
6
return TURE
9

y
P.15
Chapter 24
P.16
Corollary 24.3. Let G = (V, E) be a weighted, directed graph with
Theorem 24.4. (Correctness of the BELLMAN-FORD algorithm)
source s and weight function w: E  R .
Let BELLMAN-FORD be run on a weighted, directed graph G = (V, E)
Then for each vertex v V ,
there is a path from s to v if and only if BELLMAN-FORD terminates
with source s and weight function w: E  R .
with d [ v ]   when it run on G.
negative-weight cycles that are reachable from s, then the algorithm
If G contains no
returns TRUE, we have d [ v ]   ( s , v ) for all vertex v V , and the
predecessor subgraph G is a shortest-path tree rooted at s.
If G does
contain a negative-weight cycle reachable from s, then the algorithm
returns FALSE.
P.17
Chapter 24
P.18
Chapter 24
Proof.
 Suppose that G contains no negative-weight cycles that are reachable
Suppose that G contains a negative weight cycle C  v0 , v1 ,..., v k 
from source s.
where v0  v k , that is reachable from s.
 Claim: At termination, d [ v ]   ( s, v ) for all v V .
i 1
 Claim BELLMAN-FORD return FALSE.
 If v is reachable from s, then d [ v ]   ( s, v ) . [By Lemma 24.2]
Assume BELLMAN-FORD return TURE.
 If v is not reachable from s, then d [ v ]   ( s, v ) .
d [ vi ]  d [ vi 1 ]  w( vi 1 , vi ) for i = 1, 2,…,k.
 Claim BELLMAN-FORD return TRUE.
k
k
k
i 1
i 1
i 1
 d [ vi ]   d [ vi 1 ]   w( vi 1 , vi ) .
At termination, for all edge ( u, v )  E
d [ v ]   ( s, v )   ( s,u )  w( u, v )  d [ u ]  w( u, v )
k
 w( vi 1 , vi )  0 . (contradiction!)
BELLMAN-FORD return TURE.
Chapter 24
k
Then  w( vi 1 , vi )  0 .
i 1
P.19
Chapter 24
P.20
6
r
24.2 Single-source shortest paths in
directed acyclic graphs

5
s
2
0
v
-1

4
-2
x

6
r

s
5
2
0
t

u
7

6
r

3 for each vertex u taken in topological sorted order
5
s
2
0
t

-2
x
s
0
2
t
2
1
u
7
6
3

v
-1
5
4
-2
2
7
6
v
-1

4
6
-2
x
r


5
s
0
2
t
2
1
u
7
6
v
-1
5
4
3
2
-2
x
4
2
do RELAX(u, v, w)
Time complexity O(V+E)
6
r

5
s
2
0
t
2
7
P.21
Chapter 24
Theorem 24.5.
s
and
If a weighted, directed graph G = (V, E) has a source
no
cycles,
then
at
the
termination
of
DAG-SHORTEST-PATHS procedure, d [ u ]   ( s, v ) for all vertices
6
6
1
u
3
v
-1
6
4
-2
x
4
2
r

5
s
0
2
t
2
1
u
7
6
3
v
-1
5
4
-2
2
Chapter 24
P.22
Application:
critical paths in PERT chart analysis
 longest paths problem
v V , and the predecessor subgraph G is a shortest paths tree.
Can be solved by one of the following methods:
Proof. (By induction)
1. negating the edge weights and run DAG-SHORTEST-PATHS
2. running DAG-SHORTEST-PATHS, replacing  by - in Line 2 of
INITIALIZE-SINGLE-SOURCE and “>” by “<” in the RELAX
procedure.
Chapter 24
x
4
2
1
u
3
do for each vertex v  adj [ u ]
v
-1
4

5
2
1 topological sort the vertices of G
2 INITIALIZE-SINGLE-SOURCE(G, s)
r
1
3
DAG-SHORTEST-PATHS(G, w, s)
vertex

2
6
topological sort in O(V+E) time
5
7

3
dag  directed acyclic graph
4
1
u
t
P.23
Chapter 24
P.24
x
4
u
10
2 3
0

u
10
1 INITIALIZE-SINGLE-SOURCE(G, s)
5
4 while Q  
u
S  S  { u}
for each vertex v  Adj [ u ]
8
2 3
0
s
do RELAX(u, v, w)
5
Chapter 24
P.25
Chapter 24
u
8
10
6
5
x
u
9
8
10
4
2
4
5
v
6
7
5
9
s
y
v
1
9
9
2 3
0
7
6
7
2
x
9
v
13
7
y
1
y
1
2 3
0
7
6

2
s
2
x
10
v
14
7
5
do u  EXTRACT  MIN ( Q )
x
4
s
3 Q V[G]
5
y
9
2 3
0
2 S 
1
8
4
7
5


9
2 3
0
v
1
s
2
x
DIJKSTA(G, w, s)
8
10
6
7
 Assume that w( u, v )  0 for each ( u, v )  E .
7
u
10
4
s
5
6
v

9
24.3 Dijkstra’s algorithm
5
1

4
6
7
5
5
y
2
x
7
P.26
y
Let u be the first vertex for which d [ u ]   ( s , u ) when it is inserted into
set S. We focus our attention on the situation at the beginning of the iteration
in which u is inserted into S.
 u  s
There must be some path from s
Theorem 24.6 (Correctness of Dijkstra’s algorithm)
If we run Dijkstra’s algorithm on a weighted, directed graph G = (V, E)
with nonnegative weight function w and source s, then at termination,
to
d [ u ]   ( s, u ) for all vertices u V .
There exist a shortest path p from
Proof.
s to u. Path p connects a vertex
Claim: for each vertex u V ,
u.
(Otherwise
d [u ]  
).
in S to a vertex in V-S. Let us
d [ u ]   ( s, u ) at the time u is inserted
u
S
x
y
consider the first vertex y along p
into S and this equality is maintained forever.
such that p V  S .
continue…
Chapter 24
P.27
Chapter 24
P.28
d [ y ]   ( s, y ) when u is inserted into S.

Because u is chosen as the first vertex for which d [ u ]   ( s ,u ) when it is
inserted into S, we have d [ x ]   ( s , x ) .
When x was inserted into S.
Corollary 24.7.
If we run Dijkstra’s algorithm on a weighted,
Edge (x, y) was relaxed at that time, so the claim follows from Lemma
directed graph G = (V, E) with nonnegative weight function w and
25.7.
source s, then at termination, the predecessor subgraph G is a

 ( s , y )   ( s , u ) (because in p all edge weight are nonnegative)
shortest-path tree rooted at s.

d [ y ]   ( s , y )   ( s , u )  d [ u ] (by Lemma 25.5)

d [ u ]  d [ y ] (because u was chosen)

d [ y ]   ( s , y )   ( s , u )  d [ u ] (By the above two remarks).
Analysis:
Naïve:
O( V 2  E )
Modified: O( v lg v  E )
We get a
contradiction to the choice of u.

Thus, d [ u ]   ( s, u ) at the time u is inserted into S.

d [ u ]   ( s, u ) holds thereafter (Lemma 25.5).
P.29
Chapter 24
P.30
Chapter 24
Systems of difference constraints
x j  xi  bk ,
24.4 Difference constraints and
shortest paths
where 1  i, j  n and 1  k  m
Linear Programming:
n
max  ci xi
i 1
subject to Ax  b
(m constraints)
0
0
 1 1 0
 0 


 
1
0
0
0

1

  x    1
1
0
1
0
0  1    1 

 x2
 
1
0
0    5 
1 0
 x3  
1 0
0
1
0    4 

 x4
 
0 1 1
0      1
0
 x5 
0
  3
0 1 0
1


 
0
  3
0
0 1 1 
 simplex algorithm
 ellipsoid algorithm
 Karmarkar’s algorithm
feasible solution
feasible problem
Chapter 24
P.31
Chapter 24
x1  x2  0
x1  x5  1
x2  x5  1
x3  x1  5
x4  x1  4
x4  x3  1
x5  x3  3
x5  x5  3
P.32
Theorem 24.9.
Lemma 24.8. Let x   x1 , x2 ,..., x n  be a solution to a system
Given a system Ax  b of difference constraints.
Let G = (V, E) be the corresponding constraint graph. If G contains no
Ax  b of difference constraints, and d be any constant. Then
negative weight cycles then x   ( v0 , v1 ),..., ( v0 , v n ) is a feasible
x  d   x1  d , x2  d ,..., x n  d  is a solution to Ax  b as well.
solution for the system.
If G contains a negative-weight cycle, then
there is no feasible solution for the system.
0
Constraint graph
-5
0
Example: [-5, -3, 0, -1, -4]
v1
vo
0
0
1
-4
v5
4
-3
0
-3
-1
v4
[d-5, d-3, d, d-1, d-4]
0
-1
-1
0
Proof. (1) Assume that G contains no negative cycles. We consider
-3
v2
( vi , v j ) .
5
 ( vo , v j )   ( v0 , vi )  w( vi , v j )
0
  ( vo , v j )   ( v0 , vi )  w( vi , v j )
v3
 x j  xi  w( vi , v j )
P.33
Chapter 24
Chapter 24
(2) Assume that G contains negative-weight cycles.
Let
C  v1 , v2 ,..., v k  be a negative weight cycle with v1  v k .
Note that v0  C because no entering edge.
x1  x2  w( v1 , v2 )
x2  x3  w( v2 , v3 )

x1  x k  w( v k , v1 )
P.34
24.5 Proofs of shortest-paths properties
Lemma 24.10. (Triangle inequality)Let G = (V, E) be a weighted,
directed graph with weight function w: E  R and source vertex s.
Then for all edge ( u, v )  E , we have  ( s , v )   ( s ,u )  w( u, v ) .
0  w( C )  0 (Contradiction!)
 Complexity: O(( n  1)( n  m ))  O( n2  nm )
 Can be reduced to O(nm).
Chapter 24
P.35
Chapter 24
P.36
Effects of relaxation on shortest-path
estimates
No-path property
Lemma 24.11. (Upper-bound property)
Corollary 24.12. Suppose that in a weighted directed graph
G = (V, E) with weight function w: E  R no path connects a
Let G = (V, E) be a weighted, directed graph with weight function
w: E  R . Let s V be the source vertex, and let the graph be
initialized by INITIALIZE-SINGLE-SOURCE(G, s).
Then
d [ v ]   ( s, v ) for all v V and this invariant is maintained over any
sequence of relaxation steps on the edges of G.
Moreover, once d[v]
achieves its lower bound  ( s, v ), it never changes.
Chapter 24
source vertex s V to a given vertex v V . Then after the
graph is initialized by INITIALIZE-SINGLE-SOURCE(G, s),
we have d [ v ]   ( s, v ) =∞, and this equality maintained as an
invariant over any sequence of relaxation steps on the edges of
G.
P.37
P.38
Chapter 24
Convergence property
Lemma 24.13. Let G = (V, E) be weighted, directed graph
Lemma 24.14. Let G = (V, E) be a weighted directed graph
with weight function w : E  R, and let (u, v)  E. Then,
with weight function w: E  R , let s V be a source vertex,
immediately after relaxing edge (u, v) by executing
and let s ~ u  v be a shortest path in G for some vertices
u , v V
RELAX(u, v, w), we have d[v] ≤ d[u] + w(u, v).
.
Suppose
that
G
is
initialized
INITIALIZE_SINGLE_SOURCE(G, s) and then
by
a sequence
of relaxation steps that includes the call RELAX(u, v, w) is
executed on the edges of G.
If d [ u ]   ( s, u ) at any time
prior to the call, then d [ v ]   ( s , v ) at all time after the call.
Chapter 24
P.39
Chapter 24
P.40
Path-relaxation property
Relaxation and shortest-paths trees
Lemma 24.15. Let G = (V, E) be a weighted, directed graph with
Lemma 24.16. Let G = (V, E) be a weighted directed
weighted function w: E  R, and s  V be a source vertex.
Consider any shortest path p = v0, v1, …, vk from s = v0 to vk. If
graph with weight function w: E  R , let s V be a
G is initialized by INITIALIZE-SINGLE-SOURCE(G, s) and then a
source vertex, and assume that G contains no negative
sequence of relaxation steps occurs that includes, in order,
cycles that are reachable from s. Then after the graph is
relaxations of edges (v0, v1), (v1, v2), …, (vk-1, vk), then d[vk] =
δ(s, vk) after these relaxations and at all times afterward. This
initialized by INITIALIZE_SINGLE_SOURCE(G, s),
property holds no matter what other edge relaxations occur,
the predecessor subgraph G forms a rooted tree with
including relaxations that are intermixed with relaxations of the
root s, and any sequence of relaxation steps on edges of
edges of p.
G maintains this properties as an invariant.
Chapter 24
P.41
Predecessor-subgraph property
Lemma 24.17. Let G = (V, E) be a weighted, directed graph
with weight function w: E  R, let s  V be a source vertex,
and assume that G contains no negative-weight cycles that
are reachable from s. Let us call INITIALIZE-SINGLESOURCE(G,s) and then execute any sequence of relaxation
steps on edges of G that produces d[vi] =δ(s, v) for all v  V.
Then the predecessor subgraph Gπis a shortest-paths tree
rooted at s.
Chapter 24
P.43
Chapter 24
P.42
Computer Theory Lab.
Polynomial: A( x ) =
30.Polynomials and the FFT
n −1

i =0
ai x i
n: degree bound
a0 ,a1 ,...,an −1 coefficient
degree k
Chapter 11
P.2
Computer Theory Lab.
Computer Theory Lab.
Polynomial addition
A( x ) =
C( x ) =
n −1

j =0
n −1

j =0
a jx j ,
B( x ) =
Polynomial multiplication
n −1

j =0
bjx j
2 n −2
C( x ) =  c j x j ,
j =0
c jx j , c j = a j + bj
Example:
( 6 x 3 + 7 x 2 − 10 x + 9 )  ( −2 x 3 + 4 x − 5 ) =
Example:
3
2
− 12 x 6 − 14 x 5 + 44 x 4 − 20 x 3 − 75 x 2 + 86 x − 45
3
( 6 x + 7 x − 10 x + 9 ) + ( −2 x + 4 x − 5 )
3
2
=
4
x
+
7
x
− 6x + 4
Chapter 11
j
c j =  ak b j − k
k =0
P.3
Chapter 11
P.4
Computer Theory Lab.
Computer Theory Lab.
30.2 Representation of polynomial
degree( C ) = degree( A ) + degree( B )
degree bound(C) 
degree bound(A) + degree bound(B)
Chapter 11
◼
coefficient representation
A( x ) =
n −1

j =0
a j x j  a = ( a0 , a1 ,..., an −1 )
• evaluating the polynomial
A(x) at a given point
P.5
Chapter 11
Computer Theory Lab.
P.6
Computer Theory Lab.
Horner’s rule
◼
A( x0 ) =
◼
a0 + x( a1 + x0 ( a2 +...+ x0 ( a n −2 + x0 a n −1 )... ))
◼
◼
◼
Chapter 11
P.7
Chapter 11
Time complexity: ( n )
Adding two polynomial: (n)
Multiplication of two polynomial
the convolution of the input
c = ab
vectors a and b
Time complexity ? ( n2 ) ?
P.8
Computer Theory Lab.
Computer Theory Lab.
Theorem 30.1 (Uniqueness of
an interpolating polynomial)
Point-value representation
◼
{( x0 , y0 ), ( x1 , y1 ),..., ( xn−1 , yn−1 )} such that
◼
all xi are distinct and yi = A( xi )
for 0  i  n − 1
 A( x ) =
n −1

j =0
a jx j
(interpolation)
Chapter 11
P.9
Computer Theory Lab.
Proof.
 x0n −1   a0   y0 

 

2
n
−
1
x1
 x1   a1   y1 
=

.

.      

 

x n2−1  x nn−−11   an −1   y n −1 
1 x
0

1 x1

.
.
1 x

n −1
Aa = y
det A =
x02

jk
Chapter 11
P.10
Computer Theory Lab.
A faster algorithm for n-point
interpolation is based on
Langrange’s formula:
A( x ) =
( xk − x j )  0
 ( n3 ) algorithm by LU decomposition
Chapter 11
For any set {( x0 , y0 ),( x1 , y1 ),...,( xn−1 , yn−1 )}
of n point-value pairs, there is a
unique polynomial A(x) of degreebound n such that yk = A( xk )
for k = 0 ,1,..., n − 1 .
◼
P.11
Chapter 11
n −1

k =0
yk

(x − xj)

( xk − x j )
jk
jk
2
Takes ( n ) (Exercise 32.1-4)
P.12
Computer Theory Lab.
Computer Theory Lab.
Addition:
◼
C( x ) = A( x ) + B( x )
The point-value representation is
quite convenient for many
operations on polynomial
A  {( x0 , y0 ),...,( x n −1 , y n −1 )}
B  {( x0 , y'0 ),...,( x n −1 , y'n −1 )}
C  {( x0 , y0 + y'0 ),...,( x n −1 , y n −1 + y'n −1 )}
Takes ( n )
Chapter 11
P.13
Chapter 11
Computer Theory Lab.
P.14
Computer Theory Lab.
Multiplication:
C( x ) = A( x ) B( x )
A  {( x0 , y0 ),...,( x n −1 , y n −1 )}
◼
B  {( x0 , y'0 ),...,( x n −1 , y'n −1 )}
C  {( x0 , y0 y'0 ),...,( x n −1 , y n −1 y'n −1 )}
Takes ( n )
Chapter 11
P.15
Chapter 11
Fast multiplication of
polynomials in coefficient form:
Can we use the linear-time
multiplication method for
polynomials in point-value form to
expedite polynomial multiplication
in coefficient form?
P.16
Computer Theory Lab.
Computer Theory Lab.
Theorem 30.2
◼
◼
◼
◼
Double degree bound
Evaluate
Pointwise multiply
Interpolate
◼
Chapter 11
P.17
coefficient
representation
a0 , a1 ,..., a2 n −1
b0 , b1 ,..., b2 n −1
Interpolation
( n log n )
( n log n )


A( 12 n ),
B( 12 n )
A( 22nn −1 ), B( 22nn −1 )
30.2 The DFT and FFT
Chapter 11
◼

C( 22nn −1 )
pointwise
representation
Complex roots of unity:  n = 1

e2ik / n
k = 0 ,1,..., n − 1
 iu
e = cos u + i sin u
 = e2i / n : principal nth root of
C( 20n )
pointwise
multiplication C( 12 n )
( n )
P.18
Computer Theory Lab.
c0 , c1 ,..., c2 n −1
( n2 )
Evaluation
B( 20n )
Chapter 11
Computer Theory Lab.
Ordinary multiplication
A( 20n ),
The product of two polynomial of
degree-bound n can be computes
in time ( n log n ) with the input
and output representation in
coefficient form.
P.19
n
unity. The n complexity nth root of
unity are  n0 , n1 ,..., n n −1 .
Chapter 11
P.20
Computer Theory Lab.
Computer Theory Lab.
Lemma 30.3 (Cancellation
Lemma)
◼
Corollary 30.4
For any integers n  0 , k  0 ,
dk =  k .
and d > 0,  dn
n
Chapter 11
◼
For any even integer n > 0,
 nn / 2 = 2 = −1
P.21
Chapter 11
P.22
Computer Theory Lab.
Computer Theory Lab.
Lemma 30.6 (Summation
Lemma)
Lemma 30.5 (Halving Lemma)
◼
If n > 0 is even, then the squares
of the n complex nth roots of unity
are the n/2 complexity (n/2)th
roots of unity.
Chapter 11
P.23
◼
For any integer n  1 and
nonnegative integer k not divisible
by n, n−1 (  k ) j = 0 .
n
j =0
Chapter 11
P.24
Computer Theory Lab.
Computer Theory Lab.
The DFT
The FFT:
n −1
A( x ) =  a j x j
◼
j =0
n −1
y k = A(  nk ) =  a j nkj
j =0
◼ The vector y = y0 , y1 ,..., y n −1  is the
Discrete Fourier transform (DFT) of the
coefficient vector a = a0 ,a1 ,...,an −1  .
We write y = DFTn ( a ) .
Chapter 11
P.25
method: Fast Fourier transform
n
−1
[
0
]
2
A ( x ) = a0 + a2 x + a4 x +...+ a n − 2 x 2
n
−1
[
1
]
2
A ( x ) = a1 + a3 x + a5 x +...+ a n −1 x 2
A( x ) = A[ 0 ] ( x 2 ) + xA[ 1 ] ( x 2 )
Chapter 11
P.26
Computer Theory Lab.
Computer Theory Lab.
RECURSIVE-FFT(a)
◼
◼
Chapter 11
n
Evaluate the degree-bound
2
polynomial A[ 0 ] ( x ), A[1 ] ( x )
at (  n0 )2 ,( 1n )2 ,...,(  nn −1 )2
Combine the result using the
above equation.
P.27
1 n  length[ a ]
2 if n=1
3 then return a
4  n  e2i / n
5  1
6 a [ 0 ]  ( a0 ,a2 ,...,an − 2 )
7 a [1 ]  ( a1 ,a3 ,...,an −1 )
Chapter 11
P.28
Computer Theory Lab.
Computer Theory Lab.
RECURSIVE-FFT(a)
8 y [ 0 ]  RECURSIVE( a [ 0 ] )
9 y [1 ]  RECURSIVE( a [1 ] )
n
10 For k  1 to 2 − 1
11 Do yk  yk[ 0 ] + yk[1 ]
[0 ]
[1 ]
12 yk + n  yk − yk
2
13    n
14 return y
T ( n ) = 2T ( n / 2 ) + ( n )
= ( n log n )
Chapter 11
P.29
Chapter 11
P.30
Computer Theory Lab.
Computer Theory Lab.
Interpolation at the complexity
roots of unity
Theorem 30.7
y = Vn a
◼
1
1

1
1

 y0  
2
n −1   a0 
n
 n

 1  n


y
a1 
4
2
(
n
−
1
)
 1  = 1  2


n
 n
n
   
  

 


 y n −1  
2   a n −1 
n
−
1
2
(
n
−
1
)
(
n
−
1
)
n
n
1  n

a = DFT −1 ( y)  Vn
Chapter 11
For j, k = 0, 1, …, n-1 the (j, k)
entries of Vn −1 is  n− kj / n .
1 n −1
a j =  y k  n− kj
n k =0
−1
P.31
Chapter 11
P.32
Computer Theory Lab.
Computer Theory Lab.
Theorem 30.8 (Convolution
theorem)
DFT −1
◼
modify FFT by switch the role of a
and y, replace  n by  n −1and
divide the element of the result by
n. Hence, it takes ( n log n ) time.
Chapter 11
◼
For any two vectors a and b of length
n where n is a power of 2,
a  b = DFT2−n1( DFT2 n ( a ) * DFT2 n ( b ))
where the vectors a and b are
padded with 0’s to length 2n and *
denote the componentwise product of
two 2n-element vectors.
P.33
Chapter 11
P.34
Polynomial time algorithms: on inputs of size n, their
k
worst-case running time is O ( n ) .
34.NP Completeness
It is natural to wonder whether all problems can be solved
in polynomial time. The answer is no. For example, the
Halting Problem.
Chapter 34
P.2
Generally, we think of problems that are solvable by
polynomial-time algorithms are being tractable, and problems
that requires superpolynomial time are being intractable.
Polynomial
The subject of this chapter, however, is an interesting class of
problems, called the “ NP-complete” problems, whose status is
unknown. No polynomial-time algorithm has yet been
discovered for an NP-computer problem, nor has anyone yet
been able to prove that no polynomial-time algorithm can exist
for any one of them. This so-called P ≠ NP question has been
one of the deepest, most perplexing open research problems in
theoretical computer science since it was first posed in 1971.
Intractables
?
NP-Complete
Problem
Chapter 34
P.3
Chapter 34
P.4
The difference between these problems
NP-complete problem: status are unknown.
• Shortest vs. longest simple paths:
• Euler tour vs. hamiltonian cycle:
• 2-CNF satisfiability vs. 3 CNF satisfiablility
• NP-completeness and the classes P and NP
• Overview of showing problems to be NPcomplete
• Decision problems vs. optimization problems
If any single NP-complete problem can be solved in
polynomial time, then every NP-complete problem has a
polynomial time algorithm.
To become a good algorithm designer, you must
understand
the
rudiments
of
the
theory
of
NP-completeness.
Chapter 34
P.5
Chapter 34
P.6

Reductions
Polynomial-time
reduction algorithm

Polynomial-time
Algorithm to decide B
Polynomial-time algorithm to decide A
Suppose that there is a different decision problem,
say B, that we already know how to solve in
polynomial time. Finally, suppose that we have a
procedure that transforms any instance  of A into
some instance  of B with the following
characteristics:
yes
no
no
We can call such a procedure a polynomial-time
reduction algorithm and, it provides us a way to
solve problem A in polynomial time:
1.Given an instance  of problem A, use a polynomialtime reduction algorithm to transform it to an instance 
of problem B.
2.Run the polynomial-time decision algorithm for B on the
instance .
3.Use the answer for  as the answer for .
1.The transformation takes polynomial time.
2.The answer are the same. That is, the answer for  is “yes”
if and only if the answer for  is also “yes.”
Chapter 34
yes
P.7
Chapter 34
P.8
34.1 Polynomial time
A First NP-complete problem
Polynomial time solvable problem are regarded as tractable.

• Because the technique of reduction relies on having a problem
already known to be NP-complete in order to prove a different
problem NP-complete, we need a “first” NPC problem.
• Circuit-satisfiability problem
Even if the current best algorithm for a problem has a
running time of (n100), it is likely that an algorithm with
a much better running time will soon be discovered.

Problems for many reasonable models of computation,
can be solved in one model can be solved in polynomial
in another.

Chapter 34
P.9
Abstract Problems:
Polynomial-time solvable problems has a nice closure
property.
f , g are polynomial
 f ( g ) is also polynomial
Chapter 34
P.10
An abstract problem Q is a binary
relation on a set of problem instances and a set S of
problem solutions.
We call a problem whose instance sets is the set of binary
Decision problems: those having yes/no solution.
strings a concrete problem.
Optimization problems: recast by imposing a bound on the
We say that an algorithm solves a concrete problem in time
value to be optimized.
O( T ( n )) if when it is provided a problem instance i if
length n=|i|, the algorithm can produce the solution in at
An encoding of a set S of abstract objects is a mapping e
most O( T ( n )) time.
from S to the set of binary string, for example:
Chapter 34
{0,1,2,3,…}={0,1,10,11,…}
P.11
Chapter 34
P.12
Abstract problem  concrete problem
e: I 
{ 0 ,1 } *
encoding
A concrete problem is polynomial-time solvable if there
exists an algorithm to solve it in time
O( n k )
for some
constant k.
Problem
input k
complexity O(k)
unary
k  11...1
( k )
binary
n  lg k 
( k )  ( 2 n )
The complexity class P is the set of concrete decision
problems that are solvable in polynomial time.
Chapter 34
P.13
Chapter 34
P.14
We say that a function f : { 0 ,1 }*  { 0 ,1 } * is
polynomial-time computable if there exists a
polynomial-time algorithm A that given any x { 0 ,1 } * ,
For any set I of problem instances, we say that two
produces as output f ( x ) .
encodings e1 and e2 are polynomial related if there exist
two polynomial-time computable functions f12 and f 21
such that for any i  I , we have f12 ( e1( i ))  e2 ( i ) and
f21( e2 ( i ))  e1( i ) .
Chapter 34
P.15
Chapter 34
P.16
A formal-language framework

An alphabet  is a finite set of symbols.

A language L over  is any set of strings made up of
symbols from  .

empty string:  .

empty language:  .
Lemma 34.1. Let Q be an abstract decision problem on
an instance set I, let e1 and e2 be polynomially related
encodings on I. Then e1( Q )  P if and only if e2 ( Q )  P .
*


Let L1 , L2 be two languages. We can define
Using reasonable encoding to neglect the distinction
between abstract and concrete problems.
Chapter 34
P.17
Chapter 34
P.18
L1  L2 (union)
The set of instances of any decision problem Q is the set of
L1  L2 (intersection)
 * , where   { 0 ,1 } . Since Q is entirely characterized by
L (complement)
L1 L2  {x1 x2 | x1  L1and x2  L2 }
those problem instances that produces a 1 (yes) answer. We
(concatenation)
The closure (Kleen star) of L :
L  { x   * | Q( x )  1 } .
can view Q as the language L over  * , where
L*  { }  L  L  L  ...
2
Chapter 34
3
P.19
Chapter 34
P.20
Algorithm A accepts a string x { 0 ,1 } * if the given input
Even if language L is accepted by an algorithm A, the
x, the algorithm output A(x)=1.
algorithm will not necessarily reject a string x  L
provided as input to it.
The language accepts by an algorithm A is the set
For example, the algorithm may
loop forever.
L  { x   * | A( x )  1 } .
A language L is decided by an algorithm A if every binary
The algorithm A rejects a string x if A(x)=0.
string is either accepted or rejected by the algorithm.
Chapter 34
P.21
Chapter 34
A language L is accepted in polynomial time by an
Example:
algorithm A if for any length n string x  L , the algorithm
PATH PROBLEM:
accepts x in time
O( n k )
for some constant k.
P.22
PATH={<G,u,v,k> | G=(V,E) is an undirected graph,
A language L is decided in polynomial time by an
algorithm A if for any length n string x { 0 ,1 } * , the
u, v V , k  0 is an integer, and there is a path from u to v
whose length is at most k}.
algorithm decides x in time O( n k ) for some constant k.
Chapter 34
P.23
Chapter 34
P.24

Can be accepted in polynomial time.

Can be decided in polynomial time.
We can informally define a complexity class as a set of
languages, membership in which is determined by a
HALTING PROBLEM:
complexity measure, such as running time, on an algorithm
There exists an accepting algorithm, but no decision
that determines whether a string x belongs to language L.
algorithm exists.
Chapter 34
P.25
Chapter 34
P.26
We define the complexity class P as: P  { L  { 0 ,1 } * |
there exists an algorithm A that decides L in polynomial
time}.
HAMILTONIAN CYCLE PROBLEM:
HAM_CYCLE={<G> | G is a hamiltonian graph}
Theorem 34.2. P  { L | L is accepted by a polynomial
verification: polynomial
algorithm}.
Chapter 34
decision problem: ?
P.27
Chapter 34
P.28
34.2 Polynomial-time verification
naive algorithm:
PATH PROBLEM:
input size: If we use the reasonable encoding of a graph as
PATH={<G,u,v,k> | G=(V,E) is an undirected graph,
u, v V , k  0 is an integer, and there is a path from u to v
whose length is at most k}.
where n = |<G>| is the length of the encoding of G. There
are m! possible permutations of the vertices. Therefore the
running time is ( m !)  ( n !)  ( 2 n ) . This is not
verification: linear time.
a polynomial algorithm.
Decision problem: polynomial
Chapter 34
its adjacency matrix, the number m of vertices is ( n ) ,
P.29
Chapter 34
P.30
The complexity class NP
The complexity class NP is the class of languages that can
Verification algorithms:
be verified by a polynomial-time algorithm. More precise, a
A verification algorithm is a two-argument algorithm A,
language L belongs to NP if and only if there exists a
where one argument is an ordinary input string x and the
two-input polynomial-time algorithm A and a constant c
other is a binary string y called a certificate. A
such that
two-argument algorithm A verifies an input x if there exists
Chapter 34
L  { x  { 0 ,1 } * | there
exists
a
a certificate y such that A(x,y)=1. The language verified by
| y|  O(| x|c ) such that A(x,y)=1}.
a verification algorithm A is
 NP   (HAM_CYCLE  NP.)
L  { x  { 0 ,1 } * | y  { 0 ,1 } * s. t . A( x , y )  1 } .
 P  NP .
P.31
Chapter 34
certificate
y
with
P.32
Problem:
NP=co-NP
P=NP=co-NP
P
1. P  NP ?
2. Complexity class co-NP
co  NP  { L | L  NP } .
P
NP
NP  co  NP?
3. Obviously P  NP  co  NP .
P  NP  co  NP
P  NP  co  NP?
P
NP
Chapter 34
co-NP
P.33
P  NP  co  NP
Chapter 34
34.3 NP-completeness and reducibility
co-NP
P.34
Reducibility:
NP-completeness problem: if any one NP-complete
ax  b  0
problem can be solved in polynomial time, then every
ax 2  bx  c  0
problem in NP has a polynomial-time solution, that is
NP=P.
Chapter 34
P.35
Chapter 34
P.36
A language L1 is polynomial-time reducible to a language
L2 , written L1  P L2 if there exists a polynomial-time
computable function f : { 0 ,1 }*  { 0 ,1 } * such that for all
x  L1 if and only if f ( x )  L2 .
f
{ 0 ,1 }*
{ 0 ,1 }*
L1
We call the function f
L2
the reduction function, and a polynomial algorithm F that
computes f is called a reduction algorithm.
Chapter 34
P.37
Chapter 34
P.38
NP-Completeness
A language L { 0 ,1 } * is NP-complete if
Lemma 34.3. If L1 , L2 { 0 ,1 } * are languages
1. L  NP , and
such that L1  P L2 , then L2  P implies L1  P .
x
F
f (x)
yes, f ( x)  L2
2. L'  P L for every L'  NP
yes, f ( x)  L1
A2
no, f ( x)  L2
A1
Chapter 34
no, f ( x)  L1
P.39
Chapter 34
P.40
Theorem 34.4.
If any NP-complete problem is
polynomial-time solvable, then NP=P. If any problem is
not polynomial-time solvable, then all NP-complete
problem are not polynomial-time solvable.
 If a language L satisfies property 2, but not necessarily
property 1, we say that L is NP-hard.
Proof.
By Lemma 34.3.
 We also define NPC to be the class of NP-complete
NPC
language.
NP
P
Chapter 34
P.41
Chapter 34
P.42
Circuit satisfiability
x1 1
1
1
x2 1
1
0
1
0
1
1
1
x3
0
Circuit-satisfiability
1
1
1
problem:
Given
a
boolean
combinational circuits composed of AND, OR, or NOT
Satisfiable
1
gates, is it satisfiable?
1
x1
x2
CIRCUIT_SAT={<C> | C is a satisfiable boolean
x3
Chapter 34
Not Satisfiable!
combinational circuit}.
P.43
Chapter 34
P.44
Input bits
Lemma 34.5. The circuit-satisfiability problem belongs
C0
to the class NP.
A
Program Counter Aux Machine State x y working storage
M
Lemma 34.6. The circuit-satisfiability problem is NP-hard.
Proof. L  P CIRCUIT _ SAT
Theorem 34.7.
C1
A
Program Counter Aux Machine State x y working storage
M
L  NP.
C2
A
Program Counter Aux Machine State x y working storage
CT(n)
A
Program Counter Aux Machine State x y working storage
The circuit-satisfiability problem is
NP-Complete.
0/1 output
Chapter 34
P.45
Chapter 34
P.46
34.4 NP-Completeness Proof
Method for proving a language L is NPC:
Lemma 34.8. If L is a language such that L'  P L for
1. Prove L  NP .
some L'  NPC , then L is NP-hard. Moreover, if L  NP
2. Select a known NPC language L'
then L  NPC .
3. Describe an algorithm that computes a function f
mapping every instance of L' to an instance of L.
4. Prove that the function f satisfies x  L' if and only if
f ( x )  L for all x { 0 ,1 } * .
5. Prove that the algorithm computing f runs in polynomial
Chapter 34
P.47
Chapter 34
time.
P.48
SAT={< > |  is a satisfiability formula}
Formula satisfiability:
  (( x1  x2 )  (( x1  x3 )  x4 ))  x2
An instance of SAT is a boolean formula  composed of
x1  0 , x2  0 , x3  1, x4  1
1. boolean variables: x1 , x2 ,...
2. boolean connectives: any boolean function with one or
Example:
two input and one output
3. parentheses
Chapter 34
P.49
  (( 0  0 )  (( 0  1 )  1 ))  0
 (1  (1  1 ))  1
 (1  0 )  1
1
Chapter 34
P.50
Theorem 34.9 Satisfiability of boolean formula is
NP-complete.
Proof.
  x10  ( x4  x3 )  ( x5  ( x1  x2 ))
 SAT  NP
 ( x6  x4 )  ( x7  ( x1  x2  x4 ))
 CIRCUIT _ SAT  P SAT
 ( x8  ( x5  x6 ))  ( x9  ( x6  x7 ))
x1
x5
x2
 ( x10  ( x7  x8  x9 ))
x8
x10
x6
x9
x3
Chapter 34
x4
x7
P.51
Chapter 34
P.52
3-CNF satisfiability
 literal
 conjunction normal form
Theorem 34.10. Satisfibility boolean formula in 3-CNF
(CNF)
 3-conjunction normal form (3-CNF)
is NP complete.
( x1  x1  x2 )  ( x3  x2  x4 )
Proof.
 (x1  x3  x4 )
 3  CNF  SAT  NP
 SAT  P 3  CNF  SAT
Chapter 34
P.53
Chapter 34
P.54
  (( x1  x2 )  (( x1  x3 )  x4 )  x4
y1

y2

y3

x1

x2
x1
 ( y3  ( x1  x2 ))  ( y4  y5 )
y4


Chapter 34
  y1  ( y1  ( y2  x4 ))  ( y2  ( y3  y4 ))
x4
 ( y5  ( y6  x4 ))  ( y6  (x1  x3 ))
y5
y6
x4
x3
P.55
Chapter 34
P.56
| Ci|  3
1  y1  ( y2  x2 )
Truth Table 
Ci
| Ci | 2
1  ( y1  y2  x2 )  ( y1  y2  x2 )
 ( y1  y2  x2 )  (y1  y2  x2 )
Ci  l1  l2  (l1  l2  p)  (l1  l2  p)
De Morgan rule 
| Ci|  1
1  (y1  y2  x2 )  (y1  y2  x2 )
Ci  l  ( l  p  q )  ( l  p  q )
 (y1  y2  x2 )  ( y1  y2  x2 )
( l  p  q )  ( l  p  q )
Chapter 34
P.57
Chapter 34
P.58
CIRCUIT-SAT
34.5 NP-Complete Problems
SAT
3CNF-SAT
Clique
HAM-CYCLE
VERTEX COVER
TSP
SUBSET-SUM
Chapter 34
P.59
Chapter 34
P.60
34.5.1 The clique problem
A clique in a undirected graph G = (V,E) is a subset
V '  V of vertices, each pair of which is connected by an
CLIQUE={<G, k> |G is a graph with clique size k}
edge in E. The size of a clique is the number of vertices it
contains. The clique problem is the optimization problem
|V |
naïve algorithm: ( k 2   )
 k
of finding a clique of maximum size in a graph.
Chapter 34
P.61
Chapter 34
P.62
C1  x1  x2  x3
Theorem 34.11.
The clique problem is NP-complete.
x1
Proof.
x2
x3
 clique  NP
 3  CNF  SAT  P clique
x1
x1
x2
x2
x3
x3
  ( x1  x2  x3 )  (x1  x2  x3 )
 ( x1  x2  x3 )
C2  x1  x2  x3
Chapter 34
P.63
Chapter 34
C3  x1  x2  x3
P.64
   C1  C2 ... Ck
34.5.2 The vertex-cover problem
(1)
rs
 ( vir , v sj )  E 
( 2 ) lir  l sj
 A vertex cover of an undirected graph G=(V,E) is a
subset V '  V such that if ( u, v )  E then u V ' or
 clique size k
v V ' (or both).
 The vertex cover problem is to find a vertex cover of
minimum size in a given graph.
 VERTEX-COVER={<G, k> | graph G has a vertex cover
of size k}.
Chapter 34
P.65
Chapter 34
Theorem 34.12. The vertex-cover problem is NP-complete.
P.66
u
Proof.
v
u
v
 VERTEX  COVER  NP
z
 CLIQUE  P VERTEX  COVER
w
y
Chapter 34
P.67
Chapter 34
x
z
w
y
x
P.68
  ( x1  x2  x3 )  ( x1  x2  x3 )  ( x1  x2  x3 )
34.5.3The hamiltonian-cycle problem
b1,1
A
b1,2
Theorem 34.13.
e1
x1'
e1
B
The hamiltonian cycle problem is
b1,3
NP-complete.
x1''
A
b1,4
A
b2,1
Proof.
A
b2,2
A
B
 HAM  CYCLE  NP
b2,4
A
b3,1
x2'
e2
x2''
A
b2,3
 3CNF  SAT  P HAM  CYCLE
e2
A
e3
b3,2
x3'
e3
B
 kinds of wedges
A
b3,3
Chapter 34
P.69
b1,3
Chapter 34
x3''
P.70
b3,4
x3'
34.5.4 The traveling-salesman problem
TSP={<G, c, k> | G = (V, E) is a complete graph, c is a
function from V  V into Z, k  Z , and G has a traveling
b1,3
b1,4
b1,4
b3,3
b3,4
A
A
salesman tour with cost at most k}.
x3'
e3
x3''
u
b3,3
v
4
3
2
1
1
x
b3,4
Chapter 34
5
w
x3''
P.71
Chapter 34
P.72
Theorem 34.13

Proof.
The hamiltonian cycle problem is NPcomplete.
First, show that HAM-CYCLE belongs to NP.
 We now prove that VERTEX-COVER p HAMCYCLE, which shows that HAM-CYCLE is NPcomplete.
 Given an undirected graph G=(V,E) and an
integer k, we construct an undirected graph
G’=(V’,E’) that has a hamiltonian cycle iff G
has a vertex cover of size k.

Chapter 34
P.73
Chapter 34
The reduction of an instance of the
vertex-cover problem to an instance
of the hamiltonian-cycle problem.
widget

[u,v,1]
[v,u,1]
[u,v,2]
[v,u,2]
[u,v,3]
Wuv
[v,u,4]
[u,v,5]
[v,u,5]
[u,v,6]
[v,u,6]
(a)
[v,u,1]
[u,v,1]
[v,u,3]
[u,v,4]
[v,u,1]
[u,v,1]
Wuv
Wuv
[v,u,1]
[u,v,1]
Wuv

[v,u,6]
[u,v,6]
(b)
[v,u,6]
[u,v,6]
(c)
[v,u,6]
[u,v,6]
(a) An undirected graph G with a vertex of size 2,
consisting if the lightly shaded vertices w and y.
(b) the undirected graph G’ produced by the
reduction, with the hamiltonian path corresponding
to the vertex cover shaded.
(d)

Chapter 34
P.74
P.75
The vertex cover {w,y} corresponds to edges
(s1,[w,x,1]) and (s2,[y,x,1]) appearing in the
hamiltonian cycle.
Chapter 34
P.76
w
x
z
y
(a)
Three types of edges in E’
1.
2.
(b)
3.
[x,w,1]
[w,x,1]
[x,y,1]
Wuv
[w,x,6]
[y,x,1]
Wuv
[x,w,6]
[x,y,6]
[y,w,1]
[w,y,1]
Wuv
[y,x,6]
[w,y,6]
[z,w,1]
[w,z,1]
Edges in widget.
{([u,u(i),6],[u,u(i+1),1]) : 1  i 
degree(u)-1}
{(sj,[u,u(1),1]) : uV and 1  j  k} 
{(sj,[u,u(degree(u)),6]) : uV and 1  j  k}
Wuv
[y,w,6]
[w,z,6]
[z,w,6]
Chapter 34
P.77
Chapter 34
P.78
The reduction performed in
polynomial time


|V’| = 12|E| + k
 12|E| + |V|
The transformation from graph
G to G’ is a reduction.
 That is, G has a vertex cover of
size k iff G’ has a hamiltonian
cycle.

|E’| = (14|E|) + (2|E| -|V|) + (2k|V|)
= 16|E| + (2k-1)|V|
 16|E| + (2|V|-1)|V|
Chapter 34
P.79
Chapter 34
P.80
34.5.5 The subset-sum problem
S={1,4,16,64,256,1040,1041,1093,1284,1344}
Theorem 34.14. The traveling salesman problem is
t=3754
NP-complete.
S' ={1,16,64,256,1040,1093,1284}
Proof.
 TSP  NP
SUBSET-SUM={<S, t> | there exists a subset S'  S such
HAM  CYCLE  P TSP
that t   s
sS'
Chapter 34
P.81
Theorem 34.15

P.82
Proof.
First, show that SUBSET-SUM is in NP.
 We now show that 3-CNF-SAT p SUBSETSUM.
 Given a 3-CNF formula  over variables x1,
x2,…, xn with clauses C1, C2,…, Ck, each
containing exactly three distinct literals.
 The reduction algorithm constructs an
instance <S,t> of the subset-sum problem
such that  is satisfiable iff there is a subset
of S whose sum is exactly t.
The subset-sum problem is NPcomplete.
Chapter 34
Chapter 34

P.83
Chapter 34
P.84
The reduction of 3-CNF-SAT to
SUBSET-SUM
C has no ¬x
Example
The formula in 3-CNF is  = C1  C2 
C3  C4, where C1 = (x1  ¬x2  ¬x3), C2
= (¬x1  ¬x2  ¬x3), C3 = (¬x1  ¬x2 
x3), and C4 = (x1  x2  x3).
 A satisfying assignment of  is <x1 = 0,
x2 = 0, x3 = 1>.

Chapter 34
P.85
x1
x2
x3
C1
C2
C3
C4
v1
=
1
0
0
1
0
0
1
v1’
v2
v2’
v3
v3’
=
=
=
=
=
1
0
0
0
0
0
1
1
0
0
0
0
0
1
1
0
0
1
0
1
1
0
1
0
1
1
0
1
1
0
0
1
0
1
0
s1
=
0
0
0
1
0
0
0
s1 ’
s2
s2 ’
s3
=
=
=
=
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
1
2
0
0
0
0
1
0
0
0
0
s3 ’
s4
=
=
0
0
0
0
0
0
0
0
0
0
2
0
0
1
s4 ’
=
0
0
0
0
0
0
2
t
=
1
1
1
4
4
4
4
4
Chapter 34
1
C4 has x2
P.86
The reduction performed in
polynomial time
The set S contains 2n+2k values, each
of which has n+k digits, and the time to
produce each digit is polynomial in n+k.
 The target t has n+k digits, and the
reduction produces each in constant
time.

Chapter 34
P.87

Chapter 34
3-CNF formula  is satisfiable
iff there is a subset S’  S
whose sum is t.
P.88