CS473-Algorithms I
Lecture 3
Solving Recurrences
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
1
Solving Recurrences
• The analysis of merge sort Lecture 1 required us
to solve a recurrence.
• Recurrences are like solving integrals, differential
equations, etc.

Learn a few tricks.
• Lecture 4 : Applications of recurrences.
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
2
Recurrences
• Function expressed recursively
if n=1
1
T ( n)  
T ( n / 2)  1 if n >1
• Solve for n = 2k
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
3
Recurrences
• Claimed answer: T(n) = lgn+1 =  (lgn)
 Substitute claimed answer for T in the recurrence
 Note: resulting equations are true when n = 2k
i.e.
1
if n=20=1
lg n  1  
k >1
if
n=2
(lg(
n
/
2
)

1
)

 
Tedious technicality: haven’t shown T(n) =  (lgn)
– But, since T(n) is monotonically non-decreasing function
of n
lg n
lg n 
n2
 T (n)  T (2
 lg( 2
lg n 
)
)  1  lg n  1  (lg n)
– Thus, ceiling didn’t
matter much
Cevdet Aykanat - Bilkent University
CS473 – Lecture 3
Computer Engineering Department
4
Recurrences
• Technically, should be careful about floors and
ceilings (as in the book)
• But, usually it is okay
 To ignore floor/ceiling
 Just solve for exact powers of 2 ( or b)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
5
Boundary Conditions
• Usually assume T(n)= Θ(1) for small n
– Does not usually affect soln. (if polynomially bounded)
•
Example: Initial condition affects soln.
– Exponential T(n)=(T(n / 2))2
If T(1)= c for a constant c > 0, then
T(2) = (T(1))2=c2, T(4)= (T(2))2=c4,
T(n) = Θ(cn)
E.g.,
T (1)  2  T (n)  (2 n )

n
n
However
2


(
3
)
n 
T (1)  3  T (n)  (3 ) 

Difference in soln. is more dramatic with
T (1)  1  T (n)  (1n )  (1)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
6
Substitution Method
• The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
• Example: T(n) = 4T(n/2) + n
– [Assume that T(1) = Θ(1).]
– Guess O(n3) . (Prove O and Ω separately.)
– Assume that T(k) ≤ ck3 for k < n .
– Prove T(n) ≤ cn3 by induction.
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
7
Example of Substitution
T(n) = 4T(n/2) + n
 4c (n/2)3 + n
= (c/2) n3 + n
= cn3 –n ((c/2) n3 - n)  desired –residual
 cn3
whenever (c/2) n3 – n  0, for example,
if c  2 and n  1
CS473 – Lecture 3
residual
Cevdet Aykanat - Bilkent University
Computer Engineering Department
8
Example (Continued)
• We must also handle the initial conditions, that is,
ground the induction with base cases.
• Base: T(n) = Θ(1) for all n < n0, where n0 is a suitable
constant.
• For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big
enough.
This bound is not tight!
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
9
A Tighter Upper Bound?
We shall prove that T(n) = O(n2)
Assume that T(k)  ck2 for k < n:
T(n) = 4T(n/2) + n
 cn2 + n
= O(n2) Wrong ! We must prove the I.H.
= cn2 - (-n)
 cn2
for no choice of c > 0. Lose!
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
10
A Tighter Upper Bound!
• IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k)  c1k2 – c2k for k < n
T(n) = 4T(n/2) + n
 4 (c1 (n/2)2- c2 (n/2)) + n
= c1 n2- 2 c2 n + n
= c1 n2- c2 n –(c2 n - n)
 c1 n2- c2 n if c2 > 1
Pick c1 big enough to handle the initial conditions
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
11
Recursion-Tree Method
• A recursion tree models the costs (time) of a recursive
execution of an algorithm.
• The recursion tree method is good for generating
guesses for the substitution method.
• The recursion-tree method can be unreliable.
• The recursion-tree method promotes intuition,
however.
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
12
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
13
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
14
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4)
CS473 – Lecture 3
T(n/2)
Cevdet Aykanat - Bilkent University
Computer Engineering Department
15
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
T(n/16)
CS473 – Lecture 3
(n/2)2
T(n/8) T(n/8)
Cevdet Aykanat - Bilkent University
Computer Engineering Department
T(n/4)
16
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/16)2
(n/8)2
(n/2)2
(n/8)2
(n/4)2
(1)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
17
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/16)2
(n/8)2
n2
(n/2)2
(n/8)2
(n/4)2
(1)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
18
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/16)2
(n/8)2
5/16 n2
(n/2)2
(n/8)2
n2
(n/4)2
(1)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
19
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/16)2
(n/8)2
5/16 n2
(n/2)2
(n/8)2
n2
(n/4)2
25/256 n2
(1)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
20
Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/16)2
(1)
CS473 – Lecture 3
(n/8)2
5/16 n2
(n/2)2
(n/8)2
n2
(n/4)2
25/256 n2
Total = n2 (1 + 5/16 + (5/16)2 + (5/16)2 + ...)
= (n2) geometric series
Cevdet Aykanat - Bilkent University
Computer Engineering Department
21
The Master Method
• The master method applies to recurrences of the form
T(n) = aT(n/b) + f (n) ,
where a ≥ 1, b > 1, and f is asymptotically
positive.
• Three common cases:
– Compare f (n) with
CS473 – Lecture 3
n
logb a
Cevdet Aykanat - Bilkent University
Computer Engineering Department
22
Three Common Cases
• Compare f (n) with
1. f (n) = O( n
logb a 
n
logb a
:
) for some constant ε > 0.
• f (n) grows polynomialy slower than
n
logb a
(by an nε factor).
logb a
Solution: T(n) = Θ( n
).
2. f (n) = Θ( n logb algkn) for some constant k ≥ 0.
• f (n) and
n
logb a
Solution: T(n) = Θ(
CS473 – Lecture 3
n
grow at similar rates.
logb a
lgk+1n) .
Cevdet Aykanat - Bilkent University
Computer Engineering Department
23
Three Common Cases
• Compare f (n) with
3. f (n) =  ( n logb a 
n
logb a
:
) for some constant ε > 0.
• f (n) grows polynomially faster than
n
logb a
(by an nε factor).
and f (n) satisfies the regularity condition that
a f (n/b)  c f (n) for some constant c < 1
Solution: T(n) = Θ( f (n) ) .
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
24
Examples
• Ex: T(n) = 4T(n/2) + n
logb a
a=4, b=2  n
= n2 ; f (n) = n
CASE 1: f (n) =O (n2- ε) for ε=1
T(n) = Θ (n2)
• Ex: T(n) = 4T(n/2) + n2
logb a
a=4, b=2  n
= n2 ; f (n) = n2
CASE 2: f (n) = Θ (n2 lg0n), that is, k=0
T(n) = Θ (n2 lgn)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
25
Examples
• Ex: T(n) = 4T(n/2) + n3
a=4, b=2  n
logb a
= n2 ; f (n) = n3.
CASE 3: f (n) = (n2+ ε) for ε=1
and 4 c (n/2)3  cn3 (reg. cond.) for c=1/2.
T(n) = Θ (n3)
• Ex: T(n) = 4T(n/2) + n2 / lgn
a=4, b=2  n
logb a
= n2 ; f (n) = n2 / lgn
Master method does not apply. In particular, for every
constant ε > 0, we have nε =  (lgn)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
26
General Method (Akra-Bazzi)
k
T (n)   aiT (n / bi )  f (n)
i 1
Let p be the unique solution to
k
 (a
i 1
i
/b i) 1
p
Then, the answers are the same as for the
logb a
p
master method, but with n instead of n
(Akra and Bazzi also prove an even more general result.)
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
27
Idea of Master Theorem
Recursion tree:
f (n)
f (n)
a
f (n/b) f (n/b) f (n/b)
a
h= logbn
f (n/b2) f (n/b2) f (n/b2)
a f (n/b)
a2 f (n/b2)
#leaves = a h
T(1)
CS473 – Lecture 3
=a
logb n
=n
logb a
Cevdet Aykanat - Bilkent University
Computer Engineering Department
n
logb a
T (1)
28
Idea of Master Theorem
Recursion tree:
f (n)
a
f (n/b) f (n/b) f (n/b)
a
h= logbn
f (n/b2) f (n/b2) f (n/b2)
T(1)
CS473 – Lecture 3
f (n)
CASE 1 : The weight increases
geometrically from the root to the
leaves. The leaves hold a constant
fraction of the total weight.
Cevdet Aykanat - Bilkent University
Computer Engineering Department
a f (n/b)
a2 f (n/b2)
n
logb a
Θ(n
T (1)
logb a
29
)
Idea of Master Theorem
Recursion tree:
f (n)
a
f (n/b) f (n/b) f (n/b)
a
h= logbn
f (n/b2) f (n/b2) f (n/b2)
CASE 2 : (k = 0) The weight
T(1)
CS473 – Lecture 3
f (n)
is approximately the same on
each of the logbn levels.
Cevdet Aykanat - Bilkent University
Computer Engineering Department
a f (n/b)
a2 f (n/b2)
n
logb a
Θ(n
T (1)
logb a
30
lgn)
Idea of Master Theorem
Recursion tree:
f (n)
a
f (n/b) f (n/b) f (n/b)
a
h= logbn
f (n/b2) f (n/b2) f (n/b2)
T(1)
CS473 – Lecture 3
f (n)
CASE 3 : The weight decreases
geometrically from the root to the
leaves. The root holds a constant
fraction of the total weight.
Cevdet Aykanat - Bilkent University
Computer Engineering Department
a f (n/b)
a2 f (n/b2)
n
logb a
T (1)
Θ ( f (n) )
31
Proof of Master Theorem:
Case 1 and Case 2
• Recall from the recursion tree (note h = lgbn=tree
height)
T (n)  (n
h 1
logb a
)   a f (n / b )
i
i
i 0
Leaf cost
CS473 – Lecture 3
Non-leaf cost = g(n)
Cevdet Aykanat - Bilkent University
Computer Engineering Department
32
Proof of Case 1
logb a

n

 ( n )
f ( n)
for some  > 0
logb a
n
f ( n)


 (n )  logb a  O(n  )  f (n)  O(n logb a  )
f ( n)
n
 g ( n) 
 a O(n / b )
h 1
i
i logb a 
i 0

 h1 i
i logb a  
 O  a (n / b )

 i 0



log
a


i
log
a
i
i

b
b
  O n
a b /b


i 0


h 1
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
33
Case 1 (cont’)
h 1
i i
h 1
 i
i
h 1
ab
(
b
)
b
i
i
 i

a

a

(
b
)




i logb a
logb a i
i
a
(b
)
i 0 b
i 0
i 0
= An increasing geometric series since b > 1
h
h 
logb n 

b  1 (b )  1 (b
) 1 n 1

 
 

 
 O( n )

b 1
b 1
b 1
b 1
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
34
Case 1 (cont’)
logb a

n
logb a 

 
g ( n)  O n
O(n )  O  O(n ) 
 n



 O( n
T (n)  (n
logb a
logb a
 (n
)
)  g (n)  (n
logb a
logb a
)  O(n
logb a
)
Q.E.D.
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
35
)
Proof of Case 2 (limited to k=0)
f ( n)
 n logb a 
logb a
0
i
 (lg n)  (1)  f (n)  (n
)  f (n / b )   ( i )

logb a
n
 b

h 1

 g ( n)   a  ( n / b )
i
i logb a

i 0
 h1 i n logb a 
 logb a h1 i
1 
 logb a h1 i 1 
   a i logb a    n
a logb a i    n
a i


a 
(b
) 
i 0
i 0


 i 0 b

 logb a logb n1 
logb a
logb a

  n
1


n
log
n


n
lg n

b

i 0 


T (n)  n
logb a

n
CS473 – Lecture 3
 
 (n
logb a
lg n
logb a


lg n)
Cevdet Aykanat - Bilkent University
Computer Engineering Department
Q.E.D.
36
Conclusion
• Next time: applying the master method.
CS473 – Lecture 3
Cevdet Aykanat - Bilkent University
Computer Engineering Department
37