Recurrences
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 1
Recurrences
Coming up
Merge Sort (Chap 2.3)
Recurrences (Chap 4.1-4.3)
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 2
Merge Sort
Merge Sort
Insertion Sort
: divide-and-conquer approach
: Incremental approach
The Divide-and-Conquer Paradigm:
Divide:
Divide the problem into a number of
subproblems.
Conquer:
If the subproblem sizes are small enough, solve
them directly. Otherwise solve them recursively.
Combine:
Combine the solutions to the subproblems into
the solution for the original problem.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 3
Merge Sort
Suppose there are some people
called Mr. MergeSort. They are
identical.
They don’t know how to do sorting.
But each of them has a secretary
called Mr. Merge, who can merge 2
sorted sequences into one sorted
sequence.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 4
Merge Sort
At the beginning, a
Mr. MergeSort is
called to sort:
Then 2 other Mr.
MergeSorts are
called to sort:
Then 4 other Mr.
MergeSorts are
called to sort:
Then 8 other Mr.
MergeSorts are
called to sort:
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
5 2 4 7 1 3 2 6
5 2 4 7 1 3 2 6
5 2 4 7 1 3 2 6
5 2 4 7 1 3 2 6
http://www.cs.cityu.edu.hk/~helena
“So complicated!!, I’ll
split them and call other
Mr. MergeSorts to
handle.”
Both of them say “Still
complicated! I’ll split
them and call other Mr.
MergeSorts to handle.”
All of them say “Still
complicated! I’ll split
them and call other Mr.
MergeSorts to handle.”
All of them say
‘This is easy. No
need to do anything.’
4. Recurrences - 5
Merge Sort
Then the first Mr.
MergeSort succeeds
and returns.
Then each of the 2
Mr. MergeSorts
returns the merged
numbers.
Then the 4 Mr.
MergeSorts returns the
merged numbers.
Then the 8 Mr.
MergeSorts return.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
15 2 42 73 14 35 26 67
52 24 45 7 1 32 23 6
52 25 4 7 1 3 2 6
5 2 4 7 1 3 2 6
http://www.cs.cityu.edu.hk/~helena
The first Mr. MergeSort
calls his secretary Mr.
Merge to merge the
returned numbers
Both Mr. MergeSorts
call their secretaries Mr.
Merge to merge the
returned numbers
The 4 Mr. MergeSorts
call their secretaries Mr.
Merge to merge the
returned numbers
All of them say
‘This is easy. No
need do anything.’
4. Recurrences - 6
Merge Sort
The MERGE-SORT(A,p,r) procedure
sorts the elements A[p,..r]:
A = .. 5 2 4 7 1 3 2 6 ..
pth
rth
MERGE-SORT (A,p,r)
1 if p < r
2 Then q  (p+r)/2
3
MERGE-SORT(A,p,q)
4
MERGE-SORT(A,q+1,r)
5
MERGE (A,p,q,r)
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
x: “Floor” The least integer
greater than x
x: “Ceiling” The greatest
integer less than x
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 7
Merge Sort
The MERGE(A,p,q,r) procedure
merges 2 sorted sequences:
A[p..q] and A[q+1..r]
A = .. 2 4 5 7 1 2 3 6 ..
pth
qth
rth
Step 1: Copy A[p..q], A[q+1..r] to 2
temporary arrays L and R.
L= 2 4 5 7
R= 1 2 3 6
Step 2: Adds a largest value,  (a
sentinel: ending condition),
to each of L and R.
L= 2 4 5 7 
R= 1 2 3 6 
L= 2
754 4
75 5
7 7
R= 21
63 32
6 636 
Step 3: Continuously remove the
smallest one from L and R
back to A until finished.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
A = .. 12 42 52 73 14 25 36 67 ..
pth
http://www.cs.cityu.edu.hk/~helena
rth
4. Recurrences - 8
Merge Sort
The MERGE(A,p,q,r) procedure
merges 2 sorted sequences:
A[p..q] and A[q+1..r]
Step 1: Copy A[p..q], A[q+1..r] to 2
temporary arrays L and R.
Step 2: Adds a largest value,  (a
sentinel: ending condition),
to each of L and R.
Step 3: Continuously remove the
smallest one from L and R
back to A until finished.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
MERGE (A,p,q,r)
1 n1  q-p+1
2 n2  r-q
3 create L[1..n1+1], R[1..n2+1]
4 for i  1 to n1
5
do L[i]  A[p+i-1]
6 for j  1 to n2
7
do R[j]  A[q+j]
8 L[n1+1]  
9 R[n2+1]  
10 i  1
11 j  1
12 for k  p to r
13
do if L[i]  R[j]
14
then A[k]  L[i]
15
i  i+1
16
else
17
j  j+1
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 9
Analysis of MERGE Procedure
• Let n=n1+ n2
• Line 1-3 and 8-11 takes constant
time. So, (1).
• Line 4-5 and Line 6-7 takes
(n1+n2) = (n) time.
• In line 12-17, the loop iterates n
times, each of which takes
constant time. So, (n).
• Conclusion: The MERGE
procedure runs in (1) + (n) +
(n) = (n) time.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
MERGE (A,p,q,r)
1 n1  q-p+1
2 n2  r-q
3 create L[1..n1+1], R[1..n2+1]
4 for i  1 to n1
5
do L[i]  A[p+i-1]
6 for j  1 to n2
7
do R[j]  A[q+j]
8 L[n1+1]  
9 R[n2+1]  
10 i  1
11 j  1
12 for k  p to r
13
do if L[i]  R[j]
14
then A[k]  L[i]
15
i  i+1
16
else
17
j  j+1
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 10
Analysis of Merge Sort
MERGE-SORT (A,p,r)
1 if p < r
2 Then q  (p+r)/2
3
MERGE-SORT(A,p,q)
4
MERGE-SORT(A,q+1,r)
5
MERGE (A,p,q,r)
Then, what is the complexity
of Merge Sort?
To sort A[1..n] using Merge Sort, we
call MERGE-SORT(A,1,n)
MERGE-SORT has a recursive call to
itself, plus a call to MERGE.
The Running time:
(1)
T(n) =
Or simplified:
if n=1
Tn/2 + Tn/2 + (n) if n>1
T(n) =
c
if n=1
2T(n/2)+cn if n>1
Recurrence Equation
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 11
Solving Recurrences
T(n) =
c
2T(n/2)+cn
if n=1
if n>1
=> T(n) = (?)
When an algorithm contains recursive call(s) to itself, (eg. Divide-and-conquer
approaches), its running time can often be described by a recurrence equation.
3 methods to solve recurrences:
Recursion-tree method
Can directly prove or help to guess
MI
Substitution method
guess a bound + prove by mathematical induction
Master method
Based on the Master Theorem
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 12
Solving Recurrence (Recursion-tree method)
Recursion-tree method
T(n) =
c
if n=1
2T(n/2)+cn if n>1
Expanding the recursion tree:
T(n)
cn
T(n/2)
cn
T(n/2)
cn/2
cn/2
T(n/4) T(n/4) T(n/4) T(n/4)
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 13
Solving Recurrence (Recursion-tree method)
Fully Expanded recursion tree:
cn
cn/2
cn/4
cn
cn/2
cn/4
cn/4
cn/4
cn
cn
c*1c*1 c*1 c*1 c*1 c*1 c*1 c*1
n
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
Total: cn lg n + cn
ie. T(n) = (n lg n)
4. Recurrences - 14
Solving Recurrence (Substitution method)
MI
Substitution method
Step 1.
Guess the form of the solution
Example. Given:
1
if n=1
T(n) =
2T(n/2)+n if n>1
We guess that T(n)=O(n lg n)
To prove this we need to show
“There exists positive constants
c and n0 such that
...T(n)  cn lg n, for all n  n0 ”
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
Step 2
4. Recurrences - 15
Solving Recurrence (Substitution method)
T(n) =
1
if n=1
2T(n/2)+n if n>1
MI
T(n)=O(n lg n)
“There exists a positive constants c and
n0 such that T(n)  cn lg n, for all n  n0 ”
Step 2. Use mathematical induction to find the constants and prove.
Assume it holds for n/2, ie. There exists positive constants c
and n0 such that T(n/2)  c n/2 lg (n/2).
By substitution:
T(n)  2 (c n/2 lg (n/2)) + n
 cn lg (n/2) + n
= cn (lg(n)-lg(2))+n
= cn (lg(n)-1)+n
= cn lg n -cn + n
This holds as long as c  1
 cn lg n
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 16
Solving Recurrence (Substitution method)
T(n) =
1
if n=1
2T(n/2)+n if n>1
MI
T(n)=O(n lg n)
“There exists a positive constants c and
n0 such that T(n)  cn lg n, for all n  n0 ”
Step 2 (cont’d). Then we need to find n0.
Try n0=1
When n=1,
Try n0=2
When n=2,
Try n0=3
When n=3,
T(n)= 1
cn lg n = (c*1) lg (1) = c * 0 = 0
=> T(n)  cn lg n ?
T(n) = 2T(1) + 2 = 2(1) + 2 = 4
cn lg n = (c*2) lg (2) = c*2*1 = 2c
=> T(n)  cn lg n ?
T(n) = 2T(1) + 3 = 2(1) + 3 = 5
cn lg n = (c*3) lg (3) = 3*1.585*c = 4.755*c
=> T(n)  cn lg n ?
Since n0=2 and n0=3 form the base cases for all n>=2, we thus
completed the prove: T(n)  cn lg n is true when c>=2 and n>=2.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 17
Solving Recurrence (Master method)
Master method
A cookbook method
For solving recurrences of the form:
T(n) = a T(n/b) + f(n)
Where a >= 1 and b > 1 and
f(n) is an asymptotically positive function
These algorithms work recursively by dividing a problem of size
n into a subproblems, each of size n/b.
f(n) = cost of dividing the problem and combining the results.
We omit floors and ceilings (eg. interpret n/b to mean either
n/b or n/b.)
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 18
Solving Recurrence (Master method)
Master Theorem
Let T(n) be defined as
T(n) = a T(n/b) + f(n)
where a >=1 and b > 1.
Then
If f(n) = O(nlogba-) for some constant >0,
then T(n) = (nlogba)
If f(n) = (nlogba),
then T(n) = (nlogbalg n)
If f(n) = (nlogba+) for some constant >0,
and if a f (n/b) <= c f(n) for some constant c <1 and all sufficiently large n,
then T(n) = (f(n))
The master method can be used only if f(n) satisfies any of the 3 cases.
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 19
Solving Recurrence (Master method)
Master Theorem For T(n) = a T(n/b) + f(n) where a >=1 and b > 1.
Example 1:
If f(n) = O(nlogba-) for some constant >0,
If f(n) = (nlogba),
If f(n) = (nlogba+) for some constant >0, ..
T(n) = 9T(n/3) + n
a = 9, b = 3
=> nlogba = nlog39 = n2
Since f(n) = n = n(2-1)
= O(nlog39 -)
Where =1
=> Case 1
=> T(n) = (nlogba)
=> T(n) = (n2)
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
then T(n) = (nlogba)
then T(n) = (nlogbalg n)
then T(n) = (f(n))
T(n) = T(2n/3) + 1
a = 1, b = 3/2
=> nlogba = nlog3/21 = n0 = 1
Since
f(n) = 1 = (nlogba)
=> Case 2
=> T(n) = (nlogbalg n)
=> T(n) = (lg n)
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 20
Solving Recurrence (Master method)
Master Theorem For T(n) = a T(n/b) + f(n) where a >=1 and b > 1.
If f(n) = O(nlogba-) for some constant >0,
If f(n) = (nlogba),
If f(n) = (nlogba+) for some constant >0,
then T(n) = (nlogba)
then T(n) = (nlogbalg n)
and if a f (n/b) <= c f(n) for some constant c <1 and all sufficiently large n,
Example 3:
then T(n) = (f(n))
T(n) = 3T(n/4) + n lg n
a=3, b=4 => nlogba = nlog43 = n0.793
Since f(n) = n lg n
When n  2, f(n)  n lg 2 = n = n1 = n0.793+, where =0.201
=> f(n) = (nlogba+)
a f(n/b)
= a (n/b * lg(n/b))
= 3 (n/4 * lg (n/4))
= 3/4 n lg(n/4)
= 3/4 (n lg(n)) - 3/4 (n lg(4))
 3/4 f(n)
=> a f(n/b)  c f(n) where c=3/4. => Case 3. ie. T(n) = (f(n)) = (n lg n)
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 21
Recurrences
Summary
The Divide-and-Conquer Paradigm
Merge Sort (MERGE-SORT Procedure + MERGE Procedure)
Analysis of MERGE: (n)
Analysis of MERGE-SORT: A recurrence problem (n lg n))
3 Methods to solve recurrences:
Recursion-tree method
Substitution method
Master method
CS3381 Des & Anal of Alg (2001-2002 SemA)
City Univ of HK / Dept of CS / Helena Wong
http://www.cs.cityu.edu.hk/~helena
4. Recurrences - 22