Algorithms and Data
Structures
Lecture IV
Simonas Šaltenis
Aalborg University
simas@cs.auc.dk
September 25, 2002
1
This Lecture

Analyzing the running time of recursive
algorithms (such as divide-and-conquer)

Writing and solving recurrences
September 25, 2002
2
Recurrences



Running times of algorithms with Recursive
calls can be described using recurrences
A recurrence is an equation or inequality that
describes a function in terms of its value on
smaller inputs.
For divide-and-conquer algorithms:
solving_trivial_problem
if n  1

T ( n)  
num_pieces T (n / subproblem_size_factor)  dividing  combining if n  1

Example: Merge Sort
September 25, 2002
(1)
if n  1

T ( n)  
2T (n / 2)  (n) if n  1
3
Solving Recurrences

Repeated (backward) substitution method


Substitution method




Expanding the recurrence by substitution and noticing
a pattern
guessing the solutions
verifying the solution by the mathematical induction
Recursion-trees
Master method

templates for different classes of recurrences
September 25, 2002
4
Repeated Substitution

Let’s find the running time of the merge sort
(let’s assume that n=2b, for some b).
1
if n  1

T ( n)  
2T (n / 2)  n if n  1
T (n)  2T  n / 2   n
substitute
 2  2T  n / 4   n / 2   n expand
 22 T (n / 4)  2n substitute
 22 (2T (n /8)  n / 4)  2n expand
 23 T (n /8)  3n
observe the pattern
T (n)  2i T (n / 2i )  in
 2lg n T (n / n)  n lg n  n  n lg n
September 25, 2002
5
Repeated Substitution Method

The procedure is straightforward:








Substitute
Expand
Substitute
Expand
…
Observe a pattern and write how your expression looks
after the i-th substitution
Find out what the value of i (e.g., lgn) should be to get
the base case of the recurrence (say T(1))
Insert the value of T(1) and the expression of i into
your expression
September 25, 2002
6
Repeated Substitution (Ex. 2)

Let’s find a more exact running time of merge
sort (let’s assume that n=2b, for some b).
T ( n)
2
if n  1

T ( n)  
2T (n / 2)  2n  3 if n  1
 2T  n / 2   2n  3 substitute
 2  2T  n / 4   n  3  2n  3 expand
 22 T  n / 4   4n 2  3  3 substitute
 22  2T  n / 8   n / 2  3  4n  2  3  3 expand
 23 T  n / 23   2  3n   22 +21 +20   3 observe the pattern
i 1
T (n)  2 T (n / 2 )  2  in  3   2 j
i
i
j 0
 2lg n T (n / n)  2n lg n  3(2lg n  1)  5n  2n lg n  3
September 25, 2002
7
Repeated Substitution (Ex. 3)

Let’s find the running time of the tromino tiling
algorithm for a 2n x2n board.
1
if n  1

T ( n)  
4T (n  1)  1 if n  1
T (n)  4T  n  1  1 substitute
 4  4T  n  2   1  1 expand
 42 T  n  2   4  1 substitute
 42  4T  n  3  1  4  1 expand
 43T  n  3   4 2 +41 +40  observe the pattern
i 1
T ( n)  4 T ( n  i )   4 j
i
j 0
4n1  1 4n  1
 4 T (1) 

4 1
3
n 1
September 25, 2002
8
Substitution method
Solve T ( n)  4T ( n / 2)  n
1) Guess that T ( n)  O( n3 ), i.e., that T of the form cn 3
2) Assume T ( k )  ck 3 for k  n / 2 and
3) Prove T ( n)  cn3 by induction
T (n)  4T (n / 2)  n (recurrence)
 4c(n/2)3  n (ind. hypoth.)
c 3

n  n (simplify)
2
c

 cn3   n3  n  (rearrange)
2

 cn3 if c  2 and n  1 (satisfy)
Thus T ( n)  O(n3 )!
Subtlety: Must also check the base case (not necessarily T (1))
September 25, 2002
9
Substitution Method

Achieving tighter bounds
Try to show T ( n)  O( n 2 )
Assume T (k )  ck 2
T (n)  4T ( n / 2)  n
 4c( n / 2) 2  n
 cn 2  n
 cn 2 for no choice of c  0.
September 25, 2002
10
Substitution Method (2)

The problem? We could not rewrite the equality
T (n)  cn2 + (something positive)

as:
T (n)  cn2


in order to show the inequality we wanted
Sometimes to prove inductive step, try to
strengthen your hypothesis

T(n)  (answer you want) - (something > 0)
September 25, 2002
11
Substitution Method (3)

Corrected proof: the idea is to strengthen
the inductive hypothesis by subtracting
lower-order terms!
Assume T (k )  c1k 2  c2 k for k  n
T (n)  4T (n / 2)  n
 4(c1 (n / 2) 2  c2 (n / 2))  n
 c1n 2  2c2 n  n
 c1n 2  c2 n  (c2 n  n)
 c1n 2  c2 n if c2  1
September 25, 2002
12
Recursion Tree

A recursion tree is a convenient way to visualize
what happens when a recurrence is iterated

Good for ”guessing” asymtotic solutions to recurrences
T (n)  T (n / 4)  T (n / 2)  n2
September 25, 2002
13
Recursion Tree (2)
September 25, 2002
14
Master Method



The idea is to solve a class of recurrences that
have the form
T (n)  aT (n / b)  f (n)
a  1 and b > 1, and f is asymptotically positive.
Abstractly speaking, T(n) is the runtime for an
algorithm and we know that


a subproblems of size n/b are solved recursively, each
in time T(n/b)
f(n) is the cost of dividing the problem and combining
the results. In merge-sort T (n)  2T (n / 2)  (n)
September 25, 2002
15
Master Method (2)
Split problem into a parts at logbn
levels. There are alogb n  nlogb a leaves
September 25, 2002
16
Master Method (3)


Number of leaves: alogb n  nlogb a
Iterating the recurrence, expanding the tree
yields T (n)  f (n)  aT (n / b)

f (n)  af (n / b)  a 2T (n / b 2 )

f (n)  af (n / b)  a 2T (n / b 2 )  ...
 a logb n1 f (n / b logb n1 )  a logb nT (1)
Thus,
T ( n) 
logb n 1

a j f (n / b j )  (n logb a )
j 0


The first term is a division/recombination cost (totaled
across all levels of the tree)
log a
The second term is the cost of doing all n b subproblems
of size 1 (total of all work pushed to leaves)
September 25, 2002
17
MM Intuition

Three common cases:





Running time dominated by cost at leaves
Running time evenly distributed throughout
the tree
Running time dominated by cost at the root
Consequently, to solve the recurrence, we
need only to characterize the dominant
term
In each case compare f (n) with O(nlog a )
b
September 25, 2002
18
MM Case 1

f (n)  O(nlogb a )


for some constant   0
f(n) grows polynomially (by factor
than nlogb a

n
) slower
The work at the leaf level dominates



Summation of recursion-tree levels O(nlogb a )
Cost of all the leaves (nlogb a )
Thus, the overall cost (nlogb a )
September 25, 2002
19
MM Case 2



f (n)  (nlogb a )
f (n) and nlogb a are asymptotically the same
The work is distributed equally
logb a
throughout the tree T (n)  (n
lg n)

(level cost)  (number of levels)
September 25, 2002
20
MM Case 3

f (n)  (nlogb a ) for some constant   0



Inverse of Case 1
logb a
f(n) grows polynomially faster than n
Also need a regularity condition
c  1 and n0  0 such that af (n / b)  cf (n) n  n0

The work at the root dominates
T (n)  ( f (n))
September 25, 2002
21
Master Theorem Summarized

Given a recurrence of the form T (n)  aT (n / b)  f (n)
1.

f (n)  O n logb a 


 T (n)   n logb a
2.

f (n)   n logb a



 T (n)   n logb a lg n
3.



f (n)   n logb a  and af (n / b)  cf (n), for some c  1, n  n0
 T ( n)    f ( n) 

The master method cannot solve every recurrence of this
form; there is a gap between cases 1 and 2, as well as
cases 2 and 3
September 25, 2002
22
Strategy





Extract a, b, and f(n) from a given recurrence
log b a
Determine n
log b a
Compare f(n) and n
asymptotically
Determine appropriate MT case, and apply
Example merge sort
T (n)  2T (n / 2)  (n)
a  2, b  2; n logb a  n log2 2  n  (n)
Also f (n)  (n)


 Case 2: T (n)   nlogb a lg n    n lg n 
September 25, 2002
23
Examples of MM
T (n)  T (n / 2)  1
a  1, b  2; n log2 1  1
also f (n)  1, f ( n)  (1)
 Case 2: T (n)  (lg n)
Binary-search(A, p, r, s):
q(p+r)/2
if A[q]=s then return q
else if A[q]>s then
Binary-search(A, p, q-1, s)
else Binary-search(A, q+1, r, s)
T (n)  9T (n / 3)  n
a  9, b  3;
f (n)  n, f (n)  O( n log3 9 ) with   1
 Case 1: T (n)    n 2 
September 25, 2002
24
Examples (2)
T (n)  3T ( n / 4)  n lg n
a  3, b  4; n log 4 3  n 0.793
f (n)  n lg n, f (n)  (n log 4 3 ) with   0.2
 Case 3:
Regularity condition
af (n / b)  3(n / 4) lg(n / 4)  (3 / 4)n lg n  cf ( n) for c  3 / 4
T (n)  (n lg n)
T (n)  2T ( n / 2)  n lg n
a  2, b  2; n log 2 2  n1
f (n)  n lg n, f (n)  (n1 ) with  ?
also n lg n / n1  lg n
 neither Case 3 nor Case 2!
September 25, 2002
25
Examples: Multiplication (I)


Multiplying two n-digit (or n-bit) numbers
costs n2 digit multiplications using a
classical procedure.
Observation:
23*14 = (2101 +3)*(1101 +4) =
(2*1)102+ (3*1 + 2*4)101 + (3*4)


To save one multiplication we use a trick:
(3*1 + 2*4) = (2+3)*(1+4) - (2*1) - (3*4)
September 25, 2002
26
Examples: Multiplication (II)

To multiply a and b, which are n-digit numbers, we
use a divide and conquer algorithm. We split them
in half:


Then:


a = a1 10n/2+ a0 and b = b1 10n/2+ b0
a *b = (a1 *b1 )10n+ (a1 *b0 + a0 *b1)10n/2 + (a0 *b0 )
Use a trick to save a multiplication:
(a1 *b0 + a0 *b1) = (a1 +a0)*(b1 +b0) - (a1 *b1 ) - (a0 *b0 )
September 25, 2002
27
Examples: Multiplication (III)

The number of single-digit multiplications
performed by the algorithm can be described by
a recurrence:
1
if n  1

T ( n)  
3T (n / 2) if n  1

Solution: T (n)  nlog 3  n1.585
2
September 25, 2002
28
Next Week

Sorting


QuickSort
HeapSort
September 25, 2002
29