Asymptotics and Recurrence
Equations
Analysis of Algorithms
Prepared by
John Reif, Ph.D.
Asymptotics and Recurrence
Equations
a) Computational Complexity of a
Program
b) Worst Case and Expected Bounds
c) Definition of Asymptotic Equations
d) Solution of Recurrence Notation
Readings
• Main Reading Selection:
• CLR, Chapter 3, 4 and Appendix A
Goal
• To estimate and compare growth rates
of functions
• Ignore constant factors of growth
“f(n) is asymptotically equal to
g(n)”
f(n) ~ g(n)
if
f(n)
lim
1
n   g(n)
”f(n) is little o g(n) ”
f(n)
0
n  g(n)
• f(n) is o(g(n)) if lim
”f(n) is big O g(n) ”
f(n)
• f(n) is O(g(n)) if lim sup
c
g(n)
n 
Example of f(n) is O(g(n))
 c,n 0  0
s.t. f(n)  c  g(n)
for all n  n 0
“f(n) is order at least g(n)”
• f(n) is Ω(g(n)) if
f(n)
lim inf
c
g(n)
n 
 n 0 , c  0 s.t. f(n)  c g(n)
for all n  n 0
Example of f(n) is (g(n))
“f(n) is order tight with g(n)”
• f(n) is Θ(g(n)) if
 n 0 , c, c' s.t. c  g(n)  f(n)  c' g(n)
Suppose my algorithm runs in time
O(n)
• Don’t say:
• “his runs in time O(n2) so is worse”
• But prove:
• “his runs in time Ω(n2) so is worse”
• Must find a worst case input of length
n for which his algorithm takes time >
cn2 for all n > n0
Use of O Notation
• N is O(n2)
sometimes written
n = O(n2)
• But n2 is not O(n) so can’t use
identities!
•  The two sides of the equality do not
play a symmetric role
Use of Asymptotic Notation
• Write
• As
" f(n)  g(n) is o(h(n))"
f(n)  g(n)  o(h(n))
• Example
n
1
 1 
 1   O 2 
n -1
n
n 
1
1
 1   o 
n
n
 1  o(1) as n  
Convergent Power Sum

1
x 
 O(1), for 0  x  1

1 x
i 0
i
• A polynomial is asymptotically equal to
its leading term as x 
d
∞
 
i
d
a
x

θ
x
 i
i 0
d
 
i
d 1
a
x

o
x
 i
i 0
d
i
d
a
x
~
a
x
 i
d
i 0
Sums of Powers
• for n  ∞
n

i 1
1
i ~
n d 1
d 1
d
• Or equivalently
n

i 1
1
d 1
d 1
i 
n  o n 
d 1
d
Examples
 n
n2
 i ~
2
 i 1
 n 2
3
n

 i ~ 3
 i 1
• 2nd order asymptotic expansion
n

i 1
1
1 d
d 1
d 1
i 
n  n  o n 
d 1
2
d
Asymptotic Expansion of f(n) as
n  n0

f(n) ~  c i g i (n)
i 1
• If
(1)g i 1 (n)  o(g i (n))
for all i  1
• and
k
(2) f(n)   ci g i (n)  o(g k (n))
i 1
for all k  1
Bounding Sums by Integrals
n
n 1
n
k 1
1
k 1
 f(k)   f(x) dx   f (k  1)
Bounding Sums by Integrals
(cont’d)
n
n 1
n
k 1
1
k 1
 f(k)   f(x) dx   f (k  1)
• So
n 1
n
n 1
1
k 1
1
 f(x) dx - f(n  1)  f(1)  f(k)   f(x) dx
• Example if f(x)  ln(x)
then
 ln (x)dx  xln(x)  x
Bounding Sums by Integrals
(cont’d)
• So
n
 ln k  (n  1) ln (n  1) - n  θ(ln(n))
k 1
ln n
• Since log(n) 
ln 2
n
n
 θ (log n)
• So  log k  (n  1) log (n  1) ln 2
k 1
Other Approximations Derived from
Integrals
(n  1) 2
(n  1) 2
k log k 
log (n  1)  θ (n log n)

2
4 ln 2
k 1
n
• Harmonic Numbers
n
Hn  
k 1
1
k

1
 H n  ln(n)  γ  O  
n

 Euler' s constant γ  .577...

Stirling’s Approximation for Factorial
• Factorial
n! = 1· 2 · 3 · · · (n-1) · n
n! ~ 2 n n e
n
-n
as n  
• So
1
log(n! )  n log n - n log e  log (2 n)  θ (1)
2
 n log n - θ (n)
Recurrence Equations
(over integers)
• Homogenous of degree d
n>d
x n  a1 x n 1  a 2 x n 2  ...  a d x n d
• Given
constant coefficients
initial values
a1, …, ad
x1, x2 , …, xd
Example: Fibonacci Sequence
• n>2
Fn = Fn-1 + Fn-2
F0 = 0,
F1 =1
Solution of Fibonacci Sequence
1
r1  (1  5 )  1.618... “golden ratio”
2
1
r2  (1  5 )
2
Fn  c1 r1n  c 2 r2n
where
F0  c1  c 2  0
F1  c1 r1  c 2 r2  1
Solution of Fibonacci Sequence
(cont’d)
• Hence
n
n

1 1 5  1 5  
 
 

Fn 
5  2   2  


1
 Fn ~
5
1 5 


 2 


n
A Useful Theorem
• c > 0, d > 0
n=1
c 0

• If T (n) =   n 
d
aT
+
cn
n>1
 b
  
 


 
θ n logb a


• then T (n) = θ n d log b n

d
 θ n
a > bd
a = bd
a<b
d
Proof
d
T (n) = cn g(n) + a
log b n
d
• Is solution
2
a
 a 
g(n) = 1 + d +  d  + ... +
b
b 
 a 
 d
b 
log b n-1
Cases
 a 
(1) a > b  g(n) ~  d 
b 
is last term so
log b n-1
d

 
T(n) = θ a logb n d =θ n logb a

(2) a = bd  g(n) = log b n
so T(n) = θ  n d log b n 
(3) a < bd  g(n) upper bound by 0(1)
so T(n) = θ  n d 
Example: Mergesort
input list L of length N
if N=1 then return L
else do
let L1 be the first
elements of L

of L
let L2 be the last




 
elements
M1  Mergesort (L1)
M2  Mergesort (L2)
return
Merge (M1 , M2)
Time Bounds of Mergesort
• Initial Value T(1) = c1
  
   
  T 
  c 2 N
for N > 1T(N)  T 
  
   
for some constants c1, c2 > 1
Time Bound (cont’d)

  c2 N

• N > 1 T(N)  2T 
guess T(N)  a N log N + b
 N

N
 2 a
log   + b   c 2 N
2
 2

Holds if a = c1 + c2 ,
Solution
b = c1
T(N)   c1 + c2  N log N + c1
Time Bound (cont’d)

• N > 1 T(N)  2T    c 2 N, T(1) = c1

• Transform Variables
n = log N, N = 2n
N
n  1  log N  log 2  log  
2
• Recurrence equation:
 2X +c 
 T    = T(1) = c
Xn  T 
X0
n
n-1
2
0
1
n
• Solve by usual methods for recurrence
equations

X n  O n n

so T  N  = O (N log N)
Advanced Material
Exact Solution of Recurrence
Relations
Homogenous Recurrence Relations
(no constant additive term)
• Solve:
try
x n  a1 x n 1  a 2 x n 2  ...  a d x n d
xn  rn
Multiply by
d
r
rn
• Get characteristic equation:
r d  a1 r d -1  a 2 r d -2  ...  a d  0
Case of Distinct Roots
• Distinct Roots
r1, r2, …, rd
d
 x n   c i rin
i 1
x n ~ c i rin
• Where ri is dominant root
ri  rj  j  i
Other Case
• Roots are not distinct
r1= r2 = r3
• Then solutions not independent,
so additional terms:
d
x n  c1 r1n  c 2 n r1n  c3 n 2 r1n   ci rin
i4
Inhomogenous Recurrence
Equations
x n  a1 x n 1  a 2 x n 2  ...  a d x n d  a 0
• Nonzero constant term
a0  0
• Solution Method
(1) Solve homogenous equation
Yn  a1 Yn 1  a 2 Yn 2  ...  a n Yn d
Solution Method
1) Solve homogenous equation
Yn  a1 Yn 1  a 2 Yn 2  ...  a n Yn d
2) Case
a
i
1
, add particular solution
Solution Method (cont’d)
Case
a
i
1
, add particular solution
 a0
x n  cn= 
  ia
i


 n

3) Add particular and homogeneous
solutions, and solve for constants
This is all we usually need!!
Asymptotics and Recurrence
Equations
Analysis of Algorithms
Prepared by
John Reif, Ph.D.