Introduction to
Algorithms
Jiafen Liu
Sept. 2013
Today’s task
• Develop more asymptotic notations
• How to solve recurrences
Θ-notation
• Math:
Θ(g(n)) = { f(n) : there exist positive
constants c1, c2, and n0 such that 0 ≤ c1g(n)
≤ f (n) ≤ c2g(n) for all n ≥ n0}
• Engineering:
– Drop low-order terms;
– ignore leading constants.
O-notation
• Another asymptotic symbol used to
indicate upper bounds.
• Math:
• Ex : 2n2= O(n3)
(c= 1, n0= 2)
• “=” means “one-way” equality
Set definition of O-notation
• Math:
• Looks better?
• EX : 2n2 ∈ O(n3)
• O-notation corresponds roughly to “less
than or equal to”.
Usage of O-notation
• Macro substitution: A set in a formula
represents an anonymous function in the
set, if O-notation only occurs on the right
hand of formula.
• Ex:
f(n) = n3+ O(n2)
Means f(n) = n3+ h(n)
for some h(n) ∈O(n2), here we can think of
h(n) as error term.
Usage of O-notation
• If O-notation occurs on the left hand of
formula, such as n2+ O(n) = O(n2), which
means
for any f(n) ∈O(n):
there exists some h(n) ∈O(n2),
makes n2+ f(n) = h(n)
• O-notation is an upper-bound notation.
– It makes no sense to say f(n) is at least O(n2).
Ω-notation
• How can we express a lower bound?
• Ex:
Θ-notation
•
•
•
•
O-notation is like ≤
Ω-notation is like ≥.
And Θ-notation is like =
Now we can give another definition of Θnotation
• We also call Θ-notation as “tight bound”.
A Theorem on Asymptotic Notations
• Theorem 3.1
– For any two functions f(n) and g(n), we say
f(n)=Θ(g(n)) if and only if f(n)=O(g(n)) and
f(n)=Ω(g(n)).
• Ex:
prove that?
, how can we
Two More Strict Notations
• We have just said that:
– O-notation and Ω-notation are like ≤and ≥.
– o-notation and ω-notation are like < and >.
– Difference with O-notation : This inequality
must hold for all c instead of just for 1.
What does it mean? “for any constant c ”
• With o-notation, we mean that: no matter
what constant we put in front of g(x), f(x)
will be still less than cg(x) for sufficiently
large n. No matter how small c is.
• Ex: 2n2= o(n3)
(n0= 2/c)
• An counter example:
• 1/2n2 = Ω (n2) does not hold for each c.
Two More Strict Notations
• We have just said that:
– O-notation and Ω-notation are like ≤and ≥.
– o-notation and ω-notation are like < and >.
– Difference with Ω-notation : This inequality
must hold for all c instead of just for 1.
Solving recurrences
• The analysis of merge sort from Lecture 1
required us to solve a recurrence.
• Lecture 3: Applications of recurrences to divideand-conquer algorithms.
• We often omit some details inessential while
solving recurrences:
– n is supposed to be an integer because the size of input
is an integer typically .
– Ignore the boundary conditions for convenience.
Substitution method
• The most general method : Substitution
method
– Guess the form of the solution.
– Verify by induction.
– Solve for constants.
• Substitution method is used to
determine the upper or lower bounds of
recurrence.
Example of Substitution
• EXAMPLE: T(n) = 4T(n/2) + n (n≥1)
– (Assume that T(1) = Θ(1) )
• Can you guess the time complexity of it?
• Guess O(n3). (Prove O and Ω separately.)
– We will prove T(n) ≤ cn3 by induction.
– First ,we assume that T(k) ≤ ck3 for k < n
– T(k) ≤ ck3 holds while k=n/2 by assumption.
Example of Substitution
desired
desired
residual
All we need is
this holds as long as c ≥ 2
for n ≥ 1
Base Case in Induction
• We must also handle the initial conditions,
that is, ground the induction with base
cases.
• Base: T(n) = Θ(1) for all n < n0, where n0 is
a suitable constant.
• For 1 ≤ n< n0, we have “Θ(1)” ≤ cn3, if we
pick c big enough.
Here we are! BUT this bound is not tight !
A tighter upper bound?
• We shall prove that T(n) = O(n2) .
• Assume that T(k) ≤ ck2 for k < n.
desired
Anybody can tell me why?
Now we need –n ≥ 0
residual
But it seems impossible
for n ≥ 1
A tighter upper bound!
• IDEA: Strengthen the inductive hypothesis.
Subtract a low-order term.
• Inductive hypothesis: T(k) ≤ c1k2 – c2k for k< n.
desired
residual
Now we need
(c2-1) n ≥0,
it holds if c2 ≥ 1
For the Base Case
• We have proved now that for any value of
c1, and provided c2 ≥ 1.
• Base: We need T(1) ≤ c1 – c2
• Assumed T(1) is some constant.
• We need to choose c1 to be sufficiently
larger than c2, and c2 has to be at least 1.
• To handle the initial conditions, just pick c1
big enough with respect to c2.
About Substitution method
• We have worked for upper bounds, and
the lower bounds are similar.
– Try it yourself.
• Shortcomings:
– We had to know the answer in order to find it,
which is a bit of a pain.
– It would be nicer to just figure out the answer
by some procedure, and that will be the next
two techniques.
Recursion-tree method
• A recursion tree models the costs (time) of
a recursive execution of an algorithm.
• The recursion-tree method can be
unreliable, just like any method that uses
dot,dot,dots (…).
• The recursion-tree method promotes
intuition, however.
• The recursion tree method is good for
generating guesses for the substitution
method.
Example of recursion tree
• Solve T(n) = T(n/4) + T(n/2)+ n2:
Example of recursion tree
• Solve T(n) = T(n/4) + T(n/2)+ n2:
Example of recursion tree
• Solve T(n) = T(n/4) + T(n/2)+ n2:
Example of recursion tree
• Solve T(n) = T(n/4) + T(n/2)+ n2:
?
How many leaves? <n
We just need an upper bound.
Example of recursion tree
• Solve T(n) = T(n/4) + T(n/2)+ n2:
?
?
?
Recall 1+1/2+1/4+1/8+…
• So T(n) = Θ(n2)
?
＜2n2
The Master Method
• It looks like an application of the recursion
tree method but with more precise.
• The sad part about the master method is it
is pretty restrictive.
• It only applies to recurrences of the form:
T(n) = a T(n/b) + f(n) ,
where a ≥1, b >1, and f(n) is asymptotically
positive.
• aT(n/b) means every problem you recurse on
should be of the same size.
Three Common Cases
Three Common Cases
• Case 1:
–
Three Common Cases
• Case 1:
–
• Case 2:
–
Three Common Cases
• Case 3:
–
Examples
• Ex: T(n) = 4T(n/2) + n
– a = 4, b= 2
–
and
–
–
Examples
• Ex: T(n) = 4T(n/2) + n2
– a = 4, b= 2
–
and
–
–
Examples
• Ex: T(n) = 4T(n/2) + n3
– a = 4, b= 2
–
and
–
–
Homework
• Read Chapter 3 and 4 to be prepared for
applications of recurrences.
Proof of Master Method
Height
=
log
？bn
#(leaves) = ?
Proof of Master Method
Height
=
log
？bn
#(leaves) = ?
• CASE1: The weight increases geometrically
from the root to the leaves. The leaves hold a
constant fraction of the total weight.
Proof of Master Method
Height
=
log
？bn
#(leaves) = ?
• CASE2(k= 0) :The weight is approximately the
same on each of the logbn levels.
Proof of Master Method
Height
=
log
？bn
#(leaves) = ?
• CASE3: The weight decreases geometrically
from the root to the leaves. The root holds a
constant fraction of the total weight.