Analysis of Algorithm
Why Analysis?
• We need to know the “behavior” of
algorithms
– How much resource (time/space) does it use
• So that we know when to use which
algorithm
– So that two algorithm can be compared whether
which one is better, with respect to the
situtation
Can we?
• Sure!
• If we have the algorithm
– We can implement it
– We can test it with input
• But…
– Is that what we really want?
– If you wish to know whether falling from floor 20 of
Eng. 4 building would kill you
• Will you try?
Prediction
• We wish to know the “Behavior” of the
algorithm
– Without actually trying it
• Back to the suicidal example
– Can you guess whether you survive jumping from
20th floor of Eng. 4 building?
–
–
–
–
– Why?
What about 15th floor?
What about 10th floor?
What about 5th floor?
What about 2nd floor?
Modeling
• If floor_number > 3 then
– Die
• Else
– Survive (maybe?)
Describe behavior using some
kind of model, rule, etc.
Generalization
• What about jumping from Central World’s 20th
floor?
• What about jumping from Empire State’s 20th
floor?
• What about jumping from Bai Yok’s 20th floor?
• Can our knowledge (our analysis of the
situation) be applicable on the above
questions?
Generalization
• Knowledge from some particular instances
might be applicable to another instance
Analysis
Modeling
Generalization
• We need something that can tell us the
behavior of the algorithm that is…
– Useful (give us knowledge without actually doing it)
– Applicable (give us knowledge for similar kind of
situation)
Analysis (Measurement)
• What we really care?
RESOURCE
Time
(CPU power)
Space
(amount of RAM)
Model
• How to describe
algorithm?
Usage of
Resource
of an
how well
does an
algo use
resource?
Model
• Resource Function
Input ?
Time
Function of
algorithm A
Time
used
Space
Function of
algorithm A
Space
used
Size of
input
Input ?
Example
• Inserting a value into a sorted array
• Input:
– a sorted array A[1..N]
– A number X
• Output
– A sorted array A[1..N+1] which includes X
Algorithm
• Element Insertion
idx = N;
while (idx >= 1 && A[idx] > X) {
A[idx + 1] = A[idx];
idx--;
}
A[idx] = X;
• Assume that X = 20
– What if A = [1,2,3]?
– What if A = [101,102,103]?
Usually, resource varies
according to size of input
How much time?
Using the Model
Time
average
best
worst
Size of
Input
Resource Function
• Give us resource by “size of input”
– Why?
• Easy to compute
• Applicable (usually give meaningful, fairly accurate
result without much requirement)
Conclusion
• Measurement for algorithm
– By modeling and generalization
– For prediction of behavior
• Measurement is functions on the size of
input
– With some simplification
– Best, avg, worst case
ASYMPTOTIC NOTATION
Comparing two algorithms
• We have established that a “resource
function” is a good choice of measurement
• The next step, answering which function is
“better”
What is “better” in our sense?
• Takes less resource
• Consider this
which one is better?
f(x)
g(x)
Slice
f(x)
g(x)
What is “better” in our sense?
• which one is better?
– Performance is now a function, not a single value
– Which slice to use?
• Can we say “better” based on only one slice?
• Use the slice where it’s really matter
– i.e., when N is large
– What is large N?
• Infinity?
– Implication?
Comparison by infinite N
• There is some problem
f ( x)  
– Usually, lim
n 
– The larger the problem, the more resource used
Separation between Abstraction and
Implementation
• Rate of Growth
– by changing the size of input, how does the
and
requirement change
– Compare by how f(x) grows when x increase,
w.r.t. g(x)
f ( x) / g ( x)
f ( x)
lim
n  g ( x )
Compare by RoG
f ( x)
lim
n  g ( x )
0
: f(x) grows “slowzer” than g(x)
∞
: f(x) grows “faster” than g(x)
else
: f(x) grows “similar” to g(x)
Growth Rate Comparison
•
•
•
•
•
•
•
•
0.5n
1
log n
log6 n
n0.5
n3
2n
n!
lim f (n) / g (n)  0
n 
Sometime it is simple
Some time it is not
l’Hôpital’s Rule
• Limit of ratio of two functions equal to limit
of ratio of their derivative.
– Under specific condition
l’Hôpital’s Rule
• If
• then
The problem of this approach
• What if f(x) cannot be differentiated?
• Too complex to find derivative
Compare by Classing
• Coarse grain comparison
– Another simplification
• Work (mostly) well in practice
• Classing
Classing
• Simplification by classification
• Grading Analogy
Score
Grade
>= 80
A
70 <= x < 80
B
60 <= x < 70
C
50 <= x < 60
D
< 50
F
Compare by Classification
algo
Compare by Classification
Group B
Group A
algo
Group F
Group C
grouping
Group D
Compare by Classification
Group B
Group A
>= 80
70 <= x < 80
algo
60 <= x < 70
Group C
Describe
“simplified”
property
Group F
x < 50
Group D
50 <= x < 60
Compare by Classification
• Group by some similar property
– Select a representative of the group
– Use the representative for comparison
• If we have the comparison of the
representative
– The rest is to do the classification
Complexity Class
• We define a set of complexity class
– using rate of growth
• Here comes the so-called
• Classify by asymptotic bound
Asymptote
• Something that bounds curves
Curve
Asymptote
Asymptote
• Remember hyperbola?
O-notation
cg(x)
For function g(n), we define O(g(n)),
big-O of n, as the set:
f(x)
O(g(n)) = {f(n) :
 positive constants c and n0,
such that n  n0,
we have 0  f(n)  cg(n) }
n0
Intuitively: Set of all functions
whose
is the same
as or lower than that of g(n).
g(n) is an
for f(n).
f(x) O(g(x))
 -notation
For function g(n), we define (g(n)),
big-Omega of n, as the set:
f(x)
(g(n)) = {f(n) :
 positive constants c and n0,
such that n  n0,
cg(x)
we have 0  cg(n)  f(n)}
n0
Intuitively: Set of all functions
whose
is the same
as or higher than that of g(n).
g(n) is an
for f(n).
f(x) (g(x))
-notation
For function g(n), we define (g(n)),
big-Theta of n, as the set:
c2g(x)
(g(n)) = {f(n) :
 positive constants c1, c2, and n0,
such that n  n0,
we have 0  c1g(n)  f(n)  c2g(n)
f(x)
c1g(x)
}
n0
Intuitively: Set of all functions that
have the same
as g(n).
g(n) is an
for f(n).
f(x) (g(x))
Example
• F(n) = 300n + 10
– is a member of (30n)
– why?
• let c1 = 9
• let c2 = 11
• let n = 1
Another Example
• F(n) = 300n2 + 10n
– is a member of (10n2)
– why?
• let c1 = 29
• let c2 = 31
• let n = 11
How to Compute?
• Remove any constant
– F(n) = n3+2n2 + 4n + 10
• is a member of (n3+n2 + n)
• Remove any lower degrees
– F(n) = n3+2n2 + 4n + 10
• is a member of (n3)
Relations Between , , O
For any two functions g(n) and f(n),
f(n) = (g(n)) if and only if
f(n) = O(g(n)) and f(n) = (g(n)).
• I.e., (g(n)) = O(g(n))  (g(n))
• In practice, asymptotically tight bounds are
obtained from asymptotic upper and lower
bounds.
Practical Usage
• We say that the program has a worst case
running time of O(g(n))
• We say that the program has a best case
running time of (g(n))
• We say that the program has a tight-bound
running time of (g(n))
Example
• Insertion sort takes O(n2) in the worst case
– Meaning: at worst, insertion sort, takes time
that grows not more than quadratic of the size of
the input
• Insertion sort takes (n) in the best case
– Meaning: at best, insertion sort, takes time that
grows not less than linear to the size of the input
o-notation
For a given function g(n), the set little-oh:
o(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  f(n) < cg(n)}.
w -notation
For a given function g(n), the set little-omega:
ω(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  cg(n) < f(n)}.
Remark on Notation
• An asymptotic group is a set
– Hence f(n) is a member of an asymptotic group
– E.g., f(n)  O( n )
• Strictly speaking, f(n) = O( n ) is syntactically wrong
• But we will see this a lot
• It’s traditions
Comparison of Functions
f (n)  O(g(n))  f (n)  g(n)
f (n)  (g(n))  f (n)  g(n)
f (n)  (g(n))  f (n) = g(n)
f (n)  o(g(n))  f (n) < g(n)
f (n)  w(g(n))  f (n) > g(n)
Where < , > , = means grows slower, faster, equally
Lost of Trichotomy
• Trichotomy
– given two numbers a,b
– Only one of the following must be true
• a < b, a > b, a=b
• For our asymptotic notation
– given f(n) and g(n)
– it is possible that
• f(n) != O(g(n)) and f(n) != (g(n))
• e.g., n, n1+sin n
Properties
• Transitivity
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))
f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))
f(n) = o (g(n)) & g(n) = o (h(n))  f(n) = o (h(n))
f(n) = ω(g(n)) & g(n) = ω(h(n))  f(n) = ω(h(n))
• Reflexivity
f(n) = (f(n))
f(n) = O(f(n))
f(n) = (f(n))
Properties
• Symmetry
f(n) = (g(n))  g(n) = (f(n))
• Complementarity
f(n) = O(g(n))  g(n) = (f(n))
f(n) = o(g(n))  g(n) = ω((f(n))
Complexity Graphs
n
log(n)
Complexity Graphs
n log(n)
n
n
log(n)
Complexity Graphs
n10
n3
n2
n log(n)
Complexity Graphs (log scale)
nn
3n
n20
2n
n10
Eventually, 1.1n will overcome
n10
n
1.1
Common Class of Growth Rate
•
•
•
•
•
•
•
•
constant :
logarithmic :
polylogarithmic :
sublinear :
linear :
quadratic :
polynomial :
exponential :
Θ( 1 )
Θ( log n )
Θ( logc n ) , c ≥ 1
Θ( na ) , 0 < a < 1
Θ( n )
Θ( n2 )
Θ( nc ) , c ≥ 1
Θ( cn ) , c > 1
Logarithm
• Base of log is irrelevant
– log b n = ( log c n ) / ( log c b )
– log 10 n = ( log 2 n ) / ( log 2 10 ) = Θ( log n )
• any polynomial function of n does not
matter
– log n30 = 30 log n = Θ( log n )
Conclusion
• Compare which one is better
– By comparing their ratio when n approaches
infinity
– By comparing their asymptotic notation of the
resource function
• Asymptotic notation
– What is
– Property