Nattee Niparnan
Dept. of Computer Engineering,
Chulalongkorn University
Why Analysis?
 We need to know the “behavior” of the algorithm
 Behavior == How well does it perform?
Can we?
 Sure!
 If we have the algorithm
 We can implement it
 We can test it on any input
 But…
 Is that what we really want?
 If you wish to know whether falling from floor 20 of Eng. 4
building would kill you

Will you try?
Prediction
 We wish to know the “Behavior” of the algorithm
 Without actually trying it
 Back to the suicidal example
 Can you guess whether you survive jumping from 20th floor of
Eng. 4 building?




What about 15th floor?
What about 10th floor?
What about 5th floor?
What about 2nd floor?
 Why?
Modeling
 If floor number > 3 then
 Die
 Else
 Survive (maybe?)
Modeling
 What about jumping from Central World’s 20th floor?
 What about jumping from Empire State’s 20th floor?
 What about jumping from UCL’s 20th floor?
 Why?
Generalization
 Knowledge from some particular instances might be
applicable to another instance
Analysis
Modeling
Generalization
 We need something that can tell us the behavior of the
algorithm
Measurement
 What we really care?
RESOURCE!!!
Time
(CPU power)
Space
(amount of RAM)
Model
 How to describe
Usage of
Resource
of an algorithm?
how does
an algo use
resource?
Model
 Resource Function
Input ?
Time
Function of an
algorithm A
Time
used
Space
Function of an
algorithm A
Space
used
Size of
input
Input ?
Example
 Inserting a value into a sorted array
 Input:
 a sorted array A[1..N]
 A number X
 Output
 A sorted array A[1..N+1] which includes X
Algorithm
 Element Insertion
idx = N;
while (idx >= 1 && A[idx] > X) {
A[idx + 1] = A[idx];
idx--;
}
A[idx] = X;
 Assume that X = 20
 What if A = [1,2,3]?
 What if A = [101,102,103]?
How much time?
Using the Model
Time
best
average
worst
Size of
Input
Resource Function
 Measure resource by “size of input”
 Why?
Conclusion
 Measurement for algorithm
 My modeling and generalization
 For prediction of behavior
 Measure as a function on the size of input
 With some simplification
 Best, avg, worst case
Nattee Niparnan
What is “better” in our sense?
 Takes less resource
 Consider this
which one is better?
f(x)
g(x)
Ruler
f(x)
g(x)
What is “better” in our sense?
 which one is better?
 Which ruler to use?
 Performance is now a function, not a single value
 Can we use single value?
 Use the ruler where it’s really matter
 i.e., when N is large
 What is large N?

Infinity?
 Implication?
Comparing by infinite N
 There is some problem
 Usually,
lim f ( x)  
n 
Separation between Abstraction and
Implementation
 Rate of Growth
 by changing the size of input, how does the TIME and
SPACE requirement change
 Describe the TIME, SPACE used by function of N, the
size of input
 F(n) = n3+2n2 + 4n + 10
Rate of Growth
 Compare by how f(x) grows when n increase, w.r.t. g(x)
f ( x) / g ( x)
f ( x)
lim
n  g ( x )
Compare by RoG
f ( x)
lim
n  g ( x )
0
f(x) grows “slowzer” than g(x)
∞
f(x) grows “faster” than g(x)
else
f(x) grows “similar” to g(x)
Growth Rate Comparison
lim f (n) / g (n)  0
n 








0.5n
1
log n
log6 n
n0.5
n3
2n
n!
Sometime it is simple
Some time it is not
l’Hôpital’s Rule
 Limit of ratio of two functions equal to limit of ratio of
their derivative.
 Under specific condition
l’Hôpital’s Rule
 If
 then
Example of Growth Rate Comparison
© การวิเคราะห์และออกแบบอัลกอริทมึ , สมชาย ประสิทธิจู์ ตระกูล, 2544
The problem of this approach
 What if f(x) cannot be differentiate?
 Too lazy to diff
Compare by Classing
 Coarse grain comparison
 Another simplification
 Work (mostly) well in practice
 Classing
Classing
 Simplification by classification
 Grading Analogy
Score
Grade
>= 80
A
70 <= x < 80
B
60 <= x < 70
C
50 <= x < 60
D
< 50
F
Compare by Classification
algo
Compare by Classification
Group B
Group A
algo
Group F
Group C
grouping
Group D
Compare by Classification
Group B
Group A
algo
Group F
Group C
Group D
Select a
representative
Compare by Classification
 Group by some property
 Select representative
 Use representative for comparison
 If we have the comparison of the representative
 The rest is to do the classification
Complexity Class
 We define a set of complexity class
 using rate of growth
 Here comes the so-called Asymptotic Notation
 Q, O, W, o, w
 Classify by asymptotic bound
Asymptote
 Something that bound curve
Curve
Asymptote
Asymptote
 Remember hyperbola?
O-notation
cg(x)
For function g(n), we define O(g(n)),
big-O of n, as the set:
f(x)
O(g(n)) = {f(n) :
 positive constants c and n0,
such that n  n0,
we have 0  f(n)  cg(n) }
n0
Intuitively: Set of all functions
whose
is the same
as or lower than that of g(n).
g(n) is an
for f(n).
f(x) O(g(x))
W -notation
For function g(n), we define W(g(n)),
big-Omega of n, as the set:
f(x)
W(g(n)) = {f(n) :
 positive constants c and n0,
such that n  n0,
cg(x)
we have 0  cg(n)  f(n)}
Intuitively: Set of all functions
whose
is the same
as or higher than that of g(n).
g(n) is an
n0
for f(n).
f(x) W(g(x))
Q-notation
For function g(n), we define Q(g(n)),
big-Theta of n, as the set:
c2g(x)
Q(g(n)) = {f(n) :
 positive constants c1, c2, and n0,
such that n  n0,
we have 0  c1g(n)  f(n)  c2g(n)
f(x)
c1g(x)
}
Intuitively: Set of all functions that
have the same
as g(n).
g(n) is an
n0
for f(n).
f(x) Q(g(x))
Example
 F(n) = 300n + 10
 is a member of Q(30n)
 why?



let c1 = 9
let c2 = 11
let n = 1
Another Example
 F(n) = 300n2 + 10n
 is a member of Q(10n2)
 why?



let c1 = 29
let c2 = 31
let n = 11
How to Compute?
 Remove any constant
 F(n) = n3+2n2 + 4n + 10

is a member of Q(n3+n2 + n)
 Remove any lower degrees
 F(n) = n3+2n2 + 4n + 10

is a member of Q(n3)
Relations Between Q, W, O
For any two functions g(n) and f(n),
f(n) = Q(g(n)) iff
f(n) = O(g(n)) and f(n) = W(g(n)).
 I.e., Q(g(n)) = O(g(n))  W(g(n))
 In practice, asymptotically tight bounds are
obtained from asymptotic upper and lower
bounds.
Practical Usage
 We say that the program has a worst case running time
of O(g(n))
 We say that the program has a best case running time
of W(g(n))
 We say that the program has a tight-bound running
time of Q(g(n))
Example
 Insertion sort takes O(n2) in the worst case
 Insertion sort takes W(n) in the best case
o-notation
For a given function g(n), the set little-o:
o(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  f(n) < cg(n)}.
w -notation
For a given function g(n), the set little-omega:
w(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  cg(n) < f(n)}.
Remark on Notation
 An asymptotic group is a set
 Hence f(n) is a member of an asymptotic group
 E.g., f(n)  O( n )
 Not



f(n) = O( n )
But we will see this a lot
It’s traditions
Comparison
of
Functions
f (n) g(n)  a  b
f (n) = O(g(n))  a  b
f (n) = W(g(n))  a  b
f (n) = Q(g(n))  a = b
f (n) = o(g(n))  a < b
f (n) = w(g(n))  a > b
Lost of Trichotomy
 Trichotomy
 given two numbers a,b
 it must be one of the following

a < b, a > b, a=b
 For our asymptotic notation
 given f(n) and g(n)
 it is possible that


f(n) != O(g(n)) and f(n) != W(g(n))
e.g., n, n1+sin n
Properties
 Transitivity
f(n) = Q(g(n)) & g(n) = Q(h(n))  f(n) = Q(h(n))
f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))
f(n) = W(g(n)) & g(n) = W(h(n))  f(n) = W(h(n))
f(n) = o (g(n)) & g(n) = o (h(n))  f(n) = o (h(n))
f(n) = ω(g(n)) & g(n) = ω(h(n))  f(n) = ω(h(n))
 Reflexivity
f(n) = Q(f(n))
f(n) = O(f(n))
f(n) = W(f(n))
Properties
 Symmetry
f(n) = Q(g(n)) iff g(n) = Q(f(n))
 Complementarity
f(n) = O(g(n)) iff g(n) = W(f(n))
f(n) = o(g(n)) iff g(n) = ω((f(n))
Complexity Graphs
n
log(n)
Complexity Graphs
n log(n)
n
n
log(n)
Complexity Graphs
n10
n3
n2
n log(n)
Complexity Graphs (log scale)
nn
3n
n20
2n
n10
1.1n
Common Class of Growth Rate
 constant :
 logarithmic :
 polylogarithmic :
 sublinear :
 linear :
 quadratic :
 polynomial :
 exponential :
Θ( 1 )
Θ( log n )
Θ( logc n ) , c ≥ 1
Θ( na ) , 0 < a < 1
Θ( n )
Θ( n2 )
Θ( nc ) , c ≥ 1
Θ( cn ) , c > 1
Logarithm
 Base of log is irrelevant
 log b n = ( log c n ) / ( log c b )
 log 10 n = ( log 2 n ) / ( log 2 10 ) = Θ( log n )
 any polynomial function of n does not matter
 log n30 = 30 log n = Θ( log n )
Conclusion
 Compare which one is better
 By comparing their ratio when n approach infinity
 Compare by complexity class
 Asymptotic notation
 What is
 Property