• Time comparison
• If n =100,000 then T(100,000) = 10 seconds
• Is the algorithm linear (n), it would take 20 seconds to sort 200,000 Numbers
• Is algorithm Quadratic (n2), it would take 40 seconds to sort 200,000 Numbers
• Is algorithm cubic (n3), it would take 80 seconds to sort 200,000 Numbers
• Is the algorithm logarithmic, it would take 10.6 seconds to sort 200,000 Numbers
T(n) = c log10 n
10 = c log10 100000 (the known case)
c = 10/log10105= 10/5 =2
• For n = 100000, 2n = 200000
T(200000) = 2 log102n = 2 log102 . Log10 100000
= 2(log10105 + log102)
= 2(5 + 0.3)
= 10.6
78
• Complexity
• Algorithms can be classified as linear, quadratic, logarithmic,
etc. depending on the time they take as a function of a number
denoting the size of the problem.
• The insertion sort algorithm (worst-case) was quadratic with
a merge sort of n lg n complexity
• The task is to compare the time or space complexity of
different algorithms for large sizes of inputs
• Assume that the time (or space) required for performing the
algorithm is some function of n, which is the size of the input
• The evaluation of the comparative performance of the two
algorithms for large problem sizes can be calculated as
follows:
f ( n)
n →∞ g ( n)
lim
79
• Complexity
1)
2)
3)
f ( n)
=∞
lim
g
(
n
)
n →∞
f ( n)
=0
lim
(
)
g
n
n →∞
f ( n)
=C
lim
(
)
g
n
n →∞
1) The limit is ∞. In this case, we say that f(n) grows
asymptotically much faster than g(n) (or, equivalently, that g(n)
grows much slower than f(n)
2) The limit is 0. This is the opposite of the first case; f(n) grows
much slower than g(n) (or, equivalently, that g(n) grows much
faster)
3) The limit is some constant. In this case, both f(n) and g(n) grow
at roughly the same rate (asymptotically)
80
• Complexity
• Definition: Θ-Notation
• For a given function g(n) the set of function is denoted by Θ(g(n))
• Θ(g(n)) = { f(n) : positive constants c1, c2, and n0 exist such that
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0 }
• A function f(n) belongs to the set Θ(g(n)) if positive constants c1 and c2
exist such that it can be “sandwiched” between c1 g(n) and c2 g(n), for
sufficiently large n.
• Because Θ(g(n)) is a set, we could write “f(n) ∈ Θ(g(n))” to indicate that
f(n) is a member of Θ(g(n))
81
• Complexity, Θ-Notation
• Instead, “f(n) = Θ(g(n))” is used to
express the same notion
worst-case
• The figure gives an intuitive
diagram of functions f(n) and g(n) for
f(n) = Θ(g(n))”
• For all values of n to the right of n0,
the value of f(n) lies at or above c1
g(n) and at or c1 g(n)
best-case
• In other words, for all n ≥ n0, the
function f(n) is equal to g(n) to within
a constant factor. The g(n) is an
asymptotically tight bound for f(n)
Θ(g(n)) = { f(n) : 0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0 }
82
• Complexity
• Definition: O-Notation
• Remember that the Θ-notation
asymptotically bounds a function
from above & below
• The O-Notation is an asymptotic
upper bound as illustrated
• For a given function g(n) the set of
function is denoted by O(g(n))
O(g(n)) = { f(n) : the positive constants c, and n0 exist such that
0 ≤ f(n) ≤ c g(n) for all n ≥ n0 }
83
• Complexity
• Definition: Ω-Notation
• Remember that the O-notation
provides an asymptotic upper
bound on a function
• The Ω-Notation is an asymptotic
lower bound as illustrated below
• For a given function g(n) the set
of function is denoted by Ω(g(n))
Ω(g(n)) = { f(n) : the positive constants c, and n0 exist such that
0 ≤ c g(n) ≤ f(n) for all n ≥ n0 }
84
• Complexity
• Summary:
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
0 ≤ f(n) ≤ c g(n) for all n ≥ n0
0 ≤ c g(n) ≤ f(n) for all n ≥ n0
85
Algorithms & Data Structures
• Professor Reza Sedaghat
• COE428: Engineering Algorithms & Data Structures
•Email address: rsedagha@ee.ryerson.ca
• Course outline: www.ee.ryerson.ca/~courses/COE428/
• Course References:
1) Thomas H. Cormen, Charles E. Leiserson, Ronald L.
Rivest. Introduction to Algorithms, MIT, 2002, ISBN: 0-07013151-1 (McGraw-Hill) (Course Text)
86
87
Ordering by asymptotic growth rates
Rank the following functions by order of growth
?
3 log 2
n54
log n
n log n
88
Assignment
Calculate and assign a Θ-notation to each expression bellow.
Expression
a) 2n 2 + 1
Θ(?)
Θ(n2)
6n 3 + 12n 2 + 1
2 lg n + 4n + 3n lg n
6n 6 + n + 4
(6n + 4)(1 + lg n )
2 + 4 + 6 + 8 + ... + 2n
( n + 1)( n + 3)
n+2
( n 2 + lg n )( n + 1)
n + n2
1 + 2 + 3 + 4 + ... + n
89
Assignment
Let f(n) = (n3)
Circle True or False for each statement below. (
True or false: f(n) = O(n2)
True or false: f(n) = O(n3 lgn3)
True or false: f(n) = Θ(2lg n)
True or false: f(n) = O(n1.5)
True or false: f(n) = Ω(n3)
True or false: f(n) = Ω(n lgn3)
True or false: f(n) = Θ(n2 lgn2)
True or false: f(n) = O(2lg n)
True or false: f(n) = Ω(n1.5)
True or false: f(n) = O(n3)
90
• Running time
91
• we find that in the worst case, the running time of INSERTION-SORT is
• This worst-case running time can be expressed as an2 + bn+ c for constants a, b, and c
that again depend on the statement costs ci; it is thus a quadratic function of n
92
2.3.2 Analysis of Merge Sort Algorithm
• How can we determine the time required to perform this algorithm?
• Time to sort n numbers
T(n) = Time to divide the number +
Time to sort left side (size = n/2) +
Time to sort right side (size = n/2) +
Time to merge (total size = n/2 + n/2 = n)
Tdivide
T(n/2)
T(n/2)
c1 n
• Assumptions:
• Let T(n) represent the time to sort n numbers
• Let Tdivide be the time to divide the numbers and assume Tdivide = 0
• Let c1n +c0 be the time to merge n numbers, which is a linear algorithm
and assume and c0 = 0
T(n) =Tdivide + T(n/2) + T(n/2) + c1n + c0
T(n) = 2T(n/2) + n
93
• Complexity
• Summary:
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
0 ≤ f(n) ≤ c g(n) for all n ≥ n0
0 ≤ c g(n) ≤ f(n) for all n ≥ n0
94
Example
!
f(n) = Θ(g(n))
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
f(n) = 2n2 + 3n + 1 =! O(g(n))
Determine c and n0
To be proven: Is g(n) is an asymptotically upper bound for f(n)
0 ≤ f(n) ≤ c g(n) for all n ≥ n0
0 ≤ 2n2 + 3n +1 ≤ cn2
Lets assume g(n) for highest order-term of f(n)
g(n)
f(n)
2n2 + 3n + 1 ≤ 2n2 + 3n2 + 1n2
2n2 + 3n + 1 ≤ 6n2
f(n) ≤ c g(n),
For c = 6 and n ≥ n0=1 f(n) ≤ O(n2)
Example from slide 86:
95
Lower bound
Ω(g(n))
tight bound
Θ(g(n))
upper bound
O(g(n))
c = 6 and n ≥ n0=1 f(n) ≤ O(n2)
g(n)
f(n)
n
96
Example
!
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
f(n) = Θ(g(n))
f(n) =
!
2n2 + 3n + 1 = Ω(g(n))
Determine c and n0
To be proven: Is g(n) is an asymptotically lower bound for f(n)
0 ≤ c g(n) ≤ f(n) for all n ≥ n0
0 ≤ cn2 ≤ 2n2 + 3n +1
Lets assume g(n) for highest order-term of f(n) with its constant
f(n)
g(n)
2n2 ≤ 2n2 + 3n +1
c must be smaller than 2: c < 2
1n2 ≤ 2n2 + 3n +1
c g(n) ≤ f(n)
For c ≤ 2 and n ≥ n0>0 f(n) ≤ Ω(n2)
Example from slide 86:
97
Lower bound
Ω(g(n))
tight bound
upper bound
Θ(g(n))
O(g(n))
c ≤ 1 and n ≥ n0>0 f(n) ≤ Ω(n2)
g(n)
f(n)
n
98
Example
!
1
f(n) = n2 – 3n = Θ(n2)
2
• Determine c1, c2, and n0
• To be proven: Is g(n) is an asymptotically tight bound for f(n)
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
c1n2 ≤
1 2
n – 3n ≤ c2n2
2
• Remember that in the function the largest component gives the running time,
here we drop lower term 3n
1
1 2
n ≤ c2 n2, assuming if n0 ≥ 1 on the right side (upper bound) gives c2 ≥
2
2
• Dividing by n2 gives
c1 ≤
1
3
–
2
n
≤ c2
–
) if n = 6 ⇒ c1 = 0, which is
• On the left side, lower bound, (c1 ≤
2
n
invalid
1
1
• So, n must be n0 ≥ 7 and results in c1 ≥ 14
3
99
Lower bound, n0 ≥ 7, in c1 ≥ 1/14
100
Shortcuts
101
f(n)= ank (+/-) bn (+/-) d
- Case 1:
f(n)= ank + bn + d
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
k=0…m
Upper bound O(g(n))= ank + bnk + dnk
= (a+b+d)nk Constant c is c=a+b+d , n0 ≥ 1
Example: f(n)= 2n2 + 3n + 1
g(n)= 2n2 + 3n2 + n2
= (2+3+1)n2 Constant c is c=6 , n0 ≥ 1
- Case 1a:
f(n)= ank + bn + d
Lower bound Ω(g(n))= ank
Constant c <= a , n0 > 0
Example: f(n)= 2n2 + 3n + 1
g(n)= 2n2
Constant c <= 2 , n0 > 0
102
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
- Case 2:
f(n)= ank - bn + d
Upper bound O(g(n)) = ank
Constant c is c=a , n0 > |d/b| if d>b
Example: f(n)= 3n2 - 2n + 4
g(n)= 3n2
Constant c is c=3 , n0 > 2
- Case 2a:
f(n)= ank - bn + d
Lower bound Ω(g(n)) = (a-1)nk
Constant c is c= (a-1) for a > b, n0 ≥ 1
Constant c is c= a/b for a < b, n0 ≥
Example: f(n)= 3n2 - 2n + 1
g(n)= 2n2
Constant c is c=2 , n0 ≥ 1
Example: f(n)= 2n2 - 3n + 1
g(n)= 2/3n2
Constant c is c=2/3 , n0 ≥
|b/a|
| 3/2 =1.5 |
=2
103
f(n)= ank (+/-) bn (+/-) d
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
- Case 3:
f(n)= ank +bn - d
Upper bound O(g(n)) = (a*b)nk
Constant c is c = a*b , n0 >
|d/b| >=1
Example: f(n)= 2n2 + 3n - 1
g(n)= 6n2
Constant c is c=6 , n0 >1
- Case 3a:
f(n)= ank + bn - d
Lower bound Ω(g(n)) = ank
Constant c is c=a ≥ 0 , n0 ≥
|d/b| >=1
Enample: f(n)= 2n2+ 3n - 1
g(n)= 2n2
Constant c is c=2 , n0 > 1
104
f(n)= ank (+/-) bn (+/-) d
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
- Case 4:
f(n)= ank +bn
Upper bound O(g(n)) = an k+ bnk
= (a+b)nk
Constant c is c = a+b , n0 > 1
Example: f(n)= 2n2 + 3n
g(n)= 2n2 + 3n2
= 5n2
Constant c is c=5 , n0 > 1
- Case 4a:
f(n)= ank + bn
Lower bound Ω(g(n)) = ank
Constant c is c=a ≥ 0, n0 > 1
Example: f(n)= 3n2 + 2n
g(n)= 3n2
Constant c is c=3 , n0 > 1
105
- Case 5:
f(n)= ank -bn
Upper bound O(g(n)) = an k
Constant c is c = a , n0 >
Example: f(n)= 2n2 - 3n
g(n)= 2n2
= 2n2
Constant c is c=2 , n0 >
0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
|b/a|
|3/2|
- Case 5a:
f(n)= ank - bn
Lower bound Ω(g(n)) = ank- bnk for a>b,
, Constant c is c=a-b ≥ 0, n0 > 1
g(n) = 1nk
for a<b
Constant c is c=1 ≥ 0, n0 > | b |
Example: f(n)= 3n2 - 2n
g(n)= n2
Constant c is c=1 , n0 > 1
Example: f(n)= 2n2 - 3n
g(n)= n2
Constant c is c=1 , n0 > 3
106
L (Level)
Merge Sort
1
2
2
3
4
5
6
6
0
n / 2L = 8 / 20 = 8
2
3
6
1
n / 2L = 8 / 21 = 4
2
n / 2L = 8 / 22 = 2
3
n / 2L = 8 / 23 = 1
merge
2
4
5
6
1
merge
2
5
merge
5
2
merge
4
6
1
3
2
merge
merge
merge
4
1
2
6
3
6
6
n / 2L = 1
n = 2L
lg2 n = L
lg n
107
Complexity: Recurrence
- Recursion Tree Method
- Substitution Method
108
Complexity: Recurrence
- Recursion Tree Method
n
T(n) =
Θ(1) if n = 1;
T(n-1) + n if n > 1.
Σ
109
Complexity: Recurrence
- Substitution Method
T(n) =
Θ(1) if n = 1;
T(n-1) + n if n > 1.
note: don’t add n.
Generate a sequence
110
Complexity: Recurrence
- Recursion Tree Method
T(n)=lgn
T(n) =
Θ(1) if n = 1;
T(n-1) + lgn if n > 1.
lgn
T(lgn-1)
lg(n-1)
T(lgn-2)
lg(n-2)
T(lgn-3)
lg(n-3)
T(2)
lg(2)=1
Σ
T(1) lg1=0
Total =lgn+lg(n-1)+lg(n-2)+…+lg2+lg1=lg[n*(n-1)*(n-2)*…2*1]
= lg n!
= lgn! < nlgn
O(nlgn)
111
Growth: lgn! < nlgn
O(nlgn)
T(n)
nlgn
lgn!
n
112
Complexity: Recurrence
- Substitution Method
T(n) =
Θ(1) if n = 1;
T(n-1) + lgn if n > 1.
T(n)=T(n-1) + lgn Then T(n-1)=T(n-2)+lg(n-1)
T(n-2)=T(n-3)+lg(n-2)
T(n-3)=T(n-4)+lg(n-3)
and so on ……
[
]
Substitute T(n-2): T(n)=[T(n-3)+lg(n-2)]+lg(n-1)+lgn=T(n-3)+lg(n-2)+lg(n-1)+lgn
Substitute T(n-1): T(n)= T(n-2)+lg(n-1) +lgn =T(n-2)+lg(n-1)+lgn
If it is developed for m times then;
T(n)=T(n-m)+lg(n-(m-1))+lg(n-(m-2))+...+lg(n-1)+lgn
If n-m=0 then n=m;
=T(n-n)+lg(n-(n-1))+lg(n-(n-2))+...+lg(n-1)+lgn
=T(n-n)+
= T(0) +
= 1 +
lg1
+
lg2
+...+lg(n-1)+lgn
lg[n*(n-1)*(n-2)*…2*1]
lgn!
lgn! < nlgn
O(nlgn)
113
Complexity: Recurrence
- Recursion Tree Method
T(n)=n
T(n) =
Θ(1) if n = 1;
T(n/2) + n if n > 1.
n
T(n/2)
n/2
n/22
T(n/4)
T(n/8)
Σ
n/23
T(n/2m=1) n/2m
Total = n+(n/2)+(n/22)+(n/23)+ … +(n/2m)
Σ
m
n
= n [1+(1/2)+(1/22)+(1/23)+ … +(1/2m)]=
i=0
= Θ (n)
114
Complexity: Recurrence
- Substitution Method
T(n) =
Θ(1) if n = 1;
T(n/2) + n if n > 1.
T(n)=T(n/2) + n Then T(n/2)=T(n/22)+n/2
T(n/22)=T(n/23)+n/22
T(n/23)=T(n/24)+n/23)
and so on ……
[
]
Substitute T(n/2 ): T(n)=[T(n/2 )+n/2 ]+n/2+n=T(n/2 )+n/2 +n/2+n
Substitute T(n/2): T(n)= T(n/22)+n/2 +n = T(n/22)+n/2+n
2
3
2
3
2
If it is developed for m times then;
T(n)= T(n/2m)+ n/2m-1+n/2m-2+….+n/2+n
If n/2m=1 then n=2m and m=lgn = 1
=1
=1
=
+ n [1/2m-1+1/2m-2+….+1/2+1]
+n[
+ 2n
O(n)
1
+1]
115
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn
…
h = lg n cn/4
cn
Θ(1)
#leaves = n
Total = Θ(n lg n)
116
T (n) = 2T (n/2) + Θ(n),
Upper bound:
T (n) ≤ 2T (n/2) + n
Guess: T (n) ≤ cn lg n
117
T (n) = 2T (n/2) + Θ(n),
Lower bound:
T (n) ≥ 2T (n/2) + n
Guess: T (n) ≥ cn lg n
118
T (n) = 2T (n/2) + Θ(n),
Upper bound:
T (n) ≤ 2T (n/2) + c2n
Guess: T (n) ≤ cn lg n
119
Assignment, Recursion tree
120
T (n) = T (n-1) + Θ(n),
Guess: T (n) ≤ n2
Upper bound:
T (n) ≤ T (n-1) + n
= c (n-1)2 + n
≤ cn2 – 2cn+ c + n
- 2cn + c+ n ≤ 0
n(1-2c) + c ≤ 0
1-2c +c ≤ 0
1≤c
if n=n0=1
121
T (n) = T (n-1) + Θ(n),
Guess: T (n) ≥ n2
lower bound:
T (n) ≥ T (n-1) + n
= c (n-1)2 + n
≥ cn2 – 2cn+ c + n
- 2cn + c+ n ≥ 0
n(1-2c) + c ≥ 0 if n=n0=1
1-2c +c ≥ 0
1≥c>0
122
• Recurrence
• The complexity of recursive algorithm changes, if the cost of each level is
different. For example:
123
• Recurrence
• Example:
There is no lower growing term
f(n)= nk (+/-) n (+/-) d
Factor the common growing term
and calculate the valid c>0, n0>1
For Upper bound c>8 and for Lower bound 0 < c < 8.
124
Recursion tree
…
h = lg n
Θ(1)
#leaves = n
125
126
127
Assignment, Recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/2)2
Θ(1)
(n/8)2
(n/8)2
(
2
(n/4)2
5 n2
16
25 n2
256
…
(n/4)2
(n/16)2
n2
)
() ( )
5
5 2
5 3
1+ 16 + 16 + 16 + 
Total = n
= Θ(n2)
geometric series
128