Discrete Structures & Algorithms
Functions & Asymptotic Complexity
To understand how to perform computations
efficiently, we need to know
– How to count!
– How to sum a sequence of steps.
– Reason about the complexity/efficiency of an
algorithm.
– Prove the correctness of our analysis.
•
•
•
•
•
Logic
Methods of inference
Proof strategies
Induction
…
– (and eventually) How to create better algorithms.
2
Functions
• We can represent the number of steps needed to
achieve a goal as a function of the size of the
problem.
• Next: A quick refresher on functions
– You should know (most of this) from other
math/engineering courses.
3
Functions
Definition: A function f : A  B is a subset of AxB
where  a  A, b  B unique and <a,b>  f.
(A function is a mapping of elements from one set to another.)
4
Functions
Definition: a function f : A  B is a subset of AxB
where  a  A, b  B unique and <a,b>  f.
A point!
B
B
A
A
5
Functions
A = {Michael, Tito, Janet, Cindy, Bobby}
B = {Katherine Evans, Carol Brady, Mother Teresa}
Let f: A  B be defined as f(a) = mother(a).
Michael
Tito
Janet
Cindy
Bobby
Katherine
Carol
Mother Teresa
6
Image and preimage
image(S) = f(S)
For any set S  A, image(S) = {b : a  S, f(a) = b}
So, image({Michael, Tito}) = {Katherine Evans}
image(A) = B - {Mother Teresa}
Michael
Tito
Janet
Cindy
Bobby
Katherine
Evans
Carol Brady
Mother Teresa
7
Image and preimage
preimage(S) = f-1(S)
For any S  B, preimage(S) = {a: b  S, f(a) = b}
So, preimage({Carol Brady}) = {Cindy, Bobby}
preimage(B) = A
Michael
Tito
Janet
Cindy
Bobby
Katherine Evans
Carol Brady
Mother Teresa
8
Image and preimage
What is image(preimage(S))?
a)
b)
c)
d)
e)
S
S
{}
subset of S
superset of S
who knows?
Michael
Tito
Janet
Cindy
Bobby
Katherine
Evans
Carol Brady
Mother Teresa
9
Functions and their properties
• You should be familiar with these properties (past
classes)
or
• Functions derive many of their properties from sets
(they are defined on sets)
• As you already know much about sets ->
should be able to derive these properties
10
Functions and their properties
f(A  B)  f(A)  f(B)?
Choose an arbitrary b  f(A  B), and show that it
must also be an element of f(A)  f(B).
f(A  B) = {y : x  (A  B), f(x) = y}
So, x  (A  B) such that f(x) = y.
If x  A (it is), then f(x) = y  f(A).
If x  B (it is), then f(x) = y  f(B).
y  f(A), and y  f(B), so b  f(A)  f(B).
11
Injections
Every b  B has
at most 1
preimage.
A function f: A  B is one-to-one (injective, an
injection) if a,b,c, (f(a) = b  f(c) = b)  a = c
Not one-to-one
Michael
Tito
Janet
Cindy
Bobby
Katherine
Evans
Carol Brady
Mother Teresa
12
Surjections
Every b  B has
at least 1
preimage.
A function f: A  B is surjective if b  B, a  A
f(a) = b
Not surjective
Michael
Tito
Janet
Cindy
Bobby
Katherine
Evans
Carol Brady
Mother Teresa
13
Bijections
A function f: A  B is bijective if it is both one-toone (injective) and surjective.
Every b  B has
exactly 1
preimage.
Isaak
Bri
Lynette
Aidan
Evan
Cinda
Dee
Deb
Katrina
Dawn
An important
implication of this
characteristic:
The preimage (f-1)
is a function!
14
Examples
Suppose f: R+  R+, f(x) = x2.
Is f one-to-one?
yes
Is f surjective?
yes
Is f bijective?
yes
15
Examples
Suppose f: R  R+, f(x) = x2.
Is f one-to-one?
yes
Is f surjective?
Is f bijective?
no
no
16
Examples
Suppose f: R  R, f(x) = x2.
Is f one-to-one?
no
Is f surjective?
no
Is f bijective?
no
17
Composition of functions
Let g:XY, and f:YZ be functions. Then the
composition of f and g is:
(f o g)(x) = f(g(x))
g
f
18
Asymptotic complexity
Asymptotic complexity
• Goal: Estimate properties (e.g., running time) of
an algorithm as a function of input size n for
large n.
• Expressed using only the highest-order term in
the expression for the exact running time.
– Instead of exact running time, say Q(n2).
• Describes behavior of function in the limit.
20
Asymptotic notation
• Q, O, W, o, w
• Defined for functions over the natural numbers.
– Example: f(n)  Q(n2).
– Describes how f(n) grows in comparison to n2.
 Q(n2) define a set of functions; in practice used to
compare two function sizes.
• The notations describes the rate-of-growth for a set
of functions.
21
Q-notation (big-Theta)
For function g(n), we define Q(g(n)),
big-Theta of n, as the set:
Q(g(n)) = {f(n) :
 positive constants c1, c2, n0,
such that n  n0,
we have c1g(n)  f(n)  c2g(n) }
Intuitively: Set of all functions
that have the same rate of
growth as g(n).
g(n) is an asymptotically tight bound for f(n).
22
Q-notation
For function g(n), we define Q(g(n)),
big-Theta of n, as the set:
Q(g(n)) = {f(n) :
 positive constants c1, c2, n0,
such that n  n0,
we have c1g(n)  f(n)  c2g(n) }
Technically, f(n)  Q(g(n)).
Older usage, f(n) = Q(g(n)).
I’ll accept either…
g(n) is an asymptotically tight bound for f(n).
23
Q-notation
For function g(n), we define Q(g(n)),
big-Theta of n, as the set:
Q(g(n)) = {f(n) :
 positive constants c1, c2, n0,
such that n  n0,
we have c1g(n)  f(n)  c2g(n) }
If f(n)  Q(g(n))
f ( n)
 ?
Then lim
n  g ( n)
(0, ∞, or a constant )
24
Example
Q(g(n)) = {f(n) :  positive constants c1, c2, and n0, such
that n  n0, c1g(n)  f(n)  c2g(n)}
For f(n) = 10n2 - 3n  Q(n2)
• What constants for n0, c1, and c2 will work?
• Make c1 a little smaller than the leading
coefficient, and c2 a little bigger.
• To compare orders of growth, look at the leading
term.
Exercise: Prove that n2/2-3n  Q(n2)
25
Example
Q(g(n)) = {f(n) :  positive constants c1, c2, and n0, such
that n  n0, c1g(n)  f(n)  c2g(n)}
• Does n5  Q(n4)?
• Does 300n3  Q(n4)?
• How about 22n Q(2n)?
• Does lg(n!)  Q(n lg n)?
– Proof using Stirling’s approximation (in the text) for
lg(n!).
26
O-notationf(n) = Q(g(n))  f(n) = O(g(n)).
For function g(n), we define O(g(n)),
big-O of n, as the set:
Q(g(n))  O(g(n)).
O(g(n)) = {f(n):
 positive constants c and n0, such
that n  n0,
we have 0  f(n)  cg(n) }
Intuitively: Set of all functions f
whose rate of growth is the same
as or lower than that of g(n).
O(g(n)) is an asymptotic upper bound for f(n).
f ( n)
lim
?
n  g ( n )
27
Examples
O(g(n)) = {f(n) :  positive constants c and n0, such
that n  n0, we have 0  f(n)  cg(n) }
• Any linear function an + b is in O(n2). How?
• Show that 3n3  O(n4) for appropriate c and n0.
28
W -notation (omega)
For function g(n), we define W(g(n)),
big-Omega of n, as the set:
f(n) = Q(g(n))  f(n) = W(g(n)).
Q(g(n))  W(g(n)).
W(g(n)) = {f(n) :
 positive constants c and n0, such
that n  n0,
we have 0  cg(n)  f(n)}
Intuitively: Set of all functions
whose rate of growth is the same
as or higher than that of g(n).
W(g(n)) is an asymptotic lower bound for f(n).
f ( n)
lim
?
n  g ( n )
29
Example
W(g(n)) = {f(n) :  positive constants c and n0, such
that n  n0, we have 0  cg(n)  f(n)}
• n  W(lg n). Choose c and n0.
30
Relations between Q, O, W
31
Relations between Q, W, O
Theorem: For any two functions g(n) and f(n),
f(n)  Q(g(n))
iff
f(n)  O(g(n)) and f(n)  W(g(n)).
• Q(g(n)) = O(g(n))  W(g(n))
• In practice, asymptotically tight bounds are obtained
from asymptotic upper and lower bounds.
32
Example
• Insertion sort takes Q(n2) in the worst case, so sorting (as
a problem) is O(n2). Why?
• Any sort algorithm must look at each item, so sorting is
W(n).
• In fact, using (e.g.) merge sort, sorting is Q(n lg n) in the
worst case.
– No comparison sort to do better in the worst case. [We may
not see a proof of this result in this course.]
33
Asymptotic notation in equations
• Can use asymptotic notation in equations to
replace expressions containing lower-order
terms.
• For example,
4n3 + 3n2 + 2n + 1 = 4n3 + 3n2 + Q(n)
= 4n3 + Q(n2) = Q(n3). How do we interpret this?
• In equations, Q(f(n)) always stands for an
anonymous function g(n)  Q(f(n))
– In the example above, Q(n2) stands for
3n2 + 2n + 1.
34
o-notation
For a given function g(n), the set little-o:
o(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  f(n) < cg(n)}.
f(n) becomes insignificant relative to g(n) as n
approaches infinity:
lim [f(n) / g(n)] = 0.
n
g(n) is an upper bound for f(n) that is not
asymptotically tight.
Observe the difference in this definition from
previous ones. Why?
35
w –notation
(little-omega)
For a given function g(n), the set little-omega:
w(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  cg(n) < f(n)}.
f(n) becomes arbitrarily large relative to g(n) as n
approaches infinity:
lim [f(n) / g(n)] = .
n
g(n) is a lower bound for f(n) that is not asymptotically tight.
36
Limits
• lim [f(n) / g(n)] = 0  f(n)  o(g(n))
n
• lim [f(n) / g(n)] <   f(n)  O(g(n))
n
• 0 < lim [f(n) / g(n)] <   f(n)  Q(g(n))
n
• 0 < lim [f(n) / g(n)]  f(n)  W(g(n))
n
• lim [f(n) / g(n)] =   f(n)  w(g(n))
n
• lim [f(n) / g(n)] undefined can not say
n
37
Relationships
• Subset relations between order-of-growth sets.
RR
O( f )
W( f )
•f
o( f )
Q( f )
w( f )
38
Properties
• Transitivity
–
–
–
–
–
f(n) = Q(g(n)) & g(n) = Q(h(n))  f(n) = Q(h(n))
f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))
f(n) = W(g(n)) & g(n) = W(h(n))  f(n) = W(h(n))
f(n) = o (g(n)) & g(n) = o (h(n))  f(n) = o (h(n))
f(n) = w(g(n)) & g(n) = w(h(n))  f(n) = w(h(n))
• Reflexivity
– f(n) = Q(f(n))
– f(n) = O(f(n))
– f(n) = W(f(n))
39
Properties
• Symmetry
f(n) = Q(g(n)) iff g(n) = Q(f(n))
• Complementarity
f(n) = O(g(n)) iff g(n) = W(f(n))
f(n) = o(g(n)) iff g(n) = w((f(n))
40
Common functions
Exponentials
• Useful (and elementary)
Identities:
1
a 
a
( a m ) n  a mn
1
• Exponentials and
polynomials
a m a n  a m n
nb
lim n  0
n  a
 n b  o( a n )
43
Logarithms
x = logba is the
exponent for a =
a  b logb a
bx.
log c (ab)  log c a  log c b
log b a  n log b a
n
Natural log: ln a = logea
Binary log: lg a = log2a
lg2a
a)2
= (lg
lg lg a = lg (lg a)
log c a
log b a 
log c b
log b (1 / a )   log b a
1
log b a 
log a b
a logb c  c logb a
44
Logarithms and exponentials –
Bases
• If the base of a logarithm is changed from one
constant to another, the value is altered by a constant
factor.
– Example: log2 n = log10 n * log210
– Base of logarithm is not an issue in asymptotic notation.
• Exponentials with different bases differ by a
exponential factor (not a constant factor).
– Example: 2n = (2/3)n *3n.
45
Exercise
Express functions in A in asymptotic notation using functions in B.
A
5n2 + 100n
B
3n 2 + 2
A  Q(B)
A  Q(n2), n2  Q(B)  A  Q(B)
log3(n2)
log2(n3)
A  Q(B)
logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
lg4
lg n
A  w(B)
n
3
alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3)   as n
lg2n
n1/2
A  o(B)
lim ( lga n / nb ) = 0 (here a = 2 and b = 1/2)  A  o(B)
n
46
Wrap-up
• What are the different asymptotic bounds on
functions?
• How are the asymptotic bounds related?
• Asymptotic bounds and algorithmic efficiency
47
Procedure mergesort (L=a1, a2, ….an)
if n>1 then
m:= n/2
L1 = a1, a2, …. am
L2 = am+1, …. an
L = merge (mergesort(L1), mergesort (L2))
Recursive proof
 Correctness
 Complexity
48