Introduction to Computational Complexity
C Patvardhan
Department of Electrical Engineering
Dayalbagh Educational Institute
Comparing Functions
Big Oh and Ω
Notes on Notation
N = {0,1,2,3, ... }
N+ = {1,2,3,4, ... }
R = Set of Reals
R+ = Set of Positive Reals
R* = R+ U {0}
C Patvardhan, Dayalbagh Educational Institute, Agra
3
English Interpretations
O(f) –
Set of functions that grow no faster
W(f) –
Set of functions that grow no slower
than f
than f
C Patvardhan, Dayalbagh Educational Institute, Agra
4
Asymptotic Notation
• By now you should have an intuitive feel for
asymptotic (big-O) notation:
– What does O(n) running time mean? O(n2)? O(n lg
n)?
– How does asymptotic running time relate to
asymptotic memory usage?
• Our first task is to define this notation more formally
and completely
O(n) -> Comparing f(n) and g(n)
Let f be a function from N to R.
O(f) (Big Oh of f) is the set of all functions g
from N to R for which:
1. There exists a real number c>0
2. AND there exists an n0 in N
Such that: g(n) ≤ cf(n) whenever n ≥ n0
C Patvardhan, Dayalbagh Educational Institute, Agra
6
Big Omega
Let f be a function from N to R.
W(f) (Big W of f) is the set of all functions g from
N to R such that:
1. There exists a real number c>0
2. AND there exists an n0 in N
Such that: g(n)  cf(n) whenever n  n0
C Patvardhan, Dayalbagh Educational Institute, Agra
7
Notation and Pronunciation
Proper Notation: g O(f) (because of the
set-theoretic definition in the previous slide).
Also seen: g = O(f)
Pronounced as “g is oh of f”
C Patvardhan, Dayalbagh Educational Institute, Agra
8
Big Oh
• Consider
– 1000n and 200n2
– What are the values of c and n0 ?
C Patvardhan, Dayalbagh Educational Institute, Agra
9
Efficiency of algorithms
• What is an efficient algorithm?
• How do we measure efficiency?
• but first …
C Patvardhan, Dayalbagh Educational Institute, Agra
10
Analysis of Algorithms
• Analysis is performed with respect to a computational model
• We will usually use a generic uniprocessor random-access
machine (RAM)
– All memory equally expensive to access
– No concurrent operations
– All reasonable instructions take unit time
• Except, of course, function calls
– Constant word size
• Unless we are explicitly manipulating bits
Measuring Computational Complexity
• Consider the following algorithm for adding two m  n
matrices A and B with coefficients a( , ) and b( , ).
begin
for i = 1 to m do
for j = 1 to n do c(i,j) := a(i,j) + b(i,j)
end
What is the running time of this algorithm?
• Let’s measure it as precisely as we can as a function of
n and m.
• Is it 2nm, or 3nm, or what? How do we measure the
running time?
• What are the basic steps that we should count?
C Patvardhan, Dayalbagh Educational Institute, Agra
12
Compute the running time precisely
Operation Number (as a function of m,n)
Additions
Assignments
Comparisons
Is it possible to do such precise analysis?
Do we need this nitty-gritty?
Or are we happy with the big picture?
Turns out that such detailed analysis is not necessary at all.
C Patvardhan, Dayalbagh Educational Institute, Agra
13
Towards Computational Complexity
•
We shall count number of steps in the worst case or most
pessimistic analysis (performance guarantees).
•
But how do we get over the variety of operations?
•
All arithmetic operations are assumed to take one step (or a
number of steps that is bounded by a constant).
•
We will ignore running time constants.
•
Our running times will be stated in terms of relevant
problem parameters, e.g., nm.
C Patvardhan, Dayalbagh Educational Institute, Agra
14
A Simpler Metric for Running Time
• Operation Number (as a function of m,n)
• Additions  c1 mn for some c1 and m, n  1 - O(mn)
steps
• Assignments  c2 mn for some c2 and m, n  1 O(mn) steps
• Comparisons  c3 mn for some c3 and m, n  1 O(mn) steps
• TOTAL  c4 mn for some c4 and m, n  1
– O(mn) steps
C Patvardhan, Dayalbagh Educational Institute, Agra
15
Simplifying Assumptions and Notation
MACHINE MODEL: Random Access Machine (RAM).
• This is the computer model that everyone is used
to.
• It allows the use of arrays, and it can select any
element of an array or matrix in O(1) steps.
– c(i,j) := a(i,j) + b(i,j).
• Integrality Assumption. All numbers are integral
(unless stated otherwise.)
C Patvardhan, Dayalbagh Educational Institute, Agra
16
Asymptotic Performance
• In this course, we care most about asymptotic
performance
– How does the algorithm behave as the problem size
gets very large?
• Running time
• Memory/storage requirements
• Bandwidth/power requirements/logic gates/etc.
Time Complexity
• The time complexity is the number of operations an
algorithm performs to complete its task (considering
that each operation takes the same amount of time).
• The algorithm that performs the task in the smallest
number of operations is considered the most efficient
one in terms of the time complexity.
• Time taken to execute a program is also affected
by factors such as your operating system and
hardware, but we are not including them in this
discussion.
Time Complexity Analysis
• Best Case analysis
• Average case analysis
• Worst case analysis (more popular)
– Determines how long the algorithm will take in the worst case.
C Patvardhan, Dayalbagh Educational Institute, Agra
19
Worst case
Always consider the running time of an algorithm in the worst
case.
Motivation:
• Often worst-case running time and typical running time
are very close.
• Higher reliability.
• Average-case analysis requires to know the probability
distribution of occurrence of the inputs.
But sometimes algorithms that are bad in the worst case are
good in practice e.g. Simplex algorithm for LP.
Reason is that there are some “weird” pathological cases in
which the algorithm does not perform well. But in
“practical” cases it performs well.
C Patvardhan, Dayalbagh Educational Institute, Agra
20
Sequential v/s Binary Search
if we have 4 billion elements to search
for, then, in its worst case, linear
search will take 4 billion operations to
complete its task.
Binary search will complete this task in
just 32 operations. That’s a big
difference.
Now let’s assume that if one operation
takes 1 ms for completion, then binary
search will take only 32 ms whereas
linear search will take 4 billion ms
(that is approx. 46 days). That’s a
significant difference.
Time Complexity of algorithms for some
common problems
•
•
•
•
•
•
•
•
Accessing nth term in an array O(1)
Binary Search O(log n)
Sequential Search O(n)
Merge Sort, Heap Sort O(nlogn)
Matrix Addition O(n2)
Matrix Multiplication (naïve algorithm) O(n3)
Subset enumeration O(2n)
TSP (Brute Force approach) O(n!)
This is the reason why studying time complexity becomes
important when it comes to such a big amount of data.
C Patvardhan, Dayalbagh Educational Institute, Agra
22
Other ways of Time Complexity
Analysis
• Amortized analysis
– There is an algorithm that has linear time average case
complexity.
– In practice this algorithm is run with some other algorithms
whose worst-case complexity is constant time.
– The slower one is used much less frequently than the faster
ones.
– So the total amortized time over the use of all the algorithms
is constant time per run.
C Patvardhan, Dayalbagh Educational Institute, Agra
23
Polynomial Time Algorithms
• We say that an algorithm runs in polynomial
time if the number of steps taken by an
algorithm on any instance I is bounded by a
polynomial in the size of I.
• We say that an algorithm runs in exponential
time if it takes number of steps that is
exponential in n.
• Example 1: Two matrices of size n x n may be
multiplied in O(n3) steps.
– This is polynomial time.
C Patvardhan, Dayalbagh Educational Institute, Agra
24
On polynomial vs exponential time
• We contrast two algorithms, one that takes 30,000 n3 steps,
and one that takes 2n steps.
• Suppose that we could carry out 1 billion steps per second.
• # of nodes 30,000 n3 steps 2n steps
n = 30
0.81 seconds
1 second
n = 40
1.92 seconds
17 minutes
n = 50
3.75 seconds
12 days
n = 60
6.48 seconds
31 years
• Suppose we now have a computer that can perform 2 billion
steps per second?
C Patvardhan, Dayalbagh Educational Institute, Agra
25
function
/
n
10
20
50
100
300
Polynomial
n2
1/10,000
second
1/2,500
second
1/400
second
1/100
second
9/100
second
n5
1/10
second
3.2
seconds
5.2
minutes
2.8
hours
28.1
days
Exponential
On polynomial vs exponential time (2)
2n
1/1000
second
1
second
35.7
years
400 trillion
centuries
a 75 digitnumber of
centuries
nn
2.8
hours
3.3 trillion
years
a 70 digitnumber of
centuries
a 185 digitnumber of
centuries
a 728 digitnumber of
centuries
C Patvardhan, Dayalbagh Educational Institute, Agra
26
Polynomial time and efficiency
• Whenever an algorithm runs in O (nc ) time, where n is the
size of the input and c is a constant, we say that the
algorithm is efficient.
• We want to find polynomial-time algorithms for every
interesting problem, and with the smallest exponent.
• An algorithm running in O (n log n ) time is always
preferable to an O (n2 ) algorithm for all but finitely many
instances.
C Patvardhan, Dayalbagh Educational Institute, Agra
27
Reasonable vs. Unreasonable
• ”Good”, reasonable algorithms
– algorithms bound by a polynomial
function nk
– Tractable problems
• ”Bad”, unreasonable algorithms
– algorithms whose running time is above
nk
– Intractable problems
problems not admitting
intractable
problems
tractable
problems
reasonable algorithms
problems admitting reasonable
(polynomial-time) algorithms
C Patvardhan, Dayalbagh Educational Institute, Agra
28
Traveling Salesperson Problem
• A traveling salesperson needs to visit n cities.
• Is there a route of at most d length? (decision
problem)
• Optimization-version asks to find a shortest
cycle visiting all vertices once in a weighted
graph
C Patvardhan, Dayalbagh Educational Institute, Agra
29
Traveling Salesperson Problem
• A traveling salesperson needs to visit n cities.
• Is there a route of at most d length? (decision
problem)
• Optimization-version asks to find a shortest
cycle visiting all vertices once in a weighted
graph
C Patvardhan, Dayalbagh Educational Institute, Agra
30
TSP Algorithms
• Naive solutions take n! time in the worst-case,
where n is the number of vertices of the graph
• No polynomial-time algorithms are known
– TSP is an NP problem
• Longest Path problem between A and B in a
weighted graph is also NP.
C Patvardhan, Dayalbagh Educational Institute, Agra
31
Approximation Algorithms
• Used when optimality is not obtainable and nearoptimality is acceptable.
• Approximation algorithms produce results with a
guarantee
– Guarantee that the result would never be
worse than a lower bound determined by the
performance ratio of the algorithm.
– Performance ratio γ of an algorithm is defined
as φ/φ*, where φ is the solution produced by
the algorithm and φ* is the optimal solution to
the problem. Algorithms with γ close to 1 have
been developed.
C Patvardhan, Dayalbagh Educational Institute, Agra
32
Heuristics
• Do produce solutions but do not guarantee the
optimality of the solution.
• Have to be tested on various benchmarks to
verify their effectiveness.
• Bulk of research on VLSI physical design has
concentrated on finding heuristic algorithms.
• Focus on practical algorithms that can produce
close to optimal solutions on real world problems.
• There are general purpose heuristics applicable
to a large variety of problems and there are also
specific problem heuristics.
C Patvardhan, Dayalbagh Educational Institute, Agra
33
The End