CSCE350 Algorithms and Data Structure
Lecture 18
Jianjun Hu
Department of Computer Science and Engineering
University of South Carolina
2009.11.
Announcement
Thanksgiving…………
1) One final homework (for preparing final exam)
2) 60-70
Chapter 10: Limitations of Algorithm Power
Basic Asymptotic Efficiency Classes
1
constant
log n
logarithmic
n
linear
n log n
n log n
n2
quadratic
n3
cubic
2n
exponential
n!
factorial
Polynomial-Time Complexity
Polynomial-time complexity: the complexity of an algorithm is
ab n b  ab1n b1  ...  a1 n1  a0  (n b )
with a fixed degree b>0.
If a problem can be solved in polynomial time, it is usually
considered to be theoretically tractable in current computers.
When an algorithm’s complexity is larger than polynomial, i.e.,
exponential, theoretically it is considered to be too expensive
to be useful – intractable
This may not exactly reflect it’s usability in practice!
List of Problems
Sorting
Searching
Shortest paths in a graph
Minimum spanning tree
Primality testing
Traveling salesman problem
Knapsack problem
Chess
Towers of Hanoi …
Lower Bound
Problem A can be solved by algorithms a1, a2,…, ap
Problem B can be solved by algorithms b1, b2,…, bq
We may ask
• Which algorithm is more efficient? This makes more sense
when the compared algorithms solve the same problem
• Which problem is more complex? We may compare the
complexity of the best algorithm for A and the best algorithm
for B
For each problem, we want to know the lower bound: the best
possible algorithm’s efficiency for a problem  Ω(.)
Tight lower bound: we have found an algorithm of the this
lower-bound complexity
Trivial Lower Bound
Many problems need to ‘read’ all the necessary items and
write the ‘output’
 Their sizes provide a trivial lower bound
Example:
1. Generate all permutations of n distinct items  Ω(n!)
Why?
Is this a tight lower bound?
2. Evaluate the polynomial at a given x
an x n  an 1 x n 1  ...  a1 x  a0
 Ω(n), because of the input size. Is this tight?
Another Example
Multiplying two nxn matrices  Ω(n2)
• because we need to process 2n2 elements and output n2
elements
• We do not know whether this is tight
Many trivial lower bounds are too low to be useful
• TSP  Ω(n2) because its input is n(n-1)/2 intercity distance
and output is n+1 city in sequence
• There is no known polynomial-time algorithm to solve it
Trivial lower bound sometime have problems
• We do not need to process all the input elements
• For example: searching an element in a sorted array. What is
its complexity?
Information-Theoretic Arguments
This approach seeks to establish a lower bound based on the
amount of information it has to produce
Recall the problem of guess the number from 1..n by asking
‘yes/no’ questions
Fundamentally, it is a coding problem. If the input number can
be encoded into m bits, each ‘yes/no’ question just resolve
one bit and therefore, the lower bound is m
We know that m=log2n
We have also shown its relation to decision tree. We will apply
the decision tree to find the lower bound for several other
problems
Find the Smallest from three numbers using
comparison
Leaves in the decision tree is the possible output. The output
size is at least 3 (maybe larger than 3) here
Complexity: the height of this decision tree
Given l leaves, the height of the binary tree is at least
yes
yes
a
a< c
no
a<b
no
c
log 2 l 
yes
b
b<c
no
c
Decision Tree for Sorting Algorithms
The number of leaves: n!
The lower bound: log n!  n log2n. Is this tight?
abc
Selection Sort
yes
no
a<b
abc
yes
a< c
abc
yes
a<b<c
b< c
abc
no
yes
a<c<b
b<a
cba
bac
cba
no
no
b<c
no
c<a<b
yes
b<a<c
a< c
no
b<c<a
yes
c<b<a
b<a
Decision Tree for Searching a Sorted Array
Decision tree for binary search in a four-element array
# of leaves for n elements 2n+1 lower bound:
log 3 (2n  1)
 Ω(logn)
A[1]
<
A[0]
<
< A[0]
=
A[0]
>
(A[0], A[1] )
=
>
A[1]
A[2]
<
(A[1], A[2] )
=
>
A[2]
A[3]
<
(A[2], A[3] )
=
>
A[3]
> A[3]
P, NP, and NP-Complete Problems
As we discussed, problems that can be solved in polynomial
time are usually called tractable and the problems that cannot
be solved in polynomial time are called intractable, now
Is there a polynomial-time algorithm that solves the problem?
Possible answers:
• yes
• no
– because it can be proved that all algorithms take exponential
time
– because it can be proved that no algorithm exists at all to solve
this problem
• don’t know
• don’t know, but if such algorithm were to be found, then it
would provide a means of solving many other problems in
polynomial time
Types of problems
Optimization problem: construct a solution that maximizes or
minimizes some objective function
Decision problem: answer yes/no to a question
Many problems will have decision and optimization versions.
Eg: Traveling Salesman Problem
• optimization: find Hamiltonian cycle of minimum weight
• decision: Is there a Hamiltonian cycle of weight < k
Some More problems
Partition: Given n positive integers, determine whether it is possible
to partition them into two disjoint subsets with the same sum
Bin packing: given n items whose sizes are positive rational numbers
not larger than 1, put them into the smallest number of bins of size 1
Graph coloring: For a given graph find its chromatic number, i.e., the
smallest number of colors that need to be assigned to the graph’s
vertices so that no two adjacent vertices are assigned the same color
CNF satisfiability: Given a boolean expression in conjunctive normal
form (conjunction of disjunctions of literals), is there a truth
assignment to the variables that makes the expression true?
The class P
P: the class of decision problems that are solvable in O(p(n)),
where p(n) is a polynomial on n
Why polynomial?
if not, very inefficient
nice closure properties
machine independent in a strong sense
The class NP
NP: the class of decision problems that are solvable in
polynomial time on a nondeterministic machine
(A determinstic computer is what we know)
A nondeterministic computer is one that can “guess” the right
answer or solution
Thus NP can also be thought of as the class of problems
• whose solutions can be verified in polynomial time; or
• that can be solved in polynomial time on a machine that can
pursue infinitely many paths of the computation in parallel
Note that NP stands for “Nondeterministic Polynomial-time”
Non-deterministic computer
00
00
01
10
11
01
10
11
Example: Conjunctive Normal Form (CNF)
Satisfiability
This problem is in NP. Nondeterministic algorithm:
• Guess truth assignment
• Check assignment to see if it satisfies CNF formula
Example: (Boolean operation)
(a  b  c )  (a  b)  (a  b  c )
Truth assignments:
a  true, b  true, c  false 
the entire expression  true
Checking phase: Θ(n)
Where Are We Now?
Exhibited nondeterministic poly-time algorithm for CNFsatisfiability
CNF-sat is in NP
Similar algorithms can be found for TPS, HC, Partition, etc
proving that these problems are also in NP
All the problems in P can also be solved in this manner (but no
guessing is necessary), so we have:
P ⊆ NP
Big question: P = NP ?
NP-Complete problems
A decision problem D is NP-complete iff
1. D ∈ NP
2. every problem in NP is polynomial-time reducible to D
Cook’s theorem (1971): CNF-sat is NP-complete
Other NP-complete problems obtained through polynomialtime reductions of known NP-complete problems
The class of NP-complete problems is denoted NPC
Polynomial Reductions
A decision problem D1 is said to be polynomial reducible to a
decision problem D2 if there exists a function t that
transforms instances of D1 to instances of D2 such that
1. t maps all yes instances of D1 to yes instances of D2 and all
no instances of D1 to no instances of D2
2. t is computable by a polynomial-time algorithm
If D2 can be solved in polynomial time  D1 can be solved in
polynomial time
Polynomial Reductions
Example: Polynomial-time reduction of HC-Hamiltonian cycle
to decision version of TPS (TPS with total distance no larger
than k?) given integer distance
y
y
1
x
x
2
1
2
1
u
v
u
1
What does this prove?
• HC is harder or easier than TPS decision?
• HC is NPC  TPS(D) is NPC? or
• TPS(D) is NPC  HC is NPC?
v
To Prove a Decision Problem is in NPC
1. Prove it is in NP (verification takes polynomial time)
2. Prove that all problems in NP is reducible to this problem
or
3. Prove that a known NPC problem is reducible to this
problem
If we can prove any given NPC problem can be solve in
polynomial time  P=NP
For us, before solving a problem, we may check whether it is a
NPC problem. If it is, then …
Michael Garey and David Johnson: Computers and
Intractability - A Guide to the Theory of NP-completeness;
Freeman, 1979.