Lecture 15:
Complexity Notes
This is selected from CS200 lectures 16,
30, 37, 38 from
http://www.cs.virginia.edu/cs200/lectures/
CS588: Security and Privacy
University of Virginia
Computer Science
David Evans
http://www.cs.virginia.edu/evans
What does  really mean?
• O(x) – it is no more than x work
(upper bound)
• (x) – work scales as x (tight bound)
• (x) – it is at least x work
(lower bound)
If O(x) and (x) are true,
then (x) is true.
CS588 Spring 2005
2
Meaning of O (“big Oh”)
f(x) is O (g (x)) means:
There is a positive constant c
such that
c * f(x) < g(x)
for all but a finite number of x
values.
CS588 Spring 2005
3
O Examples
f(x) is O (g (x)) means:
There is a positive constant c such that
c * f(x) < g(x)
for all but a finite number of x values.
x is O (x2)?
10x is O (x)?
x2 is O (x)?
Yes, c = 1 works fine.
Yes, c = .09 works fine.
No, no matter what c we pick,
cx2 > x for big enough x
CS588 Spring 2005
4
Lower Bound:
 (Omega)
f(x) is  (g (x)) means:
There is a positive constant c
such that
c * f(x) > g(x)
for all but a finite number of x
values.
Difference from O – this was <
CS588 Spring 2005
5
Examples
• x is
 (x)
f(x) is  (g (x)) means:
There is a positive constant c such that
c * f(x) > g(x)
for all but a finite number of x values.
f(x) is O (g (x)) means:
There is a positive constant c such that
c * f(x) < g(x)
for all but a finite number of x values.
– Yes, pick c = 2
• 10x is
 (x)
– Yes, pick c = 1
• Is
x2
 (x)?
– Yes!
• x is O(x)
– Yes, pick c = .5
• 10x is O(x)
– Yes, pick c = .09
• x2 is not O(x)
CS588 Spring 2005
6
Tight Bound:
 (Theta)
f(x) is  (g (x)) iff:
f(x) is O (g (x))
and f(x) is  (g
(x))
CS588 Spring 2005
7
• 10x is
 Examples
 (x )
– Yes, since 10x is  (x) and 10x is O(x)
• Doesn’t matter that you choose different c
values for each part; they are independent
• x2 is/is not
 (x)?
– No, since x2 is not O (x)
• x is/is not  (x2)?
– No, since x2 is not  (x)
CS588 Spring 2005
8
Measuring Work
• When we say a procedure or a problem is
O(f(n)) what are we counting?
• Need a model computer to count steps
CS588 Spring 2005
9
How should we model a Computer?
Colossus (1944)
Cray-1 (1976)
Apollo Guidance
Computer (1969)
CS588 Spring 2005
IBM 5100 (1975)
10
Modeling Computers
• Input
– Without it, we can’t describe a problem
• Output
– Without it, we can’t get an answer
• Processing
– Need some way of getting from the input to
the output
• Memory
– Need to keep track of what we are doing
CS588 Spring 2005
11
Modeling Input
Punch Cards
Altair BASIC Paper Tape, 1976
Engelbart’s mouse and keypad
CS588 Spring 2005
12
Simplest Input
• Non-interactive: like punch cards and
paper tape
• One-dimensional: just a single tape of
values, pointer to one square on tape
0
0
1
1
0
0
1
0
0
0
How long should the tape be?
Infinitely long! We are modeling a computer, not
building one. Our model should not have silly
practical limitations (like a real computer does).
CS588 Spring 2005
13
Modeling Output
• Blinking lights are
cool, but hard to
model
• Output is what is
written on the
tape at the end of
a computation
Connection Machine CM-5, 1993
CS588 Spring 2005
14
Modeling Processing
• Evaluation Rules
– Given an input on our tape, how do we
evaluate to produce the output
• What do we need:
– Read what is on the tape at the current
square
– Move the tape one square in either direction
– Write into the current square
0
0
1
1
0
0
1
0
0
0
Is that CS588
enough
to model a computer?
Spring 2005
15
Modeling Processing
• Read, write and move is not enough
• We also need to keep track of what we
are doing:
– How do we know whether to read, write or
move at each step?
– How do we know when we’re done?
• What do we need for this?
CS588 Spring 2005
16
Finite State Machines
1
0
Start
2
1
#
0
1
HALT
CS588 Spring 2005
17
Hmmm…maybe we don’t need
those infinite tapes after all?
not a
paren
2
1
Start
)
#
ERRO
R
not a
paren
(
)
HALT
CS588 Spring 2005
What if the
next input symbol
is ( in state 2?
18
How many states do we need?
not a
paren
)
#
ERRO
R
not a
paren
2
1
Start
not a
paren
(
(
)
HALT
CS588 Spring 2005
not a
paren
3
)
(
)
4
19
(
Finite State Machine
• There are lots of things we can’t compute
with only a finite number of states
• Solutions:
– Infinite State Machine
• Hard to describe and draw
– Add a tape to the Finite State Machine
CS588 Spring 2005
20
FSM + Infinite Tape
• Start:
– FSM in Start State
– Input on Infinite Tape
– Pointer to start of input
• Move:
– Read one input symbol from tape
– Follow transition rule from current state
• To next state
• Write symbol on tape, and move L or R one square
• Finish:
– Transition to halt state
CS588 Spring 2005
21
Matching Parentheses
• Repeat until halt:
– Find the leftmost )
• If you don’t find one, the parentheses match,
write a 1 at the tape head and halt.
– Replace it with an X
– Look left for the first (
• If you find it, replace it with an X (they
matched)
• If you don’t find it, the parentheses didn’t
match – end write
0 2005
at the tape head and
CS588 a
Spring
halt
22
Matching Parentheses
(, (, R
X, X, R
1
Start
Input: )
Write: X
Move: L
(, X, R
X, X, L
2: look
for (
#, 0, #
#, 1, #
Will this report the
correct result for (()?
), X, L
HALT
CS588 Spring 2005
23
Matching Parentheses
(, (, R
X, X, L
), X, L
X, X, R
1
Start
(, X, R
2: look
for (
#, #, L
#, 1, #
X, X, L
3: look
for (
(, 0, #
#, 0, #
HALT
#, 1, #
CS588 Spring 2005
24
Turing Machine
• Alan Turing, On computable numbers: With
an application to the
Entscheidungsproblem, 1936
– Turing developed the machine abstraction to
show the halting problem really leads to a
contradiction
– Our informal argument, depended on assuming
we could do if and everything else except
halts?
CS588 Spring 2005
25
Describing Turing Machines
z z
z
z
z
z
z
z
), X, L
), #, R
(, #, L
2:
look
for (
1
Start
(, X, R
#, 1, -
HALT
#, 0, -
Finite State Machine
z
z
z
z
z
z
z
z z
z
z
TuringMachine ::= < Alphabet, Tape, FSM >
Alphabet ::= { Symbol* }
Tape ::= < LeftSide, Current, RightSide >
OneSquare ::= Symbol | #
Current ::= OneSquare
LeftSide ::= [ Square* ]
RightSide ::= [ Square* ]
Everything to left of LeftSide is #.
Everything to right of RightSide is #.
CS588 Spring 2005
26
z
), #, R
Start
#, 1, #
1
), X, L
(, X, R
HALT
(, #, L
2: look
for (
#, 0, #
Describing
Finite State
Machines
TuringMachine ::= < Alphabet, Tape, FSM >
FSM ::= < States, TransitionRules, InitialState, HaltingStates >
States ::= { StateName* }
InitialState ::= StateName
must be element of States
HaltingStates ::= { StateName* }
all must be elements of States
TransitionRules ::= { TransitionRule* }
TransitionRule ::=
< StateName, ;; Current State
Transition Rule is a procedure:
OneSquare, ;; Current square StateName X OneSquare
StateName, ;; Next State
 StateName X OneSquare X Direction
OneSquare, ;; Write on tape
Direction > ;; Move tape
CS588 Spring 2005
27
Direction ::= L, R, #
), #, R
1
Start
#, 1, #
), X, L
(, X, R
HALT
(, #, L
2: look
for (
#, 0, #
Example
Turing
Machine
TuringMachine ::= < Alphabet, Tape, FSM >
FSM ::= < States, TransitionRules, InitialState, HaltingStates >
Alphabet ::= { (, ), X }
States ::= { 1, 2, HALT }
InitialState ::= 1
HaltingStates ::= { HALT }
TransitionRules ::= { < 1, ), 2, X, L >,
< 1, #, HALT, 1, # >,
< 1, ), #, R >,
< 2, (, 1, X, R >,
< 2, #, HALT, 0, # >,
< 2, ), #, L >,}
CS588 Spring 2005
28
Enumerating Turing Machines
• Now that we’ve decided how to describe
Turing Machines, we can number them
• TM-5023582376 = balancing parens
• TM-57239683
= even number of 1s
• TM= Photomosaic Program
• TM= WindowsXP
3523796834721038296738259873
3672349872381692309875823987609823712347823
Not the real
numbers – they
would be much
bigger!
CS588 Spring 2005
29
Complexity Class P
Class P: problems that can be solved in
polynomial time by a deterministic TM.
O (nk) for some constant k.
Easy problems like sorting, making a
photomosaic using duplicate tiles,
simulating the universe are all in P.
CS588 Spring 2005
30
Deterministic Computing
• All our computing models so far are
deterministic: you know exactly what to
do every step
– TM: only one transition rule can match any
machine configuration, only one way to follow
that transition rule
– Lambda Calculus: always -reduce the
outermost redex
CS588 Spring 2005
31
Nondeterministic Computing
• Allows computer to try different options at
each step, and then use whichever one
works best
• Make a Finite State Machine nondeterministic: doesn’t change computing
time
• Make a Turing Machine non-deterministic:
might change computing time
• No one knows whether it does or not
• This is the biggest open problem in Computer
CS588 Spring 2005
32
Science
Making FSM’s Nondeterministic
0
0
0
1
1
2
1
3
#
HALT
CS588 Spring 2005
33
Nondeterministic FSM
0, 1
0
1
Input Read
1
2
3
Possible States
0
00
001
{ 1, 2 }
{ 1, 2 }
{ 1, 3 }
0010
{ 1, 2 }
00101
{ 1, 3 }
00101#
{ HALT }
#
HALT
CS588 Spring 2005
34
Power of NFSM
• Can NFSMs computing anything FSMs
cannot compute?
No – you can always convert an NFSM to a FSM by
making each possible set of states in the NFSM into one
state in the new FSM. (Number of states may increase as 2s)
• Can NFSMs compute faster than FSMs?
No – both read input one letter at a time, and transition
automatically to the next state. (Both are always (n) where
n is the number of symbols in the input.)
CS588 Spring 2005
35
Nondeterministic TMs
• Multiple transitions possible at every step
• Follow all possible steps:
– Each possible execution maintains its own
state and its own infinite tape
– If any possible execution halts (in an
accepting state), the NTM halts, and the tape
output of that execution is the NTM output
CS588 Spring 2005
36
Complexity Classes
Class P: problems that can be solved in
polynomial time by a deterministic TM.
O (nk) for some constant k.
Easy problems like simulating the universe are
all in P.
Class NP: problems that can be solved in
polynomial time by a nondeterministic TM
Hard problems like the pegboard puzzle
sorting are in NP (as well as all problems in
CS588 Spring 2005
37
P).
Problem Classes
Fill tape with 2n *s: (2n)
Simulating Universe: O(n3)
Undecidable
NP
P
Decidable
Halting Problem: ()
Sorting: (n log n)
Cracker Barrel: O(2n) and (n)
CS588 Spring 2005
38
P = NP?
• Is there a polynomial-time solution to the
“hardest” problems in NP?
• No one knows the answer!
• The most famous unsolved problem in
computer science and math
• Listed first on Millennium Prize Problems
– win $1M if you can solve it
– (also an automatic A+ in this course)
CS588 Spring 2005
39
If P  NP:
Fill tape with 2n *s: (2n)
Simulating Universe: O(n3)
Undecidable
NP
P
Decidable
Halting Problem: ()
Sorting: (n log n)
Cracker Barrel: O(2n) and (n)
CS588 Spring 2005
40
If P = NP:
Fill tape with 2n *s: (2n)
Simulating Universe: O(n3)
Undecidable
NP
P
Decidable
Halting Problem: ()
Sorting: (n log n)
Cracker Barrel: O(2n) and (n)
CS588 Spring 2005
41
Smileys Problem
Input: n square tiles
Output: Arrangement of the
tiles in a square, where the
colors and shapes match up,
or “no, its impossible”.
CS588 Spring 2005
42
Thanks to Peggy Reed for
making the Smiley Puzzles!
How much work is the
Smiley’s Problem?
• Upper bound: (O)
O (n!)
Try all possible permutations
• Lower bound: ()
 (n)
Must at least look at every tile
• Tight bound: ()
No one knows!
CS588 Spring 2005
44
NP Problems
• Can be solved by just trying all possible
answers until we find one that is right
• Easy to quickly check if an answer is right
– Checking an answer is in P
• The smileys problem is in NP
We can easily try n! different answers
We can quickly check if a guess is
correct (check all n tiles)
CS588 Spring 2005
45
Is the Smiley’s Problem in P?
No one knows!
We can’t find a O(nk) solution.
We can’t prove one doesn’t exist.
CS588 Spring 2005
46
This makes a huge difference!
1E+30
1E+28
time
since
“Big
Bang”
n!
1E+26
2n
1E+24
1E+22
1E+20
Solving a large smileys problem
either takes a few seconds, or
more time than the universe has
been in existence. But, no one
knows which for sure!
1E+18
1E+16
1E+14
1E+12
2032
1E+10
1E+08
today
1E+06
n2
n log n
10000
100
1
2
4
8
16
CS588 Spring 2005
32
64
128
log-log scale
47
Who cares about Smiley puzzles?
If we had a fast (polynomial time) procedure to solve
the smiley puzzle, we would also have a fast procedure
to solve the 3/stone/apple/tower puzzle:
3
CS588 Spring 2005
48
3SAT  Smiley




CS588 Spring 2005
Step 1: Transform
into smileys
Step 2: Solve (using
our fast smiley
puzzle solving
procedure)
Step 3: Invert
transform (back into
3SAT problem
49
The Real 3SAT Problem
(also can be quickly transformed
into the Smileys Puzzle)
CS588 Spring 2005
50
Propositional Grammar
Sentence ::= Clause
Sentence Rule: Evaluates to value of Clause
Clause ::= Clause1  Clause2
Or Rule: Evaluates to true if either clause is
true
Clause ::= Clause1  Clause2
And Rule: Evaluates to true iff both clauses
are true
CS588 Spring 2005
51
Propositional Grammar
Clause ::= Clause
Not Rule: Evaluates to the opposite value
of clause (true  false)
Clause ::= ( Clause )
Group Rule: Evaluates to value of clause.
Clause ::= Name
Name Rule: Evaluates to value associated
with Name.
CS588 Spring 2005
52
Proposition
Example
Sentence ::= Clause
Clause ::= Clause1  Clause2
Clause ::= Clause1  Clause2
Clause ::= Clause
Clause ::= ( Clause )
Clause ::= Name
(or)
(and)
(not)
a  (b  c)  b  c
CS588 Spring 2005
53
The Satisfiability Problem (SAT)
• Input: a sentence in propositional
grammar
• Output: Either a mapping from names
to values that satisfies the input
sentence or no way (meaning there
is no possible assignment that
satisfies the input sentence)
CS588 Spring 2005
54
SAT Example
Sentence ::= Clause
Clause ::= Clause1  Clause2
Clause ::= Clause1  Clause2
Clause ::= Clause
Clause ::= ( Clause )
Clause ::= Name
(or)
(and)
(not)
SAT (a  (b  c)  b  c)
 { a: true, b: false, c: true }
 { a: true, b: true, c: false }
SAT (a  a)
 no way
CS588 Spring 2005
55
The 3SAT Problem
• Input: a sentence in propositional
grammar, where each clause is a
disjunction of 3 names which may be
negated.
• Output: Either a mapping from names
to values that satisfies the input
sentence or no way (meaning there
is no possible assignment that satisfies
the input sentence)
CS588 Spring 2005
56
3SAT / SAT
Is 3SAT easier or harder than SAT?
It is definitely not harder than
SAT, since all 3SAT problems
are also SAT problems. Some
SAT problems are not 3SAT
problems.
CS588 Spring 2005
57
3SAT Example
Sentence ::= Clause
Clause ::= Clause1  Clause2
Clause ::= Clause1  Clause2
Clause ::= Clause
Clause ::= ( Clause )
Clause ::= Name
3SAT ( (a  b   c)
 (a   b  d)
 (a  b   d)
 (b   c  d ) )
 { a: true, b: false, c: false, d:
false}
CS588 Spring 2005
(or)
(and)
(not)
58
3SAT  Smiley
• Like 3/stone/apple/tower puzzle, we
can convert every 3SAT problem into
a Smiley Puzzle problem!
• Transformation is more complicated,
but still polynomial time.
• So, if we have a fast (P) solution to
Smiley Puzzle, we have a fast solution
to 3SAT also!
CS588 Spring 2005
59
NP Complete
• Cook and Levin proved that 3SAT was NPComplete (1971)
• A problem is NP-complete if it is as hard
as the hardest problem in NP
• If 3SAT can be transformed into a
different problem in polynomial time, than
that problem must also be NP-complete.
• Either all NP-complete problems are
tractable (in P) or none of them are!
CS588 Spring 2005
60
NP-Complete Problems
• Easy way to solve by trying all possible guesses
• If given the “yes” answer, quick (in P) way to
check if it is right
– Solution to puzzle (see if it looks right)
– Assignments of values to names (evaluate logical
proposition in linear time)
• If given the “no” answer, no quick way to check if
it is right
– No solution (can’t tell there isn’t one)
– No way (can’t tell there isn’t one)
CS588 Spring 2005
61
Traveling Salesperson Problem
– Input: a graph of cities and roads with
distance connecting them and a
minimum total distant
– Output: either a path that visits each
with a cost less than the minimum, or
“no”.
• If given a path, easy to check if it
visits every city with less than
minimum distance traveled
CS588 Spring 2005
62
Graph Coloring Problem
– Input: a graph of nodes with edges
connecting them and a minimum
number of colors
– Output: either a coloring of the nodes
such that no connected nodes have the
same color, or “no”.
If given a coloring, easy to check if it
no connected nodes have the same
color, and the number of colors used.
CS588 Spring 2005
63
Pegboard Problem
- Input: a configuration of n pegs on a
cracker barrel style pegboard
- Output: if there is a sequence of jumps
that leaves a single peg, output that
sequence of jumps. Otherwise, output
false.
If given the sequence of jumps, easy (O(n))
to check it is correct. If not, hard to know if
Proof that variant of this
there is a solution.
problem is NP-Complete is
attached to today’s notes.
CS588 Spring 2005
64
Minesweeper Consistency Problem
– Input: a position of n
squares in the game
Minesweeper
– Output: either a
assignment of bombs to
squares, or “no”.
• If given a bomb assignment, easy to
check if it is consistent.
CS588 Spring 2005
65
Drug Discovery Problem
– Input: a set of proteins,
a desired 3D shape
– Output: a sequence of
proteins that produces
the shape (or
impossible)
Caffeine
If given a sequence, easy (not really) to
check if sequence has the right shape.
Note: US Drug sales = $200B/year
CS588 Spring 2005
66
Complexity and Cryptography
• Next class
CS588 Spring 2005
67