CS 461 – Aug. 24
• Course overview
• Section 1.1
– Definition of finite automaton (FA)
– How to draw FAs
Overview
• What is Computational Theory?
– Not so concerned with specific HW/SW
– Search for general truths, models, imagination
Ex. What is possible vs. impossible?
• Facets
– Automata & Formal Languages (Ch. 1 – 3)
There are 3 important models of computation.
– Computability (Ch. 3 – 5)
“Some problems can’t be solved.”
– Complexity (Ch. 7)
“Some problems take too long to solve.”
Theory
• What is theory?
• You have already dealt with abstract ideas in
earlier courses
–
–
–
–
–
–
Difference between void, null, zero
Different kinds of zeros (int, float, positive, negative)
Infinity versus NaN
Correctness of an algorithm
Number of iterations in a loop
Does recursion really work?
Computation Models
• Purpose of a model is to study what computers
and programs can do.
• Models range from simple to complete:
– Model #1: Finite Automaton
– Model #2: Pushdown Automaton
– Model #3: Turing Machine
• Assume programming essentially boils down to
input  outputs.
• For simplicity / convenience,
– input = any binary string
– output = 0/1, T/F, accept/reject
Finite Automaton
• Our first model – Chapter 1
• Purpose of FA is to take (binary) input string and
either “accept” or “reject” it.
• The set of all strings accepted by an FA is called
the language of the FA.
• “Language” simply means a set of strings, e.g.
the set of all binary strings starting with “0”.
Example
• Start state
• Accept/happy state(s)
0
s1
1
s2
0, 1
– If we are in one of
these when input is
done, we accept
string.
• Transitions
• “finite” refers to # of
states. More on
limitations later.
Example
0
s1
s2
• The FA could also be
expressed in a table.
• The table tells us
where to go on each
possible input bit.
From
1
0, 1
s1
Input
0
s2
Input
1
s1
s2
s2
s2
Example
• Let’s try sample input
like 101.
0
s1
1
s2
0, 1
• Can you figure out the
language of this
machine?
– i.e. How do we get to
the happy state?
Example #2
0
s1
1
s2
s3
0
1
• What is language of this FA?
• Note: if we change which state is the accept
state, we’d have a different language!
0, 1
CS 44 – Aug. 26
• Finish section 1.1
– Definition of FA
– Examples
– Regular languages
– Union of 2 regular languages (handout)
Formal Definition
A finite automaton has 5 things:
1.
2.
3.
4.
5.
A set of states
An alphabet, for input
A start state
A set of accept states
Transition function: δ(state, input) = state
•
When we create/define an FA, must specify all
5 things. A drawing does a pretty good job.
Example
0
1
s1
s2
s3
0
1
0, 1


From state
Input 0
Input 1
s1
s2
s1
s2
s1
s3
s3
s3
s3
Let’s make FA’s
1.
2.
3.
4.
5.
L = bit strings that have exactly two 1’s
L = starts with 01
L = ends with 01
L = has an even number of 1’s
L = starts and ends with same symbol
*** Very good idea to give meaningful names to your
states.
Since L is a set… how would we create an FA for the
complement of L?
Regular Language
• A regular language is a set of strings accepted
by some FA.
• Examples
– Starting with 01; ending with 01, containing 01.
– Strings of length 2.
– {0, 11}
• Any finite set is regular. Infinite sets are more
interesting. Yet, # states always finite!
Operations on sets
• We can create new regular sets from existing
ones, rather than starting from scratch. 
• Binary operations
– Union
– Intersection
• Unary operations
– Complement
– Star: This is the concatenation of 0 or more
elements.
For example, if A = {0, 11}, then A* is { ε, 0, 11, 00,
1111, 011, 110, …}
• “Closure property”: you always get a regular set
when you use the above operations.
Union
• Book shows construction (see handout)
• We want to union two languages A1 and A2: M1
accepts A1 and M2 accepts A2. The combined
machine is called M.
• M pretends it’s running M1 and M2 at same time!
δ((r1,r2),a) = (δ1(r1,a),δ2(r2,a))
• # states increases because it’s the Cartesian
product of M1 and M2’s states.
• Next section will show easier way to do union. 
Wolf, Goat, Cabbage
• A man would like to cross a river with his wolf,
goat and head of cabbage.
• Can only ferry 1 of 3 items at a time. Plus:
– Leaving wolf & goat alone: wolf will eat goat.
– Leaving goat & cabbage alone: goat will eat
cabbage.
• Yes, we can solve this problem using an FA!
– Think about possible states and transitions.
CS 44 – Aug. 29
• Regular operations
– Union construction
• Section 1.2 - Nondeterminism
– 2 kinds of FAs
– How to trace input
– NFA design makes “union” operation easier
– Equivalence of NFAs and DFAs
Wolf, Goat, Cabbage
• A man would like to cross a river with his wolf,
goat and head of cabbage.
• Can only ferry 1 of 3 items at a time. Plus:
– Leaving wolf & goat alone: wolf will eat goat.
– Leaving goat & cabbage alone: goat will eat
cabbage.
• Yes, we can solve this problem using an FA!
– Think about possible states and transitions.
Operations on sets
• We can create new regular sets from existing
ones, rather than starting from scratch. 
• Binary operations
– Union
– Intersection
• Unary operations
– Complement
– Star: This is the concatenation of 0 or more
elements.
For example, if A = {0, 11}, then A* is { ε, 0, 11, 00,
1111, 011, 110, …}
• “Closure property”: you always get a regular set
when you use the above operations.
Union
• Book shows construction (see handout)
• We want to union two languages A1 and A2: M1
accepts A1 and M2 accepts A2. The combined
machine is called M.
• M pretends it’s running M1 and M2 at same time!
δ((r1,r2),a) = (δ1(r1,a),δ2(r2,a))
• # states increases because it’s the Cartesian
product of M1 and M2’s states.
• Next section will show easier way to do union. 
Union
• Book shows construction (see handout)
• We want to union two languages A1 and A2: M1
accepts A1 and M2 accepts A2. The combined
machine is called M.
• M pretends it’s running M1 and M2 at same time!
δ((r1,r2),a) = (δ1(r1,a),δ2(r2,a))
• # states increases because it’s the Cartesian
product of M1 and M2’s states.
• Next section will show easier way to do union. 
Non-determinism
• There are 2 kinds of FA’s
– DFA: deterministic FA
– NFA: non-deterministic FA
• NFA is like DFA except:
– A state may have any # of arrows per input symbol
– You can have ε-transitions. With this kind of transition,
you can go to another state “for free”.
• Non-determinism can make machine construction
more flexible. At first the theory seems more
complex, but NFA’s will come in handy.
Example
0, ε
1
s1
1
s2
s3
s4
0, 1
0, 1
From state
Input 0
Input 1
s1
s1
s1, s2
s2
s3
s3
s4
s3
s4
s4
Input ε
s4
continued
0, ε
1
s1
s2
1
s3
s4
0, 1
0, 1
• See the non-determinism?
• Remember, any time you reach a state that has
ε-transitions coming out, take it! It’s free.
• Let’s trace input 010110.
0, ε
1
s1
1
s2
s3
s4
0, 1
0, 1
Start
Read 0 Read 1 Read 0 Read 1 Read 1 Read 0
s1
s1
s1
s1
s1
s2
s3
s3
[dies]
s2
s3
s4
s1
s2
s3
[dies]
s1
s3
[dies]
s4
s4
Moral
• When tracing a word like 010110, we just
want to know if there is any way to get to
the accept state.
• Language is anything containing 11 or
101.
• Corresponding DFA would have more
states.
CS 461 – Aug. 31
Section 1.2 – Nondeterministic FAs
• How to trace input √
• NFA design makes “union” operation
easier
• Equivalence of NFAs and DFAs
NFA’s using “or”
ε
Old start 1
New start
ε
Old start 2
• Can you draw NFA for:
{ begin with 0 or end with 1 } ?
Amazing fact
• NFA = DFA
• In other words, the two kinds of machines have
the same power.
• Proof idea: we can always convert a DFA into
an NFA, or vice versa. Which do you think is
easier to do?
Formal NFA def’n
• The essential difference with DFA is in the
transition function:
DFA
δ: Q x Σ  Q
NFA
δ: Q x Σε  P(Q)
• Thus, converting DFA  NFA is easy. We
already satisfy the definition!
NFA  DFA construction
1. When creating DFA, states will be all possible
subsets of states from NFA.
–
–
–
This takes care of “all possible destinations.”
In practice we won’t need whole subset: only create
states as you need them.
“empty set” can be our dead state.
2. DFA start state = NFA’s start state or
anywhere you can begin for free. Happy state
will be any subset containing NFA’s happy
state.
3. Transitions: Please write as a table. Drawing
would be too cluttered. When finished, can
eliminate useless states.
Example #1
• NFA transition table
given to the right.
• DFA start state is {1, 3},
or more simply 13.
inputs
state
a
b
ε
1 
-
2
3
2,3 3
-
2
3
• DFA accept state would
be anything containing
1. Could be 1, 12, 13,
123, but we may not
need all these states.
1
-
-
continued
• The resulting DFA
could require 2n
states, but we should
only create states as
we need them.
inputs
state
a
b
ε
1 
-
2
3
2,3 3
-
2
3
1
Let’s begin:
If we’re in state 1 or 3, where do we go if we read an ‘a’ or a ‘b’?
δ(13, a) = 1, but we can get to 3 for free.
δ(13, b) = 2. We need to create a new state “2”.
Continue the construction by considering transitions from state 2.
-
-
answer
DFA
inputs
NFA
inputs
state
a
b
ε
1 
-
2
3
2,3 3
-
2
3
1
-
-
state
a
b
13 
13
2
2
23
3
23
123
3
123 
123
23
3
13




Notice that the DFA is in fact deterministic: it has exactly one destination per
transition. Also there is no column for ε.
Example #2
• NFA transition table
given to the right.
• DFA start state is A.
• DFA accept state would
be anything containing
D.
inputs
State
0
1
ε
A
A
A,C
-
B
D
-
C
C
D
B
B
D
-
continued
Let’s begin.
δ(A, 0) = A
δ(A, 1) = AC
We need new state AC.
δ(AC, 0) = A
δ(AC, 1) = ABC
Continue from ABC…
inputs
State
0
1
ε
A
A
A,C
-
B
D
-
C
C
D
B
B
D
-
answer
NFA
DFA
inputs
inputs
State
0
1
ε
State
0
1
A
A
A,C
-
A
A
AC
B
D
-
C
AC
A
ABC
C
D
B
B
D
-
ABC
AD
ABC
AD 
ABC
ACD
ACD 
ABC ABCD
ABCD  ABCD ABCD
final thoughts
• NFAs and DFAs have same computational
power.
• NFAs often have fewer states than
corresponding DFA.
• Typically, we want to design a DFA, but NFAs
are good for combining 2+ DFAs.
• After doing NFA  DFA construction, we may
see that some states can be combined.
– Later in chapter, we’ll see how to simplify FAs.
CS 461 – Sept. 2
• Review NFA  DFA
• Combining FAs to create new languages
– union, intersection, concatenation, star
– We can basically understand how these work by
drawing a picture.
• Section 1.3 – Regular expressions
– A compact way to define a regular set, rather than
drawing an FA or writing transition table.
Example #2
• NFA transition table
given to the right.
• DFA start state is A.
• DFA accept state would
be anything containing
D.
inputs
State
0
1
ε
A
A
A,C
-
B
D
-
C
C
D
B
B
D
-
continued
Let’s begin.
δ(A, 0) = A
δ(A, 1) = AC
We need new state AC.
δ(AC, 0) = A
δ(AC, 1) = ABC
Continue from ABC…
inputs
State
0
1
ε
A
A
A,C
-
B
D
-
C
C
D
B
B
D
-
answer
NFA
DFA
inputs
inputs
State
0
1
ε
State
0
1
A
A
A,C
-
A
A
AC
B
D
-
C
AC
A
ABC
C
D
B
B
D
-
ABC
AD
ABC
AD 
ABC
ACD
ACD 
ABC ABCD
ABCD  ABCD ABCD
Moral
• NFAs and DFAs have same computational
power.
• NFAs often have fewer states than
corresponding DFA.
• Typically, we want to design a DFA, but NFAs
are good for combining 2+ DFAs.
• After doing NFA  DFA construction, we may
see that some states can be combined.
– Later in chapter, we’ll see how to simplify FAs.
U and ∩
• Suppose M1 and M2 are DFAs. We want to
combine their languages.
• Union: We create new start state. √
– Do you understand formalism p. 60 ?
• How can we also do intersection?
Hint: A ∩ B = (A’ U B’)’
Concatenation
• Concat: For each happy state in M1, turn it into
a reject state and add ε-trans to M2’s start.
• Example
L1 = { does not contain 00 }
L2 = { has even # of 1’s }
Let’s draw NFA for L1L2.
• Let’s decipher formal definition of δ on p. 61.
Star
• We want to concat the language with itself 0+
times.
• Create new start state, and make it happy.
• Add ε-transitions from other happy states to the
start state.
• Example
L = { begins with 1 and ends with 0 }
Let’s draw NFA for L*.
• Formal definition of δ on p. 63.
Regular expression
• A concise way to describe a language
– Text representation, straightforward to input into
computer programs.
• Use alphabet symbols along with operators
+ means “or”
* means repetition
Concatenation
• There is no “and” or complement.
Examples
• What do you think these regular expressions
mean?
0* + 1*
0*1*
00*11*
(a shorthand would be 0+1+ )
(0 + 1)*
• What’s the difference between 10*1 and
1(0+1)*1 ? Does this mean “anything with 1 at
beginning and end?”
Practice
• Words with 2 or more 0’s. What’s wrong with
this answer: 1*01*01 ?
• Words containing substring 110.
• Every even numbered symbol is 0.
– What’s wrong with: ((0 + 1)*0)* ?
• Words of even length.
• The last 2 symbols are the same.
• What is the shortest word not in: 1*(01)*0* ?
• True or false: (111*) = (11 + 111)*
CS 461 – Sept. 7
• Section 1.3 – Regular expressions
Examples
Equivalent to FA’s.
regular expression  FA
(straightforward)
FA  regular expression
(* major skill)
Regular expression
• A concise way to describe a language
– Text representation, straightforward to input into
computer programs.
• Use alphabet symbols along with operators
+ means “or”
* means repetition
Concatenation
• There is no “and” or complement.
Examples
• What do you think these regular expressions
mean?
0* + 1*
0*1*
00*11*
(a shorthand would be 0+1+ )
(0 + 1)*
• What’s the difference between 10*1 and
1(0+1)*1 ? Does this mean “anything with 1 at
beginning and end?”
Practice
• Words with 2 or more 0’s. What’s wrong with
this answer: 1*01*01 ?
• Words containing substring 110.
• Every even numbered symbol is 0.
– What’s wrong with: ((0 + 1)*0)* ?
• Words of even length.
• The last 2 symbols are the same.
• What is the shortest word not in: 1*(01)*0* ?
• True or false: (111*) = (11 + 111)*
Conversions
• It’s useful to be able to convert among:
– English description of language
– FA drawing
– Regular expression
• Also good practice to consider complement of a
language.
• Let’s practice writing regular expressions.
– General technique: think about how to get to happy
state. Sometimes difficult, so we’ll look at general
algorithm later.
Reg exp  FA
• Just build the FA in pieces.
– We already know how to combine FA’s using union,
concatenation, star – these are the basic operations
in regular expressions. 
– You may even be able to construct the FA intuitively.
• Let’s try examples
– 0*10
– 0* + 1*
FA  reg exp
• If the FA doesn’t have too many “loops,” it’s not hard to
write down the regular expression.
• Try this one. Also think about what would happen if a
different state is the accept state.
1
s1
1
s2
0
s3
0
s4
0, 1
0, 1
In other words


From state
Input 0
Input 1
s1
s4
s2
s2
s2
s3
s3
s4
s4
s4
s4
s4
1
s1
1
s2
0
s3
0
s4
0, 1
0, 1
CS 461 – Sept. 9
• Section 1.3 – Regular expressions
Examples 
Equivalent to FA’s. 
regular expression  FA 
(straightforward)
FA  regular expression
(* major skill)
This one is more ambitious…
Think about L and L’.
0, 1
0
1
0

1
0
0, 1
0, 1
0
1
1
State
0
1

s1
s2
s2

s2
s3
s7

s3
s7
s4

s4
s5
s7

s5
s6
s6

s6
s1
s7
s7
s7
s7
Need help
• Often an FA is too complex to just figure out the
language just by looking at it.
• Need a general procedure!
0
s1
01* (01*0) ?????
s2
Nope!
Actually, it should be
0
1*0(1* + 01*0)*
1
1
State
0
1

s1
s2
s1

s2
s1
s2
Procedure
• Step-by-step transformation of FA, taking away
one state at a time until we just have one start
and one happy.
• We’ll label edges with regular expressions.
– So our FA will be a “generalized” NFA
• Step 1: Add a new start and new happy state.
• Step 2: For each of the other states:
– Pick one to be ripped out.
– Recalculate edges (except to start or from happy)
taking this removal into account.
Basic idea
Suppose we are getting rid of state X.
To go from A to B, you can either get their directly,
or via X. We would label edge from A to B as:
δ(A,B) = δ(A,B) + δ(A,X) δ(X,X)* δ(X,B)
Notes:
1. If X has no edge to itself,
Just say ε.
2. If no edge AX or XB,
we say Ø and thus
no need to recalculate AB.
X
A
B
Example
0
• First create new start
and happy states.
s1
s2
0
• New machine below…
1
0
ε
start
1
s1
ε
s2
end
0
1
1
In other words
0
State
0
1

s1
s2
s1

s2
s1
s2
s1
s2
0
1
ε
start
ε
0
s1
s2
end
0
1
1
State


0
1
start
ε
s1
s1
s2
s1
s2
s1
s2
end
1
end
Get rid of s1
0
ε
start
s1
ε
s2
end
0
1
1
???
start
ε
s2
end
???
• Need to compute δ(start,s2) and δ(s2,s2).
δ(start,s2) = Ø + ε1*0, which simplifies to 1*0
δ(s2, s2) = 1 + 01*0
In other words
State

0
1
start

ε
s1
s1
s2
s1
s2
s1
s2
end
end
Becomes:

State
1*0
start
s2
s2

end
1+01*0
ε
s2
end
Get rid of s2
ε
1*0
s2
start
end
1 + 01*0
start
end
???
• Calculate δ(start, end) = Ø + 1*0 (1+01*0)* ε
which simplifies to 1*0 (1 + 01*0)*.
Example #2
b
1
2
a
a, b
• Create new start & happy states.
• What happens when we get rid of state 1?
b
start
ε
1
2
ε
end
a
a, b
In other words
b
1
2
a
a, b
State

a
ε
b
start

1
1
1
2
2
2
2
end
end
b
start
ε
1
2
ε
end
a
a, b
Get rid of state 1
δ(start, 2) = Ø + ε a*b = a*b
δ(start, ) = Ø + ε a* Ø = Ø
δ(2, 2) = a + b + Ø… = a + b
δ(2, ) = ε
b
start
ε
1
2
ε
end
a
a, b
continued
a*b
start
2
ε
end
a+b
Finally we can get rid of state 2 to obtain:
a*b (a + b)*
CS 461 – Sept. 12
• Simplifying FA’s.
– Let’s explore the meaning of “state”. After all if we
need to simplify, this means we have too many states.
– Myhill-Nerode theorem
– Handout on simplifying
What is a state?
Example: 0*10*


need 1
1
0
state = 0*
1
0
state = 0*10*
0, 1
state = 0*10*1 (0 + 1)*
• No matter what next input symbol is, all words in
same state act alike.
x, y same state  z, xz same state as yz
• A state is a collection of words that react to input
in the same way.
Equivalent states
odd
1
0
0


even
1
1
0
0, 1
• Whether we’ve seen even or odd number of 0’s shouldn’t
matter. Only concern is ultimate outcome: will string be
accepted or not?
• Words x and y should be in same state if z, xz and yz
have the same outcome.
In other words
• The 2 machines are equivalent.
From state
Input 0
Input 1

Need 1
Need 1
Good

Good
Good
Bad
Bad
Bad
Bad
From state
Input 0
Input 1
Even
Odd
Good
Odd
Even
Good
Good
Good
Bad
Bad
Bad
Bad


Myhill-Nerode theorem
•
•
Basis for simplification algorithm.
Also gives condition for a set to be regular.
–
i.e. infinite # states not allowed.
3 parts to theorem:
1.
2.
3.
For any language L, we have equivalence relation R:
xRy if z, xz and yz same outcome.
If L is regular, # of equivalences classes is finite.
If # equivalences classes finite, language is regular.
Proof (1)
For any language L, we have equivalence relation R: xRy
if z, xz and yz same outcome.
•
To show a relation is an equivalence relation, must
show it is reflexive, symmetric and transitive.
•
Reflexive: xz and xz have same outcome. (i.e. both
are accepted, or both are rejected.) 
•
Symmetric. If xz has same outcome as yz, then yz has
same outcome as xz.
•
Transitive. If xz has same outcome as yz, and yz has
same outcome as tz, then xz has same outcome as tz.
All 3 are obviously correct.
Proof (2)
If L is regular, # of equivalences classes is finite.
• Regular means L is recognized by some FA.
• Thus, # of states is finite.
• It turns out that (# equiv classes) <= (# states)
Why? Because 2 states may be “equivalent.”
More importantly: # classes can’t exceed # states. Proof:
Assume # classes > # states. Then we have a state
representing 2 classes. In other words, x and y in the same
state but x not related to y. Meaning that z where xz, yz don’t
have same fate but travel thru the same states! This makes no
sense, so we have a contradiction.
• Since (# equiv classes) <= (# states) and # states is
finite, we have that # equiv classes is finite. 
Proof (3)
If # equivalences classes finite, language is regular.
• We prove regularity by describing how to construct an
FA.
• Class containing ε would be our start state.
• For each class: consider 2 words x and y.
–
–
–
–
x0 and y0 have to be in the same class.
x1 and y1 have to be in the same class.
From this observation, we can draw appropriate transitions.
Since states are being derived from classes, the number of
states is also finite.
• Accept states are classes containing words in L.
Example
• L = { all bit strings with exactly two 1’s } has 4
equivalence classes
[ ε ], [ 1 ], [ 11 ], [ 111 ]
• Let’s consider the class [ 11 ]. This includes
words such as 11 and 0101.
– If we append a 0, we get 110 and 01010. These
words also belong in [ 11 ].
– If we append a 1, we get 111 and 01011. These
words belong in the class [ 111 ].
• This is the same thought process we use when
creating FA’s transitions anyway.
Learning from theorem
• There exist non-regular languages! It happens if
# of equivalence classes is infinite.
– Soon we’ll discuss another method for determining
non-regularity that’s a little easier.
• Part 3 of proof tells us that there is an FA with
the minimum number of states (states =
equivalence classes).
– See simplification algorithm handout.
Example
Is { 0n1n } regular? This is the language ε, 01, 0011, 000111, etc.
no 0’s
c1 = { ε }
0
one 0
c2 = { 0 }
0
two 0’s
c3 = { 00 }
0
Etc.
c4 = { 000 }
• Should x=0 and y=00 be in the same class? No!
Consider z = 1. Then xz = 01 and yz = 001. Different
outcome!
• ∞ # classes  Can’t draw FA.
• Equivalence classes are usually hard to conceive of, so
we’ll rely more on a different way to show languages not
regular.
CS 461 – Sept. 14
• Practice simplifying FA’s
• Section 1.4 – Pumping lemma
–
–
–
–
Purpose is to show that a language is not regular
Proof
Examples
Can also help to determine if a language is
finite/infinite.
Non-regular languages
• Means there is no such FA for language.
• They do exist! How can a language be nonregular?
– Not because language is too big, because the largest
possible language (0 + 1)* is regular.
– Non-regular means the “shape” of the language is
defective. It requires an infinite number of states.
Finite # of states can’t precisely define the language.
You’ll discover that a finite # of states winds up
defining something else.
finite
regular
Essential idea
The pumping lemma basically says this:
• If L is regular, you should be able take almost
any word in L, and repeat some of its characters
in order to produce longer words in L.
– Incidentally, you should also be able to take away
some of its characters to produce a shorter word.
– Degenerate case: finite sets.
– “Almost any word”? Some words are not interesting
like ε and 0.
Motivation
• Consider language { 0n1n }.
• Suppose it’s regular. Then there must be some
FA for it.
– How many states does it have? Let’s pick a number
like 95.
– Then, how would we process the word 097197? While
reading the 0s, we have to visit the same state twice!
In other words, we go back to a previous state, for
example from s80  s81  s82  s80, and then
continue on to accept state.
– There must be another word that leaves out the s81
and s82 steps, such as 094197, or does them
repeatedly. But those words are not in L!
Contradiction implies that L is not regular.
More generally…
• Suppose L is regular and M is FA recognizing it.
Let p = # of states.
• Consider a word s, |s|  p. We take at least p
transitions, so we visit at least p+1 states. So,
we must visit some state twice while reading the
first p symbols! Suppose we call this state q.
• Let s = xyz, where x is part of string up to first
time we visit q, and y is the part of string
between our 2 visits.
• Other words of the form xyiz are also accepted
by M. In other words, we can loop as many
times as we want, even 0 times.
A litmus test
• The pumping lemma is a guaranteed property of
any regular language.
• Analogy…
– If it’s a duck, then it will quack.
– If it’s a cow, then it will moo.
• Contrapositive proves an imposter!
– If it doesn’t quack, it’s not a duck.
– If it doesn’t moo, it’s not a cow.
• However, can’t be used to prove regularity.
Pumping lemma
• See handout
– Formal statement (if it’s a duck…)
– Contrapositive used to show nonregularity
• If language is regular, then there is some part of
any word we can “pump.”
• If we have a language we suspect is nonregular, use contrapositive!
Proving non-regularity
• It’s like a 2-player game…
– Adversary picks secret number p.
– We select any string we want, in terms of p
(e.g. 0p1p)
– Adversary will break up s into xyz subject to
constraints.
• The place to pump has length at least 1.
• The place to pump appears in the first p positions.
– Be ready to show that xyiz won’t be in
language for some i.
CS 461 – Sept. 16
• Review Pumping lemma
– How do we show language non-regular?
• Applications of FA:
– Scanning (part of compilation)
– Searching for text, using regular expression
Proving non-regularity
• It’s like a 2-player game…
– Adversary picks secret number p.
– We select any string we want, in terms of p
(e.g. 0p1p)
– Adversary will break up s into xyz subject to
constraints.
• The place to pump has length at least 1.
• The place to pump appears in the first p positions.
– Be ready to show that xyiz won’t be in
language for some i.
Example languages
•
•
•
•
•
•
•
Bit strings that are palindromes
Bit strings with equal # of 0s and 1s.
More 0s than 1s.
{ 0n : n is a perfect square }
{ 0n : n is prime }
{ 03n + 2 : n  0 }
{ 0i 1j : i is even or i  j }
Notice that regular sets can’t handle counting or nonlinear
patterns.
Consider complement
Show L = { 0i 1j : i is even or i  j } is not regular.
• Let’s use some set theory…
– L regular iff L’ regular
• What is L’ ?
– Hint: Need to consider (0 + 1)* - 0*1*.
– Language L’ has “and” instead of “or”, so easier to
produce a word not in language. 
continued
L = { 0i 1j : i is even or i  j }
Then L’ is all words with:
• The definition turned around.
Let A = { i is odd and i < j }.
• Plus all words not of the form 0*1*.
Let B = (0 + 1)* - 0*1* = 0*11*0(0 + 1)*.
Then, L’ = A union B.
Let s = 02p+1 12p+2 and i = 2 and see how it works in
pumping lemma…
CS 461 – Sept. 19
• Last word on finite automata…
– Scanning tokens in a compiler
– How do we implement a “state” ?
• Chapter 2 introduces the 2nd model of
computation
Application of FA
• Compilers! When a compiler looks at your
program, it does the following:
–
–
–
–
–
Lexical analysis (scanning: break up into tokens)
Syntax analysis
Semantic analysis
Code generation
Optimization
Comment
*
Next
char
/
=
Read
/
num
Found “/=“
Div by number
Chapter 2
• How can we represent a language?
–
–
–
–
–
English description √
FA drawing (state diagram) √
Transition table √
Regular expression √
Recursive definition!
• A context-free grammar (CFG) is a compact way
of writing a recursive definition.
Expr vs. grammar
• Basic idea to creating a grammar:
– Base case: what is smallest word in language?
(Sometimes we need > 1 base case.)
– Recursive case: how to create bigger words?
What do we append to the beginning or end of word
to create more words?
• Ex. 101* in the form of a grammar:
– Shortest word is 10.
– Create more words by appending 1s to the right end.
S  10
S  S1
example
• Let’s try (00)*10 as a grammar:
– Shortest word is 10.
– Bigger words? Append 00 to the front.
S  10
S  00S
– These rules can be combined to say S  10 | 00S
examples
• Let’s try 01*0(10)*.
– Uh-oh, how do you pump in the middle? If you need
concatenation, define the language in parts. The first
part will handle 01* and the second part 0(10)*.
S  AB
A  0 | A1
B  0 | B10
• (10*1)* could be written this way:
S  ε | SAB
A  1 | A0
B1
Formal definition
• A context-free grammar has 3 things
– Set of terminal symbols (alphabet plus ε)
– Set of nonterminal symbols (i.e. variables in
grammar)
– Set of rules or “productions” for our recursive
definitions
• The rules have the following format:
nonterminal  concat of 1+ terminals/nonterminals
• Often we have several rules for the same
nonterminal, so they are joined by “|” meaning
“or.”
More power
• CFGs are more powerful than FAs or regular
expressions because they allow us to define
non-regular languages!
• For example, 0*1* and { 0n 1n }
S  AB
S  ε | 0S1
A  ε | A0
B  ε | B1
• Wow, maybe non-regular languages are easier!
• With CFGs, we now have a 2nd class of
languages, the context-free languages. 
Models
#1
#2
Model name Finite automaton Pushdown
(FA)
automaton (PDA)
Recognizes Regular
what?
languages
Context-free
languages
Text rep’n
Context-free
grammar
Regular
expression
2nd model
• Encompasses regular & non-regular languages.
• PDA is like FA, but also has a stack.
– More on PDAs in section 2.2 later.
• Section 2.1: CFGs
– Important because this is how programming
languages are defined.
– Goals:
1. Given a grammar, can we generate words;
2. Given a set, can we write a grammar for it.
Deriving words
• Here is a CFG. What is it defining?
S  AB
A  ε | 1A
B  ε | 1B0
More practice
•
•
•
•
•
•
{ 0 i 1j : i > j }
0*11*0(0 + 1)*
(000)*11
1*0 (1 + 01*0)*
(a + ba) b*a
[bb + (a + ba) b*a]*
CS 461 – Sept. 21
Context-free grammars
• Handouts
– Hints on how to create grammar
– Examples of generating words from a grammar 
• Writing a grammar for a language
• Derivations
Formal definition
• A context-free grammar has 3 things
– Set of terminal symbols (alphabet plus ε)
– Set of nonterminal symbols (i.e. variables in
grammar)
– Set of rules or “productions” for our recursive
definitions
• The rules have the following format:
nonterminal  concat of 1+ terminals/nonterminals
• Often we have several rules for the same
nonterminal, so they are joined by “|” meaning
“or.”
More power
• CFGs are more powerful than FAs or regular
expressions because they allow us to define
non-regular languages!
• For example, 0*1* and { 0n 1n }
S  AB
S  ε | 0S1
A  ε | A0
B  ε | B1
• Wow, maybe non-regular languages are easier!
• With CFGs, we now have a 2nd class of
languages, the context-free languages. 
Models
#1
#2
Model name Finite automaton Pushdown
(FA)
automaton (PDA)
Recognizes Regular
what?
languages
Context-free
languages
Text rep’n
Context-free
grammar
Regular
expression
2nd model
• Encompasses regular & non-regular languages.
• PDA is like FA, but also has a stack.
– More on PDAs in section 2.2 later.
• Section 2.1: CFGs
– Important because this is how programming
languages are defined.
– Goals:
1. Given a grammar, can we generate words;
2. Given a set, can we write a grammar for it.
Practice
Should be able to write a grammar for any regular
set, and some simple non-regular cases:
• { 0 i 1j : i > j }
• 0*11*0(0 + 1)*
• (000)*11
• 1*0 (1 + 01*0)*
• (a + ba) b*a
• [bb + (a + ba) b*a]*
Deriving words
• Here is a CFG. What is it defining?
S  AB
A  ε | 1A
B  ε | 1B0
• Handouts: Generating random strings using a
grammar. 
Old Mother Hubbard
She went to the ____P____
to buy him a […….T……..
but when she same back,
he was _____D_____ the …....V…….]
T and V have to rhyme,
So select them at the same time.
CS 461 – Sept. 23
Context-free grammars
• Derivations
• Ambiguity
• Proving correctness
Derivations
• A sequence of steps showing how a word is
generated using grammar rules.
• Drawing a diagram or tree can help.
• Example: derivation of 1110 from this grammar:
S  AB
A  ε | 1A
B  ε | 1B0
S
A
B
1A
11A
11ε
11
1B0
1ε0
10
Derivations, con’d
• There are several ways to express a derivation
– Tree 
– Leftmost derivation = as you create string, replace
leftmost variable. Example:
S  AB  1AB  11AB  11B  111B0  1110
– Rightmost derivation
• Often, grammars are ambiguous
– For some string,  2+ derivation trees. Or,
equivalently:  2+ leftmost derivations.
– Example?
More examples
• Even number of 0’s
• Words of form { 0i 1j } where j is constrained.
i  j  2i (not to be confused with { 0n 12n } )
i  j  2i + 3
i  j  1.5i
• Next topic: how do we show that our grammar is
correct?
Correctness
•
Given a language L and a grammar G, how do
we know if L = L(G)? Must show:
1. L  L(G). All words in L can be derived by grammar.
Tell how to derive the words.
2. L(G)  L. All words derived by grammar are in L.
Use induction.
First example
• Show that S  ε | 0S1 is a CFG for the
language { 0n1n }
• Step #1: Any word in 0n1n can be generated by
the grammar. We explain the derivation.
– Apply the rule 0S1 n times
– Finally, apply the ε rule.
• Step #2: All words generated by the grammar
are of the form 0n1n.
– Induction on the number of times we use a rule.
– S  ε for the basis, and S  0S1 for the inductive
step.
Second example
L is { even number of 0s } and G is
S  ε | S1 | S0S0
Step #1. Suppose w  L. How can G generate
w?
If w = ε, we’re done.
Do the following until w is ε:
• If w ends with 1s, take them off by virtue of “S1”.
• Now w ends with 0, so take 2 0’s away by virtue
of “S0S0”. We know there must be 2 0’s in w
because w had an even number to start with.
Illustration
S  ε | S1 | S0S0
Let’s see how we can derive “010100”:
• The last 2 0s: S  S0S0  ε0ε0 = 00
• 0101 is 010 + 1
010 is S  S0S0  ε0S10  ε0ε10 = 010
1 is S  S1  ε1 = 1
• We can reconstruct derivation from beginning in
form of a tree. 
L(G) is in L
S  ε | S1 | S0S0
Step #2: Need to show all generated words are in
L.
Base case: ε is in L. It has even number of 0s. 
Recursive case. Let w be a word generated by the
grammar. Assume it has even # 0s. We can
create a larger word by applying rule “S1” or
“S0S0”. S1 adds no more 0s. S0S0 adds two
0s. Either way the # of 0s stays even.
Extra example
• Let’s show that this grammar G
S  ε | 0S1 | 00S111
generates the language L = { 0i 1j | i  j  1.5i }
• Step #1: any word in L can be generated
– This is the hard part, so let’s motivate the proof with
an example or two.
• Step #2: all words generated are in L
– Just induction again.
L is in L(G)
S  ε | 0S1 | 00S111
• How could we generate 011 116?
– We have 5 extra 1s. So we use rule 00S111 five
times.
– Undoing these steps, # 0s = 11-5*2 = 1 and # 1s =
16-5*3 = 1. Then we use 0S1 once and we are left
with ε.
• Okay, how about 012 116?
– We have 4 extra 1s, so use rule 00S111 four times.
– Undoing these steps, # 0s = 12-4*2 = 4 and # 1s =
16-4*3 = 4. They match! So use “0S1” 4 times.
Thinking out loud…
S  ε | 0S1 | 00S111
Let w = 0i 1j  L. In other words, i  j  1.5i
Consider the number j – i. This is the number of
times to apply rule #3 (00S111).
Note that using rule #3 (j – i) times will account for
2(j – i) zeros.
Then apply rule #2 (0S1) the “appropriate number”
of times. How many? Well, we want i 0’s and
we’ve already got 2(j – i), so we need the
difference: i – 2(j – i) = 3i – 2j.
Finishing step 1
S  ε | 0S1 | 00S111
Let w = 0i 1j  L. In other words, i  j  1.5i
Use rule #1 once to start with empty word.
Use rule #3 (j – i) times.
Use rule #2 (3i – 2j) times.
Total # 0s = 3i – 2j + 2(j – i) = i
Total # 1s = 3i – 2j + 3(j – i) = j
Thus, the word can be generated.
L(G) is in L
S  ε | 0S1 | 00S111
Now the easy part. Need to show that when we
apply any rule, we preserve: i  j  1.5i
Base case: ε has no 0s or 1s.
0  0  1.5*0 √
Recursive case. Let w be generated by the
grammar with i 0s and j 1s satisfying i  j  1.5i .
If we apply either rule 0S1 or 00S111, we can
show
(i + 1)  (j + 1)  1.5(i + 1)
(i + 2)  (j + 3)  1.5(i + 2)
(Need to work out arithmetic.)
Think about…
• Can you write a CFG for
{ 0i 1j | 0.5i  j  1.5i } ?
in other words, the ratio of 1s to 0s must be
between ½ and 3/2.
CS 461 – Sept. 26
• CFG correctness
• Section 2.2 – Pushdown Automata
Correctness
•
Given a language L and a grammar G, how do
we know if L = L(G)? Must show:
1. L  L(G). All words in L can be derived by grammar.
Tell how to derive the words.
2. L(G)  L. All words derived by grammar are in L.
Use induction.
First example
• Show that S  ε | 0S1 is a CFG for the
language { 0n1n }
• Step #1: Any word in 0n1n can be generated by
the grammar. We explain the derivation.
– Apply the rule 0S1 n times
– Finally, apply the ε rule.
• Step #2: All words generated by the grammar
are of the form 0n1n.
– Induction on the number of times we use a rule.
– S  ε for the basis, and S  0S1 for the inductive
step.
Second example
L is { even number of 0s } and G is
S  ε | S1 | S0S0
Step #1. Suppose w  L. How can G generate
w?
If w = ε, we’re done.
Do the following until w is ε:
• If w ends with 1s, take them off by virtue of “S1”.
• Now w ends with 0, so take 2 0’s away by virtue
of “S0S0”. We know there must be 2 0’s in w
because w had an even number to start with.
Illustration
S  ε | S1 | S0S0
Let’s see how we can derive “010100”:
• The last 2 0s: S  S0S0  ε0ε0 = 00
• 0101 is 010 + 1
010 is S  S0S0  ε0S10  ε0ε10 = 010
1 is S  S1  ε1 = 1
• We can reconstruct derivation from beginning in
form of a tree. 
L(G) is in L
S  ε | S1 | S0S0
Step #2: Need to show all generated words are in
L.
Base case: ε is in L. It has even number of 0s. 
Recursive case. Let w be a word generated by the
grammar. Assume it has even # 0s. We can
create a larger word by applying rule “S1” or
“S0S0”. S1 adds no more 0s. S0S0 adds two
0s. Either way the # of 0s stays even.
Extra example
• Let’s show that this grammar G
S  ε | 0S1 | 00S111
generates the language L = { 0i 1j | i  j  1.5i }
• Step #1: any word in L can be generated
– This is the hard part, so let’s motivate the proof with
an example or two.
• Step #2: all words generated are in L
– Just induction again.
L is in L(G)
S  ε | 0S1 | 00S111
• How could we generate 011 116?
– We have 5 extra 1s. So we use rule 00S111 five
times.
– Undoing these steps, # 0s = 11-5*2 = 1 and # 1s =
16-5*3 = 1. Then we use 0S1 once and we are left
with ε.
• Okay, how about 012 116?
– We have 4 extra 1s, so use rule 00S111 four times.
– Undoing these steps, # 0s = 12-4*2 = 4 and # 1s =
16-4*3 = 4. They match! So use “0S1” 4 times.
Thinking out loud…
S  ε | 0S1 | 00S111
Let w = 0i 1j  L. In other words, i  j  1.5i
Consider the number j – i. This is the number of
times to apply rule #3 (00S111).
Note that using rule #3 (j – i) times will account for
2(j – i) zeros.
Then apply rule #2 (0S1) the “appropriate number”
of times. How many? Well, we want i 0’s and
we’ve already got 2(j – i), so we need the
difference: i – 2(j – i) = 3i – 2j.
Finishing step 1
S  ε | 0S1 | 00S111
Let w = 0i 1j  L. In other words, i  j  1.5i
Use rule #1 once to start with empty word.
Use rule #3 (j – i) times.
Use rule #2 (3i – 2j) times.
Total # 0s = 3i – 2j + 2(j – i) = i
Total # 1s = 3i – 2j + 3(j – i) = j
Thus, the word can be generated.
L(G) is in L
S  ε | 0S1 | 00S111
Now the easy part. Need to show that when we
apply any rule, we preserve: i  j  1.5i
Base case: ε has no 0s or 1s.
0  0  1.5*0 √
Recursive case. Let w be generated by the
grammar with i 0s and j 1s satisfying i  j  1.5i .
If we apply either rule 0S1 or 00S111, we can
show
(i + 1)  (j + 1)  1.5(i + 1)
(i + 2)  (j + 3)  1.5(i + 2)
(Need to work out arithmetic.)
Think about…
• Can you write a CFG for
{ 0i 1j | 0.5i  j  1.5i } ?
in other words, the ratio of 1s to 0s must be
between ½ and 3/2.
PDAs
What is a Pushdown Automaton?
• Like an FA, but PDA has a stack
– Finite number of states
– Stack can grow to unlimited depth
• Transition function often non-deterministic
• 2 flavors
– Accept by happy state
– Accept by empty stack (this one has fewer states) 
• Can be tedious to draw, so instead give a table.
– What’s new is saying what to do with stack.
Formal definition
• A PDA has 6 things.
– Same five from FA, plus:
– Stack alphabet (can be different from Σ)
• Transition function
δ(state, tos, input) = (push/pop, new state)
both are optional
{ 0 n 1n }
• While reading 0s, push them.
• If you read 1, change state. Then, as you
read 1s, pop 0s off the stack.
• Watch out for bad input!
– Unspecified transition  crash (reject)
• Now, let’s write this in the form of a table.
PDA δ for { 0n 1n }
State
Reading 0
Tos
(don’t care)
Input
0
Action Push
0
Reading 1
0
Empty
1
0
1
Go to
state
“reading
1”
Crash
Pop
0
1
Crash Crash
Notes:
Action depends on input symbol AND what’s on top of stack.
Action includes manipulating stack AND/OR changing state.
CS 461 – Sept. 28
• Section 2.2 – Pushdown Automata
– { 0n 1n }
– Palindromes
– Equal
• Next: Converting CFG  PDA
{ 0 n 1n }
• While reading 0s, push them.
• If you read 1, change state. Then, as you
read 1s, pop 0s off the stack.
• Watch out for bad input!
– Unspecified transition  crash (reject)
• Now, let’s write this in the form of a table.
PDA δ for { 0n 1n }
State
Reading 0
Tos
(don’t care)
Input
0
Action Push
0
Reading 1
0
Empty
1
0
1
POP & Go
to state
“reading
1”
Crash
Pop
0
1
Crash Crash
Notes:
Action depends on input symbol AND what’s on top of stack.
Action includes manipulating stack AND/OR changing state.
Example #2
• { w # wR }
(easier form of palindrome)
• Let’s design a PDA that accepts its input by
empty stack, as before.
• Think about: state, top-of-stack, input, action
PDA δ for { w # wR }
State
Before #
Tos
(don’t care)
Input
0
Action Push
0
After #
0
1
#
0
Push
1
Go to
“after
#”
Pop
1
1
#
0
Crash Crash Crash
1
#
Pop
Crash
Palindrome PDA?
State
Before #
Tos
(don’t care)
Input
0
Action Push
0
After #
0
1
#
0
Push
1
Go to
“after
#”
Pop
1
1
#
0
Crash Crash Crash
1
#
Pop
Crash
Changes needed:
Non-deterministically go to “after #” when you push 0 or 1.
Also, non-deterministically don’t push, in case we are dealing with oddlength palindrome!
Other examples
Think about these
• “equal” language
– How many states do we need? …
• More 1s than 0s.
• Twice as many 1s as 0s.
– Hint: think of the 0s as counting double.
Equal PDA
State
(Just one state)
ε
Tos
Input
0
Action Push
0
0
1
1
0
1
0
1
Push
1
Push
0
Pop
Pop
Push
1
CS 461 – Oct. 3
• Converting a CFG into a PDA
• Union, intersection, complement of CFL’s
• Pumping lemma for CFL’s
CFG  PDA
Algorithm described in book
• Have state 1 push S on stack, and go to state 2.
• State 2 actions:
– (Note that any symbol in grammar could be tos, and
any terminal could be input.)
– When tos is variable, ignore input. Action is to
replace variable with its rules on stack.
• Multiple rules  non-determinism 
• Push symbols out of order.
• Push ε means leave stack alone.
– When tos is terminal, need action for when input=tos:
just pop it. Don’t need action for !=, just let nondeterministic instance disappear.
S  AB
Example:
A  ε | 1A2
B  ε | B3
State
1
Tos
ignore
2
S
A
B
1
2
3
ignore
ignore
1
2
3
Pop
Pop
Pop
Input
ignore ignore
Action
Push Pop S Pop A
S
Push
goto 2
AB
Pop A
Push
1A2
Pop B
Pop B
Push B3
Notice how the grammar rules are
encoded into the machine!
Let’s trace 1122333
State
1
2
Tos
ignore
S
A
B
1
2
3
ignore
ignore
1
2
3
Pop
Pop
Pop
Input
ignore ignore
Action
Push Pop S Pop A
S
Push
goto 2
AB
1
1
A
A A 2
A 2 2 2
S B B B B
Pop A
Push
1A2
Pop B
A
2 2
B
2 2 2
B 3
B B B B 3 3
Pop B
Push B3
B
3 3
3 3 3
3 3 3 3
Combining CFLs
• Let’s write grammars for
L 1 = { 1 i 2i 3j }
and
L2 = { 1 i 2 j 3 j }
• Grammar for L1 U L2 ?
• How about ∩ ?
Simple technique.
L1 ∩ L2 = { 1n 2n 3n }
Soon, we’ll prove this is not CFL.
What can we conclude?
• Complement? Remember: A ∩ B = (A’ U B’)’
Pumping lemma
L is a CFL implies:
There is a p,
such that for any string s, |s| >= p
We can break up s into 5 parts uvxyz:
| v y | >= 1
| v x y | <= p
and uvixyiz  L.
• How does this differ from the first pumping
lemma?
In other words
• If L is a CFL, then for practically any word w in L,
we should be able to find 2 substrings within w
located near each other that you can
simultaneously pump in order to create more
words in L.
• Contrapositive: If w is a word in L, and we can
always create a word not in L by simultaneously
pumping any two substrings in w that are near
each other, then L is not a CFL.
Gist
• A string can grow in 2 places.
– When we write recursive rules, we can add symbols
to left or right of a variable.
• Ex.
S  #A#
A  ε | 1A2
Look at derivation of: #111…222#
• Even a complex rule like S  S1S1S2 can be
“simplified”, and still only need 2 places to grow the
string.
– This is because any CFG can be written in such a way that there
are no more than 2 symbols on the right side of .
Example
• L = { 1n 2n } should satisfy the pumping lemma.
Let p = 2. Strings in L of length >= 2 are:
{ 12, 1122, 111222, … }. All these words have
both a 1 and a 2, which we can pump.
Just make sure the 2 places to pump are within
p symbols of each other. So let’s say they are in
the middle.
u
v x y
z
11111…1 1 ε 2
22222….2
Could p = 1 work?
CS 461 – Oct. 5
• Pumping lemma #2
– Understanding
– Use to show a language is not a CFL
• Next: Applications of CFLs
– Expression grammars and Compiling
Pumping lemma
L is a CFL implies:
There is a p,
such that for any string s, |s| >= p
We can break up s into 5 parts uvxyz:
| v y | >= 1
| v x y | <= p
and uvixyiz  L.
• How does this differ from the first pumping
lemma?
In other words
• If L is a CFL, then for practically any word w in L,
we should be able to find 2 substrings within w
located near each other that you can
simultaneously pump in order to create more
words in L.
• Contrapositive: If w is a word in L, and we can
always create a word not in L by simultaneously
pumping any two substrings in w that are near
each other, then L is not a CFL.
Gist
• A string can grow in 2 places.
– When we write recursive rules, we can add symbols
to left or right of a variable.
• Ex.
S  #A#
A  ε | 1A2
Look at derivation of: #111…222#
• Even a complex rule like S  S1S1S2 can be
“simplified”, and still only need 2 places to grow the
string.
– This is because any CFG can be written in such a way that there
are no more than 2 symbols on the right side of .
Example
• L = { 1n 2n } should satisfy the pumping lemma.
Let p = 2. Strings in L of length >= 2 are:
{ 12, 1122, 111222, … }. All these words have
both a 1 and a 2, which we can pump.
Just make sure the 2 places to pump are within
p symbols of each other. So let’s say they are in
the middle.
u
v x y
z
11111…1 1 ε 2
22222….2
Could p = 1 work?
Example 2
• { w # wR }. Think about where we can grow.
Let p = 3. All strings of length 3 or more look like
w # wR where | w | >= 1.
As a technicality, would other values of p work?
Non-CFLs
• It’s possible for a language to be non-CFL!
Typically this will be a language that must
simultaneously pump in 3+ places.
• Ex. How could we design a PDA for { 1n2n3n }?
Adversary’s Revenge
• In Game #1, how did we win? Ex. { 0n 1n }.
–
–
–
–
Adversary chose p.
We chose s.
Adversary broke up s = xyz subject to constraints.
We were always able to pump and find words outside
L.
• Game #2 strategy
– Adversary’s constraints looser. The middle 3 parts
have to be within p of each other. Can be anywhere
in the string, not just in first p symbols of word.
{ 1n 2n 3n }
• Let p be any number.
• Choose s = 1p2p3p.
• Let s = uvxyz such that |vxy| <= p and |vy| >= 1.
Where can v and y be?
– All 1’s
– All 2’s
– All 3’s
– Straddling 1’s and 2’s
– Straddling 2’s and 3’s
• In every case, can we find a word not in L?
More examples
• { 1 i 2j 3k : i < j < k }
– What should we chose as s?
– How can the word be split up?
• {ww}
–
–
–
–
Let s = 0p1p0p1p. Where can v & y be?
Same section of 0’s and 1’s.
In neighboring sections
Either v or y straddles a border.
CS 461 – Oct. 7
• Applications of CFLs: Compiling
• Scanning vs. parsing
• Expression grammars
– Associativity
– Precedence
• Programming language (handout)
Compiling
• Grammars are used to define programming
language and check syntax.
• Phases of a compiler
source
code
scanner
stream of
tokens
parser
parse
tree
Scanning
• Scanner needs to know what to expect when
eating your program.
–
–
–
–
identifiers
numbers
strings
comments
• Specifications for tokens can be expressed by
regular expression (or regular grammar).
• While scanning, we can be in different states,
such as inside a comment.
Parser
• Purpose is to understand structure of program.
• All programming structures can be expressed as
CFG.
• Simple example for + and –
expr  expr + digit | expr – digit | digit
digit  0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
How would we derive the string 9 – 5 + 2 ?
9–5+2
expr  expr + digit | expr – digit | digit
digit  0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
expr
expr - digit
digit
5
9
expr
+
Leftmost derivation: expr  expr + digit
 expr – digit + digit
 digit – digit + digit
 9 – digit + digit
 9 – 5 + digit
9–5+2
digit
2
“parse tree”
Left & right recursion
• What is the difference between these 2
grammars? Which one is better?
expr  expr + digit | expr – digit | digit
digit  0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
expr  digit + expr | digit – expr | digit
digit  0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
• Let’s try 9 – 5 + 2 on both of these. The
grammar must convey the order of operations!
• Operators may be left associative or right
associative.
+-*/
Question:
• How do we write grammar for all 4 operators?
Can we do it this way…
expr  expr + digit | expr – digit |
expr * digit | expr / digit |
digit
digit  0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
• BTW, we can ignore “digits” and from now on just
replace them with “num”, and understand it’s any single
number.
Precedence
• (* /) bind stronger than (+ -)
• (+ -) separate better than (* /)
• Need to break up expression into terms
– Ex. 9 – 8 * 2 + 4 / 5
– We want to say that an expression consists of “terms”
separated by + and –
– And each term consists of numbers separated by *
and /
– But which should we define first, expr or term?
Precedence (2)
• Which grammar is right?
expr  expr + term | expr – term | term
term  term * num | term / num | num
Or this one:
expr  expr * term | expr / term | term
term  term + num | term – num | num
Let’s try examples 1 + 2 * 3
and
1*2+3
Moral
• If a grammar is defining something hierarchical,
like an expression, define large groupings first.
• Lower precedence operators appear first in
grammar. (They separate better)
– Ex. * appears lower in parse tree than + because it
gets evaluated first.
• In a real programming language, there can be
more than 10 levels of precedence. C has ~15!
C language 
• Handout
– How does the grammar begin?
– Where are the mathematical expressions?
– Do you agree with the precedence?
– Do you see associativity?
– What else is defined in grammar?
– Where are the terminals?
CS 461 – Oct. 10
• Review PL grammar as needed
• How to tell if a word is in a CFL?
– Convert to PDA and run it. 
– CYK algorithm
– Modern parsing techniques
Accepting input
• How can we tell if a given source file (input
stream of tokens) is a valid program?
Language defined by CFG, so …
– Can see if there is some derivation from grammar?
– Can convert CFG to PDA?
• Exponential performance not acceptable. (e.g.
doubling every time we add token)
• Two improvements:
– CYK algorithm, runs in O(n3)
– Bottom-up parsing, generally linear, but restrictions
on grammar.
CYK algorithm
• In 1965-67, discovered independently by Cocke,
Younger, Kasami.
• Given any CFG and any string, can tell if
grammar generates string.
• The grammar needs to be in CNF first.
– This ensures that the rules are simple. Rules are of
the form X  a or X  YZ
• Consider all substrings of len 1 first. See if
these are in language. Next try all len 2, len 3,
…. up to length n.
continued
• Maintain results in an
NxN table. Top right
portion not used.
– Example on right is for
testing word of length
3.
• Start at bottom; work
your way up.
• For length 1, just look
for “unit rules” in
grammar, e.g. Xa.
1..3
1..2
1..1
X X
X
2..3
2..2
3..3
continued
• For general case i..j
– Think of all possible
ways this string can be
broken into 2 pieces.
– Ex. 1..3 = 1..2 + 3..3
or 1..1 + 2..3
– We want to know if
both pieces  L. This
handles rules of form
A  BC.
• Let’s try example from
3+7+. (in CNF)
1..3
1..2
1..1
X X
X
2..3
2..2
3..3
337  3+7+ ?
S  AB
A  3 | AC
B  7 | BD
C3
D7
For each len 1 string, which
variables generate it?
1..1 is 3. Rules A and C.
2..2 is 3. Rules A and C.
3..3 is 7. Rules B and D.
1..3
1..2
1..1
A, C
X X
X
2..3
2..2
A, C
3..3
B, D
337  3+7+ ?
S  AB
A  3 | AC
B  7 | BD
C3
D7
Length 2:
1..2 = 1..1 + 2..2 =
(A or C)(A or C) = rule A
2..3 = 2..2 + 3..3 =
(A or C)(B or D) = rule S
1..3
1..2
A
1..1
A, C
X X
X
2..3
S
2..2
A, C
3..3
B, D
337  3+7+ ?
S  AB
A  3 | AC
B  7 | BD
C3
D7
Length 3: 2 cases for 1..3:
1..2 + 3..3: (A)(B or D) = S
1..1 + 2..3: (A or C)(S) no!
We only need one case to
work.
1..3
S
1..2
A
1..1
A, C
X X
X
2..3
S
2..2
A, C
3..3
B, D
CYK example #2
Let’s test the word baab
S  AB | BC
A  BA | a
B  CC | b
C  AB | a
1..4
1..3
1..2
Length 1:
‘a’ generated by A, C
‘b’ generated by B
1..1
B
X X X
2..4
X X
2..3 3..4
X
2..2 3..3 4..4
A, C A, C B
baab
S  AB | BC
A  BA | a
B  CC | b
C  AB | a
Length 2:
1..2 = 1..1 + 2..2 = (B)(A, C) = S,A
2..3 = 2..2 + 3..3 = (A,C)(A,C) = B
3..4 = 3..3 + 3..4 = (A,C)(B) = S,C
1..4
1..2
S, A
X X X
2..4
X X
2..3 3..4
B
S, C X
1..1
B
2..2 3..3 4..4
A, C A, C B
1..3
baab
S  AB | BC
A  BA | a
B  CC | b
C  AB | a
Length 3: [ each has 2 chances! ]
1..3 = 1..2 + 3..3 = (S,A)(A,C) = Ø
1..3 = 1..1 + 2..3 = (B)(B) = Ø
2..4 = 2..3 + 4..4 = (B)(B) = Ø
2..4 = 2..2 + 3..4 = (A,C)(S,C) = B
1..4
1..2
S, A
X X X
2..4
X X
B
2..3 3..4
B
S, C X
1..1
B
2..2 3..3 4..4
A, C A, C B
1..3
Ø
Finally…
S  AB | BC
A  BA | a
B  CC | b
C  AB | a
Length 4 [has 3 chances!]
1..4 = 1..3 + 4..4 = (Ø)(B) = Ø
1..4 = 1..2 + 3..4 = (S,A)(S,C) = Ø
1..4 = 1..1 + 2..4 = (B)(B) = Ø
Ø means we lose!
baab  L.
However, in general don’t give up if
you encounter Ø in the middle of
the process.
1..4
Ø
1..2
S, A
X X X
2..4
X X
B
2..3 3..4
B
S, C X
1..1
B
2..2 3..3 4..4
A, C A, C B
1..3
Ø
CS 461 – Oct. 12
Parsing
• Running a parse machine
– “Goto” (or shift) actions
– Reduce actions: backtrack to earlier state
– Maintain stack of visited states
• Creating a parse machine
– Find the states: sets of items
– Find transitions between states, including reduce.
– If many states, write table instead of drawing 
Parsing
• CYK algorithm still too slow
• Better technique: bottom-up parsing
• Basic idea
S  AB
A  aaa
B  bb
At any point in time, think about where we could be while parsing the
string “aaabb”.
When we arrive at aaabb. We can reduce the “aaa” to A.
When we arrive at Abb, we can reduce the “bb” to B.
Knowing that we’ve just read AB, we can reduce this to S.
• See handouts for details.
Sets of items
• We’re creating states.
• We start with a grammar.
First step is to augment it
with the rule S’  S.
• The first state I0 will
contain S’   S
• Important rule: Any time
you write  before a
variable, you must
“expand” that variable.
So, we add items from
the rules of S to I0.
Example: { 0n 1n+1 }
S  1 | 0S1
We add new start rule
S’  S
State 0 has these 3 items:
I 0:
S’   S
S 1
Expand
S
S   0S1
continued
• Next, determine
transitions out of state 0.
δ(0, S) = 1
δ(0, 1) = 2
δ(0, 0) = 3
I’ve written destinations
along the right side.
State 0 has these 3 items:
I 0:
S’   S
1
S 1
2
S   0S1
3
• Now we’re ready for state
1. Move cursor to right to
become S’  S 
I 1:
S’  S 
continued
• Any time an item ends
with , this represents a
reduce, not a goto.
• Now, we’re ready for
state 2. The item S  1
moves its cursor to the
right: S  1 
This also become a
reduce.
I 0:
S’   S
S 1
S   0S1
1
2
3
I1:
S’  S 
r
I 2:
S 1
r
continued
• Next is state 3. From
S  0S1, move cursor.
Notice that now the  is in
front of a variable, so we
need to expand.
• Once we’ve written the
items, fill in the
transitions. Create new
state only if needed.
δ(3, S) = 4 (a new state)
δ(3, 1) = 2 (as before)
δ(3, 0) = 3 (as before)
I 0:
I 1:
I2:
I 3:
S’   S
S 1
S   0S1
S’  S 
S 1
1
2
3
r
r
S  0  S1
S 1
S   0S1
4
2
3
continued
• Next is state 4. From
item S  0  S1, move
cursor.
• Determine transition.
δ(4, 1) = 5
Notice we need new state
since we’ve never seen
“0 S  1” before.
I 0:
I 1:
I2:
I 3:
I 4:
S’   S
S 1
S   0S1
S’  S 
S 1
S  0  S1
S 1
S   0S1
1
2
3
r
r
4
2
3
S  0S  1
5
Last state!
• Our last state is #5.
Since the cursor is at the
end of the item, our
transition is a reduce.
• Now, we are done finding
states and transitions!
• One question remains,
concerning the reduce
transitions: On what
input should we reduce?
I 4:
S’   S
S 1
S   0S1
S’  S 
S 1
S  0  S1
S 1
S   0S1
S  0S  1
1
2
3
r
r
4
2
3
5
I 5:
S  0S1 
r
I 0:
I 1:
I2:
I 3:
CS 461 – Oct. 17
• Creating parse machine
– Convert grammar into sets of items
– Determine goto and reduce actions
• On what input do we reduce?
– Whatever “follows” the nonterminal we’re reducing to.
• Declaration grammar
0n 1n+1
• There are 5 states.
When the cursor is at the
end of the item, our
transition is a reduce.
• Now, we are done finding
states and transitions!
• One question remains,
concerning the reduce
transitions: On what
input should we reduce?
I 4:
S’   S
S 1
S   0S1
S’  S 
S 1
S  0  S1
S 1
S   0S1
S  0S  1
1
2
3
r
r
4
2
3
5
I 5:
S  0S1 
r
I 0:
I 1:
I2:
I 3:
When to reduce
If you are at the end of an item such as S  1 ,
there is no symbol after the  telling us what
input to wait for.
– The next symbol should be whatever “follows” the
variable we are reducing. In this case, what follows
S. We need to look at the original grammar to find
out.
– For example, if you were reducing A, and you saw a
rule S  A1B, you would say that 1 follows A.
– Since S is start symbol, $ (end of input) follows S.
• For more info, see parser worksheet.
– New skill: for each grammar variable, what follows?
First( )
To calculate first(A), look at A’s rules.
• If you see A  c…, add c to first(A)
• If you see A  B…, add first(B) to first(A).
Note: don’t put $ in first( ).
Follow( )
What should be included in follow(A) ?
•
•
•
•
If A is start symbol, add $.
If you see Q  …Ac…, add c.
If you see Q  …AB…, add first(B).
If you see Q  …A, add follow(Q).
Note: don’t put ε in follow( ).
CS 461 – Oct. 19
• Examples
– Calculator grammar
– Review first( ) and follow( )
– Declaration grammar
• How to handle ε in grammar
– Need to change how we find first( ) and follow( ).
First( )
To calculate first(A), look at A’s rules.
• If you see A  c…, add c to first(A)
• If you see A  B…, add first(B) to first(A).
– If B can yield ε, continue to next symbol in rule until
you reach a symbol that can represent a terminal.
• If A can yield ε, add ε to first(A).
Note: don’t put $ in first( ).
Follow( )
What should be included in follow(A) ?
•
•
•
•
If A is start symbol, add $.
If you see Q  …Ac…, add c.
If you see Q  …AB…, add first(B).
If you see Q  …A, add follow(Q).
• If you see Q  …ABC, and B yields ε, add first (C).
• If you see Q  …AB, and B yields ε, add follow(Q).
Note: don’t put ε in follow( ).
Example
Try this grammar:
S  AB
A  ε | 1A2
B  ε | 3B
First(B) = ε, 3
First(A) = ε, 1
First(S) = ε, 1, 3
(note in this case A  ε)
Follow(S) = $
since S is start symbol
Follow(A) = 2, 3, $
we need first(B)
since B  ε, we need $
Follow(B) = $
Try this one
Let’s try the language ((1*2(3+4)*(56)*)*
Rules
S’  S
S  ε | SABC
A  2 | 1A
B  ε | 3B | 4B
C  ε | 56C
First
Follow
answer
Let’s try the language ((1*2(3+4)*(56)*)*
Rules
First
Follow
S’  S
ε, 1, 2
$
S  ε | SABC
ε, 1, 2
1, 2, $
A  2 | 1A
1, 2
3, 4, 5, $, 1, 2
B  ε | 3B | 4B
ε, 3, 4
5, $, 1, 2
C  ε | 56C
ε, 5
$, 1, 2
CS 461 – Oct. 21
Begin chapter 3
• We need a better (more encompassing) model
of computation.
• Ex. { 1n 2n 3n } couldn’t be accepted by PDA.
– How could any machine accept this language?
– Somehow need to check 1s, 2s and 3s at same time.
– We should also be able to handle 1n 2n 3n 4n , etc.
Turing Machine
• 3rd model of computation
– Alan Turing, 1936
– Use “tape” instead of stack
– Can go left or right over the input!
• Every TM has:
–
–
–
–
Set of states, including 1 start and 1 accept state
Input alphabet
Tape alphabet, including special “blank” symbol
Transition function
δ (state, input) = (state, output, direction)
The tape and δ
• Tape is infinite in both directions
– Blanks straddle the input on the tape.
• Begin reading at leftmost input symbol.
• As soon as you enter accept state, halt.
– You don’t have to read all the input.
δ (state, input) = (state, output, direction)
– You may change state.
– You may change the symbol that’s on the tape.
– The “direction” is either L or R.
States
• A TM has at least 3 states
– One accept state
– One reject state
– Ordinary states that we describe in our transition
table, beginning with a start state.
• We don’t need transitions for the accept or reject
states.
• Sometimes a transition “doesn’t matter” because
we can’t be in that situation.
– Ex. If the leftmost symbol isn’t blank, then the
rightmost symbol can’t be blank. You can just go to
reject state.
Example
• A TM to accept the language 0(0+1)*1.
– Make sure first symbol is 0. Else reject.
– Keep moving right until you see blank.
– After blank, go left. If 1, accept. Else reject.
State
Input 0
Input 1
Input _
s1
s2, 0, R
reject
reject
s2
s2, 0, R
s2, 1, R
s3, _, L
s3
reject

Doesn’t matter
Let’s trace 011
State
s1
s2
s3
Input 0
s2, 0, R
s2, 0, R
reject
Input 1
reject
s2, 1, R

Tape
Transition
Effect
[s1]011
s2,0,R
0[s2]11
0[s2]11
s2,1,R
01[s2]1
01[s2]1
s2,1,R
011[s2]
011[s2]_
s3,_,L
01[s3]1_
01[s3]1

Input _
reject
s3, _, L
Doesn’t matter
Input  output
• TMs can easily modify the input on the tape. Thus we
can have useful output!
• Example: let’s flip the bits of our binary input, and
automatically accept.
– Note: the reject state is unreachable.
State
s1
Input 0
s1, 1, R
Input 1
s1, 0, R
Input _

Infinite loop
• It’s possible for a TM to have an infinite loop. We say
that the TM “loops.” In this case, it doesn’t accept or
reject its input.
• Simple example: From start state, go left on any input!
State
Input 0
Input 1
Input _
s1
s1, 0, L
s1, 1, L
s1, _, L
Mystery #1
• What does this TM do? Assume s1 is the start
state, and unspecified transitions are crash.
State
Input 0
s1
s2, 1, R
s2
Input 1
Input _
s1, 0, R

Hint: notice we never move left. This means it doesn’t
matter what we write on tape.
Mystery #2
• Can you figure out what this machine does?
– See something wrong?
State
s1
s2
Input 0
s1, 0, R

Input 1
s2, 1, R
s2, 1, R
Input _
s1, _, R
s2, _, R
CS 461 – Oct. 24
• Turing machines √
– Can go left or right
– Can write on tape
– Can halt early or go on forever!
• More examples of TMs.
–
–
–
–
Going in both directions along the tape.
Practice!
Optional features
Doing arithmetic on a TM.
Review:
Mystery #2
• Can you figure out what this machine does?
– See something wrong?
State
s1
s2
Input 0
s1, 0, R

Input 1
s2, 1, R
s2, 1, R
Input _
s1, _, R
s2, _, R
Back & forth
• Often it’s useful for a TM to scan its input in both
directions.
• Ex: Cut and paste, as in 001  ___#001.
– First, put a # at the end of the input.
– Copy first input symbol onto the end, and continue for
all other symbols.
– How do we know we’re “at the end”?
– How do we know a given symbol has already been
duplicated?
– How will we know we are finished?
algorithm
1.
2.
3.
4.
5.
6.
Go right until you see _. Upon blank, change it to #
and head left. Go to 2.
Keep going left until you see _. Upon blank, go right
and go to state 3.
If “0”, change it to _, head right and go to state 4.
If “1”, change it to _, head right and go to state 6.
If “#”, accept.
Go right until you see _. At the _, change it to 0 and
head left in state 5.
Keep left until _. At _, go right. Go to state 3.
Just like state 4, but change _ to 1.
δ and trace
State
0
1
#
_
1
1,0,R
1,1,R
2
2,0,L
2,1,L
2,#,L
3
4,_,R
6,_,R

4
4,0,R
4,1,R
4,#,R
5,0,L
5
5,0,L
5,1,L
5,#,L
3,_,R
6
6,0,R
6,1,R
6,#,R
5,1,L
2,#,L
3,_,R
[s1]01
0[s1]1
01[s1]_
0[s2]1#
[s2]01#
[s2]_01#
[s3]01#
_[s4]1#
_1[s4]#
_1#[s4]_
_1[s5]#0
_[s5]1#0
Q: How can you tell if a transition basically says
“do nothing” and move L/R?
[s5]_1#0
_[s3]1#0
__[s6]#0
__#[s6]0
__#0[s6]_
__#[s5]01
__[s5]#01
_[s5]_#01
__[s3]#01

Some practice
• See handout for some more examples.
• Let’s design TMs that accept these languages.
0*1*
Palindrome
Palindrome
1. If “0”/”1”, change to _ and go to 2/4. If _, accept.
2. Continue right until _. Then go left and go to 3.
3. If “0”, change to _, go left. Go to 6. If “1” reject
4. (states 4,5 are analogous to 2,3)
5. If “1”, change to _, go left. Go to 6. If “0” reject.
6. Continue left until _. Turn right. Start over at 1.
Closer look at δ
State
0
1
_
1
2, _, R
4, _, R

2
2, 0, R
2, 1, R
3, _, L
3
6, _, L
Reject
4
4, 0, R
4, 1, R
5
Reject
6, _, L
6
6, 0, L
6, 1, L
5, _, L
1, _, R
[s1]101
_[s4]01
_0[s4]1
_01[s4]_
_0[s5]1
_[s6]0_
[s6]_0_
_[s1]0_
__[s2]_
_[s3]__
At this point we realize the
missing δ should be  also.
CS 461 – Oct. 26
• TM practice
–
–
–
–
0*1*
Twice as many 0’s as 1’s
Palindrome…
Adding register(s) to simplify programming
• Arithmetic on a TM
• TM variants (handout)
– Adding more features doesn’t add power.
If a TM can’t do it, nothing can.
Palindrome
1. If “0”/”1”, change to _ and go to 2/4. If _, accept.
2. Continue right until _. Then go left and go to 3.
3. If “0”, change to _, go left. Go to 6. If “1” reject
4. (states 4,5 are analogous to 2,3)
5. If “1”, change to _, go left. Go to 6. If “0” reject.
6. Continue left until _. Turn right. Start over at 1.
Closer look at δ
State
0
1
_
1
2, _, R
4, _, R

2
2, 0, R
2, 1, R
3, _, L
3
6, _, L
Reject

4
4, 0, R
4, 1, R
5, _, L
5
Reject
6, _, L

6
6, 0, L
6, 1, L
1, _, R
[s1]101
_[s4]01
_0[s4]1
_01[s4]_
_0[s5]1
_[s6]0_
[s6]_0_
_[s1]0_
__[s2]_
_[s3]__
 transitions have been added.
Registers
• In palindrome example, we needed 2 extra
states because leftmost input symbol could be 0
or 1.
• What if alphabet were { a, b, c, … z } ? Do we
literally want 2*25 more states? 
• In place of those extra states, store a finite
amount of information in TM’s “control unit”, just
like registers in CPU. With a register, our state
is just an ordered pair like (2, ‘t’).
• More and more, a TM is looking like a computer!
New palindrome
1. If _, accept. Otherwise copy first symbol to
register. Change tape to _ and go to 2.
2. Continue right until _. Then go left and go to 3.
3. If input matches register, change to _, go left.
Go to 6. If no match, reject. (if _, accept)
6. Continue left until _. Turn right. Start over at
1.
Arithmetic
• Can do binary addition on a TM.
– Method #1: Unary programming 
Input is of the form 1101111 for 2 + 4.
– Method #2: With binary notation: Modify our
definition of TM so it can have 3 parallel tracks on the
tape. The third track can hold the answer.
• Can do other operations analogously.
Unary +
1.
Make sure input is of the form 1*01*. If not, reject. If
so, put = at end and go back to beginning of word.
10. Copy all the 1s on left side of 0 to the right end of
word.
20. Copy all the 1s between the 0 and = to the right end of
word.
•
•
This is just an outline, and I didn’t attempt to count
exact number of states needed.
If we didn’t care about changing the input, we could
just get rid of the 0 and squeeze the 1s together!
TM Variants
• Based on additional features.
– Can save a lot of effort.
– Multi-tape probably most useful.
• We can show that a new feature can be
emulated by the ordinary TM model.
•  Nothing can beat a TM. The extra bells and
whistles just make programming easier.
Some variants
• Allowing a TM to “stay put” instead of having to
move left or right.
•  Multi-tape (not to be confused with multi-track
tape)
– Makes use of stay-put
– Scanning the “virtual heads”
•  Non-deterministic
– Makes use of multi-tape
Non-determinism
• The non-deterministic computations can be
arranged in a tree.
– If we want to deterministically search this tree, DFS or
BFS?
– P. 151: Tape 3 will contain a list of transitions to
experiment on. Copy tape 1 to tape 2, and use tape 2
for scratch work. If reject, increment tape 3 to next
possible set of transitions.
• Ex. Accept the union of 2+ languages
• Ex. For some language L, design a TM that will
accept substrings (of length 3) of these words.
Computer
• A TM with multiple tapes can simulate a
computer.
– One tape contains data and instruction memory.
Unfortunately, need to label each with address.
– One tape can store the address of current instruction.
– Another tape can store address of data operand.
– Finally, a tape for scratch work.
CS 461 – Oct. 28
• TM applications
–
–
–
–
Recognize a language √
Arithmetic √
Enumerate a set
Encode a data structure or problem to solve
• Two kinds of TMs
– Decider: will always halt for any input
This one is useful for encoding an algorithm. 
– Non-decider: for some input, may loop forever
Enumerators
• List all the words in the language
– In order of length
– Within each length, canonical order such as
alphabetical
• Technique
– Ignore input
– Use multiple parallel tapes. One for output, the rest
for scratch work.
Example
• Let’s enumerate { 0n 1n }.
–
–
–
–
One tape will hold value of n.
One tape will hold for i = 1 to n.
One tape will hold the words we write.
(need to separate words by special symbol such as #)
– The machine will go on forever.
• Similarly, we could enumerate all positive integers, all
prime numbers, etc. (They’re not algorithms.)
TM input encoding
• Often we use TMs to solve problems or answer
mathematical questions.
• For input, we need a specific encoding or rep’n.
– Ex. Given an equation, can we solve for x?
– Ex. Given a graph, can we tell if it’s connected?
– If “yes,” there is an algorithm… so we can design a
TM (or program). We encode the input to feed to TM
because it can’t see the graph like we can.
Yes
String input
TM
No
Encode these
If an object doesn’t have a text or numerical
representation, need to create one.
How would you encode…?
• Graph
• Binary tree
• finite automaton
• context-free grammar
• Polynomial
• Function (or one-to-many relation)
Algorithm
• A sequence of steps that always halts (  or  ) no
matter what the input is.
• Decider: TM will always enter  or  state, even if it
takes a long time.
• If we can create a decider (i.e. algorithm), we say the
problem is “decidable”.
• 3 ways we specify a TM
– List all transitions formally (as in a table)
– Write pseudocode for each state
– High-level description – making it clear algorithm will
always halt, but doesn’t spell out states.
Examples
• Let’s look at informal TM descriptions.
– Must clearly define the input, the overall steps, and
how to accept/reject.
• Graph connectedness
– See how book writes it on p. 158
• Solving an equation for “x”
– What is wrong with TM description on p. 156?
• Determine if a number is prime
• Does a given FA accept its input
CS 461 – Oct. 31
• Chapter 4 – Decidable questions
“Is it decidable to determine if …”
We want a TM that will always halt.
• A closer look at ∞
Question about FAs
• Given any DFA and any input string, is there an
algorithm to determine if the DFA will accept the
string?
• This is called the “acceptance problem” for
DFAs: ADFA.
• Approach (can visualize as diagram)
–
–
–
–
Encode the DFA and input string.
Feed into TM, and have TM run the DFA on input.
If DFA accepts, say “yes.” If DFA rejects, say “no.”
Note that this TM will always halt!
Other FA questions
• Once we can solve one problem about FAs, try
to use similar technique…
• ANFA
• For 2 FA’s, L(M1) = L(M2) ?
• L(M) = Ø ?
• Is L(M) infinite?
• Does L(M) contain a word of length 5?
• Does L(M) contain all words of length 5?
• Does L(M) contain all words of the form *01?
• Are all words in L(M) of the form *01?
Questions about CFG
• ACFG
– How would we phrase this question?
– Is it decidable? (i.e. can we do it in finite time?)
• Also decidable
– L(G) = Ø ?
– L(G) contains some word or finite subset.
– L(G) is itself infinite.
• The “=“ question is harder than for FA’s
– For CFLs, intersection isn’t closed.
– For complicated reason, not decidable anyway.
– Questions about infinite intersections undecidable.
Question about TM
• ATM is an important question
– Why can’t we run a TM just like we did for FAs?
– Does this prove that it’s impossible?
• To resolve the question of ATM
– We’ll prove undecidable by contradiction.
– First, need to understand something about ∞
– What do the words “finite” and “infinite” mean?
CS 461 – Nov. 2
• Sets
– finite vs. infinite
– Infinite sets
• Countable
• Uncountable
• Prepare for ATM
– Proving undecidable is similar to proving a set
uncountable.
– Please be sure you understand reasoning.
Finite/infinite
• Finite set – means that its elements can be
numbered. Formally: its elements can be put
into a 1-1 correspondence with the sequence 1,
2, 3, … n, where n is some positive integer.
• An infinite set is one that’s not finite. 
• However, there are 2 kinds of infinity!
Countable
• Countable set: An infinite set whose elements
can be put into a 1-1 correspondence with the
positive integers.
– Examples we can show are countable:
Even numbers, all integers, rational numbers, ordered
pairs.
• Uncountable set = ∞ set that’s not countable. 
– Examples we can show are uncountable:
Real numbers, functions of integers, infinite-length
strings
Examples
• The set of even numbers
is countable.
2
4
6
8
10
12
…
1
2
3
4
5
6
…
• The set of integers is
countable.
0 1 -1 2 -2 3 -3 4 -4 …
1 2 3 4 5 6 7 8 9
Ordered pairs
j=1
j=2
j=3
j=4
i=1
1
2
4
7
i=2
3
5
8
…
i=3
6
9
…
i=4
10
…
…
…
The number assigned to (i , j) is (i + j – 1)*(i + j – 2)/2 + i
Real numbers
• Suppose real numbers
were countable. The
numbering might go
something like this:
• The problem is that we
can create a value that
has no place in the
correspondence!
Real #
Value
1
2.71828…
2
3.14159…
3
0.55555…
4
-1.23456…
5
5.676767...
…
…
X?
. 85667…
Infinite bit strings
#
Value
1
00000…
2
100000…
3
011010…
4
0010001…
5
11111001011…
…
?
11010…
Universal TM
Let’s design “U” – the Universal TM:
• Input consists of <M> and w:
– <M> is the encoding of some TM
– w is any (binary) string.
• Assume: U is a decider (i.e. ATM is decidable.)
U
<M>,w
M
w
yes
yes
no*
no
ATM solution
• Start with U, the Universal Turing Machine
• Suppose U decides ATM. Let’s build new TM D.
– D takes in a Turing machine, and returns opposite of U’s answer.
D
<M>
U
<M>,<M>
no
yes
no
yes
If M accepts its own string rep’n, D rejects <M>.
If M doesn’t accept <M>, D accepts <M>.
What does D do with <D> as input?
For example
<M1> <M2> <M3> <M4> …
M1
Yes
No
No
Yes
M2
No
No
Yes
Yes
M3
Yes
No
Yes
No
M4
Yes
No
No
No
No
Yes
No
Yes
<D>
…
D
Contradiction  The TM D can’t exist  So U is not a decider.
Uh-oh
In other words
•
Let U = universal TM.
–
–
–
•
Create 2nd TM called D.
–
–
–
•
Its input is a TM description <M> and a word <w>.
Determines if M accepts w.
Assume U halts for all inputs. (is a decider)
Its input is a TM description <M>.
Gives <M> to U as the TM to run as well as the input.
D returns the opposite of what U returns.
What happens when the input to D is <D>?
–
–
–
–
According to U, if D accepts <D>, U accepts, so D must reject!
According to U, if D rejects <D>, U rejects, so D must accept!
Both cases give a contradiction.
Thus, U is not a decider. ATM is undecidable.
CS 461 – Nov. 7
• Decidability concepts
–
–
–
–
Countable = can number the elements 
Uncountable = numbering scheme impossible 
ATM undecidable
Language classes
• Next
– { languages } uncountable, but { TMs } countable
There are more languages than TMs! …
– Be on the lookout for ∞ rep’n.
Universal TM
Let’s design “U” – the Universal TM:
• Input consists of <M> and w:
– <M> is the encoding of some TM
– w is any (binary) string.
• Assume: U is a decider (i.e. ATM is decidable.)
U
<M>,w
M
w
yes
yes
no*
no
ATM solution
• Start with U, the Universal Turing Machine
• Suppose U decides ATM. Let’s build new TM D.
– D takes in a Turing machine, and returns opposite of U.
D
<M>
U
<M>,<M>
no
yes
no
yes
If M accepts its own string rep’n, D rejects <M>.
If M doesn’t accept <M>, D accepts <M>.
What does D do with <D> as input?
For example
<M1> <M2> <M3> <M4> …
M1
Yes
No
No
Yes
M2
No
No
Yes
Yes
M3
Yes
No
Yes
No
M4
Yes
No
No
No
No
Yes
No
Yes
<D>
…
D
Contradiction  The TM D can’t exist  So U is not a decider.
Uh-oh
In other words
•
Let U = universal TM.
–
–
–
•
Create 2nd TM called D.
–
–
–
•
Its input is a TM description <M> and a word <w>.
Determines if M accepts w.
Assume U halts for all inputs. (is a decider)
Its input is a TM description <M>.
Gives <M> to U as the TM to run as well as the input.
D returns the opposite of what U returns.
What happens when the input to D is <D>?
–
–
–
–
According to U, if D accepts <D>, U accepts, so D must reject!
According to U, if D rejects <D>, U rejects, so D must accept!
Both cases give a contradiction.
Thus, U is not a decider. ATM is undecidable.
Language classes
Working from the inside out:
•
•
•
•
•
•
•
Finite set
Regular
CFL (deterministic)
CFL (non-deterministic)
Decidable
Turing-recognizable
Outer space!
– Yes – it’s possible for a language not to be
recognized by any TM whatsoever
• Note: all languages are countable (or finite).
Language beyond TM
• The set of all TM’s is countable.
– Finite representation
• The set of all languages is uncountable.
– Infinite representation
• Not enough TM’s to go around  There must be
a language unrecognized by any TM.
• Let’s find one!
Other properties
• 2 kinds of TMs  2 kinds of languages.
– Turing-recognizable (a.k.a. recursively enumerable)
• Example: ATM
– Decidable (a.k.a. recursive)
• Example: 0*
• If L is decidable, then L’ is decidable.
• If L and L’ are both Turing-recognizable,
then L is decidable. (since either L or L’ must accept)
• Therefore, the complement of ATM is not even Turing
recognizable.
CS 461 – Nov. 9,11
• Chomsky hierarchy of language classes
– Review
– Let’s find a language outside the TM world!
– Hints: languages and TM are countable, but the set
of all languages is uncountable
• Undecidability
– We have seen that ATM is undecidable.
– Let’s look at some more undecidable problems.
TM properties
• 2 kinds of TMs  2 kinds of languages.
– Turing-recognizable (a.k.a. recursively enumerable)
• Example: ATM
– Decidable (a.k.a. recursive)
• Example: 0*
• If L is decidable, then L’ is decidable.
• If L and L’ are both Turing-recognizable,
then L is decidable. (since either L or L’ must accept)
• Therefore, the complement of ATM is not even Turing
recognizable.
Decidable?
for n = 3 .. maxint:
for each combo of (x, y, z) starting at 1:
if (xn + yn == zn)
print (“hello, world”);
• Are we able to tell if this program will print
“hello, world” or not?
– Only since 1993 . For 300 years we didn’t know.
– Solving this problem in general (for any program)
must be hard… but what if it could be done?
“hello, world” tester
H
Print “yes”
program & input
Print “no”
• Suppose H can decide the question.
• Then we can create similar algorithm H2.
– Returns “hello, world” instead of “no”.
– Let’s run H2 on itself!
H2
Print “yes”
H2
Print “hello, world”
Regular?
• REG = “Is there an algorithm to determine if a TM
recognizes a regular language?”
• Proof by contradiction, p. 191
• Assume REG is decidable.
– Then,  TM “R” that can tell if an input <TM> recognizes a
regular language.
– We’ll use R to solve the ATM problem by creating a decider S.
– In other words:
Machine R solves REG -- which implies existence of ….
Machine S which solves ATM.
– Do we understand the I/O of R and S?
Regular, continued
• Here is how to build machine S, which will “solve” ATM.
• Input to S is <T, w>.
• Create a new TM “M2”
– M2 sees if its input, x, is of the form 0n1n. If so, M2 accepts x.
– If x is not of the form 0n1n,
• M2 runs T on w.
• If T accepts w, M2 accepts x.
– Thus, L(M2) is either Σ*, if T accepts w, or 0n1n.
– The only way for M2 to accept Σ* is for T to accept w. 
• Feed M2 into R.
– If R says that M2 accepts a regular language, we can conclude T
accepts w, so S can say “yes”, i.e. accept. Otherwise S rejects.
– But S can’t exist. So we have a contradiction. REG undecidable.
Rice’s Theorem
• Any question Q about a TM language is undecidable.
• Proof by contradiction:
• Assume Q is decidable. Then  TM “QD” that
– always halts, and
– answers Q for any input TM it receives.
• Using QD, we can build a machine AD to decide ATM.
• (But we know that AD can’t exist, therefore QD can’t exist.
That is the contradiction.)
Building TM “AD”
• AD will do the following:
– Its input is <T, w>.
– Build a TM “Q1” that satisfies Q.
– Build a TM “M2” whose input is x.
• M2 runs T on w. If T loops/rejects, M2 will loop/reject.
• If T accepts w, next run Q1 on x. M2 accepts if Q1
accepts.
• Note that L(M2) = L(Q1) if T accepts w. L(M2) =  if T
does not accept w.
• In other words, if T accepts, then M2 satisfies Q.
– Feed M2 into QD. If QD accepts, AD accepts. Otherwise
AD rejects.
• Thus: QD is a decider  AD is a decider.
Glossary to proof
•
•
•
•
•
•
•
•
Q = some question or property about a TM language
QD = a TM that can decide Q
Q1 = a TM for which Q is true, built inside AD
ATM = the acceptance question for TM’s, already known
to be undecidable
AD = a hypothetical decider for ATM
T = the TM input to AD
w = the corresponding word also input to AD
M2 = a second TM built inside AD
• Rice’s theorem is a little complex because  5 TM’s!
Examples
•
•
•
•
Let Q = “Is the language infinite?” or ….
“Does the language include 001?”
“Is the language finite?”
“Is the language nonregular?”
• Can also prove by contradiction:
– Is it decidable to determine if one TM language is the
subset of another?
CS 461 – Nov. 14
• Section 5.2 – Undecidable problems
starting from ATM:
 Matching sets of strings
 Ambiguity of CFGs
String matching
• Formal name is the Post Correspondence
Problem, (by mathematician Emil Post)
• Given a set of dominoes
– Each contains a string on the top and bottom
– Use the dominoes so that the strings on the top and
bottom match.
– You may use each domino as many times as you like.
But there must be one domino. 
– The solution is the sequence of dominoes (e.g. 1,2,3)
11
111
100
001
111
11
Another PCP
• Can you find a solution to this one?
1
111
10111
10
10
0
011
11
101
011
Or this one?
10
101
Undecidability
• To show the problem is undecidable in general,
we draw a parallel with ATM, which we know is
undecidable.
• Take any TM with an input word. If we follow the
TM’s steps, there is a parallel sequence of
dominoes! Accept  there is a match, and
reject  there is no match.
• For each possible TM action, we specify what
the dominoes look like.
Example
• This is best seen via an example. Let’s say we
have a TM that accepts the language
1*00*1(0+1)*, and we feed it the word 01.
State
0
1
_
s1
s2, 0, R
s1, 1, R
rej, _, R
s2
s2, 0, R
acc, 1, r
rej, _, R
[s1] 0
1
0 [s2] 1
0
1 [acc]
Create dominoes!
• There are 7 kinds of
dominoes we need.
– The first domino
represents the initial
configuration of the
TM, which is
#
#[s1] 0 1 #
[s1] 0
1
0 [s2] 1
0
1 [acc]
Dominoes (2,3)
• Second type of
domino represents
moves to the right. In
our case we see
[s1] 0
0 [s2]
[s1] 0
1
0 [s2] 1
0
1 [acc]
[s2] 1
1 [acc]
The third type of domino would represent moves going left, but
we don’t have any of these.
In general, right moves are qa/br and left moves are cqa/rcb.
Dominoes (4, 5)
• Allow ourselves to do
nothing to a tape
symbol away from the
read-write head.
0
0
1
1
[s1] 0
1
0 [s2] 1
0
1 [acc]
_
_
Also, allow use to put a
delimiter (#) between steps,
and put blanks on either
side of the input as needed.
#
#
#
#_
#
_#
Dominoes (6, 7)
• Allow ourselves to eat
the tape once we
accept! There will be
6 dominoes like this:
0 [acc]
[acc]
[s1] 0
1
0 [s2] 1
0
1 [acc]
[acc] 0
[acc]
And finally, we need a special
case domino to “eat” the accept
state notation on the tape. 
[acc] # #
#
CS 461 – Nov. 16
• Decidable vs. undecidable √
– The goal was to create a TM that always halts.
– Last undecidable problem: ambiguity of CFG
• Tractable vs. intractable
– Now, not only should it halt, we want an algorithm that
won’t take too long!
– Two kinds of problems “P” and “NP”
Ambiguous grammars
• If we can’t tell if PCP has a solution, then we can’t tell if a
CFG is ambiguous.
• The idea is to convert a domino game into a grammar
that has to be ambiguous exactly when PCP has a
solution.
• First, begin with an instance of a PCP problem, like we
saw before.
• Notice that we have a top & bottom. And we’re
wondering if we can get them to match! Create a
grammar for the top (A) and bottom (B) and say
SA|B
continued
• Our grammar will look like
SA|B
A  ______________________
B  ______________________
In our case, we have:
A  1Ai1 | 10111Ai2 | 10Ai3 |
1i1 | 10111i2 | 10i3
B  111Bi1 | 10Bi2 | 0Bi3 |
111i1 | 10i2 | 0i3
Derivations
• Let’s see how our solution 10111 1 1 10 is
derived from our A and B grammars.
• We wind up with these strings
10111 1 1 10 i3 i1 i1 i2
and
10 111 111 0 i3 i1 i1 i2
These strings are equal, so there are 2 derivations
 ambiguous grammar. Note that within A and
B it couldn’t be ambiguous, because each time
we lay a domino (and choose a grammar rule)
we are writing down a distinct “i” number.
Complexity
• Algorithms are often classified according to
complexity.
• Big-O notation is pretty common. General
worst-case behavior.
• Example: Sorting
–
–
–
–
–
Radix sort, O(n)
Merge sort, O(n log n)
Insertion sort, O(n2)
Stooge sort, O(2.7n)
Monkey sort, O(n!)
Good
Bad
Non-determinism
• An algorithm should be deterministic!
• Non-determinism is “cheating.”
– For sorting, non-determinism means pick the right arrangement of
elements, and verify it’s in order!
• Ex. Spell checking
– Deterministic, O(log n)
– Non-deterministic, O(1)
• Looks like non-determinism makes things easier, but the
trick is there’s no way to know what data to pick. Must try
them all, so deterministic algorithm often exponential.
P and NP
• P = Problems where  deterministic polynomialtime algorithm.
– “can quickly decide” (in the TM sense)
– The run time is O(n2) or O(n3), etc. 
• NP = Problems where  non-deterministic
polynomial-time algorithm.
– “can quickly verify”
– A deterministic algorithm would require exponential
time, which isn’t too helpful.
– (NP – P) consists of problems where we don’t know
of any deterministic polynomial algorithm.
CS 461 – Nov. 18
Section 7.1
• Overview of complexity issues
– “Can quickly decide” vs. “Can quickly verify”
• Measuring complexity
• Dividing decidable languages into complexity classes.
• Next, please finish sections 7.1 – 7.2:
– Algorithm complexity depends on what kind of TM
you use
– Formal definition of P, NP, NP-complete
Revisit:
P and NP
• P = Problems where  deterministic polynomialtime algorithm.
– “can quickly decide” (in the TM sense)
– The run time is O(n2) or O(n3), etc. 
• NP = Problems where  non-deterministic
polynomial-time algorithm.
– “can quickly verify”
– A deterministic algorithm would require exponential
time, which isn’t too helpful.
– (NP – P) consists of problems where we don’t know
of any deterministic polynomial algorithm.
Conjecture
• P and NP are distinct.
– Meaning that some NP problems are not in P.
– There are some problems that seem inherently
exponential.
• Major unsolved question!
– For each NP problem, try to find a deterministic
polynomial algorithm, so it can be reclassed as P.
– Or, prove that such an algorithm can’t exist. We don’t
know how to do this. Therefore, it’s still possible that
P = NP.
• Ex. Primality was recently shown to be in P.
Example
• Consider this problem: subset-sum. Given a set
S of integers and a number n, is there a subset
of S that adds up to n?
– If we’re given the subset, easy to check.  NP
– Nobody knows of a deterministic polynomial
algorithm.
• What about the complement?
– In other words, there is no subset with that sum.
– Seems even harder. Nobody knows of a nondeterministic algorithm to check. Seems like we need
to check all subsets and verify none adds up to n. 
– Another general unsolved problem: are complements
of NP problems also NP?
Graph examples
• The “clique” problem. Given a graph, does it
contain a subgraph that’s complete?
– Non-deterministically, we would be “given” the clique,
then verify that it’s complete.
– What is the complexity?
– (Complement of Clique: not known to be in NP.)
• Hamiltonian path: Given a graph, can we visit
each vertex exactly once?
– Non-deterministically, we’d be given the itinerary.
Inside NP
• There are generally 2 kinds of NP problems.
– Smaller category:
Problems where a deterministic polynomial algorithm
is lurking out there, and we’ll eventually find it.
– Larger category:
Problems that seem hopelessly exponential. When
you distill these problems, they all have the same
structure. If a polynomial solution exists for one, they
would all be solvable!
These problems are called NP-complete.
Examples
• Some graph problems
–
–
–
–
Finding the shortest path
Finding the cheapest network (spanning tree)
Hamiltonian and Euler cycles
Traveling salesman problem
• Why do similar sounding problems have vastly different
complexity?
CS 461 – Nov. 21
Sections 7.1 – 7.2
• Measuring complexity
• Dividing decidable languages into complexity classes.
• Algorithm complexity depends on what kind of TM you
use.
• Formal definition of “P”
• Next: Concepts of NP and NP-complete
Examples
• Some graph problems
–
–
–
–
Finding the shortest path
Finding the cheapest network (spanning tree)
Hamiltonian and Euler cycles
Traveling salesman problem
• Why do similar sounding problems have vastly different
complexity?
“O” review
• Order of magnitude upper bound
• Rationale: 1000 n2 is fundamentally less than n3, and is
in the same family as n2.
• Definition: f(n) is O(g(n)) if there exist integer constants
c and n0 such that f(n)  c g(n) for all n  n0.
– In other words: in the long run, f(n) can be bounded by g(n)
times a constant.
– e.g. 7n2 + 1 is O(n2) and also O(n3) but not O(n). O(n2) would
be a tight or more useful upper bound.
– e.g. Technically, 2n is O(3n) but 3n is not O(2n).
– log(n) is between 1 and n.
• Ordering of common complexities:
O(1), O(log n), O(n), O(n log n), O(n2), O(n3), O(2n), O(n!)
Measuring complexity
• Complexity can be defined:
– as a function of n, the input length
– It’s the number of Turing machine moves needed.
– We’re interested in order analysis, not exact count.
• E.g. About how many moves would we need to
recognize { 0n1n } ?
– Repeatedly cross of outermost 0 and 1.
– Traverse n 0’s, n 1’s twice, (n-1) 0’s twice, (n-1) 1’s twice, etc.
– The total number of moves is approximately:
3n + 4((n-1)+(n-2)+(n-3)+…+1) = 3n + 2n(n-1) = 2n2 + n ~ 2n2
– 2n2 steps for input size 2n  O(n2).
Complexity class
• We can classify the decidable languages.
• TIME(t(n)) = set of all languages that can be decided by
a TM with running time O(t(n)).
– { 0n 1n }  TIME(n2).
– 1*00*1(0+1)*  TIME(n).
– { 1n 2n 3n }  TIME(n2).
– CYK algorithm  TIME(n3).
– { ε, 0, 1101 }  TIME(1).
• Technically, you can also belong to a “worse” time
complexity class. L  TIME(n)  L  TIME(n2) .
•  It turns out that { 0n 1n }  TIME(n log n). (p. 252)
TM variants
• It turns out that the complexity depends on the variant!
– Ordinary one-tape TM.
– Multi-tape TM.
– Non-deterministic TM.
(for definition of TIME(t(n))
(useful for programming)
• E.g. { 0n 1n } can be decided in O(n) on a multi-tape TM.
– See page 253.
• In general, a multi-tape TM that runs in O(t(n)) can be
converted to a single-tape TM that runs in O((t(n)2).
– See page 254.
• In general, a non-deterministic TM that runs in O(t(n))
can be converted to an ordinary TM running in 2 O(t(n)).
– See page 256.
– This is why we say that NP problems are exponential. 
Multi-tape { 0n 1n }
• We have seen an O(n2) algorithm on a single-tape TM.
• And a better algorithm that runs in O(n log n), also with
single tape.
• With multiple tapes, we can do it in O(n).
–
–
–
–
–
Scan input to make sure of the form 0*1*.
Return to beginning.
Read 0’s and copy them to 2nd tape.
When you reach the first 1, start the 2nd tape’s head at the first 0.
Now we match 0’s and 1’s simultaneously.
• So, the problem is O(n), although technically it’s still in
TIME(n log n). Not a big deal in the long run.
Single vs. multi tape
O(t(n)) multi-tape  O(t(n)2) single tape. Proof:
• We simulate the multi-tape operation on single tape TM.
– Each of the k tapes is performing a computation that runs in t(n)
time. In other words, we take up to t(n) steps.
• Scan the tape, taking note of where virtual heads are.
• How long is the tape contents?
– Each virtual section of the tape could be t(n) symbols long,
because each step could write a symbol on the tape.
• Final calculation:
– Scanning the entire tape takes O(t(n)) time.
– And we have to scan the tape t(n) times!
– Multiply together to obtain O(t(n)2).
DTM vs. NTM
O(t(n)) NTM  2 O(t(n)) DTM. Proof:
• The NTM has a tree of possible computational futures.
• For input length n, the depth of tree is up to t(n).
• Each node of the tree can have up to c children.
– Total number of nodes in the tree is on the order of O(c t(n)).
• The time it takes to reach the appropriate node in the
tree is t(n)
– Multiply this by the number of nodes to obtain the upper bound
on the complexity: O(t(n) c t(n)).
– Claim: this is equivalent to 2 O(t(n)). Actually it’s greater but ok.
• Finally, need to convert multi-tape to single tape. But
squaring an exponential is still just an exponential.
Definition of P
• The union of all language classes of the form:
TIME(1)  TIME(n)  TIME(n2)  TIME(n3) …
• Thus, it’s all algorithms that can be decided by an
ordinary (one-tape, deterministic) TM in polynomial time.
– Even if we allowed multiple tapes, the time complexity would still
by polynomial, by our earlier theorem.
– We also generalize “TM” to computer program. When we speak
of a O(f(n)) algorithm, this is generally assumed. But it’s still
deterministic! So, it’s ok to say { 0n 1n } is O(n) problem.
• Many algorithms are in P. This is where we like to be.
– No recursion
– Nested loops ok.
CS 461 – Nov. 28
Sections 7.3 – 7.4
• Classes of problems “NP” and “NP-complete”
• We don’t know if P = NP
• Proving that  an NP-complete problem.
– And then showing that we have other problems of similar
complexity.
Beyond P
• We classify decidable problems by their complexity. √
• Some problems apparently have no polynomial-time
algorithm. In other words, they appear not to be in P.
• Most of these are in “NP”. They can be verified by DTM
in polynomial time. (Also: decided by NTM)
– Nobody knows if “Hamiltonian Path” problem is in P.
– But, given a path in a graph, we can easily verifying if it’s
Hamiltonian.
• Worse, some problems are apparently even outside NP.
– “Does this graph not have a Hamiltonian path?” What given info
could we use to verify? Brute force 
NP
• Up to 3 types of problems in NP
– P (the easy ones)
– NP-complete (the hard ones)
– NP-intermediate (we don’t know if any exist!)
• But, if P = NP, then the above 3 types collapse into 1
• NP-complete:
– Many NP problems are intrinsically related to one another. We
can prove so by constructing a polynomial time reduction. Thus,
if one of them turns out to be in P, they all are.
• Examples of NP problems:
– Hamiltonian, Clique & subset sum (p. 268)
– Isomorphic
NP-complete
• Formal definition. Q is NP-complete if:
– The problem is in NP
– Every other problem in NP can be reduced to Q in polynomial
time.
• Step 1 is usually straightforward.
• For step 2, we have to select some existing problem
already known to be NP-complete and show that it
corresponds to Q.
• But, from first principles, we have to show that first of all
there exists one NP-complete problem.
– Analogous to a base case in an inductive proof.
The SAT problem
• Given a boolean formula, is there some assignment of
truth values to the variables that makes the formula true?
– e.g. (p  q’  r’  s)  (p’  s)  (q  r’  s’)
– In general, there could be a lot of different variables used in a
boolean formula, so we give our variables subscripted names x1,
x2, x3, … And if there are lots of possible variables, we can even
see multiple subscripts like x16,40,1.
• Cook-Levin theorem: SAT is NP-complete
– We need to show that it’s NP, and that any other NP problem
can be reduced (in polynomial time) to SAT.
– SAT is in NP because if we are given a truth assignment of the
variables, we can determine if the boolean formula is true.
– The polynomial reduction is very interesting! (pp. 277-281)
Any NP  SAT
• We’ll show that any NP language/problem can be
reduced to SAT, and this reduction can be performed in
polynomial time.
• If A is any NP problem, then there exists a NTM that
decides A.
– Furthermore, it runs in polynomial time, so the number of steps
in its execution can be bounded by nk.
• One possible execution of this NTM can be expressed in
the form of a (nk x nk) table.
– Rows correspond to individual steps of the NTM computation,
and literally represent the ID’s/configurations of the state & tape,
along with blanks to fill up nk columns.
– Columns are for individual tape & state symbols on tape. First
and last columns are # delimiters.
NTM execution table
• The table corresponds to one possible run of the NTM.
It may represent an accepting computation or not.
• Although the table has nk rows, we may accept the input
before running that many steps.
• Each cell in the table contains a symbol in an ID, which
could be:
– Delimiter # on the far left & right ends of the table.
– State
– Tape symbol
• The NTM accepts w if there exists a table showing an
accepting configuration.
Creating 
• We now convert the table into a corresponding boolean
formula.
• The possible variables of our formula are of the form xi,j,s
where
– i is a row number in the table
– j is a column number in the table
– s is a symbol that may appear in cell [ i, j ], such as a state, tape
symbol or #.
• What does this variable mean? xi,j,s is “true” if cell [ i, j ]
contains the symbol s.
•  is created in such a way that  is true when the
corresponding execution table shows that the NTM
accepts w.

•  =  cell   start   accept   move
• In order for the table to represent an “accept”, 4
conditions must be satisfied.
•  cell says that for all cells in the table, there can only be
one symbol in that cell.  For all i and j, there is exactly
one s for which xi,j,s is “true”.
•  start says that the first row of the table contains the
initial configuration of the NTM, including the # delimiters
and blanks.
•  accept says that at some point we must reach an
accept state. One cell must contain qaccept.  For some
i and j, xi,j,q accept is true.
 continued
•  move says that the rows of the table represent legal
moves of the NTM.
– We specify this by examining 2x3 windows of the table.
– Reminiscent of our PCP dominoes.
– Some windows center on activity around the state/cursor; some
are far away from the state and don’t change from one row to the
next.
– All windows must be “legal” according to  and proper move
behavior.
• This reduction from the NTM to  can be done in
polynomial time. We have a polynomial number of
variables and terms to create. At most we need O(n2k)
terms for .
Other problems
• We now have 1 problem, “SAT”, proved to be
NP-complete by definition.
• To show that some other problem is NP-complete, all we
need to do is create a polynomial-time correspondence
with SAT, or other problem already proved to be NPcomplete.
– Not necessary to start over with arbitrary NTM.
• Examples:
– 3SAT: the satisfiability question assuming the boolean formula
is of the form (_  _  _)  (_  _  _)  (_  _  _)  …
i.e. CNF in which each factor has exactly 3 terms.
– Clique, subset-sum, isomorphic, Hamiltonian path, etc.
CS 461 – Nov. 30
Section 7.5
• How to show a problem is NP-complete
– Show it’s in NP.
– Show that it corresponds to another problem already known to
be NP-complete.
• Example NP-complete problems
– 3SAT, special case of SAT
– Clique
– Subset sum
Review:
Other problems
• We now have 1 problem, “SAT”, proved to be
NP-complete by definition.
• To show that some other problem is NP-complete, all we
need to do is create a polynomial-time correspondence
with SAT, or other problem already proved to be NPcomplete.
– Not necessary to start over with arbitrary NTM.
• Examples:
– 3SAT: the satisfiability question assuming the boolean formula
is of the form (_  _  _)  (_  _  _)  (_  _  _)  …
i.e. CNF in which each factor has exactly 3 terms.
– Clique, subset-sum, isomorphic, Hamiltonian path, etc.
3SAT
• Want to show this is NP-complete.
• Note that we can’t simply say “3SAT is a special case of
SAT, therefore it’s NP-complete.”
– Consider this special case of SAT: all boolean formulas
containing just one variable and no operators. Now the problem
is trivially in P.
– It’s possible that a “special case” of an NP-problem could
become simple enough to be in P. But in the case of 3SAT we
show it’s still NP-complete.
• Simple approach: re-do our proof of SAT, but this time
make sure each  formula we write is expressed:
– in CNF (conjunctive normal form)
– with exactly 3 terms per factor
3SAT in CNF
• The  we created for SAT is almost already in CNF.
– CNF means an AND of OR’s.
–  has 4 factors. See book for precise definitions.
– 3 of the 4 factors are in CNF. Only 4th factor move
needs modifying.
– move is more complex: an AND of OR’s of AND’s.
We can use DeMorgan’s Law to convert OR’s of
AND’s into AND’s of OR’s. Now we have CNF.
continued
• We also need exactly 3 terms per factor (_  _  _) .
– If fewer than 3, just repeat a term. e.g. (p  q  q)
– To reduce 4 terms to 3, we would do this:
(x1  x2  x3  x4) = (x1  x2  d)  (d’  x3  x4)
where “d” is a new dummy variable.
See why it works? Consider all xi false. Consider one xi true.
– For more than 4 terms the technique generalizes:
(x1  x2  x3  … xn) =
(x1  x2  d1)  (d1’  x3  d2)  (d2’  x4  d3)…
 (dn-3’  xn-1  xn)
Clique problem
• Given a graph G and a number k, does G contain a
complete subgraph of k vertices?
– Alternatively, given a graph G, find the size (k) of the largest
complete subgraph.
• This problem is in NP, because we could be given a
subgraph to consider, and we can verify in O(k2) time that
it’s complete by checking that all required edges are
present.
• To complete the proof that “Clique” is NP-complete, we
will give a polynomial-time reduction from 3SAT to clique.
– We already know that 3SAT is NP-complete.
– We must convert a boolean formula  into a graph G, such that
the  is true iff G has a clique.
3SAT  Clique
• We’re given , which has k factors, 3 terms per factor.
• Create the graph G as follows.
• The vertices are arranged in groups of 3.
– Each “triple” of vertices corresponds to a factor in 
– Label each vertex with corresponding term (e.g. x1’). Note that
the same label may appear in other triples.
• Edges connect all vertices except:
– Vertices within the same triple
– Vertices that are opposites of each other (as x1 and x1’)
• Why does this construction work?
– We’ll show:  is “true”  G has a clique with k vertices.
 True  clique
• Suppose  = true.
– One of the terms in each triple must be true.
– Looking at G, the vertices are grouped in k triples.
– There are lots of edges between triples. The only type of edge
between triples we don’t have is between contradictory terms
like x1 and x1’.
– A “true term” in a triple can always be connected to a “true term”
in another triple.
– Therefore, there is a clique of size k.
• Suppose G has a clique of size k.
– Each vertex in the clique must be in a different triple.
– Let’s assign truth values to each vertex to make each clique
vertex true. There will be no contradictions.
– Therefore, each triple will have one term that is true, so  is
true.
Summary
• Thus far we have shown:
– SAT is NP-complete
– Therefore, 3SAT is NP-complete
– Therefore, Clique is NP-complete
• Please watch this video lecture by Ron Graham:
RonaldLG1988.mpeg on the class Web site
• Next:
– Wrap up section 7.5
– Review
CS 461 – Dec. 2
Major results
• Automata: 3 models of computation
– Definitions, properties, corresponding language classes, text
representations
• Decidability
• Complexity: classes of decidable languages
– P: polynomial algorithm
– NP-complete: exponential algorithm & polynomial verifier
– NP-hard: exponential algorithm, no known polynomial verifier
Graham’s talk
• Minimum spanning tree vs. minimum Steiner tree
– Helpful lemmas don’t reduce exponential complexity
– NP-complete: “I can’t find an efficient algorithm, but neither can
all these famous people!”
• One common feature of NP-complete problems:
– A solution that’s almost right is no good.
– TSP tour that is 2 miles too long.
– Subset sum that might be off by 1.
• It may be good enough to
– approximate solution
• e.g. Unfortunately, sum of square roots can be difficult to compare!
– Restricted problem – special case
• Sometimes even the verifying requires exponential time.
Major results
• NFA = DFA
• FA = regular expression
• Can simplify FA’s (MyhillNerode theorem)
• Pumping lemmas
• PDA = CFG
• CYK algorithm O(n3) time
• Bottom-up parsing (linear
time)
• Variants of TM
• Countable, uncountable
• ATM undecidable
•  non Turing
recognizable language
• Rice’s theorem
• PCP undecidable
• SAT is NP-complete
• Many other NP-complete
problems
• We don’t know if P = NP
CS 461 – Dec. 5
Endgame
• Review questions today
• Study day (Wednesday)
– Conference room food/drinks/fun at 2:00
– I’m available afterward if you have questions
• As Carol Burnett would say, …
I’m so glad we had this time together
Just to have a laugh or sing a song.
Seems we just get started and before you know it
Comes the time we have to say so long.