Models of Computability
•Church's Thesis
•Primitive Recursive
•TM to Primitive Recursive?
•Primitive Recursive << TM
•TM to Recursive
•Register Machine
•And/Or/Not Circuits
•Computer Circuits
•Circuits Depth
•Arithmetic Circuits
•Neural Nets
•Poly-Time
•Non-Deterministic Machine
•Quantum Machine
•Context Free Grammars
•JAVA to Machine Code
(Compiling/Parsing)
•Context Sensitive Grammars
•TM to Grammar
•Decide vs Accept
•Humans
•Complexity Classes
Jeff Edmonds
York University
COSC 4111
Lecture 1
Please feel free
to ask questions!
Please give me feedback
so that I can better serve
you.
Thanks for the feedback
that you have given me
already.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
•by a (simple) Register Machine
NonUniform
•by a Circuit: And/Or/Not, Computer,
 unreasonable
Arithmetic, & Neural Nets.
Uniform
 reasonable
Computes
the same,
•by a Non-Deterministic Machine
but faster.
Computes the same,
•by a Quantum Machine
but faster.
Decide vs Accept
•by a Context Sensitive Grammar
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
A Java program can
easily simulate each
•by a Turing Machine
of them.
•by a (simple) Recursive Program
•by a (simple) Register Machine
•by a Circuit: And/Or/Not, Computer,
Arithmetic, & Neural Nets.
•by a Non-Deterministic Machine
•by a Quantum Machine
•by a Context Sensitive Grammar
Church’s Thesis
Proof
A computational problem is computable
A Java program
•by a Java Program
can be simulated by
A TM
•by a Turing Machine
a TM.
can be simulated by
•by a (simple) Recursive Program Aa Recursive
Recursive Program
Program.
can be simulated by a
•by a (simple) Register Machine
Register
Machine.
A TM
•by a Circuit: And/Or/Not, Computer,can be simulated by a
Non-Deterministic
Circuit.
Arithmetic, & Neural Nets.
Machine
•by a Non-Deterministic Machine can be simulated by a
ADeterministic
Quantum
A TMMachine
one.
canbe
besimulated
simulatedby
byaa
can
•by a Quantum Machine
Deterministic
one.
Context
Sensitive
•by a Context Sensitive Grammar
Grammar.
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
•by a (simple) Register Machine
•by a Circuit: And/Or/Not, Computer,
Arithmetic, & Neural Nets.
•by a Non-Deterministic Machine
•by a Quantum Machine
•by a Context Sensitive Grammar
Note you can get
anywhere from
anywhere.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
Define
•by a (simple) Recursive Program
MULT(X,Y):
Recursion
Friends & Strong Induction
If |X| = |Y| = 1 then RETURN XY
Break X into a,b and Y into c,d
e = MULT(a,c) and f =MULT(b,d)
RETURN
e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f
X = 33
Y = 12
ac = 3
bd = 6
(a+b)(c+d) = 18
XY = 396
X=3
Y=1
XY = 3
X=3
Y=2
XY = 6
X=6
Y=3
XY = 18
•Consider your input instance
•Allocate work
•Construct one or more sub-instances
• It must be smaller
• and meet the precondition
•Assume by magic your friends give you the
answer for these.
• Don’t micro-manage them by tracing
out what they and their friends friends
do.
•Use this help to solve your own instance.
•Do not worry about anything else.
•Who your boss is…
•How your friends solve their instance…
Primitive Recursive
We will now define two models of computation:
• Primitive Recursion Functions and
• μ-Recursive Functions
• They were developed about the same time as
Turing Machines
• Like TMs, the allowed set of allowed operations
is very restricted.
• It was a big historical deal when it was prove that
The Class of μ-Recursive Problems
= The Class of Turing Computable Problems
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Smaller instance (very specific)
Help from friend
My solution
Base case
Does not change. Could be x1,x2,..,xn.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
f(x,y) could be computed iteratively:
yˈ = 0
f = g(x)
loop Loop Invariant:
fˈ= f(x,yˈ)
exit when yˈ=y
yˈ = yˈ+1
fˈ = h(x,yˈ-1,f’)
end loop
return(fˈ)
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
f is defined from g & h by composition if
•f(x,y) = g(h1(x,y),…,hn(x,y))
Initial functions:
•Zero(x) = 0
•Successor(x) = x+1
•In,i(x1,x2,..,xn ) = xi
Ex
•If you have defined g(y,x)
and want f(x,y) = g(y,x)
•f(x,y) = g(I2,2(x,y),I2,1(x,y))
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
f is defined from g & h by composition if
•f(x,y) = g(h1(x,y),…,hn(x,y))
Initial functions:
•Zero(x) = 0
•Successor(x) = x+1
•In,i(x1,x2,..,xn ) = xi
No really dude.
If it is not on this page,
you can’t do it.
And you want me to be able to
compute everything?
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
f is defined from g & h by composition if
•f(x,y) = g(h1(x,y),…,hn(x,y))
There is a beauty to the
simplicity of these programs.
Initial functions:
Each routine f() is defined
•Zero(x) = 0
by specifying
•Successor(x) = x+1
• not lots of fancy code, but only
•In,i(x1,x2,..,xn ) = xi
• the number of arguments
f(x1,x2,..,xn,y)
• and which functions
g & h to call.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
x+y = [x+(y-1)] +1
x+0 = x
Get help from your friend.
My solution from his.
This is not formal
enough.
Ok.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Sum(x,y) = h( x,y-1,Sum(x,y-1) )
Sum(x,0) = g(x)
•We define a new primitive recursive function Sum
•from a previously define h & g.
•h has arguments:
• All the other variables x left unchanged.
• y one smaller.
• Recursive solution on input with y one smaller.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Sum(x,y) = h( x,y-1,Sum(x,y-1) )
Sum(x,0) = g(x)
•We define a new primitive recursive function Sum
•from a previously define ones h & g.
•When y is zero, Sum is defined from
• g on all the other variables x left unchanged.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Sum(x,y) = h( x,y-1,Sum(x,y-1) )
= Successor( I3,3( x,y-1,Sum(x,y-1)) )
= Sum(x,y-1) + 1
•We want to use the recursive solution
•But it must be extracted from the tuple
•using the I function.
•Then we add one to this recursive solution.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Sum(x,y) = h( x,y-1,Sum(x,y-1) )
= Successor( I3,3( x,y-1,Sum(x,y-1)) )
= Sum(x,y-1) + 1
Sum(x,0) = g(x)
= I1,1(x)
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
x×y = [x×(y-1)] +x
x×0 = 0
Get help from your friend.
My solution from his.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Prod(x,y) = h( x,y-1,Prod(x,y-1) )
= Sum(
I3,3( x,y-1,Prod(x,y-1)),
I3,1( x,y-1,Prod(x,y-1)) )
= Sum( Prod(x,y-1), x )
•Now we define Prod from previously defined Sum.
•Sum has two arguments
Extracted from tuple
• Prod(x,y-1)
• x
Extracted from tuple
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
xy =
[xy-1]
x0 = 1
×x
Get help from your friend.
My solution from his.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
log(16) = 4
because
log(20) = 4
because
24 = 16
first power of 2 below it is 16.
20/2=10, 10/2=5, 5/2=2, 2/2=1
log(20)=4
log(y) = # of time to divide by 2 until you get 1
log(y/2) = one less
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
log(y) = log(y/2) +1
= log( div(y,2) ) +1
Get help from your friend.
My solution from his.
This is true. But is it
formally Primitive
Recursive?
Sure!
We showed y/2 is prim. rec.
So we use it to show log is.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
log(y) = log(y/2) +1
= log( div(y,2) ) +1
Get help from your friend.
My solution from his.
You must use
f(y) = h(y-1,f(y-1))
You must give your
friend <y-1>.
Oops
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Is this legal?
f(x,y) = h(x/2,y+5,f(x,y-1))
Not officially.
But you can legally define
h’(x,y,c) = h(x/2,y+5,c)
and then define
f(x,y) = h’(x,y,f(x,y-1))
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Now a hard one.
How do you make
Examples:
something smaller?
Predecessor(y)
= Predecessor(y-1)+1 Get help from your friend.
My solution from his.
This looks fine.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x) This does not make anything smaller,
but defers to its friend to makes
something else smaller.
Predecessor(y) What is the first thing to get smaller?
= Predecessor(y-1)+1 Get help from your friend.
My solution from his.
Examples:
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
But what is the base case?
Examples:
Predecessor(y)
= Predecessor(y-1)+1 Get help from your friend.
My solution from his.
Predecessor(0) = -1
Bug: Negative values are not defined!
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
But what is the base case?
Examples:
Predecessor(y)
= Predecessor(y-1)+1
Predecessor(0) = -1 0
Predecessor(2) = 1,
Predecessor(1) = 0,
Predecessor(0) = 0.
Get help from your friend.
My solution from his.
Ok, as a special case,
we will set it to zero.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
What about y=1?
Examples:
Predecessor(y)
= Predecessor(y-1)+1
Predecessor(0) = -1 0
Get help from your friend.
My solution from his.
Predecessor(1) = Predecessor(0)+1
=
0
+1  0
We need more cases.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Predecessor(y)
= h(y-1,Pred(y-1)) =
Predecessor(0) = 0
Pred+1
if y > 1
0
if y = 1
Bug: “if “ and “equality” are not defined.
Oh dear! I don’t know.
It seems impossible.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Lets start back at the formal definition
Predecessor(y) =h(x,y-1,f(x,y-1))
That is what we want.
h(x,y-1,f) = I3,2(x,y-1,f)
Predecessor(0) = g(x) = 0
Yay!
That does it.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Lets start back at the formal definition
Predecessor(y) =h(x,y-1,f(x,y-1))
That is what we want.
h(x,y-1,f) = I3,2(x,y-1,f)
Predecessor(0) = g(x) = 0
Actually, to avoid negative values,
the actual definition is
•f(x, Successor(y)) = h(x,y,f(x,y))
I changed it because I thought it looked better.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
x-y =
x-0 = x
[x-(y-1)] -1
Get help from your friend.
My solution from his.
3-5 = Pred(3-4)
= Pred(Pred(Pred(Pred(Pred(3-0)))))
= Pred(Pred(Pred(Pred(Pred(3)))))
= Pred(Pred(0))
=0
Fair enough.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
max(x,y) = (x-y)+y
(8-5)+5 = (3)+5 = 8
(5-8)+8 = (0)+8 = 8
min(x,y) = x+y-max(x,y)
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Truth(y) =
0 if y=0
1 else
f(x,0) = g(x) = 0
f(x,y) = h(x,y-1,f(x,y-1)) = 1
Here true≥1 and false=0.
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Truth(y) =
0 if y=0
1 else
Here true≥1 and false=0.
Neg(R) = 1-Truth(R)
(R and S) = min(R,S)
(R or S) = max(R,S)
(x>y)
= x-y
(x=y)
= Neg(x>y) and Neg(y>x)
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
Select(x) =
5 if x = 0
3 if x = 1
7 if x = 2
…
4 if x = r
=
5×(x=0)
+ 3×(x=1)
+ 7×(x=2)
+…
+ 4×(x=r)
3 = Successor(Successor(Successor(Zero)))
Primitive Recursive
mod(y,x) = modx(y) = Remainder(y/x)
eg mod(12,5) = 2
This is a hugely important in
math & computer science
In field mod 5, the universe has 5 “numbers” {0,1,2,3,4}
Think of them as equivalence classes
… -8 =mod 5 -3 =mod 5 2 =mod 5 7 …
Two operations + and ×
3+4 = 7 =mod 5 2
3×4 = 12 =mod 5 2
Additive inverse
-3 = 2 because -3+2 =mod 5 0
Multiplicative Inverse
½ = 3 because 2×3=mod 5 1
Primitive Recursive
mod(y,x) = modx(y) = Remainder(y/x)
eg mod(8,3) = 2
Counting Mod 3
1
2
SuccModx(r) =
0
1
2
1 if r = 0
2 if r = 2
3 if r = 3
…
x-1 if r= x-2
0 if r= x-1
0
1
2
= (r+1)×(r=x)
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Get help from your friend.
Examples:
My solution from his.
modx(y) = SuccModx( modx(y-1) )
modx(0) = 0
SuccModx(r) =
1 if r = 0
2 if r = 2
3 if r = 3
…
x-1 if r= x-2
0 if r= x-1
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
y/x =
13/5 = 2
10/5 = 2
[(y-1)/x]
[(y-1)/x] +1
12/5 = 2
9/5 = 1
Get help from your friend.
My solution from his.
When does each occur?
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Examples:
y/x =
13/5 = 2
10/5 = 2
0/x = 0
[(y-1)/x]
[(y-1)/x] +1
12/5 = 2
9/5 = 1
else
if modx(y)=0
When does each occur?
Primitive Recursive
mod(y,x) = modx(y) = Remainder(y/x)
eg mod(8,3) = 2 8/3 = 2
Counting Mod 3
# people/3 =
The number of time go
back to zero.
1
1
2
2
0
1
2
0
1
2
Primitive Recursive
How do I check if 11 is prime?
Make sure r = 2,3,4,…,10
don’t divide into it,
I can do each with
mod(11,r) ≠ 0
Lets call such an r good.
2
3
4
5
6
7
8
9
10
Primitive Recursive
How do I know all these n guys are good?
Get help from your friend.
The first n-1 are good.
My solution from his.
Ok they must all be good.
2
3
4
5
6
7
8
9
I’m good
10
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Get help from your friend.
Examples:
My solution from his.
AllGood(x,y) = r ≤ y Good(x,r)
= AllGood(x,y-1) and Good(x,y)
AllGood(x,0) = Good(x,0)
Prime(x) = r ≤ x-1 mod(x,r) ≠0 or r{0,1}
= AllGood(x,x-1)
where Good(x,r) = [mod(x,r) ≠0 or r{0,1}]
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Get help from your friend.
Examples:
My solution from his.
ExistsGood(x,y) =  r ≤ y Good(x,r)
= ExitsGood(x,y-1) or Good(x,y)
ExistsGood(x,0) = Good(x,0)
PowerOf2(x) = yes for 1,2,4,8,16,32,…
=  r ≤ x 2r=x
= ExistsGood(x,x)
where Good(x,r) = [2r=x]
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Get help from your friend.
Examples:
My solution from his.
NumGood(x,y) = |{ r≤ y | Good(x,r) }|
= NumGood(x,y-1) + Good(x,y)
NumGood(x,0) = Good(x,0)
or because
recursively
=log(8)+1
log(16) = 4 because
= 16
log(20) = 4 because first power of 2 below it is 16.
24
How many powers of 2 are below? |{1,2,4,8,16}=5
log(2) =1 log(1) =0
one more than log(20)
Primitive Recursive
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Get help from your friend.
Examples:
My solution from his.
NumGood(x,y) = |{ r≤ y | Good(x,r) }|
= NumGood(x,y-1) + Good(x,y)
NumGood(x,0) = Good(x,0)
log(x) = |{r ≤ x | PowerOf2(r) and r≠1 }|
= NumGood(x,x)
where Good(x,r) = PowerOf2(r) and r≠1
Home work
to do starting now
4.
Church’s Thesis
Proof
A computational problem is computable
•by a Turing Machine
•by a (simple) Recursive Program
Primitive Recursive Programs
seem to be able to compute a lot.
But can they compute everything a TM can?
The input to a TM M is a binary string Istring.
The input to a PR P is an integer
Iinteger.
Istring is the binary representation of Iinteger.
Church’s Thesis
Proof
A computational problem is computable
•by a Turing Machine
•by a (simple) Recursive Program
To prove this,
we must do a simulation of a TM computation
by a primitive recursive program.
2)
TM to Primitive Recursive
•Consider some computational problem P(I)
that is computable by some TM M.
•We must construct a recursive program computing P(I).
•Consider some input I.
•Consider some time step t.
•Consider the configuration of M on I at time t.
q
1
0
1
0
0
1
0
•We want to encode this as the configuration of
•A recursive program at time t,
•i.e. as a tuple <q,tape,head> of whole numbers.
TM to Primitive Recursive
Consider some configuration of a TM
q
1
0
1
0
0
1
0
Grows this way.
Grows this way.
•Let tape = 10100102
be the binary number with the bits on the tape.
•Flip around the tape.
TM to Primitive Recursive
Consider some configuration of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Let tape = 01001012
be the binary number with the bits on the tape.
•Let head = 00001002
be the binary number with a one only where the head is.
•<q,tape,head> specifies the current configuration of the TM.
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Let NextQ(q,tape,head) = q’
NextTape(q,tape,head) = tape’
NextHead(q,tape,head) = head’
giving that <q’,tape’,head’> is the next configuration
that the TM will be in.
•We need to show that these functions are primitive recursive.
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
c1
0
1
= tape
0
0
0
0
1
0
0
= head
•rightTape = Contents 01001 of tape starting at head
= tape/head
•Char c under head = Remainder( rightTape/2 )
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
c1
0
1
= tape
0
0
0
0
1
0
0
= head
•TM specifies the transition function δ(q,c) = <q’,c’,direction
•Note that q and c both are from a finite range.
•Hence δ can be computed using a primitive recursive select.
Select(x) =
5 if x = 0
3 if x = 1
7 if x = 2
…
4 if x = r
=
5×(x=0)
+ 3×(x=1)
+ 7×(x=2)
+…
+ 4×(x=r)
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
c1
0
1
= tape
0
0
0
0
1
0
0
= head
•TM specifies the transition function δ(q,c) = <q’,c’,direction
•Note that q and c both are from a finite range.
•Hence δ can be computed using a primitive recursive select.
•Hence NextQ(q,tape,head) = q’
NextC(q,tape,head) = c’
NextDirection(q,tape,head) = direction
are primitive recursive.
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•NextTape(q,tape,head)
tape + head if c=0 & c’=1
if c=1 & c’=0
= tape - head
if c=c’
tape
•NextHead(q,tape,head)
if direction = right
= head×2
if direction = left
head/2
are also primitive recursive
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Let Config(x,y) = <q,tape,head> be the configuration
that the TM will be in on input x after y time steps.
Config(x,y) = Next( Config(x,y-1)) Get help from your friend.
My solution from his.
Config(x,0) = <qstart,x,1> is the configuration at time zero.
Note a TM is designed to stay in the same configuration
once it reaches a halting state qhalt.
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Let Output(x,t) be the output of the TM on input x
if it halts within t time steps.
Note a TM is designed to have the output
on the tape when it halts.
Output(x,t) = Tape(Config(x,t)) if TM has halted by time t
“Has not halted” else
Careful!
•We are done because everything is primitive recursive!
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
•Let Output(x) =
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
the output of the TM
∞
•We need this to be primitive recursive!
Is it?
if it halts.
else
TM to Primitive Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Suppose the TM is known to halt in time at most
2n
t ≤ 2x = 2 where n = log2 x = size(x)
•Then Output(x) = Output(x,2x)
which is primitive recursive.
Church’s Thesis
Proof
A computational problem is computable
•by a Turing Machine in double exponential time
Primitive
•by a (simple) Recursive Program
•Comparing these models:
•Is there a danger a primitive recursive program
will run forever on an input?
No!
•Is there a danger a TM
Definitely
will run forever on an input?
•Is it possible that they
have the same computing power? No!
Church’s Thesis
Proof
A computational problem is computable
•by a Turing Machine
Primitive
•by a (simple) Recursive Program
•Comparing these models:
• Having a TM run forever is a bad thing.
But being able to compute as long as it needs
to is a good thing.
• There are computational problems computable
by TMs but not by primitive recursive
programs.
Church’s Thesis
A computational problem is computable
•by a Turing Machine
Primitive
•by a (simple) Recursive Program
•Comparing these models:
•Let us bound the running time of
a given primitive recursive program.
For this we need
Ackermann’s function.
Proof
Ackermann’s Function
How big is A(5,5)?
Define Ak (n)  A(k,n)
i.e. a different function Ak for each k .
Base Case :
Proof by induction on k , that
Ak ( 0 )  Ak ( 0 )
A0(n)  2  n
Ak (n)  Ak-1( Ak-1( Ak-1( Ak-1(  Ak-1( Ak ( 0 ) ))))) Inductive
Step :
n applications
Ak (n)  Ak-1( Ak (n  1 ) )
Ackermann’s Function
Proof by induction on k , that
Ak (n)  Ak-1( Ak-1( Ak-1( Ak-1(  Ak-1( Ak ( 0 ) )))))
n applications
A0(n)  2  n
A1(n) 2  2  2  2   T1( 0 )  2  n
n applications
Ackermann’s Function
Proof by induction on k , that
Ak (n)  Ak-1( Ak-1( Ak-1( Ak-1(  Ak-1( Ak ( 0 ) )))))
n applications
A0(n)  2  n
A1(n)  2  n
A2(n) 2  2  2  2  A1( 0 )  2 n
n applications
Ackermann’s Function
Proof by induction on k , that
Ak (n)  Ak-1( Ak-1( Ak-1( Ak-1(  Ak-1( Ak ( 0 ) )))))
n applications
A0(n)  2  n
A1(n)  2  n
A2(n)  2n
A3(n) 
A4(n) 
Ackermann’s Function
A3(n)
A4(0)
A4(1)
Ackermann’s Function
A3(n)
A4(1)
A4(1)
Ackermann’s Function
Ackermann’s Function
Ackermann’s Function
Ackermann’s Function
For which k can Ackermann’s Ak(y) be computed?
A0(y)  2  y
A1(y)  2  y
R0(y) = succ(succ(y))
= 2+2×(n-1)
R1(y) = R0( R1(y-1) )
Get help from your friend.
My solution from his.
Ackermann’s Function
For which k can Ackermann’s Ak(y) be computed?
A0(y)  2  y
A1(y)  2  y
A2(y)  2 y
R0(y) = succ(succ(y))
= 2+2×(n-1)
= 2×2n-1
R1(y) = R0( R1(y-1) )
R2(y) = R1( R2(y-1) )
Get help from your friend.
My solution from his.
Ackermann’s Function
For which k can Ackermann’s Ak(y) be computed?
A0(y)  2  y
A1(y)  2  y
A2(y)  2 y
R0(y) = succ(succ(y))
= 2+2×(n-1)
= 2×2n-1
A3(y) 
=2
R1(y) = R0( R1(y-1) )
R2(y) = R1( R2(y-1) )
R3(y) = R2( R3(y-1) )
-1
Get help from your friend.
My solution from his.
Ackermann’s Function
For which k can Ackermann’s Ak(y) be computed?
R0(y) = succ(succ(y))
Get help from your friend.
My solution from his.
Proof by induction on k , that
Ak (y)  Ak-1( Ak-1( Ak-1( Ak-1(  Ak-1( Ak ( 0 ) )))))
]
 Ak-1 Ak-1( Ak-1( Ak-1( Ak-1( Ak( 0 ) ))))
y-1 applications
R3(y) = R2( R3(y-1) )
Rk(y) = Rk-1( Rk(y-1) )
y applications
[
R1(y) = R0( R1(y-1) )
R2(y) = R1( R2(y-1) )
Ackermann’s Function
For which k can Ackermann’s Ak(y) be computed?
Hence (by induction)
For each k, Ackermann’s Ak(y)
is computed by the primitive
recursive program Rk(y)
k,  PR Rk, y Rk (y) = Ak(y)
R0(y) = succ(succ(y))
R1(y) = R0( R1(y-1) )
R2(y) = R1( R2(y-1) )
R3(y) = R2( R3(y-1) )
Rk(y) = Rk-1( Rk(y-1) )
This is a different program for each k!
Is there one PR program R(k,y) that works for all k?
 PR R, y,k R (k,y) = A(k,y)
How does the complexity of
Sorry no.
Rk increase with k in a way
Applications of Prim Recur that cant grow with input size?
Constant vs Finite
Resources
Grow with Input
Constant Resources
Fixed at Compile Time
Java: • # Lines of code
•
•
# & Range of Variables
Instruction Set
(Computed in One Step)
TM: • # of States
•
•
PR: •
•
•
(# Bits on Black Board)
Tape Alphabet
State Transitions
(Computed in One Step)
Will Halt
# Lines of code
Time is
# of Variables
Function Names Bounded
Mult(x,y), Sum(x,y), Succ(y)
•
Applications of Prim Recur
•
•
Allocated memory
Running time
(Might not Halt)
•
•
# Cells used
Running time
(Might not Halt)
•
•
Range of Variables
Running time
x+y
x+(y-1)
x+(y-2)
….
x+0
Depth of Rec Tree
First Order Logic
Model of Computation: Turing Machine
q
q’
k,  TM M, x,yk
M multiplies xy in one time step.
Yes. We showed you can.
The number of states needed grows as a
function of k.
First Order Logic
Model of Computation: Turing Machine
q
q’
 TM M, x,y
M multiplies xy in one time step.
No, it cannot remember arbitrary
integers in its states.
The number of states needed grows as a
function of x & y.
First Order Logic
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
k,  PR Rk, n Rk (n) = Ackk(n)
Yes. We showed you can.
It needs k applications
of primitive recursion.
First Order Logic
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
 PR R, k,n R(k,n) = Ack(k,n)
No, it cannot have a growing number of
applications of primitive recursion.
Ackermann’s Function
•Claim: For each k, Ak(y) can be computed
with primitive recursive Rk(y)
with k applications of primitive recursion.
•Proof:
• True for k = 0, because R0(y) = y+2
• Assume by way of induction that it is true for k-1.
Rk(y) = Rk-1( Rk(y-1)). Get help from your friend.
My solution from his.
Ackermann’s Function
•Claim: No primitive recursive
with k applications of primitive recursion
grows as fast as than Ak+1(y).
•Proof: By seeing that this is tight.
To design the fastest growing Fastk(y)
with k applications of primitive recursion:
• First let Fastk-1(y) be the fastest growing
with k-1 applications of primitive recursion:
• By induction assume Fastk-1(y)≈Ak-1(y).
• Make Fastk(y) as big as you can.
Fastk(y) = Fastk-1( Fastk-1( Fastk-1( Fastk(y-1)
Get help from your friend.
Gives Ak(y).
Gives Ak(3y) << Ak+1(y)
My solution from his.
Ackermann’s Function
•Claim: A(k,n) is not primitive recursive.
•Proof: Suppose it was by program R(k,n).
•Let kR denote the number of applications of
primitive recursion in R(k,n).
•For k>kR, A(k,n) grows much faster than R(k,n).
•Contradiction.
First Order Logic
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
k,  PR Rk, n Rk (n) = Ackk(n)
Yes. We showed you can.
It needs k applications
of primitive recursion.
First Order Logic
f is defined from g & h by primitive recursion if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
 PR R, k,n R(k,n) = Ack(k,n)
No, it cannot have a growing number of
applications of primitive recursion.
Church’s Thesis
Proof
A computational problem is computable
•by a Turing Machine
Primitive
•by a (simple) Recursive Program
•There are computational problems computable
by TMs but not by primitive recursive programs.
•We need to give primitive recursive programs more power.
•For one, it needs to be given the possibility of
running as long as it wants to.
(at the danger of running forever.)
Recursive
•Any primitive recursive function f is recursive.
But we need more.
Recursive
Let μ denote the least number operator.
If g is recursive then so is
f(x) = μy [g(x,y)] computed as follows.
procedure f(x)
for y = 0..∞
if g(x,y) = 0
return(y)
Note if
• y g(x,y) ≠ 0
• or first a y is tried for which g(x,y) does not halt.
(denoted g(x,y) = ∞)
Then this code does not halt
(denoted f(x) = ∞)
Recursive
Let μ denote the least number operator.
If g is recursive then so is
f(x) = μy [g(x,y)] computed as follows.
procedure f(x)
for y = 0..∞
if g(x,y) = 0
return(y)
=
the least number y such that
•g(x,y) = 0
• y’<y g(x,y’) ≠ ∞
∞ if no such y exists
TM to Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Recall Config(x,t) = <q,tape,head> is the configuration
that the TM will be in on input x after t time steps.
•Let HaltTime(x) = t be the time at which the TM will halt.
HaltTime(x) = μt HasHalted( Config(x,t) )
HasHalted(<q,tape,head>) = 0
iff this is a halting configuration.
TM to Recursive
Consider some configuration <q,tape,head> of a TM
q
0
1
0
0
1
0
1
= tape
0
0
0
0
1
0
0
= head
•Recall Config(x,t) = <q,tape,head> is the configuration
that the TM will be in on input x after t time steps.
•Let HaltTime(x) = t be the time at which the TM will halt.
•Recall Output(x,t) is the output of the TM on input x
if it halts after t time steps.
and Output(x) is the output of the TM on input x.
Output(x) = Output(x,HaltTime(x))
Proving that this is Recursive!
Church’s Thesis
A computational problem is computable
•by a Turing Machine
Done
•by a (simple) Recursive Program
Define and prove this now.
•by a (simple) Register Machine
Proof
Register Machine Model
Hilbert’s 10th problem was
is there an algorithm to find an integer solution
to a set (or even single) polynomial equation.
Register Machines were designed to perhaps be better than
Turing Machines at this task.
Contents of
Memory Cell
Number of
Memory Cells
Turing Machines
constant size
(0/1)
arbitrarily
many
Register Machines
arbitrarily
large integer
constant number
(10)
Fixed size
as input grows
 c,  I,
Size grows
with input
 I,  c,
Register Machine Model
Model of Computation: Register Machine
•The model has a constant number of registers
that can take on any integer ≥ 0.
•A program is specified by code made from
the following commands:
•Ri = 0
•Ri = Ri+1
•if Ri=Rj goto line k
Surely this is not enough!
How do you decrement?
Register Machine Model
1.
2.
3.
4.
5.
Copy: Ri = Rj
Ri
Rj
Ri = 0
if Ri=Rj goto line 5
Ri = Ri+1
if Ri=Ri goto line 2
done
Register Machine Model
1.
2.
3.
4.
5.
6.
7.
Add: Ri = Rj + Rk
Ri
Rj
Rk Rk’
Ri = Rj
Rk’ = 0
if Rk’=Rk goto line 7
Ri = Ri+1
Rk’ = Rk’+1
goto line 3
done
Register Machine Model
1.
2.
3.
4.
5.
6.
7.
Subtract: Ri = Rj - Rk
Ri
Rj Rj’
Rk
Ri = 0
Rj’ = Rk
if Rj’=Rj goto line 7
Ri = Ri+1
Rj’ = Rj’+1
goto line 3
done
Register Machine Model
subtract(Rj,Rk)
1. Ri = 0
Subroutine Call:
2. Rj’ = Rk
Ri = subtract(x,y)
3. if Rj’=Rj goto line 7
Solutions:
4. Ri = Ri+1
1. Copy the code in every
5. Rj’ = Rj’+1
where you use it.
6. goto line 3
2. Ri = 0
7. done
•Rj = 0
8. Rreturn = 5 goto line 12
•if Ri=Rj goto line k
3. But how do you where to return to?
• Before calling save a pointer
where to return to
10. Rreturn = 5
11. goto line 1
Recursive to Register Machine
Recursive
Register Machine
•Zero(x) = 0
•Successor(x) = x+1
•In,i(x1,x2,..,xn ) = xi
•f(x,y) = g(h1(x,y),…,hn(x,y))
Ri = 0
Ri = Ri+1
Rk = Ri
Use results as inputs
Primitive recursion
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
f = g(x)
for y’=1..y
f = h(x,y’-1,f)
return(f)
Least Number Operator
f(x) = μy [g(x,y)]
for y = 0..∞
if g(x,y) = 0
return(y)
Church’s Thesis
A computational problem is computable
•by a Turing Machine
•by a (simple) Recursive Program
Done
•by a (simple) Register Machine
Proof
Home work
to do starting now
Home work
to do starting now
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by an And/Or/Not Circuit
Proof
And/Or/Not Circuits
•A circuit is a directed acyclic graph of and/or/not gates
•An input X assigns a bit to each incoming wire.
x1 0
x2 1
x30
AND
AND
0
OR
0
1
NOT
OR
0
0
OR
0
The bits percolate
down to the output wires.
And/Or/Not Circuits
•Key differences in this model.
A given TM
A given circuit
• can take an input with
• has a fixed number n of
an arbitrarily large
input wires an hence can only
number of bits.
take inputs with this number
of bits.
And/Or/Not Circuits
•Key differences in this model.
A given TM
A given circuit
• has a fixed finite
• For each n, the circuit Cn
description
has a finite description,
but in order handle
inputs of all size,
we need a different one
for each n, C1, C2, C3, ….
And/Or/Not Circuits
•Key differences in this model.
A given TM
A given circuit
• has a fixed finite
• Allowing it to have
description
an infinite description
C1, C2, C3, …. is not fair.
• To make it fair, each Cn
should be constructed in
a Uniform way.
And/Or/Not Circuits
•Key differences in this model.
A given TM
A given circuit
• the contents of the tape
• the 0/1 value on
and location of the head
each wire (input & internal)
depend on the input
depend on the input
(and on the time step)
(but not on time)
and hence are not known
at “compile time”.
And/Or/Not Circuits
•Key differences in this model.
A given TM
A given circuit
• the contents of the tape
• the gates and
and location of the head
their connections
depend on the input
must be fixed at
(and on the time step)
“compile time”
and hence are not known
at “compile time”.
And/Or/Not Circuits
•Clearly circuits compute.
• Any function f(X) of n bits can be computed
with a nonuniform circuit of size O(2n).
2n
n
X
f(X)
000000
0
000001
1
000010
0
000011
0
000100
1

= Cn
f(x)
And/Or/Not Circuits
¬x1¬x2 ¬x3 …xn ¬x1 ¬x2 ¬x3 x4 … ¬xn
AND
AND
Outputs 1
iff
X = 000001
Outputs 1
iff
X = 000100
2n
n
X
f(X)
000000
0
000001
1
000010
0
000011
0
000100
1

…
x1 x2 ¬x3 … ¬xn
AND
Outputs 1
iff
X = 110..0
Repeat this
for every value of X
for which f(X)=1.
And/Or/Not Circuits
¬x1¬x2 ¬x3 …xn ¬x1 ¬x2 ¬x3 x4 … ¬xn
AND
AND
…
OR
Outputs 1 iff f(X)=1.
x1 x2 ¬x3 … ¬xn
AND
History of Computability
Computable
•Which computational problems
are computable by such circuits?
•Unreasonable because for each n,
a separate circuit C1, C2, C3, ….
of size 2n needs to be defined.
•This is big enough to list all the answers.
This “algorithm” does not have finite size.
•An algorithm that needs something different
specified for each n is called “nonuniform”.
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by a Turing Machine
NonUniform
& arbitrarily large input
 unreasonable
NonUniform
•by an And/Or/Not Circuit
& fixed input size

reasonable
If a TM can compute it in time T(n),
Uniform
then it can be computed by
a uniform circuit of size O(T2(n)). & arbitrarily large input
 reasonable
TM to Circuits
Given a TM M, whose running time
on inputs of size n is
at most T(n)..
n
T(n)
f(x)
T(n)
A given circuit
• has a fixed number n of
input wires an hence can only
take inputs with this number
of bits.
Need a separate circuit
C1, C2, C3, ….
for each input size n.
TM to Circuits
Given a TM M, whose running time
on inputs of size n is
at most T(n)..
n
T(n)
f(x)
T(n)
• the 0/1 value on
each wire at the tth level
t encodes the tape and head
position of the TM at time t.
All dependent on the input.
TM to Circuits
ASCII-celled TM at time t
H
E
L
L
O
Encoded by wires at layer t of the circuits.
H
E
L
L
q
01001000
01000011
01001101
01001101
O
00110100
ASCII-celled TM at time t+1
H
E
A
L
O
Encoded by wires at layer t+1 of the circuits.
H
E
A
q’ L
01001000
01000011
01000000
01001101
O
00110100
TM to Circuits
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
Put gates between these layers to compute t+1 from t.
H
01001000
E
01000011
A
01000000
q’
L
01001101
O
00110100
TM to Circuits
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
These bits
(i.e. if head is here and contents of this cell)
influence which bits?
H
01001000
E
01000011
A
01000000
q’
L
01001101
O
00110100
TM to Circuits
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
These bits
or move left leave Write
put
Move head right
and changeand
state
change
state and change state
Character
H
01001000
E
01000011
A
01000000
q’
L
01001101
O
00110100
TM to Circuits
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
Conversely, these output bits
depend on which input bits?
H
01001000
E
01000011
A
01000000
q’
L
01001101
O
00110100
TM to Circuits
H
01001000
E
01000011
q
Computes the entire
transition function
δ(q,c) = <q’,c’,direction>
H
01001000
E
01000011
L
01001101
L
01001101
A
01000000
O
00110100
Some small
circuit
q’
L
01001101
O
00110100
TM to Circuits
A given TM
• the contents of the tape
and location of the head
depend on the input
(and on the time step)
and hence are not known
at “compile time”.
H
01001000
E
01000011
A given circuit
• the gates and
their connections
must be fixed at
“compile time”
q
Hence, the same small circuit
gets copied all along.
H
01001000
E
01000011
L
01001101
L
01001101
A
01000000
O
00110100
Some small
circuit
q’
L
01001101
O
00110100
TM to Circuits
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
H
01001000
E
01000011
A
01000000
q’
L
01001101
O
00110100
TM to Circuits
qstart x1
qstart
H
01001000
Top layer is specified by input X.
x2
x3
x4
E
01000011
L
01001101
L
01001101
x5
….
O
00110100
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
TM to Circuits
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
qhalt
H
01001000
y1
E
01000011
y2
A
01000000
L
01001101
O
00110100
y3
y4
y5
The bottom layer specifies the output.
….
TM to Circuits
• If a TM can compute it in time T(n),
• then it uses at most T(n) cells of tape,
• then the size of the circuit is O(T(n)×T(n)).
• The circuit is constructed in a uniform way
from many copies of the same small circuit.
qstart x1
x2
x3
x4
x5
qstart
H
01001000
E
01000011
L
01001101
L
01001101
….
O
00110100
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by a Turing Machine
Done
•by an And/Or/Not Circuit
A Java program
• on an input of size n
• can produce this
uniform T(n)×T(n) circuit
• and evaluate it on the input.
Uniform
If a TM can compute it in time T(n), & arbitrarily large input
 reasonable
then it can be computed by
a uniform circuit of size O(T2(n)).
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by an And/Or/Not Circuit
Computer Circuit
•A computer circuit differs from a circuit in two ways:
• Memory
•Cycles in the graphs of gates
Computer Circuits
•Time is kept by a clock.
•Value xt may change through time.
•But when clock is high, current value
is stored in memory.
•When clock is low, it remembers
previous value.
•Output of memory is this value stored.
xt
Memory xt-1
t
xt
Clock
Computer Circuits
•Time is kept by a clock.
•Value xt may change through time.
•But when clock is high, current value
is stored in memory.
•When clock is low, it remembers
previous value.
•Output of memory is this value stored.
zt =
xt
yt
if clock=1
if clock=0
zt
AND
AND
OR
xt
Memory yt
¬c
xt+1
c
yt
Clock
Computer Circuits
ASCII-celled TM at time t
q
H
E
L
L
O
Encoded by wires at layer t of the circuits.
H
E
L
L
q
01001000
01000011
01001101
01001101
O
00110100
ASCII-celled TM at time t+1
q’
H
E
A
L
O
Encoded by wires at layer t+1 of the circuits.
H
E
A
q’ L
01001000
01000011
01000000
01001101
O
00110100
Computer Circuits
memory
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
H
01001000
E
01000011
A
01000000
q’
L
01001101
O
00110100
Computer Circuits
memory
H
01001000
E
01000011
q
L
01001101
L
01001101
O
00110100
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by an And/Or/Not Circuit
Computer Circuit Done
Circuit Depth
Circuits Depth
•The depth of a circuit:
•is the length of the longest path
from an input to an output.
•It indicates evaluation time.
x1 0
•It relates to parallel
computation time.
x2 1
AND
AND
•The Size of a circuit:
•is the number of gates.
•It relates to sequential
computation time.
x30
0
OR
0
1
NOT
OR
0
0
OR
0
Circuits Depth
Any function f(X) of n bits can be computed
with a circuit of size 2n.
x1 ¬x2 x3 …¬xn ¬x1 x2 ¬x3 … xn
AND
x1 x2 ¬x3 … ¬xn
…
AND
OR
AND
1
If you allow arbitrary fan.
But we assume 2 inputs.
Outputs 1 iff f(X)=1.
Circuits Depth
Any function f(X) of n bits can be computed
with a circuit of size 2n.
x1 ¬x2 x3 …¬xn ¬x1 x2 ¬x3 … xn
AND
x1 x2 ¬x3 … ¬xn
…
AND
OR
Outputs 1 iff f(X)=1.
AND
O(log(n))
O(n)
Circuits Depth
• If a TM can compute it in time T(n),
• then it uses at most T(n) cells of tape,
• then the size of the circuit is O(T(n)×T(n)).
• The circuit is constructed in a uniform way
from many copies of the same small circuit. O(T(n))
qstart x1
x2
x3
x4
x5 ….
qstart
H
01001000
E
01000011
L
01001101
L
01001101
O
00110100
Some smallSome smallSome smallSome smallSome small
circuit
circuit
circuit
circuit
circuit
How to add 2 n-bit numbers.
+
* * * * * * * * * * *
* * * * * * * * * * *
How to add 2 n-bit numbers.
+
*
* * * * * * * * * * *
* * * * * * * * * * *
*
How to add 2 n-bit numbers.
+
* *
* * * * * * * * * * *
* * * * * * * * * * *
* *
How to add 2 n-bit numbers.
+
* * *
* * * * * * * * * * *
* * * * * * * * * * *
* * *
How to add 2 n-bit numbers.
+
* * * *
* * * * * * * * * * *
* * * * * * * * * * *
* * * *
How to add 2 n-bit numbers.
* * * * * * * * * * *
* * * * * * * * * * *
+ * * * * * * * * * * *
* * * * * * * * * * * *
How to add 2 n-bit numbers.
***********
+ ***********
***********
************
Takes O(n) time.
Even with parallel help.
O(n) circuit depth.
How to add 2 n-bit numbers.
***********
+ ***********
***********
************
ci = ith carry bit.
xi = ith input bit of X.
yi = ith input bit of Y.
zi = ith output bit of Z.
zi = lowBit( ci+xi+yi )
ci+1 = highBit( ci+xi+yi )
xi yi ci
+
ci+1 zi
How to add 2 n-bit numbers.
x0 y0 c0
x1 y1 c1 +
z0
x2 y2 c2 +
z1
x3 y3 c3 +
z2 Is O(n) depth
x4 y4 c4 +
intrinsic to adding
z3
or can it be done
+
x5 y5 c5
with less depth?
z4
x5 y5 c5 +
We need to compute
z5
+
the carries ci sooner.
z z
O(n) circuit depth.
How to add 2 n-bit numbers.
+
* * * *
* * * *
Size (# of gates)
Depth (path leng)
* * * * * * * *
* * * * * * * *
Previous alg
I was taught
Little extra
thought.
O(n)
O(n2)
O(n)
O(n)
O(logn)
O(logn)
Trade off between size and depth! Win-Win
How to add 2 n-bit numbers.
n/
d
+
= 3 blocks
* * * *
* * * *
d = 4 bits per blocks
* * * * * * * *
* * * * * * * *
At level l= 1,2,3,…,logn from the circuit inputs.
Break the input bits into n/d blocks of size d = 2l.
How to add 2 n-bit numbers.
n/
d
= 3 blocks
Xd2
Yd2
d = 4 bits per blocks
1
Xd1
Yd1
Xd0
Yd0
Note 0 ≤ Xli ≤ 2d-1
Value of 2d gives a carry out of these d bits.
Xdi + Ydi ≥ 2d, then there is definitely a carry
to the next block.
We say the block “Generates” a carry.
Wire gdi = true.
Xdi + Ydi ≥ 2d-1, then there is a carry out
only if there is a carry in.
We say the block “Propagates” a carry.
Wire pdi = true.
How to add 2 n-bit numbers.
n/
+
1
= 12 blocks
* * * * * * *
* * * * * * *
d = 1 bits per blocks
1 1*
* 10 * *
* 1 * *
Wire g1i = block “Generates” a carry (ie. one out for sure)
xi+yi ≥ 21 = 2
g1i = And( xi, yi )
Wire pdi = block “Propagates” a carry (ie. one out if one in)
xi+yi ≥ 21-1 = 1
pdi = Or( xi, yi )
xi yi xi yi
AND
g1i
OR
p1i
How to add 2 n-bit numbers.
n/
d
= 3 blocks
1
1
1
d = 4 bits per blocks
Xd/22i+1 Xd/22i
Yd/22i+1 Yd/22i
Wire gdi = block “Generates” a carry (ie. one out for sure)
iff first half generates a carry
and second half propagates it
or second half generates it
= (gd/22i and pd/22i+1) or (gd/22i+1)
Wire pdi = block “Propogates” a carry (ie. one out if one in)
iff
first half propagates a carry
and second half propagates a carry
= (pd/22i and pd/22i+1)
How to add 2 n-bit numbers.
n/
d
= 3 blocks
d = 4 bits per blocks
Xd/22i+1 Xd/22i Xd/22i+1 Xd/22i
Yd/22i+1 Yd/22i Yd/22i+1 Yd/22i
gd/22i+3 pd/22i+3 gd/22i+2pd/22i+2 gd/22i+1 pd/22i+1 gd/22i pd/22i
gdi+1pdi+1
gdi pdi
Little extra
thought.
g2di/2 p2di/2
Size
Depth
n+n/2+n/4+…1
= O(n)
O(logn)
How to add 2 n-bit numbers.
n/
d
= 3 blocks
1*
1*
d = 4 bits per blocks
Xd/22i+1 Xd/22i
Yd/22i+1 Yd/22i
Suppose you know the carries between blocks of size d.
Then there is a carry between these block of size d/2,
iff carry at beginning and propagate
or generate.
Compute ALL carries
between bits.
How to add 2 n-bit numbers.
*
* * * *
* *
* *
Compute ALL carries
between bits.
* * * *
* * * *
Little extra
thought.
Size
Depth
n+n/2+n/4+…1
= O(n)
O(logn)
How to add 2 n-bit numbers.
***********
+ ***********
***********
************
ci = ith carry bit.
xi = ith input bit of X.
yi = ith input bit of Y.
zi = ith output bit of Z.
zi = lowBit( ci+xi+yi )
Little extra
thought.
xi yi ci
+
zi
Size
Depth
O(n)
O(1)
How to add 2 n-bit numbers.
***********
+ ***********
***********
************
Size (# of gates)
Depth (path leng)
Previous alg
I was taught
Little extra
thought.
O(n)
O(n2)
O(n)
O(n)
O(logn)
O(logn)
Add n
2n-bit
numbers
How to multiply 2 n-bit numbers.

x1
x2
x3
********
********
********
********
********
********
********
********
********
xn
********
****************
How to multiply 2 n-bit numbers.
x1 x2
O(logn) circuit depth.
x3
O(n logn) circuit depth.
x4
Can we multiply
+
2
x
5
in O(log n) depth?
Can we multiply
+
x6
in O(logn) depth?
+
x7
+
+
+
Home work
to do starting now
6.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by an And/Or/Not Circuit
Computer Circuit
Circuit Depth
Arithmetic Circuits
Done
Arithmetic Circuits
•An arithmetic circuit has +, -, ×, & / gates.
•An input X assigns a real number to each incoming wire.
x1 3
x2 5
x37
×
+
8
+
21
12
×
168
-12
/
-14
The real numbers percolate
down to the output wires.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by an And/Or/Not Circuit
Computer Circuit
Circuit Depth
Arithmetic Circuits Done
Neural Nets
Neural Nets
Neural Nets
x1 x2 x3 … xn
w1 w2 w3 … wn
Threshold
T
y
•Inputs x1, x2 , x3 , …, xn and output y are binary.
•Weights w1 , w2 , w3 , …, wn are real numbers
(possibly negative).
•y = 1 iff Σi wi×xi ≥ T
•The neural net learns by adjusting weights wi.
Neural Nets
x1 x2 x3 … xn
w1 w2 w3 … wn
Threshold
T
y
•You can build an AND, OR, and NOT gate from these.
•Hence, they are as powerful as circuits.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by an And/Or/Not Circuit
Computer Circuit
Circuit Depth
Arithmetic Circuits
Done
Neural Nets
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
•by a (simple) Register Machine
•by a Circuit: And/Or/Not, Computer,
Arithmetic, & Neural Nets.
Done
History of Computability
Are these definitions
equivalent?
Church says “Yes”
All reasonable models
of computation are equivalent
as far as what they can compute.
Turing 1936
What about time?
History of Computability
Computable
Halting
Exp
Time: input size  running time
For large n, n100 << 2n
Poly
Jack Edmonds 1965
History of Computability
Computable
Halting
Exp
Revised Church says:
All reasonable models
of computation are equivalent
within a polynomial in time
Poly
Jack Edmonds 1965
History of Computability
Revised Church says:
All reasonable models
of computation are equivalent
within a polynomial in time
•For any two models of computation M1 and M2,
•there is a constant c,
•for any computation problem P,
•if P can be solved in time T(n) in M1,
•then it can be solved in at most time (nT(n))c in M2.
Note c depends on the models M1 and M2,
but not on the problem P.
Home work
to do starting now
7.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
Proof
Reasonable
Reasonable?
Yes, polynomially slower
•by a (simple) Recursive Program
Reasonable?
See assignment
Reasonable?
•by a (simple) Register Machine
See assignment
•by a Circuit: And/Or/Not, Computer,
Reasonable
Arithmetic, & Neural Nets.
•by a Non-Deterministic Machine?
Non-Deterministic Machines
A Non-Deterministic TM/Java/… is
•the same as a deterministic one,
except that its moves are not deterministic.
•It has a “choice” as to which step to take next.
•Deterministic:
(qi,c) = <qj,c',right>
•Non-Deterministic: <qi,c; qj,c',right> ϵ 
•A non-deterministic machine M on input I
is said to accept its input
•if there exists an accepting computation.
•Note problems computed need yes/no answers.
Jack Edmonds
Steve Cook
Non-Deterministic Machines
• If I is a “yes” instance:
• A non-deterministic (fairy god mother) could prove
it to you by telling you which moves to make.
• This information is said to be a witness.
• Given an instance I and a witness/solution S,
there is a poly-time deterministic alg Valid(I,S)
to test whether or not S is a valid.
• With this you can convince your non-believing boss
that I is a “yes” instance:
• If I is a “no” instance:
•There is no witness she could tell you.
•You cannot convince your boss.
• An equivalent definition:
• Problem P can be computed non-deterministically
if P(I) =  S Valid(I,S)
Non-Deterministic Machines
• Example: Satisfiablity:
• Instance: A circuit I.
• Solution: A satisfying assignment S.
• I is a yes instance if there is such an assignment .
• Given an instance I and a solution S,
there is a poly-time alg Valid(I,S) to test
whether or not S is a valid solution of I.
Assignment S:
0110101
Circuit I:
1
P(I) =  S Valid(I,S)
Non-Deterministic Machines
• Suppose you and your boss only have polynomial time
i.e. Time(Valid(I,S))  n1000, where n = |I|.
Note
|S|  n1000.
Then the problem P is said to be in Exp
Non-Deterministic Polynomial Time (NP).
If P NP
 Deterministic Time is?
NP
SAT
Given an1000input I,
n
try all 2
witnesses S.
Poly
P(I) =  S Valid(I,S)
Non-Deterministic Machines
• Suppose you and your boss only have polynomial time
i.e. Time(Valid(I,S))  n1000, where n = |I|.
Note
|S|  n1000.
Then the problem P is said to be in Exp
Non-Deterministic Polynomial Time (NP).
If P Polynomial Time,
You don’t even need
NP
SAT
the fairy good mother.
 P NP
Poly
GCD
P(I) =  S Valid(I,S)
Non-Deterministic Machines
Acceptable
• Suppose you and your boss have TM computable time
Computable
i.e. Valid(I,S)) is computed by a TM.
Note |S| can be arbitrarily long.
Then the problem P is said to be in Acceptable
Exp
If I don’t have
Fairy God mother,
why don’t I just try every
possible witness S?
•If I is a “yes” instance
 there is a witness
 alg halts and answers “yes”
•If I is a “no” instance
 there is no witness
 alg runs forever
NP
SAT
Poly
GCD
Non-Deterministic Machines
Acceptable
Computable
A Problem P is said to be
computable/decidable/recursive
•if on every input the machine
halts and give the correct answer.
Non-Deterministic Machines
Acceptable
Computable
Three equivalent definitions for a problem P
Acceptable:  TM M
•On every yes input, the machine halts
and gives the correct answer.
•On every no input, the machine
could halt and gives the correct answer
or could run for ever.
•Witnesses:
 computable Valid such that P(I) =  S Valid(I,S)
•Enumerable:  TM M
•Every yes input I, is eventually printed.
•No no input is ever printed.
Non-Deterministic Machines
Acceptable
Is there a language
that can be accepted
but not computed?
Halting
Halting(M,I)
= Does TM M halt
on input I.
Turing proves uncomputable.
Alg: Run M on I.
If it halts halt and say “yes”
If it does not halt run forever.
Computable
Non-Deterministic Machines
The Post Correspondence Problem (PCP)
•The input is a finite collection of dominoes
b
a ca abc
I = ca , ab , a , c
•A solution is a finite sequence of the dominoes
a b ca a abc
ab ca a ab c
I can give a
solution as a
•So that the combined string on the top
witness.
is the same as that on the bottom
abcaaabc
abcaaabc
Non-Deterministic Machines
Acceptable
But how big is
this witness?
PCP?
With
Repeats
Computable
Exp
a b ca a abc
ab ca a ab c
NP
PCP?
Without
Repeats
•If dominos can’t be repeated,
|S|  |I|
•If dominos can be repeated,
|S| can be arbitrarily long.
I can give a
solution as a
witness.
Poly
Non-Deterministic
Machines
Acceptable
CoAcceptable
Halting
Not Halting
Computable
Non-Deterministic
Yes instances I
•have accepting
computations
•have witness/solutions.
No instances I
•Do not.
Co-Non-Deterministic
No instances I
•have accepting
computations
•have witness/solutions.
Yes instances I
•Do not.
Exp
CoNP
NP
SAT
not SAT
Poly
GCD
NP-Complete Problems
Problem Pnew is NP-Complete
• Pnew not too hard.
•Pnew  NP
Test in poly-time
if a given solution
is valid
Computable
Exp
NP
complete
Pnew
Poly
Known
GCD
NP-Complete Problems
Problem Pnew is NP-Complete
• Pnew not too hard.
•Pnew  NP
Computable
Exp
• Pnew sufficiently hard.
If being able to solve
this problem fast
means that you can
solve every problem
in the class fast.
NP
Sat
complete
Pnew
Poly
Known
GCD
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
Proof
Reasonable
Reasonable?
Yes, polynomially slower
•by a (simple) Recursive Program
Reasonable?
See assignment
Reasonable?
•by a (simple) Register Machine
See assignment
•by a Circuit: And/Or/Not, Computer,
Reasonable
Arithmetic, & Neural Nets.
Reasonable
wrt computing power
•by a Non-Deterministic Machine
Unreasonably fast.
•by a Quantum Machine?
Quantum Machines
What about Quantum Machines?
•Based on quantum mechanics,
at each point in time a quantum TM
is in the super-position of any number
of configurations of normal TMs.
•In time T, it can only flip T quantum coins so can only
be in the super-position of 2T configurations.
•Hence, can be simulated by a normal TM in time 2T×T
time.
•Hence, Quantum Machines can’t compute more,
just sometimes faster.
•Factoring can be done in poly-time, ie 6=2×3.
•It is believed that NP-complete problems still take
exponential time.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
Proof
Reasonable
Reasonable?
Yes, polynomially slower
•by a (simple) Recursive Program
Reasonable?
See assignment
Reasonable?
•by a (simple) Register Machine
See assignment
•by a Circuit: And/Or/Not, Computer,
Reasonable
Arithmetic, & Neural Nets.
Reasonable
wrt computing power
•by a Non-Deterministic Machine
Unreasonablely fast.
Reasonable
•by a Quantum Machine
wrt computing power
Unreasonablely fast.
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
These are all
•by a (simple) Register Machine
Automata Machines
•by a Circuit: And/Or/Not, Computer, computing to
accept or reject inputs.
Arithmetic, & Neural Nets.
•by a Non-Deterministic Machine
•by a Quantum Machine
•by a Context Sensitive Grammar
What about
Grammars that
generate strings?
Grammars
Context Free Grammar:
Generates Strings:
Generates Language/Computational Problem:
Grammar generates string
iff string is in the language
iff Computational Problem says Yes.
Grammars
Context Free Grammar:
•Terminals: a,b,c,…
characters in the final string being generated.
•NonTerminals: A,B,C,…
characters that generate more substrings.
•Rules: A  aAb
 cBccBcc
At any time, you can replace A with aAb
•Start Symbol: S
•Stop when no more nonterminals in your string.
•Examples:
•English, JAVA, …
•All a CFG can do is
•union,
•concatenate,
•and spew & plop.
Grammars
Grammars
Grammar for union L1  L2:
•S S1
S
or
 S2
S1
L1
S
S2
L2
Grammars
Grammar for concatenation L1  L2:
•S S1 S2
S
S1
S2
L1
L2
Grammars
Grammar for spew & plop an # bn:
•Spew: SaSb
S
•Plop: S #
aSb
aaSbb
aaaSbbb
aaaaSbbbb
aaaaaSbbbbb
aaaaa#bbbbb
Grammars
•Step 1: Look for links and concatenations
Grammar for S  {0,1}* an bm c2m+3 dn {0,1}*
Linked
Linked
because must be
because must be
the same size n. the double the size.
Not Linked
because *
means anything.
Grammars
•Step 1: Look for links and concatenations
Grammar for S  {0,1}* an bm c2m+3 dn {0,1}*
Extra
concatenated on
Extra
concatenated on
Extra
stuck in
Grammars
•Step 1: Look for links and concatenations
•Step 2: Give names to the parts.
•Step 3: Build Grammar rules.
•Step 4: Recurse.
Grammar for S  {0,1}* an bm c2m+3 dn {0,1}*
A
•Concatenate:
Q
S  AQA
A
Grammars
•Step 1: Look for links and concatenations
•Step 2: Give names to the parts.
•Step 3: Build Grammar rules.
•Step 4: Recurse.
Grammar for A  {0,1}i
C
Spew
•Spew: A  CA
•Plop: A  ε (empty string)
A
CA
CCA
CCCA
CCCCA
CCCCCA
CCCCC
Grammars
•Step 1: Look for links and concatenations
•Step 2: Give names to the parts.
•Step 3: Build Grammar rules.
Grammar for C  {0,1}
Union
•Union: C  0 | 1
CCCCC
10010
Grammars
•Step 1: Look for links and concatenations.
Grammar for Q  an bm c2m+3 dn
Linked
Linked
because must be
because must be
the same size n. the double the size.
•Which to spew first a&d or b&c
Grammars
•Step 1: Look for links and concatenations
•Step 2: Give names to the parts.
•Step 3: Build Grammar rules.
•Step 4: Recurse.
Grammar for Q  an bm c2m+3 dn
T
•Spew: Q  aQd
•Plop: Q  Tccc
Q
aQd
aaQdd
aaaQddd
aaaaQdddd
aaaaaQddddd
aaaaaTcccddddd
Grammars
•Step 1: Look for links and concatenations
•Step 2: Give names to the parts.
•Step 3: Build Grammar rules.
Grammar for T  bm c2m
•Spew: T  bTcc
•Plop: T  ε
Q
aQd
aaQdd
aaaQddd
aaaaQdddd
aaaaaQddddd
aaaaaTcccddddd
aaaaabTcccccddddd
aaaaabbTcccccccddddd
aaaaabbbTcccccccccddddd
aaaaabbbcccccccccddddd
Grammars
•Step 1: Look for links and concatenations
•Step 2: Give names to the parts.
•Step 3: Build Grammar rules.
Grammar for S  {0,1}* an bm c2m+3 dn {0,1}*
T
A
Q
(concatenate)
S  AQA
(spew & plop)
A  CA | ε
(union)
C0|1
Q  aQd | Tccc (spew & plop)
T  bTcc | ε
(spew & plop)
A
S
A
C
Q
T

 {0,1}*
 {0,1}
 an bm c2m+3 dn
 bm c2m
Grammars
•Step 1: Look for links and concatenations
Grammar for S  an bm cn dm
Linked
because must be
the same size.
Grammars
•Step 1: Look for links and concatenations
Plop
Grammar for S  an bm cn dm
Spew Not linked
Grammars
•Step 1: Look for links and concatenations
Plop
Grammar for S  an bm cn dm
Not linked
Spew
Links over lap and hence
can’t be done by a CFG!
Pumping Lemma (ugly)
Grammars
•Step 1: Look for links and concatenations
Grammar for S  an bn cn
Linked
because must be
the same size.
Links over lap and hence
can’t be done by a CFG!
Grammars
•Step 1: Look for links and concatenations
Grammar for S  α1 α2 … αn # α1 α2 … αn
…
Linked
because must be
the same string.
Links over lap and hence
can’t be done by a CFG!
Grammars
•Step 1: Look for links and concatenations
…
Grammar for S  α1 α2 … αn # αn αn-1 … α1
Linked
because must be
the same string.
Links do not over lap and hence
can be done by a CFG!
S  0S0 | 1S1 | ε
Grammars
•Step 1: Look for links and concatenations
S
•begin Alg
•x = 5
•begin loop
•x = ( [ { 3 } ] )
•exit loop
•return x
•end Alg
Linked
This is what CFG is really good at!
Home work
to do starting now
8.7.
Machine Code vs Java
Machine Model:
• one line of code
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
Machine Code vs Java
Machine Model:
• one line of code
• one line of memory cells.
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
Machine Code vs Java
Machine Model:
• one line of code
• one line of memory cells.
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
If a computational problem is computable
by a Machine Code
by Java Program
Proof:
• This is what a compiler does.
Parsing/Compiling
Input: s=6*8+((2+42)*(5+12)+987*7*123+15*54)
Output:
Parsing/Compiling
Input: Java Code
Output: MARIE Machine Code
simulating the Java code.
Parsing/Compiling
Input: Java Code
Output: MARIE Machine Code
simulating the Java code.
Challenge: Keep track of three algorithms simultaneously
• The compiler
• The Java code being compiled
• The MARIE code being produced.
Parsing/Compiling
Parsing/Compiling
Parsing/Compiling
Friends - Strong Induction View of Recursion:
•The sub-instance given must be
•smaller and
•must be an instance to the same problem.
•Combine solution given by friend
to construct your own solution for your
instance.
•Focus on one step.
•Do not talk of their friends friends friends.
•Solve small instances on your own.
Parsing/Compiling
Algorithm: GetExp( s, i )
Input: s is a string of tokens
i is a start index
Output: p is a parsing of the longest valid expression
j is the end index
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
Parsing/Compiling
Algorithm: GetTerm( s, i )
Input: s is a string of tokens
i is a start index
Output: p is a parsing of the longest valid term
j is the end index
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
Parsing/Compiling
Algorithm: GetFact( s, i )
Input: s is a string of tokens
i is a start index
Output: p is a parsing of the longest valid factor
j is the end index
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
Algorithm: GetExp( s, i )
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
p<term,1>
p<term,2>
p<term,3>
Algorithm: GetExp( s, i )
Exp
p<term,1>
*
p<term,2>
* …*
p<term,k>
Algorithm: GetExp( m )
Output: MARIE Machine Code that
evaluates an expression and stores its value
in memory cell indexed by m.
Algorithm: GetTerm( s, i )
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
p<fact,1>
p<fact,2>
Algorithm: GetTerm( s, i )
Term
p<fact,1>
+
p<fact,2>
+ …+
p<fact,k>
Algorithm: GetTerm( m )
Output: MARIE Machine Code that
evaluates a term and stores its value
in memory cell indexed by m.
Parsing
Algorithm: GetFact( s, i )
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
Parsing
Algorithm: GetFact( s, i )
Fact
42
Algorithm: GetFact( s, i )
s=6*8+((2+42)*(5+12)+987*7*123+15*54)
p<exp>
Algorithm: GetFact( s, i )
Fact
(
p<exp>
)
Algorithm: GetFact( m )
Output: MARIE Machine Code that
evaluates a factor and stores its value
in memory cell indexed by m.
Next token determines which case the factor is.
Algorithm: GetFactArray( m )
Output: MARIE Machine Code that
evaluates a factor and stores its value
in memory cell indexed by m.
Algorithm: GetIfStatement()
Output: MARIE Machine Code that
executes an IfStatement
Home work
to do starting now
9.
Grammars
A language (a set if strings)
• is accepted by a TM
if the TM accepts every string in the language
and no other.
• is generated by a Grammar
if the grammar generates every string in the language
and no other.
The language anbn is generated by the grammar
• SaSb
• S 
The language anbncn is generated by
no context free grammars
Grammars
Which strings the does the
S
following grammar generate?
A
D
B
• A a
• S A
bB
• B bB
D
a
bbB
B
bbbB
This grammar only
bbbbB
bbbbbB
generates the one string a.
• Start with the start symbol, S.
• You can think of choosing which rule to apply
Non-Deterministically.
• If all characters in the string become terminal symbols,
then this string is generated
• If the process ends because no rule can be applied
then this process fails to generate a string.
• or if it continues forever.
Grammars
Context Free Grammar:
•C aBc
•Context does not matter.
Context Sensitive Grammar:
•dCc aBcc
•Context does matter.
C
acBadCcdaBAb
aBc acBad aBc cdaBAb
dCa
acBadCcdaBAb
acBa aBcc daBAb
Grammars
S
ABCABCABCS
Grammar for anbncn :
•Spew equal numbers: SABCS
ABCABCACBS
•Allow sorting: BAAB
CAAC
AAABBBCCCS
CBBC
AAABBBCCCTc
•Turn Cc:
S Tc
CTc Tcc
AAABBBCCTcc
•Turn Bb: Tc  Tb
AAABBBTcccc
BTb Tbb
•Turn Aa: Tb  Ta
AAABBBTbccc
ATa Taa
AAATbbbbccc
•End:
Ta ε
•Nothing else can be generated
Taaaabbbccc
because, no other way to get rid of
aaabbbccc
all the nonterminals.
Home work
to do starting now
10.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a Context Sensitive Grammar
Proof
TM to Grammar
Consider some configuration of a TM
q
1
0
1
1
0
1
0
•Encode configuration of the TM as a string
of terminals and nontermnials.
•Encode tape as string of terminals: 10110100
•Mark the beginning and the end of the tape nonterms 1011010
•Encode state as a nonterminal Qi for each state qi.
•Encode head location as 10Qi11010
TM to Grammar
Beginning
TM: Qstart1011010
TM starting configuration on input 10110110:
qstart
1
0
1
1
0
Grammar: Qstart1011010
1
0
TM to Grammar
Beginning
TM: Qstart1011010
Ending
Qaccept
We assume that if the input is in the language,
then the TM halts in the state qaccept
with the tape empty.
TM accepts with configuration:
qaccept
Grammar: Qaccept
TM to Grammar
Beginning
TM: Qstart1011010
Computation
Once input is given,
computation
is deterministic.
Ending
Qaccept
TM to Grammar
Beginning
TM: Qstart1011010
Grammar:
S
Computation
Deterministic.
Ending
Qaccept
1011010
The grammar must
be able to generate
each string in the language.
Hence, some of the steps
of the derivation must
be nondeterministic.
Rule: A  0A
A  1A
TM to Grammar
Beginning
TM: Qstart1011010
Grammar:
S
Computation
Deterministic.
Nondeterministic
Ending
Qaccept
1011010
The TM and the Grammar seem to
go in opposite directions.
The standard solution is to have
the Grammar simulate
the TM backwards.
TM to Grammar
Computation
Ending
Beginning
Deterministic.
TM: Qstart1011010
Qaccept
Transitions: (qi,0) = <qj,c,right>
(qi,1) = <qj,c,right>
The TM deterministically
overwrites the 0/1 with c.
Backwards
TM:
The backwards TM
nondeterministically guesses
whether the overwritten
value was 0 or 1.
We, however, will stick with forward moving TMs.
TM to Grammar
Beginning
TM: Qstart1011010
Grammar:
S
Computation
Deterministic.
Ending
Qaccept
1011010
Nondeterministic
Our Grammar:
1011010 Qstart1011010
1011010 Qaccept
Remembers the input at the end,
so that it can be “outputted”.
Needs to have two copies of
the input at the beginning.
TM to Grammar
Beginning
TM: Qstart1011010
Grammar:
S
Computation
Deterministic.
Nondeterministic
Ending
Qaccept
1011010
Our Grammar:
1011010 Qstart1011010
1011010 Qaccept
Nondeterministically guesses the input
and then simulates the TM
deterministically.
TM to Grammar
Our Grammar:
•Start:
Generates:
S
Nondeterministically 1011010 Qstart1011010
guesses the input. Copy of Start config
of TM
input
TM to Grammar
Our Grammar:
•Start: S  S’ 
S’  0S’0
 1S’1
 Q’start
Q’start 
See assignment.
Generates:
S
1011010 Q’start0101101
Reverse input
1011010 Qstart1011010
Two copies of the input
TM to Grammar
Our Grammar:
Generates:
S
•Start: See assignment.
•Blanks: Add trailing blanks.
  b
1011010 Qstart1011010
1011010 Qstart1011010bbbb
•Accepting TM Computation:
•Blanks: Remove trailing blanks.
b  
1011010 Qacceptbbbb
1011010 Qaccept
•Accepting: Qaccept  ε
1011010
TM to Grammar
Our Grammar:
Generates:
S
•Start: See assignment.
•Blanks: Add trailing blanks.
  b
1011010 Qstart1011010
1011010 Qstart1011010bbbb
•Rejecting or not halting TM computation:
1011010 ???Q’????
The derivation never removes
all nonterminals and
hence no string is generated!
1011010
TM to Grammar
Transformations
TM
qi
qj
1
0
1
1
0
1
0
TM is in state qi and sees a 1.
Transition (qi,1) = <qj,c,right>
1
0
c
1
Grammar: 10Qi11010
10Qj11010
10Qjc1010
10cQj1010
0
1
0
Rule: Qi1  cQj
TM to Grammar
Transformations
TM
qi
qj
1
0
1
1
0
1
0
TM is in state qi and sees a 1.
Transition (qi,1) = <qj,c,left>
1
0
c
1
Grammar: 10Qi11010
10Qj11010
10Qjc1010
1Qj0c1010
0
1
0
Rule: 0Qi1  Qj0c
1Qi1  Qj1c
TM to Grammar
Our Grammar:
Generates:
S
•Start: See assignment.
•Blanks: Add trailing blanks.
  b
1011010 Qstart1011010
1011010 Qstart1011010bbbb
•Transitions: Qi1  cQj
0Qi1  Qj0c
1Qi1  Qj1c
•Blanks: Remove trailing blanks.
b  
1011010 Qacceptbbbb
1011010 Qaccept
•Accepting: Qaccept  ε
1011010
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by a Turing Machine
done
•by a Context Sensitive Grammar
A Java program can
easily determine if a
string can be
generated by a
grammar.
Or can it?
Grammar to Java
•Given a string, a JAVA program would have to determine
if it can be parsed by the context sensitive grammar.
•It wont know which rules to choose.
•But it can try all combinations of rules, ie. all derivations.
Grammar to Java
•But how many possible derivations are there
and how long is each?
•With a context free grammar,
•each rule makes the string longer.
•Hence, no derivation can be longer then the length n
of the string.
•Hence, there can’t be more than (#rules)n derivations.
•With a context sensitive grammar,
•some rules make the string longer and others make it
shorter.
•Hence, derivation can be arbitrarily long.
•Hence, there are an infinite number of derivations.
Grammar to Java
•Given that there an infinite number of derivations,
how can a Java program try them all?
•If it follows one path of rules too long it will never get to
try other paths.
•It should try all paths of length one, then two, then three, …
•Each derivation will eventually be reached.
•If there is a derivation of the string, then
the Java program will eventually halt and accept the string.
•If there is a not derivation of the string, then
the Java program will run for ever!
Grammar to Java
The set of languages generated by
context sensitive grammars
is equal to the class of languages
accepted by a TM.
Computable
Acceptable
•if on every yes input
•the TM halts and says yes.
•the grammar generates the input
•on every no input
•the TM runs for ever or says no.
•the grammar fails to generates the input
•gets to a nonterminal Qno with no rule
•or never stops derivation.
Acceptable
Grammar to Java
Is there a
context sensitive grammar
that generates all <P,I>
for which P halts on I?
Computable
Yes!
•Let MU be a “universal” TM
which halts on input <P,I> iff P halts on I.
•Let GHalt be the grammar that generates <P,I>
iff MU halts on <P,I>.
Acceptable
Halting
Church’s Thesis
A computational problem is computable
acceptable
•by a Java Program
•by a Turing Machine
done
•by a Context Sensitive Grammar
Proof
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
•by a (simple) Register Machine
•by a Circuit: And/Or/Not, Computer,
Arithmetic, & Neural Nets.
•by a Quantum Machine
•by a Context Sensitive Grammar
NonUniform
 unreasonable
Uniform
Computes
the
same,
 reasonable
but faster.
Decide vs Accept
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
•by a (simple) Register Machine
•by a Circuit: And/Or/Not, Computer,
Arithmetic, & Neural Nets.
•by a Quantum Machine
?
•by a Context Sensitive Grammar
•by a Human
Human
What about the human brain?
Can it compute more than a TM?
•Science:
•The brain is just an elaborate machine (neural net).
•Hence can’t do any more than a TM.
•New Age:
•Quantum mechanics is magical.
•The brain is based on quantum mechanics.
•Hence, the brain can do much more than a machine.
•But we already showed QM can’t compute more.
•Religion:
•The human has a soul.
•Hence, the brain can do much more than a machine.
Complexity Classes
A complexity class is
a set of computational problems
that have a similar difficulty in
computing them.
Design a new class:
1. Choose some model
Java or Circuits
2. Deterministic or
Nondeterministic
3. Limit some resource
Time or Space
to Log, Poly, Exp, ….
Computable
Exp
Co-NP
NP
Poly
Complexity Classes
A complexity class is
a set of computational problems
that have a similar difficulty in
computing them.
Proving C1  C2 is not too hard.
Prove C2 can simulate C1.
Proving C1  C2 is hard.
Prove P ϵ C2 and P ϵ C1.
Computable
Exp
Co-NP
NP
Poly
Complexity Classes
A complexity class is
a set of computational problems
that have a similar difficulty in
computing them.
A problem is complete for a class
if being able to solve this problem
fast means that you can solve
every problem in the class fast.
There are complete problems
for every class.
Computable
Exp
Co-NP
NP
complete
Poly
Non-Deterministic Machines
Acceptable
CoAcceptable
Halting
Computable
Exp
Games
CoNP
NP
SAT
not SAT
complete
complete
Poly
GCD
Complexity Classes
ComputableHalting
Poly-Time
Acceptable, Witness, EnumerableNC: Poly-log depth Circuits
Computable
NC2
Ackermann's Time
NL: Non-Det Log Space
Double Exponential Time
L: Log-Space
E: Exponential Time
AC0, Thres0, Arith0: Constant
Exp Time
Depth
PSpace
NC0: Constant Fan-out
PH: Poly-Time Hierarchy
NP & Co-NP
Lecture 2
Jeff Edmonds
York University
COSC 4111
End