Models of Computability
•History & Overview
•Church's Thesis
•Goto vs Loop Models
•Turing Machines
•Loop Invariant Example
•Adding Example (single tape)
•Adding Example (multi tape)
•Table Lookup Example
•Single vs Multi Tape TMs
•Java to TM
•Constant vs Finite vs Infinite
•Universal TM
•Other Models of Computation
•Problems, Languages, Machines,
and Classes
Jeff Edmonds
ECS 2001
York University
Lecture 2
Please feel free
to ask questions!
Please give me feedback
so that I can better serve
you.
Thanks for the feedback
that you have given me
already.
History of Computability
In the beginning:
Euclid said,
“ Let there be an algorithm for GCD”
And so it was.
GCD(a,b) = GCD(x,y)
<x,y>  <y,x mod y>
And it was good.
Euclid (300 BC)
He gave a new understanding of
“Algorithm”
History of Computability
As one gets closer to God,
everything is possible.
We must only learn how to do it.
Unknown
Euclid (300 BC)
Known
GCD
History of Computability
The Jacquard loom (1801) was one of the
first programmable devices.
Weaves complex patterns in textiles.
Controlled by punched cards
This portrait of Jacquard was woven in silk
on a Jacquard loom and required 24,000
punched cards to create (1839).
History of Computability
Charles Babbage’s Analytical Engine (1837)
• Memory
• An arithmetical unit,
• Conditional branching and loops,
Making it the first Turing-complete general-purpose computer.
History of Computability
The 1890 census compiled by machine
that sorted punched cards.
Reduced required time from eight to one year
Population of 62,947,714
History of Computability
Can every problem be computed?
Or are some Uncomputable?
One of his 23 influential
open problems (1900)
Hilbert
Need to formally define
“Algorithm”
History of Computability
TM
Computable
Halting
Problem
A problem is Computable
if it can be computed by a
Turing Machine. (1936)
Turing
Need to formally define
“Algorithm”
History of Computability
TM
Computable
Halting
Problem
A problem is Computable
if it can be computed by a
Turing Machine. (1936)
Turing
Though not intended to be practical,
the Turing Machine hugely helped
develop and understand
the power and limits
of mechanical computation.
History of Computability
Vacuum tubes made electronic computing possible
• Colossus (1943) 2,400 tubes, ENIAC (1946) 17,000 tubes
• 5,000 characters per second
• $500,000 ($6 million with inflation).
• Helped breaking German codes (with Turing)
• Tube failure (every two days, 15 min to locate)
• First “bug” was a moth in a tube.
History of Computability
The first transistorised computer (1953)
Integrated circuits (1968)
Intel 4004 Microprocessors (1971)
Church’s Thesis
TM
Computable
Are these definitions
equivalent?
Church says “Yes”
All reasonable models
of computation are equivalent
as far as what they can compute.
Turing
Need to formally define
“Algorithm”
And within a polynomial in time
(except for Quantum Machines)
Church’s Thesis
A Computational Problem P states
•for each possible input I
•what the required output P(I) is.
A Language L is
•a computational problem where
•the required output is yes/no
Eg: Sorting
Eg: Is the input sorted?
An Algorithm/Program/Machine M is
•a set of instructions
•for solving a given computational problem P
Eg: Merge Sort
•on a given input I
A Model of Computation is
•What is a legal algorithm/program/machine
Eg: Java
Church’s Thesis
A Model of Computation
(What is a legal Algorithm/Machine/Program)
•How the algorithm is specified by finite description.
•What defines the current state of the machine.
• Eg, how information computed so far is stored.
•What line of code it is on or state it is in.
•What are the legal operations?
Should be a “reasonable” mechanical task.
•How the input/output are specified.
Question: Which computational problems
are computable by a given model of computation?
Church’s Thesis
Model of Computation: Java
•An algorithm/program is specified by Java code.
•The current state of the machine is specified by the
values of all the variables & data structures
and by which is the next line of code to be executed.
•The legal operations are the legal Java commands.
•Input/output are read and written to a file.
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
•by a (simple) Recursive Program
•by a (simple) Register Machine
•by a Circuit: And/Or/Not, Computer,
Arithmetic, & Neural Nets.
•by a Non-Deterministic Machine
•by a Quantum Machine
•by a Context Sensitive Grammar
Church’s Thesis
A computational problem is computable
•by a Java Program
•by a Turing Machine
If a boy can do it, a girl can do it better.
If a Java program can do it,
Turing Machine can simulate it.
Proof
Goto vs Loop Models
Goto Model:
• In the 70’s, programs
were controlled with
goto (and if) commands.
0:
1:
2:
3:
4:
x=0
print(“Hi”)
x = x+1
if( x<10 ) goto line 1
halt
Called spaghetti code because
computation flow can go all
over the place.
How many “Hi”s get printed?
Loop Model:
• In the early 80’s, structured
programming was enforced.
• “Goto”s were not allowed.
• Programs were controlled with
while (and if) commands.
for( x = 0; x<10; ++x )
print(“Hi”)
Goto vs Loop Models
Goto Model:
• In the 70’s, programs
were controlled with
goto (and if) commands.
0:
1:
2:
3:
4:
x=0
print(“Hi”)
x = x+1
if( x<10 ) goto line 1
halt
Called spaghetti code because
computation flow can go all
over the place.
How many “Hi”s get printed?
Loop Model:
• In the early 80’s, structured
programming was enforced.
• “Goto”s were not allowed.
• Programs were controlled with
while (and if) commands.
for( x = 0; x<10; ++x )
print(“Hi”)
Goto vs Loop Models
Goto Model:
• In the 70’s, programs
were controlled with
goto (and if) commands.
0:
1:
2:
3:
4:
x=0
print(“Hi”)
x = x+1
if( x<10 ) goto line 1
halt
Loop Model:
• In the early 80’s, structured
programming was enforced.
• “Goto”s were not allowed.
• Programs were controlled with
while (and if) commands.
for( x = 0; x<10; ++x )
print(“Hi”)
Do these two models have the same power?
Can they compute the same things?
Goto vs Loop Models
Goto Model:
Loop Model:
Do these two models have the same power?
Can they compute the same things?
Goto vs Loop Models
We design a goto program
that simulates it.
0:
1:
2:
3:
x=0
print(“Hi”)
x = x+1
if( x<10 ) goto line 1
Pretty straight forward
to replace each loop
with a goto.
Given an arbitrary
loop program:
for( x = 0; x<10; ++x )
print(“Hi”)
Goto vs Loop Models
Given an arbitrary
goto program:
11:
38:
99:
…
x=0
…
goto line 18
…
halt
We design a loop program
that simulates it.
Have one big loop
that iterates once
per line of the goto program.
Loop Invariantt:
At each iteration each program
has all the same variable values.
The loop program has a variable
line that specifies
which line of code
the goto program is on.
Goto vs Loop Models
Given an arbitrary
goto program:
We design a loop program
that simulates it.
line = 1
while( line  -1 )
11:
38:
99:
…
x=0
…
goto line 18
…
halt
if( line = 11 )
x=0
++line
if( line = 38 )
line = 18
if( line = 99 )
line = -1
end while
Goto vs Loop Models
Goto Model:
• In the 70’s, programs
were controlled with
goto (and if) commands.
0:
1:
2:
3:
4:
Loop Model:
• In the early 80’s, structured
programming was enforced.
• “Goto”s were not allowed.
• Programs were controlled with
while (and if) commands.
x=0
print(“Hi”)
for( x = 0; x<10; ++x )
x = x+1
print(“Hi”)
if( x<10 ) goto line 1
halt
Hence these do have the same power
Home work
to do starting now
Goto vs Loop Models
Loop Invariant Model:
• In the 90’s, the importance of
loop invariants was realized.
Loop Model:
• In the early 80’s, structured
programming was enforced.
• “Goto”s were not allowed.
• Programs were controlled with
while (and if) commands.
for( x = 0; x<10; ++x )
print(“Hi”)
It is still hard for me to know
how many “Hi”s get printed 
Goto vs Loop Models
Loop Invariant Model:
• In the 90’s, the importance of
loop invariants was realized.
x=0
loop
% Loop Inv: x = # “Hi”s printed so far.
exit when x=10
Focus on sequence of states
print(“Hi”)
instead of sequence of commands!
x = x+1
end loop
% Post Cond:
# “Hi”s get printed is 10.
Establishing Loop Invariant
<preCond>
codeA
<loop-invariant>
Goto vs Loop Models
Loop Invariant Model:
• In the 90’s, the importance of
loop invariants was realized.
x=0
loop
% Loop Inv: x = # “Hi”s printed so far.
exit when x=10
print(“Hi”)
x = x+1
end loop
% Post Cond:
# “Hi”s get printed is 10.
Maintaining Loop Invariant
Exit
<loop-invariant>
¬<exit Cond>
codeB
<loop-invariant>
Goto vs Loop Models
Loop Invariant Model:
• In the 90’s, the importance of
loop invariants was realized.
x=0
loop
% Loop Inv: x = # “Hi”s printed so far.
exit when x=10
print(“Hi”)
x = x+1
end loop
% Post Cond:
# “Hi”s get printed is 10.
Clean up loose ends
Exit
<loop-invariant>
<exit Cond>
codeC
<postCond>
Goto vs Loop Models
Loop Invariant Model:
• In the 90’s, the importance of
loop invariants was realized.
x=0
loop
% Loop Inv: x = # “Hi”s printed so far.
exit when x=10
print(“Hi”)
x = x+1
end loop
% Post Cond:
# “Hi”s get printed is 10.
Proves that IF the program terminates then it works
<PreCond> & <code> <PostCond>
Machine Code vs Java
Machine Model:
• one line of code
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
Machine Code vs Java
Machine Model:
• one line of code
• one line of memory cells.
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
Machine Code vs Java
Machine Model:
• one line of code
• one line of memory cells.
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
If a computational problem is computable
by a Machine Code
Proof:
• Easy simulation
by Java Program
Machine Code vs Java
Machine Model:
• one line of code
• one line of memory cells.
Java Model:
• fancy data structures
• loops
• recursions
• object oriented
If a computational problem is computable
by a Machine Code
by Java Program
Proof:
• This is what a compiler does.
• We will prove this after covering
Context Free Grammars
TM Model of Computation
A Turing Machine (1936)
is the simplest model of computation.
Motivations:
• Define the first general model of computation
(Previous machines/models did specific predefined tasks.)
TM Model of Computation
A Turing Machine (1936)
is the simplest model of computation.
Motivations:
• Show that a machine/algorithm does not need
a huge set of complex operation
but can itself be very simple.
• Somehow the power of computation is an
“emergent property”
Defn: In philosophy, systems theory, science, and art,
emergence is the way complex properties and patterns arise out
of a multiplicity of relatively simple interactions.
TM Model of Computation
A Turing Machine (1936)
is the simplest model of computation.
Motivations:
• Formally define what a legal computation is
so that he could prove that the
Halting problem can’t be computed.
Halting
Problem
Computable
TM Model of Computation
A Turing Machine (1936)
is the simplest model of computation.
Motivations:
• Just as Euclid/Gödel wanted the simplest set of axioms
from which to define truth of all mathematical statements,
Turing wanted the simplest set of operations
from which to define all algorithms.
Giving multiplication away for free
would have felt like cheating.
TM Model of Computation
A Turing Machine (1936)
is the simplest model of computation.
Motivations:
• It is cool that such a simple model of computation
computes everything computable.
TM Model of Computation
A Turing Machine (1936)
is the simplest model of computation.
Motivations:
• Though not intended to be practical,
the Turing Machine hugely helped
develop and understand the power and limits
of mechanical computation.
TM Model of Computation
Model of Computation: Turing Machine
•The current configuration of the machine is specified by
•The contents of tape
•Each cell contains either a zero, one, or blank.
(or characters from some other finite alphabet )
•Tape is infinite in one direction
•But after a finite number of cells, all are blanks.
•Initially the input 101 is presented as b101bb…
•As is the output.
TM Model of Computation
Model of Computation: Turing Machine
q
•The current configuration of the machine is specified by
•The contents of tape
•The current state q
•One of some fixed number
•One of which is the start state
•And one of which is the halting state
TM Model of Computation
Model of Computation: Turing Machine
q
•The current configuration of the machine is specified by
•The contents of tape
•The current state
•The current location of the head.
•The machine can move this head left or right.
•But does not “know” its current location.
•It only knows the value of this current cell.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
•The legal operations are:
•Given the current state q
•And value c of the cell with the head
•The TM looks up in a preset table
•The next state q’
•Whether to write a new value to the current cell
•Whether to move the head one cell
to the right or left
ie Transition function δ(q,c) = <q’,c’,direction>
TM Model of Computation
Model of Computation: Turing Machine
δ
0
1
qstart
Current State
This is a TM.
Nothing more.
Nothing less.
It gives instructions
for “computing”
It is abstract.
It is NOT practical.
We will try to give
it “meaning”.
Instead of trying to design
TMs, we will compile any
Java like program into a
TM.
cϵ∑ seen on tape
q1
q2
<q16, 0, R>
q3
q4
….
q100
qhalt
Transition function δ(q,c) = <q’,c’,direction>
Blank
contents of tape
 current state
T
i
m
e
Simple rules like these
produce complex patterns.
TM step:
• Current state & cell content at head
determines which rule to use giving:
• how cell contents changes
• how head moves
• how state changes
Tape Cells
contents of tape
 current state
T
i
m
e
Simple rules like these
produce complex patterns.
TM step:
• Current state & cell content at head
determines which rule to use giving:
• how cell contents changes
• how head moves
• how state changes
Tape Cells
T
i
m
e
Simple rules like these
produce complex patterns.
But any meaning has to be
added by us.
Tape Cells
TM Model of Computation
We must impose order and meaning:
Input:
0
Loop Invariants
We will rename the states to have
meaningful names:
q<line=39,x=3,y=2,p=even>
1
1
0
0
…
Spaghetti
Structured
TM
Java
like
computation
Code
δ(qi,c) = <qj,c’,direction>
Output:
Instead of trying to design TMs,
1
1
we will compile any Java like program
into a TM.
0
0
1
…
TM Model of Computation
Model of Computation: Turing Machine
q
q’
The finite state control can be thought of
as the computation done in hardware.
•For example, the cpu may have addition built into the
gates of the chip ie hardware.
•An and/or/not circuit can compute any function on a
constant r number of bits using at most 2r gates.
•In contrast, the TM head executes the software,
eg. Multiplications
TM Model of Computation
Model of Computation: Turing Machine
q
q’
The finite state control can be
thought of being a JAVA object with:
•a finite length of code (& a current line)
•a finite number of variables
•each taking on a current value from a finite range
•Its periscope can look at one cell of the tape
TM Model of Computation
Model of Computation: Turing Machine
q
q’
The finite state control can be
thought of being a JAVA object with:
•Each time step it decides
•How to change its values of its internal variables
and its current line of code
•What character to write in the current cell.
•Whether to move forward or backward one cell.
TM Model of Computation
If the states are just a set of meaningless objects,
how can a TM use them to “know” anything?
Set of all possible inputs I
Set of all possible states q
0 0 0 0 0 0 0 0 0
For each I, put an edge to
0 0 0 0 0 0 0 0 1
the state the TM is in q1
q2
0 0 0 0 0 0 0 1 0
at time 3.
q3
0 0 0 0 0 0 0 1 1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
1
q7
…
0
0
0
0
…
…
1
1
1
1
…
What does the TM “know”
when in state q3?
It seems to know that I1=1.
TM Model of Computation
If the states are just a set of meaningless objects,
how can a TM use them to “know” anything?
Set of all possible times t
Set of all possible states q
Put an edge to the states
q1
1
the
TM
might
be
in
q2
2
at this time.
q
3
3
…
…
q7
…
4
What else does the TM
“know” when in state q3?
It seems to know that t=3.
TM Model of Computation
Model of Computation: Turing Machine
q
q
q’
•The finite state control can be thought of as having
•A fixed sized black board on which to write
some bounded amount of information q.
•Eg, If it remembers a total of r bits, then
then the number of different states q is 2r
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=39,
q
x=3,
y=2,
p=even
•The finite state control can be thought of as having
•A fixed sized black board on which to write
some bounded amount of information q.
•Eg, If it remembers
•x ϵ {1,2,3,4}, y ϵ {1,2,3,4,5}, & p ϵ {even,odd}
•The number of different states q is 4×5×2 = 40
•Give the states meaningful names like
q<line=39,x=3,y=2,p=even>. Programs also must remember
which “line of code” they are on.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=39,
x=3,
y=2,
p=even
•The finite state control can be thought of as having
•A fixed sized black board on which to write q.
•And learning value c of the cell under new head position.
•Can instantly do any computation on what it knows.
•(even solving incomputable problems)
δ(qknows,c) = <qcomputed,c’,direction>
qknows is the state indicating what the TM currently knows.
qcomputed is the state indicating what the TM computes.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=39,
x=3,
y=0
Suppose the current state is q<line=39,x=3,y=0>
line 39: Remember x, y = value at head, move head right.
δ(q,c) = <q’,c’,direction>
δ(q<line=39,x=3,y=0>,2) = <q<line=40,x=3,y=2>,2,right>
Name of previous state.
Character at head.
Name of next state.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=39,
x=3,
y=0
2
Suppose the current state is q<line=39,x=3,y=0>
line 39: Remember x, y = value at head, move head right.
δ(q,c) = <q’,c’,direction>
δ(q<line=39,x=3,y=0>,2) = <q<line=40,x=3,y=2>,2,right>
Change the value of y.
Don’t change the value of cell.
Move head.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
More generally, suppose the current state is q<line=39,x=x’,y=0>
line 39: Remember x, y = value at head, move head right.
δ(q,c) = <q’,c’,direction>
x’,y’, δ(q<line=39,x=x’,y=0>,y’) = <q<line=40,x=x’,y=y’>,y’,right>
Recall: To keep the number of states finite
(and size of the table for δ)
the range of values must be bounded.
x’ ϵ {1,2,3,4}, y’ ϵ {1,2,3,4,5},
TM Model of Computation
cϵ∑ seen on tape
δ
1
Blank
qstart
Current State
A TM is simply this table.
All we have done is given the
states meaningful names
and given a pattern to the
transition rules.
0
q1
q2
<q16, 0, R>
q3
q4
….
q100
qhalt
x’,y’, δ(q<line=39,x=x’,y=0>,y’) = <q<line=40,x=x’,y=y’>,y’,right>
Recall: To keep the number of states finite
(and size of the table for δ)
the range of values must be bounded.
x’ ϵ {1,2,3,4}, y’ ϵ {1,2,3,4,5},
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
More generally, suppose the current state is q<line=39,x=x’,y=0>
line 39: Remember x, y = value at head, move head right.
δ(q,c) = <q’,c’,direction>
x’,y’, δ(q<line=39,x=x’,y=0>,y’) = <q<line=40,x=x’,y=y’>,y’,right>
It is important to differentiate between
•variables x
•values x’
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
z=5
More generally, suppose the current state is q<line=39,x=x’,y=0>
line 39: Remember x, y = value at head, move head right.
line 40: z=x+y. Do not move head.
x’,y’, δ(q<line=39,x=x’,y=0>,y’) = <q<line=40,x=x’,y=y’>,y’,right>
δ(q<line=40,x=x’,y=y’>,c) = <q<line=41,x=x’,y=,y’,z=x’+y’>,c,stay>
All done with table lookup.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
z=5
More generally, suppose the current state is q<line=39,x=x’,y=0>
line 39: Remember x, y = value at head, move head right.
line 40: z=x+y. Do not move head.
x’,y’, δ(q<line=39,x=x’,y=0>,y’) = <q<line=40,x=x’,y=y’>,y’,right>
δ(q<line=40,x=x’,y=y’>,c) = <q<line=41,x=x’,y=y’,z=x’+y’>,c,stay>
y’,right>
Or we can combine these two steps.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
z=5
z=5
a=6
b=216
…
More generally, suppose the current state is q<line=39,x=x’,y=0>
line 39: Remember x, y = value at head,
z=x+y, a=xy, b=ax, …, move head right.
x’,y’, δ(q<line=39,x=x’,y=0>,y’)
= <q<line=40,x=x’,y=,y’,z=x’+y’,a=x’y’,b=a’x’>, y’,right>
A TM can instantly do any computation on what it knows.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
z=5
z=5
a=6
b=216
…
More generally, suppose the current state is q<line=39,x=x’,y=0>
line 39: Remember x, y = value at head,
z=x+y, a=xy, b=ax, …, move head right.
x’,y’, δ(q<line=39,x=x’,y=0>,y’)
= <q<line=40,x=x’,y=,y’,z=x’+y’,a=x’y’,b=a’x’>, y’,right>
Evaluating z, a, & b does not require extra time.
Does it require more states (a bigger black board)?
TM Model of Computation
Implied
Model of Computation: Turing Machine
q
q’
l=40,
x=3,
y=2
z=5
Knowledge
z=5
a=6
b=216
…
No, it really is just a renaming of the states.
x’,y’, q<line=40,x=x’,y=,y’>
= q<line=40,x=x’,y=,y’,z=x’+y’,a=x’y’,b=a’x’>
For x,y ϵ {0,1,2,3,4},
As long as the TM knows x & y,
it implicitly knows x+y, xy, (xy)y , … there are still only
25=55 states.
The fancy state names are only for the programmer.
Later renaming them to q1, q2, q3, … changes nothing.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
Can a TM multiply two arbitrary
integers in one time step?
We just showed you can.
No, it cannot remember arbitrary
integers in its states.
Only: x ϵ {1,2,3,4}, y ϵ {1,2,3,4,5},
TM Model of Computation
Model of Computation: Turing Machine
q
q’
k,  TM M, x,yk
M multiplies xy in one time step.
Yes. We just showed you can.
The number of states needed grows as
a function of k.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
 TM M, x,y
M multiplies xy in one time step.
No, it cannot remember arbitrary
integers in its states.
The number of states needed grows as a
function of x & y.
TM Model of Computation
Resources
Grow with Input
Constant Resources
Fixed at Compile Time
Java: • # Lines of code
•
•
# & Range of Variables
Instruction Set
(Computed in One Step)
TM: • # of States
•
•
(# Bits on Black Board)
Tape Alphabet
State Transitions
(Computed in One Step)
•
•
Allocated memory
Running time
•
•
# Cells used
Running time
TM Model of Computation
Model of Computation: Turing Machine
q
q’
So how does a TM compute?
It can scan the input writing
in its states what
it wants to remember.
But it can’t remember the entire input
because it is too big.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
So how does a TM compute?
But it will remember what it can,
compute what ever it wants
from what it can remember.
TM Model of Computation
Model of Computation: Turing Machine
q
q’
So how does a TM compute?
Then move the head to learn
something else about the input
(forgetting something else)
TM Model of Computation
Model of Computation: Turing Machine
q
q’
So how does a TM compute?
Or it writes something that it has
computed somewhere on the tape
so that it can come back an reread it
later.
Home work
to do starting now
Oops I did it for you.
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
• Do not worry about the entire computation.
next
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
Leap into the middle of the algorithm.
What would you like your data structure to
look like when you are half done?
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
A loop invariant is an assertion that must be true
about the state of the data structure
every time the algorithm is
at the top of the loop.
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
The loop invariant should flow smoothly from the
beginning to the end of the algorithm.
– At the beginning, it should follow easily from
the preconditions.
– It should progress in small natural steps.
– Once the exit condition has been met, the
postconditions should easily follow.
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
Some i of the characters have been handled.
Input:
b10010b
Loop Invariant: bbb010b00010000b
Output:
b00010000000000010000b
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
Some i of the characters have been handled.
Input:
b10010b
Loop Invariant: bbb010b00010000b
Output:
b00010000000000010000b
By Induction the loop invariant will always be true.
Exit
S(0)
 i S(i)
i S(i)  S(i+1)
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
Some i of the characters have been handled.
Input:
b10010b
Loop Invariant: bbb010b00010000b
Output:
Exit
79 km
75 km
b00010000000000010000b
Measure of progress:
is the number of characters handled.
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
Some 0 of the characters have been handled.
Input:
b10010b
Loop Invariant: b10010bb
Output:
b00010000000000010000b
Establishing Loop Invariant
<preCond>
codeA
<loop-invariant>
Loop Invariant Example
Input: b10010b
Expand alphabet:
Output: b00010000000000010000b
All of the characters have been handled.
Input:
b10010b
Loop Invariant: bbbbbbb00010000000000010000b
Output:
b00010000000000010000b
Clean up loose ends
Exit
<loop-invariant>
<exit Cond>
codeC
<postCond>
Loop Invariant Example
Expand alphabet:
Output: b00010000000000010000b
Input: b10010b
Loop Invarianti: bbb010b00010000b
Loop Invarianti+1: bbbb10b000100000000b
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
Handle one character.
<loop-invarianti+1>
Only allowed actions are:
• Read character at head
• Move head
• Write character at head.
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbb010b00010000b
δ(q<line=1>, b) = <q<line=2>, b, right>
δ(q<line=1>, 0) = <q<panic>>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbb010b00010000b
a=0
i=2
No!
Cant store i.
Takes more
than
constant
memory.
δ(q<line=2>, 0) = <q<line=3,a=0>, b, right>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000b
δ(q<line=3,a=0>, 1) = <q<line=3,a=0>, 1, right>
a=0
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000b
δ(q<line=3,a=0>, 1) = <q<line=3,a=0>, 1, right>
δ(q<line=3,a=0>, 0) = <q<line=3,a=0>, 0, right>
a=0
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000b
δ(q<line=3,a=0>, 1) = <q<line=3,a=0>, 1, right>
δ(q<line=3,a=0>, 0) = <q<line=3,a=0>, 0, right>
δ(q<line=3,a=0>, b) = <q<line=5,a=0>, b, right>
a=0
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000b
δ(q<line=5,a=0>, 0) = <q<line=5,a=0>, 0, right>
δ(q<line=5,a=0>, 1) = <q<line=5,a=0>, 1, right>
δ(q<line=5,a=0>, b) = <q<line=6,a=0>, b, stay>
a=0
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000b
δ(q<line=6,a=0>, b) = <q<line=7,a=0>, 0, right>
δ(q<line=7,a=0>, b) = <q<line=8,a=0>, 0, right>
δ(q<line=8,a=0>, b) = <q<line=9,a=0>, 0, right>
a=0
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000000bb
a=0
Can forget a
if you like.
δ(q<line=9,a=0>, b) = <q<line=10,a=0>, 0, stay>
δ(q<line=9,a=1>, b) = <q<line=10,a=1>, 1, stay>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b00010000000bb
a=0
Can forget a
if you like.
δ(q<line=9,a=0>, b) = <q<line=10>, 0, stay>
δ(q<line=9,a=1>, b) = <q<line=10>, 1, stay>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
δ(q<line=10>, 0) =
δ(q<line=10>, 1) =
<q<line=10>, 0, left>
<q<line=10>, 1, left>
a=0
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
δ(q<line=10>, b) =
<q<line=12>, b, left>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
δ(q<line=12>, b) =
δ(q<line=12>, 0) =
δ(q<line=12>, 1) =
<q<line=15,>, b, right>
<q<line=13>, 0, stay>
<q<line=13>, 1, stay>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
δ(q<line=13>, 0) =
δ(q<line=13>, 1) =
<q<line=13>, 0, left>
<q<line=13>, 1, left>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
δ(q<line=13>, b) =
<q<line=1>, b, stay>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
q<line=1>
Loop Invariant Example
Expand alphabet:
Loop Invarianti: bbbb10b000100000000b
Loop Invarianti: bbb010b00010000b
Loop Invarianti+1: bbbb10b000100000000b
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
Handle one character.
<loop-invarianti+1>
Only allowed actions are:
• Read character at head
• Move head
• Write character at head.
Loop Invariant Example
Expand alphabet:
Output: b00010000000000010000b
Input: b10010b
Loop Invarianti: bbb010b00010000b
Loop Invarianti+1: bbbb10b000100000000b
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
Handle one character.
<loop-invarianti+1>
Only allowed actions are:
• Read character at head
• Move head
• Write character at head.
Adding Example
+
* * * * * * * * * * *
* * * * * * * * * * *
Adding Example
+
*
* * * * * * * * * * *
* * * * * * * * * * *
*
Adding Example
+
* *
* * * * * * * * * * *
* * * * * * * * * * *
* *
Adding Example
+
* * *
* * * * * * * * * * *
* * * * * * * * * * *
* * *
Adding Example
+
* * * *
* * * * * * * * * * *
* * * * * * * * * * *
* * * *
Algorithm?
Do not worry about
the entire computation.
next
Adding Example
+
* * * *
* * * * * * * * * * *
* * * * * * * * * * *
Leap into the middle of the algorithm.
*
What would you like your data structure
to look like when you are half done?
* * *
Adding Example
+
* * * *
* * * * * * * * * * *
* * * * * * * * * * *
* * * *
A loop invariant is an assertion that must be true
about the state of the data structure
every time the algorithm is
at the top of the loop.
Adding Example
+
* * * *
* * * * * * * * * * *
* * * * * * * * * * *
* * * *
Loop invariant:
The lowest i-1 digits of x and y have been read.
The lowest i-1 digits of the sum z have been produced
The carry to the ith digits has been remembered.
Adding Example
+
* * * *
* * * * * * * * * * *
* * * * * * * * * * *
* * * *
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
<loop-invarianti+1>
Adding Example
+
* * * * *
* * * * * * * * * * *
* * * * * * * * * * *
* * * * *
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
<loop-invarianti+1>
Adding Example
* * * * * * * * * * *
* * * * * * * * * * *
+ * * * * * * * * * * *
* * * * * * * * * * * *
Adding Example
Adding Example
How should we encode the input and the output?
Input:
0
1
1
0
0
…
The contents of tape
•Each cell contains either a zero, one, or blank
(or characters from some other finite alphabet )
Let  = {0,1,2,…,9} or even better ASCII.
Output:
1
1
0
0
1
…
Adding Example
How should we encode the input and the output?
Input:
2
9
3
6
7
x
y
Output:
4
1
7
z=x+y
5
2
3
8
1
5
6
Adding Example
How does the algorithm proceed?
Input:
2
9
x
3
6
7
1
8
1
5
6
y
Loop invariant:
The lowest i-1 digits of x and y have been read.
The lowest i-1 digits of the sum z have been produced.
The carry to the ith digits has been remembered.
Head is on ist digit of x.
Output:
z=x+y
Adding Example
How does the algorithm proceed?
Input:
2
9
3
6
7
x
1
8
1
5
6
y
carry = 0
Establishing Loop Invariant
<preCond>
codeA
Output:
z=x+y
<loop-invariant>
Adding Example
How does the algorithm proceed?
Input:
2
9
3
6
7
x
1
8
1
5
6
y
carry = 0
Maintaining Loop Invariant
Exit
Output:
z=x+y
<loop-invarianti>
¬<exit Cond>
codeB
<loop-invarianti+1>
Adding Example
How does the algorithm proceed?
Input:
2
9
x
3
6
7
1
8
1
z=x+y
6
y
States can’t remember more than a constant
number of digits of x.
or how many digits of x have been read.
Output:
5
carry = 0
xi=7
Adding Example
How does the algorithm proceed?
Input:
2
9
x
3
6
7
1
8
1
5
6
y
carry = 0
xi=7
yi=6
zi=3
carry = 1
Output:
z=x+y
Adding Example
How does the algorithm proceed?
Input:
2
9
3
6
7
1
x
8
1
5
6
y
carry = 0
xi=7
yi=6
zi=3
carry = 1
Output:
3
z=x+y
Adding Example
How does the algorithm proceed?
Input:
2
9
3
6
7
1
x
8
1
5
6
y
Loop invariant:
The lowest i-1 digits of x and y have been read.
The lowest i-1 digits of the sum z have been produced.
The carry to the ith digits has been remembered.
Head is on ist digit of x.
Output:
3
z=x+y
carry = 1
Adding Example
How does the algorithm proceed?
Input:
2
9
3
6
7
1
x
8
1
5
6
y
carry = 1
A bulk of the work will be
simply moving the head!
Output:
3
z=x+y
Adding Example
Multi-tape ASCII-celled TM
q
H
E
Z
L
T
H
E
2
t
5
4
L
R
O
7
E
3
•Knowing
•the one state q
•the value ci under each head
•Automata decides
•New state q’
•For each head what to write & whether to move.
ie δ(q,c1,c2,c3) = <q’, c’1,direc1, c’2,direc2, c’3,direc3 >
δ(q,L,E,5) = <q’, Z, right, 7, left, t, stay >
Adding Example
Multi-tape ASCII-celled TM
2
9
3
6
7
x
1
8
1
7
6
y
2
3
z=x+y
4
7
5
Guess how many states/lines of
code will be needed!
Would you believe 2?
Adding Example
Multi-tape ASCII-celled TM
2
9
3
6
7
x
1
8
1
7
6
y
2
3
z=x+y
4
7
5
Loop invariant:
The lowest i-1 digits of x and y have been read.
The lowest i-1 digits of the sum z have been produced.
The carry to the ith digits has been remembered.
Adding Example
Multi-tape ASCII-celled TM
carry = 1
2
9
3
6
7
x
1
8
1
7
6
y
2
3
z=x+y
Loop invariant:
The lowest i-1 digits of x and y have been read.
The lowest i-1 digits of the sum z have been produced.
The carry to the ith digits has been remembered.
Heads on the i digits.
Adding Example
What is remembered between iterations?
Only the carry bit.
How many states are needed for this?
Only two.
q<line=0,carrie=0> and q<line=0,carrie=1>
Adding Example
Multi-tape ASCII-celled TM
carry = 1
2
9
3
6
7
x
1
8
1
7
6
y
5
2
3
z=x+y
δ(q,c1,c2,c3) = <q’, c’1,direc1, c’2,direc2, c’3,direc3 >
δ(q<line=0,carrie=1> , 3, 1, blank)
= < q<line=0,carrie=0>, 3, left, 1, left, 5, left >
xi = 3, yi = 1
s = 3+1+1 = 5
zi = 5
carrie’ = 0
Adding Example
Multi-tape ASCII-celled TM
carry = 0
2
9
3
6
7
x
1
8
1
7
6
y
7
5
2
3
z=x+y
δ(q,c1,c2,c3) = <q’, c’1,direc1, c’2,direc2, c’3,direc3 >
δ(q<line=0,carrie=0> , 9, 8, blank)
= < q<line=0,carrie=1>, 9, left, 8, left, 7, left >
xi = 9, yi = 8
s = 9+8+0 = 17
zi = 7
carrie’ = 1
Adding Example
Multi-tape ASCII-celled TM
carry = 1
2
9
3
6
7
x
1
8
1
7
6
y
4
7
5
2
3
z=x+y
δ(q,c1,c2,c3) = <q’, c’1,direc1, c’2,direc2, c’3,direc3 >
δ(q<line=0,carrie=c> , xi, yi, blank)
= < q<line=0,carrie=c’> xi, left, yi, left, zi, left >
,
Table Lookup Example
Undecidable
Hint: Knowing whether J halts and
says yes on I=0 is undecidable.
Table Lookup Example
Hint: What can Pooh compute from what’s
on his black board? Anything!
Algorithm: Write the two million character
input on the blackboard and then do table
lookup to get the answer.
Any finite language can be computed this way.
Table Lookup Example
Input: I = b100101b
Output: f(I)
if
|I| ≤ 6
• Leap into the middle of the algorithm.
• If you have read the prefix α = 100, what do
you want to remember on the black board?
– The entire prefix read so far is on the board.
• What state should you be in?
– q100.
α=100
Table Lookup Example
Input: I = b100101b
Output: f(I)
if
|I| ≤ 6
δ(q100, 1) = ?
• You are in state q100 and the next character is a 1.
• Hence the input prefix you have read is 100.
• Hence after reading the 1, the prefix read will be 1001.
• Hence the next state should be q1001.
δ(q100, 1) = <q1001, 1, right>
α∑<6, c∑ δ(qα, c) = <qαc, c, right>
For each string α from alphabet ∑ of length at most 6
there is a state qα.
For each character c in this alphabet, when reading this character
what should the next state (and actions) should be?
Table Lookup Example
Input: I = b100101b
0
q00
0
1
q0
0
1 q
0
01
1
Output: f(I)
if
|I| ≤ 6
q000
α, c δ(qα, c) = <qαc, c, right>
q001
α |α|=6, c δ(qα, c) = ?
= <qtoo long, c, right>
q010
q011
q
1
0
q100
1
q101
0
q110
1
q111
q10
q1
1
0
q11
…
q100101
0,1
qtoo long
Table Lookup Example
Input: I = b100101b
0
q00
0
1
q0
0
1 q
0
01
1
Output: f(I)
q000
q001
q010
q011
q
1
0
q100
1
q101
0
q110
1
q111
q10
q1
1
0
q11
if
|I| ≤ 6
α |α|≤6, δ(qα, b) = ?
• When we read the closing blank
we know that the input is over.
• Hence, the input is I=α.
• Hence, the required output is f(α).
• How do we learn this value?
• Table lookup
• δ(qα, b) = <qanswer f(α), b, stay>
…
q100101
b
qanswer f(α)
0,1
qtoo long
Table Lookup Example
Input: I = b100101b
0
q00
0
1
q0
0
1 q
0
01
1
q000
Output: f(I)
if |I| ≤ 6
α |α|≤6,
(qα, b) = <qanswer f(α), b, stay>
q001
q010
q011
qanswer Yes
b
qanswer NO
q
1
0
q100
1
q101
0
q110
1
q111
q10
q1
1
0
q11
…
q100101
0,1
qtoo long
We will later see that this is a
Deterministic Finite Automata.
TM Model of Computation
Model of Computation: Turing Machine
δ
cϵ∑ seen on tape
0
1
Blank
<q10, 0, R>
<q11, 0, R>
<qanswer Yes, b, R>
q
Current State
q0
q1
q00
q01
….
q111111
qhalt
<qanswer No, b, R>
Church’s Thesis
Proof
A computational problem is computable
•by a Turing Machine
•by a Multi-Tape TM
A TM
simulates
Multi-Tape TM
by interweaving tapes.
Multi to Single Tape
Multi-tape ASCII-celled TM
H
2
E
5
L
4
L
O
3
•Knowing
•the one state q
•the value ci under each head
•Automata decides
•New state q’
•For each head what to write & whether to move.
ie δ(q,c1,c2) = <q’,c’1,direction1,c’2,direction2>
Multi to Single Tape
Multi-tape ASCII-celled TM
H
2
E
5
L
4
L
O
3
Simulated by:
Single-tape 0/1-celled TM
• Do not worry about the entire simulation.
next
Multi to Single Tape
Multi-tape ASCII-celled TM
H
2
E
5
L
4
L
O
3
Simulated by:
Single-tape 0/1-celled TM
Leap into the middle of the algorithm.
What would you like your data structure
to look like when you are half done?
Multi to Single Tape
Multi-tape ASCII-celled TM
H
2
E
5
L
4
L
O
3
Simulated by:
Single-tape 0/1-celled TM
A loop invariant is an assertion that must be true
about the state of the data structure
every time the algorithm is
at the top of the loop.
Multi to Single Tape
Multi-tape ASCII-celled TM
H
2
E
5
L
4
L
O
3
Simulated by:
Single-tape 0/1-celled TM
H
01001000
2
00110010
E
01000011
5
00110101
L
01001101
4
00110
• The cells from the multi-tape TM are interwoven on the
single tape TM
• Each ASCII cell needs a block of 8 bit cells to store the
character. Plus a bit to indicate whether the head is there.
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
01001000
00110010
O
E
01000011
5
00110101
• Msingle stores in states the state of Mmulti
L
01001101
4
00110
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
01001000
00110010
O
E
01000011
5
00110101
L
01001101
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
<loop-invarianti+1>
4
00110
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
01001000
00110010
O
E
01000011
5
00110101
L
01001101
4
00110
18
line 1: Move right to location of white tape’s head.
i.e. Remember i = head position mod 18
move right till i=0 and head sees 1.
δ(q<line=1,q=qmult, i=0>,0) =  (q<line=1,q=qmult, i=1>,0,right) 
δ(q<line=1,q=qmult, i=0>,1) =  (q<line=2,q=qmult, i=0>,1,stay) 
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
01001000
00110010
O
E
01000011
5
00110101
LL
01001101
line 1: Move right to location of white tape’s head.
line 2: Learn the character on the tape there.
4
00110
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
L
01001000
00110010
O
E
01000011
55
00110101
L
01001101
line 1: Move right to location of white tape’s head.
line 2: Learn the character on the tape there.
line 3: Move right to location of yellow tape’s head.
line 4: Learn the character on the tape there.
4
00110
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
L 5
01001000
00110010
O
E
01000011
5
00110101
L
01001101
line 5: Use table look up to learn what Mmulti will do.
δMulti(qmulti,L,5) = q’multi,H,right,4,left
δsingle( qline=5,q=qmulti,c1=L,c2=5, 0) =
 qline=6,q=q’multi,c1=H,left,c2=4,right,, 0,stay
4
00110
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
O
3
Simulated by:
Single-tape 0/1-celled TM
qmulti
H
2
E
5
L
01001000
00110010
01000011
00110101
01001101
HL
4,R
line 6: Move right to location of white tape’s head.
line 7: Write the required value.
line 8: Move white tape’s head.
4
00110
Multi to Single Tape
Multi-tape ASCII-celled TM
qmulti
H
2
E
5
L
4
L
O
3
Simulated by:
Single-tape 0/1-celled TM
H
qmulti
H
2
E
5
L
01001000
00110010
01000011
00110101
01001101
HL
4 R
line 9: Move right to location of yellow tape’s head.
line 10: Write the required value.
line 11: Move white tape’s head.
4
00110
Church’s Thesis
Proof
A computational problem is computable
A Java program
•by a Java Program
can be simulated by
a TM.
•by a Turing Machine
2)
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by Machine Code
By a Java compiler
•by a Turing Machine
•by a Multi-Tape TM
We just did this.
Multi-Tape TM
simulates
Machine Code
by moving head where
needed
We do this now.
Machine Code to TM
Machine Code
2
5
4
3
•
•Move value in cell addressed by cell 101 into register.
•
Simulated by:
Multi-tape ASCII-celled TM
2
5
4
3
Copy value 101 onto second tape.
Move first tape head counting this many cells.
Copy the value at head to “register” on second tape.
Machine Code to TM
Machine Code
2
5
4
3
•
•Add two 32-bit integers
•
Simulated by:
Multi-tape ASCII-celled TM
2
5
We did this already.
4
3
Church’s Thesis
Proof
A computational problem is computable
•by a Java Program
•by Machine Code
Done.
Done
•by a Turing Machine
•by a Multi-Tape TM
Variations in TM Model
•The most important requirement of
a Model of Computation is:
•that each algorithm/machine has a finite description.
•Otherwise:
every computational problem can be computed.
Variations in TM Model
Which changes to the TM are reasonable?
Put in state qI
• Allow a growing but
finite number of states.
Put I in one cell
• Allow an arbitrarily large
integers in a cell
Put head at cell
• Allow TM to “know”
indexed by I
where its head is.
Think of the entire input string as a single integer.
Itape = 1 0 0 1 0 1 1 1 0
I = 100101102
Variations in TM Model
Which changes to the TM are reasonable?
Put in state qI
• Allow a growing but
finite number of states.
Put I in one cell
• Allow an arbitrarily large
integers in a cell
Put head at cell
• Allow TM to “know”
indexed by I
where its head is.
Then next action of the TM can depend on entire input.
And with table lookup know the required answer f(I).
It could solve the halting problem.
But would require infinite space to write down its description.
Variations in TM Model
Which changes to the TM are reasonable?
• Allow an arbitrarily large
integers in a cell
Later we will look at the models of computation
- Recursive Programs
- Register Machines
that allow this.
But their next action is not a function of
the value of this integer.
Variations in TM Model
Which changes to the TM are reasonable?
• Suppose a TM does not have a head,
but can specify the index i of the cell to read
and that i’ to write to.
Then it would be weaker because a Finite State control
can only specify a fixed number of cells.
Variations in TM Model
I had to let the TM move its
head so that it can access all of
the arbitrarily long input.
Turing
Need to formally define
“Algorithm”
But the TM cant do something
different with every head
position or else the description
would be infinite.
Constant vs Finite vs Infinite
I wanted to give a TM an
arbitrarily large number of
states so that it can compute an
arbitrarily hard any
“computable” problem
But not give it an infinite
number, or else it would know
Turing
the answer to any problem
Need to formally define
using table look up.
“Algorithm”
Constant vs Finite vs Infinite
How many states can a TM use?
•Let #States(M,I) denote the number of states of TM M on input I.
•Designer can give her TM as many states as she wants
(given the needs of the problem at hand)
•  integer k,  a TM M, #States(M)=k
If the problem or my adversarial boss gives me
some integer k
(even 1,000,000,000,000,000),
I can build a TM with this many states.
Constant vs Finite vs Infinite
How many states can a TM use?
•Let #States(M,I) denote the number of states of TM M on input I.
•Designer can give her TM as many states as she wants
(given the needs of the problem at hand)
•  integer k,  a TM M, #States(M)=k
•But once set, this number is fixed/constant.
i.e. can depend on M, but not on I (or on |I|)
•  TM M,  an integer k,  inputs I, K(M,I) = k
If my adversary gives me even the grandest TM M,
it is limited in that its number of states
cannot grow beyond some preset number k
even if the input size |I| gets bigger and bigger.
Constant vs Finite vs Infinite
I come up with a hard problem P
•  computable problems P,
 a TM M,  an integer k,
 inputs I,
K(M,I) = k
Constant vs Finite vs Infinite
I write a TM M
and give it some k states.
I can use as MANY as I like.
1,000,000,000,000,000!
Wow. That’s not fair.
With more and more states,
you can memorize
more and more answers.
•  computable problems P,
 a TM M,  an integer k,
 inputs I,
K(M,I) = k
Constant vs Finite vs Infinite
I write a TM M
and give it some k states.
I can use as MANY as I like.
1,000,000,000,000,000!
I will let him use any
number of states he wants,
But this fixed M must work
for all inputs.
•  computable problems P,
 a TM M,  an integer k,
 inputs I,
K(M,I) = k
Constant vs Finite vs Infinite
I write a TM M
and give it some k states.
I can use as MANY as I like.
1,000,000,000,000,000!
If he uses more states,
I will give him a bigger input I.
Hee Hee Hee
M must still solve the problem.
•  computable problems P,
 a TM M,  an integer k,
 inputs I,
K(M,I) = k
Constant vs Finite vs Infinite
•Given the input,
•a TM,
•can dynamically allocated as many cells of tape
as it wants.
•This number, though still finite, is not constant. Meaning:
•If K(M,I) is the number cells used
by TM M on input I,
then this function can depend on M,
and on I. (hence on |I|)
•  computable problems P,
 a TM M,
 inputs I,
 an integer k,
K(M,I) = k
Constant vs Finite vs Infinite
I come up with a hard problem P
•  computable problems P,
 a TM M,
 inputs I,
 an integer k,
K(M,I) = k
Constant vs Finite vs Infinite
I write a TM M.
I don’t need to specify how many
cells of tape k it will allocate.
Oops.
I now need to come up with
the input I before knowing this k.
•  computable problems P,
 a TM M,
 inputs I,
 an integer k,
K(M,I) = k
Constant vs Finite vs Infinite
No problem.
If he gives me a bigger input I,
I will use even more tape!
•  computable problems P,
 a TM M,
 inputs I,
 an integer k,
K(M,I) = k
Constant vs Finite vs Infinite
No problem.
If he gives me a bigger input I,
I will use even more tape!
Good thing he cannot increase
the number of states
in the same way or else
•it would take infinite space
to write down the algorithm
•such an algorithm could
solve any problem.
Universal TM
10101,1110,01011,0111,1
00110,1000,00111,0101,0
A Universal TM Muniversal is given
a description of a TM M and its input I
and simulates this computation.
Muniversal has a fixed set of states
and tape alphabet
But for M these are arbitrarily large. (ie bigger)
δM(q,c) = <q’,c’,direction>
δM(q10101,c1110) = <q01011,c0111,right>
…
Universal TM
10101,1110,01011,0111,1
00110,1000,00111,0101,0
A Universal TM is a TM Muniversal is given
a description of a TM M and its input I
and simulates this computation.
Muniversal has a fixed set of states
and tape alphabet {0, 1, , , ’,’}
But for M these are arbitrarily large. (ie bigger)
δM(q,c) = <q’,c’,direction>
δM(q10101,c1110) = <q01011,c0111,right>
…
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
Muniversal is given input
I = c1110,c0101,c1000,…
…
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
After simulating t steps of M,
this second Muniversal tape gives
the contents of M’s tape.
Muniversal ’s second head is on the “character”
that M’s head is on.
…
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
What Muniversal ’s finite state automata “knows”
is what is in its states
– i.e. what is written on its black board.
Can it “know” what state M is in?
No. M was chosen by an adversary
AFTER Muniversal was designed.
Hence M has way more states.
Hence Muniversal ’s states cant remember one.
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
What Muniversal ’s finite state automata “knows”
is what is in its states
– i.e. what is written on its black board.
Can it “know” what state M is in?
Need separators , because Muniversal can’t even
know the width of M’s state and character
descriptions!
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
What Muniversal ’s finite state automata “knows”
is what is in its states
– i.e. what is written on its black board.
Can it “know” what state M is in?
In contrast, Msingle tape simulating Mmulti tape
was designed knowing Mmulti tape’s states.
δsingle( qline=5,q=qmulti,c1=L,c2=5, 0) =
 qline=6,q=q’multi,c1=H,left,c2=4,right,, 0,stay
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
00110
Muniversal will store M’s current state
on at third tape.
Maintaining Loop Invariant
Exit
<loop-invarianti>
¬<exit Cond>
codeB
<loop-invarianti+1>
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
,1000
00110,1000,00111,0101,0
…
…
00110,1000
00110
line 1: Copy the current “character” onto third tape.
(One character at a time).
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
00110,1000
line 2: Search the first tape for the string on the third tape.
All rules before the head do not match.
All bits in the current rule before the head do match
the bits before the third head.
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
00110,1000
line 2: Search the first tape for the string on the third tape.
line 2.1: If the current bits are the same, move heads.
δ(q<line=2>,c,?,c) =  q<line=2>,c,right,?,stay,c,right 
Note Muniversal is remembering
nothing but code line.
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
00110,1000
line 2: Search the first tape for the string on the third tape.
line 2.1: If the current bits are the same, move heads.
δ(q<line=2>,c,?,c) =  q<line=2>,c,right,?,stay,c,right 
1st tape and 3rd tape bits
are the same.
So move right.
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
…
00110,1000
line 2: Search the first tape for the string on the third tape.
line 2.2: If the current bits are the different,
move to next rule.
δ(q<line=2>,c,?,c’) =  q<line=2’>,c,stay,?,stay,c’,stay 
δ(q<line=2’>,?,?,?) =  q<line=2’>,c,right,?,stay,c’,left 
Move till  or .
Universal TM
10101,1110,01011,0111,1
1110 0101 1000
00110,1000,00111,0101,0
…
00110,1000
line 3: Having found the rule,
copy the new state to the third line,
the new character to the second line,
and move second head to the next “character”
…
Universal TM
10101,1110,01011,0111,1
1110 0101 0101
00110,1000,00111,0101,0
…
…
00111
This completes the simulation of one more step of M
and maintains the loop invariant.
Universal TM
10101,1110,01011,0111,1
1110 0101 0101
…
00110,1000,00111,0101,0
…
m
t
00111
Running Time
• Let m = |M| = size of description of TM M
• Let n = |I| = size of description of input I
• Let t = Time(M,I) = # time steps of M running on I
• Assume M also uses t tape so Muniversal’s second tape has length t.
Running time of Muniversal.
• With 3 tapes:
Time to search for rule match = ? O(m)
Number of simulation steps = t
O(mt)
Total time = ?
Universal TM
Tape1Tape3Tape2
m
t
Running Time
Small “constant”
• Let m = |M| = size of description of TM M
as n and t grow.
• Let n = |I| = size of description of input I
• Let t = Time(M,I) = # time steps of M running on I
• Assume M also uses t tape so Muniversal’s second tape has length t.
Running time of Muniversal.
• With one tape:
•
Time to search for rule match = ? O(|c|t + |qc|m) ≈ O(t)
Number of simulation steps = t
O(t2)
Total time = ?
Universal TM
Tape2 after M’s headTape1Tape3Tape2 after M’s head
t
m
t
Running Time
Small “constant”
• Let m = |M| = size of description of TM M
as n and t grow.
• Let n = |I| = size of description of input I
• Let t = Time(M,I) = # time steps of M running on I
• Assume M also uses t tape so Muniversal’s second tape has length t.
Running time of Muniversal.
• With one tape:
•
Time to search for rule match = ? O(|qc|m)
Simulation time is
Number of simulation steps = t
≈ O(t) “linear” time in M’s.
Total time = ?
Other Famous Models
of Computation
Are these definitions
equivalent?
Church says “Yes”
All reasonable models
of computation are equivalent
as far as what they can compute.
Turing 1936
Primitive Recursive
f is defined from g & h if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Smaller instance (very specific)
Help from friend
My solution
Base case
Does not change. Could be x1,x2,..,xn.
Primitive Recursive
f is defined from g & h if
•f(x,y) = h(x,y-1,f(x,y-1))
•f(x,0) = g(x)
Initial functions:
•Zero(x) = 0
•Successor(x) = x+1
•In,i(x1,x2,..,xn ) = xi
Because PR had bounded and TM can run forever,
TM can compute more problems.
n
2
But if a TM’s time is limited to time 2
2
then they are equivalent.
Because PR can only add&subtract one
and a TM can manipulate bits,
PR takes exponentially longer.
Register Machines
•The model has a constant number of registers
that can take on any integer ≥ 0.
•A program is specified by code made from
the following commands:
•Ri = 0
•Ri = Ri+1
Can compute the same
•if Ri=Rj goto line k
problems as TM.
Ri
Rj
Because it can only add one
and a TM can manipulate bits,
it takes exponentially longer.
And/Or/Not Circuits
•A circuit is a directed acyclic graph of and/or/not gates
•An input X assigns a bit to each incoming wire.
x1 0
x2 1
x30
AND
AND
0
OR
0
1
NOT
OR
0
0
OR
0
The bits percolate
down to the output wires.
And/Or/Not Circuits
•Clearly circuits compute.
• Any function f(x) of n bits can be computed
with a nonuniform circuit of size O(2n).
• If a TM can compute it in time T(n), then it can be
computed by a uniform circuit of size O(T2(n)).
• Parallel time relates to circuit depth.
Arithmetic Circuits
•An arithmetic circuit has +, -, ×, & / gates.
•An input X assigns a real number to each incoming wire.
x1 3
x2 5
x37
×
+
8
+
21
12
×
168
-12
/
-14
The real numbers percolate
down to the output wires.
Neural Nets
Neural Nets
x1 x2 x3 … xn
w1 w2 w3 … wn
Threshold
T
y
•Inputs x1, x2 , x3 , …, xn and output y are binary.
•Weights w1 , w2 , w3 , …, wn are real numbers
(possibly negative).
•y = 1 iff Σi wi×xi ≥ T
•The neural net learns by adjusting weights wi.
Non-Deterministic Machines
A Non-Deterministic TM/Java/… is
•the same as a deterministic one,
except that its moves are not deterministic.
•It has a “choice” as to which step to take next.
•Deterministic:
(qi,c) = <qj,c',right>
•Non-Deterministic: <qi,c; qj,c',right> ϵ 
•A non-deterministic machine M on input I
is said to accept its input
•if there exists an accepting computation.
•Note problems computed need yes/no answers.
Jack Edmonds
Steve Cook
Non-Deterministic Machines
• If I is a “yes” instance:
• A non-deterministic (fairy god mother) could prove
it to you by telling you which moves to make.
• This information is said to be a witness.
• Given an instance I and a witness/solution S,
there is a poly-time deterministic alg Valid(I,S)
to test whether or not S is a valid.
• With this you can convince your non-believing boss
that I is a “yes” instance:
• If I is a “no” instance:
•There is no witness she could tell you.
•You cannot convince your boss.
• An equivalent definition:
• Problem P can be computed non-deterministically
if P(I) =  S Valid(I,S)
Non-Deterministic Machines
• Example: Satisfiablity:
• Instance: A circuit I.
• Solution: A satisfying assignment S.
• I is a yes instance if there is such an assignment .
• Given an instance I and a solution S,
there is a poly-time alg Valid(I,S) to test
whether or not S is a valid solution of I.
Assignment S:
0110101
Circuit I:
1
P(I) =  S Valid(I,S)
Quantum Machines
What about Quantum Machines?
•Based on quantum mechanics,
at each point in time a quantum TM
is in the super-position of any number
of configurations of normal TMs.
•In time T, it can only flip T quantum coins so can only
be in the super-position of 2T configurations.
•Hence, can be simulated by a normal TM in time 2T×T
time.
•Hence, Quantum Machines can’t compute more,
just sometimes faster.
•Factoring can be done in poly-time, ie 6=2×3.
•It is believed that NP-complete problems still take
exponential time.
Human
What about the human brain?
Can it compute more than a TM?
•Science:
•The brain is just an elaborate machine (neural net).
•Hence can’t do any more than a TM.
•New Age:
•Quantum mechanics is magical.
•The brain is based on quantum mechanics.
•Hence, the brain can do much more than a machine.
•But we already showed QM can’t compute more.
•Religion:
•The human has a soul.
•Hence, the brain can do much more than a machine.
Problems, Languages, Machines, and Classes
A Computational Problem P states
•For each possible input I
•What the required output P(I) is.
An Algorithm/Program/Machine M is
•A set of instructions
I
•On a given input I,
following the instructions
•produces output M(I)
•or runs forever.
Eg: Sorting
M(I)
=P(I)
M
Eg: Insertion Sort
M computes P iff
•It gives the correct answer for every input.
•I M(I) = P(I)
Problems, Languages, Machines, and Classes
A Computational Problem P states
•For each possible input I
•What the required output P(I) is.
A Language L is a yes/no problem
•P(I)  {yes,no}
Eg: Sorting
Eg: IsSorted
I
L
This square is the set of all
possible inputs I.
01101
An input could be any type of object,
but because any can be described by
a 0/1 string, we tend to assume the
inputs are 0/1 strings.
Problems, Languages, Machines, and Classes
A Computational Problem P states
•For each possible input I
•What the required output P(I) is.
A Language L is a yes/no problem
•P(I)  {yes,no}
•L = { I  {0,1}* | P(I) = yes }
Eg: Sorting
Eg: IsSorted
I
L
The inputs I in the set L are those
that require a yes answer.
01101
Those outside L require a no.
Iyes
Ino
Problems, Languages, Machines, and Classes
I
M(I)
=
•M halts with yes/accept
•M halts with no/reject
•M runs forever
 IL
 IL
 IL
M
I
The language L=L(M) is the set
of inputs I for which machine M
halts with yes/accept.
L(M)
01101
Iyes
Irun forever
Ino
The goal of machine M
is to distinguish between
IL and IL
Problems, Languages, Machines, and Classes
I
M(I)
=
•M halts with yes/accept
•M halts with no/reject
•M runs forever
 IL
 IL
 IL
M
This machine M is said to
Recognize/Accept
this language L=L(M)
(even though M might run
forever on IL.)
I
L(M)
01101
Iyes
Irun forever
Ino
The machine M is said to
Compute/Decide
the language L=L(M)
iff M halts on each input I.
Problems, Languages, Machines, and Classes
A complexity class is
a set of computational problems
that have a similar difficulty in
computing them.
Computable/Decidable
Problems which have
TM/Java programs that solve them
Exp
Problems which have
TM/Java programs that solve them
exponential time.
NP
Co-NP
Problems whose solutions can be checked Poly
in polynomial time.
Polynomial Time
Problems, Languages, Machines, and Classes
A complexity class is
Recognizable Co-Recognizable
a set of computational problems
Computable/Decidable
that have a similar difficulty in
computing them.
Exp
Decidable:
TM must halt on every input
and give the correct answer.
Recognizable:
On Yes instances,
TM must halt and accept.
On No instances
may halt and reject or run forever.
Co-Recognizable:
On Yes instances,
TM may halt and accept or run forever.
On No instances
TM must halt and reject.
Co-NP
NP
Poly
Problems, Languages, Machines, and Classes
A complexity class is
Recognizable Co-Recognizable
a set of computational problems
Computable/Decidable
that have a similar difficulty in
computing them.
Exp
L
A problem/language
LRecognizable
is a set of yes inputs.
I
Co-NP
NP
L(M)
Poly
01101
Iyes
Irun forever
Ino
Problems, Languages, Machines, and Classes
A complexity class is
Recognizable Co-Recognizable
a set of computational problems
Computable/Decidable
that have a similar difficulty in
computing them.
Exp
Proving C1  C2 is not too hard.
Prove C2 can simulate C1.
Proving C1  C2 is hard.
Prove P ϵ C2 and P ϵ C1.
Co-NP
NP
Poly
End