COMP 382: Reasoning about
algorithms
Unit 9: Undecidability
1
[Slides adapted from Amos Israeli’s]
Limits to computation
There are problems for which there cannot be
an algorithm, provably.
– Undecidable problems
There are problems for which polynomial-time
algorithms are unlikely to exist
– NP-complete problems
Diagonalization
Technique for proving results about the
cardinality of sets
Invented by German mathematician Georg
Cantor
Cardinality
How many natural numbers are there? Infinity!
How many real numbers are there? Infinity!
Does the number of natural numbers equal the
number of real numbers?
How is the size of infinite sets measured?
Cardinality
The cardinality of a set is a property marking its
size.
Two sets have the same cardinality if there is a
(one-to-one) correspondence between their
elements
Example
A  1,2,3,4
B  2,4,6,8
A 1 2 3 4
B 2 4 6 8
Clearly, the cardinality of A is equal to the
cardinality of B.
How about the cardinality of infinite sets?
7
Question
Is the cardinality of natural numbers larger than
the cardinality of even natural numbers?
8
Naturals and even naturals
N = {0,1, 2,3, 4,..., n,...}
EN = {0, 2, 4, 6,8,..., 2n,...}
Correspondence:
f n   2n
9
Countable Sets
A set A is countable if it is either finite or its
cardinality is equal to the cardinality of N.
“A set is countable if a list of its elements can be
created”.
10
Countable Sets
“A set is countable if a list of its elements can be
created”.
This list does not have to be finite, but for each
natural number i, one should be able to
specify the i-th element on the list.
For example, for EN the i-th element on the list
is 2i .
11
Question
Is the set Q of rational numbers countable?
12
The set of rationals is countable
Recall that each natural number is defined by a
pair of natural numbers.
List rationals in
an infinite
rectangle.
1 / 1 1 / 2 1 / 3 1 / 4 1 / 5 ………
2 / 1 2 / 2 2 / 3 2 / 4 2 / 5 ………
3 / 1 3 / 2 3 / 3 3 / 4 3 / 5 ………
4 / 1 4 / 2 4 / 3 4 / 4 4 / 5 ………
5 / 1 5 / 2 5 / 3 5 / 4 5 / 5 ………
………………………………………………...
14
The set of rationals is countable
How can we form a list including all these
numbers?
Can’t reach all
numbers
15
1 / 1 1 / 2 1 / 3 1 / 4 1 / 5 ………
2 / 1 2 / 2 2 / 3 2 / 4 2 / 5 ………
3 / 1 3 / 2 3 / 3 3 / 4 3 / 5 ………
4 / 1 4 / 2 4 / 3 4 / 4 4 / 5 ………
5 / 1 5 / 2 5 / 3 5 / 4 5 / 5 ………
………………………………………………...
The set of rationals is countable
1/1 1/ 2 1/ 3 1/ 4 1/ 5
2 /1 2 / 2 2 / 3 2 / 4 2 / 5
3 /1 3 / 2 3 / 3 3 / 4 3 / 5
4 /1 4 / 2 4 / 3 4 / 4 4 / 5
5 /1 5 / 2 5 / 3 5 / 4 5 / 5
A better way
16
The set of rationals is countable
Note that some rational numbers appear more
than once. For example: all numbers on the
main diagonal are equal to 1, so this list is
not final.
In order to compute the actual place of a given
rational, we need to erase all duplicates, but
this is a technicality…
17
Are all sets countable?
The set of infinite binary sequences is not
countable.
18
Uncountable Sets
Assume that there exists a list of all binary
sequences. Such a list may look like this:
1 0 1 1 0
1 1 0 0 1
0 0 0 0 1
………
………
………
………
………
1 1 1 0 1
1 0 0 0 1
…………………….................
19
Uncountable Sets
But can you be sure that all sequences are in
this list?
In fact, There exist
infinitely many
sequences that
are not on the list:
20
1 0 1 1 0
1 1 0 0 1
0 0 0 0 1
………
………
………
………
………
1 1 1 0 1
1 0 0 0 1
…………………….................
Uncountable Sets
Consider for example S=0,0,1,1,0,… . The
sequence S is formed so that
S 1  1st elt. Of 1st seq.
1 0 1 1 0 ………
1 1 0 0 1 ………
S 2   2nd elt. Of 2nd seq.
S 3 
3rd
elt. Of
And so on …
21
3rd
seq.
0 0 0 0 1 ………
1 1 1 0 1 ………
1 0 0 0 1 ………
……………………...........
Uncountable Sets
Define S i   1  Si i  . Obviously, for every
i  N , the i-th element of S, S i  differs from
the i-th element of the i-th sequence in the
list, that is: The element on the diagonal.
Can the sequence S appear on the list?
22
Uncountable Sets
S1
1
0 1 1
0 ... ...
S 2 1 1 0 0 1 ... ...
S3 0 0 0 0 1 ... ...
S4
1 1 1
S5
1
Contradiction
0 1 ... ...
0 0 0 1 ... ...
... ... ... ... ... ... ... ...
S 0 0 1 1 0 ... ?
Diagonalization
23
Back to Turing machines
Turing machine = model of a program
Inputs coded as strings over a finite input
alphabet
Language = Set of strings
Intuition: A language corresponds to a
computational decision problem
Example
Problem: check if a given number n is even
Language:
{0, 2, 4, 6,8,..., 2n,...}
All inputs on which the answer is “yes”
Back to Turing machines
The collection of strings that M accepts is the
language of M, denoted LM  .
A language is Turing Recognizable if there exists
a Turing machine that recognizes it.
Turing deciders (algorithms)
Since it is hard to tell whether a running machine
is looping, we prefer machines that halt on all
inputs. These machines are called deciders.
A decider that recognizes a language L is said to
decide L.
A language is Turing decidable if there exists a
Turing machine that decides it, and
undecidable otherwise.
27
Turing unrecognizable languages
Claim
Some Languages are not Turing-recognizable.
Proof
For any (finite) alphabet,  , the set of (finite)
*
strings  , is countable. A list of all elements
in * is obtained by first listing strings of
length 1, then 2, …, then n…
28
Proof (cont.)
The set of all TM-s is also countable because
every TM, M , can be described by its
encoding M , which is a string over  . So
*
the set of TM-s corresponds to a subset of  .
Note: Here we use the (unproven but correct)
fact that the cardinality of a set is always not
greater then the cardinality of any of its
supersets.
29
Proof (cont.)
Since each TM recognizes exactly a single
language, a list of all TM-s can be used as a
list of all recognizable languages.
If we show that the set of languages over  is
uncountable, we can deduce that at least a
single language is not on the list, that is: it is
not recognized by any TM.
30
Proof (cont.)
We have already seen that the set of infinite
binary sequences is uncountable. Now we
form a correspondence between the set of
languages over  and the set of infinite
binary sequences to show that the set of
languages is uncountable.
31
Proof (cont.)
We have already seen that the set  is
countable. Consider a fixed list l of all words
*
in  . A correspondence between the set of
*
infinite binary sequences and the set  of
languages is formed as follows: For every
infinite binary sequence S, corresponds the
language:
L S   l i  | where si   1 QED
*
32
Moral of the story
There are more problems than programs
The Halting Problem
Now we’ll show an undecidable language.
The language that we prove to be undecidable is
a very natural language namely the language
consisting of pairs of the form M , w where M
is a TM accepting string w:


ATM  M , w  M is a TM acceptingw
34
The Halting Problem
Since this language requires to decide whether
the computation of TM M halts on input w, it
is often called The Halting Problem.
Is the halting problem Turing
Recognizable?
35
The Halting Problem
Since this language requires to decide whether
the computation of TM M halts on input w, it
is often called The Halting Problem.
Claim
The halting problem is Turing Recognizable.
36
Proof
Consider a TM U that gets a pair M , w as input
and simulates the run of M on input w. If M
accepts or rejects so does U. Otherwise, U
loops.
Note: U recognizes ATM ,since it accepts any pair
M , w  L , that is: any pair in which M accepts
input w.
37
A TM N that simulates an input TM N
TM N works as follows:
1. Mark M’s initial state and w’s initial symbol as
the “current state” and “current head
location”.
2. Look for M’s next transition on the
description of its transition function.
3. Execute M’s transition.
38
Simulating an Input TM
4. Move M’s “current state” and “current head
location” to their new places.
5. If M’s new state is a deciding state decide
according to the state, otherwise – repeat
stages 2-5.
39
The language ATM is undecidable
So far we proved the existence of a language
which is not Turing recognizable. Now we
continue our quest to prove:
Theorem
The language


ATM  M , w  M is a TM acceptingw
is undecidable.
40
The Language ATM is undecidable
Before we start the proof let us consider two
ancient questions:
Question1:
Can god create a boulder so heavy such that he
(god) cannot lift?
41
The Language ATM is undecidable
Question2:
In a small town, there is a single barber:
Over the barber’s chair there is a note saying:
“I will shave you on one condition: You shall
never shave yourself.”
Who Shaves the Barber?
42
Proof
Assume, by way of contradiction, that ATM is
decidable and let H be a TM deciding ATM .
That is H  M , w   accept

reject
if M acceptsw
if M does not acceptw
Define now another TM, D, that uses H as a
subroutine as follows:
43
Proof
Define now another TM new D that uses H as a
subroutine as follows:
D=“On input M where M is a TM:
1. Run H on input M, M
.
2. Output the opposite of H’s output
namely: If H accepts reject, otherwise
accept.“
44
Proof
Note: What we do here is taking advantage of
the two facts:
Fact1: TM M should be able to compute with
any string as input.
Fact2: The encoding of M, M , is a string.
45
Proof
Running a machine on its encoding is analogous
to using a compiler for the computer
language C, to compile itself (the compiler is
written in C).
46
Proof
What we got now is:
accept if M rejects M
D M 
 reject if M accepts M


Consider now the result of running D with
input D . What we get is:
accept if D rejects D
D D 
 reject if D accepts D
 
47
Proof
accept if D rejects D
D D 
 reject if D accepts D
 
So if D accepts, it rejects and if it rejects it
accepts.
And it’s all caused by our assumption that TM H
exists!
48
Proof review
1. Define ATM   M , w  M is a TM acceptingw .
2. Assume that ATM id decidable and let H be a
TM deciding it.
3. Use H to build TM D that gets a string and
behaves exactly opposite to H’s behavior,
namely:



accept if M rejects M
D M 
 reject if M accepts M
49
Proof review
4. Run TM D on its encoding D and conclude:
accept if D rejects D
D D 
 reject if D accepts D
 
Contradiction.
50
So where is the diagonalization?
The following table describes the behavior of
each machine on some machine encodings:
M1
M2
M3
M4
accept accept accept accept 

accept accept


51

M2

M3
accept

M4


M1
accept
So where is the diagonalization?
This table describes the behavior of TM H.
Note: TM H rejects where M i loops.
52
M3
M4

accept reject 
M1
M1
M2
accept reject
M2
M3
M4
accept accept accept accept 
reject reject reject reject 
accept accept reject reject 
Proof review
Now TM D is added to the table…
53
M1
M1
M2
M3
M4

D
accept reject accept reject  accept
M2
M3
M4
accept accept accept accept  accept
reject reject reject reject  reject
accept accept reject reject  accept

D

reject






reject accept accept 



???

Reductions
Use the undecidability of ATM to prove many
other languages undecidable.
54
Reductions
Show that a solution for some problem A
induces a solution for problem B.
If we know that B does not have a solution, we
may deduce that A is also unsolvable. In this
case we say that B is reducible to A.
55
Reductions
Say you want to prove that a certain language L
is undecidable.
Assume that L is decidable, and show that a
decider for L, can be used to devise a decider
for ATM .
Since ATM is undecidable, so is the language L.


ATM  M , w  M is a TM acceptingw
56
Reductions
Using a decider for L to construct a decider for
ATM , is called reducing ATM to L.
Note: Once we prove that a certain language L
is undecidable, we can prove that some other
language, say L’ , is undecidable, by reducing
L’ to L.
57
Schematic of a Reduction
1. We know that A is undecidable.
2. We want to prove B is undecidable.
3. We assume that B is decidable and use this
assumption to prove that A is decidable.
4. We conclude that B is undecidable.
Note: The reduction is from A to B.
58
The “Real” halting problem
Consider


HALTTM  M , w  M is a TM that halts on w
Theorem
HALTTM
is undecidable.
Proof
By reducing ATM to HALTTM


ATM  M , w  M is a TM acceptingw
59
Intuition
Assume by way of contradiction that HALTTM is
decidable.
Recall that a decidable set has a decider R: A TM
that halts on every input and either accepts
or rejects, but never loops!.
We will use the assumed decider of HALTTM to
devise a decider for ATM .
60
Proof
Assume that HALTTM is decidable and let R be a
TM deciding it.
We give a TM S that uses R as a subroutine and
decides ATM .
Since ATM is undecidable, this is a contradiction.
62
Proof
S=“On input M , w where M is a TM:
1. Run R on input M , w until it halts.
2. If R rejects, (i.e. M loops on w ) - reject.
(At this stage we know that R accepts, and we
conclude that M halts on input w.)
3. Simulate M on w until it halts.
4. If M accepts - accept, otherwise - reject. “
63
The TM Emptiness Problem
Show that the following language is
undecidable.

ETM  M  M is a TM And LM   
64

Proof Outline
The proof is by reduction from ATM :
1. We know that ATM is undecidable.
2. We want to prove ETM is undecidable.
3. We assume toward a contradiction that ETM
is decidable and devise a decider for ATM .
4. We conclude that ETM is undecidable.
65
Proof
Assume that ETM is decidable and let R be a TM
deciding it.
Now we give a TM S that uses R as a subroutine
and decides ATM .
66
Proof
Given an instance for ATM , M , w , we may try
to run R on this instance. If R accepts, we
know that L M    . In particular, M does not
accept w so a decider for ATM must reject
M,w .
67
Proof
What happens if R rejects? The only conclusion
we can draw is that L M    . What we need
to know though is whether w  LM  .
In order to use our decider R for ETM , we once
again modify the input machine M to obtain
TM M 1 :
68
Description of M 1
We start with a TM satisfying LM1   LM  .
M1
nq start
nqaccept
M
qstart
qaccept
qreject
nq reject
69
Description of M 1
Now we add a filter to divert all inputs but w.
M1
nqaccept
x  w
M
filter
yes
nq start
no
qaccept
xw
qstart
xw
w if M acceptsw
LM1   
  if M rejectsw
70
qreject
nq reject
Proof
TM M 1 has a filter that rejects all inputs
excepts w, so the only input reaching M, is w.
Therefore, M 1 satisfies:
w if M acceptsw
LM1   
  if M rejectsw
71
Proof
Here is a formal description of M 1 :
M 1  “On input x :
1. If x  w - reject .
2. If x  w - run M on w and accept if M
accepts. ”
Note: M accepts w if and only if LM 1    .
72
Proof
S=“On input M , w where M is a TM:
1. Compute an encoding M1 of TM M 1 .
2. Run R on input M1 .
3. If R rejects - accept, otherwise - reject.
74
Proof
Recall that R is a decider for ETM . If R rejects the
modified machine M 1 , LM 1    , hence by
the specification of M 1 , w  LM  , and a
decider for ATM must accept M , w .
If however R accepts, it means that LM 1    ,
hence w  LM  , and S must reject M , w .
QED
75
Rice’s Theorem
We will show that for every nontrivial property P of
recognizable languages is undecidable.
Consider the language
LP = { <M> | M satisfies property P }
We show how to reduce ATM to LP.
Since ATM is undecidable, it follows that LP is also
undecidable.
76
The Reduction
Our reduction algorithm takes M and w and
produces a TM M’.
L(M’) has property P if and only if M accepts w.
M’ has two tapes, used for:
1. Simulates another TM ML on the input to M’.
2. Simulates M on w.
•
neither M, ML, nor w is input to M’.
77
The Reduction (continued)
Assume that  does not have property P.
– If it does, consider the complement of P, which
would also be decidable.
Let L be any language with property P, and let
ML be a TM that accepts L.
M’ is constructed to work as follows.
78
Design of M’
1. On the second tape, write w and then
simulate M on w.
2. If M accepts w, then simulate ML on the
input x to M’, which appears initially on the
first tape.
3. M’ accepts its input x if and only if ML
accepts x.
79
Action of M’ if M Accepts w
On accept
x
Simulate M
on input w
Simulate ML
on input x
Accept
iff x is
in ML
80
Design of M’ (continued)
Suppose M accepts w.
Then M’ simulates ML and therefore accepts
x if and only if x is in L.
That is, L(M’) = L, L(M’) has property P.
81
Design of M’ – (3)
Suppose M does not accept w.
Then M’ never starts the simulation of ML, and
never accepts its input x.
Thus, L(M’) = , and L(M’) does not have
property P.
That is, M’ is not in LP.
82
Action of M’ if M Does not Accept w
x
Simulate M
on input w
Never accepts, so
nothing else happens
and x is not accepted
83
Design of M’ – Conclusion
• Thus, the algorithm that converts M and w
to M’ is a reduction of ATM to LP.
• Thus, LP is undecidable.
84
Picture of the Reduction
M, w
A real
reduction
algorithm
M’
Accept
iff M
accepts w
Hypothetical
algorithm for
property P
Otherwise
halt without
accepting
This would be an algorithm
for Lu, which doesn’t exist
85
Applications of Rice’s Theorem
• We now have any number of undecidable
questions about TM’s:
– Is a TM equivalent to a finite automaton?
– Does a TM accept any palindromes?
– Does a TM accept more than 1000 strings?
– ...
86