Remaining Topics




Decidability Concept
The Halting Problem
P vs. NP
NP-completeness &
Cook-Levin Theorem
4.1
4.2
7.2 and 7.3
7.4
Review: Turing Machines in a nutshell
Church-Turing Thesis

Turing Machine equal Notion of an Algorithm
Turing Machine


Most simple machine possible
Computational power of
modern computer and
 high-level language


Not particularly efficient in a practical sense
Review: Turing Machines in a nutshell
Most simply model possible…
 Adding another tape can improve
 efficiency
(time or computational “speed”)
 but not computational “power”
(ability to solve a problem).
 Every multi-tape TM has an equivalent single-tape TM
(Theorem 3.13, p.149)

Similar to an Automata
Non-determinism does NOT add any
“computational power” (Theorem 3.15, p.150)
Review: Turing Machines in a nutshell

Computational power of
modern computer and
 high-level language

Every operation and statement in a high
level language can be implemented with
a Turing Machine (TM)
 Just as statements can be combined
So can TMs (HW5 illustrates this)

Decidability 4.1

Is there an algorithm that can decide if
 An
item is in a set.
 A string is in a language
 A formula is a member of a theory

These are all variations of the same concept, i.e.,
the concept of decidability
Decidability in Languages


We will concentrate on this:
 Algorithms for deciding if a string is in a
language
But, the strings and languages are going to
represent deeper problems
ADFA = {<B,w> | B is a DFA that accepts input string w}
ADFA
ADFA =
{<B,w> | B is a DFA that accepts input string w}
 B is the encoding of a DFA
 Remember that you can encode a DFA as follows:


B = (Q,Σ, δ, qstart, F)
We are literally encoding the machine and the
input (w) as a string
“<({1,2,3},{a,b},{(1,2,a),(1,3,b)},1,{3}),abc>”
ADFA
ADFA =
{<B,w> | B is a DFA that accepts input string w}
 Testing whether a DFA accepts an input w is the
same as the problem of testing whether the string
<B,w> is a member of the language ADFA
Just as A = {w | w = (11)*} would accept the set {ε, 11,
1111, 111111, …}
 ADFA would enumerate all the <B,w>’s such that w is
accepted by the encoded B.

ADFA is decidable
ADFA =
{<B,w> | B is a DFA that accepts input string
w}
 What does this mean in plain English?
 How can we prove it?
ADFA is decidable
ADFA =
{<B,w> | B is a DFA that accepts input string
w}
 What does this mean in plain English?
 “An algorithm exists that can accept strings that
adhere to the definition of ADFA and reject string
that don’t”
ADFA is decidable
ADFA =
{<B,w> | B is a DFA that accepts input string
w}
 How can we prove it?
 Proof is on p.167
ANFA is decidable
ANFA =
{<B,w> | B is a NFA that accepts input string
w}
 How do we know this to be true?
 Hint: How are NFAs and DFAs different?
AREX is decidable
AREX =
{<R,w> | R is a Regular Expression that
generates the string w}
 How do we know this to be true?
 Hint: How are Regular Expressions and DFAs
related?
EDFA is decidable
EDFA =
{<A> | A is a DFA and L(A) is empty}
 Prove it
 Hint: Just as Turing Machine can “simulate” a
DFA it can also determine if a state is
unreachable.
EQDFA is decidable
EQDFA =
{<A,B> | A and B are DFAs and L(A) = L(B)}
 How do we know this to be true?
 Hint: Symmetric Difference formula
Decidability of Regular Languages

Deciding if
a
language is Regular or not
 If
a
given DFA, NFA or REX
Regular language is empty
 two Regular languages are equal
Decidability of Context Free

Deciding if
a
language is Context Free or not (Theorem 4.7)
 If
a
given CFG
Context Free language is empty (Theorem 4.8)
 two Context Free languages are equal
Classes of languages
Turing-recognized
Decidable
Context-Free (ACFG)
Regular (ADFA)
The Halting Problem


Will an algorithm halt on a given input.
Intuition:
 Can
you ever be sure that a loop is infinite?
 It might just terminate in a few minutes, hours, years,
millenniums, etc.

Sometimes you can make such a determination:
 while
(x > 0) {x=1;}
 But is it always possible to make such a determination?
Infinite Looping

DFA:



PDA:




by definition, upon consuming the input, the machine rejects unless
it is in an accept state.
Looping is simply not an option by definition.
very, very hard to make deterministic PDA’, but it can be done.
Once the input is consumed, empty transitions can move to a
reject/accept state.
Every CF language has a PDA that will halt (not loop).
TM:


Just like a high-level language TMs can loop forever.
Intuition: you don’t consume the input, you can move on the tape
infinitely, and the states can have a loop with no accept or reject.
ATM

ATM= {<M,w> | M is a TM and M accepts w}
U
= “on input <M,w> simulate M on w”
 If
M accepts, U accepts
 If M rejects, U rejects

Simple intuition:
M
could be a Turing Machine that loops forever on
certain input.
 If M loops forever, U cannot be a decider for ATM
Is ATM decidable?

ATM= {<M,w> | M is a TM and M accepts w}
U
= “on input <M,w> simulate M on w”
 If
M accepts, U accepts
 If M rejects, U rejects



BUT! Perhaps there is a way to implement M such
that we can detect the infinite loop?
Upon infinite loop detection, U rejects.
U could still be a decider for ATM
The Halting Problem

H(<M,w>) =
if M accepts w
if M rejects w
accept
reject
The Halting Problem is Undecidable

Proof: First, consider the machine/algorithm D:

D = “on input <M>, where M is a TM:
Run H on input <M,<M>>
 Output the opposite of what H outputs; that is; if H accepts,
reject and if H rejects, accept.”

Recall H:
 H(<M,w>) =
if M accepts w
if M rejects w
accept
reject
D is a crazy Decider Algorithm




D is implemented with a Turing Machine
D(<M>) = if D does not accept <M>,
if D accepts <M>,
accept
reject
What happens if we run D with its own Turing Machine
description?
D(<D>) = if D does not accept <D>,
accept
if D accepts <D>,
reject
A paradox emerges



D(<D>) = if D does not accept <D>,
if D accepts <D>,
If D accepts, how can D(<D>) reject?
We assumed that H could decide ATM because it could
‘somehow detect an infinite loop”


Think of H as a deterministic decider if a Turing Machine loops
Then, we use H to build D (the crazy decider)


accept
reject
Here we assume H can stop D from looping infinitely
Then, we run D on its own encoding, which creates a paradox.
Paradox resolved




Either H or D cannot exist.
Which one?
D is a TM machine that can simulate another Turing
Machine, which has been elegantly proven.
Intuition: Consider a program that can take another
program and simulate its execution.
Program, Algorithm, and Turing Machine are all synonymous
(Church-Turing Thesis)
 Compilers
 Virtual Machines

Significance of Turing Machines

Turing Machines are the “tool” we used to prove
that the Halting Problem is un-decidable.
 In
other words, no algorithm exists to determine if a
general algorithm will halt or not.
 Note: There are some algorithms where its easy to
show/prove that it will halt, but we are interested in the
general case (any/all algorithms).
Un-decidable Languages…

…there are many, but this is the interesting one:


Obviously, this language can’t be generated by a REX
or CFG.


ATM= {<M,w> | M is a TM and M accepts w}
So, a NFA, DFA, and PDA can’t be used as a decider to
accept/reject strings
But, even a Turing machine cannot act as a decider.
It may be able to decide some input on some machines, but
not all.
 There are strings in ATM that will cause the decider to loop
infinitely. Specifically <D,<D>> and likely other strings.

Significance of ATM

A formal language that cannot be decided by
Turing Machine.
 We
can define this language’s concept
 But we cannot create an algorithm (TM) to determine if
a string is in this language or not.
ATM
Turing-Decidable
Context-Free (ACFG)
Regular (ADFA)
Decidable vs. Recognizable
Turing Decidable Languages
 Language such that some TM
will accept all of its strings
 And, reject strings in the
language’s compliment
 Halts on all input
Turing Recognizable Languages
 Language such that some TM
will accept all of its strings
 But, might not halt on strings in
the language’s compliment
 Its it looping infinitely or will it
accept?

We don’t know.
ATM is Turing Recognizable

ATM= {<M,w> | M is a TM and M accepts w}

U = “on input <M,w> simulate M on w”



If M accepts, U accepts
If M rejects, U rejects
U will always halt if M halts. If M doesn’t halt on w than M
doesn’t accept w, so <M,w> isn’t in the language. By its
very definition U will always halt on strings in ATM.


The un-decidability is when U has been looping for 10 million
years,
we really don’t know


Is it eventually going to be an accepted w or
an infinite loop caused by a rejected w. This is why infinity is trouble.
Time Complexity




TM’s are a formal way to describe algorithms
Some problems don’t have algorithms that will
always halt, i.e., determining if a string is in ATM.
Algorithms that do halt can still take a long time.
How long is long?
General Time Unit



With Turing Machines we can define a unit of time
to be the execution time of one TM transition.
With more practical machines, a time unit could be
a CPU clock cycle, which might execute one
machine-level instruction.
Some machines can execute 1 billion instructions per
second, so the time unit would be 1/100000000
seconds.
Time as a function of input size




N is the size of the input
f(N) is the number of time unit to solve the problem.
The running time of algorithms can be expressed as
functions:
f(N) = 2N + 5;
 Two
loops of size N and 5 setup instructions
 Or, on loop of size N with two instruction inside and 5
instructions outside the loop.
Constants don’t matter

For really big problems, constants don’t matter
 f(N)


= 2N is the same as g(N) = 100N
While 100 days seems like forever compared to 2
days, parallel computation and faster computers
can eventually make up the difference (we hope).
For big problems,
= N4 is much different than g(N) = N2
 A faster computer may not help, why?
 f(N)
1.15 Days vs. 11 billion years
N
N^2
days
N^4
years
1
1
1.15741E-14
1
3.17E-17
2
4
4.62963E-14
16
5.07E-16
3
9
1.04167E-13
81
2.57E-15
4
16
1.85185E-13
256
8.12E-15
10
100
1.15741E-12
10000
3.17E-13
20
400
4.62963E-12
160000
5.07E-12
50
2500
2.89352E-11
6250000
1.98E-10
1000
1000000
1.15741E-08
1E+12
3.17E-05
10000
100000000
1.15741E-06
1E+16
3.17E-01
100000 10000000000
0.000115741
1E+20
3.17E+03
1000000
1E+12
0.011574074
1E+24
3.17E+07
10000000
1E+14
1.157407407
1E+28
3.17E+11
Big-O




Review
Constants don’t matter
Only the leading exponent matters
Why?
1.157407 days vs. 1.157419 days
N
N^2
days
N^2 + 100N
days
1
1
1.15741E-14
101
1.16898E-12
2
4
4.62963E-14
204
2.36111E-12
3
9
1.04167E-13
309
3.57639E-12
4
16
1.85185E-13
416
4.81481E-12
10
100
1.15741E-12
1100
1.27315E-11
20
400
4.62963E-12
2400
2.77778E-11
50
2500
2.89352E-11
7500
8.68056E-11
1000
1000000
1.15741E-08
1100000
1.27315E-08
10000
100000000
1.15741E-06
101000000
1.16898E-06
100000 10000000000
0.000115741
10010000000
0.000115856
1000000
1E+12
0.011574074
1.0001E+12
0.011575231
10000000
1E+14
1.157407407
1.00001E+14
1.157418981
Why do we only care about big N’s


Same reason I would worry about a $10,000 bill in
my wallet but not a penny.
Same reason I would worry about a trip to Mars
but not a trip to Menands.
Real Algorithm  TM Decider





Prepare for “hand-waving magic:”
Any algorithm that can be programmed can be
reduced into a language problem.
A = {<p,i,o> | p is the encoding/description of a
problem, i is the input, and o is the correct output.}
Deciding if a string is in L is the same thing as
solving the problem.
The TM that decides A solves problem p.
The class P




The class of languages that can be decided in
polynomial time.
Corresponds, the set of problems that can be solved
in polynomial time.
Polynomial is O(nk)
What are some problem in P that you have studied?
Did you know?

Every context free language is in P
The class NP





Non-deterministically Polynomial.
One way to think of this is NOT Polynomial.
Or, exponential
Or N!
But that is not the whole story.