Strings and Languages Operations
Concatenation
Exponentiation
Kleene Star
Regular Expressions
1
Strings and Language Operations
• Concatenation
• Exponentiation
• Kleene star
• Pages 28-32 of the recommended text
• Regular expressions
• Pages 85-90 of the recommended text
2
String Concatenation
• If x and y are strings over alphabet S, the
concatenation of x and y is the string xy
formed by writing the symbols of x and the
symbols of y consecutively.
• Suppose x = abb and y = ba
• xy = abbba
• yx = baabb
3
Properties of String Concatenation
• Suppose x, y, and z are strings.
• Concatenation is not commutative.
• xy is not guaranteed to be equal to yx
• Concatenation is associative
• (xy)z = x(yz) = xyz
• The empty string is the identity for
concatenation
• x/\ = /\x = x
4
Language Concatenation
• Suppose L1 and L2 are languages (sets of strings).
• The concatenation of L1 and L2, denoted L1L2,is
defined as
• L1L2 = { xy | x  L1 and y  L2 }
• Example,
• Let L1 = { ab, bba } and L2 = { aa, b, ba }
• What is L1L2?
• Solution
• Let x1= ab, x2= bba, y1= aa, y2= b, y3= ba
• L1L2 = { x1y1, x1y2, x1y3, x2y1, x2y2, x2y3 }
= { abaa, abb, abba, bbaaa, bbab, bbaba}
5
Language Concatenation is not
commutative
• Let L1 = { aa, bb, ba } and L2 = { /\, aba }
• Let x1= aa, x2= bb, x3=ba, y1= /\, y2= aba
• L1L2 = { x1y1, x1y2, x2y1, x2y2, x3y1, x3y2 }
= { aa, aaaba, bb, bbaba, ba, baaba }
• L2L1 = { y1x1, y1x2, y1x3, y2x1, y2x2, y2x3 }
= { aa, bb, ba, abaaa, ababb, ababa }
• L2L2 = { y1y1, y1y2, y2y1, y2y2 }
= { /\, aba, aba, abaaba }
= { /\, aba, abaaba } (dropped extra aba)
6
Associativity of Language
Concatenation
• (L1L2)L3 = L1(L2L3) = L1L2L3
• Example
• Let L1={a,b}, L2={c,d}, and L3={e,f}
• L1L2L3=({a,b}{c,d}){e,f}
={ac, ad, bc, bd}{e,f}
={ ace,acf,ade,aef,bce,bcf,bde,bdf }
• L1L2L3={a,b}({c,d}{e,f})
={a,b}{ce, df, ce, df}
={ ace,acf,ade,aef,bce,bcf,bde,bdf }
7
Special Cases
• What language is the identity for language
concatenation?
• The set containing only the empty string /\: {/\}
• Example
• {aab,ba,abc}{/\} = {/\}{aab,ba,abc} = {aab,ba,abc}
• What about {}?
• For any language L, L {} = {} L = {}
• Thus {} for concatenation is like 0 for multiplication
• Example
• {aab,ba,abc}{} = {}{aab,ba,abc} = {}
• The intuitive reason is that we must choose a string from both
sets that are being concatenated, but there is nothing to choose
from {}.
8
Exponentiation
• We use exponentiation to indicate the number of
items being concatenated
•
•
•
•
•
•
•
•
Symbols
Strings
Set of symbols (S for example)
Set of strings (languages)
a3 = aaa
x3 = xxx
S3 = SSS = { x  S* | |x|=3 }
L3 = LLL
9
Examples of Exponentiation
Let x=abb, S={a,b}, L={ab,b}
a4 = aaaa
x3 = (abb)(abb)(abb) = abbabbabb
S3 = SSS = {a,b}{a,b}{a,b}
={aaa,aab,aba,abb,baa,bab,bba,bbb}
• L3 = LLL = {ab,b}{ab,b}{ab,b}
= {ababab,ababb,abbab,abbb,
babab,babb,bbab,bbb}
•
•
•
•
10
Results of Exponentiation
• Exponentiation of a symbol or a string
results in a string.
• Exponentiation of a set of symbols or a set
of strings results in a set of strings
•
•
•
•
a symbol  a string
a string  a string
a set of symbols  a set of strings
a set of strings  a set of strings
11
Special Cases of Exponentiation
•
•
•
•
a0 = /\
x0 = /\
S0 = { /\ }
L0 = { /\ } for any language L
•
•
•
•
{aa,bb}0 = { /\ }
{ a, aa, aaa, aaaa, …}0 = { /\ }
{ /\ }0 = { /\ }
0 = { }0 = { /\ }
12
Kleene Star
• Kleene * is a unary operation on languages.
• Kleene * is not an operation on strings
• However, see the pages on regular expressions.
• L* represents any finite number of concatenations
of L.
L* = Uk>0 Lk = L0 U L1 U L2 U …
• For any L, /\ is always an element of L*
• because L0 = { /\ }
• Thus, for any L, L* != 
13
Example of Kleene Star
•
•
•
•
•
•
•
•
Let L={aa}
L0={ /\ }
L1=L={aa }
L2={ aaaa }
L3= …
L* = L0  L1  L2  L3 …
= { /\, aa, aaaa, aaaaaa, … }
= set of all strings that can be obtained by
concatenating 0 or more copies of aa
14
Example of Kleene Star
•
•
•
•
•
•
•
Let L={aa, b}
L0={ /\ }
L1=L={aa,b}
L2= LL={ aaaa, aab, baa, bb}
L3= …
L* = L0  L1  L2  L3 …
= set of all strings that can be obtained by
concatenating 0 or more copies of aa and b
15
Regular Languages
• Regular languages are languages that can be
obtained from the very simple languages
over S, using only
• Union
• Concatenation
• Kleene Star
16
Examples of Regular Languages
• {aab} (i.e. {a}{a}{b} )
• {aa,b} (i.e. {a}{a}  {b} )
• {a,b}*
language of strings that can be obtained by
concatenating any number of a’s and b’s
• {bb}{a,b}* language of strings that begin with bb
(followed by any number of a’s and b’s)
• {a}*{bb,/\} language of strings that begin with any
number of a’s and end with an optional bb.
• {a}*{b}* language of strings that consist of only a’s or
only b’s and /\.
17
Regular Expressions
• We can simplify the formula for regular
languages slightly by
• leaving out the set brackets { } and
• replacing  with +
• The results are called regular expressions.
18
Examples of Regular Expressions
Set notation
Regular
Expressions
aab
{aab}
{aa,b}
= {aa}{b}
aa+b
{a,b}*
= ({a}{b})*
(a+b)*
{bb}{a,b}* = {bb}({a}{b})*
bb(a+b)*
{a}*{bb,/\} = {a}*({bb}{/\})
a*(bb+/\)
{a}*{b}*
a*+b*
19
String or Language?
• Consider the regular expression a*(bb+/\)
• a*(bb+/\) is a string over alphabet {a, b, *, +, /\, (, ),  }
• a*(bb+/\) represents a language over alphabet {a, b}
• It represents the language of strings over {a,b} that begin with any
number of a’s and end with an optional bb.
• Some regular expressions look just like strings over
alphabet {a,b}
• Regular expression aaba represents the language {aaba}
• Regular expression /\ represents the language {/\}
• It should be clear from the context whether a sequence of
symbols is a regular expression or just a string.
20
Module 1: Course Overview
• Course: CSE 460
• Instructor: Dr. Eric Torng
• Grader: Yi Liu
21
What is this course?
• Philosophy of computing course
• We take a step back to think about computing in
broader terms
• Science of computing course
• We study fundamental ideas/results that shape
the field of computer science
• “Applied” computing course
• We learn study a broad range of material with
relevance to computing today
22
Philosophy
• Phil. of life
• What is the purpose of
life?
• What are we capable of
accomplishing in life?
• Are there limits to
what we can do in life?
• Why do we drive on
parkways and park on
driveways?
• Phil. of computing
• What is the purpose of
programming?
• What can we achieve
through programming?
• Are there limits to what we
can do with programs?
• Why don’t debuggers
actually debug programs?
23
Science
• Physics
• Study of fundamental
physical laws and
phenomenon like
gravity and electricity
• Engineering
• Governed by physical
laws
• Our material
• Study of fundamental
computational laws
and phenomenon like
undecidability and
universal computers
• Programming
• Governed by
computational laws
24
Applied computing
• Applications are not immediately obvious
• In some cases, seeing the applicability of this
material requires advanced abstraction skills
• Every year, there are people who leave this course
unable to see the applicability of the material
• Others require more material in order to
completely understand their application
• for example, to understand how regular expressions
and context-free grammars are applied to the design
of compilers, you need to take a compilers course
25
Some applications
• Important programming languages
• regular expressions (perl)
• finite state automata (used in hardware design)
• context-free grammars
• Proofs of program correctness
• Subroutines
• Using them to prove problems are unsolvable
• String searching/Pattern matching
• Algorithm design concepts such as recursion
26
Fundamental Theme *
• What are the capabilities and limitations of
computers and computer programs?
• What can we do with computers/programs?
• Are there things we cannot do with
computers/programs?
27
Module 2: Fundamental Concepts
• Problems
• Programs
• Programming languages
28
Problems
We view solving problems as the
main application for computer
programs
29
Definition
• A problem is a mapping or function between
a set of inputs and a set of outputs
• Example Problem: Sorting
(4,2,3,1)
(3,1,2,4)
(7,5,1)
(1,2,3)
Inputs
(1,2,3,4)
(1,5,7)
(1,2,3)
Outputs
30
How to specify a problem
• Input
• Describe what an input instance looks like
• Output
• Describe what task should be performed on the
input
• In particular, describe what output should be
produced
31
Example Problem Specifications*
• Sorting problem
• Input
– Integers n1, n2, ..., nk
• Output
– n1, n2, ..., nk in nondecreasing order
• Find element problem
• Input
– Integers n1, n2, …, nk
– Search key S
• Output
– yes if S is in n1, n2, …, nk, no otherwise
32
Programs
Programs solve problems
33
Purpose
• Why do we write programs?
• One answer
• To solve problems
• What does it mean to solve a problem?
• Informal answer: For every legal input, a correct
output is produced.
• Formal answer: To be given later
34
Programming Language
• Definition
• A programming language defines what
constitutes a legal program
• Example: a pseudocode program may not be a
legal C++ program which may not be a legal C
program
• A programming language is typically referred to
as a “computational model” in a course like this.
35
C++
• Our programming language will be C++
with minor modifications
• Main procedure will use input parameters in a
fashion similar to other procedures
• no argc/argv
• Output will be returned
• type specified by main function type
36
Maximum Element Problem
• Input
• integer n ≥ 1
• List of n integers
• Output
• The largest of the n integers
37
C++ Program which solves the
Maximum Element Problem*
int main(int A[], int n) {
int i, max;
if (n < 1)
return (“Illegal Input”);
max = A[0];
for (i = 1; i < n; i++)
if (A[i] > max)
max = A[i];
return (max);
}
38
Fundamental Theme
Exploring capabilities and limitations
of C++ programs
39
Restating the Fundamental Theme *
• We will study the capabilities and limits of
C++ programs
• Specifically, we will try and identify
• What problems can be solved by C++ programs
• What problems cannot be solved by C++
programs
40
Question
• Is C++ general enough?
• Or is it possible that there exists some
problem P such that
• P can be solved by some program P in some
other reasonable programming language
• but P cannot be solved by any C++ program?
41
Church’s Thesis (modified)
• We have no proof of an answer, but it is
commonly accepted that the answer is no.
• Church’s Thesis (three identical statements)
• C++ is a general model of computation
• Any algorithm can be expressed as a C++
program
• If some algorithm cannot be expressed by a
C++ program, it cannot be expressed in any
reasonable programming language
42
Summary *
• Problems
• When we talk about what programs can or
cannot “DO”, we mean what PROBLEMS can
or cannot be solved
43
Module 3: Classifying Problems
• One of the main themes of this course will be to
classify problems in various ways
• By solvability
• Solvable, “half-solvable”, unsolvable
• We will focus our study on decision problems
• function (one correct answer for every input)
• finite range (yes or no is the correct output)
44
Classification Process
• Take some set of problems and partition it
into two or more subsets of problems where
membership in a subset is based on some
shared problem characteristic
Subset 1
Set of Problems
Subset 2
Subset 3
45
Classify by Solvability
• Criteria used is whether or not the problem
is solvable
• that is, does there exist a C++ program which
solves the problem?
Set of All Problems
Solvable Problems Unsolvable Problems
46
Function Problems
• We will focus on problems where the
mapping from input to output is a function
Set of All Problems
Non-Function Problems Function Problems
47
General (Relation) Problem
• the mapping is a relation
• that is, more than one output is possible for a
given input
Inputs
Outputs
48
Criteria for Function Problems
• mapping is a function
• unique output for each input
Inputs
Outputs
49
Example Non-Function Problem
• Divisor Problem
• Input: Positive integer n
• Output: A positive integral divisor of n
9
Inputs
1
3
9
Outputs
50
Example Function Problems
• Sorting
• Multiplication Problem
• Input: 2 integers x and y
• Output: xy
2,5
Inputs
10
Outputs
51
Another Example *
• Maximum divisor problem
• Input: Positive integer n
• Output: size of maximum divisor of n smaller
than n
9
Inputs
1
3
9
Outputs
52
Decision Problems
• We will focus on function problems where
the correct answer is always yes or no
Set of Function Problems
Non-Decision Problems Decision Problems
53
Criteria for Decision Problems
• Output is yes or no
• range = {Yes, No}
• Note, problem must be a function problem
• only one of Yes/No is correct
Yes
No
Inputs
Outputs
54
Example
• Decision sorting
• Input: list of integers
• Yes/No question: Is the list in nondecreasing
order?
(1,3,2,4)
(1,2,3,4)
Inputs
Yes
No
Outputs
55
Another Example
• Decision multiplication
• Input: Three integers x, y, z
• Yes/No question: Is xy = z?
(3,5,14)
(3,5,15)
Inputs
Yes
No
Outputs
56
A Third Example *
• Decision Divisor Problem
• Input: Two integers x and y
• Yes/No question: Is y a divisor of x?
(14,5)
(14,7)
Inputs
Yes
No
Outputs
57
Focus on Decision Problems
Set of All Problems
Solvable Problems Unsolvable Problems
Other
Probs
Decision
Problems
• When studying solvability, we are going to
focus specifically on decision problems
• There is no loss of generality, but we will not
explore that here
58
Finite Domain Problems
• These problems have only a finite number of
inputs
Set of All Problems
Finite Domain Problems Infinite Domain Problems
59
Lack of Generality
Set of All Problems
Solvable Problems Unsolvable Problems
Infinite
Domain
Finite
Domain
Empty
• All finite domain problems can be solved
using “table lookup” idea
60
Table Lookup Program
int main(string x) {
switch x {
case “Bill”: return(3);
case “Judy”: return(25);
case “Tom”: return(30);
default: cerr << “Illegal input\n”;
}
61
Key Concepts
• Classification Theme
• Decision Problems
• Important subset of problems
• We can focus our attention on decision
problems without loss of generality
• Same is not true for finite domain problems
• Table lookup
62
Module 4: Formal Definition of
Solvability
• Analysis of decision problems
• Two types of inputs:yes inputs and no inputs
• Language recognition problem
• Analysis of programs which solve decision problems
• Four types of inputs: yes, no, crash, loop inputs
• Solving and not solving decision problems
• Classifying Decision Problems
• Formal definition of solvable and unsolvable decision
problems
63
Analyzing Decision Problems
Can be defined by two sets
64
Decision Problems and Sets
• Decision problems consist of 3 sets
• The set of legal input instances (or universe of
input instances)
• The set of “yes” input instances
• The set of “no” input instances
Set of All Legal Inputs
Yes Inputs
No Inputs
65
Redundancy *
• Only two of these sets are needed; the third
is redundant
• Given
• The set of legal input instances (or universe of input
instances)
– This is given by the description of a typical input instance
• The set of “yes” input instances
– This is given by the yes/no question
• We can compute
• The set of “no” input instances
66
Typical Input Universes
• S*: The set of all finite length strings over finite
alphabet S
• Examples
• {a}*: {/\, a, aa, aaa, aaaa, aaaaa, … }
• {a,b}*: {/\, a, b, aa, ab, ba, bb, aaa, aab, aba, abb, … }
• {0,1}*: {/\, 0, 1, 00, 01, 10, 11, 000, 001, 010, 011, … }
• The set of all integers
• If the input universe is understood, a decision
problem can be specified by just giving the set of
yes input instances
67
Language Recognition Problem
• Input Universe
• S* for some finite alphabet S
• Yes input instances
• Some set L subset of S*
• No input instances
• S* - L
• When S is understood, a language
recognition problem can be specified by just
stating what L is.
68
Language Recognition Problem *
• Traditional
Formulation
• Input
• A string x over some
finite alphabet S
• Task
• Is x in some language L
subset of S*?
• 3 set formulation
• Input Universe
•
S* for a finite alphabet S
• Yes input instances
• Some set L subset of S*
• No input instances
•
S* - L
• When S is understood, a
language recognition
problem can be specified
by just stating what L is.
69
Equivalence of Decision Problems and
Languages
• All decision problems can be formulated as
language recognition problems
• Simply develop an encoding scheme for
representing all inputs of the decision problem
as strings over some fixed alphabet S
• The corresponding language is just the set of
strings encoding yes input instances
• In what follows, we will often use decision
problems and languages interchangeably
70
Visualization *
Yes Inputs
Language L
Encoding
Scheme over
alphabet S
No Inputs
Original Decision
Problem
S* - L
Corresponding
Language Recognition
Problem
71
Analyzing Programs which Solve
Decision Problems
Four possible outcomes
72
Program Declaration
• Suppose a program P is designed to solve
some decision problem P. What does P’s
declaration look like?
• What should P return on a yes input instance?
• What should P return on a no input instance?
73
Program Declaration II
• Suppose a program P is designed to solve a
language recognition problem P. What does P’s
declaration look like?
• bool main(string x) {
• We will assume that the string declaration is correctly defined
for the input alphabet S
– If S = {a,b}, then string will define variables consisting of only
a’s and b’s
– If S = {a, b, …, z, A, …, Z}, then string will define variables
consisting of any string of alphabet characters
74
Programs and Inputs
• Notation
• P denotes a program
• x denotes an input for program P
• 4 possible outcomes of running P on x
• P halts and says yes: P accepts input x
• P halts and says no: P rejects input x
• P halts without saying yes or no: P crashes on input x
• We typically ignore this case as it can be combined with rejects
• P never halts: P infinite loops on input x
75
Programs and the Set of Legal Inputs
• Based on the 4 possible outcomes of running P on
x, P partitions the set of legal inputs into 4 groups
• Y(P): The set of inputs P accepts
• When the problem is a language recognition problem, Y(P) is often
represented as L(P)
• N(P): The set of inputs P rejects
• C(P): The set of inputs P crashes on
• I(P): The set of inputs P infinite loops on
• Because L(P) is often used in place of Y(P) as described above, we
use notation I(P) to represent this set
76
Illustration
All Inputs
Y(P)
N(P)
C(P)
I(P)
77
Analyzing Programs and Decision
Problems
Distinguish the two carefully
78
Program solving a
decision problem
Y(P)
N(P)
C(P)
I(P)
• Formal Definition:
• A program P solves decision problem P if and only if
• The set of legal inputs for P is identical to the set of input
instances of P
• Y(P) is the same as the set of yes input instances for P
• N(P) is the same as the set of no input instances for P
• Otherwise, program P does not solve problem P
• Note C(P) and I(P) must be empty in order for P to solve
problem P
79
Solvable Problem
• A decision problem P is solvable if and only
if there exists some C++ program P which
solves P
• When the decision problem is a language
recognition problem for language L, we often
say that L is solvable or L is decidable
• A decision problem P is unsolvable if and
only if all C++ programs P do not solve P
• Similar comment as above
80
Illustration of Solvability
Inputs of Program P
Y(P)
N(P)
Yes Inputs
No Inputs
C(P)
I(P)
Inputs of Problem P
81
Program halfsolving a problem
Y(P)
N(P)
C(P)
I(P)
• Formal Definition:
• A program P half-solves problem P if and only if
• The set of legal inputs for P is identical to the set of
input instances of P
• Y(P) is the same as the set of yes input instances for P
• N(P) union C(P) union I(P) is the same as the set of
no input instances for P
• Otherwise, program P does not half-solve
problem P
• Note C(P) and I(P) need not be empty
82
Half-solvable Problem
• A decision problem P is half-solvable if and
only if there exists some C++ program P
which half-solves P
• When the decision problem is a language
recognition problem for language L, we often
say that L is half-solvable
• A decision problem P is not half-solvable if
and only if all C++ programs P do not halfsolve P
83
Illustration of Half-Solvability *
Inputs of Program P
Y(P)
Yes Inputs
N(P)
C(P)
I(P)
No Inputs
Inputs of Problem P
84
Hierarchy of Decision Problems
All decision problems
Half-solvable
Solvable
The set of half-solvable decision problems is a proper
subset of the set of all decision problems
The set of solvable decision problems is a proper subset of
the set of half-solvable decision problems.
85
Why study half-solvable problems?
• A correct program must halt on all inputs
• Why then do we define and study half-solvable problems?
• One Answer: the set of half-solvable problems is the
natural class of problems associated with general
computational models like C++
• Every program half-solves some decision problem
• Some programs do not solve any decision problem
• In particular, programs which do not halt do not solve their
corresponding decision problems
86
Key Concepts
• Four possible outcomes of running a program on
an input
• The four subsets every program divides its set of
legal inputs into
• Formal definition of
• a program solving (half-solving) a decision problem
• a problem being solvable (half-solvable)
• Be precise: with the above two statements!
87
Module 5
• Topics
• Proof of the existence of unsolvable problems
• Proof Technique
– There are more problems/languages than there are
programs/algorithms
– Countable and uncountable infinities
88
Overview
• We will show that there are more problems
than programs
• Actually more problems than programs in any
computational model (programming language)
• Implication
• Some problems are not solvable
89
Preliminaries
Define set of problems
Observation about programs
90
Define set of problems
• We will restrict the set of problems to be the
set of language recognition problems over
the alphabet {a}.
• That is
• Universe: {a}*
• Yes Inputs: Some language L subset of {a}*
• No Inputs: {a}* - L
91
Set of Problems *
• The number of distinct problems is given by
the number of languages L subset of {a}*
• 2{a}* is our shorthand for this set of subset
languages
• Examples of languages L subset of {a}*
•
•
•
•
0 elements: { }
1 element: {/\}, {a}, {aa}, {aaa}, {aaaa}, …
2 elements: {/\, a}, {/\, aa}, {a, aa}, …
Infinite # of elements: {an | n is even}, {an | n is
prime}, {an | n is a perfect square}
92
Infinity and {a}*
• All strings in {a}* have finite length
• The number of strings in {a}* is infinite
• The number of languages L in 2{a}* is
infinite
• The number of strings in a language L in
2{a}* may be finite or infinite
93
Define set of programs
• The set of programs we will consider are
the set of legal C++ programs as defined in
earlier lectures
• Key Observation
• Each C++ program can be thought of as a finite
length string over alphabet SP
• SP = {a, …, z, A, …, Z, 0, …, 9, white space,
punctuation}
94
Example *
int main(int A[], int n){
int i, max;
{26 characters including newline}
{13 characters including initial tab}
{1 character: newline}
if (n < 1)
return (“Illegal Input”);
max = A[0];
for (i = 1; i < n; i++)
if (A[i] > max)
max = A[i];
return (max);
}
{12 characters}
{28 characters including 2 tabs}
{13 characters}
{25 characters}
{18 characters}
{15 characters}
{15 characters}
{2 characters including newline}
95
Number of programs
• The set of legal C++ programs is clearly
infinite
• It is also no more than |SP*|
• SP = {a, …, z, A, …, Z, 0, …, 9, white space,
punctuation}
96
Goal
• Show that the number of languages L in
2{a}* is greater than the number of strings in
SP*
• SP = {a, …, z, A, …, Z, 0, …, 9, white space,
punctuation}
• Problem
• Both are infinite
97
How do we compare the relative sizes
of infinite sets?
Bijection (yes)
Proper subset (no)
98
Bijections
• Two sets have EQUAL size if there exists a
bijection between them
• bijection is a 1-1 and onto function between
two sets
• Examples
• Set {1, 2, 3} and Set {A, B, C}
• Positive even numbers and positive integers
99
Bijection Example
• Positive Integers Positive Even Integers
1
2
3
...
i
…
2
4
6
...
2i
...
100
Proper subset
• Finite sets
• S1 proper subset of S2 implies S2 is strictly
bigger than S1
• Example
– women proper subset of people
– number of women less than number of people
• Infinite sets
• Counterexample
• even numbers and integers
101
Two sizes of infinity
Countable
Uncountable
102
Countably infinite set S *
• Definition 1
• S is equal in size (bijection) to N
• N is the set of natural numbers {1, 2, 3, …}
• Definition 2 (Key property)
• There exists a way to list all the elements of set
S (enumerate S) such that the following is true
• Every element appears at a finite position in the
infinite list
103
Uncountable infinity *
• Any set which is not countably infinite
• Examples
• Set of real numbers
• 2{a}*, the set of all languages L which are a
subset of {a}*
• Further gradations within this set, but we
ignore them
104
Proof
105
(1) The set of all legal C++ programs is
countably infinite
• Every C++ program is a finite string
• Thus, the set of all legal C++ programs is a
language LC
• This language LC is a subset of SP*
106
For any alphabet S, S* is countably
infinite
• Enumeration ordering
• All length 0 strings
• |S|0 = 1 string: l
• All length 1 strings
• |S| strings
• All length 2 strings
• |S|2 strings
• …
• Thus, SP* is countably infinite
107
Example with alphabet {a,b} *
• Length 0 strings
• 0 and l
• Length 1 strings
• 1 and a, 2 and b
• Length 2 strings
• 3 and aa, 4 and ab, 5 and ba, 6 and bb, ...
• Question
• write a program that takes a number as input and
computes the corresponding string as output
108
(2) The set of languages in 2{a}* is
uncountably infinite
• Diagonalization proof technique
• “Algorithmic” proof
• Typically presented as a proof by contradiction
109
Algorithm Overview *
• To prove this set is uncountably infinite, we
construct an algorithm D that behaves as
follows:
• Input
• A countably infinite list of languages L[] subset of {a}*
• Output
• A language D(L[]) which is a subset of {a}* that is not
on list L[]
110
Visualizing D
List L[]
L[0]
L[1]
L[2]
Algorithm D
Language
D(L[]) not in
list L[]
L[3]
...
111
Why existence of D implies result
• If the number of languages in 2{a}* is countably
infinite, there exists a list L[] s.t.
• L[] is complete
• it contains every language in 2{a}*
• L[] is countably infinite
• The existence of algorithm D implies that no
list of languages in 2{a}* is both complete and
countably infinite
• Specifically, the existence of D shows that any
countably infinite list of languages is not complete
112
Visualizing One Possible L[ ] *
L[0]
L[1]
L[2]
L[3]
L[4]
l
a
aa aaa aaaa ...
IN
IN
IN
IN
IN
IN OUT IN OUT IN
OUT OUT OUT OUT OUT
OUT IN
IN OUT OUT
•#Rows is countably infinite
•Given
•#Cols is countably infinite
• {a}* is countably infinite
OUT IN OUT OUT IN
...
• Consider each string to be a feature
• A set contains or does not contain each string
113
Constructing D(L[ ]) *
• We construct D(L[]) by using a unique feature
(string) to differentiate D(L[]) from L[i]
• Typically use ith string for language L[i]
• Thus the name diagonalization
D(L[]) l
a
L[0]
L[1]
L[2]
OUT
IN IN
L[3]
L[4]
OUT IN
aa aaa aaaa ...
IN
IN
IN
IN OUT
IN IN OUT IN
OUT OUT OUT
IN OUT OUT
IN OUT
IN OUT
OUT IN OUT OUT OUT
IN
...
114
D(L[]) cannot be any language L[i]
• D(L[]) cannot be language L[0] because D(L[])
differs from L[0] with respect to string a0 = /\.
• If L[0] contains /\, then D(L[]) does not, and vice versa.
• D(L[]) cannot be language L[1] because D(L[])
differs from L[1] with respect to string a1 = a.
• If L[1] contains a, then D(L[]) does not, and vice versa.
• D(L[]) cannot be language L[2] because D(L[])
differs from L[2] with respect to string a2 = aa.
• If L[2] contains aa, then D(L[]) does not, and vice versa.
• …
115
Questions
L[0]
L[1]
L[2]
L[3]
L[4]
l
a
aa aaa aaaa ...
IN
IN
IN
IN
IN
IN OUT IN OUT IN
OUT OUT OUT OUT OUT
OUT IN
IN OUT OUT
OUT IN OUT OUT IN
...
• Do we need to use the diagonal?
• Every other column and every row?
• Every other row and every column?
• What properties are needed to construct D(L[])?
116
Visualization
All problems
Solvable Problems
The set of solvable problems is a proper subset of the set
of all problems.
117
Summary
• Equal size infinite sets: bijections
• Countable and uncountable infinities
• More languages than algorithms
• Number of algorithms countably infinite
• Number of languages uncountably infinite
• Diagonalization technique
• Construct D(L[]) using infinite set of features
• The set of solvable problems is a proper
subset of the set of all problems
118
Module 6
• Topics
• Program behavior problems
• Input of problem is a program/algorithm
• Definition of type program
• Program correctness
• Testing versus Proving
119
Number Theory Problems
• These are problems where we investigate
properties of numbers
• Primality
• Input: Positive integer n
• Yes/No Question: Is n a prime number?
• Divisor
• Input: Integers m,n
• Yes/No question: Is m a divisor of n?
120
Graph Theory Problems
• These are problems where we investigate
properties of graphs
• Connected
• Input: Graph G
• Yes/No Question: Is G a connected graph?
• Subgraph
• Input: Graphs G1 and G2
• Yes/No question: Is G1 a subgraph of G2?
121
Program Behavior Problems
• These are problems where we investigate
properties of programs and how they
behave
• Give an example problem with one input
program P
• Give an example problem with two input
programs P1 and P2
122
Program Representation
• Program variables
• Abstractly, we define the type “program”
• graph G, program P
• More concretely, we define type program to be a
string over the program alphabet SP = {a, …, z,
A, …, Z, 0, …, 9, punctuation, white space}
• Note, many strings over SP are not legal programs
• We consider them to be programs that always crash
• Possible declaration of main procedure
• bool main(program P)
123
Program correctness
• How do we determine whether or not a
program P we have written is correct?
• What are some weaknesses of this approach?
• What might be a better approach?
124
Testing versus Analyzing
Test Inputs
x1
x2
x3
...
Program P
Program
P
Analyzer
Outputs
P(x1)
P(x2)
P(x3)
...
Analysis of
Program P
125
2 Program Behavior Problems *
• Correctness
• Input
• Program P
• Yes/No Question
• Does P correctly solve the primality problem?
• Functional Equivalence
• Input
• Programs P1, P2
• Yes/No Question
• Is program P1 functionally equivalent to program P2
126
Module 7
• Halting Problem
• Fundamental program behavior problem
• A specific unsolvable problem
• Diagonalization technique revisited
• Proof more complex
127
Definition
• Input
• Program P
• Assume the input to program P is a single nonnegative integer
– This assumption is not necessary, but it simplifies the following
unsolvability proof
– To see the full generality of the halting problem, remove this
assumption
• Nonnegative integer x, an input for program P
• Yes/No Question
• Does P halt when run on x?
• Notation
• Use H as shorthand for halting problem when space is a
constraint
128
Example Input *
• Program with one input of type unsigned
bool main(unsigned Q) {
int i=2;
if ((Q = = 0) || (Q= = 1)) return false;
while (i<Q) {
if (Q%i = = 0) return (false);
i++;
}
return (true);
}
• Input x
4
129
Three key definitions
130
Definition of list L *
SP* is countably infinite where SP = {characters, digits, white
space, punctuation}
• Type program will be type string with SP as the alphabet
• Define L to be the strings in SP* listed in enumeration order
•
• length 0 strings first
• length 1 strings next
• …
• Every program is a string in SP
• For simplicity, consider only programs that have
• one input
• the type of this input is an unsigned
• Consider strings in SP* that are not legal programs to be
programs that always crash (and thus halt on all inputs)
131
Definition of PH *
• If H is solvable, some program must solve H
• Let PH be a procedure which solves H
• We declare it as a procedure because we will use PH as
a subroutine
• Declaration of PH
• bool PH(program P, unsigned x)
• In general, the type of x should be the type of the input to P
• Comments
• We do not know how PH works
• However, if H is solvable, we can build programs
which call PH as a subroutine
132
Definition of program D
bool main(unsigned y) /* main for program D */
{ program P = generate(y);
if (PH(P,y)) while (1>0); else return (yes); }
/* generate the yth string in SP* in enumeration order */
program generate(unsigned y)
/* code for program of slide 21 from module 5 did this
for {a,b}* */
bool PH(program P, unsigned x)
/* how PH solves H is unknown */
133
Generating Py from y *
• We won’t go into this in detail here
• This was the basis of the question at the bottom of slide 21 of
lecture 5 (alphabet for that problem was {a,b} instead of SP).
• This is the main place where our assumption about the input type for
program P is important
• for other input types, how to do this would vary
• Specification
•
•
•
•
•
•
•
•
0 maps to program l
1 maps to program a
2 maps to program b
3 maps to program c
…
26 maps to program z
27 maps to program A
…
134
Proof that H is not solvable
135
Argument Overview *
H is solvable
Definition of
Solvability
PH exists
H is NOT solvable
PH does NOT exist
D’s code
D exists
L is list of
all programs
D is on list L
D does NOT exist
D is NOT on list L
p → q is logically equivalent to (not q) → (not p)
136
Proving D is not on list L
• Use list L to specify a program behavior B that is
distinct from all real program behaviors (for
programs with one input of type unsigned)
• Diagonalization argument similar to the one for proving the
number of languages over {a} is uncountably infinite
• No program P exists that exhibits program behavior B
• Argue that D exhibits program behavior B
• Thus D cannot exist and thus is not on list L
137
Non-existent program behavior B
138
Visualizing List L *
P0
P1
P2
P3
P4
0
1
2
3
4
H
H
H
H
H
H NH
H
NH
H
NH NH NH NH NH
NH
H
H
H
H
H
NH NH
H
H
...
•#Rows is countably infinite
• Sp* is countably infinite
•#Cols is countably infinite
• Set of nonnegative integers
is countably infinite
...
• Consider each number to be a feature
• A program halts or doesn’t halt on each integer
• We have a fixed L this time
139
Diagonalization to specify B *
•We specify a non-existent program behavior B by using a unique feature
(number) to differentiate B from Pi
B
0
1
2
3
4
P0
NH
H
H
H
H
H
H NH
H
H
NH
H
P1
P2
P3
P4
...
NH NH NH
H NH NH
NH
H
H
H
H
H
NH
H NH
H
NH
H
...
140
Arguing D exhibits
program behavior B
141
Code for D
bool main(unsigned y) /* main for program D */
{ program P = generate(y);
if (PH(P,y)) while (1>0); else return (yes); }
/* generate the yth string in SP* in enumeration order */
program generate(unsigned y)
/* code for extra credit program of slide 21 from lecture
5 did this for {a,b}* */
bool PH(program P, unsigned x)
/* how PH solves H is unknown */
142
Visualization of D in action on input y
• Program D with input y
• (type for y: unsigned)
0
1
2 ...
P0
P1
P2
H
H
H
H NH
H
NH NH
NH
y ...
...
Py
NH
H
...
Given input y, generate the
program (string) Py
Run PH on Py and y
• Guaranteed to halt since
PH solves H
IF (PH(Py,y)) while (1>0);
else return (yes);
D
143
D cannot be any program on list L
• D cannot be program P0 because D behaves
differently on 0 than P0 does on 0.
• If P0 halts on 0, then D does not, and vice versa.
• D cannot be program P1 because D behaves
differently on 1 than P1 does on 1.
• If P1 halts on 1, then D does not, and vice versa.
• D cannot be program P2 because D behaves
differently on 2 than P2 does on 2.
• If P2 halts on 2, then D does not, and vice versa.
• …
144
Alternate Proof
145
Alternate Proof Overview
• For every program Py, there is a number y
that we associate with it
• The number we use to distinguish program
Py from D is this number y
• Using this idea, we can arrive at a
contradiction without explicitly using the
table L
• The diagonalization is hidden
146
H is not solvable, proof II
• Assume H is solvable
• Let PH be the program which solves H
• Use PH to construct program D which cannot exist
• Contradiction
• This means program PH cannot exist.
• This implies H is not solvable
• D is the same as before
147
Arguing D cannot exist
• If D is a program, it must have an associated number y
• What does D do on this number y?
• 2 cases
• D halts on y
• This means PH(D,y) = NO
– Definition of D
• This means D does not halt on y
– PH solves H
• Contradiction
• This case is not possible
148
Continued
• D does not halt on this number y
• This means PH(D,y) = YES
– Definition of D
• This means D halts on y
– PH solves H
• Contradiction
• This case is not possible
• Both cases are not possible, but one must be for D to exist
• Thus D cannot exist
149
Implications *
• The Halting Problem is one of the simplest problems
we can formulate about program behavior
• We can use the fact that it is unsolvable to show that
other problems about program behavior are also
unsolvable
• This has important implications restricting what we can
do in the field of software engineering
• In particular, “perfect” debuggers/testers do not exist
• We are forced to “test” programs for correctness even though
this approach has many flaws
150
Summary
• Halting Problem definition
• Basic problem about program behavior
• Halting Problem is unsolvable
• We have identified a specific unsolvable
problem
• Diagonalization technique
• Proof more complicated because we actually need to
construct D, not just give a specification B
151
Module 8
• Closure Properties
• Definition
• Language class definition
• set of languages
• Closure properties and first-order logic
statements
• For all, there exists
152
Closure Properties
• A set is closed under an operation if
applying the operation to elements of the set
produces another element of the set
• Example/Counterexample
• set of integers and addition
• set of integers and division
153
Integers and Addition
2
7
5
Integers
154
Integers and Division
.4
2
5
Integers
155
Language Classes
• We will be interested in closure properties of
language classes
• A language class is a set of languages
• Thus, the elements of a language class (set of
languages) are languages which are sets themselves
• Crucial Observation
• When we say that a language class is closed under
some set operation, we apply the set operation to the
languages (elements of the language classes) rather
than the language classes themselves
156
Example Language Classes *
• In all these examples, we do not explicitly
state what the underlying alphabet S is
• Finite languages
• Languages with a finite number of strings
• CARD-3
• Languages with at most 3 strings
157
Finite Sets and Set Union *
{0,1,00}
{0,1,00,11}
{0,1,11}
Finite Sets
158
CARD-3 and Set Union
{0,1,00,11}
{0,1,00}
{0,1,11}
CARD-3
CARD-3: sets with at most 3 elements
159
Finite Sets and Set Complement
{/\,00,10,11,000,...}
{0,1,01}
Finite Sets
160
Infinite Number of Facts
• A closure property often represents an infinite
number of facts
• Example: The set of finite languages is closed
under the set union operation
•
•
•
•
•
•
{} union {} is a finite language
{} union {l} is a finite language
{} union {0} is a finite language
...
{l} union {} is a finite language
...
161
First-order logic and closure
properties *
• A way to formally write (not prove) a
closure property
• " L1, ...,Lk in LC, op (L1, ... Lk) in LC
• Only one expression is needed because of the
for all quantifier
• Number of languages k is determined by
arity of the operation op
162
Example F-O logic statements *
• " L1,L2 in FINITE, L1 union L2 in FINITE
• " L1,L2 in CARD-3, L1 union L2 in CARD3
• " L in FINITE, Lc in FINITE
• " L in CARD-3, Lc in CARD-3
163
Stating a closure property is false
• What is true if a set is not closed under
some k-ary operator?
• There exist k elements of that set which, when
combined together under the given operator,
produce an element not in the set
• $ L1, ...,Lk in LC, op (L1, …, Lk) not in LC
• Example
• Finite sets and set complement
164
Complementing a F-O logic statement
• Complement “" L1,L2 in CARD-3, L1 union
L2 in CARD-3”
• not (" L1,L2 in CARD-3, L1 union L2 in
CARD-3)
• $ L1,L2 in CARD-3, not (L1 union L2 in
CARD-3)
• $ L1,L2 in CARD-3, L1 union L2 not in
CARD-3
165
Proving/Disproving
• Which is easier and why?
• Proving a closure property is true
• Proving a closure property is false
166
Module 9
• Recursive and r.e. language classes
• representing solvable and half-solvable problems
• Proofs of closure properties
• for the set of recursive (solvable) languages
• for the set of r.e. (half-solvable) languages
• Generic element/template proof technique
• Relationship between RE and REC
• pseudoclosure property
167
RE and REC language classes
• REC
• A solvable language is commonly referred to as
a recursive language for historical reasons
• REC is defined to be the set of solvable or
recursive languages
• RE
• A half-solvable language is commonly referred
to as a recursively enumerable or r.e. language
• RE is defined to be the set of r.e. or halfsolvable languages
168
Why study closure properties of RE
and REC?
• It tests how well we really understand the
concepts we encounter
• language classes, REC, solvability, halfsolvability
• It highlights the concept of subroutines and
how we can build on previous algorithms to
construct new algorithms
• we don’t have to build our algorithms from
scratch every time
169
Example Application
• Setting
• I have two programs which can solve the
language recognition problems for L1 and L2
• I want a program which solves the language
recognition problem for L1 intersect L2
• Question
• Do I need to develop a new program from
scratch or can I use the existing programs to
help?
• Does this depend on which languages L1 and L2 I am
working with?
170
Closure Properties of REC *
• We now prove REC is closed under two set
operations
• Set Complement
• Set Intersection
• In these proofs, we try to highlight intuition
and common sense
171
Set Complement Example
• Even: the set of even length strings over {0,1}
• Complement of Even?
• Odd: the set of odd length strings over {0,1}
• Is Odd recursive (solvable)?
• How is the program P’ that solves Odd related to
the program P that solves Even?
172
Set Complement Lemma
• If L is a solvable language, then L
complement is a solvable language
• Proof
• Let L be an arbitrary solvable language
• First line comes from For all L in REC
• Let P be the C++ program which solves L
• P exists by definition of REC
173
proof continued
• Modify P to form P’ as follows
• Identical except at very end
• Complement answer
– Yes → No
– No → Yes
• Program P’ solves L complement
• Halts on all inputs
• Answers correctly
• Thus L complement is solvable
• Definition of solvable
174
P’ Illustration
P’
P
YES
No
No
YES
Input x
175
Code for P’
bool main(string y)
{
if (P (y)) return no; else return yes;
}
bool P (string y)
/* details deleted; key fact is P is guaranteed to halt on all inputs */
176
Set Intersection Example
• Even: the set of even length strings over {0,1}
• Mod-5: the set of strings of length a multiple of 5
over {0,1}
• What is Even intersection Mod-5?
• Mod-10: the set of strings of length a multiple of 10
over {0,1}
• How is the program P3 (Mod-10) related to
programs P1 (Even) and P2 (Mod-5)
177
Set Intersection Lemma
• If L1 and L2 are solvable languages, then L1
intersection L2 is a solvable language
• Proof
• Let L1 and L2 be arbitrary solvable languages
• Let P1 and P2 be programs which solve L1 and
L2, respectively
178
proof continued
• Construct program P3 from P1 and P2 as
follows
• P3 runs both P1 and P2 on the input string
• If both say yes, P3 says yes
• Otherwise, P3 says no
• P3 solves L1 intersection L2
• Halts on all inputs
• Answers correctly
• L1 intersection L2 is a solvable language
179
P3 Illustration
P1
Yes/No
P3
P2
AND
Yes/No
Yes/No
180
Code for P3
bool main(string y)
{
if (P1(y) && P2(y)) return yes;
else return no;
}
bool P1(string y) /* details deleted; key fact is P1 always halts. */
bool P2(string y) /* details deleted; key fact is P2 always halts. */
181
Other Closure Properties
• Unary Operations
• Language Reversal
• Kleene Star
• Binary Operations
•
•
•
•
Set Union
Set Difference
Symmetric Difference
Concatenation
182
Closure Properties of RE *
• We now try to prove RE is closed under the
same two set operations
• Set Intersection
• Set Complement
• In these proofs
• We define a more formal proof methodology
• We gain more intuition about the differences
between solvable and half-solvable problems
183
RE Closed Under Set Intersection
• Expressing this closure property as an
infinite set of facts
• Let Li denote the ith r.e. language
•
•
•
•
•
L1 intersect L1 is in RE
L1 intersect L2 is in RE
...
L2 intersect L1 is in RE
...
184
Generic Element or Template Proofs
• Since there are an infinite number of facts
to prove, we cannot prove them all
individually
• Instead, we create a single proof that proves
each fact simultaneously
• I like to call these proofs generic element or
template proofs
185
Basic Proof Ideas
• Name your generic objects
• In this case, we use L1 and L2
• Only use facts which apply to any relevant
objects
• We will only use the fact that there must exist
P1 and P2 which half-solve L1 and L2
• Work from both ends of the proof
• The first and last lines are usually obvious, and
we can often work our way in
186
Set Intersection Example *
• Let L1 and L2 be arbitrary r.e. languages
• There exist P1 and P2 s.t. Y(P1)=L1 and Y(P2)=L2
• By definition of half-solvable languages
• Construct program P3 from P1 and P2
• Note, we can assume very little about P1 and P2
• Prove Program P3 half-solves L1 intersection L2
• There exists a program P which half-solves L1
intersection L2
• L1 intersection L2 is an r.e. language
187
Constructing P3 *
• Run P1 and P2 in parallel
• One instruction of P1, then one instruction of
P2, and so on
• If both halt and say yes, halt and say yes
• If both halt but both do not say yes, halt and
say no
188
P3 Illustration
P1
Input
Yes/No/P3
P2
AND
Yes/No/-
Yes/No/-
189
Code for P3
bool main(string y){
parallel-execute(P1(y), P2(y)) until both return;
if ((P1(y) && P2(y)) return yes;
else return no;
}
bool P1(string y)
/* key fact is P1 only guaranteed to halt on yes input instances */
bool P2(string y)
/* key fact is P2 only guaranteed to halt on yes input instances */
190
Proving P3 Is Correct
• 2 steps to showing P3 half-solves L1
intersection L2
• For all x in L1 intersection L2, must show P3
• accepts x
– halts and says yes
• For all x not in L1 intersection L2, must show P3
does what?
191
Part 1 of Correctness Proof
• P3 accepts x in L1 intersection L2
• Let x be an arbitrary string in L1 intersection L2
• Note, this subproof is a generic element proof
• P1 accepts x
• L1 intersection L2 is a subset of L1
• P1 accepts all strings in L1
• P2 accepts x
• P3 accepts x
• We reach the AND gate because of the 2 previous facts
• Since both P1 and P2 accept, AND evaluates to YES
192
Part 2 of Correctness Proof
• P3 does not accept x not in L1 intersection L2
• Let x be an arbitrary string not in L1 intersection L2
• By definition of intersection, this means x is not in L1 or L2
• Case 1: x is not in L1
• 2 possibilities
• P1 rejects (or crashes on) x
– One input to AND gate is No
– Output cannot be yes
– P3 does not accept x
• P1 loops on x
– One input never reaches AND gate
– No output
– P3 loops on x
• P3 does not accept x when x is not in L1
• Case 2: x is not in L2
• Essentially identical analysis
• P3 does not accept x not in L1 intersection L2
193
RE closed under set complement?
• First-order logic formulation?
• What this really means
• Let Li denote the ith r.e. language
• L1 complement is in RE
• L2 complement is in RE
• ...
194
Set complement proof overview
• Let L be an arbitrary r.e. language
• There exists P s.t. Y(P)=L
• By definition of r.e. languages
• Construct program P’ from P
• Note, we can assume very little about P
• Prove Program P’ half-solves L complement
• There exists a program P’ which half-solves L
complement
• L complement is an r.e. language
195
Constructing P’
• What did we do in recursive case?
• Run P and then just complement answer at end
• Accept → Reject
• Reject → Accept
• Does this work in this case?
• No. Why not?
• Does this prove that RE is not closed under
set complement?
196
Other closure properties
• Unary Operations
• Language reversal
• Kleene Closure
• Binary operations
• union (on practice hw)
• concatenation
• Not closed
• Set difference (on practice hw)
197
Closure Property Applications
• How can we use closure properties to prove
a language LT is r.e. or recursive?
• Unary operator op (e.g. complement)
• 1) Find a known r.e. or recursive language L
• 2) Show LT = L op
• Binary operator op (e.g. intersection)
• 1) Find 2 known r.e or recursive languages L1 and L2
• 2) Show LT = L1 op L2
198
Closure Property Applications *
• How can we use closure properties to prove
a language LT is not r.e. or recursive?
• Unary operator op (e.g. complement)
• 1) Find a known not r.e. or non-recursive language L
• 2) Show LT op = L
• Binary operator op (e.g. intersection)
• 1) Find a known r.e. or recursive language L1
• 2) Find a known not r.e. or non-recursive language L2
• 3) Show L2 = L1 op LT
199
Example
• Looping Problem
• Input
• Program P
• Input x for program P
• Yes/No Question
• Does P loop on x?
• Looping Problem is unsolvable
• Looping Problem complement = H
200
Closure Property Applications
• Proving a new closure property
• Theorem: Unsolvable languages are closed under set
complement
• Let L be an arbitrary unsolvable language
• If Lc is solvable, then L is solvable
• (Lc)c = L
• Solvable languages closed under complement
• However, we are assuming that L is unsolvable
• Therefore, we can conclude that Lc is unsolvable
• Thus, unsolvable languages are closed under complement
201
Pseudo Closure Property
• Lemma: If L and Lc are half-solvable, then
L is solvable.
• Question: What about Lc?
202
High Level Proof
• Let L be an arbitrary language where L and Lc
are both half-solvable
• Let P1 and P2 be the programs which half-solve
L and Lc, respectively
• Construct program P3 from P1 and P2
• Argue P3 solves L
• L is solvable
203
Constructing P3
• Problem
• Both P1 and P2 may loop on some input strings,
and we need P3 to halt on all input strings
• Key Observation
• On all input strings, one of P1 and P2 is
guaranteed to halt. Why?
204
Illustration
S*
L
Lc
P1 halts
P2 halts
205
Construction and Proof
• P3’s Operation
• Run P1 and P2 in parallel on the input string x
until one accepts x
• Guaranteed to occur given previous argument
• Also, only one program will accept any string x
• IF P1 is the accepting machine THEN yes ELSE
no
206
P3 Illustration
P1
Input
Yes
Yes
P3
P2
Yes
No
207
Code for P3
*
bool main(string y)
{
parallel-execute(P1(y), P2(y)) until one returns yes;
if (P1(y)) return yes;
if (P2(Y)) return no;
}
bool P1(string y) /* guaranteed to halt on strings in L*/
bool P2(string y) /* guaranteed to halt on strings in Lc */
208
RE and REC
All Languages
RE
L
REC
Lc
209
RE and REC
All Languages
Lc
L
Lc
RE
Lc REC
Are there any languages L in RE - REC?
210
Module 10
• Universal Algorithms
• moving beyond one problem at a time
• operating system/general purpose computer
211
Observation
• So far, each program solves one specific
problem
•
•
•
•
Divisor
Sorting
Multiplication
Language L
212
Universal Problem/Program
• Universal Problem (nonstandard term)
• Input
• Program P
• Input x to program P
• Task
• Compute P(x)
• Univeral Program
• Program which solves universal problem
• Universal Turing machine
213
Example Input *
int main(A[6]) {
int i,temp;
for (i=1;i<=3;i++)
if (A[i] > A[i+3]) {
temp = A[i+3];
A[i+3] = A[i];
A[i] = temp;
}
Input
A[1] = 6
A[2] = 4
A[3] = 2
A[4] = 3
A[5] = 5
A[6] = 1
for (i=1; i<=5; i++)
for (j=i+1;j<=6;j++)
if (A[j-1] > A[j]) {
temp = A[j];
A[j] = A[j-1];
A[j-1] = temp;
}
}
214
Organization
Universal Program’s Memory
Program P
Program P’s Memory
Program Counter
Program P
int main(A[6]){
int i,temp;
for (i=1;i<=3;i++)
if (A[i] > A[i+3]) {
temp = A[i+3];
6
A[i+3] = A[i];
4
A[i] = temp;
2
int A[6],i,temp;
}
5
3 for (i=1; i<=5; i++)
1
for (j=i+1;j<=6;j++)
Line 1
}
if (A[j-1] > A[j]) {
temp = A[j];
A[j] = A[j-1];
A[j-1] = temp;
}
215
Description of Universal Program
• Basic Loop
• Find current line of program P
• Execute current line of program P
• Update program P’s memory
• Update program counter
• Return to Top of Loop
216
Past, Present, Future
• Turing came up with the concept of a universal program
(Universal Turing machine) in the 1930’s
• This is well before the invention of the general purpose computer
• People were still thinking of computing devices as special-purpose
devices (calculators, etc.)
• Turing helped move people beyond this narrow perspective
• Turing/Von Neumann perspective
• Computers are general purpose/universal algorithms
• Focused on computation
• Stand-alone
• Today, we are moving beyond this view
• Computation, communication, cyberspace
• However, results in Turing perspective still relevant
217
Halting Problem Revisited *
• Halting Problem is half-solvable
• Modified Universal Program (MUP) halfsolves H
Run P on x
Output yes
– This step only executed if first step halts
• Behavior
• What does MUP do on all yes instances of H?
• What does MUP do on all no inputs of H?
218
Debuggers
• How are debugger’s like gdb or ddd related
to universal programs?
• How do debuggers simplify the debugging
process?
219
RE and REC
• We now have a problem that is halfsolvable but not solvable
• What do we now know about the
complement of the Halting Problem?
• What additional fact about RE and set
complement does this prove?
220
RE and REC
All Languages
RE
Hc
H
REC
221
Summary
• Universal Programs
• 1930’s, Turing
• Introduces general purpose computing concept
• Not a super intelligent program, merely a
precise follower of instructions
• Halting Problem half-solvable but not
solvable
• RE not closed under set complement
222
Module 11
• Proving more specific problems are not
solvable
• Input transformation technique
• Use subroutine theme to show that if one
problem is unsolvable, so is a second problem
• Need to clearly differentiate between
• use of program as a subroutine and
• a program being an input to another program
223
Basic Idea/Technique
224
Proving a problem L is unsolvable
• Assume PL is a procedure that solves problem L
• We have no idea how PL solves L
• Construct a program PH that solves H using PL
as a subroutine
• We use PL as a black box
• (We could use any unsolvable problem in place of H)
• Argue PH solves H
• Conclude that L is unsolvable
• Otherwise PL would exist and then H would be solvable
• L will be a problem about program behavior
225
Focusing on H
• In this module, we will typically use H, the
Halting Problem, as our known unsolvable
problem
• The technique generalizes to using any unsolvable
problem L’ in place of H.
• You would need to change the proofs to work with L’
instead of H, but in general it can be done
• The technique also can be applied to solvable
problems to derive alternative consequences
• We focus on H to simplify the explanation
226
Constructing PH using PL
Answer-preserving input
transformations and Program PT
227
PH has two subroutines
• There are many ways to construct PH using
program PL that solves L
• We focus on one method in which PH
consists of two subroutines
• Procedure PL that solves L
• Procedure PT which computes a function f that I
call an answer-preserving (or answer-reversing)
input transformation
• More about this in a moment
228
Pictoral Representation of PH *
x
PT
PT(x)
PL
Y/N
Yes/No
PH
229
Answer-preserving input
transformation PT
• Input
• An input to H
• Output
• An input to L such that
• yes inputs of H map to yes inputs of L
• no inputs of H map to no inputs of L
• Note, PT must not loop when given any
legal input to H
230
Why this works *
yes input to H
no input to H
PT
yes input to L
no input to L
PL
yes
no
PH
We have assumed that PL solves L
231
Answer-reversing input
transformation PT
• Input
• An input to H
• Output
• An input to L such that
• yes inputs of H map to no inputs of L
• no inputs of H map to yes inputs of L
• Note, PT must not loop when given any
legal input to H
232
Why this works
yes input to H
no input to H
PT
no input to L
yes input to L
PL
no
yes
yes
no
PH
We have assumed that PL solves L
233
Yes->Yes and No>No
x
PT PT(x) PL
PH
Yes/No
No inputs
for H
Yes inputs
for H
Domain of H
Yes inputs
for L
No inputs
for L
Domain of L
234
Notation and
Terminology
Yes inputs
No inputs
Domain of H
Yes inputs
No inputs
Domain of
L
answer-preserving
(or
• If there is such an
answer-reversing) input transformation f
(and the corresponding program PT), we say
that H transforms to (many-one reduces to)
L
• Notation
H≤L
235
Examples not involving the Halting
Problem
236
Generalization
• As noted earlier, while we focus on
transforming H to other problems, the
concept of transformation generalizes
beyond H and beyond unsolvable program
behavior problems
• We work with some solvable, language
recognition problems to illustrate some
aspects of the transformation process in the
next few slides
237
Yes inputs
Example 1
No inputs
Domain of L1
Yes inputs
No inputs
Domain of L2
• L1 is the set of even length strings over {0,1}
• What are the set of legal input instances and no inputs for
the L1 LRP?
• L2 is the set of odd length strings over {0,1}
• Same question as above
• Tasks
• Give an answer-preserving input transformation f that
shows that L1 LRP ≤ L2 LRP
• Give a corresponding program PT that computes f
238
Program PT
string main(string x)
{
return(x concatenate “0”);
}
239
Yes inputs
Example 2
No inputs
Domain of L1
Yes inputs
No inputs
Domain of L2
• L1 is the set of all strings over {0,1}
• What is the set of all inputs, yes inputs, no inputs for the L1
LRP?
• L2 is {0}
• Same question as above
• Tasks
• Give an answer-preserving input transformation f which
shows that the L1 LRP ≤ L2 LRP
• Give a corresponding program PT which computes f
240
Program PT
string main(string x)
{
return( “0”);
}
241
Yes inputs
Example 3
No inputs
Domain of L1
Yes inputs
No inputs
Domain of L2
• L1
• Input: Java program P that takes as input an unsigned int
• Yes/No Question: Does P halt on all legal inputs
• L2
• Input: C++ program P that takes as input an unsigned int
• Yes/No Question: Does P halt on all legal inputs
• Tasks
• Describe what an answer-preserving input transformation f
that shows that L1 ≤ L2 would be/do?
242
Proving a program behavior problem
L is unsolvable
243
Problem Definitions *
• Target Problem L
• Halting Problem H
• Input
• Program QH that has one
input of type unsigned int
• non-negative integer y
that is input to program
QH
• Yes/No Question
• Does QH halt on y?
• Input
• Program QL that has
one input of type string
• Yes/No question
• Does Y(QL) = the set of
even length strings?
• Assume program PL
solves L
244
Construction review
x
PT
PT(x)
PL
Y/N
Yes/No
PH
•We are building a program PH to solve the halting
problem H
•PH will use PL as a subroutine, and we have no idea
how PL accomplishes its task
•PH will use PT as a subroutine, and we must explicitly
construct PT using specific properties of H and L
245
P’s and Q’s
• Programs which are PART of program PH
and thus “executed” when PH executes
• Program PT, an actual program we construct
• Program PL, an assumed program which solves
problem L
• Programs which are INPUTS/OUTPUTS of
programs PH, PL, and PT and which are not
“executed” when PH executes
• Programs QH, QL, and QYL
• code for QYL is available to PT
246
Two inputs for L *
• Target Problem L
• Input
• Program Q that has one
input of type string
• Yes/No question
• Does Y(Q) = the set of
even length strings?
• Program PL
• Solves L
• We don’t know how
• Consider the following program Q1
bool main(string z)
{while (1>0) ;}
• What does PL output when given Q1 as
input?
• Consider the following program Q2
bool main(string z)
{ if ((z.length %2) = = 0) return (yes)
else return (no); }
• What does PL output when given Q2 as
input?
247
Another input for L *
• Target Problem L
• Input
• Program Q that has one
input of type string
• Yes/No question
• Does Y(Q) = the set of
even length strings?
• Program PL
• Solves L
• We don’t know how
• Consider the following program QL
with 2 procedures Q1 and QYL
bool main(string z) {
Q1(5); /* ignore return value */
return(QYL(z));
}
bool Q1(unsigned x) {
if (x > 3) return (no); else loop;
}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return (yes);
else return(no);
}
• What does PL output when given QL
as input?
248
Input and Output of PT *
• Program QL that is the output of PT
• (Also input of L)
• Input of PT
• (Also Input of H)
• Program QH
• one input of type
unsigned int
• Non-negative integer y
QH,y
PT
QL
bool main(string z) {
QH(y); /* QH and y come left-hand side */
/* ignore return value */
return(QYL(z));
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return (yes);
else return(no);
}
249
Example 1 *
QH,y
PT
QL
Input to PT
Output of PT
Program QH
bool main(unsigned y) {
if (y ==5) return yes;
else if (y ==4) return no;
else while (1>0) {};
}
Program QL
bool QH(unsigned y) {
if (y ==5) return yes;
else if (y ==4) return no;
else while (1>0) {};
}
bool QYL(string z) {
if ((z.length % 2) == 0) return (yes)
else return (no);
}
bool main(string z) {
unsigned y = 5;
QH(y);
return (QYL(z));
}
Input y
5
250
Example 2
QH,y
PT
QL
Input to PT
Output of PT
Program QH
bool main(unsigned y) {
if (y ==5) return yes;
else if (y ==4) return no;
else while (1>0) {};
}
Program QL
bool QH(unsigned y) {
if (y ==5) return yes;
else if (y ==4) return no;
else while (1>0) {};
}
bool QYL(string z) {
if ((z.length % 2) == 0) return (yes)
else return (no);
}
bool main(string z) {
unsigned y = 3;
QH(y);
return (QYL(z));
}
Input y
3
251
PT in more detail
252
QH,y
Declaration of PT
PT
QL
PL
Yes/No
PH
• What is the return type of PT?
• Type program1 (with one input of type string)
• What are the input parameters of PT?
• The same as the input parameters to H; in this case,
• type program2 (with one input of type unsigned int)
• unsigned int (input type to program2)
program1 main(program2 QH, unsigned y)
253
Code for PT
QH,y
PT
QL
PL
Yes/No
PH
program1 main(program2 P, unsigned y) {
/* Will be viewing types program1 and program2 as STRINGS over the
program alphabet SP */
program1 QL = replace-main-with-QH(P);
/* Insert line break */
QL += “\n”;
/* Insert QYL */
QL += “bool QYL(string z) {\n \t if ((z.length % 2) == 0) return (yes) else
return (no);\n }”;
/* Add main routine of QL */
QL += “bool main(string z) {\n\t”; /* determined by L */
QL += “unsigned y =”
QL += convert-to-string(y);
QL += “;\n\t QH(y)\n\t return(QYL(z));\n}”;
return(QL);
}
program1 replace-main-with-QH(program2 P) /* Details hidden */
string convert-to-string(unsigned y) /* Details hidden */
254
QH,y
PT in action
PT
QL
PL
Yes/No
PH
Program QH
Program QL
bool main(unsigned y) {
if (y ==5) return yes;
else if (y ==4) return no;
else while (1>0) {};
}
bool QH(unsigned y) {
if (y ==5) return yes;
else if (y ==4) return no;
else while (1>0) {};
}
bool QYL(string z) {
if ((z.length % 2) == 0) return (yes) else return (no);
}
bool main(string z) {
unsigned y = 5;
QH(y);
return (QYL(z));
}
PT
Input y
code
for QYL
5
QH
PT
QL
z
y
unsigned y
QYL
QH
halt start
QYL
Y/N Y/N
255
Constructing QL (and thus PT)
How to choose QYL or QNL
256
Start with no input for H
• Program QL
• If QH, y is a no input
bool main(string z) {
to the Halting problem
QH(y); /* ignore return value */
return(Q?L(z)); /* yes or no? */
– QH loops on y
– Thus Y(QL) = {}
– Determine if this
makes QL a no or yes
input instance to L
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool Q?L(string y) {
}
257
Answer-preserving input
transformation
• Program QL
• If QH, y is a no input
bool main(string z) {
to the Halting problem
QH(y); /* ignore return value */
return(QYL(z)); /* yes */
– QH loops on y
– Thus Y(QL) = {}
– Determine if this
makes QL a no or yes
input instance to L
– Now choose a QYL (or
QNL) that is a yes (or
no) input instance to L
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool QYL(string y) {
}
258
Make yes for H map to yes for L
• Program QL
• If QH, y is a no input
bool main(string z) {
to the Halting problem
QH(y); /* ignore return value */
return(QYL(z)); /* yes */
– QH loops on y
– Thus Y(QL) = {}
– Determine if this
makes QL a no or yes
input instance to L
– Now choose a QYL (or
QNL) that is a yes (or
no) input instance to L
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return (yes);
else return (no);
}
259
Possible shortcut
Program QL
bool main(string z) {
QH(y); /* ignore return value */
return(QYL(z)); /* yes */
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return
(yes);
else return (no);
Program QL
bool main(string z) {
QH(y); /* ignore return value */
if ((z.length( ) % 2) = = 0) return (yes);
else return (no);
}
bool QH(unsigned x) {
/* comes from left-hand side
}
}
260
Another Example
261
Problem Definitions
• Target Problem L
• Halting Problem H
• Input
• Program QH that has one
input of type unsigned int
• non-negative integer y
that is input to program
QH
• Yes/No Question
• Input
• Program QL that has
one input of type string
• Yes/No question
• Is Y(QL) finite?
• Assume program PL
solves L
• Does QH halt on y?
262
Start with no input for H
• Program QL
• If QH, y is a no input
bool main(string z) {
to the Halting problem
QH(y); /* ignore return value */
return(Q?L(z)); /* yes or no? */
– QH loops on y
– Thus Y(QL) = {}
– Determine if this
makes QL a no or yes
input instance to L
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool Q?L(string y) {
}
263
Answer-reversing input transformation
• Program QL
• If QH, y is a no input
bool main(string z) {
to the Halting problem
QH(y); /* ignore return value */
return(QNL(z)); /* no */
– QH loops on y
– Thus Y(QL) = {}
– Determine if this
makes QL a no or yes
input instance to L
– Now choose a QYL (or
QNL) that is a yes (or
no) input instance to L
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool QNL(string y) {
}
264
Make yes for H map to no for L
• Program QL
• If QH, y is a no input
bool main(string z) {
to the Halting problem
QH(y); /* ignore return value */
return(QNL(z)); /* no */
– QH loops on y
– Thus Y(QL) = {}
– Determine if this
makes QL a no or yes
input instance to L
– Now choose a QYL (or
QNL) that is a yes (or
no) input instance to L
}
bool QH(unsigned x) {
/* comes from left-hand side
}
bool QNL(string y) {
if ((y.length( ) % 2) = = 0) return(yes);
else return(no);
}
265
Analyzing proposed transformations
4 possibilities
266
Problem Setup
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L
•
bool main(string z) {
QH(y); /* ignore return value */
return(QYL(z)); /* yes or no */
}
bool QH(unsigned x) {}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return (yes);
else return (no);
• Input: Program P
• Yes/No Question: Is Y(P) =
{aa}?
Question: Is the
transformation on the
left an answerpreserving or answerreversing input
transformation from H
to problem L?
}
267
Key Step
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L
•
bool main(string z) {
QH(y); /* ignore return value */
return(QYL(z)); /* yes or no */
}
bool QH(unsigned x) {}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return (yes);
else return (no);
• Input: Program P
• Yes/No Question: Is Y(P) =
{aa}?
The output of the
transformation is the
input to the problem.
• Plug QL in for
program P above
• Is Y(QL) = {aa}?
}
268
Is Y(QL) = {aa}?
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L
• Input: Program P
• Yes/No Question: Is Y(P) =
{aa}?
• Analysis
• If QH loops on x, Y(QL)={}
bool main(string z) {
• No input to H creates a QL
QH(y); /* ignore return value */
that is a no input for L
return(QYL(z)); /* yes or no */
• If QH halts on x, Y(QL) =
}
{even length strings}
bool QH(unsigned x) {}
• Yes input to H creates a QL
bool QYL(string y) {
that is a no input for L
if ((y.length( ) % 2) = = 0) return (yes);
• Transformation does not work
else return (no);
• All inputs map to no inputs
}
269
Three other problems
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L1
• Input: Program P
• Yes/No Question: Is Y(P)
infinite?
• Problem L2
bool main(string z) {
QH(y); /* ignore return value */
return(QYL(z)); /* yes or no */
}
•
bool QH(unsigned x) {}
bool QYL(string y) {
if ((y.length( ) % 2) = = 0) return (yes);
else return (no);
• Input: Program P
• Yes/No Question: Is Y(P)
finite?
Problem L3
• Input: Program P
• Yes/No Question: Is Y(P) =
{} or is Y(P) infinite?
}
270
Is Y(QL) infinite?
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L1
• Input: Program P
• Yes/No Question: Is Y(P)
infinite?
• Analysis
• If QH loops on x, Y(QL)={}
bool main(string z) {
• No input to H creates a QL
QH(y); /* ignore return value */
that is a no input for L
return(QYL(z)); /* yes or no */
• If QH halts on x, Y(QL) =
}
{even length strings}
bool QH(unsigned x) {}
• Yes input to H creates a QL
bool QYL(string y) {
that is a yes input for L
if ((y.length( ) % 2) = = 0) return (yes); • Transformation works
else return (no);
• Answer-preserving
}
271
Is Y(QL) finite?
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L2
• Input: Program P
• Yes/No Question: Is Y(P)
finite?
• Analysis
• If QH loops on x, Y(QL)={}
bool main(string z) {
• No input to H creates a QL
QH(y); /* ignore return value */
that is a yes input for L
return(QYL(z)); /* yes or no */
• If QH halts on x, Y(QL) =
}
{even length strings}
bool QH(unsigned x) {}
• Yes input to H creates a QL
bool QYL(string y) {
that is a no input for L
if ((y.length( ) % 2) = = 0) return (yes); • Transformation works
else return (no);
• Answer-reversing
}
272
Is Y(QL) = {} or is Y(QL) infinite?
• Input of Transformation
• Program QH, unsigned x
• Output of Transformation
• Program QL
• Problem L3
• Input: Program P
• Yes/No Question: Is Y(P) =
{} or is Y(P) infinite?
• Analysis
• If QH loops on x, Y(QL)={}
bool main(string z) {
• No input to H creates a QL
QH(y); /* ignore return value */
that is a yes input for L
return(QYL(z)); /* yes or no */
• If QH halts on x, Y(QL) =
}
{even length strings}
bool QH(unsigned x) {}
• Yes input to H creates a QL
bool QYL(string y) {
that is a yes input for L
if ((y.length( ) % 2) = = 0) return (yes); • Transformation does not work
else return (no);
• All inputs map to yes inputs
}
273