What is an Algorithm?
Algorithms and
Algorithmics
Part I
• Let’s focus on gastronomy!
• Given a kitchen containing
• a baker, supply of ingredients, baking utensils, an oven, etc.
• Baking a cake is
• a process that is carried out from the ingredients, by the
baker, with the aid of the oven, and most significantly,
according to the recipe!
Algorithms
Computability Theory
Computational Complexity Theory
Cryptography
• Ingredients: input to the process
• Cake: output of the process
• Recipe: the algorithm (an abstract entity)
• the formal written version: program (Software)
• Utensils, oven, baker: (Hardware)
Baking a Cake
Input
Ingredients
8 ounces of semisweet chocolate pieces, 2 tablespoons of water, 1/4 cup of powdered sugar, 6 separated eggs, etc.
Software
Hardware
Recipe
Oven
Utensils
Baker
Output
A real recipe: Mousseline au chocolat
Cake
Effective
Process
Computational
Thinking!
Melt chocolate and 2 tablespoons water in double boiler. When melted
stir in powdered sugar; add butter bit by bit. Set aside. Beat egg yolks until
thick and lemon-colored, about 5 minutes. Gently fold in chocolate. Reheat
slightly to melt chocolate, if necessary. Stir in Rum and vanilla. Beat egg whites
until foamy. Beat in 2 tablespoons sugar; beat until stiff peaks form.Gently
fold whites into chocolate-yolk mixture. Pour into individual serving dishes.
Chill at least 4 hours. Serve with whipped cream, if desired.
6-8 servings of Mousseline au chocolat
Level of Detail / Basic Actions
“Stir in powdered sugar”
Problem:
Basic action because it is assumed the hardware knows how to do it.
Level of detail is very important when it comes to an algorithm’s elementary instructions.
• Should be tailored to fit the hardware’s capabilities
• Should also be appropriate for the comprehension level of the potential reader or user of the algorithm.
Ambiguities or Fuzzy Phrases not allowed
in specifying instructions to a computer!
“served with whipped cream if desired”
“about 5 minutes”
Is it the serving or addition of WC
that is dependent on person’s desires?
The Algorithmic Problem and its Solution
Characterization
of all legal inputs
and
Characterization of
desired outputs as a
function of inputs
Short Algorithms for Long Processes
Any legal input
Given a list of personnel records, one for each employee in a certain company,
each containing the employee’s name, personal details, and salary, find the total
sum of all salaries of all employees
Algorithm:
1. Make a note of the number 0;
2. proceed through the list, adding each employee’s salary to the noted number;
3. having reached the end of the list, produce the noted number as output.
The text of the algorithm is short and fixed, yet the process it invokes
varies with the length of the input list and can be very, very long!
We have:
A fixed algorithm prescribing many processes of varying lengths, the precise duration and nature of the process being dependent
on the inputs to the algorithm. In fact, the potential choice of inputs may be infinite!
The problem and its solution
• An algorithmic problem consists of:
• a characterization of a legal, possibly infinite, collection of potential
input sets, and
• a specification of the desired outputs as a function of the inputs
Algorithm
A
• Assumption
• a description of allowed basic actions or a hardware configuration,
together with its built-in basic actions is provided in advance.
• Each of the actions must be carried out in a finite amount of time.
Desired Output
• An algorithmic solution consists of:
• an algorithm, composed of elementary instructions prescribing
actions from the agreed upon set
Algorithmic
Problem
Algorithmic
Solution
• the algorithm, when executed for any legal input set, solves the
problem, producing the output as required.
Solving Algorithmic Problems
•
Algorithmic problems can be incredibly complex
and take years of work to solve satisfactorily
Course Book Definition
An algorithm is an ordered set of unambiguous finitely
executable steps that defines a terminating process
• Many problems do not admit satisfactory
solutions
• Many problems do not admit solutions at all!
• For many problems, the status as far as good
algorithmic solutions is still unknown.
Control Flow or Ordering in Program Execution
• Sequencing: Statements are to be executed (or expressions evaluated) in a
certain specified order.
•
Selection: Depending on some runtime condition, a choice is to be made
among two or more statements or expressions
• The most common selection constructs are if and case (switch)
• Iteration: A given fragment of code is executed repeatedly, either a certain
•
An algorithm must contain control instructions that
determine the sequence or order in which the executable
steps or instructions are carried out
Control Flow or Ordering in Program Execution
• Concurrency: Two or more program fragments are to be executed/evaluated
“at the same time”
• either in parallel on separate processors,
• or interleaved on a single processor in a way that achieves the same effect.
• Exception handling and speculation: A program fragment is executed
number of times, or until a certain run-time condition is true.
optimistically, on the assumption that the expected condition will be true. If
the condition turns out to be false
• Iteration constructs include for/do, while, and repeat loops
• execution branches to a handler that executes in place of the remainder of
Procedural Abstraction: A potentially complex collection of control
constructs ( a subroutine) is encapsulated in a way that allows it to be
treated as a single unit, usually subject to parameterization
• Recursion: An expression is defined in terms of (simpler versions of )
itself, either directly or indirectly
• the computational model requires a stack on which to save information
about partially evaluated instances of the expression.
the protected fragment ( in the case of exception handling), or
• in place of the entire protected fragment (in the case of speculation)
• In this case, the language implementation must be able to rollback or
undo any visible effects of the protected code.
• Nondeterminacy: The ordering or choice among expressions or statements is
deliberately left unspecified, implying that any alternative will lead to correct
results.
Control Flow or Ordering in Program Execution
• These 8 principle categories cover all the
control-flow constructs and mechanisms found in
most programming languages
• Though the syntax and semantics details vary
from language to language, thinking in these
terms will make it easier to
• learn new languages
• evaluate the tradeoffs among languages
• and design and reason about algorithms in a
language-independent way!
Computational Problems
• An “algorithm” is an informal intuitive concept, but associated with it is
the concept of a computational process.
• Many attempts have been made to provide formal definitions of what
the most general mathematical notion of a computational process is.
• In other words, a formal equivalent of the informal notion of an
algorithm.
• For example:
• A Turing machine that halts on all inputs is the precise formal notion
corresponding to the intuitive notion of an algorithm.
• The area that studies such issues is called Computability Theory
• It asks the fundamental question:
• What can be computed by a computational device?
Computational Problems
Computational Problems that
CANNOT be solved by any algorithm
Computability Theory
Computational Problems that CAN
be solved by an algorithm
Computability Theory
Computational Problems that CAN
be solved by an algorithm
CANNOT be solved in any practical
sense due to excessive time/space
requirements
CAN be solved in a practical
sense with reasonable time/space
requirements
Intractable Class NP
Computational
Complexity Theory
Tractable
Class P
Leibniz: Step Reckoner
Calculus Ratiocinator
•A universal artificial mathematical language
•All human knowledge could be represented Some History and
Context
Leibniz (1646 - 1716)
Let us
Calculate!
in this language
•Calculational rules would reveal all logical
relationships among these propositions
•Machines would be capable of carrying out such calculations
In 1673, Leibniz built the first true four-function calculator. His
unique, drum-shaped gears formed the basis of many successful
calculator designs for the next 275 years, an unbroken record for
a single underlying calculator mechanism.
Addition
Subtraction
Multiplication
Square root extraction
Leibniz Step Reckoner
Hilbert (1862-1943)
Early use of binary system
(not in step reckoner)
Boole (1815-1864)
Turned “Logic” into Algebra
1st Problem: Decide the truth of Cantor’s
Continum Hypothesis
2nd Problem: Establish the consistency of the
axioms for the arithmetic of real numbers
24 problems
for the
20th century
23rd Problem: Does there exist an algorithm that can
determine the truth or falsity of any logical proposition in
a system of logic that is powerful enough to represent
the natural numbers? (Entscheidungsproblem)
Classes and terms (thoughts) could be manipulated
using algebraic rules resulting in valid inferences
Logical deduction could
be developed as a branch
of mathematics
Subsumed Aristotle’s syllogisms
In essence Leibniz’
calculus rationator (lite)
Boolean Logic
Frege (1848-1925)
Gödel (1906-1978)
Begriffsschrift “Concept Script”
Showed the completeness
of 1st-order logic in his PhD Thesis
The 1st fully developed system of logic
encompassing all of the deductive
reasoning in ordinary mathematics.
•1st example of formal artificial language with formal syntax
•logical inference as purely mechanical
operations (rules of inference)
Intention was to show that all of mathematics
could be based on logic! (Logicism)
Turing (1912-1954)
Turing wanted to disprove the 23rd problem
23rd Problem: Does there exist an algorithm that can determine the truth or falsity of
any logical proposition in a system of logic that is powerful enough to represent the
natural numbers? (Entscheidungsproblem)
To do this, he had to come up with a formal
characterization of the generic process underlying
the computation of an algorithm
He then showed that there were functions that were
not effectively computable including the
Entscheidungsproblem!
As a byproduct he found a mathematical model of an
all-purpose computing machine!
Develop metamathematics inside
a formal logical system by encoding
propositions as numbers
The logic of PM
(and consequently PA)
is incomplete
There are true
sentences not provable
within the logical
system
Hilbert’s 2nd Problem
As a consequence, the
consistency of the
mathematics of the real
numbers can not be
proven within any
system as strong as PA
“Effective” Computation
What is it, that humans do, when humans “compute”?
What actions are undertaken by a human agent when
manipulating a finite set of symbols according to fixed rules?
Strip this undertaking of any particular notion of mental activity,
thought, imagination, creativity....
What are the essentials of human computation itself?
Observe!:This is 1930.There are no computers in existence!
Turing Machine
• Blueprint for a Turing Machine (1st of 2 parts):
• Architectural Part:
• An infinitely long tape divided into squares stretching in two directions
• there will never be a shortage of space
• A finite set of symbols
• 0 and 1 (can use any finite alphabet)
• executes absolutely simple, primitive operations on these symbols.
• A reading head
• Can scan squares one at a time
• Can move to the left or right one square at a time, or not at all
• Can inscribe, or erase symbols on squares
• A finite set of states
• The states correspond to various finite configurations of a Turing
machine’s reading head.
Turing Machine
• Blueprint for a Turing Machine (2nd of 2 parts):
• Procedural Part:
• The behavior of a Turing machine is controlled by a finite series of
instructions
• Each instruction is governed by
• the state of the reading head and
• the current symbol being scanned
The
• Turing machine understands only two commands:
• The first instructs the machine what to write or erase
• The second, whether it is to move to the left or the right by one
square
• Each command given to a Turing machine has 4 parts:
• If the machine is in this state, and it is scanning that symbol, then
it must write this symbol or that symbol, and move one square to
the left or one square to the right, or not at all.
Effective Computability: Turing Machine
• finite alphabet of symbols
• finite set of states
• infinite tape marked off with squares
In the Course Book
A Turing Machine for Incrementing a Value
each of which is capable of carrying a
single symbol
• mobile sensing-and-writing head that
can travel along the tape one square at a
time
• state-transition diagram containing
the instructions that cause changes to
take place at each step
Uses a state transition table instead of finite state diagram
A Turing machine is computing a function:
a mapping from input to output!
An Example: Palindromes
Can generalize to any finite alphabet
Can encode decision problems (recall hilbert’s 23rd problem!)
#/#, L
move-a
test-a
#/#, L
a/#, R
mark
move-b
a/#, L
b/b, L
YES
a/a, R
b/b, R
b/#, R
NO
#/#, L
a/a, L
b/b, L
return
a/a, L
#/#, L
test-b
#
a b b a #
b/#, L
with a given set of input values
• Effectively Computable Function: Output values can be
determined algorithmically from input values.
• Turing Computable Function: Any function that can be
• Place input value in binary form on a tape
• Run the Turing machine associated with the function until it
halts
#
a b a a #
On Algorithms and Computable Functions
• The notion of an algorithm is a human artifact
• It stems from the human concept of effectively
getting something done (mentally).
• In essence, it may be synonymous with anything a
human mind can effectively do.
•
• Computing a Function: Determining the output value associated
computed from a Turing machine in the following manner:
#/#, R
Examples:
• Function: A correspondence between a collection of possible
input values and a collection of possible output values so that each
possible input is assigned a single output
a/a, R
b/b, R
#/#, L
Functions
• Controversial!
The notion of computability is a formally defined
mathematical concept.
• Turing Computability is one such formal definition.
• For any computable function, there is a Turing
machine which effectively computes that function.
• Read the output value from the tape
Church-Turing Thesis
Turing machines are capable of solving any effectively computable
algorithmic problem! Put differently, any algorithmic problem for
which we can find an algorithm that can be programmed in some
programming language, any language, running on some computer,
any computer, even one that has not yet been built, and even one
requiring unbounded amounts of time and memory space for ever
larger inputs, is also solvable by a Turing machine!
(Partial) Recursive Functions: Gödel,Kleene
Lambda Calculus: Church
Post Production Systems: Post
Turing Machines: Turing
Unlimited Register Machines: Cutland
Scheme, C++,
C, Ada, Phython,
Java, Ruby, C#,
LISP ,.......
Pentium,
Multicore, Big Blue,
super computer, ...
Turing
Machine
A Truly Astonishing and Remarkable (formally well-grounded )Claim!
Uncomputable Functions?
• Are there well-defined functions that are not effectively computable?
• If so, how would we prove such a thing?
• Turing did just that!
• First he required a formal mathematical notion of effective computability.
• Turing Machine
• His well-defined problem was that asked by Hilbert:
• Does there exist an algorithm that can determine the truth or falsity of any
logical proposition in a system of logic that is powerful enough to represent
the natural numbers? (Entscheidungsproblem)
• He formally proved that the function (decision problem) associated with this
An Unsolvable Problem: The Halting Problem
A Program R
what a computer can do.
• The result may in fact place limitations on what the human mind can do
also....but this is a controversial (but important) topic.
Universal Programming Language
• It would be tedious to represent algorithms at such a low
level of abstraction using Turing Machines.
• A Universal Programming Language is one in which a
solution to any computable function can be expressed
• Most popular programming languages are universal in
this sense.
• Turing Completeness
• A system of data-manipulation rules (such as a computer’s
instruction set, a programming language, etc.) is said to be
• Turing complete or computationally universal if it can be
used to simulate any single-taped Turing machine.
Potential Input
Does R halt
on X?
If R(X)
diverges
If R(X)
terminates
Yes
question is not effectively computable. There is no Turing machine that can
compute this function!
• The fact that there are functions which are not computable, places limitations on
X
No
Is there a Turing machine which computes this function?
Before answering, let’s define a bare bones language for writing programs!
Bare Bones Language
• Variables:
• refer to bit patterns interpreted as non-negative
integers
• Three Assignment Statements:
• clear name
• incr name
• decr name
• One control structure:
• while name not 0 do: <statement sequence> end;
Examples
Swap variable values
Universality of Bare Bones
• Has been shown formally to be Turing Complete
• Relation to a Turing machine:
• Input: Initial values of all variables
• Output: Final values of all variables
• The Bare Bones program itself directs the
computation of a function
Multiply X x Y
Back to the Halting Problem
Testing a Program for Self-Termination
R
Encode the program R in bare bones as a bit sequence
We define a program as self-terminating if executing
the program with all its variables initialized to the
program’s encoded representation leads to a terminating process
Observe! Any program is either self-terminating or not self-terminating
Examples: Testing a Program for Self-Termination
Self-referential Programs
While X not 0
do;
incr X;
end:
Clear X
While X not 0
do;
incr X;
end:
Binding X to the program’s
encoded representation results
in the program not halting
Binding X to the program’s
encoded representation results
in the program halting
By definition it is
not self-terminating
By definition it is
self-terminating
Case 2
Proving the Unsolvability of the
Halting Problem
Case 1
Assume:
First: Propose the existence
of a program that,
Now: If this new program were
self-terminating and
we started it with
its own encoding
as its input
given any encoded
version of a program
execution would
reach this point
with X equal to 1
Proposed
Program
so execution
would become
trapped in this
loop forever
will halt with variable
X equal to 1 if the
input represents a
self-terminating
program, or with X
equal to 0 otherwise.
Proposed
Program
adding a
while-end
structure
execution would
reach this point
with X equal to 0
Proposed
Program
so execution of
this loop would be
skipped
while X
not 0 do;
end;
while X
not 0 do;
end;
to produce
a new
program
Proposed
Program
while X
not 0 do;
end;
and execution
would halt;
• Using Diagonalization
i.e., if the new program
is not self-terminating,
then it is self-terminating!!!
i.e., if the new program is self-terminating,
then it is not self-terminating!!!
Then: If such a program exists,
we could modify it by
Another Proof
However: If this new program were
not self-terminating and
we started it with
its own encoding
as its input
Consequently:
The existence of
the proposed
program
Proposed
Program
Contradiction
the existence of
a new program
would
lead to
Proposed
Program
while X
not 0 do;
that is neither
self-terminating
nor non selfterminating
Negate the
assumption
So the existence of the proposed
program is impossible!
Universal Turing Machine
• One consequence of the Church-Turing thesis is the existence of universal
algorithms.
• A universal algorithm A has the ability to act like any algorithm
whatsoever
• Input: the description of any algorithm P
• Input: any legal input value to the algorithm, X
• It runs, or simulates P(X), halting if P(X) halts with the outputs that
Universal Turing Machine: A blueprint for a general
purpose computer
Formal
mathematical
abstraction of a
general computing
device
Ace Computer
algorithm A
P(X) would normally give had P been running on X
•
• It is presented with a program P in a language with legal input X, to P; It
executes P on X and outputs the result.
•
P implements A;
is written in
language L2
A computer or interpreter is very much like a universal algorithm
• Stored Program Concept (program as data)
Universal though means that it is insensitive to the choice of language or
machine, which a computer or interpreter is not.
LISP: Eval
Programs as data
program P
Run P
on X
input X
Universal program U,
written in language L1;
simulates the effect of a
program in L2 on an input
Let both L1 and L2 be the language of Turing Machines!
Observe: U can simulate any computer in existence
Observe: L1 can simulate algorithms in its own language L1
Conclusions
• Gödel’s result:
• Any formal system expressive enough to axiomatize arithmetic is and always will be
incomplete!
• There are mathematical formulas which are true but can not be shown within the
formal system to be true!
• One such formula is the consistency of the axioms of arithmetic themselves.
• Turing’s result:
• There is no algorithm that can determine the truth or falsity of any logical proposition
in a system of logic that is powerful enough to represent the natural numbers
(Entscheidungsproblem)
•
• As a side effect:
• Turing proposed a formal model of effective computability
• Turing defined a mathematical abstraction of a computing device
Both Gödel and Turing formally showed the limits to formal systems and the degree of
mechanization possible.
• If mind is mechanism, what does this say about human intelligence?
• What does this say about the goals of artificial intelligence?
Computational Problems
Computational Problems that
CANNOT be solved by any algorithm
Computability Theory
Computational Problems that CAN
be solved by an algorithm
Computational Problems that CAN
be solved by an algorithm
CANNOT be solved in any practical
sense due to excessive time/space
requirements
CAN be solved in a practical
sense with reasonable time/space
requirements
Intractable Class NP
Computational
Complexity Theory
Tractable
Class P