Chapter 1
Introduction
What is an Algorithm?
1-1
Algorithm
Historical Perspective
• An algorithm is a sequence of unambiguous
instructions for solving a problem, i.e., for obtaining a
required output for any legitimate input in a finite
amount of time.
• Can be represented various forms
• Muhammad ibn Musa Al-Khwarizmi
– 9th century mathematician
– “father of algebra”
– al-Khwarizmi (Algorizm) (770 - 840 C.E.)
• Euclid’s algorithm for finding the greatest common divisor
• Unambiguity/clearness
• The word algorism originally referred only to the rules of
performing arithmetic using Hindu-Arabic numerals but
evolved via European Latin translation of Al-Khwarizmi's
name into algorithm by the 18th century. The use of the
word evolved to include all definite procedures for solving
problems or performing tasks.
• Effectiveness
• Finiteness/Termination
• Correctness
1-2
1-3
Notions of algorithm and problem
Example of computational problem: sorting
• Statement of problem:
– Input: A sequence of n numbers <a1, a2, …, an>
– Output: A reordering of the input sequence <a’1, a’2, …,
a’n> so that a’i ≤ a’j whenever i < j
Problem
Algorithm
input
(or instance)
“computer”
• Instance: The sequence <5, 3, 2, 8, 3>
• Algorithms:
output
–
–
–
–
algorithmic solution
(different from a conventional solution)
Selection sort
Insertion sort
Merge sort
… many others
1-4
1-5
Selection Sort
•
•
•
•
•
•
•
•
•
•
• Input: array a[1], …, a[n]
• Output: array a sorted in non-decreasing
order
• Algorithm:
for i=1 to n
swap a[i] with smallest of a[i], …, a[n]
• Is this unambiguous? Effective?
• See also pseudocode, Section 3.1.
1-6
Some Well-known Computational
Problems
Sorting
Searching
Shortest paths in a graph
Minimum spanning tree
Primality testing
Traveling salesman problem
Knapsack problem
Chess
Towers of Hanoi
Program termination
Some of these problems don’t
have efficient algorithms, or
algorithms at all!
1-7
Basic Issues Related to Algorithms
• How to design algorithms?
• How to express algorithms?
What is an algorithm?
Recipe, process, method, technique, procedure,
routine,… with the following requirements:
1. Finiteness
• terminates after a finite number of steps
• Efficiency (or complexity) analysis
2. Definiteness
– Theoretical analysis
– Empirical analysis
• rigorously and unambiguously specified
3. Clearly specified input
• valid inputs are clearly specified
4. Clearly specified/expected output
• Does there exist a better algorithm?
• can be proved to produce the correct output given a valid input
– Lower bounds
– Optimality
5. Effectiveness
• steps are sufficiently simple and basic
1-8
What does algorithm look like?
Pseudocode
Program-like
Algorithm Sample
Input: …
Output: …
Step 1: …
Step 2: …
Step 3: …
…
Step n: …
Type Sample ( i1, i2, …, ik )
{
…
…
…
…
return output
}
1-9
Why study algorithms?
• Theoretical importance
– the core of computer science
• Practical importance
– A practitioner’s toolkit of known algorithms
– Framework for designing and analyzing algorithms
for new problems
實用 & 實際
1-10
1-11
Euclid’s Algorithm
Two descriptions of Euclid’s algorithm
Problem: Find gcd(m,n), the greatest common divisor
of two nonnegative, not both zero integers m and n
Examples: gcd(60,24)=12, gcd(60,0)=60, gcd(0,0)=?0
Euclid’s algorithm is based on repeated application of
equality
gcd(m,n) = gcd(n, m mod n)
until the second number becomes 0, which makes the
problem trivial, m≥n.
Example: gcd(60,24)=gcd(24,12)=gcd(12,0)=12
A Step 1: If n = 0, return m and stop; otherwise go
to Step 2.
Step 2: Divide m by n and assign the value of
the remainder to r.
Step 3: Assign the value of n to m and the value
of r to n. Go to Step 1.
Pseudocode
B while n ≠ 0 do
Program-like
r ← m mod n
m← n
n←r
return m
1-12
Other methods for gcd(m,n) (1/2)
Consecutive integer checking algorithm
Step 1 Assign the value of min{m,n} to t
Step 2 Divide m by t. If the remainder is 0, go to
Step 3; otherwise, go to Step 4
Step 3 Divide n by t. If the remainder is 0, return t
and stop; otherwise, go to Step 4
Step 4 Decrease t by 1 and go to Step 2
Is this slower than Euclid’s algorithm?
How much slower?
O(n), if n ≤ m , vs. O(log n)
1-13
Other methods for gcd(m,n) (2/2)
Middle-school procedure
Step 1 Find the prime factorization of m
Step 2 Find the prime factorization of n
Step 3 Find all the common prime factors
Step 4 Compute the product of all the common
prime factors and return it as gcd(m,n)
Is this an algorithm?
How efficient is it? Time complexity: O(
1-14
)
1-15
Sieve of Eratosthenes (ca. 200 B.C.)
Sieve of Eratosthenes - Example
Input: Integer n ≥ 2
Output: List of primes less than or equal to n
for p ← 2 to n do A[p] ← p
for p ← 2 to n do
if A[p] ≠ 0 //p hasn’t been previously eliminated from the list
j ← p* p
while j ≤ n do
A[j] ← 0 //mark element as eliminated
j←j+p
Algorithm steps for primes below 120 (including optimization of
terminating when square of prime exceeds upper limit)
1-16
Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1-17
Fundamentals of Algorithmic
Problem Solving
1-18
1-19
Ascertaining the Capabilities of the
Computational Device
Understanding the Problem
• An input to an algorithm specifies an instance of the
problem the algorithm solves. It is very important to
specify exactly the set of instances the algorithm needs
to handle. (As an example, recall the variations in the
set of instances for the three greatest common divisor
algorithms discussed in the previous section.)
• If you fail to do this, your algorithm may work correctly
for a majority of inputs but crash on some “boundary”
value. Remember that a correct algorithm is not one
that works most of the time, but one that works
correctly for all legitimate inputs.
• Once you completely understand a problem, you need to
ascertain the capabilities of the computational device the
algorithm is intended for. The vast majority of algorithms in
use today are still destined to be programmed for a computer
closely resembling the von Neumann machine—a computer
architecture outlined by the prominent Hungarian-American
mathematician John von Neumann (1903–1957), in
collaboration with A. Burks and H. Goldstine, in 1946.
• The essence of this architecture is captured by the so-called
random-access machine (RAM). Its central assumption is that
instructions are executed one after another, one operation at
a time. Accordingly, algorithms designed to be executed on
such machines are called sequential algorithms.
• Parallel Algorithms ???
1-20
1-21
Choosing between Exact and
Approximate Problem Solving
Von Neumann architecture scheme
• The next principal decision is to choose between solving the
problem exactly or solving it approximately. In the former case,
an algorithm is called an exact algorithm; in the latter case, an
algorithm is called an approximation algorithm. Why would
one opt for an approximation algorithm?
Pascal GP100 Full GPU with 60 SM Units
(NVIDIA Tesla P100)
1-22
– First, there are important problems that simply cannot be solved
exactly for most of their instances.
– Second, available algorithms for solving a problem exactly can be
unacceptably slow because of the problem’s intrinsic complexity. This
happens, in particular, for many problems involving a very large
number of choices; you will see examples of such difficult problems in
Chapters 3, 11, and 12.
– Third, an approximation algorithm can be a part of a more
sophisticated algorithm that solves a problem exactly.
1-23
Designing an Algorithm and Data
Structures
Algorithm Design Techniques
• An algorithm design technique (or “strategy” or “paradigm”)
is a general approach to solving problems algorithmically that
is applicable to a variety of problems from different areas of
computing
–
–
–
–
–
Brute force
Decrease and conquer
Divide and conquer
Transform and conquer
Space and time
tradeoffs
–
–
–
–
–
Greedy approach
Dynamic programming
Iterative improvement
Backtracking
Branch and bound
1-24
Analysis of Algorithms
• Pseudocode is a mixture of a natural language
and programming language-like constructs.
• Pseudocode is usually more precise than
natural language, and its usage often yields
more succinct algorithm descriptions.
Algorithms + Data Structures
+ Programming Language
||
Programs
1-25
Coding an Algorithm
• How good is the algorithm?
– time efficiency
– space efficiency
– correctness ignored in this course
• In the academic world, the question leads to an interesting
but usually difficult investigation of an algorithm’s optimality.
Actually, this question is not about the efficiency of an
algorithm but about the complexity of the problem it solves:
What is the minimum amount of effort any algorithm
will need to exert to solve the problem?
• Characteristics of an Algorithm
– simplicity
– generality
• Does there exist a better algorithm?
– lower bounds
– optimality
1-26
• For some problems, the answer to this question is known. For
example, any algorithm that sorts an array by comparing
values of its elements needs about nlog2n comparisons for
some arrays of size n.
• But for many seemingly easy problems such as integer
multiplication, computer scientists do not yet have a final
1-27
answer.
In conclusion
Example
Question: #9 in Exercises 1.2
Consider the following algorithm for finding the distance between the two
closet elements in an array of numbers.
As a rule, a good algorithm is
a result of repeated effort and
rework.
Algorithm MinDistance(A[0..n − 1])
Example:
100, 77, 20, 50, 82, 33, 120, 180
//Input: Array A[0..n − 1] of numbers
//Output: Minimum distance between two of its elements
dmin ←∞
for i ← 0 to n − 1 do
for j ← 0 to n − 1 do
if i ≠ j and |A[i] − A[j]| < dmin
dmin ← |A[i] − A[j]|
return dmin
1-28
Make as many improvements as you can in this algorithmic solution to the
problem. (If you need to, you may change the algorithm altogether; if not,
improve the implementation given.)
1-29
Important Problem Types
•
•
•
•
•
•
•
Important Problem Types
1-30
Sorting
Searching
String processing
Graph problems
Combinatorial problems
Geometric problems
Numerical problems
1-31
Sorting (I)
Sorting (II)
• Examples of sorting algorithms
• Rearrange the items of a given list in
ascending order.
–
–
–
–
–
– Input: A sequence of n numbers <a1, a2, …, an>
– Output: A reordering <a’1, a’2, …, a’n> of the input
sequence such that a’1≤ a’2 ≤ … ≤ a’n.
• Why sorting?
Selection sort
Bubble sort
Insertion sort
Merge sort
Heap sort …
• Evaluate sorting algorithm complexity: the
number of key comparisons.
• Two properties
– Help searching
– Algorithms often use sorting as a key subroutine.
– Stability: A sorting algorithm is called stable if it
preserves the relative order of any two equal elements
in its input.
– In place: A sorting algorithm is in place if it does not
require extra memory, except, possibly for a few
memory units.
• Sorting key
– A specially chosen piece of information used to guide
sorting. E.g., sort student records by names.
1-32
Selection Sort
1-33
Searching
• Find a given value, called a search key, in a
given set.
• Examples of searching algorithms
Algorithm SelectionSort(A[0..n-1])
//The algorithm sorts a given array by selection sort
//Input: An array A[0..n-1] of orderable elements
//Output: Array A[0..n-1] sorted in ascending order
for i  0 to n – 2 do
min  i
for j  i + 1 to n – 1 do
if A[j] < A[min]
min  j
swap A[i] and A[min]
1-34
– Sequential search
– Binary search, See below
Input: sorted array ai < … < aj and key x;
m (i+j)/2;
while i < j and x != am do
Time: O(log n)
if x < am then j  m-1
else i  m+1;
if x = am then output am;
1-35
String Processing
Graph Problems
• A string is a sequence of characters from an alphabet.
• Text strings: letters, numbers, and special characters.
• String matching: searching for a given word/pattern
in a text.
Examples:
• Informal definition
– A graph is a collection of points called vertices, some of
which are connected by line segments called edges.
• Modeling real-life problems
–
–
–
–
Modeling WWW
Communication networks
Project scheduling
…
–
–
–
–
–
Graph traversal algorithms
Shortest-path algorithms
Topological sorting
Graph-coloring problems (#8 in Exercises 1.3)
…
• Examples of graph algorithms
(i) searching for a word or phrase on WWW or
in a Word document
(ii) searching for a short read in the reference
genomic sequence
1-36
1-37
Find a Euler circuit
Fundamental Data Structures
Find a Hamiltonian circuit
Graph-coloring problems1-38
1-39
Fundamental data structures
• list
– array
– linked list
– string
Linear Data Structures
• Arrays
• graph
• tree and binary tree
• set and dictionary

– A sequence of n items of the
same data type that are stored
contiguously in computer
memory and made accessible by
specifying a value of the array’s
index.
• stack
• queue
• priority queue/heap




• Linked List
– A sequence of zero or more
nodes each containing two kinds
of information: some data and
one or more links called pointers
to other nodes of the linked list.
– Singly linked list (next pointer)
– Doubly linked list (next +
previous pointers)
Arrays

Linked Lists




a1
fixed length (need preliminary
reservation of memory)
contiguous memory locations
direct access
Insert/delete
dynamic length
arbitrary memory locations
access by following links
Insert/delete
a2
…
an
.
1-41
1-40
Stacks and Queues
Priority Queue and Heap
• Stacks
• Priority queues (implemented using heaps)
– A stack of plates
– A data structure for maintaining a set of elements,
each associated with a key/priority, with the
following operations
• insertion/deletion can be done only at the top.
• LIFO
– Two operations (push and pop)
• Queues
• Finding the element with the highest priority
• Deleting the element with the highest priority
• Inserting a new element
9
6
8
– Scheduling jobs on a shared computer
5 2 3
– A queue of customers waiting for services
• Insertion/enqueue from the rear and deletion/dequeue
from the front.
• FIFO
– Two operations (enqueue and dequeue)
9 6 8 5 2 3
1-42
1-43
Graphs
Graph Representation
• Formal definition
• Adjacency matrix
– A graph G = <V, E> is defined by a pair of two sets: a finite
set V of items called vertices and a set E of vertex pairs
called edges.
– n x n boolean matrix if |V| is n.
– The element on the ith row and jth column is 1 if there’s an edge
from ith vertex to the jth vertex; otherwise 0.
– The adjacency matrix of an undirected graph is symmetric.
• Undirected and directed graphs (digraphs).
• What’s the maximum number of edges in an
undirected graph with |V| vertices?
• Complete, dense, and sparse graphs
• Adjacency linked lists
– A collection of linked lists, one for each vertex, that contain all the
vertices adjacent to the list’s vertex.
• Which data structure would you use if the graph is a 100node star shape?
– A graph with every pair of its vertices connected by an
edge is called complete, K|V|
1
2
3
4
0111
1000
1000
1000
1-44
Weighted Graphs
•
• Weighted graphs
– Graphs or digraphs with numbers assigned to the edges.
6
1
3
5
9
8
2
2
1
1
1
3
4
1-45
Graph Properties - Paths and
Connectivity
Paths
– A path from vertex u to v of a graph G is defined as a
sequence of adjacent (connected by an edge) vertices that
starts with u and ends with v.
– Simple paths: All edges of a path are distinct.
– Path lengths: the number of edges, or the number of
vertices – 1.
• Connected graphs
7
4
– A graph is said to be connected if for every pair of its
vertices u and v there is a path from u to v.
• Connected component
– The maximum connected subgraph of a given graph.
1-46
1-47
Graph Properties - Acyclicity
• Cycle
– A tree (or free tree) is a connected acyclic graph.
– Forest: a graph that has no cycles but is not
necessarily connected.
– A simple path of a positive length that starts and ends
a the same vertex.
• Acyclic graph
• Properties of trees
– A graph without cycles
– DAG (Directed Acyclic Graph)
1
2
3
4
Trees
• Trees
– For every two vertices in a tree there always exists
exactly one simple path from one of these vertices to
the other. Why?
• Rooted trees: The above property makes it possible to select an
arbitrary vertex in a free tree and consider it as the root of the so
called rooted tree.
rooted
• Levels in a rooted tree.

|E| = |V| - 1
1-48
Rooted Trees (I)
3
2
4
3
4
1
2
5
1-49
• Depth of a vertex
– For any vertex v in a tree T, all the vertices on the simple path
from the root to that vertex are called ancestors.
– The length of the simple path from the root to the vertex.
Descendants
• Height of a tree
– All the vertices for which a vertex v is an ancestor are said to be
descendants of v.
– The length of the longest simple path from the root to a leaf.
• Parent, child and siblings
– If (u, v) is the last edge of the simple path from the root to
vertex v, u is said to be the parent of v and v is called a child of
u.
– Vertices that have the same parent are called siblings.
h=2
3
4
• Leaves
– A vertex without children is called a leaf.
1
5
2
• Subtree
– A vertex v with all its descendants is called the subtree of T
rooted at v.
5
Rooted Trees (II)
• Ancestors
•
1
1-50
1-51
Ordered Trees
Summary (1/2)
• Ordered trees
– An ordered tree is a rooted tree in which all the children of each vertex
are ordered.
• Binary trees
– A binary tree is an ordered tree in which every vertex has no more than
two children and each children is designated s either a left child or a
right child of its parent.
• Binary search trees
– Each vertex is assigned a number.
– A number assigned to each parental vertex is larger than all the
numbers in its left subtree and smaller than all the numbers in its right
subtree.
• log2n ≤ h ≤ n – 1, where h is the height of a binary tree with n
nodes.
9
6
6
8
3
9
5 2 3
2 5
8
1-52
Summary (2/2)
• A good algorithm is usually the result of repeated
efforts and rework.
• The same problem can often be solved by several
algorithms.
• Algorithms operate on data. This makes the issue of
data structuring critical for efficient algorithmic
problem solving.
• An abstract collection of objects with several
operations that can be performed on them is called an
abstract data type (ADT). Modern object-oriented
languages support implementation of ADTs by means
of classes.
1-54
• An algorithm is a sequence of nonambiguous instructions
for solving a problem in a finite amount of time. An input to
an algorithm specifies an instance of the problem the
algorithm solves.
• Algorithms can be specified in a natural language or
pseudocode; they can also be implemented as computer
programs.
• Among several ways to classify algorithms, the two
principal alternatives are:
– to group algorithms according to types of problems they solve
– to group algorithms according to underlying design techniques
they are based upon
1-53