MATH 224 – Discrete Mathematics
Algorithms and Complexity
An algorithm is a precise recipe or set of instruction for solving a problem. In addition,
to be considered an algorithm the set of instructions must solve the problem with a finite
number of steps. In other words, if the algorithm is implemented on a computer, it must
terminate with a solution. An algorithm should leave nothing to chance and must not
have any infinite loops. (There are some algorithms that do use random or probabilistic
techniques, and therefore may leave some things to chance.)
Algorithms solve problems such as sorting, find an object, finding the shortest path
between two points, determining mathematical functions such as logarithms among
others.
Specifying an Algorithms
• Informal English description
• Formal English description
• Pseudo-Code
• Code in programming language (e.g., C++)
4/13/2015
1
MATH 224 – Discrete Mathematics
Describing an Algorithm – Sorting an Array 0..n−1
Informal description of selection sort
• Find the smallest number and put it in the first position
• Find the second smallest number and put it in the second position
• Continue as above until the array is sorted
Formal Description
1.
2.
3.
4.
5.
Set i = 0
Find the smallest value between positions i and n−1
Swap the value at i with the smallest value
Set i = i+1
If i < n −1 go back to Step 2, otherwise quit
4/13/2015
2
MATH 224 – Discrete Mathematics
Describing an Algorithm – Sorting an Array 0.. n−1
Pseudo-Code
void function sort(Array A[n])
for i in [0..n-2] →
pos := i
for j in [i+1.. n-1] →
if A[j] < A[pos] →
pos := j
fi
rof
swap(A[i], A[pos])
rof
end sort
C++ Version
void function sort (A_type A[ ], int n) {
for (int i = 0; i < n-1; i++) {
pos = i;
for (int j = i+1; j < n; j++)
if (A[j] < A[pos])
pos = j;
// fi
// rof
swap(A[i], A[pos]);
} // rof
} // end sort
Note the use of “:=” for assignment as do Pascal and Ada.
4/13/2015
3
MATH 224 – Discrete Mathematics
Describing an Algorithm – Insertion Sort Array 0.. n−1
Pseudo-Code
C++ Version
procedure insert_sort (A_type A[ ], int n)
for i1 in [1..n-1] →
i2 := i1–1
item := A[i1]
do (i2 ≥ 0) and (item < A[i2]) →
A[i2+1] := A[i2];
i2 := i2 –1
od
A[i2+1] := item
rof
end insert_sort
void function insert_sort (A_type A[ ], int n) {
int i1. i2;
A_type item;
for (int i1 = 1; i < n; i++) {
i2 = i1 –1 ;
item = A[i1];
while (i2 >= 0 && item < A[i2]) {
A[i2+1] = A[i2];
i2 – – ;
} // end while
A[i2 + 1] = item;
} // rof
} // end insert_sort
4/13/2015
4
MATH 224 – Discrete Mathematics
Binary search of an Array 1.. n
Pseudo-Code – Algorithm 3 from the textbook (Page 172)
procedure binary_search(x: integer a[1..n] : integer)
i := 1;
j := n
while i < j
m := (i+j)/2
// just integer division in C++
└
┘
if x > a[m]
i := m+1
else
j := m
fi
end while
if x = a[i] location is i
// here = corresponds to
else x is not in the array
// C++ = =
end binary_search
Note the text uses “:=” for assignment as do Pascal and Ada.
4/13/2015
5
MATH 224 – Discrete Mathematics
Exponentiation ap
Incremental version
int function exp(int a, int p)
if p ≤ 0 → return 1
|p> 0 →
val := a
for i in [1..p-1] →
val := val*a
rof
fi
return val
end exp
How times does the code inside the for statement execute?
4/13/2015
6
MATH 224 – Discrete Mathematics
Exponentiation ap
Divide-and-Conquer version (using recursion)
int function exp(int a, int p)
if p ≤ 0 → return 1
| p > 0→
val := exp(a, p/2)
if p is even →
return val*val
| p is odd →
fi
return val*val*a
fi
end exp
How many times does the if statement execute?
4/13/2015
7
MATH 224 – Discrete Mathematics
Exponentiation ap
The Program (Run Time) Stack
int function exp(int a, int p)
if p ≤ 0 → return 1
| p > 0→
val := exp(a, p/2)
if p is even →
return val*val
| p is odd →
return val*val*a
fi
fi
end exp
exp(3, 0)
Return 1
1
exp(3, 1)
val = 1
3
exp(3, 2)
val = 3
9
exp(3, 5)
val = 9
exp(3, 11)
243
val = 243
MAIN a=3, p=11
4/13/2015
177,147
8
MATH 224 – Discrete Mathematics
Greedy Algorithm for Change
Algorithm 6 from the textbook (Page 175)
procedure change(n : integer c[1..r] : integer) // C is an array of coins sorted from
for i := 1 to r
// largest to smallest, and n is the
while n > c[i]
// the total amount of change.
add c[i] to change
n = n – c[i]
end while
rof
end change
4/13/2015
9
MATH 224 – Discrete Mathematics
Asymptotic Growth of Functions
Big-0 – an Upper Bound
Omega – a Lower Bound
Theta – a Tight Bound
4/13/2015
10
MATH 224 – Discrete Mathematics
Examples Functions and their Asymptotic Growth
4/13/2015
11
MATH 224 – Discrete Mathematics
Polynomial Functions
1600
1400
O(N2)
1200
T(N)
1000
n0 = 10 for O
800
n0 = 0 for 
600
(N2)
400
200
0
1
1
4/13/2015
2
3
4
5
6
7
8
9
10
3N2 +10N
+3
3N^2
+10N+3
11
10
12
13
4N2
4N^2
14
15
3N2
16
17
18
19
20
20
3N^2
12
MATH 224 – Discrete Mathematics
Log Function
30
n0 = 5.7*106
25
N 0.2
Lg(N)
20
15
10
5
0
1
4
7
10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97
X axis * 105
4/13/2015
N0.2
N^0.2
lg(N)
13
MATH 224 – Discrete Mathematics
Evaluating Algorithms and Complexity
• Execute the algorithm on a standard computer with standard tests
• Problems: depends on the input and the computer
• Count the number of steps
• Problem: depends on the input and can be difficult to do
• Use big-O (asymptotic) evaluation
•Problem: provides an approximation and does not give the full picture
4/13/2015
14
MATH 224 – Discrete Mathematics
Searching an Array
VALUE
INDEX
4/13/2015
-4 -1 7 12 21 23 31 45 72 83 86 92
0 1 2
9 10 11
15
MATH 224 – Discrete Mathematics
Linear Search of an Array
VALUE
INDEX
-4 -1 7 12 21 23 31 45 72 83 86 92
0 1 2 3 4 5 6 7 8 9 10 11
Inspects 11 out of 12 elements O(N)
4/13/2015
16
MATH 224 – Discrete Mathematics
Binary Search of an Array
VALUE
INDEX
-4 -1 7 12 21 23 31 45 72 83 86 92
0 1 2
5
8 9 10 11
1
2 3 4
Searches O(log n) locations (4 in this example)
Divides segments in half until a value is found
4/13/2015
17
MATH 224 – Discrete Mathematics
Complexity of Algorithms
The complexity of an algorithm refers to how long it takes for an algorithm to
solve a problem (time complexity) or how much memory the algorithm takes
(space complexity). Complexity normally expressed as a function of one or more
variables. For example, in the previous slides the binary search algorithm is said
to take lg(n) steps, where n is the number of elements in the array being searched.
Complexity is normally express using O, Ω, or Θ notation since the precise
running time is often difficult to calculate and will depend on the computer that is
used.
4/13/2015
18
MATH 224 – Discrete Mathematics
Types of Complexity
Most often when considering an algorithm either the worst-case or average-case
complexity is analyzed. More rarely the best-case complexity may be considered.
Worst-case refers to the maximum for a given size problem. So for example the
worst-case time complexity for binary search is Θ(lg(x)) for an array of size x.
Average-case refers to the average for an algorithm of a given size. It turns out
that the average case is also Θ(lg(x)) for binary search since the average case takes
one less iteration than does the worst-case.
Best-case refers to the best an algorithm can do in the ideal case. So, for
example, if the item being searched is in the exact middle of an array, binary search
will find it on the first iteration. What would be the best-case time complexity
for binary search? Best case is rarely used.
4/13/2015
19
MATH 224 – Discrete Mathematics
Worst-Case and Big-O
Because it is the easiest to calculate, worst-case analysis is most often used. In
addition, it is often the most useful since it is often necessary to minimize the
worst-case time or memory usage.
Also Big-O is most often used when stating the complexity of an algorithm since
it is often easier to calculate than Θ. In most cases, people are in the habit of
using Big-O when they really mean Θ. Do you have any idea why Big-O is
more common than Θ?
4/13/2015
20
MATH 224 – Discrete Mathematics
Complexity of Selection Sort
void function sort(Array A[n])
for i in [0..n-2] →
pos := i
for j in [i+1.. n−1] →
if A[j] < A[pos] →
pos := j
fi
rof
swap(A[i], A[pos])
rof
end sort
How many times does the outer
for-loop execute?
The inner loop is more complex
since the starting value keeps
increasing. When i = 0, the inner
loop executes n − 1 times, but when
i = n − 2 it only executes once. So
the number of steps is indicated by
a sum of n − 1 + n − 2, n − 3, … 1.
How is this sum written using the summation symbol Σ?
4/13/2015
21
MATH 224 – Discrete Mathematics
Complexity of Selection Sort
void function sort(Array A[n])
for i in [0..n-2] →
pos := i
for j in [i+1.. n−1] →
if A[j] < A[pos] →
pos := j
fi
rof
swap(A[i], A[pos])
rof
end sort
4/13/2015
Thus the total number of steps in
the worst-case is something like
Σn−1 (2i ) + 2(n−1) +1
i=1
What does this look like in
simplified form?
n(n–1) + 2(n−1) + 1 = n2 + n − 1
What is this in Big-O or Θ?
Note that Σn (i) = n(n+1)/2
i=1
22