Lecture 1
Jeff Edmonds
York University
Iterative Algorithms
& Loop Invariants
Steps in an Iterative Algorithm
Typical Loop Invariants
Lower Bound for Sort
(same as in 3101)

A Sequence of Actions vs
A Sequence of Assertions
Max( a,b,c )
“preCond: Input has 3 numbers.” m = a
“assert: m is max in {a}” if( b>m ) m = b endif
“assert: m is max in {a,b}” if( c>m ) m = c endif
“assert: m is max in {a,b,c}” return(m)
“postCond: return max in {a,b,c}”
It is helpful to have different ways of looking at it.
2

Purpose of Assertions
Useful for
– thinking about algorithms
– developing
– describing
– proving correctness
3

Definition of Assertions
An assertion is a statement about the current state of the data structure that is either true or false. eg. the amount in your bank account is not negative.
4

Definition of Assertions
It is made at some particular point during the execution of an algorithm.
It should be true independent of the path followed through the code and independent of the input.
If it is false, then something has gone wrong in the logic of the algorithm.
5

Definition of Assertions
An assertion is not a task for the algorithm to perform.
It is only a comment that is added for the benefit of the reader.
6

Definition of Assertions
• The
– Everything that is true at a particular instant in time.
– Imagine flying in from Mars, what do you need to know about the present to be able to continue.
– It does not say anything about the past, i.e. how the computation got here.
– Eg
<x=8, y=3, current line of code = line 84>
7

Definition of Assertions
• The current state of the computation
• An
, A
– is a function
• that takes as input the current state
• and that outputs
– Eg A = “x is odd”
True or False
•
A(<x=5, y=3, line
84
> ) = True
• A(<x=8, y=3, line
84
> ) = False

Computation is on the path

Something has gone wrong!
8

Algorithm Termination
You need to define some
Measure of progress to prove that your algorithm eventually terminates.
• An
, M
– is a function
• that takes as input the current state
• and that outputs a real number
– Eg Measure = “value of y”
•
M(<x=5, y=3, line
84
> ) = 3
– We must prove that this number goes down (up) each iteration.
9

Designing an Algorithm Is this sufficient?
Define Problem Define Loop
Invariants
Define Measure of
Progress
79 km to school
Define Step Define Exit Condition Maintain Loop Inv
Exit
Exit
Make Progress
Exit
79 km 75 km
Initial Conditions
 km
Ending
0 km Exit
Exit
10

Explaining Insertion Sort
We maintain a subset of elements sorted within a list. The remaining elements are off to the side somewhere. Initially, think of the first element in the array as a sorted list of length one. One at a time, we take one of the elements that is off to the side and we insert it into the sorted list where it belongs. This gives a sorted list that is one element longer than it was before. When the last element has been inserted, the array is completely sorted.
88
31
25
52
14
98
62
79
30
23
14,23,25,30,31,52,62,79,88,98
11

Partial Correctness
Establishing Loop Invariant
<preCond> codeA
<loop-invariant>
Maintaining Loop Invariant
Exit
<loop-invariant>
¬<exit Cond> codeB
Clean up loose ends
<loop-invariant>
<loop-invariant>
<exit Cond> <postCond>
Exit codeC
Proves that IF the program terminates then it works
<PreCond> & <code>

<PostCond>
12

Exit
Consider an Algorithm
Exit
79 km 75 km
 km 0 km Exit Exit
13

Loop Invariant Maintained
Exit
14

Computation can always proceed
15

Computation always makes progress
79 km 75 km
Exit
79 km 75 km
16

Computation Terminates
 km
79 km
75 km
0 km
0 km
0 km Exit
Exit
17

Computation Terminates
Exit
Exit
18

Exit
Consider an Algorithm
Exit
79 km 75 km
 km 0 km Exit Exit
This is sufficient!
19

Typical Types of Loop Invariants
• More of the input
• More of the output
• Narrowed the search space
• GCD Like
• Fairy God Mother
20

More of the Input
Loop Invariant
The input consists of an array of objects
Solution
Extra
We have read in the first i objects.
We will pretend that this prefix is the entire input.
We have a solution for this prefix plus some additional information
21

Loop Invariants and
Deterministic Finite Automaton
L = { a
{0,1}* | a has length at most three and the number of 1's is odd }.
22

More of the Input
Loop Invariant
The input consists of an array of objects
Solution
Solution Extra
Extra
We read in the i +1 st object.
We will pretend that this larger prefix is the entire input.
We extend the solution we have to one for this larger prefix.
(plus some additional information)
23

More of the Input
Loop Invariant
The input consists of an array of objects
Solution
Solution Extra
Extra
Exit
79 km 75 km i to i +1
24

More of the Input
Loop Invariant
The input consists of an array of objects
Exit
Solution
In the end, we have read in the entire input.
The LI gives us that we have a solution for this entire input.
25

Insertion Sort
The input consists of an array of integers
52,23,88,31,25,30,98,62,14,79
23,31,52,88
Solution
We have read in the first i objects.
We will pretend that this prefix is the entire input.
We have a solution for this prefix
26

Insertion Sort
The input consists of an array of integers
52,23,88,31,25,30,98,62,14,79
23,31,52,88
23,25,31,52,88
We read in the i +1 st object.
We will pretend that this larger prefix is the entire input.
We extend the solution we have to one for this larger prefix.
27

Typical Types of Loop Invariants
• More of the input
• More of the output
• Narrowed the search space
• GCD Like
• Fairy God Mother
28

More of the Output
The output
Loop Invariant consists of an array of objects
I have produced the first i objects.
Produce the i +1 st output object.
Exit
79 km 75 km i to i +1
Exit
Done when output n objects.
29

Exit
Selection Sort
Input:
52,23,88,31,25,30,98,62,14,79
The output consists of an array of integers
14,23,25,30,31,52,62,79,88,98
I have produced the first i objects.
Produce the i +1 st output object.
Done when output n objects.
30

Typical Types of Loop Invariants
• More of the input
• More of the output
• Narrowed the search space
• GCD Like
• Fairy God Mother
31

Define Loop Invariant
• Maintain a sub-list.
• If the key is contained in the original list, then the key is contained in the sub-list.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
32

Typical Types of Loop Invariants
• More of the input
• More of the output
• Narrowed the search space
• GCD Like
•
An instance has been produced that is smaller than the actual instance
•
These two instances require the same output.
Hence, it is sufficient for me to forget about the actual instance and simply give the output for the new smaller one.
• Fairy God Mother
33

GCD(a,b)
Input: <a,b>
Output: GCD(a,b)
= <21,9>
= 3
Maintain values <x,y> such that
GCD(a,b) = GCD(x,y)
GCD(a,b)
= GCD(a-b,b)
GCD(21,9)
= GCD(12,9) = 3
Replace <x,y> with <x-y,y>
GCD(a,b) = GCD(x,y)
= GCD(x-y,y) = GCD(x’,y’)
Exit
79 km 75 km smaller Running Time?
34

GCD( a , b )
Input: < a , b > = <9999999999999,2>
< x , y > = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Time = O( a )
Poly Time?
35

Size of Input Instance
5
1’’ 83920
2’’
• Size of paper
• # of bits
• # of digits
• Value
- n = 2 in 2
- n = 17 bits
- n = 5 digits
- n = 83920
• Intuitive
• Formal
• Reasonable
• Unreasonable
# of bits = log
2
(Value)
Value = 2 # of bits
36

GCD( a , b )
Input: < a , b > = <9999999999999,2>
< x , y > = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Time =
Size =
O( a ) = 2 O( n ) number of digits n = O(log( a ))
37

GCD( a , b )
Input: <a,b>
Output: GCD(a,b)
= <21,9>
= 3
Maintain values <x,y> such that
GCD(a,b) = GCD(x,y)
Exit
79 km 75 km
GCD(a,b)
= GCD(a-b,b)
GCD(21,9)
= GCD(12,9) = 3
= GCD(a mod b, b) = GCD(3,9)
Much faster
We want the first number the biggest
38

GCD( a , b )
Input: <a,b>
Output: GCD(a,b)
= <21,9>
= 3
Maintain values <x,y> such that
GCD(a,b) = GCD(x,y)
GCD(a,b)
= GCD(a-b,b)
GCD(21,9)
= GCD(12,9) = 3
= GCD(a mod b, b)
= GCD(b, a mod b)
= GCD(3,9)
= GCD(9,3)
39

GCD(a,b)
Input: <a,b> = <10000000000001,9999999999999>
<x,y> = <10000000000001,9999999999999>
= <9999999999999,2>
Little progress
= <2,1>
= <1,0>
GCD(a,b) = GCD(x,y) = 1
Every two iterations: the value x decreases by at least a factor of 2.
the size of x decreases by at least one bit
Time = O(log(a)+log(b)) = O(n)
Size = n = O(log(a)+log(b))
40

GCD( a , b )
41

Typical Types of Loop Invariants
• More of the input
• More of the output
• Narrowed the search space
• GCD Like
• Fairy God Mother
•
I have arrived from Mars.
•
Some amount of progress has been made already.
•
My Fairy God Mother has set the state of the computation to be just what I want so that I feel comfortable.
42

Two Graph Problems
• Tournament
• Euclidian Cycle
43

A Lower Bound on Sorting
Merge, Quick, and Heap Sort can sort N numbers using O(N log N) comparisons between the values.
Theorem: No algorithm can sort faster.
44

A Lower Bound on Sorting
"
A,
$
I, [ A(I)
=
P(I) or Time(A,I)

T lower
(|I|)]
I have an algorithm A that I claim works and is fast.
Oh yeah, I have an input I for which it does not .
I win if A on input I gives
•the wrong output or
•runs slow.
45

A Lower Bound on Sorting
• Sorting
– Input: Values:
Indexes:
23
1
67
2
34
3
21
4
87
5
– Output:
Values:
Indexes:
3
6
3
6
13
7
11
8
11
8
13
7
21
4
23
1
34
3
67
2
87
5
• Lower Bound:
– Input: An Algorithm for Sorting
A
– Output: An instance I on which alg A either
Values:
Indexes:
• Takes too much time
• Or gives the wrong answer
23
1
Values:
Indexes:
11
8
67
2
3
6
34
3
13
7
21
4
21
4
87
5
23
1
3
6
34
3
13
7
67
2
11
8
87
5
46

Algorithm Definition Completed
Define Problem Define Loop
Invariants
Define Measure of
Progress
79 km to school
Define Step Define Exit Condition Maintain Loop Inv
Exit
Exit
Make Progress
Exit
79 km 75 km
Initial Conditions
 km
Ending
0 km Exit
Exit
47

A Lower Bound on Sorting
I give you algorithm A
I claim it sorts.
I must output an instance
Values:
Indexes:
23
1
67
2
34
3
21
4
87
5
3
6
13
7
11
8 on which alg A either
•Takes too much time
•Or gives the wrong answer
It might as well be a permutation of 1..N
(indexes not shown)
4 7 5 2 8 1 3 6
48

A Lower Bound on Sorting
I give you algorithm A
I claim it sorts.
Need to know what the algorithm does before we can know what input to give it.
Break this cycle, one iteration at a time.
Need to give the algorithm an input before we can know what the algorithm does with the input.
49

A Lower Bound on Sorting
Restrict the search space
50

A Lower Bound on Sorting
I maintain a set of instances
(permutations of 1..n)
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Oh dear!
The first t time steps of my algorithm A are the same given any of these instances!!!
51

A Lower Bound on Sorting
Initially, I consider all N! permutations of 1..N
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Oh dear!
The first 0 time steps of my algorithm A are the same given any of these instances!!!
52

A Lower Bound on Sorting
79 km  km
Initially, I consider all N! permutations of 1..N
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
The measure of progress is the number of instances.
Initially, there are N! of them.
53

A Lower Bound on Sorting
My t+1 st time step
• a
7
< a
2
•
5 th bit of a
7
•
Any yes/no
+ a
2 question
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Because the first t time steps are the same, what the alg knows is the same.
Hence, its t+1 st step is the same.
54

A Lower Bound on Sorting
I partition my instances based on what they do on this t+1 st step.
My t+1 st time step
• a
7
< a
2
•
5 th bit of a
7
•
Any yes/no
+ a
2 question
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Because the first t time steps are the same, what the alg knows is the same.
Hence, its t+1 st step is the same.
55

A Lower Bound on Sorting
I keep the larger half.
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
56

A Lower Bound on Sorting
I keep the larger half.
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
Oh dear!
The first t+1 time steps of my algorithm A are the same given any of these instances!!!
57

A Lower Bound on Sorting
I keep the larger half.
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
79 km
Initially, I have n! Permutations.
After t time steps I have N!/2 t
58

A Lower Bound on Sorting
I keep the larger half.
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
Exit
Initially, I have n! Permutations.
After t time steps I have N!/2 t
We exit when there are two instances.
T = log(N!)
59

A Lower Bound on Sorting
Exit
N /
2 factors each at least
N! = 1
×
2
×
3
× … × N /
2
× … × N
N /
2
.
N factors each at most N.
N!
N N
We exit when there are two instances.
T = log(N!) =
Q
(N log(N)).
60

Exit
A Lower Bound on Sorting
I give you algorithm A
I must output an instance I claim it sorts.
Values:
Indexes:
23
1
67
2
34
3
21
4
87
5
3
6
13
7
11
8 on which alg A either
•Takes too much time
•Or gives the wrong answer
61

Exit
Exit
A Lower Bound on Sorting
I must output an instance
Values:
Indexes:
23
1
67
2
34
3
21
4
87
5
3
6
13
7
11
8 on which alg A either
•Takes too much time
•Or gives the wrong answer
Case 1: The algorithm does not stop at time T on these two instances.
T = log(N!) =
Q
(N log(N)).
Oops!
62

Exit
Exit
A Lower Bound on Sorting
I must output an instance
Values:
Indexes:
23
1
67
2
34
3
21
4
87
5
3
6
13
7
11
8 on which alg A either
•Takes too much time
•Or gives the wrong answer
Oops!
I must give the wrong answer on one of these!
Case 2: The algorithm stops at time T and gives an answer.
We exit when there are two instances and these need different answers.
The first T time steps of alg A are the same on these two instances and hence the same answer is given.
63

A Lower Bound on Sorting
Theorem: For every sorting algorithm A , on the worst case input instance I ,
Q
(N log
2
N) comparisons (or other bit operations) need to be executed to sort N objects.
"
A,
$
I, A(I)
=
P(I) or Time(A,I)

N log
2
N
Proof: Prover/Adversary Game
64

End Iterative Algorithms
Math Review
Recursive Algorithms
65