Analysis of Algorithms
The Sorting Problem
• Input:
– A sequence of n numbers a1, a2, . . . , an
• Output:
– A permutation (reordering) a1’, a2’, . . . , an’ of the input
sequence such that a1’ ≤ a2’ ≤ · · · ≤ an’
2
Structure of data
3
Insertion Sort
To insert 12, we need to
make room for it by moving
first 36 and then 24.
4
Insertion Sort
5
Insertion Sort
6
Insertion Sort
input array
5
2
4
6
1
3
at each iteration, the array is divided in two sub-arrays:
left sub-array
sorted
right sub-array
unsorted
7
Insertion Sort
8
INSERTION-SORT
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
1
2
3
4
5
6
7
8
a1 a2 a3 a4 a5 a6 a7 a8
key
Insert A[ j ] into the sorted sequence A[1 . . j -1]
i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
• Insertion sort – sorts the elements in place
9
Loop Invariant for Insertion Sort
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
Insert A[ j ] into the sorted sequence A[1 . . j -1]
i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
Invariant: at the start of the for loop the elements in A[1 . . j-1]
are in sorted order
10
Proving Loop Invariants
• Proving loop invariants works like induction
• Initialization (base case):
– It is true prior to the first iteration of the loop
• Maintenance (inductive step):
– If it is true before an iteration of the loop, it remains true before
the next iteration
• Termination:
– When the loop terminates, the invariant gives us a useful
property that helps show that the algorithm is correct
– Stop the induction when the loop terminates
11
Loop Invariant for Insertion Sort
• Initialization:
– Just before the first iteration, j = 2:
the subarray A[1 . . j-1] = A[1],
(the element originally in A[1]) – is
sorted
12
Loop Invariant for Insertion Sort
• Maintenance:
– the while inner loop moves A[j -1], A[j -2], A[j -3],
and so on, by one position to the right until the proper
position for key (which has the value that started out in
A[j]) is found
– At that point, the value of key is placed into this
position.
13
Loop Invariant for Insertion Sort
• Termination:
– The outer for loop ends when j = n + 1  j-1 = n
– Replace n with j-1 in the loop invariant:
• the subarray A[1 . . n] consists of the elements originally in
A[1 . . n], but in sorted order
j-1
j
• The entire array is sorted!
Invariant: at the start of the for loop the elements in A[1 . . j-1]
are in sorted order
14
Analysis of Insertion Sort
INSERTION-SORT(A)
cost
c1
c2
Insert A[ j ] into the sorted sequence A[1 . . j -1] 0
c4
i←j-1
c5
while i > 0 and A[i] > key
c6
do A[i + 1] ← A[i]
c7
i←i–1
c8
A[i + 1] ← key
for j ← 2 to n
do key ← A[ j ]
times
n
n-1
n-1
n-1



n
t
j 2 j
n
j 2
n
j 2
(t j  1)
(t j  1)
n-1
tj: # of times the while statement is executed at iteration j
T (n)  c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6  t j  1  c7  t j  1  c8 (n  1)
n
n
n
j 2
j 2
j 2
15
Best Case Analysis
• The array is already sorted “while i > 0 and A[i] > key”
– A[i] ≤ key upon the first time the while loop test is run
(when i = j -1)
– tj = 1
• T(n) = c1n + c2(n -1) + c4(n -1) + c5(n -1) + c8(n-1)
= (c1 + c2 + c4 + c5 + c8)n + (c2 + c4 + c5 + c8)
= an + b = (n)
T (n)  c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6  t j  1  c7  t j  1  c8 (n  1)
n
n
n
j 2
j 2
j 2
16
Worst Case Analysis
• The array is in reverse sorted order“while i > 0 and A[i] > key”
– Always A[i] > key in while loop test
– Have to compare key with all elements to the left of the j-th
position  compare with j-1 elements  tj = j
using
n
n(n  1)
n(n  1)
j
  j 
 1 

2
2
j 1
j 2
n
n
 ( j  1) 
j 2
n(n  1)
2
we have:
n(n  1)
n(n  1)
 n(n  1) 
T (n )  c1n  c2 (n  1)  c4 (n  1)  c5 
 1  c6
 c7
 c8 (n  1)
2
2
2


 an 2  bn  c
• T(n) = (n2)
a quadratic function of n
order of growth in n2
T (n)  c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6  t j  1  c7  t j  1  c8 (n  1)
n
n
n
j 2
j 2
j 2
17
Comparisons and Exchanges in
Insertion Sort
INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
cost
times
c1
n
c2
n-1
Insert A[ j ] into the sorted sequence A[1 . . j -1] 0
n-1
 n2/2
n-1
i←j-1
comparisons c4
while i > 0 and A[i] > key
c5
do A[i + 1] ← A[i]
c6
i←i–1
A[i + 1] ← key
 n2/2
exchanges c7
c8



n
t
j 2 j
n
j 2
n
j 2
(t j  1)
(t j  1)
n-1
18
Insertion Sort - Summary
• Advantages
– Good running time for “almost sorted” arrays (n)
• Disadvantages
– (n2) running time in worst and average case
–  n2/2 comparisons and exchanges
19
Bubble Sort (Ex. 2-2, page 38)
• Idea:
– Repeatedly pass through the array
– Swaps adjacent elements that are out of order
i
1
2
3
8
4
6
n
9
2
3
1
j
• Easier to implement, but slower than Insertion
sort
20
Example
8
4
6
9
2
3
i=1
8
4
6
9
2
4
6
3
9
1
2
4
4
6
1
9
6
i=1
1
6
1
2
8
4
6
1
i=1
j
1
8
i=1
j
4
4
6
6
9
i=3
3
1
2
3
8
4
6
i=4
2
3
1
2
3
2
3
1
2
3
9
9
2
2
3
3
1
2
3
3
4
4
4
9
j
8
6
i=5
9
3
j
6
j
8
2
j
j
4
9
i=2
j
i=1
8
1
8
j
i=1
8
1
j
i=1
8
1
6
9
j
8
9
i=6
j
8
9
i=7
j
21
Bubble Sort
Alg.: BUBBLESORT(A)
for i  1 to length[A]
do for j  length[A] downto i + 1
do if A[j] < A[j -1]
then exchange A[j]  A[j-1]
i
8
i=1
4
6
9
2
3
1
j
22
Bubble-Sort Running Time
Alg.: BUBBLESORT(A)
for i  1 to length[A] c1
do for j  length[A] downto i + 1 c2
c3
Comparisons:  n2/2 do if A[j] < A[j -1]
Exchanges:  n2/2
then exchange A[j]  A[j-1]
n
n
i 1
i 1
T(n) = c1(n+1) + c2  (n  i  1)  c3  (n  i )  c4
= (n) + (c2 + c2 + c4)
c4
n
 (n  i )
i 1
n
 (n  i )
i 1
2
n
(
n

1)
n
n
2
where  (n  i)  n   i  n 
 
2
2 2
i 1
i 1
i 1
n
n
Thus,T(n) = (n2)
n
23
Selection Sort (Ex. 2.2-2, page 27)
• Idea:
– Find the smallest element in the array
– Exchange it with the element in the first position
– Find the second smallest element and exchange it with
the element in the second position
– Continue until the array is sorted
• Disadvantage:
– Running time depends only slightly on the amount of
order in the file
24
Example
8
4
6
9
2
3
1
1
2
3
4
9
6
8
1
4
6
9
2
3
8
1
2
3
4
6
9
8
1
2
6
9
4
3
8
1
2
3
4
6
8
9
1
2
3
9
4
6
8
1
2
3
4
6
8
9
25
Selection Sort
Alg.: SELECTION-SORT(A)
n ← length[A]
8 4 6
for j ← 1 to n - 1
do smallest ← j
for i ← j + 1 to n
do if A[i] < A[smallest]
then smallest ← i
exchange A[j] ↔ A[smallest]
9
2
3
1
26
Analysis of Selection Sort
Alg.: SELECTION-SORT(A)
times
n ← length[A]
c1
1
for j ← 1 to n - 1
c2
n
c3
n-1
n2/2
do smallest ← j
for i ← j + 1 to n
comparisons
n
cost
c4 nj11 (n  j  1)
do if A[i] < A[smallest]
c5

then smallest ← i
c6

exchanges
exchange A[j] ↔ A[smallest] c7
n 1
n 1
n 1
j 1
n 1
j 1
(n  j )
(n  j )
n-1
n 1
T (n)  c1  c2 n  c3 (n  1)  c4  (n  j  1)  c5   n  j   c6   n  j   c7 (n  1)  (n 2 )
j 1
j 1
j 2
27
Time
Sort
Average
Best
Worst
Space
Remarks
Bubble sort
O(n^2)
O(n^2)
O(n^2)
Constant
Always use a modified
bubble sort
Constant
Even a perfectly sorted
input requires
scanning the entire
array
Constant
In the best case
(already sorted), every
insert requires
constant time
Depends
On arrays, merge sort
requires O(n) space;
on linked lists, merge
sort requires constant
space
Constant
Randomly picking a
pivot value (or
shuffling the array prior
to sorting) can help
avoid worst case
scenarios such as a
perfectly sorted array.
Selection Sort
Insertion Sort
Merge Sort
Quicksort
O(n^2)
O(n^2)
O(n*log(n))
O(n*log(n))
O(n^2)
O(n)
O(n*log(n))
O(n*log(n))
O(n^2)
O(n^2)
O(n*log(n))
O(n^2)
28