Foundations of Algorithms, Fourth Edition
Richard Neapolitan, Kumarss Naimipour
Chapter 2
Divide-and-Conquer
Divide and Conquer
• In this approach a problem is divided into
sub-problems and the same algorithm is
applied to every subproblem ( often this is
done recursively)
• Examples
– Binary Search (review algorithm in book)
– Mergesort (review algorithm in book)
– Quicksort
Figure 2.1 : The steps down by a human when searching with Binary
Search. (Note: x = 18)
Complexity of Binary Search
Since this and many other divide and conquer
algorithms are recursive you will recall that we can
determine their complexity using recurrence
relations.
For Binary Search we have
T(n) = T(n/2) + 1
=[T(n/4)+1]+1 = T(n/22) + 2
=[T(n/8+1]+ 2 = T(n/23)+ 3
…
=T(n/2k)+k
What is
k
T(n/2 )+k
We if we let k get larger until n=2k then
we see that k = log2n. Why?
Consequently the relation becomes
T(n) = T(1) + log2n
T(n) = log2n
Since n/2k is 1 if they are equal and T(1) =1
MergeSort
Recall in this algorithm we divide the array
into two equal parts and sort each half prior
to merging. The recurrence relation is
clearly
T(n) = 2T(n/2) + n
Recall that Merging is O(n) right?
T(n/2)
T(n/4)
O(n)
Figure 2.2: The steps done by a human when sorting with Mergesort.
T(n) = 2T(n/2) + n
T(n) = 2T(n/2) + n = 2[ 2T(n/22) + n/2] + n
= 22T(n/22) + 2n = 2[2T(n/23) + n/22] +2n
=23T(n/23) + 3n
…
=2kT(n/2k) + kn
If n=2k then we have
T(n) = nT(1) + (log2n)n = n+ nlog2n
= O(nlog2n)
QuickSort
Works in situ!
Void quicksort(int low, int high)
{
int pivot;
if (high > low){
partition(low, high, pivot);
quicksort(low, pivot-1);
quicksort(pivot+1,high);
}
Figure 2.3: The steps done by a human when sorting with Quicksort. The
subarrays are enclosed in rectangles whereas the pivot points are free.
Partition
Study this carefully
void partition (int low, int high, int&pivot)
{
int I,j, pivotitem;
pivotitem = S[low]; // select left item (hmmm)
j=low;
for (i=low+1; i<=high; i++)
if (S[i] < pivotitem){
j++;
swap S[i] and S[j];
}
pivot= j;
swap S[low] and S[pivot];
}
There are many ways
to write this function!
All have a complexity
of O(n).
Complexity of Quicksort
The complexity of this algorithm depends on
how good the pivot value selection is . If the
value is always in the middle of array then
the best case complexity is
T(n) = n + 2T(n/2)
Which we already have determined is
T(n) = n log2 n
Worst case for Quicksort
This clearly will occur if each pivot value is
less than (or greater) all the elements of the
array. IE the array is split into 1 and n-1
size pieces. This gives a recurrence relation
of
T(n) = T(1) + T(n-1) + n-1
Time to sort left array right array partition
Worst Case analysis
T(n) = T(1) + T(n-1) + n-1 = T(n-1) + n
Assume the answer is n(n-1)/2
check it out !
n(n-1)/2 = 0 + (n-1)(n-2)/2 + n-1
= (n-1)(n-2)/2 + 2(n-1)/2
=((n-1)(n-2)+ 2(n-1))/2
= (n-1)(n-2+2)/2 = n(n-1)/2 ☺
Quick Sort Analysis
Quicksort’s worst case is θ(n2)
Does this mean that quick sort is just as bad
as say selection sort, insertion sort and/or
bubble sort. No!
Its all about average case performance.
The average case performance for these
three is θ(n2) as well.
What is the average case complexity for QS?
Average Case Analysis
assume prob. pivotpoint is p
𝑛
𝐴 𝑛 =
1
𝐴 𝑝 − 1 + 𝐴(𝑛 − 𝑝) + 𝑛 − 1
𝑛
𝑝=1
𝑛
1
𝐴 𝑛 =
𝑛
𝐴 𝑝 − 1 + 𝐴(𝑛 − 𝑝) + 𝑛 − 1
𝑝=1
2
𝐴 𝑛 =
𝑛
𝑛
𝐴 𝑝−1 +𝑛−1
𝑝=1
See HW 22 p 86 for above conversion
Average case continued
Multiplying by n
𝑛
𝑛𝐴 𝑛 = 2
(𝑛 − 1)𝐴 𝑛 − 1 = 2
𝐴 𝑝−1 +𝑛 𝑛−1
𝑝=1
𝑛−1
𝑝=1 𝐴
𝑝 − 1 + (𝑛 − 1)(𝑛 − 2)
Subtracting these equations we have
𝑛𝐴 𝑛 − 𝑛 − 1 𝐴 𝑛 − 1 = 2𝐴 𝑛 − 1 + 2(𝑛 − 1)
𝐴 𝑛
𝐴 𝑛−1
2 𝑛−1
=
+
𝑛+1
𝑛
𝑛 𝑛+1
Average case QS continued
𝐴 𝑛
Assume 𝑎𝑛 =
𝑛+1
2 𝑛−1
𝑎𝑛 = 𝑎𝑛−1 +
,
𝑛 𝑛+1
we get
𝑎0 = 0
Applying some simple math we have
𝑎𝑛 ≈ 2 ln 𝑛
Which give
𝐴(𝑛) ≈ 𝑛 + 1 lg 𝑛 = 𝑛 + 1 2((ln 2)(lg 𝑛)
≈ 1.38 𝑛 + 1 lg 𝑛 ∈ 𝜃(𝑛 lg 𝑛)
Matrix Multiplication (Strassen)
Lets look at the product of two 2 by 2’s
𝑐11 𝑐12
𝑎11 𝑎12
𝑏11 𝑏12
=
×
𝑐21 𝑐22
𝑎21 𝑎22
𝑏21 𝑏22
Clearly after you do the homework #26
m1=(a11+a22)(b11+b22)
m2=(a11+a22)b11
m3=a11(b12+b22)
m4=a22(b21+b11)
m5=(a11+a12)b22
m6=(a21+a11)(b11+b12)
m7=(a12+a22)(b21+b22)
will give the following!
And the answer is
𝑚1 + 𝑚4 − 𝑚5 + 𝑚7
𝐶=
𝑚2 + 𝑚4
𝑚3 + 𝑚5
𝑚1 + 𝑚3 − 𝑚2 + 𝑚6
Original method
8 mult, four add/sub
Strassen’s method
7 mult, and 18 add/sub
Hmmmm! So what’s the big deal?
Big Matrices
𝐶11
𝐶21
n
2 by
n
2
𝑐12
𝐴11 𝐴12
𝐵11 𝐵12
=
×
𝐵21 𝐵22
𝑐22
𝐴21 𝐴22
Where
C11 is the upper left hand corner of the
matrix of size n/2 by n/2. The others are
similarly defined. Now
m1=(A11+A22)(B11+B22)
Is the sum and product of matrices
Our function is then
void Strassen(int n, A,B, C)// these are nxn mats
{
if (n<= threshold) computer C=AxB normally
Partition A and B into eight submatrices
strassen(n/2, A11+A22, B11+B22, M1);
strassen(n/2, A21+A22, B11, M2)
etc // making 7 recursive calls
}
NOT EIGHT!
Complexity
T(n) = 7T(n/2) + cn2
Which is
T(n) = θ(nlg7) = O(n2.81)
Using the general theorem.
The best know is
Coppersmith and Winograd with a time complexity of
O(n2.376)
Why am I using big O here?
Recalling General Theorem
See page 588
Assume 𝑇 𝑛 = 𝑎𝑇
𝑛
𝑏
+ 𝑐𝑛𝑘 for n>1 and n
a power of b, T(1)=d
𝜃 𝑛𝑘 𝑖𝑓 𝑎 < 𝑏 𝑘
𝑇 𝑛 ∈ 𝜃(𝑛𝑘 lg 𝑛) 𝑖𝑓 𝑎 = 𝑏 𝑘
𝜃 𝑛𝑙𝑜𝑔𝑏 𝑎 𝑖𝑓 𝑎 > 𝑏 𝑘
Just a side note
Suppose we has 8 recursive calls instead of
7 in the above case. Then the recurrence
relation would be
T(n) = 8T(n/2) + cn2
This has a complexity of what?
When not to use divide and conquer
• An instance of size n is divided into two or
more instances each almost of size n.
• An instance of size n is divided into almost
n instances of size n/c, where c is a
constant