Chapter 4 Divide-and-Conquer
1
About this lecture
• Recall the divide-and-conquer paradigm, which
we used for merge sort:
– Divide the problem into a number of sub-problems
that are smaller instances of the same problem.
– Conquer the sub-problems by solving them recursively.
• Base case: If the sub-problems are small enough, just solve
them by brute force.
– Combine the sub-problem solutions to give a solution
to the original problem.
• We look at two more algorithms based on divideand-conquer.
2
About this lecture
• Analyzing divide-and-conquer algorithms
• Introduce some ways of solving recurrences
– Substitution Method (If we know the answer)
– Recursion Tree Method (Very useful !)
– Master Theorem (Save our effort)
3
Maximum-subarray problem
• Input: an array A[1..n] of n numbers
– Assume that some of the numbers are negative,
because this problem is trivial when all numbers
are nonnegative
• Output: a nonempty subarray A[i..j] having the
largest sum S[i, j] = ai + ai+1 +... + aj
A
1
2
13
-3
3
4
5
6
7
8
9
10 11 12 13 14 15 16
-25 20
-3
-16 -23 18
20
-7
12
-5
-22 15
-4
7
maximum subarray
4
A brute-force solution
n
 
2
• Examine all
possible S[i, j]
• Two implementations:
–
–
–
–
compute each S[i, j] in O(n) time  O(n3) time
compute each S[i, j+1] from S[i, j] in O(1) time
(S[i, i] = A[i] and S[i, j+1] = S[i, j] + A[j+1])
 O(n2) time
Ex: i 1 2 3 4 5 6
A[i] 13 -3 -25 20 -3 -16
S[2, 2] =
S[2, 3] =
S[2, 4] =
S[2, 5] =
-3
-28
-8
-11
5
A divide-and-conquer solution
• Possible locations of a maximum subarray A[i..j] of
A[low..high], where mid = (low+high)/2
– entirely in A[low..mid] (low  i  j  mid)
– entirely in A[mid+1..high] (mid < i  j  high)
– crossing the midpoint (low  i  mid < j  high)
crossing the midpoint
low
mid
high
mid +1
entirely in A[low..mid]
entirely in A[mid+1..high]
Possible locations of subarrays of A[low..high]
6
FIND-MAX-CROSSING-SUBARRAY (A, low, mid, high)
left-sum = - // Find a maximum subarray of the form A[i..mid]
sum = 0
for i = mid downto low
sum = sum + A[i ]
if sum > left-sum
left-sum = sum
max-left = i
right-sum = - 
// Find a maximum subarray of the form A[mid + 1 .. j ]
sum =0
for j = mid +1 to high
sum = sum + A[j]
if sum > right-sum
right-sum = sum
max-right = j
// Return the indices and the sum of the two subarrays
Return (max-left, max-right, left-sum + right-sum)
7
A[mid+1..j]
low
i
mid
high
mid +1
j
A[i..mid]
A[i..j] comprises two subarrays A[i..mid] and A[mid+1..j]
8
Example:
A
mid =5
1
2
3
4
5
6
7
8
9
10
13
-3
-25
20
-3
-16
-23
18
20
-7
S[5 .. 5] =
S[4 .. 5] =
S[3 .. 5] =
S[2 .. 5] =
S[1 .. 5] = 2
1
A 13
-3
17  (max-left = 4)
-8
-11
mid =5
2
3
4
5
6
7
8
9
10
-3
-25
20
-3
-16
-23
18
20
-7
S[6 .. 6] =
S[6 .. 7] =
S[6 .. 8] =
S[6 .. 9] =
S[6..10] =
-16
-39
-21
(max-right = 9) 
-1
-8
 maximum subarray crossing mid is S[4..9] = 16
9
FIND-MAXIMUM-SUBARRAY (A, low, high)
if high == low
Return (low, high, A[low])
// base case: only one element
else mid = low high/ 2
(left-low, left-high, left-sum) =
FIND-MAXIMUM-SUBARRAY(A, low, mid)
(right-low, right-high, right-sum) =
FIND-MAXIMUM-SUBARRAY(A, mid + 1, high)
(cross-low, cross-high, cross-sum) =
FIND-MAX-CROSSING-SUBARRAY(A, low, mid, high)
if left-sum ≧ right-sum and left-sum ≧ cross-sum
return (left-low, left-high, left-sum)
elseif right-sum ≧ left-sum and right-sum ≧ cross-sum
return (right-low, right-high, right-sum)
else return (cross-low, cross-high, cross-sum)
Initial call: FIND-MAXIMUM-SUBARRAY (A, 1, n)
10
Analyzing time complexity
• FIND-MAX-CROSSING-SUBARRAY : (n),
where n = high  low + 1
• FIND-MAXIMUM-SUBARRAY
T(n) = 2T(n/2) + (n) (with T(1) = (1))
= (nlg n) (similar to merge-sort)
11
Matrix multiplication
• Input: two n  n matrices A and B
n
• Output: C = AB,
cij   aikbkj .
k 1
An O(n3) time naive algorithm
SQUARE-MATRIX-MULTIPLY(A, B)
n  A.rows
let C be an n  n matrix
for i  1 to n
for
j  1 to n
cij  0
for k  1 to n
cij  cij + aikbkj
return C
12
Divide-and-Conquer Algorithm
• Assume that n is an exact power of 2
 A11 A12 
 B11B12 
 C11C12 
, B  
, C  

A  
 A21 A22 
 B21B22 
 C21C22 
(4.1)
 C11C12   A11 A12   B11B12 

  
  

 C21C22   A21 A22   B21B22 
13
Divide-and-Conquer Algorithm
C11  A11B11  A12 B21
C12  A11B12  A12 B22
C21  A21B11  A22 B21
C22  A21B12  A22 B22
A straightforward divide-and-conquer algorithm
T(n) = 8T(n/2) + (n2)
= (n3)
Computing A+B  O(n2)
14
Strassen’s method
S1  B12  B22 , S2  A11  A12 , S3  A21  A22
S4  B21  B11 , S5  A11  A22 , S6  B11  B22
S7  A12  A22 , S8  B21  B22 , S9  A11  A21
S10  B11  B12
(4.2)
15
Strassen’s method
P1  A11S1
P2  S2 B22
P3  S3 B11
P4  A22 S4
C11  P5  P4  P2  P6
C12  P1  P2
(4.4)
P5  S5 S6
C21  P3  P4
P6  S7 S8
C22  P5  P1  P3  P7
P7  S9 S10
(4.3)
16
Strassen’s divide-and-conquer
algorithm
• Step 1: Divide each of A, B, and C into four
sub-matrices as in (4.1)
• Step 2: Create 10 matrices S1, S2, …, S10 as in
(4.2)
• Steep 3: Recursively, compute P1, P2, …, P7 as
in (4.3)
• Step 4: Compute C11 , C12 , C21 , C22 according to
(4.4)
17
Time complexity
T(n) = 7T(n/2) + (n2)
= (nlg 7 ) (why?)
= (n2.81)
18
Discussion
• Strassen’s method is largely of theoretical
interest for n  45
• Strassen’s method is based on the fact that we
can multiply two 2  2 matrices using only 7
multiplications (instead of 8).
• It was shown that it is impossible to multiply
two 2  2 matrices using less than 7
multiplications.
19
Discussion
• We can improve Strassen’s algorithm by finding an
efficient way to multiply two k  k matrices using a
smaller number q of multiplications, where k > 2. The
time is T(n) = qT(n/k) + θ(n2).
• A trivial lower bound for matrix multiplication is
(n2). The current best upper bound known is
O(n2.376).
• Open problems:
– Can the upper bound O(n2.376) be improved?
– Can the lower bound (n2) be improved?
20
Substitution Method
(if we know the answer)
How to solve this?
T(n) = 2T( n / 2 ) + n,
1. Make a guess
e.g., T(n) = O(n log n)
with T(1) = 1
2. Show it by induction
•
e.g., to show upper bound, we find constants c and
n0 such that T(n)  c f(n) for n = n0, n0+1, n0+2, …
21
Substitution Method
(if we know the answer)
How to solve this?
T(n) = 2T( n / 2 ) + n,
1. Make a guess
e.g., T(n) = O(n log n)
2. Show it by induction
•
with T(1) = 1
Firstly, T(2) = 4, T(3) = 5.
 We want to have T(n)  cn lg n
 Let c = 2  T(2) and T(3) okay
•
Other Cases ?
22
Substitution Method
(if we know the answer)
•
Induction Case:
Assume the guess is true for all n = 2,3,…,k
For n = k+1, we have:
T(n) = 2T(n / 2 ) + n
 2cn / 2 lgn / 2  n
 cn lg n / 2  n
Induction
case is true
= cn lg n – cn + n  cn log n
23
Substitution Method
(if we know the answer)
Q. How did we know the value of c and n0 ?
A. If induction works, the induction case must be
correct  c ≥ 1
Then, we find that by setting c = 2, our guess is
correct as soon as n0 = 2
Alternatively, we can also use c = 1.5 Then, we
just need a larger n0 = 4
(What will be the new base cases? Why?)
24
Substitution Method
(New Challenge)
How to solve this?
T (n)  T ( n / 2)  T ( n / 2)  1, T (1)  1
1. Make a guess (T(n) = O(n)), and
2. Show T(n) ≤ cn by induction
– What will happen in induction case?
25
Substitution Method
(New Challenge)
Induction Case:
(assume guess is true for some base cases)
T (n )  T ( n / 2)  T ( n / 2)  1
 c n / 2  c n / 2  1
 cn  1
This term is
not what we
want …
26
Substitution Method
(New Challenge)
•
The 1st attempt was not working because our
guess for T(n) was a bit “loose”
Recall: Induction may become easier if we
prove a “stronger” statement
2nd Attempt: Refine our statement
Try to show T(n) ≤ cn - b instead
27
Substitution Method
(New Challenge)
Induction Case:
T (n)  T ( n / 2)  T ( n / 2)  1
 c n / 2  b  c n / 2  b  1
 cn  b
We get the
desired term
(when b  1)
It remains to find c and n0, and prove the base
case(s), which is relatively easy
28
Substitution Method
(New Challenge 2)
How to solve this?
T(n) = 2T( n ) + lg n ?
Hint: Change variable: Set m = lg n
29
Substitution Method
(New Challenge 2)
Set m = lg n , we get
T(2m) = 2T(2m/2) + m
Next, set S(m) = T(2m) = T(n)
S(m) = 2S(m/2) + m
We solve S(m) = O(m lg m)

T(n) = O(lg n lg lg n)
30
Recursion Tree Method
( Nothing Special… Very Useful ! )
How to solve this?
T(n) = 2T(n/2) + n2,
with T(1) = 1
31
Recursion Tree Method
( Nothing Special… Very Useful ! )
Expanding the terms, we get:
T(n) = n2 + 2T(n/2)
= n2 + 2n2/4 + 4T(n/4)
= n2 + 2n2/4 + 4n2/16 + 8T(n/8)
=...
=lg n 1 (1 / 2)k n 2  2lg n T (1)
k 0
= (n2) + (n) = (n2)
32
Recursion Tree Method
( Recursion Tree View )
We can express the previous recurrence by:
33
Further expressing gives us:
This term is
from T(n/2)
34
Recursion Tree Method
( New Challenge )
How to solve this?
T(n) = T(n/3) + T(2n/3) + n, with T(1) = 1
What will be the recursion tree view?
35
The corresponding recursion tree view is:
36
Master Method
( Save our effort )
When the recurrence is in a special form, we can
apply the Master Theorem to solve the
recurrence immediately
The Master Theorem has 3 cases …
37
Master Theorem
Let
T(n) = aT(n/b) + f(n)
with a  1 and b  1 are constants, where
we interpret n/b to mean either n/b or
n/b.
Theorem: (Case 1)
If f(n) = O(nlogb a - e) for some constant e  0
then T(n) = (nlogb a)
38
Theorem: (Case 2)
If f(n) = (nlogb a),
then T(n) = (nlogb a lg n)
Theorem: (Case 3)
If f(n) = (nlogb a + e) for some constant e  0,
and if af(n/b)  c f(n) for some constant c  1,
and all sufficiently large n,
then T(n) =
(f(n))
39
Master Theorem
1. Solve T(n) = 9T(n/3) + n (case 1)
2. Solve T(n) = T(2n/3) + 1 (case 2)
3. Solve T(n) = 3T(n/4) + nlgn (case 3)
4. How about this?
T(n) = 2T(n/2) + n lg n ?
5. T(n) = 8T(n/2) + n2 , T(n) = 8T(n/2) + n
6. T(n) = 7T(n/2) + n2 , T(n) = 7T(n/2) + 1
40
Homework
•
•
•
•
Exercise 4.1-3 (due Oct. 12)
Practice at home: 4.1-5, 4.2-1,4.2-5
Exercise 4.5-4 Problem 4.1(b,d,f)(due Oct. 19)
Practice at home: 4.3-7,4.4-8,4.3-6,4.4-7,4.5-1
41