8.1 Recurrence Relations
Definition:
A recurrence relation for the sequence {an} is an equation that expresses an in terms of
one or more previous terms of the sequence a0, a1, ..., an-1. A sequence is called a
solution of a recurrence relation if its terms satisfy the recurrence relation.
The recurrence relation together with the initial conditions uniquely determines a
sequence, i.e., a solution.
1
e.g. Consider the recurrence relation (Fibonacci numbers):
an = an-1 + an-2 , n ≥ 2
where a0 = 0 and a1 = 1.
Then {0,1,1,2,3,5,8,…} is the solution.
Question: Is it possible to find an explicit formula for an?
In some conditions yes!
2
8.2 Solving Recurrence Relations
Definition:
A linear homogeneous recurrence relation of degree k with constant coefficients is a
recurrence relation of the form
an = c1an-1 + c2an-2 + ××× + ckan-k where c1, c2, ..., ck are real numbers and ck ¹ 0.
3
8.2 Solving Recurrence Relations
Definition:
A linear homogeneous recurrence relation of degree k with constant coefficients is a
recurrence relation of the form
an = c1an-1 + c2an-2 + ××× + ckan-k where c1, c2, ..., ck are real numbers and ck ¹ 0.
Basic approach is to look for a solution of the form an = rn.
Þ rn = c1rn-1 + c2rn-2 + ××× + ckrn-k
4
8.2 Solving Recurrence Relations
Definition:
A linear homogeneous recurrence relation of degree k with constant coefficients is a
recurrence relation of the form
an = c1an-1 + c2an-2 + ××× + ckan-k where c1, c2, ..., ck are real numbers and ck ¹ 0.
Basic approach is to look for a solution of the form an = rn.
Þ rn = c1rn-1 + c2rn-2 + ××× + ckrn-k
Divide both sides by rn-k
Þ rk - c1rk-1 - c2rk-2 - ××× - ck-1r - ck = 0
5
8.2 Solving Recurrence Relations
Definition:
A linear homogeneous recurrence relation of degree k with constant coefficients is a
recurrence relation of the form
an = c1an-1 + c2an-2 + ××× + ckan-k where c1, c2, ..., ck are real numbers and ck ¹ 0.
Basic approach is to look for a solution of the form an = rn.
Þ rn = c1rn-1 + c2rn-2 + ××× + ckrn-k
Divide both sides by rn-k
Þ rk - c1rk-1 - c2rk-2 - ××× - ck-1r - ck = 0: Characteristic equation
Þ If r is a solution of the characteristic equation, then{an} with an= rn, is a solution of
the recurrence relation.
6
Theorem:
Consider the recurrence relation an = c1an-1 + c2 an-2 . Suppose that r2 - c1r - c2 has two
distinct roots r1 and r2. Then the solution is given by an = a1r1n + a2r2n, where a1 and a2
are real constants.
Proof: See page 462 in the textbook 6th edition (page 499 in the textbook 7th edition).
7
e.g. Solve ƒn = ƒn-1 + ƒn-2, where ƒ0 = 0 and ƒ1 = 1, i.e., find an explicit formula for
Fibonacci numbers.
The roots of the characteristic equation r2- r - 1= 0 are
r1 = (1+ 5 )/2 and r2 = (1- 5 )/2.
8
e.g. Solve ƒn = ƒn-1 + ƒn-2, where ƒ0 = 0 and ƒ1 = 1, i.e., find an explicit formula for
Fibonacci numbers.
The roots of the characteristic equation r2- r - 1= 0 are
r1 = (1+ 5 )/2 and r2 = (1- 5 )/2.
By the theorem, ƒn = a1r1n + a2r2n for some a1 and a2.
9
e.g. Solve ƒn = ƒn-1 + ƒn-2, where ƒ0 = 0 and ƒ1 = 1, i.e., find an explicit formula for
Fibonacci numbers.
The roots of the characteristic equation r2- r - 1= 0 are
r1 = (1+ 5 )/2 and r2 = (1- 5 )/2.
By the theorem, ƒn = a1r1n + a2r2n for some a1 and a2.
Using the initial conditions ƒ0 = 0 and ƒ1 = 1,
ƒ0 = a 1 + a 2 = 0
ƒ1 = a1 (1+ 5 )/2 + a2 (1- 5 )/2 = 1
Þ a1=1/ 5 and a2 = -1/ 5
n
Þ
n
1 æ1+ 5 ö
1 æ1- 5 ö
ç
÷ ç
÷
fn =
ç
÷
ç
5è 2 ø
5 è 2 ÷ø .
10
Theorem:
Consider the recurrence relation an = c1an-1 + c2an-2 . Suppose that r2 - c1r - c2 has only
one root r0 with multiplicity 2. Then the solution is given by an = a1r0n + a2nr0n, where
a1and a2 are real constants.
11
Theorem:
Consider the recurrence relation an = c1an-1 + c2an-2 . Suppose that r2 - c1r - c2 has only
one root r0 with multiplicity 2. Then the solution is given by an = a1r0n + a2nr0n, where
a1and a2 are real constants.
e.g. Solve the recurrence relation an = 6an-1 - 9an-2 with a0 = 1 and a1 = 6.
Characteristic equation: r2- 6r +9 = 0, which is (r-3)2 = 0 Þ r0 = 3
Solution is then given by an = a1r0n + a2nr0n
a0 = a1r00 + a20r00 = a130 = a1 = 1
a1 = a1r01 + a21r01 = a131 + a2131 = 3a1 + 3a2 = 6
Þ a2 = 1
an = a1r0n + a2nr0n
an = 3n + n3n
12
8.4 Generating Functions
Using Generating Functions to solve recurrence relations
n -1
Example: Solve an = 8an-1 + 10 with initial condition a0 = 1.
(an corresponds to the number of valid code words of length n, supposing that a valid
code word is an n-digit number in decimal notation containing an even number of 0s. )
Can’t use characteristic equation in this case because the recurrence relation is not
linear.
13
8.4 Generating Functions
Using Generating Functions to solve recurrence relations
n -1
Example: Solve an = 8an-1 + 10 with initial condition a0 = 1.
(an corresponds to the number of valid code words of length n, supposing that a valid
code word is an n-digit number in decimal notation containing an even number of 0s. )
Can’t use characteristic equation in this case because the recurrence relation is not
linear.
One possible way is to use generating functions:
Definition:
The generating function for the sequence a0 , a1 ,..., a k ,... of real numbers is the infinite
power series
¥
G ( x) = a0 + a1 x + ... + ak x + ... = å ak x k
k
k =0
14
n -1
a
=
8
a
+
10
Example: Solve n
with initial condition a0 = 1.
n -1
(an corresponds to the number of valid code words of length n, supposing that a valid
code word is an n-digit number in decimal notation containing an even number of 0s. )
Solution:
∞
n
G(x)
=
a
x
∑
a
n
Let G(x) be the generating function for the sequence { n }, that is,
n=0
15
n -1
a
=
8
a
+
10
Example: Solve n
with initial condition a0 = 1.
n -1
(an corresponds to the number of valid code words of length n, supposing that a valid
code word is an n-digit number in decimal notation containing an even number of 0s. )
Solution:
∞
n
G(x)
=
a
x
∑
a
n
Let G(x) be the generating function for the sequence { n }, that is,
n=0
∞
∞
∞
⇒ G(x) = 1+ ∑ an x = 1+ ∑ (8an−1 x +10
n
n=1
n
n−1
n
x ) = 1+ 8x∑ an−1 x
n=1
n=1
16
n−1
∞
+ x∑10n−1 x n−1
n=1
n -1
a
=
8
a
+
10
Example: Solve n
with initial condition a0 = 1.
n -1
(an corresponds to the number of valid code words of length n, supposing that a valid
code word is an n-digit number in decimal notation containing an even number of 0s. )
Solution:
∞
n
G(x)
=
a
x
∑
a
n
Let G(x) be the generating function for the sequence { n }, that is,
n=0
∞
∞
∞
⇒ G(x) = 1+ ∑ an x = 1+ ∑ (8an−1 x +10
n
n
n=1
∞
n
x ) = 1+ 8x∑ an−1 x
n=1
n=1
n−1
∞
+ x∑10n−1 x n−1
n=1
∞
= 1+ 8x∑ an x + x∑10n x n = 1+ 8x ⋅ G(x) + x (1−10x)
n=0
n
n−1
n=0
1- 9x
1æ 1 ö 1æ 1 ö
Þ G ( x) =
= ç
÷+ ç
÷
(1 - 8 x)(1 - 10 x) 2 è 1 - 8 x ø 2 è 1 - 10 x ø
1æ ¥ n n ¥ n nö ¥ 1 n
= ç å 8 x + å10 x ÷ = å (8 + 10 n )x n
2 è n =0
n =0
ø n =0 2
1
Þ an = (8 n + 10 n )
2
17
since by geometric series:
¥
1/(1 - rx) = å r k x k
k =0
Example: Solve ak = 3ak -1 with initial condition a0 = 2.
(You could solve this recurrence relation by using its characteristic equation, but we’ll
try using generating functions.)
18
Example: Solve ak = 3ak -1 with initial condition a0 = 2.
(You could solve this recurrence relation by using its characteristic equation, but we’ll
try using generating functions.)
¥
k
Let G(x) be the generating function for the sequence { ak }, that is, G ( x) = å ak x .
k =0
¥
¥
G ( x) = å ak x = 2 + å 3ak -1 x k (by the recurrence relation ak = 3ak -1 )
k
k =0
k =1
¥
= 2 + 3 x å ak -1 x k -1 = 2 + 3 x × G ( x)
k =1
Þ G ( x) - 3 x × G ( x) = (1 - 3 x) × G ( x) = 2
Þ G ( x) = 2 /(1 - 3 x)
¥
1/(1 - rx) = å r k x k ,
Using the identity,
k =0
¥
¥
k =0
k =0
G ( x) = 2 × å 3k x k = å 2 × 3k x k \ ak = 2 × 3k
19
8.3 Divide-and-Conquer Recurrence Relations
Many algorithms divide a problem into one or more smaller problems.
Recall binary search: (from Chapter 3)
Search x in the list a0, a1, …, an-1 where a0 < a1 < … < an-1.
1. Compare x with the middle term of the sequence, am, where m = ë(n–1) / 2û.
2. If x > am, search x on the second half {am+1, am+2, … an}
else search x on the first half {a1, a2, … am}
3. Repeat the first two steps until a list with one single term is obtained.
4. Determine whether this one term is x or not.
Reduces the search in a sequence of size n to a search in a sequence of size n/2,
assuming n is even.
20
8.3 Divide-and-Conquer Recurrence Relations
Many algorithms divide a problem into one or more smaller problems.
Recall binary search: (from Chapter 3)
Search x in the list a0, a1, …, an-1 where a0 < a1 < … < an-1.
1. Compare x with the middle term of the sequence, am, where m = ë(n–1) / 2û.
2. If x > am, search x on the second half {am+1, am+2, … an}
else search x on the first half {a1, a2, … am}
3. Repeat the first two steps until a list with one single term is obtained.
4. Determine whether this one term is x or not.
Reduces the search in a sequence of size n to a search in a sequence of size n/2,
assuming n is even.
Two comparisons are needed at each iteration (one to determine which half of the list
to use and the other to determine whether any terms of the list remain).
Þ f (n) = f (n / 2) + 2, f (1) = 2
f (n): # of operations (comparisons) for a search sequence of size n.
21
Definition:
The recurrence relation
f ( n) = a × f ( n / b) + g ( n)
is called a divide-and-conquer recurrence relation.
Sometimes we aren’t really interested in solving a given recurrence relation, but we
rather want to find the complexity of the function.
22
e.g. The problem of finding the maximum element of a sequence, a1 , a2 ,..., an , can be
solved using a divide-and-conquer algorithm:
• If n = 1 then a1 is the maximum.
• If n > 1, split the sequence into two sequences. The overall maximum is
maximum of the maximum elements of these two sub-sequences. The problem is
hence reduced to finding the maximum of each of the two smaller sequences.
23
e.g. The problem of finding the maximum element of a sequence, a1 , a2 ,..., an , can be
solved using a divide-and-conquer algorithm:
• If n = 1 then a1 is the maximum.
• If n > 1, split the sequence into two sequences. The overall maximum is
maximum of the maximum elements of these two sub-sequences. The problem is
hence reduced to finding the maximum of each of the two smaller sequences.
int MAX(int a[], int i, int j) {
if (i == j)
p = a[i];
else{
max1 = MAX(a, i,(i+j)/2));
max2 = MAX(a,((i+j)/2)+1,j);
if (max1 > max2) p = max1;
else p = max2;
}
return p;
}
// initially i=0,j=n-1
24
Complexity analysis:
Let f (n) be the number of comparisons to find the maximum of the sequence with n
elements. Using two comparisons (one to compare the current maxima, and one to
determine whether any terms of the list remain at each iteration),
f (n) = 2 f (n / 2) + 2, f (1) = 1, where n is even.
int MAX(int a[], int i, int j) {
if (i == j)
p = a[i];
else{
max1 = MAX(a, i,(i+j)/2));
max2 = MAX(a,((i+j)/2)+1,j);
if (max1 > max2) p = max1;
else p = max2;
}
return p;
}
// initially i=0,j=n-1
25
Theorem: Let f be an increasing function that satisfies the recurrence relation:
f (n) = a ⋅ f (n / b) + c
where n is divisible by b, a ≥ 1, b is an integer greater than 1 and c is a positive real
number. Then
f(n) =
{
0 (n logb a )
0 (log n)
if a > 1
if a = 1
26
Theorem: Let f be an increasing function that satisfies the recurrence relation:
f (n) = a ⋅ f (n / b) + c
where n is divisible by b, a ≥ 1, b is an integer greater than 1 and c is a positive real
number. Then
f(n) =
{
0 (n logb a )
0 (log n)
if a > 1
if a = 1
e.g. Finding the maximum with f (n) = 2 × f (n / 2) + 2 , n is even
log 2 2
) = O ( n) .
By the theorem f (n) is O(n
27
Theorem: Let f be an increasing function that satisfies the recurrence relation:
f (n) = a ⋅ f (n / b) + c
where n is divisible by b, a ≥ 1, b is an integer greater than 1 and c is a positive real
number. Then
f(n) =
{
0 (n logb a )
0 (log n)
if a > 1
if a = 1
e.g. Finding the maximum with f (n) = 2 × f (n / 2) + 2 , n is even
log 2 2
) = O ( n) .
By the theorem f (n) is O(n
e.g. Binary search with f (n) = f (n / 2) + 2 , n is even
From the theorem, f (n) is O(log n).
28
k
Proof: Let n = b
Þ f ( n) = a × f ( n / b) + c
= a 2 × f (n / b 2 ) + ac + c
!
k -1
= a × f (n / b ) + å a j c
k
k
j =0
k -1
Þ f (n) = a × f (1) + c å a j
k
j =0
Case i:
Let a =1, then f (n) = f (1) + ck = f (1) + c × log b n \
f (n) is 0 (log n)
When n is not a power of b, we have b k < n < b k +1 for some k.
Since f is an increasing function,
f (n) £ f (b k +1 ) = f (1) + c(k + 1) = f (1) + c + ck
£ f (1) + c + c × log b n
\ f (n) is 0 (log n)
29
in both cases.
Case ii:
k
Let a >1 and n = b
f (n) = a k × f (1) + c × (a k - 1) /(a - 1) (from geometric series, see Section 2.4)
= a k [ f (1) + c /(a - 1) ] - c /(a - 1)
= C1 × n logb a + C 2
(Note that in general n log a = a log n )
where C1 = f (1) + c /(a - 1) and C2 = -c /(a - 1) .
\ f (n) is 0 (n logb a )
Suppose that n ¹ b k , then b k < n < b k +1
Þ
f (n) £ f (b k +1 ) = C1a k +1 + C 2
£ (C1a)a logb n + C 2 since k £ log b n < k + 1
£ (C1a)n logb a + C 2
log b a
).
Hence f(n) is 0 (n
30
Master Theorem: Let f (n) be an increasing function that satisfies
f (n) = a × f (n / b) + cn d
where n = b k , k is a positive integer, a ≥ 1, b is an integer greater than 1, c and d are
positive real numbers. Then
f(n)=
{
0 (n d )
if a < b d
0 (n d log n) if a = b d
0 (n log b a )
if a > b d
See the textbook for the proof.
31
Master Theorem: Let f (n) be an increasing function that satisfies
f (n) = a × f (n / b) + cn d
where n = b k , k is a positive integer, a ≥ 1, b is an integer greater than 1, c and d are
positive real numbers. Then
f(n)=
{
0 (n d )
if a < b d
0 (n d log n) if a = b d
0 (n log b a )
if a > b d
See the textbook for the proof.
e.g., Merge Sort
f(n) = 2 f (n/2) + n, which is O(n log n).
32