Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Algorithms – I (CS29003/203)
Autumn 2022, IIT Kharagpur
Divide and Conquer, Recursion
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Divide and Conquer
• General techniques
• Merge sort
• Recursion Tree Method
• Master Theorem
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and COnquer
2
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Divide and Conquer
• Divide and conquer involves three steps
• Divide a problem into pieces (smaller instances of same problem)
•
E.g., divide into halves
• Conquer the subproblems by solving them recursively
•
Can just call the same algorithm on the subproblems (recursively solving)
•
Base case: when n=1 (or is small)
• Combine solutions to pieces to get answer to original problem
•
Combining results from recursive calls.
•
Usually the hardest part in the algorithm design
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
3
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort
8 3 2 9 5 1 7 4
8 3 2 9
8 3
8
5 1
2 9
3
3 8
5 1 7 4
2
9
2 9
2 3 8 9
7 4
1
5
1 5
7
4
4 7
1 4 5 7
1 2 3 4 5 7 8 9
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
4
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort
• Dividing is easy. The key operation, though, is merge.
• How to do merging of two sorted arrays to produce a merged
sorted array?
2 5 9
Aug 10,11,12,17, 2022
3 4 8
CS21003/CS21203 / Algorithms - I | Divide and Conquer
5
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort Pseudocode
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
6
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort – In What Order?
8 3 2 9 5 1 7 4
8 3 2 9
8 3
8
5 1
2 9
3
3 8
5 1 7 4
2
9
2 9
2 3 8 9
7 4
1
5
1 5
7
4
4 7
1 4 5 7
1 2 3 4 5 7 8 9
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
7
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort – Runtime Analysis
Total number of elements in two arrays is 𝑛1 + 𝑛2
Θ 1 [Constant time]
Θ 𝑛1
Θ 𝑛2
Θ 1
Overall runtime of “MERGE”
subroutine is Θ(total number
of elements after merging)
Θ 𝑛1 + 𝑛2
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
8
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort – Runtime Analysis
𝑝
Θ 1
Θ 1
𝑝
𝑛
𝑛ൗ
2
𝑟
𝑟
𝑝
𝑛ൗ
2
𝑟
2 ∗ 𝑛ൗ2
4 ∗ 𝑛ൗ4
𝑝 𝑛ൗ4 𝑟 𝑝 𝑛ൗ4 𝑟
𝑝 𝑛ൗ4 𝑟 𝑝 𝑛ൗ4 𝑟
𝑝 = 𝑟 𝑝 = 𝑟 𝑝 = 𝑟𝑝 = 𝑟
𝑝 = 𝑟𝑝 = 𝑟 𝑝 = 𝑟𝑝 = 𝑟
Θ 𝑛
8 ∗ 𝑛ൗ8
𝑛ൗ 𝑛ൗ
4
4
𝑛ൗ 𝑛ൗ
4
4
𝑛ൗ 𝑛ൗ
4
4
𝑛ൗ 𝑛ൗ
4
4
• How do we count the recursive calls?
• Recursive calls belong to the next level
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
9
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Merge Sort – Runtime Analysis
• What are the number of levels in mergesort?
• log 𝑛
• There are log 𝑛 levels and at each level the number of operations is
Θ(𝑛)
• The runtime of mergesort is Θ(𝑛 log 𝑛)
• Next we will learn how to write an equation that represents the
running time of a divide and conquer algorithm.
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
10
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Recurrence Relation
• Let 𝑇 𝑛 be the running time of the
algorithm when the input size is 𝑛
• Base case takes constant time
Base case
Divide time = const
𝑐
• 𝑇 𝑛 = ൝𝑑 + Θ 𝑛 + 2𝑇
𝑛
2
𝑛=1
𝑛>1
𝑛
Conquer time = 𝑇(2 ) each
Combine time = Θ(𝑛)
𝑐
• 𝑇 𝑛 = ൝2𝑇
𝑛
2
+Θ 𝑛
𝑛=1
𝑛>1
• Recurrence relation: Represents the
running time of a recursive algorithm
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
11
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑐
• 𝑇 𝑛 = ൝2𝑇
𝑛
2
+Θ 𝑛
𝑛=1
𝑛>1
• We can write Θ 𝑛 as some const.*𝑛
• Does the base case need to be always for 𝑛 = 1?
• General form:
• 𝑇 𝑛 = ቐ
𝑑
Aug 10,11,12,17, 2022
2𝑇
𝑛 = 𝑛0
𝑛
2
+ 𝑐𝑛
𝑛 > 𝑛0
CS21003/CS21203 / Algorithms - I | Divide and Conquer
12
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
•
•
•
•
𝑑
𝑛=1
𝑛
𝑇 𝑛 = ቐ
2𝑇
+ 𝑐𝑛 𝑛 > 1
2
Left hand side is 𝑇(𝑛). This is a node
It expands to 2, 𝑇(𝑛Τ2) nodes and one 𝑐𝑛 node
The convention is to replace 𝑇(𝑛) with these nodes
And continue
𝑐𝑛
𝑇(𝑛)
𝑇(𝑛)
𝑇(𝑛ൗ2)
𝑇(𝑛ൗ4)
Aug 10,11,12,17, 2022
𝑇(𝑛ൗ2)
𝑇(𝑛ൗ4)
𝑐𝑛ൗ
2
𝑛ൗ )
𝑇(𝑐𝑛
2
𝑇(𝑛ൗ4)
𝑇(𝑛ൗ4)
CS21003/CS21203 / Algorithms - I | Divide and Conquer
𝑐𝑛ൗ
2
13
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑑
𝑛=1
𝑛
𝑇 𝑛 = ቐ
2𝑇
+ 𝑐𝑛 𝑛 > 1
2
•
•
•
•
Left hand side is 𝑇(𝑛). This is a node
It expands to 2, 𝑇(𝑛Τ2) nodes and one 𝑐𝑛 node
The convention is to replace 𝑇(𝑛) with these nodes
And continue
𝑐𝑛
The tree ends
when the base
case is reached
𝑇(𝑛)
𝑐𝑛ൗ
2
𝑇(𝑛ൗ4)
How many 𝑑 ?
Base
Aug 10,11,12,17, 2022
𝑇(𝑛ൗ4) 𝑇(𝑛ൗ4)
𝑐𝑛ൗ
2
𝑇(𝑛ൗ4)
𝑑 𝑑 𝑑
CS21003/CS21203 / Algorithms - I | Divide and Conquer
𝑑
14
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑐𝑛
𝑐𝑛ൗ
2
𝑐𝑛ൗ
2
𝑑
𝑛=1
𝑇 𝑛 = ቐ2𝑇 𝑛 + 𝑐𝑛 𝑛 > 1
2
𝑐𝑛ൗ
4
Base
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑑 𝑑 𝑑
𝑑
• Total runtime comes from counting number of operations from the
internal nodes and counting them from base nodes
• Lets get introduced to trees and some related terminologies as we need
• More will come later
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
15
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
• Graphs are sets of nodes and edges
• Trees are a form of a graph with some specialties
• Trees have direction (parent/child relationships) and don't contain
cycles
• Some terminologies
Root
A
D
Internal
C
B
E
F
G
Leaf
• In a Binary Tree, a node can have at most two children
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
16
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Some Terminologies
Parent
A
Right child
Left child
B
D
C
Siblings
G
F
E
Root
A
Right subtree
Left subtree
C
B
D
Aug 10,11,12,17, 2022
E
F
G
CS21003/CS21203 / Algorithms - I | Divide and Conquer
17
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Some Terminologies
A
C
B
D
D
•
•
•
•
•
E
D
G
F
E
E
D
E
D
E
Level 0
Max # of
Nodes
1 = 20
Level 1
2 = 21
Level 2
4 = 22
Level 3
8 = 23
Number of nodes at level 𝑖 = 2𝑖
Number of leaves 𝐿 = Number of nodes at level 𝐻 𝑖𝑠 2𝐻 [𝐻 is height]
Naturally height 𝐻 = log 2 𝐿
Note: Number of Levels = Height (H) + 1
𝑖
𝐻+1
Total number of nodes, 𝑁 = σ𝐻
−1
𝑖=0 2 = 2
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
18
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Some Terminologies
A
C
B
D
D
E
D
E
D
For general `k’-array Tree
G
F
E
E
D
E
• Number of nodes at level 𝑖 = 𝑘 𝑖
• Number of leaves 𝐿 = Number of nodes at level 𝐻 = 𝑘 𝐻 [𝐻 is height]
• Naturally height 𝐻 = log 𝑘 𝐿
𝑖
• Total number of nodes, 𝑁 = σ𝐻
𝑖=0 𝑘 =
Aug 10,11,12,17, 2022
𝑘 𝐻+1 −1
𝑘−1
CS21003/CS21203 / Algorithms - I | Divide and Conquer
19
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
Level 0
𝑐𝑛
𝑐𝑛ൗ
2
𝑐𝑛ൗ
2
𝑑
𝑛=1
𝑇 𝑛 = ቐ2𝑇 𝑛 + 𝑐𝑛 𝑛 > 1
2
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
Level 1
𝑐𝑛ൗ
4
Level 𝐻 − 1
Base
𝑑 𝑑 𝑑
𝑑
Level 𝐻
• To get the total runtime, first we need to get the height of the tree
𝐻 = log 2 𝑛
𝐻
log
• Number of leaves 𝐿 = 2 = 2 2 𝑛 = 𝑛
• Base cost = 𝑑 × 𝐿 = 𝑑𝑛 = Θ(𝑛)
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
20
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑐𝑛
Level 0
𝑐𝑛
2 ∗ 𝑐𝑛ൗ2 = 𝑐𝑛
𝑐𝑛ൗ
2
𝑐𝑛ൗ
2
4 ∗ 𝑐𝑛ൗ4 = 𝑐𝑛
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
Level 1
𝑐𝑛ൗ
4
Level 𝐻 − 1
Base
•
•
•
•
•
𝑑 𝑑 𝑑
𝑑
Level 𝐻
What about the internal costs?
We need to sum the costs at each internal node
Since the costs at each level is same we can multiply by height
Internal cost = 𝑐𝑛 ∗ 𝐻 = 𝑐𝑛 log 2 𝑛 = Θ (𝑛 log2 𝑛)
Total cost =Θ 𝑛 + 𝜃 𝑛 log2 𝑛 = 𝜃 𝑛 log2 𝑛
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
21
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
• Lets take another example
𝑑
𝑛=1
𝑛
𝑇 𝑛 = ቐ
4𝑇
+ 𝑐𝑛 𝑛 > 1
2
• Does it make sense?
• The general form of the recurrence can be
T n = 𝑎𝑇
•
•
•
𝑛
+ 𝑓(𝑛)
𝑏
Where, 𝑎 is number of subproblems (branching factor)
𝑏 is the size of each subproblem (division ratio)
𝑓(𝑛) total time for divide and combine operations
• What are these values for binary search?
T n =𝑇
• with base case time - constant
Aug 10,11,12,17, 2022
𝑛
+𝑐
2
CS21003/CS21203 / Algorithms - I | Divide and Conquer
22
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑑
𝑛=1
𝑇 𝑛 = ቐ4𝑇 𝑛 + 𝑐𝑛 𝑛 > 1
2
c𝑛
𝑐𝑛𝑛
𝑇(
ൗ2ൗ2)
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
Base
𝑐𝑛𝑛
𝑇(
ൗ2ൗ2)
𝑐𝑛𝑛ൗൗ )
𝑇(
22
𝑛ൗൗ )
𝑇(𝑐𝑛
22
𝑐𝑛ൗ
4
𝑑 𝑑 𝑑
𝑑
• What will be the number of leaves? More than or less than previous?
• What about the height? More than or less than previous?
• Height 𝐻 = log 𝑏 𝑛 = log 2 𝑛
• Number of leaves 𝐿 = 𝑎𝐻 = 4log2 𝑛 = 𝑛2
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
23
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑑
𝑛=1
𝑇 𝑛 = ቐ4𝑇 𝑛 + 𝑐𝑛 𝑛 > 1
2
c𝑛
𝑐𝑛ൗ
2
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
2
𝑐𝑛ൗ
2
𝑐𝑛
Level 0
𝑐𝑛ൗ
2
𝑐𝑛ൗ
4
𝑐𝑛
= 2𝑐𝑛
2
Level 1
4
Level 2
16
𝑐𝑛
= 4𝑐𝑛
4
Level 𝐻 − 1
Base
𝑑 𝑑 𝑑
𝑑
Level 𝐻
• So base cost, = 𝑑 × 𝐿 = 𝑑𝑛2 = 𝜃(𝑛2 )
• Now the internal cost
• So, number of operations at level 𝑖 is 2𝑖 𝑐𝑛
𝑖
𝐻
log2 𝑛
• Internal cost = σ𝐻−1
− 1 = 𝑐𝑛 𝑛 − 1 =
𝑖=0 2 𝑐𝑛 = 𝑐𝑛 2 − 1 = 𝑐𝑛 2
2
𝑐𝑛 − 𝑐𝑛
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
24
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Recursion Tree
𝑑
𝑛=1
𝑇 𝑛 = ቐ4𝑇 𝑛 + 𝑐𝑛 𝑛 > 1
2
c𝑛
𝑐𝑛ൗ
2
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
4
𝑐𝑛ൗ
2
𝑐𝑛ൗ
2
𝑐𝑛
Level 0
𝑐𝑛ൗ
2
𝑐𝑛ൗ
4
𝑐𝑛
= 2𝑐𝑛
2
Level 1
4
Level 2
16
𝑐𝑛
= 4𝑐𝑛
4
Level 𝐻 − 1
Base
𝑑 𝑑 𝑑
𝑑
Level 𝐻
• So base cost, = 𝑑 × 𝐿 = 𝑑𝑛2 = 𝜃(𝑛2 )
• Internal cost = 𝑐𝑛2 − 𝑐𝑛
• Total cost = 𝑑𝑛2 + 𝑐𝑛2 − 𝑐𝑛 = 𝜃(𝑛2 )
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
25
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Master Theorem
• Review of the general procedure
𝑎
• General form:
• 𝑇 𝑛 = ቐ
•
•
•
•
𝑑
𝑎𝑇
𝑛
𝑓( )
𝑏
𝑛 ≤ 𝑛0
𝑛
𝑏
+ 𝑓(𝑛)
𝑓(𝑛)
𝑛
𝑓( )
𝑏
𝑛
𝑓( )
𝑏
𝑛
𝑓( )
𝑏
𝑛 > 𝑛0
where 𝑑 is base case runtime
𝑎 is number of subproblems (branching factor) Base 𝑑
𝑏 is the size of each subproblem (division ratio)
𝑓(𝑛) total time for divide and combine operations
𝑑
𝑑
• Compute Height 𝐻 = log 𝑏 𝑛
• Compute number of leaves 𝐿 = 𝑎𝐻 = 𝑎log𝑏 𝑛 = 𝑛log𝑏 𝑎
• Compute base cost (Generally - 𝐿 × 𝑑)
𝑛
• Compute internal cost 𝑓 𝑛 + 𝑎𝑓
+ 𝑎2 𝑓
𝑏
• Sum base and internal cost
Aug 10,11,12,17, 2022
𝑛
𝑏2
+⋯
CS21003/CS21203 / Algorithms - I | Divide and Conquer
26
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Master Theorem
• Master theorem provides a consolidated set of rules
• It depends on the relation between the number of leaves (𝐿 =
𝑛log𝑏 𝑎 ) and the function 𝑓(𝑛)
• Case I: If 𝐿 is polynomially larger i.e., 𝑓 𝑛 = 𝑂 𝑛log𝑏 𝑎−∆ , where ∆
is some constant > 0, then 𝑇 𝑛 = Θ(𝑛log𝑏 𝑎 )
• ∆ is the difference in the polynomial degree between 𝐿 and 𝑓(𝑛)
• If 𝐿 = 𝑛2 and 𝑓 𝑛 = 𝑛, then ∆=?  1
3
• Similarly, if 𝐿 = 𝑛2 and 𝑓 𝑛 = 𝑛2 , then ∆=?  0.5
• Another way of thinking this is – the ratio between 𝐿 and 𝑓(𝑛)
must be at least a polynomial
𝑛2
𝐿
• Take 𝐿 = 𝑛2 and 𝑓 𝑛 =
, Then
= log 𝑛 and log 𝑛 is less
log 𝑛
𝑓(𝑛)
than polynomial
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
27
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Solving Recurrences – Master Theorem
• Case I: If 𝐿 is polynomially larger i.e., 𝑓 𝑛 = 𝑂 𝑛log𝑏 𝑎−∆ , where ∆
is some constant > 0, then 𝑇 𝑛 = Θ(𝑛log𝑏 𝑎 )
•
∆ is the difference in the polynomial degree between 𝐿 and 𝑓(𝑛)
• Case II: If 𝑓 𝑛 and 𝐿 are asymptotically same, i.e., 𝑓 𝑛 = Θ 𝑛log𝑏 𝑎
then 𝑇 𝑛 = Θ 𝑛log𝑏 𝑎 log 𝑛 = Θ(𝑓(𝑛) log 𝑛)
• Case III: If 𝑓(𝑛) is polynomially larger i.e., 𝑓 𝑛 = Ω 𝑛log𝑏 𝑎+∆ ,
𝑛
then 𝑇 𝑛 = Θ(𝑓(𝑛)) if 𝑎𝑓( ) ≤ 𝑘𝑓(𝑛) for 𝑘 < 1
Regularity Condition
𝑏
𝑛
+ 𝑐𝑛
2
𝑎 = 2, 𝑏 = 2
𝐿 = 𝑛log𝑏 𝑎 = 𝑛
𝑓 𝑛 = 𝑐𝑛
Case II: 𝑇 𝑛 = Θ(𝑛 log 𝑛)
𝑇 𝑛 = 2𝑇
Aug 10,11,12,17, 2022
𝑛
+ 𝑐𝑛
2
𝑎 = 4, 𝑏 = 2
𝐿 = 𝑛log𝑏 𝑎 = 𝑛2
𝑓 𝑛 = 𝑐𝑛
𝑇 𝑛 = 4𝑇
Case I: 𝑇 𝑛 = Θ 𝑛log𝑏 𝑎 = Θ(𝑛2 )
CS21003/CS21203 / Algorithms - I | Divide and Conquer
28
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort
• Quicksort is a sorting algorithm which puts elements in the right
places one by one
• What is a “right place”?
• Is there any element which is at the right place?
3
7
9
5
15 11
4
7
9
5
15 11 17
4
7
5
9
15 11 12
Aug 10,11,12,17, 2022
4
CS21003/CS21203 / Algorithms - I | Divide and Conquer
29
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort
• Lets walkthrough an example to understand how quicksort works
7
6
10
5
9
2
1
15
7
• The crucial step in quicksort is partioning (divide into two parts
such that left of pivot is less and right of pivot id more)
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
30
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
31
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
Θ 1
Θ 𝑛
Overall runtime of “PARTITION”
subroutine is Θ(𝑛)
Θ 1
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
32
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
Θ 𝑛
Hard to tell about these.
• Lets take two extreme cases
Partitions are perfectly balanced everytime
• Number of levels = log 2 𝑛
• Recurrence relation
𝑛
+ 𝑐𝑛
2
• Same as mergesort – Θ(𝑛 log 𝑛)
𝑇 𝑛 = 2𝑇
•
Aug 10,11,12,17, 2022
However, this is the best case scenario
CS21003/CS21203 / Algorithms - I | Divide and Conquer
33
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
Θ 𝑛
Hard to tell about these.
• Lets take two extreme cases
Partitions are most unbalanced everytime
• Number of levels = Θ(𝑛)
• Recurrence relation
𝐶𝑜𝑛𝑠𝑡.
𝑇 𝑛 = 𝑇 𝑛 − 1 + 𝑇(0) + 𝑐𝑛
• Lets try to solve this recusrsive eqn.
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
34
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
• Base case = 𝑐𝑜𝑛𝑠𝑡.
c𝑛
• Internal cost at level 𝑖 is 𝑐(𝑛 − 𝑖)
C(𝑛 − 1)
• Total internal cost
𝑐 2+ 3+ ⋯𝑛
C(𝑛 − 2)
• Number of levels = Θ(𝑛)
2c
• Recurrence relation
Base case
𝐶𝑜𝑛𝑠𝑡.
𝑇 𝑛 = 𝑇 𝑛 − 1 + 𝑇 0 + 𝑐𝑛
• Same as insertion sort – Θ 𝑛2
•
Aug 10,11,12,17, 2022
This is the worst case scenario
CS21003/CS21203 / Algorithms - I | Divide and Conquer
35
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
• What should the input look like for the best case?
•
The medians are chosen as pivots always
• What should the input look like for the worst case?
•
The input array is already sorted
• Probability of getting an already sorted array is
1
𝑛!
• However, the problem can be tackled if instead of always taking
the first element as pivot, we take any element as pivot in random
• The average-case time complexity of the randomized version of
quicksort is Θ(𝑛 log 𝑛)
• The proof of this involves a more complicated recurrence than we
have seen so far.
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
36
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
• We will follow “Introduction to Algorithms and Data Structures” by
M. J. Dinneen, G. Gimelfarb, M. C. Wilson for the proof
• Let 𝑇(𝑛) denote the average-case running time for size 𝑛 list
• In the first step, the time taken to compare all the elements with
the pivot is 𝑐𝑛
• Let 0 ≤ 𝑖 ≤ 𝑛 − 1 is the final position of the pivot. So, the two
sublists are of size 𝑖 and 𝑛 − 𝑖 − 1 respectively
𝑖
𝑖
𝑛−𝑖−1
• The recurrence relation for this 𝑖 is 𝑇 𝑛 = 𝑇 𝑖 + 𝑇 𝑛 − 1 − 𝑖 + 𝑐𝑛
•
Small detail: Last term should be Θ 𝑛 , as it depends on the exact implementaiton
(e.g., whether comparison starts from pivot or one element after, etc.)
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
37
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
𝑖
•
•
•
•
•
𝑖
𝑛−𝑖−1
The recurrence relation for this 𝑖 is 𝑇 𝑛 = 𝑇 𝑖 + 𝑇 𝑛 − 1 − 𝑖 + 𝑐𝑛
The final pivot position 𝑖 can be any of the 𝑛 positions with equal
probability
So, the value of 𝑇(𝑛) to be 𝑇 0 + 𝑇 𝑛 − 1 + 𝑐𝑛 (i.e., 𝑖 = 0) has
1
probability
𝑛
Similarly, the value of 𝑇(𝑛) to be 𝑇 1 + 𝑇 𝑛 − 2 + 𝑐𝑛 (i.e., 𝑖 = 1)
1
has probability and so on
𝑛
So the expected value of 𝑇(𝑛) is given by,
𝑛−1
𝑛−1
1
1
𝑇 𝑛 = ෍ 𝑇 𝑖 + 𝑇 𝑛 − 1 − 𝑖 + 𝑐𝑛 = ෍ 𝑇 𝑖 + 𝑇 𝑛 − 1 − 𝑖
𝑛
𝑛
𝑖=0
Aug 10,11,12,17, 2022
+ 𝑐𝑛
𝑖=0
CS21003/CS21203 / Algorithms - I | Divide and Conquer
38
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
1
• 𝑇 𝑛 = σ𝑛−1
(𝑇 𝑖 + 𝑇 𝑛 − 1 − 𝑖 ) + 𝑐𝑛
𝑛 𝑖=0
• 𝑇 0 + 𝑇 𝑛 − 1 = 𝑇 𝑛 − 1 + 𝑇 0 𝑓𝑜𝑟 𝑖 = 0 𝑎𝑛𝑑 𝑖 = 𝑛 − 1
• 𝑇 1 + 𝑇 𝑛 − 2 = 𝑇 𝑛 − 2 + 𝑇 1 𝑓𝑜𝑟 𝑖 = 1 𝑎𝑛𝑑 𝑖 = 𝑛 − 2 and so
on
𝑖
𝑖
𝑖
• 𝑇 𝑛 =
𝑛−𝑖−1
1 𝑛−1
σ (𝑇
𝑛 𝑖=0
𝑛−𝑖−1
𝑖
2
𝑛
𝑖 + 𝑇 𝑛 − 1 − 𝑖 ) + 𝑐𝑛 = σ𝑛−1
𝑖=0 𝑇 𝑖 + 𝑐𝑛
2
• Let us rewrite this as 𝑛𝑇 𝑛 = 2 σ𝑛−1
𝑖=0 𝑇 𝑖 + 𝑐𝑛
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
39
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
2
• 𝑛𝑇 𝑛 = 2 σ𝑛−1
𝑖=0 𝑇 𝑖 + 𝑐𝑛
• Replace 𝑛 by 𝑛 − 1 and subtract from the above
•
𝑛𝑇 𝑛 = 2 𝑇 0 + 𝑇 1 + ⋯ 𝑇 𝑛 − 2 + 𝑇 𝑛 − 1 + 𝑐𝑛2
𝑛 − 1 𝑇 𝑛 − 1 = 2 𝑇 0 + 𝑇 1 + ⋯ + 𝑇 𝑛 − 3 + 𝑇(𝑛 − 2) + 𝑐 𝑛 − 1
• 𝑛𝑇 𝑛 − 𝑛 − 1 𝑇 𝑛 − 1 = 2𝑇 𝑛 − 1 + 𝑐 𝑛2 − 𝑛 − 1
• 𝑛𝑇 𝑛 = 𝑛 + 1 𝑇 𝑛 − 1 + 𝑐(2𝑛 − 1)
• Divide everything by 𝑛(𝑛 + 1)
•
𝑇(𝑛)
𝑛+1
=
𝑇(𝑛−1)
+
𝑛
𝑐
2𝑛−1
𝑛(𝑛+1)
=
𝑇(𝑛−1)
+
𝑛
𝑐
2
2
3
1
−
𝑛+1
𝑛
• Last term uses partial fraction decomposition
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
40
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
𝑇(𝑛)
𝑛+1
=
𝑇(𝑛−1)
+
𝑛
𝑐
•
𝑇(1)
2
=
𝑇(0)
3𝑐
+
1
2
−
𝑐
1
•
𝑇(2)
3
=
𝑇(1)
3𝑐
+
2
3
−
𝑐
2
•
𝑇(3)
4
=
𝑇(2)
3𝑐
+
3
4
−
𝑐
3
•
𝑇(𝑛)
𝑛+1
=
𝑇(𝑛−1)
3𝑐
𝑐
+
−
𝑛
𝑛+1
𝑛
•
𝑇(𝑛)
𝑛+1
=
𝑇(0)
+
1
•
2𝑛−1
𝑛(𝑛+1)
=
• Lets put 𝑛 = 1, 2,3, … 𝑛
Aug 10,11,12,17, 2022
3𝑐
𝑇(𝑛−1)
+
𝑛
𝑐
1
1
1
1
+ + +⋯+
2
3
4
𝑛+1
3
1
−
𝑛+1
𝑛
−𝑐
1
1
1
+ + +
1
2
3
CS21003/CS21203 / Algorithms - I | Divide and Conquer
⋯+
1
𝑛
41
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
Quicksort – Runtime Analysis
•
𝑇(𝑛)
𝑛+1
=
𝑇(0)
+
1
•
𝑇(𝑛)
𝑛+1
=
𝑇(0)
−𝑐
1
1
2
3𝑐
+
1
3
1
1
1
1
+ + +⋯+
2
3
4
𝑛+1
3𝑐
𝑛+1
+ 2𝑐
1
1
+
2
3
1
4
−𝑐
+ + ⋯+
1
𝑛
1
1
1
+ + +
1
2
3
=𝑑−𝑐+
⋯+
3𝑐
+
𝑛+1
1
𝑛
2𝑐(𝐻𝑛 − 1)
1
𝑛
• 𝐻𝑛 = 1 + + + ⋯ + ; 𝑑 = 𝑇(0)
• 𝑇 𝑛 = 𝑑 − 𝑐 𝑛 + 1 + 3𝑐 + 2𝑐 𝑛 + 1 𝐻𝑛 − 1
•
= 𝑑 + 𝑑 − 3𝑐 𝑛 + 2𝑐𝐻𝑛 + 2𝑐𝑛𝐻𝑛
• Using a result 𝐻𝑛 = Θ(log 𝑛), we can write
• 𝑇 𝑛 = 𝑑 + 𝑑 − 3𝑐 𝑛 + 2𝑐Θ log 𝑛 + 2𝑐Θ 𝑛 log 𝑛 = Θ(𝑛 log 𝑛)
• 𝐻𝑛 = Θ log 𝑛 -> This will shown to be true for a special case, next
Aug 10,11,12,17, 2022
CS21003/CS21203 / Algorithms - I | Divide and Conquer
42
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in
To be continued …