Created by Turbolearn AI
Heapsort
Heapsort is a sorting algorithm with the following characteristics:
Like Merge sort, Heapsort has a running time of O(n lg n).
Like Insertion sort, Heapsort works in place, requiring only Θ(1) extra space
during execution.
It uses a data structure to manage information, representing an algorithm
design technique.
Arrays as Binary Trees
Arrays can be interpreted as binary trees. The tree's height is the number of edges in
the longest path from the root to a leaf.
The height of a node in the tree is the number of edges in the longest simple
downward path from the node to a leaf.
The height of the tree is the height of its root.
Given an index $i$:
Parent: $PARENT(i) = i/2$
Left Child: $LEFT(i) = 2i$
Right Child: $RIGHT(i) = 2i + 1$
These procedures (PARENT, LEFT, RIGHT) can be efficiently implemented using bit
shifting operations.
Max-Heaps and Min-Heaps
There are two types of binary heaps:
Page 1
Created by Turbolearn AI
Max-Heaps: In a max-heap, the value of each node is at most the value of its
parent.
$A[PARENT(i)] ≥ A[i]$
The largest element is stored at the root.
Min-Heaps: Organized in the opposite way such that the smallest element is at
the root.
$A[PARENT(i)] ≤ A[i]$
Heap Attributes
A heap has two main attributes:
Length: The number of elements in the array.
Heap-size: The number of elements of the heap stored in the array.
The heap-size is always less than or equal to the length of the array. $0 ≤ heap-size
≤ A.length$
Quiz Answers
Page 2
Created by Turbolearn AI
1. What is the height of the tree corresponding to an $n$-element array?
The path from the root to the leftmost leaf has maximal length.
The height equals the number of times we can move to a left subtree: $n ≥
2^h$.
Therefore, $h = ⌊lg n⌋$.
2. What are the minimum and maximum number of elements in a tree of height
$h$?
Min number is $2^h$. This is a tree with only one maximal path.
Max number is $2^(h+1) - 1$. This is a tree where if you add one more
node, it becomes a tree of height $h + 1$.
3. What are the indices in the tree that are leaves?
$⌊n/2⌋ + 1, ⌊n/2⌋ + 2, …, n$
$⌊i/2⌋$ is the parent of the index $i$.
$⌊n/2⌋$ is the last index that is a parent; the following indices are leaves.
Heapsort Algorithm
Heapsort involves several procedures:
MAX-HEAPIFY: Maintains the max-heap property. Runs in $O(\lg n)$ time.
BUILD-MAX-HEAP: Produces a max-heap from an unordered input array. Runs
in linear time.
HEAPSORT: Sorts an array in place. Runs in $O(n \lg n)$ time.
Operations on Heaps
Max-Heapify
Focus on max-heaps. Procedures can be adapted to min-heaps.
Max-Heapify: Maintains the max-heap property. Runs in time O(lgn).
Build-Max-Heap: Produces a max-heap from an unordered input array. Runs in
O(n).
Maintaining the Heap Property
Page 3
Created by Turbolearn AI
Input for Max-Heapify
Array A[1.. n]
Index i, where 1 ≤ i ≤ n
Binary trees rooted at Lef t(i) and Right(i) are max-heaps but might be
smaller than their children A[i].
Output for Max-Heapify
Array reordered to satisfy the max-heap property for the binary tree rooted at i
.
Main Idea of Max-Heapify
Compare A[i] with A[Lef t(i)] and A[Right(i)].
"Float down" A[i] by swapping it with the largest of the three.
Recursively call on the selected subtree.
The problem is solved when A[i] is the largest, or when i is a leaf.
Max-Heapify: Pseudocode
MAX-HEAPIFY(A, i)
1. l = LEFT(i)
2. r = RIGHT(i)
3. if l <= A.heap-size and A[l] > A[i]
4.
largest = l
5. else largest = i
6. if r <= A.heap-size and A[r] > A[largest]
7.
largest = r
8. if largest != i
9.
exchange A[i] with A[largest]
10. MAX-HEAPIFY(A, largest)
Max-Heapify: Explanation
Page 4
Created by Turbolearn AI
1. Determine the largest among A[i], A[Lef t(i)], and A[Right(i)].
2. If A[i] is the largest, the subtree rooted at node i is already a max-heap, and
the procedure terminates.
3. Otherwise, swap A[i] with A[largest], causing node i and its children to satisfy
the max-heap property.
4. Recursively restore the heap property in the subtree.
Max-Heapify: Example
Figure 6.2 demonstrates Max-Heapify's action with A[2], where
A. heap − size = 10.
Initial configuration: A[2] violates the max-heap property.
The max-heap property is restored by exchanging A[2] with A[4].
The recursive call Max-Heapify(A, 4) then has i = 4.
Max-Heapify: Running Time
The running time of Max-Heapify on a subtree of size n rooted at node i is Θ(1)
to fix relationships among A[i], A[Lef t(i)], and A[Right(i)], plus the time to run
Max-Heapify on a subtree rooted at one of i's children (if the recursive call
occurs).
Children's subtrees have size at most 2n/3 (worst case: bottom level is half
full).
Recurrence for running time: T (n) ≤ T (2n/3) + Θ(1).
Max-Heapify: Run-Time Analysis
Problem size is the number of elements in the tree rooted at node i.
What is the maximum size of the subproblem (as a function of n)?
Answer: Worst case is when the bottom level of the tree is exactly half
full.
If b is the nodes at the bottom level, then b = n + 2(n − 1) + 1 = 3n − 1.
The size of the largest subtree is then ≤ 2n/3
*Recall: A full binary tree with m leaves has m − 1 internal nodes.
Page 5
Created by Turbolearn AI
Max-Heapify: Time Complexity
T (n) ≤ T (2n/3) + Θ(1)
By the Master theorem (case 2), T (n) = O(lgn).
Building a Heap
Main Idea of Building a Heap
Indices ⌊n/2⌋ + 1, ⌊n/2⌋ + 2, … , n are leaf nodes of the tree (i.e., 1-element
heap).
Use Max-Heapify in a bottom-up manner to convert an array into a max-heap.
Build-Max-Heap: Procedure
1. Set A. heap − size = A. length.
2. Iterate from i = ⌊A. length/2⌋ down to 1.
3. Call Max-Heapify(A, i) for each i.
BUILD-MAX-HEAP(A)
1. A.heap-size = A.length
2. for i = floor(A.length/2) downto 1
3.
MAX-HEAPIFY(A, i)
Loop Invariant for Correctness
At the start of each iteration of the for loop, each node i + 1, i + 2, … , n is
the root of a max-heap.
Initialization: Prior to the first iteration, i = ⌊n/2⌋. Each node
⌊n/2⌋ + 1, ⌊n/2⌋ + 2, … , n is a leaf and thus the root of a trivial max-heap.
Maintenance: Each iteration maintains the loop invariant. The children of node i
are numbered higher than i. By the loop invariant, they are both roots of maxheaps. The call Max-Heapify(A, i) makes node i a max-heap root. The MaxHeapify call preserves the property that nodes i + 1, i + 2, … , n are all roots of
max-heaps.
Termination: At termination, i = 0. Each node 1, 2, … , n is the root of a maxheap.
Page 6
Created by Turbolearn AI
Build-Max-Heap: Running Time
Each call to Max-Heapify costs O(lgn) time, and Build-Max-Heap makes O(n)
such calls. Thus, the running time is O(nlgn). This upper bound, though correct,
is not asymptotically tight.
The time for Max-Heapify to run at a node varies with the height of the node in
the tree, and the heights of most nodes are small.
Build-Max-Heap: Tighter Analysis
An n-element heap has height lgn.
There are at most n/2 nodes of any height h.
The time required by Max-Heapify when called on a node of height h is O(h).
h
Building a Heap
BUILD-MAX-HEAP
The BUILD-MAX-HEAP procedure is used to produce a max-heap from an unordered
array. The procedure goes through the remaining nodes of the tree and runs MAXHEAPIFY on each one.
BUILD-MAX-HEAP(A)
A: heap-size = A: length
for i = A: length / 2 downto 1
MAX-HEAPIFY(A, i)
Figure 6.3 from the lecture notes illustrates the operation of BUILD-MAX-HEAP.
To show why BUILD-MAX-HEAP works correctly, we use the following loop invariant:
At the start of each iteration of the for loop of lines 2-3, each node i + 1,
i + 2, ..., n is the root of a max-heap.
Page 7
Created by Turbolearn AI
Initialization: Prior to the first iteration of the loop, i = n/2. Each node
n/2 + 1, n/2 + 2, . . . , n is a leaf and is thus the root of a trivial max-heap.
Maintenance: The children of node i are numbered higher than i. By the loop
invariant, therefore, they are both roots of max-heaps. This is precisely the
condition required for the call MAX-HEAPIFY(A, i) to make node i a max-heap
root. Moreover, the MAX-HEAPIFY call preserves the property that nodes i + 1, i
+ 2, ..., n are all roots of max-heaps. Decrementing i in the for loop update
reestablishes the loop invariant for the next iteration.
Termination: At termination, i = 0. By the loop invariant, each node 1, 2, . . . , n
is the root of a max-heap. In particular, node 1 is.
BUILD-MAX-HEAP: Runtime
The simple upper bound on the running time of BUILD-MAX-HEAP is O(n lg n)
because each call to MAX-HEAPIFY costs O(lg n) time, and BUILD-MAX-HEAP makes
O(n) such calls.
The tighter analysis relies on the properties that an n-element heap has height
lg n and at most n/2
nodes of any height h.
The time required by MAX-HEAPIFY when called on a node of height h is O(h),
and so we can express the total cost of BUILD-MAX-HEAP as being bounded
from above.
Rough Big-O Analysis: O(n lg n)
Actually Θ(n): observe that the time for MAX-HEAPIFY varies with the height of
the node in the tree
h+1
Heapsort Algorithm
The main idea behind the Heapsort algorithm:
1. Build a max-heap from A[1.. . n].
2. Take the max element and swap it with A[n].
3. Restore the max-heap property for A[1.. . n − 1] by calling MAX-HEAPIFY(A, 1).
4. Repeat steps 2-4 on the subarray A[1.. . n − 1].
Order matters when performing Heapsort!
Page 8
Created by Turbolearn AI
HEAPSORT(A)
BUILD-MAX-HEAP(A)
for i = A:length downto 2
exchange A[1] with A[i]
A:heap-size = A:heap-size - 1
MAX-HEAPIFY(A, 1)
At line 5 the subtrees rooted at Left(1) and Right(1) are max-heaps.
Figure 6.4 from the lecture notes shows an example of the operation of HEAPSORT after
line 1 has built the initial max-heap. The figure shows the max-heap before the first
iteration of the for loop of lines 2-5 and after each iteration.
The HEAPSORT procedure takes time O(n lg n), since the call to BUILD-MAX-HEAP
takes time O(n) and each of the n − 1 calls to MAX-HEAPIFY takes time O(lg n).
Worst-case runtime: O(n lg n)
Quick Sort
Why Quick Sort?
Like Merge Sort, based on divide and conquer design
Like Insertion Sort and Heapsort, works in-place
Worst-case runtime O(n )
Average-case runtime Θ(n lg n)
The constant factors hidden behind Θ(n lg n) are quite small ⟹ often best
practical choice
2
Main Idea
Divide: rearrange the array into two partitions (subarrays) L = A[p. . . q − 1] and
R = A[q + 1. . . r] such that
Each element of L is less than or equal to A[q].
Each element of R is greater than or equal to A[q].
Conquer: sort L and R by recursive calls to Quicksort.
Combine: the two subarrays are already sorted, thus no work is needed to
combine them.
Page 9
Created by Turbolearn AI
Partitioning the array
Input:
array A[1.. . n]
indices 1 ≤ p ≤ r ≤ n
Goal: pick x ∈ A and rearrange the elements of A[p. . . r] so that
each element of A[p. . . q − 1] is less than or equal to x
each element of A[q + 1. . . r] is greater than or equal to x
x = A[q]
Partitioning the array: Main Idea
Use x = A[r] (i.e., x is the last element of the subarray)
Grow the partitions scanning elements from left to right
When arrived at the last element, place it in the middle
Partition: Pseudocode
PARTITION(A, p, r)
x = A[r]
i = p - 1
for j = p to r - 1
if A[j] <= x
i = i + 1
exchange A[i] with A[j]
exchange A[i + 1] with A[r]
return i + 1
Quicksort: Pseudocode
QUICKSORT(A, p, r)
if p < r
q = PARTITION(A, p, r)
QUICKSORT(A, p, q - 1)
QUICKSORT(A, q + 1, r)
To sort an entire array A, the initial call is QUICKSORT(A, 1, A:length).
Page 10
Created by Turbolearn AI
Partition: Loop Invariant
The key to the algorithm is the PARTITION procedure, which rearranges the
subarray A[p. . . r] in place.
Use x = A[r] to set the pivot element.
At the beginning both partitions are empty: separator out of the range [p, r].
i = p − 1
Partition: Loop Invariant Properties
The following is a visual representation of the four regions maintained by the
procedure PARTITION on a subarray A[p. . . r]:
A[p. . . i]
A[i + 1. . . j − 1]
A[j. . . r − 1]
A[r]
≤ x
> x
unrestricted
x
Initialization: Prior to the first iteration of the loop, i = p − 1 and j = p. Because
no values lie between p and i and no values lie between i + 1 and j − 1, the
first two conditions of the loop invariant are trivially satisfied.
Maintenance: We consider two cases, depending on the outcome of the test in
line 4:
If A[j] > x, the only action in the loop is to increment j. After j is
incremented, condition 2 holds for A[j − 1] and all other entries remain
unchanged.
If A[j] ≤ x, the loop increments i, swaps A[i] and A[j], and then
increments j. Because of the swap, we now have that A[i] ≤ x, and
condition 1 is satisfied. Similarly, we also have that A[j − 1] > x, since the
item that was swapped into A[j − 1] is, by the loop invariant, greater than
x.
Termination: At termination, j = r. Therefore, every entry in the array is in one
of the three sets described by the invariant, and we have partitioned the values
in the array into three sets: those less than or equal to x, those greater than x,
and a singleton set containing x.
Partition: Final steps
Page 11
Created by Turbolearn AI
The final two lines of PARTITION finish up by swapping the pivot element with the
leftmost element greater than x, thereby moving the pivot into its correct place in the
partitioned array, and then returning the pivot’s new index.
The output of PARTITION now satisfies the specifications given for the divide step. In
fact, it satisfies a slightly stronger condition: after line 2 of QUICKSORT, A[q] is strictly
less than every element of A[q + 1. . . r].
Partition: Running time
The running time of PARTITION on the subarray A[p. . . r] is Θ(n), where n = r − p + 1.
Combine: In Quicksort
Because the subarrays are already sorted, no work is needed to combine them: the
entire array A[p. . . r] is now sorted.
Quicksort Overview
Quicksort is a sorting algorithm employing a divide-and-conquer strategy. It works
by partitioning an array around a chosen pivot element.
Divide-and-Conquer Paradigm
Quicksort uses the divide-and-conquer paradigm, which involves three steps:
1. Divide: Partition the array into two subarrays around a pivot element x such
that elements less than or equal to x are in one subarray and elements greater
than x are in the other.
2. Conquer: Recursively sort the two subarrays.
3. Combine: Because the subarrays are already sorted, no work is needed to
combine them as the entire array is now sorted.
Quicksort Procedure
Page 12
Created by Turbolearn AI
QUICKSORT(A, p, r)
1 if p < r
2
q = PARTITION(A, p, r)
3
QUICKSORT(A, p, q - 1)
4
QUICKSORT(A, q + 1, r)
To sort an entire array A, the initial call is QUICKSORT(A, 1, A.length).
Partitioning the Array
The key to quicksort is the PARTITION procedure, which rearranges the subarray
A[p. . . r] in place.
PARTITION(A, p, r)
1 x = A[r]
2 i = p - 1
3 for j = p to r - 1
4
if A[j] <= x
5
i = i + 1
6
exchange A[i] with A[j]
7 exchange A[i + 1] with A[r]
8 return i + 1
Figure 7.1 illustrates how PARTITION works on an 8-element array.
Loop Invariant of Partition
PARTITION always selects an element x = A[r] as a pivot element around which to
partition the subarray A[p. . . r]. As the procedure runs, it partitions the array into four
(possibly empty) regions. At the start of each iteration of the for loop, these regions
satisfy certain properties, which are maintained as a loop invariant.
At the beginning of each iteration of the loop, for any array index k:
1. If p ≤ k ≤ i, then A[k] ≤ x.
2. If i + 1 ≤ k ≤ j − 1, then A[k] > x.
3. If k = r, then A[k] = x.
Figure 7.2 visually represents these regions.
Page 13
Created by Turbolearn AI
Loop Invariant Properties
Region
First Region
Second Region
Pivot
Indices
Property
p ≤ k ≤ i
A[k] ≤ x
i + 1 ≤ k ≤ j − 1
A[k] > x
k = r
A[k] = x
Maintenance of the Loop Invariant
As Figure 7.3 shows, we consider two cases, depending on the outcome of the test in
line 4. If A[j] > x, the only action in the loop is to increment j. If A[j] ≤ x, the loop
increments i, swaps A[i] and A[j], and then increments j.
Termination of Partition
At termination, j = r. Every entry in the array is in one of the three sets described by
the invariant. The final two lines of PARTITION finish up by swapping the pivot
element with the leftmost element greater than x, thereby moving the pivot into its
correct place in the partitioned array, and then returning the pivot’s new index.
Running Time of Partition
The running time of PARTITION on the subarray A[p. . . r] is Θ(n), where
n = r − p + 1
Quicksort
Quicksort is a sorting algorithm based on the divide-and-conquer paradigm.
Main Idea
Page 14
Created by Turbolearn AI
Divide: Rearrange the array into two partitions (subarrays) such that
A[p. . q − 1] ≤ A[q] ≤ A[q + 1.. r].
Conquer: Sort partitions by recursive calls to Quicksort.
Combine: No work is needed to combine the subarrays because they are
already sorted.
Quicksort Procedure
The following procedure implements quicksort:
QUICKSORT(A, p, r)
1 if p < r
2
q = PARTITION(A, p, r)
3
QUICKSORT(A, p, q - 1)
4
QUICKSORT(A, q + 1, r)
To sort an entire array A, the initial call is QUICKSORT(A, 1, A.length).
Partitioning the Array
The key to Quicksort is the PARTITION procedure, which rearranges the subarray
A[p. . r] in place.
PARTITION(A, p, r)
1 x = A[r] // pivot element
2 i = p - 1
3 for j = p to r - 1
4
if A[j] <= x
5
i = i + 1
6
exchange A[i] with A[j]
7 exchange A[i + 1] with A[r]
8 return i + 1
The PARTITION procedure always selects an element x = A[r] as a pivot element
around which to partition the subarray A[p. . r]. As the procedure runs, it partitions
the array into four (possibly empty) regions.
Loop Invariant of Partition
Page 15
Created by Turbolearn AI
At the beginning of each iteration of the for loop in lines 3–6 of the PARTITION
procedure, for any array index k:
1. If p ≤ k ≤ i, then A[k] ≤ x.
2. If i + 1 ≤ k ≤ j − 1, then A[k] > x.
3. If k = r, then A[k] = x. The indices between j and r − 1 are not covered by any
of the three cases, meaning the values in these entries have no particular
relationship to the pivot x.
Quicksort Runtime Analysis
The runtime of Quicksort can be expressed as: T (n) = T (q − p) + T (r − q) + Θ(n)
where n = r − p + 1. Using a change of variable m = q − p, the equation becomes:
T (n) = T (m) + T (n − m − 1) + Θ(n) The runtime depends on how balanced the
partitioning is at each step, which in turn depends on the values of the pivots.
Worst-Case Scenario
Occurs when the partitioning routine produces one subproblem with n − 1
elements and one with 0 elements (i.e., m = n − 1 or m = 0).
This unbalanced partitioning arises in each recursive call.
This occurs, for example, when the array is already sorted.
The recurrence becomes: T (n) = T (n − 1) + T (0) + Θ(n) Since T (0) = Θ(1), we
have: T (n) = T (n − 1) + Θ(n) This leads to a worst-case time complexity of:
2
T (n) = Θ(n )
Best-Case Scenario
Occurs when the partitioning routine produces two balanced subproblems (i.e.,
m = ⌊n/2⌋).
And keeps being balanced in each recursive call The recurrence becomes:
T (n) = T (⌊n/2⌋) + T (⌈n/2⌉ − 1) + Θ(n) Which simplifies to:
T (n) ≈ 2T (n/2) + Θ(n) This results in a best-case time complexity of:
T (n) = Θ(n lg n) The same asymptotic growth occurs even when there is not
perfect balancing, for example: T (n) = T (9n/10) + T (n/10) + Θ(n) = Θ(n lg n)
Page 16
Created by Turbolearn AI
Average Case
On average, Quicksort runs in Θ(n lg n).
Counting Sort
Why Counting Sort?
Counting sort does not use comparisons to sort.
It sorts in linear time, Θ(n).
It uses extra memory.
Counting sort is stable.
Stable Sort Definition: Numbers with the same value appear in the same
order as they do in the resulting array.
∀1 ≤ i, j ≤ n. i < j ∧ A[i] = A[j] ⇒ π(i) < π(j)
Counting Sort Procedure
Input: Input array A[1.. n] of integer numbers, such that A[i] ∈ [0.. k], and output
array B[1.. n].
Output: The array B contains the elements of A but in sorted order.
Main Idea
Exploit the fact that numbers are integers in the range 0.. k.
Count occurrences of each number in A.
Determine the final (sorted) position of each element, and copy them in the
array B.
Counting Sort Pseudocode
Page 17
Created by Turbolearn AI
COUNTING-SORT(A, B, k)
1 let C[0..k] be a new array
2 for i = 0 to k
3
C[i] = 0
4 for j = 1 to A.length
5
C[A[j]] = C[A[j]] + 1
6 // C[i] now contains the number of elements equal to i.
7 for i = 1 to k
8
C[i] = C[i] + C[i - 1]
9 // C[i] now contains the number of elements less than or equal to i.
10 for j = A.length downto 1
11
B[C[A[j]]] = A[j]
12
C[A[j]] = C[A[j]] - 1
Counting Sort Example
Below is a step-by-step illustration of how Counting Sort works on an input array A.
The code initializes an array C to store counts, calculates cumulative counts to
determine positions, and copies elements to the output array B in sorted order.
1. Initialize the counters
2. Count number of occurrences of each element in A
3. C[i] now contains the number of elements equal to i
4. C[i] now contains the number of elements less than or equal to i
5. Copy the element A[j] in B in position C[A[j]], and update the position in case
there are other occurrences of the same element
6. Starting from the last element and going down ensure the stability of the
sorting procedure
Counting Sort
Counting sort is an algorithm that sorts elements by counting the number of
occurrences of each unique element in the array. The count is then used to determine
the position of each element in the sorted array.
Here's the pseudocode:
Page 18
Created by Turbolearn AI
COUNTING-SORT(A, B, k)
1 let C[0..k] be a new array
2 for i = 0 to k
3
C[i] = 0
4 for j = 1 to A.length
5
C[A[j]] = C[A[j]] + 1
6 // C[i] now contains the number of elements equal to i.
7 for i = 1 to k
8
C[i] = C[i] + C[i-1]
9 // C[i] now contains the number of elements less than or equal to i.
10 for j = A.length downto 1
11
B[C[A[j]]] = A[j]
12
C[A[j]] = C[A[j]] - 1
Counting Sort Example
Here is an example of how counting sort works with an array A.
A = [2, 5, 3, 0, 2, 3, 0, 3]
C is an auxillary array.
After the initialization of array C to all zeros and inspecting each input element, C will
hold the number of input elements equal to i for each integer i = 0, 1, . . . , k.
C = [2, 0, 2, 3, 0, 1]
After determining how many input elements are less than or equal to i by keeping a
running sum of the array C:
C = [2, 2, 4, 7, 7, 8]
The final sorted array B:
B = [0, 0, 2, 2, 3, 3, 3, 5]
Counting Sort: Run-time
The run-time of counting sort can be broken down as follows:
Page 19
Created by Turbolearn AI
Lines
Time
2 to 3
4 to 5
7 to 8
10 to 12
Θ(k)
Θ(n)
Θ(k)
Θ(n)
In practice counting sort is:
, used when k = O(n), in which case T (n) = Θ(n).
T (n) = Θ(n + k)
Radix Sort
Radix sort is a non-comparative sorting algorithm that sorts data with integer keys
by grouping keys by the individual digits which share the same significant position
and value. Radix sort uses counting sort as a subroutine. It can sort in linear time and
is stable. Radix sort is typically used to sort records of information that are keyed by
multiple fields.
How Radix Sort Works
The main idea is to repeatedly sort the elements according to the i-th digit, starting
from the least significant digit up to the most significant one. It is crucial to use a
stable sorting subroutine.
Here's the pseudocode for radix sort:
RADIX-SORT(A, d)
1 for i = 1 to d
2
use a stable sort to sort array A on digit i
Radix Sort: Example
Given the input: [329, 457, 657, 839, 436, 720, 355]
Sort by the ones column: [720, 329, 436, 457, 355, 657, 839]
Sort by the tens column: [720, 329, 436, 355, 457, 657, 839]
Page 20
Created by Turbolearn AI
Sort by the hundreds column: [329, 355, 436, 457, 657, 720, 839]
Lemma 8.3
Given n d-digit numbers in which each digit can take on up to k possible
values, RADIX-SORT correctly sorts these numbers in Θ(d(n + k)) time if
the stable sort it uses takes Θ(n + k) time.
When d is constant and k = O(n), we can make radix sort run in linear time.
Radix Sort
Radix sort works by iterating from i = 1 to d, using a stable sort to sort array A on
digit i.
Lemma 8.3: Given n d-digit numbers, where each digit can take on up to k
possible values, radix sort correctly sorts these numbers in Θ(d ⋅ (n + k))
time if the stable sort it uses takes Θ(n + k) time.
The correctness of radix sort is proven by induction on the column being sorted. The
running time depends on the stable sort used as the intermediate sorting algorithm.
When each digit is in the range 0 to k - 1, counting sort is often the obvious choice.
Each pass over n d-digit numbers then takes time Θ(n + k). With d passes, the total
time for radix sort is Θ(d ⋅ (n + k)).
If d is constant and k = O(n), we can make radix sort run in linear time. We have
flexibility in how to break each key into digits. Using counting sort as a subroutine
requires linear memory, while Quicksort works in place. The constant factors hidden
in the Θ-notation may be big, depending on the underlying architecture.
Sorting Algorithm Comparison
Here's a comparison of different sorting algorithms:
Page 21
Created by Turbolearn AI
Algorithm
Worst-case Running Time
Insertion Sort Θ(n )
Merge Sort
Θ(n lg n)
Heapsort
O(n lg n)
Quicksort
Θ(n )
Counting Sort Θ(k + n)
Radix Sort
Θ(d ⋅ (n + k))
Bucket Sort
Θ(n )
2
2
2
Average-case/Expected Running Time
2
Θ(n )
Θ(n lg n)
—
Θ(n lg n)
(expected)
Θ(k + n)
Θ(d ⋅ (n + k))
Θ(n)
(average-case)
Order Statistics
The ith order statistic of a set of n numbers is the ith smallest number in the set. We
can select the ith order statistic by sorting the input and indexing the ith element of
the output, which takes !(n lg n) time. However, it's possible to find the ith smallest
element in O(n) time. There exists a randomized algorithm with tight pseudocode
that runs in Θ(n ) time in the worst case but has an expected running time of O(n).
There's also a more complicated algorithm that runs in O(n) worst-case time.
2
Learned Concepts Recap
Heapsort: Design exploiting data organization
Comparison-based sorting
Quicksort: Divide & Conquer + in place
Comparison-based sorting
Counting Sort: Requires elements to be in a known range 0..k
Stable sorting procedure
Uses linear additional space
Radix Sort: Requires stable sorting subroutine
Extends applicability of Counting sort
Page 22