6
Heapsort
In this chapter, we introduce another sorting algorithm: heapsort. Like merge sort,
but unlike insertion sort, heapsort’s running time is O.n lg n/. Like insertion sort,
but unlike merge sort, heapsort sorts in place: only a constant number of array
elements are stored outside the input array at any time. Thus, heapsort combines
the better attributes of the two sorting algorithms we have already discussed.
Heapsort also introduces another algorithm design technique: using a data structure, in this case one we call a “heap,” to manage information. Not only is the heap
data structure useful for heapsort, but it also makes an efficient priority queue. The
heap data structure will reappear in algorithms in later chapters.
The term “heap” was originally coined in the context of heapsort, but it has since
come to refer to “garbage-collected storage,” such as the programming languages
Java and Lisp provide. Our heap data structure is not garbage-collected storage,
and whenever we refer to heaps in this book, we shall mean a data structure rather
than an aspect of garbage collection.
6.1 Heaps
The (binary) heap data structure is an array object that we can view as a
nearly complete binary tree (see Section B.5.3), as shown in Figure 6.1. Each
node of the tree corresponds to an element of the array. The tree is completely filled on all levels except possibly the lowest, which is filled from the
left up to a point. An array A that represents a heap is an object with two attributes: A:length, which (as usual) gives the number of elements in the array, and
A:heap-size, which represents how many elements in the heap are stored within
array A. That is, although AŒ1 : : A:length# may contain numbers, only the elements in AŒ1 : : A:heap-size#, where 0 $ A:heap-size $ A:length, are valid elements of the heap. The root of the tree is AŒ1#, and given the index i of a node, we
can easily compute the indices of its parent, left child, and right child:
152
Chapter 6 Heapsort
1
16
2
14
4
8
2
8
9
4
5
10
6
7
1
3
9
(a)
10
7
3
1
2
3
4
5
6
7
8
9
10
16 14 10 8
7
9
3
2
4
1
(b)
Figure 6.1 A max-heap viewed as (a) a binary tree and (b) an array. The number within the circle
at each node in the tree is the value stored at that node. The number above a node is the corresponding
index in the array. Above and below the array are lines showing parent-child relationships; parents
are always to the left of their children. The tree has height three; the node at index 4 (with value 8)
has height one.
PARENT .i/
1 return bi=2c
L EFT .i/
1 return 2i
R IGHT .i/
1 return 2i C 1
On most computers, the L EFT procedure can compute 2i in one instruction by
simply shifting the binary representation of i left by one bit position. Similarly, the
R IGHT procedure can quickly compute 2i C 1 by shifting the binary representation
of i left by one bit position and then adding in a 1 as the low-order bit. The
PARENT procedure can compute bi=2c by shifting i right one bit position. Good
implementations of heapsort often implement these procedures as “macros” or “inline” procedures.
There are two kinds of binary heaps: max-heaps and min-heaps. In both kinds,
the values in the nodes satisfy a heap property, the specifics of which depend on
the kind of heap. In a max-heap, the max-heap property is that for every node i
other than the root,
AŒPARENT.i/# % AŒi# ;
that is, the value of a node is at most the value of its parent. Thus, the largest
element in a max-heap is stored at the root, and the subtree rooted at a node contains
6.1 Heaps
153
values no larger than that contained at the node itself. A min-heap is organized in
the opposite way; the min-heap property is that for every node i other than the
root,
AŒPARENT.i/# $ AŒi# :
The smallest element in a min-heap is at the root.
For the heapsort algorithm, we use max-heaps. Min-heaps commonly implement priority queues, which we discuss in Section 6.5. We shall be precise in
specifying whether we need a max-heap or a min-heap for any particular application, and when properties apply to either max-heaps or min-heaps, we just use the
term “heap.”
Viewing a heap as a tree, we define the height of a node in a heap to be the
number of edges on the longest simple downward path from the node to a leaf, and
we define the height of the heap to be the height of its root. Since a heap of n elements is based on a complete binary tree, its height is ‚.lg n/ (see Exercise 6.1-2).
We shall see that the basic operations on heaps run in time at most proportional
to the height of the tree and thus take O.lg n/ time. The remainder of this chapter
presents some basic procedures and shows how they are used in a sorting algorithm
and a priority-queue data structure.
!
!
!
!
The M AX -H EAPIFY procedure, which runs in O.lg n/ time, is the key to maintaining the max-heap property.
The B UILD -M AX -H EAP procedure, which runs in linear time, produces a maxheap from an unordered input array.
The H EAPSORT procedure, which runs in O.n lg n/ time, sorts an array in
place.
The M AX -H EAP -I NSERT, H EAP -E XTRACT-M AX, H EAP -I NCREASE -K EY,
and H EAP -M AXIMUM procedures, which run in O.lg n/ time, allow the heap
data structure to implement a priority queue.
Exercises
6.1-1
What are the minimum and maximum numbers of elements in a heap of height h?
6.1-2
Show that an n-element heap has height blg nc.
6.1-3
Show that in any subtree of a max-heap, the root of the subtree contains the largest
value occurring anywhere in that subtree.
154
Chapter 6 Heapsort
6.1-4
Where in a max-heap might the smallest element reside, assuming that all elements
are distinct?
6.1-5
Is an array that is in sorted order a min-heap?
6.1-6
Is the array with values h23; 17; 14; 6; 13; 10; 1; 5; 7; 12i a max-heap?
6.1-7
Show that, with the array representation for storing an n-element heap, the leaves
are the nodes indexed by bn=2c C 1; bn=2c C 2; : : : ; n.
6.2
Maintaining the heap property
In order to maintain the max-heap property, we call the procedure M AX -H EAPIFY.
Its inputs are an array A and an index i into the array. When it is called, M AX H EAPIFY assumes that the binary trees rooted at L EFT .i/ and R IGHT .i/ are maxheaps, but that AŒi# might be smaller than its children, thus violating the max-heap
property. M AX -H EAPIFY lets the value at AŒi# “float down” in the max-heap so
that the subtree rooted at index i obeys the max-heap property.
M AX -H EAPIFY .A; i/
1 l D L EFT .i/
2 r D R IGHT .i/
3 if l $ A:heap-size and AŒl# > AŒi#
4
largest D l
5 else largest D i
6 if r $ A:heap-size and AŒr# > AŒlargest#
7
largest D r
8 if largest ¤ i
9
exchange AŒi# with AŒlargest#
10
M AX -H EAPIFY .A; largest/
Figure 6.2 illustrates the action of M AX -H EAPIFY. At each step, the largest of
the elements AŒi#, AŒL EFT .i/#, and AŒR IGHT .i/# is determined, and its index is
stored in largest. If AŒi# is largest, then the subtree rooted at node i is already a
max-heap and the procedure terminates. Otherwise, one of the two children has the
largest element, and AŒi# is swapped with AŒlargest#, which causes node i and its
6.2 Maintaining the heap property
155
1
1
4
8
14
2
16
2
4
i
9
5
10
8
6
7
1
10
9
16
2
3
7
3
i
8
2
(a)
14
4
4
9
8
5
10
1
3
6
7
9
10
7
3
(b)
1
16
2
14
4
8
2
8
9
4
i
5
10
1
3
6
7
9
10
7
3
(c)
Figure 6.2 The action of M AX -H EAPIFY.A; 2/, where A: heap-size D 10. (a) The initial configuration, with AŒ2# at node i D 2 violating the max-heap property since it is not larger than
both children. The max-heap property is restored for node 2 in (b) by exchanging AŒ2# with AŒ4#,
which destroys the max-heap property for node 4. The recursive call M AX -H EAPIFY.A; 4/ now
has i D 4. After swapping AŒ4# with AŒ9#, as shown in (c), node 4 is fixed up, and the recursive call
M AX -H EAPIFY.A; 9/ yields no further change to the data structure.
children to satisfy the max-heap property. The node indexed by largest, however,
now has the original value AŒi#, and thus the subtree rooted at largest might violate
the max-heap property. Consequently, we call M AX -H EAPIFY recursively on that
subtree.
The running time of M AX -H EAPIFY on a subtree of size n rooted at a given
node i is the ‚.1/ time to fix up the relationships among the elements AŒi#,
AŒL EFT .i/#, and AŒR IGHT .i/#, plus the time to run M AX -H EAPIFY on a subtree
rooted at one of the children of node i (assuming that the recursive call occurs).
The children’s subtrees each have size at most 2n=3—the worst case occurs when
the bottom level of the tree is exactly half full—and therefore we can describe the
running time of M AX -H EAPIFY by the recurrence
T .n/ $ T .2n=3/ C ‚.1/ :
156
Chapter 6 Heapsort
The solution to this recurrence, by case 2 of the master theorem (Theorem 4.1),
is T .n/ D O.lg n/. Alternatively, we can characterize the running time of M AX H EAPIFY on a node of height h as O.h/.
Exercises
6.2-1
Using Figure 6.2 as a model, illustrate the operation of M AX -H EAPIFY .A; 3/ on
the array A D h27; 17; 3; 16; 13; 10; 1; 5; 7; 12; 4; 8; 9; 0i.
6.2-2
Starting with the procedure M AX -H EAPIFY, write pseudocode for the procedure
M IN -H EAPIFY .A; i/, which performs the corresponding manipulation on a minheap. How does the running time of M IN -H EAPIFY compare to that of M AX H EAPIFY?
6.2-3
What is the effect of calling M AX -H EAPIFY .A; i/ when the element AŒi# is larger
than its children?
6.2-4
What is the effect of calling M AX -H EAPIFY .A; i/ for i > A:heap-size=2?
6.2-5
The code for M AX -H EAPIFY is quite efficient in terms of constant factors, except
possibly for the recursive call in line 10, which might cause some compilers to
produce inefficient code. Write an efficient M AX -H EAPIFY that uses an iterative
control construct (a loop) instead of recursion.
6.2-6
Show that the worst-case running time of M AX -H EAPIFY on a heap of size n
is !.lg n/. (Hint: For a heap with n nodes, give node values that cause M AX H EAPIFY to be called recursively at every node on a simple path from the root
down to a leaf.)
6.3
Building a heap
We can use the procedure M AX -H EAPIFY in a bottom-up manner to convert an
array AŒ1 : : n#, where n D A:length, into a max-heap. By Exercise 6.1-7, the
elements in the subarray AŒ.bn=2cC1/ : : n# are all leaves of the tree, and so each is
6.3 Building a heap
157
a 1-element heap to begin with. The procedure B UILD -M AX -H EAP goes through
the remaining nodes of the tree and runs M AX -H EAPIFY on each one.
B UILD -M AX -H EAP .A/
1 A:heap-size D A:length
2 for i D bA:length=2c downto 1
3
M AX -H EAPIFY .A; i/
Figure 6.3 shows an example of the action of B UILD -M AX -H EAP.
To show why B UILD -M AX -H EAP works correctly, we use the following loop
invariant:
At the start of each iteration of the for loop of lines 2–3, each node i C 1;
i C 2; : : : ; n is the root of a max-heap.
We need to show that this invariant is true prior to the first loop iteration, that each
iteration of the loop maintains the invariant, and that the invariant provides a useful
property to show correctness when the loop terminates.
Initialization: Prior to the first iteration of the loop, i D bn=2c. Each node
bn=2c C 1; bn=2c C 2; : : : ; n is a leaf and is thus the root of a trivial max-heap.
Maintenance: To see that each iteration maintains the loop invariant, observe that
the children of node i are numbered higher than i. By the loop invariant, therefore, they are both roots of max-heaps. This is precisely the condition required
for the call M AX -H EAPIFY .A; i/ to make node i a max-heap root. Moreover,
the M AX -H EAPIFY call preserves the property that nodes i C 1; i C 2; : : : ; n
are all roots of max-heaps. Decrementing i in the for loop update reestablishes
the loop invariant for the next iteration.
Termination: At termination, i D 0. By the loop invariant, each node 1; 2; : : : ; n
is the root of a max-heap. In particular, node 1 is.
We can compute a simple upper bound on the running time of BUILD -M AX H EAP as follows. Each call to M AX -H EAPIFY costs O.lg n/ time, and B UILD M AX -H EAP makes O.n/ such calls. Thus, the running time is O.n lg n/. This
upper bound, though correct, is not asymptotically tight.
We can derive a tighter bound by observing that the time for M AX -H EAPIFY to
run at a node varies with the height of the node in the tree, and the heights of most
nodes are small. Our tighter analysis relies on the properties
˙
)that an n-element heap
has height blg nc (see Exercise 6.1-2) and at most n=2hC1 nodes of any height h
(see Exercise 6.3-3).
The time required by M AX -H EAPIFY when called on a node of height h is O.h/,
and so we can express the total cost of B UILD -M AX -H EAP as being bounded from
above by
158
Chapter 6 Heapsort
A 4
1
3
2 16 9 10 14 8
7
1
4
2
1
4
8
2
14
5
7
4
2
3
7
9
10
8
3
6
i 16
9
1
10
i
8
2
14
(a)
1
4
9
5
10
8
8
14
2
9
5
10
8
7
16
4
8
2
14
9
8
9
1
i
7
10
4
8
14
2
i
4
2
1
9
5
10
8
7
3
6
9
(e)
10
9
7
3
(d)
1
4
7
10
3
6
16
1
5
10
3
(c)
i
2
3
6
16
9
7
1
4
1
4
3
(b)
1
2
6
16
7
3
10
16
2
7
3
14
4
8
2
8
9
4
5
10
1
3
6
7
9
10
7
3
(f)
Figure 6.3 The operation of B UILD -M AX -H EAP, showing the data structure before the call to
M AX -H EAPIFY in line 3 of B UILD -M AX -H EAP. (a) A 10-element input array A and the binary tree it represents. The figure shows that the loop index i refers to node 5 before the call
M AX -H EAPIFY.A; i/. (b) The data structure that results. The loop index i for the next iteration
refers to node 4. (c)–(e) Subsequent iterations of the for loop in B UILD -M AX -H EAP. Observe that
whenever M AX -H EAPIFY is called on a node, the two subtrees of that node are both max-heaps.
(f) The max-heap after B UILD -M AX -H EAP finishes.
6.4 The heapsort algorithm
blg nc l
X
hD0
n m
O.h/ D O n
D
1=2
.1 ! 1=2/2
2hC1
159
blg nc
X h
hD0
2h
!
:
We evalaute the last summation by substituting x D 1=2 in the formula (A.8),
yielding
1
X
h
hD0
2h
D 2:
Thus, we can bound the running time of B UILD -M AX -H EAP as
!
!
blg nc
1
X h
X
h
O n
D O n
2h
2h
hD0
hD0
D O.n/ :
Hence, we can build a max-heap from an unordered array in linear time.
We can build a min-heap by the procedure B UILD -M IN -H EAP, which is the
same as B UILD -M AX -H EAP but with the call to M AX -H EAPIFY in line 3 replaced
by a call to M IN -H EAPIFY (see Exercise 6.2-2). B UILD -M IN -H EAP produces a
min-heap from an unordered linear array in linear time.
Exercises
6.3-1
Using Figure 6.3 as a model, illustrate the operation of B UILD -M AX -H EAP on the
array A D h5; 3; 17; 10; 84; 19; 6; 22; 9i.
6.3-2
Why do we want the loop index i in line 2 of B UILD -M AX -H EAP to decrease from
bA:length=2c to 1 rather than increase from 1 to bA:length=2c?
6.3-3
˙
)
Show that there are at most n=2hC1 nodes of height h in any n-element heap.
6.4 The heapsort algorithm
The heapsort algorithm starts by using B UILD -M AX -H EAP to build a max-heap
on the input array AŒ1 : : n#, where n D A:length. Since the maximum element
of the array is stored at the root AŒ1#, we can put it into its correct final position
160
Chapter 6 Heapsort
by exchanging it with AŒn#. If we now discard node n from the heap—and we
can do so by simply decrementing A:heap-size—we observe that the children of
the root remain max-heaps, but the new root element might violate the max-heap
property. All we need to do to restore the max-heap property, however, is call
M AX -H EAPIFY .A; 1/, which leaves a max-heap in AŒ1 : : n ! 1#. The heapsort
algorithm then repeats this process for the max-heap of size n ! 1 down to a heap
of size 2. (See Exercise 6.4-2 for a precise loop invariant.)
H EAPSORT .A/
1 B UILD -M AX -H EAP .A/
2 for i D A:length downto 2
3
exchange AŒ1# with AŒi#
4
A:heap-size D A:heap-size ! 1
5
M AX -H EAPIFY .A; 1/
Figure 6.4 shows an example of the operation of H EAPSORT after line 1 has built
the initial max-heap. The figure shows the max-heap before the first iteration of
the for loop of lines 2–5 and after each iteration.
The H EAPSORT procedure takes time O.n lg n/, since the call to B UILD -M AX H EAP takes time O.n/ and each of the n ! 1 calls to M AX -H EAPIFY takes
time O.lg n/.
Exercises
6.4-1
Using Figure 6.4 as a model, illustrate the operation of H EAPSORT on the array
A D h5; 13; 2; 25; 7; 17; 20; 8; 4i.
6.4-2
Argue the correctness of H EAPSORT using the following loop invariant:
At the start of each iteration of the for loop of lines 2–5, the subarray
AŒ1 : : i# is a max-heap containing the i smallest elements of AŒ1 : : n#, and
the subarray AŒi C 1 : : n# contains the n ! i largest elements of AŒ1 : : n#,
sorted.
6.4-3
What is the running time of H EAPSORT on an array A of length n that is already
sorted in increasing order? What about decreasing order?
6.4-4
Show that the worst-case running time of H EAPSORT is !.n lg n/.
6.4 The heapsort algorithm
161
16
14
14
10
8
2
7
4
1
9
8
3
10
4
2
(a)
7
1
16 i
9
i 10
14
3
16
1
2
10
14
14
16
8
1
i 9
3
1
10
2
14
16
10
(g)
7
8
14
16
2 3
4 7
8 i
9
(f)
2
1
i 4
3
(c)
4
(e)
2
9
1
7
2
3
16
7
3
2
10
i
14 16
3
4
(d)
i 7
4
2
7
4
1
3
9
8
8
7
8
(b)
9
4
10
1
9
(h)
3 i
4
10
7
14
16
8
9
(i)
1
i 2
3
4
10
7
14
16
8
(j)
9
A
1
8 9 10 14 16
(k)
Figure 6.4 The operation of H EAPSORT . (a) The max-heap data structure just after B UILD -M AX H EAP has built it in line 1. (b)–(j) The max-heap just after each call of M AX -H EAPIFY in line 5,
showing the value of i at that time. Only lightly shaded nodes remain in the heap. (k) The resulting
sorted array A.
162
Chapter 6 Heapsort
6.4-5 ?
Show that when all elements are distinct, the best-case running time of H EAPSORT
is !.n lg n/.
6.5
Priority queues
Heapsort is an excellent algorithm, but a good implementation of quicksort, presented in Chapter 7, usually beats it in practice. Nevertheless, the heap data structure itself has many uses. In this section, we present one of the most popular applications of a heap: as an efficient priority queue. As with heaps, priority queues
come in two forms: max-priority queues and min-priority queues. We will focus
here on how to implement max-priority queues, which are in turn based on maxheaps; Exercise 6.5-3 asks you to write the procedures for min-priority queues.
A priority queue is a data structure for maintaining a set S of elements, each
with an associated value called a key. A max-priority queue supports the following
operations:
I NSERT .S; x/ inserts the element x into the set S, which is equivalent to the operation S D S [ fxg.
M AXIMUM .S/ returns the element of S with the largest key.
E XTRACT-M AX .S/ removes and returns the element of S with the largest key.
I NCREASE -K EY .S; x; k/ increases the value of element x’s key to the new value k,
which is assumed to be at least as large as x’s current key value.
Among their other applications, we can use max-priority queues to schedule
jobs on a shared computer. The max-priority queue keeps track of the jobs to
be performed and their relative priorities. When a job is finished or interrupted,
the scheduler selects the highest-priority job from among those pending by calling
E XTRACT-M AX. The scheduler can add a new job to the queue at any time by
calling I NSERT.
Alternatively, a min-priority queue supports the operations I NSERT, M INIMUM,
E XTRACT-M IN, and D ECREASE -K EY. A min-priority queue can be used in an
event-driven simulator. The items in the queue are events to be simulated, each
with an associated time of occurrence that serves as its key. The events must be
simulated in order of their time of occurrence, because the simulation of an event
can cause other events to be simulated in the future. The simulation program calls
E XTRACT-M IN at each step to choose the next event to simulate. As new events are
produced, the simulator inserts them into the min-priority queue by calling I NSERT.
6.5 Priority queues
163
We shall see other uses for min-priority queues, highlighting the D ECREASE -K EY
operation, in Chapters 23 and 24.
Not surprisingly, we can use a heap to implement a priority queue. In a given application, such as job scheduling or event-driven simulation, elements of a priority
queue correspond to objects in the application. We often need to determine which
application object corresponds to a given priority-queue element, and vice versa.
When we use a heap to implement a priority queue, therefore, we often need to
store a handle to the corresponding application object in each heap element. The
exact makeup of the handle (such as a pointer or an integer) depends on the application. Similarly, we need to store a handle to the corresponding heap element
in each application object. Here, the handle would typically be an array index.
Because heap elements change locations within the array during heap operations,
an actual implementation, upon relocating a heap element, would also have to update the array index in the corresponding application object. Because the details
of accessing application objects depend heavily on the application and its implementation, we shall not pursue them here, other than noting that in practice, these
handles do need to be correctly maintained.
Now we discuss how to implement the operations of a max-priority queue. The
procedure H EAP -M AXIMUM implements the M AXIMUM operation in ‚.1/ time.
H EAP -M AXIMUM .A/
1 return AŒ1#
The procedure H EAP -E XTRACT-M AX implements the E XTRACT-M AX operation. It is similar to the for loop body (lines 3–5) of the H EAPSORT procedure.
H EAP -E XTRACT-M AX .A/
1 if A:heap-size < 1
2
error “heap underflow”
3 max D AŒ1#
4 AŒ1# D AŒA:heap-size#
5 A:heap-size D A:heap-size ! 1
6 M AX -H EAPIFY .A; 1/
7 return max
The running time of H EAP -E XTRACT-M AX is O.lg n/, since it performs only a
constant amount of work on top of the O.lg n/ time for M AX -H EAPIFY.
The procedure H EAP -I NCREASE -K EY implements the I NCREASE -K EY operation. An index i into the array identifies the priority-queue element whose key we
wish to increase. The procedure first updates the key of element AŒi# to its new
value. Because increasing the key of AŒi# might violate the max-heap property,
164
Chapter 6 Heapsort
the procedure then, in a manner reminiscent of the insertion loop (lines 5–7) of
I NSERTION -S ORT from Section 2.1, traverses a simple path from this node toward
the root to find a proper place for the newly increased key. As H EAP -I NCREASE K EY traverses this path, it repeatedly compares an element to its parent, exchanging their keys and continuing if the element’s key is larger, and terminating if the element’s key is smaller, since the max-heap property now holds. (See Exercise 6.5-5
for a precise loop invariant.)
H EAP -I NCREASE -K EY .A; i; key/
1 if key < AŒi#
2
error “new key is smaller than current key”
3 AŒi# D key
4 while i > 1 and AŒPARENT.i/# < AŒi#
5
exchange AŒi# with AŒPARENT.i/#
6
i D PARENT.i/
Figure 6.5 shows an example of a H EAP -I NCREASE -K EY operation. The running
time of H EAP -I NCREASE -K EY on an n-element heap is O.lg n/, since the path
traced from the node updated in line 3 to the root has length O.lg n/.
The procedure M AX -H EAP -I NSERT implements the I NSERT operation. It takes
as an input the key of the new element to be inserted into max-heap A. The procedure first expands the max-heap by adding to the tree a new leaf whose key is !1.
Then it calls H EAP -I NCREASE -K EY to set the key of this new node to its correct
value and maintain the max-heap property.
M AX -H EAP -I NSERT .A; key/
1 A:heap-size D A:heap-size C 1
2 AŒA:heap-size# D !1
3 H EAP -I NCREASE -K EY .A; A:heap-size; key/
The running time of M AX -H EAP -I NSERT on an n-element heap is O.lg n/.
In summary, a heap can support any priority-queue operation on a set of size n
in O.lg n/ time.
Exercises
6.5-1
Illustrate the operation of H EAP -E XTRACT-M AX on the heap A D h15; 13; 9; 5;
12; 8; 7; 4; 0; 6; 2; 1i.
Introduction
Sets are as fundamental to computer science as they are to mathematics. Whereas
mathematical sets are unchanging, the sets manipulated by algorithms can grow,
shrink, or otherwise change over time. We call such sets dynamic. The next five
chapters present some basic techniques for representing finite dynamic sets and
manipulating them on a computer.
Algorithms may require several different types of operations to be performed on
sets. For example, many algorithms need only the ability to insert elements into,
delete elements from, and test membership in a set. We call a dynamic set that
supports these operations a dictionary. Other algorithms require more complicated
operations. For example, min-priority queues, which Chapter 6 introduced in the
context of the heap data structure, support the operations of inserting an element
into and extracting the smallest element from a set. The best way to implement a
dynamic set depends upon the operations that must be supported.
Elements of a dynamic set
In a typical implementation of a dynamic set, each element is represented by an
object whose attributes can be examined and manipulated if we have a pointer to
the object. (Section 10.3 discusses the implementation of objects and pointers in
programming environments that do not contain them as basic data types.) Some
kinds of dynamic sets assume that one of the object’s attributes is an identifying
key. If the keys are all different, we can think of the dynamic set as being a set
of key values. The object may contain satellite data, which are carried around in
other object attributes but are otherwise unused by the set implementation. It may
230
Part III Data Structures
also have attributes that are manipulated by the set operations; these attributes may
contain data or pointers to other objects in the set.
Some dynamic sets presuppose that the keys are drawn from a totally ordered
set, such as the real numbers, or the set of all words under the usual alphabetic
ordering. A total ordering allows us to define the minimum element of the set, for
example, or to speak of the next element larger than a given element in a set.
Operations on dynamic sets
Operations on a dynamic set can be grouped into two categories: queries, which
simply return information about the set, and modifying operations, which change
the set. Here is a list of typical operations. Any specific application will usually
require only a few of these to be implemented.
S EARCH .S; k/
A query that, given a set S and a key value k, returns a pointer x to an element
in S such that x:key D k, or NIL if no such element belongs to S.
I NSERT .S; x/
A modifying operation that augments the set S with the element pointed to
by x. We usually assume that any attributes in element x needed by the set
implementation have already been initialized.
D ELETE .S; x/
A modifying operation that, given a pointer x to an element in the set S, removes x from S. (Note that this operation takes a pointer to an element x, not
a key value.)
M INIMUM .S/
A query on a totally ordered set S that returns a pointer to the element of S
with the smallest key.
M AXIMUM .S/
A query on a totally ordered set S that returns a pointer to the element of S
with the largest key.
S UCCESSOR .S; x/
A query that, given an element x whose key is from a totally ordered set S,
returns a pointer to the next larger element in S, or NIL if x is the maximum
element.
P REDECESSOR .S; x/
A query that, given an element x whose key is from a totally ordered set S,
returns a pointer to the next smaller element in S, or NIL if x is the minimum
element.
Part III
Data Structures
231
In some situations, we can extend the queries S UCCESSOR and P REDECESSOR
so that they apply to sets with nondistinct keys. For a set on n keys, the normal
presumption is that a call to M INIMUM followed by n ! 1 calls to S UCCESSOR
enumerates the elements in the set in sorted order.
We usually measure the time taken to execute a set operation in terms of the size
of the set. For example, Chapter 13 describes a data structure that can support any
of the operations listed above on a set of size n in time O.lg n/.
Overview of Part III
Chapters 10–14 describe several data structures that we can use to implement
dynamic sets; we shall use many of these later to construct efficient algorithms
for a variety of problems. We already saw another important data structure—the
heap—in Chapter 6.
Chapter 10 presents the essentials of working with simple data structures such
as stacks, queues, linked lists, and rooted trees. It also shows how to implement
objects and pointers in programming environments that do not support them as
primitives. If you have taken an introductory programming course, then much of
this material should be familiar to you.
Chapter 11 introduces hash tables, which support the dictionary operations I N SERT, D ELETE, and S EARCH . In the worst case, hashing requires ‚.n/ time to perform a S EARCH operation, but the expected time for hash-table operations is O.1/.
The analysis of hashing relies on probability, but most of the chapter requires no
background in the subject.
Binary search trees, which are covered in Chapter 12, support all the dynamicset operations listed above. In the worst case, each operation takes ‚.n/ time on a
tree with n elements, but on a randomly built binary search tree, the expected time
for each operation is O.lg n/. Binary search trees serve as the basis for many other
data structures.
Chapter 13 introduces red-black trees, which are a variant of binary search trees.
Unlike ordinary binary search trees, red-black trees are guaranteed to perform well:
operations take O.lg n/ time in the worst case. A red-black tree is a balanced search
tree; Chapter 18 in Part V presents another kind of balanced search tree, called a
B-tree. Although the mechanics of red-black trees are somewhat intricate, you can
glean most of their properties from the chapter without studying the mechanics in
detail. Nevertheless, you probably will find walking through the code to be quite
instructive.
In Chapter 14, we show how to augment red-black trees to support operations
other than the basic ones listed above. First, we augment them so that we can
dynamically maintain order statistics for a set of keys. Then, we augment them in
a different way to maintain intervals of real numbers.
10
Elementary Data Structures
In this chapter, we examine the representation of dynamic sets by simple data structures that use pointers. Although we can construct many complex data structures
using pointers, we present only the rudimentary ones: stacks, queues, linked lists,
and rooted trees. We also show ways to synthesize objects and pointers from arrays.
10.1 Stacks and queues
Stacks and queues are dynamic sets in which the element removed from the set
by the D ELETE operation is prespecified. In a stack, the element deleted from
the set is the one most recently inserted: the stack implements a last-in, first-out,
or LIFO, policy. Similarly, in a queue, the element deleted is always the one that
has been in the set for the longest time: the queue implements a first-in, first-out,
or FIFO, policy. There are several efficient ways to implement stacks and queues
on a computer. In this section we show how to use a simple array to implement
each.
Stacks
The I NSERT operation on a stack is often called P USH, and the D ELETE operation, which does not take an element argument, is often called P OP. These names
are allusions to physical stacks, such as the spring-loaded stacks of plates used
in cafeterias. The order in which plates are popped from the stack is the reverse
of the order in which they were pushed onto the stack, since only the top plate is
accessible.
As Figure 10.1 shows, we can implement a stack of at most n elements with
an array SŒ1 : : n". The array has an attribute S:top that indexes the most recently
10.1 Stacks and queues
1
2
S 15 6
3
2
4
9
5
6
7
233
1
2
S 15 6
3
2
4
6
9 17 3
S:top D 4
(a)
5
7
S:top D 6
(b)
1
2
S 15 6
3
2
4
5
6
9 17 3
7
S:top D 5
(c)
Figure 10.1 An array implementation of a stack S. Stack elements appear only in the lightly shaded
positions. (a) Stack S has 4 elements. The top element is 9. (b) Stack S after the calls P USH.S; 17/
and P USH.S; 3/. (c) Stack S after the call P OP.S/ has returned the element 3, which is the one most
recently pushed. Although element 3 still appears in the array, it is no longer in the stack; the top is
element 17.
inserted element. The stack consists of elements SŒ1 : : S:top", where SŒ1" is the
element at the bottom of the stack and SŒS:top" is the element at the top.
When S:top D 0, the stack contains no elements and is empty. We can test to
see whether the stack is empty by query operation S TACK -E MPTY. If we attempt
to pop an empty stack, we say the stack underflows, which is normally an error.
If S:top exceeds n, the stack overflows. (In our pseudocode implementation, we
don’t worry about stack overflow.)
We can implement each of the stack operations with just a few lines of code:
S TACK -E MPTY .S/
1 if S:top == 0
2
return TRUE
3 else return FALSE
P USH .S; x/
1 S:top D S:top C 1
2 SŒS:top" D x
P OP.S/
1 if S TACK -E MPTY .S/
2
error “underflow”
3 else S:top D S:top ! 1
4
return SŒS:top C 1"
Figure 10.1 shows the effects of the modifying operations P USH and P OP. Each of
the three stack operations takes O.1/ time.
234
Chapter 10 Elementary Data Structures
(a)
Q
1
2
3
4
5
6
7
8
15 6
9
9
10 11 12
8
Q:head D 7
(b)
1
Q 3
2
5
3
4
5
(c)
Q 3
2
5
3
4
Q:tail D 3
7
8
15 6
Q:tail D 12
9
10 11 12
9
10 11 12
9
8
4 17
Q:head D 7
Q:tail D 3
1
6
4
5
6
7
8
15 6
9
8
4 17
Q:head D 8
Figure 10.2 A queue implemented using an array QŒ1 : : 12". Queue elements appear only in the
lightly shaded positions. (a) The queue has 5 elements, in locations QŒ7 : : 11". (b) The configuration
of the queue after the calls E NQUEUE.Q; 17/, E NQUEUE.Q; 3/, and E NQUEUE.Q; 5/. (c) The
configuration of the queue after the call D EQUEUE.Q/ returns the key value 15 formerly at the
head of the queue. The new head has key 6.
Queues
We call the I NSERT operation on a queue E NQUEUE, and we call the D ELETE
operation D EQUEUE; like the stack operation P OP, D EQUEUE takes no element argument. The FIFO property of a queue causes it to operate like a line of customers
waiting to pay a cashier. The queue has a head and a tail. When an element is enqueued, it takes its place at the tail of the queue, just as a newly arriving customer
takes a place at the end of the line. The element dequeued is always the one at
the head of the queue, like the customer at the head of the line who has waited the
longest.
Figure 10.2 shows one way to implement a queue of at most n ! 1 elements
using an array QŒ1 : : n". The queue has an attribute Q:head that indexes, or points
to, its head. The attribute Q:tail indexes the next location at which a newly arriving element will be inserted into the queue. The elements in the queue reside in
locations Q:head; Q:head C 1; : : : ; Q:tail ! 1, where we “wrap around” in the
sense that location 1 immediately follows location n in a circular order. When
Q:head D Q:tail, the queue is empty. Initially, we have Q:head D Q:tail D 1.
If we attempt to dequeue an element from an empty queue, the queue underflows.
10.1 Stacks and queues
235
When Q:head D Q:tail C 1, the queue is full, and if we attempt to enqueue an
element, then the queue overflows.
In our procedures E NQUEUE and D EQUEUE, we have omitted the error checking
for underflow and overflow. (Exercise 10.1-4 asks you to supply code that checks
for these two error conditions.) The pseudocode assumes that n D Q:length.
E NQUEUE .Q; x/
1 QŒQ:tail" D x
2 if Q:tail == Q:length
3
Q:tail D 1
4 else Q:tail D Q:tail C 1
D EQUEUE .Q/
1 x D QŒQ:head"
2 if Q:head == Q:length
3
Q:head D 1
4 else Q:head D Q:head C 1
5 return x
Figure 10.2 shows the effects of the E NQUEUE and D EQUEUE operations. Each
operation takes O.1/ time.
Exercises
10.1-1
Using Figure 10.1 as a model, illustrate the result of each operation in the sequence
P USH .S; 4/, P USH .S; 1/, P USH .S; 3/, P OP.S/, P USH .S; 8/, and P OP.S/ on an
initially empty stack S stored in array SŒ1 : : 6".
10.1-2
Explain how to implement two stacks in one array AŒ1 : : n" in such a way that
neither stack overflows unless the total number of elements in both stacks together
is n. The P USH and P OP operations should run in O.1/ time.
10.1-3
Using Figure 10.2 as a model, illustrate the result of each operation in the
sequence E NQUEUE .Q; 4/, E NQUEUE .Q; 1/, E NQUEUE .Q; 3/, D EQUEUE .Q/,
E NQUEUE .Q; 8/, and D EQUEUE .Q/ on an initially empty queue Q stored in
array QŒ1 : : 6".
10.1-4
Rewrite E NQUEUE and D EQUEUE to detect underflow and overflow of a queue.
236
Chapter 10 Elementary Data Structures
10.1-5
Whereas a stack allows insertion and deletion of elements at only one end, and a
queue allows insertion at one end and deletion at the other end, a deque (doubleended queue) allows insertion and deletion at both ends. Write four O.1/-time
procedures to insert elements into and delete elements from both ends of a deque
implemented by an array.
10.1-6
Show how to implement a queue using two stacks. Analyze the running time of the
queue operations.
10.1-7
Show how to implement a stack using two queues. Analyze the running time of the
stack operations.
10.2 Linked lists
A linked list is a data structure in which the objects are arranged in a linear order.
Unlike an array, however, in which the linear order is determined by the array
indices, the order in a linked list is determined by a pointer in each object. Linked
lists provide a simple, flexible representation for dynamic sets, supporting (though
not necessarily efficiently) all the operations listed on page 230.
As shown in Figure 10.3, each element of a doubly linked list L is an object with
an attribute key and two other pointer attributes: next and pre&. The object may
also contain other satellite data. Given an element x in the list, x:next points to its
successor in the linked list, and x:pre& points to its predecessor. If x:pre& D NIL,
the element x has no predecessor and is therefore the first element, or head, of
the list. If x:next D NIL , the element x has no successor and is therefore the last
element, or tail, of the list. An attribute L:head points to the first element of the
list. If L:head D NIL , the list is empty.
A list may have one of several forms. It may be either singly linked or doubly
linked, it may be sorted or not, and it may be circular or not. If a list is singly
linked, we omit the pre& pointer in each element. If a list is sorted, the linear order
of the list corresponds to the linear order of keys stored in elements of the list; the
minimum element is then the head of the list, and the maximum element is the
tail. If the list is unsorted, the elements can appear in any order. In a circular list,
the pre& pointer of the head of the list points to the tail, and the next pointer of
the tail of the list points to the head. We can think of a circular list as a ring of
10.2 Linked lists
237
prev
key
next
(a)
L:head
9
16
4
1
(b)
L:head
25
9
16
4
(c)
L:head
25
9
16
1
1
Figure 10.3 (a) A doubly linked list L representing the dynamic set f1; 4; 9; 16g. Each element in
the list is an object with attributes for the key and pointers (shown by arrows) to the next and previous
objects. The next attribute of the tail and the pre& attribute of the head are NIL , indicated by a diagonal
slash. The attribute L: head points to the head. (b) Following the execution of L IST-I NSERT.L; x/,
where x: key D 25, the linked list has a new object with key 25 as the new head. This new object
points to the old head with key 9. (c) The result of the subsequent call L IST-D ELETE.L; x/, where x
points to the object with key 4.
elements. In the remainder of this section, we assume that the lists with which we
are working are unsorted and doubly linked.
Searching a linked list
The procedure L IST-S EARCH .L; k/ finds the first element with key k in list L
by a simple linear search, returning a pointer to this element. If no object with
key k appears in the list, then the procedure returns NIL. For the linked list in
Figure 10.3(a), the call L IST-S EARCH .L; 4/ returns a pointer to the third element,
and the call L IST-S EARCH .L; 7/ returns NIL.
L IST-S EARCH .L; k/
1 x D L:head
2 while x ¤ NIL and x:key ¤ k
3
x D x:next
4 return x
To search a list of n objects, the L IST-S EARCH procedure takes ‚.n/ time in the
worst case, since it may have to search the entire list.
Inserting into a linked list
Given an element x whose key attribute has already been set, the L IST-I NSERT
procedure “splices” x onto the front of the linked list, as shown in Figure 10.3(b).
238
Chapter 10 Elementary Data Structures
L IST-I NSERT .L; x/
1 x:next D L:head
2 if L:head ¤ NIL
3
L:head:pre& D x
4 L:head D x
5 x:pre& D NIL
(Recall that our attribute notation can cascade, so that L:head:pre& denotes the
pre& attribute of the object that L:head points to.) The running time for L ISTI NSERT on a list of n elements is O.1/.
Deleting from a linked list
The procedure L IST-D ELETE removes an element x from a linked list L. It must
be given a pointer to x, and it then “splices” x out of the list by updating pointers.
If we wish to delete an element with a given key, we must first call L IST-S EARCH
to retrieve a pointer to the element.
L IST-D ELETE .L; x/
1 if x:pre& ¤ NIL
2
x:pre&:next D x:next
3 else L:head D x:next
4 if x:next ¤ NIL
5
x:next:pre& D x:pre&
Figure 10.3(c) shows how an element is deleted from a linked list. L IST-D ELETE
runs in O.1/ time, but if we wish to delete an element with a given key, ‚.n/ time
is required in the worst case because we must first call L IST-S EARCH to find the
element.
Sentinels
The code for L IST-D ELETE would be simpler if we could ignore the boundary
conditions at the head and tail of the list:
L IST-D ELETE0 .L; x/
1 x:pre&:next D x:next
2 x:next:pre& D x:pre&
A sentinel is a dummy object that allows us to simplify boundary conditions. For
example, suppose that we provide with list L an object L:nil that represents NIL
10.2 Linked lists
239
(a)
L:nil
(b)
L:nil
9
16
4
1
(c)
L:nil
25
9
16
4
(d)
L:nil
25
9
16
4
1
Figure 10.4 A circular, doubly linked list with a sentinel. The sentinel L: nil appears between the
head and tail. The attribute L: head is no longer needed, since we can access the head of the list
by L: nil: next. (a) An empty list. (b) The linked list from Figure 10.3(a), with key 9 at the head and
key 1 at the tail. (c) The list after executing L IST-I NSERT0 .L; x/, where x: key D 25. The new object
becomes the head of the list. (d) The list after deleting the object with key 1. The new tail is the
object with key 4.
but has all the attributes of the other objects in the list. Wherever we have a reference to NIL in list code, we replace it by a reference to the sentinel L:nil. As
shown in Figure 10.4, this change turns a regular doubly linked list into a circular, doubly linked list with a sentinel, in which the sentinel L:nil lies between the
head and tail. The attribute L:nil:next points to the head of the list, and L:nil:pre&
points to the tail. Similarly, both the next attribute of the tail and the pre& attribute of the head point to L:nil. Since L:nil:next points to the head, we can
eliminate the attribute L:head altogether, replacing references to it by references
to L:nil:next. Figure 10.4(a) shows that an empty list consists of just the sentinel,
and both L:nil:next and L:nil:pre& point to L:nil.
The code for L IST-S EARCH remains the same as before, but with the references
to NIL and L:head changed as specified above:
L IST-S EARCH0 .L; k/
1 x D L:nil:next
2 while x ¤ L:nil and x:key ¤ k
3
x D x:next
4 return x
We use the two-line procedure L IST-D ELETE 0 from before to delete an element
from the list. The following procedure inserts an element into the list:
240
Chapter 10 Elementary Data Structures
L IST-I NSERT0 .L; x/
1 x:next D L:nil:next
2 L:nil:next:pre& D x
3 L:nil:next D x
4 x:pre& D L:nil
Figure 10.4 shows the effects of L IST-I NSERT 0 and L IST-D ELETE 0 on a sample list.
Sentinels rarely reduce the asymptotic time bounds of data structure operations,
but they can reduce constant factors. The gain from using sentinels within loops
is usually a matter of clarity of code rather than speed; the linked list code, for
example, becomes simpler when we use sentinels, but we save only O.1/ time in
the L IST-I NSERT 0 and L IST-D ELETE 0 procedures. In other situations, however, the
use of sentinels helps to tighten the code in a loop, thus reducing the coefficient of,
say, n or n2 in the running time.
We should use sentinels judiciously. When there are many small lists, the extra
storage used by their sentinels can represent significant wasted memory. In this
book, we use sentinels only when they truly simplify the code.
Exercises
10.2-1
Can you implement the dynamic-set operation I NSERT on a singly linked list
in O.1/ time? How about D ELETE?
10.2-2
Implement a stack using a singly linked list L. The operations P USH and P OP
should still take O.1/ time.
10.2-3
Implement a queue by a singly linked list L. The operations E NQUEUE and D E QUEUE should still take O.1/ time.
10.2-4
As written, each loop iteration in the L IST-S EARCH 0 procedure requires two tests:
one for x ¤ L:nil and one for x:key ¤ k. Show how to eliminate the test for
x ¤ L:nil in each iteration.
10.2-5
Implement the dictionary operations I NSERT, D ELETE, and S EARCH using singly
linked, circular lists. What are the running times of your procedures?
10.3 Implementing pointers and objects
241
10.2-6
The dynamic-set operation U NION takes two disjoint sets S1 and S2 as input, and
it returns a set S D S1 [ S2 consisting of all the elements of S1 and S2 . The
sets S1 and S2 are usually destroyed by the operation. Show how to support U NION
in O.1/ time using a suitable list data structure.
10.2-7
Give a ‚.n/-time nonrecursive procedure that reverses a singly linked list of n
elements. The procedure should use no more than constant storage beyond that
needed for the list itself.
10.2-8 ?
Explain how to implement doubly linked lists using only one pointer value x:np per
item instead of the usual two (next and pre&). Assume that all pointer values can be
interpreted as k-bit integers, and define x:np to be x:np D x:next XOR x:pre&,
the k-bit “exclusive-or” of x:next and x:pre&. (The value NIL is represented by 0.)
Be sure to describe what information you need to access the head of the list. Show
how to implement the S EARCH, I NSERT, and D ELETE operations on such a list.
Also show how to reverse such a list in O.1/ time.
10.3 Implementing pointers and objects
How do we implement pointers and objects in languages that do not provide them?
In this section, we shall see two ways of implementing linked data structures without an explicit pointer data type. We shall synthesize objects and pointers from
arrays and array indices.
A multiple-array representation of objects
We can represent a collection of objects that have the same attributes by using an
array for each attribute. As an example, Figure 10.5 shows how we can implement
the linked list of Figure 10.3(a) with three arrays. The array key holds the values
of the keys currently in the dynamic set, and the pointers reside in the arrays next
and pre&. For a given array index x, the array entries keyŒx", nextŒx", and pre&Œx"
represent an object in the linked list. Under this interpretation, a pointer x is simply
a common index into the key, next, and pre& arrays.
In Figure 10.3(a), the object with key 4 follows the object with key 16 in the
linked list. In Figure 10.5, key 4 appears in keyŒ2", and key 16 appears in keyŒ5",
and so nextŒ5" D 2 and pre&Œ2" D 5. Although the constant NIL appears in the next
242
Chapter 10 Elementary Data Structures
L
7
next
key
prev
1
2
3
4
5
3
1
2
4
5
2
16
7
6
7
8
5
9
Figure 10.5 The linked list of Figure 10.3(a) represented by the arrays key, next, and pre&. Each
vertical slice of the arrays represents a single object. Stored pointers correspond to the array indices
shown at the top; the arrows show how to interpret them. Lightly shaded object positions contain list
elements. The variable L keeps the index of the head.
attribute of the tail and the pre& attribute of the head, we usually use an integer
(such as 0 or !1) that cannot possibly represent an actual index into the arrays. A
variable L holds the index of the head of the list.
A single-array representation of objects
The words in a computer memory are typically addressed by integers from 0
to M ! 1, where M is a suitably large integer. In many programming languages,
an object occupies a contiguous set of locations in the computer memory. A pointer
is simply the address of the first memory location of the object, and we can address
other memory locations within the object by adding an offset to the pointer.
We can use the same strategy for implementing objects in programming environments that do not provide explicit pointer data types. For example, Figure 10.6
shows how to use a single array A to store the linked list from Figures 10.3(a)
and 10.5. An object occupies a contiguous subarray AŒj : : k". Each attribute of
the object corresponds to an offset in the range from 0 to k ! j , and a pointer to
the object is the index j . In Figure 10.6, the offsets corresponding to key, next, and
pre& are 0, 1, and 2, respectively. To read the value of i:pre&, given a pointer i, we
add the value i of the pointer to the offset 2, thus reading AŒi C 2".
The single-array representation is flexible in that it permits objects of different
lengths to be stored in the same array. The problem of managing such a heterogeneous collection of objects is more difficult than the problem of managing a homogeneous collection, where all objects have the same attributes. Since most of the
data structures we shall consider are composed of homogeneous elements, it will
be sufficient for our purposes to use the multiple-array representation of objects.
10.3 Implementing pointers and objects
L
1
19
A
2
3
4
5
6
7
4
7 13 1
8
9
4
243
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
16 4 19
9 13
key prev
next
Figure 10.6 The linked list of Figures 10.3(a) and 10.5 represented in a single array A. Each list
element is an object that occupies a contiguous subarray of length 3 within the array. The three
attributes key, next, and pre& correspond to the offsets 0, 1, and 2, respectively, within each object.
A pointer to an object is the index of the first element of the object. Objects containing list elements
are lightly shaded, and arrows show the list ordering.
Allocating and freeing objects
To insert a key into a dynamic set represented by a doubly linked list, we must allocate a pointer to a currently unused object in the linked-list representation. Thus,
it is useful to manage the storage of objects not currently used in the linked-list
representation so that one can be allocated. In some systems, a garbage collector is responsible for determining which objects are unused. Many applications,
however, are simple enough that they can bear responsibility for returning an unused object to a storage manager. We shall now explore the problem of allocating
and freeing (or deallocating) homogeneous objects using the example of a doubly
linked list represented by multiple arrays.
Suppose that the arrays in the multiple-array representation have length m and
that at some moment the dynamic set contains n " m elements. Then n objects
represent elements currently in the dynamic set, and the remaining m!n objects are
free; the free objects are available to represent elements inserted into the dynamic
set in the future.
We keep the free objects in a singly linked list, which we call the free list. The
free list uses only the next array, which stores the next pointers within the list.
The head of the free list is held in the global variable free. When the dynamic
set represented by linked list L is nonempty, the free list may be intertwined with
list L, as shown in Figure 10.7. Note that each object in the representation is either
in list L or in the free list, but not in both.
The free list acts like a stack: the next object allocated is the last one freed. We
can use a list implementation of the stack operations P USH and P OP to implement
the procedures for allocating and freeing objects, respectively. We assume that the
global variable free used in the following procedures points to the first element of
the free list.
244
Chapter 10 Elementary Data Structures
free
4
L
7
1
2
3
4
5
next
key
prev
3
1
2
4
5
6
7
8
8
2 1
16
7
5
9
6
(a)
free
5
L
4
1
next
key
prev
4
free
8
L
4
1
next
key
prev
2
3
4
5
6
3
4
5
7 2 1
1 25 16
2
7
7
8
5 6
9
4
(b)
2
3
5
6
7
8
3
4
7
7 8
1 25
2
1
2
9
4
6
(c)
Figure 10.7 The effect of the A LLOCATE -O BJECT and F REE -O BJECT procedures. (a) The list
of Figure 10.5 (lightly shaded) and a free list (heavily shaded). Arrows show the free-list structure.
(b) The result of calling A LLOCATE -O BJECT./ (which returns index 4), setting keyŒ4" to 25, and
calling L IST-I NSERT.L; 4/. The new free-list head is object 8, which had been nextŒ4" on the free
list. (c) After executing L IST-D ELETE.L; 5/, we call F REE -O BJECT.5/. Object 5 becomes the new
free-list head, with object 8 following it on the free list.
A LLOCATE -O BJECT ./
1 if free == NIL
2
error “out of space”
3 else x D free
4
free D x:next
5
return x
F REE -O BJECT .x/
1 x:next D free
2 free D x
The free list initially contains all n unallocated objects. Once the free list has been
exhausted, running the A LLOCATE -O BJECT procedure signals an error. We can
even service several linked lists with just a single free list. Figure 10.8 shows two
linked lists and a free list intertwined through key, next, and pre& arrays.
The two procedures run in O.1/ time, which makes them quite practical. We
can modify them to work for any homogeneous collection of objects by letting any
one of the attributes in the object act like a next attribute in the free list.
10.3 Implementing pointers and objects
free 10
L2 9
L1 3
1
2
3
4
5
6
245
7
next 5
6 8
2 1
key k1 k2 k3
k5 k6 k7
prev 7 6
1 3 9
8
9
7
k9
10
4
Figure 10.8 Two linked lists, L1 (lightly shaded) and L2 (heavily shaded), and a free list (darkened) intertwined.
Exercises
10.3-1
Draw a picture of the sequence h13; 4; 8; 19; 5; 11i stored as a doubly linked list
using the multiple-array representation. Do the same for the single-array representation.
10.3-2
Write the procedures A LLOCATE -O BJECT and F REE -O BJECT for a homogeneous
collection of objects implemented by the single-array representation.
10.3-3
Why don’t we need to set or reset the pre& attributes of objects in the implementation of the A LLOCATE -O BJECT and F REE -O BJECT procedures?
10.3-4
It is often desirable to keep all elements of a doubly linked list compact in storage,
using, for example, the first m index locations in the multiple-array representation.
(This is the case in a paged, virtual-memory computing environment.) Explain
how to implement the procedures A LLOCATE -O BJECT and F REE -O BJECT so that
the representation is compact. Assume that there are no pointers to elements of the
linked list outside the list itself. (Hint: Use the array implementation of a stack.)
10.3-5
Let L be a doubly linked list of length n stored in arrays key, pre&, and next of
length m. Suppose that these arrays are managed by A LLOCATE -O BJECT and
F REE -O BJECT procedures that keep a doubly linked free list F . Suppose further
that of the m items, exactly n are on list L and m ! n are on the free list. Write
a procedure C OMPACTIFY-L IST .L; F / that, given the list L and the free list F ,
moves the items in L so that they occupy array positions 1; 2; : : : ; n and adjusts the
free list F so that it remains correct, occupying array positions nC1; nC2; : : : ; m.
The running time of your procedure should be ‚.n/, and it should use only a
constant amount of extra space. Argue that your procedure is correct.
246
Chapter 10 Elementary Data Structures
10.4 Representing rooted trees
The methods for representing lists given in the previous section extend to any homogeneous data structure. In this section, we look specifically at the problem of
representing rooted trees by linked data structures. We first look at binary trees,
and then we present a method for rooted trees in which nodes can have an arbitrary
number of children.
We represent each node of a tree by an object. As with linked lists, we assume
that each node contains a key attribute. The remaining attributes of interest are
pointers to other nodes, and they vary according to the type of tree.
Binary trees
Figure 10.9 shows how we use the attributes p, left, and right to store pointers to
the parent, left child, and right child of each node in a binary tree T . If x:p D NIL,
then x is the root. If node x has no left child, then x:left D NIL , and similarly for
the right child. The root of the entire tree T is pointed to by the attribute T:root. If
T:root D NIL, then the tree is empty.
Rooted trees with unbounded branching
We can extend the scheme for representing a binary tree to any class of trees in
which the number of children of each node is at most some constant k: we replace
the left and right attributes by child 1 ; child 2 ; : : : ; child k . This scheme no longer
works when the number of children of a node is unbounded, since we do not know
how many attributes (arrays in the multiple-array representation) to allocate in advance. Moreover, even if the number of children k is bounded by a large constant
but most nodes have a small number of children, we may waste a lot of memory.
Fortunately, there is a clever scheme to represent trees with arbitrary numbers of
children. It has the advantage of using only O.n/ space for any n-node rooted tree.
The left-child, right-sibling representation appears in Figure 10.10. As before,
each node contains a parent pointer p, and T:root points to the root of tree T .
Instead of having a pointer to each of its children, however, each node x has only
two pointers:
1. x:left-child points to the leftmost child of node x, and
2. x:right-sibling points to the sibling of x immediately to its right.
If node x has no children, then x:left-child D NIL, and if node x is the rightmost
child of its parent, then x:right-sibling D NIL.
10.4 Representing rooted trees
247
T:root
Figure 10.9 The representation of a binary tree T . Each node x has the attributes x: p (top), x: left
(lower left), and x: right (lower right). The key attributes are not shown.
T:root
Figure 10.10 The left-child, right-sibling representation of a tree T . Each node x has attributes x: p
(top), x: left-child (lower left), and x: right-sibling (lower right). The key attributes are not shown.
248
Chapter 10 Elementary Data Structures
Other tree representations
We sometimes represent rooted trees in other ways. In Chapter 6, for example,
we represented a heap, which is based on a complete binary tree, by a single array
plus the index of the last node in the heap. The trees that appear in Chapter 21 are
traversed only toward the root, and so only the parent pointers are present; there
are no pointers to children. Many other schemes are possible. Which scheme is
best depends on the application.
Exercises
10.4-1
Draw the binary tree rooted at index 6 that is represented by the following attributes:
index
1
2
3
4
5
6
7
8
9
10
key
12
15
4
10
2
18
7
14
21
5
left
7
8
10
5
right
3
NIL
NIL
9
NIL
NIL
NIL
NIL
NIL
NIL
1
6
NIL
4
2
NIL
10.4-2
Write an O.n/-time recursive procedure that, given an n-node binary tree, prints
out the key of each node in the tree.
10.4-3
Write an O.n/-time nonrecursive procedure that, given an n-node binary tree,
prints out the key of each node in the tree. Use a stack as an auxiliary data structure.
10.4-4
Write an O.n/-time procedure that prints all the keys of an arbitrary rooted tree
with n nodes, where the tree is stored using the left-child, right-sibling representation.
10.4-5 ?
Write an O.n/-time nonrecursive procedure that, given an n-node binary tree,
prints out the key of each node. Use no more than constant extra space outside
Problems for Chapter 10
249
of the tree itself and do not modify the tree, even temporarily, during the procedure.
10.4-6 ?
The left-child, right-sibling representation of an arbitrary rooted tree uses three
pointers in each node: left-child, right-sibling, and parent. From any node, its
parent can be reached and identified in constant time and all its children can be
reached and identified in time linear in the number of children. Show how to use
only two pointers and one boolean value in each node so that the parent of a node
or all of its children can be reached and identified in time linear in the number of
children.
Problems
10-1 Comparisons among lists
For each of the four types of lists in the following table, what is the asymptotic
worst-case running time for each dynamic-set operation listed?
S EARCH .L; k/
I NSERT .L; x/
D ELETE .L; x/
S UCCESSOR .L; x/
P REDECESSOR .L; x/
M INIMUM .L/
M AXIMUM .L/
unsorted,
singly
linked
sorted,
singly
linked
unsorted,
doubly
linked
sorted,
doubly
linked
250
Chapter 10 Elementary Data Structures
10-2 Mergeable heaps using linked lists
A mergeable heap supports the following operations: M AKE -H EAP (which creates
an empty mergeable heap), I NSERT, M INIMUM, E XTRACT-M IN, and U NION.1
Show how to implement mergeable heaps using linked lists in each of the following
cases. Try to make each operation as efficient as possible. Analyze the running
time of each operation in terms of the size of the dynamic set(s) being operated on.
a. Lists are sorted.
b. Lists are unsorted.
c. Lists are unsorted, and dynamic sets to be merged are disjoint.
10-3 Searching a sorted compact list
Exercise 10.3-4 asked how we might maintain an n-element list compactly in the
first n positions of an array. We shall assume that all keys are distinct and that the
compact list is also sorted, that is, keyŒi" < keyŒnextŒi"" for all i D 1; 2; : : : ; n such
that nextŒi" ¤ NIL . We will also assume that we have a variable L that contains
the index of the first element on the list. Under these assumptions, you will show
p
that we can use the following randomized algorithm to search the list in O. n/
expected time.
C OMPACT-L IST-S EARCH .L; n; k/
1 i DL
2 while i ¤ NIL and keyŒi" < k
3
j D R ANDOM.1; n/
4
if keyŒi" < keyŒj " and keyŒj " " k
5
i Dj
6
if keyŒi" == k
7
return i
8
i D nextŒi"
9 if i == NIL or keyŒi" > k
10
return NIL
11 else return i
If we ignore lines 3–7 of the procedure, we have an ordinary algorithm for
searching a sorted linked list, in which index i points to each position of the list in
1 Because we have defined a mergeable heap to support M INIMUM and E XTRACT-M IN , we can also
refer to it as a mergeable min-heap. Alternatively, if it supported M AXIMUM and E XTRACT-M AX,
it would be a mergeable max-heap.
Problems for Chapter 10
251
turn. The search terminates once the index i “falls off” the end of the list or once
keyŒi" $ k. In the latter case, if keyŒi" D k, clearly we have found a key with the
value k. If, however, keyŒi" > k, then we will never find a key with the value k,
and so terminating the search was the right thing to do.
Lines 3–7 attempt to skip ahead to a randomly chosen position j . Such a skip
benefits us if keyŒj " is larger than keyŒi" and no larger than k; in such a case, j
marks a position in the list that i would have to reach during an ordinary list search.
Because the list is compact, we know that any choice of j between 1 and n indexes
some object in the list rather than a slot on the free list.
Instead of analyzing the performance of C OMPACT-L IST-S EARCH directly, we
shall analyze a related algorithm, C OMPACT-L IST-S EARCH 0 , which executes two
separate loops. This algorithm takes an additional parameter t which determines
an upper bound on the number of iterations of the first loop.
C OMPACT-L IST-S EARCH0 .L; n; k; t/
1 i DL
2 for q D 1 to t
3
j D R ANDOM.1; n/
4
if keyŒi" < keyŒj " and keyŒj " " k
5
i Dj
6
if keyŒi" == k
7
return i
8 while i ¤ NIL and keyŒi" < k
9
i D nextŒi"
10 if i == NIL or keyŒi" > k
11
return NIL
12 else return i
To compare the execution of the algorithms C OMPACT-L IST-S EARCH .L; n; k/
and C OMPACT-L IST-S EARCH 0 .L; n; k; t/, assume that the sequence of integers returned by the calls of R ANDOM.1; n/ is the same for both algorithms.
a. Suppose that C OMPACT-L IST-S EARCH .L; n; k/ takes t iterations of the while
loop of lines 2–8. Argue that C OMPACT-L IST-S EARCH 0 .L; n; k; t/ returns the
same answer and that the total number of iterations of both the for and while
loops within C OMPACT-L IST-S EARCH 0 is at least t.
In the call C OMPACT-L IST-S EARCH 0 .L; n; k; t/, let X t be the random variable that
describes the distance in the linked list (that is, through the chain of next pointers)
from position i to the desired key k after t iterations of the for loop of lines 2–7
have occurred.
252
Chapter 10 Elementary Data Structures
b. Argue that the expected running time of C OMPACT-L IST-S EARCH 0 .L; n; k; t/
is O.t C E ŒX t "/.
Pn
c. Show that E ŒX t " " rD1 .1 ! r=n/t . (Hint: Use equation (C.25).)
d. Show that
Pn"1
rD0 r
t
" nt C1 =.t C 1/.
e. Prove that E ŒX t " " n=.t C 1/.
f. Show that C OMPACT-L IST-S EARCH 0 .L; n; k; t/ runs in O.t C n=t/ expected
time.
p
g. Conclude that C OMPACT-L IST-S EARCH runs in O. n/ expected time.
h. Why do we assume that all keys are distinct in C OMPACT-L IST-S EARCH? Argue that random skips do not necessarily help asymptotically when the list contains repeated key values.
Chapter notes
Aho, Hopcroft, and Ullman [6] and Knuth [209] are excellent references for elementary data structures. Many other texts cover both basic data structures and their
implementation in a particular programming language. Examples of these types of
textbooks include Goodrich and Tamassia [147], Main [241], Shaffer [311], and
Weiss [352, 353, 354]. Gonnet [145] provides experimental data on the performance of many data-structure operations.
The origin of stacks and queues as data structures in computer science is unclear, since corresponding notions already existed in mathematics and paper-based
business practices before the introduction of digital computers. Knuth [209] cites
A. M. Turing for the development of stacks for subroutine linkage in 1947.
Pointer-based data structures also seem to be a folk invention. According to
Knuth, pointers were apparently used in early computers with drum memories. The
A-1 language developed by G. M. Hopper in 1951 represented algebraic formulas
as binary trees. Knuth credits the IPL-II language, developed in 1956 by A. Newell,
J. C. Shaw, and H. A. Simon, for recognizing the importance and promoting the
use of pointers. Their IPL-III language, developed in 1957, included explicit stack
operations.
11
Hash Tables
Many applications require a dynamic set that supports only the dictionary operations I NSERT, S EARCH, and D ELETE. For example, a compiler that translates a
programming language maintains a symbol table, in which the keys of elements
are arbitrary character strings corresponding to identifiers in the language. A hash
table is an effective data structure for implementing dictionaries. Although searching for an element in a hash table can take as long as searching for an element in a
linked list—‚.n/ time in the worst case—in practice, hashing performs extremely
well. Under reasonable assumptions, the average time to search for an element in
a hash table is O.1/.
A hash table generalizes the simpler notion of an ordinary array. Directly addressing into an ordinary array makes effective use of our ability to examine an
arbitrary position in an array in O.1/ time. Section 11.1 discusses direct addressing in more detail. We can take advantage of direct addressing when we can afford
to allocate an array that has one position for every possible key.
When the number of keys actually stored is small relative to the total number of
possible keys, hash tables become an effective alternative to directly addressing an
array, since a hash table typically uses an array of size proportional to the number
of keys actually stored. Instead of using the key as an array index directly, the array
index is computed from the key. Section 11.2 presents the main ideas, focusing on
“chaining” as a way to handle “collisions,” in which more than one key maps to the
same array index. Section 11.3 describes how we can compute array indices from
keys using hash functions. We present and analyze several variations on the basic
theme. Section 11.4 looks at “open addressing,” which is another way to deal with
collisions. The bottom line is that hashing is an extremely effective and practical
technique: the basic dictionary operations require only O.1/ time on the average.
Section 11.5 explains how “perfect hashing” can support searches in O.1/ worstcase time, when the set of keys being stored is static (that is, when the set of keys
never changes once stored).
254
Chapter 11 Hash Tables
11.1 Direct-address tables
Direct addressing is a simple technique that works well when the universe U of
keys is reasonably small. Suppose that an application needs a dynamic set in which
each element has a key drawn from the universe U D f0; 1; : : : ; m ! 1g, where m
is not too large. We shall assume that no two elements have the same key.
To represent the dynamic set, we use an array, or direct-address table, denoted
by T Œ0 : : m ! 1", in which each position, or slot, corresponds to a key in the universe U . Figure 11.1 illustrates the approach; slot k points to an element in the set
with key k. If the set contains no element with key k, then T Œk" D NIL.
The dictionary operations are trivial to implement:
D IRECT-A DDRESS -S EARCH .T; k/
1 return T Œk"
D IRECT-A DDRESS -I NSERT .T; x/
1 T Œx:key" D x
D IRECT-A DDRESS -D ELETE .T; x/
1 T Œx:key" D NIL
Each of these operations takes only O.1/ time.
T
9
1
U
(universe of keys)
0
6
7
4
K
(actual
keys)
2
5
0
1
2
3
4
5
3
6
8
key
satellite data
2
3
5
7
8
9
8
Figure 11.1 How to implement a dynamic set by a direct-address table T . Each key in the universe
U D f0; 1; : : : ; 9g corresponds to an index in the table. The set K D f2; 3; 5; 8g of actual keys
determines the slots in the table that contain pointers to elements. The other slots, heavily shaded,
contain NIL .
11.1 Direct-address tables
255
For some applications, the direct-address table itself can hold the elements in the
dynamic set. That is, rather than storing an element’s key and satellite data in an
object external to the direct-address table, with a pointer from a slot in the table to
the object, we can store the object in the slot itself, thus saving space. We would
use a special key within an object to indicate an empty slot. Moreover, it is often
unnecessary to store the key of the object, since if we have the index of an object
in the table, we have its key. If keys are not stored, however, we must have some
way to tell whether the slot is empty.
Exercises
11.1-1
Suppose that a dynamic set S is represented by a direct-address table T of length m.
Describe a procedure that finds the maximum element of S. What is the worst-case
performance of your procedure?
11.1-2
A bit vector is simply an array of bits (0s and 1s). A bit vector of length m takes
much less space than an array of m pointers. Describe how to use a bit vector
to represent a dynamic set of distinct elements with no satellite data. Dictionary
operations should run in O.1/ time.
11.1-3
Suggest how to implement a direct-address table in which the keys of stored elements do not need to be distinct and the elements can have satellite data. All
three dictionary operations (I NSERT, D ELETE, and S EARCH) should run in O.1/
time. (Don’t forget that D ELETE takes as an argument a pointer to an object to be
deleted, not a key.)
11.1-4 ?
We wish to implement a dictionary by using direct addressing on a huge array. At
the start, the array entries may contain garbage, and initializing the entire array
is impractical because of its size. Describe a scheme for implementing a directaddress dictionary on a huge array. Each stored object should use O.1/ space;
the operations S EARCH, I NSERT, and D ELETE should take O.1/ time each; and
initializing the data structure should take O.1/ time. (Hint: Use an additional array,
treated somewhat like a stack whose size is the number of keys actually stored in
the dictionary, to help determine whether a given entry in the huge array is valid or
not.)
256
Chapter 11 Hash Tables
11.2 Hash tables
The downside of direct addressing is obvious: if the universe U is large, storing
a table T of size jU j may be impractical, or even impossible, given the memory
available on a typical computer. Furthermore, the set K of keys actually stored
may be so small relative to U that most of the space allocated for T would be
wasted.
When the set K of keys stored in a dictionary is much smaller than the universe U of all possible keys, a hash table requires much less storage than a directaddress table. Specifically, we can reduce the storage requirement to ‚.jKj/ while
we maintain the benefit that searching for an element in the hash table still requires
only O.1/ time. The catch is that this bound is for the average-case time, whereas
for direct addressing it holds for the worst-case time.
With direct addressing, an element with key k is stored in slot k. With hashing,
this element is stored in slot h.k/; that is, we use a hash function h to compute the
slot from the key k. Here, h maps the universe U of keys into the slots of a hash
table T Œ0 : : m ! 1":
h W U ! f0; 1; : : : ; m ! 1g ;
where the size m of the hash table is typically much less than jU j. We say that an
element with key k hashes to slot h.k/; we also say that h.k/ is the hash value of
key k. Figure 11.2 illustrates the basic idea. The hash function reduces the range
of array indices and hence the size of the array. Instead of a size of jU j, the array
can have size m.
T
U
(universe of keys)
K
(actual
keys)
k1
k4
k2
0
h(k1)
h(k4)
k5
h(k2) = h(k5)
k3
h(k3)
m–1
Figure 11.2 Using a hash function h to map keys to hash-table slots. Because keys k2 and k5 map
to the same slot, they collide.
11.2 Hash tables
257
T
U
(universe of keys)
K
(actual
keys)
k1
k4
k5
k2
k3
k8
k6
k1
k4
k6
k2
k7
k8
k5
k3
k7
Figure 11.3 Collision resolution by chaining. Each hash-table slot T Œj " contains a linked list of
all the keys whose hash value is j . For example, h.k1 / D h.k4 / and h.k5 / D h.k7 / D h.k2 /.
The linked list can be either singly or doubly linked; we show it as doubly linked because deletion is
faster that way.
There is one hitch: two keys may hash to the same slot. We call this situation
a collision. Fortunately, we have effective techniques for resolving the conflict
created by collisions.
Of course, the ideal solution would be to avoid collisions altogether. We might
try to achieve this goal by choosing a suitable hash function h. One idea is to
make h appear to be “random,” thus avoiding collisions or at least minimizing
their number. The very term “to hash,” evoking images of random mixing and
chopping, captures the spirit of this approach. (Of course, a hash function h must be
deterministic in that a given input k should always produce the same output h.k/.)
Because jU j > m, however, there must be at least two keys that have the same hash
value; avoiding collisions altogether is therefore impossible. Thus, while a welldesigned, “random”-looking hash function can minimize the number of collisions,
we still need a method for resolving the collisions that do occur.
The remainder of this section presents the simplest collision resolution technique, called chaining. Section 11.4 introduces an alternative method for resolving
collisions, called open addressing.
Collision resolution by chaining
In chaining, we place all the elements that hash to the same slot into the same
linked list, as Figure 11.3 shows. Slot j contains a pointer to the head of the list of
all stored elements that hash to j ; if there are no such elements, slot j contains NIL.
258
Chapter 11 Hash Tables
The dictionary operations on a hash table T are easy to implement when collisions are resolved by chaining:
C HAINED -H ASH -I NSERT .T; x/
1 insert x at the head of list T Œh.x:key/"
C HAINED -H ASH -S EARCH .T; k/
1 search for an element with key k in list T Œh.k/"
C HAINED -H ASH -D ELETE .T; x/
1 delete x from the list T Œh.x:key/"
The worst-case running time for insertion is O.1/. The insertion procedure is fast
in part because it assumes that the element x being inserted is not already present in
the table; if necessary, we can check this assumption (at additional cost) by searching for an element whose key is x:key before we insert. For searching, the worstcase running time is proportional to the length of the list; we shall analyze this
operation more closely below. We can delete an element in O.1/ time if the lists
are doubly linked, as Figure 11.3 depicts. (Note that C HAINED -H ASH -D ELETE
takes as input an element x and not its key k, so that we don’t have to search for x
first. If the hash table supports deletion, then its linked lists should be doubly linked
so that we can delete an item quickly. If the lists were only singly linked, then to
delete element x, we would first have to find x in the list T Œh.x:key/" so that we
could update the next attribute of x’s predecessor. With singly linked lists, both
deletion and searching would have the same asymptotic running times.)
Analysis of hashing with chaining
How well does hashing with chaining perform? In particular, how long does it take
to search for an element with a given key?
Given a hash table T with m slots that stores n elements, we define the load
factor ˛ for T as n=m, that is, the average number of elements stored in a chain.
Our analysis will be in terms of ˛, which can be less than, equal to, or greater
than 1.
The worst-case behavior of hashing with chaining is terrible: all n keys hash
to the same slot, creating a list of length n. The worst-case time for searching is
thus ‚.n/ plus the time to compute the hash function—no better than if we used
one linked list for all the elements. Clearly, we do not use hash tables for their
worst-case performance. (Perfect hashing, described in Section 11.5, does provide
good worst-case performance when the set of keys is static, however.)
The average-case performance of hashing depends on how well the hash function h distributes the set of keys to be stored among the m slots, on the average.
11.2 Hash tables
259
Section 11.3 discusses these issues, but for now we shall assume that any given
element is equally likely to hash into any of the m slots, independently of where
any other element has hashed to. We call this the assumption of simple uniform
hashing.
For j D 0; 1; : : : ; m ! 1, let us denote the length of the list T Œj " by nj , so that
n D n0 C n1 C # # # C nm"1 ;
(11.1)
and the expected value of nj is E Œnj " D ˛ D n=m.
We assume that O.1/ time suffices to compute the hash value h.k/, so that
the time required to search for an element with key k depends linearly on the
length nh.k/ of the list T Œh.k/". Setting aside the O.1/ time required to compute
the hash function and to access slot h.k/, let us consider the expected number of
elements examined by the search algorithm, that is, the number of elements in the
list T Œh.k/" that the algorithm checks to see whether any have a key equal to k. We
shall consider two cases. In the first, the search is unsuccessful: no element in the
table has key k. In the second, the search successfully finds an element with key k.
Theorem 11.1
In a hash table in which collisions are resolved by chaining, an unsuccessful search
takes average-case time ‚.1C˛/, under the assumption of simple uniform hashing.
Proof Under the assumption of simple uniform hashing, any key k not already
stored in the table is equally likely to hash to any of the m slots. The expected time
to search unsuccessfully for a key k is the expected time to search to the end of
list T Œh.k/", which has expected length E Œnh.k/ " D ˛. Thus, the expected number
of elements examined in an unsuccessful search is ˛, and the total time required
(including the time for computing h.k/) is ‚.1 C ˛/.
The situation for a successful search is slightly different, since each list is not
equally likely to be searched. Instead, the probability that a list is searched is proportional to the number of elements it contains. Nonetheless, the expected search
time still turns out to be ‚.1 C ˛/.
Theorem 11.2
In a hash table in which collisions are resolved by chaining, a successful search
takes average-case time ‚.1C˛/, under the assumption of simple uniform hashing.
Proof We assume that the element being searched for is equally likely to be any
of the n elements stored in the table. The number of elements examined during a
successful search for an element x is one more than the number of elements that
260
Chapter 11 Hash Tables
appear before x in x’s list. Because new elements are placed at the front of the
list, elements before x in the list were all inserted after x was inserted. To find
the expected number of elements examined, we take the average, over the n elements x in the table, of 1 plus the expected number of elements added to x’s list
after x was added to the list. Let xi denote the ith element inserted into the table, for i D 1; 2; : : : ; n, and let ki D xi :key. For keys ki and kj , we define the
indicator random variable Xij D I fh.ki / D h.kj /g. Under the assumption of simple uniform hashing, we have Pr fh.ki / D h.kj /g D 1=m, and so by Lemma 5.1,
E ŒXij " D 1=m. Thus, the expected number of elements examined in a successful
search is
" n
!#
n
X
1X
E
1C
Xij
n i D1
j Di C1
!
n
n
X
1X
D
1C
E ŒXij "
(by linearity of expectation)
n i D1
j Di C1
!
n
n
X
1X
1
D
1C
n i D1
m
j Di C1
n
1 X
D 1C
.n ! i/
nm i D1
!
n
n
X
1 X
D 1C
n!
i
nm i D1
i D1
%
&
1
n.n C 1/
2
D 1C
n !
(by equation (A.1))
nm
2
n!1
D 1C
2m
˛
˛
D 1C !
:
2 2n
Thus, the total time required for a successful search (including the time for computing the hash function) is ‚.2 C ˛=2 ! ˛=2n/ D ‚.1 C ˛/.
What does this analysis mean? If the number of hash-table slots is at least proportional to the number of elements in the table, we have n D O.m/ and, consequently, ˛ D n=m D O.m/=m D O.1/. Thus, searching takes constant time
on average. Since insertion takes O.1/ worst-case time and deletion takes O.1/
worst-case time when the lists are doubly linked, we can support all dictionary
operations in O.1/ time on average.
11.2 Hash tables
261
Exercises
11.2-1
Suppose we use a hash function h to hash n distinct keys into an array T of
length m. Assuming simple uniform hashing, what is the expected number of
collisions? More precisely, what is the expected cardinality of ffk; lg W k ¤ l and
h.k/ D h.l/g?
11.2-2
Demonstrate what happens when we insert the keys 5; 28; 19; 15; 20; 33; 12; 17; 10
into a hash table with collisions resolved by chaining. Let the table have 9 slots,
and let the hash function be h.k/ D k mod 9.
11.2-3
Professor Marley hypothesizes that he can obtain substantial performance gains by
modifying the chaining scheme to keep each list in sorted order. How does the professor’s modification affect the running time for successful searches, unsuccessful
searches, insertions, and deletions?
11.2-4
Suggest how to allocate and deallocate storage for elements within the hash table
itself by linking all unused slots into a free list. Assume that one slot can store
a flag and either one element plus a pointer or two pointers. All dictionary and
free-list operations should run in O.1/ expected time. Does the free list need to be
doubly linked, or does a singly linked free list suffice?
11.2-5
Suppose that we are storing a set of n keys into a hash table of size m. Show that if
the keys are drawn from a universe U with jU j > nm, then U has a subset of size n
consisting of keys that all hash to the same slot, so that the worst-case searching
time for hashing with chaining is ‚.n/.
11.2-6
Suppose we have stored n keys in a hash table of size m, with collisions resolved by
chaining, and that we know the length of each chain, including the length L of the
longest chain. Describe a procedure that selects a key uniformly at random from
among the keys in the hash table and returns it in expected time O.L # .1 C 1=˛//.
262
Chapter 11 Hash Tables
11.3 Hash functions
In this section, we discuss some issues regarding the design of good hash functions
and then present three schemes for their creation. Two of the schemes, hashing by
division and hashing by multiplication, are heuristic in nature, whereas the third
scheme, universal hashing, uses randomization to provide provably good performance.
What makes a good hash function?
A good hash function satisfies (approximately) the assumption of simple uniform
hashing: each key is equally likely to hash to any of the m slots, independently of
where any other key has hashed to. Unfortunately, we typically have no way to
check this condition, since we rarely know the probability distribution from which
the keys are drawn. Moreover, the keys might not be drawn independently.
Occasionally we do know the distribution. For example, if we know that the
keys are random real numbers k independently and uniformly distributed in the
range 0 " k < 1, then the hash function
h.k/ D bkmc
satisfies the condition of simple uniform hashing.
In practice, we can often employ heuristic techniques to create a hash function
that performs well. Qualitative information about the distribution of keys may be
useful in this design process. For example, consider a compiler’s symbol table, in
which the keys are character strings representing identifiers in a program. Closely
related symbols, such as pt and pts, often occur in the same program. A good
hash function would minimize the chance that such variants hash to the same slot.
A good approach derives the hash value in a way that we expect to be independent of any patterns that might exist in the data. For example, the “division method”
(discussed in Section 11.3.1) computes the hash value as the remainder when the
key is divided by a specified prime number. This method frequently gives good
results, assuming that we choose a prime number that is unrelated to any patterns
in the distribution of keys.
Finally, we note that some applications of hash functions might require stronger
properties than are provided by simple uniform hashing. For example, we might
want keys that are “close” in some sense to yield hash values that are far apart.
(This property is especially desirable when we are using linear probing, defined in
Section 11.4.) Universal hashing, described in Section 11.3.3, often provides the
desired properties.
11.3 Hash functions
263
Interpreting keys as natural numbers
Most hash functions assume that the universe of keys is the set N D f0; 1; 2; : : :g
of natural numbers. Thus, if the keys are not natural numbers, we find a way to
interpret them as natural numbers. For example, we can interpret a character string
as an integer expressed in suitable radix notation. Thus, we might interpret the
identifier pt as the pair of decimal integers .112; 116/, since p D 112 and t D 116
in the ASCII character set; then, expressed as a radix-128 integer, pt becomes
.112 # 128/ C 116 D 14452. In the context of a given application, we can usually
devise some such method for interpreting each key as a (possibly large) natural
number. In what follows, we assume that the keys are natural numbers.
11.3.1
The division method
In the division method for creating hash functions, we map a key k into one of m
slots by taking the remainder of k divided by m. That is, the hash function is
h.k/ D k mod m :
For example, if the hash table has size m D 12 and the key is k D 100, then
h.k/ D 4. Since it requires only a single division operation, hashing by division is
quite fast.
When using the division method, we usually avoid certain values of m. For
example, m should not be a power of 2, since if m D 2p , then h.k/ is just the p
lowest-order bits of k. Unless we know that all low-order p-bit patterns are equally
likely, we are better off designing the hash function to depend on all the bits of the
key. As Exercise 11.3-3 asks you to show, choosing m D 2p ! 1 when k is a
character string interpreted in radix 2p may be a poor choice, because permuting
the characters of k does not change its hash value.
A prime not too close to an exact power of 2 is often a good choice for m. For
example, suppose we wish to allocate a hash table, with collisions resolved by
chaining, to hold roughly n D 2000 character strings, where a character has 8 bits.
We don’t mind examining an average of 3 elements in an unsuccessful search, and
so we allocate a hash table of size m D 701. We could choose m D 701 because
it is a prime near 2000=3 but not near any power of 2. Treating each key k as an
integer, our hash function would be
h.k/ D k mod 701 :
11.3.2
The multiplication method
The multiplication method for creating hash functions operates in two steps. First,
we multiply the key k by a constant A in the range 0 < A < 1 and extract the
264
Chapter 11 Hash Tables
w bits
k
×
s D A # 2w
r1
r0
extract p bits
h.k/
Figure 11.4 The multiplication method of hashing. The w-bit representation of the key k is multiplied by the w-bit value s D A # 2w . The p highest-order bits of the lower w-bit half of the product
form the desired hash value h.k/.
fractional part of kA. Then, we multiply this value by m and take the floor of the
result. In short, the hash function is
h.k/ D bm .kA mod 1/c ;
where “kA mod 1” means the fractional part of kA, that is, kA ! bkAc.
An advantage of the multiplication method is that the value of m is not critical.
We typically choose it to be a power of 2 (m D 2p for some integer p), since we
can then easily implement the function on most computers as follows. Suppose
that the word size of the machine is w bits and that k fits into a single word. We
restrict A to be a fraction of the form s=2w , where s is an integer in the range
0 < s < 2w . Referring to Figure 11.4, we first multiply k by the w-bit integer
s D A # 2w . The result is a 2w-bit value r1 2w C r0 , where r1 is the high-order word
of the product and r0 is the low-order word of the product. The desired p-bit hash
value consists of the p most significant bits of r0 .
Although this method works with any value of the constant A, it works better
with some values than with others. The optimal choice depends on the characteristics of the data being hashed. Knuth [211] suggests that
p
(11.2)
A % . 5 ! 1/=2 D 0:6180339887 : : :
is likely to work reasonably well.
As an example, suppose we have k D 123456, p D 14, m D 214 D 16384,
and w D 32. Adapting Knuth’spsuggestion, we choose A to be the fraction of the
form s=232 that is closest to . 5 ! 1/=2, so that A D 2654435769=232 . Then
k # s D 327706022297664 D .76300 # 232 / C 17612864, and so r1 D 76300
and r0 D 17612864. The 14 most significant bits of r0 yield the value h.k/ D 67.
11.3 Hash functions
?
11.3.3
265
Universal hashing
If a malicious adversary chooses the keys to be hashed by some fixed hash function,
then the adversary can choose n keys that all hash to the same slot, yielding an average retrieval time of ‚.n/. Any fixed hash function is vulnerable to such terrible
worst-case behavior; the only effective way to improve the situation is to choose
the hash function randomly in a way that is independent of the keys that are actually
going to be stored. This approach, called universal hashing, can yield provably
good performance on average, no matter which keys the adversary chooses.
In universal hashing, at the beginning of execution we select the hash function
at random from a carefully designed class of functions. As in the case of quicksort, randomization guarantees that no single input will always evoke worst-case
behavior. Because we randomly select the hash function, the algorithm can behave differently on each execution, even for the same input, guaranteeing good
average-case performance for any input. Returning to the example of a compiler’s
symbol table, we find that the programmer’s choice of identifiers cannot now cause
consistently poor hashing performance. Poor performance occurs only when the
compiler chooses a random hash function that causes the set of identifiers to hash
poorly, but the probability of this situation occurring is small and is the same for
any set of identifiers of the same size.
Let H be a finite collection of hash functions that map a given universe U of
keys into the range f0; 1; : : : ; m ! 1g. Such a collection is said to be universal
if for each pair of distinct keys k; l 2 U , the number of hash functions h 2 H
for which h.k/ D h.l/ is at most jH j =m. In other words, with a hash function
randomly chosen from H , the chance of a collision between distinct keys k and l
is no more than the chance 1=m of a collision if h.k/ and h.l/ were randomly and
independently chosen from the set f0; 1; : : : ; m ! 1g.
The following theorem shows that a universal class of hash functions gives good
average-case behavior. Recall that ni denotes the length of list T Œi".
Theorem 11.3
Suppose that a hash function h is chosen randomly from a universal collection of
hash functions and has been used to hash n keys into a table T of size m, using chaining to resolve collisions. If key k is not in the table, then the expected
length E Œnh.k/ " of the list that key k hashes to is at most the load factor ˛ D n=m.
If key k is in the table, then the expected length E Œnh.k/ " of the list containing key k
is at most 1 C ˛.
Proof We note that the expectations here are over the choice of the hash function and do not depend on any assumptions about the distribution of the keys.
For each pair k and l of distinct keys, define the indicator random variable
266
Chapter 11 Hash Tables
Xkl D I fh.k/ D h.l/g. Since by the definition of a universal collection of hash
functions, a single pair of keys collides with probability at most 1=m, we have
Pr fh.k/ D h.l/g " 1=m. By Lemma 5.1, therefore, we have E ŒXkl " " 1=m.
Next we define, for each key k, the random variable Yk that equals the number
of keys other than k that hash to the same slot as k, so that
X
Yk D
Xkl :
l2T
l¤k
Thus we have
2X
E ŒYk " D E4
l2T
l¤k
D
X
"
X 1
3
Xkl 5
E ŒXkl "
(by linearity of expectation)
l2T
l¤k
l2T
l¤k
m
:
The remainder of the proof depends on whether key k is in table T .
!
!
If k 62 T , then nh.k/ D Yk and jfl W l 2 T and l ¤ kgj D n. Thus E Œnh.k/ " D
E ŒYk " " n=m D ˛.
If k 2 T , then because key k appears in list T Œh.k/" and the count Yk does not
include key k, we have nh.k/ D Yk C 1 and jfl W l 2 T and l ¤ kgj D n ! 1.
Thus E Œnh.k/ " D E ŒYk " C 1 " .n ! 1/=m C 1 D 1 C ˛ ! 1=m < 1 C ˛.
The following corollary says universal hashing provides the desired payoff: it
has now become impossible for an adversary to pick a sequence of operations that
forces the worst-case running time. By cleverly randomizing the choice of hash
function at run time, we guarantee that we can process every sequence of operations
with a good average-case running time.
Corollary 11.4
Using universal hashing and collision resolution by chaining in an initially empty
table with m slots, it takes expected time ‚.n/ to handle any sequence of n I NSERT,
S EARCH, and D ELETE operations containing O.m/ I NSERT operations.
Proof Since the number of insertions is O.m/, we have n D O.m/ and so
˛ D O.1/. The I NSERT and D ELETE operations take constant time and, by Theorem 11.3, the expected time for each S EARCH operation is O.1/. By linearity of
11.3 Hash functions
267
expectation, therefore, the expected time for the entire sequence of n operations
is O.n/. Since each operation takes !.1/ time, the ‚.n/ bound follows.
Designing a universal class of hash functions
It is quite easy to design a universal class of hash functions, as a little number
theory will help us prove. You may wish to consult Chapter 31 first if you are
unfamiliar with number theory.
We begin by choosing a prime number p large enough so that every possible
key k is in the range 0 to p ! 1, inclusive. Let Zp denote the set f0; 1; : : : ; p ! 1g,
and let Zp# denote the set f1; 2; : : : ; p ! 1g. Since p is prime, we can solve equations modulo p with the methods given in Chapter 31. Because we assume that the
size of the universe of keys is greater than the number of slots in the hash table, we
have p > m.
We now define the hash function hab for any a 2 Zp# and any b 2 Zp using a
linear transformation followed by reductions modulo p and then modulo m:
hab .k/ D ..ak C b/ mod p/ mod m :
(11.3)
For example, with p D 17 and m D 6, we have h3;4 .8/ D 5. The family of all
such hash functions is
˚
+
Hpm D hab W a 2 Zp# and b 2 Zp :
(11.4)
Each hash function hab maps Zp to Zm . This class of hash functions has the nice
property that the size m of the output range is arbitrary—not necessarily prime—a
feature which we shall use in Section 11.5. Since we have p ! 1 choices for a
and p choices for b, the collection Hpm contains p.p ! 1/ hash functions.
Theorem 11.5
The class Hpm of hash functions defined by equations (11.3) and (11.4) is universal.
Proof Consider two distinct keys k and l from Zp , so that k ¤ l. For a given
hash function hab we let
r D .ak C b/ mod p ;
s D .al C b/ mod p :
We first note that r ¤ s. Why? Observe that
r ! s & a.k ! l/ .mod p/ :
It follows that r ¤ s because p is prime and both a and .k ! l/ are nonzero
modulo p, and so their product must also be nonzero modulo p by Theorem 31.6.
Therefore, when computing any hab 2 Hpm , distinct inputs k and l map to distinct
268
Chapter 11 Hash Tables
values r and s modulo p; there are no collisions yet at the “mod p level.” Moreover,
each of the possible p.p!1/ choices for the pair .a; b/ with a ¤ 0 yields a different
resulting pair .r; s/ with r ¤ s, since we can solve for a and b given r and s:
#
$
a D .r ! s/..k ! l/"1 mod p/ mod p ;
b D .r ! ak/ mod p ;
where ..k ! l/"1 mod p/ denotes the unique multiplicative inverse, modulo p,
of k ! l. Since there are only p.p ! 1/ possible pairs .r; s/ with r ¤ s, there
is a one-to-one correspondence between pairs .a; b/ with a ¤ 0 and pairs .r; s/
with r ¤ s. Thus, for any given pair of inputs k and l, if we pick .a; b/ uniformly
at random from Zp# ' Zp , the resulting pair .r; s/ is equally likely to be any pair of
distinct values modulo p.
Therefore, the probability that distinct keys k and l collide is equal to the probability that r & s .mod m/ when r and s are randomly chosen as distinct values
modulo p. For a given value of r, of the p ! 1 possible remaining values for s, the
number of values s such that s ¤ r and s & r .mod m/ is at most
dp=me ! 1 " ..p C m ! 1/=m/ ! 1 (by inequality (3.6))
D .p ! 1/=m :
The probability that s collides with r when reduced modulo m is at most
..p ! 1/=m/=.p ! 1/ D 1=m.
Therefore, for any pair of distinct values k; l 2 Zp ,
Pr fhab .k/ D hab .l/g " 1=m ;
so that Hpm is indeed universal.
Exercises
11.3-1
Suppose we wish to search a linked list of length n, where each element contains
a key k along with a hash value h.k/. Each key is a long character string. How
might we take advantage of the hash values when searching the list for an element
with a given key?
11.3-2
Suppose that we hash a string of r characters into m slots by treating it as a
radix-128 number and then using the division method. We can easily represent
the number m as a 32-bit computer word, but the string of r characters, treated as
a radix-128 number, takes many words. How can we apply the division method to
compute the hash value of the character string without using more than a constant
number of words of storage outside the string itself?
11.4 Open addressing
269
11.3-3
Consider a version of the division method in which h.k/ D k mod m, where
m D 2p ! 1 and k is a character string interpreted in radix 2p . Show that if we
can derive string x from string y by permuting its characters, then x and y hash to
the same value. Give an example of an application in which this property would be
undesirable in a hash function.
11.3-4
Consider a hash table of sizepm D 1000 and a corresponding hash function h.k/ D
bm .kA mod 1/c for A D . 5 ! 1/=2. Compute the locations to which the keys
61, 62, 63, 64, and 65 are mapped.
11.3-5 ?
Define a family H of hash functions from a finite set U to a finite set B to be
!-universal if for all pairs of distinct elements k and l in U ,
Pr fh.k/ D h.l/g " % ;
where the probability is over the choice of the hash function h drawn at random
from the family H . Show that an %-universal family of hash functions must have
%$
1
1
!
:
jBj jU j
11.3-6 ?
Let U be the set of n-tuples of values drawn from Zp , and let B D Zp , where p
is prime. Define the hash function hb W U ! B for b 2 Zp on an input n-tuple
ha0 ; a1 ; : : : ; an"1 i from U as
!
n"1
X
hb .ha0 ; a1 ; : : : ; an"1 i/ D
aj b j mod p ;
j D0
and let H D fhb W b 2 Zp g. Argue that H is ..n ! 1/=p/-universal according to
the definition of %-universal in Exercise 11.3-5. (Hint: See Exercise 31.4-4.)
11.4 Open addressing
In open addressing, all elements occupy the hash table itself. That is, each table
entry contains either an element of the dynamic set or NIL. When searching for
an element, we systematically examine table slots until either we find the desired
element or we have ascertained that the element is not in the table. No lists and
270
Chapter 11 Hash Tables
no elements are stored outside the table, unlike in chaining. Thus, in open addressing, the hash table can “fill up” so that no further insertions can be made; one
consequence is that the load factor ˛ can never exceed 1.
Of course, we could store the linked lists for chaining inside the hash table, in
the otherwise unused hash-table slots (see Exercise 11.2-4), but the advantage of
open addressing is that it avoids pointers altogether. Instead of following pointers,
we compute the sequence of slots to be examined. The extra memory freed by not
storing pointers provides the hash table with a larger number of slots for the same
amount of memory, potentially yielding fewer collisions and faster retrieval.
To perform insertion using open addressing, we successively examine, or probe,
the hash table until we find an empty slot in which to put the key. Instead of being
fixed in the order 0; 1; : : : ; m ! 1 (which requires ‚.n/ search time), the sequence
of positions probed depends upon the key being inserted. To determine which slots
to probe, we extend the hash function to include the probe number (starting from 0)
as a second input. Thus, the hash function becomes
h W U ' f0; 1; : : : ; m ! 1g ! f0; 1; : : : ; m ! 1g :
With open addressing, we require that for every key k, the probe sequence
hh.k; 0/; h.k; 1/; : : : ; h.k; m ! 1/i
be a permutation of h0; 1; : : : ; m!1i, so that every hash-table position is eventually
considered as a slot for a new key as the table fills up. In the following pseudocode,
we assume that the elements in the hash table T are keys with no satellite information; the key k is identical to the element containing key k. Each slot contains
either a key or NIL (if the slot is empty). The H ASH -I NSERT procedure takes as
input a hash table T and a key k. It either returns the slot number where it stores
key k or flags an error because the hash table is already full.
H ASH -I NSERT .T; k/
1 i D0
2 repeat
3
j D h.k; i/
4
if T Œj " == NIL
5
T Œj " D k
6
return j
7
else i D i C 1
8 until i == m
9 error “hash table overflow”
The algorithm for searching for key k probes the same sequence of slots that the
insertion algorithm examined when key k was inserted. Therefore, the search can
11.4 Open addressing
271
terminate (unsuccessfully) when it finds an empty slot, since k would have been
inserted there and not later in its probe sequence. (This argument assumes that keys
are not deleted from the hash table.) The procedure H ASH -S EARCH takes as input
a hash table T and a key k, returning j if it finds that slot j contains key k, or NIL
if key k is not present in table T .
H ASH -S EARCH .T; k/
1 i D0
2 repeat
3
j D h.k; i/
4
if T Œj " == k
5
return j
6
i D i C1
7 until T Œj " == NIL or i == m
8 return NIL
Deletion from an open-address hash table is difficult. When we delete a key
from slot i, we cannot simply mark that slot as empty by storing NIL in it. If
we did, we might be unable to retrieve any key k during whose insertion we had
probed slot i and found it occupied. We can solve this problem by marking the
slot, storing in it the special value DELETED instead of NIL. We would then modify
the procedure H ASH -I NSERT to treat such a slot as if it were empty so that we can
insert a new key there. We do not need to modify H ASH -S EARCH, since it will pass
over DELETED values while searching. When we use the special value DELETED,
however, search times no longer depend on the load factor ˛, and for this reason
chaining is more commonly selected as a collision resolution technique when keys
must be deleted.
In our analysis, we assume uniform hashing: the probe sequence of each key
is equally likely to be any of the mŠ permutations of h0; 1; : : : ; m ! 1i. Uniform hashing generalizes the notion of simple uniform hashing defined earlier to a
hash function that produces not just a single number, but a whole probe sequence.
True uniform hashing is difficult to implement, however, and in practice suitable
approximations (such as double hashing, defined below) are used.
We will examine three commonly used techniques to compute the probe sequences required for open addressing: linear probing, quadratic probing, and double hashing. These techniques all guarantee that hh.k; 0/; h.k; 1/; : : : ; h.k; m ! 1/i
is a permutation of h0; 1; : : : ; m ! 1i for each key k. None of these techniques fulfills the assumption of uniform hashing, however, since none of them is capable of
generating more than m2 different probe sequences (instead of the mŠ that uniform
hashing requires). Double hashing has the greatest number of probe sequences and,
as one might expect, seems to give the best results.
272
Chapter 11 Hash Tables
Linear probing
Given an ordinary hash function h0 W U ! f0; 1; : : : ; m ! 1g, which we refer to as
an auxiliary hash function, the method of linear probing uses the hash function
h.k; i/ D .h0 .k/ C i/ mod m
for i D 0; 1; : : : ; m ! 1. Given key k, we first probe T Œh0 .k/", i.e., the slot given
by the auxiliary hash function. We next probe slot T Œh0 .k/ C 1", and so on up to
slot T Œm ! 1". Then we wrap around to slots T Œ0"; T Œ1"; : : : until we finally probe
slot T Œh0 .k/ ! 1". Because the initial probe determines the entire probe sequence,
there are only m distinct probe sequences.
Linear probing is easy to implement, but it suffers from a problem known as
primary clustering. Long runs of occupied slots build up, increasing the average
search time. Clusters arise because an empty slot preceded by i full slots gets filled
next with probability .i C 1/=m. Long runs of occupied slots tend to get longer,
and the average search time increases.
Quadratic probing
Quadratic probing uses a hash function of the form
h.k; i/ D .h0 .k/ C c1 i C c2 i 2 / mod m ;
(11.5)
where h0 is an auxiliary hash function, c1 and c2 are positive auxiliary constants,
and i D 0; 1; : : : ; m ! 1. The initial position probed is T Œh0 .k/"; later positions
probed are offset by amounts that depend in a quadratic manner on the probe number i. This method works much better than linear probing, but to make full use of
the hash table, the values of c1 , c2 , and m are constrained. Problem 11-3 shows
one way to select these parameters. Also, if two keys have the same initial probe
position, then their probe sequences are the same, since h.k1 ; 0/ D h.k2 ; 0/ implies h.k1 ; i/ D h.k2 ; i/. This property leads to a milder form of clustering, called
secondary clustering. As in linear probing, the initial probe determines the entire
sequence, and so only m distinct probe sequences are used.
Double hashing
Double hashing offers one of the best methods available for open addressing because the permutations produced have many of the characteristics of randomly
chosen permutations. Double hashing uses a hash function of the form
h.k; i/ D .h1 .k/ C ih2 .k// mod m ;
where both h1 and h2 are auxiliary hash functions. The initial probe goes to position T Œh1 .k/"; successive probe positions are offset from previous positions by the
11.4 Open addressing
0
1
2
3
4
5
6
7
8
9
10
11
12
273
79
69
98
72
14
50
Figure 11.5 Insertion by double hashing. Here we have a hash table of size 13 with h1 .k/ D
k mod 13 and h2 .k/ D 1 C .k mod 11/. Since 14 & 1 .mod 13/ and 14 & 3 .mod 11/, we insert
the key 14 into empty slot 9, after examining slots 1 and 5 and finding them to be occupied.
amount h2 .k/, modulo m. Thus, unlike the case of linear or quadratic probing, the
probe sequence here depends in two ways upon the key k, since the initial probe
position, the offset, or both, may vary. Figure 11.5 gives an example of insertion
by double hashing.
The value h2 .k/ must be relatively prime to the hash-table size m for the entire
hash table to be searched. (See Exercise 11.4-4.) A convenient way to ensure this
condition is to let m be a power of 2 and to design h2 so that it always produces an
odd number. Another way is to let m be prime and to design h2 so that it always
returns a positive integer less than m. For example, we could choose m prime and
let
h1 .k/ D k mod m ;
h2 .k/ D 1 C .k mod m0 / ;
where m0 is chosen to be slightly less than m (say, m ! 1). For example, if
k D 123456, m D 701, and m0 D 700, we have h1 .k/ D 80 and h2 .k/ D 257, so
that we first probe position 80, and then we examine every 257th slot (modulo m)
until we find the key or have examined every slot.
When m is prime or a power of 2, double hashing improves over linear or quadratic probing in that ‚.m2 / probe sequences are used, rather than ‚.m/, since
each possible .h1 .k/; h2 .k// pair yields a distinct probe sequence. As a result, for
274
Chapter 11 Hash Tables
such values of m, the performance of double hashing appears to be very close to
the performance of the “ideal” scheme of uniform hashing.
Although values of m other than primes or powers of 2 could in principle be
used with double hashing, in practice it becomes more difficult to efficiently generate h2 .k/ in a way that ensures that it is relatively prime to m, in part because the
relative density '.m/=m of such numbers may be small (see equation (31.24)).
Analysis of open-address hashing
As in our analysis of chaining, we express our analysis of open addressing in terms
of the load factor ˛ D n=m of the hash table. Of course, with open addressing, at
most one element occupies each slot, and thus n " m, which implies ˛ " 1.
We assume that we are using uniform hashing. In this idealized scheme, the
probe sequence hh.k; 0/; h.k; 1/; : : : ; h.k; m ! 1/i used to insert or search for
each key k is equally likely to be any permutation of h0; 1; : : : ; m ! 1i. Of course,
a given key has a unique fixed probe sequence associated with it; what we mean
here is that, considering the probability distribution on the space of keys and the
operation of the hash function on the keys, each possible probe sequence is equally
likely.
We now analyze the expected number of probes for hashing with open addressing under the assumption of uniform hashing, beginning with an analysis of the
number of probes made in an unsuccessful search.
Theorem 11.6
Given an open-address hash table with load factor ˛ D n=m < 1, the expected
number of probes in an unsuccessful search is at most 1=.1!˛/, assuming uniform
hashing.
Proof In an unsuccessful search, every probe but the last accesses an occupied
slot that does not contain the desired key, and the last slot probed is empty. Let us
define the random variable X to be the number of probes made in an unsuccessful
search, and let us also define the event Ai , for i D 1; 2; : : :, to be the event that
an ith probe occurs and it is to an occupied slot. Then the event fX $ ig is the
intersection of events A1 \ A2 \ # # # \ Ai "1 . We will bound Pr fX $ ig by bounding
Pr fA1 \ A2 \ # # # \ Ai "1 g. By Exercise C.2-5,
Pr fA1 \ A2 \ # # # \ Ai "1 g D Pr fA1 g # Pr fA2 j A1 g # Pr fA3 j A1 \ A2 g # # #
Pr fAi "1 j A1 \ A2 \ # # # \ Ai "2 g :
Since there are n elements and m slots, Pr fA1 g D n=m. For j > 1, the probability
that there is a j th probe and it is to an occupied slot, given that the first j ! 1
probes were to occupied slots, is .n ! j C 1/=.m ! j C 1/. This probability follows
11.4 Open addressing
275
because we would be finding one of the remaining .n ! .j ! 1// elements in one
of the .m ! .j ! 1// unexamined slots, and by the assumption of uniform hashing,
the probability is the ratio of these quantities. Observing that n < m implies that
.n ! j /=.m ! j / " n=m for all j such that 0 " j < m, we have for all i such that
1 " i " m,
n!i C2
n n!1 n!2
#
#
###
m m!1 m!2
m!i C2
' n (i "1
"
m
D ˛ i "1 :
Pr fX $ ig D
Now, we use equation (C.25) to bound the expected number of probes:
E ŒX " D
"
D
D
1
X
i D1
1
X
i D1
1
X
Pr fX $ ig
˛ i "1
˛i
i D0
1
:
1!˛
This bound of 1=.1 ! ˛/ D 1 C ˛ C ˛ 2 C ˛ 3 C # # # has an intuitive interpretation.
We always make the first probe. With probability approximately ˛, the first probe
finds an occupied slot, so that we need to probe a second time. With probability
approximately ˛ 2 , the first two slots are occupied so that we make a third probe,
and so on.
If ˛ is a constant, Theorem 11.6 predicts that an unsuccessful search runs in O.1/
time. For example, if the hash table is half full, the average number of probes in an
unsuccessful search is at most 1=.1 ! :5/ D 2. If it is 90 percent full, the average
number of probes is at most 1=.1 ! :9/ D 10.
Theorem 11.6 gives us the performance of the H ASH -I NSERT procedure almost
immediately.
Corollary 11.7
Inserting an element into an open-address hash table with load factor ˛ requires at
most 1=.1 ! ˛/ probes on average, assuming uniform hashing.
276
Chapter 11 Hash Tables
Proof An element is inserted only if there is room in the table, and thus ˛ < 1.
Inserting a key requires an unsuccessful search followed by placing the key into the
first empty slot found. Thus, the expected number of probes is at most 1=.1!˛/.
We have to do a little more work to compute the expected number of probes for
a successful search.
Theorem 11.8
Given an open-address hash table with load factor ˛ < 1, the expected number of
probes in a successful search is at most
1
1
ln
;
˛ 1!˛
assuming uniform hashing and assuming that each key in the table is equally likely
to be searched for.
Proof A search for a key k reproduces the same probe sequence as when the
element with key k was inserted. By Corollary 11.7, if k was the .i C 1/st key
inserted into the hash table, the expected number of probes made in a search for k
is at most 1=.1 ! i=m/ D m=.m ! i/. Averaging over all n keys in the hash table
gives us the expected number of probes in a successful search:
n"1
1X m
n i D0 m ! i
n"1
D
D
"
D
D
mX 1
n i D0 m ! i
1
˛
m
X
kDm"nC1
1
k
Z
1 m
.1=x/ dx (by inequality (A.12))
˛ m"n
1
m
ln
˛ m!n
1
1
ln
:
˛ 1!˛
If the hash table is half full, the expected number of probes in a successful search
is less than 1:387. If the hash table is 90 percent full, the expected number of probes
is less than 2:559.
11.5 Perfect hashing
277
Exercises
11.4-1
Consider inserting the keys 10; 22; 31; 4; 15; 28; 17; 88; 59 into a hash table of
length m D 11 using open addressing with the auxiliary hash function h0 .k/ D k.
Illustrate the result of inserting these keys using linear probing, using quadratic
probing with c1 D 1 and c2 D 3, and using double hashing with h1 .k/ D k and
h2 .k/ D 1 C .k mod .m ! 1//.
11.4-2
Write pseudocode for H ASH -D ELETE as outlined in the text, and modify H ASH I NSERT to handle the special value DELETED.
11.4-3
Consider an open-address hash table with uniform hashing. Give upper bounds
on the expected number of probes in an unsuccessful search and on the expected
number of probes in a successful search when the load factor is 3=4 and when it
is 7=8.
11.4-4 ?
Suppose that we use double hashing to resolve collisions—that is, we use the hash
function h.k; i/ D .h1 .k/ C ih2 .k// mod m. Show that if m and h2 .k/ have
greatest common divisor d $ 1 for some key k, then an unsuccessful search for
key k examines .1=d /th of the hash table before returning to slot h1 .k/. Thus,
when d D 1, so that m and h2 .k/ are relatively prime, the search may examine the
entire hash table. (Hint: See Chapter 31.)
11.4-5 ?
Consider an open-address hash table with a load factor ˛. Find the nonzero value ˛
for which the expected number of probes in an unsuccessful search equals twice
the expected number of probes in a successful search. Use the upper bounds given
by Theorems 11.6 and 11.8 for these expected numbers of probes.
? 11.5 Perfect hashing
Although hashing is often a good choice for its excellent average-case performance, hashing can also provide excellent worst-case performance when the set of
keys is static: once the keys are stored in the table, the set of keys never changes.
Some applications naturally have static sets of keys: consider the set of reserved
words in a programming language, or the set of file names on a CD-ROM. We
278
Chapter 11 Hash Tables
0
1
2
T
S
m0 a0 b0 0
1 0 0 10
m2 a2 b2
9 10 18
3
4
5
6
7
8
S2
0
0
S5
1
2
60 72
3
4
5
6
75
7
m5 a5 b5
1 0 0 70
m7 a7 b7
16 23 88
S7
0
0
8
1
2
3
4
5
6
40 52 22
7
8
9
10
11
12
13
37
14
15
Figure 11.6 Using perfect hashing to store the set K D f10; 22; 37; 40; 52; 60; 70; 72; 75g. The
outer hash function is h.k/ D ..ak C b/ mod p/ mod m, where a D 3, b D 42, p D 101, and
m D 9. For example, h.75/ D 2, and so key 75 hashes to slot 2 of table T . A secondary hash
table Sj stores all keys hashing to slot j . The size of hash table Sj is mj D nj2 , and the associated
hash function is hj .k/ D ..aj k C bj / mod p/ mod mj . Since h2 .75/ D 7, key 75 is stored in slot 7
of secondary hash table S2 . No collisions occur in any of the secondary hash tables, and so searching
takes constant time in the worst case.
call a hashing technique perfect hashing if O.1/ memory accesses are required to
perform a search in the worst case.
To create a perfect hashing scheme, we use two levels of hashing, with universal
hashing at each level. Figure 11.6 illustrates the approach.
The first level is essentially the same as for hashing with chaining: we hash
the n keys into m slots using a hash function h carefully selected from a family of
universal hash functions.
Instead of making a linked list of the keys hashing to slot j , however, we use a
small secondary hash table Sj with an associated hash function hj . By choosing
the hash functions hj carefully, we can guarantee that there are no collisions at the
secondary level.
In order to guarantee that there are no collisions at the secondary level, however,
we will need to let the size mj of hash table Sj be the square of the number nj of
keys hashing to slot j . Although you might think that the quadratic dependence
of mj on nj may seem likely to cause the overall storage requirement to be excessive, we shall show that by choosing the first-level hash function well, we can limit
the expected total amount of space used to O.n/.
We use hash functions chosen from the universal classes of hash functions of
Section 11.3.3. The first-level hash function comes from the class Hpm , where as
in Section 11.3.3, p is a prime number greater than any key value. Those keys
11.5 Perfect hashing
279
hashing to slot j are re-hashed into a secondary hash table Sj of size mj using a
hash function hj chosen from the class Hp;mj .1
We shall proceed in two steps. First, we shall determine how to ensure that
the secondary tables have no collisions. Second, we shall show that the expected
amount of memory used overall—for the primary hash table and all the secondary
hash tables—is O.n/.
Theorem 11.9
Suppose that we store n keys in a hash table of size m D n2 using a hash function h
randomly chosen from a universal class of hash functions. Then, the probability is
less than 1=2 that there are any collisions.
#$
Proof There are n2 pairs of keys that may collide; each pair collides with probability 1=m if h is chosen at random from a universal family H of hash functions.
Let X be a random variable that counts the number of collisions. When m D n2 ,
the expected number of collisions is
!
n
1
E ŒX " D
# 2
2
n
n2 ! n 1
# 2
2
n
< 1=2 :
D
(This analysis is similar to the analysis of the birthday paradox in Section 5.4.1.)
Applying Markov’s inequality (C.30), Pr fX $ tg " E ŒX " =t, with t D 1, completes the proof.
In the situation described in Theorem 11.9, where m D n2 , it follows that a hash
function h chosen at random from H is more likely than not to have no collisions.
Given the set K of n keys to be hashed (remember that K is static), it is thus easy
to find a collision-free hash function h with a few random trials.
When n is large, however, a hash table of size m D n2 is excessive. Therefore,
we adopt the two-level hashing approach, and we use the approach of Theorem 11.9
only to hash the entries within each slot. We use an outer, or first-level, hash
function h to hash the keys into m D n slots. Then, if nj keys hash to slot j , we
use a secondary hash table Sj of size mj D nj2 to provide collision-free constanttime lookup.
1 When n D m D 1, we don’t really need a hash function for slot j ; when we choose a hash
j
j
function hab .k/ D ..ak C b/ mod p/ mod mj for such a slot, we just use a D b D 0.
280
Chapter 11 Hash Tables
We now turn to the issue of ensuring that the overall memory used is O.n/.
Since the size mj of the j th secondary hash table grows quadratically with the
number nj of keys stored, we run the risk that the overall amount of storage could
be excessive.
If the first-level table size is m D n, then the amount of memory used is O.n/
for the primary hash table, for the storage of the sizes mj of the secondary hash
tables, and for the storage of the parameters aj and bj defining the secondary hash
functions hj drawn from the class Hp;mj of Section 11.3.3 (except when nj D 1
and we use a D b D 0). The following theorem and a corollary provide a bound on
the expected combined sizes of all the secondary hash tables. A second corollary
bounds the probability that the combined size of all the secondary hash tables is
superlinear (actually, that it equals or exceeds 4n).
Theorem 11.10
Suppose that we store n keys in a hash table of size m D n using a hash function h
randomly chosen from a universal class of hash functions. Then, we have
"m!1 #
X
nj2 < 2n ;
E
j D0
where nj is the number of keys hashing to slot j .
Proof We start with the following identity, which holds for any nonnegative integer a:
!
a
:
(11.6)
a2 D a C 2
2
We have
"m!1 #
X
E
nj2
j D0
"m!1
X
!!#
nj
D E
nj C 2
(by equation (11.6))
2
j D0
"m!1 #
"m!1
!#
X
X nj
D E
nj C 2 E
(by linearity of expectation)
2
j D0
j D0
!#
"m!1
X nj
(by equation (11.1))
D E Œn! C 2 E
2
j D0
11.5 Perfect hashing
281
"m!1
!#
X nj
D n C 2E
2
j D0
(since n is not a random variable) .
Pm!1 ! "
To evaluate the summation j D0 n2j , we observe that it is just the total number
of pairs of keys in the hash table that collide. By the properties of universal hashing,
the expected value of this summation is at most
!
n 1
n.n ! 1/
D
2m
2 m
D
n!1
;
2
since m D n. Thus,
"m!1 #
X
n!1
E
nj2
" nC2
2
j D0
D 2n ! 1
< 2n :
Corollary 11.11
Suppose that we store n keys in a hash table of size m D n using a hash function h randomly chosen from a universal class of hash functions, and we set the
size of each secondary hash table to mj D nj2 for j D 0; 1; : : : ; m ! 1. Then,
the expected amount of storage required for all secondary hash tables in a perfect
hashing scheme is less than 2n.
Proof Since mj D nj2 for j D 0; 1; : : : ; m ! 1, Theorem 11.10 gives
"m!1 #
"m!1 #
X
X
E
mj
D E
nj2
j D0
j D0
< 2n ;
(11.7)
which completes the proof.
Corollary 11.12
Suppose that we store n keys in a hash table of size m D n using a hash function h
randomly chosen from a universal class of hash functions, and we set the size
of each secondary hash table to mj D nj2 for j D 0; 1; : : : ; m ! 1. Then, the
probability is less than 1=2 that the total storage used for secondary hash tables
equals or exceeds 4n.
282
Chapter 11 Hash Tables
Proof Again we apply Markov’s inequality
Pm!1 (C.30), Pr fX # tg " E ŒX ! =t, this
time to inequality (11.7), with X D j D0 mj and t D 4n:
(m!1
)
#Pm!1 $
X
E
j D0 mj
Pr
mj # 4n
"
4n
j D0
2n
4n
D 1=2 :
<
From Corollary 11.12, we see that if we test a few randomly chosen hash functions from the universal family, we will quickly find one that uses a reasonable
amount of storage.
Exercises
11.5-1 ?
Suppose that we insert n keys into a hash table of size m using open addressing
and uniform hashing. Let p.n; m/ be the probability that no collisions occur. Show
!n.n!1/=2m
. (Hint: See equation (3.12).) Argue that when n exthat p.n;
p m/ " e
ceeds m, the probability of avoiding collisions goes rapidly to zero.
Problems
11-1 Longest-probe bound for hashing
Suppose that we use an open-addressed hash table of size m to store n " m=2
items.
a. Assuming uniform hashing, show that for i D 1; 2; : : : ; n, the probability is at
most 2!k that the ith insertion requires strictly more than k probes.
b. Show that for i D 1; 2; : : : ; n, the probability is O.1=n2 / that the ith insertion
requires more than 2 lg n probes.
Let the random variable Xi denote the number of probes required by the ith insertion. You have shown in part (b) that Pr fXi > 2 lg ng D O.1=n2 /. Let the random
variable X D max1"i "n Xi denote the maximum number of probes required by
any of the n insertions.
c. Show that Pr fX > 2 lg ng D O.1=n/.
d. Show that the expected length E ŒX ! of the longest probe sequence is O.lg n/.
Problems for Chapter 11
283
11-2 Slot-size bound for chaining
Suppose that we have a hash table with n slots, with collisions resolved by chaining, and suppose that n keys are inserted into the table. Each key is equally likely
to be hashed to each slot. Let M be the maximum number of keys in any slot after
all the keys have been inserted. Your mission is to prove an O.lg n= lg lg n/ upper
bound on E ŒM !, the expected value of M .
a. Argue that the probability Qk that exactly k keys hash to a particular slot is
given by
!
% &k %
&
1
1 n!k n
Qk D
1!
:
n
n
k
b. Let Pk be the probability that M D k, that is, the probability that the slot
containing the most keys contains k keys. Show that Pk " nQk .
c. Use Stirling’s approximation, equation (3.18), to show that Qk < e k =k k .
d. Show that there exists a constant c > 1 such that Qk0 < 1=n3 for k0 D
c lg n= lg lg n. Conclude that Pk < 1=n2 for k # k0 D c lg n= lg lg n.
e. Argue that
'
(
'
(
c lg n
c lg n
c lg n
E ŒM ! " Pr M >
$ n C Pr M "
$
:
lg lg n
lg lg n lg lg n
Conclude that E ŒM ! D O.lg n= lg lg n/.
11-3 Quadratic probing
Suppose that we are given a key k to search for in a hash table with positions
0; 1; : : : ; m ! 1, and suppose that we have a hash function h mapping the key space
into the set f0; 1; : : : ; m ! 1g. The search scheme is as follows:
1. Compute the value j D h.k/, and set i D 0.
2. Probe in position j for the desired key k. If you find it, or if this position is
empty, terminate the search.
3. Set i D i C 1. If i now equals m, the table is full, so terminate the search.
Otherwise, set j D .i C j / mod m, and return to step 2.
Assume that m is a power of 2.
a. Show that this scheme is an instance of the general “quadratic probing” scheme
by exhibiting the appropriate constants c1 and c2 for equation (11.5).
b. Prove that this algorithm examines every table position in the worst case.
284
Chapter 11 Hash Tables
11-4 Hashing and authentication
Let H be a class of hash functions in which each hash function h 2 H maps the
universe U of keys to f0; 1; : : : ; m ! 1g. We say that H is k-universal if, for every
fixed sequence of k distinct keys hx .1/ ; x .2/ ; : : : ; x .k/ i and for any h chosen at
random from H , the sequence hh.x .1/ /; h.x .2/ /; : : : ; h.x .k/ /i is equally likely to be
any of the mk sequences of length k with elements drawn from f0; 1; : : : ; m ! 1g.
a. Show that if the family H of hash functions is 2-universal, then it is universal.
b. Suppose that the universe U is the set of n-tuples of values drawn from
Zp D f0; 1; : : : ; p ! 1g, where p is prime. Consider an element x D
hx0 ; x1 ; : : : ; xn!1 i 2 U . For any n-tuple a D ha0 ; a1 ; : : : ; an!1 i 2 U , define the hash function ha by
!
n!1
X
ha .x/ D
aj xj mod p :
j D0
Let H D fha g. Show that H is universal, but not 2-universal. (Hint: Find a key
for which all hash functions in H produce the same value.)
c. Suppose that we modify H slightly from part (b): for any a 2 U and for any
b 2 Zp , define
h0ab .x/ D
n!1
X
j D0
aj xj C b
!
mod p
and H 0 D fh0ab g. Argue that H 0 is 2-universal. (Hint: Consider fixed n-tuples
x 2 U and y 2 U , with xi ¤ yi for some i. What happens to h0ab .x/
and h0ab .y/ as ai and b range over Zp ?)
d. Suppose that Alice and Bob secretly agree on a hash function h from a
2-universal family H of hash functions. Each h 2 H maps from a universe of
keys U to Zp , where p is prime. Later, Alice sends a message m to Bob over the
Internet, where m 2 U . She authenticates this message to Bob by also sending
an authentication tag t D h.m/, and Bob checks that the pair .m; t/ he receives
indeed satisfies t D h.m/. Suppose that an adversary intercepts .m; t/ en route
and tries to fool Bob by replacing the pair .m; t/ with a different pair .m0 ; t 0 /.
Argue that the probability that the adversary succeeds in fooling Bob into accepting .m0 ; t 0 / is at most 1=p, no matter how much computing power the adversary has, and even if the adversary knows the family H of hash functions
used.
Notes for Chapter 11
285
Chapter notes
Knuth [211] and Gonnet [145] are excellent references for the analysis of hashing algorithms. Knuth credits H. P. Luhn (1953) for inventing hash tables, along
with the chaining method for resolving collisions. At about the same time, G. M.
Amdahl originated the idea of open addressing.
Carter and Wegman introduced the notion of universal classes of hash functions
in 1979 [58].
Fredman, Komlós, and Szemerédi [112] developed the perfect hashing scheme
for static sets presented in Section 11.5. An extension of their method to dynamic
sets, handling insertions and deletions in amortized expected time O.1/, has been
given by Dietzfelbinger et al. [86].
12
Binary Search Trees
The search tree data structure supports many dynamic-set operations, including
S EARCH, M INIMUM, M AXIMUM, P REDECESSOR, S UCCESSOR, I NSERT, and
D ELETE. Thus, we can use a search tree both as a dictionary and as a priority
queue.
Basic operations on a binary search tree take time proportional to the height of
the tree. For a complete binary tree with n nodes, such operations run in ‚.lg n/
worst-case time. If the tree is a linear chain of n nodes, however, the same operations take ‚.n/ worst-case time. We shall see in Section 12.4 that the expected
height of a randomly built binary search tree is O.lg n/, so that basic dynamic-set
operations on such a tree take ‚.lg n/ time on average.
In practice, we can’t always guarantee that binary search trees are built randomly, but we can design variations of binary search trees with good guaranteed
worst-case performance on basic operations. Chapter 13 presents one such variation, red-black trees, which have height O.lg n/. Chapter 18 introduces B-trees,
which are particularly good for maintaining databases on secondary (disk) storage.
After presenting the basic properties of binary search trees, the following sections show how to walk a binary search tree to print its values in sorted order, how
to search for a value in a binary search tree, how to find the minimum or maximum
element, how to find the predecessor or successor of an element, and how to insert
into or delete from a binary search tree. The basic mathematical properties of trees
appear in Appendix B.
12.1 What is a binary search tree?
A binary search tree is organized, as the name suggests, in a binary tree, as shown
in Figure 12.1. We can represent such a tree by a linked data structure in which
each node is an object. In addition to a key and satellite data, each node contains
attributes left, right, and p that point to the nodes corresponding to its left child,
12.1 What is a binary search tree?
287
6
5
2
2
5
7
5
7
8
6
8
5
(a)
(b)
Figure 12.1 Binary search trees. For any node x, the keys in the left subtree of x are at most x: key,
and the keys in the right subtree of x are at least x: key. Different binary search trees can represent
the same set of values. The worst-case running time for most search-tree operations is proportional
to the height of the tree. (a) A binary search tree on 6 nodes with height 2. (b) A less efficient binary
search tree with height 4 that contains the same keys.
its right child, and its parent, respectively. If a child or the parent is missing, the
appropriate attribute contains the value NIL. The root node is the only node in the
tree whose parent is NIL.
The keys in a binary search tree are always stored in such a way as to satisfy the
binary-search-tree property:
Let x be a node in a binary search tree. If y is a node in the left subtree
of x, then y:key " x:key. If y is a node in the right subtree of x, then
y:key # x:key.
Thus, in Figure 12.1(a), the key of the root is 6, the keys 2, 5, and 5 in its left
subtree are no larger than 6, and the keys 7 and 8 in its right subtree are no smaller
than 6. The same property holds for every node in the tree. For example, the key 5
in the root’s left child is no smaller than the key 2 in that node’s left subtree and no
larger than the key 5 in the right subtree.
The binary-search-tree property allows us to print out all the keys in a binary
search tree in sorted order by a simple recursive algorithm, called an inorder tree
walk. This algorithm is so named because it prints the key of the root of a subtree
between printing the values in its left subtree and printing those in its right subtree.
(Similarly, a preorder tree walk prints the root before the values in either subtree,
and a postorder tree walk prints the root after the values in its subtrees.) To use
the following procedure to print all the elements in a binary search tree T , we call
I NORDER -T REE -WALK .T:root/.
288
Chapter 12 Binary Search Trees
I NORDER -T REE -WALK .x/
1 if x ¤ NIL
2
I NORDER -T REE -WALK .x:left/
3
print x:key
4
I NORDER -T REE -WALK .x:right/
As an example, the inorder tree walk prints the keys in each of the two binary
search trees from Figure 12.1 in the order 2; 5; 5; 6; 7; 8. The correctness of the
algorithm follows by induction directly from the binary-search-tree property.
It takes ‚.n/ time to walk an n-node binary search tree, since after the initial call, the procedure calls itself recursively exactly twice for each node in the
tree—once for its left child and once for its right child. The following theorem
gives a formal proof that it takes linear time to perform an inorder tree walk.
Theorem 12.1
If x is the root of an n-node subtree, then the call I NORDER -T REE -WALK .x/
takes ‚.n/ time.
Proof Let T .n/ denote the time taken by I NORDER -T REE -WALK when it is
called on the root of an n-node subtree. Since I NORDER -T REE -WALK visits all n
nodes of the subtree, we have T .n/ D ".n/. It remains to show that T .n/ D O.n/.
Since I NORDER -T REE -WALK takes a small, constant amount of time on an
empty subtree (for the test x ¤ NIL ), we have T .0/ D c for some constant c > 0.
For n > 0, suppose that I NORDER -T REE -WALK is called on a node x whose
left subtree has k nodes and whose right subtree has n ! k ! 1 nodes. The time to
perform I NORDER -T REE -WALK .x/ is bounded by T .n/ " T .k/CT .n!k!1/Cd
for some constant d > 0 that reflects an upper bound on the time to execute the
body of I NORDER -T REE -WALK .x/, exclusive of the time spent in recursive calls.
We use the substitution method to show that T .n/ D O.n/ by proving that
T .n/ " .c C d /n C c. For n D 0, we have .c C d / $ 0 C c D c D T .0/. For n > 0,
we have
T .n/ "
D
D
D
T .k/ C T .n ! k ! 1/ C d
..c C d /k C c/ C ..c C d /.n ! k ! 1/ C c/ C d
.c C d /n C c ! .c C d / C c C d
.c C d /n C c ;
which completes the proof.
12.2 Querying a binary search tree
289
Exercises
12.1-1
For the set of f1; 4; 5; 10; 16; 17; 21g of keys, draw binary search trees of heights 2,
3, 4, 5, and 6.
12.1-2
What is the difference between the binary-search-tree property and the min-heap
property (see page 153)? Can the min-heap property be used to print out the keys
of an n-node tree in sorted order in O.n/ time? Show how, or explain why not.
12.1-3
Give a nonrecursive algorithm that performs an inorder tree walk. (Hint: An easy
solution uses a stack as an auxiliary data structure. A more complicated, but elegant, solution uses no stack but assumes that we can test two pointers for equality.)
12.1-4
Give recursive algorithms that perform preorder and postorder tree walks in ‚.n/
time on a tree of n nodes.
12.1-5
Argue that since sorting n elements takes ".n lg n/ time in the worst case in
the comparison model, any comparison-based algorithm for constructing a binary
search tree from an arbitrary list of n elements takes ".n lg n/ time in the worst
case.
12.2 Querying a binary search tree
We often need to search for a key stored in a binary search tree. Besides the
S EARCH operation, binary search trees can support such queries as M INIMUM,
M AXIMUM, S UCCESSOR, and P REDECESSOR. In this section, we shall examine
these operations and show how to support each one in time O.h/ on any binary
search tree of height h.
Searching
We use the following procedure to search for a node with a given key in a binary
search tree. Given a pointer to the root of the tree and a key k, T REE -S EARCH
returns a pointer to a node with key k if one exists; otherwise, it returns NIL.
290
Chapter 12 Binary Search Trees
15
6
7
3
2
18
4
17
20
13
9
Figure 12.2 Queries on a binary search tree. To search for the key 13 in the tree, we follow the path
15 ! 6 ! 7 ! 13 from the root. The minimum key in the tree is 2, which is found by following
left pointers from the root. The maximum key 20 is found by following right pointers from the root.
The successor of the node with key 15 is the node with key 17, since it is the minimum key in the
right subtree of 15. The node with key 13 has no right subtree, and thus its successor is its lowest
ancestor whose left child is also an ancestor. In this case, the node with key 15 is its successor.
T REE -S EARCH .x; k/
1 if x == NIL or k == x:key
2
return x
3 if k < x:key
4
return T REE -S EARCH .x:left; k/
5 else return T REE -S EARCH .x:right; k/
The procedure begins its search at the root and traces a simple path downward in
the tree, as shown in Figure 12.2. For each node x it encounters, it compares the
key k with x:key. If the two keys are equal, the search terminates. If k is smaller
than x:key, the search continues in the left subtree of x, since the binary-searchtree property implies that k could not be stored in the right subtree. Symmetrically,
if k is larger than x:key, the search continues in the right subtree. The nodes
encountered during the recursion form a simple path downward from the root of
the tree, and thus the running time of T REE -S EARCH is O.h/, where h is the height
of the tree.
We can rewrite this procedure in an iterative fashion by “unrolling” the recursion
into a while loop. On most computers, the iterative version is more efficient.
12.2 Querying a binary search tree
291
I TERATIVE -T REE -S EARCH .x; k/
1 while x ¤ NIL and k ¤ x:key
2
if k < x:key
3
x D x:left
4
else x D x:right
5 return x
Minimum and maximum
We can always find an element in a binary search tree whose key is a minimum by
following left child pointers from the root until we encounter a NIL, as shown in
Figure 12.2. The following procedure returns a pointer to the minimum element in
the subtree rooted at a given node x, which we assume to be non-NIL:
T REE -M INIMUM .x/
1 while x:left ¤ NIL
2
x D x:left
3 return x
The binary-search-tree property guarantees that T REE -M INIMUM is correct. If a
node x has no left subtree, then since every key in the right subtree of x is at least as
large as x:key, the minimum key in the subtree rooted at x is x:key. If node x has
a left subtree, then since no key in the right subtree is smaller than x:key and every
key in the left subtree is not larger than x:key, the minimum key in the subtree
rooted at x resides in the subtree rooted at x:left.
The pseudocode for T REE -M AXIMUM is symmetric:
T REE -M AXIMUM .x/
1 while x:right ¤ NIL
2
x D x:right
3 return x
Both of these procedures run in O.h/ time on a tree of height h since, as in T REE S EARCH, the sequence of nodes encountered forms a simple path downward from
the root.
Successor and predecessor
Given a node in a binary search tree, sometimes we need to find its successor in
the sorted order determined by an inorder tree walk. If all keys are distinct, the
292
Chapter 12 Binary Search Trees
successor of a node x is the node with the smallest key greater than x:key. The
structure of a binary search tree allows us to determine the successor of a node
without ever comparing keys. The following procedure returns the successor of a
node x in a binary search tree if it exists, and NIL if x has the largest key in the
tree:
T REE -S UCCESSOR .x/
1 if x:right ¤ NIL
2
return T REE -M INIMUM .x:right/
3 y D x:p
4 while y ¤ NIL and x == y:right
5
x Dy
6
y D y:p
7 return y
We break the code for T REE -S UCCESSOR into two cases. If the right subtree
of node x is nonempty, then the successor of x is just the leftmost node in x’s
right subtree, which we find in line 2 by calling T REE -M INIMUM .x:right/. For
example, the successor of the node with key 15 in Figure 12.2 is the node with
key 17.
On the other hand, as Exercise 12.2-6 asks you to show, if the right subtree of
node x is empty and x has a successor y, then y is the lowest ancestor of x whose
left child is also an ancestor of x. In Figure 12.2, the successor of the node with
key 13 is the node with key 15. To find y, we simply go up the tree from x until we
encounter a node that is the left child of its parent; lines 3–7 of T REE -S UCCESSOR
handle this case.
The running time of T REE -S UCCESSOR on a tree of height h is O.h/, since we
either follow a simple path up the tree or follow a simple path down the tree. The
procedure T REE -P REDECESSOR, which is symmetric to T REE -S UCCESSOR, also
runs in time O.h/.
Even if keys are not distinct, we define the successor and predecessor of any
node x as the node returned by calls made to T REE -S UCCESSOR .x/ and T REE P REDECESSOR.x/, respectively.
In summary, we have proved the following theorem.
Theorem 12.2
We can implement the dynamic-set operations S EARCH, M INIMUM, M AXIMUM,
S UCCESSOR, and P REDECESSOR so that each one runs in O.h/ time on a binary
search tree of height h.
12.2 Querying a binary search tree
293
Exercises
12.2-1
Suppose that we have numbers between 1 and 1000 in a binary search tree, and we
want to search for the number 363. Which of the following sequences could not be
the sequence of nodes examined?
a. 2, 252, 401, 398, 330, 344, 397, 363.
b. 924, 220, 911, 244, 898, 258, 362, 363.
c. 925, 202, 911, 240, 912, 245, 363.
d. 2, 399, 387, 219, 266, 382, 381, 278, 363.
e. 935, 278, 347, 621, 299, 392, 358, 363.
12.2-2
Write recursive versions of T REE -M INIMUM and T REE -M AXIMUM.
12.2-3
Write the T REE -P REDECESSOR procedure.
12.2-4
Professor Bunyan thinks he has discovered a remarkable property of binary search
trees. Suppose that the search for key k in a binary search tree ends up in a leaf.
Consider three sets: A, the keys to the left of the search path; B, the keys on the
search path; and C , the keys to the right of the search path. Professor Bunyan
claims that any three keys a 2 A, b 2 B, and c 2 C must satisfy a " b " c. Give
a smallest possible counterexample to the professor’s claim.
12.2-5
Show that if a node in a binary search tree has two children, then its successor has
no left child and its predecessor has no right child.
12.2-6
Consider a binary search tree T whose keys are distinct. Show that if the right
subtree of a node x in T is empty and x has a successor y, then y is the lowest
ancestor of x whose left child is also an ancestor of x. (Recall that every node is
its own ancestor.)
12.2-7
An alternative method of performing an inorder tree walk of an n-node binary
search tree finds the minimum element in the tree by calling T REE -M INIMUM and
then making n ! 1 calls to T REE -S UCCESSOR. Prove that this algorithm runs
in ‚.n/ time.
294
Chapter 12 Binary Search Trees
12.2-8
Prove that no matter what node we start at in a height-h binary search tree, k
successive calls to T REE -S UCCESSOR take O.k C h/ time.
12.2-9
Let T be a binary search tree whose keys are distinct, let x be a leaf node, and let y
be its parent. Show that y:key is either the smallest key in T larger than x:key or
the largest key in T smaller than x:key.
12.3 Insertion and deletion
The operations of insertion and deletion cause the dynamic set represented by a
binary search tree to change. The data structure must be modified to reflect this
change, but in such a way that the binary-search-tree property continues to hold.
As we shall see, modifying the tree to insert a new element is relatively straightforward, but handling deletion is somewhat more intricate.
Insertion
To insert a new value # into a binary search tree T , we use the procedure T REE I NSERT. The procedure takes a node ´ for which ´:key D #, ´:left D NIL,
and ´:right D NIL . It modifies T and some of the attributes of ´ in such a way that
it inserts ´ into an appropriate position in the tree.
T REE -I NSERT .T; ´/
1 y D NIL
2 x D T:root
3 while x ¤ NIL
4
y Dx
5
if ´:key < x:key
6
x D x:left
7
else x D x:right
8 ´:p D y
9 if y == NIL
10
T:root D ´
// tree T was empty
11 elseif ´:key < y:key
12
y:left D ´
13 else y:right D ´
12.3 Insertion and deletion
295
12
5
2
18
9
19
15
13
17
Figure 12.3 Inserting an item with key 13 into a binary search tree. Lightly shaded nodes indicate
the simple path from the root down to the position where the item is inserted. The dashed line
indicates the link in the tree that is added to insert the item.
Figure 12.3 shows how T REE -I NSERT works. Just like the procedures T REE S EARCH and I TERATIVE -T REE -S EARCH, T REE -I NSERT begins at the root of the
tree and the pointer x traces a simple path downward looking for a NIL to replace
with the input item ´. The procedure maintains the trailing pointer y as the parent
of x. After initialization, the while loop in lines 3–7 causes these two pointers
to move down the tree, going left or right depending on the comparison of ´:key
with x:key, until x becomes NIL. This NIL occupies the position where we wish to
place the input item ´. We need the trailing pointer y, because by the time we find
the NIL where ´ belongs, the search has proceeded one step beyond the node that
needs to be changed. Lines 8–13 set the pointers that cause ´ to be inserted.
Like the other primitive operations on search trees, the procedure T REE -I NSERT
runs in O.h/ time on a tree of height h.
Deletion
The overall strategy for deleting a node ´ from a binary search tree T has three
basic cases but, as we shall see, one of the cases is a bit tricky.
!
!
!
If ´ has no children, then we simply remove it by modifying its parent to replace ´ with NIL as its child.
If ´ has just one child, then we elevate that child to take ´’s position in the tree
by modifying ´’s parent to replace ´ by ´’s child.
If ´ has two children, then we find ´’s successor y—which must be in ´’s right
subtree—and have y take ´’s position in the tree. The rest of ´’s original right
subtree becomes y’s new right subtree, and ´’s left subtree becomes y’s new
left subtree. This case is the tricky one because, as we shall see, it matters
whether y is ´’s right child.
296
Chapter 12 Binary Search Trees
The procedure for deleting a given node ´ from a binary search tree T takes as
arguments pointers to T and ´. It organizes its cases a bit differently from the three
cases outlined previously by considering the four cases shown in Figure 12.4.
!
!
!
If ´ has no left child (part (a) of the figure), then we replace ´ by its right child,
which may or may not be NIL. When ´’s right child is NIL, this case deals with
the situation in which ´ has no children. When ´’s right child is non-NIL, this
case handles the situation in which ´ has just one child, which is its right child.
If ´ has just one child, which is its left child (part (b) of the figure), then we
replace ´ by its left child.
Otherwise, ´ has both a left and a right child. We find ´’s successor y, which
lies in ´’s right subtree and has no left child (see Exercise 12.2-5). We want to
splice y out of its current location and have it replace ´ in the tree.
!
!
If y is ´’s right child (part (c)), then we replace ´ by y, leaving y’s right
child alone.
Otherwise, y lies within ´’s right subtree but is not ´’s right child (part (d)).
In this case, we first replace y by its own right child, and then we replace ´
by y.
In order to move subtrees around within the binary search tree, we define a
subroutine T RANSPLANT, which replaces one subtree as a child of its parent with
another subtree. When T RANSPLANT replaces the subtree rooted at node u with
the subtree rooted at node #, node u’s parent becomes node #’s parent, and u’s
parent ends up having # as its appropriate child.
T RANSPLANT .T; u; #/
1 if u:p == NIL
2
T:root D #
3 elseif u == u:p:left
4
u:p:left D #
5 else u:p:right D #
6 if # ¤ NIL
7
#:p D u:p
Lines 1–2 handle the case in which u is the root of T . Otherwise, u is either a left
child or a right child of its parent. Lines 3–4 take care of updating u:p:left if u
is a left child, and line 5 updates u:p:right if u is a right child. We allow # to be
NIL, and lines 6–7 update #:p if # is non-NIL. Note that T RANSPLANT does not
attempt to update #:left and #:right; doing so, or not doing so, is the responsibility
of T RANSPLANT’s caller.
12.3 Insertion and deletion
297
q
q
(a)
z
r
r
NIL
q
q
(b)
z
l
l
NIL
q
q
(c)
z
l
y
y
l
x
NIL
q
q
(d)
z
l
y
q
z
r
NIL
x
l
y
NIL
x
y
r
l
r
x
x
Figure 12.4 Deleting a node ´ from a binary search tree. Node ´ may be the root, a left child of
node q, or a right child of q. (a) Node ´ has no left child. We replace ´ by its right child r, which
may or may not be NIL . (b) Node ´ has a left child l but no right child. We replace ´ by l. (c) Node ´
has two children; its left child is node l, its right child is its successor y, and y’s right child is node x.
We replace ´ by y, updating y’s left child to become l, but leaving x as y’s right child. (d) Node ´
has two children (left child l and right child r), and its successor y ¤ r lies within the subtree rooted
at r. We replace y by its own right child x, and we set y to be r’s parent. Then, we set y to be q’s
child and the parent of l.
298
Chapter 12 Binary Search Trees
With the T RANSPLANT procedure in hand, here is the procedure that deletes
node ´ from binary search tree T :
T REE -D ELETE .T; ´/
1 if ´:left == NIL
2
T RANSPLANT .T; ´; ´:right/
3 elseif ´:right == NIL
4
T RANSPLANT .T; ´; ´:left/
5 else y D T REE -M INIMUM .´:right/
6
if y:p ¤ ´
7
T RANSPLANT .T; y; y:right/
8
y:right D ´:right
9
y:right:p D y
10
T RANSPLANT .T; ´; y/
11
y:left D ´:left
12
y:left:p D y
The T REE -D ELETE procedure executes the four cases as follows. Lines 1–2
handle the case in which node ´ has no left child, and lines 3–4 handle the case in
which ´ has a left child but no right child. Lines 5–12 deal with the remaining two
cases, in which ´ has two children. Line 5 finds node y, which is the successor
of ´. Because ´ has a nonempty right subtree, its successor must be the node in
that subtree with the smallest key; hence the call to T REE -M INIMUM .´:right/. As
we noted before, y has no left child. We want to splice y out of its current location,
and it should replace ´ in the tree. If y is ´’s right child, then lines 10–12 replace ´
as a child of its parent by y and replace y’s left child by ´’s left child. If y is
not ´’s left child, lines 7–9 replace y as a child of its parent by y’s right child and
turn ´’s right child into y’s right child, and then lines 10–12 replace ´ as a child of
its parent by y and replace y’s left child by ´’s left child.
Each line of T REE -D ELETE, including the calls to T RANSPLANT, takes constant
time, except for the call to T REE -M INIMUM in line 5. Thus, T REE -D ELETE runs
in O.h/ time on a tree of height h.
In summary, we have proved the following theorem.
Theorem 12.3
We can implement the dynamic-set operations I NSERT and D ELETE so that each
one runs in O.h/ time on a binary search tree of height h.
12.4 Randomly built binary search trees
299
Exercises
12.3-1
Give a recursive version of the T REE -I NSERT procedure.
12.3-2
Suppose that we construct a binary search tree by repeatedly inserting distinct values into the tree. Argue that the number of nodes examined in searching for a
value in the tree is one plus the number of nodes examined when the value was
first inserted into the tree.
12.3-3
We can sort a given set of n numbers by first building a binary search tree containing these numbers (using T REE -I NSERT repeatedly to insert the numbers one by
one) and then printing the numbers by an inorder tree walk. What are the worstcase and best-case running times for this sorting algorithm?
12.3-4
Is the operation of deletion “commutative” in the sense that deleting x and then y
from a binary search tree leaves the same tree as deleting y and then x? Argue why
it is or give a counterexample.
12.3-5
Suppose that instead of each node x keeping the attribute x:p, pointing to x’s
parent, it keeps x:succ, pointing to x’s successor. Give pseudocode for S EARCH,
I NSERT, and D ELETE on a binary search tree T using this representation. These
procedures should operate in time O.h/, where h is the height of the tree T . (Hint:
You may wish to implement a subroutine that returns the parent of a node.)
12.3-6
When node ´ in T REE -D ELETE has two children, we could choose node y as
its predecessor rather than its successor. What other changes to T REE -D ELETE
would be necessary if we did so? Some have argued that a fair strategy, giving
equal priority to predecessor and successor, yields better empirical performance.
How might T REE -D ELETE be changed to implement such a fair strategy?
? 12.4 Randomly built binary search trees
We have shown that each of the basic operations on a binary search tree runs
in O.h/ time, where h is the height of the tree. The height of a binary search
300
Chapter 12 Binary Search Trees
tree varies, however, as items are inserted and deleted. If, for example, the n items
are inserted in strictly increasing order, the tree will be a chain with height n ! 1.
On the other hand, Exercise B.5-4 shows that h # blg nc. As with quicksort, we
can show that the behavior of the average case is much closer to the best case than
to the worst case.
Unfortunately, little is known about the average height of a binary search tree
when both insertion and deletion are used to create it. When the tree is created
by insertion alone, the analysis becomes more tractable. Let us therefore define a
randomly built binary search tree on n keys as one that arises from inserting the
keys in random order into an initially empty tree, where each of the nŠ permutations
of the input keys is equally likely. (Exercise 12.4-3 asks you to show that this notion
is different from assuming that every binary search tree on n keys is equally likely.)
In this section, we shall prove the following theorem.
Theorem 12.4
The expected height of a randomly built binary search tree on n distinct keys is
O.lg n/.
Proof We start by defining three random variables that help measure the height
of a randomly built binary search tree. We denote the height of a randomly built
binary search on n keys by Xn , and we define the exponential height Yn D 2Xn .
When we build a binary search tree on n keys, we choose one key as that of the
root, and we let Rn denote the random variable that holds this key’s rank within
the set of n keys; that is, Rn holds the position that this key would occupy if the
set of keys were sorted. The value of Rn is equally likely to be any element of the
set f1; 2; : : : ; ng. If Rn D i, then the left subtree of the root is a randomly built
binary search tree on i ! 1 keys, and the right subtree is a randomly built binary
search tree on n ! i keys. Because the height of a binary tree is 1 more than the
larger of the heights of the two subtrees of the root, the exponential height of a
binary tree is twice the larger of the exponential heights of the two subtrees of the
root. If we know that Rn D i, it follows that
Yn D 2 $ max.Yi !1 ; Yn!i / :
As base cases, we have that Y1 D 1, because the exponential height of a tree with 1
node is 20 D 1 and, for convenience, we define Y0 D 0.
Next, define indicator random variables Zn;1 ; Zn;2 ; : : : ; Zn;n , where
Zn;i D I fRn D ig :
Because Rn is equally likely to be any element of f1; 2; : : : ; ng, it follows that
Pr fRn D ig D 1=n for i D 1; 2; : : : ; n, and hence, by Lemma 5.1, we have
E ŒZn;i ! D 1=n ;
(12.1)
12.4 Randomly built binary search trees
301
for i D 1; 2; : : : ; n. Because exactly one value of Zn;i is 1 and all others are 0, we
also have
Yn D
n
X
i D1
Zn;i .2 $ max.Yi !1 ; Yn!i // :
We shall show that E ŒYn ! is polynomial in n, which will ultimately imply that
E ŒXn ! D O.lg n/.
We claim that the indicator random variable Zn;i D I fRn D ig is independent
of the values of Yi !1 and Yn!i . Having chosen Rn D i, the left subtree (whose
exponential height is Yi !1 ) is randomly built on the i ! 1 keys whose ranks are
less than i. This subtree is just like any other randomly built binary search tree
on i ! 1 keys. Other than the number of keys it contains, this subtree’s structure
is not affected at all by the choice of Rn D i, and hence the random variables
Yi !1 and Zn;i are independent. Likewise, the right subtree, whose exponential
height is Yn!i , is randomly built on the n ! i keys whose ranks are greater than i.
Its structure is independent of the value of Rn , and so the random variables Yn!i
and Zn;i are independent. Hence, we have
" n
#
X
E ŒYn ! D E
Zn;i .2 $ max.Yi !1 ; Yn!i //
D
D
D
n
X
i D1
n
X
i D1
i D1
E ŒZn;i .2 $ max.Yi !1 ; Yn!i //!
E ŒZn;i ! E Œ2 $ max.Yi !1 ; Yn!i /! (by independence)
n
X
1
n
i D1
$ E Œ2 $ max.Yi !1 ; Yn!i /!
n
D
2X
E Œmax.Yi !1 ; Yn!i /!
n i D1
n
"
(by linearity of expectation)
2X
.E ŒYi !1 ! C E ŒYn!i !/
n i D1
(by equation (12.1))
(by equation (C.22))
(by Exercise C.3-4) .
Since each term E ŒY0 ! ; E ŒY1 ! ; : : : ; E ŒYn!1 ! appears twice in the last summation,
once as E ŒYi !1 ! and once as E ŒYn!i !, we have the recurrence
E ŒYn ! "
n!1
4X
E ŒYi ! :
n i D0
(12.2)
302
Chapter 12 Binary Search Trees
Using the substitution method, we shall show that for all positive integers n, the
recurrence (12.2) has the solution
!
1 nC3
E ŒYn ! "
:
4
3
In doing so, we shall use the identity
!
!
n!1
X
i C3
nC3
D
:
3
4
i D0
(12.3)
(Exercise 12.4-1 asks you to prove this identity.)
!"
For the base cases, we note that the bounds 0 D Y0 D E ŒY0 ! " .1=4/ 33 D 1=4
! "
and 1 D Y1 D E ŒY1 ! " .1=4/ 1C3
D 1 hold. For the inductive case, we have that
3
E ŒYn ! "
n!1
4X
E ŒYi !
n i D0
n!1
"
D
D
D
D
D
4 X1 i C3
n i D0 4
3
!
n!1
1 X i C3
n i D0
3
!
1 nC3
n
4
1 .n C 3/Š
$
n 4Š .n ! 1/Š
1 .n C 3/Š
$
4
3Š nŠ!
1 nC3
:
4
3
!
(by the inductive hypothesis)
(by equation (12.3))
We have bounded E ŒYn !, but our ultimate goal is to bound E ŒXn !. As Exercise 12.4-4 asks you to show, the function f .x/ D 2x is convex (see page 1199).
Therefore, we can employ Jensen’s inequality (C.26), which says that
#
$
2EŒXn ! " E 2Xn
D E ŒYn ! ;
as follows:
2EŒXn !
"
1 nC3
4
3
!
Problems for Chapter 12
303
1 .n C 3/.n C 2/.n C 1/
$
4
6
n3 C 6n2 C 11n C 6
D
:
24
Taking logarithms of both sides gives E ŒXn ! D O.lg n/.
D
Exercises
12.4-1
Prove equation (12.3).
12.4-2
Describe a binary search tree on n nodes such that the average depth of a node in
the tree is ‚.lg n/ but the height of the tree is !.lg n/. Give an asymptotic upper
bound on the height of an n-node binary search tree in which the average depth of
a node is ‚.lg n/.
12.4-3
Show that the notion of a randomly chosen binary search tree on n keys, where
each binary search tree of n keys is equally likely to be chosen, is different from
the notion of a randomly built binary search tree given in this section. (Hint: List
the possibilities when n D 3.)
12.4-4
Show that the function f .x/ D 2x is convex.
12.4-5 ?
Consider R ANDOMIZED -Q UICKSORT operating on a sequence of n distinct input
numbers. Prove that for any constant k > 0, all but O.1=nk / of the nŠ input
permutations yield an O.n lg n/ running time.
Problems
12-1 Binary search trees with equal keys
Equal keys pose a problem for the implementation of binary search trees.
a. What is the asymptotic performance of T REE -I NSERT when used to insert n
items with identical keys into an initially empty binary search tree?
We propose to improve T REE -I NSERT by testing before line 5 to determine whether
´:key D x:key and by testing before line 11 to determine whether ´:key D y:key.
304
Chapter 12 Binary Search Trees
If equality holds, we implement one of the following strategies. For each strategy,
find the asymptotic performance of inserting n items with identical keys into an
initially empty binary search tree. (The strategies are described for line 5, in which
we compare the keys of ´ and x. Substitute y for x to arrive at the strategies for
line 11.)
b. Keep a boolean flag x:b at node x, and set x to either x:left or x:right based
on the value of x:b, which alternates between FALSE and TRUE each time we
visit x while inserting a node with the same key as x.
c. Keep a list of nodes with equal keys at x, and insert ´ into the list.
d. Randomly set x to either x:left or x:right. (Give the worst-case performance
and informally derive the expected running time.)
12-2 Radix trees
Given two strings a D a0 a1 : : : ap and b D b0 b1 : : : bq , where each ai and each bj
is in some ordered set of characters, we say that string a is lexicographically less
than string b if either
1. there exists an integer j , where 0 " j " min.p; q/, such that ai D bi for all
i D 0; 1; : : : ; j ! 1 and aj < bj , or
2. p < q and ai D bi for all i D 0; 1; : : : ; p.
For example, if a and b are bit strings, then 10100 < 10110 by rule 1 (letting
j D 3) and 10100 < 101000 by rule 2. This ordering is similar to that used in
English-language dictionaries.
The radix tree data structure shown in Figure 12.5 stores the bit strings 1011,
10, 011, 100, and 0. When searching for a key a D a0 a1 : : : ap , we go left at a
node of depth i if ai D 0 and right if ai D 1. Let S be a set of distinct bit strings
whose lengths sum to n. Show how to use a radix tree to sort S lexicographically
in ‚.n/ time. For the example in Figure 12.5, the output of the sort should be the
sequence 0, 011, 10, 100, 1011.
12-3 Average node depth in a randomly built binary search tree
In this problem, we prove that the average depth of a node in a randomly built
binary search tree with n nodes is O.lg n/. Although this result is weaker than
that of Theorem 12.4, the technique we shall use reveals a surprising similarity
between the building of a binary search tree and the execution of R ANDOMIZED Q UICKSORT from Section 7.3.
We define the total path length P .T / of a binary tree T as the sum, over all
nodes x in T , of the depth of node x, which we denote by d.x; T /.
Problems for Chapter 12
305
0
0
1
1
1
011
0
100
10
0
1
1
1011
Figure 12.5 A radix tree storing the bit strings 1011, 10, 011, 100, and 0. We can determine each
node’s key by traversing the simple path from the root to that node. There is no need, therefore, to
store the keys in the nodes; the keys appear here for illustrative purposes only. Nodes are heavily
shaded if the keys corresponding to them are not in the tree; such nodes are present only to establish
a path to other nodes.
a. Argue that the average depth of a node in T is
1X
1
d.x; T / D P .T / :
n x2T
n
Thus, we wish to show that the expected value of P .T / is O.n lg n/.
b. Let TL and TR denote the left and right subtrees of tree T , respectively. Argue
that if T has n nodes, then
P .T / D P .TL / C P .TR / C n ! 1 :
c. Let P .n/ denote the average total path length of a randomly built binary search
tree with n nodes. Show that
n!1
1X
.P .i/ C P .n ! i ! 1/ C n ! 1/ :
P .n/ D
n i D0
d. Show how to rewrite P .n/ as
n!1
2X
P .k/ C ‚.n/ :
P .n/ D
n
kD1
e. Recalling the alternative analysis of the randomized version of quicksort given
in Problem 7-3, conclude that P .n/ D O.n lg n/.
306
Chapter 12 Binary Search Trees
At each recursive invocation of quicksort, we choose a random pivot element to
partition the set of elements being sorted. Each node of a binary search tree partitions the set of elements that fall into the subtree rooted at that node.
f. Describe an implementation of quicksort in which the comparisons to sort a set
of elements are exactly the same as the comparisons to insert the elements into
a binary search tree. (The order in which comparisons are made may differ, but
the same comparisons must occur.)
12-4 Number of different binary trees
Let bn denote the number of different binary trees with n nodes. In this problem,
you will find a formula for bn , as well as an asymptotic estimate.
a. Show that b0 D 1 and that, for n # 1,
bn D
n!1
X
bk bn!1!k :
kD0
b. Referring to Problem 4-4 for the definition of a generating function, let B.x/
be the generating function
B.x/ D
1
X
bn x n :
nD0
Show that B.x/ D xB.x/2 C 1, and hence one way to express B.x/ in closed
form is
B.x/ D
p
"
1 !
1 ! 1 ! 4x :
2x
The Taylor expansion of f .x/ around the point x D a is given by
f .x/ D
1
X
f .k/ .a/
kD0
kŠ
.x ! a/k ;
where f .k/ .x/ is the kth derivative of f evaluated at x.
c. Show that
1
2n
bn D
nC1 n
!
Notes for Chapter 12
307
p
(the nth Catalan number) by using the Taylor expansion of 1 ! 4x around
x D 0. (If you wish, instead of using the Taylor expansion, you may use
the generalization of the binomial expansion (C.4) to nonintegral exponents
n,
! "
where for any real number n and for any integer k, we interpret kn to be
n.n ! 1/ $ $ $ .n ! k C 1/=kŠ if k # 0, and 0 otherwise.)
d. Show that
bn D p
4n
.1 C O.1=n// :
%n3=2
Chapter notes
Knuth [211] contains a good discussion of simple binary search trees as well as
many variations. Binary search trees seem to have been independently discovered
by a number of people in the late 1950s. Radix trees are often called “tries,” which
comes from the middle letters in the word retrieval. Knuth [211] also discusses
them.
Many texts, including the first two editions of this book, have a somewhat simpler method of deleting a node from a binary search tree when both of its children
are present. Instead of replacing node ´ by its successor y, we delete node y but
copy its key and satellite data into node ´. The downside of this approach is that
the node actually deleted might not be the node passed to the delete procedure. If
other components of a program maintain pointers to nodes in the tree, they could
mistakenly end up with “stale” pointers to nodes that have been deleted. Although
the deletion method presented in this edition of this book is a bit more complicated,
it guarantees that a call to delete node ´ deletes node ´ and only node ´.
Section 15.5 will show how to construct an optimal binary search tree when
we know the search frequencies before constructing the tree. That is, given the
frequencies of searching for each key and the frequencies of searching for values
that fall between keys in the tree, we construct a binary search tree for which a
set of searches that follows these frequencies examines the minimum number of
nodes.
The proof in Section 12.4 that bounds the expected height of a randomly built
binary search tree is due to Aslam [24]. Martı́nez and Roura [243] give randomized
algorithms for insertion into and deletion from binary search trees in which the
result of either operation is a random binary search tree. Their definition of a
random binary search tree differs—only slightly—from that of a randomly built
binary search tree in this chapter, however.
13
Red-Black Trees
Chapter 12 showed that a binary search tree of height h can support any of the basic
dynamic-set operations—such as S EARCH, P REDECESSOR, S UCCESSOR, M INI MUM , M AXIMUM , I NSERT, and D ELETE—in O.h/ time. Thus, the set operations
are fast if the height of the search tree is small. If its height is large, however, the
set operations may run no faster than with a linked list. Red-black trees are one
of many search-tree schemes that are “balanced” in order to guarantee that basic
dynamic-set operations take O.lg n/ time in the worst case.
13.1 Properties of red-black trees
A red-black tree is a binary search tree with one extra bit of storage per node: its
color, which can be either RED or BLACK. By constraining the node colors on any
simple path from the root to a leaf, red-black trees ensure that no such path is more
than twice as long as any other, so that the tree is approximately balanced.
Each node of the tree now contains the attributes color, key, left, right, and p. If
a child or the parent of a node does not exist, the corresponding pointer attribute
of the node contains the value NIL. We shall regard these NILs as being pointers to
leaves (external nodes) of the binary search tree and the normal, key-bearing nodes
as being internal nodes of the tree.
A red-black tree is a binary tree that satisfies the following red-black properties:
1. Every node is either red or black.
2. The root is black.
3. Every leaf (NIL) is black.
4. If a node is red, then both its children are black.
5. For each node, all simple paths from the node to descendant leaves contain the
same number of black nodes.
13.1 Properties of red-black trees
309
Figure 13.1(a) shows an example of a red-black tree.
As a matter of convenience in dealing with boundary conditions in red-black
tree code, we use a single sentinel to represent NIL (see page 238). For a red-black
tree T , the sentinel T:nil is an object with the same attributes as an ordinary node
in the tree. Its color attribute is BLACK, and its other attributes—p, left, right,
and key—can take on arbitrary values. As Figure 13.1(b) shows, all pointers to NIL
are replaced by pointers to the sentinel T:nil.
We use the sentinel so that we can treat a NIL child of a node x as an ordinary
node whose parent is x. Although we instead could add a distinct sentinel node
for each NIL in the tree, so that the parent of each NIL is well defined, that approach would waste space. Instead, we use the one sentinel T:nil to represent all
the NILs—all leaves and the root’s parent. The values of the attributes p, left, right,
and key of the sentinel are immaterial, although we may set them during the course
of a procedure for our convenience.
We generally confine our interest to the internal nodes of a red-black tree, since
they hold the key values. In the remainder of this chapter, we omit the leaves when
we draw red-black trees, as shown in Figure 13.1(c).
We call the number of black nodes on any simple path from, but not including, a
node x down to a leaf the black-height of the node, denoted bh.x/. By property 5,
the notion of black-height is well defined, since all descending simple paths from
the node have the same number of black nodes. We define the black-height of a
red-black tree to be the black-height of its root.
The following lemma shows why red-black trees make good search trees.
Lemma 13.1
A red-black tree with n internal nodes has height at most 2 lg.n C 1/.
Proof We start by showing that the subtree rooted at any node x contains at least
2bh.x/ ! 1 internal nodes. We prove this claim by induction on the height of x. If
the height of x is 0, then x must be a leaf (T:nil), and the subtree rooted at x indeed
contains at least 2bh.x/ ! 1 D 20 ! 1 D 0 internal nodes. For the inductive step,
consider a node x that has positive height and is an internal node with two children.
Each child has a black-height of either bh.x/ or bh.x/ ! 1, depending on whether
its color is red or black, respectively. Since the height of a child of x is less than
the height of x itself, we can apply the inductive hypothesis to conclude that each
child has at least 2bh.x/!1 ! 1 internal nodes. Thus, the subtree rooted at x contains
at least .2bh.x/!1 ! 1/ C .2bh.x/!1 ! 1/ C 1 D 2bh.x/ ! 1 internal nodes, which proves
the claim.
To complete the proof of the lemma, let h be the height of the tree. According
to property 4, at least half the nodes on any simple path from the root to a leaf, not
310
Chapter 13 Red-Black Trees
3
3
2
2
1
1
7
3
NIL
1
NIL
12
1
NIL
21
2
1
NIL
41
17
14
10
16
15
NIL
26
1
NIL
19
NIL
NIL
2
1
1
20
NIL
23
NIL
1
NIL
30
1
28
NIL
NIL
NIL
1
1
38
35
1
NIL
NIL
2
NIL
47
NIL
NIL
39
NIL
NIL
(a)
26
41
17
14
21
16
10
7
12
19
15
47
30
23
28
38
20
35
39
3
T:nil
(b)
26
41
17
14
21
16
10
7
3
12
15
19
47
30
23
28
20
38
35
39
(c)
Figure 13.1 A red-black tree with black nodes darkened and red nodes shaded. Every node in a
red-black tree is either red or black, the children of a red node are both black, and every simple path
from a node to a descendant leaf contains the same number of black nodes. (a) Every leaf, shown
as a NIL , is black. Each non-NIL node is marked with its black-height; NIL s have black-height 0.
(b) The same red-black tree but with each NIL replaced by the single sentinel T: nil, which is always
black, and with black-heights omitted. The root’s parent is also the sentinel. (c) The same red-black
tree but with leaves and the root’s parent omitted entirely. We shall use this drawing style in the
remainder of this chapter.
13.1 Properties of red-black trees
311
including the root, must be black. Consequently, the black-height of the root must
be at least h=2; thus,
n # 2h=2 ! 1 :
Moving the 1 to the left-hand side and taking logarithms on both sides yields
lg.n C 1/ # h=2, or h " 2 lg.n C 1/.
As an immediate consequence of this lemma, we can implement the dynamic-set
operations S EARCH, M INIMUM, M AXIMUM, S UCCESSOR, and P REDECESSOR
in O.lg n/ time on red-black trees, since each can run in O.h/ time on a binary
search tree of height h (as shown in Chapter 12) and any red-black tree on n nodes
is a binary search tree with height O.lg n/. (Of course, references to NIL in the
algorithms of Chapter 12 would have to be replaced by T:nil.) Although the algorithms T REE -I NSERT and T REE -D ELETE from Chapter 12 run in O.lg n/ time
when given a red-black tree as input, they do not directly support the dynamic-set
operations I NSERT and D ELETE, since they do not guarantee that the modified binary search tree will be a red-black tree. We shall see in Sections 13.3 and 13.4,
however, how to support these two operations in O.lg n/ time.
Exercises
13.1-1
In the style of Figure 13.1(a), draw the complete binary search tree of height 3 on
the keys f1; 2; : : : ; 15g. Add the NIL leaves and color the nodes in three different
ways such that the black-heights of the resulting red-black trees are 2, 3, and 4.
13.1-2
Draw the red-black tree that results after T REE -I NSERT is called on the tree in
Figure 13.1 with key 36. If the inserted node is colored red, is the resulting tree a
red-black tree? What if it is colored black?
13.1-3
Let us define a relaxed red-black tree as a binary search tree that satisfies redblack properties 1, 3, 4, and 5. In other words, the root may be either red or black.
Consider a relaxed red-black tree T whose root is red. If we color the root of T
black but make no other changes to T , is the resulting tree a red-black tree?
13.1-4
Suppose that we “absorb” every red node in a red-black tree into its black parent,
so that the children of the red node become children of the black parent. (Ignore
what happens to the keys.) What are the possible degrees of a black node after all
312
Chapter 13 Red-Black Trees
its red children are absorbed? What can you say about the depths of the leaves of
the resulting tree?
13.1-5
Show that the longest simple path from a node x in a red-black tree to a descendant
leaf has length at most twice that of the shortest simple path from node x to a
descendant leaf.
13.1-6
What is the largest possible number of internal nodes in a red-black tree with blackheight k? What is the smallest possible number?
13.1-7
Describe a red-black tree on n keys that realizes the largest possible ratio of red internal nodes to black internal nodes. What is this ratio? What tree has the smallest
possible ratio, and what is the ratio?
13.2 Rotations
The search-tree operations T REE -I NSERT and T REE -D ELETE, when run on a redblack tree with n keys, take O.lg n/ time. Because they modify the tree, the result
may violate the red-black properties enumerated in Section 13.1. To restore these
properties, we must change the colors of some of the nodes in the tree and also
change the pointer structure.
We change the pointer structure through rotation, which is a local operation in
a search tree that preserves the binary-search-tree property. Figure 13.2 shows the
two kinds of rotations: left rotations and right rotations. When we do a left rotation
on a node x, we assume that its right child y is not T:nil; x may be any node in
the tree whose right child is not T:nil. The left rotation “pivots” around the link
from x to y. It makes y the new root of the subtree, with x as y’s left child and y’s
left child as x’s right child.
The pseudocode for L EFT-ROTATE assumes that x:right ¤ T:nil and that the
root’s parent is T:nil.
13.2 Rotations
LEFT-ROTATE(T, x)
y
γ
x
α
β
313
RIGHT-ROTATE(T, y)
x
y
α
β
γ
Figure 13.2 The rotation operations on a binary search tree. The operation L EFT-ROTATE.T; x/
transforms the configuration of the two nodes on the right into the configuration on the left by changing a constant number of pointers. The inverse operation R IGHT-ROTATE.T; y/ transforms the configuration on the left into the configuration on the right. The letters ˛, ˇ, and & represent arbitrary
subtrees. A rotation operation preserves the binary-search-tree property: the keys in ˛ precede x: key,
which precedes the keys in ˇ, which precede y: key, which precedes the keys in & .
L EFT-ROTATE .T; x/
1 y D x:right
2 x:right D y:left
3 if y:left ¤ T:nil
4
y:left:p D x
5 y:p D x:p
6 if x:p == T:nil
7
T:root D y
8 elseif x == x:p:left
9
x:p:left D y
10 else x:p:right D y
11 y:left D x
12 x:p D y
// set y
// turn y’s left subtree into x’s right subtree
// link x’s parent to y
// put x on y’s left
Figure 13.3 shows an example of how L EFT-ROTATE modifies a binary search
tree. The code for R IGHT-ROTATE is symmetric. Both L EFT-ROTATE and R IGHTROTATE run in O.1/ time. Only pointers are changed by a rotation; all other
attributes in a node remain the same.
Exercises
13.2-1
Write pseudocode for R IGHT-ROTATE.
13.2-2
Argue that in every n-node binary search tree, there are exactly n ! 1 possible
rotations.
314
Chapter 13 Red-Black Trees
7
4
3
11 x
6
9
18 y
2
14
12
LEFT-ROTATE(T, x)
19
17
22
20
7
4
3
2
18 y
6
x 11
9
19
14
12
22
17
20
Figure 13.3 An example of how the procedure L EFT-ROTATE.T; x/ modifies a binary search tree.
Inorder tree walks of the input tree and the modified tree produce the same listing of key values.
13.2-3
Let a, b, and c be arbitrary nodes in subtrees ˛, ˇ, and & , respectively, in the left
tree of Figure 13.2. How do the depths of a, b, and c change when a left rotation
is performed on node x in the figure?
13.2-4
Show that any arbitrary n-node binary search tree can be transformed into any other
arbitrary n-node binary search tree using O.n/ rotations. (Hint: First show that at
most n ! 1 right rotations suffice to transform the tree into a right-going chain.)
13.2-5 ?
We say that a binary search tree T1 can be right-converted to binary search tree T2
if it is possible to obtain T2 from T1 via a series of calls to R IGHT-ROTATE. Give
an example of two trees T1 and T2 such that T1 cannot be right-converted to T2 .
Then, show that if a tree T1 can be right-converted to T2 , it can be right-converted
using O.n2 / calls to R IGHT-ROTATE.
13.3 Insertion
315
13.3 Insertion
We can insert a node into an n-node red-black tree in O.lg n/ time. To do so, we
use a slightly modified version of the T REE -I NSERT procedure (Section 12.3) to
insert node ´ into the tree T as if it were an ordinary binary search tree, and then we
color ´ red. (Exercise 13.3-1 asks you to explain why we choose to make node ´
red rather than black.) To guarantee that the red-black properties are preserved, we
then call an auxiliary procedure RB-I NSERT-F IXUP to recolor nodes and perform
rotations. The call RB-I NSERT .T; ´/ inserts node ´, whose key is assumed to have
already been filled in, into the red-black tree T .
-
-
-
RB-I NSERT .T; ´/
1 y D T:nil
2 x D T:root
3 while x ¤ T:nil "Tree not
empty
4
y Dx
5
if ´:key < x:key
6
x D x:left
7
else x D x:right
8 ´:p D y
9 if y == T:nil
10
T:root D ´
11 elseif ´:key < y:key
12
y:left D ´
13 else y:right D ´
14 ´:left D T:nil
15 ´:right D T:nil
16 ´:color D RED
17 RB-I NSERT-F IXUP .T; ´/
The procedures T REE -I NSERT and RB-I NSERT differ in four ways. First, all
instances of NIL in T REE -I NSERT are replaced by T:nil. Second, we set ´:left
and ´:right to T:nil in lines 14–15 of RB-I NSERT, in order to maintain the
proper tree structure. Third, we color ´ red in line 16. Fourth, because coloring ´ red may cause a violation of one of the red-black properties, we call
RB-I NSERT-F IXUP .T; ´/ in line 17 of RB-I NSERT to restore the red-black properties.
316
Chapter 13 Red-Black Trees
RB-I NSERT-F IXUP .T; ´/
1 while ´:p:color == RED
2
if ´:p == ´:p:p:left
3
y D ´:p:p:right
4
if y:color == RED
5
´:p:color D BLACK
6
y:color D BLACK
7
´:p:p:color D RED
8
´ D ´:p:p
9
else if ´ == ´:p:right
10
´ D ´:p
11
L EFT-ROTATE .T; ´/
12
´:p:color D BLACK
13
´:p:p:color D RED
14
R IGHT-ROTATE .T; ´:p:p/
15
else (same as then clause
with “right” and “left” exchanged)
16 T:root:color D BLACK
nthe
// case 1
// case 1
// case 1
// case 1
// case 2
// case 2
// case 3
// case 3
// case 3
To understand how RB-I NSERT-F IXUP works, we shall break our examination
of the code into three major steps. First, we shall determine what violations of
the red-black properties are introduced in RB-I NSERT when node ´ is inserted
and colored red. Second, we shall examine the overall goal of the while loop in
lines 1–15. Finally, we shall explore each of the three cases1 within the while
loop’s body and see how they accomplish the goal. Figure 13.4 shows how RBI NSERT-F IXUP operates on a sample red-black tree.
Which of the red-black properties might be violated upon the call to RBI NSERT-F IXUP? Property 1 certainly continues to hold, as does property 3, since
both children of the newly inserted red node are the sentinel T:nil. Property 5,
which says that the number of black nodes is the same on every simple path from
a given node, is satisfied as well, because node ´ replaces the (black) sentinel, and
node ´ is red with sentinel children. Thus, the only properties that might be violated are property 2, which requires the root to be black, and property 4, which
says that a red node cannot have a red child. Both possible violations are due to ´
being colored red. Property 2 is violated if ´ is the root, and property 4 is violated
if ´’s parent is red. Figure 13.4(a) shows a violation of property 4 after the node ´
has been inserted.
1 Case 2 falls through into case 3, and so these two cases are not mutually exclusive.
13.3 Insertion
317
11
14
2
1
(a)
7
15
5
8 y
4
z
Case 1
11
14 y
2
1
(b)
7
5
15
z
8
4
Case 2
11
14 y
7
(c)
z
8
2
1
15
5
Case 3
4
7
z
(d)
2
11
1
5
4
8
14
15
Figure 13.4 The operation of RB-I NSERT-F IXUP. (a) A node ´ after insertion. Because both ´
and its parent ´: p are red, a violation of property 4 occurs. Since ´’s uncle y is red, case 1 in the
code applies. We recolor nodes and move the pointer ´ up the tree, resulting in the tree shown in (b).
Once again, ´ and its parent are both red, but ´’s uncle y is black. Since ´ is the right child of ´: p,
case 2 applies. We perform a left rotation, and the tree that results is shown in (c). Now, ´ is the left
child of its parent, and case 3 applies. Recoloring and right rotation yield the tree in (d), which is a
legal red-black tree.
318
Chapter 13 Red-Black Trees
The while loop in lines 1–15 maintains the following three-part invariant at the
start of each iteration of the loop:
a. Node ´ is red.
b. If ´:p is the root, then ´:p is black.
c. If the tree violates any of the red-black properties, then it violates at most
one of them, and the violation is of either property 2 or property 4. If the
tree violates property 2, it is because ´ is the root and is red. If the tree
violates property 4, it is because both ´ and ´:p are red.
Part (c), which deals with violations of red-black properties, is more central to
showing that RB-I NSERT-F IXUP restores the red-black properties than parts (a)
and (b), which we use along the way to understand situations in the code. Because
we’ll be focusing on node ´ and nodes near it in the tree, it helps to know from
part (a) that ´ is red. We shall use part (b) to show that the node ´:p:p exists when
we reference it in lines 2, 3, 7, 8, 13, and 14.
Recall that we need to show that a loop invariant is true prior to the first iteration of the loop, that each iteration maintains the loop invariant, and that the loop
invariant gives us a useful property at loop termination.
We start with the initialization and termination arguments. Then, as we examine how the body of the loop works in more detail, we shall argue that the loop
maintains the invariant upon each iteration. Along the way, we shall also demonstrate that each iteration of the loop has two possible outcomes: either the pointer ´
moves up the tree, or we perform some rotations and then the loop terminates.
Initialization: Prior to the first iteration of the loop, we started with a red-black
tree with no violations, and we added a red node ´. We show that each part of
the invariant holds at the time RB-I NSERT-F IXUP is called:
a. When RB-I NSERT-F IXUP is called, ´ is the red node that was added.
b. If ´:p is the root, then ´:p started out black and did not change prior to the
call of RB-I NSERT-F IXUP.
c. We have already seen that properties 1, 3, and 5 hold when RB-I NSERTF IXUP is called.
If the tree violates property 2, then the red root must be the newly added
node ´, which is the only internal node in the tree. Because the parent and
both children of ´ are the sentinel, which is black, the tree does not also
violate property 4. Thus, this violation of property 2 is the only violation of
red-black properties in the entire tree.
If the tree violates property 4, then, because the children of node ´ are black
sentinels and the tree had no other violations prior to ´ being added, the
13.3 Insertion
319
violation must be because both ´ and ´:p are red. Moreover, the tree violates
no other red-black properties.
Termination: When the loop terminates, it does so because ´:p is black. (If ´ is
the root, then ´:p is the sentinel T:nil, which is black.) Thus, the tree does not
violate property 4 at loop termination. By the loop invariant, the only property
that might fail to hold is property 2. Line 16 restores this property, too, so that
when RB-I NSERT-F IXUP terminates, all the red-black properties hold.
Maintenance: We actually need to consider six cases in the while loop, but three
of them are symmetric to the other three, depending on whether line 2 determines ´’s parent ´:p to be a left child or a right child of ´’s grandparent ´:p:p.
We have given the code only for the situation in which ´:p is a left child. The
node ´:p:p exists, since by part (b) of the loop invariant, if ´:p is the root,
then ´:p is black. Since we enter a loop iteration only if ´:p is red, we know
that ´:p cannot be the root. Hence, ´:p:p exists.
We distinguish case 1 from cases 2 and 3 by the color of ´’s parent’s sibling,
or “uncle.” Line 3 makes y point to ´’s uncle ´:p:p:right, and line 4 tests y’s
color. If y is red, then we execute case 1. Otherwise, control passes to cases 2
and 3. In all three cases, ´’s grandparent ´:p:p is black, since its parent ´:p is
red, and property 4 is violated only between ´ and ´:p.
Case 1: ´’s uncle y is red
Figure 13.5 shows the situation for case 1 (lines 5–8), which occurs when
both ´:p and y are red. Because ´:p:p is black, we can color both ´:p and y
black, thereby fixing the problem of ´ and ´:p both being red, and we can
color ´:p:p red, thereby maintaining property 5. We then repeat the while loop
with ´:p:p as the new node ´. The pointer ´ moves up two levels in the tree.
Now, we show that case 1 maintains the loop invariant at the start of the next
iteration. We use ´ to denote node ´ in the current iteration, and ´0 D ´:p:p
to denote the node that will be called node ´ at the test in line 1 upon the next
iteration.
a. Because this iteration colors ´:p:p red, node ´0 is red at the start of the next
iteration.
b. The node ´0 :p is ´:p:p:p in this iteration, and the color of this node does not
change. If this node is the root, it was black prior to this iteration, and it
remains black at the start of the next iteration.
c. We have already argued that case 1 maintains property 5, and it does not
introduce a violation of properties 1 or 3.
320
Chapter 13 Red-Black Trees
new z
C
(a)
A
A
D y
B z
α
β
δ
ε
D
B
α
γ
β
new z
B
z
α
A
β
δ
A
ε
α
ε
C
B
D y
γ
δ
γ
C
(b)
C
D
γ
δ
ε
β
Figure 13.5 Case 1 of the procedure RB-I NSERT-F IXUP. Property 4 is violated, since ´ and its
parent ´: p are both red. We take the same action whether (a) ´ is a right child or (b) ´ is a left
child. Each of the subtrees ˛, ˇ, & , ı, and " has a black root, and each has the same black-height.
The code for case 1 changes the colors of some nodes, preserving property 5: all downward simple
paths from a node to a leaf have the same number of blacks. The while loop continues with node ´’s
grandparent ´: p: p as the new ´. Any violation of property 4 can now occur only between the new ´,
which is red, and its parent, if it is red as well.
If node ´0 is the root at the start of the next iteration, then case 1 corrected
the lone violation of property 4 in this iteration. Since ´0 is red and it is the
root, property 2 becomes the only one that is violated, and this violation is
due to ´0 .
If node ´0 is not the root at the start of the next iteration, then case 1 has
not created a violation of property 2. Case 1 corrected the lone violation
of property 4 that existed at the start of this iteration. It then made ´0 red
and left ´0 :p alone. If ´0 :p was black, there is no violation of property 4.
If ´0 :p was red, coloring ´0 red created one violation of property 4 between ´0
and ´0 :p.
Case 2: ´’s uncle y is black and ´ is a right child
Case 3: ´’s uncle y is black and ´ is a left child
In cases 2 and 3, the color of ´’s uncle y is black. We distinguish the two cases
according to whether ´ is a right or left child of ´:p. Lines 10–11 constitute
case 2, which is shown in Figure 13.6 together with case 3. In case 2, node ´
is a right child of its parent. We immediately use a left rotation to transform
the situation into case 3 (lines 12–14), in which node ´ is a left child. Because
13.3 Insertion
321
C
C
δ y
A
α
z
γ
α
β
Case 2
δ y
B
B z
A
B
γ
z
α
A
C
β
γ
δ
β
Case 3
Figure 13.6 Cases 2 and 3 of the procedure RB-I NSERT-F IXUP. As in case 1, property 4 is violated
in either case 2 or case 3 because ´ and its parent ´: p are both red. Each of the subtrees ˛, ˇ, & , and ı
has a black root (˛, ˇ, and & from property 4, and ı because otherwise we would be in case 1), and
each has the same black-height. We transform case 2 into case 3 by a left rotation, which preserves
property 5: all downward simple paths from a node to a leaf have the same number of blacks. Case 3
causes some color changes and a right rotation, which also preserve property 5. The while loop then
terminates, because property 4 is satisfied: there are no longer two red nodes in a row.
both ´ and ´:p are red, the rotation affects neither the black-height of nodes
nor property 5. Whether we enter case 3 directly or through case 2, ´’s uncle y
is black, since otherwise we would have executed case 1. Additionally, the
node ´:p:p exists, since we have argued that this node existed at the time that
lines 2 and 3 were executed, and after moving ´ up one level in line 10 and then
down one level in line 11, the identity of ´:p:p remains unchanged. In case 3,
we execute some color changes and a right rotation, which preserve property 5,
and then, since we no longer have two red nodes in a row, we are done. The
while loop does not iterate another time, since ´:p is now black.
We now show that cases 2 and 3 maintain the loop invariant. (As we have just
argued, ´:p will be black upon the next test in line 1, and the loop body will not
execute again.)
a. Case 2 makes ´ point to ´:p, which is red. No further change to ´ or its color
occurs in cases 2 and 3.
b. Case 3 makes ´:p black, so that if ´:p is the root at the start of the next
iteration, it is black.
c. As in case 1, properties 1, 3, and 5 are maintained in cases 2 and 3.
Since node ´ is not the root in cases 2 and 3, we know that there is no violation of property 2. Cases 2 and 3 do not introduce a violation of property 2,
since the only node that is made red becomes a child of a black node by the
rotation in case 3.
Cases 2 and 3 correct the lone violation of property 4, and they do not introduce another violation.
322
Chapter 13 Red-Black Trees
Having shown that each iteration of the loop maintains the invariant, we have
shown that RB-I NSERT-F IXUP correctly restores the red-black properties.
Analysis
What is the running time of RB-I NSERT? Since the height of a red-black tree on n
nodes is O.lg n/, lines 1–16 of RB-I NSERT take O.lg n/ time. In RB-I NSERTF IXUP, the while loop repeats only if case 1 occurs, and then the pointer ´ moves
two levels up the tree. The total number of times the while loop can be executed
is therefore O.lg n/. Thus, RB-I NSERT takes a total of O.lg n/ time. Moreover, it
never performs more than two rotations, since the while loop terminates if case 2
or case 3 is executed.
Exercises
13.3-1
In line 16 of RB-I NSERT, we set the color of the newly inserted node ´ to red.
Observe that if we had chosen to set ´’s color to black, then property 4 of a redblack tree would not be violated. Why didn’t we choose to set ´’s color to black?
13.3-2
Show the red-black trees that result after successively inserting the keys 41; 38; 31;
12; 19; 8 into an initially empty red-black tree.
13.3-3
Suppose that the black-height of each of the subtrees ˛; ˇ; &; ı; " in Figures 13.5
and 13.6 is k. Label each node in each figure with its black-height to verify that
the indicated transformation preserves property 5.
13.3-4
Professor Teach is concerned that RB-I NSERT-F IXUP might set T:nil:color to
RED , in which case the test in line 1 would not cause the loop to terminate when ´
is the root. Show that the professor’s concern is unfounded by arguing that RBI NSERT-F IXUP never sets T:nil:color to RED.
13.3-5
Consider a red-black tree formed by inserting n nodes with RB-I NSERT. Argue
that if n > 1, the tree has at least one red node.
13.3-6
Suggest how to implement RB-I NSERT efficiently if the representation for redblack trees includes no storage for parent pointers.
13.4 Deletion
323
13.4 Deletion
Like the other basic operations on an n-node red-black tree, deletion of a node takes
time O.lg n/. Deleting a node from a red-black tree is a bit more complicated than
inserting a node.
The procedure for deleting a node from a red-black tree is based on the T REE D ELETE procedure (Section 12.3). First, we need to customize the T RANSPLANT
subroutine that T REE -D ELETE calls so that it applies to a red-black tree:
RB-T RANSPLANT .T; u; #/
1 if u:p == T:nil
2
T:root D #
3 elseif u == u:p:left
4
u:p:left D #
5 else u:p:right D #
6 #:p D u:p
The procedure RB-T RANSPLANT differs from T RANSPLANT in two ways. First,
line 1 references the sentinel T:nil instead of NIL. Second, the assignment to #:p in
line 6 occurs unconditionally: we can assign to #:p even if # points to the sentinel.
In fact, we shall exploit the ability to assign to #:p when # D T:nil.
The procedure RB-D ELETE is like the T REE -D ELETE procedure, but with additional lines of pseudocode. Some of the additional lines keep track of a node y
that might cause violations of the red-black properties. When we want to delete
node ´ and ´ has fewer than two children, then ´ is removed from the tree, and we
want y to be ´. When ´ has two children, then y should be ´’s successor, and y
moves into ´’s position in the tree. We also remember y’s color before it is removed from or moved within the tree, and we keep track of the node x that moves
into y’s original position in the tree, because node x might also cause violations
of the red-black properties. After deleting node ´, RB-D ELETE calls an auxiliary
procedure RB-D ELETE -F IXUP, which changes colors and performs rotations to
restore the red-black properties.
324
Chapter 13 Red-Black Trees
RB-D ELETE .T; ´/
1 y D´
2 y-original-color D y:color
3 if ´:left == T:nil
4
x D ´:right
5
RB-T RANSPLANT .T; ´; ´:right/
6 elseif ´:right == T:nil
7
x D ´:left
8
RB-T RANSPLANT .T; ´; ´:left/
9 else y D T REE -M INIMUM .´:right/
10
y-original-color D y:color
11
x D y:right
12
if y:p == ´
13
x:p D y
14
else RB-T RANSPLANT .T; y; y:right/
15
y:right D ´:right
16
y:right:p D y
17
RB-T RANSPLANT .T; ´; y/
18
y:left D ´:left
19
y:left:p D y
20
y:color D ´:color
21 if y-original-color == BLACK
22
RB-D ELETE -F IXUP .T; x/
Although RB-D ELETE contains almost twice as many lines of pseudocode as
T REE -D ELETE, the two procedures have the same basic structure. You can find
each line of T REE -D ELETE within RB-D ELETE (with the changes of replacing
NIL by T:nil and replacing calls to T RANSPLANT by calls to RB-T RANSPLANT),
executed under the same conditions.
Here are the other differences between the two procedures:
!
!
We maintain node y as the node either removed from the tree or moved within
the tree. Line 1 sets y to point to node ´ when ´ has fewer than two children
and is therefore removed. When ´ has two children, line 9 sets y to point to ´’s
successor, just as in T REE -D ELETE, and y will move into ´’s position in the
tree.
Because node y’s color might change, the variable y-original-color stores y’s
color before any changes occur. Lines 2 and 10 set this variable immediately
after assignments to y. When ´ has two children, then y ¤ ´ and node y
moves into node ´’s original position in the red-black tree; line 20 gives y the
same color as ´. We need to save y’s original color in order to test it at the
13.4 Deletion
325
end of RB-D ELETE; if it was black, then removing or moving y could cause
violations of the red-black properties.
!
!
As discussed, we keep track of the node x that moves into node y’s original
position. The assignments in lines 4, 7, and 11 set x to point to either y’s only
child or, if y has no children, the sentinel T:nil. (Recall from Section 12.3
that y has no left child.)
Since node x moves into node y’s original position, the attribute x:p is always
set to point to the original position in the tree of y’s parent, even if x is, in fact,
the sentinel T:nil. Unless ´ is y’s original parent (which occurs only when ´ has
two children and its successor y is ´’s right child), the assignment to x:p takes
place in line 6 of RB-T RANSPLANT. (Observe that when RB-T RANSPLANT
is called in lines 5, 8, or 14, the second parameter passed is the same as x.)
When y’s original parent is ´, however, we do not want x:p to point to y’s original parent, since we are removing that node from the tree. Because node y will
move up to take ´’s position in the tree, setting x:p to y in line 13 causes x:p
to point to the original position of y’s parent, even if x D T:nil.
!
Finally, if node y was black, we might have introduced one or more violations
of the red-black properties, and so we call RB-D ELETE -F IXUP in line 22 to
restore the red-black properties. If y was red, the red-black properties still hold
when y is removed or moved, for the following reasons:
1. No black-heights in the tree have changed.
2. No red nodes have been made adjacent. Because y takes ´’s place in the
tree, along with ´’s color, we cannot have two adjacent red nodes at y’s new
position in the tree. In addition, if y was not ´’s right child, then y’s original
right child x replaces y in the tree. If y is red, then x must be black, and so
replacing y by x cannot cause two red nodes to become adjacent.
3. Since y could not have been the root if it was red, the root remains black.
If node y was black, three problems may arise, which the call of RB-D ELETE F IXUP will remedy. First, if y had been the root and a red child of y becomes the
new root, we have violated property 2. Second, if both x and x:p are red, then
we have violated property 4. Third, moving y within the tree causes any simple
path that previously contained y to have one fewer black node. Thus, property 5
is now violated by any ancestor of y in the tree. We can correct the violation
of property 5 by saying that node x, now occupying y’s original position, has an
“extra” black. That is, if we add 1 to the count of black nodes on any simple path
that contains x, then under this interpretation, property 5 holds. When we remove
or move the black node y, we “push” its blackness onto node x. The problem is
that now node x is neither red nor black, thereby violating property 1. Instead,
326
Chapter 13 Red-Black Trees
node x is either “doubly black” or “red-and-black,” and it contributes either 2 or 1,
respectively, to the count of black nodes on simple paths containing x. The color
attribute of x will still be either RED (if x is red-and-black) or BLACK (if x is
doubly black). In other words, the extra black on a node is reflected in x’s pointing
to the node rather than in the color attribute.
We can now see the procedure RB-D ELETE -F IXUP and examine how it restores
the red-black properties to the search tree.
RB-D ELETE -F IXUP .T; x/
1 while x ¤ T:root and x:color == BLACK
2
if x == x:p:left
3
w D x:p:right
4
if w:color == RED
5
w:color D BLACK
6
x:p:color D RED
7
L EFT-ROTATE .T; x:p/
8
w D x:p:right
9
if w:left:color == BLACK and w:right:color == BLACK
10
w:color D RED
11
x D x:p
12
else if w:right:color == BLACK
13
w:left:color D BLACK
14
w:color D RED
15
R IGHT-ROTATE .T; w/
16
w D x:p:right
17
w:color D x:p:color
18
x:p:color D BLACK
19
w:right:color D BLACK
20
L EFT-ROTATE .T; x:p/
21
x D T:root
22
else (same as then clause with “right” and “left” exchanged)
23 x:color D BLACK
// case 1
// case 1
// case 1
// case 1
// case 2
// case 2
// case 3
// case 3
// case 3
// case 3
// case 4
// case 4
// case 4
// case 4
// case 4
The procedure RB-D ELETE -F IXUP restores properties 1, 2, and 4. Exercises
13.4-1 and 13.4-2 ask you to show that the procedure restores properties 2 and 4,
and so in the remainder of this section, we shall focus on property 1. The goal of
the while loop in lines 1–22 is to move the extra black up the tree until
1. x points to a red-and-black node, in which case we color x (singly) black in
line 23;
2. x points to the root, in which case we simply “remove” the extra black; or
3. having performed suitable rotations and recolorings, we exit the loop.
13.4 Deletion
327
Within the while loop, x always points to a nonroot doubly black node. We
determine in line 2 whether x is a left child or a right child of its parent x:p. (We
have given the code for the situation in which x is a left child; the situation in
which x is a right child—line 22—is symmetric.) We maintain a pointer w to
the sibling of x. Since node x is doubly black, node w cannot be T:nil, because
otherwise, the number of blacks on the simple path from x:p to the (singly black)
leaf w would be smaller than the number on the simple path from x:p to x.
The four cases2 in the code appear in Figure 13.7. Before examining each case
in detail, let’s look more generally at how we can verify that the transformation
in each of the cases preserves property 5. The key idea is that in each case, the
transformation applied preserves the number of black nodes (including x’s extra
black) from (and including) the root of the subtree shown to each of the subtrees
˛; ˇ; : : : ; '. Thus, if property 5 holds prior to the transformation, it continues to
hold afterward. For example, in Figure 13.7(a), which illustrates case 1, the number of black nodes from the root to either subtree ˛ or ˇ is 3, both before and after
the transformation. (Again, remember that node x adds an extra black.) Similarly,
the number of black nodes from the root to any of & , ı, ", and ' is 2, both before and after the transformation. In Figure 13.7(b), the counting must involve the
value c of the color attribute of the root of the subtree shown, which can be either
RED or BLACK . If we define count.RED / D 0 and count.BLACK / D 1, then the
number of black nodes from the root to ˛ is 2 C count.c/, both before and after
the transformation. In this case, after the transformation, the new node x has color
attribute c, but this node is really either red-and-black (if c D RED ) or doubly black
(if c D BLACK ). You can verify the other cases similarly (see Exercise 13.4-5).
Case 1: x’s sibling w is red
Case 1 (lines 5–8 of RB-D ELETE -F IXUP and Figure 13.7(a)) occurs when node w,
the sibling of node x, is red. Since w must have black children, we can switch the
colors of w and x:p and then perform a left-rotation on x:p without violating any
of the red-black properties. The new sibling of x, which is one of w’s children
prior to the rotation, is now black, and thus we have converted case 1 into case 2,
3, or 4.
Cases 2, 3, and 4 occur when node w is black; they are distinguished by the
colors of w’s children.
2 As in RB-I NSERT-F IXUP , the cases in RB-D ELETE -F IXUP are not mutually exclusive.
328
Chapter 13 Red-Black Trees
Case 2: x’s sibling w is black, and both of w’s children are black
In case 2 (lines 10–11 of RB-D ELETE -F IXUP and Figure 13.7(b)), both of w’s
children are black. Since w is also black, we take one black off both x and w,
leaving x with only one black and leaving w red. To compensate for removing
one black from x and w, we would like to add an extra black to x:p, which was
originally either red or black. We do so by repeating the while loop with x:p as
the new node x. Observe that if we enter case 2 through case 1, the new node x
is red-and-black, since the original x:p was red. Hence, the value c of the color
attribute of the new node x is RED, and the loop terminates when it tests the loop
condition. We then color the new node x (singly) black in line 23.
Case 3: x’s sibling w is black, w’s left child is red, and w’s right child is black
Case 3 (lines 13–16 and Figure 13.7(c)) occurs when w is black, its left child
is red, and its right child is black. We can switch the colors of w and its left
child w:left and then perform a right rotation on w without violating any of the
red-black properties. The new sibling w of x is now a black node with a red right
child, and thus we have transformed case 3 into case 4.
Case 4: x’s sibling w is black, and w’s right child is red
Case 4 (lines 17–21 and Figure 13.7(d)) occurs when node x’s sibling w is black
and w’s right child is red. By making some color changes and performing a left rotation on x:p, we can remove the extra black on x, making it singly black, without
violating any of the red-black properties. Setting x to be the root causes the while
loop to terminate when it tests the loop condition.
Analysis
What is the running time of RB-D ELETE? Since the height of a red-black tree of n
nodes is O.lg n/, the total cost of the procedure without the call to RB-D ELETE F IXUP takes O.lg n/ time. Within RB-D ELETE -F IXUP, each of cases 1, 3, and 4
lead to termination after performing a constant number of color changes and at
most three rotations. Case 2 is the only case in which the while loop can be repeated, and then the pointer x moves up the tree at most O.lg n/ times, performing
no rotations. Thus, the procedure RB-D ELETE -F IXUP takes O.lg n/ time and performs at most three rotations, and the overall time for RB-D ELETE is therefore
also O.lg n/.
13.4 Deletion
329
Case 1
B
(a)
x A
α
D w
C
β
γ
E
δ
ε
x A
ζ
α
C
β
E
δ
ε
δ
C
γ
C
β
ε
c′
δ
δ
ε
C new w
α
β
D
γ
ζ
E
δ
ε
ζ
D c
B
E
A
ζ
ζ
B c
E
ε
E
γ
D w
β
ζ
D
Case 4
x A
α
B c
x A
B c
(d)
new x
ζ
E
C
γ
δ
α
D w
β
γ
Case 3
x A
ε
A
B c
α
β
D w
γ
(c)
α
new w C
Case 2
x A
E
B
B c
(b)
D
α
C
β
γ
c′ ε
δ
ζ
new x D T:root
Figure 13.7 The cases in the while loop of the procedure RB-D ELETE -F IXUP. Darkened nodes
have color attributes BLACK, heavily shaded nodes have color attributes RED, and lightly shaded
nodes have color attributes represented by c and c 0 , which may be either RED or BLACK. The letters
˛; ˇ; : : : ; ' represent arbitrary subtrees. Each case transforms the configuration on the left into the
configuration on the right by changing some colors and/or performing a rotation. Any node pointed
to by x has an extra black and is either doubly black or red-and-black. Only case 2 causes the loop to
repeat. (a) Case 1 is transformed to case 2, 3, or 4 by exchanging the colors of nodes B and D and
performing a left rotation. (b) In case 2, the extra black represented by the pointer x moves up the
tree by coloring node D red and setting x to point to node B. If we enter case 2 through case 1, the
while loop terminates because the new node x is red-and-black, and therefore the value c of its color
attribute is RED. (c) Case 3 is transformed to case 4 by exchanging the colors of nodes C and D and
performing a right rotation. (d) Case 4 removes the extra black represented by x by changing some
colors and performing a left rotation (without violating the red-black properties), and then the loop
terminates.
330
Chapter 13 Red-Black Trees
Exercises
13.4-1
Argue that after executing RB-D ELETE -F IXUP, the root of the tree must be black.
13.4-2
Argue that if in RB-D ELETE both x and x:p are red, then property 4 is restored by
the call to RB-D ELETE -F IXUP .T; x/.
13.4-3
In Exercise 13.3-2, you found the red-black tree that results from successively
inserting the keys 41; 38; 31; 12; 19; 8 into an initially empty tree. Now show the
red-black trees that result from the successive deletion of the keys in the order
8; 12; 19; 31; 38; 41.
13.4-4
In which lines of the code for RB-D ELETE -F IXUP might we examine or modify
the sentinel T:nil?
13.4-5
In each of the cases of Figure 13.7, give the count of black nodes from the root of
the subtree shown to each of the subtrees ˛; ˇ; : : : ; ', and verify that each count
remains the same after the transformation. When a node has a color attribute c
or c 0 , use the notation count.c/ or count.c 0 / symbolically in your count.
13.4-6
Professors Skelton and Baron are concerned that at the start of case 1 of RBD ELETE -F IXUP, the node x:p might not be black. If the professors are correct,
then lines 5–6 are wrong. Show that x:p must be black at the start of case 1, so that
the professors have nothing to worry about.
13.4-7
Suppose that a node x is inserted into a red-black tree with RB-I NSERT and then
is immediately deleted with RB-D ELETE. Is the resulting red-black tree the same
as the initial red-black tree? Justify your answer.
Problems for Chapter 13
331
Problems
13-1 Persistent dynamic sets
During the course of an algorithm, we sometimes find that we need to maintain past
versions of a dynamic set as it is updated. We call such a set persistent. One way to
implement a persistent set is to copy the entire set whenever it is modified, but this
approach can slow down a program and also consume much space. Sometimes, we
can do much better.
Consider a persistent set S with the operations I NSERT, D ELETE, and S EARCH,
which we implement using binary search trees as shown in Figure 13.8(a). We
maintain a separate root for every version of the set. In order to insert the key 5
into the set, we create a new node with key 5. This node becomes the left child
of a new node with key 7, since we cannot modify the existing node with key 7.
Similarly, the new node with key 7 becomes the left child of a new node with
key 8 whose right child is the existing node with key 10. The new node with key 8
becomes, in turn, the right child of a new root r 0 with key 4 whose left child is the
existing node with key 3. We thus copy only part of the tree and share some of the
nodes with the original tree, as shown in Figure 13.8(b).
Assume that each tree node has the attributes key, left, and right but no parent.
(See also Exercise 13.3-6.)
4
r
3
r
8
2
7
4
4
3
10
2
7
r′
8
8
7
10
5
(a)
(b)
Figure 13.8 (a) A binary search tree with keys 2; 3; 4; 7; 8; 10. (b) The persistent binary search
tree that results from the insertion of key 5. The most recent version of the set consists of the nodes
reachable from the root r 0 , and the previous version consists of the nodes reachable from r. Heavily
shaded nodes are added when key 5 is inserted.
332
Chapter 13 Red-Black Trees
a. For a general persistent binary search tree, identify the nodes that we need to
change to insert a key k or delete a node y.
b. Write a procedure P ERSISTENT-T REE -I NSERT that, given a persistent tree T
and a key k to insert, returns a new persistent tree T 0 that is the result of inserting k into T .
c. If the height of the persistent binary search tree T is h, what are the time and
space requirements of your implementation of P ERSISTENT-T REE -I NSERT?
(The space requirement is proportional to the number of new nodes allocated.)
d. Suppose that we had included the parent attribute in each node. In this case,
P ERSISTENT-T REE -I NSERT would need to perform additional copying. Prove
that P ERSISTENT-T REE -I NSERT would then require ".n/ time and space,
where n is the number of nodes in the tree.
e. Show how to use red-black trees to guarantee that the worst-case running time
and space are O.lg n/ per insertion or deletion.
13-2 Join operation on red-black trees
The join operation takes two dynamic sets S1 and S2 and an element x such that
for any x1 2 S1 and x2 2 S2 , we have x1 :key " x:key " x2 :key. It returns a set
S D S1 [ fxg [ S2 . In this problem, we investigate how to implement the join
operation on red-black trees.
a. Given a red-black tree T , let us store its black-height as the new attribute T:bh.
Argue that RB-I NSERT and RB-D ELETE can maintain the bh attribute without requiring extra storage in the nodes of the tree and without increasing the
asymptotic running times. Show that while descending through T , we can determine the black-height of each node we visit in O.1/ time per node visited.
We wish to implement the operation RB-J OIN .T1 ; x; T2 /, which destroys T1 and T2
and returns a red-black tree T D T1 [ fxg [ T2 . Let n be the total number of nodes
in T1 and T2 .
b. Assume that T1 :bh # T2 :bh. Describe an O.lg n/-time algorithm that finds a
black node y in T1 with the largest key from among those nodes whose blackheight is T2 :bh.
c. Let Ty be the subtree rooted at y. Describe how Ty [ fxg [ T2 can replace Ty
in O.1/ time without destroying the binary-search-tree property.
d. What color should we make x so that red-black properties 1, 3, and 5 are maintained? Describe how to enforce properties 2 and 4 in O.lg n/ time.
Problems for Chapter 13
333
e. Argue that no generality is lost by making the assumption in part (b). Describe
the symmetric situation that arises when T1 :bh " T2 :bh.
f. Argue that the running time of RB-J OIN is O.lg n/.
13-3 AVL trees
An AVL tree is a binary search tree that is height balanced: for each node x, the
heights of the left and right subtrees of x differ by at most 1. To implement an AVL
tree, we maintain an extra attribute in each node: x:h is the height of node x. As
for any other binary search tree T , we assume that T:root points to the root node.
a. Prove that an AVL tree with n nodes has height O.lg n/. (Hint: Prove that
an AVL tree of height h has at least Fh nodes, where Fh is the hth Fibonacci
number.)
b. To insert into an AVL tree, we first place a node into the appropriate place in binary search tree order. Afterward, the tree might no longer be height balanced.
Specifically, the heights of the left and right children of some node might differ
by 2. Describe a procedure BALANCE .x/, which takes a subtree rooted at x
whose left and right children are height balanced and have heights that differ
by at most 2, i.e., jx:right:h ! x:left:hj " 2, and alters the subtree rooted at x
to be height balanced. (Hint: Use rotations.)
c. Using part (b), describe a recursive procedure AVL-I NSERT .x; ´/ that takes
a node x within an AVL tree and a newly created node ´ (whose key has already been filled in), and adds ´ to the subtree rooted at x, maintaining the
property that x is the root of an AVL tree. As in T REE -I NSERT from Section 12.3, assume that ´:key has already been filled in and that ´:left D NIL
and ´:right D NIL; also assume that ´:h D 0. Thus, to insert the node ´ into
the AVL tree T , we call AVL-I NSERT .T:root; ´/.
d. Show that AVL-I NSERT, run on an n-node AVL tree, takes O.lg n/ time and
performs O.1/ rotations.
13-4 Treaps
If we insert a set of n items into a binary search tree, the resulting tree may be
horribly unbalanced, leading to long search times. As we saw in Section 12.4,
however, randomly built binary search trees tend to be balanced. Therefore, one
strategy that, on average, builds a balanced tree for a fixed set of items would be to
randomly permute the items and then insert them in that order into the tree.
What if we do not have all the items at once? If we receive the items one at a
time, can we still randomly build a binary search tree out of them?
334
Chapter 13 Red-Black Trees
G: 4
B: 7
A: 10
H: 5
E: 23
K: 65
I: 73
Figure 13.9 A treap. Each node x is labeled with x: key : x: priority. For example, the root has
key G and priority 4.
We will examine a data structure that answers this question in the affirmative. A
treap is a binary search tree with a modified way of ordering the nodes. Figure 13.9
shows an example. As usual, each node x in the tree has a key value x:key. In
addition, we assign x:priority, which is a random number chosen independently
for each node. We assume that all priorities are distinct and also that all keys are
distinct. The nodes of the treap are ordered so that the keys obey the binary-searchtree property and the priorities obey the min-heap order property:
!
!
!
If # is a left child of u, then #:key < u:key.
If # is a right child of u, then #:key > u:key.
If # is a child of u, then #:priority > u:priority.
(This combination of properties is why the tree is called a “treap”: it has features
of both a binary search tree and a heap.)
It helps to think of treaps in the following way. Suppose that we insert nodes
x1 ; x2 ; : : : ; xn , with associated keys, into a treap. Then the resulting treap is the
tree that would have been formed if the nodes had been inserted into a normal
binary search tree in the order given by their (randomly chosen) priorities, i.e.,
xi :priority < xj :priority means that we had inserted xi before xj .
a. Show that given a set of nodes x1 ; x2 ; : : : ; xn , with associated keys and priorities, all distinct, the treap associated with these nodes is unique.
b. Show that the expected height of a treap is ‚.lg n/, and hence the expected time
to search for a value in the treap is ‚.lg n/.
Let us see how to insert a new node into an existing treap. The first thing we do
is assign to the new node a random priority. Then we call the insertion algorithm,
which we call T REAP -I NSERT, whose operation is illustrated in Figure 13.10.
Problems for Chapter 13
335
G: 4
B: 7
A: 10
G: 4
H: 5
E: 23
(a)
C: 25
K: 65
B: 7
A: 10
I: 73
H: 5
E: 23
C: 25
G: 4
D: 9
B: 7
A: 10
E: 23
B: 7
K: 65
A: 10
I: 73
D: 9
H: 5
E: 23
D: 9
(d)
G: 4
A: 10
F: 2
H: 5
D: 9
C: 25
F: 2
K: 65
E: 23
(e)
K: 65
I: 73
C: 25
(c)
B: 7
I: 73
G: 4
H: 5
C: 25
(b)
K: 65
I: 73
…
A: 10
B: 7
G: 4
D: 9
C: 25
H: 5
E: 23
(f)
K: 65
I: 73
Figure 13.10 The operation of T REAP -I NSERT . (a) The original treap, prior to insertion. (b) The
treap after inserting a node with key C and priority 25. (c)–(d) Intermediate stages when inserting a
node with key D and priority 9. (e) The treap after the insertion of parts (c) and (d) is done. (f) The
treap after inserting a node with key F and priority 2.
336
Chapter 13 Red-Black Trees
15
9
3
15
18
9
12
6
25
(a)
21
3
18
12
6
25
(b)
21
Figure 13.11 Spines of a binary search tree. The left spine is shaded in (a), and the right spine is
shaded in (b).
c. Explain how T REAP -I NSERT works. Explain the idea in English and give pseudocode. (Hint: Execute the usual binary-search-tree insertion procedure and
then perform rotations to restore the min-heap order property.)
d. Show that the expected running time of T REAP -I NSERT is ‚.lg n/.
T REAP -I NSERT performs a search and then a sequence of rotations. Although
these two operations have the same expected running time, they have different
costs in practice. A search reads information from the treap without modifying it.
In contrast, a rotation changes parent and child pointers within the treap. On most
computers, read operations are much faster than write operations. Thus we would
like T REAP -I NSERT to perform few rotations. We will show that the expected
number of rotations performed is bounded by a constant.
In order to do so, we will need some definitions, which Figure 13.11 depicts.
The left spine of a binary search tree T is the simple path from the root to the node
with the smallest key. In other words, the left spine is the simple path from the
root that consists of only left edges. Symmetrically, the right spine of T is the
simple path from the root consisting of only right edges. The length of a spine is
the number of nodes it contains.
e. Consider the treap T immediately after T REAP -I NSERT has inserted node x.
Let C be the length of the right spine of the left subtree of x. Let D be the
length of the left spine of the right subtree of x. Prove that the total number of
rotations that were performed during the insertion of x is equal to C C D.
We will now calculate the expected values of C and D. Without loss of generality,
we assume that the keys are 1; 2; : : : ; n, since we are comparing them only to one
another.
Notes for Chapter 13
337
For nodes x and y in treap T , where y ¤ x, let k D x:key and i D y:key. We
define indicator random variables
Xi k D I fy is in the right spine of the left subtree of xg :
f. Show that Xi k D 1 if and only if y:priority > x:priority, y:key < x:key, and,
for every ´ such that y:key < ´:key < x:key, we have y:priority < ´:priority.
g. Show that
Pr fXi k D 1g D
D
h. Show that
E ŒC ! D
k!1
X
.k ! i ! 1/Š
.k ! i C 1/Š
1
:
.k ! i C 1/.k ! i/
1
j.j C 1/
j D1
D 1!
1
:
k
i. Use a symmetry argument to show that
E ŒD! D 1 !
1
:
n!kC1
j. Conclude that the expected number of rotations performed when inserting a
node into a treap is less than 2.
Chapter notes
The idea of balancing a search tree is due to Adel’son-Vel’skiı̆ and Landis [2], who
introduced a class of balanced search trees called “AVL trees” in 1962, described in
Problem 13-3. Another class of search trees, called “2-3 trees,” was introduced by
J. E. Hopcroft (unpublished) in 1970. A 2-3 tree maintains balance by manipulating
the degrees of nodes in the tree. Chapter 18 covers a generalization of 2-3 trees
introduced by Bayer and McCreight [35], called “B-trees.”
Red-black trees were invented by Bayer [34] under the name “symmetric binary
B-trees.” Guibas and Sedgewick [155] studied their properties at length and introduced the red/black color convention. Andersson [15] gives a simpler-to-code
338
Chapter 13 Red-Black Trees
variant of red-black trees. Weiss [351] calls this variant AA-trees. An AA-tree is
similar to a red-black tree except that left children may never be red.
Treaps, the subject of Problem 13-4, were proposed by Seidel and Aragon [309].
They are the default implementation of a dictionary in LEDA [253], which is a
well-implemented collection of data structures and algorithms.
There are many other variations on balanced binary trees, including weightbalanced trees [264], k-neighbor trees [245], and scapegoat trees [127]. Perhaps
the most intriguing are the “splay trees” introduced by Sleator and Tarjan [320],
which are “self-adjusting.” (See Tarjan [330] for a good description of splay trees.)
Splay trees maintain balance without any explicit balance condition such as color.
Instead, “splay operations” (which involve rotations) are performed within the tree
every time an access is made. The amortized cost (see Chapter 17) of each operation on an n-node tree is O.lg n/.
Skip lists [286] provide an alternative to balanced binary trees. A skip list is a
linked list that is augmented with a number of additional pointers. Each dictionary
operation runs in expected time O.lg n/ on a skip list of n items.
14
Augmenting Data Structures
Some engineering situations require no more than a “textbook” data structure—such as a doubly linked list, a hash table, or a binary search tree—but many
others require a dash of creativity. Only in rare situations will you need to create an entirely new type of data structure, though. More often, it will suffice to
augment a textbook data structure by storing additional information in it. You can
then program new operations for the data structure to support the desired application. Augmenting a data structure is not always straightforward, however, since the
added information must be updated and maintained by the ordinary operations on
the data structure.
This chapter discusses two data structures that we construct by augmenting redblack trees. Section 14.1 describes a data structure that supports general orderstatistic operations on a dynamic set. We can then quickly find the ith smallest
number in a set or the rank of a given element in the total ordering of the set.
Section 14.2 abstracts the process of augmenting a data structure and provides a
theorem that can simplify the process of augmenting red-black trees. Section 14.3
uses this theorem to help design a data structure for maintaining a dynamic set of
intervals, such as time intervals. Given a query interval, we can then quickly find
an interval in the set that overlaps it.
14.1 Dynamic order statistics
Chapter 9 introduced the notion of an order statistic. Specifically, the ith order
statistic of a set of n elements, where i 2 f1; 2; : : : ; ng, is simply the element in the
set with the ith smallest key. We saw how to determine any order statistic in O.n/
time from an unordered set. In this section, we shall see how to modify red-black
trees so that we can determine any order statistic for a dynamic set in O.lg n/ time.
We shall also see how to compute the rank of an element—its position in the linear
order of the set—in O.lg n/ time.
340
Chapter 14 Augmenting Data Structures
26
20
17
12
14
7
10
7
3
1
2
4
12
1
16
14
1
2
41
21
4
19
2
20
1
7
30
21
key
1
size
28
1
5
1
38
35
1
3
47
39
1
Figure 14.1 An order-statistic tree, which is an augmented red-black tree. Shaded nodes are red,
and darkened nodes are black. In addition to its usual attributes, each node x has an attribute x: size,
which is the number of nodes, other than the sentinel, in the subtree rooted at x.
Figure 14.1 shows a data structure that can support fast order-statistic operations.
An order-statistic tree T is simply a red-black tree with additional information
stored in each node. Besides the usual red-black tree attributes x:key, x:color, x:p,
x:left, and x:right in a node x, we have another attribute, x:size. This attribute
contains the number of (internal) nodes in the subtree rooted at x (including x
itself), that is, the size of the subtree. If we define the sentinel’s size to be 0—that
is, we set T:nil:size to be 0—then we have the identity
x:size D x:left:size C x:right:size C 1 :
We do not require keys to be distinct in an order-statistic tree. (For example, the
tree in Figure 14.1 has two keys with value 14 and two keys with value 21.) In the
presence of equal keys, the above notion of rank is not well defined. We remove
this ambiguity for an order-statistic tree by defining the rank of an element as the
position at which it would be printed in an inorder walk of the tree. In Figure 14.1,
for example, the key 14 stored in a black node has rank 5, and the key 14 stored in
a red node has rank 6.
Retrieving an element with a given rank
Before we show how to maintain this size information during insertion and deletion, let us examine the implementation of two order-statistic queries that use this
additional information. We begin with an operation that retrieves an element with
a given rank. The procedure OS-S ELECT .x; i/ returns a pointer to the node containing the ith smallest key in the subtree rooted at x. To find the node with the ith
smallest key in an order-statistic tree T , we call OS-S ELECT .T:root; i/.
14.1 Dynamic order statistics
341
OS-S ELECT .x; i/
1 r D x:left:size C 1
2 if i == r
3
return x
4 elseif i < r
5
return OS-S ELECT .x:left; i/
6 else return OS-S ELECT .x:right; i ! r/
In line 1 of OS-S ELECT, we compute r, the rank of node x within the subtree
rooted at x. The value of x:left:size is the number of nodes that come before x
in an inorder tree walk of the subtree rooted at x. Thus, x:left:size C 1 is the
rank of x within the subtree rooted at x. If i D r, then node x is the ith smallest
element, and so we return x in line 3. If i < r, then the ith smallest element
resides in x’s left subtree, and so we recurse on x:left in line 5. If i > r, then
the ith smallest element resides in x’s right subtree. Since the subtree rooted at x
contains r elements that come before x’s right subtree in an inorder tree walk, the
ith smallest element in the subtree rooted at x is the .i ! r/th smallest element in
the subtree rooted at x:right. Line 6 determines this element recursively.
To see how OS-S ELECT operates, consider a search for the 17th smallest element in the order-statistic tree of Figure 14.1. We begin with x as the root, whose
key is 26, and with i D 17. Since the size of 26’s left subtree is 12, its rank is 13.
Thus, we know that the node with rank 17 is the 17 ! 13 D 4th smallest element
in 26’s right subtree. After the recursive call, x is the node with key 41, and i D 4.
Since the size of 41’s left subtree is 5, its rank within its subtree is 6. Thus, we
know that the node with rank 4 is the 4th smallest element in 41’s left subtree. After the recursive call, x is the node with key 30, and its rank within its subtree is 2.
Thus, we recurse once again to find the 4! 2 D 2nd smallest element in the subtree
rooted at the node with key 38. We now find that its left subtree has size 1, which
means it is the second smallest element. Thus, the procedure returns a pointer to
the node with key 38.
Because each recursive call goes down one level in the order-statistic tree, the
total time for OS-S ELECT is at worst proportional to the height of the tree. Since
the tree is a red-black tree, its height is O.lg n/, where n is the number of nodes.
Thus, the running time of OS-S ELECT is O.lg n/ for a dynamic set of n elements.
Determining the rank of an element
Given a pointer to a node x in an order-statistic tree T , the procedure OS-R ANK
returns the position of x in the linear order determined by an inorder tree walk
of T .
342
Chapter 14 Augmenting Data Structures
OS-R ANK .T; x/
1 r D x:left:size C 1
2 y Dx
3 while y ¤ T:root
4
if y == y:p:right
5
r D r C y:p:left:size C 1
6
y D y:p
7 return r
The procedure works as follows. We can think of node x’s rank as the number of
nodes preceding x in an inorder tree walk, plus 1 for x itself. OS-R ANK maintains
the following loop invariant:
At the start of each iteration of the while loop of lines 3–6, r is the rank
of x:key in the subtree rooted at node y.
We use this loop invariant to show that OS-R ANK works correctly as follows:
Initialization: Prior to the first iteration, line 1 sets r to be the rank of x:key within
the subtree rooted at x. Setting y D x in line 2 makes the invariant true the
first time the test in line 3 executes.
Maintenance: At the end of each iteration of the while loop, we set y D y:p.
Thus we must show that if r is the rank of x:key in the subtree rooted at y at the
start of the loop body, then r is the rank of x:key in the subtree rooted at y:p
at the end of the loop body. In each iteration of the while loop, we consider
the subtree rooted at y:p. We have already counted the number of nodes in the
subtree rooted at node y that precede x in an inorder walk, and so we must add
the nodes in the subtree rooted at y’s sibling that precede x in an inorder walk,
plus 1 for y:p if it, too, precedes x. If y is a left child, then neither y:p nor any
node in y:p’s right subtree precedes x, and so we leave r alone. Otherwise, y is
a right child and all the nodes in y:p’s left subtree precede x, as does y:p itself.
Thus, in line 5, we add y:p:left:size C 1 to the current value of r.
Termination: The loop terminates when y D T:root, so that the subtree rooted
at y is the entire tree. Thus, the value of r is the rank of x:key in the entire tree.
As an example, when we run OS-R ANK on the order-statistic tree of Figure 14.1
to find the rank of the node with key 38, we get the following sequence of values
of y:key and r at the top of the while loop:
iteration
1
2
3
4
y:key
38
30
41
26
r
2
4
4
17
14.1 Dynamic order statistics
343
The procedure returns the rank 17.
Since each iteration of the while loop takes O.1/ time, and y goes up one level in
the tree with each iteration, the running time of OS-R ANK is at worst proportional
to the height of the tree: O.lg n/ on an n-node order-statistic tree.
Maintaining subtree sizes
Given the size attribute in each node, OS-S ELECT and OS-R ANK can quickly
compute order-statistic information. But unless we can efficiently maintain these
attributes within the basic modifying operations on red-black trees, our work will
have been for naught. We shall now show how to maintain subtree sizes for both
insertion and deletion without affecting the asymptotic running time of either operation.
We noted in Section 13.3 that insertion into a red-black tree consists of two
phases. The first phase goes down the tree from the root, inserting the new node
as a child of an existing node. The second phase goes up the tree, changing colors
and performing rotations to maintain the red-black properties.
To maintain the subtree sizes in the first phase, we simply increment x:size for
each node x on the simple path traversed from the root down toward the leaves. The
new node added gets a size of 1. Since there are O.lg n/ nodes on the traversed
path, the additional cost of maintaining the size attributes is O.lg n/.
In the second phase, the only structural changes to the underlying red-black tree
are caused by rotations, of which there are at most two. Moreover, a rotation is
a local operation: only two nodes have their size attributes invalidated. The link
around which the rotation is performed is incident on these two nodes. Referring
to the code for L EFT-ROTATE .T; x/ in Section 13.2, we add the following lines:
13
14
y:size D x:size
x:size D x:left:size C x:right:size C 1
Figure 14.2 illustrates how the attributes are updated. The change to R IGHTROTATE is symmetric.
Since at most two rotations are performed during insertion into a red-black tree,
we spend only O.1/ additional time updating size attributes in the second phase.
Thus, the total time for insertion into an n-node order-statistic tree is O.lg n/,
which is asymptotically the same as for an ordinary red-black tree.
Deletion from a red-black tree also consists of two phases: the first operates
on the underlying search tree, and the second causes at most three rotations and
otherwise performs no structural changes. (See Section 13.4.) The first phase
either removes one node y from the tree or moves upward it within the tree. To
update the subtree sizes, we simply traverse a simple path from node y (starting
from its original position within the tree) up to the root, decrementing the size
344
Chapter 14 Augmenting Data Structures
93
42
11
19
x
6
LEFT-ROTATE(T, x)
y
42
19
7
RIGHT-ROTATE(T, y)
4
x
93
12
6
4
y
7
Figure 14.2 Updating subtree sizes during rotations. The link around which we rotate is incident
on the two nodes whose size attributes need to be updated. The updates are local, requiring only the
size information stored in x, y, and the roots of the subtrees shown as triangles.
attribute of each node on the path. Since this path has length O.lg n/ in an nnode red-black tree, the additional time spent maintaining size attributes in the first
phase is O.lg n/. We handle the O.1/ rotations in the second phase of deletion
in the same manner as for insertion. Thus, both insertion and deletion, including
maintaining the size attributes, take O.lg n/ time for an n-node order-statistic tree.
Exercises
14.1-1
Show how OS-S ELECT .T:root; 10/ operates on the red-black tree T of Figure 14.1.
14.1-2
Show how OS-R ANK .T; x/ operates on the red-black tree T of Figure 14.1 and
the node x with x:key D 35.
14.1-3
Write a nonrecursive version of OS-S ELECT.
14.1-4
Write a recursive procedure OS-K EY-R ANK .T; k/ that takes as input an orderstatistic tree T and a key k and returns the rank of k in the dynamic set represented
by T . Assume that the keys of T are distinct.
14.1-5
Given an element x in an n-node order-statistic tree and a natural number i, how
can we determine the ith successor of x in the linear order of the tree in O.lg n/
time?
14.2 How to augment a data structure
345
14.1-6
Observe that whenever we reference the size attribute of a node in either OSS ELECT or OS-R ANK, we use it only to compute a rank. Accordingly, suppose
we store in each node its rank in the subtree of which it is the root. Show how to
maintain this information during insertion and deletion. (Remember that these two
operations can cause rotations.)
14.1-7
Show how to use an order-statistic tree to count the number of inversions (see
Problem 2-4) in an array of size n in time O.n lg n/.
14.1-8 ?
Consider n chords on a circle, each defined by its endpoints. Describe an O.n lg n/time algorithm to determine the number of pairs of chords that intersect inside the
circle. (For example, !if"the n chords are all diameters that meet at the center, then
the correct answer is n2 .) Assume that no two chords share an endpoint.
14.2 How to augment a data structure
The process of augmenting a basic data structure to support additional functionality
occurs quite frequently in algorithm design. We shall use it again in the next section
to design a data structure that supports operations on intervals. In this section, we
examine the steps involved in such augmentation. We shall also prove a theorem
that allows us to augment red-black trees easily in many cases.
We can break the process of augmenting a data structure into four steps:
1. Choose an underlying data structure.
2. Determine additional information to maintain in the underlying data structure.
3. Verify that we can maintain the additional information for the basic modifying
operations on the underlying data structure.
4. Develop new operations.
As with any prescriptive design method, you should not blindly follow the steps
in the order given. Most design work contains an element of trial and error, and
progress on all steps usually proceeds in parallel. There is no point, for example, in
determining additional information and developing new operations (steps 2 and 4)
if we will not be able to maintain the additional information efficiently. Nevertheless, this four-step method provides a good focus for your efforts in augmenting
a data structure, and it is also a good way to organize the documentation of an
augmented data structure.
346
Chapter 14 Augmenting Data Structures
We followed these steps in Section 14.1 to design our order-statistic trees. For
step 1, we chose red-black trees as the underlying data structure. A clue to the
suitability of red-black trees comes from their efficient support of other dynamicset operations on a total order, such as M INIMUM, M AXIMUM, S UCCESSOR, and
P REDECESSOR.
For step 2, we added the size attribute, in which each node x stores the size of the
subtree rooted at x. Generally, the additional information makes operations more
efficient. For example, we could have implemented OS-S ELECT and OS-R ANK
using just the keys stored in the tree, but they would not have run in O.lg n/ time.
Sometimes, the additional information is pointer information rather than data, as
in Exercise 14.2-1.
For step 3, we ensured that insertion and deletion could maintain the size attributes while still running in O.lg n/ time. Ideally, we should need to update only
a few elements of the data structure in order to maintain the additional information.
For example, if we simply stored in each node its rank in the tree, the OS-S ELECT
and OS-R ANK procedures would run quickly, but inserting a new minimum element would cause a change to this information in every node of the tree. When we
store subtree sizes instead, inserting a new element causes information to change
in only O.lg n/ nodes.
For step 4, we developed the operations OS-S ELECT and OS-R ANK. After all,
the need for new operations is why we bother to augment a data structure in the first
place. Occasionally, rather than developing new operations, we use the additional
information to expedite existing ones, as in Exercise 14.2-1.
Augmenting red-black trees
When red-black trees underlie an augmented data structure, we can prove that insertion and deletion can always efficiently maintain certain kinds of additional information, thereby making step 3 very easy. The proof of the following theorem is
similar to the argument from Section 14.1 that we can maintain the size attribute
for order-statistic trees.
Theorem 14.1 (Augmenting a red-black tree)
Let f be an attribute that augments a red-black tree T of n nodes, and suppose that
the value of f for each node x depends on only the information in nodes x, x:left,
and x:right, possibly including x:left:f and x:right:f . Then, we can maintain the
values of f in all nodes of T during insertion and deletion without asymptotically
affecting the O.lg n/ performance of these operations.
Proof The main idea of the proof is that a change to an f attribute in a node x
propagates only to ancestors of x in the tree. That is, changing x:f may re-
14.2 How to augment a data structure
347
quire x:p:f to be updated, but nothing else; updating x:p:f may require x:p:p:f
to be updated, but nothing else; and so on up the tree. Once we have updated
T:root:f , no other node will depend on the new value, and so the process terminates. Since the height of a red-black tree is O.lg n/, changing an f attribute in a
node costs O.lg n/ time in updating all nodes that depend on the change.
Insertion of a node x into T consists of two phases. (See Section 13.3.) The
first phase inserts x as a child of an existing node x:p. We can compute the value
of x:f in O.1/ time since, by supposition, it depends only on information in the
other attributes of x itself and the information in x’s children, but x’s children are
both the sentinel T:nil. Once we have computed x:f , the change propagates up
the tree. Thus, the total time for the first phase of insertion is O.lg n/. During the
second phase, the only structural changes to the tree come from rotations. Since
only two nodes change in a rotation, the total time for updating the f attributes
is O.lg n/ per rotation. Since the number of rotations during insertion is at most
two, the total time for insertion is O.lg n/.
↳
3
Like insertion, deletion has two phases. (See Section 13.4.) In the first phase,
changes to the tree occur when the deleted node is removed from the tree. If the
deleted node had two children at the time, then its successor moves into the position
of the deleted node. Propagating the updates to f caused by these changes costs
at most O.lg n/, since the changes modify the tree locally. Fixing up the red-black
tree
during the second phase requires at most three rotations, and each rotation
requires at most O.lg n/ time to propagate the updates to f . Thus, like insertion,
the total time for deletion is O.lg n/.
F
In many cases, such as maintaining the size attributes in order-statistic trees, the
cost of updating after a rotation is O.1/, rather than the O.lg n/ derived in the proof
of Theorem 14.1. Exercise 14.2-3 gives an example.
-
-
-
-
-
-
-
-
-
Exercises
14.2-1
Show, by adding pointers to the nodes, how to support each of the dynamic-set
queries M INIMUM, M AXIMUM, S UCCESSOR, and P REDECESSOR in O.1/ worstcase time on an augmented order-statistic tree. The asymptotic performance of
other operations on order-statistic trees should not be affected.
14.2-2
Can we maintain the black-heights of nodes in a red-black tree as attributes in the
nodes of the tree without affecting the asymptotic performance of any of the redblack tree operations? Show how, or argue why not. How about maintaining the
depths of nodes?
348
Chapter 14 Augmenting Data Structures
14.2-3 ?
Let ˝ be an associative binary operator, and let a be an attribute maintained in each
node of a red-black tree. Suppose that we want to include in each node x an additional attribute f such that x:f D x1 :a ˝ x2 :a ˝ $ $ $ ˝ xm :a, where x1 ; x2 ; : : : ; xm
is the inorder listing of nodes in the subtree rooted at x. Show how to update the f
attributes in O.1/ time after a rotation. Modify your argument slightly to apply it
to the size attributes in order-statistic trees.
14.2-4 ?
We wish to augment red-black trees with an operation RB-E NUMERATE .x; a; b/
that outputs all the keys k such that a " k " b in a red-black tree rooted at x.
Describe how to implement RB-E NUMERATE in ‚.m C lg n/ time, where m is the
number of keys that are output and n is the number of internal nodes in the tree.
(Hint: You do not need to add new attributes to the red-black tree.)
14.3 Interval trees
In this section, we shall augment red-black trees to support operations on dynamic
sets of intervals. A closed interval is an ordered pair of real numbers Œt1 ; t2 !, with
t1 " t2 . The interval Œt1 ; t2 ! represents the set ft 2 R W t1 " t " t2 g. Open and
half-open intervals omit both or one of the endpoints from the set, respectively. In
this section, we shall assume that intervals are closed; extending the results to open
and half-open intervals is conceptually straightforward.
Intervals are convenient for representing events that each occupy a continuous
period of time. We might, for example, wish to query a database of time intervals
to find out what events occurred during a given interval. The data structure in this
section provides an efficient means for maintaining such an interval database.
We can represent an interval Œt1 ; t2 ! as an object i, with attributes i:low D t1
(the low endpoint) and i:high D t2 (the high endpoint). We say that intervals i
and i 0 overlap if i \ i 0 ¤ ;, that is, if i:low " i 0 :high and i 0 :low " i:high. As
Figure 14.3 shows, any two intervals i and i 0 satisfy the interval trichotomy; that
is, exactly one of the following three properties holds:
a. i and i 0 overlap,
b. i is to the left of i 0 (i.e., i:high < i 0 :low),
c. i is to the right of i 0 (i.e., i 0 :high < i:low).
An interval tree is a red-black tree that maintains a dynamic set of elements, with
each element x containing an interval x:int. Interval trees support the following
operations:
14.3 Interval trees
349
i
i′
i
i′
i
i′
i
i′
(a)
i
i′
(b)
i′
i
(c)
Figure 14.3 The interval trichotomy for two closed intervals i and i 0 . (a) If i and i 0 overlap, there
are four situations; in each, i: low " i 0 : high and i 0 : low " i: high. (b) The intervals do not overlap,
and i: high < i 0 : low. (c) The intervals do not overlap, and i 0 : high < i: low.
I NTERVAL -I NSERT .T; x/ adds the element x, whose int attribute is assumed to
contain an interval, to the interval tree T .
I NTERVAL -D ELETE .T; x/ removes the element x from the interval tree T .
I NTERVAL -S EARCH .T; i/ returns a pointer to an element x in the interval tree T
such that x:int overlaps interval i, or a pointer to the sentinel T:nil if no such
element is in the set.
Figure 14.4 shows how an interval tree represents a set of intervals. We shall track
the four-step method from Section 14.2 as we review the design of an interval tree
and the operations that run on it.
Step 1: Underlying data structure
We choose a red-black tree in which each node x contains an interval x:int and the
key of x is the low endpoint, x:int:low, of the interval. Thus, an inorder tree walk
of the data structure lists the intervals in sorted order by low endpoint.
Step 2: Additional information
In addition to the intervals themselves, each node x contains a value x:max, which
is the maximum value of any interval endpoint stored in the subtree rooted at x.
Step 3: Maintaining the information
We must verify that insertion and deletion take O.lg n/ time on an interval tree
of n nodes. We can determine x:max given interval x:int and the max values of
node x’s children:
350
Chapter 14 Augmenting Data Structures
26 26
17
(a)
16
0
3
6
5
0
8
9
8
5
19
25
20
19
15
21
30
23
10
10
15
20
25
30
[16,21]
30
[8,9]
[25,30]
23
(b)
[5,8]
[15,23]
10
[0,3]
3
30
23
[17,19]
[6,10]
10
int
max
[26,26]
20
26
[19,20]
20
Figure 14.4 An interval tree. (a) A set of 10 intervals, shown sorted bottom to top by left endpoint.
(b) The interval tree that represents them. Each node x contains an interval, shown above the dashed
line, and the maximum value of any interval endpoint in the subtree rooted at x, shown below the
dashed line. An inorder tree walk of the tree lists the nodes in sorted order by left endpoint.
x:max D max.x:int:high; x:left:max; x:right:max/ :
Thus, by Theorem 14.1, insertion and deletion run in O.lg n/ time. In fact, we
can update the max attributes after a rotation in O.1/ time, as Exercises 14.2-3
and 14.3-1 show.
Step 4: Developing new operations
The only new operation we need is I NTERVAL -S EARCH .T; i/, which finds a node
in tree T whose interval overlaps interval i. If there is no interval that overlaps i in
the tree, the procedure returns a pointer to the sentinel T:nil.
14.3 Interval trees
351
I NTERVAL -S EARCH .T; i/
1 x D T:root
2 while x ¤ T:nil and i does not overlap x:int
3
if x:left ¤ T:nil and x:left:max # i:low
4
x D x:left
5
else x D x:right
6 return x
The search for an interval that overlaps i starts with x at the root of the tree and
proceeds downward. It terminates when either it finds an overlapping interval or x
points to the sentinel T:nil. Since each iteration of the basic loop takes O.1/ time,
and since the height of an n-node red-black tree is O.lg n/, the I NTERVAL -S EARCH
procedure takes O.lg n/ time.
Before we see why I NTERVAL -S EARCH is correct, let’s examine how it works
on the interval tree in Figure 14.4. Suppose we wish to find an interval that overlaps
the interval i D Œ22; 25!. We begin with x as the root, which contains Œ16; 21! and
does not overlap i. Since x:left:max D 23 is greater than i:low D 22, the loop
continues with x as the left child of the root—the node containing Œ8; 9!, which also
does not overlap i. This time, x:left:max D 10 is less than i:low D 22, and so the
loop continues with the right child of x as the new x. Because the interval Œ15; 23!
stored in this node overlaps i, the procedure returns this node.
As an example of an unsuccessful search, suppose we wish to find an interval
that overlaps i D Œ11; 14! in the interval tree of Figure 14.4. We once again begin with x as the root. Since the root’s interval Œ16; 21! does not overlap i, and
since x:left:max D 23 is greater than i:low D 11, we go left to the node containing Œ8; 9!. Interval Œ8; 9! does not overlap i, and x:left:max D 10 is less than
i:low D 11, and so we go right. (Note that no interval in the left subtree overlaps i.) Interval Œ15; 23! does not overlap i, and its left child is T:nil, so again we
go right, the loop terminates, and we return the sentinel T:nil.
To see why I NTERVAL -S EARCH is correct, we must understand why it suffices
to examine a single path from the root. The basic idea is that at any node x,
if x:int does not overlap i, the search always proceeds in a safe direction: the
search will definitely find an overlapping interval if the tree contains one. The
following theorem states this property more precisely.
Theorem 14.2
Any execution of I NTERVAL -S EARCH .T; i/ either returns a node whose interval
overlaps i, or it returns T:nil and the tree T contains no node whose interval overlaps i.
352
Chapter 14 Augmenting Data Structures
i′
i′
i
(a)
i
i′′
i′
i′′
i′′
(b)
Figure 14.5 Intervals in the proof of Theorem 14.2. The value of x: left: max is shown in each case
as a dashed line. (a) The search goes right. No interval i 0 in x’s left subtree can overlap i. (b) The
search goes left. The left subtree of x contains an interval that overlaps i (situation not shown),
or x’s left subtree contains an interval i 0 such that i 0 : high D x: left: max. Since i does not overlap i 0 ,
neither does it overlap any interval i 00 in x’s right subtree, since i 0 : low " i 00 : low.
Proof The while loop of lines 2–5 terminates either when x D T:nil or i overlaps x:int. In the latter case, it is certainly correct to return x. Therefore, we focus
on the former case, in which the while loop terminates because x D T:nil.
We use the following invariant for the while loop of lines 2–5:
If tree T contains an interval that overlaps i, then the subtree rooted at x
contains such an interval.
We use this loop invariant as follows:
Initialization: Prior to the first iteration, line 1 sets x to be the root of T , so that
the invariant holds.
Maintenance: Each iteration of the while loop executes either line 4 or line 5. We
shall show that both cases maintain the loop invariant.
If line 5 is executed, then because of the branch condition in line 3, we
have x:left D T:nil, or x:left:max < i:low. If x:left D T:nil, the subtree
rooted at x:left clearly contains no interval that overlaps i, and so setting x
to x:right maintains the invariant. Suppose, therefore, that x:left ¤ T:nil and
x:left:max < i:low. As Figure 14.5(a) shows, for each interval i 0 in x’s left
subtree, we have
i 0 :high " x:left:max
< i:low :
By the interval trichotomy, therefore, i 0 and i do not overlap. Thus, the left
subtree of x contains no intervals that overlap i, so that setting x to x:right
maintains the invariant.
14.3 Interval trees
353
If, on the other hand, line 4 is executed, then we will show that the contrapositive of the loop invariant holds. That is, if the subtree rooted at x:left contains no interval overlapping i, then no interval anywhere in the tree overlaps i.
Since line 4 is executed, then because of the branch condition in line 3, we
have x:left:max # i:low. Moreover, by definition of the max attribute, x’s left
subtree must contain some interval i 0 such that
i 0 :high D x:left:max
# i:low :
(Figure 14.5(b) illustrates the situation.) Since i and i 0 do not overlap, and
since it is not true that i 0 :high < i:low, it follows by the interval trichotomy
that i:high < i 0 :low. Interval trees are keyed on the low endpoints of intervals,
and thus the search-tree property implies that for any interval i 00 in x’s right
subtree,
i:high < i 0 :low
" i 00 :low :
By the interval trichotomy, i and i 00 do not overlap. We conclude that whether
or not any interval in x’s left subtree overlaps i, setting x to x:left maintains
the invariant.
Termination: If the loop terminates when x D T:nil, then the subtree rooted at x
contains no interval overlapping i. The contrapositive of the loop invariant
implies that T contains no interval that overlaps i. Hence it is correct to return
x D T:nil.
Thus, the I NTERVAL -S EARCH procedure works correctly.
Exercises
14.3-1
Write pseudocode for L EFT-ROTATE that operates on nodes in an interval tree and
updates the max attributes in O.1/ time.
14.3-2
Rewrite the code for I NTERVAL -S EARCH so that it works properly when all intervals are open.
14.3-3
Describe an efficient algorithm that, given an interval i, returns an interval overlapping i that has the minimum low endpoint, or T:nil if no such interval exists.
21
Data Structures for Disjoint Sets
Some applications involve grouping n distinct elements into a collection of disjoint
sets. These applications often need to perform two operations in particular: finding
the unique set that contains a given element and uniting two sets. This chapter
explores methods for maintaining a data structure that supports these operations.
Section 21.1 describes the operations supported by a disjoint-set data structure
and presents a simple application. In Section 21.2, we look at a simple linked-list
implementation for disjoint sets. Section 21.3 presents a more efficient representation using rooted trees. The running time using the tree representation is theoretically superlinear, but for all practical purposes it is linear. Section 21.4 defines
and discusses a very quickly growing function and its very slowly growing inverse,
which appears in the running time of operations on the tree-based implementation,
and then, by a complex amortized analysis, proves an upper bound on the running
time that is just barely superlinear.
21.1 Disjoint-set operations
A disjoint-set data structure maintains a collection S D fS1 ; S2 ; : : : ; Sk g of disjoint dynamic sets. We
identify each set by a representative, which is some member of the set. In some applications, it doesn’t matter which member is used as the
representative; we care only that if we ask for the representative of a dynamic set
twice without modifying the set between the requests, we get the same answer both
times. Other applications may require a prespecified rule for choosing the representative, such as choosing the smallest member in the set (assuming, of course,
that the elements can be ordered).
As in the other dynamic-set implementations we have studied, we represent each
element of a set by an object. Letting x denote an object, we wish to support the
following operations:
-
-
-
562
Chapter 21 Data Structures for Disjoint Sets
M AKE -S ET .x/ creates a new set whose only member (and thus representative)
X
is x. Since the sets are disjoint, we require that x not already be in some other
set.
-
-
-
-
U NION .x; y/ unites the dynamic sets that contain x and y, say Sx and Sy , into a
new set that is the union of these two sets. We assume that the two sets are disjoint prior to the operation. The representative of the resulting set is any member
of Sx [ Sy , although many implementations of U NION specifically choose the
representative of either Sx or Sy as the new representative. Since we require
the sets in the collection to be disjoint, conceptually we destroy sets Sx and Sy ,
removing them from the collection S . In practice, we often absorb the elements
of one of the sets into the other set.
-
-
F IND -S ET .x/ returns a pointer to the representative of the (unique) set containing x.
Throughout this chapter, we shall analyze the running times of disjoint-set data
structures in terms of two parameters: n, the number of M AKE -S ET operations,
and m, the total number of M AKE -S ET, U NION, and F IND -S ET operations. Since
the sets are disjoint, each U NION operation reduces the number of sets by one.
After n ! 1 U NION operations, therefore, only one set remains. The number of
U NION operations is thus at most n ! 1. Note also that since the M AKE -S ET
operations are included in the total number of operations m, we have m $ n. We
assume that the n M AKE -S ET operations are the first n operations performed.
An application of disjoint-set data structures
One of the many applications of disjoint-set data structures arises in determining the connected components of an undirected graph (see Section B.4). Figure 21.1(a), for example, shows a graph with four connected components.
The procedure C ONNECTED -C OMPONENTS that follows uses the disjoint-set
operations to compute the connected components of a graph. Once C ONNECTED C OMPONENTS has preprocessed the graph, the procedure S AME -C OMPONENT
answers queries about whether two vertices are in the same connected component.1
(In pseudocode, we denote the set of vertices of a graph G by G:V and the set of
edges by G:E.)
1 When the edges of the graph are static—not changing over time—we can compute the connected
components faster by using depth-first search (Exercise 22.3-12). Sometimes, however, the edges
are added dynamically and we need to maintain the connected components as each edge is added. In
this case, the implementation given here can be more efficient than running a new depth-first search
for each new edge.
21.1 Disjoint-set operations
a
b
e
c
d
g
563
f
h
j
i
(a)
Edge processed
initial sets
(b,d)
(e,g)
(a,c)
(h,i)
(a,b)
(e, f )
(b,c)
{a}
{a}
{a}
{a,c}
{a,c}
{a,b,c,d}
{a,b,c,d}
{a,b,c,d}
Collection of disjoint sets
{b}
{c} {d} {e}
{f} {g}
{b,d} {c}
{e}
{f} {g}
{b,d} {c}
{e,g}
{f}
{b,d}
{e,g}
{f}
{b,d}
{e,g}
{f}
{e,g}
{f}
{e, f,g}
{e, f,g}
{h}
{h}
{h}
{h}
{h,i}
{h,i}
{h,i}
{h,i}
{i}
{i}
{i}
{i}
{j}
{j}
{j}
{j}
{j}
{j}
{j}
{j}
(b)
Figure 21.1 (a) A graph with four connected components: fa; b; c; d g, fe; f; gg, fh; ig, and fj g.
(b) The collection of disjoint sets after processing each edge.
C ONNECTED -C OMPONENTS .G/
1 for each vertex $ 2 G:V
2
M AKE -S ET .$/
3 for each edge .u; $/ 2 G:E
4
if F IND -S ET .u/ ¤ F IND -S ET .$/
5
U NION .u; $/
S AME -C OMPONENT .u; $/
1 if F IND -S ET .u/ == F IND -S ET .$/
2
return TRUE
3 else return FALSE
The procedure C ONNECTED -C OMPONENTS initially places each vertex $ in its
own set. Then, for each edge .u; $/, it unites the sets containing u and $. By
Exercise 21.1-2, after processing all the edges, two vertices are in the same connected component if and only if the corresponding objects are in the same set.
Thus, C ONNECTED -C OMPONENTS computes sets in such a way that the procedure S AME -C OMPONENT can determine whether two vertices are in the same con-
564
Chapter 21 Data Structures for Disjoint Sets
nected component. Figure 21.1(b) illustrates how C ONNECTED -C OMPONENTS
computes the disjoint sets.
In an actual implementation of this connected-components algorithm, the representations of the graph and the disjoint-set data structure would need to reference
each other. That is, an object representing a vertex would contain a pointer to
the corresponding disjoint-set object, and vice versa. These programming details
depend on the implementation language, and we do not address them further here.
Exercises
21.1-1
Suppose that C ONNECTED -C OMPONENTS is run on the undirected graph G D
.V; E/, where V D fa; b; c; d; e; f; g; h; i; j; kg and the edges of E are processed in the order .d; i/; .f; k/; .g; i/; .b; g/; .a; h/; .i; j /; .d; k/; .b; j /; .d; f /;
.g; j /; .a; e/. List the vertices in each connected component after each iteration of
lines 3–5.
21.1-2
Show that after all edges are processed by C ONNECTED -C OMPONENTS, two vertices are in the same connected component if and only if they are in the same set.
21.1-3
During the execution of C ONNECTED -C OMPONENTS on an undirected graph G D
.V; E/ with k connected components, how many times is F IND -S ET called? How
many times is U NION called? Express your answers in terms of jV j, jEj, and k.
21.2 Linked-list representation of disjoint sets
Figure 21.2(a) shows a simple way to implement a disjoint-set data structure: each
set is represented by its own linked list. The object for each set has attributes head,
pointing to the first object in the list, and tail, pointing to the last object. Each
object
in the list contains a set member, a pointer to the next object in the list, and
a pointer back to the set object. Within each linked list, the objects may appear in
any order. The representative is the set member in the first object in the list.
With this linked-list representation, both M AKE -S ET and F IND -S ET are easy,
requiring O.1/ time. To carry out M AKE -S ET .x/, we create a new linked list
whose only object is x. For F IND -S ET .x/, we just follow the pointer from x back
to its set object and then return the member in the object that head points to. For
example, in Figure 21.2(a), the call F IND -S ET .g/ would return f .
-
-
21.2 Linked-list representation of disjoint sets
(a)
S1
(b)
S1
head
f
g
d
S2
tail
head
f
g
d
c
565
c
head
h
e
b
tail
h
e
b
tail
Figure 21.2 (a) Linked-list representations of two sets. Set S1 contains members d , f , and g, with
representative f , and set S2 contains members b, c, e, and h, with representative c. Each object in
the list contains a set member, a pointer to the next object in the list, and a pointer back to the set
object. Each set object has pointers head and tail to the first and last objects, respectively. (b) The
result of U NION.g; e/, which appends the linked list containing e to the linked list containing g. The
representative of the resulting set is f . The set object for e’s list, S2 , is destroyed.
A simple implementation of union
The simplest implementation of the U NION operation using the linked-list set representation takes significantly more time than M AKE -S ET or F IND -S ET. As Figure 21.2(b) shows, we perform U NION .x; y/ by appending y’s list onto the end
of x’s list. The representative of x’s list becomes the representative of the resulting
set. We use the tail pointer for x’s list to quickly find where to append y’s list. Because all members of y’s list join x’s list, we can destroy the set object for y’s list.
Unfortunately, we must update the pointer to the set object for each object originally on y’s list, which takes time linear in the length of y’s list. In Figure 21.2, for
example, the operation U NION .g; e/ causes pointers to be updated in the objects
for b, c, e, and h.
In fact, we can easily construct a sequence of m operations on n objects that
requires ‚.n2 / time. Suppose that we have objects x1 ; x2 ; : : : ; xn . We execute
the sequence of n M AKE -S ET operations followed by n ! 1 U NION operations
shown in Figure 21.3, so that m D 2n ! 1. We spend ‚.n/ time performing the n
M AKE -S ET operations. Because the ith U NION operation updates i objects, the
total number of objects updated by all n ! 1 U NION operations is
-
-
-
566
Chapter 21 Data Structures for Disjoint Sets
Operation
M AKE -S ET.x1 /
M AKE -S ET.x2 /
::
:
M AKE -S ET.xn /
U NION.x2 ; x1 /
U NION.x3 ; x2 /
U NION.x4 ; x3 /
::
:
U NION.xn ; xn!1 /
o
ocris[
Number of objects updated
1
1
::
:
1
1
2
3
::
:
n!1
T
[updaten-1
times
Figure 21.3 A sequence of 2n ! 1 operations on n objects that takes ‚.n2 / time, or ‚.n/ time
per operation on average, using the linked-list set representation and the simple implementation of
U NION.
n!1
X
i D1
i D ‚.n2 / :
-
#n]M-1 =
make (n) + union (n-1)
The total number of operations is 2n! 1, and so each operation on average requires
‚.n/ time. That is, the amortized time of an operation is ‚.n/.
-
A weighted-union heuristic
n
to
-
1
be
In the worst case, the above implementation of the U NION procedure requires an
average of ‚.n/ time per call because we may be appending a longer list onto
exact
a shorter list; we must update the pointer to the set object for each member of
the longer list. Suppose instead that each list also includes the length of the list
(which we can easily maintain) and that we always append the shorter list onto the
longer, breaking ties arbitrarily. With this simple weighted-union heuristic, a single U NION operation can still take !.n/ time if both sets have !.n/ members. As
the following theorem shows, however, a sequence of m M AKE -S ET, U NION, and
F IND -S ET operations, n of which are M AKE -S ET operations, takes O.m C n lg n/
time.
-
-
-
-
-
Theorem 21.1
Using the linked-list representation of disjoint sets and the weighted-union heuristic, a sequence of m M AKE -S ET, U NION, and F IND -S ET operations, n of which
are M AKE -S ET operations, takes O.m C n lg n/ time.
21.2 Linked-list representation of disjoint sets
567
Proof Because each U NION operation unites two disjoint sets, we perform at
most n ! 1 U NION operations over all. We now bound the total time taken by these
U NION operations. We start by determining, for each object, an upper bound on the
number of times the object’s pointer back to its set object is updated. Consider a
2
particular object x. We know that each time x’s pointer was updated, x must have + 1
-)
started in the smaller set. The first time x’s pointer was updated, therefore, the
sets
resulting
set must have had at least 2 members. Similarly, the next time x’s pointer E both
was updated, the resulting set must have had at least 4 members. Continuing on, are equally
we observe that for any k " n, after x’s pointer has been updated dlg2ke times, small 2 + 2 4
the resulting set must have at least k members. Since the largest set has at most n
members, each object’s pointer is updated at most dlg ne times over all the U NION
oclgn) perminion
operations. Thus the total time spent updating object pointers over all U NION
operations is O.n lg n/. We must also account for updating the tail pointers and
the list lengths, which take only ‚.1/ time per U NION operation. The total time
spent in all U NION operations is thus O.n lg n/.
The time for the entire sequence of m operations follows easily. Each M AKE S ET and F IND -S ET operation takes O.1/ time, and there are O.m/ of them. The
total time for the entire sequence is thus O.m C n lg n/.
=
-
-
e
-
=
-
-
Exercises
21.2-1
Write pseudocode for M AKE -S ET, F IND -S ET, and U NION using the linked-list
representation and the weighted-union heuristic. Make sure to specify the attributes
that you assume for set objects and list objects.
21.2-2
Show the data structure that results and the answers returned by the F IND -S ET
operations in the following program. Use the linked-list representation with the
weighted-union heuristic.
1
2
3
4
5
6
7
8
9
10
11
for i D 1 to 16
M AKE -S ET .xi /
for i D 1 to 15 by 2
U NION .xi ; xi C1 /
for i D 1 to 13 by 4
U NION .xi ; xi C2 /
U NION.x1 ; x5 /
U NION.x11 ; x13 /
U NION.x1 ; x10 /
F IND -S ET .x2 /
F IND -S ET .x9 /
568
Chapter 21 Data Structures for Disjoint Sets
Assume that if the sets containing xi and xj have the same size, then the operation
U NION .xi ; xj / appends xj ’s list onto xi ’s list.
21.2-3
Adapt the aggregate proof of Theorem 21.1 to obtain amortized time bounds
of O.1/ for M AKE -S ET and F IND -S ET and O.lg n/ for U NION using the linkedlist representation and the weighted-union heuristic.
21.2-4
Give a tight asymptotic bound on the running time of the sequence of operations in
Figure 21.3 assuming the linked-list representation and the weighted-union heuristic.
21.2-5
Professor Gompers suspects that it might be possible to keep just one pointer in
each set object, rather than two (head and tail), while keeping the number of pointers in each list element at two. Show that the professor’s suspicion is well founded
by describing how to represent each set by a linked list such that each operation
has the same running time as the operations described in this section. Describe
also how the operations work. Your scheme should allow for the weighted-union
heuristic, with the same effect as described in this section. (Hint: Use the tail of a
linked list as its set’s representative.)
21.2-6
Suggest a simple change to the U NION procedure for the linked-list representation
that removes the need to keep the tail pointer to the last object in each list. Whether
or not the weighted-union heuristic is used, your change should not change the
asymptotic running time of the U NION procedure. (Hint: Rather than appending
one list to another, splice them together.)
21.3 Disjoint-set forests
In a faster implementation of disjoint sets, we represent sets by rooted trees, with
each node containing one member and each tree representing one set. In a disjointset forest, illustrated in Figure 21.4(a), each member points only to its parent. The
root of each tree contains the representative and is its own parent. As we shall
see, although the straightforward algorithms that use this representation are no
faster than ones that use the linked-list representation, by introducing two heuristics—“union by rank” and “path compression”—we can achieve an asymptotically
optimal disjoint-set data structure.
21.3 Disjoint-set forests
c
h
569
f
e
f
d
b
g
(a)
c
h
b
d
e
g
(b)
Figure 21.4 A disjoint-set forest. (a) Two trees representing the two sets of Figure 21.2. The
tree on the left represents the set fb; c; e; hg, with c as the representative, and the tree on the right
represents the set fd; f; gg, with f as the representative. (b) The result of U NION.e; g/.
We perform the three disjoint-set operations as follows. A M AKE -S ET operation
simply creates a tree with just one node. We perform a F IND -S ET operation by
following
parent pointers until we find the root of the tree. The nodes visited on
this simple path toward the root constitute the find path. A U NION operation,
shown in Figure 21.4(b), causes the root of one tree to point to the root of the other.
-
-
-
-
Heuristics to improve the running time
So far, we have not improved on the linked-list implementation. A sequence of
n ! 1 U NION operations may create a tree that is just a linear chain of n nodes. By
using two heuristics, however, we can achieve a running time that is almost linear
in the total number of operations m.
The first heuristic, union by rank, is similar to the weighted-union heuristic we
used with the linked-list representation. The obvious approach would be to make
the root of the tree with fewer nodes point to the root of the tree with more nodes.
Rather than explicitly keeping track of the size of the subtree rooted at each node,
we shall use an approach that eases the analysis. For each node, we maintain a
rank, which is an upper bound on the height of the node. In union by rank, we
make the root with smaller rank point to the root with larger rank during a U NION
operation.
The second heuristic, path compression, is also quite simple and highly effective. As shown in Figure 21.5, we use it during F IND -S ET operations to make each
node on the find path point directly to the root. Path compression does not change
any ranks.
-
-
-
-
-
-
-
-
570
Chapter 21 Data Structures for Disjoint Sets
f
e
f
d
c
a
b
c
d
e
b
a
(a)
(b)
Figure 21.5 Path compression during the operation F IND -S ET . Arrows and self-loops at roots are
omitted. (a) A tree representing a set prior to executing F IND -S ET.a/. Triangles represent subtrees
whose roots are the nodes shown. Each node has a pointer to its parent. (b) The same set after
executing F IND -S ET.a/. Each node on the find path now points directly to the root.
Pseudocode for disjoint-set forests
To implement a disjoint-set forest with the union-by-rank heuristic, we must keep
track of ranks. With each node x, we maintain the integer value x:rank, which is
an upper bound on the height of x (the number of edges in the longest simple path
between x and a descendant leaf). When M AKE -S ET creates a singleton set, the
single node in the corresponding tree has an initial rank of 0. Each F IND -S ET operation leaves all ranks unchanged. The U NION operation has two cases, depending
on whether the roots of the trees have equal rank. If the roots have unequal rank,
we make the root with higher rank the parent of the root with lower rank, but the
ranks themselves remain unchanged. If, instead, the roots have equal ranks, we
arbitrarily choose one of the roots as the parent and increment its rank.
Let us put this method into pseudocode. We designate the parent of node x
by x:p. The L INK procedure, a subroutine called by U NION, takes pointers to two
roots as inputs.
21.3 Disjoint-set forests
571
M AKE -S ET .x/
1 x:p D x
2 x:rank D 0
U NION .x; y/
1 L INK .F IND -S ET .x/; F IND -S ET .y//
L INK .x; y/
1 if x:rank > y:rank
2
y:p D x
3 else x:p D y
4
if x:rank == y:rank
5
y:rank D y:rank C 1
The F IND -S ET procedure with path compression is quite simple:
F IND -S ET .x/
1 if x ¤ x:p
2
x:p D F IND -S ET .x:p/
3 return x:p
1
The F IND -S ET procedure is a two-pass method: as it recurses, it makes one pass
up the find path to find the root, and as the recursion unwinds, it makes a second
pass back down the find path to update each node to point directly to the root. Each
call of F IND -S ET .x/ returns x:p in line 3. If x is the root, then F IND -S ET skips
line 2 and instead returns x:p, which is x; this is the case in which the recursion
bottoms out. Otherwise, line 2 executes, and the recursive call with parameter x:p
returns a pointer to the root. Line 2 updates node x to point directly to the root,
and line 3 returns this pointer.
&
-
Effect of the heuristics on the running time
Separately, either union by rank or path compression improves the running time of
the operations on disjoint-set forests, and the improvement is even greater when
we use the two heuristics together. Alone, union by rank yields a running time
of O.m lg n/ (see Exercise 21.4-4), and this bound is tight (see Exercise 21.3-3).
Although we shall not prove it here, for a sequence of n M AKE -S ET operations (and hence at most n ! 1 U NION operations) and f F IND -S ET operations, the path-compression heuristic alone gives a worst-case running time of
‚.n C f # .1 C log2Cf =n n//.
1
572
Chapter 21 Data Structures for Disjoint Sets
When we use both union by rank and path compression, the worst-case running
time is O.m ˛.n//, where ˛.n/ is a very slowly growing function, which we define in Section 21.4. In any conceivable application of a disjoint-set data structure,
˛.n/ " 4; thus, we can view the running time as linear in m in all practical situations. Strictly speaking, however, it is superlinear. In Section 21.4, we prove this
upper bound.
Exercises
21.3-1
Redo Exercise 21.2-2 using a disjoint-set forest with union by rank and path compression.
21.3-2
Write a nonrecursive version of F IND -S ET with path compression.
21.3-3
Give a sequence of m M AKE -S ET, U NION, and F IND -S ET operations, n of which
are M AKE -S ET operations, that takes !.m lg n/ time when we use union by rank
only.
21.3-4
Suppose that we wish to add the operation P RINT-S ET .x/, which is given a node x
and prints all the members of x’s set, in any order. Show how we can add just
a single attribute to each node in a disjoint-set forest so that P RINT-S ET .x/ takes
time linear in the number of members of x’s set and the asymptotic running times
of the other operations are unchanged. Assume that we can print each member of
the set in O.1/ time.
21.3-5 ?
Show that any sequence of m M AKE -S ET, F IND -S ET, and L INK operations, where
all the L INK operations appear before any of the F IND -S ET operations, takes only
O.m/ time if we use both path compression and union by rank. What happens in
the same situation if we use only the path-compression heuristic?
22
Elementary Graph Algorithms
This chapter presents methods for representing a graph and for searching a graph.
Searching a graph means systematically following the edges of the graph so as to
visit the vertices of the graph. A graph-searching algorithm can discover much
about the structure of a graph. Many algorithms begin by searching their input
graph to obtain this structural information. Several other graph algorithms elaborate on basic graph searching. Techniques for searching a graph lie at the heart of
the field of graph algorithms.
Section 22.1 discusses the two most common computational representations of
graphs: as adjacency lists and as adjacency matrices. Section 22.2 presents a simple graph-searching algorithm called breadth-first search and shows how to create a breadth-first tree. Section 22.3 presents depth-first search and proves some
standard results about the order in which depth-first search visits vertices. Section 22.4 provides our first real application of depth-first search: topologically sorting a directed acyclic graph. A second application of depth-first search, finding the
strongly connected components of a directed graph, is the topic of Section 22.5.
22.1 Representations of graphs
We can choose between two standard ways to represent a graph G D .V; E/:
as a collection of adjacency lists or as an adjacency matrix. Either way applies
to both directed and undirected graphs. Because the adjacency-list representation
provides a compact way to represent sparse graphs—those for which jEj is much
less than jV j2 —it is usually the method of choice. Most of the graph algorithms
presented in this book assume that an input graph is represented in adjacencylist form. We may prefer an adjacency-matrix representation, however, when the
graph is dense—jEj is close to jV j2 —or when we need to be able to tell quickly
if there is an edge connecting two given vertices. For example, two of the all-pairs
-
-
I
sommet
pex
a
·
590
Chapter 22 Elementary Graph Algorithms
1
1
2
3
4
5
2
3
5
4
2
1
2
2
4
(a)
5
5
4
5
1
3
1
2
3
4
5
4
3
2
1
0
1
0
0
1
2
1
0
1
1
1
(b)
3
0
1
0
1
0
4
0
1
1
0
1
5
1
1
0
1
0
(c)
Figure 22.1 Two representations of an undirected graph. (a) An undirected graph G with 5 vertices
and 7 edges. (b) An adjacency-list representation of G. (c) The adjacency-matrix representation
of G.
1
2
3
4
5
6
(a)
1
2
3
4
5
6
2
5
6
2
4
6
(b)
4
5
1
2
3
4
5
6
1
0
0
0
0
0
0
2
1
0
0
1
0
0
3
0
0
0
0
0
0
4
1
0
0
0
1
0
5
0
1
1
0
0
0
6
0
0
1
0
0
1
(c)
Figure 22.2 Two representations of a directed graph. (a) A directed graph G with 6 vertices and 8
edges. (b) An adjacency-list representation of G. (c) The adjacency-matrix representation of G.
shortest-paths algorithms presented in Chapter 25 assume that their input graphs
are represented by adjacency matrices.
The adjacency-list representation of a graph G D .V; E/ consists of an array Adj of jV j lists, one for each vertex in V . For each u 2 V , the adjacency list
AdjŒu" contains all the vertices # such that there is an edge .u; #/ 2 E. That is,
AdjŒu" consists of all the vertices adjacent to u in G. (Alternatively, it may contain
pointers to these vertices.) Since the adjacency lists represent the edges of a graph,
in pseudocode we treat the array Adj as an attribute of the graph, just as we treat
the edge set E. In pseudocode, therefore, we will see notation such as G:AdjŒu".
Figure 22.1(b) is an adjacency-list representation of the undirected graph in Figure 22.1(a). Similarly, Figure 22.2(b) is an adjacency-list representation of the
directed graph in Figure 22.2(a).
If G is a directed graph, the sum of the lengths of all the adjacency lists is jEj,
since an edge of the form .u; #/ is represented by having # appear in AdjŒu". If G is
22.1 Representations of graphs
591
an undirected graph, the sum of the lengths of all the adjacency lists is 2 jEj, since
if .u; #/ is an undirected edge, then u appears in #’s adjacency list and vice versa.
For both directed and undirected graphs, the adjacency-list representation has the
desirable property that the amount of memory it requires is ‚.V C E/.
We can readily adapt adjacency lists to represent weighted graphs, that is, graphs
for which each edge has an associated weight, typically given by a weight function
w W E ! R. For example, let G D .V; E/ be a weighted graph with weight
function w. We simply store the weight w.u; #/ of the edge .u; #/ 2 E with
vertex # in u’s adjacency list. The adjacency-list representation is quite robust in
that we can modify it to support many other graph variants.
A potential disadvantage of the adjacency-list representation is that it provides
no quicker way to determine whether a given edge .u; #/ is present in the graph
than to search for # in the adjacency list AdjŒu". An adjacency-matrix representation of the graph remedies this disadvantage, but at the cost of using asymptotically
more memory. (See Exercise 22.1-8 for suggestions of variations on adjacency lists
that permit faster edge lookup.)
For the adjacency-matrix representation of a graph G D .V; E/, we assume
that the vertices are numbered 1; 2; : : : ; jV j in some arbitrary manner. Then the
adjacency-matrix representation of a graph G consists of a jV j % jV j matrix
A D .aij / such that
(
1 if .i; j / 2 E ;
aij D
0 otherwise :
Figures 22.1(c) and 22.2(c) are the adjacency matrices of the undirected and directed graphs in Figures 22.1(a) and 22.2(a), respectively. The adjacency matrix of
a graph requires ‚.V 2 / memory, independent of the number of edges in the graph.
Observe the symmetry along the main diagonal of the adjacency matrix in Figure 22.1(c). Since in an undirected graph, .u; #/ and .#; u/ represent the same
edge, the adjacency matrix A of an undirected graph is its own transpose: A D AT .
In some applications, it pays to store only the entries on and above the diagonal of
the adjacency matrix, thereby cutting the memory needed to store the graph almost
in half.
Like the adjacency-list representation of a graph, an adjacency matrix can represent a weighted graph. For example, if G D .V; E/ is a weighted graph with edgeweight function w, we can simply store the weight w.u; #/ of the edge .u; #/ 2 E
as the entry in row u and column # of the adjacency matrix. If an edge does not
exist, we can store a NIL value as its corresponding matrix entry, though for many
problems it is convenient to use a value such as 0 or 1.
Although the adjacency-list representation is asymptotically at least as spaceefficient as the adjacency-matrix representation, adjacency matrices are simpler,
and so we may prefer them when graphs are reasonably small. Moreover, adja-
592
Chapter 22 Elementary Graph Algorithms
cency matrices carry a further advantage for unweighted graphs: they require only
one bit per entry.
Representing attributes
Most algorithms that operate on graphs need to maintain attributes for vertices
and/or edges. We indicate these attributes using our usual notation, such as #:d
for an attribute d of a vertex #. When we indicate edges as pairs of vertices, we
use the same style of notation. For example, if edges have an attribute f , then we
denote this attribute for edge .u; #/ by .u; #/:f . For the purpose of presenting and
understanding algorithms, our attribute notation suffices.
Implementing vertex and edge attributes in real programs can be another story
entirely. There is no one best way to store and access vertex and edge attributes.
For a given situation, your decision will likely depend on the programming language you are using, the algorithm you are implementing, and how the rest of your
program uses the graph. If you represent a graph using adjacency lists, one design
represents vertex attributes in additional arrays, such as an array d Œ1 : : jV j" that
parallels the Adj array. If the vertices adjacent to u are in AdjŒu", then what we call
the attribute u:d would actually be stored in the array entry d Œu". Many other ways
of implementing attributes are possible. For example, in an object-oriented programming language, vertex attributes might be represented as instance variables
within a subclass of a Vertex class.
Exercises
22.1-1
Given an adjacency-list representation of a directed graph, how long does it take
to compute the out-degree of every vertex? How long does it take to compute the
in-degrees?
22.1-2
Give an adjacency-list representation for a complete binary tree on 7 vertices. Give
an equivalent adjacency-matrix representation. Assume that vertices are numbered
from 1 to 7 as in a binary heap.
22.1-3
The transpose of a directed graph G D .V; E/ is the graph G T D .V; E T /, where
E T D f.#; u/ 2 V % V W .u; #/ 2 Eg. Thus, G T is G with all its edges reversed.
Describe efficient algorithms for computing G T from G, for both the adjacencylist and adjacency-matrix representations of G. Analyze the running times of your
algorithms.
22.1 Representations of graphs
593
22.1-4
Given an adjacency-list representation of a multigraph G D .V; E/, describe an
O.V C E/-time algorithm to compute the adjacency-list representation of the
“equivalent” undirected graph G 0 D .V; E 0 /, where E 0 consists of the edges in E
with all multiple edges between two vertices replaced by a single edge and with all
self-loops removed.
22.1-5
The square of a directed graph G D .V; E/ is the graph G 2 D .V; E 2 / such that
.u; #/ 2 E 2 if and only G contains a path with at most two edges between u and #.
Describe efficient algorithms for computing G 2 from G for both the adjacencylist and adjacency-matrix representations of G. Analyze the running times of your
algorithms.
22.1-6
Most graph algorithms that take an adjacency-matrix representation as input require time $.V 2 /, but there are some exceptions. Show how to determine whether
a directed graph G contains a universal sink—a vertex with in-degree jV j " 1 and
out-degree 0—in time O.V /, given an adjacency matrix for G.
22.1-7
The incidence matrix of a directed graph G D .V; E/ with no self-loops is a
jV j % jEj matrix B D .bij / such that
bij D
! "1 if edge j leaves vertex i ;
1
0
if edge j enters vertex i ;
otherwise :
Describe what the entries of the matrix product BB T represent, where B T is the
transpose of B.
22.1-8
Suppose that instead of a linked list, each array entry AdjŒu" is a hash table containing the vertices # for which .u; #/ 2 E. If all edge lookups are equally likely, what
is the expected time to determine whether an edge is in the graph? What disadvantages does this scheme have? Suggest an alternate data structure for each edge list
that solves these problems. Does your alternative have disadvantages compared to
the hash table?
594
Chapter 22 Elementary Graph Algorithms
22.2 Breadth-first search
Breadth-first search is one of the simplest algorithms for searching a graph and
the archetype for many important graph algorithms. Prim’s minimum-spanningtree algorithm (Section 23.2) and Dijkstra’s single-source shortest-paths algorithm
(Section 24.3) use ideas similar to those in breadth-first search.
Given a graph G D .V; E/ and a distinguished source vertex s, breadth-first
search systematically explores the edges of G to “discover” every vertex that is
reachable from s. It computes the distance (smallest number of edges) from s
to each reachable vertex. It also produces a “breadth-first tree” with root s that
contains all reachable vertices. For any vertex # reachable from s, the simple path
in the breadth-first tree from s to # corresponds to a “shortest path” from s to #
in G, that is, a path containing the smallest number of edges. The algorithm works
on both directed and undirected graphs.
Breadth-first search is so named because it expands the frontier between discovered and undiscovered vertices uniformly across the breadth of the frontier. That
is, the algorithm discovers all vertices at distance k from s before discovering any
vertices at distance k C 1.
To keep track of progress, breadth-first search colors each vertex white, gray, or
black. All vertices start out white and may later become gray and then black. A
vertex is discovered the first time it is encountered during the search, at which time
it becomes nonwhite. Gray and black vertices, therefore, have been discovered, but
breadth-first search distinguishes between them to ensure that the search proceeds
in a breadth-first manner.1 If .u; #/ 2 E and vertex u is black, then vertex #
is either gray or black; that
is, all vertices adjacent to black vertices have been
nesee
discovered.
Gray
vertices
may
have some adjacent white vertices; they represent
the frontier between discovered and undiscovered vertices.
Breadth-first search constructs a breadth-first tree, initially containing only its
root, which is the source vertex s. Whenever the search discovers a white vertex #
in the course of scanning the adjacency list of an already discovered vertex u, the
vertex # and the edge .u; #/ are added to the tree. We say that u is the predecessor
or parent of # in the breadth-first tree. Since a vertex is discovered at most once, it
has at most one parent. Ancestor and descendant relationships in the breadth-first
tree are defined relative to the root s as usual: if u is on the simple path in the tree
from the root s to vertex #, then u is an ancestor of # and # is a descendant of u.
-
-
-
1 We distinguish between gray and black vertices to help us understand how breadth-first search operates. In fact, as Exercise 22.2-3 shows, we would get the same result even if we did not distinguish
between gray and black vertices.
22.2 Breadth-first search
595
The breadth-first-search procedure BFS below assumes that the input graph
G D .V; E/ is represented using adjacency lists. It attaches several additional
attributes to each vertex in the graph. We store the color of each vertex u 2 V
in the attribute u:color and the predecessor of u in the attribute u:%. If u has no
predecessor (for example, if u D s or u has not been discovered), then u:% D NIL .
The attribute u:d holds the distance from the source s to vertex u computed by the
algorithm. The algorithm also uses a first-in, first-out queue Q (see Section 10.1)
to manage the set of gray vertices.
-
-
-
-
BFS.G; s/
initialize for every
1 for each vertex u 2 G:V " fsg
vertex
except sources
2
u:color D WHITE
3
u:d D 1
4
u:% D NIL
5 s:color D GRAY
6 s:d D 0
7 s:% D NIL
8 Q D ; G was empty
9 E NQUEUE .Q; s/1 add the first gray vertex
10 while Q ¤ ;
11
u D D EQUEUE .Q/
12
for each # 2 G:AdjŒu"
13
if #:color == WHITE
14
#:color D GRAY
15
#:d D u:d C 1
16
#:% D u
17
E NQUEUE .Q; #/
18
u:color D BLACK
Figure 22.3 illustrates the progress of BFS on a sample graph.
The procedure BFS works as follows. With the exception of the source vertex s,
lines 1–4 paint every vertex white, set u:d to be infinity for each vertex u, and set
the parent of every vertex to be NIL. Line 5 paints s gray, since we consider it to be
discovered as the procedure begins. Line 6 initializes s:d to 0, and line 7 sets the
predecessor of the source to be NIL. Lines 8–9 initialize Q to the queue containing
just the vertex s.
The while loop of lines 10–18 iterates as long as there remain gray vertices,
which are discovered vertices that have not yet had their adjacency lists fully examined. This while loop maintains the following invariant:
At the test in line 10, the queue Q consists of the set of gray vertices.
596
Chapter 22 Elementary Graph Algorithms
(a)
(c)
(e)
(g)
(i)
r
∞
s
0
t
∞
u
∞
∞
v
∞
w
∞
x
∞
y
r
1
s
0
t
2
u
∞
∞
v
1
w
2
x
∞
y
r
1
s
0
t
2
u
3
2
v
1
w
2
x
∞
y
r
1
s
0
t
2
u
3
2
v
1
w
2
x
3
y
r
1
s
0
t
2
u
3
2
v
1
w
2
x
3
y
Q
Q
Q
(b)
s
0
r
1
t x
2 2
x v u
2 2 3
Q
u y
3 3
Q
;
(d)
(f)
(h)
r
1
s
0
t
∞
u
∞
∞
v
1
w
∞
x
∞
y
r
1
s
0
t
2
u
∞
2
v
1
w
2
x
∞
y
r
1
s
0
t
2
u
3
2
v
1
w
2
x
3
y
r
1
s
0
t
2
u
3
2
v
1
w
2
x
3
y
Q
w r
1 1
Q
t x v
2 2 2
Q
v u y
2 3 3
Q
y
3
Figure 22.3 The operation of BFS on an undirected graph. Tree edges are shown shaded as they
are produced by BFS. The value of u: d appears within each vertex u. The queue Q is shown at the
beginning of each iteration of the while loop of lines 10–18. Vertex distances appear below vertices
in the queue.
Although we won’t use this loop invariant to prove correctness, it is easy to see
that it holds prior to the first iteration and that each iteration of the loop maintains
the invariant. Prior to the first iteration, the only gray vertex, and the only vertex
in Q, is the source vertex s. Line 11 determines the gray vertex u at the head of
the queue Q and removes it from Q. The for loop of lines 12–17 considers each
vertex # in the adjacency list of u. If # is white, then it has not yet been discovered,
and the procedure discovers it by executing lines 14–17. The procedure paints
vertex # gray, sets its distance #:d to u:dC1, records u as its parent #:%, and places
it at the tail of the queue Q. Once the procedure has examined all the vertices on u’s
22.2 Breadth-first search
597
adjacency list, it blackens u in line 18. The loop invariant is maintained because
whenever a vertex is painted gray (in line 14) it is also enqueued (in line 17), and
whenever a vertex is dequeued (in line 11) it is also painted black (in line 18).
The results of breadth-first search may depend upon the order in which the neighbors of a given vertex are visited in line 12: the breadth-first tree may vary, but the
distances d computed by the algorithm will not. (See Exercise 22.2-5.)
Analysis
Before proving the various properties of breadth-first search, we take on the somewhat easier job of analyzing its running time on an input graph G D .V; E/. We
use aggregate analysis, as we saw in Section 17.1. After initialization, breadth-first
search never whitens a vertex, and thus the test in line 13 ensures that each vertex
is enqueued at most once, and hence dequeued at most once. The operations of
enqueuing and dequeuing take O.1/ time, and so the total time devoted to queue
operations is O.V /. Because the procedure scans the adjacency list of each vertex
only when the vertex is dequeued, it scans each adjacency list at most once. Since
the sum of the lengths of all the adjacency lists is ‚.E/, the total time spent in
scanning adjacency lists is O.E/. The overhead for initialization is O.V /, and
thus the total running time of the BFS procedure is O.V C E/. Thus, breadth-first
search runs in time linear in the size of the adjacency-list representation of G.
Shortest paths
At the beginning of this section, we claimed that breadth-first search finds the distance to each reachable vertex in a graph G D .V; E/ from a given source vertex
s 2 V . Define the shortest-path distance ı.s; #/ from s to # as the minimum number of edges in any path from vertex s to vertex #; if there is no path from s to #,
then ı.s; #/ D 1. We call a path of length ı.s; #/ from s to # a shortest path2
from s to #. Before showing that breadth-first search correctly computes shortestpath distances, we investigate an important property of shortest-path distances.
2 In Chapters 24 and 25, we shall generalize our study of shortest paths to weighted graphs, in which
every edge has a real-valued weight and the weight of a path is the sum of the weights of its constituent edges. The graphs considered in the present chapter are unweighted or, equivalently, all
edges have unit weight.
598
Chapter 22 Elementary Graph Algorithms
Lemma 22.1
Let G D .V; E/ be a directed or undirected graph, and let s 2 V be an arbitrary
vertex. Then, for any edge .u; #/ 2 E,
ı.s; #/ $ ı.s; u/ C 1 :
Proof If u is reachable from s, then so is #. In this case, the shortest path from s
to # cannot be longer than the shortest path from s to u followed by the edge .u; #/,
and thus the inequality holds. If u is not reachable from s, then ı.s; u/ D 1, and
the inequality holds.
We want to show that BFS properly computes #:d D ı.s; #/ for each vertex # 2 V . We first show that #:d bounds ı.s; #/ from above.
Lemma 22.2
Let G D .V; E/ be a directed or undirected graph, and suppose that BFS is run
on G from a given source vertex s 2 V . Then upon termination, for each vertex # 2 V , the value #:d computed by BFS satisfies #:d # ı.s; #/.
Proof We use induction on the number of E NQUEUE operations. Our inductive
hypothesis is that #:d # ı.s; #/ for all # 2 V .
The basis of the induction is the situation immediately after enqueuing s in line 9
of BFS. The inductive hypothesis holds here, because s:d D 0 D ı.s; s/ and
#:d D 1 # ı.s; #/ for all # 2 V " fsg.
For the inductive step, consider a white vertex # that is discovered during the
search from a vertex u. The inductive hypothesis implies that u:d # ı.s; u/. From
the assignment performed by line 15 and from Lemma 22.1, we obtain
#:d D u:d C 1
# ı.s; u/ C 1
# ı.s; #/ :
Vertex # is then enqueued, and it is never enqueued again because it is also grayed
and the then clause of lines 14–17 is executed only for white vertices. Thus, the
value of #:d never changes again, and the inductive hypothesis is maintained.
To prove that #:d D ı.s; #/, we must first show more precisely how the queue Q
operates during the course of BFS. The next lemma shows that at all times, the
queue holds at most two distinct d values.
22.2 Breadth-first search
599
Lemma 22.3
Suppose that during the execution of BFS on a graph G D .V; E/, the queue Q
contains the vertices h#1 ; #2 ; : : : ; #r i, where #1 is the head of Q and #r is the tail.
Then, #r :d $ #1 :d C 1 and #i :d $ #i C1 :d for i D 1; 2; : : : ; r " 1.
Proof The proof is by induction on the number of queue operations. Initially,
when the queue contains only s, the lemma certainly holds.
For the inductive step, we must prove that the lemma holds after both dequeuing
and enqueuing a vertex. If the head #1 of the queue is dequeued, #2 becomes the
new head. (If the queue becomes empty, then the lemma holds vacuously.) By the
inductive hypothesis, #1 :d $ #2 :d. But then we have #r :d $ #1 :d C 1 $ #2 :d C 1,
and the remaining inequalities are unaffected. Thus, the lemma follows with #2 as
the head.
In order to understand what happens upon enqueuing a vertex, we need to examine the code more closely. When we enqueue a vertex # in line 17 of BFS, it
becomes #rC1 . At that time, we have already removed vertex u, whose adjacency
list is currently being scanned, from the queue Q, and by the inductive hypothesis,
the new head #1 has #1 :d # u:d. Thus, #rC1 :d D #:d D u:d C1 $ #1 :d C1. From
the inductive hypothesis, we also have #r :d $ u:d C 1, and so #r :d $ u:d C 1 D
#:d D #rC1 :d, and the remaining inequalities are unaffected. Thus, the lemma
follows when # is enqueued.
The following corollary shows that the d values at the time that vertices are
enqueued are monotonically increasing over time.
Corollary 22.4
Suppose that vertices #i and #j are enqueued during the execution of BFS, and
that #i is enqueued before #j . Then #i :d $ #j :d at the time that #j is enqueued.
Proof Immediate from Lemma 22.3 and the property that each vertex receives a
finite d value at most once during the course of BFS.
We can now prove that breadth-first search correctly finds shortest-path distances.
Theorem 22.5 (Correctness of breadth-first search)
Let G D .V; E/ be a directed or undirected graph, and suppose that BFS is run
on G from a given source vertex s 2 V . Then, during its execution, BFS discovers
every vertex # 2 V that is reachable from the source s, and upon termination,
#:d D ı.s; #/ for all # 2 V . Moreover, for any vertex # ¤ s that is reachable
600
Chapter 22 Elementary Graph Algorithms
from s, one of the shortest paths from s to # is a shortest path from s to #:%
followed by the edge .#:%; #/.
Proof Assume, for the purpose of contradiction, that some vertex receives a d
value not equal to its shortest-path distance. Let # be the vertex with minimum ı.s; #/ that receives such an incorrect d value; clearly # ¤ s. By
Lemma 22.2, #:d # ı.s; #/, and thus we have that #:d > ı.s; #/. Vertex # must be
reachable from s, for if it is not, then ı.s; #/ D 1 # #:d. Let u be the vertex immediately preceding # on a shortest path from s to #, so that ı.s; #/ D ı.s; u/ C 1.
Because ı.s; u/ < ı.s; #/, and because of how we chose #, we have u:d D ı.s; u/.
Putting these properties together, we have
#:d > ı.s; #/ D ı.s; u/ C 1 D u:d C 1 :
(22.1)
Now consider the time when BFS chooses to dequeue vertex u from Q in
line 11. At this time, vertex # is either white, gray, or black. We shall show
that in each of these cases, we derive a contradiction to inequality (22.1). If # is
white, then line 15 sets #:d D u:d C 1, contradicting inequality (22.1). If # is
black, then it was already removed from the queue and, by Corollary 22.4, we have
#:d $ u:d, again contradicting inequality (22.1). If # is gray, then it was painted
gray upon dequeuing some vertex w, which was removed from Q earlier than u
and for which #:d D w:d C 1. By Corollary 22.4, however, w:d $ u:d, and so we
have #:d D w:d C 1 $ u:d C 1, once again contradicting inequality (22.1).
Thus we conclude that #:d D ı.s; #/ for all # 2 V . All vertices # reachable
from s must be discovered, for otherwise they would have 1 D #:d > ı.s; #/. To
conclude the proof of the theorem, observe that if #:% D u, then #:d D u:d C 1.
Thus, we can obtain a shortest path from s to # by taking a shortest path from s
to #:% and then traversing the edge .#:%; #/.
Breadth-first trees
The procedure BFS builds a breadth-first tree as it searches the graph, as Figure 22.3 illustrates. The tree corresponds to the % attributes. More formally, for
a graph G D .V; E/ with source s, we define the predecessor subgraph of G as
G! D .V! ; E! /, where
V! D f# 2 V W #:% ¤ NILg [ fsg
and
E! D f.#:%; #/ W # 2 V! " fsgg :
The predecessor subgraph G! is a breadth-first tree if V! consists of the vertices
reachable from s and, for all # 2 V! , the subgraph G! contains a unique simple
22.2 Breadth-first search
601
path from s to # that is also a shortest path from s to # in G. A breadth-first tree
is in fact a tree, since it is connected and jE! j D jV! j " 1 (see Theorem B.2). We
call the edges in E! tree edges.
The following lemma shows that the predecessor subgraph produced by the BFS
procedure is a breadth-first tree.
Lemma 22.6
When applied to a directed or undirected graph G D .V; E/, procedure BFS constructs % so that the predecessor subgraph G! D .V! ; E! / is a breadth-first tree.
Proof Line 16 of BFS sets #:% D u if and only if .u; #/ 2 E and ı.s; #/ < 1—
that is, if # is reachable from s—and thus V! consists of the vertices in V reachable
from s. Since G! forms a tree, by Theorem B.2, it contains a unique simple path
from s to each vertex in V! . By applying Theorem 22.5 inductively, we conclude
that every such path is a shortest path in G.
The following procedure prints out the vertices on a shortest path from s to #,
assuming that BFS has already computed a breadth-first tree:
P RINT-PATH .G; s; #/
1 if # == s
2
print s
3 elseif #:% == NIL
4
print “no path from” s “to” # “exists”
5 else P RINT-PATH .G; s; #:%/
6
print #
This procedure runs in time linear in the number of vertices in the path printed,
since each recursive call is for a path one vertex shorter.
Exercises
22.2-1
Show the d and % values that result from running breadth-first search on the directed graph of Figure 22.2(a), using vertex 3 as the source.
22.2-2
Show the d and % values that result from running breadth-first search on the undirected graph of Figure 22.3, using vertex u as the source.
602
Chapter 22 Elementary Graph Algorithms
22.2-3
Show that using a single bit to store each vertex color suffices by arguing that the
BFS procedure would produce the same result if lines 5 and 14 were removed.
22.2-4
What is the running time of BFS if we represent its input graph by an adjacency
matrix and modify the algorithm to handle this form of input?
22.2-5
Argue that in a breadth-first search, the value u:d assigned to a vertex u is independent of the order in which the vertices appear in each adjacency list. Using
Figure 22.3 as an example, show that the breadth-first tree computed by BFS can
depend on the ordering within adjacency lists.
22.2-6
Give an example of a directed graph G D .V; E/, a source vertex s 2 V , and a
set of tree edges E! & E such that for each vertex # 2 V , the unique simple path
in the graph .V; E! / from s to # is a shortest path in G, yet the set of edges E!
cannot be produced by running BFS on G, no matter how the vertices are ordered
in each adjacency list.
22.2-7
There are two types of professional wrestlers: “babyfaces” (“good guys”) and
“heels” (“bad guys”). Between any pair of professional wrestlers, there may or
may not be a rivalry. Suppose we have n professional wrestlers and we have a list
of r pairs of wrestlers for which there are rivalries. Give an O.n C r/-time algorithm that determines whether it is possible to designate some of the wrestlers as
babyfaces and the remainder as heels such that each rivalry is between a babyface
and a heel. If it is possible to perform such a designation, your algorithm should
produce it.
22.2-8 ?
The diameter of a tree T D .V; E/ is defined as maxu;"2V ı.u; #/, that is, the
largest of all shortest-path distances in the tree. Give an efficient algorithm to
compute the diameter of a tree, and analyze the running time of your algorithm.
22.2-9
Let G D .V; E/ be a connected, undirected graph. Give an O.V C E/-time algorithm to compute a path in G that traverses each edge in E exactly once in each
direction. Describe how you can find your way out of a maze if you are given a
large supply of pennies.
22.3 Depth-first search
603
22.3 Depth-first search
The strategy followed by depth-first search is, as its name implies, to search
“deeper” in the graph whenever possible. Depth-first search explores edges out
of the most recently discovered vertex # that still has unexplored edges leaving it.
Once all of #’s edges have been explored, the search “backtracks” to explore edges
leaving the vertex from which # was discovered. This process continues until we
have discovered all the vertices that are reachable from the original source vertex.
If any undiscovered vertices remain, then depth-first search selects one of them as
a new source, and it repeats the search from that source. The algorithm repeats this
entire process until it has discovered every vertex.3
As in breadth-first search, whenever depth-first search discovers a vertex # during a scan of the adjacency list of an already discovered vertex u, it records this
event by setting #’s predecessor attribute #:% to u. Unlike breadth-first search,
whose predecessor subgraph forms a tree, the predecessor subgraph produced by
a depth-first search may be composed of several trees, because the search may
repeat from multiple sources. Therefore, we define the predecessor subgraph of
a depth-first search slightly differently from that of a breadth-first search: we let
G! D .V; E! /, where
-
-
-
-
-
E! D f.#:%; #/ W # 2 V and #:% ¤ NIL g :
-
The predecessor subgraph of a depth-first search forms a depth-first forest comprising several depth-first trees. The edges in E! are tree edges.
As in breadth-first search, depth-first search colors vertices during the search to
indicate their state. Each vertex is initially white, is grayed when it is discovered
in the search, and is blackened when it is finished, that is, when its adjacency list
has been examined completely. This technique guarantees that each vertex ends up
in exactly one depth-first tree, so that these trees are disjoint.
Besides creating a depth-first forest, depth-first search also timestamps each vertex. Each vertex # has two timestamps: the first timestamp #:d records when #
is first discovered (and grayed), and the second timestamp #:f records when the
search finishes examining #’s adjacency list (and blackens #). These timestamps
-
=
-
-
3 It may seem arbitrary that breadth-first search is limited to only one source whereas depth-first
search may search from multiple sources. Although conceptually, breadth-first search could proceed
from multiple sources and depth-first search could be limited to one source, our approach reflects how
the results of these searches are typically used. Breadth-first search usually serves to find shortestpath distances (and the associated predecessor subgraph) from a given source. Depth-first search is
often a subroutine in another algorithm, as we shall see later in this chapter.
604
Chapter 22 Elementary Graph Algorithms
provide important information about the structure of the graph and are generally
helpful in reasoning about the behavior of depth-first search.
The procedure DFS below records when it discovers vertex u in the attribute u:d
and when it finishes vertex u in the attribute u:f . These timestamps are integers
between 1 and 2 jV j, since there is one discovery event and one finishing event for
each of the jV j vertices. For every vertex u,
(22.2)
u:d < u:f :
Vertex u is WHITE before time u:d, GRAY between time u:d and time u:f , and
BLACK thereafter.
The following pseudocode is the basic depth-first-search algorithm. The input
graph G may be undirected or directed. The variable time is a global variable that
we use for timestamping.
DFS.G/
1 for each vertex u 2 G:V
2
u:color D WHITE
3
u:% D NIL
4 time D 0
5 for each vertex u 2 G:V
6
if u:color == WHITE
7
DFS-V ISIT .G; u/
-
DFS-V ISIT .G; u/
1 time D time C 1
2 u:d D time
3 u:color D GRAY
4 for each # 2 G:AdjŒu"
5
if #:color == WHITE
6
#:% D u
7
DFS-V ISIT .G; #/
8 u:color D BLACK
9 time D time C 1
10 u:f D time
// white vertex u has just been discovered
// explore edge .u; #/
// blacken u; it is finished
Figure 22.4 illustrates the progress of DFS on the graph shown in Figure 22.2.
Procedure DFS works as follows. Lines 1–3 paint all vertices white and initialize their % attributes to NIL. Line 4 resets the global time counter. Lines 5–7
check each vertex in V in turn and, when a white vertex is found, visit it using
DFS-V ISIT. Every time DFS-V ISIT .G; u/ is called in line 7, vertex u becomes
22.3 Depth-first search
605
u
1/
v
w
u
1/
v
2/
w
u
1/
v
2/
x
y
(a)
z
x
y
(b)
z
x
3/
y
(c)
u
1/
v
2/
w
u
1/
v
2/
w
u
1/
v
2/
B
B
4/
x
3/
y
u
1/
v
2/7
F
z
(e)
w
B
z
w
3/
y
u
1/8
v
2/7
z
(f)
w
B
z
u
1/8
v
2/7
w
9/
u
1/8
v
2/7
w
9/
F
(g)
F
3/6
y
(k)
z
u
1/8
v
2/7
w
9/
u
1/8
v
2/7
w
9/
u
1/8
v
2/7
w
9/
(m)
10/
z
4/5
x
C
3/6
y
(n)
10/
z
B
F
4/5
x
4/5
B
4/5
x
B
w
3/6
y
z
3/6
y
v
2/7
x
&
3/6
y
(j)
4/5
x
u
1/
z
z
4/5
x
F
3/
y
(d)
3/6
y
z
C
4/
x
w
4/5
x
3/6
y
(i)
B
v
2/
B
4/5
x
F
u
1/
B
4/5
x
F
w
B
C
3/6
y
(o)
10/11
z
B
4/5
x
C
3/6
y
(l)
z
v
2/7
w
9/12
steel here
u
1/8
B
(h)
F
4/5
x
B
C
3/6
y
(p)
10/11
z
B
Figure 22.4 The progress of the depth-first-search algorithm DFS on a directed graph. As edges
are explored by the algorithm, they are shown as either shaded (if they are tree edges) or dashed
(otherwise). Nontree edges are labeled B, C, or F according to whether they are back, cross, or
forward edges. Timestamps within vertices indicate discovery time/finishing times.
the root of a new tree in the depth-first forest. When DFS returns, every vertex u
has been assigned a discovery time u:d and a finishing time u:f .
In each call DFS-V ISIT .G; u/, vertex u is initially white. Line 1 increments
the global variable time, line 2 records the new value of time as the discovery
time u:d, and line 3 paints u gray. Lines 4–7 examine each vertex # adjacent to u
and recursively visit # if it is white. As each vertex # 2 AdjŒu" is considered in
line
4, we say that edge .u; #/ is explored by the depth-first search. Finally, after
every edge leaving u has been explored, lines 8–10 paint u black, increment time,
and record the finishing time in u:f .
Note that the results of depth-first search may depend upon the order in which
line 5 of DFS examines the vertices and upon the order in which line 4 of DFSV ISIT visits the neighbors of a vertex. These different visitation orders tend not
-
-
s
606
Chapter 22 Elementary Graph Algorithms
to cause problems in practice, as we can usually use any depth-first search result
effectively, with essentially equivalent results.
What is the running time of DFS? The loops on lines 1–3 and lines 5–7 of DFS
take time ‚.V /, exclusive of the time to execute the calls to DFS-V ISIT. As we did
for breadth-first search, we use aggregate analysis. The procedure DFS-V ISIT is
called exactly once for each vertex # 2 V , since the vertex u on which DFS-V ISIT
is invoked must be white and the first thing DFS-V ISIT does is paint vertex u gray.
During an execution of DFS-V ISIT .G; #/, the loop on lines 4–7 executes jAdjŒ#"j
times. Since
X
jAdjŒ#"j D ‚.E/ ;
-
"2V
the total cost of executing lines 4–7 of DFS-V ISIT is ‚.E/. The running time of
DFS is therefore ‚.V C E/.
Properties of depth-first search
Depth-first search yields valuable information about the structure of a graph. Perhaps the most basic property of depth-first search is that the predecessor subgraph G! does indeed form a forest of trees, since the structure of the depthfirst trees exactly mirrors the structure of recursive calls of DFS-V ISIT. That is,
u D #:% if and only if DFS-V ISIT .G; #/ was called during a search of u’s adjacency list. Additionally, vertex # is a descendant of vertex u in the depth-first
forest if and only if # is discovered during the time in which u is gray.
Another important property of depth-first search is that discovery and finishing
times have parenthesis structure. If we represent the discovery of vertex u with
a left parenthesis “.u” and represent its finishing by a right parenthesis “u/”, then
the history of discoveries and finishings makes a well-formed expression in the
sense that the parentheses are properly nested. For example, the depth-first search
of Figure 22.5(a) corresponds to the parenthesization shown in Figure 22.5(b). The
following theorem provides another way to characterize the parenthesis structure.
Theorem 22.7 (Parenthesis theorem)
In any depth-first search of a (directed or undirected) graph G D .V; E/, for any
two vertices u and #, exactly one of the following three conditions holds:
!
!
!
the intervals Œu:d; u:f " and Œ#:d; #:f " are entirely disjoint, and neither u nor #
is a descendant of the other in the depth-first forest,
the interval Œu:d; u:f " is contained entirely within the interval Œ#:d; #:f ", and u
is a descendant of # in a depth-first tree, or
the interval Œ#:d; #:f " is contained entirely within the interval Œu:d; u:f ", and #
is a descendant of u in a depth-first tree.
22.3 Depth-first search
y
3/6
607
z
2/9
B
(a)
4/5
x
s
1/10
t
11/16
C
B
F
7/8
w
C
12/13
v
C
14/15
u
C
s
t
z
(b)
v
y
u
w
x
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
(s (z (y (x x) y) (w w) z) s) (t (v v) (u u) t)
s
(c)
z
B
y
C
t
C
F
C
v
C
B
u
w
x
Figure 22.5 Properties of depth-first search. (a) The result of a depth-first search of a directed
graph. Vertices are timestamped and edge types are indicated as in Figure 22.4. (b) Intervals for
the discovery time and finishing time of each vertex correspond to the parenthesization shown. Each
rectangle spans the interval given by the discovery and finishing times of the corresponding vertex.
Only tree edges are shown. If two intervals overlap, then one is nested within the other, and the
vertex corresponding to the smaller interval is a descendant of the vertex corresponding to the larger.
(c) The graph of part (a) redrawn with all tree and forward edges going down within a depth-first tree
and all back edges going up from a descendant to an ancestor.
608
Chapter 22 Elementary Graph Algorithms
Proof We begin with the case in which u:d < #:d. We consider two subcases,
according to whether #:d < u:f or not. The first subcase occurs when #:d < u:f ,
so # was discovered while u was still gray, which implies that # is a descendant
of u. Moreover, since # was discovered more recently than u, all of its outgoing edges are explored, and # is finished, before the search returns to and finishes u. In this case, therefore, the interval Œ#:d; #:f " is entirely contained within
the interval Œu:d; u:f ". In the other subcase, u:f < #:d, and by inequality (22.2),
u:d < u:f < #:d < #:f ; thus the intervals Œu:d; u:f " and Œ#:d; #:f " are disjoint.
Because the intervals are disjoint, neither vertex was discovered while the other
was gray, and so neither vertex is a descendant of the other.
The case in which #:d < u:d is similar, with the roles of u and # reversed in the
above argument.
Corollary 22.8 (Nesting of descendants’ intervals)
Vertex # is a proper descendant of vertex u in the depth-first forest for a (directed
or undirected) graph G if and only if u:d < #:d < #:f < u:f .
Proof
Immediate from Theorem 22.7.
The next theorem gives another important characterization of when one vertex
is a descendant of another in the depth-first forest.
Theorem 22.9 (White-path theorem)
In a depth-first forest of a (directed or undirected) graph G D .V; E/, vertex # is
a descendant of vertex u if and only if at the time u:d that the search discovers u,
there is a path from u to # consisting entirely of white vertices.
Proof ): If # D u, then the path from u to # contains just vertex u, which is still
white when we set the value of u:d. Now, suppose that # is a proper descendant
of u in the depth-first forest. By Corollary 22.8, u:d < #:d, and so # is white at
time u:d. Since # can be any descendant of u, all vertices on the unique simple
path from u to # in the depth-first forest are white at time u:d.
(: Suppose that there is a path of white vertices from u to # at time u:d, but #
does not become a descendant of u in the depth-first tree. Without loss of generality, assume that every vertex other than # along the path becomes a descendant of u.
(Otherwise, let # be the closest vertex to u along the path that doesn’t become a descendant of u.) Let w be the predecessor of # in the path, so that w is a descendant
of u (w and u may in fact be the same vertex). By Corollary 22.8, w:f $ u:f . Because # must be discovered after u is discovered, but before w is finished, we have
u:d < #:d < w:f $ u:f . Theorem 22.7 then implies that the interval Œ#:d; #:f "
22.3 Depth-first search
609
is contained entirely within the interval Œu:d; u:f ". By Corollary 22.8, # must after
all be a descendant of u.
Classification of edges
Another interesting property of depth-first search is that the search can be used
to classify the edges of the input graph G D .V; E/. The type of each edge can
provide important information about a graph. For example, in the next section, we
shall see that a directed graph is acyclic if and only if a depth-first search yields no
“back” edges (Lemma 22.11).
We can define four edge types in terms of the depth-first forest G! produced by
a depth-first search on G:
-
1. Tree edges are edges in the depth-first forest G! . Edge .u; #/ is a tree edge if #
was first discovered by exploring edge .u; #/.
2. Back edges are those edges .u; #/ connecting a vertex u to an ancestor # in a
depth-first tree. We consider self-loops, which may occur in directed graphs, to
be back edges.
3. Forward edges are those nontree edges .u; #/ connecting a vertex u to a descendant # in a depth-first tree.
4. Cross edges are all other edges. They can go between vertices in the same
depth-first tree, as long as one vertex is not an ancestor of the other, or they can
go between vertices in different depth-first trees.
In Figures 22.4 and 22.5, edge labels indicate edge types. Figure 22.5(c) also shows
how to redraw the graph of Figure 22.5(a) so that all tree and forward edges head
downward in a depth-first tree and all back edges go up. We can redraw any graph
in this fashion.
The DFS algorithm has enough information to classify some edges as it encounters them. The key idea is that when we first explore an edge .u; #/, the color of
vertex # tells us something about the edge:
1. WHITE indicates a tree edge,
2. GRAY indicates a back edge, and
3. BLACK indicates a forward or cross edge.
The first case is immediate from the specification of the algorithm. For the second case, observe that the gray vertices always form a linear chain of descendants
corresponding to the stack of active DFS-V ISIT invocations; the number of gray
vertices is one more than the depth in the depth-first forest of the vertex most recently discovered. Exploration always proceeds from the deepest gray vertex, so
610
Chapter 22 Elementary Graph Algorithms
an edge that reaches another gray vertex has reached an ancestor. The third case
handles the remaining possibility; Exercise 22.3-5 asks you to show that such an
edge .u; #/ is a forward edge if u:d < #:d and a cross edge if u:d > #:d.
An undirected graph may entail some ambiguity in how we classify edges,
since .u; #/ and .#; u/ are really the same edge. In such a case, we classify the
edge as the first type in the classification list that applies. Equivalently (see Exercise 22.3-6), we classify the edge according to whichever of .u; #/ or .#; u/ the
search encounters first.
We now show that forward and cross edges never occur in a depth-first search of
an undirected graph.
Theorem 22.10
In a depth-first search of an undirected graph G, every edge of G is either a tree
edge or a back edge.
Proof Let .u; #/ be an arbitrary edge of G, and suppose without loss of generality
that u:d < #:d. Then the search must discover and finish # before it finishes u
(while u is gray), since # is on u’s adjacency list. If the first time that the search
explores edge .u; #/, it is in the direction from u to #, then # is undiscovered
(white) until that time, for otherwise the search would have explored this edge
already in the direction from # to u. Thus, .u; #/ becomes a tree edge. If the
search explores .u; #/ first in the direction from # to u, then .u; #/ is a back edge,
since u is still gray at the time the edge is first explored.
We shall see several applications of these theorems in the following sections.
Exercises
22.3-1
Make a 3-by-3 chart with row and column labels WHITE, GRAY, and BLACK. In
each cell .i; j /, indicate whether, at any point during a depth-first search of a directed graph, there can be an edge from a vertex of color i to a vertex of color j .
For each possible edge, indicate what edge types it can be. Make a second such
chart for depth-first search of an undirected graph.
22.3-2
Show how depth-first search works on the graph of Figure 22.6. Assume that the
for loop of lines 5–7 of the DFS procedure considers the vertices in alphabetical
order, and assume that each adjacency list is ordered alphabetically. Show the
discovery and finishing times for each vertex, and show the classification of each
edge.
22.3 Depth-first search
611
q
s
v
r
t
w
x
u
y
z
Figure 22.6 A directed graph for use in Exercises 22.3-2 and 22.5-2.
22.3-3
Show the parenthesis structure of the depth-first search of Figure 22.4.
22.3-4
Show that using a single bit to store each vertex color suffices by arguing that
the DFS procedure would produce the same result if line 3 of DFS-V ISIT was
removed.
22.3-5
Show that edge .u; #/ is
a. a tree edge or forward edge if and only if u:d < #:d < #:f < u:f ,
b. a back edge if and only if #:d $ u:d < u:f $ #:f , and
c. a cross edge if and only if #:d < #:f < u:d < u:f .
22.3-6
Show that in an undirected graph, classifying an edge .u; #/ as a tree edge or a back
edge according to whether .u; #/ or .#; u/ is encountered first during the depth-first
search is equivalent to classifying it according to the ordering of the four types in
the classification scheme.
22.3-7
Rewrite the procedure DFS, using a stack to eliminate recursion.
22.3-8
Give a counterexample to the conjecture that if a directed graph G contains a path
from u to #, and if u:d < #:d in a depth-first search of G, then # is a descendant
of u in the depth-first forest produced.
612
Chapter 22 Elementary Graph Algorithms
22.3-9
Give a counterexample to the conjecture that if a directed graph G contains a path
from u to #, then any depth-first search must result in #:d $ u:f .
22.3-10
Modify the pseudocode for depth-first search so that it prints out every edge in the
directed graph G, together with its type. Show what modifications, if any, you need
to make if G is undirected.
22.3-11
Explain how a vertex u of a directed graph can end up in a depth-first tree containing only u, even though u has both incoming and outgoing edges in G.
22.3-12
Show that we can use a depth-first search of an undirected graph G to identify the
connected components of G, and that the depth-first forest contains as many trees
as G has connected components. More precisely, show how to modify depth-first
search so that it assigns to each vertex # an integer label #:cc between 1 and k,
where k is the number of connected components of G, such that u:cc D #:cc if
and only if u and # are in the same connected component.
22.3-13 ?
A directed graph G D .V; E/ is singly connected if u ❀ # implies that G contains
at most one simple path from u to # for all vertices u; # 2 V . Give an efficient
algorithm to determine whether or not a directed graph is singly connected.
22.4 Topological sort
This section shows how we can use depth-first search to perform a topological sort
of a directed acyclic graph, or a “dag” as it is sometimes called. A topological sort
of a dag G D .V; E/ is a linear ordering of all its vertices such that if G contains an
edge .u; #/, then u appears before # in the ordering. (If the graph contains a cycle,
then no linear ordering is possible.) We can view a topological sort of a graph as
an ordering of its vertices along a horizontal line so that all directed edges go from
left to right. Topological sorting is thus different from the usual kind of “sorting”
studied in Part II.
Many applications use directed acyclic graphs to indicate precedences among
events. Figure 22.7 gives an example that arises when Professor Bumstead gets
dressed in the morning. The professor must don certain garments before others
(e.g., socks before shoes). Other items may be put on in any order (e.g., socks and
22.4 Topological sort
undershorts
11/16
(a)
613
12/15
pants
6/7
belt
socks 17/18
shirt 1/8
tie
shoes 13/14
watch 9/10
2/5
jacket 3/4
(b)
socks
undershorts
pants
shoes
watch
shirt
belt
tie
jacket
17/18
11/16
12/15
13/14
9/10
1/8
6/7
2/5
3/4
Figure 22.7 (a) Professor Bumstead topologically sorts his clothing when getting dressed. Each
directed edge .u; #/ means that garment u must be put on before garment #. The discovery and
finishing times from a depth-first search are shown next to each vertex. (b) The same graph shown
topologically sorted, with its vertices arranged from left to right in order of decreasing finishing time.
All directed edges go from left to right.
pants). A directed edge .u; #/ in the dag of Figure 22.7(a) indicates that garment u
must be donned before garment #. A topological sort of this dag therefore gives an
order for getting dressed. Figure 22.7(b) shows the topologically sorted dag as an
ordering of vertices along a horizontal line such that all directed edges go from left
to right.
The following simple algorithm topologically sorts a dag:
T OPOLOGICAL -S ORT .G/
1 call DFS.G/ to compute finishing times #:f for each vertex #
2 as each vertex is finished, insert it onto the front of a linked list
3 return the linked list of vertices
Figure 22.7(b) shows how the topologically sorted vertices appear in reverse order
of their finishing times.
We can perform a topological sort in time ‚.V C E/, since depth-first search
takes ‚.V C E/ time and it takes O.1/ time to insert each of the jV j vertices onto
the front of the linked list.
We prove the correctness of this algorithm using the following key lemma characterizing directed acyclic graphs.
614
Chapter 22 Elementary Graph Algorithms
Lemma 22.11
A directed graph G is acyclic if and only if a depth-first search of G yields no back
edges.
Proof ): Suppose that a depth-first search produces a back edge .u; #/. Then
vertex # is an ancestor of vertex u in the depth-first forest. Thus, G contains a path
from # to u, and the back edge .u; #/ completes a cycle.
(: Suppose that G contains a cycle c. We show that a depth-first search of G
yields a back edge. Let # be the first vertex to be discovered in c, and let .u; #/ be
the preceding edge in c. At time #:d, the vertices of c form a path of white vertices
from # to u. By the white-path theorem, vertex u becomes a descendant of # in the
depth-first forest. Therefore, .u; #/ is a back edge.
Theorem 22.12
T OPOLOGICAL -S ORT produces a topological sort of the directed acyclic graph
provided as its input.
Proof Suppose that DFS is run on a given dag G D .V; E/ to determine finishing times for its vertices. It suffices to show that for any pair of distinct vertices u; # 2 V , if G contains an edge from u to #, then #:f < u:f . Consider any
edge .u; #/ explored by DFS.G/. When this edge is explored, # cannot be gray,
since then # would be an ancestor of u and .u; #/ would be a back edge, contradicting Lemma 22.11. Therefore, # must be either white or black. If # is white,
it becomes a descendant of u, and so #:f < u:f . If # is black, it has already been
finished, so that #:f has already been set. Because we are still exploring from u, we
have yet to assign a timestamp to u:f , and so once we do, we will have #:f < u:f
as well. Thus, for any edge .u; #/ in the dag, we have #:f < u:f , proving the
theorem.
Exercises
22.4-1
Show the ordering of vertices produced by T OPOLOGICAL -S ORT when it is run on
the dag of Figure 22.8, under the assumption of Exercise 22.3-2.
22.4-2
Give a linear-time algorithm that takes as input a directed acyclic graph G D
.V; E/ and two vertices s and t, and returns the number of simple paths from s
to t in G. For example, the directed acyclic graph of Figure 22.8 contains exactly
four simple paths from vertex p to vertex #: po#, pory#, posry#, and psry#.
(Your algorithm needs only to count the simple paths, not list them.)
22.5 Strongly connected components
m
n
q
t
o
r
u
x
615
p
s
v
y
w
z
Figure 22.8 A dag for topological sorting.
22.4-3
Give an algorithm that determines whether or not a given undirected graph G D
.V; E/ contains a cycle. Your algorithm should run in O.V / time, independent
of jEj.
22.4-4
Prove or disprove: If a directed graph G contains cycles, then T OPOLOGICAL S ORT.G/ produces a vertex ordering that minimizes the number of “bad” edges
that are inconsistent with the ordering produced.
22.4-5
Another way to perform topological sorting on a directed acyclic graph G D
.V; E/ is to repeatedly find a vertex of in-degree 0, output it, and remove it and
all of its outgoing edges from the graph. Explain how to implement this idea so
that it runs in time O.V C E/. What happens to this algorithm if G has cycles?
22.5 Strongly connected components
We now consider a classic application of depth-first search: decomposing a directed graph into its strongly connected components. This section shows how to do
so using two depth-first searches. Many algorithms that work with directed graphs
begin with such a decomposition. After decomposing the graph into strongly connected components, such algorithms run separately on each one and then combine
the solutions according to the structure of connections among components.
Recall from Appendix B that a strongly connected component of a directed
graph G D .V; E/ is a maximal set of vertices C & V such that for every pair
of vertices u and # in C , we have both u ❀ # and # ❀ u; that is, vertices u and #
are reachable from each other. Figure 22.9 shows an example.
-
616
Chapter 22 Elementary Graph Algorithms
a
13/14
b
11/16
c
1/10
d
8/9
12/15
3/4
2/7
5/6
a
b
c
d
e
f
g
h
(a)
e
f
g
h
(b)
cd
(c)
abe
fg
h
Figure 22.9 (a) A directed graph G. Each shaded region is a strongly connected component of G.
Each vertex is labeled with its discovery and finishing times in a depth-first search, and tree edges
are shaded. (b) The graph
G T , the transpose of G, with the depth-first forest computed in line 3
of S TRONGLY-C ONNECTED -C OMPONENTS shown and tree edges shaded. Each strongly connected
component corresponds to one depth-first tree. Vertices b, c, g, and h, which are heavily shaded, are
the roots of the depth-first trees produced by the depth-first search of G T . (c) The acyclic component
graph G SCC obtained by contracting all edges within each strongly connected component of G so
that only a single vertex remains in each component.
-
-
-
Our algorithm for finding strongly connected components of a graph G D
.V; E/ uses the transpose of G, which we defined in Exercise 22.1-3 to be the
graph G T D .V; E T /, where E T D f.u; #/ W .#; u/ 2 Eg. That is, E T consists of
the edges of G with their directions reversed. Given an adjacency-list representation of G, the time to create G T is O.V C E/. It is interesting to observe that G
and G T have exactly the same strongly connected components: u and # are reachable from each other in G if and only if they are reachable from each other in G T .
Figure 22.9(b) shows the transpose of the graph in Figure 22.9(a), with the strongly
connected components shaded.
22.5 Strongly connected components
617
The following linear-time (i.e., ‚.V CE/-time) algorithm computes the strongly
connected components of a directed graph G D .V; E/ using two depth-first
searches, one on G and one on G T .
S TRONGLY-C ONNECTED -C OMPONENTS .G/
1 call DFS.G/ to compute finishing times u:f for each vertex u
2 compute G T
3 call DFS.G T /, but in the main loop of DFS, consider the vertices
in order of decreasing u:f (as computed in line 1)
4 output the vertices of each tree in the depth-first forest formed in line 3 as a
separate strongly connected component
The idea behind this algorithm comes from a key property of the component
graph G SCC D .V SCC ; E SCC /, which we define as follows. Suppose that G
has strongly connected components C1 ; C2 ; : : : ; Ck . The vertex set V SCC is
f#1 ; #2 ; : : : ; #k g, and it contains a vertex #i for each strongly connected component Ci of G. There is an edge .#i ; #j / 2 E SCC if G contains a directed edge .x; y/
for some x 2 Ci and some y 2 Cj . Looked at another way, by contracting all
edges whose incident vertices are within the same strongly connected component
of G, the resulting graph is G SCC . Figure 22.9(c) shows the component graph of
the graph in Figure 22.9(a).
The key property is that the component graph is a dag, which the following
lemma implies.
Lemma 22.13
Let C and C 0 be distinct strongly connected components in directed graph G D
.V; E/, let u; # 2 C , let u0 ; # 0 2 C 0 , and suppose that G contains a path u ❀ u0 .
Then G cannot also contain a path # 0 ❀ #.
Proof If G contains a path # 0 ❀ #, then it contains paths u ❀ u0 ❀ # 0 and
# 0 ❀ # ❀ u. Thus, u and # 0 are reachable from each other, thereby contradicting
the assumption that C and C 0 are distinct strongly connected components.
We shall see that by considering vertices in the second depth-first search in decreasing order of the finishing times that were computed in the first depth-first
search, we are, in essence, visiting the vertices of the component graph (each of
which corresponds to a strongly connected component of G) in topologically sorted
order.
Because the S TRONGLY-C ONNECTED -C OMPONENTS procedure performs two
depth-first searches, there is the potential for ambiguity when we discuss u:d
or u:f . In this section, these values always refer to the discovery and finishing
times as computed by the first call of DFS, in line 1.
618
Chapter 22 Elementary Graph Algorithms
We extend the notation for discovery and finishing times to sets of vertices.
If U & V , then we define d.U / D minu2U fu:dg and f .U / D maxu2U fu:f g.
That is, d.U / and f .U / are the earliest discovery time and latest finishing time,
respectively, of any vertex in U .
The following lemma and its corollary give a key property relating strongly connected components and finishing times in the first depth-first search.
Lemma 22.14
Let C and C 0 be distinct strongly connected components in directed graph G D
.V; E/. Suppose that there is an edge .u; #/ 2 E, where u 2 C and # 2 C 0 . Then
f .C / > f .C 0 /.
Proof We consider two cases, depending on which strongly connected component, C or C 0 , had the first discovered vertex during the depth-first search.
If d.C / < d.C 0 /, let x be the first vertex discovered in C . At time x:d, all vertices in C and C 0 are white. At that time, G contains a path from x to each vertex
in C consisting only of white vertices. Because .u; #/ 2 E, for any vertex w 2 C 0 ,
there is also a path in G at time x:d from x to w consisting only of white vertices:
x ❀ u ! # ❀ w. By the white-path theorem, all vertices in C and C 0 become
descendants of x in the depth-first tree. By Corollary 22.8, x has the latest finishing
time of any of its descendants, and so x:f D f .C / > f .C 0 /.
If instead we have d.C / > d.C 0 /, let y be the first vertex discovered in C 0 .
At time y:d, all vertices in C 0 are white and G contains a path from y to each
vertex in C 0 consisting only of white vertices. By the white-path theorem, all vertices in C 0 become descendants of y in the depth-first tree, and by Corollary 22.8,
y:f D f .C 0 /. At time y:d, all vertices in C are white. Since there is an edge .u; #/
from C to C 0 , Lemma 22.13 implies that there cannot be a path from C 0 to C .
Hence, no vertex in C is reachable from y. At time y:f , therefore, all vertices in C
are still white. Thus, for any vertex w 2 C , we have w:f > y:f , which implies
that f .C / > f .C 0 /.
The following corollary tells us that each edge in G T that goes between different
strongly connected components goes from a component with an earlier finishing
time (in the first depth-first search) to a component with a later finishing time.
Corollary 22.15
Let C and C 0 be distinct strongly connected components in directed graph G D
.V; E/. Suppose that there is an edge .u; #/ 2 E T , where u 2 C and # 2 C 0 . Then
f .C / < f .C 0 /.
22.5 Strongly connected components
619
Proof Since .u; #/ 2 E T , we have .#; u/ 2 E. Because the strongly connected components of G and G T are the same, Lemma 22.14 implies that
f .C / < f .C 0 /.
Corollary 22.15 provides the key to understanding why the strongly connected
components algorithm works. Let us examine what happens when we perform the
second depth-first search, which is on G T . We start with the strongly connected
component C whose finishing time f .C / is maximum. The search starts from
some vertex x 2 C , and it visits all vertices in C . By Corollary 22.15, G T contains
no edges from C to any other strongly connected component, and so the search
from x will not visit vertices in any other component. Thus, the tree rooted at x
contains exactly the vertices of C . Having completed visiting all vertices in C ,
the search in line 3 selects as a root a vertex from some other strongly connected
component C 0 whose finishing time f .C 0 / is maximum over all components other
than C . Again, the search will visit all vertices in C 0 , but by Corollary 22.15,
the only edges in G T from C 0 to any other component must be to C , which we
have already visited. In general, when the depth-first search of G T in line 3 visits
any strongly connected component, any edges out of that component must be to
components that the search already visited. Each depth-first tree, therefore, will be
exactly one strongly connected component. The following theorem formalizes this
argument.
Theorem 22.16
The S TRONGLY-C ONNECTED -C OMPONENTS procedure correctly computes the
strongly connected components of the directed graph G provided as its input.
Proof We argue by induction on the number of depth-first trees found in the
depth-first search of G T in line 3 that the vertices of each tree form a strongly
connected component. The inductive hypothesis is that the first k trees produced
in line 3 are strongly connected components. The basis for the induction, when
k D 0, is trivial.
In the inductive step, we assume that each of the first k depth-first trees produced
in line 3 is a strongly connected component, and we consider the .k C 1/st tree
produced. Let the root of this tree be vertex u, and let u be in strongly connected
component C . Because of how we choose roots in the depth-first search in line 3,
u:f D f .C / > f .C 0 / for any strongly connected component C 0 other than C
that has yet to be visited. By the inductive hypothesis, at the time that the search
visits u, all other vertices of C are white. By the white-path theorem, therefore, all
other vertices of C are descendants of u in its depth-first tree. Moreover, by the
inductive hypothesis and by Corollary 22.15, any edges in G T that leave C must be
to strongly connected components that have already been visited. Thus, no vertex
620
Chapter 22 Elementary Graph Algorithms
in any strongly connected component other than C will be a descendant of u during
the depth-first search of G T . Thus, the vertices of the depth-first tree in G T that is
rooted at u form exactly one strongly connected component, which completes the
inductive step and the proof.
Here is another way to look at how the second depth-first search operates. Consider the component graph .G T /SCC of G T . If we map each strongly connected
component visited in the second depth-first search to a vertex of .G T /SCC , the second depth-first search visits vertices of .G T /SCC in the reverse of a topologically
sorted order. If we reverse the edges of .G T /SCC , we get the graph ..G T /SCC /T .
Because ..G T /SCC /T D G SCC (see Exercise 22.5-4), the second depth-first search
visits the vertices of G SCC in topologically sorted order.
Exercises
22.5-1
How can the number of strongly connected components of a graph change if a new
edge is added?
22.5-2
Show how the procedure S TRONGLY-C ONNECTED -C OMPONENTS works on the
graph of Figure 22.6. Specifically, show the finishing times computed in line 1 and
the forest produced in line 3. Assume that the loop of lines 5–7 of DFS considers
vertices in alphabetical order and that the adjacency lists are in alphabetical order.
22.5-3
Professor Bacon claims that the algorithm for strongly connected components
would be simpler if it used the original (instead of the transpose) graph in the
second depth-first search and scanned the vertices in order of increasing finishing
times. Does this simpler algorithm always produce correct results?
22.5-4
Prove that for any directed graph G, we have ..G T /SCC /T D G SCC . That is, the
transpose of the component graph of G T is the same as the component graph of G.
22.5-5
Give an O.V C E/-time algorithm to compute the component graph of a directed
graph G D .V; E/. Make sure that there is at most one edge between two vertices
in the component graph your algorithm produces.
Problems for Chapter 22
621
22.5-6
Given a directed graph G D .V; E/, explain how to create another graph G 0 D
.V; E 0 / such that (a) G 0 has the same strongly connected components as G, (b) G 0
has the same component graph as G, and (c) E 0 is as small as possible. Describe a
fast algorithm to compute G 0 .
22.5-7
A directed graph G D .V; E/ is semiconnected if, for all pairs of vertices u; # 2 V ,
we have u ❀ # or # ❀ u. Give an efficient algorithm to determine whether
or not G is semiconnected. Prove that your algorithm is correct, and analyze its
running time.
Problems
22-1 Classifying edges by breadth-first search
A depth-first forest classifies the edges of a graph into tree, back, forward, and
cross edges. A breadth-first tree can also be used to classify the edges reachable
from the source of the search into the same four categories.
a. Prove that in a breadth-first search of an undirected graph, the following properties hold:
1. There are no back edges and no forward edges.
2. For each tree edge .u; #/, we have #:d D u:d C 1.
3. For each cross edge .u; #/, we have #:d D u:d or #:d D u:d C 1.
b. Prove that in a breadth-first search of a directed graph, the following properties
hold:
1. There are no forward edges.
2. For each tree edge .u; #/, we have #:d D u:d C 1.
3. For each cross edge .u; #/, we have #:d $ u:d C 1.
4. For each back edge .u; #/, we have 0 $ #:d $ u:d.
22-2 Articulation points, bridges, and biconnected components
Let G D .V; E/ be a connected, undirected graph. An articulation point of G is
a vertex whose removal disconnects G. A bridge of G is an edge whose removal
disconnects G. A biconnected component of G is a maximal set of edges such
that any two edges in the set lie on a common simple cycle. Figure 22.10 illustrates
622
Chapter 22 Elementary Graph Algorithms
2
1
6
4
3
5
Figure 22.10 The articulation points, bridges, and biconnected components of a connected, undirected graph for use in Problem 22-2. The articulation points are the heavily shaded vertices, the
bridges are the heavily shaded edges, and the biconnected components are the edges in the shaded
regions, with a bcc numbering shown.
these definitions. We can determine articulation points, bridges, and biconnected
components using depth-first search. Let G! D .V; E! / be a depth-first tree of G.
a. Prove that the root of G! is an articulation point of G if and only if it has at
least two children in G! .
b. Let # be a nonroot vertex of G! . Prove that # is an articulation point of G if and
only if # has a child s such that there is no back edge from s or any descendant
of s to a proper ancestor of #.
c. Let
#:low D min
(
#:d ;
w:d W .u; w/ is a back edge for some descendant u of # :
Show how to compute #:low for all vertices # 2 V in O.E/ time.
d. Show how to compute all articulation points in O.E/ time.
e. Prove that an edge of G is a bridge if and only if it does not lie on any simple
cycle of G.
f. Show how to compute all the bridges of G in O.E/ time.
g. Prove that the biconnected components of G partition the nonbridge edges of G.
h. Give an O.E/-time algorithm to label each edge e of G with a positive integer e:bcc such that e:bcc D e 0 :bcc if and only if e and e 0 are in the same
biconnected component.
Notes for Chapter 22
623
22-3 Euler tour
An Euler tour of a strongly connected, directed graph G D .V; E/ is a cycle that
traverses each edge of G exactly once, although it may visit a vertex more than
once.
a. Show that G has an Euler tour if and only if in-degree.#/ D out-degree.#/ for
each vertex # 2 V .
b. Describe an O.E/-time algorithm to find an Euler tour of G if one exists. (Hint:
Merge edge-disjoint cycles.)
22-4 Reachability
Let G D .V; E/ be a directed graph in which each vertex u 2 V is labeled with
a unique integer L.u/ from the set f1; 2; : : : ; jV jg. For each vertex u 2 V , let
R.u/ D f# 2 V W u ❀ #g be the set of vertices that are reachable from u. Define
min.u/ to be the vertex in R.u/ whose label is minimum, i.e., min.u/ is the vertex #
such that L.#/ D min fL.w/ W w 2 R.u/g. Give an O.V C E/-time algorithm that
computes min.u/ for all vertices u 2 V .
Chapter notes
Even [103] and Tarjan [330] are excellent references for graph algorithms.
Breadth-first search was discovered by Moore [260] in the context of finding
paths through mazes. Lee [226] independently discovered the same algorithm in
the context of routing wires on circuit boards.
Hopcroft and Tarjan [178] advocated the use of the adjacency-list representation
over the adjacency-matrix representation for sparse graphs and were the first to
recognize the algorithmic importance of depth-first search. Depth-first search has
been widely used since the late 1950s, especially in artificial intelligence programs.
Tarjan [327] gave a linear-time algorithm for finding strongly connected components. The algorithm for strongly connected components in Section 22.5 is adapted
from Aho, Hopcroft, and Ullman [6], who credit it to S. R. Kosaraju (unpublished)
and M. Sharir [314]. Gabow [119] also developed an algorithm for strongly connected components that is based on contracting cycles and uses two stacks to make
it run in linear time. Knuth [209] was the first to give a linear-time algorithm for
topological sorting.
23
Minimum Spanning Trees
Electronic circuit designs often need to make the pins of several components electrically equivalent by wiring them together. To interconnect a set of n pins, we can
use an arrangement of n " 1 wires, each connecting two pins. Of all such arrangements, the one that uses the least amount of wire is usually the most desirable.
We can model this wiring problem with a connected, undirected graph G D
.V; E/, where V is the set of pins, E is the set of possible interconnections between
pairs of pins, and for each edge .u; #/ 2 E, we have a weight w.u; #/ specifying
the cost (amount of wire needed) to connect u and #. We then wish to find an
acyclic subset T & E that connects all of the vertices and whose total weight
X
w.u; #/
w.T / D
.u;"/2T
is minimized. Since T is acyclic and connects all of the vertices, it must form a tree,
which we call a spanning tree since it “spans” the graph G. We call the problem of
determining the tree T the minimum-spanning-tree problem.1 Figure 23.1 shows
an example of a connected graph and a minimum spanning tree.
In this chapter, we shall examine two algorithms for solving the minimumspanning-tree problem: Kruskal’s algorithm and Prim’s algorithm. We can easily
make each of them run in time O.E lg V / using ordinary binary heaps. By using
Fibonacci heaps, Prim’s algorithm runs in time O.E C V lg V /, which improves
over the binary-heap implementation if jV j is much smaller than jEj.
The two algorithms are greedy algorithms, as described in Chapter 16. Each
step of a greedy algorithm must make one of several possible choices. The greedy
strategy advocates making the choice that is the best at the moment. Such a strategy does not generally guarantee that it will always find globally optimal solutions
1 The phrase “minimum spanning tree” is a shortened form of the phrase “minimum-weight spanning
tree.” We are not, for example, minimizing the number of edges in T , since all spanning trees have
exactly jV j " 1 edges by Theorem B.2.
23.1 Growing a minimum spanning tree
4
11
a
8
8
b
h
7
i
1
7
c
2
4
6
g
d
14
2
f
625
9
e
10
Figure 23.1 A minimum spanning tree for a connected graph. The weights on edges are shown,
and the edges in a minimum spanning tree are shaded. The total weight of the tree shown is 37. This
minimum spanning tree is not unique: removing the edge .b; c/ and replacing it with the edge .a; h/
yields another spanning tree with weight 37.
to problems. For the minimum-spanning-tree problem, however, we can prove that
certain greedy strategies do yield a spanning tree with minimum weight. Although
you can read this chapter independently of Chapter 16, the greedy methods presented here are a classic application of the theoretical notions introduced there.
Section 23.1 introduces a “generic” minimum-spanning-tree method that grows
a spanning tree by adding one edge at a time. Section 23.2 gives two algorithms
that implement the generic method. The first algorithm, due to Kruskal, is similar
to the connected-components algorithm from Section 21.1. The second, due to
Prim, resembles Dijkstra’s shortest-paths algorithm (Section 24.3).
Because a tree is a type of graph, in order to be precise we must define a tree in
terms of not just its edges, but its vertices as well. Although this chapter focuses
on trees in terms of their edges, we shall operate with the understanding that the
vertices of a tree T are those that some edge of T is incident on.
23.1 Growing a minimum spanning tree
Assume that we have a connected, undirected graph G D .V; E/ with a weight
function w W E ! R, and we wish to find a minimum spanning tree for G. The
two algorithms we consider in this chapter use a greedy approach to the problem,
although they differ in how they apply this approach.
This greedy strategy is captured by the following generic method, which grows
the minimum spanning tree one edge at a time. The generic method manages a set
of
edges A, maintaining the following loop invariant:
-
Prior to each iteration, A is a subset of some minimum spanning tree.
At
each step, we determine an edge .u; #/ that we can add to A without violating
this invariant, in the sense that A [ f.u; #/g is also a subset of a minimum spanning
-
626
Chapter 23 Minimum Spanning Trees
tree. We call such an edge a safe edge for A, since we can add it safely to A while
maintaining the invariant.
-
G ENERIC -MST.G; w/
1 AD;
2 while A does not form a spanning tree
3
find an edge .u; #/ that is safe for A
4
A D A [ f.u; #/g
5 return A
We use the loop invariant as follows:
Initialization: After line 1, the set A trivially satisfies the loop invariant.
Maintenance: The loop in lines 2–4 maintains the invariant by adding only safe
edges.
Termination: All edges added to A are in a minimum spanning tree, and so the
set A returned in line 5 must be a minimum spanning tree.
The tricky part is, of course, finding a safe edge in line 3. One must exist, since
when line 3 is executed, the invariant dictates that there is a spanning tree T such
that A & T . Within the while loop body, A must be a proper subset of T , and
therefore there must be an edge .u; #/ 2 T such that .u; #/ 62 A and .u; #/ is safe
for A.
In the remainder of this section, we provide a rule (Theorem 23.1) for recognizing safe edges. The next section describes two algorithms that use this rule to find
safe edges efficiently.
We first need some definitions. A cut .S; V " S/ of an undirected graph G D
.V; E/ is a partition of V . Figure 23.2 illustrates this notion. We say that an edge
.u; #/ 2 E crosses the cut .S; V " S/ if one of its endpoints is in S and the other
is in V " S. We say that a cut respects a set A of edges if no edge in A crosses the
cut. An edge is a light edge crossing a cut if its weight is the minimum of any edge
crossing the cut. Note that there can be more than one light edge crossing a cut in
the case of ties. More generally, we say that an edge is a light edge satisfying a
given property if its weight is the minimum of any edge satisfying the property.
Our rule for recognizing safe edges is given by the following theorem.
Theorem 23.1
Let G D .V; E/ be a connected, undirected graph with a real-valued weight function w defined on E. Let A be a subset of E that is included in some minimum
spanning tree for G, let .S; V " S/ be any cut of G that respects A, and let .u; #/
be a light edge crossing .S; V " S/. Then, edge .u; #/ is safe for A.
23.1 Growing a minimum spanning tree
627
a
4
h
8
7
11
i
6
b
2
S
V–S
4
a
11
8
8
b
h
7
i
1
7
c
2
g
2
f
d
9
e
14
4
6
d
10
(a)
S
V–S
g
8
7
9
e
1
14
10
c
2
4
f
S V–S
(b)
Figure 23.2 Two ways of viewing a cut .S; V " S/ of the graph from Figure 23.1. (a) Black
vertices are in the set S, and white vertices are in V " S. The edges crossing the cut are those
connecting white vertices with black vertices. The edge .d; c/ is the unique light edge crossing the
cut. A subset A of the edges is shaded; note that the cut .S; V " S/ respects A, since no edge of A
crosses the cut. (b) The same graph with the vertices in the set S on the left and the vertices in the
set V " S on the right. An edge crosses the cut if it connects a vertex on the left with a vertex on the
right.
Proof Let T be a minimum spanning tree that includes A, and assume that T
does not contain the light edge .u; #/, since if it does, we are done. We shall
construct another minimum spanning tree T 0 that includes A [ f.u; #/g by using a
cut-and-paste technique, thereby showing that .u; #/ is a safe edge for A.
The edge .u; #/ forms a cycle with the edges on the simple path p from u
to # in T , as Figure 23.3 illustrates. Since u and # are on opposite sides of the
cut .S; V " S/, at least one edge in T lies on the simple path p and also crosses
the cut. Let .x; y/ be any such edge. The edge .x; y/ is not in A, because the cut
respects A. Since .x; y/ is on the unique simple path from u to # in T , removing .x; y/ breaks T into two components. Adding .u; #/ reconnects them to form
a new spanning tree T 0 D T " f.x; y/g [ f.u; #/g.
We next show that T 0 is a minimum spanning tree. Since .u; #/ is a light edge
crossing .S; V "S/ and .x; y/ also crosses this cut, w.u; #/ $ w.x; y/. Therefore,
w.T 0 / D w.T / " w.x; y/ C w.u; #/
$ w.T / :
628
Chapter 23 Minimum Spanning Trees
x
p
u
y
v
Figure 23.3 The proof of Theorem 23.1. Black vertices are in S, and white vertices are in V " S.
The edges in the minimum spanning tree T are shown, but the edges in the graph G are not. The
edges in A are shaded, and .u; #/ is a light edge crossing the cut .S; V " S/. The edge .x; y/ is
an edge on the unique simple path p from u to # in T . To form a minimum spanning tree T 0 that
contains .u; #/, remove the edge .x; y/ from T and add the edge .u; #/.
But T is a minimum spanning tree, so that w.T / $ w.T 0 /; thus, T 0 must be a
minimum spanning tree also.
It remains to show that .u; #/ is actually a safe edge for A. We have A & T 0 ,
since A & T and .x; y/ 62 A; thus, A [ f.u; #/g & T 0 . Consequently, since T 0 is a
minimum spanning tree, .u; #/ is safe for A.
Theorem 23.1 gives us a better understanding of the workings of the G ENERIC MST method on a connected graph G D .V; E/. As the method proceeds, the
set A is always acyclic; otherwise, a minimum spanning tree including A would
contain a cycle, which is a contradiction. At any point in the execution, the graph
GA D .V; A/ is a forest, and each of the connected components of GA is a tree.
(Some of the trees may contain just one vertex, as is the case, for example, when
the method begins: A is empty and the forest contains jV j trees, one for each
vertex.) Moreover, any safe edge .u; #/ for A connects distinct components of GA ,
since A [ f.u; #/g must be acyclic.
The while loop in lines 2–4 of G ENERIC -MST executes jV j " 1 times because
it finds one of the jV j " 1 edges of a minimum spanning tree in each iteration.
Initially, when A D ;, there are jV j trees in GA , and each iteration reduces that
number by 1. When the forest contains only a single tree, the method terminates.
The two algorithms in Section 23.2 use the following corollary to Theorem 23.1.
23.1 Growing a minimum spanning tree
629
Corollary 23.2
Let G D .V; E/ be a connected, undirected graph with a real-valued weight function w defined on E. Let A be a subset of E that is included in some minimum
spanning tree for G, and let C D .VC ; EC / be a connected component (tree) in the
forest GA D .V; A/. If .u; #/ is a light edge connecting C to some other component
in GA , then .u; #/ is safe for A.
Proof The cut .VC ; V " VC / respects A, and .u; #/ is a light edge for this cut.
Therefore, .u; #/ is safe for A.
Exercises
23.1-1
Let .u; #/ be a minimum-weight edge in a connected graph G. Show that .u; #/
belongs to some minimum spanning tree of G.
23.1-2
Professor Sabatier conjectures the following converse of Theorem 23.1. Let G D
.V; E/ be a connected, undirected graph with a real-valued weight function w defined on E. Let A be a subset of E that is included in some minimum spanning
tree for G, let .S; V " S/ be any cut of G that respects A, and let .u; #/ be a safe
edge for A crossing .S; V " S/. Then, .u; #/ is a light edge for the cut. Show that
the professor’s conjecture is incorrect by giving a counterexample.
23.1-3
Show that if an edge .u; #/ is contained in some minimum spanning tree, then it is
a light edge crossing some cut of the graph.
23.1-4
Give a simple example of a connected graph such that the set of edges f.u; #/ W
there exists a cut .S; V " S/ such that .u; #/ is a light edge crossing .S; V " S/g
does not form a minimum spanning tree.
23.1-5
Let e be a maximum-weight edge on some cycle of connected graph G D .V; E/.
Prove that there is a minimum spanning tree of G 0 D .V; E " feg/ that is also a
minimum spanning tree of G. That is, there is a minimum spanning tree of G that
does not include e.
630
Chapter 23 Minimum Spanning Trees
23.1-6
Show that a graph has a unique minimum spanning tree if, for every cut of the
graph, there is a unique light edge crossing the cut. Show that the converse is not
true by giving a counterexample.
23.1-7
Argue that if all edge weights of a graph are positive, then any subset of edges that
connects all vertices and has minimum total weight must be a tree. Give an example
to show that the same conclusion does not follow if we allow some weights to be
nonpositive.
23.1-8
Let T be a minimum spanning tree of a graph G, and let L be the sorted list of the
edge weights of T . Show that for any other minimum spanning tree T 0 of G, the
list L is also the sorted list of edge weights of T 0 .
23.1-9
Let T be a minimum spanning tree of a graph G D .V; E/, and let V 0 be a subset
of V . Let T 0 be the subgraph of T induced by V 0 , and let G 0 be the subgraph of G
induced by V 0 . Show that if T 0 is connected, then T 0 is a minimum spanning tree
of G 0 .
23.1-10
Given a graph G and a minimum spanning tree T , suppose that we decrease the
weight of one of the edges in T . Show that T is still a minimum spanning tree
for G. More formally, let T be a minimum spanning tree for G with edge weights
given by weight function w. Choose one edge .x; y/ 2 T and a positive number k,
and define the weight function w 0 by
(
w.u; #/
if .u; #/ ¤ .x; y/ ;
w 0 .u; #/ D
w.x; y/ " k if .u; #/ D .x; y/ :
Show that T is a minimum spanning tree for G with edge weights given by w 0 .
23.1-11 ?
Given a graph G and a minimum spanning tree T , suppose that we decrease the
weight of one of the edges not in T . Give an algorithm for finding the minimum
spanning tree in the modified graph.
23.2 The algorithms of Kruskal and Prim
631
23.2 The algorithms of Kruskal and Prim
The two minimum-spanning-tree algorithms described in this section elaborate on
the generic method. They each use a specific rule to determine a safe edge in line 3
of G ENERIC -MST. In Kruskal’s algorithm, the set A is a forest whose vertices are
all those of the given graph. The safe edge added to A is always a least-weight
edge in the graph that connects two distinct components. In Prim’s algorithm, the
set A forms a single tree. The safe edge added to A is always a least-weight edge
connecting the tree to a vertex not in the tree.
Kruskal’s algorithm
Kruskal’s algorithm finds a safe edge to add to the growing forest by finding, of all
the edges that connect any two trees in the forest, an edge .u; #/ of least weight.
Let C1 and C2 denote the two trees that are connected by .u; #/. Since .u; #/ must
be a light edge connecting C1 to some other tree, Corollary 23.2 implies that .u; #/
is a safe edge for C1 . Kruskal’s algorithm qualifies as a greedy algorithm because
at each step it adds to the forest an edge of least possible weight.
Our implementation of Kruskal’s algorithm is like the algorithm to compute
connected components from Section 21.1. It uses a disjoint-set data structure to
maintain several disjoint sets of elements. Each set contains the vertices in one tree
of the current forest. The operation F IND -S ET .u/ returns a representative element
from the set that contains u. Thus, we can determine whether two vertices u and #
belong to the same tree by testing whether F IND -S ET .u/ equals F IND -S ET .#/. To
combine trees, Kruskal’s algorithm calls the U NION procedure.
MST-K RUSKAL .G; w/
1 AD;
2 for each vertex # 2 G:V
3
M AKE -S ET .#/
4 sort the edges of G:E into nondecreasing order by weight w
5 for each edge .u; #/ 2 G:E, taken in nondecreasing order by weight
6
if F IND -S ET .u/ ¤ F IND -S ET .#/
7
A D A [ f.u; #/g
8
U NION .u; #/
9 return A
Figure 23.4 shows how Kruskal’s algorithm works. Lines 1–3 initialize the set A
to the empty set and create jV j trees, one containing each vertex. The for loop in
lines 5–8 examines edges in order of weight, from lowest to highest. The loop
632
Chapter 23 Minimum Spanning Trees
4
(a)
11
a
8
4
(c)
8
4
(e)
8
4
(g)
8
7
i
g
2
h
i
g
2
g
8
7
i
1
7
4
6
2
7
4
6
g
f
d
f
d
f
d
2
(b)
8
9
4
e
(d)
10
8
9
4
e
(f)
10
8
9
4
e
10
(h)
8
7
i
g
2
h
i
g
2
g
8
7
i
1
7
4
6
2
7
4
6
g
f
d
e
10
9
f
d
e
10
9
14
2
c
9
14
2
c
1
b
7
4
6
d
14
2
c
8
7
4
6
1
b
h
2
8
h
11
a
i
7
c
1
b
11
a
7
h
11
a
8
b
11
a
10
14
f
4
e
14
2
c
9
14
2
c
1
b
7
4
6
d
14
2
c
8
7
4
6
1
b
h
2
8
h
11
a
i
7
c
1
b
11
a
7
h
11
a
8
b
f
d
e
10
9
14
2
f
e
10
Figure 23.4 The execution of Kruskal’s algorithm on the graph from Figure 23.1. Shaded edges
belong to the forest A being grown. The algorithm considers each edge in sorted order by weight.
An arrow points to the edge under consideration at each step of the algorithm. If the edge joins two
distinct trees in the forest, it is added to the forest, thereby merging the two trees.
checks, for each edge .u; #/, whether the endpoints u and # belong to the same
tree. If they do, then the edge .u; #/ cannot be added to the forest without creating
a cycle, and the edge is discarded. Otherwise, the two vertices belong to different
trees. In this case, line 7 adds the edge .u; #/ to A, and line 8 merges the vertices
in the two trees.
23.2 The algorithms of Kruskal and Prim
4
(i)
11
a
8
4
(k)
8
4
(m)
8
2
7
i
g
2
g
8
7
i
1
7
4
6
2
7
4
6
g
9
f
d
f
d
2
(j)
8
9
4
e
(l)
10
8
9
4
e
10
(n)
8
2
7
i
g
2
g
8
7
i
1
7
4
6
2
7
4
6
g
9
f
d
e
10
9
14
2
c
d
14
2
c
1
b
4
6
8
h
h
i
7
c
1
b
11
a
7
h
11
a
8
b
11
a
10
14
f
4
e
14
2
c
d
14
2
c
1
b
4
6
8
h
h
i
7
c
1
b
11
a
7
h
11
a
8
b
633
f
d
e
10
9
14
2
f
e
10
Figure 23.4, continued Further steps in the execution of Kruskal’s algorithm.
The running time of Kruskal’s algorithm for a graph G D .V; E/ depends
on how we implement the disjoint-set data structure. We assume that we use
the disjoint-set-forest implementation of Section 21.3 with the union-by-rank and
path-compression heuristics, since it is the asymptotically fastest implementation
known. Initializing the set A in line 1 takes O.1/ time, and the time to sort the
edges in line 4 is O.E lg E/. (We will account for the cost of the jV j M AKE -S ET
operations in the for loop of lines 2–3 in a moment.) The for loop of lines 5–8
performs O.E/ F IND -S ET and U NION operations on the disjoint-set forest. Along
with the jV j M AKE -S ET operations, these take a total of O..V C E/ ˛.V // time,
where ˛ is the very slowly growing function defined in Section 21.4. Because we
assume that G is connected, we have jEj # jV j " 1, and so the disjoint-set operations take O.E ˛.V // time. Moreover, since ˛.jV j/ D O.lg V / D O.lg E/, the total running time of Kruskal’s algorithm is O.E lg E/. Observing that jEj < jV j2 ,
we have lg jEj D O.lg V /, and so we can restate the running time of Kruskal’s
algorithm as O.E lg V /.
634
Chapter 23 Minimum Spanning Trees
Prim’s algorithm
Like Kruskal’s algorithm, Prim’s algorithm is a special case of the generic minimum-spanning-tree method from Section 23.1. Prim’s algorithm operates much
like Dijkstra’s algorithm for finding shortest paths in a graph, which we shall see in
Section 24.3. Prim’s algorithm has the property that the edges in the set A always
form a single tree. As Figure 23.5 shows, the tree starts from an arbitrary root
vertex r and grows until the tree spans all the vertices in V . Each step adds to the
tree A a light edge that connects A to an isolated vertex—one on which no edge
of A is incident. By Corollary 23.2, this rule adds only edges that are safe for A;
therefore, when the algorithm terminates, the edges in A form a minimum spanning
tree. This strategy qualifies as greedy since at each step it adds to the tree an edge
that contributes the minimum amount possible to the tree’s weight.
In order to implement Prim’s algorithm efficiently, we need a fast way to select
a new edge to add to the tree formed by the edges in A. In the pseudocode below,
the connected graph G and the root r of the minimum spanning tree to be grown
are inputs to the algorithm. During execution of the algorithm, all vertices that
are not in the tree reside in a min-priority queue Q based on a key attribute. For
each vertex #, the attribute #:key is the minimum weight of any edge connecting #
to a vertex in the tree; by convention, #:key D 1 if there is no such edge. The
attribute #:% names the parent of # in the tree. The algorithm implicitly maintains
the set A from G ENERIC -MST as
A D f.#; #:%/ W # 2 V " frg " Qg :
When the algorithm terminates, the min-priority queue Q is empty; the minimum
spanning tree A for G is thus
A D f.#; #:%/ W # 2 V " frgg :
MST-P RIM .G; w; r/
1 for each u 2 G:V
2
u:key D 1
3
u:% D NIL
4 r:key D 0
5 Q D G:V
6 while Q ¤ ;
7
u D E XTRACT-M IN .Q/
8
for each # 2 G:AdjŒu"
9
if # 2 Q and w.u; #/ < #:key
10
#:% D u
11
#:key D w.u; #/
23.2 The algorithms of Kruskal and Prim
4
(a)
a
11
8
4
(c)
a
4
(e)
a
4
(g)
a
8
4
(i)
a
8
7
g
2
7
i
g
2
g
8
7
h
i
2
g
8
7
i
1
7
4
6
2
7
4
6
g
d
f
d
f
d
f
d
2
a
4
e
(d)
a
10
9
4
e
(f)
a
10
9
4
e
10
(h)
a
8
7
i
g
2
h
i
g
2
g
8
7
i
1
7
4
6
2
7
4
6
g
f
d
e
10
9
f
d
e
10
9
14
2
c
9
14
2
c
1
b
7
4
6
d
14
2
c
8
7
4
6
1
b
h
2
8
h
11
i
7
c
1
b
11
8
7
h
11
8
8
b
11
8
9
f
d
e
10
9
14
2
f
e
10
9
e
14
f
(b)
10
14
2
c
f
4
e
14
2
c
1
b
7
4
6
9
14
2
c
1
b
7
4
6
d
14
2
c
8
h
h
i
4
6
1
b
11
2
8
h
11
i
7
c
1
b
11
8
7
h
11
8
8
b
635
10
Figure 23.5 The execution of Prim’s algorithm on the graph from Figure 23.1. The root vertex
is a. Shaded edges are in the tree being grown, and black vertices are in the tree. At each step of
the algorithm, the vertices in the tree determine a cut of the graph, and a light edge crossing the cut
is added to the tree. In the second step, for example, the algorithm has a choice of adding either
edge .b; c/ or edge .a; h/ to the tree since both are light edges crossing the cut.
636
Chapter 23 Minimum Spanning Trees
Figure 23.5 shows how Prim’s algorithm works. Lines 1–5 set the key of each
vertex to 1 (except for the root r, whose key is set to 0 so that it will be the
first vertex processed), set the parent of each vertex to NIL, and initialize the minpriority queue Q to contain all the vertices. The algorithm maintains the following
three-part loop invariant:
Prior to each iteration of the while loop of lines 6–11,
1. A D f.#; #:%/ W # 2 V " frg " Qg.
2. The vertices already placed into the minimum spanning tree are those in
V " Q.
3. For all vertices # 2 Q, if #:% ¤ NIL , then #:key < 1 and #:key is
the weight of a light edge .#; #:%/ connecting # to some vertex already
placed into the minimum spanning tree.
Line 7 identifies a vertex u 2 Q incident on a light edge that crosses the cut
.V " Q; Q/ (with the exception of the first iteration, in which u D r due to line 4).
Removing u from the set Q adds it to the set V " Q of vertices in the tree, thus
adding .u; u:%/ to A. The for loop of lines 8–11 updates the key and % attributes
of every vertex # adjacent to u but not in the tree, thereby maintaining the third
part of the loop invariant.
The running time of Prim’s algorithm depends on how we implement the minpriority queue Q. If we implement Q as a binary min-heap (see Chapter 6), we
can use the B UILD -M IN -H EAP procedure to perform lines 1–5 in O.V / time. The
body of the while loop executes jV j times, and since each E XTRACT-M IN operation takes O.lg V / time, the total time for all calls to E XTRACT-M IN is O.V lg V /.
The for loop in lines 8–11 executes O.E/ times altogether, since the sum of the
lengths of all adjacency lists is 2 jEj. Within the for loop, we can implement the
test for membership in Q in line 9 in constant time by keeping a bit for each vertex
that tells whether or not it is in Q, and updating the bit when the vertex is removed
from Q. The assignment in line 11 involves an implicit D ECREASE -K EY operation on the min-heap, which a binary min-heap supports in O.lg V / time. Thus,
the total time for Prim’s algorithm is O.V lg V C E lg V / D O.E lg V /, which is
asymptotically the same as for our implementation of Kruskal’s algorithm.
We can improve the asymptotic running time of Prim’s algorithm by using Fibonacci heaps. Chapter 19 shows that if a Fibonacci heap holds jV j elements, an
E XTRACT-M IN operation takes O.lg V / amortized time and a D ECREASE -K EY
operation (to implement line 11) takes O.1/ amortized time. Therefore, if we use a
Fibonacci heap to implement the min-priority queue Q, the running time of Prim’s
algorithm improves to O.E C V lg V /.