Expected Running Times
and Randomized Algorithms
Instructor
Neelima Gupta
ngupta@cs.du.ac.in
Expected Running Time of
Insertion Sort
(at rth position)
x1,x2,........., xi-1,xi,.......…,xn
For I = 2 to n
Insert the ith element xi in the partially
sorted list x1,x2,........., xi-1.
Expected Running Time of
Insertion Sort
• Let Xi be the random variable which represents
the number of comparisons required to insert ith
element of the input array in the sorted sub array
of first i-1 elements.
• Xi : xi1,xi2,..................…,xii
E(Xi) = Σj xijp(xij )
where E(Xi) is the expected value Xi
And, p(xij) is the probability of inserting xi in
the jth position 1≤j≤i
Expected Running Time of
Insertion Sort
(at jth position)
x1,x2,........., xi-1,xi,.......…,xn
How many comparisons it makes to insert
ith element in jth position?
• Position
i
i-1
i-2
.
.
.
2
1
# of Comparisions
1
2
3
.
.
.
i-1
i-1
Note: Here, both position 2 and 1 have # of Comparisions equal to i-1. Why?
Because to insert element at position 2 we have to compare with previously
first element. and after that comparison we know which of them come first
and which at second.
Thus, E(Xi) = (1/i)
{ i-1Σk=1k + (i-1) }
where 1/i is the probability to insert at jth
position in the i possible positions.
For n elements,
E(X1 + X2 + .............+Xn)
= nΣi=2 E(Xi)
= nΣ
i=2 (1/i)
{ i-1Σk=1k + (i-1)
}
= (n-1)(n-4)/4
For n number of elements, expected time
taken is,
T = nΣi=2 (1/i)
{ i-1Σk=1k + (i-1) }
where 1/i is the probability to insert at rth
position in the i possible positions.
E(X1 + X2 + .............+Xn) = nΣi=1 E(Xi)
Where,Xi is expected value of inserting Xi element.
T = (n-1)(n-4)/4
Therefore average case of insertion sort takes
Θ(n2)
Quick-Sort
• Pick the first item from the array--call it the pivot
• Partition the items in the array around the pivot so all
elements to the left are  to the pivot and all elements
to the right are greater than the pivot
• Use recursion to sort the two partitions
partition 1: items  pivot
pivot
partition: items > pivot
Quicksort: Expected number of
comparisons
• Partition may generate splits
(0:n-1, 1:n-2, 2:n-3, … , n-2:1, n-1:0)
each with probability 1/n
• If T(n) is the expected running time,
n1
1
T n  T k  T n 1 k  n
n k 0
Randomized Quick-Sort
• Pick an element from the array--call it the pivot
• Partition the items in the array around the pivot so all
elements to the left are  to the pivot and all elements
to the right are greater than the pivot
• Use recursion to sort the two partitions
partition 1: items  pivot
pivot
partition: items > pivot
Remarks
• Not much different from the Q-sort except
that earlier, the algorithm was
deterministic and the bounds were
probabilistic.
• Here the algorithm is also randomized. We
pick an element to be a pivot randomly.
Notice that there isn’t any difference as to
how does the algorithm behave there
onwards?
• In the earlier case, we can identify the
Randomized Select
1 n1
T n    max{T k , T n  1  k }  n 
n k 0
Randomized Algorithms
• A randomized algorithm performs coin tosses (i.e.,
uses random bits) to control its execution
• b ← random()
if b = 0
do A …
else { i.e. b = 1}
do B …
• Its running time depends on the outcomes of the coin
tosses
Assumptions
• 􀂄 the coins are unbiased, and
• 􀂄 the coin tosses are independent
• The worst-case running time of a randomized
algorithm may be large but occurs with very low
probability (e.g., it occurs when all the coin
tosses give “heads”)
Monte Carlo Algorithms
• Running times are guaranteed but the
output may not be completely correct.
• Probability of error is low.
Las Vegas Algorithms
• Output is guaranteed to be correct.
• Bounds on running times hold with high
probability.
• What type of algorithm is Randomized
Qsort?
Why expected running times?
• Markov’s inequality
P( X > k E(X)) < 1/k
i.e. the probability that the algorithm will take
more than O(2 E(X)) time is less than 1/2.
Or the probability that the algorithm will take
more than O(10 E(X)) time is less than
1/10.
This is the reason why Qsort does well in
practice.
Markov’s Bound
P(X<kM)< 1/k ,
where k is a constant.
Chernouff’s Bound
P(X>2μ)< ½
A More Stronger Result
P(X>k μ )< 1/nk,
where k is a constant.
RANDOMLY BUILT BST
Binary search tree can be built randomly.
x
<
>
Rank(x)=i
Randomly selected key becomes the root.
Pivot element=root
HEIGHT OF THE TREE
• Xi : the height of the tree rooted at a node
with rank=i.
• Yi : exponential height of the tree=2^Xi
• H=max{H1,H2} + 1
where H1 : ht. of left subtree
H2 : ht.of right subtree
H : ht. of the tree rooted at x.
• Y=2^H
=2.max{2^H1,2^H2}
• Expected value of exponential ht. of the
tree with ‘n’ nodes:
=E(EH(T(X)))
=2/n ∑ max{EH(T(k)),EH(T(n-1-k))}
=O(n^3)=E(H(T(n)))=O(log n)
Skip List : Dictionary as ADT
Skip list is a data structure that can be used to maintain dictionary.
Given n keys ,we insert these n keys in a linked list that has -∞ as first node and ∞ as
last node .
Initial list S0
Then we flip coin a coin for each element until only one is left in Si ,if a tail occurs,we
insert it into next list Si+1 and so on.
-∞
5
9
25
30
35
38
40
∞
∞
S3
-∞
S2
∞
-∞
S1
S0
-∞
-∞
∞
30
5
9
25
head
30
Tail
35
38
40
∞
Operations that can be performed on skip list
Each node has two pointers right and down
1. right
Drop down
• This operation
is performed when after(p)>key.
down
• In this operation, pointer p moves down to immediate lower
level list.
p
S1
-∞
S0
-∞
9
30 (after drop down)
∞
30
∞
38
2. Scan forward
• This operation is performed when after(p)<key.
• Here the pointer p moves to the next element in the list.
• eg. here key=28 & p is at 9 after(9)<28, so scan forward
p p
S0
-∞
9
25
p
30
∞
Searching a key k
Keep a ptr p to the first node in the highest list Sh
while (after(p)>k)
if (Scur==S0) // Scur is the current skip list
then “key k not found”;
exit
if (after(p)>k)
drop down to next skip list
If (after(p)<k)
.
scan forward i.e. update pafter(p)
if (after(p)==k)
return after(p);
Searching for a key 25
S3
p
∞
-∞
S2
p
S1
-∞
∞
30
S0
p
p
-∞
9
30
∞
38
p
-∞
5
9
p
25
Key found
30
35
38
40
∞
Searching for a key 28
S3
p
∞
-∞
S2
p
S1
-∞
∞
30
S0
p
p
-∞
9
30
∞
38
p
p
-∞
5
9
p
25
Key not found
30
35
38
40
∞
Deletion of a key
eg .delete 30
S3
p
∞
-∞
S2
p
p
-∞
∞
30
S1
p
S0
-∞
9
30
∞
38
p
-∞
5
9
25
30
35
38
40
∞
Analysis
1. An element xk is in Si with probability 1/2i , true ∀ elements .
n
E(Si ) = ∑ 1/2i .Xk,i
where Xk,i = 1 if xk is in Si,
k=1
0, otherwise.
= n/2i
E(total size) = E(∑ ISi I)
∞
= ∑ n/2i ≤ 2n
k=1
2. Expected height of a skip list,h = log n
n/2h =1
⇛
h ≃ log n
Analysis(contd.)
3.
Drop down
O(log n)
Since pointer p can drop atmost h times
i.e.height of the skip list until S0 is reached
and h = logn .
4.
# of elements
Scan forward
O(log n)
Total no. of levels
Total Cost
to scan at each level
O(1)
O(log n )
O(log n )
The number of elements scanned at ith level is no more than 2 because
The key lies between p and after(p) on the (i+1)th level (that’s why we came
down to ith level). And, there is only one element between p and after(p) of
(i+1)th level in Si : the element pointed to by after(p) in Si.
Thus we scan at most two elements at Si the element pointed to by p (when
we came down) and after(p) in Si.
Hashing
• Motivation: symbol tables
– A compiler uses a symbol table to relate
symbols to associated data
• Symbols: variable names, procedure names, etc.
• Associated data: memory location, call graph, etc.
– For a symbol table (also called a dictionary),
we care about search, insertion, and deletion
– We typically don’t care about sorted order
Hash Tables
• More formally:
– Given a table T and a record x, with key (= symbol)
and satellite data, we need to support:
• Insert (T, x)
• Delete (T, x)
• Search(T, x)
– We want these to be fast, but don’t care about sorting
the records
• The structure we will use is a hash table
– Supports all the above in O(1) expected time!
Hash Functions
• Next problem: collision
U
(universe of keys)
k2
0
h(k1)
h(k4)
k1
k4
K
(actual
keys)
T
k5
h(k2) = h(k5)
k3
h(k3)
m-1
Resolving Collisions
• How can we solve the problem of
collisions?
• One of the solution is : chaining
• Other solutions: open addressing
Chaining
• Chaining puts elements that hash to the
same slot in a linked list:
T
U
(universe of keys)
k4
K
k5
(actual
k7
keys)
k6
k8
k1
k4 ——
k5
k2
——
——
——
k1
k2
——
——
k3
k3 ——
k8
——
k6 ——
k7 ——
Chaining
• How do we insert an element?
U
(universe of keys)
k4
K
k5
(actual
k7
keys)
k6
k8
k1
k4 ——
k5
k2
——
——
——
k1
k2
T
——
——
k3
k3 ——
k8
——
k6 ——
k7 ——
Chaining
• How do we delete an element?
U
(universe of keys)
k4
K
k5
(actual
k7
keys)
k6
k8
k1
k4 ——
k5
k2
——
——
——
k1
k2
T
——
——
k3
k3 ——
k8
——
k6 ——
k7 ——
Chaining
• How do we search for a element with a
given key?
T
U
(universe of keys)
k4
K
k5
(actual
k7
keys)
k6
k8
k1
k4 ——
k5
k2
——
——
——
k1
k2
——
——
k3
k3 ——
k8
——
k6 ——
k7 ——
Analysis of Chaining
• Assume simple uniform hashing: each key
in table is equally likely to be hashed to
any slot
• Given n keys and m slots in the table: the
load factor  = n/m = average # keys per
slot
• What will be the average cost of an
unsuccessful search for a key?
Analysis of Chaining
• Assume simple uniform hashing: each key
in table is equally likely to be hashed to
any slot
• Given n keys and m slots in the table, the
load factor  = n/m = average # keys per
slot
• What will be the average cost of an
unsuccessful search for a key? A:
O(1+)
Analysis of Chaining
• Assume simple uniform hashing: each key in
table is equally likely to be hashed to any slot
• Given n keys and m slots in the table, the
load factor  = n/m = average # keys per slot
• What will be the average cost of an
unsuccessful search for a key? A: O(1+)
• What will be the average cost of a successful
search?
Analysis of Chaining
• Assume simple uniform hashing: each key in
table is equally likely to be hashed to any slot
• Given n keys and m slots in the table, the
load factor  = n/m = average # keys per slot
• What will be the average cost of an
unsuccessful search for a key? A: O(1+)
• What will be the average cost of a successful
search? A: O((1 + )/2) = O(1 + )
Analysis of Chaining Continued
• So the cost of searching = O(1 + )
• If the number of keys n is proportional to
the number of slots in the table, what is ?
• A:  = O(1)
– In other words, we can make the expected
cost of searching constant if we make 
constant
A Final Word About Randomized Algorithms
If we could prove this,
P(failure)<1/k
(we are sort of happy)
P(failure)<1/nk
(most of times this is
true and we’re happy )
P(failure)<1/2n
(this is difficult but still
we want this)
Acknowledgements
• Kunal Verma
• Nidhi Aggarwal
• And other students of MSc(CS) batch
2009.
END