Randomized Algorithms
CS648
Lecture 11
• BPWM : The final algorithm
• Hashing – (part 1)
1
Boolean Product Witness Matrix (BPWM)
Problem: Given two Boolean matrices A and B, and their Boolean product C,
compute a matrix 𝑾, such that:
𝑾𝑖𝑗 stores a witness for each (𝑖, 𝑗) with 𝑪𝑖𝑗 = 1.
Conclusion of last lecture
Theorem:
Given two Boolean matrices 𝑨 and 𝑩, and integer 𝒕,
there is a randomized Monte Carlo algorithm to compute witnesses
for all those pairs which have 𝒕 witnesses.
• The running time is O(𝒏𝝎 𝐥𝐨𝐠 𝒏).
• The error probability is at most
𝟏
𝒏𝒄−𝟐
?
Questions:
1. How to compute witnesses for all pairs in O(𝒏𝝎 𝐥𝐨𝐠 𝟐 𝒏) time.
2. How to convert Monte Carlo to Las Vegas ?
Randomized algorithm for Computing Witnesses for
all pairs with exactly 𝒕 witnesses
//The pseudo code for sampling the (indices of) columns
Sample(𝑛, 𝑝)
{ 𝑅 ∅;
For each 𝑖 ∈ [𝑛] do:
add 𝑖 to 𝑅 with probability 𝑝;
return 𝑅;
}
Randomized algorithm for Computing Witnesses for
all pairs with exactly 𝒕 witnesses
Compute-Witnesses(𝐴,𝐵,𝑡)
{
{ 𝑅 Sample(𝑛, 1/𝑡);
For each 𝑖, 𝑗 ∈ [𝑛]
if 𝑗 ∈ 𝑅 then 𝐴𝑖𝑗  𝑗∙ 𝐴𝑖𝑗 ;
else
𝐴𝑖𝑗 0;
𝐷′  𝐴 ∙ 𝐵;
For each 𝑖, 𝑗 ∈ [𝑛]
′
If 𝐷𝑖𝑗
is a witness for (𝑖, 𝑗)
′
𝑊𝑖𝑗  𝐷𝑖𝑗
}
}
Time complexity: O(𝒏𝝎 )
1
Probability of finding a witness for a single pair (𝑖, 𝑗): ??
>
𝑒
Focus on a single pair (𝑖,𝑗)
Let there be 𝑤 witnesses for (𝑖,𝑗) such that 𝑡 ≤ 𝑤 < 2𝑡
Question: If each column is selected independently with 𝑝 = 1/𝑡 ,
what is the probability that exactly one out of 𝑤 witnesses for (𝑖,𝑗) survives ?
Answer:
=
𝑤 1
1 𝑡
1 𝑤−1
𝑡
1−
1 𝑤
1−𝑡
1 2𝑡
≥
≥ 1−𝑡
1
≥ 𝑒2
= 0.135 …
>
1
8
Randomized algorithm for Computing Witnesses for
all pairs with witness count ∈ [𝒕, 𝟐𝒕)
Compute-Witnesses(𝐴,𝐵,𝑡)
{
{ 𝑅 Sample(𝑛, 1/𝑡);
For each 𝑖, 𝑗 ∈ [𝑛]
if 𝑗 ∈ 𝑅 then 𝐴𝑖𝑗  𝑗∙ 𝐴𝑖𝑗 ;
else
How to reduce the error
𝟏
probability to < 𝒏𝒄 ?
𝐴𝑖𝑗 0;
𝐷′  𝐴 ∙ 𝐵;
For each 𝑖, 𝑗 ∈ [𝑛]
′
If 𝐷𝑖𝑗
is a witness for (𝑖, 𝑗)
Repeat the entire process
𝒄 𝐥𝐨𝐠 8/7 𝒏 times.
′
𝑊𝑖𝑗  𝐷𝑖𝑗
}
}
Time complexity: O(𝒏𝝎 )
1 7
Probability of failing to find a witness for a single pair (𝑖, 𝑗): ??< 1 − <
8 8
Randomized algorithm for Computing Witnesses for
all pairs with witness count ∈ [𝒕, 𝟐𝒕)
Compute-Witnesses(𝐴,𝐵,𝑡)
{ Repeat 𝒄 𝐥𝐨𝐠 𝟖/𝟕 𝒏 times
{ 𝑅 Sample(𝑛, 1/𝑡);
For each 𝑖, 𝑗 ∈ [𝑛]
if 𝑗 ∈ 𝑅 then 𝐴𝑖𝑗  𝑗∙ 𝐴𝑖𝑗 ;
else
𝐴𝑖𝑗 0;
𝐷′  𝐴 ∙ 𝐵;
For each 𝑖, 𝑗 ∈ [𝑛]
′
If 𝐷𝑖𝑗
is a witness for (𝑖, 𝑗)
′
𝑊𝑖𝑗  𝐷𝑖𝑗
}
}
Time complexity: O(𝒏𝝎 𝐥𝐨𝐠 𝒏)
𝟏
< 𝒄
Probability of failing to find a witness for a single pair (𝑖, 𝑗): ??
𝒏
Randomized algorithm for Computing Witnesses for
all pairs with witness count ∈ [𝒕, 𝟐𝒕)
Compute-Witnesses(𝐴,𝐵,𝑡)
{ Repeat 𝒄 𝐥𝐨𝐠 𝟖/𝟕 𝒏 times
{ 𝑅 Sample(𝑛, 1/𝑡);
For each 𝑖, 𝑗 ∈ [𝑛]
if 𝑗 ∈ 𝑅 then 𝐴𝑖𝑗  𝑗∙ 𝐴𝑖𝑗 ;
else
How to compute
witnesses for all pairs
in O(𝒏𝝎 𝐥𝐨𝐠 𝟐 𝒏) time ?
𝐴𝑖𝑗 0;
𝐷′  𝐴 ∙ 𝐵;
For each 𝑖, 𝑗 ∈ [𝑛]
′
If 𝐷𝑖𝑗
is a witness for (𝑖, 𝑗)
Let there be 𝑚 pairs that
have exactly 𝑡 witnesses.
Apply Union theorem …
′
𝑊𝑖𝑗  𝐷𝑖𝑗
}
}
Time complexity: O(𝒏𝝎 𝐥𝐨𝐠 𝒏)
𝒎
Prob. of failing to find a witness for any pair having witness count ∈ [𝒕, 𝟐𝒕) : ??
< 𝒏𝒄 <
𝟏
𝒏𝒄−𝟐
Randomized algorithm for Computing Witnesses for
all pairs
Compute-Witnesses(𝐴,𝐵)
{ For 𝑘 = 1 to 𝐥𝐨𝐠 𝟐 𝒏
{ Repeat 𝒄 𝐥𝐨𝐠 𝟖/𝟕 𝒏 times
{ 𝑅 Sample(𝑛, 1/2𝑘 );
For each 𝑖, 𝑗 ∈ [𝑛]
if 𝑗 ∈ 𝑅 then 𝐴𝑖𝑗  𝑗∙ 𝐴𝑖𝑗 ;
else
𝐴𝑖𝑗 0;
𝐷′  𝐴 ∙ 𝐵;
For each 𝑖, 𝑗 ∈ [𝑛]
′
If 𝐷𝑖𝑗
is a witness for (𝑖, 𝑗)
′
𝑊𝑖𝑗  𝐷𝑖𝑗
}
}
}
Time complexity: O(𝒏𝝎 𝐥𝐨𝐠 𝟐 𝒏)
Question:
What is Prob. that witness is not found for a
given pair (𝑖,𝑗) ?
Answer: ? < 𝟏
𝒄
𝒏
Question:
What is Prob. that witness is not found for at
least one pair ?
Answer: < 𝟏
𝒏𝒄−𝟐
How to
If there is transform
even a single
to Laspair with
nonzero witnesses
Vegas but
algowe
? fail to find
even one, run the O(𝒏𝟑 ) algo.
Expected running time = O(𝒏𝝎 𝐥𝐨𝐠 𝟐 𝒏)
Boolean Product Witness Matrix (BPWM)
Theorem: Given two Boolean matrices A and B, and their Boolean product C,
there exists a Las Vegas algorithm for computing a matrix 𝑾, such that:
𝑾𝑖𝑗 stores a witness for each (𝑖, 𝑗) with 𝑪𝑖𝑗 = 1.
The expected running time of the algorithm is O(𝒏𝝎 𝐥𝐨𝐠 𝟐 𝒏).
Homework:
Show that the running time of the algo is concentrated around O(𝒏𝝎 𝐥𝐨𝐠 𝟐 𝒏).
(modify the algorithm if needed)
Hashing
(part 1)
Problem Definition
• 𝑼 = 1,2, … , 𝑚 called universe
• 𝑺 ⊆ 𝑼 and 𝑠 = |𝑺|
• 𝑠≪𝑚
Examples:
𝑚 = 1018 , 𝑠 = 103
Aim
Maintain a data structure for storing 𝑺 to support the search query :
“Does 𝑖 ∈ 𝑺 ?” for any given 𝑖 ∈ 𝑼.
Solutions
Solutions with worst case guarantees
Solution for static 𝑺 :
• Array storing 𝑺 in sorted order
Solution for dynamic 𝑺 :
• Height Balanced Search trees (AVL trees, Red-Black trees,…)
Time per operation: O(log 𝑠),
Space: O(𝑠)
Alternative:
Time per operation: O(1),
Space: O(𝑚)
Solutions used in practice with no worst case guarantees
Hashing.
Hashing
• Hash table:
𝑻: an array of size 𝒏.
• Hash function
𝒉 : 𝑼 [𝒏]
Answering a Query: “Does 𝑖 ∈ 𝑺 ?”
1. 𝑘𝒉(𝑖);
2. Search the list stored at 𝑻[𝑘].
Properties of 𝒉 :
• 𝒉 𝑖 computable in O(1) time.
• Space required by 𝒉: O(1).
How many words
needed to encode 𝒉 ?
𝑻
0
1
⋮
⋮
𝒏−𝟏
Elements of 𝑺
Collision
Definition: Two elements 𝑖, 𝑗 ∈ 𝑼 are
said to collide under hash function 𝒉 if
𝒉 𝑖 =𝒉 𝑗
Worst case time complexity of searching
an item 𝑖 :
No. of elements in 𝑺 colliding with 𝑖.
A Discouraging fact:
No hash function can be found
which is good for all 𝑺.
Proof:
At least 𝑚/𝑛 elements from 𝑼 are
mapped to a single index in 𝑻.
𝑻
0
1
⋮
⋮
𝒏−𝟏
Collision
Definition: Two elements 𝑖, 𝑗 ∈ 𝑼 are
said to collide under hash function 𝒉 if
𝒉 𝑖 =𝒉 𝑗
Worst case time complexity of searching
an item 𝑖 :
No. of elements in 𝑺 colliding with 𝑖.
A Discouraging fact:
No hash function can be found
which is good for all 𝑺.
Proof:
At least 𝑚/𝑛 elements from 𝑼 are
mapped to a single index in 𝑻.
𝑻
0
1
⋮
⋯
⋮
𝒏−𝟏
𝑚/𝑛
Hashing
• A very popular heuristic since 1950’s
• Achieves O(1) search time in practice
• Worst case guarantee on search time: O(𝒔)
Question: Can we have a hashing ensuring
• O(1) worst case guarantee on search time.
• O(𝒔) space.
• Expected O(𝒔) preprocessing time.
The following result gave an answer in affirmative
Michael Fredman, Janos Komlos, Endre Szemeredy. Storing a Sparse Table
with O(1) Worst Case Access Time. Journal of the ACM (Volume 31, Issue 3),
1984.
WHY DOES HASHING WORK SO
WELL IN PRACTICE ?
Why does hashing work so well in Practice ?
Question: What is the simplest hash function 𝒉 : 𝑼 [𝒏] ?
Answer: 𝒉 𝑖 = 𝑖 𝐦𝐨𝐝 𝑛
Hashing works so well in practice because the set 𝑺 is usually a uniformly
random subset of 𝑼.
Let us give a theoretical reasoning for this fact.
Why does hashing work so well in Practice ?
Let 𝑦1 , 𝑦2 , … , 𝑦𝑠 denote 𝑠 elements
selected randomly uniformly from 𝑼 to
form 𝑺.
Question:
What is expected number of elements
colliding with 𝑦1 ?
Answer: Let 𝑦1 takes value 𝑖.
P(𝑦𝑗 collides with 𝑦1 ) = ??
1
2
⋮
𝑖−𝑛
𝑖
𝑖+𝑛
How many possible values
can 𝑦𝑗 take ? 𝑚 − 1
How many possible values
can collide with 𝑖 ?
𝑖 + 2𝑛
𝑖 + 3𝑛
⋮
m
Why does hashing work so well in Practice ?
Let 𝑦1 , 𝑦2 , … , 𝑦𝑠 denote 𝑠 elements
selected randomly uniformly from 𝑼 to
form 𝑺.
Question:
What is expected number of elements
colliding with 𝑦1 ?
Answer: Let 𝑦1 takes value 𝑖.
P(𝑦𝑗 collides with 𝑦1 ) =
𝑚
𝑛
1
2
⋮
𝑖−𝑛
𝑖
𝑖+𝑛
−1
𝑚−1
𝑖 + 2𝑛
Expected number of elements of 𝑺
colliding with 𝑦1 =
=
𝑚
𝑛
−1
𝑚−1
𝑖 + 3𝑛
⋮
(𝑠 − 1)
= 𝑂 1 for 𝑛 = 𝐎(𝑠)
m
Values which may
collide with 𝑖
under the hash
function 𝒉 𝑥 =
𝒙 𝐦𝐨𝐝 𝑛
Why does hashing work so well in Practice ?
Conclusion
1.
𝒉 𝑖 = 𝑖 𝐦𝐨𝐝 𝑛 works so well because
for a uniformly random subset of 𝑼,
the expected number of collision at an index of 𝑻 is O(1).
It is easy to fool this hash function such that it achieves O(s) search time.
(do it as a simple exercise).
This makes us think:
“How can we achieve worst case O(1) search time for a given set 𝑺.”
HOW TO ACHIEVE WORST CASE
O(1) SEARCH TIME
Key idea to achieve worst case O(1) search time
Observation: Of course, no single hash function is good for every possible 𝑺.
But we may strive for a hash function which is good for a given 𝑺.
A promising direction:
Find out a family of hash functions H such that
• For any given 𝑺, many of them are good.
The notion of goodness is captured
formally by Universal hash family in
the following slide.
• Select a function randomly from H and try for 𝑺.
UNIVERSAL HASH FAMILY
Universal Hash Family
Definition: A collection 𝑯 of hash-functions is said to be c-universal
if there exists a constant 𝑐 such that for any 𝑖, 𝑗 ∈ 𝑼,
𝐏𝒉∈𝑟 𝑯 𝒉 𝑖 = 𝒉 𝑗
≤
𝑐
𝑛
Question: Does there exist a Universal hash family whose hash functions have a compact
encoding?
Answer:
Yes and it is very simple too 
STATIC HASHING
WORST CASE O(1) SEARCH TIME
The Journey
One Milestone in Our Journey:
• A perfect hash function using hash table of size O(𝑠 2 )
Tools Needed:
• Universal Hash Family where 𝑐 is a small constant
• Elementary Probability
Spend at least 30 minutes today to
see how the milestone can be
achieved. It is simple and beautiful 