Probability Theory Overview and
Analysis of Randomized Algorithms
Analysis of Algorithms
Prepared by
John Reif, Ph.D.
Probability Theory Topics
a) Random Variables: Binomial and
Geometric
b) Useful Probabilistic Bounds and
Inequalities
Readings
• Main Reading Selections:
– CLR, Chapter 5 and Appendix C
Probability Measures
• A probability measure (Prob) is a
mapping from a set of events to the
reals such that:
1) For any event A
0 < Prob(A) < 1
2) Prob (all possible events) = 1
3) If A, B are mutually exclusive
events, then
Prob(A  B) = Prob (A) + Prob (B)
Conditional Probability
• Define
Prob(A  B)
Prob(A | B) 
Prob(B)
for Prob(B) > 0
Bayes’ Theorem
• If A1, …, An are mutually exclusive and
contain all events then
Prob(A i | B) 
Pi
n
P
j
j=1
where Pj = Prob(B | A j )  Prob(A j )
Random Variable A
(Over Real Numbers)
• Density Function
f A (x) = Prob(A=x)
Random Variable A (cont’d)
• Prob Distribution Function
x
FA (x) = Prob(A  x) =  f A (x) dx
-
Random Variable A (cont’d)
• If for Random Variables A,B
x FA (x)  FB (x)
• Then “A upper bounds B” and
“B lower bounds A”
FA (x)  Prob (A  x)
FB (x)  Prob (B  x)
Expectation of Random Variable A

E(A) = A =  x f A (x) dx
-
• Ā is also called “average of A” and
“mean of A” = μA
Variance of Random Variable A
σ 2A  (A  A) 2  A 2  (A) 2

where 2nd momoment A 
2
x
-
2
f A (x) dx
Variance of Random Variable A
(cont’d)
Discrete Random Variable A
Discrete Random Variable A
(cont’d)
Discrete Random Variable A
Over Nonnegative Numbers
• Expectation

E(A) = A =  x f A (x)
x=0
Pair-Wise Independent Random
Variables
• A,B independent if
Prob(A ∧ B) = Prob(A) * Prob(B)
• Equivalent definition of independence
f A  B (x) = f A (x)  f B (x)
M A  B (s) = M A (s)  M B (s)
G A  B (z) = G A (z)  G B (z)
Bounding Numbers of Permutations
• n! = n * (n-1) * 2 * 1
= number of permutations of n objects
• Stirling’s formula
n! = f(n) (1+o(1)), where
n -n
f(n) = n e
2πn
Bounding Numbers of Permutations
(cont’d)
• Note
– Tighter bound
f(n) e
n!
(n-k)!
1
(12n+1)
< n! < f(n) e
1
12n
= number of permutations of n
objects taken k at a time
Bounding Numbers of Combinations
n
n!
 
 k  k! (n-k)!
= number of (unordered)
combinations of n objects
taken k at a time
• Bounds (due to Erdos & Spencer, p. 18)
k 2 k3


k
2n 6n 2
n n e
 ~
k!
k
(1  o(1))
 
for k = o  n 
 
3
4
Bernoulli Variable
• Ai is 1 with prob P and 0 with prob 1-P
• Binomial Variable
• B is sum of n independent Bernoulli
variables Ai each with some probability p
procedure
begin
BINOMIAL with parameters n,p
B0
for i=1 to n do
with probability P do B  B+1
output
end
B is Binomial Variable with
Parameters n,p
mean   n  p
variance  2  np (1-p)
B is Binomial Variable with
Parameters n,p (cont’d)
n x
density fn  Prob(B=x) =   p (1-p)n-x
x
n k
n-k
distribution fn  Prob(B  x) =    p (1-p)
k=0  k 
x
Poisson Trial
• Ai is 1 with prob Pi
and 0 with prob 1-Pi
• Suppose B’ is the
sum of n independent Poisson trials
Ai with probability Pi for i > 1, …, n
Hoeffding’s Theorem
• B’ is upper bounded by a Binomial
Variable B
n
P
i
• Parameters n,p where
p=
i=1
n
Geometric Variable V
• parameter p
x  0
Prob(V=x) = p(1-p)x
procedure
GEOMETRIC parameter p
begin
V 0
loop with probability 1-p
goto exit
Probabilistic Inequalities
• For Random Variable A
mean   A
variance   A  (A)
2
2
2
Markov and Chebychev
Probabilistic Inequalities
• Markov Inequality
Prob (A  x) 
(uses only mean)

x
• Chebychev Inequality
(uses mean and variance)
Prob ( A    ) 
2
2
Example of use of Markov and
Chebychev Probabilistic Inequalities
• If B is a Binomial with parameters n,p
np
Then Prob (B  x) 
x
np (1-p)
Prob ( B  np  ) 
2

Gaussian Density Function
 (x) =
1
e
2π
x2
2
Normal Distribution
x
 (X) =
 (Y) dY
-
• Bounds x > 0 (Feller, p. 175)
Ψ(x)
1 1 
 (x)   3   1   (x) 
x
x x 
x  [0,1]
x
1
 x  (1)  (x) -   (0) =
2
2πe
x
2π
Sums of Independently Distributed
Variables
• Let Sn be the sum of n independently
distributed variables A1, …, An
2

• Each with mean
and variance
n
n
• So Sn has mean μ and variance σ2
Strong Law of Large Numbers:
Limiting to Normal Distribution
• The probability density function of
Tn =
(Sn - )

limits as n  
to normal distribution Φ(x)
• Hence
Prob (
Sn -   x)  (x) as n  
Strong Law of Large Numbers
(cont’d)
• So
Prob (
Sn -   x)  2(1  (x))
 2(x)/x
(since 1- Φ(x) < Ψ(x)/x)
Advanced Material
Moment Generating Functions and
Chernoff Bounds
Moments of Random Variable A
(cont’d)
• n’th Moments of Random Variable A

A 
n
x
n
f A (x) dx
-
• Moment generating function

M A (s)   esx f A (x) dx
-
= E(esA )
Moments of Random Variable A
(cont’d)
• Note
S is a formal parameter
 d M A (s) 


n
ds


n
A
n
s0
Moments of Discrete Random
Variable A
• n’th moment

A   x f A (x)
n
n
x=0
Probability Generating Function of
Discrete Random Variable A

G A (z) =  z f A (x)  E(z )
x
A
x=0
1st derivative G 'A (1) = A
2nd derivative G"A (1) = A 2  A
variance σ  G (1)  G (1)  (G (1))
2
A
"
A
'
A
'
A
2
Moments of AND of Independent
Random Variables
• If A1, …, An independent with same
distribution
f A1 (x) = f Ai (x) for i = 1, ... n
Then if B = A1  A 2  ...  A n

f B (x) = f A1 (x)


n

n

M B (s) = M A1 (s) , G B (z) = G A1 (z)

n
Generating Function of Binomial
Variable B with Parameters n,p
n k
n-k
G(z) = (1-p+pz) =  z   p (1-p)
k=0
k
x
n
k
• Interesting fact
 1 
Prob(B= ) = 

 n
Generating Function of Geometric
Variable with parameter p

p
G(Z) =  Z (p(1-p) ) =
1-(1-p)Z
k=0
k
k
Chernoff Bound of Random
Variable A
• Uses all moments
• Uses moment generating function
Prob (A  x)  e
-sx
=e
M A (s) for s  0
γ(s) - sx
 e
where γ(s) = ln (M A (s))
γ(s) - s γ'(s)
By setting x = ɣ’ (s)
1st derivative minimizes bounds
Chernoff Bound of Discrete Random
Variable A
Prob (A  x)  z G A (z) for z  1
-x
• Choose z = z0 to minimize above
bound
• Need Probability Generating function
G A (z) =  z f A (x) = E(z )
x
x 0
A
Chernoff Bounds for Binomial B
with parameters n,p
• Above mean x > μ
Prob (B  x)
 n- 


 n-x 
n-x

 
x
x

 1
1
 e   since  1    e
x
 x
-x - μ
2
 e
for x  μe
x
x-μ
x
Chernoff Bounds for Binomial B
with parameters n,p (cont’d)
• Below mean x < μ
Prob (B  x)
 n- 


 n-x 
n-x

 
x
x
Anguin-Valiant’s Bounds for
Binomial B with Parameters n,p
• Just above mean
m = np for 0 < e < 1
Prob (B ³ (1+e )m ) £ e
-e 2
m
2
• Just below mean
m < np for 0 < e < 1
Prob (B £ ë (1-e )mû ) £ e
-e 2
m
3
Anguin-Valiant’s Bounds for
Binomial B with Parameters n,p
(cont’d)
Tails are bounded by Normal
distributions
Binomial Variable B with
Parameters p,N and Expectation μ= np
• By Chernoff
Bound for p < ½
N
N
Prob (B  )  2 p
2
n
2
¶¶
• Raghavan-Spencer bound
for any ∂ > 0
m
æ
ö
e
Prob (B ³ (1+¶)m ) £ ç
÷
(1+¶)
(1+¶)
è
ø
¶
Probability Theory
Analysis of Algorithms
Prepared by
John Reif, Ph.D.