Probability Theory Overview and Analysis of Randomized Algorithms Analysis of Algorithms Prepared by John Reif, Ph.D. Probability Theory Topics a) Random Variables: Binomial and Geometric b) Useful Probabilistic Bounds and Inequalities Readings • Main Reading Selections: – CLR, Chapter 5 and Appendix C Probability Measures • A probability measure (Prob) is a mapping from a set of events to the reals such that: 1) For any event A 0 < Prob(A) < 1 2) Prob (all possible events) = 1 3) If A, B are mutually exclusive events, then Prob(A B) = Prob (A) + Prob (B) Conditional Probability • Define Prob(A B) Prob(A | B) Prob(B) for Prob(B) > 0 Bayes’ Theorem • If A1, …, An are mutually exclusive and contain all events then Prob(A i | B) Pi n P j j=1 where Pj = Prob(B | A j ) Prob(A j ) Random Variable A (Over Real Numbers) • Density Function f A (x) = Prob(A=x) Random Variable A (cont’d) • Prob Distribution Function x FA (x) = Prob(A x) = f A (x) dx - Random Variable A (cont’d) • If for Random Variables A,B x FA (x) FB (x) • Then “A upper bounds B” and “B lower bounds A” FA (x) Prob (A x) FB (x) Prob (B x) Expectation of Random Variable A E(A) = A = x f A (x) dx - • Ā is also called “average of A” and “mean of A” = μA Variance of Random Variable A σ 2A (A A) 2 A 2 (A) 2 where 2nd momoment A 2 x - 2 f A (x) dx Variance of Random Variable A (cont’d) Discrete Random Variable A Discrete Random Variable A (cont’d) Discrete Random Variable A Over Nonnegative Numbers • Expectation E(A) = A = x f A (x) x=0 Pair-Wise Independent Random Variables • A,B independent if Prob(A ∧ B) = Prob(A) * Prob(B) • Equivalent definition of independence f A B (x) = f A (x) f B (x) M A B (s) = M A (s) M B (s) G A B (z) = G A (z) G B (z) Bounding Numbers of Permutations • n! = n * (n-1) * 2 * 1 = number of permutations of n objects • Stirling’s formula n! = f(n) (1+o(1)), where n -n f(n) = n e 2πn Bounding Numbers of Permutations (cont’d) • Note – Tighter bound f(n) e n! (n-k)! 1 (12n+1) < n! < f(n) e 1 12n = number of permutations of n objects taken k at a time Bounding Numbers of Combinations n n! k k! (n-k)! = number of (unordered) combinations of n objects taken k at a time • Bounds (due to Erdos & Spencer, p. 18) k 2 k3 k 2n 6n 2 n n e ~ k! k (1 o(1)) for k = o n 3 4 Bernoulli Variable • Ai is 1 with prob P and 0 with prob 1-P • Binomial Variable • B is sum of n independent Bernoulli variables Ai each with some probability p procedure begin BINOMIAL with parameters n,p B0 for i=1 to n do with probability P do B B+1 output end B is Binomial Variable with Parameters n,p mean n p variance 2 np (1-p) B is Binomial Variable with Parameters n,p (cont’d) n x density fn Prob(B=x) = p (1-p)n-x x n k n-k distribution fn Prob(B x) = p (1-p) k=0 k x Poisson Trial • Ai is 1 with prob Pi and 0 with prob 1-Pi • Suppose B’ is the sum of n independent Poisson trials Ai with probability Pi for i > 1, …, n Hoeffding’s Theorem • B’ is upper bounded by a Binomial Variable B n P i • Parameters n,p where p= i=1 n Geometric Variable V • parameter p x 0 Prob(V=x) = p(1-p)x procedure GEOMETRIC parameter p begin V 0 loop with probability 1-p goto exit Probabilistic Inequalities • For Random Variable A mean A variance A (A) 2 2 2 Markov and Chebychev Probabilistic Inequalities • Markov Inequality Prob (A x) (uses only mean) x • Chebychev Inequality (uses mean and variance) Prob ( A ) 2 2 Example of use of Markov and Chebychev Probabilistic Inequalities • If B is a Binomial with parameters n,p np Then Prob (B x) x np (1-p) Prob ( B np ) 2 Gaussian Density Function (x) = 1 e 2π x2 2 Normal Distribution x (X) = (Y) dY - • Bounds x > 0 (Feller, p. 175) Ψ(x) 1 1 (x) 3 1 (x) x x x x [0,1] x 1 x (1) (x) - (0) = 2 2πe x 2π Sums of Independently Distributed Variables • Let Sn be the sum of n independently distributed variables A1, …, An 2 • Each with mean and variance n n • So Sn has mean μ and variance σ2 Strong Law of Large Numbers: Limiting to Normal Distribution • The probability density function of Tn = (Sn - ) limits as n to normal distribution Φ(x) • Hence Prob ( Sn - x) (x) as n Strong Law of Large Numbers (cont’d) • So Prob ( Sn - x) 2(1 (x)) 2(x)/x (since 1- Φ(x) < Ψ(x)/x) Advanced Material Moment Generating Functions and Chernoff Bounds Moments of Random Variable A (cont’d) • n’th Moments of Random Variable A A n x n f A (x) dx - • Moment generating function M A (s) esx f A (x) dx - = E(esA ) Moments of Random Variable A (cont’d) • Note S is a formal parameter d M A (s) n ds n A n s0 Moments of Discrete Random Variable A • n’th moment A x f A (x) n n x=0 Probability Generating Function of Discrete Random Variable A G A (z) = z f A (x) E(z ) x A x=0 1st derivative G 'A (1) = A 2nd derivative G"A (1) = A 2 A variance σ G (1) G (1) (G (1)) 2 A " A ' A ' A 2 Moments of AND of Independent Random Variables • If A1, …, An independent with same distribution f A1 (x) = f Ai (x) for i = 1, ... n Then if B = A1 A 2 ... A n f B (x) = f A1 (x) n n M B (s) = M A1 (s) , G B (z) = G A1 (z) n Generating Function of Binomial Variable B with Parameters n,p n k n-k G(z) = (1-p+pz) = z p (1-p) k=0 k x n k • Interesting fact 1 Prob(B= ) = n Generating Function of Geometric Variable with parameter p p G(Z) = Z (p(1-p) ) = 1-(1-p)Z k=0 k k Chernoff Bound of Random Variable A • Uses all moments • Uses moment generating function Prob (A x) e -sx =e M A (s) for s 0 γ(s) - sx e where γ(s) = ln (M A (s)) γ(s) - s γ'(s) By setting x = ɣ’ (s) 1st derivative minimizes bounds Chernoff Bound of Discrete Random Variable A Prob (A x) z G A (z) for z 1 -x • Choose z = z0 to minimize above bound • Need Probability Generating function G A (z) = z f A (x) = E(z ) x x 0 A Chernoff Bounds for Binomial B with parameters n,p • Above mean x > μ Prob (B x) n- n-x n-x x x 1 1 e since 1 e x x -x - μ 2 e for x μe x x-μ x Chernoff Bounds for Binomial B with parameters n,p (cont’d) • Below mean x < μ Prob (B x) n- n-x n-x x x Anguin-Valiant’s Bounds for Binomial B with Parameters n,p • Just above mean m = np for 0 < e < 1 Prob (B ³ (1+e )m ) £ e -e 2 m 2 • Just below mean m < np for 0 < e < 1 Prob (B £ ë (1-e )mû ) £ e -e 2 m 3 Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d) Tails are bounded by Normal distributions Binomial Variable B with Parameters p,N and Expectation μ= np • By Chernoff Bound for p < ½ N N Prob (B ) 2 p 2 n 2 ¶¶ • Raghavan-Spencer bound for any ∂ > 0 m æ ö e Prob (B ³ (1+¶)m ) £ ç ÷ (1+¶) (1+¶) è ø ¶ Probability Theory Analysis of Algorithms Prepared by John Reif, Ph.D.