Markov’s & Chebyshev’s Inequalities
Markov’s Inequality
For any random variable X ≥ 0 and constant a > 0, then
Lecture 7: Chebyshev’s Inequality, LLN, and the CLT
P(X ≥ a) ≤
E (X )
a
Corollary - Chebyshev’s Inequality:
Sta 111
Colin Rundel
P(|X − E (X )| ≥ a) ≤
May 22, 2014
Var (X )
a2
“The inequality says that the probability that X is far away from its mean
is bounded by a quantity that increases as Var (X ) increases.”
Sta 111 (Colin Rundel)
Markov’s & Chebyshev’s Inequalities
Lecture 7
May 22, 2014
1 / 28
Markov’s & Chebyshev’s Inequalities
Derivation of Markov’s Inequality
Derivation of Chebyshev’s Inequality
Let X be a random variable such that X ≥ 0 then
Proposition - if f (x) is a non-decreasing function then
P(X ≥ a) = P f (X ) ≥ f (a) .
Therefore,
E f (X )
P(X ≥ a) ≤
.
f (a)
If we define the positive valued random variable to be |X − E (X )| and
f (x) = x 2 then
2
P |X − E (X )| ≥ a = P (X − E (X )) ≥ a
2
E (X − E (X ))2
Var (X )
≤
=
a2
a2
Var (X )
P |X − E (X )| ≥ a ≤
a2
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
2 / 28
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
3 / 28
Markov’s & Chebyshev’s Inequalities
Markov’s & Chebyshev’s Inequalities
Using Markov’s and Chebyshev’s Inequalities
Chebyshev’s Inequality - Example
Suppose that it is known that the number of items produced in a factory during a week
is a random variable X with mean 50. (Note that we don’t know anything about the
distribution of the pmf)
Lets use Chebyshev’s inequality to make a statement about the bounds for
the probability of being with in 1, 2, or 3 standard deviations of the mean
for all random variables.
p
If we define a = kσ where σ = Var (X ) then
(a) What can be said about the probability that this week’s production will exceed 75
units?
(b) If the variance of a week’s production is known to equal 25, then what can be said
about the probability that this week’s production will be between 40 and 60 units?
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
4 / 28
P(|X − E (X )| ≥ kσ) ≤
Sta 111 (Colin Rundel)
Markov’s & Chebyshev’s Inequalities
X −µ
σ
Lecture 7
May 22, 2014
5 / 28
Law of Large Numbers
Accuracy of Chebyshev’s inequality, cont.
If X ∼ N (µ, σ 2 ) then let Z =
Var (X )
1
= 2
2
2
k σ
k
Independent and Identically Distributed (iid)
∼ N (0, 1).
A collection of random variables that share the same probability
distribution and all are mutually independent.
Empirical Rule:
Chebyshev’s inequality:
1 − P(−1 ≤ Z ≤ 1) ≈ 1 − 0.66 = 0.34
1 − P(−1 ≤ U ≤ 1) ≤
1
11
1 − P(−2 ≤ Z ≤ 2) ≈ 1 − 0.96 = 0.04
1 − P(−2 ≤ U ≤ 2) ≤
1
22
= 0.25
1 − P(−3 ≤ Z ≤ 3) ≈ 1 − 0.997 = 0.003
1 − P(−3 ≤ U ≤ 3) ≤
1
32
= 0.11
Example
=1
If X ∼ Binom(n, p) then X =
Pn
i=1 Yi
iid
where Y1 , · · · , Yn ∼ Bern(p)
Why do we care if it is so inaccurate?
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
6 / 28
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
7 / 28
Law of Large Numbers
Law of Large Numbers
Sums of iid Random Variables
Average of iid Random Variables
iid
iid
Let X1 , X2 , · · · , Xn ∼ D where D is some probability distribution with
E (Xi ) = µ and Var (Xi ) = σ 2 .
Let X1 , X2 , · · · , Xn ∼ D where D is some probability distribution with
E (Xi ) = µ and Var (Xi ) = σ 2 .
We defined Sn = X1 + X2 + · · · + Xn
We defined X n = (X1 + X2 + · · · + Xn )/n = Sn /n then
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
8 / 28
Sta 111 (Colin Rundel)
Law of Large Numbers
Lecture 7
May 22, 2014
9 / 28
Law of Large Numbers
Weak Law of Large Numbers
Law of Large Numbers
Based on these results and Markov’s Inequality we can show the following:
Weak Law of Large Numbers (X̄n converges in probability to µ):
lim P(|X̄n − µ| > ) = 0
n→∞
Strong Law of Large Numbers (X̄n converges almost surely to µ):
P lim X̄n = µ = 1
n→∞
Strong LLN is a more powerful result (Strong LLN implies Weak LLN),
but its proof is more complicated.
Therefore, as long as σ 2 < ∞
lim P(|X̄n − µ| ≥ ) = 0
n→∞
Sta 111 (Colin Rundel)
⇒
Lecture 7
lim P(|X̄n − µ| < ) = 1
n→∞
May 22, 2014
10 / 28
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
11 / 28
Law of Large Numbers
Law of Large Numbers
LLN - Example
LLN and CLT
How large a random sample must be taken from a given distribution in
order for the probability to be at least 0.99 that the sample mean will be
within 2 standard deviations of the mean of the distribution?
Law of large numbers shows us that
Sn − nµ
= lim (X̄n − µ) → 0
n→∞
n→∞
n
lim
which shows that for large n, n S̄n − nµ.
What happens if we divide by something that grows slower than n like
What about 0.95 probability to be within 1 standard deviations of the
mean?
√
n?
√
Sn − nµ
d
√
= lim n(X̄n − µ) → N(0, σ 2 )
n→∞
n→∞
n
lim
This is the Central Limit Theorem, of which the DeMoivre-Laplace
theorem for the normal approximation to the binomial is a special case.
Hopefully by the end of this class we will have the tools to prove this.
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
12 / 28
Sta 111 (Colin Rundel)
Lecture 7
Law of Large Numbers
May 22, 2014
13 / 28
Law of Large Numbers
Equivalence of de Moivre-Laplace with CLT
Equivalence of de Moivre-Laplace with CLT, cont.
We have already seen that a Binomial random variable is equivalent to the
sum of n iid Bernoulli random variables.
P
iid
Let X ∼ Binom(n,p) where X = ni=1 Yi with Y1 , · · · , Yn ∼ Bern(p) and
E (Yi ) = p, Var (Yi ) = p(1 − p).
The Central Limit theorem gives us,
Sn − nµ
√
P c≤
≤ d ≈ Φ(d) − Φ(c)
σ n
P
then for X = Sn = Yi
de Moivre-Laplace tells us:
P(a ≤ X ≤ b) = P
≈Φ
X − np
b − np
a − np
p
≤p
≤p
np(1 − p)
np(1 − p)
np(1 − p)
!
!
b − np
a − np
p
−Φ p
np(1 − p)
np(1 − p)
P(a ≤ X ≤ b) = P
!
=P
≈Φ
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
14 / 28
Sta 111 (Colin Rundel)
a − nµ
X − nµ
b − nµ
√
≤ √
≤ √
nσ
nσ
nσ
a − np
Sn − np
b − np
p
≤p
≤p
np(1 − p)
np(1 − p)
np(1 − p)
!
!
b − np
a − np
p
−Φ p
np(1 − p)
np(1 − p)
Lecture 7
May 22, 2014
!
15 / 28
Law of Large Numbers
Law of Large Numbers
Why do we care? (Statistics)
Astronomy Example
In general we are interested in making statements about how the world
works, we usually do this based on empirical evidence. This tends to
involve observing or measuring something a bunch of times.
An astronomer is interested in measuring, in light years, the distance from his
observatory to a distant star. Although the astronomer has a measuring
technique, he knows that, because of changing atmospheric conditions and
measurement error, each time a measurement is made it will not yield the exact
distance but merely an estimate. As a result the astronomer plans to make a
series of measurements and then use the average value of these measurements as
his estimated value of the actual distance.
LLN tells us that if we average our results we’ll get closer and closer
to the true value as we take more measurements
CLT tells us the distribution of our average is normal if we take
enough measurements, which tells us our uncertainty about the ‘true’
value
Example - Polling: I can’t interview everyone about their opinions but if I
interview enough my estimate should be close and I can use the CLT to
approximate my margin of error.
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
16 / 28
If the astronomer believes that the values of the measurements are independent
and identically distributed random variables having a common mean d (the actual
distance) and a common variance of 4 (light years2 ), how many measurements
should he make to be reasonably sure that his estimated distance is accurate to
within ±0.5 light years?
Sta 111 (Colin Rundel)
Law of Large Numbers
Lecture 7
May 22, 2014
17 / 28
Law of Large Numbers
Astronomy Example, cont.
Astronomy Example, cont.
We need to decide on a level to account for being “reasonably sure”,
usually taken to be 95% but the choice somewhat arbitrary.
Our previous estimate depends on how long it takes for the distribution of
the average of the measurements to converge to the normal distribution.
CLT only guarantees convergence as n → ∞, but most distributions
converge much more quickly (think about the np ≥ 10, np ≥ 10
requirement for Normal approximation to Binomial).
We can also solve this problem using Chebyshev’s inequality
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
18 / 28
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
19 / 28
Central Limit Theorem
Central Limit Theorem
Moment Generating Function - Properties
Moment Generating Function - Unit Normal
If X and Y are independent random variables then the moment generating
function for the distribution of X + Y is
Let Z ∼ N (0, 1) then
Z ∞
2
1
MZ (t) = E [e tZ ] =
e tx √ e −x /2 dx
2π
−∞
Z ∞
2
x −tx
1
=√
e − 2 dx
2π −∞
Z ∞
(x−t)2
t2
1
=√
e − 2 + 2 dx
2π −∞
Z ∞
(x−t)2
1
t 2 /2
√ e − 2 dx
=e
2π
−∞
MX +Y (t) = E [e t(X +Y ) ] = E [e tX e tY ] = E [e tX ]E [e tY ] = MX (t)MY (t)
Similarly, the moment generating function for Sn , the sum of iid random
variables X1 , X2 , . . . , Xn is
MSn (t) = [MXi (t)]n
Sta 111 (Colin Rundel)
Lecture 7
= et
May 22, 2014
20 / 28
Sta 111 (Colin Rundel)
Central Limit Theorem
Sketch of Proof
Let X1 , . . . , Xn be a sequence of independent and identically distributed
random variables each having mean µ and variance σ 2 . Then the
distribution of
Proposition
X1 + · · · + Xn − nµ
√
σ n
Sta 111 (Colin Rundel)
May 22, 2014
21 / 28
Let X1 , X2 , . . . be a sequence of independent and identically distributed random
variables and Sn = X1 + · · · + Xn . The distribution of Sn is given by the distribution
function fSn which has a moment generating function MSn with n ≥ 1.
If MSn (t) → MZ (t) for all t, then fSn (t) → fZ (t) for all t at which fZ (t) is
continuous.
That is, for −∞ < a < ∞,
P
Lecture 7
Let Z being a random variable with distribution function fZ and moment generating function MZ .
tends to the unit normal as n → ∞.
X1 + · · · + Xn − nµ
√
≤a
σ n
/2
Central Limit Theorem
Central Limit Theorem
2
1
→√
2π
Lecture 7
Z
a
e
−x 2 /2
dx = Φ(a) as n → ∞
−∞
May 22, 2014
22 / 28
We can prove the CLT by letting Z ∼ N (0, 1), MZ (t) = e t
2
showing for any Sn that MSn /√n → e t /2 as n → ∞.
Sta 111 (Colin Rundel)
Lecture 7
2 /2
and then
May 22, 2014
23 / 28
Central Limit Theorem
Central Limit Theorem
Proof of the CLT
Proof of the CLT, cont.
√
The moment generating function of Xi / n is given by
tX
t
i
MXi /√n (t) = E exp √
= MXi √
n
n
Some simplifying assumptions and notation:
E (Xi ) = 0
Var (Xi ) = 1
P
√
√
and this the moment generating function of Sn / n = ni=1 Xi / n is
given by
n
t
MSn /√n (t) = MXi √
n
MXi (t) exists and is finite
LX (t) = log MX (t)
Also, remember L’Hospital’s Rule:
f (x)
f 0 (x)
= lim 0
x→∞ g (x)
x→∞ g (x)
lim
Sta 111 (Colin Rundel)
Lecture 7
Therefore in order to show MSn /√n → MZ (t) we need to show
n
t
2
MXi √
→ e t /2
n
May 22, 2014
24 / 28
Sta 111 (Colin Rundel)
Central Limit Theorem
√
2
[MXi (t/ n)]n → e t /2
√
nLXi (t/ n) → t 2 /2
LXi (t) = log MXi (t)
MX0 (t)
d
i
log MXi (t) =
dt
MXi (t)
MX0 (0)
µ
i
=
=0
L0Xi (0) =
MXi (0)
1
L00
Xi (0) =
=
Sta 111 (Colin Rundel)
25 / 28
Proof of the CLT, cont.
LXi (0) = log MXi (0) = log 1 = 0
L00
Xi (t) =
May 22, 2014
Central Limit Theorem
Proof of the CLT, cont.
L0Xi (t) =
Lecture 7
lim
n→∞
0
MXi (t)MX00 (t) − [MX0 (t)]2
d MXi (t)
i
i
=
dt MXi (t)
[MXi (t)]2
MXi (0)MX00 (0) − [MX0 (0)]2
i
i
[MXi (0)]2
E (Xi0 )E (Xi2 ) − E (Xi )2
= E (Xi2 ) − E (Xi )2 = σ 2 = 1
E (Xi0 )2
Lecture 7
May 22, 2014
26 / 28
√
√
L0 (t/ n)(− 21 tn−3/2 )
L(t/ n)
=
lim
n→∞
n−1
−n−2
√
0
L (t/ n)t
= lim
n→∞ 2n−1/2
√
L00 (t/ n)t(− 12 tn−3/2 )
= lim
n→∞
−n−3/2
√ t2
= lim L00 (t/ n)
n→∞
2
t2
=
2
Sta 111 (Colin Rundel)
Lecture 7
by L’Hospital’s rule
by L’Hospital’s rule
May 22, 2014
27 / 28
Central Limit Theorem
Proof of the CLT, Final Comments
The preceding proof assumes that E (Xi ) = 0 and Var (Xi ) = 1.
We can generalize this result to any collection of random variables Yi by
considering the standardized form Yi∗ = (Yi − µ)/σ.
√
Y1 + · · · + Yn − nµ
Y1 − µ
Yn − µ
√
n
=
+ ··· +
σ
σ
σ n
√
= (Y1∗ + · · · + Yn∗ ) / n
E (Yi∗ ) = 0
Var (Yi∗ ) = 1
Sta 111 (Colin Rundel)
Lecture 7
May 22, 2014
28 / 28