Lecture Notes 4
Convergence (Chapter 5)
1
Random Samples
Let X1 , . . . , Xn ∼ F . A statistic is any function Tn = g(X1 , . . . , Xn ). Recall that the sample
mean is
n
1X
Xi
Xn =
n i=1
and sample variance is
n
Sn2
1 X
(Xi − X n )2 .
=
n − 1 i=1
Let µ = E(Xi ) and σ 2 = Var(Xi ). Recall that
E(X n ) = µ,
Var(X n ) =
σ2
,
n
E(Sn2 ) = σ 2 .
Theorem 1 If X1 , . . . , Xn ∼ N (µ, σ 2 ) then X n ∼ N (µ, σ 2 /n).
Proof. We know that MXi (s) = eµs+σ
t
MX n (t) = E(etX n ) = E(e n
Pn
i=1
Xi
2 s2 /2
. So,
)
= (EetXi /n )n = (MXi (t/n))n = e(µt/n)+σ
2 t2 /(2n2 )
n
(
= exp µt +
σ2 2
t
n
)
2
which is the mgf of a N (µ, σ 2 /n). Example 2 Let X(1) , . . . , X(n) denoted the ordered values:
X(1) ≤ X(2) ≤ · · · ≤ X(n) .
Then X(1) , . . . , X(n) are called the order statistics. And Tn = (X(1) , . . . , X(n) ) is a statistic.
1
2
Convergence
Let X1 , X2 , . . . be a sequence of random variables and let X be another random variable.
Let Fn denote the cdf of Xn and let F denote the cdf of X. We are going to study different
types of convergence.
Example: A good example to keep in mind is the folllowing. Let Y1 , Y2 , . . . , be a
sequence of iid random variables. Let
n
1X
Yi
Xn =
n i=1
be the average of the first n of the Yi ’s. This defines a new sequence X1 , X2 , . . . ,. In other
words, the sequence of interest X1 , X2 , . . . , might be a sequence of statistics based on some
other sequence of iid random variables. Note that the original sequence Y1 , Y2 , . . . , is
iid but the sequence X1 , X2 , . . . , is not iid.
a.s.
1. Xn converges almost surely to X, written Xn → X, if, for every > 0,
P lim |Xn − X| < = 1.
n→∞
(1)
a.s.
Xn converges almost surely to a constant c, written Xn → c, if, for every > 0,
P lim Xn = c = 1.
(2)
n→∞
P
2. Xn converges to X in probability, written Xn → X, if, for every > 0,
P(|Xn − X| > ) → 0
(3)
as n → ∞. In other words, Xn − X = oP (1).
P
Xn converges to c in probability, written Xn → c, if, for every > 0,
P(|Xn − c| > ) → 0
(4)
as n → ∞. In other words, Xn − c = oP (1).
3. Xn converges to X in quadratic mean (also called convergence in L2 ), written
qm
Xn → X, if
E(Xn − X)2 → 0
(5)
as n → ∞.
qm
Xn converges to c in quadratic mean, written Xn → c, if
E(Xn − c)2 → 0
as n → ∞.
2
(6)
4. Xn converges to X in distribution, written Xn
X, if
lim Fn (t) = F (t)
(7)
n→∞
at all t for which F is continuous.
Xn converges to c in distribution, written Xn
c, if
lim Fn (t) = δc (t)
n→∞
(8)
at all t 6= c where δc (t) = 0 if t < c and δc (t) = 1 if t ≥ c.
Theorem 3 Convergence in probability does not imply almost sure convergence.
Proof. Let Ω = [0, 1]. Let P be uniform on [0, 1]. We draw S ∼ P . Let X(s) = s and
let
X1 = s + I[0,1] (s),
X4 = s + I[0,1/3] (s),
X2 = s + I[0,1/2] (s), X3 = s + I[1/2,1] (s)
X5 = s + I[1/3,2/3] (s), X6 = s + I[2/3,1] (s)
P
etc. Then Xn → X. But, for each s, Xn (s) does not converge to X(s). Hence, Xn does not
converge almost surely to X. In fact, P ({s ∈ Ω : limn Xn (s) = X(s)}) = 0. Example 4 Let Xn ∼ N (0, 1/n). Intuitively, Xn is concentrating at 0 so we would like to
say that Xn converges to 0. Let’s
√ see if this is true. Let F be the distribution function for
a point mass at 0. Note that nXn ∼ N (0, 1). Let Z denote a standard normal random
variable. For t < 0,
√
√
√
Fn (t) = P(Xn < t) = P( nXn < nt) = P(Z < nt) → 0
√
since nt → −∞. For t > 0,
√
√
√
Fn (t) = P(Xn < t) = P( nXn < nt) = P(Z < nt) → 1
√
since nt → ∞. Hence, Fn (t) → F (t) for all t 6= 0 and so Xn
0. Notice that Fn (0) =
1/2 6= F (1/2) = 1 so convergence fails at t = 0. That doesn’t matter because t = 0 is
not a continuity point of F and the definition of convergence in distribution only requires
convergence at continuity points.
Now consider convergence in probability. For any > 0, using Markov’s inequality,
1
E(Xn2 )
n
P(|Xn | > ) = P(|Xn | > ) ≤
= 2 →0
2
2
2
P
as n → ∞. Hence, Xn → 0.
3
The next theorem gives the relationship between the types of convergence.
Theorem 5 The following relationships hold:
qm
P
(a) Xn → X implies that Xn → X.
P
(b) Xn → X implies that Xn
X.
P
(c) If Xn
X and if P(X = c) = 1 for some real number c, then Xn → X.
as
P
(d) Xn → X implies Xn → X.
In general, none of the reverse implications hold except the special case in (c).
qm
Proof. We start by proving (a). Suppose that Xn → X. Fix > 0. Then, using
Markov’s inequality,
P(|Xn − X| > ) = P(|Xn − X|2 > 2 ) ≤
E|Xn − X|2
→ 0.
2
Proof of (b). Fix > 0 and let x be a continuity point of F . Then
Fn (x) = P(Xn ≤ x) = P(Xn ≤ x, X ≤ x + ) + P(Xn ≤ x, X > x + )
≤ P(X ≤ x + ) + P(|Xn − X| > )
= F (x + ) + P(|Xn − X| > ).
Also,
F (x − ) = P(X ≤ x − ) = P(X ≤ x − , Xn ≤ x) + P(X ≤ x − , Xn > x)
≤ Fn (x) + P(|Xn − X| > ).
Hence,
F (x − ) − P(|Xn − X| > ) ≤ Fn (x) ≤ F (x + ) + P(|Xn − X| > ).
Take the limit as n → ∞ to conclude that
F (x − ) ≤ lim inf Fn (x) ≤ lim sup Fn (x) ≤ F (x + ).
n→∞
n→∞
This holds for all > 0. Take the limit as → 0 and use the fact that F is continuous at x
and conclude that limn Fn (x) = F (x).
Proof of (c). Fix > 0. Then,
P(|Xn − c| > ) =
≤
=
→
=
P(Xn < c − ) + P(Xn > c + )
P(Xn ≤ c − ) + P(Xn > c + )
Fn (c − ) + 1 − Fn (c + )
F (c − ) + 1 − F (c + )
0 + 1 − 1 = 0.
4
Proof of (d). Omitted.
Let us now show that the reverse implications do not hold.
Convergence in
√in quadratic mean. Let U ∼ Unif(0, 1)
√probability does not imply convergence
and let Xn = nI(0,1/n) (U ). Then P(|Xn | > ) = P( nI(0,1/n) (U ) > ) = P(0 ≤ U < 1/n) =
R 1/n
P
1/n → 0. Hence, Xn → 0. But E(Xn2 ) = n 0 du = 1 for all n so Xn does not converge in
quadratic mean.
Convergence in distribution does not imply convergence in probability. Let X ∼ N (0, 1).
Let Xn = −X for n = 1, 2, 3, . . .; hence Xn ∼ N (0, 1). Xn has the same distribution
function as X for all n so, trivially, limn Fn (x) = F (x) for all x. Therefore, Xn
X. But
P(|Xn − X| > ) = P(|2X| > ) = P(|X| > /2) 6= 0. So Xn does not converge to X in
probability. The relationships between the types of convergence can be summarized as follows:
q.m.
↓
a.s. → prob → distribution
P
Example 6 One might conjecture that if Xn → b, then E(Xn ) → b. This is not true.
Let Xn be a random variable defined by P(Xn = n2 ) = 1/n and P(Xn = 0) = 1 − (1/n).
P
Now, P(|Xn | < ) = P(Xn = 0) = 1 − (1/n) → 1. Hence, Xn → 0. However, E(Xn ) =
[n2 × (1/n)] + [0 × (1 − (1/n))] = n. Thus, E(Xn ) → ∞.
Example 7 Let X1 , . . . , Xn ∼ Uniform(0, 1). Let X(n) = maxi Xi . First we claim that
P
X(n) → 1. This follows since
P(|X(n) − 1| > ) = P(X(n) ≤ 1 − ) =
Y
P(Xi ≤ 1 − ) = (1 − )n → 0.
i
Also
P(n(1 − X(n) ) ≤ t) = P(X(n) ≥ 1 − (t/n)) = 1 − (1 − t/n)n → 1 − e−t .
So n(1 − X(n) )
Exp(1).
Some convergence properties are preserved under transformations.
Theorem 8 Let Xn , X, Yn , Y be random variables.
P
P
P
(a) If Xn → X and Yn → Y , then Xn + Yn → X + Y .
qm
qm
qm
(b) If Xn → X and Yn → Y , then Xn + Yn → X + Y .
P
P
P
(c) If Xn → X and Yn → Y , then Xn Yn → XY .
5
In general, Xn
X and Yn
are cases when it does:
Y does not imply that Xn + Yn
Theorem 9 (Slutzky’s Theorem) If Xn
If Xn
X and Yn
c, then Xn Yn
cX.
X and Yn
X + Y . But there
c, then Xn + Yn
X + c. Also,
Theorem 10 (The Continuous Mapping Theorem) Let Xn , X, Yn , Y be random variables. Let g be a continuous function.
P
P
(a) If Xn → X, then g(Xn ) → g(X).
(b) If Xn
X, then g(Xn )
g(X).
Exercise: Prove the continuous mapping theorem.
3
The Law of Large Numbers
The law of large numbers (LLN) says that the mean of a large sample is close to the mean
of the distribution. For example, the proportion of heads of a large number of tosses of a
fair coin is expected to be close to 1/2. We now make this more precise.
Let X1 , X2 , . . . be an iid sample,
let µ = E(X1 ) and σ 2 = Var(X1 ). Recall that the sample
P
n
mean is defined as X n = n−1 i=1 Xi and that E(X n ) = µ and Var(X n ) = σ 2 /n.
Theorem 11 (The Weak Law of Large Numbers (WLLN))
P
If X1 , . . . , Xn are iid, then X n → µ. Thus, X n − µ = oP (1).
Interpretation of the WLLN: The distribution of X n becomes more concentrated
around µ as n gets large.
Proof. Assume that σ < ∞. This is not necessary but it simplifies the proof. Using
Chebyshev’s inequality,
Var(X n )
σ2
=
P |X n − µ| > ≤
2
n2
which tends to 0 as n → ∞. Theorem 12 The Strong Law of Large Numbers. Let X1 , . . . , Xn be iid with mean µ.
as
Then X n → µ.
The proof is beyond the scope of this course.
6
4
The Central Limit Theorem
The law of large numbers says that the distribution of X n piles up near µ. This isn’t enough
to help us approximate probability statements about X n . For this we need the central limit
theorem.
2
Suppose that X1 , . . . , Xn are iid
P with mean µ and variance σ . The central limit the−1
orem (CLT) says that X n = n
i Xi has a distribution which is approximately Normal
with mean µ and variance σ 2 /n. This is remarkable since nothing is assumed about the
distribution of Xi , except the existence of the mean and variance.
Theorem 13 (The Central Limit
P Theorem (CLT)) Let X1 , . . . , Xn be iid with mean µ
and variance σ 2 . Let X n = n−1 ni=1 Xi . Then
√
Xn − µ
n(X n − µ)
Zn ≡ q
=
Z
σ
Var(X n )
where Z ∼ N (0, 1). In other words,
Z
z
lim P(Zn ≤ z) = Φ(z) =
n→∞
−∞
1
2
√ e−x /2 dx.
2π
Interpretation: Probability statements about X n can be approximated using a Normal distribution. It’s the probability statements that we are approximating, not the
random variable itself.
Remark: We often write
Xn ≈ N
as short form for
√
n(X n − µ)
σ2
µ,
n
N (0, 1).
Recall that if X is a random variable, its moment generating function (mgf) is ψX (t) =
EetX . Assume in what follows that the mgf is finite in a neighborhood around t = 0.
Lemma 14 Let Z1 , Z2 , . . . be a sequence of random variables. Let ψn be the mgf of Zn . Let
Z be another random variable and denote its mgf by ψ. If ψn (t) → ψ(t) for all t in some
open interval around 0, then Zn
Z.
P
−1/2
Proof of the central limit theorem.
Let
Y
=
(X
−
µ)/σ.
Then,
Z
=
n
i
i
n
i Yi .
P
√
Let ψ(t) be the mgf of Yi . The mgf of i Yi is (ψ(t))n and mgf of Zn is [ψ(t/ n)]n ≡ ξn (t).
Now ψ 0 (0) = E(Y1 ) = 0, ψ 00 (0) = E(Y12 ) = Var(Y1 ) = 1. So,
t2 00
t3 000
ψ(t) = ψ(0) + tψ (0) + ψ (0) + ψ (0) + · · ·
2!
3!
t2 t3 000
= 1 + 0 + + ψ (0) + · · ·
2
3!
t2 t3 000
= 1 + + ψ (0) + · · ·
2
3!
0
7
Now,
n
t
ξn (t) = ψ √
n
n
2
t
t3
000
= 1+
+
ψ (0) + · · ·
2n 3!n3/2
#n
"
t3
t2
000
+
ψ
(0)
+
·
·
·
1/2
3!n
=
1+ 2
n
2 /2
→ et
which is the mgf of a N(0,1). The result follows from Lemma 14. In the last step we used
the fact that if an → a then
an n
1+
→ ea . n
√
The central limit theorem tells us that Zn = n(X n − µ)/σ is approximately N(0,1).
However, we rarely know σ. We can estimate σ 2 from X1 , . . . , Xn by
n
Sn2
1 X
(Xi − X n )2 .
=
n − 1 i=1
This raises the following question: if we replace σ with Sn , is the central limit theorem still
true? The answer is yes.
Theorem 15 Assume the same conditions as the CLT. Then,
√
n(X n − µ)
Tn =
N (0, 1).
Sn
Proof. Here is a brief proof. We have that
Tn = Zn Wn
where
√
Zn =
n(X n − µ)
σ
and
Wn =
Now Zn
P
σ
.
Sn
N (0, 1) and Wn → 1. The result follows from Slutzky’s theorem. 8
Here is an extended proof.
P
Step 1. We first show that Rn2 → σ 2 where
n
Rn2
1X
(Xi − X n )2 .
=
n i=1
Note that
n
1X 2
2
Rn =
X −
n i=1 i
n
1X
Xi
n i=1
!2
.
Define Yi = Xi2 . Then, using the LLN (law of large numbers)
n
n
1X
1X 2
P
Xi =
Yi → E(Yi ) = E(Xi2 ) = µ2 + σ 2 .
n i=1
n i=1
Next, by the LLN,
n
1X
P
Xi → µ.
n i=1
Since g(t) = t2 is continuous, the continuous mapping theorem implies that
!2
n
1X
P
Xi
→ µ2 .
n i=1
Thus
P
Rn2 → (µ2 + σ 2 ) − µ2 = σ 2 .
Step 2. Note that
Sn2
=
n
n−1
P
Rn2 .
P
Since, Rn2 → σ 2 and n/(n − 1) → 1, we have that Sn2 → σ 2 .
Step 3. Since g(t) =
P
implies that Sn → σ.
√
t is continuous, (for t ≥ 0) the continuous mapping theorem
Step 4. Since g(t) = t/σ is continuous, the continuous mapping theorem implies that
P
Sn /σ → 1.
Step 5. Since g(t) = 1/t is continuous (for t > 0) the continuous mapping theorem
P
implies that σ/Sn → 1. Since convergence in probability implies convergence in distribution,
σ/Sn
1.
9
Step 5. Note that
√
Tn =
n(X n − µ)
σ
σ
Sn
≡ Vn Wn .
Now Vn
Z where Z ∼ N (0, 1) by the CLT. And we showed that Wn
1. By Slutzky’s
theorem, Tn = Vn Wn
Z × 1 = Z.
The next result is very important. It tells us how close the distribution of X is to the
Normal distribution.
Theorem 16 (Berry-Esseen Theorem) Let X1 , . . . , Xn ∼ P . Let µ = E[Xi ] and σ 2 =
Var[Xi ]. Assume that µ3 = E[|Xi − µ|3 ] < ∞. Let
√
n(X n − µ)
≤z .
Fn (z) = P
σ
Then
sup |Fn (z) − Φ(z)| ≤
z
33 µ3
√ .
4 σ3 n
There is also a multivariate version of the central limit theorem. Recall that X =
(X1 , . . . , Xk )T has a multivariate Normal distribution with mean vector µ and covariance
matrix Σ if
1
1
T −1
f (x) =
exp − (x − µ) Σ (x − µ) .
(2π)k/2 |Σ|1/2
2
In this case we write X ∼ N (µ, Σ).
Theorem 17 (Multivariate central limit theorem) Let X1 , . . . , Xn be iid random vectors where Xi = (X1i , . . . , Xki )T withP
mean µ = (µ1 , . . . , µk )T and covariance matrix Σ. Let
X = (X 1 , . . . , X k )T where X j = n−1 ni=1 Xji . Then,
√
n(X − µ)
N (0, Σ).
Remark: There is also a multivariate version of the Berry-Esseen theorem but it is more
complicated than the one-dimensional version.
5
The Delta Method
If Yn has a limiting Normal distribution then the delta method allows us to find the limiting
distribution of g(Yn ) where g is any smooth function.
10
Theorem 18 (The Delta Method) Suppose that
√
n(Yn − µ)
N (0, 1)
σ
and that g is a differentiable function such that g 0 (µ) 6= 0. Then
√
n(g(Yn ) − g(µ))
N (0, 1).
|g 0 (µ)|σ
In other words,
σ2
Yn ≈ N µ,
n
!
!
2
σ
g(Yn ) ≈ N g(µ), (g 0 (µ))2
.
n
implies that
Example 19 Let
variance σ 2 . By the central
√ X1 , . . . , Xn be iid with finite mean µ and Xfinite
limit theorem, n(X n − µ)/σ
N (0, 1). Let Wn = e n . Thus, Wn = g(X n ) where
s
0
s
g(s) = e . Since g (s) = e , the delta method implies that Wn ≈ N (eµ , e2µ σ 2 /n).
There is also a multivariate version of the delta method.
Theorem 20 (The Multivariate Delta Method) Suppose that Yn = (Yn1 , . . . , Ynk ) is a
sequence of random vectors such that
√
n(Yn − µ)
N (0, Σ).
Let g : Rk → R and let

∂g
∂y1



∇g(y) =  ...  .
∂g
∂yk
Let ∇µ denote ∇g(y) evaluated at y = µ and assume that the elements of ∇µ are nonzero.
Then
√
n(g(Yn ) − g(µ))
N 0, ∇Tµ Σ∇µ .
Example 21 Let
X11
X21
X12
X1n
,
, ...,
X22
X2n
be iid random vectors with mean µ = (µ1 , µ2 )T and variance Σ. Let
n
n
1X
X1 =
X1i ,
n i=1
1X
X2 =
X2i
n i=1
and define Yn = X 1 X 2 . Thus, Yn = g(X 1 , X 2 ) where g(s1 , s2 ) = s1 s2 . By the central limit
theorem,
√
X 1 − µ1
n
N (0, Σ).
X 2 − µ2
11
Now
∇g(s) =
∂g
∂s1
∂g
∂s2
!
=
s2
s1
and so
∇Tµ Σ∇µ
= (µ2 µ1 )
σ11 σ12
σ12 σ22
µ2
µ1
= µ22 σ11 + 2µ1 µ2 σ12 + µ21 σ22 .
Therefore,
√
n(X 1 X 2 − µ1 µ2 )
N
0, µ22 σ11
12
+ 2µ1 µ2 σ12 +
µ21 σ22
.