2. Independent random variables.
The Law of large Numbers. If {Xi : i ≥ 1} are a sequence of independent identically
distributed random variables (on some (Ω, F , P )) with E[Xi] = m, then
1
P ω : lim [X1 (ω) + · · · + Xn (ω)] = m = 1
n→∞ n
This is the strong law of large numbers. The weak law, which is naturally weaker than the
strong law, asserts that for any ǫ > 0,
1
lim P ω : [X1 (ω) + · · · + Xn (ω)] − m ≥ ǫ = 0
n→∞
n
The strong law requires the existence of the mean, i.e {Xi } have to be integrable. On
the other hand the weak law can be valid some times even if we can only have the mean
defined as
Z a
m = lim
a→∞
x dF (x)
−a
where F is the common distribution of {Xi }.
The Central Limit Theorem. If {Xi } are independent, identically distributed and have
mean 0 and variance σ 2 , then the distribution of Zn = √1n [X1 + · · · + Xn ], approaches the
normal distribution with mean , i.e.
Z x
y2
1
1
e− 2σ2 dy
lim P [ √ [X1 + · · · + Xn ] ≤ x] = √
n→∞
n
2πσ −∞
The central limit theorem is valid also if {Xi } are not identically distributed,Pbut Xi has
n
mean 0 and variance σi2 . Then with Zn = s1n [X1 + · · · + Xn ], where s2n = i=1 σi2 , we
have
Z x
y2
1
e− 2 dy
lim P [Zn ≤ x] = √
n→∞
2π −∞
provided sn → ∞ and some additional condition known as Lindeberg’s condition is satisfied. It is that, for any ǫ > 0,
n Z
1 X
x2 dFi (x) = 0
lim
n→∞ s2
n i=1 |x|≥ǫsn
where Fi is the distribution of Xi .
Kolmogorov’s Two Series Theorem. Let {Xi } be a sequence of independent random
variables uniformly bounded by a constant C with Xi having mean 0 and variance σi2 .
Then the necessary and sufficient condition for the convergence with probability 1 of the
series
∞
X
Xi
S=
i=1
1
is
∞
X
i=1
σi2 < ∞
The sufficiency part of the proof depends on the important inequality known as Kolmogorov’s inequality.
Kolmogorov’s Inequality. Let {Xi } be indpendent random variables with mean 0 and
variance σi2 . Then
n
1 X 2
σi
P [ sup |X1 + X2 + · · · + Xj | ≥ ℓ] ≤ 2
ℓ
1≤j≤n
I=1
Tchebecheff’s inequality gives
j
1 X 2
P [|X1 + X2 + · · · + Xj | ≥ ℓ] ≤ 2
σi
ℓ
I=1
Naive calculation will yield
j
n
1 XX 2
σi
P [ sup |X1 + X2 + · · · + Xj | ≥ ℓ] ≤ 2
ℓ j=1
1≤j≤n
I=1
which is not good enough. Let us define
Ej = {ω : |X1 + X2 + · · · + Xk | < ℓ for k ≤ (j − 1),
|X1 + X2 + · · · + Xj | ≥ ℓ}
We are interested in estimating
P (E) =
n
X
P (Ej )
j=1
Denoting by Sj =
Pj
i=1
Z
Xi ,
Sn2 dP
=
Z
=
Z
≥
Z
Ej
Ej
(Sj + (Sn − Sj ))2 dP
Ej
[Sj2 + 2Sj (Sn − Sj ) + (Sn − Sj )2 ] dP
Ej
Sj2 dP ≥ ℓ2 P (Ej )
The cross term is 0 because Ej , Sj are independent of Sn − Sj . Because Ej are disjoint
and their union is E, summing over j = 1, 2, . . . , n, we get
Z
2
sn ≥
s2n dP ≥ ℓ2 P (E)
E
2
This allows the tail of the series
Tn = sup |Sj − Sn |
j≥n
to be estimated by
∞
1 X 2
σ
P [Tn ≥ ǫ] ≤ 2
ǫ j=n+1 j
proving Tn → 0 with probability 1. The converse depends on an inequality as well. Assume
lim sup |Sn | < ∞ with positive probbaility. Let
Fn = {ω : sup |Sj | ≤ ℓ}
1≤j≤n
so that En+1 = Fn − Fn+1 are disjoint. Then there is an ℓ such that P (Fn ) ≥ δ > 0 for all
Z
Z
Z
Z
Z
2
2
2
2
Sn+1 dP −
Sn2 dP
Sn dP =
Sn+1 dP +
Sn+1 dP −
Fn
F
Fn+1
En+1
Fn
Z n
2
]dP
=
[2Xn+1 Sn + Xn+1
Fn
Z
2
≥
Xn+1
dP
Fn
2
= P (Fn )σn+1
2
≥ δ σn+1
On the other hand |Sn+1 | ≤ C + ℓ on En+1 . Therefore
Z
Z
1
2
2
2
σn+1 ≤ [(C + ℓ) P (En+1 ) +
Sn+1 dP −
Sn2 dP ]
δ
En+1
Fn
And the telescoping sum
XZ
[
2
Sn+1
dP
En+1
−
Z
Sn2 dP ]
Fn
is bounded by ℓ2 . Providing the estimate
X
j
σj2 ≤
1
[(C + ℓ)2 + ℓ2 ]
δ
Actually we have shown some thing stronger. If
X
σj2 = +∞
j
3
then
P [lim sup |Sn | = ∞] = 1
n→∞
An easy corollary is that if {Xi } are independent random variables with E[Xi] = 0 and
E[Xi2 ] = σi2 then for any sequence of constants {aj } satisfying
X
j
σj2 a2j < ∞
the series
S=
X
j
will converge with probability 1,
4
aj X j