PTE 5TH EDITION SOLUTIONS MANUAL
HOIL LEE AND WONJUN SEO
Contents
1.
Measure Theory
3
1.1.
Probability Theory
1.2.
Distributions
4
1.3.
Random Variables
5
1.4.
Integration
7
1.5.
Properties of Integral
9
1.6.
Expected Value
12
1.7.
Product Measures, Fubini’s Theorem
16
2.
3
Laws of Large Numbers
18
2.1.
Independence
18
2.2.
Weak Laws of Large Numbers
21
2.3.
Borel-Cantelli Lemmas
26
2.4. Strong Law of Large Numbers
3. Central Limit Theorems
36
37
3.1.
The De Moivre-Laplace Theorem
37
3.2.
Weak Convergence
37
3.3.
Characteristic Functions
47
3.4.
Central Limit Theorems
58
3.5.
Local Limit Theorems
64
3.6.
Poisson Convergence
64
3.7.
Poisson Processes
64
∗
3.8.
Stable Laws
3.9.
Infinitely Divisible Distributions∗
3.10.
4.
Limit Theorems in R
64
64
d
65
Martingales
67
4.1.
Conditional Expectation
67
4.2.
Martingales, Almost Sure Convergence
70
4.3.
Examples
72
4.4.
p
Doob’s Inequality, Convergence in L , p > 1
Date: September 9, 2021.
1
76
4.6.
Uniform Integrability, Convergence in L1
4.7.
Backwards Martingales
80
4.8.
Optional Stopping Theorems
81
4.9.
Combinatorics of Simple Random Walk
84
5.
Markov Chains
78
84
5.1.
Examples
84
5.2.
Construction, Markov Properties
86
5.3.
Recurrence and Transience
89
5.4.
Recurrence of Random Walks
89
5.5.
Stationary Measures
89
5.6.
Asymptotic Behavior
89
5.7.
Periodicity, Tail σ-Field
89
5.8.
6.
General State Space
89
Ergodic Theorems
89
6.1.
Definitions and Examples
89
6.2.
Birkhoff’s Ergodic Theorem
92
6.3.
Recurrence
93
6.5. Applications
7. Brownian Motion
95
95
7.1.
Definition and Construction
95
7.2.
Markov Property, Blumenthal’s 0-1 Law
95
7.3.
Stopping Times, Strong Markov Property
96
7.4.
Path Properties
96
7.5.
Martingales
96
7.6.
8.
Itô’s Formula
96
Applications to Random Walk
96
8.3.
Donsker’s Theorem
96
8.4.
Empirical Distributions, Brownian Bridge
96
8.5.
9.
Laws of Iterated Logarithm
96
Multidimensional Brownian Motion
96
9.4.
Martingales
96
9.5.
Dirichlet Problem
96
2
1. Measure Theory
1.1. Probability Theory.
Exercise 1.1.1.
(i) F is a σ-algebra.
From the definition, ∅, Ω are in F, and F is closed under operation of complement.
Now, suppose A1 , A2 , · · · are in F. If all An ’s are countable, then ∪∞
n=1 An , which
is countable union of countable sets, is also countable, and thus is in F. Otherwise,
c
∞
∪∞
n=1 An is countable, and thus ∪n=1 An is in F.
(ii) P is a probability measure.
By definition, P (A) ≥ P (∅) = 0, for any A ∈ F. For disjiont A1 , A2 , · · · ∈ F, if all An ’s
are countable, then
P (∪∞
n=1 An ) = 0 =
∞
X
P (An ).
n=1
If one of An is uncountable(say Ak ), then since Ack is countable and Aj ⊂ Ack for j 6= k,
c
c
(∪∞
n=1 An ) ⊂ Ak and Aj ’s are countable for j 6= k. Thus,
X
P (∪∞
An .
n=1 An ) = 1 = P (Ak ) +
n6=k
Therefore, (Ω, F, P ) is a probability space.
Exercise 1.1.2. Let Od = {(a1 , b1 ) × · · · × (ad , bd ) : ai , bi ∈ R}. It suffices to show that
σ(Od ) = σ(Sd ). Since,
(a1 , b1 ] × · · · × (ad , bd ] =
(a1 , b1 ) × · · · × (ad , bd ) =
∞
\
(a1 , b1 +
n=1
∞
[
(a1 , b1 −
n=1
1
1
) × · · · × (ad , bd + ).
n
n
1
1
) × · · · × (ad , bd − )
n
n
Exercise 1.1.3. C = {(a1 , b1 ) × · · · × (ad , bd ) : rational numbers a < b}.
Exercise 1.1.4. Let F = ∪i Fi .
(i) Clearly, ∅, Ω ∈ F. If A ∈ F, then there exists j such that A ∈ Fj . Then, Ac ∈ Fj , and
thus Ac ∈ Fj . Let A and B are in F. Then, there exist j1 , j2 such that A ∈ Fj1 and
B ∈ Fj2 . WLOG, let j1 ≤ j2 . Since Fn are increasing sequence, A is also in Fj2 . Thus,
A ∪ B ∈ Fj ⊂ F. Therefore, F is an algebra.
j
n
∀
(ii) Let Ω = (0, 1] and In = {( j−1
2n , 2n ], for j = 1, . . . , 2 } for n. Define
Fn ≡ σ(In ) for n = 1, 2, · · ·
2(j−1) 2j−1
j
2j−1
2j
Since ( j−1
2n , 2n ] = ( 2n+1 , 2n+1 ] ∪ ( 2n+1 , 2n+1 ], Fn ⊂ Fn+1 .
Now,
(0, 1) =
∞
[
2n − 2 2n − 1
],
( n ,
2
2n
n=1
3
n
n
2 −1
∀
/ Fn for ∀ n, (0,1) ∈
/ F.
where ( 2 2−2
n ,
2n ] ∈ Fn for n. However, since (0,1) ∈
Therefore, F is not a σ-algebra.
Exercise 1.1.5. (Counterexample) Let A be the set of even numbers. Then, A ∈ A with
θ = 21 . Let I0 = {1}, Ik = {n : 2k−1 < n ≤ 2k }, for k = 1, · · · . Construct B as follows.
c
B = ∪∞
k=0 {(I2k ∩ A ) ∪ (I2k+1 ∩ A)}
Then, B ∈ A with θ = 12 . However, A ∩ B ∈
/ A for following reasons.
(1) [n = 22m ]
|(A ∩ B) ∩ {1, 2, . . . , 22m }| =
Pm
1
k=1 2 |I2k−1 |.
|(A ∩ B) ∩ {1, 2, . . . , 22m }|
=
22m
=
=
Pm
1
k=1 2 |I2k−1 |
22m
1+
Pm
k=2
22m
22m−1 +1
3
22m
22k−3
→
1
,
6
(2) [n = 22m+1 ]
Similarly,
|(A ∩ B) ∩ {1, 2, . . . , 22m+1 }|
=
22m+1
=
=
By (1), (2), lim sup |(A∩B)∩{1,2,...,n}|
≥
n
Pm+1
1
k=1 2 |I2k−1 |
22m+1
1+
Pm+1
2k−3
k=2 2
22m+1
22m+1 +1
3
22m+1
→
1
3
|(A∩B)∩{1,2,...,n}|
1
3 , lim inf
n
≤
1
6.
Thus, A ∩ B ∈
/ A.
Therefore, A is neither an algebra nor a σ-algebra.
1.2. Distributions.
Exercise 1.2.1. To show that Z is a r.v., it suffices to show that for any B ∈ B(R), Z −1 (B) ∈
F.
Z −1 (B) = [Z −1 (B) ∩ A] ∪ [Z −1 (B) ∩ Ac ]
= [X −1 (B) ∩ A] ∪ [Y −1 (B) ∩ Ac ]
∈ B(R),
since X, Y are r.v.s, and A ∈ F.
Exercise 1.2.2. Omit.
Exercise 1.2.3. Let A be the discontinuity set of distribution function F . Since F is rightcontinuous, for x ∈ A, F (x− ) < F (x) = F (x+ ) holds. For x < y who are in A, we have
4
F (x− ) < F (x) ≤ F (y − ) < F (y). Pick rationals qx and qy such that qx ∈ (F (x− ), F (x))
and qy ∈ (F (y − ), F (y)). Then, qx < qy , and thus |A| ≤ ℵ0 . Therefore, there exist at most
countably many discontinuities.
Exercise 1.2.4. Let F −1 (y) = inf{x : y ≤ F (x)}. Then followings hold.
(i) F (F −1 (y)) = y for y ∈ (0, 1).
(ii) F −1 is strictly increasing.
(iii) {F −1 (F (X)) ≤ F −1 (y)} = {X ≤ F −1 (y)}
(i) comes from continuity of F , and (ii) comes from monotonicity of F . For (iii),
{F −1 (F (X)) ≤ F −1 (y)} = {F (X) ≤ y}
= {X ≤ F
−1
(by (ii))
(y)}. (by definition)
Thus, for 0 < y < 1,
P (F (X) ≤ y) = P (F −1 (F (X)) ≤ F −1 (y))
(by (ii))
= P (X ≤ F −1 (y))
= F (F
−1
(by (iii))
(y))
=y
(by (i))
For y = 0, P (F (X) ≤ 0) = limn P (F (X) ≤
1
n)
= limn
For y = 1, P (F (X) ≤ 1) = limn P (F (X) ≤ 1 −
1
n)
1
n
= 0.
= limn 1 −
1
n
= 1.
Thus, F (X) = Y ∼ U nif (0, 1).
Exercise 1.2.5. Omit.
Exercise 1.2.6. Omit.
Exercise 1.2.7. Omit.
1.3. Random Variables.
Exercise 1.3.1. We want to show that σ(X) = σ(X −1 (A)).
(⊃) By definition of σ(X) = {{X ∈ B} : B ∈ S}, it is clear.
(⊂) It suffices to show that X is σ(X −1 (A))-measurable. Firstly, for any σ-field D on
Ω, T = {B ∈ S : {X ∈ B} ∈ D} is also a σ-field on S. Construct T with D =
σ(X −1 (A)). Then, T is a σ-field containing A. Since A generates S, S = T , which
means that X is σ(X −1 (A))-measurable.
Exercise 1.3.2.
{X1 + X2 < x} = {X1 < x − X2 }
[
=
{X1 < 1 < x − X2 }
q∈Q
=
[
[{X1 < q} ∩ {X2 < x − q}] ∈ F.
q∈Q
5
Exercise 1.3.3. Since f is continuous, f (Xn ) and f (X) are r.v. Moreover, on Ω0 = {ω :
Xn (ω) → X(ω)}, f (Xn )(ω) → f (X)(ω), which means that Ω0 ⊂ Ω00 = {ω : f (Xn )(ω) →
f (X)(ω)}, since f is continuous. Thus, P (Ω0 ) ≥ P (Ω) = 1, that is, f (Xn ) → f (X) a.s.
Exercise 1.3.4.
(i) Let f be a continuous function from Rd → R and let O, Od be col-
lections of open sets in R, Rd , respectively. Since pre-image of open set in R under
f is also open(by continuity), σ(f −1 (O)) ⊂ σ(Od ) = B(Rd ). Also, by Exercise 1.3.1,
σ(f −1 (O)) = f −1 (B(R)). Thus, f is B(Rd ), B(R) -measurable.
(ii) Let A be the smallest σ-field that makes all the continuous functions measurable. By
(i), we showed that A ⊂ B(Rd ). To show the converse, define fi ((x1 , . . . , xd )) = xi , for
i = 1, · · · , d, who are continuous. Then, fi−1 ((ai , bi )) ∈ B(Rd ), and thus,
f1−1 ((a1 , b1 )) ∩ · · · ∩ fd−1 ((ad , bd )) = (a1 , b1 ) × · · · × (ad , bd ),
for any ai , bi ∈ R. This means that any open boxes in Rd is in A. Therefore, B(Rd ) ⊂ A.
Exercise 1.3.5. Let f be a lower semicontinuous function. f is l.s.c if and only if for any
> 0, there exists δ > 0 such that f (y) > f (x) − for y ∈ B(x, δ). We want to show that f
is l.s.c if and only if {x : f (x) ≤ a} is closed for each a ∈ R.
(⇒) Suppose not. Then, there exists a0 ∈ R such that {x : f (x) ≤ a} is not closed, that
is, there exists a limit point x0 of {x : f (x) ≤ a} with f (x0 ) > a0 . For =
there exists δ > 0 such that f (y) > f (x0 ) − =
f (x0 )+a0
2
f (x0 )−a0
,
2
> a0 for y ∈ B(x0 , δ). This
contradicts that x0 is a limit point of {x : f (x) ≤ a}.
(⇐) Fix x0 . Let’s show that f is l.s.c. at x0 . For any > 0, let a = f (x0 ) − . Then,
x0 ∈ {x : f (x) > a}. Since {x : f (x) > a} = {x : f (x) ≤ a}c is an open set, there
exists δ > 0 such that B(x0 , δ) ⊂ {x : f (x) > a}. That is, f (y) > f (x) − for
y ∈ B(x0 , δ). Thus, f is l.s.c. at x0 . Since x0 is arbitrary, f is l.s.c.
For measurability, by Exercise 1.3.1, it suffices to show that f −1 ((a, ∞)) is in B(R). If f is
l.s.c, then f −1 ((−∞, a]) = {x : f (x) ≤ a} is closed. Thus, f ∈ B(R).
Exercise 1.3.6. Let’s show that f δ = sup{f (y) : |y−x| < δ} is l.s.c. By Exercise 1.3.5, it suffices to show that Aa ≡ {x : f δ (x) ≤ a} is closed for any a ∈ R. Let x0 be a limit point of Aa .
For any y ∈ B(x0 , δ), there exists zy ∈ Aa different from x0 such that |x0 − zy | < δ − |x0 − y|.
Since |zy − y| ≤ |zy − x0 | + |x0 − y| < δ and z ∈ Aa , f (y) ≤ a. Since this holds for all
y ∈ B(x0 , δ), sup{f (y) : |y − x0 | < δ} ≤ a. That is, x0 ∈ Aa , which implies that Aa is closed.
Similar argument gives us that fδ is u.s.c.
Since semicontinuous functions are measurable (Exercise 1.3.5), f 0 and f0 , limits of semicontinuous functions, are also measurable. Thus,
∞ [
1
0
0
{f 6= f0 } =
x : f (x) − f0 (x) >
n
n=0
is measurable. (Clearly, f 0 ≥ f0 )
6
Exercise 1.3.7. (1) Obviously, simple functions are F-measurable. And, for any fn ∈ F,
lim sup fn is also F-measurable.
(2) Any F-measurable function is a pointwise limit of simple functions.
By (1) and (2), class of F-measurable functions is the smallest class containing the simple
functions and closed under pointwise limit.
Exercise 1.3.8.
f
X
(Ω, σ(X)) → (R, B(R)) → (R, B(R))
(⇐) Since X is σ(X)-measurable, Y = f (X) is σ(X)-measurable. (Composition of measurable functions is measurable.)
(⇒) If one can show that T = {Y = f (X) : f ∈ B(R)} contains simple functions with
respect to (Ω, σ(X)) and is closed under pointwise limit, then by previous Exercise
1.3.7, {Y : Y ∈ σ(X)} ⊂ T .
(a) Let ϕ be a simple function with respect to (Ω, σ(X)). That is,
ϕ(ω) =
n
X
cm IAm (ω) where disjoint Am ∈ σ(X).
m=1
Since Am ∈ σ(X), there exist disjoint Bm ∈ B(R)X such that Am = X −1 (Bm ).
Thus,
ϕ(ω) =
=
=
n
X
m=1
n
X
m=1
n
X
cm IAm (ω)
cm IX −1 (Bm ) (ω)
cm IBm (X(ω))
m=1
Therefore, T contains simple functions with respect to (Ω, σ(X)).
(b) Now, let Y be a pointwise limit of Yn = fn (X) where f ∈ B(R). That is,
(?)
Y (ω) = limn fn (X(ω)) for all ω ∈ Ω. Define f (x) = lim supn fn (x). Since
fn ∈ B(R), f ∈ B(R). Also, Y (ω) = f (X(ω)) for all ω ∈ Ω. Thus, T is closed
under pointwise limit.
(?) : From the definition of fn and f , we can ensure that fn converges to f only
on x ∈ X(Ω). Since we don’t know whether convergence also holds on
R\X(Ω) or not, we define f (x) by limsup of fn (x).
Exercise 1.3.9. Omit.
1.4. Integration.
7
Exercise 1.4.1. Suppose not. Then, there exists n such that µ(An ) > 0, where An =
{f (x) >
1
n }.
Since f ≥ 0,
Z
Z
f dµ ≥
f dµ ≥
An
1
µ(An ) > 0,
n
which is a contradiction.
Exercise 1.4.2. Define fn (x) =
P∞
m
m=1 2n IEn,m (x).
By MCT,
R
fn (x) =
P∞
m
m=1 2n µ(En,m ).
Note that if x ∈ En,m , then x ∈ En+1,2m ∪ En+1,2m+1 . Thus, fn (x) ≤ fn+1 (x). And, since
R
R
f (x) − fn (x) < 21n , fn (x) → f (x) for all x. Thus, by MCT, fn dµ = f dµ. Combining two
results, we get the desired result.
R
R
(i) Let g ≥ 0. Since gdm = sup{ sdm : 0 ≤ s ≤ f, s simple}, there
R
R
R
R
exists simple 0 ≤ s ≤ g such that gdm − < sdm. Thus, |g − s|dm = (g − s)dm <
Exercise 1.4.3.
.
For integrable g = g + − g − , find simple s+ and s− such that
Z
Z
|g − − s− |dm < .
|g + − s+ |dm <
2
2
Then,
Z
|g − (s+ − s− )|dm ≤
Z
|g − s+ |dm +
Z
|g − s− |dm
< .
s+ − s− is the desired simple function.
(ii) Firstly, let ϕ = IA , where A is measurable. By outer regularity of Lebesgue measure,
U∞
there exists open G ⊃ A such that m(G − A) > 2 . Since G = i=1 (ai , bi ), by continuity
Un
of measure, there exists n such that m(G\ i=1 (ai , bi )) < 2 . By relabeling (ai , bi ) for
i = 1, . . . , n, we can construct a step function q such that
Z
Z
Z
|ϕ − q|dm ≤ |IA − IG |dm + |IG − q|dm
= m(G − A) + m(G\ ]ni=1 (ai , bi ))
< .
Now, let ϕ =
Pn
k=1 bk IAk .
For each k, there exists step function qk such that
Z
.
|IAk − qk |dm < Pn
k=1 |bk |
Then,
Z
|ϕ −
n
X
bk qk |dm =
Z X
n
k=1
|bk ||IAk − qk |dm
k=1
≤
n
X
k=1
8
Z
|bk |
|IAk − qk |dm
< .
P
And we can construct a step function from
(iii) For given , pick sufficiently small positive δ
k bk qk .
< Pk |c | .
j
j=1
Define a continuous function r
as follows.
On x ∈ [aj−1 + δ, aj − δ], r(x) = cj .
On x ∈ [aj − δ, aj ], connect (aj − δ, cj ) and (aj , 0) continuously.
On x ∈ [aj , aj + δ], connect (aj , 0) and (aj + δ, cj+1 ) continuously.
Then,
Z
|q − r|dµ < δ
n
X
|cj | < .
j=1
Exercise 1.4.4. Suppose g is a step function. Then,
Z aj
Z
k
X
cj
cos nxdx
g(x) cos nxdx =
aj−1
j=1
=
≤
k
X
1
cj (sin(naj ) − sin(naj−1 ))
n
j=1
k
X
2|cj |
j=1
n
→ 0.
For integrable g, previous exercise says there exists a step function q such that
R
|g − q|dx < ,
for given > 0. Then,
Z
Z
|g(x) cos nx|dx = |g(x) cos nx − q(x) cos nx + q(x) cos nx|dx
Z
Z
≤ |g − q|| cos nx|dx + |q(x) cos nx|dx
Z
Z
≤ |g − q|dx + |q(x) cos nx|dx
Z
< + |q(x) cos nx|dx → .
Since q is a step function, it goes to 0 as n increases. Letting → 0, we get the desired result.
1.5. Properties of Integral.
Exercise 1.5.1. Our claim is that if ||g||∞ = M , then |g| ≤ M a.e. Suppose not. Then,
µ({x : |g(x)| > M }) = µ(∪∞
n=1 {x : |g(x)| > M +
Thus, there exists N ∈ N such that µ({x : |g(x)| > M +
1
N })
1
}) > 0.
n
> 0, which contradicts on
definition of ||g||∞ . Therefore, |f g| ≤ |f |||g||∞ a.e. By integrating, we get the desired result.
Exercise 1.5.2. As shown in the previous exercise, we know that |f | ≤ ||f ||∞ a.e. Then,
R
|f |p dµ ≤ ||f ||p∞ , and thus lim supp ||f ||p ≤ ||f ||∞ . For the converse, let B = {x : |f (x)| >
9
||f ||∞ − }. Then, for sufficiently small > 0, 0 < µ(B ≤ 1. Note that
Z
Z
p
|f (x)| dµ ≥
|f (x)|p dµ
B
Z
>
(||f ||∞ − )p dµ
B
= µ(B (||f ||∞ − )p
Letting p → ∞ and letting → 0+ give us lim inf p ||f ||p ≥ ||f ||∞
Exercise 1.5.3.
(i)
Z
Z
||f + g||pp = |f + g|p dµ ≤ 2p (|f |p + |g|p )dµ = 2p ||f ||pp + 2p ||g||pp < ∞.
Thus, ||f + g||p < ∞. If ||f + g||p = 0, then clearly ||f + g||p ≤ ||f ||p + ||g||p . Suppose
not. Applying Holder Inequality to |f ||f + g|p−1 and |g||f + g|p−1 with p, q =
Z
|f ||f + g|p−1 dµ ≤ ||f ||p
Z
p
p−1 ,
p−1
p
p
(|f + g|p−1 ) p−1 dµ
= ||f ||p ||f + g||pp−1
p−1
Z
Z
p
p
|g||f + g|p−1 dµ ≤ ||g||p
(|f + g|p−1 ) p−1 dµ
= ||g||p ||f + g||p−1
p
Then,
||f + g||pp =
Z
Z
≤
|f + g|p dµ
(|f | + |g|)|f + g|p−1 dµ
≤ (||f ||p + ||g||p )||f + g||pp−1 .
Since ||f + g||p < ∞, we get ||f + g||p ≤ ||f ||p + ||g||p .
(ii) (p = 1) |f + g| ≤ |f | + |g|
(p = ∞) Let ||f ||∞ = M1 and ||g||∞ = M2 . Since
{x : |f (x)| + |g(x)| > M1 + M2 } ⊂ {x : |f (x)| + |g(x)| > M1 + M2 }
⊂ {x : |f (x)| > M1 } ∪ {x : |g(x)| > M2 },
µ({x : |f (x)| + |g(x)| > M1 + M2 }) ≤ 0. Thus, ||f + g||∞ ≤ M1 + M2 .
Pn
n
Exercise 1.5.4. Let fn :=
m=0 f 1Em = f 1]m=0 Em . Then, fn → f 1E and |fn | ≤ f :
integrable. By DCT, we get the desired result.
Exercise 1.5.5. Let fn := gn + g1− . Since gn ↑ g, −gn− ≥ −g1− . Thus, fn = gn+ − gn− + g1− ≥ 0.
By MCT,
Z
Z
fn dµ =
Z
gn dµ +
g1− dµ →
10
Z
Z
gdµ +
g1− dµ.
Since
R
g1− dµ < ∞, by substracting, we get the desired result.
Exercise 1.5.6. MCT
Exercise 1.5.7.
(i) MCT
(ii) Step 1 g ≥ 0.
By (i), for given , there exists N such that
R
R
gdµ < (g ∧ N )dµ + 2 . Pick δ =
2N .
Then, for A with µ(A) < δ,
Z
Z
gdµ < (g ∧ N )dµ + ≤ N µ(A) + < .
2
2
A
A
Step 2 g : integrable.
|g| = g + + g − . For given , there exists N such that
Z
Z
+
g dµ < (g + ∧ N )dµ +
4
Z
Z
−
−
g dµ < (g ∧ N )dµ + .
4
Pick δ =
4N .
Then, for A with µ(A) < δ,
Z
Z
Z
+
|g|dµ =
g dµ +
g − dµ
A
A
A
Z
Z
< (g + ∧ N )dµ + (g − ∧ N )dµ +
2
A
A
≤ 2N µ(A) + < .
2
Exercise 1.5.8. Let c ∈ (a, b). Since f is integrable, by previous exercise(1.5.7), for given
R
> 0, there exists δ > 0 such that if µ(A) < δ, then A |f |dµ < . For |x − c| < δ,
Z x
Z x
|g(x) − g(c)| =
f (y)dy ≤
|f (y)|dy < .
c
c
Since c is arbitrary in (a, b), g is continuous on (a, b).
Exercise 1.5.9.
(i) (1 ≤ p < ∞)
Step 1 f ≥ 0.
There exists (ϕn )n∈N such that 0 ≤ ϕn ≤ f and ϕn ↑ f . Let gn = |f − ϕn |p . gn → 0 and
R
R
|gn | ≤ (|f | + |ϕn |)p ≤ 2p |f |p : integrable. By DCT, gn dµ = |f − ϕn |p → 0. Thus,
||ϕn − f ||p → 0.
Step 2 Integrable f = f + − f − .
−
There exist (ϕ+
n )n∈N , (ϕn )n∈N such that
+
+
0 ≤ ϕ+
and ϕ+
n ≤f
n ↑f
−
−
0 ≤ ϕ−
and ϕ−
n ≤f
n ↑f .
−
For ϕn ≡ ϕ+
n − ϕn ,
+
−
−
||ϕn − f ||p ≤ ||ϕ+
n − f ||p + ||ϕn − f ||p → 0.
11
R P∞
P∞ R
Pm
Exercise 1.5.10. By MCT,
n=1 |fn |dµ =
n=1 |fn |dµ < ∞. For gm ≡
n=1 fn ,
P∞
Pm
P∞
gm → g ≡ n=1 fn and |gm | ≤ n=1 |fn | ≤ n=1 |fn | : integrable. By DCT,
Z
Z X
Z
Z X
m
m Z
∞
X
gm dµ =
fn dµ =
fn dµ → gdµ =
fn dµ.
n=1
n=1
n=1
1.6. Expected Value.
Exercise 1.6.1. For a support line l(x) = a(x − x0 ) + ϕ(x0 ) at x0 of ϕ(x) (l(x) ≤ ϕ(x)),
strict convexity implies that (∗) l(x) < ϕ(x) if x 6= x0 . For x0 = EX, Eϕ(X) − El(X) =
Eϕ(X) − l(EX) = Eϕ(X) − ϕ(EX) = 0 implies ϕ(X) = l(X) a.s. since ϕ(X) ≥ l(X). Thus,
X = EX a.s.
(∗) : Suppose l(y) = ϕ(y) for y 6= x0 . Then, with λ = 21 ,
x
y 1
1
0
ϕ
+
> ϕ(x0 ) + ϕ(y)
2
2
2
2
1
1
= l(x0 ) + l(y)
2
2
x
y
0
=l
+
2
2
which contradicts that l(x) is a support line of ϕ(x). The last equality comes from the linearity
of l(x).
Exercise 1.6.2. Let l(x1 , . . . , xn ) = ϕ(EX1 , . . . , EXn )+gx> (x1 −EX1 , . . . , xn −EXn ), where
gx ∈ ∂f (x). Then, ϕ(x) ≥ l(x) for any x = (x1 , . . . , xn ) ∈ Rn . Taking expectation gives us
the desired result.
Eϕ(X) ≥ El(X) = ϕ(EX).
Exercise 1.6.3.
(i) Let ϕ(x) = x2 and A = (−∞, −a] ∪ [a, ∞). Then, it is clear that
iA = inf{ϕ(y) : y ∈ A} = a2 . By Chebyshev’s inequality, we have
b2
.
a2
If some random variable X attains equality above, then X has to satisfy following
a2 P (|X| ≥ a) ≤ EX 2 = b2 ⇒ P (|X| ≥ a) ≤
identities.
b2
dP = 2 ,
a
A
Z
b2
dP = 1 − 2 ,
a
Ac
Z
It is clear that X with P (X = a) =
b2
a2 ,
Z
x2 dP = b2 .
2
P (X = 0) = 1− ab 2 satisfies identities. Therefore,
inequality is sharp.
(ii) For a > 0, it’s clear that if x ∈ {x : |x| ≥ a}, then x2 ≥ a2 . Thus,
a2 1(|X|≥a) ≤ X 2 1(|X|≥a) = X 2 − X 2 1(|X|<a) .
Since EX 2 is finite, both functions on each sides are integrable. Thus, linearity of
integration and order-preserving property of integration give us
Z
Z
Z
a2 1(|X|≥a) dP ≤ X 2 dP − X 2 1(|X|<a) dP.
12
We know that LHS = a2 P (|X| ≥ a). Let’s focus on RHS. The first term in RHS is
clearly EX 2 . For the second term, we can apply MCT. (It is clear that conditions of
MCT hold.) By MCT,
Z
lim a2 P (|X| ≥ a) ≤ EX 2 − lim
X 2 1(|X|<a) dP
a→∞
a→∞
Z
= EX 2 −
lim X 2 1(|X|<a) dP
a→∞
= EX 2 − EX 2 = 0
The last equality holds since EX 2 < ∞. Therefore, we have
lim a2 P (|X| ≥ a)/EX 2 = 0.
a→∞
However, Chebyshev’s inequality with ϕ(x) = x2 and A = (−∞, −a] ∪ [a, ∞), so that
iA = a2 , gives us following inequality.
a2 P (|X| ≥ a)/EX 2 ≤ 1,
which is not sharp.
Exercise 1.6.4.
(i) Let A = [a, ∞), ϕ(y) = (y + b)2 where 0 < b < a. It is clear that
iA = inf{ϕ(y) : y ∈ A} = (a + b)2 . Then by Chebyshev’s inequality,
(a + b)2 P (Y ≥ a) ≤ E[ϕ(Y )] = E[Y 2 + 2bY + b2 ] = E[X 2 + 2bX + b2 ] = E[ϕ(X)].
The second equality holds since EY = EX, V ar(Y ) = V ar(X). Note that
E[(X + b)2 ] = p(a + b)2 .
Thus,
P (Y ≥ a) ≤ p.
If Y = X, then equality holds.
(ii) Let p =
σ2
(a2 +σ 2 ) ,
b=
p
1−p a,
A = [a, ∞), and ϕ(y) = (y + b)2 . Note that 0 < p < 1 and
b > 0. Then, it is clear that iA = (a + b)2 . By Chebyshev’s inequality,
(a + b)2 P (Y ≥ a) ≤ E[ϕ(Y )] = E[Y 2 + 2bY + b2 ] = σ 2 + b2 ,
since E(Y ) = 0, V ar(Y ) = E(Y 2 ) − E(Y )2 = E(Y 2 )σ 2 . Thus, we have
P (Y ≥ a) ≤
Since b =
p
1−p a
=
σ2
a ,
σ 2 + b2
.
(a + b)2
we have
4
σ 2 + σa2
σ 2 + b2
P (Y ≥ a) ≤
=
=
2
(a + b)2
(a + σa )2
σ2
2
2
a2 (a + σ )
1
2
2 2
a2 (a + σ )
=
σ2
.
(a2 + σ 2 )
And equality holds when Y has P (Y = a) = p and P (Y = −b) = 1 − p, which satisfies
E(Y ) = ap − b(1 − p) = 0
13
V ar(Y ) = E[(Y − 0)2 ] = a2 p + b2 (1 − p) =
Exercise 1.6.5.
P (|X| > ) ≤
σ4
a2 σ 2
+ 2
= σ2 .
2
+σ
a + σ2
a2
(i) It suffices to show that ∀ n, ∃ X such that EX = 0, V ar(X) = 1 and
1
n.
Construct X as follows.

1 − 1 ,
n
P (X = x) =
1,
if x = 0
√
if x = ± n
2n
Then, EX = 0, V ar(X) = 1 and P (|X| > ) ≤ P (X 6= 0) =
1
n.
(ii) Similarly,

1 − 1 ,
n
P (X = x) =
1,
2n
if x = 1
if x = 1 ±
√
nσ
Then, EX = 1, V ar(X) = σ 2 and P (|X| > y) ≤ P (X 6= 1) =
1
n.
Exercise 1.6.6. Let f = Y and g = I(Y >0) . Then,
E|f g| = E|Y I(Y >0) = EY I(Y >0) = EY
E|f |2 = EY 2
E|g|2 = EI(Y >0) = P (Y > 0).
By Cauchy-Schwarz Ineq., (E|f g|)2 ≤ E|f |2 E|g|2 . Thus, (EY )2 ≤ EY 2 P (Y > 0).
Exercise 1.6.7. Checking (i), (ii) in Theorem 1.6.8 is obvious. For (iii),
Z
2
Eg(Xn ) = |Xn | α dm
Z
1
1 dm
= n2 I( n+1
,n
)
=
n
< 1 for all n.
n+1
Thus, Theorem 1.6.8 can be applied.
Suppose that Y ≥ |Xn | = Xn . Then,
Z
∞
X
Y dm ≥
nα
=
n=1
∞
X
1
n(n + 1)
nα−1
=∞
n+1
n=1
The last equality holds since 0 < α−1 < 1. Therefore, Xn are not dominated by an integrable
function.
Exercise 1.6.8. Step 1 g : Indicator function.
Let g(x) = IA (x) where A ∈ B(R). Then,
Z
Z
g(x)µ(dx) = IA (x)µ(dx)
14
Z
= µ(A)
Z
g(x)f (x)dx = IA (x)f (x)d(x)
Z
=
f (x)dx
A
= µ(A).
Step 2 g : simple function.
Linearity of Integral.
Step 3 g ≥ 0.
There exists non-negative simple function sn ↑ g. Then,
Z
Z
g(x)µ(dx) =
lim sn (x)µ(dx)
n→∞
Z
(MCT)
= lim
sn (x)µ(dx)
n→∞
Z
Step 2
= lim
sn (x)f (x)dx
n→∞
Z
(MCT)
=
g(x)f (x)dx.
Step 4 g : Integrable.
g = g + − g − . By Step 3,
R
g(x)µ(dx) =
R
g(x)f (x)dx.
Exercise 1.6.9. Induction.
Exercise 1.6.10. Induction.
(i) Let A = (|X| > 1). Then, |X|j < |X|k for 0 < j < k on A.
Z
Z
Z
|X|j dP =
|X|j dP +
|X|j dP
c
A
ZA
k
≤
|X| dP + 1
A
Z
≤ |X|k dP + 1 < ∞.
Exercise 1.6.11.
k
(ii) Let ϕ(x) = x j I(x≥0) . It is easy to check that ϕ is convex. Let Y = |X|j . Then,
|EY | = E|X|j < ∞(by (i)) and |Eϕ(Y )| = E|X|k < ∞. By Jensen’s Ineq.,
k
E|X j | j = ϕ(EY ) ≤ Eϕ(Y ) = E|X|k .
j
Thus, we get E|X|j ≤ (E|X|k ) k .
Exercise 1.6.12. P eX = ym = P (X = log ym ) = p(m).
EeX =
n
X
p(m)ym
m=1
15
e
EX
n
X
= exp
!
p(m) log ym
m=1
p(m)
= exp log Πnm=1 ym
p(m)
= Πnm=1 ym
.
By Jensen’s Ineq., EeX ≥ exp(EX). And, we get the desired result.
Exercise 1.6.13. Exercise 1.5.5
Exercise 1.6.14.
(i)
E(1/X; X > y) = E(1/X; 1/X < 1/y) ≤
1
1
P (1/X < 1/y) = P (X > y).
y
y
Thus, limy→∞ yE(1/X; X > y) ≤ limy→∞ P (X > y) = 0.
(ii) Let 0 < y < .
yE(1/X; X > y) = E(y/X; X > y and 0 ≤ X < ) + E(y/X; X > y and ≤ X).
≤ P (X > y, 0 ≤ X < ) + E(y/X; ≤ X).
≤ P (0 < X < ) + E(y/X; ≤ X).
For the second term on right hand side,
y
X I≤X
≤ y I≤X ≤ 1 and
y
X I≤X
→ 0 as y → 0.
Thus, by DCT,
lim sup yE(1/X; X > y) ≤ P (0 < X < ).
y→0
Letting → 0, we get the desired result.
Exercise 1.6.15. Exercise 1.5.6
Exercise 1.6.16. Let Xk = XIUk
n=0
An .
Then, Xk → XIA and |Xk | ≤ |X| : integrable. By
DCT, EXk → E(X; A). Note that
Z
k Z
X
EXk = U
XdP =
k
n=0
An
Thus, E(X; A) = limk→∞ EXk =
n=0
P∞
n=0
XdP =
An
k
X
E(X; An ).
n=0
EXk .
1.7. Product Measures, Fubini’s Theorem.
Exercise 1.7.1. Since |f (x, y)| ≥ 0, by Fubini’s Theorem,
R
X×Y
|f |dµ =
R R
X
Y
|f (x, y)|µ2 (dy)µ1 (dx) <
∞. And by Fubini’s Theorem again, we get the desired result.
Exercise 1.7.2. Construct a probability space (Y, B, ν) = ([0, ∞), B([0, ∞)), λ), where λ
denotes a Lebesgue measure. Let f (x, y) = I{(x,y) : 0≤y<g(x)} . Then,
Z Z
Z
f dλdµ =
gdµ
X Y
X
Z
f d(µ × λ) = (µ × λ)({(x, y) : 0 ≤ y < g(x)})
X×Y
16
Z Z
∞
Z
f dµdλ =
Y
µ({x : g(x) > y}dy.
X
0
Since f ≥ 0, Fubini’s Theorem implies that
Z Z
Z
Z Z
f dλdµ =
f d(µ × λ) =
f dµdλ.
X
Y
X×Y
Y
X
And we get the desired result.
ExerciseZ1.7.3. (i) Fubini’s
Z Theorem with f (x, y) = I{(x,y):a<x≤y≤b} ≥ 0.
(ii)
F (y)dG(y) +
G(x)dF (x)
(a,b]
(a,b]
Z
Z
(a,b]
(i)
=
Z
{F (y) − F (a)}dG(y) +
=
{G(x) − G(a)}dF (x) +
(a,b]
Z
Z
F (a)dG(y) +
(a,b]
G(a)dF (x)
(a,b]
Z
I{(x,y):a<x≤y≤b} d(µ × ν) +
I{(x,y):a<y≤x≤b} d(µ × ν) + F (a)(G(b) − G(a)) + G(a)(F (b) − F (a))
= (µ × ν)((a, b] × (a, b]) + (µ × ν)({(x, y) : a < x = y ≤ b} + F (a)G(b) + F (b)G(a) − 2F (a)G(a)
X
= (F (b) − F (a))(G(b) − G(a)) +
µ({x})ν({x}) + F (a)G(b) + F (b)G(a) − 2F (a)G(a)
x∈(a,b]
= F (b)(G(b) − F (a)G(a) +
X
µ({x})ν({x}).
x∈(a,b]
(iii) If F=G, then the third term in right hand side of (ii) equals zero. Thus,
Z
2F (y)dF (y) = F 2 (b) − F 2 (a).
(a,b]
Exercise 1.7.4. Let f = I{(x,y):x<y≤x+c} . Then,
Z Z
Z
f dµ(y)dx = (F (x + c) − F (x))dx
ZR ZR
ZR
f dxdµ(y) =
cdµ(y)
R
R
R
= cµ(R).
Since f ≥ 0, Fubini’s Theorem gives us that
Z Z
Z Z
f dµ(y)dx =
f dxdµ
R
R
R
R
a
∞
and we get the desired result.
Exercise 1.7.5.
Z
0
a
Z
∞
|e−xy sin x|dydx =
0
Z
0
Z
0
≤
| sin x|
dx
x
a
1dx < ∞.
0
17
e−xy | sin x|dydx
0
a
=
Z
Z
By Exercise 1.7.1, e−xy sin xI{(x,y):0<x<a,0<y} is integrable and we can apply Fubini’s Theorem.
Z
0
a
sin x
dx =
x
Z
a
∞
Z
e−xy sin xdydx =
0
0
∞
Z
e−xy sin xdxdy
0
0
∞
Z
a
Z
=
0
e−xy (−y sin x − cos x)
1 + y2
a
dy
0
∞
1 − e−ay (y sin a + cos a)
dy
1 + y2
0
Z ∞ −ay
Z ∞ −ay
ye
π
e
= − cos a
dy − sin a
dy.
2
2
1
+
y
1
+ y2
0
0
R 1
−1
The last equality comes from the fact that 1+y
(y). Moreover,
2 dy = tan
Z ∞ −ay
Z a
Z ∞ −ay
e
sin x
π
ye
= cos a
dx −
dy + sin a
dy
2
x
2
1
+
y
1
+ y2
0
0
0
Z ∞
| cos a| + y| sin a|
≤
e−ay
dy
1 + y2
0
Z ∞
1+y
≤
e−ay
dy
1 + y2
0
Z ∞
2
≤
e−ay · 2dy = .
a
0
Z
=
Letting a → ∞, we get
Z
∞
0
Note that since
sin x
x
π
sin x
dx = .
x
2
is not integrable on R≥0 , we can’t calculate its integration directly.
2. Laws of Large Numbers
2.1. Independence.
Exercise 2.1.1. By Thm 2.1.8, it suffices to show that P (X1 ≤ z1 , . . . , Xn ≤ zn ) =
R∞
Πni=1 P (Xi ≤ zi ). Let ci = −∞ gi (xi )dxi , for i = 1, . . . , n. Then, we have c1 · · · cn = 1.
Our claim is that
P (Xi ≤ zi ) =
1
ci
Z
zi
gi (xi )dxi .
−∞
WLOG, let i = 1.
P (X1 ≤ z1 ) = P (−∞ < X1 ≤ z1 , −∞ < X2 < ∞, . . . , −∞ < Xn < ∞)
Z z1
= (c2 · · · cn )
g1 (x1 )dx1
−∞
1
=
c1
Z
z1
g1 (x1 )dx1 .
−∞
Now,
Z
z1
P (X1 ≤ z1 , . . . , Xn ≤ zn ) =
Z
g1 (x1 )dx1 · · ·
−∞
zn
−∞
18
gn (xn )dxn
Z zn
Z
1
1 z1
g1 (x1 )dx1 · · ·
gn (xn )dxn
c1 −∞
cn −∞
n
Y
=
P (Xi ≤ zi )
=
i=1
Exercise 2.1.2. Let Ai =
{Xi−1 (xi )
: xi ∈ Si } ∪ {∅, Ω}. Let I = {n1 , . . . , nk } ⊂ {1, . . . , n}
and J = {1, . . . , n} \I = {m1 , . . . , mn−k }. Then,
X
P (Xn1 = xn1 , · · · , Xnk = xnk ) =
···
xm1 ∈Sm1
=
Y
X
P (X1 = x1 , · · · , Xn = xn )
xmn−1 ∈Smn−k
P (Xi = xi ).
i∈I
By Lemma 2.1.5, Ai ’s are independent. By Theorem 2.1.7, σ(Ai )’s are independent. Sine
Xi−1 (A) ∈ σ(Ai ) for all A ∈ 2Si , σ(Xi ) ⊂ σ(Ai ). Therefore, X1 , . . . , Xn are independent.
Exercise 2.1.3. (1) h(ρ(x, y)) → [0, ∞)
(i) h(ρ(x, y)) = 0 ⇐⇒ ρ(x, y) = 0 ⇐⇒ x = y.
First ⇐⇒ holds since h(0) = 0 and h0 (x) > 0 for x > 0.
(ii) h(ρ(x, y)) = h(ρ(y, x)) by symmetry of metric ρ.
Z ρ(x,z)
(iii) h(ρ(x, z)) =
h0 (u)du
0
Z
ρ(x,y)+ρ(y,z)
h0 (u)du
≤
0
Z
≤
ρ(x,y)
0
Z
ρ(y,z)
h0 (u)du
h (u)du +
0
0
= h(ρ(x, y)) + h(ρ(y, z)).
The first inequality holds since ρ is metric. The second inequality holds since h0 is
decreasing.
(2) By simple calculus, one can show that h(x) =
x
x+1
satisfies (1).
Exercise 2.1.4. It is easy to check the uncorrelatedness. To show that Xn ’s are not independent, it suffices to show that there exist A and B such that P (Xn ∈ A, Xm ∈ B) 6= P (Xn ∈
A)P (Xm ∈ B) for 
different m and n.
A = X −1 (0, )
n
For 0 < < 1, let
. Then, A ⊂ B. Note that we have 2n intervals in
B = X −1 X (A)
m
m
1
. This is possible since
A. Let l be the length of one interval. Now, pick so that 2nl < m
P (A) & 0 as & 0. Then, it is clear that P (B) < 1. Thus, P (A ∩ B) = P (A) > P (A)P (B).
Therefore, Xn and Xm are not independent.
Exercise 2.1.5.
(i) Let Dµ be a set of discontinuities of µ. By Exercise 1.2.3, Dµ is at
most countable.
Z Z
P (X + Y = 0) =
1(x+y=0) dµdν
19
Z
=
µ({−y})dν
Z
Z
=
µ({−y})dν +
µ({−y})dν.
c
Dµ
Dµ
Note that for −y ∈ Dµc , µ({−y}) = P (X ≤ −y) − P (X < −y) = 0. Thus,
Z
P (X + Y = 0) =
µ({−y})1Dµ (y)dν
R
X
=
µ({−y})ν({y}).
y∈Dµ
The second equality holds by Exercise 1.5.6.
(ii) With Y = −Y in (i), P (X = Y ) = 0 since Dµ = ∅.
Exercise 2.1.6. Note that
σ(f (X)) = {X −1 (f −1 (B)) : B ∈ B(R)}
σ(f (Y )) = {Y −1 (g −1 (B)) : B ∈ B(R)}
Since f and g are measurable, f −1 (B) and g −1 (B) are in B(R) for any B ∈ B(R). Thus,
σ(f (X)) ⊂ σ(X) and σ(g(Y )) ⊂ σ(Y ). Therefore, independence of X and Y implies the
independence of f (X) and g(Y ).
Exercise 2.1.7. By Exercise 2.1.2, it suffices to show that P (Zi = α, Zj = β) = P (Zi =
α)P (Zj = β) for all α, β ∈ {0, 1, . . . , K − 1}.
Firstly, for given Y = y, P (Zi = α|Y = y) =
P (Zi = α) =
K−1
X
1
K
since K is a prime. Thus,
P (Zi = α|Y = y)P (Y = y)
y=0
1
1
= .
2
K
K
Now, for given Y = y, Zi = α and Zj = β means that X + iy ≡ α and X + jY ≡ β. That is,
=K·
(i − j)y ≡ α − β (mod K).
Since gcd(i − j, K) = 1, there exists unique y0 ∈ {0, 1, . . . , K − 1} satisfying Zi = α and
Zj = β. And, with y0 , x0 ∈ {0, 1, . . . , K − 1} uniquely determined. Thus,
P (Zi = α, Zj = β) =
1
.
K2
Therefore,
P (Zi = α, Zj = β) =
1
1 1
=
·
= P (Zi = α)P (Zj = β) for all α, β ∈ {0, 1, . . . , K − 1} .
2
K
K K
i.i.d
Exercise 2.1.8. X1 , X2 , X3 ∼ 21 1{1} + 12 1{−1} and X4 = X1 X2 X3 .
n
o
n
o
Exercise 2.1.9. A1 = {1, 2}, Ω and A2 = {1, 3}, {1, 4}, {2, 3}, {2, 4}, Ω
20
Exercise 2.1.10.
P (X + Y = n) =
X
P (X = m, Y = n − m) =
m
X
P (X = m)P (Y = n − m)
m
The last equality holds by the independence.
Exercise 2.1.11. Omit.
Exercise 2.1.12. Omit.
Exercise 2.1.13. (a) F (z) = 91 1[0,1) (z) + 92 1[1,2) (z) + 31 1[2,3) (z) + 29 1[3,4) (z) + 19 1[4,∞) (z)
(b) For some c ∈ 0, 29
0
0
1
2
2
9
1
2
1
9
c
−c
1
9
c
−c
1
9
c
2
9
2
9
−c
Exercise 2.1.14.
Z Z
P (XY ≤ z) =
1(xy≤z) dF dG
Z Z
1(x≤z/y,y6=0) + 1(0≤z,y=0) dF dG
Z
z
=
F
dG + P (Y = 0)I(z ≥ 0).
y
R\{0}
=
Exercise 2.1.15. For simplicity, let Ω = [0, 1). We have

0, if ω ∈ 2m , 2m+1 : m = 0, 1, . . . , 2n−1 − 1
2n
2n
Yn (ω) =
1, if ω ∈ 2m+1 , 2m+2 : m = 0, 1, . . . , 2n−1 − 1
2n
.
2n
By construction, it is obvious that P (Yn = 0) = P (Yn = 1) =
1
2.
To show that Y1 , Y2 , . . .
are independent, we have to show that Y1 , . . . , Yn are independent for all n ∈ N. By Exercise
Qn
2.1.2, it suffices to show that P (Y1 = y1 , . . . , Yn = yn ) = i=1 P (Yi = yi ), where yi ∈ {0, 1}.
By induction, one can show that {ω : Y1 (ω) = y1 , . . . , Yn (ω) = yn } = 2ln , l+1
for some l ∈
2n
Qn
1
n
{0, 1, . . . , 2 − 1} for any yi ’s. Therefore, P (Y1 = y1 , . . . , Yn = yn ) = 2n = i=1 P (Yi = yi ),
where yi ∈ {0, 1}.
2.2. Weak Laws of Large Numbers.
Exercise 2.2.1. We assume that EXi2 < ∞ so that Var(Xi ) < ∞ for all i. Since L2
convergence implies convergence in probability, it suffices to show that
E|Sn /n − νn |2 → 0 as n → ∞.
Note that
E|Sn /n − νn |2 =
1
E|Sn − ESn |2
n2
21
1
Var(Sn )
n2
n
1 X
= 2
Var(Xi )
n i=1
=
n
≤
1 X Var(Xi )
n i=1
i
n)
The last equality holds since Xi ’s are uncorrelated. Now, fix > 0. Since Var(X
→ 0, there
PN Var(Xi ) n
Var(Xn )
exists N ∈ N such that
< for all n > N . Let M = i=1
< ∞. Then, for
n
i
n > N,
!
n
n
X
1
1 X Var(Xi )
<
M+
n i=1
i
n
i=N +1
1
≤ ·M +
n
By taking limsup and letting → 0, we get
E|Sn /n − νn |2 → 0,
which is the desired result.
Exercise 2.2.2. We assume that r(0) is finite. Let Sn = X1 + · · · + Xn . Since ESn = 0,
" #
2
Sn
Sn
E
= Var
n
n
1 X
EXi Xj
= 2
n
1≤i,j≤n
n
n−1
n−2
2
X
X
X
1 X
≤ 2
EXi2 + 2
EXi Xi+1 + 2
EXi Xi+2 + · · · + 2
EXi Xi+n−2 + 2EX1 Xn
n i=1
i=1
i=1
i=1
Now, since r(k) → 0 as k → ∞, for given > 0, ∃ N such that for n ≥ N , |r(n)| < . Moreover,
since EXi2 ≤ r(0) < ∞, by Cauchy-Schwartz Inequality, E|Xi Xj | ≤ (EXi2 )1/2 (EXj2 )1/2 ≤
r(0) < ∞. Then, for n ≥ N + 1, we have
" #
2
Sn
E
n
n
n−1
n−N
n−N
X
X+1
X
1 X
2
≤ 2
EXi + 2
E|Xi Xi+1 | + · · · + 2
E|Xi Xi+N −1 | + 2
EXi Xi+N + · · · + 2EX1 Xn
n i=1
i=1
i=1
i=1
≤
n−N
n−N
X
X−1
1
nr(0)
+
2(n
−
1)r(0)
+
·
·
·
+
2(n
−
N
+
1)r(0)
+
2
r(N
)
+
2
r(N + 1) + · · · + 2r(n − 1)
2
n
i=1
i=1
≤
n−N
n−N
X
X−1
1
2N
nr(0)
+
2
|r(N
)|
+
2
|r(N + 1)| + · · · + 2|r(n − 1)|
2
n
i=1
i=1
22
≤
n−N
X
1
2N
nr(0)
+
2
i
·
n2
i=1
≤
n
X
1
2N
nr(0)
+
2
i·
2
n
i=1
≤
2N r(0) n(n + 1)
+
n
n2
Since > 0 is arbitrary, E
h
Sn 2
n
i
→ 0, which means that Sn /n → 0 in L2 . Therefore,
Sn /n → 0 in probability, which is the desired result.
Exercise 2.2.3.
(i) Since Ui ’s are i.i.d. and f is a measurable function, f (Ui )’s are also
R1
i.i.d. Note that since Ui follows Uniform distribution on [0,1], Ef (U1 ) = 0 f (x)dx. By
R1
assumption, E|f (U1 )| = 0 |f (x)|dx < ∞. By WLLN,
Z 1
f (U1 ) + · · · + f (Un )
In =
→ Ef (U1 ) =
f (x)dx = I in prob.
n
0
R1
(ii) Since 0 |f (x)|2 dx < ∞,
Z 1
2
Z 1
2
2
2
Var(f (U1 )) = Ef (U1 ) − (Ef (U1 )) =
f (x) dx −
f (x)dx < ∞.
0
0
2
By Chebyshev’s inequality with ϕ(x) = x and A = {x ∈ R : |x| > a/n1/2 }, we have
iA = inf{ϕ(y) : y ∈ A} = a2 /n and
n
E(In − I)2
a2
n
= 2 Var(In )
a
1
= 2 Var(f (U1 ))
a
"Z
Z 1
2 #
1
1
2
= 2
f (x) dx −
f (x)dx
a
0
0
P (|In − I| > a/n1/2 ) ≤
Exercise 2.2.4. (1) E|Xi | = ∞.
∞
R∞ 1
P∞
P∞
1
Since k=2 k log
k=2
k ≥ 2 x log x dx = log(log x) 2 = ∞, E|Xi | =
C
k log k
= ∞.
(2) There exists µ < ∞ so that Sn /n → µ in probability.
Since
nP (|X1 | > n) ≤ n
∞
X
k=n+1
Z ∞
∞
X
n
C
n
C
C
C
≤
≤
dx =
,
2
2
2
k log k
log n
k
log n n x
log n
k=n+1
nP (|X1 | > n) → 0 as n → ∞. Also, there exists µ < ∞ such that
µn := E(X1 1(|X1 |≤n) ) =
∞
X
∞
(−1)k · k ·
k=2
X
C
C
=
(−1)k
→µ
k 2 log k
k log k
k=2
since it is alternating series. Therefore, by Theorem 2.2.12, Sn /n → µ in probability.
23
Exercise 2.2.5. By definition, Xi takes only positive values.
(1) There exists a sequence µn → ∞ such that Sn /n − µn → 0 in probability.
Firstly, for x ≥ 2, xP (|Xi | > x) = xP (Xi > x) =
2
log x
→ 0 as x → ∞. Let µn =
E(X1 1(|X1 |≤n) ). Then, for n ≥ 3,
Z n
e
µn =
dx = [e log(log x)]ne = e log(log n) → ∞.
x
log
x
e
Therefore, by Theorem 2.2.12, Sn /n − µn → 0 in probability.
(2) E|Xi | = ∞.
Since E|Xi | = EXi ≥ µn for all n, E|Xi | = ∞.
P∞
Exercise 2.2.6. (i) X = n=1 1(X≥n) .
∞
X
x2 P (X = x)
(ii) EX 2 =
x=1
= P (X = 1) + (1 + 3)P (X = 2) + · · · + (1 + 3 + · · · + (2n − 1))P (X = n) + · · ·
= P (X ≥ 1) + 3P (X ≥ 2) + · · · + (2n − 1)P (X ≥ n) + · · ·
=
∞
X
(2n − 1)P (X ≥ n).
n=1
Exercise 2.2.7.
Z
∞
Z
∞
Z
h(y)P (X ≥ y)dy =
−∞
h(y)1(X≥y) (ω)dP dy
−∞
Z Z
Ω
∞
=
h(y)1(X≥y) (ω)dP dy
Ω
−∞
Z
=
H(X)dP = EH(X).
Ω
The second equality holds by Fubini.
Exercise 2.2.8. Let’s use Theorem 2.2.11 in Textbook with Xn,k = Xk for k = 1, . . . , n and
bn = 2m(n) where m(n) = min m : 2−m m−3/2 ≥ n−1 . Let’s check the conditions.
(0) bn > 0 with bn → ∞.
By definition of m(n).
Pn
(i)
k=1 P (|Xn,k | > bk ) → 0.
n
X
∞
X
P (|Xn,k | > bk ) = nP (|X1 | > 2m(n) ) = n
k=1
pj
j=m(n)+1
=
∞
X
j=m(n)+1
n
≤
2j · j(j + 1)
∞
X
j=m(n)+1
1
n
= m(n)
≤ m(n)− 2 → 0
2
m(n)2
24
n
2j · m(n)2
(ii) b−2
n
Pn
k=1
2
E X̄n,k
→ 0.
b−2
n
n
X
2
E X̄n,k
=
k=1
n
E X12 1{|X1 |≤2m(n) }
22m(n)
m(n)3/2
E X12 1{|X1 |≤2m(n) }
m(n)
2


m(n)
2
m(n)3/2  X k
2 − 1 pk + p0  .
= m(n)
2
k=1
≤
(1)
Note that

m(n)
X

2k − 1
2

m(n)
pk + p0  ≤ 1 +
X
k=1
22k pk
k=1
m(n)
X
=1+
1
k(k + 1)
2k ·
k=1
X
=1+
1≤k<
m(n)
2
X
≤1+
1≤k<
=1+2
≤
2k ·
1
+
k(k + 1)
2k +
m(n)
2
m(n)
2 +1
2m(n)
·C
m(n)2
−2+
X
2k ·
m(n)
2 ≤k≤m(n)
4
m(n) · (m(n) + 2)
1
k(k + 1)
X
2k ·
m(n)
2 ≤k≤m(n)
m(n)
4
2m(n)+1 − 2 2 +1
m(n) · (m(n) + 2)
for large n.
Thus,
3
1
m(n) 2 2m(n)
·
· C = C · m(n)− 2 → 0
2
m(n)
m(n)
2
Pn
p
n
By (0)-(ii), Theorem 2.2.11 gives us that Snb−a
→ 0, where an = k=1 E X̄n,k .
n
(1) ≤
Now,
an = nE(X1 1{|X1 |≤2m(n) }


m(n)
X
= n
(2k − 1)pk − p0 
k=1

1
+
m(n) + 1

∞
X
1

2k · k(k + 1)
k=m(n)+1


∞
X
1
1
1
≤ n −
+
m(n) + 1 m(n)2
2k
= n −
k=m(n)+1
25
1
k(k + 1)
∼−
n
.
m(n)
Since n ≤ 2m(n) m(n)3/2 ≤ Cn, log n ∼ m(n). Thus,
n
an ∼
log n
bn = 2m(n) ∼ n(m(n))−3/2 ∼ n(log n)−3/2 .
Thus,
Sn + logn n p
Sn − an
∼
3 → 0.
bn
n(log n)− 2
1
Since (log n)− 2 → 0,
Sn + n(log n)−1 p
→ 0.
n(log n)−1
Finally,
Sn
p
→ 0.
n/ log n
Exercise 2.2.9.
2.3. Borel-Cantelli Lemmas.
Exercise 2.3.1. Since ∪k≥n Ak ⊃ Am for all m ≥ n, P (∪k≥n Ak ) ≥ supk≥n P (Ak ). Thus,
P (lim sup An ) = P (∩∞
n=1 ∪k≥n Ak ) = lim P (∪k≥n Ak ) ≥ lim sup P (Ak ) = lim sup P (An ).
n→∞
n→∞ k≥n
By applying above result to Bn = Acn , we get
P (lim inf An ) ≤ lim inf P (An ).
Exercise 2.3.2. > 0 given. Since f is uniformly continuous on [−M − 1, M + 1], there
exists δ ∈ (0, 1) such that if x, y ∈ [−M − 1, M + 1] and |x − y| < δ, then |f (x) − f (y)| < .
Then, |f (x) − f (y)| ≥ implies |x − y| ≥ δ or |x| ≥ M . Thus,
P (|f (Xn ) − f (X)| ≥ ) ≤ P (|Xn − X| ≥ δ) + P (|X| ≥ M ).
Since Xn → X in probability,
lim sup P (|f (Xn ) − f (X)| ≥ ) ≤ P (|X| ≥ M ).
n
By letting M → ∞, we get
lim P (|f (Xn ) − f (X)| ≥ ) = 0.
n
Exercise 2.3.3. We know that
P (ln = k) =


1
,
2k+1
1,
2n
k = 0, 1, . . . , n − 1
k = n.
(1) lim sup logln n = 1 a.s.
2
26
(i) lim sup logln n ≤ 1 a.s.
2
Fix > 0. Let An = {ln > (1 + ) log2 n} and m = b(1 + ) log2 nc. Then,
P (An ) = 1 − P (ln ≤ (1 + ) log2 n
m+1
n
X
1
1
=1−
=
2k+1
2
k=1
(1+) log2 n
1
= n−(1+) .
≤
2
P∞
Since 1 + > 1, we get n=1 P (An ) < ∞. By Borel-Cantelli Lemma 1,
ln
P (An i.o.) = P
> 1 + i.o. = 0,
log2 n
that is,
P
ln
≤ 1 + eventually
log2 n
= 1.
Since > 0 was arbitrarily chosen, we get lim sup logln n ≤ 1 a.s.
2
(ii) lim sup logln n ≥ 1 a.s.
2
Construct a sequence (rn )n∈N by r1 = 1, r2 = 2, r3 = rn−1 + [log2 n] for r ≥ 3. Let
Bn = {Xm = 1, for all m such that rn−1 ≤ m ≤ rn }. Then,
rn −rn−1 [log2 n] log2 n
1
1
1
1
=
≥
= .
P (Bn ) =
2
2
2
n
P∞
Since n=1 P (Bn ) = ∞ and Bn ’s are independent, by Borel-Cantelli Lemma 2,
P (Bn i.o.) = 1. Since Bn ⊂ {lrn ≥ [log2 n]}, P (lrn ≥ [log2 n] i.o.) = 1. Note
that rn ≤ n log2 o
n fornn ≥ 2. (It can be shown easily by induction.)
Then, since
o
n
lrn
lrn
[log2 n]
[log2 n]
[log2 n]
≥
⊂
≥
,
we
get
=
log2 rn
log2 rn
log2 rn
log2 (n log2 n)
log2 n+log2 log2 n
l rn
[log2 n]
P
≥
i.o. = 1.
log2 rn
log2 n + log2 log2 n
Since limn→∞
[log2 n]
log2 n+log2 log2 n
= 1,
lim sup
(2) lim inf
ln
log2 n
ln
≥ 1 a.s.
log2 n
= 0 a.s.
It suffices to show that P (ln = 0i.o.) = 1. Note that {ln = 0} = {Xn = 0} are independent
and P (Xn = 0) = 21 . Thus, by Borel-Cantelli Lemma 2, P (ln = 0 i.o.) = 1.
Exercise 2.3.4. Firstly, find a subsequence (Xn(m) )m∈N such that EXn(m) → lim inf EXn .
Since Xn → X in probability, there exists a further subsequence Xn(mk ) such that Xn(mk ) →
X a.s. by Theorem 2.3.2 in Textbook. Then, by Fatou’s Lemma,
EX = E(lim inf Xn(mk ) ) ≤ lim inf EXn(mk ) = lim inf EXn .
27
The inequality comes from Fatou’s Lemma.
Exercise 2.3.5. (a) For all subsequence Xnk , there exists a sub-subsequence Xm(nk ) converging almost surely to X. Since |Xm(nk ) | ≤ Y : integrable, by DCT, EXm(nk ) → EX.
This means that every subsequence (EXnk ) of (EXn ) has a sub-subsequence EXm(nk )
converging to EX. Therefore, EXn → EX.
(b) See Theorem 1.6.8.
Exercise 2.3.6. (a) Note that f (x) =
f (x) = 0 if and only if x = 0.
x
1+x
is non-decreasing, 0 ≤ f (x) ≤ 1 for all x ≥ 0 and
Firstly, d(X, Y ) = E(f (|X − Y |)) ≥ [0, ∞).
(i)
d(X, Y ) = E(f (|X − Y |)) = 0 ⇐⇒ f (|X − Y |) = 0 a.s.
⇐⇒ |X − Y | = 0 a.s.
(ii) Clear.
(iii) Since |X − Z| ≤ |X − Y | + |Y − Z|,
|X − Z|
= f (|X − Z|)
1 + |X − Z|
≤ f (|X − Y | + |Y − Z|)
|X − Y | + |Y − Z|
1 + |X − Y | + |Y − Z|
|X − Y |
|Y − Z|
=
+
1 + |X − Y | 1 + |Y − Z|
=
Therefore,
d(X, Z) = E(f (|X − Z|)) ≤ E(f (|X − Y |)) + E(f (|Y − Z|)) = d(X, Y ) + d(Y, Z).
(b) (⇒) Fix > 0. Let ϕ(x) =
A} =
x
1+x
≥ 0 and A = (, ∞). Then, we have iA = inf{ϕ(y) : y ∈
1+ .
By Chebyshev’s inequality, we get
1+
|Xn − X|
1+
P (|Xn − X| > ) ≤
·E
≤
· d(Xn , X) → 0.
1 + |Xn − X|
Thus, Xn → X in probability.
(⇐) For any > 0,
|Xn − X|
d(Xn , X) = E
1 + |Xn − X|
|Xn − X|
|Xn − X|
=E
; |Xn − X| > + E
; |Xn − X| ≤ 1 + |Xn − X|
1 + |Xn − X|
≤ E(1; |Xn − X| > ) + E
; |Xn − X| ≤ 1+
≤ P (|Xn − X| < ) +
.
1+
28
Since Xn → X in probability, lim supn d(Xn , X) =
1+ .
Since > 0 was arbitrarily
chosen, by letting → 0, we get limn d(Xn , X) = 0.
Exercise 2.3.7. By Theorem 2.3.2 in Textbook, it suffices to show that there exists a r.v. X∞
such that for any subsequence Xn(m) , there exists a further subsequence Xn(mk ) that converges
almost surely to X∞ . Since d(Xm , Xn ) → 0, there exists Nk ∈ N such that d(Xm , Xn ) < 2−k
if m, n > Nk . Now, fix a subsequence Xn(m) . Choose increasing subsequence (mk ) so that
y
n(mk ) ≥ Nk . Using Chebyshev’s Inequality with ϕ(y) = 1+y
and A = k12 , ∞ , we get
1
k2 + 1
.
P Xn(mk ) − Xn(mk−1 ) > 2 ≤
k
2k
P∞ 2
P∞
Since k=1 = P Xn(mk ) − Xn(mk−1 ) > k12 ≤ k=1 k 2+1
< ∞, by Borel-Cantelli Lemma
k
1,
1
Xn(mk ) − Xn(mk−1 ) > 2 i.o. = 0,
k
1
that is, P Xn(mk ) − Xn(mk−1 ) ≤ k2 eventually = 1. For ω ∈ Ω0 := Xn(mk ) − Xn(mk−1 ) ≤
Xn(mk ) (ω) k∈N is a Cauchy sequence in R. Thus, there exists X∞ (ω) := limk Xn(mk ) (ω).
P
Since P (Ω0 ) = 1, Xnm(k) → X∞ a.s.
It remains to check the well-definedness of X∞ . Consider other subsequence Xn0 (m0 ) such
0
0
) = 0. By Exercise
that Xn0 (m0k ) → X∞
. Since d(Xn(mk ) , Xn0 (m0k ) → 0, by DCT, d(X∞ , X∞
0
.
2.3.6, it’s equivalent to that X∞ = X∞
Exercise 2.3.8. Since P (∪n An ) = 1, P (∩n Acn ) = 0. Moreover, since An ’s are independent
events, so are Acn . And this implies that
!
∞
∞
\
Y
c
P
An =
P (Acn )
n=1
=
n=1
∞
Y
(1 − P (An )) = 0.
n=1
Since P (An ) < 1,
Qm−1
n=1
(1 − P (An )) > 0 for m = 2, 3, · · · . Thus, we have for all m ∈ N,
!
∞
\
c
P
An = 0.
n=m
Since
{Acn
eventually} =
S∞
m=1
P (Acn
T∞
c
n=m An ,
eventually) = P
∞ \
∞
[
!
Acn
m=1 n=m
≤
∞
X
m=1
P
∞
\
!
Acn
= 0.
n=m
Then, P (An i.o.) = 1. Thus, by Borel-Cantelli Lemma 1(Theorem 2.3.1),
29
P∞
n=1
P (An ) = ∞.
1
k2
eventually ,
(i) Let Bn = Acn ∩ An+1 . Then,
Exercise 2.3.9.
P (∪m≥n Am ) ≤ P (An ) +
∞
X
P (Bm ) → 0
as n → ∞.
m=n
Thus, P (An i.o.) = limn→ P (∪m≥n Am ) = 0.
(ii) An = 0, n1 with Lebesgue measure.
Exercise 2.3.10. 10
(i) For 0 < < 1, P (|Xn | > ) = pn → 0 ⇐⇒ pn → 0.
P∞
P∞
(ii) Let An = {|Xn | > }. Then, P (An i.o.) = 0 ⇐⇒
n=1 pn < ∞. Note
n=1 P (An ) =
Exercise 2.3.11.
that (⇒) holds since Xn ’s are independent.
Exercise 2.3.12. Let’s enumerate Ω = {ω1 , ω2 , · · · }. Let Ω0 = {ωi : P (ωi ) > 0}. Since
P (Ωc0 ) = 0, if one can show that Xn (ω) → X(ω) for ω ∈ Ω0 , then it’s done. Fix > 0. Since
Xn → X in probability, for each ωi ∈ Ω0 , there exists Ni ∈ N such that P (|Xn − X|) > ) <
1
2 P (ωi )
if n > Ni . Since 12 P (ωi ) < P (ωi ),
ωi ∈
/ {ω : |Xn − X| > for n > Ni } .
Thus, Xn (ωi ) → X(ωi ) as n → ∞.
P∞
1
n
>
Exercise 2.3.13. Pick cn such that P |Xn | > |cnn | ≤ 21n . Since n=1 P X
cn
n < ∞,
Xn
1
1
n
by Borel-Cantelli Lemma 1, P X
>
≤
i.o.
=
0.
That
is,
P
eventually
= 1.
cn
n
cn
n
Now, for given > 0, choose N ∈ N such that
1
N
< . Then, since {|Xn | ≤ |cn |} ⊃ {|Xn | ≤
|cn |/N },
Xn
1
Xn
≤ eventually ≥ P
≤
eventually = 1.
P
cn
cn
N
Since is arbitrarily chosen, Xn /cn → 0 a.s.
Exercise 2.3.14.
(⇒) Suppose
P
n
P (Xn > A) = ∞ for all A. Then, by Borel-Cantelli
Lemma 2, P (Xn > A i.o.) = 1 for all A. That is, sup Xn = ∞ a.s. which is the
contradiction.
P
(⇐) Since n P (Xn > A) < ∞ for some A, by Borel-Cantelli Lemma 1, P (Xn > A i.o.) =
0. Thus, sup Xn ≤ A < ∞ a.s.
Exercise 2.3.15.
Xn
=1
log
n
Xn
P log
n >
(i) lim supn→∞
a.s.
(≤) It suffices to show that
1 + i.o. = 0 for any > 0. By definition,
P Xn > (1 + ) log n = n−(1+) . Since 1 + > 1, we have
∞
∞
X
X
P Xn > (1 + ) log n =
n=1
n=1
1
n1+
< ∞.
By Borel-Cantelli Lemma 1, P Xn > (1 + ) log n i.o. = 0.
30
Xn
0
(≥) It suffices to show that P log
n > 1 i.o. = 1. Since Xn s are independent and
P∞
P∞ 1
n=1 P (Xn > log n) =
n=1 n = ∞, by Borel-Cantelli Lemma 2, we get
P (Xn > log n i.o.) = 1.
(ii) If one can show that
(a)
1 ≤ lim inf
n→∞
Mn
Mn (b)
≤ lim sup
≤ 1
log n
n→∞ log n
a.s.,
then it’s done.
Mn
(a) It suffices to show that P log
<
1
−
i.o.
= 0 for any > 0. Note that
n
Mn
P
<1− =P
max Xm < (1 − ) log n
1≤m≤n
log n
n
Y
=
P (Xm < (1 − ) log n)
m=1
=
1−
1
n
n1−
≤ e−n .
The second equality holds since Xi ’s are i.i.d. By L’Hopital’s rule, e−n / n12 → 0.
Thus, by comparison test,
∞
X
Mn
< 1 − < ∞.
log n
n=1
Mn
By Borel-Cantelli Lemma 1, P log
<
1
−
i.o.
= 0.
n
Mn
(b) It suffices to show that P log
n < 1 + eventually = 1 for any > 0.
P
Fix > 0. Let Ω0 = {ω : lim supn→∞
Xn (ω)
log n
= 1}.
(?) For ω ∈ Ω0 , there exists N1 (ω) ∈ N such that
Xn (ω)
log n
< 1 + if n > N1 (ω).
Now, for n > N1 (ω),
Mn (ω)
Xn (ω)
= max
1≤m≤n log n
log n
= max
max
Xn (ω)
Xm (ω)
,
max
1≤m≤N1 (ω) log n N1 (ω)<m≤n log n
- There exists N2 (ω) ∈ N such that max1≤m≤N1 (ω)
- By (?),
Xm (ω)
log n
≤
Xm (ω)
log m
Xm (ω)
log n
< 1 + for N1 (ω) < m ≤ n.
Thus, for n > N (ω) = max{N1 (ω), N2 (ω)}, we have
Mn (ω)
< 1 + .
log n
31
.
< 1 + if n > N2 (ω).
n
Mn
log n
P
n (1
That is, Ω0 ⊂
Exercise 2.3.16. (1)
o
< 1 + eventually . Since P (Ω0 ) = 1 by (i),
Mn
< 1 + eventually = 1.
P
log n
− F (λn )) < ∞.
Let Bn = {Xn > λn }. Since 1 − F (λn ) = P (Xn > λn ), by Borel-Cantelli Lemma
1, P (Xn > λn i.o.) = 0. That is, P (Xn ≤ λn eventually) = 1. Note that (?) for
ω ∈ Ω0 := {ω : Xn (ω) ≤ λn eventually}, there exists N1 (ω) ∈ N such that Xn (ω) ≤ λn
if n > N1 (ω).
Now, for n > N1 (ω),
max Xm (ω) = max
max
1≤m≤n
1≤m≤N1 (ω)
Xm (ω),
max
N1 (ω)<m≤n
Xm (ω) .
- Since λn ↑ ∞, there exists N2 (ω) ∈ N such that max1≤m≤N1 (ω) Xm (ω) ≤ λn if n >
N2 (ω).
- By (?), Xm (ω) ≤ λm ≤ λn for N1 (ω) < m ≤ n.
Thus, for n > N (ω) = max{N1 (ω), N2 (ω)}, we have
max Xm (ω) ≤ λn .
1≤m≤n
That is, Ω0 ⊂ {max1≤m≤n Xm ≤ λn eventually}.
Since P (Ω0 ) = 1, P (max1≤m≤n Xm ≤ λn eventually) = 1, which means that
P
max Xm > λn i.o. = 0.
1≤m≤n
(2)
P
n (1
− F (λn )) = ∞.
By Borel-Cantelli Lemma 2, P (Xn > λn i.o.) = 1. Then, clearly, P (max1≤m≤n Xn >
λn i.o.) = 1.
Exercise 2.3.17. The answers are
(i) E|Y1 | < ∞,
(ii) EY1+ < ∞,
(iii) nP(|Y1 | > n) → 0, and
(iv) P(|Y1 | < ∞) = 1.
(i) If E|Y1 | < ∞, then
X
Z ∞
∞
∞
X
Y1
> =
P (|Y1 | > n) ≤
P (|Y1 | > t) = E|Y1 | < ∞
P
n
0
n=1
n=1
since P(|Y1 | > ·) decreases. By Borel-Cantelli, we have the convergence. Conversely, if
E|Y1 | = ∞, then
Z
∞ = E|Y1 | =
∞
P(|Y1 | > t) dt ≤
0
∞
X
P(|Y1 | > n)
n=0
since P(|Y1 | > ·) decreases. By the second Borel-Cantelli, the convergence fails.
32
(ii) If EY1+ < ∞, then Yn+ /n → 0 a.s. from (i). Thus for a given > 0, there exists N such
that Yn /n+ < for all n > N . Then,
maxm≤n Ym+
≤ max
n
YN+
Y1+
,··· ,
, → 0
n
n
almost surely as n → ∞. That is,
maxm≤n Ym+
=0
n→∞
n
lim
almost surely. Now we have
lim sup
n→∞
maxm≤n Ym
maxm≤n Ym+
≤ lim sup
= 0.
n
n
n→∞
We claim that
lim sup
n→∞
maxm≤n Ym
≥ 0.
n
Note that
P
maxm≤n Ym
< −
n
= F (−n)n
where F is the distribution of Y1 . Let M be large so that F (−n) < δ < 1 for all n > M .
Then we have
X
∞
M
∞
X
X
maxm≤n Ym
maxm≤n Ym
P
< − ≤
P
< − +
δn
n
n
n=1
n=1
n=M +1
M +1
≤M+
δ
1−δ
< ∞.
Therefore, by Borel-Cantelli,
lim sup
n→∞
maxm≤n Ym
≥ −
n
almost surely. This proves our claim.
Conversely, if EY1+ = ∞, then
Z ∞
∞
X
∞ = EY1+ =
P(Y1 > t) dt ≤
P(Y1 > n),
0
n=0
and thus Yn /n > 1 infinitely often.
(iii) If
(iv) By definition, Yn /n → 0 in probability if and only if
P(|Yn | > n) → 0
for any > 0. This is equivalent to P(|Y1 | = ∞) = 0.
Exercise 2.3.18 and 2.3.19 will be solved by the way we did in Theorem 2.3.9 in Textbook.
33
Exercise 2.3.18. Step 1.
Xn
anα
→ 1 in probability.
Fix > 0. Since β < 2α,
Xn − EXn
Var(Xn )
Bnβ
B
P
≤
∼ 2 2 · nβ−2α → 0.
> ≤ 2
2
2
2
EXn
(EXn )
(EXn )
a Thus,
Xn −EXn
EXn
→ 0 in probability. Therefore,
Xn
EXn
→ 1 in probability.
Step 2. Find subsequence (nk ).
Let nk = k [2/(2α−β)]+1 (+1 to avoid the case where
h
2
2α−β
i
= 0.) and Tk = Xnk .
Then,
Bnβk
Tk − ETk
Var(Tk )
B
> ≤ 2
≤
∼ 2 2 · nkβ−2α
P
2
2
2
ETk
(ETk )
(ETk )
a 2
B
≤ 2 2 · k ( 2α−β )·(β−2α)
a = Ck −2
P∞
−ETk
The last inequality holds by construction of nk . Since k=1 P TkET
>
< ∞,
k
by Borel-Cantelli Lemma 1,
Tk − ETk
> i.o. = 0.
P
ETk
That is,
Step 3.
Xn
anα
Tk
ETk
→ 1 a.s.
→ 1 a.s.
For sufficiently large n such that nk ≤ n < nk+1 and ω ∈ Ω0 := {ω :
Tk
ETk
→ 1}, we
have
Tk (ω)
Xn (ω)
Tk+1 (ω)
Tk+1 (ω) ETk+1
ETk Tk (ω)
·
=
≤
≤
=
·
.
ETk+1 ETk
ETk+1
EXn
ETk
ETk+1
ETk
Note that
anα
ETk+1
k+1
∼
=
ETk
anα
k
Therefore, we have
a.s.
Step 4.
Xn
nα
Xn (ω)
EXn
"
2 +1 #α
1 [ 2α−β ]
→ 1.
1+
k
→ 1. Since P (Ω0 ) = 1 by Step 2, we conclude that
Xn
EXn
→1
→ a a.s.
Xn
=a·
nα
EXn
anα
Xn
·
→a
EXn
a.s.
Exercise 2.3.19. Suppose Y ∼ P oisson(λ), where λ > 1. Then, we can decompose Y into
Y = Y1 + · · · + Yn , where Yi ’s are independent and Yi ∼ P oisson(λi . Thus, without loss of
generality, we may assume that Xn are independent Poisson r.v.’s with EXn = λn , where
λn ≤ 1.
34
Step 1.
Sn
ESn
→ 1 in probability.
P
Fix > 0. Since n λn = ∞,
P
λn
Sn − ESn
Var(Sn )
1
P
> ≤ 2
= 2 Pn
= 2P
→ 0.
2
2
ESn
(ESn )
( n λn )
n λn
Thus,
Sn −ESn
ESn
→ 0 in probability. Therefore,
Sn
ESn
→ 1 in probability.
Step 2. Find subsequence (nk ).
P
Pn
Since n λn = ∞, one can find subsequence (nk ) such that nk = inf{n : i=1 λi ≥
k 2 }. Let Tk = Snk . Then,
Tk − ETk
Var(Tk )
1
1
> ≤ 2
= 2 Pnk
≤ 2 2.
P
ETk
(ETk )2
k
λ
i=1 i
P∞
−ETk
Since k=1 P TkET
> < ∞, by Borel-Cantelli Lemma 1,
k
Tk − ETk
P
> i.o. = 0.
ETk
That is,
Step 3.
Sn
ESn
Tk
ETk
→ 1 a.s.
→ 1 a.s.
For n ∈ N such that nk ≤ n < nk+1 , we have
k2 ≤
nk
X
λi = ETk ≤ k 2 + 1 ≤ (k + 1)2 ≤ ETk+1 ≤ (k + 1)2 + 1.
i=1
Note that the assumption that λn ≤ 1 is used here.
With such n and ω ∈ Ω0 := {ω :
Tk (ω)
ETk
→ 1},
ETk Tk (ω)
Tk (ω)
Sn (ω)
Tk+1 (ω)
Tk+1 (ω) ETk+1
·
=
≤
≤
=
·
.
ETk+1 ETk
ETk+1
ESn
ETk
ETk+1
ETk
Note that
2(k + 1)
ETk+1
(k + 1)2 + 1
=1+
→ 1.
≤
ETk
k2
k2
Therefore, we have
Sn (ω)
ESn
→ 1. Since P (Ω0 ) = 1 by Step 2, we conclude that
Sn
ESn
→1
a.s.
Exercise 2.3.20. It suffices to show that for any M > 0, P
Xn
n log2 n
≥ M i.o. = 1. Fix
M > 0. Let kn = blog2 (M · n log2 n)c + 1. Then, for n ≥ 2,
P (X1 ≥ M · n log2 n) =
∞
X
1
1
= kn −1
2j
2
j=kn
=
1
1
≥
.
M · n log2 n
2blog2 (M ·n log2 n)c
35
Then, by Exercise 2.2.4,
∞
X
X1
≥M
n log2 n
∞
X
1
= ∞.
M
·
n
log2 n
n=2
n=2
Xn
Since Xn ’s are independent, by Borel-Cantelli Lemma 2, we get P n log
P
≥
2n
≥ M i.o. = 1.
2.4. Strong Law of Large Numbers.
i.i.d
i.i.d
Exercise 2.4.1. Xi , Yi ≥ 0. Xi ∼ F , Yi ∼ G. EX1 , EY1 < ∞.
Let Tn := (X1 + Y1 ) + · · · + (Xn + Yn ) and Nt := min {n : Tn ≤ t}. Note that Nt → ∞ as
t → ∞. Then,
PNt
i=1
Xi
Rt
≤
≤1−
t
PNt
i=1
t
By SLLN and Theorem 2.4.7, we have
PNt
PNt
i=1 Yi a.s.
i=1 Xi a.s.
→ EX1 ,
→ EY1 ,
Nt
Nt
t
Yi
.
Nt a.s.
1
→
.
t
EX1 + EY1
Thus, we get
Rt a.s.
EX1
→
.
t
EX1 + EY1
Exercise 2.4.2. Let Yn+1 = Xn+1 /|Xn |. Since Yn is independent of X1 , · · · , Xn , (Yn )n is
an iid sequence. Note that
log |Xn |
log |Y1 | + · · · + log |Yn |
=
n
n
converges to E log |Y1 | whenever it exists by the Strong Law of Large Numbers. Now to find
the expectation,
Z
E log |Y1 | =
B1
1
log |x| dx =
π
Z
0
1
Z
∂Br
1
log r dS dr =
π
Z
0
1
1
2r log r dr = − .
2
Remark. For d-dimensional vectors, one can derive that it converges to −1/d by a similar
argument.
Exercise 2.4.3.
(i)
Wn = (ap + (1 − p)Vn−1 )Wn−1
= (ap + (1 − p)Vn−1 )(ap + (1 − p)Vn−2 )Wn−2
..
.
=
n−1
Y
(ap + 1 − p)Vi ).
i=1
Pn−1
log(ap + (1 − p)Vi ). By SLLN,
Pn−1
log(ap + (1 − p)Vi ) a.s. log Wn
n−1
=
· i=1
→ E log(ap + (1 − p)V1 ) =: c(p).
n
n
n−1
Thus, log Wn =
i=1
36
(ii) c(p) =
R
log(ap + (1 − p)ν)µ(dν), where µ : distribution of V . By Theorem A.5.1,
2
Z a−ν
00
c (p) = −
µ(dν) < 0.
ap + (1 − p)ν
(iii)
a−ν
µ(dν) = −1 + aE(V1−1 )
ν
Z
1
a−ν
µ(dν) = 1 − EV1 .
c0 (1) =
a
a
0
Z
c (0) =
To guarantee that the optimal choice of p lies in (0,1), E(V1−1 ) >
should hold.
(iv) p =
8−5a
2a2 −10a+8 .
3. Central Limit Theorems
3.1. The De Moivre-Laplace Theorem.
3.2. Weak Convergence.
Exercise 3.2.1. Omit.
Exercise 3.2.2.
(i) P Mn /n1/α ≤ y = P Mn ≤ yn1/α
h in
= P Xi ≤ yn1/α
−α n
= 1 − yn1/α
n
y −α
= 1−
→ exp −y −α .
n
(ii) P n1/β Mn ≤ y = P Mn ≤ yn−1/β
h in
= P Xi ≤ yn−1/β
n
|y|β
= 1−
→ exp −|y|β .
n
(iii) P (Mn − log n ≤ y) = P (Mn ≤ y + log n)
n
= [P (Xi ≤ y + log n)]
n
= 1 − e−y−log n
n
e−y
= 1−
→ exp −e−y .
n
37
1
a
and E(V1 ) > a
Z
Exercise 3.2.3.
(i)
∞
e−
y2
2
x
The first equality gives us
∞ Z ∞
1 y2
1 − y2
dy = − e− 2
e 2 dy
−
y
y2
x
x
∞ ∞ Z ∞
1 − y2
3 − y2
1 − y2
2
2
2
e
= − e
dy .
− − 3e
−
y
y
y4
x
x
x
Z
∞
e−
y2
2
dy ≤
x
1 − x2
e 2.
x
The second equality gives us
Z ∞
2
y2
1
1
− x2
e− 2 dy.
≤
− 3 e
x x
x
Thus, P (Xi > x) ∼
1
x
·
1 2
√1 e− 2 x .
2π
P (Xi > x + θ/x)
∼
P (Xi > x)
Then,
√
2
1
exp − 21 x + xθ
θ
2π(x+ x
)
√1
exp − 12 x2
2πx
→ e−θ
(ii) It’s clear that bn → ∞ as n → ∞. By (i), P Xi > bn + bxn · n → e−x .
x
P (bn (Mn − bn ) ≤ x) = P Mn ≤ bn +
bn
n
x
= P Xi ≤ bn +
bn
n
x
= 1 − P Xi > bn +
bn
n

P Xi > bn + bxn

= 1 − P (Xi > bn ) ·
P (Xi > bn )
n

x
P
X
>
b
+
i
n
bn
1

= 1 − ·
n
P (Xi > bn )
→ exp −e−x .
√
(iii) Claim. bn ∼ 2 log n.
√
2 log n) ∼ n√4π1 log n . Thus, for large n, bn ≤
√
(≥) Similar argument with 2 log n − 2 log log n
Proof. (≤) By (i), P (Xi >
√
2 log n.
Now, by (iii), P (bn (Mn − bn ) ≤ x) → exp (−e−x ). Let xn and yn be xn → ∞ and
yn → −∞. Then,
xn
P Mn − bn ≤
→1
bn
38
yn
M n − bn ≤
→0
bn
yn
xn
⇒P
≤ M n − bn ≤
→1
bn
bn
Mn
bn
xn
y
√ n
≤√
−√
≤ √
→ 1.
⇒P
bn 2 log n
2 log n
2 log n
bn 2 log n
By choosing xn , yn with o b2n , we can show that
P
√
Mn
P
→ 1.
2 log n
Exercise 3.2.4. By Skorohod’s Representation Theorem(Thm 3.2.8 in Textbook), there
d
d
a.s.
exists Yn and Y such that Yn = Xn and Y = X with Yn −−→ Y . By Continuous Mapping
a.s.
Theorem (Exercise 1.3.3), Eg(Yn ) −−→ Eg(Y ). By Fatou’s Lemma, we get
lim inf Eg(Xn ) = lim inf Eg(Yn ) ≥ Eg(Y ) = Eg(X).
n
n
Exercise 3.2.5. By Skorohod’s Representation Theorem, there exists Yn and Y with distria.s.
bution Fn and F , respectively, such that Yn −−→ Y . By Theorem 1.6.8 in Textbook,
Z
h(x)dFn (x) →∈ h(x)dF (x).
Exercise 3.2.6. Define L(F, G) = {c : F (x − c) − c ≤ G(x) ≤ F (x + c) + c for all x}.
Since any distribution is non-decreasing,
(a) if a ∈ L(F, G), then for any b ≥ a are also in L(F, G)
(b) L(F, G) contains only non-negative constants.
(1) ρ is a metric.
(i) ρ(F, G) = 0 ⇐⇒ F = G.
(⇐) Trivial.
(⇒) For any > 0, F (x − ) − ≤ G(x) ≤ F (x + ) + for all x. Letting & 0,
we get F (x− ) ≤ G(x) ≤ F (x) for all x. Thus, for x ∈ C(F ), F (x) = G(x).
For x ∈ D(F ), since D(F ) is at most countable(Exercise 1.2.3), we can find
(xn )n∈N such that xn ∈ C(F ) with xn ↓ x. Since F (xn ) = G(xn ), F (x) =
limn F (xn ) = limn G(xn ) = G(x). Therefore, F = G
(ii) ρ(F, G) = ρ(G, F ).
Suppose a ∈ L(F, G). Then, for all x,
F (x) − a ≤ G(x + a) ≤ F (x + 2a) + a
F (x − 2a) − a ≤ G(x − a) ≤ F (x) + a.
Thus, G(x − a) − a ≤ F (x) ≤ G(x + a) + a for all x. That is, a ∈ L(G, F ). Therefore,
ρ(G, F ) ≤ ρ(F, G). By symmetry, it also holds that ρ(F, G) ≤ ρ(G, F ).
(iii) ρ(F, H) ≤ ρ(F, G) + ρ(G, H).
Suppose a ∈ L(F, G) and b ∈ L(G, H). Then, for all x,
F (x − a − b) − a − b ≤ G(x − b) − b ≤ H(x) ≤ G(x + b) + b ≤ F (x + a + b) + a + b.
39
That is, a + b ∈ L(F, H). Thus, ρ(F, H) ≤ ρ(F, G) + ρ(G, H).
By (i)-(iii), ρ is a metric.
(2) ρ(Fn , F ) → 0 if and only if Fn ⇒ F .
(⇒) Since ρ(Fn , F ) → 0, for given > 0, there exists N ∈ N such that ρ(Fn , F ) < for
n > N . By (a), ∈ L(Fn , F ) for n > N . That is, if n > N ,
F (x − ) − ≤ Fn (x) ≤ F (x + ) + ,
∀x.
Letting n → ∞,
F (x − ) − ≤ lim inf Fn (x) ≤ lim sup Fn (x) ≤ F (x + ) + ,
∀x.
For x ∈ C(F ), letting & 0,
F (x) ≤ lim inf Fn (x) ≤ lim sup Fn (x) ≤ F (x).
Thus, Fn (x) → F (x).
(⇐) Since Fn ⇒ F , limn→∞ Fn (x) = F (x) for x ∈ C(F ). Fix > 0. Since D(F ) is at
most countable, we can find {x1 , . . . , xk } ∈ C(F ) such that
F (x1 ) < , F (xk ) > 1 − and |xi − xi−1 | < for i = 2, . . . , k .
For each xi , there exists Ni () ∈ N such that F (xi ) − < Fn (xi ) < F (xi ) + if
n > Ni (). Let N () = max {N1 (), . . . , Nk ()}. Then, for n > N (),
(i) if x < x1 ,
F (x − 2) − 2 ≤ F (x1 ) − 2 < 0 ≤ Fn (x) ≤ Fn (x1 ) < F (x1 ) + < 2 ≤ F (x + 2) + 2.
(ii) if x > xk ,
F (x − 2) − 2 ≤ 1 − 2 < F (xk ) − < Fn (xk ) ≤ Fn (x) ≤ 1 < F (xk ) + 2 ≤ F (x + 2) + 2.
(iii) if xi−1 ≤ x ≤ xi+1 for i = 2, . . . , k ,
F (x − ) − ≤ F (xi−1 ) − < Fn (xi−1 ) ≤ Fn (x) ≤ Fn (xi ) < F (xi ) + ≤ F (x + ) + .
By (i)-(iii), 2 ∈ L(F, Fn ) for n > N (). Since > 0 is arbitrarily chosen, we can
conclude that ρ(Fn , F ) → 0.
Exercise 3.2.7. X ∼ F and Y ∼ G. Recall L(F, G) defined in Exercise 3.2.6.
Define K(X, Y ) = { ≥ 0 : P (|X − Y | > ) ≤ }. If one can show that K(X, Y ) ⊂ L(F, G),
then it’s done. Suppose a ∈ K(X, Y ). Then, P (|X − Y | > a) ≤ a. Since
P (X ≤ x − a) − P (|X − Y | > a) ≤ P (Y ≤ x) ≤ P (X ≤ x + a) + P (|X − Y | > a) ∀x,
we get
P (X ≤ x − a) − a ≤ P (Y ≤ x) ≤ P (X ≤ x + a) + a, ∀x.
Thus, a ∈ L(F, G)
40
Exercise 3.2.8. Recall K(X, Y ) defined in Exercise 3.2.7. It is easy to check that if a ∈
K(X, Y ), then any b > a is also in K(X, Y ).
If a = 0, then X = Y , and thus β(X, Y ) = 0. Now, suppose a > 0.
(1)
a2
1+a
≤ β(X, Y ).
Fix 0 < < a. Since ∈
/ K(X, Y ), < P (|X − Y | > ). By Exercise 2.3.6 (b) (⇒), we
get
1+
< P (|X − Y | > ) ≤
E
Thus,
2
1+
|X − Y |
1 + |X − Y |
.
< β(X, Y ). Letting % a, we get
a2
≤ β(X, Y ).
1 + a2
(2) β(X, Y ) ≤
2a
1+a .
For all b > a,
|X − Y |
dP
1 + |X − Y |
Z
Z
|X − Y |
|X − Y |
=
dP +
dP
|X−Y |>b 1 + |X − Y |
|X−Y |≤b 1 + |X − Y |
Z
Z
b
≤
1dP +
dP
1
+
b
|X−Y |>b
|X−Y |≤b
Z
β(X, Y ) =
b
= P (|X − Y | > b) +
· P (|X − Y | ≤ b)
1+b
b
b
+
= P (|X − Y | > b) · 1 −
1+b
1+b
b
b
2b
≤
+
=
.
1+b 1+b
1+b
The last inequality holds since b ∈ K(X, Y ). Letting b & a, we get β(X, Y ) ≤
2a
1+a .
Exercise 3.2.9. Since Fn ⇒ F and F is continuous, Fn (x) → F (x), ∀x. Fix k ∈ N.
Pick xj,k = F −1 kj for j = 1, . . . , k − 1 with x0,k = −∞ and xk,k = ∞. For each
j, there exists Nk (j) ∈ N such that |Fn (xj,k ) − F (xj,k )| <
1
k
if n > Nk (j). Let Nk =
max {Nk (1), . . . , Nk (k − 1)}. Then, for n > Nk , if xj−1,k < x < xj,k , then
F (x) −
2
1
1
2
≤ F (xj−1,k) − < Fn (xj−1,k ) ≤ Fn (x) ≤ Fn (xj,k ) < F (xj,k ) + ≤ F (x) + .
k
k
k
k
That is, |Fn (x) − F (x)| < k2 . Thus, supx |Fn (x) − F (x)| → 0 as n → ∞.
Pn
iid
Exercise 3.2.10. Let Xi ∼ F and Fn (ω, x) = n1 i=1 I(Xi (ω) ≤ x). By Glivenko-Cantelli
Theorem(Thm 2.4.9), for each x ∈ R, Fn (ω, x) → F (x) a.s. Let Ωx be a set where Fn (ω, x) →
F (x) and let Ω0 = ∩q∈Q Ωq . Then, P (Ω0 ) = 1 and for each ω0 ∈ Ω0 , we have Fn (ω, q) → F (q),
∀q ∈ Q. Now, for x ∈ C(F ), choose r, s ∈ Q s.t. r < x < s and
|F (r) − F (x)| < 41
|F (s) − F (x)| < .
Then,
Fn (x) − F (x) ≤ Fn (s) − F (r) → F (s) − F (r) < 2
Fn (x) − F (x) ≥ Fn (r) − F (s) → F (r) − F (s) > −2.
Therefore, −2 ≤ lim inf(Fn (x) − F (x)) ≤ lim sup(Fn (x) − F (x)) ≤ 2.
Exercise 3.2.11. Since Xn , 1 ≤ n ≤ ∞, are integer valued, R\Z ⊂ C(Fn ), where Fn is df of
Xn .
(⇒) For 1 ≤ n < ∞,
(i) m ∈ R\Z
P (Xn = m) = 0 = P (X∞ = m).
(ii) m ∈ Z
1
1
P (Xn = m) = Fn m +
− Fn m −
2
2
1
1
− F∞ m −
→ F∞ m +
2
2
= P (X∞ = m).
By (i), (ii), P (Xn = m) → P (X∞ = m), ∀m.
(⇐) Want to show that Fn (x) → F∞ (x), ∀x ∈ C(F∞ ).
Fix > 0. Since F∞ can have positive probability only on Z, which is countable, we
can pick K() := {k1 , · · · , kl } ⊂ Z such that
l
X
(11a)
P (X∞ = ki ) > 1 − .
i=1
By assumption, for each ki , there exists N (i) ∈ N such that for n > N (i),
|P (Xn = ki ) − P (X∞ = ki )| < .
l
Let N = max {N (1), · · · , N (l )}. Then, for n < N ,
l
l
X
X
[P (Xn = ki ) − P (X∞ = ki )] ≤
|P (Xn = ki ) − P (X∞ = ki )| =: P0 < i=1
i=1
Also, by (11a), we have
P1 :=
X
P (X∞ = k) < .
k∈Z\K()
Thus,
P2 : =
X
P (Xn = k)
k∈Z\K()
42
= 1−
l
X
P (Xn = ki )
i=1
= 1−
l
X
P (X∞ = ki ) +
i=1
X
≤
l
X
P (X∞ = ki ) −
i=1
P (X∞ = k) +
l
X
P (Xn = ki )
i=1
l
X
P (Xn = ki ) − P (X∞ = ki )
i=1
k∈Z\K()
< 2.
Now, for x ∈ C(F∞ ), let Zx = Z ∩ (−∞, x].
X
P (Xn = k)
Fn (x) =
k∈Zx
X
=
X
P (Xn = k) +
k∈Zx ∩K()
P (Xn = k).
k∈Zx \K()
Similarly,
X
F∞ (x) =
X
P (X∞ = k) +
k∈Zx ∩K()
P (X∞ = k).
k∈Zx \K()
For n > N ,
X
|Fn (x) − F∞ (x)| =
k∈Zx ∩K()
X
≤
X
P (Xn = k) − P (X∞ = k) +
X
P (Xn = k) − P (X∞ = k)
k∈Zx \K()
k∈Zx ∩K()
≤
X
P (Xn = k) − P (X∞ = k) +
P (Xn = k) +
k∈Zx \K()
X
|P (Xn = k) − P (X∞ = k)| +
k∈Zx ∩K()
P (Xn = k) +
k∈Zx \K()

Z ∩ K() ⊂ K() = {k , · · · , k }
x
1
l
Note that
Z \K() ⊂ Z\K()
. Thus,
x
|Fn (x) − F∞ (x)| ≤ P0 + P1 + P2 < 4.
Therefore, Fn (x) → F∞ (x), and thus Xn ⇒ X∞ .
Exercise 3.2.12. (1) If Xn → X in probability, then Xn ⇒ X.
p
Since Xn −
→ X, for given > 0, P (|Xn − X| > ) → 0.
(i) P (Xn ≤ x) = P (Xn ≤ x, |Xn − X| ≤ ) + P (Xn ≤ x, |Xn − x| > )
≤ P (X ≤ x + ) + P (|Xn − X| > ).
Then, lim supn P (Xn ≤ x) ≤ P (X ≤ x + ). By letting & 0, we get
lim sup P (Xn ≤ x) ≤ P (X ≤ x).
n
43
X
P (X∞ = k)
k∈Zx \K()
X
k∈Zx \K()
P (X∞ = k) .
(ii) P (Xn > x) = P (Xn > x, |Xn − X| ≤ ) + P (Xn > x, |Xn − X| > )
≤ P (X > x − ) + P (|Xn − X| > ).
Then,
lim sup P (Xn > x) ≤ P (X > x − )
n
⇒ lim sup [1 − P (Xn ≤ x)] ≤ 1 − P (X ≤ x − )
n
⇒ lim inf P (Xn ≤ x) ≥ P (X ≤ x − ).
n
For x ∈ C(FX ), by letting & 0, we get
lim inf P (Xn ≤ x) ≥ P (X ≤ x).
n
By (i), (ii), for x ∈ C(FX ),
Fn (x) = P (Xn ≤ x) → P (X ≤ x) = F (x).
Thus, Xn ⇒ X.

0,
(2) Since Xn ⇒ c, P (Xn ≤ y) →
1,
y<c
. Thus, for any > 0,
y>c
P (|Xn − c| > ) = P (Xn > c + ) + P (Xn < c − )
= 1 − P (Xn ≤ c + ) + P (Xn < c − )
→ 0.

X ⇒ X, P (X ≤ x) → P (X ≤ x), ∀x ∈ C(F ).
n
n
X
Exercise 3.2.13. Since
P
Y →
c, P (|Yn − c| > ) → 0, ∀ > 0.
n
(i) P (Xn + Yn ≤ z) = P (Xn + Yn ≤ z, |Yn − c| ≤ ) + P (Xn + Yn ≤ z, |Yn − c| > )
≥ P (Xn ≤ z − c + ) + P (|Yn − c| > ).
Since D(FX ) is at most countable, we may assume that z − c + ∈ C(FX ). Then,
lim supn P (Xn + Yn ≤ z) ≤ P (X ≤ z − c + ). Letting & 0,
lim sup P (Xn + Yn ≤ z) ≤ P (X ≤ z − c) = P (X + c ≤ z).
n
(ii) P (Xn + Yn > z) = P (Xn + Yn > z, |Yn − c| ≤ ) + P (Xn + Yn > z, |Yn − c| > )
≤ P (Xn > z − c − ) + P (|Yn − c| > ).
By similar argument, we get
lim inf P (Xn + Yn ≤ z) ≥ P (X ≤ z − c − ).
n
For z − c ∈ C(FX ), by letting → 0,
lim inf P (Xn + Yn ≤ z) ≥ P (X ≤ z − c) = P (X + c ≤ z).
n
44
Note that z ∈ C(FX+c ) if and only if z − c ∈ C(FX ). By (i), (ii), we get
P (Xn + Yn ≤ z) → P (X + c ≤ z).

X ⇒ X, P (X ≤ x) → P (X ≤ x), ∀x ∈ C(F ).
n
n
X
Exercise 3.2.14. Since
P
Y →
c, P (|Yn − c| > ) → 0, ∀ > 0.
n
For simple proof, let’s assume 0 < < c.
(i) P (Xn Yn ≤ z) = P (Xn Yn ≤ z, |Yn − c| ≤ ) + P (Xn Yn ≤ z, |Yn − c| > )
z
≤ P Xn ≤
+ P (|Yn − c| > ).
c−
z
Let c− ∈ C(FX ). Then,
z
lim sup P (Xn Yn ≤ z) ≤ P X ≤
.
c−
n
Letting → 0,
z
lim sup P (Xn Yn ≤ z) ≤ P X ≤
= P (cX ≤ z).
c
n
(ii) P (Xn Yn > z) = P (Xn Yn > z, |Yn − c| ≤ ) + P (Xn Yn > z, |Yn − c| > ) Similarly,
z
≤ P Xn >
+ P (|Yn − c| > ).
c+
z
.
lim inf P (Xn Yn ≤ z) ≥ P X ≤
n
c+
For
z
c
∈ C(FX ), by letting → 0,
z
lim inf P (Xn Yn ≤ z) ≥ P X ≤
= P (cX ≤ z).
n
c
Note that z ∈ C(FcX ) if and only if
z
c
∈ C(FX ). By (i),(ii), we get
P (Xn Yn ≤ z) → P (cX ≤ z).
Pn
d
Exercise 3.2.15. Step 1. Xni ≡ Yi /( m=1 Ym2 /n)1/2 .
Note that uniformly distributed over the surface of the unit sphere is the only distribution
which is invariant to rotation and belongs to unit sphere.
iid
Since Yi ∼ N (0, 1), Y = (Y1 , · · · , Yn ) ∼ N (0, In ). For any orthogonal matrix A, AY ∼
Pn
Pn
d
d
AY
Y
2
2
N (0, In ) ≡ Y . Then, ||YY||2 ≡ ||AY
m=1 Ym , · · · , Ym /
m=1 Ym ) is
||2 . Thus, ||Y ||2 = (Y1 /
√
d
uniformly distributed over the surface of the unit sphere. Then, Xn ≡ n ||YY||2 , and thus
d
Xni ≡
Yi
n
X
m=1
45
!1/2
Ym2 /n
.
Step 2. Yi /(
Pn
m=1
Ym2 /n)1/2 ⇒ N (0, 1).
For Yi ∼ N (0, 1), EYi2 = 1 < ∞. Thus, by WLLN,
n
X
P
Ym2 n → 1.
m=1
Pn
By Exercise 3.2.12(1), m=1 Ym2 /n ⇒ 1. Note that f (x) = √1x I(x>0) is continuous on R\{0}
Pn
and P ( m=1 Ym2 /n = 0) = P (Y1 = · · · = Yn = 0) = 0. By Continuous Mapping Theorem
(Theorem 3.2.10), we get
n
X
Ym2
!−1/2
⇒ 1.
n
m=1
By Exercise 3.2.14,
Yi ·
n
X
Ym2
!−1/2
⇒ N (0, 1).
n
m=1
Thus,
Xn1
⇒ standard normal.
Exercise 3.2.16. For 1 < M < ∞, since EYnβ → 1, there exists N ∈ N such that for
n > N , EYnβ ≤ M . Therefore, we can drop finitely many Yn ’s and say {Fn } is tight. (with
ϕ(y) = |y|β )
For every subsequence (Fnk ), by Helly and Tightness Theorem, there exists (Fn(mk ) ) such
that Fn(mk ) ⇒ F , where F is a df. Let Y be a r.v. with F.
For g(y) = |y|β and h(y) = |y|α , we have
(0) g, h are continuous.
(i) g(y) > 0
(ii) |h(y)|/g(y) = |y|α−β → 0 as |y| → ∞
(iii) Fn(mk ) ⇒ F
R
β
(iv) |y|β dFn(mk ) (y) = EYn(m
≤ M < ∞.
k)
By Exercise 3.2.5,
Z
α
|y|α dFn(mk ) (y) = EYn(m
→ EY α =
k)
Z
|y|α dF (y).
Since convergent real sequence has the unique limit, EY α = 1. Also, similar argument for
h(y) = |y|γ with γ ∈ (α, β) gives us that
γ
EYn(m
→ EY γ .
k)
From Jensen’s Inequality with following functions, we get EYnγ → 1.
γ
(1) ϕ1 (y) = |y| α .
γ
γ
(EYnα ) α ≤ E (Ynα ) α = EYnγ
46
⇒
1 ≤ lim inf EYnγ .
n
β
(2) ϕ2 (y) = |y| γ .
β
β
(EYnγ ) γ ≤ E (Ynγ ) γ = EYnγ
⇒
lim sup EYnγ ≤ 1.
n
Therefore, EY γ = 1.
γ
γ
γ
For strictly convex ϕ(y) = |y| α , we have (EY α ) α = 1 = E(Y α ) α = EY γ . By Exercise 1.6.1,
P
Y α = 1 a.s. Since α > 0, Y = 1 a.s. Since Yn(mk ) → 1, by Theorem 2.3.2,
P
Yn → 1.
Exercise 3.2.17. Suppose not. Then, for some K < ∞ and y < 1, there exists a r.v. X
with EX 2 = 1 and EX 4 ≤ K such that P (|X| > y) < c for all c > 0. With c =
we can construct a sequence (Xn ) such that P (|Xn | > y) → 0,
EXn2
1
n,
n ∈ N,
= 1 and EXn4 ≤ K.
Theorem 3.2.14 with ϕ(x) = x2 gives us that {Xn } is tight. Then, by Helly and Tightness
Theorem, there exists (Xnm ) such that Xnm ⇒ X∞ . By Generalized DCT with h(x) = x2
2
and g(x) = x4 , we get EX∞
= limm→∞ EXn2m = 1. However, since P (|Xn | > y) → 0,
2
P (|Xn | ≤ y) → 1. Then, EX∞
≤ y 2 < 1, which is a contradiction.
3.3. Characteristic Functions.
Exercise 3.3.1. Let X be a r.v. with ch.f ϕ and X 0 be an independent copy of −X. Then,
its ch.f ϕ0 (t) = Eeit(−X) = ϕ(t). By Theorem 3.3.2 and Lemma 3.3.9 in Textbook, Re ϕ is
ch.f of 12 (X + X 0 ) and |ϕ|2 is ch.f of X + X 0 .
Exercise 3.3.2.
(i) Let
T
1
2T
Z
1
=
2T
Z
Z
−T
Z T
−∞
Z ∞
−T
−∞
IT =
=
1
2T
−T
T
e−ita ϕ(t) dt
∞
e−ita eitx µ(dx) dt
eit(x−a) µ(dx) dt
Since eit(x−a) ≤ 1 and [−T, T ] is a finite interval, by Fubini,
Z ∞Z T
1
cos(t(x − a)) + i sin(t(x − a)) dt µ(dx).
IT =
2T −∞ −T
Since sin t is symmetric about the origin,
Z ∞Z T
1
IT =
cos(t(x − a)) dt µ(dx)
2T −∞ −T
Rx
RT
1
Since 0 cos(y)dy = | sin(y)| ≤ |x|, 2T
cos(t(x − a)) dt ≤ 1 : integrable. By
−T
DCT,
Z
∞
lim IT =
T →∞
−∞
"
1
lim
T →∞ 2T
Z
T
#
cos(t(x − a)) dt µ(dx).
−T
47
(1) x 6= a
1
lim
T →∞ 2T
Z
T
sin((x − a)T )
= 0.
(x − a)T
cos(t(x − a)) dt = lim
T →∞
−T
(2) x = a
1
T →∞ 2T
Z
T
cos(t(x − a)) dt = 1.
lim
−T
By (1) and (2), we get
lim IT = µ {a} .
T →∞
(ii) Let µ be the distribution of X.
ϕ
2π
+t
h
Z
=
2π
h +t)x
ei(
Z
2π
h +t)x
ei(
=
µ(dx)
µ(dx)
hZ
Z
eitx µ(dx)
=
hZ
Z
(2a)
=
eitx µ(dx) = ϕ(t).
By (i),
1
P (X = x) = µ({x}) = lim
T →∞ 2T
Let Tn =
π
h
+
2π
h
Z
T
e−itx ϕ(t)dt.
−T
· n, with n ≥ 0 . Then, since limit of RHS above exists,
Z Tn
1
e−itx ϕ(t)dt.
P (X = x) = lim
n→∞ 2Tn −T
n
Now, for 1 ≤ k ∈ N,
Z Tk
e−itx ϕ(t)dt =
Z
π
h
e−i(t+
2kπ
h )x
−π
h
Tk−1
= e−i(
2kπx
h )
π
h
Z
2kπ
·ϕ t+
dt
h
e−it ϕ(t)dt
−π
h
by (2a). Since x ∈ hZ, 2kπx
∈ N. Thus, e−i(
h
Z Tk
Z
e−itx ϕ(t)dt =
2kπx
h )
π
h
= 1. Therefore,
e−it ϕ(t)dt
−π
h
Tk−1
Z
T0
=
−T0
48
e−itx ϕ(t)dt.
R −T
RT
This also holds for −Tkk−1 e−itx ϕ(t)dt = −T0 0 e−itx ϕ(t)dt. Then,
Z Tn
Z T0
1
1
e−itx ϕ(t)dt =
·
(2n
+
1)
e−itx ϕ(t)dt
2Tn −Tn
2 · (2n+1)
π
−T
0
h
Z T0
h
e−itx ϕ(t)dt
=
2π −T0
Z πh
h
=
e−itx ϕ(t)dt.
2π − πh
Therefore,
h
2π
P (X = x) =
Z
h
π
e−itx ϕ(t)dt.
h
−π
(iii) X = Y + b. If P (X ∈ b + hZ) = 1, then P (Y ∈ hZ) = 1. For x ∈ b + hZ, P (X = x) =
P (Y = x − b). Then,
h
2π
Z
h
=
2π
Z
h
=
2π
Z
P (Y = x − b) =
π
h
e−it(x−b) ϕY (t)dt
−π
h
π
h
−π
h
π
h
e−itx · eitb · e−itb ϕX (t)dt
e−itx ϕX (t)dt.
−π
h
Exercise 3.3.3. Since X and Y are independent, X and −Y are also independent. Then,
ch.f of X − Y is given by ϕ(t) · ϕ(t) = |ϕ(t)|2 . By Exercise 3.3.2,
Z T
1
P (X − Y = 0) = lim
|ϕ(t)|2 dt.
T →∞ 2T −T
By Exercise 2.1.5,
P (X − Y = 0) =
X
µ {x}
2
.
x
Exercise 3.3.4. For exponential distribution,
f (y) = e−y I(y>0)
ϕ(t) =
1
1 − it
And,
Z
Z
|ϕ(t)|dt =
1
dt =
1 − it
Z
√
1
dt = ∞.
1 + t2
Exercise 3.3.5. ch.f of X1 is given by
ϕ1 (t) = EeitX
1
Z 1
1 itx
itx 1
e
=
e · dx =
2
2it
−1
−1
1
=
eit − e−it
2it
49
1
(cos t + i sin t − cos t + i sin t)
2it
sin t
=
.
t
n
Thus, ch.f of X1 + · · · + Xn is given by ϕ(t) = sint t .
=
Note that
Z
∞
Z
sin t
t
|ϕ(t)|dt = 2
0
n
dt.
For n ≥ 2,
Z
∞
0
sin t
t
n
Z
1
dt =
0
Z
≤
sin t
t
1
n
Z
dt +
1
∞
Z
1dt +
0
1
∞
sin t
t
n
dt
1
dt < ∞.
tn
Therefore, by Thm 3.3.14,
n
sin t
dt
t
−∞
n
n
Z ∞ sin t
sin t
1
=
· cos tx − i
· sin tx dt
2π −∞
t
t
n
Z 1 ∞ sin t
cos txdt.
=
π 0
t
f (x) =
1
2π
Z
∞
e−itx ·
Exercise 3.3.6. In Example 3.3.16, we showed that ch.f of Cauchy distribution is given by
exp(−|t|).
Since X1 , · · · , Xn are independent, by Thm 3.3.2,
ϕX1 +···+Xn (t) =
n
Y
ϕXi (t) =
i=1
n
Y
exp(−|t|) = exp(−n|t|)
i=1
Bu Thm 3.3.1(e),
t
ϕ X1 +···+Xn (t) = ϕX1 +···+Xn
= exp(−|t|),
n
n
which is ch.f of Cauchy. Thus, by Inversion Formula,
X1 +···+Xn d
≡
n
X1 .
2 2
σ t
Exercise 3.3.7. ϕXn (t) = exp − n2 .
2
σ
Since Xn ⇒ X, ϕXn (t) → ϕ∞ (t), ∀t. For t = 1, ϕXn (1) = exp − 2n . Thus,
ϕXn (1) converges ⇐⇒ log ϕXn (1) converges
⇐⇒ −
σn2
converges.
2
Therefore, there exists σ 2 such that σn2 → σ 2. Sinceσn2 ∈ (0, ∞), we can say that σ 2 ∈ [0, ∞].
σ 2 t2
Now, suppose σ 2 = ∞. Then ϕXn (t) = exp − n2
→ 0 if t 6= 0. However, ϕXn (0) = 1, ∀n.
Thus, continuity of ϕX at 0 is violated, which means that Xn ; X. Thus, σn2 → σ 2 ∈ [0, ∞).
50
Exercise 3.3.8. Let ϕn (t) be the ch.f of Xn and ψn (t) be the ch.f of Yn , for 1 ≤ n ≤ ∞.
By Theorem 3.3.17(Continuity Theorem),
ϕn (t) → ϕ∞ (t), ∀t
ψn (t) → ψ∞ (t), ∀t.
Since Xn and Yn are independent, ch.f of Xn + Yn = ϕn (t)ψn (t) , for 1 ≤ n ≤ ∞. Then,
ϕn (t)ψn (t) → ϕ∞ (t)ψ∞ (t), ∀t. Since ϕ∞ , ψ∞ are ch.f, they are continuous at 0. Thus, ϕ∞ ψ∞
is also continuous at 0. By Thm 3.3.17, Xn + Yn ⇒ X∞ + Y∞ .
Exercise 3.3.9. By Thm 3.3.2, ch.f of Sn is given by
ϕSn (t) =
n
Y
ϕj (t).
j=1
Since a.s. convergence implies weak convergence, Sn ⇒ S∞ . By Thm 3.3.17(i),
ϕS∞ (t) = lim ϕSn (t) =
n→∞
∞
Y
ϕj (t), ∀t.
j=1
Exercise 3.3.10. By Exercise 3.3.5, sint t is ch.f of U nif (−1, 1). Let
Y1 , . . . , Yn be i.i.d sym1
metric Bernoulli random variables. P (Y1 = 1) = P (Y1 = −1) = 2 . Define Xi = Y2ii . Then,
P∞
Pn
Q∞
ch.f of Xi = cos 2ti , and thus ch.f of i=1 Xi = i=1 cos 2ti . Claim. i=1 Xi ⇒ U nif (−1, 1).
P
n
X
2k + 1
Xi =
2n
i=1
!
=
1
for k = −2n−1 , · · · , −1, 0, 1, · · · , 2n−1 − 1.
2n
Then,
P
n
X
!
Xi ≤ x
b2n xc − 1
+ 2n−1 , 0
2
n
1
1 b2 xc − 1
= max
+ n
,0
2 2
2
= max
i=1
1
2n
Now, for x ∈ (−1, 1), letting n → ∞ gives us
P
∞
X
!
Xi ≤ x
=
i=1
1 1
+ x.
2 2
Note that for x ∈ (−1, 1), P (U nif (−1, 1) ≤ x) = 21 + 21 x. Therefore,
Pn
i=1
Xi ⇒ U nif (−1, 1).
By Theorem 3.3.17, we get the desired result.
Exercise 3.3.11. ch.f of
2Xj
3j
= Eeit
2Xj
3j
=
by
ϕX (t) =
1
2
+
1
2
∞ Y
1
j=1
51
2it
· e 3j . By Exercise 3.3.9, ch.f of X is given
1 2it
+ e 3j .
2 2
Then,
k ∞ Y
2i3 π
1 1
ϕX 3k π =
+ exp
2
2
3j
j=1
=
∞ Y
1
k Y
1 1
1
−l
k−j
+ exp 2π · 3 i ·
+ exp 2π · 3 i
2 2
2 2
j=1
l=1
Since 3
k−j
∈ N ∪ {0}, we get
k
ϕX 3 π =
∞ Y
1
1
−l
+ exp 2π3 i .
2 2
j=1
R∞
R∞
1 2
Exercise 3.3.12. We have −∞ |x|e
dx = 2 0 xe− 2 x dx < ∞. By induction, for n ≥ 2,
Z ∞
Z ∞
1 2
n − 21 x2
xn e− 2 x dx
dx = 2
|x| e
− 12 x2
−∞
0
Z ∞
h
i
1 2 ∞
1 2
+2
xn−1 e− 2 x dx
= 2 −xn−1 e− 2 x
0
0
Z ∞
1 2
=0+2
xn−1 e− 2 x dx < ∞.
0
Note that
1 2
ϕ(n) (t) = exp− 2 t
2
3
1
1 2
1
1 2
1 2
+ · − t
+ ···
=1− t + · − t
2
2!
2
3!
2
∞
X
1
=1+
t2k .
k! · (−2)k
k=1
By Thm 3.3.18, we have in EX n = ϕ(n) (t)|t=0 . Therefore,
EX 2n =
Exercise 3.3.13.
(i) Fix > 0. For 0 < h <
(2n)!
.
2n n!
2
4 ,
|ϕi (t + h) − ϕi (t)| ≤ E|eihXi − 1|
≤ E min(|hXi |, 2)
2
2
≤ E |hXi |; |Xi | ≤ √
+ E 2; |Xi | > √
h
h
√
2
≤ 2 h + 2P |Xi | > √
h
√
≤2 h+
≤ 2.
The first inequality comes from Theorem 3.3.1(d). The second inequality comes from
(3.3.3). The fifth inequality comes from tightness.
52
(ii) Since µn ⇒ µ∞ , {µn }n∈N is tight, and thus {µn }1≤n≤∞ is tight.
Then, by (i), for > 0, choose δ > 0 so that
if |h| < δ, then |ϕn (t + h) − ϕn (t)| < , for 1 ≤ n ≤ ∞.
Without loss of generality, assume the compact set is [−K, K].
Choose integer m >
1
δ
and let xj =
j
m
· K for −m ≤ j ≤ m. By Continuity Theorem,
we have ϕn (t) → ϕ∞ (t) for all t. Thus, for xj , there exists nj ∈ N such that if n > nj ,
then |ϕn (xj ) − ϕ∞ (xj )| < . Let n0 = max {n−m , · · · , nm }. Then, for all t ∈ [−K, K],
there exists j such that t − jK
m < δ. Now, for n > n0 ,
jK
jK
jK
jK
+ ϕn
+ ϕ∞
|ϕn (t) − ϕ∞ (t)| ≤ ϕn (t) − ϕn
− ϕ∞
− ϕ∞ (t)
m
m
m
m
< 3.
Therefore, ϕn → ϕ∞ uniformly on a compact set.
(iii) Let Xn =
it
1
n.
Then, ϕn (t) = e n

 Sn = ϕ
n
(i) ch.f of
a = eiat
Exercise 3.3.14.
Claim. ϕ
t
n
t n
n
.
→ eiat , for all t.
Proof. Since ϕ0 (0) = ia,
ϕ( nt ) − ϕ(0)
t
= ia ⇒ lim n ϕ
− 1 = iat.
n→∞
n→∞
t/n
n
lim
Thus, for all t,
!n
n
n ϕ( nt ) − 1
t
lim ϕ
= lim 1 +
n→∞
n→∞
n
n
= eiat .
The second equality holds by Theorem 3.4.2.
By Claim and Theorem 3.3.17,
Sn
a
⇒ a, and thus
Sn P
a →
a.
(ii) In probability convergence implies weak convergence. By Theorem 3.3.17, we get the
desired result.
(iii) (I think (ϕ(h) − 1)/h → +ia.)
(a) Since log z is differentiable at z = 1,
log ϕ( nt ) − log 1
log ϕ( nt )
=
lim
= (log z)0
n→∞
n→∞ ϕ( t ) − 1
ϕ( nt ) − 1
n
t n
iat
(b) By (ii), ϕ n → e . By taking logarithm, we get
t
n log ϕ
→ iat.
n
lim
53
z=1
= 1.
By (a), (b), we get
ϕ( nt ) − 1
t
t
n ϕ
−1 =
·
n
log
ϕ
n
n
log ϕ( nt )
→ iat.
By Exercise 3.3.13,
Sn
n
⇒ a implies that ϕ( nt )n → eiat uniformly on a compact set. Fix
small 0 < δ < 1. For any h > 0, we can express h =
t
n
with large n and t ∈ [1 − δ, 1 + δ].
Then,
ϕ( nt ) − 1
ϕ(h) − 1
− ia =
− ia
h
t/n
=
1
(n(ϕ(t/n) − 1) − iat)
t
≤
1
· |n(ϕ(t0 /n) − 1) − iat0 |
t0
sup
t0 ∈[1−δ,1+δ]
≤
1
1−δ
|n(ϕ(t0 /n) − 1) − iat0 |.
sup
t0 ∈[1−δ,1+δ]
Since n(ϕ(t/n) − 1) → iat uniformly on compact set,
"
#
((((
(
(
1
ϕ(h) − 1
(
((
lim sup
sup ((
|n(ϕ(t/n)
− 1) − iat| = 0.
lim sup
− ia ≤
(((
h
1 − δ n→∞
(t∈[1−δ,1+δ]
h→0
(
(
(
Therefore,
ϕ(h)−1
h
→ ia as h ↓ 0.
Exercise 3.3.15. From Example 3.3.15,
R∞
−∞
1−cos x
πx2 dx
= 1.
Let x = |y|t. Then,
Z
∞
−∞
Thus, 2
R∞
0
1−cos yt
dt
πt2
1 − cos |y|t
1
|y|dt =
π|y|2 t2
|y|
Z
∞
−∞
1 − cos yt
dt = 1.
πt2
= |y|.
Now, integrate with repect to F (y). Then, we get
Z ∞Z ∞
Z ∞
1 − cos yt
2
dtdF (y) =
|y|dF (y).
πt2
−∞
−∞ 0
Since
1−cos yt
πt2
≥ 0, by Fubini,
Z ∞
Z ∞
∞Z ∞
1 − Re(ϕ(t))
1 − cos yt
2
dF
(y)dt
=
2
dt
=
|y|dF (y).
πt2
πt2
0
−∞
0
−∞
Z
= c > −∞,
Exercise 3.3.16. Since limt↓0 ϕ(t)−1
t2
ϕ(t) − 1
ϕ(t) − 1
ϕ(−t) − 1
lim
= lim
= lim
= c.
2
2
t↓0
t↓0
t↓0
t
t
t2
Thus,
lim
t↓0
ϕ(t) + ϕ(−t) − 2ϕ(0)
= 2c > −∞.
t2
54
By Theorem 3.3.21, EX 2 < ∞.
By Theorem 3.3.20, ϕ(t) = 1 + itEX −
t2
2
2 EX
+ o(t2 ).
Then,
2
lim
t↓0
itEX − t2 EX 2 + o(t2 )
ϕ(t) − 1
=
lim
t↓0
t2
t2
iEX
1
= lim
− EX 2
t↓0
t
2
= c > −∞.
Thus, EX = 0 and EX 2 = −2c < ∞.
ϕ(t)−1
t2
Especially, if ϕ(t) = 1 + o(t2 ), we get limt↓0
= limt↓0
o(t2 )
t2
= 0. Then, EX = 0 and
EX 2 = 0. Thus, Var(X) = 0, which means that X is constant a.s. Therefore, X = 0 a.s.
Thus, ϕ(t) = EeitX = 1.
Exercise 3.3.17.
(⇒) By Continuity Theorem, ϕn (t) → ϕ0 (t), for all t, where ϕ0 is ch.f of 0. Since ϕ0 (t) =
Eeit0 = 1, ϕn (t) → 1, for all t.
(⇐) Step 1. {Yn } is tight.

|f (t) ≤ 2 I
n
α [−α,α] (t) : integrable
Let fn (t) = α1 (1−ϕn (t))I[−α,α] (t) for α < δ. Then,
f (t) → 0 by assumption
n
By DCT, we have
Z
fn (t) =
1
α
Z
α
(1 − ϕn (t))dt → 0.
−α
Note that
Z
Z Z ∞
1 u
1 u
ity
(1 − ϕn (t))dt =
1−
e dµn (y) dt
u −u
u −u
−∞
Z
Z
1 ∞ u
=
1 − eity dtdµn (y)
u −∞ −u
u
Z
1 ∞
1
=
t − eity
dµn (y)
u −∞
iy
−u
"
Z
1 ∞
1
=
u − (cos uy + i sin uy)
u −∞
iy
#
1
− −u − (cos uy − i sin uy) dµn (y)
iy
Z ∞
1
2
=
2u − sin uy dµn (y)
u −∞
y
Z ∞
sin uy
=2
1−
dµn (y)
uy
−∞
Z
1
≥2
1−
dµn (y)
2
u|y|
|y|≥ u
55
.
Z
≥
1dµn (y)
2
|y|≥ u
2
= µn y : |y| ≥
u
2
.
≥ µn y : |y| >
u
The second equality holds by Fubini. The first inequality holds since
sin uy
uy
≤
sin uy
uy
≤ 1 and
1
u|y| .
Step 2. Yn ⇒ 0.
Fix a subsequence {Ynk }. By Helly and Tightness Theorem, there exists sub-subsequence
Ynm(k) such that Ynm(k) ⇒ X. Since ϕnm(k) → 1 for |t| < δ, ϕX (t) = 1 for |t| < δ.
Then, for |t| < δ,
ϕX (t)−1
t2
= 0, and thus
lim
t↓0
ϕ(t) − 1
= 0.
t2
By Exercise 3.3.16, ϕX (t) ≡ 1. This means that X ≡ 0. Since this holds for all
subsequence, by Theorem 3.2.15, Yn ⇒ 0.
P
Exercise 3.3.18. Step 1. Sm − Sn → 0.
Let ϕ(t) be ch.f of X1 . Then, ch.f of Sn is given by ϕ(t)n and it converges to for some ch.f
ψ(t). Since ψ is ch.f, there exists δ > 0 such that ψ(t) > 0 for |t| ≤ δ. Now, suppose m > n.
Pm
ch.f of Sm − Sn = l=n+1 Xl is given by
ϕ(t)m−n =
ψ(t)m
.
ψ(t)n
For |t| ≤ δ, as m, n → ∞, ϕ(t)m−n → 1.
By Exercise 3.3.17, Sm − Sn ⇒ 0.
P
By Exercise 3.2.12, Sm − Sn → 0.
Step 2. Sn converges in prob.
By Exercise 2.3.6,
d(Sn − Sm , 0) → 0 ⇐⇒ d(Sn , Sm ) → 0.
P
By Exercise 2.3.7, there exists a r.v. S∞ such that Sn → S∞ .
Exercise 19-25 (?)
Exercise 3.3.19.
Exercise 3.3.20.
Exercise 3.3.21.
Exercise 3.3.22.
Exercise 3.3.23.
56
Exercise 3.3.24.
Exercise 3.3.25.
57
3.4. Central Limit Theorems.
Exercise 3.4.1. Φ(−1).
Exercise 3.4.2. (a) Let σ 2 = Var(Xi ) ∈ (0, ∞). By CLT, we have
√Sn
nσ
⇒ Z ∼ N (0, 1).
Then, for any M ,
Sn
Sn
M
lim sup P √ > M = lim sup P √
>
= P (Z > M/σ) > 0.
σ
n
nσ
n
n
Therefore,
S
S
√n > M i.o. ≥ lim sup P √n > M > 0.
n
n
n
o
n
S
Note that lim supn √nn > M ∈ T : tail σ-algebra.
Sn
By Kolmogorov’s 0-1 law, P lim supn √
>
M
= 1, for all M .
n
P
Sn
√
= ∞ a.s.
n
P
Sn
√
→ U for some
n
Thus, lim supn
(b) Assume that
U.
Then, as m → ∞,
S(m+1)!
S
P
√m! − p
→ 0.
m!
(m + 1)!
Since Sm! and S(m+1)! − Sm! are independent, we get
!
S(m+1)! − Sm!
Sm!
Sm!
P 1< √
< 2, p
< −3 = P 1 < √
<2 P
m!
m!
(m + 1)!
!
S(m+1)! − Sm!
p
< −3
(m + 1)!
→ P (1 < T < 2)P (T < −3) > 0,
(2a)
where T ∼ N (0, σ 2 ). For the second term, √
1
Sm!
(m+1)!
=√
1
(m+1)!
·
Sm! P
√
→
m!
0.
Also,
P
!
S(m+1)! − Sm!
Sm!
1< √
< −3 = P
< 2, p
m!
(m + 1)!
S(m+1)!
Sm!
Sm!
1< √
< −3 + p
< 2, p
m!
(m + 1)!
(m + 1)!
!
S(m+1)!
Sm!
Sm!
< −3 + √
1< √
< 2, p
m!
m!
(m + 1)!
!
S(m+1)!
Sm!
1< √ , p
< −1
m!
(m + 1)!
!
S(m+1)!
Sm!
√ −p
>2 .
m!
(m + 1)!
≤P
≤P
≤P
(2b)
By combining (2a) and (2b), we get
lim inf P
m
!
S(m+1)!
Sm!
√ −p
> 2 > 0,
m!
(m + 1)!
which contradicts to the assumption that
Sm!
√
m!
58
S
P
− √(m+1)! → 0.
(m+1)!
!
Exercise 3.4.3.



Xi0 : independent copy of Xi




Y = X − X 0
i
i
i


Ui = Yi 1(|Yi |≤A)




V = Y 1
i
i (|Yi |>A)
We know that (Xi ), (Xi0 ), (Yi ), (Ui ), (Vi ) are i.i.d sequences. Suppose EXi2 = ∞.
Pn
Sn
⇒ W , Sn0 = i=1 Xi0 also weakly converges
Since there exists some r.v. W such that √
n
to W . That is, for all t, ϕW (t) = limn→∞ ϕSn /√n (t) = limn→∞ ϕSn0 /√n (t). Then, ch.f of
Pn
Ym
m=1
√
n
is given by
Sn
Sn0
√
√
ϕ Pnm=1
(t)
=
E
exp
it
−
Ym
√
n
n
n
Sn (t) · ϕ S 0 (−t)
= ϕ√
√n
n
n
→ ϕW (t) · ϕW (−t), ∀t.
SincePϕW (t) · ϕW (−t) is continuous at 0, by Continuity Theorem, there exists a r.v. T such
that
n
m=1
√
Ym
n
⇒ T . By hint, for any K, which is a continuity point of d.f of T ,
Pn
1
m=1 Ym
√
≥K ≥ .
P (Y ≥ K) = lim P
n→∞
5
n
This means that limK→∞ P (T ≥ K) ≥
1
5
since discontinuity points of d.f of T is at most
countable, which is a contradiction. One can derive hint from CLT and MCT.
Exercise 3.4.4. Let Sn = X1 + · · · + Xn . CLT for i.i.d sequence implies that
(4a)
Note that
S√
n −n
nσ
√
=
By WLLN, we have
Sn − ESn
Sn − n
√
= √
⇒ χ.
nσ
nσ
√
√
· ( Sn − n).
√
S√
n+ n
nσ
Sn P
n → 1.
(EX1 = 1, Var(X1 ) < ∞).
By Continuous Mapping Theorem,
(4b)
1
σ
√
S
P 2
√ n +1 → .
σ
n
By Converging Together Lemma with (4a) and (4b),
√
−1
p
√
1
σ
Sn − n
S
√
√ n +1
Sn − n =
·
⇒ χ
σ
2
nσ
n
Exercise 3.4.5. By CLT,
(5a)
By WLLN,
Pn
m=1 Xm
√
⇒ χ.
nσ
Pn
m=1
2
Xm
n
59
P
→ σ2 .
By Continuous Mapping Theorem,
r Pn
m=1
(5b)
2
Xm
n
P
→ σ2 .
By Converging Together Lemma with (5a) and (5b),
s
Pn
Pn
nσ 2
m=1 Xm
m=1 Xm
√
⇒ χ.
=
· Pn
Pn
2
1/2
2 )
nσ
( m=1 Xm
m=1 Xm
Exercise 3.4.6. By CLT for i.i.d sequence, we know that
quence
San
√
σ an
S√n
σ n
⇒ N (0, 1). Thus, its subseP
⇒ N (0, 1). For Yn and Zn defined above, if one can show that Yn − Zn → 0,
SNn
Sa
Sa − SNn
Sa
= √ n + (Yn − Zn ) ⇒ N (0, 1)
√ = √n + n √
σ an
σ an
σ an
σ an
P
by Converging Together Lemma.
Claim. Yn − Zn → 0.
Nn
Nn P
Fix > 0. Since an → 1, P an − 1 > = P |Nn − an | > an → 0. Then,
h
i
lim sup P (|Yn − Zn | > δ) = lim sup P (|Yn − Zn | > δ, |Nn − an | > an ) + P (|Yn − Zn | > δ, |Nn − an | ≤ an )
n
n
≤ lim sup P (|Yn − Zn | > δ, |Nn − an | > an ) + lim sup P (|Yn − Zn | > δ, |Nn − an | ≤ an )
n
n
= lim sup P (|Yn − Zn | > δ, |Nn − an | ≤ an ).
n
Note that
√
P (|Yn − Zn | > δ, |Nn − an | ≤ an ) = P (|SNn − San | > δ · σ an , |Nn − an | ≤ an ).
Since |Nn − an | ≤ an ,
h
i
√
P |SNn − San | > δσ an , |Nn − an | ≤ an ≤ P
√ δσ an
max
|Sm − S[(1−)an ] | >
2
(1−)an ≤m≤(1+)an
4
≤ 2 2 Var S[(1+)an ] − S[(1−)an ]
δ σ an
4
= 2 2 σ 2 [(1 + )an ] − [(1 − )an ]
δ σ an
8
≤ 2
δ
The second inequality holds by Kolmogorov’s Maximal Inequality.
Thus, lim supn P (|Yn − Zn > δ) ≤
8
δ2 .
Since was arbitrarily chosen, P (|Yn − Zn | > δ) → 0,
P
for any δ > 0. Therefore, Yn → Zn .
Exercise 3.4.7. 7
Exercise 3.4.8. 8
Exercise 3.4.9. (1) Var(Sn )/n → 2.
2
Var(Xn ) = EXm
= 2 − m12 . Then, Var(Sn ) = 2n −
we get the desired result.
60
Pn
1
m=1 m2 .
Since
Pn
1
m=1 m2
converges,
√
(2) Sn / n ⇒ χ.
Define Ym =

1
, Xm > 0
. Then, Ym ’s are i.i.d sequence with EYm = 0 and
, Xm < 0
Var(Ym ) = 1. By CLT for i.i.d, we have
−1
n
X
Ym
.√
n ⇒ χ.
m=1
P∞
Note that P (Xm 6= Ym ) = P (Xm = ±m) = m12 . Then,
m=1 P (Xm 6= Ym ) =
P∞
1
<
∞.
By
Borel-Cantelli
Lemma
1,
P
(X
=
6
Y
i.o.)
= 0. Thus, P (Xm =
m
m
m=1 m2
Ym eventually) = 1. This means that
Pn
Pn
Ym
Sn − m=1 Ym
S
√n = m=1
√
√
+
⇒χ
n
n
n
by converging together Lemma since the second term converges to 0.
Pn
Exercise 3.4.10. Let s2n = m=1 Var(Xm ) and Ynm =
P
n
t2n := m=1 Var(Ynm ) = 1. For Ynm ,
Pn
2
2
(i)
m=1 EYnm = tn = 1
Xm −EXm
.
sn
Then, EYmn = 0 and
(ii) For > 0, |Ynm | > ⇐⇒ |Xm − EXm | > sn .
Note that since |Xi | ≤ M , |Xm − EXm | ≤ 2M , ∀m. However, since
P∞
n=1
Var(Xn ) =
∞, Sn > 2M for sufficiently large n. This means that {|Ynm | > } does not occur
eventually. Thus,
lim
n→∞
n Z
X
m=1
|Ynm |2 dP = 0.
|Ynm |>
Pn
By (i) and (ii), Lindeberg Theorem gives us that m=1 Ynm ⇒ χ. Therefore,
Pn
Pn
Sn − ESn
m=1 Xm − E (
m=1 Xm )
pPn
=p
⇒ χ.
Var(Sn )
Var(X
)
m
m=1
Exercise 3.4.11. By Exercise 3.4.12, it suffices to show that {Xn } satisfies Lyapounov’s
condition.
lim
n→∞
= lim
n→∞
αn−(2+δ)
1
n1+δ/2
n
X
E(|Xm − EXm |2+δ )
m=1
n
X
E|Xm |2+δ
m=1
C
= lim δ/2
n→∞ n
=0.
Exercise 3.4.12. Let Xn,m =
Xm −EXm
.
αn
It suffices to show that {Xn,m } satisfies the Lin-
deberg condition.
Pn
2
(i)
m=1 EXn,m = 1.
61
(ii) For given > 0,
n
X
2
E(Xn,m
; |Xn,m |
> ) =
m=1
n Z
X
|Xn,m |>
m=1
=
1
αn2
1
≤ 2
αn
2
Xn,m
dP
n
X
Z
|Xm − EXm |2 dP
|Xm −EXm |>αn
m=1
n Z
X
m=1
|Xm −EXm |>αn
"
≤
|Xm − EXm |
αn
#
δ
|Xm − EXm |2 dP
n Z
1 X
1
·
|Xm − EXm |2+δ dP → 0
δ
αn2+δ m=1
by assumption.
By (i) and (ii), we get the desired result.
Exercise 3.4.13.
(i) β > 1.
P (Xj 6= 0) = P (Xj = ±j) =
∞
X
1
.
jβ
Thus,
P (Xj 6= 0) =
j=1
∞
X
1
< ∞.
β
j
j=1
By Borel-Cantelli Lemma 1, P (Xj = 0eventually) = 1. For ω ∈ Ω0 := {ω : Xj (ω) = 0 eventually},
P∞
j=1 Xj (ω) converges. Thus, Sn → S∞ a.s. , where S∞ is a r.v. such that S∞ (ω) =
P∞
j=1 Xj (ω) for ω ∈ Ω0
(ii) β < 1.
Let



X

 nm
Ynm



T
n
= Xm for 1 ≤ m ≤ n, ∀n
=
Xnm
n(3−β)/2
=
Pn
m=1
Ynm =
Sn
n(3−β)/2
1
2
Then, EYnm = 0 and EYnm
= (3−β
m2−β .
Pn
Pn n 2−β
1
1
2
≤ n(3−β)
(a)
· n · n2−β = 1.
m=1 Ynm = n(3−β)
m=1 m
Pn
Pn
2
2
→ c2 ≤ 1
Since m=1 Ynm
is bounded and monotonically increasing, m=1 Ynm
for some c > 0.
Pn
Pn
2
(3−β)/2
2
).
m=1 E(|Ynm | ; |Ynm | > ) = limn→∞
m=1 E(|Ynm | ; |Xnm | > ·n
3−β
(3−β)/2
(3−β)/2
Since β < 1, 2 > 1. Then, for large n, ·n
> n.Then, |Xnm | > · n
=
Pn
2
(3−β)/2
∅. Thus, limn→∞ m=1 E(|Ynm | ; |Xnm | > · n
) = 0.
Sn
(a) and (b), Lindeberg Theorem implies that Tn = n(3−β)/2 ⇒ cχ.
(b) limn→∞
By
(iii) β = 1.
ϕSn /n (t) =
n
Y
ϕXj (t/n)
j=1
62
n Y
jt
1
=
1 − cos
1−
j
n
j=1
=
n
Y
j=1
1
1− ·
n
−1 !
j
jt
· 1 − cos
.
n
n
Note that
−1 Z 1
n
X
1
jt
1 j
1 − cos
→
(1 − cos xt)dx.
n n
n
0 x
j=1
jt
1
Also, 1j cos jt
n − 1 ≤ 0 for all j and limn→∞ j cos n − 1 → 0.
Thus, max1≤j≤n 1j cos jt
n − 1 → 0.
Exercise ?? with cj,n = − 1j 1 − cos jt
n implies
n Y
jt
1
Sn (t) =
1 − cos
1+ −
ϕ√
n
j
n
j=1
Z 1
1
→ exp −
(1 − cos tx)dx .
0 x
Since the last one is the ch.f of χ, it is continuous at t = 0. By Continuity theorem, we
get Sn /n ⇒ χ.
63
3.5. Local Limit Theorems. No exercise
3.6. Poisson Convergence. No exercise
3.7. Poisson Processes.
Exercise 3.7.1. 1
Exercise 3.7.2. 2
Exercise 3.7.3. 3
Exercise 3.7.4. 4
Exercise 3.7.5. 5
Exercise 3.7.6. 6
Exercise 3.7.7. 7
Exercise 3.7.8. 8
Exercise 3.7.9. 9
3.8. Stable Laws∗ .
Exercise 3.8.1. 1
Exercise 3.8.2. 2
Exercise 3.8.3. 3
Exercise 3.8.4. 4
Exercise 3.8.5. 5
Exercise 3.8.6. 6
Exercise 3.8.7. 7
3.9. Infinitely Divisible Distributions∗ .
Exercise 3.9.1. 1
Exercise 3.9.2. 2
Exercise 3.9.3. 3
Exercise 3.9.4. 4
64
3.10. Limit Theorems in Rd .
Exercise 3.10.1.
P (Xi ≤ x) = P (Xi ≤ x, Xj ∈ (−∞, ∞) for j 6= i)
= lim P (X1 ≤ y, · · · , Xi−1 ≤ y, Xi ≤ x, Xi+1 ≤ y, · · · , Xd ≤ y)
y→∞
= lim F (y, · · · , y, x, y, · · · , y),
y→∞
where x lies at i-th coordinate.
Exercise 3.10.2. Omit.
Exercise 3.10.3. Omit.
Exercise 3.10.4. Portmanteu
Exercise 3.10.5. Let X be a r.v. whose ch.f is ϕ and Y = (Y1 , . . . , Yn ) be a random vector
whose ch.f is ψ.
Then,
ϕ(t1 + · · · + td ) = E exp[i(t1 + · · · td )X]
ψ(t1 , . . . , td ) = E exp[itT Y]
= E exp[i(t1 Y1 + · · · + td Yd )].
Thus, Y = (X, . . . , X)
Exercise 3.10.6. Let X = (X1 , . . . , Xk )T .
T
(⇒) ϕX1 ,··· ,Xk (t) = Eeit
X
= Eeit1 X1 +···+tk Xk
= Eeit1 X1 · · · Eeitk Xk
=
k
Y
ϕXj (tj )
j=1
The third equality holds since Xi ’s are independent.
(⇐) For A = [a1 , b1 ] × · · · × [ak , bk ] with µX (∂A) = 0,




Z
k
Y

µX (A) = lim (2π)−k
ψj (tj ) ϕX (t)dt
T →∞
[−T,T ]k

= lim (2π)−k
T →∞

= lim 
T →∞
k
Y
j=1
j=1
k Y
Z
[−T,T ]k j=1
(2π)−1

ψj (tj )ϕXj (tj ) dt1 · · · dtj 
!
Z
ψj (tj )ϕXj (tj )dtj 
[−T,T ]
65
=
k
Y
j=1
=
k
Y
"
−1
#
Z
lim (2π)
T →∞
ψj (tj )ϕXj (tj )dtj
[−T,T ]
µXj [aj , bj ] .
j=1
Since it holds for any A, X1 , · · · , Xk are independent.
Exercise 3.10.7. Let X = (X1 , . . . , Xd )T , θ = (θ1 , . . . , θd )T and Γ = (Γij )


d X
d
X
1
Γij ti tj 
(⇒) ϕX (t) = exp itT θ −
2 i=1 j=1
= E exp(itT X)
= E exp(it1 X1 + · · · itd Xd )
= E exp(it1 X1 ) · · · E exp(itd Xd )
#
"
d
X
1
Γll t2l
= exp itT θ −
2
l=1
The fourth equality holds since Xi ’s are independent. The fifth equality comes from
Exercise 3.10.8 with c = ei . Thus, Γij = 0 for i 6= j.
(⇐) Similarly,


d X
d
X
1
ϕX (t) = exp itT θ −
Γij ti tj 
2 i=1 j=1
"
#
d
X
1
= exp itT θ −
Γll t2l
2
l=1
d
Y
1
=
exp iti θi − Γii t2i
2
i=1
=
d
Y
ϕXi (ti )
i=1
The last equality comes from Exercise 3.10.8 with c = ei .
Exercise 3.10.8. Let X = (X1 , . . . , Xd )T , θ = (θ1 , . . . , θd )T , c = (c1 , . . . , cd )T and Γ = (Γij )
: d × d matrix.
We want to show that
X ∼ Nd (θ, Γ) ⇐⇒ ∀c ∈ Rd , cT X ∼ N (cT θ, cT Γc).
Note that ch.f of X = Nd (θ, Γ) is given by
1
ϕX (t) = exp itT θ − tT Γt .
2
66
T
(⇒) ϕcT X (t) = Eeitc
X
T
= Eei(tc) X
1
= exp i(tc)T θ − (tc)T Γ(tc)
2
1
= exp it(cT θ) − (cT Γc)t2 ,
2
which is ch.f of N (cT θ, cT Γc). Thus, cT X = c1 X1 + · · · + cd Xd ∼ N (cT θ, cT Γc).
T
(⇐) ϕX (t) = Eeit
X
T
= Eei·1·(t
X)
= ϕtT X (1)
1 T
T
2
= exp i · 1 · (t θ) − t Γt · 1
2
1 T
T
= exp it θ − t Γt ,
2
which is ch.f of Nd (θ, Γ). Thus, X ∼ Nd (θ, Γ).
4. Martingales
4.1. Conditional Expectation.
Exercise 4.1.1. Since G, Ω ∈ G,
Z
Z
P(A|G) dP =
1A dP = P(A ∩ G)
ZG
ZG
P(A|G) dP =
1A dP = P(A)
Ω
Ω
and the result follows.
Exercise 4.1.2. Let F ∈ F.
Z
Z
Z
Z
|X|2
P(|X| ≥ a|F) dP =
1|X|≥a dP ≤
dP
=
a−2 E |X|2 |F dP.
2
F
F
F a
F
Since F ∈ F was arbitrary, the result follows.
Exercise 4.1.3. Almost surely, we have
0 ≤ EG [(|X| + θ|Y |)2 ] = EG [X 2 ] + 2θEG |XY | + θ2 EG [Y 2 ].
Thus, almost surely,
D/4 = (EG |XY |)2 − EG [X 2 ]EG [Y 2 ] ≤ 0.
Exercise 4.1.4. Let µ : Ω × F → [0, 1] be an r.c.p. given G.
Z
EG |XY | = |X(ω)Y (ω)|µ(ω̃, dω)
67
Z
≤
1/p Z
1/q
q
|X(ω)| µ(ω̃, dω)
|Y (ω)| µ(ω̃, dω)
p
= (EG |X|p )1/p (EG |Y |q )1/q .
Exercise 4.1.5. Let P(a) = P(b) = P(c) = 1/3 and let F1 = σ({a}), F2 = σ({b}). Define X
as



a 7→ 1


X=
b 7→ 2 .



 c 7→ 3
Then,
EF1 X =





a 7→ 1




c 7→ 2.5
b 7→ 2.5 ,



a 7→ 2


F2
E X=
b 7→ 2



 c 7→ 2
and
F2
E
F1
E
X=









a 7→ 1.75
b 7→ 2.5 ,
F1
E
F2
E
X=
c 7→ 1.75





a 7→ 2




c 7→ 2
b 7→ 2 .
Exercise 4.1.6.
E[(X − EF X)2 ] + E[(EF X − EG X)2 ]
= EX 2 + E[(EF X)2 ] − 2E[XEF X] + E[(EF X)2 ] + E[(EG X)2 ] − 2E[(EF X)(EG X)].
Note that E[(EF X)2 ] = E[(EF X)(EF X)] = E[EF [XEF X]] = E[XEF X], and E[(EF X)(EG X)] =
E[EF [XEG X]] = E[XEG X] = E[(EG X)2 ]. Thus the above becomes
= EX 2 − E[(EG X)2 ] = E[(X − EG X)2 ].
Exercise 4.1.7. Note that
Var[X|F] = EF X 2 − (EF X)2 = EF [(X − EF X)2 ].
Thus,
E[Var[X|F]] = E[(X − EF X)2 ].
Also,
Var[EF X] = E[(EF X − EX)2 ].
Thus the result follows from Exercise 4.1.6.
68
Exercise 4.1.8. Let F = σ(N ). First, we need to check that E|X|2 < ∞. Indeed,
X
E|X|2 =
E[|X|2 1{N =n} ]
n
=
X
=
X
=
X
≤
X
E[(Y1 + · · · + Yn )2 1{N =n} ]
n
E[(Y1 + · · · + Yn )2 ]E[1{N =n} ]
n
(nσ 2 + n2 µ2 )P[N = n]
n
Cn2 P[N = n]
n
= CEN 2 < ∞
where for the third equality we used the independence of Yi and N . Now we claim that
EF X = N µ and Var[X|F] = N σ 2 . Note that these are F-measurable. For An = {N = n},
we have
Z
Z
(Y1 + · · · + Yn ) dP
X dP =
An
An
= E[(Y1 + · · · + Yn )1An ]
= E[Y1 + · · · + Yn ]E[1An ]
= nµE[1An ]
Z
=
N µ dP
An
and
Z
X 2 dP =
An
Z
(Y1 + · · · + Yn )2 dP
An
= E[(Y1 + · · · + Yn )2 1An ]
= E[(Y1 + · · · + Yn )2 ]E[1An ]
= (nσ 2 + n2 µ2 )E[1An ]
Z
=
(N σ 2 + N 2 µ2 ) dP.
An
Since F is generated by An , n ≥ 1 by complement and countable disjoint union (cf. Dynkin’s
π-λ theorem), we have the above equalities for all A ∈ F. Thus EF X = N µ and Var[X|F] =
EF X 2 − (EF X)2 = N σ 2 . Finally, from Exercise 4.1.7,
Var[X] = Var[EF X] + E[Var[X|F]] = µ2 Var[N ] + σ 2 EN.
Exercise 4.1.9. From Exercise 4.1.7,
E[(Y − X)2 ] + E[(X − EX)2 ] = E[(Y − EY )2 ].
69
Note that E[(X − EX)2 ] = EX 2 − (EX)2 = EY 2 − (EY )2 = E[(Y − EY )2 ], where the second
equality holds since EX 2 = EY 2 and EX = EY . Thus E[(Y − X)2 ] = 0 and X = Y a.s.
d
Exercise 4.1.10. Fix c ∈ Q. Since E[Y |G] − c = Y − c, we have E|E[Y |G] − c| = E|Y − c|.
Meanwhile, we also have
E|E[Y |G] − c| = E|E[Y − c|G]| ≤ EE[|Y − c||G] = E|Y − c|
and thus
E|E[Y − c|G]| = EE[|Y − c||G]
a.s.
Since |E[Y − c|G]| ≤ E[|Y − c||G], it follows that
|E[Y − c|G]| = E[|Y − c||G]
a.s.
Now let A = {E[Y − c|G] > 0} ∈ G. Multiplying 1A both sides, the last display becomes
E[(Y − c)1A |G] = E[Y − c|G]1A
= |E[Y − c|G]|1A
= E[|Y − c||G]1A
= E[|Y − c|1A |G].
Since (Y − c)1A ≤ |Y − c|1A , it again implies that (Y − c)1A = |Y − c|1A a.s. A similar
argument with B = {E[Y − c|G] ≤ 0} in place of A will show that −(Y − c)1B = |Y − c|1B
a.s. Thus sgn(Y − c) = sgn(E[Y − c|G]) = sgn(E[Y |G] − c). Now suppose that {Y 6= E[Y |G]}
has positive probability. Without loss of generality we may assume that {Y < E[Y |G]} has a
S
positive probability. Since {Y < E[Y |G]} = c∈Q {Y < c < E[Y |G]}, there must exist some
c ∈ Q such that {Y < c < E[Y |G]} has a positive probability. This is a contradiction because
then with a positive probability, sgn(Y − c) 6= sgn(E[Y |G] − c). Therefore Y = E[Y |G] a.s.
4.2. Martingales, Almost Sure Convergence.
Exercise 4.2.1. Since X1 , · · · , Xn are measurable with respect to Gn , Fn ⊂ Gn . Note that
EFn [Xn+1 ] = EFn [EGn Xn+1 ] = EFn Xn = Xn . Thus Xn is a martingale with respect to Fn .
Exercise 4.2.2. Let Xn = −1/n. Then EFn Xn+1 = −1/(n + 1) > −1/n = Xn and
2
= 1/(n + 1)2 < 1/n2 = Xn2 .
EFn Xn+1
Exercise 4.2.3. Note that, almost surely,
EFn [Xn+1 ∨ Yn+1 ] = EFn [Xn+1 1{Xn+1 ≥Yn+1 } ] + EFn [Yn+1 1{Xn+1 <Yn+1 } ]
≥ EFn [Yn+1 1{Xn+1 ≥Yn+1 } ] + EFn [Yn+1 1{Xn+1 <Yn+1 } ]
= EFn [Yn+1 ] = Yn ,
and similarly EFn [Xn+1 ∨ Yn+1 ] ≥ Xn a.s. Thus EFn [Xn+1 ∨ Yn+1 ] ≥ Xn ∨ Yn .
70
Exercise 4.2.4. Let TM = inf{n ≤ 0 : Xn+ > M }. Consider the stopped process XTM ∧n .
Note that XT+M ∧n ≤ M + supj ξj+ , which leads to EXT+M ∧n ≤ M + E supj ξj+ < ∞ for all n.
Thus XTM ∧n converges a.s. That is, Xn converges on the event {TM = ∞}. Since we have
supn Xn < ∞, if we let M → ∞, {TM = ∞} ↑ Ω up to a measure zero set. Therefore, Xn
converges a.s.
Exercise 4.2.5. Let ξj be defined as

1

with probability
−1
j + 1
ξj = 1
j


with probability
j
j + 1
P
where {j } satisfies j j < ∞. By the Borel-Cantelli lemma, P(ξj 6= −1 i.o.) = 0 since
P
P(ξj 6= −1) < ∞. That is, ξj = −1 eventually a.s. and thus Xn → −∞ a.s.
Exercise 4.2.6.
(i) Note that Xn ≥ 0. Since X is a martingale, it is a supermartingale,
and thus Xn converges a.s. to an L1 random variable X. To see X = 0 a.s., note that
P(|Xn+1 − Xn | > δ) = P(Xn |Yn+1 − 1| > δ)
≥ P(Xn > δ, |Yn+1 − 1| > )
= P(Xn > δ)P(|Yn+1 − 1| > ).
Here, P(|Xn+1 − Xn | > δ) converges to 0 since Xn converges a.s. Since P(|Yn+1 − 1| >
p
) > 0 for small , this implies that P(Xn > δ) → 0. That is, Xn → 0. Therefore X = 0
a.s.
(ii) By applying Jensen’s inequality to log(Y ∨ δ), we have E log(Y ∨ δ) ≤ log E[Y ∨ δ].
Letting δ → 0, we have E log Y ≤ log EY = 0. By the strong law of large numbers, it
follows that
1 X
1
log Xn =
log Ym → E log Y ≤ 0.
n
n
m≤n
(Note that this holds even when E log Y = −∞.)
Exercise 4.2.7. Confer Proposition 3.1 in Chapter 5 of Complex Analysis by Stein and
Shakarchi.
Exercise 4.2.8. Observe that, from the given inequality,
Xn+1
Xn
E Qn
Fn ≤ Qn−1
.
(1
+
Y
)
m
m=1
m=1 (1 + Ym )
Thus the random variable
Wn = Qn−1
Xn
m=1 (1
+ Ym )
,
which is also adapted and L1 , is a nonnegative supermartingale, and thus converges a.s. Since
Qn−1
m=1 (1 + Ym ) converges a.s. by Exercise 4.2.7, it follows that Xn converges a.s.
71
Exercise 4.2.9.
(i) On {N > n},
1
2
E[Yn+1 1N >n |Fn ] = E[Xn+1
1N >n+1 + Xn+1
1n<N ≤n+1 |Fn ]
1
1
≤ E[Xn+1
1N >n+1 |Fn ] + E[Xn+1
1n<N ≤n+1 |Fn ]
1
= E[Xn+1
1N >n |Fn ]
1
= E[Xn+1
|Fn ]1N >n
= Xn1 1N >n .
Meanwhile, on {N ≤ n},
2
E[Yn+1 1N ≤n |Fn ] = E[Xn+1
1N ≤n |Fn ]
2
= E[Xn+1
|Fn ]1N ≤n
≤ Xn2 1N ≤n .
Thus E[Yn+1 |Fn ] ≤ Xn1 1N >n + Xn2 1N ≤n = Yn , and Yn is a supermartingale.
(ii) Use the similar argument as the above.
Exercise 4.2.10. content
4.3. Examples.
Exercise 4.3.1. Let X0 = 0. Define Xn+1 as follows: If Xn = 0, then Xn+1 = 1 or −1 with
probability 1/2 each. If Xn 6= 0, then Xn+1 = 0 or n2 Xn with probability 1 − 1/n2 and 1/n2
respectively. Note that E[Xn+1 |Fn ] = Xn . i.e. Xn is indeed a martingale. Note also that,
P
since n 1/n2 < ∞, by Borel-Cantelli, P(Xn+1 = n2 Xn i.o.|Xn 6= 0) = 0. That is, given
Xn 6= 0, Xn+1 = 0 except for finitely many times. Thus P(Xn = a i.o.) = 1 for a = −1, 0, 1.
Exercise 4.3.2.
Exercise 4.3.3. Observe that from the given inequality,
E[Xn+1 −
n
X
Yk |Fn ] ≤ Xn −
k=1
n−1
X
Yk .
k=1
Thus the random variable
Zn = Xn −
n−1
X
Yk ,
k=1
Pn
which is also adapted and L1 , is a supermartingale. Let NM = inf{ k=1 Yk > M }. Then
PNM ∧n
ZNM ∧n is also a supermartingale. Since ZNM ∧n + M = XNM ∧n + M − k=1
Yk ≥ 0,
it is a nonnegative supermartingale, and thus converges a.s. to an L1 limit. That is, Zn
P
converges to an L1 limit on the event {NM = ∞}. Since
Yk < ∞ a.s., if we let M → ∞,
{NM = ∞} ↑ Ω up to a measure zero set. Therefore, Zn converges a.s. to an L1 limit Z∞
Pn−1
P
a.s.
and thus Xn = Zn + k=1 Yk → Z∞ + Yk < ∞.
72
Exercise 4.3.4. Let (Am )m be a sequence of independent events with P(Am ) = pm . By the
P
Borel-Cantelli lemma, we know that
pm = ∞ if and only if P(Am i.o.) = 1. Thus we show
Q
that P(Am i.o.) = 1 if and only if (1 − pm ) = 0. Suppose first that P(Am i.o.) = 1. Then
0 = P(Acm eventually)
\
Acm )
= lim P(
n→∞
m≥n
Y
= lim
n→∞
(1 − pm ).
m≥n
Note that
∞
Y
(1 − pm ) ≤
m=1
Y
(1 − pm ) → 0
m≥n
Q
Q
and thus (1 − pm ) = 0. Now for the other direction, suppose that (1 − pm ) = 0. Since
Q
pm < 1, 1≤m≤n (1 − pm ) 6= 0, and thus it follows that
Y
\
0=
(1 − pm ) = P(
Acm )
m≥n+1
for all n. Letting n → ∞, we get
P(Acm
m≥n+1
eventually) = 0 and thus P(Am i.o.) = 1.
c
Exercise 4.3.5. Suppose first that P(An | ∩n−1
m=1 Am ) = 1 for some n. This means that
n−1 c
n
c
c
An ⊃ ∩n−1
m=1 Am up to a measure zero set. That is, ∩m=1 Am = ∩m=1 Am \ An is measure zero.
n−1
c
c
Then we immediately have P(∩∞
m=1 Am ) = 0. Thus we may assume that P(An | ∩m=1 Am ) < 1
for all n. By Exercise 4.3.4, it follows that
0=
=
=
∞
Y
c
P(Acn | ∩n−1
m=1 Am )
n=2
∞
Y
P(∩nm=1 Acm )
c
P(∩n−1
m=1 Am )
n=2
c
P(∩∞
m=1 Am )
c
P(A1 )
c
c
and thus P(∩∞
m=1 Am ) = 0. (Note that P(A1 ) 6= 0 from the above assumption.)
Exercise 4.3.6.
(i) Note first that Xn is Fn -measurable and in L1 . To see it is a martin-
gale, we show that
Z
Z
Xn+1 dν =
Ik,n
Xn dν
Ik,n
for all k. If ν(Ik,n ) = 0, then Xn 1Ik,n = 0 and Xn+1 1Ik,n = 0, and thus the above holds.
Now assume ν(Ik,n ) 6= 0 and suppose that Ik,n = Ip+1,n+1 ∪ · · · ∪ Iq,n+1 ∪ Iq+1,n+1 ∪ · · · ∪
Ir,n+1 where ν(Ip+1,n+1 ), · · · , ν(Iq,n+1 ) are nonzero and ν(Iq+1,n+1 ) = · · · = ν(Ir,n+1 ) =
73
0. Then we have
Z
Xn+1 dν = µ(Ip+1,n+1 ) + · · · + µ(Iq,n+1 )
Ik,n
= µ(Ip+1,n+1 ) + · · · + µ(Iq,n+1 ) + µ(Iq+1,n+1 ) + · · · + µ(Ir,n+1 )
= µ(Ik,n )
Z
=
Xn dν.
Ik,n
Thus Xn is a martingale.
(ii) Again, note that Xn is Fn -measurable and in L1 . Here, we show that
Z
Z
Xn+1 dν ≤
Xn dν
Ik,n
Ik,n
for all k. If ν(Ik,n ) = 0, then the inequality trivially holds as the above. Assume
ν(Ik,n ) 6= 0. With the same notation as the above, we have
Z
Xn+1 dν = µ(Ip+1,n+1 ) + · · · + µ(Iq,n+1 )
Ik,n
≤ µ(Ip+1,n+1 ) + · · · + µ(Iq,n+1 ) + µ(Iq+1,n+1 ) + · · · + µ(Ir,n+1 )
= µ(Ik,n )
Z
=
Xn dν.
Ik,n
Exercise 4.3.7. We first assume that µ and ν are finite measures. Let Fn = σ(Am : 1 ≤
m ≤ n) and let µn and νn be restrictions of µ and ν to Fn respectively. Obviously µn νn .
Since Fn is finitely generated for each n, there exists a family of partitions (Ik,n )k,n such that
Fn = σ(Ik,n : 1 ≤ k ≤ Kn ) and {Ik,n+1 : 1 ≤ k ≤ Kn+1 } is a refinement of {Ik,n : 1 ≤ k ≤
Kn }. (Check this.) Define Xn as in Example 4.3.7. Then by definition, Xn satisfies
Z
(∗)
µn (A) =
Xn dνn
A
for A ∈ Fn . (We don’t define Xn as the Radon-Nikodym derivative because we are in the
way of proving the Radon-Nikodym theorem.) Moreover, as we saw in Exercise 4.3.6, Xn
is a martingale, and in particular it is a nonnegative supermartingale. By Theorem 4.2.12,
a.s.
Xn → X where X ∈ L1 (ν). Now by the proof of Theorem 4.3.5,
Z
µ(A) =
X dν + µ(A ∩ {X = ∞}).
A
(Even though Xn is not a Radon-Nikodym derivative, it plays the same role as in the proof
of Theorem 4.3.5 because of (∗) above. Thus the proof works.) Since X ∈ L1 (ν), ν({X =
∞}) = 0 and by the absolute continuity µ({X = ∞}) = 0. Therefore,
Z
µ(A) =
X dν.
A
74
Finally, note that X is measurable with respect to F = σ(An : n ∈ N). This proves the
Radon-Nikodym theorem for finite measures.
Now, let µ and ν be σ-finite. Let Ωk ↑ Ω be satisfying µ(Ωk ) < ∞ and ν(Ωk ) < ∞ for all k.
(Check that we can always find such Ωk ’s.) Define µ(k) (·) = µ(· ∩ Ωk ) and ν (k) (·) = ν(· ∩ Ωk ).
Then µ(k) and ν (k) are finite measures. By the above, we have
Z
(k)
µ (A) =
X (k) dν (k)
A
Where X
(k)
is F-measurable and supported on Ωk . Define X to be equal a.s. to Xk on Ωk .
Note that for j < k,
µ(j) (A) = µ(k) (A ∩ Ωj ) =
Z
X (k) dν (k) =
A∩Ωj
Z
X (k) dν (j) .
A
Because of the uniqueness of (almost sure) limit, X
(k)
=X
(j)
on Ωj . Thus X is well-defined.
Letting k → ∞ and using the continuity of measures and the definition of Lebesgue integral,
Z
Z
(k)
(k)
µ(A) = lim µ (A) = lim
X dν =
X dν.
k→∞
k→∞
A
A
Finally it is obvious that X is F-measurable.
Exercise 4.3.8.
(i) Note that
Z
1 − αn = Fn (0) =
qn (x) dGn (x) = qn (0)(1 − βn ),
(−∞,0]
Z
qn (x) dGn (x) = qn (0)(1 − βn ) + qn (1)βn .
1 = Fn (1) =
(−∞,1]
Thus qn (0) = (1 − αn )/(1 − βn ) and qn (1) = αn /βn . By Theorem 4.3.8, µ ν if and
only if
∞ Z
Y
√
qm dGm > 0.
m=1
Note that
∞ Z
Y
√
qm dGm =
m=1
=
∞ r
Y
1 − αm
m=1
∞ p
Y
1 − βm
r
(1 − βm ) +
(1 − αm )(1 − βm ) +
αm
βm
βm
p
αm βm .
m=1
Thus the condition becomes
∞ p
Y
p
(1 − αm )(1 − βm ) + αm βm > 0.
m=1
(ii)
75
P
Exercise 4.3.9. By the Borel-Cantelli lemma,
αn < ∞ implies that ξn = 0 eventually µP
a.s. and βn = ∞ implies that ξn = 1 infinitely often ν-a.s. Since {ξn = 0 eventually}∩{ξn =
1 i.o.} = ∅, we have µ ⊥ ν.
Exercise 4.3.10. content
Exercise 4.3.11. content
Exercise 4.3.12. content
Exercise 4.3.13. content
4.4. Doob’s Inequality, Convergence in Lp , p > 1.
Exercise 4.4.1. Since X is a submartingale, E[Xk 1N =j |Fj ] = E[Xk |Fj ]1N =j ≥ Xj 1N =j .
Now take the expectation of both sides.
Exercise 4.4.2. On {M = N }, the result is trivial. On {M < N }, let Kn = 1M <n≤N . Then
K is predictable. Note that (K · X)n = XN ∧n − XM ∧n . Since (K · X) is a submartingale,
from E(K · X)0 ≤ E(K · X)k , we have EXM ≤ EXN .
(Another proof ) Let Yn = XN ∧n . Then Y is a submartingale. By Theorem 4.4.1, EXM =
EYM ≤ EYk = EXN .
Exercise 4.4.3. We first observe that A ∈ FN . To see this, note that {N ≤ n} ∩ A = {N ≤
n} ∩ {M ≤ n} ∩ A, and that {M ≤ n} ∩ A ∈ Fn and {N ≤ n} ∈ Fn . Thus {N ≤ n} ∩ A ∈ Fn ,
meaning that A ∈ FN . Now consider {L ≤ n}. Observe that
{L ≤ n} = ({L ≤ n} ∩ A) ∪ ({L ≤ n} ∩ Ac )
= ({M ≤ n} ∩ A) ∪ ({N ≤ n} ∩ Ac ) ∈ Fn
since A ∈ FM and A ∈ FN . Thus L is a stopping time.
Exercise 4.4.4. Let A ∈ FM and L be the stopping time defined as above. Since L ≤
N , by Exercise 4.4.2, EXL ≤ EXN . Note that EXL = EXL 1A + EXL 1Ac = EXM 1A +
EXN 1Ac . Plugging this into the inequality, we have EXN ≥ EXM 1A + EXN 1Ac , which leads
to EXN 1A ≥ EXM 1A . Since A ∈ FM was arbitrary, it follows that E[XN |FM ] ≥ XM .
Exercise 4.4.5. We show that E[E[Y |G]E[Y |F]] = E[E[Y |F]2 ]. Since F ⊂ G,
E[E[Y |G]E[Y |F]] = E[E[Y E[Y |F]|G]]
= E[Y E[Y |F]]
= E[E[Y E[Y |F]|F]]
= E[E[Y |F]2 ].
This proves the result.
76
Exercise 4.4.6. Let A = {max1≤m≤n |Sm | > x}, and let N = inf{m ≥ 0 : |Sm | > x} and
T = N ∧ n. Then N and T are stopping times. Since Sn2 − s2n is a martingale, by Exercise
4.4.1,
0 = E[S02 − s20 ] = E[ST2 − s2T ]
2
= E[(SN
− s2N )1A ] + E[(Sn2 − s2n )1Ac ]
≤ (x + K)2 P(A) + (x2 − s2n )P(Ac ).
Thus we have
0 ≤ (x + K)2 − (x + K)2 P(Ac ) + (x2 − s2n )P(Ac )
and it follows that
P(Ac ) ≤
(x + K)2
(x + K)2
≤
.
s2n + (x + K)2 − x2
s2n
Exercise 4.4.7. Let c ≥ −λ. Since (Xn + c)2 is a nonnegative submartingale, by Doob’s
inequality,
P( max Xm ≥ λ) = P( max (Xm + c)2 ≥ (λ + c)2 )
1≤m≤n
1≤m≤n
≤
E(Xn + c)2
(λ + c)2
=
EXn2 + c2
(λ + c)2
Optimizing over c ≥ −λ, the right-hand-side is minimized when c = EXn2 /λ. Plugging this c
into the inequality, the result follows.
Exercise 4.4.8.
(i)
Z
∞
E(X̄n ∧ M ) =
P(X̄n ∧ M > x) dx
0
Z
1
≤
0
Z
dx +
Z ∞
∞
P(X̄n ∧ M > x) dx
1
1
E(X̄n+ 1X̄n ∧M >x ) dx
x
1
Z ∞Z
1 +
=1+
X̄n 1X̄n ∧M >x dP dx
x
1
Z Z X̄n ∧M
1 +
=1+
X̄ dx dP
x n
Z 1
= 1 + X̄n+ log(X̄n ∧ M ) dP
≤1+
77
(ii) Let f (a, b) = (a log b − a log a)/b. From ∇f = 0 we get log b = log a + 1, or simply
b = ea. Thus the maximum of f is
f (a, ea) = 1/e
which proves the result.
Exercise 4.4.9. From EXn < ∞ and EYn < ∞, all the expectations are finite. Note
that EXm−1 Ym = E[E[Xm−1 Ym |Fm−1 ]] = E[Xm−1 E[Ym |Fm−1 ]] = EXm−1 Ym−1 . Thus
Pn
Pn
m=1 E(Xm − Xm−1 )(Ym − Ym−1 ) =
m=1 (EXm Ym − EXm−1 Ym−1 ) = EXn Yn − EX0 Y0 .
Exercise 4.4.10. Note that for all n,
EXn2 = E(X0 + ξ1 + · · · + ξn )2
= EX02 +
n
X
2
Eξm
+2
m=1
= EX02 +
≤ EX02 +
n
X
m=1
∞
X
n
X
EX0 ξm + 2
m=1
n
X
Eξm ξk
m,k=1
2
Eξm
2
Eξm
<∞
m=1
where the third equality is due to Theorem 4.4.7. By Theorem 4.4.6, the result follows.
Exercise 4.4.11. For all n,
n
Xn2
1
1 X 2
2
=
EX
+
Eξ .
0
b2n
b2n
b2n m=1 m
Pn
2
By Kronecker’s lemma (Theorem 2.5.9), m=1 Eξm
/b2m converges to 0. Also, since EX02 < ∞
E
and bn ↑ ∞, EX02 /b2n converges to 0. Therefore EXn2 /b2n converges to 0, which implies that
sup EXn2 /b2n < ∞. Since Xn /bn is a submartingale, by Theorem 4.4.6 Xn /bn a.s. and in L2 .
As Xn /bn → 0 in L2 , Xn /bn → 0 a.s.
4.6. Uniform Integrability, Convergence in L1 .
Exercise 4.6.1. Let Fn = σ(Y1 , · · · , Yn ) and define Xn = E[θ|Fn ]. Then X is a martingale
with filtration (Fn ). By Theorem 4.6.8, Xn converges a.s. to E[θ|F∞ ]. Now it remains to
show that θ = E[θ|F∞ ]. i.e. θ is F∞ -measurable. It follows by the strong law of large numbers
that almost surely
Y1 + · · · + Yn
→θ
n
and thus θ is F∞ -measurable.
Exercise 4.6.2. Obviously Xn is Fn -measurable and Xn ∈ L1 for all n since |Xn | ≤ K as f
is Lipschitz. Note that
EXn+1 1Ik,n = EXn+1 1I2k+1,n+1 + EXn+1 1I2k,n+1
78
=
f
2k+2
2n+1
2k+1
2n+1
−f
1
1
2n+1
2n+1
2k + 2
2k
=f
−f
2n+1
2n+1
+
f
2k+1
2n+1
−f
1
2n+1
2k
2n+1
1
2n+1
and that
f
k+1
2n
k
2n
−f
1
2n
2k + 2
2k
=f
−
f
.
2n+1
2n+1
EXn 1Ik,n =
1
2n
Thus EXn+1 1Ik,n = EXn 1Ik,n for all k, and by the definition of conditional expectation we
have Xn = E[Xn+1 |Fn ]. That is, (Xn ) is a martingale. Since |Xn | ≤ K for all n, the family
(Xn ) is uniformly integrable. Therefore, by Theorem 4.6.7, Xn converges a.s. and in L1 to
an L1 random variable X∞ . Let ak and bk be endpoints of some Ik,n . Observe that
Z bk
f (bk ) − f (ak ) =
Xn (ω) dω.
ak
By the dominated convergence, it follows that
Z
bk
f (bk ) − f (ak ) =
X∞ (ω) dω
ak
for all such points ak and bk . Since f is continuous and |X∞ | ≤ K, letting ak → a and bk → b
we have the last assertion.
Exercise 4.6.3. By Theorem 4.6.8, E[f |Fn ] → E[f |F∞ ] a.s. and in L1 . Now it remains to
show that f is F∞ -measurable, which is obvious.
Exercise 4.6.4. Let ω ∈ {lim sup Xn < ∞}. We will show that ω ∈ D. Let M (ω) =
lim sup Xn (ω) < ∞. Then there exists N (ω) such that Xn (ω) < 2M (ω) for all n ≥ N (ω).
From the given assumption, it follows that
P(D|X1 , · · · , Xn )(ω) ≥ δ(x)
for all n ≥ N (ω). Since D ∈ σ(X1 , X2 , · · · ), by Theorem 4.6.9, the left-hand-side of the above
display converges to 1D (ω) for all ω ∈ {lim sup Xn < ∞} except on a measure zero set. That
is, we have 1D ≥ δ(x) > 0 for all ω ∈ {lim sup Xn < ∞} \ E where P(E) = 0. It follows that
1D (ω) = 1 and thus ω ∈ D. i.e. {lim sup Xn < ∞} \ E ⊂ D. The result now immediately
follows.
Exercise 4.6.5. content
Exercise 4.6.6. Note that Xn is a martingale as E[Xn+1 |Fn ] = Xn (α+βXn )+(1−Xn )βXn =
Xn . Since |Xn | ≤ 1 for all n, it is a uniformly integrable martingale. Thus Xn converges to
an L1 random variable X almost surely (and in L1 ).
79
To see X ∈ {0, 1}, we first let Bn = {Xn+1 = α + βXn } and B = lim sup Bn . For ω ∈ B,
infinitely often,
Xn+1 (ω) − Xn (ω) = α + βXn (ω) − Xn (ω)
= α(1 − Xn (ω)).
Since Xn converges almost surely, |Xn+1 (ω) − Xn (ω)| < for large enough n. That is,
infinitely often we have
α|1 − Xn (ω)| < .
Thus X(ω) = 1 for almost every ω ∈ B. Now consider C = B c = lim inf Bnc . For ω ∈ C,
eventually,
Xn+1 = βXn
which then immediately implies that X(ω) = 0. Thus X(ω) = 0 for almost every ω ∈ C.
This proves that X ∈ {0, 1}
Finally, since Xn is a martingale, we have EX0 = EXn and by the dominated convergence
EXn → EX. Thus
θ = EX0 = EX = 1 · P(X = 1) + 0 · P(X = 0) = P(X = 1).
Exercise 4.6.7. Note that
E|E[Yn |Fn ] − E[Y |F∞ ]| ≤ E|E[Yn |Fn ] − E[Y |Fn ]| + E|E[Y |Fn ] − E[Y |F∞ ]|
≤ E|Yn − Y | + E|E[Y |Fn ] − E[Y |F∞ ]|
By Theorem 4.6.8, E[Y |Fn ] → E[Y |F∞ ] in L1 and Yn → Y in L1 by the assumption.
4.7. Backwards Martingales.
Exercise 4.7.1. By the Lp maximum inequality, for all n ≤ 0,
p
p
p
E sup |Xm | ≤
E|X0 |p .
p−1
n≤m≤0
Letting n → −∞, it follows that supm≤0 |Xm | ∈ Lp . Let X−∞ be the limit of the backward
martingale Xn (which exists by Theorem 4.7.1). Noting that |Xn −X−∞ |p ≤ (2 supm≤0 |Xm |)p ,
by the dominated convergence we have
E|Xn − X−∞ |p → 0.
Exercise 4.7.2. Note that
|E[Yn |Fn ] − E[Y−∞ |F−∞ ]| ≤ |E[Yn |Fn ] − E[Y−∞ |Fn ]| + |E[Y−∞ |Fn ] − E[Y−∞ |F−∞ ]|
≤ E[|Yn − Y−∞ ||Fn ] + |E[Y−∞ |Fn ] − E[Y−∞ |F−∞ ]|.
80
By Theorem 4.7.3, |E[Y−∞ |Fn ] − E[Y−∞ |F−∞ ]| → 0. To see the convergence of the first term,
let WN = supm,n≤N |Ym − Yn |. Then WN ≤ 2Z ∈ L1 and WN → 0. It follows that
lim sup E[|Yn − Y−∞ ||Fn ] ≤ lim sup E[WN |Fn ]
n→−∞
n→−∞
= E[WN |F−∞ ]
→0
where the last convergence is due to the dominated convergence theorem.
Exercise 4.7.3. Let p = P(X1 = 1, · · · , Xm = 1). Because (Xn ) is exchangeable, P(Xn1 =
n
1, · · · , Xnm = 1) = p for any subsequence (nk ). Thus P(Sn = m) = m
p. Similarly,
P(X1 = 1, · · · , Xk = 1, Xn1 = 1, · · · , Xnm−k = 1) = p for any (nk ), and thus P(X1 =
n−k
1, · · · , Xk = 1, Sn = m) = m−k
p. Therefore,
n−k
n
n−k
n
P(X1 = 1, · · · , Xk = 1|Sn = m) =
=
m−k
m
n−m
m
Exercise 4.7.4. Note that from the exchangeability,
0 ≤ E(X1 + · · · + Xn )2 = nEX12 + n(n − 1)EX1 X2
and thus
EX1 X2 ≥ −
EX12
→ 0.
(n − 1)
Exercise 4.7.5. Let ϕ(x, y) = (x − y)2 and let
−1 X
n
An (ϕ) =
2
ϕ(Xi , Xj ).
1≤i<j≤n
Since An (ϕ) ∈ En , we have
An (ϕ) = E[An (ϕ)|En ]
−1 X
n
=
2
E[ϕ(Xi , Xj )|En ]
1≤i<j≤n
= E[ϕ(X1 , X2 )|En ].
By Theorem 4.7.3, E[ϕ(X1 , X2 )|En ] → E[ϕ(X1 , X2 )|E]. (Note that this is possible because
ϕ(X1 , X2 ) is integrable.) Now since E is trivial due to Example 4.7.6, it follows that
E[ϕ(X1 , X2 )|E] = Eϕ(X1 , X2 )
= EX12 + EX22 − 2EX1 EX2
= 2σ 2 .
4.8. Optional Stopping Theorems.
81
Exercise 4.8.1. Let A ∈ FL and let N = L1A + M 1Ac . Note that N is a stopping time
according to Exercise 4.4.3 and that N ≤ M . Since YM ∧n is a uniformly integrable submartingale, applying Theorem 4.8.3 to this leads to
EY0 ≤ EYN ≤ EYM .
Observe that EYN = EYL 1A + EYM 1Ac . It follows that
EYL 1A ≤ EYM 1A .
Since A ∈ FL was arbitrary, by the definition of conditional expectation,
YL ≤ E[YM |FL ]
and also EYL ≤ EYM .
Exercise 4.8.2. Let N = inf{m : Xm > λ}. Since XN 1N <∞ ≥ λ,
λP(N < ∞) ≤ EXN 1N <∞ ≤ EX0
by Theorem 4.8.4. Observe that {N < ∞} = {supm Xm > λ}.
Exercise 4.8.3. Let Xn = Sn2 − nσ 2 . Note that
E|XT | ≤ EST2 = EST2 1T <∞ + EST2 1T =∞
≤ E(a + ξ)2 1T <∞ + a2
≤ 2a2 + σ 2 < ∞.
To see
E[|Xn 1T >n |; |Xn 1T >n | > a2 + M ] = 0
for all n, we observe that |Xn 1T >n | ≤ a2 1T >n ≤ a2 for all n and thus {|Xn 1T >n | > a2 +M } =
∅. By Theorem 4.8.2, XT ∧n is a uniformly integrable martingale. It follows that
0 = EXT ∧n → EXT = E[ST2 − T σ 2 ].
If ET = ∞, then the result is trivial. For ET < ∞, we have T < ∞ a.s. and thus
σ 2 ET = EST2 ≥ a2 .
Exercise 4.8.4. Let Xn = Sn2 − nσ 2 . Then Xn is a martingale. Since T is a stopping time
with ET < ∞, by the monotone convergence,
EST2 ∧n = σ 2 E[T ∧ n] → σ 2 ET.
In particular, EST2 ∧n ≤ σ 2 ET for all n. i.e. supn EST2 ∧n < ∞. Since ST ∧n is a martingale,
this implies that ST ∧n converges a.s. and in L2 to ST . Therefore, EST2 ∧n → EST2 and the
result follows.
82
Exercise 4.8.5. (a) Let A = p − q and B = 1 − (p − q)2 . Using the same argument as in
Exercise 4.8.3, it follows that (SV0 ∧n − (V0 ∧ n)A)2 − (V0 ∧ n)B is a uniformly integrable
martingale. Thus we have
BEV0 = E(SV0 − V0 A)2
= ESV20 − 2AESV0 V0 + A2 EV02 .
Now since p < 1/2, V0 < ∞ a.s. and SV0 = 0. Thus the above display becomes
BEV0 = A2 EV02 .
Therefore, the second moment of V0 is (not the variance; probably a typo)
EV02 =
B
1 − (p − q)2
EV
=
x.
0
A2
(q − p)3
Exercise 4.8.6. (a) Let φ(θ) = E exp(θξi ) = peθ + qe−θ and Xn = exp(θSn )/φ(θ)n . Then
Xn is a martingale and thus
eθx = EX0 = EXV0 ∧n
=E
exp(θSV0 ∧n )
.
φ(θ)V0 ∧n
Now note that, since θ ≤ 0 and SV0 ∧n ≥ 0, exp(θSV0 ∧n ) ≤ 1 and φ(θ) ≥ 1. By the
bounded convergence theorem, it follows that
exp(θSV0 ∧n )
φ(θ)V0 ∧n
exp(θSV0 )
→E
φ(θ)V0
eθx = E
= E[φ(θ)−V0 ].
(b) From φ(θ) = 1/s, we get pe2θ − eθ /s + q = 0. Solving this with respect to eθ and noting
that eθ ≤ 1, we have
θ
e =
1−
p
1 − 4pqs2
.
2ps
Thus
V0
E[s ] =
1−
p
1 − 4pqs2
2ps
!x
.
4
2
Exercise 4.8.7. Note that E[Sn+1
|Fn ] = Sn4 + 6Sn2 + 1 and E[Sn+1
|Fn ] = Sn2 + 1. From
E[Yn+1 |Fn ] = Yn , we get (2b − 6)n + (b + c − 5) = 0 and thus b = 3 and c = 2. Now from
EY0 = EYT ∧n ,
0 = EST4 ∧n − 6E(T ∧ n)ST2 ∧n + 3E(T ∧ n)2 + 2E(T ∧ n).
83
Noting that |ST ∧n | ≤ a and that ET < ∞ (from the proof of Theorem 4.8.7), apply the DCT
and MCT to get
0 = a4 − 6a2 ET + 3ET 2 + 2ET.
Now ET = a2 leads to
ET 2 =
5a4 − 2a2
.
3
Exercise 4.8.8. Note that Sτ ∧n ≥ a, and hence Xn ≤ eaθ0 for all n. Thus Xτ ∧n is a
uniformly integrable martingale, and by Theorem 4.8.2, 1 = EX0 = EXτ . Now note that
1 = EXτ ≥ E[Xτ ; Sτ ≤ a]
≥ eaθ0 P(Sτ ≤ a)
from which the result follows.
Exercise 4.8.9. By the same argument as in Exercise 4.8.8, XT ∧n is a uniformly integrable
martingale and we have 1 = EX0 = EXT . Now observe that
1 = EXT = E[XT ; T < ∞] + E[XT ; T = ∞]
= eaθ0 P(T < ∞) + E[X∞ ; T = ∞].
Here, X∞ = exp(θ0 S∞ ) = 0 since S∞ = lim Sn = ∞ a.s. by the strong law of large numbers.
Therefore,
P(T < ∞) = e−aθ0 .
Exercise 4.8.10. Note that φ(θ) = E exp(θξ) = (e−θ + eθ + e2θ )/3. Solving φ(θ) = 1 with
√
√
θ ≤ 0, we get eθ = 2 − 1. By Exercise 4.8.9, P(T < ∞) = eθi = ( 2 − 1)i .
Exercise 4.8.11. Let φ(θ) = E exp(θξ) = exp((c − µ)θ + σ 2 θ2 /2). Note that θ0 = −2(c −
µ)/σ 2 ≤ 0 satisfies φ(θ0 ) = 1. For this θ0 , Xn = exp(θ0 Sn ) is a martingale. Now let
T = inf{Sn ≤ 0}. By the same argument as in Exercise 4.8.8, XT ∧n is a uniformly integrable
martingale. Thus
E exp(θ0 S0 ) = E exp(θ0 ST )
≥ E[exp(θ0 ST ); T < ∞]
≥ P(T < ∞).
Now plugging θ0 = −2(c − µ)/σ 2 into the last display, the result follows.
4.9. Combinatorics of Simple Random Walk.
Exercise 4.9.1. content
5. Markov Chains
5.1. Examples.
84
Exercise 5.1.1.
P(Xn+1 = j|Xn = i, Xn−1


1 − i
N
= in−1 , · · · , X0 = i0 ) =
i


N
=: p(i, j)
for j = i + 1,
for j = i.
Exercise 5.1.2. Note that
P(X4 = 2, X3 = 1, X2 = 0, X1 = 0)
P(X3 = 1, X2 = 0, X1 = 0)
P(S4 = 2, S3 = 1, S2 = 0, S1 = −1)
1
=
= ,
P(S3 = 1, S2 = 0, S1 = −1)
2
P(X4 = 2|X3 = 1, X2 = 0, X1 = 0) =
P(X4 = 2, X3 = 1, X2 = 1, X1 = 1)
P(X3 = 1, X2 = 1, X1 = 1)
P(S4 = 2, S3 = 1, S2 = 0, S1 = 1)
1
=
= .
P(S2 = 0, S1 = 1)
4
P(X4 = 2|X3 = 1, X2 = 1, X1 = 1) =
Thus Xn is not a Markov chain.
Exercise 5.1.3. Note that Xi ∈ σ(ξn , · · · , ξ0 ) for i ≤ n − 1. Since Xn+1 = (ξn+1 , ξn+2 ),
Xn+1 is independent of {Xn−1 , · · · , X0 }. Thus,
P(Xn+1 = j|Xn = i, Xn−1 = in−1 , · · · , X0 = i0 ) = P(Xn+1 = j|Xn = i).
That is, Xn is a Markov chain. The transition probabilities are
p
HH
HT
TH
TT
HH HT TH
1/2 1/2 0
0
0 1/2
1/2 1/2 0
0
0 1/2
TT
0
1/2
0
1/2
p2
HH
HT
TH
TT
HH
1/4
1/4
1/4
1/4
TT
1/4
1/4
1/4
1/4
and
HT
1/4
1/4
1/4
1/4
TH
1/4
1/4
1/4
1/4
Indeed, the distribution of Xn+2 = (ξn+2 , ξn+3 ) is independent of Xn = (ξn , ξn+1 ).
Exercise 5.1.4. The transition probabilities are
85
AA,AA
AA,Aa
AA,aa
Aa,Aa
Aa,aa
aa,aa
AA,AA
1
1/4
0
1/16
0
0
AA,Aa
0
1/2
0
1/4
0
0
AA,aa
0
0
0
1/8
0
0
Aa,Aa
0
1/4
1
1/4
1/4
0
Aa,aa
0
0
0
1/4
1/2
0
aa,aa
0
0
0
1/16
1/4
1
Exercise 5.1.5.
P(Xn+1

m−i


× b−i

m
 m
m−i
m−b+i
= j|Xn = i) =
+
m ×
m



 i × m−b+i
m
for j = i + 1
i
m
×
b−i
m
for j = i
for j = i − 1
m
Exercise 5.1.6. Let Kn = |{1 ≤ i ≤ n : Xi = 1}|. Note that Sn = Kn − (n − Kn ) = 2Kn − n,
or Kn = (Sn + n)/2. From the elementary definition of conditional probability,
R 1 K +1
θ n (1 − θ)n−Kn
.
P(Xn+1 = 1|Xn , · · · , X1 ) = 0R 1
θKn (1 − θ)n−Kn
0
Here, integrating by parts,
Z 1
Z
θm (1 − θ)k = (−1)2m
0
0
1
m!k!
m!
(1 − θ)k+m =
.
(k + 1) · · · (k + m)
(k + m + 1)!
Thus,
P(Xn+1 = 1|Xn , · · · , X1 ) =
Kn + 1
Sn + n + 2
=
.
n+2
2(n + 2)
Now, note that
P(Sn+1 = sn+1 |Sn , · · · , S1 ) =

P(X
n+1
= 1|Xn , · · · , X1 )
for sn+1 = Sn + 1
1 − P(X
for sn+1 = Sn − 1
n+1 = 1|Xn , · · · , X1 )

 Sn +n+2
for sn+1 = Sn + 1
2(n+2)
=
1 − Sn +n+2
for s
= S − 1.
2(n+2)
n+1
n
That is, the transition probability depends only on Sn . Therefore, Sn is a Markov chain.
5.2. Construction, Markov Properties.
Exercise 5.2.1.
Pµ (A ∩ B|Xn ) = Eµ [Eµ [1A 1B |Fn ]|Xn ]
= Eµ [1A Eµ [1B |Fn ]|Xn ]
(∗)
= Eµ [1A Eµ [1B |Xn ]|Xn ]
= Eµ [1A |Xn ] Eµ [1B |Xn ]
Here, (∗) is from the Markov property.
86
Exercise 5.2.2. Let Fn = σ(X1 , · · · , Xn ), F∞ = σ(X1 , X2 , · · · ). Note that
S
m≥n+1 {Xm
∈
Bm } ∈ F∞ . By Lévy’s 0-1 law, on {Xn ∈ An },
[
[
(∗)
δ ≤ P(
{Xm ∈ Bm }|Xn ) = P(
{Xm ∈ Bm }|Fn )
m≥n+1
m≥n+1
→
1S
m≥n+1 {Xm ∈Bm }
where (∗) is from the Markov property. That is, for ω ∈ {Xn ∈ An }, 1Sm≥n+1 {Xm ∈Bm } (ω) =
1, or in other words,
[
{Xn ∈ An } ⊂
{Xm ∈ Bm }.
m≥n+1
Thus,
[
[
{Xn ∈ An } ⊂
n≥k
{Xm ∈ Bm }
m≥k+1
and letting k → ∞, {Xn ∈ An i.o.} ⊂ {Xn ∈ Bn i.o.}.
Exercise 5.2.3. Let Aδ = {x : Px (D) ≥ δ}. Then obviously PXn (D) ≥ δ on {Xn ∈ Aδ }.
Note that

PXn (D) = P 

[
{Xm = a}|Xn  ≥ δ
m≥n+1
on {Xn ∈ Aδ }. By the previous exercise,
P({Xn ∈ Aδ i.o.} − {Xn = a i.o.}) = 0.
From {Xn = a i.o.} ⊂ D, we have
{Xn ∈ Aδ i.o.} ⊂ {Xn = a i.o.} ⊂ D.
This implies that P(Xn ∈ Acδ eventually) = 1 on Dc . That is, on Dc , PXn (D) ≤ δ eventually
almost surely. Since δ is arbitrary, we have the desired result.
Exercise 5.2.4.
pn (x, y) = Px (Xn = y) =
=
(∗)
=
=
n
X
m=1
n
X
Px (Ty = m)Px (Xn = y|Ty = m)
Px (Ty = m)Px (Xn = y|X1 6= y, · · · , Xm−1 6= y, Xm = y)
m=1
n
X
Px (Ty = m)Px (Xn = y|Xm = y)
m=1
n
X
Px (Ty = m)pn−m (y, y)
m=1
where (∗) is by the Markov property.
87
Exercise 5.2.5. Let T (k) = inf{i ≥ k : Xi = x} ≥ k. Then, similarly to the previous exercise,
Px (Xm = x) =
m
X
Px (T (k) = j)Px (Xm−j = x).
j=k
Summing over m and applying Fubini,
n+k
X
Px (Xm = x) =
m=k
n+k
m
XX
Px (T (k) = j)Px (Xm−j = x)
m=k j=k
=
n+k
X n+k
X
Px (T (k) = j)Px (Xm−j = x)
j=k m=j
=
n+k
X
n−(j−k)
Px (T
(k)
X
= j)
≤
n+k
X
Px (Xm = x)
m=0
j=k
n
X
Px (T (k) = j)
Px (Xm = x)
m=0
j=k
= Px (T (k) ≤ n + k)
n
X
Px (Xm = x)
m=0
≤
n
X
Px (Xm = x).
m=0
Exercise 5.2.6. Since Px (TC < ∞) > 0, there exist nx and x such that Px (TC ≤ nx ) = x >
0. Let N = maxs∈S−C nx and = mins∈S−C x . Then for all y ∈ S − C, Py (TC > N ) ≤ 1 − ,
and thus the desired result holds for k = 1. Now we proceed by induction. Note that
Py (TC > kN ) = Py (X1 ∈
/ C, · · · , XkN ∈
/ C)
= Py (XN +1 ∈
/ C, · · · , XkN ∈
/ C|X1 ∈
/ C, · · · , XN ∈
/ C)Py (X1 ∈
/ C, · · · , XN ∈
/ C)
≤ (1 − )Py (XN +1 ∈
/ C, · · · , XkN ∈
/ C|XN ∈
/ C)
Here,
Py (XN ∈
/ C, · · · , XkN ∈
/ C)
Py (XN ∈
/ C)
Z
1
=
1{XN +1 ∈C,···
dPy
/
,XkN ∈C}
/
Py (XN ∈
/ C) {XN ∈C}
/
Z
1
=
Ey [1{XN +1 ∈C,···
|XN ] dPy
/
,XkN ∈C}
/
Py (XN ∈
/ C) {XN ∈C}
/
Z
1
=
EXN [1{X1 ∈C,···
] dPy .
/
,X(k−1)N ∈C}
/
Py (XN ∈
/ C) {XN ∈C}
/
Py (XN +1 ∈
/ C, · · · , XkN ∈
/ C|XN ∈
/ C) =
By the induction hypothesis, EXN [1{X1 ∈C,···
] ≤ (1−)k−1 on {XN ∈
/ C} Py -almost
/
,X(k−1)N ∈C}
/
surely. Thus the desired result follows for general k.
Exercise 5.2.7.
88
5.3. Recurrence and Transience.
5.4. Recurrence of Random Walks.
5.5. Stationary Measures.
5.6. Asymptotic Behavior.
5.7. Periodicity, Tail σ-Field.
5.8. General State Space.
6. Ergodic Theorems
6.1. Definitions and Examples.
(i) Since ϕ−1 Ω ⊂ Ω and ϕ−1 Ω has probability 1, ϕ−1 Ω = Ω up to a null
Exercise 6.1.1.
set. Thus Ω ∈ I.
(ii) Let A ∈ I. Then up to a null set,
ϕ−1 (Ω \ A) = ϕ−1 Ω \ ϕ−1 A = Ω \ A,
and thus Ω \ A ∈ I.
(iii) Let Aj ∈ Ω for j ≥ 1. Then up to a null set,
[
[
[
ϕ−1 ( Aj ) =
ϕ−1 Aj =
Aj ,
j
and thus
S
j
j
j
Aj ∈ I.
Now to see that X ∈ I if and only if X ◦ ϕ = X a.s., suppose first that X ∈ I. Then up
to a null set,
{X ◦ ϕ > a} = ϕ−1 {X > a} = {X > a}
since {X > a} ∈ I. Since this is true for arbitrary a, X ◦ ϕ = X a.s. (Confer the argument
in Exercise 4.1.10.) For the other direction, suppose X ◦ ϕ = X a.s. Then up to a null set,
{X > a} = {X ◦ ϕ > a} = ϕ−1 {X > a},
and thus {X > a} ∈ I.
Exercise 6.1.2.
(i)
ϕ−1 (B) = ϕ−1 (
∞
[
ϕ−n (A)) =
n=0
∞
[
ϕ−n (A) ⊂ B
n=1
(ii)
−1
ϕ
−1
(C) = ϕ
(
∞
\
−n
ϕ
n=0
(B)) =
∞
\
−n
ϕ
n=1
since ϕ−n (B) ∩ B = ϕ−n (B).
89
(B) =
∞
\
(ϕ−n (B) ∩ B) = C
n=1
(iii) Suppose that A is almost invariant. Define B =
S∞
n=0
ϕ−n (A) and C =
T∞
n=0
ϕ−n (B).
Then by the above, C is invariant in the strict sense. Note that
C=
=
=
∞
\
ϕ−n (
n=0
∞ [
∞
\
∞
[
ϕ−k (A))
k=0
ϕ−(n+k) (A)
n=0 k=0
∞ [
\
ϕ−k (A)
n=0 k≥n
= lim sup ϕ−n (A).
n→∞
−n
Thus both C\A = lim sup(ϕ
(A)\A) and A\C = lim inf(A\ϕ−n (A)) are measure-zero.
Conversely, suppose that there exists C invariant in the strict sense with P(A∆C) = 0.
Then both ϕ−1 (A \ C) = ϕ−1 (A) \ C and ϕ−1 (C \ A) = C \ ϕ−1 (A) are measurezero. It follows that both ϕ−1 (A) \ A ⊂ (ϕ−1 (A) \ C) ∪ (C \ A) and A \ ϕ−1 (A) ⊂
(A \ C) ∪ (C \ ϕ−1 (A)) are measure-zero.
Exercise 6.1.3.
(i) Suppose for the sake of contradiction that xn = xm for some n 6= m.
Then (n − m)θ = k ∈ N and θ ∈ Q, which is a contradiction. Thus xn are distinct in
[0, 1), and hence for any N , |xm − xn | < 1/N for some m < n ≤ N . Now let > 0 and
x ∈ [0, 1). Fix N > 1/. Let k be satisfying x ∈ [k/N, (k + 1)/N ).
(ii)
(iii)
Exercise 6.1.4. Let (Yk )k∈Z be a sequence of random variables with joint distribution defined
by
d
(Y−k , · · · , Yk ) = (X0 , · · · , X2k ).
To see (Yk ) is well-defined, note that, by the stationarity of X,
d
d
(Y−k , · · · , Yk ) = (X1 , · · · , X2k+1 ) = (X0 , · · · , X2k ).
Thus, distributions of (Y−k , · · · , Yk )k are consistent, and by the Kolmogorov extension theorem, the sequence (Yk ) is well-defined. To check (Yk ) is stationary, observe that
d
d
d
(Y0 , · · · , Yk ) = (Xk , · · · , X2k ) = (X2n+k , · · · , X2n+2k ) = (Yn , · · · , Yn+k ).
d
d
Finally, note that (X0 , · · · , Xk ) = (Xk , · · · , X2k ) = (Y0 , · · · , Yk ), and therefore (Xk ) is indeed
embedded in (Yk ).
Exercise 6.1.5. To see that Yk is stationary, note that
(Y0 , · · · , Yk ) = (g((Xj )j≥0 ), · · · , g((Xk+j )j≥0 ))
d
= (g((Xn+j )j≥0 ), · · · , g((Xn+k+j )j≥0 ))
90
= (Yn , · · · , Yn+k ).
To see that Yk is ergodic, note that
Yk+1 = g(Xk+1 , Xk+2 , · · · ) = g(Xk ◦ ϕ, Xk+1 ◦ ϕ, · · · )) = Yk ◦ ϕ.
Thus, for B ∈ B(R{0,1,··· } ),
{(Y1 , Y2 , · · · ) ∈ B} = {(Y0 , Y1 , · · · ) ◦ ϕ ∈ B}
= ϕ−1 {(Y0 , Y1 , · · · ) ∈ B}
Thus, if we let A = {(Y0 , Y1 , · · · ) ∈ B}, then since Xn is ergodic (or in other words, since ϕ
is ergodic), A = ϕ−1 A implies that A is trivial. Therefore Yn is also ergodic.
Exercise 6.1.6. Fix N > n. Let xν and yν be the smallest integer ≥ 0 that satisfy ν +xν ≡ 0
mod n and ν + N − yν ≡ 0 mod n. Then xν and yν are uniformly distributed on {1, · · · , n}.
Note that
(Z1 , · · · , ZN ) = (Yν+1 , · · · , Yν+N )
= (Yν+1 , · · · , Yν+xν , iid blocks of length n, Yν+N −yν +1 , · · · , Yν+N )
and all the blocks that appear in the above (including the first and the last incomplete
blocks) are independent. Now fix k and let x̃ν and ỹν be the smallest integer ≥ 0 that satisfy
k + ν + x̃ν ≡ 0 mod n and k + ν + N − ỹν ≡ 0 mod n. Then again x̃ν and ỹν are uniformly
distributed on {1, · · · , n}. Note again that
(Zk+1 , · · · , Zk+N ) = (Yk+ν+1 , · · · , Yk+ν+N )
= (Yk+ν+1 , · · · , Yk+ν+x̃ν , iid blocks of length n, Yk+ν+N −ỹν +1 , · · · , Yk+ν+N ).
Since ν is independent of Y , so is xν , yν , x̃ν , and ỹν . Thus, the distribution of (Z1 , · · · , ZN )
and (Zk+1 , · · · , Zk+N ) are equal. To see that Z is ergodic, note that the tail σ-field T of Z is
contained in the tail σ-field of blocks (Ykn+1 , · · · , Ykn+n ). Since the blocks (Ykn+1 , · · · , Ykn+n )
are iid, T is trivial. As the invariant σ-field is contained in T , it is also trivial.
Exercise 6.1.7. Note first that
µ([a, b]) =
1
log 2
Z
a
b
dx
1
1+b
=
log
.
1+x
log 2
1+a
From the definition of ϕ, note that x = 1/(ϕ(x) + n) for n ∈ N. It follows that
1
∞ Z
1 X n+a dx
µ(ϕ−1 [a, b]) =
1
log 2 n=1 n+b
1+x
=
∞
1 X
1+n+a n+b
log
log 2 n=1
n+a 1+n+b
=
1
1+n+a 1+b
lim log
log 2 n→∞
1+a 1+n+b
91
=
1
1+b
log
.
log 2
1+a
Thus µ(ϕ−1 [a, b]) = µ([a, b]). Now let L = {A ⊂ (0, 1) : µ(ϕ−1 A) = µ(A)}. Then L is a
λ-system (check) and contains all closed intervals. By Dynkin’s theorem, µ(ϕ−1 A) = µ(A)
for any Borel set A.
6.2. Birkhoff ’s Ergodic Theorem.
0
00
0
Exercise 6.2.1. As in the proof of Theorem 6.2.1, let XM
= X1|X|≤M and XM
= X − XM
.
Following the proof, we know that
n−1
1 X 0
0
X (ϕm ω) → E[XM
|I]
n m=0 M
a.s..
Since
n−1
1 X 0
0
X (ϕm ω) − E[XM
|I]
n m=0 M
p
≤ (2M )p ,
00
by the bounded convergence theorem, the convergence occurs in Lp as well. To handle XM
part, observe that,
n−1
1 X 00 m
00
X (ϕ ω) − E[XM
|I]
n m=0 M
≤
p
n−1
1 X
00
00
kX 00 (ϕm ω)kp + kE[XM
|I]kp ≤ 2 kXM
kp .
n m=0 M
Let
A=
n−1
1 X 0
0
X (ϕm ω) − E[XM
|I]
n m=0 M
B=
n−1
1 X 00 m
00
X (ϕ ω) − E[XM
|I].
n m=0 M
and
It follows that
lim sup
n→∞
n−1
1 X
X(ϕm ω) − E[X|I]
n m=0
00
≤ lim sup(kAkp + kBkp ) ≤ 2 kXM
kp .
n→∞
p
00 p
Letting M → ∞ and using the dominated convergence theorem, E|XM
| → 0 and the result
follows.
Exercise 6.2.2.
(i) Let hN = supm,n≥N |gm − gn |. hN ≤ 2 supk |gk |, so EhN < ∞. Note
that
n−1
1 X
gm (ϕm ω) − E[g|I]
n m=0
≤
N −1
n−1
n−1
1 X
1 X
1 X
gm (ϕm ω) +
|gm (ϕm ω) − g(ϕm ω)| +
g(ϕm ω) − E[g|I]
n m=0
n
n
m=N
m=N
92
≤
N −1
n−1
n−1
1 X
1 X
1 X
gm (ϕm ω) +
hN (ϕm ω) +
g(ϕm ω) − E[g|I] .
n m=0
n
n
m=N
m=N
Since E supk |gk | < ∞, gm < ∞ a.s. and thus the first term → 0 a.s. as n → ∞. Also,
since hN and g are integrable, the second term → E[hN |I] and the third term → 0 a.s.
Thus
lim sup
n→∞
n−1
1 X
gm (ϕm ω) − E[g|I] ≤ E[hN |I].
n m=0
Now letting N → ∞ and using the dominated convergence, the result follows.
(ii) Note that
E
n−1
1 X
gm (ϕm ω) − E[g|I]
n m=0
≤E
≤
n−1
n−1
n−1
1 X
1 X
1 X
gm (ϕm ω) −
g(ϕm ω) + E
g(ϕm ω) − E[g|I]
n m=0
n m=0
n m=0
n−1
n−1
1 X
1 X
E |gm (ϕm ω) − g(ϕm ω)| + E
g(ϕm ω) − E[g|I]
n m=0
n m=0
Since gm → g in L1 , the first term → 0 as n → ∞. The second term → 0 from Theorem
6.2.1.
0
, and Mk0 =
Exercise 6.2.3. Let X 0 = X − α, Xj0 (ω) = X 0 (ϕj ω), Sk0 = X00 + · · · + Xk−1
max(0, S10 , · · · , Sk0 ). By Lemma 6.2.2, E[X 0 ; Mk0 > 0] ≥ 0. That is, E[X; Mk0 > 0] ≥ αP(Mk0 >
0). Note that Mk0 > 0 if and only if Dk > α. It follows that
E|X| ≥ E[X; Mk0 > 0] ≥ αP(Mk0 > 0) = αP(Dk > α).
6.3. Recurrence.
Exercise 6.3.1. Note that
Rn (ω) = 1{S2 −S1 6=0,··· ,Sn −S1 6=0} (ω) + · · · + 1{Sn −Sn−1 6=0} (ω) + 1
= 1{S1 6=0,··· ,Sn−1 6=0} (ϕω) + · · · + 1{S1 6=0} (ϕn−1 ω) + 1.
Taking the expectation, we get
ERn = P(S1 6= 0, · · · , Sn−1 6= 0) + · · · + P(S1 6= 0) + 1
=
n
X
gm−1 .
m=1
Exercise 6.3.2. Note that, since P(Xi > 1) = 0,
1, · · · , max Sm ≤ {S1 , · · · , Sn } ≤ min Sm , · · · , max Sm ,
m≤n
m≤n
93
m≤n
and thus
max Sm ≤ Rn ≤ max Sm − min Sm
m≤n
m≤n
m≤n
Since our sequence is ergodic, Sn /n → EXi > 0 by the ergodic theorem. This implies that
Sn → ∞ and that minm Sm > −∞ a.s. Thus,
lim sup max
n→∞ m≤n
Rn
Sm
Sm
≤ lim
≤ lim inf max
,
n→∞ n
n→∞ m≤n n
n
and by Theorem 6.3.1,
P(A) = lim max
n→∞ m≤n
Sm
.
n
To evaluate the right-hand-side, note that, for some K,
lim
n→∞
Sn
Sm
≤ lim inf max
n→∞
m≤n
n
n
Sm
≤ lim sup max
n→∞ m≤n n
Sm
≤ lim sup max
n→∞ K≤m≤n n
Sm
≤ max
.
K≤m m
Since K is arbitrary, by letting K → ∞, limn→∞ maxm≤n Sm /n = EXi . This completes the
proof.
Exercise 6.3.3. Note that


X
X E
1{Xm ∈B} X0 ∈ A =
E 1{Xm ∈B} 1{T1 ≥m} |X0 ∈ A
1≤m≤T1
1≤m
=
X
P(Xm ∈ B, T1 ≥ m|X0 ∈ A)
1≤m
= P(X0 ∈ A)−1
X
P(Xm ∈ B, T1 ≥ m, X0 ∈ A).
1≤m
Let Cm = {X−m ∈ A, X−m+1 ∈
/ A, · · · , X−1 ∈
/ A, X0 ∈ B}. Observe that
!
c
K
[
Cm
= {Xj ∈ A for some − K ≤ j ≤ −1, X0 ∈ B}c
m=1
= {Xj ∈
/ A for all − K ≤ j ≤ −1} ∪ {X0 ∈
/ B}.
As K → ∞, from P(Xn ∈ A at least once) = 1, it follows that P(Xj ∈
/ A for all − K ≤ j ≤
S
−1) = P(Xj ∈
/ A for all 1 ≤ j ≤ K) → 0. Thus P( Cm ) = P(X0 ∈ B). Now the first display
becomes
= P(X0 ∈ A)−1
X
P(Cm ) =
1≤m
94
P(X0 ∈ B)
.
P(X0 ∈ A)
Exercise 6.3.4. From Theorem 6.3.3,
P̄(T1 ≥ n)
= P(T1 ≥ n|X0 = 1)P(X0 = 1)
ĒT1
= P(T1 ≥ n, X0 = 1).
Note that
P(T1 = n) = P(X1 = · · · = Xn−1 = 0, Xn = 1)
∞
X
=
=
=
m=0
∞
X
m=0
∞
X
P(X−m = 1, X−m+1 = · · · = Xn−1 = 0, Xn = 1)
P(X0 = 1, X1 = · · · = Xm+n−1 = 0, Xm+n = 1)
P(X0 = 1, T1 = m + n)
m=0
= P(X0 = 1, T1 ≥ n)
which completes the proof.
6.5. Applications.
Exercise 6.5.1. content
Exercise 6.5.2. content
Exercise 6.5.3. content
Exercise 6.5.4. content
Exercise 6.5.5. content
7. Brownian Motion
7.1. Definition and Construction.
Exercise 7.1.1. content
Exercise 7.1.2. Let X = B1 , Y = B2 − B1 , and Z = B3 − B2 . Then X, Y , Z are iid normal
random variables with mean 0 and variance 1. Note that
B12 B2 B3 = X 2 (X + Y )(X + Y + Z)
= X 4 + 2X 3 Y + X 2 Y 2 + X 3 Z + X 2 Y Z.
Taking the expectation, it follows that
EB12 B2 B3 = EX 4 + EX 2 Y 2 = 4.
7.2. Markov Property, Blumenthal’s 0-1 Law.
95
7.3. Stopping Times, Strong Markov Property.
7.4. Path Properties.
7.5. Martingales.
7.6. Itô’s Formula.
8. Applications to Random Walk
8.1. Donsker’s Theorem.
8.4. Empirical Distributions, Brownian Bridge.
8.5. Laws of Iterated Logarithm.
9. Multidimensional Brownian Motion
9.1. Martingales.
9.5. Dirichlet Problem. end of document
Email address: hoil.lee@kaist.ac.kr
Email address: swj777888@snu.ac.kr
96