Chapter 4
LIMIT THEOREMS
§ 4.1 Law of large numbers
We are now in a position to prove our first fundamental theorem of probability.We have seen that an
intuitive way to view the probability of a certain outcome is as the frequency with which that outcome occurs in
the long run, when the experiment is repeated a large number of times. We have also defined probability
mathematically as a value of a distribution function for the random variable representing the experiment. The
Law of Large Numbers(LLN for short), which is a theorem proved about the mathematical model of probability,
shows that this model is consistentwith the frequency interpretation of probability. This theorem is sometimes
called the law of averages.
Definition 4.1
Suppose that X 1 , X 2 ,  is random variable siery and there exist constants  1 ，
 2 ，…such that for any  ＞0,the following condition holds :
1 n

lim P   X k   n     1 ，
n 
 n k 1

or equivalently,
1 n

lim P   X k   n     0 ,
n 
 n k 1

then, it is said that random variable siery X n  follows the law of large number.
Suppose that the expectation of X n , named EX n , exists，n =1，2，…，put  n 
1 n
 EX k in definition
n k 1
4.1，it is the classical LLN：for any  ＞0，we have
1 n

1 n
lim P   X k   EX k     1 ，
n 
n k 1
 n k 1

or equivalently
 1 n

1 n
lim P   X k   EX k     0 .

n  n
n k 1
 k 1

This chapter just focus on classical LLN.
Law of Large Number
Let X 1 ， X 2 ，…be a sery of r.v.s and assume that EX n , n  1,2,  exists.
Theorem 4.1(Chebyshev’s LLN) Let
X n 
a sery of indenpendent random variables and there exists a
constant C such that DX n ≤ C ， n =1，2，…，then
84
X n 
follows the law of large number.
Proof
for any  ＞0，By Chebyshev’s inequaility,
1
P
n

1
 P
n

D(
≤
Note that
X n 
n

Xk 
k 1
n

n
 EX
1
n
k 1
X k  E(
k 1
1
n





Xk ) 

k 1

n

1
n
n
X
k
n
k
)
k 1
2
D(
k
)
k 1
2 2


X
；
n 
n
are indepedent r.v.s， DX k ≤ C ，thus D(

EX k ) 
k 1
1
P
n

n

Xk 
k 1
1
n
n
 DX
k
≤ nC and
k 1
 nC
C
EX k    ≤ 2 2  2 ，
 n 
n
k 1

n

i.e.
lim P(
n 
1
n
n

Xk 
k 1
n
 EX
1
n
k
≥  )  0.
k 1
This completes the proof .
Theorem 4.2(Bernoulli LLN) Set  n counting the numbers of outcomes of A in n Bernoulli
experiment, P( A)  p ＞0，then for any  ＞0，


lim P n  p    =1.
 n

 
1,
the outcome of kth exper i ment i s A;
0,
the outcome of kth exper i ment i s not A
Proof Set X k  
Obviously,  n 
n
X
k
， since
X n 
k =1，2，…，
are independent r.v.s with indentically distribution as follows
k 1

 0
Xk ~ 
1  p

EX k  p ， DX k  p (1  p ) ≤

1
 ， k =1，2，…，
p

1
， k  1，2，…，By Theorem 4.1, we have
4
1


lim P n  p     lim P
n
 n
 n  n
Theorem 4.3(Khinchine LLN) Assume that
X n 
85
n

k 1
Xk 
1
n
n
 EX
k 1
k

   =1.


are indpendent r.v.s with EX n  a ，then
1
lim P
n  n

That is to say
X n 
n
X
k
k 1

 a    =1.


follows LLN.
Example 1 Assume that
X n 
is a sequence of i.i.d. r.v.s with common density funciton
1  
 2  , x  1
p( x)   x
0,
x 1

Prove that
Proof
X n 
0 ＜  ≤ 1 ，
follows LLN.
EX 1 




xp( x)dx  (1   )


x
1
1
x
2
dx
1
＜+  ，

By Khinchine LLN，for any  ＞0，we have
1
lim P
n   n

n
X
k

k 1
1 


   =1.


This completes the proff.
Example 2 Assume that
X n 
  follows tha
are i.i.d. r.v.s and EX nk exists， n =1，2，…，then X nk
LLN.
  are i.i.d. and
Proof Since X n  are i.i.d., thus X nk
EX nk exists, by Khninchine LLN, one can find that
X  follows LLN.
k
n
§4.2 Central limit theorem
According to the weak law of large numbers, the distribution of the sample mean
M n is increasingly
concentrated in the near vicinity of the true mean µ. In particular, its variance tends to zero. On the other hand,
the variance of the sum Sn  X 1  ···  X n  nM n increases to infinity, and the distribution of S n
cannot be said to converge to anything meaningful. An intermediate view is obtained by considering the
deviation Sn  n of S n from its mean n  , and scaling it by a factor proportional to
1
. What is special
n
about this particular scaling is that it keeps the variance at a constant level. The central limit theorem，cope with
the limit behavior of so-called following “standardized sum” :
86
n
Yn 
n
 X   EX
k 1
k
k 1
k
n
 DX
k 1
k
One question is under what condition that
X n 
satisfied, we have
lim PYn ≤ x  = ( x) 
n 
1
2


x
e

t2
2
dx ， x  R ?
Central Limit theorem
Now, we introduce some well-known CLT.
Theorem 4.4 (Lindeberg-lévy CLT) Assume that
EX n  a, DX n   2 ＞0，then
X n 
X n 
follows the central limit theorem, i.e. for any real number x ，we have
 n

  X k  na

1
lim P  k 1
 x 
n 

n
2




That is to say for i.i.d. X1 , X 2 ,
is a sequence of i.i.d. random varibables with
x

e

t2
2 dx
 ( x) .
X n with mean a and variance  2 , the “ standardized sum ” of
n
 Xk
is approximately normal distribtued, i.e.
k 1
n
 X k  na 近似地
k 1
n
N (0,1) when n is large enough .
n
Usually, it is not so easy to determine the distribution of the sum of dependent r.v.s
 X k , by Lindeberg-l
k 1
évy theorem, we can approximate its distribution by  ( x) when n is large enought, where  ( x) is the
distribution of standard normal distribution.
Theorem 4.5 (DeMoiver-Lapace CLT) Suppose that  n records the number of outcome A appeared
in n repeated Bernoulli trials with P( A)  P ＞0，then for any real number x ，we have
   np

lim P n
 x 

n  
 npq

Proof Set
87
1
2


x

e
t2
2
dt .

1, the outcome is A in kth trial;
Xk  

0, the outcome is not A in kth trial.
Obvioulsy,
X n 
are i.i.d. r.v.s with common distribution law

0

q

EX n  p, DX n  pq, and  n 

1

p

(q  1  p) ，
n
X
k
，by theorem 4.4, it follows that
k 1


  n  np


P
 x   p
 npq






n
X
k
 np
k 1
npq



 x    ( x)



( n  ) .
The applications of central limit theorems : the De Moivre – Laplace Approximation to the Binomial
If n is very large number, by theorem 4.5 the binomial distribution can be approximate by normal
distribution, the way is as follows. Suppose that X ~ B(n, p) ，then
P (k1 ＜ X ＜ k 2 ) = P (k1 ＜  n ≤ k 2 )
 k  np  n  np k 2  np 

 P 1


 npq

npq
npq


 k  np 


   k1  np  .
  2
 npq 
 npq 




Remark. In Bernoulli trials,suppose that P ( A)  p and let n
appeared in n repeated Bernoulli trials， then
n
n
records the number of outcome A
is the frequency of even A ,By theorem 4.5, when n is
large enough, we have


P  n  p     P(
 n


≈  



n 
   

pq 

  np
n
 n

pq
npq

n 
 2 

pq 

n
)
pq
n 
1.
pq 
This formula can be used for measure the error of frequncy approximation for probability.
Example 3 Toss an uneven coin repeatedly，suppose that head appeared in each trial with probability
p ，let
n
records the frequency of outcome A appeared in n trial, to make the error of n and p is
88
not greater than
1
with probability 95%, how many tosses we should made the least ? The problem is to find
100
n such that


1 
P n  p  ＜
 ≥0.95,
 n
 100 
By theorem 4.5,


1 
n 
 ≈ 2 0.01
P n  p 
-1，

100 
pq 
 n


n 
Thus, approcimately 2 0.01
-1≥0.95，

pq 


n
n 
or equivalently，  0.01
≥0.975，that is 0.01
≥1.96, i.e. n ≥ 196 2 pq ，


pq
pq


Note that pq ≤
1
1
，it is sufficient for all n ≥ 196 2  =9604.
4
4
Exercise 4
1．Suppose that
X n 
are i.i.d. with density funcion
13
 , x  1,
p( x)   x

 0, x  1,
Prove that
X n 
follows the law of large numbers.
2．A system composed with n dependent working parts with probability 0.9 that each part is well-operated,
Only 80% parts are well-operated canc that the system be operated, try to determine that the least number of n
such that the systme is well-operated with probability 0.95？
3*．Suppse X ~ P(100) ，determine P (80≤ X ≤100)with Poisson approximation and central limit theorem
respectively.
4*．Prove the following limit with cental limit theorem
 n n k  n 1
e  .
lim 

n 
k
!
2
 k 0


89