Hindawi Publishing Corporation
Journal of Applied Mathematics
Volume 2008, Article ID 639145, 10 pages
doi:10.1155/2008/639145
Research Article
A Strong Limit Theorem for Functions of
Continuous Random Variables and an Extension of
the Shannon-McMillan Theorem
Gaorong Li,1, 2 Shuang Chen,3 and Sanying Feng4
1
School of Finance and Statistics, East China Normal University, Shanghai 200241, China
College of Applied Sciences, Beijing University of Technology, Beijing 100022, China
3
School of Sciences, Hebei University of Technology, Tianjin 300130, China
4
College of Mathematics and Science, Luoyang Normal University, Henan 471022, China
2
Correspondence should be addressed to Gaorong Li, l gaorong@163.com
Received 1 November 2007; Revised 10 January 2008; Accepted 29 March 2008
Recommended by Onno Boxma
By means of the notion of likelihood ratio, the limit properties of the sequences of arbitrarydependent continuous random variables are studied, and a kind of strong limit theorems
represented by inequalities with random bounds for functions of continuous random variables
is established. The Shannon-McMillan theorem is extended to the case of arbitrary continuous
information sources. In the proof, an analytic technique, the tools of Laplace transform, and moment
generating functions to study the strong limit theorems are applied.
Copyright q 2008 Gaorong Li et al. This is an open access article distributed under the Creative
Commons Attribution License, which permits unrestricted use, distribution, and reproduction in
any medium, provided the original work is properly cited.
1. Introduction
Let {Xn , n ≥ 1} be a sequence of arbitrary continuous real random variables on the probability
space Ω, F, P with the joint density function
fn x1 , . . . , xn > 0, n 1, 2, . . . ,
1.1
where xi ∈ −∞, ∞, 1 ≤ i ≤ n. Let Q be another probability measure on F, and {Xn , n ≥ 1}
is a sequence of independent random variables on the probability space Ω, F, Q with the
marginal density functions gk xk 1 ≤ k ≤ n, and let
n
πn x1 , . . . , xn g k xk .
k1
1.2
2
Journal of Applied Mathematics
In order to indicate the deviation between {Xn , n ≥ 1} on the probability measure P and
Q, we first introduce the following definitions.
Definition 1.1. Let {Xn , n ≥ 1} be a sequence of random variables with joint distribution 1.1,
and let gk xk k 1, 2, . . . , n be defined by 1.2. Let
f n X1 , . . . , X n
Zn ω .
πn X1 , . . . , Xn
1.3
In statistical terms, Zn ω is called the likelihood ratio, which is of fundamental importance in
the theory of testing the statistical hypotheses cf. 1, page 388; 2, page 483.
The random variable
1
1
rω lim sup ln Zn ω −lim inf ln
n→∞ n
n→∞ n
n
g k Xk
f n X1 , . . . , X n
1.4
k1
is called asymptotic logarithmic likelihood ratio, relative to the product of marginal
distribution of 1.2, of Xn , n ≥ 1, where ln is the natural logarithm, ω is the sample point.
For the sake of brevity, we denote Xk ω by Xk .
Although rω is not a proper metric between probability measures, we nevertheless
think of it as a measure of “dissimilarity” between their joint distribution fn x1 , . . . , xn and
the product πn x1 , . . . , xn of their marginals.
Obviously, rω 0, a.s. if and only if {Xn , n ≥ 1} are independent.
A stochastic process of fundamental importance in the theory of testing hypotheses is
the sequence of likelihood ratio. In view of the above discussion of the asymptotic logarithmic
likelihood ratio, it is natural to think of rω as a measure how far the random deviation of
Xn is from being independent, how dependent they are. The smaller rω is, the smaller the
deviation is cf. 3–5.
In 3, the strong deviation theorems for discrete random variables were discussed by
using the generating function method. Later, the approach of Laplace transform to study
the strong limit theorems was first proposed by Liu 4. Yang 6 further studied the limit
properties for Markov chains indexed by a homogeneous tree through the analytic technique.
Many comprehensive works may be found in Liu 7. The purpose of this paper is to establish
a kind of strong deviation theorems represented by inequalities with random bounds for
functions of arbitrary continuous random variables, by combining the analytic technique
with the method of Laplace transform, and to extend the strong deviation theorems to the
differential entropy for arbitrary-dependent continuous information sources in more general
settings.
Definition 1.2. Let {hn xn , n ≥ 1} be a sequence of nonnegative B orel measurable functions
defined on R, the Laplace transform of random variables hn Xn on the probability space
Ω, F, Q is defined by
f
n s ∞
−∞
e−shn xn gn xn dxn EQ e−shn Xn ,
where EQ denotes the expectation under Q.
n 1, 2, . . . ,
1.5
Gaorong Li et al.
3
We have the following assumptions in this paper.
1 Assume that there exists s0 ∈ 0, ∞ such that
f
n s < ∞, s ∈ − s0 , s0 , n 1, 2, . . . .
1.6
2 Assume M > 0 is a constant, satisfying
supEQ hn Xn ≤ M < ∞.
1.7
n
In order to prove our main results, we first give a lemma, and it will be shown that it
plays a central role in the proofs.
Lemma 1.3. Let fn x1 , . . . , xn , gn x1 , . . . , xn be two probability functions on Ω, F, P , let
g n X1 , . . . , X n
Tn ω ,
f n X1 , . . . , X n
1.8
then
lim sup
n→∞
1
ln Tn ω ≤ 0 a.s.
n
1.9
Proof. By 8, {Tn , F, n ≥ 1} is a nonnegative martingale and ETn 1, we have by the Doob
martingale convergence theorem, there exists an integral random variable T∞ ω, such that
Tn → T∞ , a.s. and 1.9 follows.
2. Main results
Theorem 2.1. Let {Xn , n ≥ 1}, Zn ω, rω, f
n s be defined as before, and under the assumptions of
(1) and (2), let
lim sup
n→∞
n
n
1
1
EQ es0 hk Xk lim sup
f
k − s0 M s0 < ∞.
n k1
n→∞ n k1
2.1
Then
lim inf
n→∞
n
1
hk Xk − EQ hk Xk ≥ −β rω ,
n k1
lim sup
n→∞
n
1
hk Xk − EQ hk Xk ≤ β rω ,
n k1
a.s.
2.2
a.s.
2.3
where
βx inf φs, x, −s0 < s < 0 , x ≥ 0,
2e−2 sM s0
x
φs, x − 2 − s , x ≥ 0,
s0 − |s|
√ √ x 1 x −2 0 ≤ βx ≤
2e M s0 1 ,
lim βx β0 0.
x→0
s0
2.4
2.5
2.6
4
Journal of Applied Mathematics
Remark 2.2. Let
αx sup φs, x, 0 < s < s0 ,
x ≥ 0,
2.7
then
αx −βx.
2.8
Proof. Let s be an arbitrary real number in −s0 , s0 , let
gn s, xn e−shn xn gn xn /f
n s, xn ∈ −∞, ∞, n 1, 2, . . . ,
∞
then −∞ gn s, xn dxn 1, and let
n
gk s, xk .
qn s; x1 , . . . , xn 2.9
2.10
k1
Therefore, qn s; x1 , . . . , xn is an n multivariate probability density function, let
qn s; X1 , . . . , Xn
tn s, ω , n 1, 2, . . . .
f n X1 , . . . , X n
2.11
By Lemma 1.3, there exists a set As such that P As 1, so we have
lim sup
n→∞
1
ln tn s, ω ≤ 0,
n
ω ∈ As.
By 1.3, 2.9, 2.11, and 2.12, we have
n
n
1
− s hk Xk −
lim sup
ln f
k s − ln Zn ω ≤ 0,
n→∞ n
k1
k1
2.12
ω ∈ As.
2.13
Therefore,
rω ≥ 0,
ω ∈ A0.
2.14
By 2.13 and 1.4, the property of the superior limit
lim sup an − bn ≤ 0 ⇒ lim sup an ≤ lim sup bn ,
n→∞
n→∞
2.15
n→∞
and the inequality ln x ≤ x − 1 x > 0, we have
n
1
lim sup −s
hk Xk − EQ hk Xk
n
n→∞
k1
≤ lim sup
n→∞
n
1
ln f
k s sEQ hk Xk rω
n k1
2.16
n
1
f
k s − 1 sEQ hk Xk rω
≤ lim sup
n
n→∞
k1
lim sup
n→∞
n
1
f
k s − 1 − −sEQ hk Xk rω,
n k1
ω ∈ As.
Gaorong Li et al.
5
By the inequality 0 ≤ ex − 1 − x ≤ 1/2x2 e|x| , which can be found in 9, we have
2
0 ≤ e−shk Xk − 1 − −shk Xk ≤ 1/2 shk Xk e|shk Xk | .
2.17
By 2.5 and 2.17, we have
n
1
hk Xk − EQ hk Xk
lim sup −s
n→∞ n
k1
n
2
1
≤ 1/2s lim sup
EQ hk Xk e|s|hk Xk rω,
n→∞ n k1
2
2.18
ω ∈ As.
It is easy to see that ϕx tx x2 t > 1 attains its largest value ϕ−2/ ln t 4e−2 /ln t2
on the interval −∞, 0, and ϕx tx x2 0 < t < 1 attains its largest value ϕ−2/ ln t 4e−2 /ln t2 on the interval 0, ∞, we have
2
4e−2
4e−2
sup es−s0 hk Xk hk Xk , k ≥ 1 ≤ 2 , 0 < s < s0 ,
2 s − s0
s0 − s
2
4e−2
sup es0 s−hk Xk − hk Xk , k ≥ 1 ≤ 2 , −s0 < s < 0.
s0 s
2.19
2.20
Let 0 < s < s0 in 2.18, by 2.19 and 2.1, we obtain
−slim inf
n→∞
n
1
hk Xk − EQ hk Xk
n k1
≤ 1/2s2 lim sup
n→∞
1/2s2 lim sup
n→∞
n
2
1
EQ hk Xk e|s|hk Xk rω
n k1
n
2 1
EQ es0 hk Xk es−s0 hk Xk hk Xk
rω
n k1
2e s M s0
≤ 2 rω,
s0 − s
−2 2
2.21
ω ∈ As.
Dividing the two sides of 2.21 by −s, we obtain
n
2e−2 sM s0
1
rω
lim inf
hk Xk − EQ hk Xk ≥ − 2 − s
n→∞ n
s0 − s
k1
φ s, rω , 0 < s < s0 , ω ∈ As ∩ A0.
2.22
By 2.14 and 0 < s < s0 , obviously φs, rω ≤ 0, hence αrω ≤ 0. Let Q be the set of
rational numbers in the interval 0, s0 , and let A∗ ∩s∈Q As, then P A∗ 1. By 2.22, then
we have
lim inf
n→∞
n
1
hk Xk − EQ hk Xk ≥ φ s, rω ,
n k1
ω ∈ A∗ ∩ A0, ∀s ∈ Q .
2.23
6
Journal of Applied Mathematics
It is easy to see that φs, x is a continuous function with respect to s on the interval
0, s0 . For each ω ∈ A∗ ∩ A0 0 ≤ rω < ∞, take sn ω ∈ Q , n 1, 2, . . . , such that
2.24
lim φ sn ω, rω α rω .
n→∞
By 2.23, 2.24, and 2.8, we have
lim inf
n→∞
n
1
hk Xk − EQ hk Xk ≥ −β rω ,
n k1
ω ∈ A∗ ∩ A0.
2.25
Since P A∗ ∩ A0 1, 2.2 follows from 2.25.
Let −s0 < s < 0 in 2.18, by 2.20 and 2.1, we have
lim sup
n→∞
n
1
hk Xk − EQ hk Xk
n k1
≤ −1/2s lim sup
n→∞
−1/2s lim sup
n→∞
n
2
rω
1
EQ hk Xk e|s|hk Xk −
n k1
s
n
2 rω
1
EQ es0 hk Xk es0 s−hk Xk − hk Xk
−
n k1
s
2.26
2e−2 sM s0
rω
≤− 2 − s
s0 s
φ s, rω , −s0 < s < 0, ω ∈ As ∩ A0.
By 2.14 and −s0 < s < 0, obviously φs, rω ≥ 0, hence βrω ≥ 0. Let Q− be the set of
rational numbers in the interval −s0 , 0, and let A∗ ∩s∈Q− As, then P A∗ 1. Then we have
by 2.26
lim sup
n→∞
n
1
hk Xk − EQ hk Xk ≤ φ s, rω ,
n k1
ω ∈ A∗ ∩ A0, ∀s ∈ Q− .
2.27
It is clear that φs, x is a continuous function with respect to s on the interval −s0 , 0. For each
ω ∈ A∗ ∩ A0 0 ≤ rω < ∞, take λn ω ∈ Q− , n 1, 2, . . . , such that
2.28
lim φ λn ω, rω β rω .
n→∞
By 2.27 and 2.28, we have
lim sup
n→∞
n
1
hk Xk − EQ hk Xk ≤ β rω ,
n k1
ω ∈ A∗ ∩ A0.
Since P A∗ ∩ A0 1, 2.3 follows from 2.29.
By 2.4, 2.5, and 2.14, if x > 0, we have
√ √ √
x 1 x −2 s0 x
0 ≤ βx ≤ φ −
2e M s0 1 .
√ ,x s0
1 x
If x 0, we have
2e−2 M s0
β0 ≤ φ − n , 0 2 ,
n s0 − n−1
−1
n ≥ 1.
Noticing that βx ≥ 0, x ≥ 0, 2.6 follows from 2.30 and 2.31.
2.29
2.30
2.31
Gaorong Li et al.
7
Corollary 2.3. If P Q, or {Xn , n ≥ 1} is a sequence of independent random variables, and under the
assumptions of (1) and (2), then
n
1
hk Xk − Ehk Xk 0
n→∞ n
k1
lim
Proof. In this case, fn x1 , . . . , xn from 2.2 and 2.3.
n
k1 gk xk ,
a.s.
2.32
and rω 0 a.s. Hence, 2.32 follows directly
3. An extension of the Shannon-McMillan theorem
In order to understand better, we first introduce some definitions in information theory in this
section.
Let {Xn , n ≥ 1} be a sequence produced by an arbitrary continuous information source
on the probability space Ω, F, P with the joint density function
fn x1 , . . . , xn > 0, xk ∈ −∞, ∞, 1 ≤ k ≤ n, n 1, 2, . . . .
3.1
For the sake of brevity, we denote fn > 0, and Xk stands for Xk ω. Let
pn ω −1/n ln fn X1 , . . . , Xn ,
3.2
where ω is the sample point, pn ω is called the sample entropy or the entropy density of
{Xk , 1 ≤ k ≤ n}. Also let Q be another probability measure on F with the density function
3.3
qn x1 , . . . , xn > 0, xk ∈ −∞, ∞, 1 ≤ k ≤ n, n 1, 2, . . . .
Let
Ln ω ln fn X1 , . . . , Xn /qn X1 , . . . , Xn ,
Lω lim sup1/nLn ω
n→∞
−lim inf1/n ln qn X1 , . . . , Xn /fn X1 , . . . , Xn ,
n→∞
D fn qn EP Ln
EP ln fn X1 , . . . , Xn /qn X1 , . . . , Xn .
3.4
Ln ω, Lω, and Dfn qn are called the sample relative entropy, the sample relative
entropy rate, and the relative entropy, respectively, relative to the reference density function
qn x1 , . . . , xn . Indeed, they all are the measure of the deviation between the true joint
distribution density function fn x1 , . . . , xn and the reference distribution density function
qn x1 , . . . , xn cf. 10, pages 12, 18.
A question of importance in information theory is the study of the limit properties of the
relative entropy density fn ω. Since Shannon’s initial work was published cf. 11, there has
been a great deal of investigation about this question e.g., cf. 12–20.
In this paper, a class of small deviation theorems i.e., the strong limit theorems
represented by inequalities is established by using the analytical technique, and an extension
of the Shannon-McMillan theorem to the arbitrary-dependent continuous information sources
is given. Especially, an approach of applying the tool of Laplace transform to the study of the
strong deviation theorems on the differential entropy is proposed.
8
Journal of Applied Mathematics
Let hk xk − ln gk xk 1 ≤ k ≤ n, n 1, 2, . . . in 1.5, then we give the following
definitions.
Definition 3.1. The Laplace transform of − ln gk xk is defined by
f
k s Ee−s− ln gk Xk ∞
−∞
e−s− ln gk xk gk xk dxk .
3.5
Definition 3.2. The differential entropy for continuous random variables Xk is defined by
h Xk E − ln gk Xk −
∞
−∞
gk xk ln gk xk dxk .
3.6
In the following theorem, let {Xn , n ≥ 1} be independent random variables with respect
n
to Q, then the reference density function qn x1 , . . . , xn k1 gk xk , and let hk Xk − ln gk Xk 1 ≤ k ≤ n in Theorem 2.1.
Theorem 3.3. Let {Xn , n ≥ 1}, Ln ω, Lω, f
n s be given as above, and under the assumptions of
(1) and (2), let
lim sup
n→∞
n
n
1
1
f
k − s0 M s0 < ∞.
EQ es0 − ln gk Xk lim sup
n k1
n
n→∞
k1
3.7
Then
n
1
− ln gk Xk − h Xk ≥ −β Lω ,
n k1
a.s.
n
1
− ln gk Xk − h Xk ≤ β Lω ,
lim sup
n
n→∞
k1
a.s.
lim inf
n→∞
3.8
where
βx inf φs, x, −s0 < s < 0 , x ≥ 0,
2e−2 sM s0
x
φs, x − 2 − s , x ≥ 0,
s0 − |s|
√ √ x 1 x −2 0 ≤ βx ≤
2e M s0 1 ,
lim βx β0 0.
x→0
s0
3.9
3.10
3.11
Remark 3.4. Let
αx sup φs, x, 0 < s < s0 ,
x ≥ 0,
3.12
then
αx −βx.
3.13
Gaorong Li et al.
9
Corollary 3.5. Let pn ω be defined by 3.2. Under the condition of Theorem 3.3, then
lim inf pn ω − 1/nh X1 , . . . , Xn ≥ α Lω − Lω H∗
n→∞
lim sup pn ω − 1/nh X1 , . . . , Xn ≤ β Lω H ∗
a.s.
a.s.
3.14
n→∞
where hX1 , . . . , Xn E− ln fn X1 , . . . , Xn is the differential entropy for X1 , . . . , Xn , and
n
H∗ lim inf1/n
h Xk − h X 1 , . . . , Xn
n→∞
k1
lim inf1/nD fn qn ,
n→∞
n
∗
H lim sup1/n
h Xk − h X 1 , . . . , Xn
n→∞
3.15
k1
lim sup1/nD fn qn ,
n→∞
where αLω, βLω are denoted by 3.9–3.13.
Corollary 3.6. If P Q, or {Xn , n ≥ 1} are independent random variables, and there exists s0 > 0,
such that 2.1 holds, then
lim pn ω − 1/nh X1 , . . . , Xn 0
n→∞
a.s.
3.16
Acknowledgments
This research is supported by the National Natural Science Foundation of China Grants
nos. 10671052 and 10571008, the Natural Science Foundation of Beijing Grant no. 1072004,
Funding Project for Academic Human Resources Development in Institutions of Higher
Learning Under the Jurisdiction of Beijing Municipality, the Basic Research and Frontier
Technology Foundation of Henan Grant no. 072300410090, and the Natural Science Research
Project of Henan Grant no. 2008B110009. The authors would like to thank the editor and the
referees for helpful comments, which helped to improve an earlier version of the paper.
References
1 R. G. Laha and V. K. Rohatgi, Probability Theory, Wiley Series in Probability and Mathematical Statistic,
John Wiley & Sons, New York, NY, USA, 1979.
2 P. Billingsley, Probability and Measure, Wiley Series in Probability and Mathematical Statistics, John
Wiley & Sons, New York, NY, USA, 2nd edition, 1986.
3 W. Liu, Relative entropy densities and a class of limit theorems of the sequence of m-valued random
variables, The Annals of Probability, vol. 18, no. 2, pp. 829839, 1990.
4 W. Liu, A class of strong deviation theorems and Laplace transform methods, Chinese Science Bulletin,
vol. 43, no. 10, pp. 10361041, 1998.
5 W. Liu and Y. Wang, A strong limit theorem expressed by inequalities for the sequences of absolutely
continuous random variables, Hiroshima Mathematical Journal, vol. 32, no. 3, pp. 379387, 2002.
6 W. Yang, Some limit properties for Markov chains indexed by a homogeneous tree, Statistics &
Probability Letters, vol. 65, no. 3, pp. 241250, 2003.
10
Journal of Applied Mathematics
7 W. Liu, Strong Deviation Theorems and Analytic Method, Science Press, Beijing, China, 2003.
8 J. L. Doob, Stochastic Processes, John Wiley & Sons, New York, NY, USA, 1953.
9 W. Liu and J. Wang, A strong limit theorem on gambling systems, Journal of Multivariate Analysis, vol.
84, no. 2, pp. 262273, 2003.
10 T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley Series in Telecommunications,
John Wiley & Sons, New York, NY, USA, 1991.
11 C. E. Shannon, A mathematical theory of communication, The Bell System Technical Journal, vol. 27, pp.
379423, 623–656, 1948.
12 P. H. Algoet and T. M. Cover, A sandwich proof of the Shannon-McMillan-Breiman theorem, The
Annals of Probability, vol. 16, no. 2, pp. 899909, 1988.
13 A. R. Barron, The strong ergodic theorem for densities: generalized Shannon-McMillan-Breiman
theorem, The Annals of Probability, vol. 13, no. 4, pp. 12921303, 1985.
14 K. L. Chung, A note on the ergodic theorem of information theory, The Annals of Mathematical Statistics,
vol. 32, no. 2, pp. 612614, 1961.
15 J. C. Kieffer, A simple proof of the Moy-Perez generalization of the Shannon-McMillan theorem, Pacific
Journal of Mathematics, vol. 51, pp. 203206, 1974.
16 J. C. Kieffer, A counterexample to Perez’s generalization of the Shannon-McMillan theorem, The
Annals of Probability, vol. 1, no. 2, pp. 362364, 1973.
17 B. McMillan, The basic theorems of information theory, The Annals of Mathematical Statistics, vol. 24,
no. 2, pp. 196219, 1953.
18 W. Liu and W. Yang, An extension of Shannon-McMillan theorem and some limit properties for
nonhomogeneous Markov chains, Stochastic Processes and Their Applications, vol. 61, no. 1, pp. 129145,
1996.
19 W. Liu and W. Yang, The Markov approximation of the sequences of N-valued random variables and
a class of small deviation theorems, Stochastic Processes and Their Applications, vol. 89, no. 1, pp. 117130,
2000.
20 R. M. Gray, Entropy and Information Theory, Springer, New York, NY, USA, 1990.