Proof of Cesaro Means
In order to show that our alternative definition of entropy rate:
H (X) = lim H Xn | X1 , . . . , Xn−1
n→∞
is equivalent to the canonical definition
1
H(X1 , . . . , Xn )
n→∞ n
H(X) = lim
when the stochastic process is stationary, we used the following theorem
Theorem Cesaro Means:
Let an → a, let bn = n−1 ni=1 ai , then
lim bn = a.
n→∞
Proof Recall the meaning of limn→∞ bn = a:
For every δ > 0 there exists a nδ such that, for every n > nd elta, bn −a < δ.
Now, since limn→∞ an = a, we know that ∀, ∃n such that ∀n > n , |an −a| <
.
Choose δ = 2.
Fix, , determine n , let n >> n , and look at bn − a.
n −1
ai − a
|bn − a| = n
i=1
n n −1
= n
ai − a n
i=1
n
−1
= n
−na +
ai i=1
n
−1 = n
(ai − a)
i=1
1
Now divide the sum on the right hand side into two parts: the first is the
sum over the indexes between 1 and n , the second is the over the remaining
terms
n
1 (ai − a)
|bn − a| = n
i=1
n
1 ≤ (ai − a)
(1)
n
i=1
n
1 (2)
+
(ai − a)
n
i=n +1
We are now going to bound the two sums (1) and (2).
First we bound (1)
n
n
1 1
(ai − a) ≤
|ai − a|
n
n
i=1
i=1
1
n
≤
max |aj − a|
n i=1 j=1
n
1
1
= max |aj − a|
j=1
n i=1
n
n
= max |aj − a|
j=1
n
n
n
and, if we pick n satisfying
|aj − a|
n maxnj=1
n>
,
then sum (1) satisfies
n
1
(ai − a) < n
i=1
Now we bound (2) using a similar trick
n
n
1 1 (ai − a) ≤
|ai − a|
n
n
i=n +1
i=n +1
2
(3)
(4)
n
1 n
≤
max |ai − a|
n i=n +1 i=n +1
n
1 1
= max |ai − a|
i=n +1
n i=n +1
n
n − n
n
= max |ai − a|
i=n +1
n
n
≤ max |ai − a|
i=n +1
≤ (5)
where Inequality 5 is a consequence of how we selected n .
We have therefore shown that, if we fix δ, let = δ/2, determine n , there
n maxn |aj −a|
such that, for all n > nδ , |bn − a| < δ;
exists a nδ = j=1
In other words, we have shown that limn→∞ bn = a.
3