Exercises II
1. Let X be a stationary process over an alphabet with k letters.
(a) Show that X is iid iff h(X) = h(X1).
(b) Show that X is iid uniform iff h(X) = log k.
2. Show that a function of an iid process is iid, i.e. if X is iid and
Yi = f (Xi), then Y is iid.
3. Prove directly and combinatorially the AEP Corollary for iid
processes only:
(a) For suff. large n, µ(An) > 1 − (b) |An| ≤ 2n(h2(X)+)
(c) For sufficiently large n, |An| ≥ (1 − )2n(h2(X)−)
4. Let (X, Y ) be jointly distributed with distribution p(x, y).
Let q(x, y) = p1(x)p2(y) where p1 is the X-marginal and p2 is
the y-marginal.
Show that I(X; Y ) = D(p, q) where D is the Kullback-Leibler
divergence.
5. Show that I(X; Y ) ≤ min(H(X), H(Y )) with equality iff X is
a function of Y or Y is a function of X.
6. Consider the binary symmetric channel with errors and erasures.
That is, the input alphabet is {0, 1}, alphabet is {0, 1, e} with
p(1|0) = = p(0|1), p(e|0) = δ = p(e|1), p(0|0) = 1−−δ = p(1|1),
here ‘e’ is an erasure symbol, and the channel is memoryless,
i.e., independent from time-slot to time-slot. Find the capacity
of this channel.
1