EECE 577: Information and Coding Theory
Midterm Exam # 1 (October 2010)
Time allowed: 5 Hours
Name:
(Return this test booklet with your answer sheets.)
Problem 1:
(a)
(b)
(k)
(l)
/ 40
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(m)
(n)
(o)
(p)
(q)
(r)
(s)
(t)
True
False
True
False
Problem 2:
/ 30
Problem 3:
/ 20
Problem 4:
/ 15
Problem 5:
/ 25
Total:
/ 130
Problem 1. (40 points: 1 point each) Say True or False to the following statements. You
don’t need to justify your answer. Mark your answers in the table on the cover page. (Correct
answer= 2, No answer= 0, Incorrect answer= −1)
(a) Rate-distortion theory is about channel coding.
(b) The fixed-length source coding that relies on the Asymptotic Equi-partition Property
(AEP) is in general classified as lossless source coding.
(c) Continuous source is a source that generates the output modeled as a continuous-time
random process.
(d) In zero-error data compression using a variable-length code, a zero-error code is defined
as a non-singular code.
(e) A zero-error variable-length source code is always a prefix-free (PF) code.
P|X |
(f) If a D-ary PF code has the codeword-length profile [l1 , l2 , ..., l|X | ], then i=1 1/Dli ≤ 1.
(g) Given a source distribution [p1 , p2 , ..., p|X | ] of a DMS, there always exists an integer D(≥ 2)
such that logD 1/pi is an integer for all i.
(h) If two discrete random variables X and Y are independent, then HD (X, Y ) = HD (X) +
HD (Y ) for any integer D(≥ 2).
(i) If the codewords assigned to the least two probability masses are siblings for a PF code,
then it is an optimal PF code.
(j) If there exists a D-ary uniquely decodable (UD) code with the codeword-length profile [l1 , l2 , ..., l|X | ], then there always exists a D-ary PF code that has exactly the same
codeword-length profile as the UD code even without rearranging the codeword lengths.
(k) Given a discrete source with |X | outcomes of non-zero probabilities and a D-ary code with
|X | codewords, the minimum expected codeword length is achieved by assigning longer
codewords to less probable outcomes.
(l) Given a discrete source with |X | outcomes of non-zero probabilities, a Huffman procedure
that considers up to the level K of a D-ary tree can assign at least D + (K − 1)(D − 1)
distinct codewords to the outcomes.
(m) To achieve the entropy lower bound by using variable-length PF codes constructed through
Huffman procedures, we always need to encode asymptotically large number of data symbols together.
(n) The convergence of a sequence of random variables is essentially the convergence of a
sequence of functions.
(o) The central limit theorem is a special case of the convergence of a sequence of random
variables in distribution.
(p) The almost sure convergence of a sequence of random variables always implies the convergence in distribution.
2
(q) The weak law of large numbers is a special case of the convergence of a sequence of random
variables with probability one.
(n)
(r) Given a discrete-random variable with the sample space X , if x ∈ Aε where xi ∈ X and
(n)
(n)
Aε is the set of ε-typical sequences of length n, then |Aε | ≥ (1 − ε)Dn(HD (X)−ε) for
any integers D(≥ 2) and n.
(n)
(s) Given a discrete-random variable with the sample space X , if x ∈ Aε where xi ∈ X
(n)
(n)
and Aε is the set of ε-typical sequences of length n, then |Aε | ≤ Dn(HD (X)+ε) for any
integers D(≥ 2) and n.
(t) If
i ’s are i.i.d. discrete random variables with pmf pX (x) and the sample space X , then
PX
n
log
D (1/pX (Xi ))/n converges to HD (X) in distribution.
i
3
Problem 2. (30 points) Answer the following questions:
(a) (5 points) When C (1) = {0, 10, 110, 111}, find its 2nd order extension C (2) .
(b) (5 points) Find
1
lim (ax) x ,
x→∞
where a is a positive real number.
(c) (5 points) When D ≥ 2 and li ’s are natural numbers, show that
Ã
!
m
n
X
X
−li
ln+1
ln+1 −li
D ≤ 1 =⇒ D
−
D
≥ 1, ∀n < m.
i=1
i=1
(d) (5 points) When a source is modeled by a discrete random variable X having the probability masses 0.05, 0.15, 0.25, 0.25, and 0.3 for the outcomes x1 , x2 , x3 , x4 , and x5 , respectively,
find a ternary (D= 3) UD code that achieves the minimum average codeword length. Also
compute the average codeword length and compare it with the entropy of X.
(e) (5 points) When a and x are real numbers greater than 1, show that
lim
n→∞
dn loga xe
= loga x.
n
(f) (5 points) Fill in the black.
lim Pr(|Xn − X| > ε) = 0 if and only if
Ã
n→∞
∀ε0 > 0, ∃N (ε0 ) ∈ N, such that n ≥ N (ε0 ) ⇒
4
!
.
Problem 3. (20 points) Suppose that Xi ’s are i.i.d. binary random variables with Pr(Xi =
0) = 0.25 and Pr(Xi = 1) = 0.75. Answer the following questions.
(a) (5 points) Find the minimum expected codeword length per source symbol achievable by
a binary UD code, when Xi ’s are encoded separately for i = 1, 2, 3, ....
(b) (5 points) Find the minimum expected codeword length per source symbol achievable by
a binary UD code, when X2i−1 and X2i are encoded together for i = 1, 2, 3, ....
(c) (5 points) Find the minimum expected codeword length per source symbol achievable by
a binary UD code, when X3i−2 , X3i−1 , and X3i are encoded together for i = 1, 2, 3, ....
(d) (5 points) Compare the code rates of (a), (b), and (c) with the entropy H2 (X) of X.
5
Problem 4. (15 points) Suppose that we have i.i.d. discrete random variables Xi ’s with a
finite sample X and the probability mass function pX (x). When the binary entropy of X is
H(X), X (n) is defined as
X (n) , [X1 , X2 , ..., Xn ],
(n)
and Aε denotes the n-th order ε-typical set with respect to the base 2, answer the following
questions.
(n)
(a) (5 points) Show that Pr(X (n) ∈ Aε ) ≤ 1 − ε for sufficiently large n.
(n)
(b) (5 points) Show that |Aε | > (1 − ε)2n(H(x)−ε) for sufficiently large n.
(n)
(c) (5 points) Show that all the members in Aε can be assigned distinct binary codewords
of a fixed length that is less than or equal to n(H(X) + ε) + 1 [bits] for all n.
6
Problem 5. (25 points) Suppose that Xi ’s are i.i.d. binary random variables with Pr(Xi =
(n)
0) = 0.25 and Pr(Xi = 1) = 0.75. When Aε is defined as the set of ε-typical binary sequences
of length n, answer the following questions.
(n)
for n = 1 and ε = 0.5.
(n)
for n = 2 and ε = 0.3.
(n)
for n = 3 and ε = 0.2.
(a) (1 points) Find Aε
(b) (1 points) Find Aε
(c) (1 points) Find Aε
(d) (1 points) Show that the least probable all-zero sequence (0, 0, ..., 0) of length n is not a
(n)
member of Aε for any n when ε < 0.1.
(e) (1 points) Show that the most probable all-one sequence (1, 1, ..., 1) of length n is not a
(n)
member of Aε for any n when ε < 0.1.
(f) (10 points) As shown in (e), the encoder-decoder design method that utilizes the sequence
of the ε-typical sets of length n not only excludes the least probable dataword but also
excludes the most probable dataword from the sequences that are correctly decoded.
Rather, it assigns distinct codewords to the datawords with each probability close to
2−nH2 (X) . Let the code rate of this method be Rn and the error rate be Pe,n . Now,
suppose that you instead adopt the design method that assigns distinct codewords to the
datawords such a way that after you sort the datawords in the decreasing order of the
probability and the most probable ones with total probability 1 − ε are to be correctly
0 .
decoded. Let the code rate of this second method be Rn0 and the error rate be Pe,n
0 = ε > 0.
Compare Rn and Rn0 , when Pe,n = Pe,n
n
(g) (10 points) In (g), compare limn→∞ Rn and limn→∞ Rn0 when εn & 0 as n → ∞. Fully
justify your answer.
7