DOCUMENT ROOM,
~T
ROOK 36-412
RESEARCH LABORATORY CF ELECTRONICS
MASSACHUSETTS INS' ! UTE OF TECHVOCLO '
LIST DECODING FOR NOISY CHANNELS
PETER ELIAS
Technical Report 335
September 20, 1957
RESEARCH LABORATORY OF ELECTRONICS
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
CAMBRIDGE, MASSACHUSETTS
Reprinted from the
1957 IRE WESCON CONVENTION
Part 2
PRINTED IN
_
THE U.S.A.
RECORD
The Research Laboratory of Electronics is
an interdepartmental laboratory of the Department of Electrical Engineering and the
Department of Physics.
The research reported in this document
was made possible in part by support extended the Massachusetts Institute of Technology,
Research Laboratory of Electronics, jointly
by the U. S. Army (Signal Corps), the U. S.
Navy (Office of Naval Research), and the
U. S. Air Force (Office of Scientific Research,
Air Research and Development Command),
under Signal Corps Contract DA36-039-sc64637, Department of the Army Task 3-9906-108 and Project 3-99-00-100.
_
LIST DECODING FOR NOISY CHANNELS
PETER ELIAS
Technical Report 335
September 20, 1957
RESEARCH LABORATORY OF ELECTRONICS
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
CAMBRIDGE, MASSACHUSETTS
Reprinted from the
1957 IRE WESCON CONVENTION RECORD
Part 2
PRINTED IN THE U.SA.
-_II--UXI
LIST DECODING FOR NOISY CHANNELS
Peter Elias
Department of Electrical Engineering and Researcn Laboratory -of Electronics
Massachusetts Institute of Technology, Cambridge, Massachusetts
Summary.
Shannon's fundamental coding
q = 1-p that it will be received correctly.
theorem for noisy channels states that such a
Successive transmitted digits are treated by the
channel has a capacity C, and that for any trans-
channel with statistical independence.
mission rate R less than C it is possible for the
The capacity C of this channel is given by
receiver to use a received sequence of n symbols
C = 1
to select one of the 2n R possible transmitted
H(p)
bits/symbol
(1)
where
H(p) = -p log p - q log q
sequences, with an error probability Pe which can
(2)
be made arbitrarily small by increasing n, keeping
is the entropy of the p,q probability distribution.
R and C fixed.
C is plotted in Fig. 1 as a function of p.
Recently upper and lower bounds
Note
have been found for the best obtainable Pe as a
that if p ? q, the output symbols can be rein-
function of C,R and n. This paper investigates
terpreted, an output 1 being read as a 0 and vice
this relationship for a modified decoding pro-
versa,
cedure, in which the receiver lists L messages,
considered.
rather than one, after reception.
out the paper are taken to the base 2.)
In this case
for given C and R, it is possible to choose L
so that only values of p C q will be
(The logarithms in Eq. 2 and through-
C can be given two interpretations.
First,
large enough so that the ratio of upper and lower
if 0's and l's are transmitted with equal proba-
bounds to the error probability is arbitrarily
bility and statistical independence, then one bit
near to 1 for all large n. This implies that for
per symbol is supplied at the
large L, the average of all codes is almost as
average of H(p) bits per symbol of equivocation
good as the best code, and in fact that almost
remains in the output.
all codes are almost as good as the best code.
coding theorem for noisy channels, if the input
Second, by the fundamental
information rate is reduced from 1 bit per symbol
Introduction
The binary symmetric channel (abbreviated
BSC) is illustrated in Fig. 1.
input, and an
to any rate R < C bits per symbol, it is possible
It accepts two
for the receiver to reconstruct the transmitted
input symbols, 0 and 1, and produces the same two
sequence of input symbols with as low an
symbols as outputs.
probability,as is desired, at the cost of a&delay
There is probability p that
rror
an input symbol will be altered in transmission,
of n symbols between transmission and decoding,
from 0 to 1 or from 1'to 0, and probability
where n must be increased to reduce the error
probability.
This work was supported in part by the Army
(Signal Corps), the Air Force (Office of Scientific Research, Air Research and Development
Command), and the Navy (Office of Naval Research).
In block coding, the input information rate
is reduced by using only a restricted number,
94
_
__
___
_
M = 2R, of the 2n possible sequences of length n.
therefore even the exponent of Popt is not known,
This gives an average rate of (l/n)log M - R bits
although it is
per symbol; the sequences being used with equal
rates, it can be shown that the' exponent of Popt
probability.
is
The receiver then lists the single
Finally, for very low
definitely different from that for either P
av
or Pt'.
transmitter sequence which is most probable, on
As can be seen-from this brief description,
the evidence of the received sequence of n noisy
symbols.
bounded.
It is sometimes wrong, with a probability
the behavior of Popt is complicated, and not well-
which depends on the particular set of sequences
defined, even for the channel which has been
which have been selected for transmission.
studied most intensively.
define Popt
We
The present paper
discusses a somewhat more complicated transmission
a function of n, R and C, as the
minimum average probability of error for a block
procedure for use with the BSC, which has the
code: i.e. as the average error probabilityforthe
advantage that the description of its error prob-
code whose transmitter sequences have been chosen
ability behavior is much simpler.
most
block coding with list decoding - leaves a little
udiciously, the average being taken over the
equivocation in the received message.
different transmitter sequences.
receiver provides a list of L messages, L <
for small n, a
number are known (see Slepian (3,4)).
Instead of
a unique selection of one message out of M, the
Unfortunately we do not know the optimum
codes for large values of n:
Our procedure -
However it
M.
As soon as a little residual equivocation is per-
is possible to define a lower bound to Popto
mitted, it turns out that P
has the same exponent
av
which we denote by Pt. which is a fairly simple
as Pt' and thus as Popt' for all transmission rates
function of the code and channel parameters.
and not just those which are near capacity.
It
This
is also possible to define Pa , the average error
is essentially Theorem 1.
probability of all possible codes, which is an
as L increases, Pa
upper bound to Popt'
that Popt itself, and not just its exponent,
The behavior of Popt is
discussed in some detail in references (1,2).
actually approaches Pt. so
becomes well-defined.
As
It also turns out that
This is the content of
the transmission rate approaches the channel
Theorem 2.
capacity, Pav approaches Pt. so that the error
the cause of detection errors in a system using
probability Popt is well defined.
list detection can be interpreted simply: mistakes
For somewhat
As a result of this simplification,
lower rates the two bounds differ, but for fixed
are made when, in a particular block of n symbols,
R and C their ratio is bounded above and below by
the channel gets too noisy.
constants as n increases.
ordinary block coding, on the other hand, errors
Since each is decreas-
In the case of
ing exponentially with n, at least the exponent of
also occur because of unavoidable weak spots in
Popt is well-defined.
the code, which causes the complication in the
For still lower rates, how-
description of Popt for such codes.
ever, Pav and Pt decrease with different exponents
as n increases with C and R fixed.
In this region
95
_I____·______
I_
I
_
__
_
Operation
being ranked in the same manner as the noise
The operation of a block code with list
sequences.
The received sequence is decoded by
decoding is illustrated in Fig. 2. There are M
means of a decoding codebook, as illustrated in
equiprobable messages, represented by the integers
Fig. 3, which provides a list of L = 3 messages
from 1 to M, and the source selects one, say the
for each received sequence.
integer m = 2.
This is coded into a sequence
The decoding codebook is constructed by list-
um = U2 of n binary digits, 011 in the example,
ing, after each received sequence, the L messages
for which n = 3.
whose transmitter sequences are converted into the
The sequence um is transmitted
over the noisy BSC.
the digits:
The channel alters some of
given received sequence by noise sequences of the
its effect is to add a noise sequence
lowest possible rank.
Because of the equal prob-
wr, in this case the sequence 010, to the trans-
ability of the transmitter sequences and the
mitted sequence um to produce the received noisy
statistical independence of transmitter sequence
sequence,
= 001.
2
(The addition is modulo 2,
and noise sequence in the BSC, the L messages
digit by digit: 1+1 = 0+0 = 0, 1+0 = 0+1 = 1.)
which are most probable on the evidence of the
The received sequence is decoded into a list of
received sequence are those which are converted
L messages - L = 3 in the Figure.
into the received sequence by the L most probable
If the list
includes the message which the source selected,
noise sequences.
the system has operated correctly and no error has
will be listed in the .codebook, except that the
occurred.
ranking of the noise sequences will provide an
If not there has been an error.
Since
These are just the ones which
2 is on the list 1,2,3, the transmission illustrated
unambiguous decision when several noise sequences
was successful.
have the same probability.
The behavior of the system is determined by
the choice of u I ,...u
sequences.
m ....
,uM,
This follows because
the probability Prob[wr] of a noise sequence wr
the transmitter
with k l's is just
An encoding codebook, illustrated for
Prob[wr
pkqn-k
qn (p/q)k
(3)
the example-in Fig. 3, assigns M = 4 such se-
which is monotone decreasing in k for p <
quences each of length n = 3, to the four messages
while rank is monotone increasing with k.
m = 1,2,3,4.
The 2 = 8 possible noise sequences
The behavior of the system under all possible
of length n are ranked, by an index r, so that the
sequence with no l's, w
q,
circumstances is illustrated in the table of
= 000, comes first, a
possible events in Fig. 3.
The message sequence
sequence with k l's precedes one with k+l l's, and
U2 and the noise sequence w3 determine a row and a
sequences with the same number of l's
column in the table.
by their size as binary numbers.
are ranked
In the example
The received sequence v2 =001
which is their digit-by-digit sum modulo two is
the noise sequence w3 = 010 is added to the trans-
entered at the intersection.
mitted sequence to produce the received sequence
v2 : the 2n possible received sequences vl...,v2n
book shaws that v2 will be decoded as the list
The decoding code-
1,2,3 which includes the message m = 2 which was
96
I
-
-
-
-
--
-'
actually transmitted.
Therefore the number of circled entries
sequence.
This entry in the table is
therefore circled, as an event which the system
in row m is equal to the total number of times
will handle without error, and so are all such
n
that the integer m appears in all of the 2 lists
of L messages in the decoding codebook.
The probability of error for the system
events.
total of L
is then just the sum of the probabilities of the
2n
n
uncircled events, which are all of the entries in
total of M-2
the last two columns.
of
2 n(l
Thus a
entries are circled, out of the
entries in the table, or an average
R) circles per message.
Clearly the minimum probability of error
Rate of Transmission
would be obtained if all of the circles were as
The average amount of information received
over this kind of system when no error has been
far to the left as possible in the table.
made depends on the probabilities of the different
true for the code in Fig. 3, but will not be possi-
messages on the decoded list.
ble for other values of M, L and n.
To make it constant,
This is
However it
we assume that the messages on the list are
provides a lower bound to error probability in any
shuffled in order before being given to the sink,
case.
so that they are equiprobable to him.
code (perhaps unrealizeable) with such a table.
Then the
We define Pt as the error probability for a
Let r
rate is
be the last column in which any circled
entries appear:
R =
log M - log L
n
(4)
bits/symbol,
rlM 2 L2
n
> (rl-)M, or
and to transmit at rate R requires that
(6)
r1
M = L.2nR
messages be transmitted.
>
2 n(1-R)>
1
(5)
Then Pt is bounded:
In the example of Figs.2
and 3, we have M = 4, L = 3, n = 3, and R = (1/3)
n
n
2
2
log (4/3)
- 0.14 bits/symbol.
Prob[Wr]
Note that the largest M which can be used
> Pt
r=r I
Prob[wr]
(7)
r=rl+l
effectively is M = 2n, since there are only that
many distinct sequences.
Thus L =
2 n(
1- R
large a list as would be of any interest.
The quantity Pt is characteristic of the
) is as
channel, and of its use at rate R in blocks of
This
would correspond to transmitting uncoded infor-
N symbols, but is independent of the code used.
mation-i.e., all possible sequences.
The channel parameters p,q enter in evaluating
Error Probability: Lower Bound
Prob[wr ]
n
In any row of the table in Fig. 3, all 2
ProbEwr] = pk(r)qn-k(r)
(8)
possible received sequences appear, since each of
the
2n
where k(r) is determined by means of the binomial
noise sequences produces a distinct received
coefficients
sequence when added to the given transmitter
()
= n!/j!(n-j)
,
97
_I__I___ILWLP·_III1II__--L-_·^L-LLI
,;
giving equal weights to each.
k(r)-l
k(r)
Pay can be expressed as a sum over r of the
(9)
(/)
(in
j =0
probability of occurrence of the rth noise sequence
wr, times the ensemble average conditional prob-
Using Eq. 8, and introducing k1 as an
abbreviation for k(r1) defined through Eq. (9), we
ability QL(r) of making a decoding erro
have the following bounds for Pt:
noise sequence wr occurs:
2n
L
pk(r)qn-k(r) > Pt
r=r 1
av
pk(r)n-k(r)
when the
r(12)
(10)
r=rl+l
Now we can bound Pav above and below.
Since
it is an average of quantities all of which are
n
(n) pkqn-k
(kI pq
p
greater than Pt. Pt is a lower bound.
>
kk
k=kl
(l+l
n-k
I(11)
Since QL(r)
is a conditional probability and thus less than
1rr~
one, it can be bounded above by one for a portion
of the range of summation.
The first of these is needed for Theorem 2.
Thus, using Eq. 10,
The second, which is simpler to evaluate, suffices
pk(r) n-k(r)((r
for Theorem 1.
Error Probability: Upper Bound
)
2n
k(r) n-k(r)
An upper bound to the best attainable average
(13)
r+l l
error probability is obtained by Shannon's procedure of random coding - i.e. by computing the
r,
average error probability for all possible codes.
pk(r)n-k(r)%(r) +
This turns out to be much easier than computing
More crudely, we define Q
the error probability for any one particular code.
(k):
Since a code is uniquely determined by its set
ul,...,uM of transmitter sequences, there are as
q(k(r)) =
nR
length n, as there are selections of M = L.2
2n
)
Q(r) ,
(14)
j =0
many codes possible, for rate R, list size L and
objects out of
(n)
(
by Eq. 9 and the fact that QL(r) is monotone
increasing in r (See Eq. 16, below).
possibilities, repetitions
Then
permitted, or
(15)
k0
2 nM
2 nL*
k=O
2n R
Finally, an explicit expression can be given
We define P as the average error
av
probability, averaged over all transmitter
for Q(r).
sequences in any one code and over all codes,
occurred in a transmission, the probability of a
in all.
Given that the rth noise sequence has
98
_
_____
The value of L
decoding error is just the probability that L or
0
needed for the theorem is
more of the other M-1 transmitter sequences differ
log(q/p)
L
from the received sequence by error sequences of
rank
0
=
-1 .
log(ql/pl)
(19)
r . This is the probability of L successes
Corollary
in M-1 statistically independent tries, when the
Under the same conditions, the exponent of
average probability of success is r/2 n, for in the
ensemble Gffcodes the different transmitter sequences are si.seically independent.
P
opt
as n increases is
Thus QL(r)
is a binomial sum:
lim l/n)log opt
cs
= nlim
(l/n)log a)
=
QL(r)
L(Ml )
=
(r/2 n) (1-r/2n)
. (16)
lim
(l/n)lo
-gPt
(20)
)
3JL
=
(P1)-
(p)
-
(pl-p)log(q/p)
The Parameter pi . Theorem 1 .
The proof of the corollary given the theorem
It would be desirable to express the error
probability directly as a function of rate, capaci-
is direct.
ty, block size n and list size L.
shows that Pv and Pt have a common exponent.
However it
Taking logarithms and limits in Eq. 18
turns out to be more convenient for Theorem 1 to
Since they bound Popt above and below, it shares
use, instead of rate, the parameter p
the same exponent.
where k
= k(rl) is defined by Eq.9.
= kl/n,
large n the relation is simple:
the exponent of Pt, as found in references (1,2).
Fixing P1
and n determines the rate R through Eq. 6.
The value given in Eq. 20 is
It is illustrated geometrically in Fig. lb.
For
Proof of Theorem 1.
if we define the
Bounding Pav in terms of Pt requires bounding
rate R1 by the equation
the sum in Eq. 15.
R1 = 1 - H(pl1 )
For the last term in the sum we
(17)
use the bound Q(kl) =1
For the earlier terms
as illustrated in Fig. lb, then for fixed Pi,
we use the bound of Eq. A6, derived in Appendix A.
R approaches R1 from above as n increases.
Eq. 15 then becomes
This
is shown in references (1,2).
We can now state
PaPay -%
eL (n P-
j
e
pkqnk
{ (k
()
/ (knq
/
k=o
Theorem 1
(n
Q+
)
Given a BSC with error probability p, and
kI n-k I
p q
(21)
+ Pt
given any P1 , with o p (p1< i, and any
L > Lo(p,p), the ratio of PaV to Pt for list
decoding with list size L is bounded, independent
of n:
1
Now the sum in Eq. 21 can be bounded by a
Pav/Pt
=
A(pp
1 ,L)
(18)
geometric series, which sums:
99
U·
_ I(~U
~ ^ )__I_
---- y ~ C~~^_
I·-tI
.IY-.. ·-ll·_1X----·_
I-~Y-.
the weakness of the bound of Eq. A6 on Q(k) for
kl-l
kqn-k
()
)
p
k near k1 . Actually A(p,pl,L) decreases as L
/
(kllY
L+l
k-
increases.
k=-o
In fact we have
Theorem 2
kll n-kl+l
P
(22)
Given a BSC with capacity C, and a rate R
q
i - (q/p)(pl/ql)L~
9
with 0 < R < C < 1 , and given any e >
0 ,it
is possible to find an L(e) so large that for all
where we have used k1 = npl and n-k
= nql.
I he
n > n(L),
convergence of the series is guaranteed by thee
requirement that L > L as given by Eq. 19,
1
s0
PaV/Pt
=
.
1+
(26)
that the denominator on the right in Eq. 22 ii
s
positive.
Corollary
Substituting Eq. 22 in Eq. 21 and regrou]ping,
For sufficiently large L and n, almost all
codes are almost as good as the best code.
Pa Is
av
n )
lkn-k1
1 +
Given the theorem, the corollary follows by
\k)
(23)
an argument of Shannon's in reference (5). The
L
+qP
difference between Popt and the error probability
+ Pt
Pql
1 - (q/p)(pl/ql)L+
of any code is positive or zero, and the average
value of the difference, by Theorem 2, is < c.
Now, from Eq. 11,
Thus only a fraction 1/T of the codes may have
such a difference
t
=t
kl+l
k
p
+
n-k -l
(l-l/T) have an error probability
q
(24)
= (P/q)
k)
Te , and at least a fraction
QED.
k1 n-k1
P q
(l+Tc)Pop t ,
The result here is stronger than in Shannon's
application because of the nearby lower bound.
Proof of Theorem 2.
Substituting Eq. 24, read in reverse, in
For a constant rate R,p1 = kl/n is not a
Eq. 23 gives
constant, but depends on n . However for sufficiently large n it is arbitrarily close to the
Pav/Pt
(q/p)
1
+
(25)
limiting value given by setting R
L
+
Let p
qPl
Pq
1
-
(q/ppl
/
L+
(q/p)(p 1 /q1 )LJ
be this limit.
R1 in Eq. 17.
Then
+
QED.
lim
Theorem 2.
The proof of Theorem 1 suggests that
A(p,pl,L) increases with L, but this is due to
100
kl*
/
~k
-1)
=
(27)
Choose J so that
(pl/,l)
=
-(l+a)
r n<
1 k
(28)
,a > O .
11
(33)
p/l
Substituting this in Eq. 32, we have
Then breaking the sum in Eq. 13 at
k1-J
L
°
(29)
( k)o
C1
r1
Q(r) < p
k=o
q
L
r=r
0
and using the bound of Eq. A6 with the substi-
(34)
C2
"tution of Eq. 27 below this point gives a term
n
k,
which is bounded by
(n
-(L+l)a
pkqn-kl
Now the denominator under C1 increases
1-(q/p)(p*/q*L+l
/
(30:
arbitrarily with L, and the denominator under C2
LP)(l
ql)
increases arbitrarily with n.
Next, the portion of the sum in Eq. 13 which
is
between r,
bounded.
in Eq. 34 becomes negligible for large n and L
given in Eq. 29, and rl, is
kn-k
Since p q
Thus the summation
compared to the term
is monotone decreasing in
k1 n-k
q
k, we have
n
k
(35)
(31)
So does the expression in Eq. 30 which bounds the
Z:
pk(r)qn-k(r)QL(r)
(q/p)Jpklqn-kl
(r).
remainder of the sum in EQ. 13, because of its
r=r
r=rr
exponential factor.
We now bound QL(r) by Eq. B4, derived in
appendix B.
true for Pt
This gives
r1
o
e-(L/2)(rl
r=r
Thus in the limit the right
Conclusion.
- r )
/r
Two points should be noted in conclusion.
o
First, if
if
1+
in fact of the order of the
13 approaches Pt, QSD.
2
r2
(r)
r=r
which is
expression in Eq. 35.
side of E.
But from Eq. 24, this is not
f-Lr
2
1
2
/2r
dr
L grows exponentially with n - that is,
there is a residual percentage of equivocation,
rather than a
esidual amount - then the ratio
0
, C
1 +
(r 1/2) V&-IL
From Eq. 9, using a geometric series to
Pav /Pt approaches 1 exponentially,
since\Lf
then a negative exponential
Second,
in n .
discussion of Theorem 2 shows that Popt is
bound the sum of binomial coefficients as in
near Pt
references (1,2,6) gives
them are.
is
the
very
but does not say Just how big either of
Pt is of course given by Eq. 9, with r
101
-I~IP~^___^^_111_·
-
___._
Now from Eq.'s 6 and 9,
kl-l
(1-) =
( n1
defined by -Eq. 6, and the exponent is given by
Eq. 20.
But no really tight bounds for Pt or
(A4)
Popt in terms of C, R, and finite n and L have
been given.
And, by term-by-term comparison, for k - k 1-l,
These take a good deal more of
detailed algebra, which will be presented elsewhere.
Acknowledgements.
k -l
k
1- 1
Theorem 1 was discovered independently by
E
Professor J. M. Wozencraft in the course of his
doctoral research.
J-o
(in)
j=o
.e.
=
(k
/ (k 1 1) , (A5 Y
It does not appear in his
thesis (10) since it was not directly relevent to
so from Eq. A2 and the definition in Eq. 14 we
the final topic, but he will publish a note on the
have for k - kl-,
subject shortly.
Shannon introduced the author to
the work of Chernoff and Cramer (7,8), which makes
the bound of Appendix B possible.
Without this
kind of bound the argument for Theorem 2 is even
more tedious.
QL(k) = (eL/
APPENDIX A:
)
/
{()
(kl
}
A BOUND ON Q( r).
(A6)
4
L
n
G
n
The probability of L or more successes in-M-1
tries, with probability r/2n of success on each
try, can be bounded by the product of (r/2n)L , the
probability of success in a part&cular L o
the
APPENDIX B:
M-1 tries, times the number of ways of selecting
ANO1HERO
BOUND ON
(r).
L from M-1 :
A result due to Chernoff (7), which may also
QL(r)
(M1)
(r/2n)L
be derived from Cramer (8), provides bounds on the
(Al)
tails of distributions of sums of random variables
< ( /Lt)(r/ 2n)
Stirling's lower bound to L
in terms of the moment-generating function of the
gives
parent distribution. Applying this to the binomial
distribution, Shannon (9)
(r)
)(/L(rn)
< (L/E'
i
(e /'2nL
L
)(Mr/L2n)L
kb = N%> Npa ,
Pb (a) } N
(A2)
)(r/2n(1-R),
k=kb
making use of Eq. 5. From Eq. 6 we have
QL(r) = (e/
shows that for
)(r/(rfl))L
(A3)
102
{(
)
Taking 1/N times the natural logarithm of the
right side of Eq. B1, and defining & =Pb-Pa >
QL(r)
O,
exp ( - (M/2)(rl-r)2 /rl.2n )
gives
Iexp( - (L/2)(rl-r)/2n(1-R)rl
1 )
Pb
(B4)
+ b loge(l+6/%b)
loge(1-S/)
= exp ( - (L/2)(r 1 -r)2 /r
References
(B2)
(S/J)
cs
We take qb > Pb
coefficient of 6
( )J
1
)
ED.
q1J.
Pbb-
1.
Elias, P., Coding for Noisy Channels", I.R.E.
Convention record part 4, p.37-46 (1955).
Then 1/qb < 1/Pb
in Eq. B2 is >
0.
so the
2. Elias, P., Coding for Two Noisy Channels", in
Cherry, (editor) Informtion Theory, p.61-74,
Butterworth (1956).
Then the
sum in Eq. B2 is more negative than its leading
3. Slepian, D., "A Class of Binary Signalling
Alphabets", Bell Syst. Tech. J. 25, p.203-234
(1956).
term, and
4. Slepian, D., A Note on Two Binary Signalling
Alphabets', Trans. I.R.E. PGIT IT-2, 61-74 (1956).
N
5.
L
(k)
k=kb
Pa qb
2
= exp ( - N(
/2)((lpb)
Shannon, C. E., 'The Mathematical Theory of
Communication , University of Illinois Press
(1949) see p.bO .
-
6. Feller, W., Probability Theory and its
Applications , Wiley (1950) see .126, Eq. (8.1)
+,(l/qb)) )
(B3)
<= ep ( - s 6/2pbqb)
< exp ( - N
2
/2pb)
In applying this to QL(r), we have pa = r/2
Pb
=
rl/2n
n .
= (rl-r)/2 n , and we may use N = M,
,
since increasixg M-1 to M in Eq. 16 increases the
sum.
Thus, using Eqs. 5 and 6,
7. Chernoff, H. 'A Measure of Asymptotic
Efficiency for Tests of a ypothesis Based
on a Sum of Observations', Ann. Math. Stat. 2,
(1952).
8. Cramer, . Sur un Nouveau Theoreme-Limite
de la Theorie des Probabilites' in Actualites Sci.
et Indust., no. 736, Hermsnn et cie (1938).
9. Shannon, C. ., Information Seminar Notes
M.I.T. (1956) (unpublished).
10. Wozencraft, J. M., Sequential Decoding for
Reliable Communications , Dissertation submitted
to Elect. Eng. Dept., M.I.T. (1957).
q
INPUT
OUTPUT
SYMBOLS
I
_
IE--C - H(P)
R,C t
SYMBOLS
q
THE BINARY SYMMETRIC CHANNEL
O ' I
Fig. la.
P
-
CHANNEL CAPACITY
Fig. lb.
103
·I__·I
I_^L_
I_
n SYMBOLS
O11
n SYMBOLS
L MESSAGES
TRANSMITTED SEQUENCE
NOISE SEQUENCE
RECEIVED SEQUENCE
W 010
=
1,2,3
001
OI
NOISY
CHANNEL
OPERATION
DECODED
MESSAGE LIST
RECEIVED
SEQUENCE
TRANSMITTED
SEQUENCE
MESSAGE
2
001
BLOCK CODING AND LIST DECODING
Fig. 2.
ENCODING
MESSAGE
m
I
2
,
3
4
NOISE
CODE BOOK
_
TRANSMITTER
SEQUENCE um
RANK
SEQUENCE
r
Wr
u
000
I
W
=
I
u2
0 11
2
-
w2
u3
I101
3
-
w3
U4"
I 10
4
,
5
,
6
DECODING CODE BOOK
SEQUENCES
w4
W
5 -
w6
7
-
w7
8
_
w
000
001
010
1 00
011
101
10
I II
RECEIVED
SEQUENCE v,
VI
V2
V3
V4=
V5
V6
V7
=
000
001
010
1 00
011
101
110
Vs
8
DECODED
MESSAGE LIST
1,2,3
1,2,3
1, 2,4
I, 3,4
1,2,4
1,3,4
2,3,4
2,3,4
EXAMPLE OF LIST DECODING SYSTEM
Fig. 3a.
NOISE
SEQUENCES
PROB [wr]
CIRCLED
ENTRIES
ARE
CORRECTLY
TRANSMITTER
SEQUENCES
DECODED
EXAMPLE
OF LIST DECODING SYSTEM
Fig. 3b.
104