Module 5: Lecture 41
Information Theory: Channel Coding
Dr. S .M. Zafaruddin
Associate Professor
Deptt. of EEE, BITS Pilani, Pilani Campus
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
1 / 50
Objectives of Today Lecture
Shannon Limit
Block Codes
What next...
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
2 / 50
Shannon’s Channel Capacity for AWGN Channel
y =x+n
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
3 / 50
Shannon Channel Capacity of Band-Limited
AWGN
I(x; y) = H(y) − H(y|x) = H(y) − H(n)
Maximum H(y): 21 log2 [2πe(S + N )]
For Gaussian noise: H(n) = 12 log2 [2πeN ], N = N0 B.
For signal: best is Gaussian
For noise: worst is Gaussian
C = 12 log2 [2πe(S + N )] − 21 log2 [2πeN ]
C = 12 log2 [1 + NS ] bits per symbol
For a BW of B: maximum 2B symbols per second (Nyquist
ISI criteria)
C = 2B 12 log2 [1 + NS ] = B log2 (1 + S/N ) = B log2 (1 + SN R)
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
4 / 50
Shannon Channel Capacity of Band-Limited
AWGN
C = B log2 (1 + SN R)
Assumptions to get the above?
Channel: Band-limited
Noise: Additive
Noise: Gaussian distributed
Input signals: White Gaussian
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
5 / 50
Shannon Channel Capacity: Asymptotic
Capacity when transmitted signal power → ∞?
C→∞
Capacity when channel BW → ∞?
C = B log2 (1 + NS )
C = B log2 (1 + NS0 B ), where N0 : PSD, N : Power.
S
)
N0 B
S N0 B
limB→∞ N0 [ S log2 (1 + NS0 B )]
= NS0 log2 e = 1.44 NS0
limB→∞ C = limB→∞ B log2 (1 +
limB→∞ C
limB→∞ C
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
6 / 50
Spectral Efficiency
R<C
R < B log2 (1 +
S
)
N0 B
R
efficiency B
bps
bps/Hz: how many bits per second per
Spectral
Hz:
R
= log2 (1 + NS0 B )
B
Bit energy Eb = S/R, i.e., S = Eb × R
Eb R
R
< log2 (1 + N
)
B
0B
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
7 / 50
Shannon Limit
Eb
N0
>
2R/B −1
R/B
Eb
required to achieve the absolute
What is minimum N
0
R
minimum spectral efficiency B
→0
R/B
R/B
> limB→∞ 2 R/B−1 = limR/B→0 2 R/B−1 = loge 2 = 0.693 =
−1.6dB.
Shannon-Limit: Minimum Eb /N0 required to achieve the
absolute minimum spectral efficiency = Eb /N0 = −1.6dB.
For reliable transmission: Eb /N0 > 0.693.
Eb
N0
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
8 / 50
Reliable Transmission
102
Non-Achievable Rate Region R>C
Spectral Efficiency ( )
101
Capacity Boundary (R=C)
100
Achievable Rate Region R<C
10-1
10-2
10-3
-1.6 dB: Shannon Limit
10-4
-50
0
50
100
150
200
250
300
Eb/N0 (dB)
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
9 / 50
Communication Systems with Channel Coding
Discrete
Source
Channel
Endoder
Modulator
Channel
Dr. Zafar (BITS Pilani)
Demodulator
Detector
CommSys: M5L41 IT
Channel
Decoder
Sink
10 / 50
Error Correcting Codes: Motivation
For a single data bits k = 1, transmit n = 3 bits.
Code length: number of bits in a code = 3.
Code word: combinations of n = 3 bits
Check bits m = n − k = 2, code = (n, k).
How correction works when instead of a single digit, we repeat the
same digit by 3 times.
For 1, we encode as 111, and for 0, we encode as 000.
Received with possible combinations
Rx
000
001
010
100
111
110
101
011
Dr. Zafar (BITS Pilani)
Detected
0
0
0
0
1
1
1
1
CommSys: M5L41 IT
11 / 50
Shannon’s Noise Channel Coding Theorem
For a noisy channel with a capacity C, there exists a code of
rate R < C such that optimal decoding results:
Pe ≤ 2−nEb (R)
FEC: forward error correcting codes.
Probability of error can be minimized by increasing the block
code length n while keeping the code rate of data bits
constant.
Problem: Find good codes that minimizes the error
probability by increasing n but with a simpler decoding.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
12 / 50
Error Correcting Codes
Block Codes
Cyclic codes: a type of block codes
Convolution Codes
Turbo Codes: a type of convolution codes
Low density parity check codes (LDPC): a type of block codes
Trellis Coded Modulation (TCM)
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
13 / 50
Modulo-2 Addition
XOR operation
1 mod 1 = 0 mod 0 = 0
0 mod 1 = 1 mod 0 = 1
Subtraction is same as addition.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
14 / 50
Hamming Distance
If two binary sequences of the same length differ in j places:
Hamming distance d = j.
Hamming distance between 000 and 111?
Ans: d = 3
weight of a code word: number of 1s.
Minimum weight (except 0 code)= Hamming distance.
To correct upto t bits in errors: dmin = 2t + 1.
To detect upto t bits in errors: dmin = t + 1 (retransmission
ARQ)
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
15 / 50
Hamming Codes
P
A code that satisfies the Hamming bound: 2m ≥ tj=0 nj
P
If 2m = tj=0 nj : Perfect Code
Hamming codes: Binary, single-error correcting, and perfect
codes
Hence, t = 1, dmin = 3.
P
2m = 1j=0 nj = 1 + n
Thus: n = 2m − 1
Hamming codes (n, k): n = 2m − 1, k = 2m − 1 − m, and
parity check bits m ≥ 3.
Hamming codes (2m − 1, 2m − 1 − m, m)
(7, 4, 3) is an example of Hamming code.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
16 / 50
Linear Block Codes
Data digits d = {d1 , d2 , d3 , ..., dk }
Here, d1 , d2 ...are single digits.
Code words c = {c1 , c2 , c3 , ..., cn }
Code rate : Rc = k/n.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
17 / 50
Linear Block Codes
Systematic codes: If c1 = d1 , c2 = d2 , ..., ck = dk and the
remaining digits from ck+1 to cn are linear combinations of
d1 , d2 , d3 ,...,dk .
Thus, in a systematic code, the leading k digits are the data
or information digits and the remaining m = n − k digits are
the parity check digits, formed by linear combinations of data
digits d1 , d2 , ..., dk .
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
18 / 50
Example
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
19 / 50
Solution
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
20 / 50
Linear Block Codes: Systematic
Generator matrix G(k × n) = [Ik (k × k) P(k × m)]
Elements of P are either 0 or 1.
Code word c = dG.
c = d[Ik P].
c = [d dP].
c = [d cp ].
where cp = dP: checksum or parity bits.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
21 / 50
Example
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
22 / 50
Solution
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
23 / 50
Problem
For a (6, 3) systematic linear block code
For systematic: Generator matrix
G(k × n) = [Ik (k × k) P(k × m)]
Generator matrix:


1 0 0 1 1 1
G = 0 1 0 1 1 0
0 0 1 1 0 1
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
24 / 50
Solution: Codes
For systematic: Generator matrix
G(k × n) = [Ik (k × k) P(k × m)]
P matrix:


1 1 1
P = 1 1 0
1 0 1
Code generation formula 1: c = dG
Code generation formula 2: c = [d dP]
Use the second one because it is systematic code.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
25 / 50
Solution: Codes
Information bits 000, 001, 010, 011, 100, 101, 110, 111
Codes can be generated as


1 1 1
[000] 1 1 0 = [000]
1 0 1
Code 1: [000000]
Similarly other codes can be obtained.
All codes [000000], [001101],
[010110],[011011],[100111],[101010],[110001],[111100].
dmin = 3
t=1
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
26 / 50
Problem
A generator matrix is given as
1 0 1 1
G=
0 1 1 0
Find the codewords for all possible input bits.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
27 / 50
Solution: Generator matrix
Data bits k = 2.
Code word length n = 4.
Redundancy m = n − k = 2
Generator matrix G(k × n) = [Ik (k × k) P(k × m)]
1 0 1 1
G=
0 1 1 0
I2 ?
1 1
P=
1 0
Systematic code: Yes (if the generator matrix G has the
above form )
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
28 / 50
Solution: Code 1
Corresponding to d = [00]
1 1
dP =[00]
= [00]: Modulo-2 addition
1 0
Code 1: c = [d dP] = [0000]
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
29 / 50
Solution: Code 2
Corresponding to d = [01]
1 1
dP =[01]
= [10]: Modulo-2 addition
1 0
Code 2: c = [d dP] = [0110]
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
30 / 50
Solution: Code 3
Corresponding to d = [10]
1 1
dP =[10]
= [11]: Modulo-2 addition
1 0
Code 3: c = [d dP] = [1011]
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
31 / 50
Solution: Code 4
Corresponding to d = [11]
1 1
dP =[11]
= [01]: Modulo-2 addition
1 0
Code 4: c = [d dP] = [1101]
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
32 / 50
Solution: Minimum Distance
Codes [0000], [0110], [1011], [1101]
In linear block codes [0000] must be a code.
Distance between two codes: how many bits they differ.
Minimum Hamming distance = minimum of all distances.
Minimum Hamming distance dmin = 2.
Error correction capability dmin = 2t + 1: t: number of bits
that can be corrected.
Error detection capability dmin = t + 1: t: number of bits
that can be detected.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
33 / 50
Solution: Minimum Distance
Also minimum Hamming distance for linear block codes =
minimum weight of a non-zero code
Weight = number of 1’s
Using the above property: dmin ?
Error correction capability dmin = 2t + 1: t: number of bits
that can be corrected.
Error detection capability dmin = t + 1: t: number of bits
that can be detected.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
34 / 50
Problem
Consider a generator matrix G for a nonsystematic (6, 3) code:


0 1 1 1 0 1
G = 1 1 1 0 1 0
1 1 0 0 0 1
Construct the code for this G, and find dmin , the minimum
distance between codewords.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
35 / 50
Solution
Why the above code is non-systematic?
For systematic: Generator matrix
G(k × n) = [Ik (k × k) P(k × m)]
Given

0 1 1 1 0
G = 1 1 1 0 1
1 1 0 0 0

1
0
1
There is no identity matrix of order k.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
36 / 50
Solution 3b
(6, 3) code, k = 3, n = 6, m = 3.
Information bits 000, 001, 010, 011, 100, 101, 110, 111
Given


0 1 1 1 0 1
G = 1 1 1 0 1 0
1 1 0 0 0 1
Code generation formula 1: c = dG
Code generation formula 2: c = [d dP]
Use the first one because it is not systematic code.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
37 / 50
Solution 3: code 1
Information bits 000, 001, 010, 011, 100, 101, 110, 111
Given


0 1 1 1 0 1
[000] 1 1 1 0 1 0 = [000000]
1 1 0 0 0 1
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
38 / 50
Solution 3: code 2...
Information bits 000, 001, 010, 011, 100, 101, 110, 111
Similarly, we can multiply each information bits with the
generator matrix G to get all eight codes.
All codes: [000000], [110001], [111010], [001011], [101100],
[100111], 010110].
dmin = 3.
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
39 / 50
Thanks !
Dr. Zafar (BITS Pilani)
CommSys: M5L41 IT
40 / 50