Algorithms in the Real World
Error Correcting Codes II
– Cyclic Codes
– Reed-Solomon Codes
Page 1
Viewing Messages as Polynomials
A (n, k, n-k+1) code:
Consider the polynomial of degree k-1
p(x) = ak-1 xk-1 + L + a1 x + a0
Message: (ak-1, …, a1, a0)
Codeword: (p(y0),p(y1), …, p(yn-1)) for distinct y0,…,yn-1
To keep the p(yi) fixed size, we use yi, ai  GF(pr)
To make the yi distinct, n < pr
Unisolvence Theorem: Any subset of size k of (p(y0),
p(y2), …, p(yn-1)) is enough to (uniquely) reconstruct
p(x) using polynomial interpolation, e.g., LaGrange’s
Formula.
Page 2
Polynomial-Based Code
A (n, k, 2s +1) code:
k
2s
n
Can detect 2s errors
Can correct s errors
Can correct 2s erasures
Generally can correct a erasures and b errors if
a + 2b  2s
2s = n-k, so this is an (n,k,n-k+1) code
Page 3
Correcting Errors
Correcting s errors:
1. Find k + s symbols that agree on a polynomial p(x).
These must exist since originally k + 2s symbols
agreed and only s are in error
2. There are no k + s symbols that agree on the
wrong polynomial p’(x)
- Any subset of k symbols will define p’(x)
- Since at most s out of the k+s symbols are in
error, p’(x) = p(x)
Page 4
A Systematic Code
Message: (m0, m1, …, mk-1)
Find polynomial p(x) = ak-1 xk-1 + L + a1 x + a0 such that
p(y0)= m0, p(y2)= m2, …, p(yk-1) = m0
Codeword: (m0, m1, …, mk-1, p(yk), p(yk+1), …, p(yn-1))
This has the advantage that if we know there are no errors (e.g.,
all points lie on the same degree-(k-1) polynomial), decoding is
trivial.
The version of RS used in practice uses something slightly
different.
This will allow us to use the “Parity Check” ideas from linear
codes (i.e., HcT = 0?) to quickly test for errors.
Page 5
Reed-Solomon Codes in the Real World
(204,188,17)256 : ITU J.83(A)2
(128,122,7)256 : ITU J.83(B)
(255,223,33)256 : Common in Practice
– Note that they are all byte based
(i.e., symbols are from GF(28)).
Decoding rate on 1.8GHz Pentium 4:
– (255,251,5) = 89Mbps
– (255,223,33) = 18Mbps
Dozens of companies sell hardware cores that
operate 10x faster (or more)
– (204,188,17) = 320Mbps (Altera decoder)
296.3
Page 6
Applications of Reed-Solomon Codes
•
•
•
•
•
Storage: CDs, DVDs, “hard drives”,
Wireless: Cell phones, wireless links
Sateline and Space: TV, Mars rover, …
Digital Television: DVD, MPEG2 layover
High Speed Modems: ADSL, DSL, ..
Good at handling burst errors.
Other codes are better for random errors.
– e.g., Gallager codes, Turbo codes
Page 7
RS and “burst” errors
Let’s compare to Hamming Codes (which are “optimal”).
code bits
check bits
RS (255, 253, 3)256
2040
16
Hamming (211-1, 211-11-1, 3)2
2047
11
They can both correct 1 error, but not 2 random errors.
– The Hamming code does this with fewer check bits
However, RS can fix 8 contiguous bit errors in one byte
– Much better than lower bound for 8 arbitrary errors
  n
 n
log1     L      8 log( n  7)  88 check bits


 8
 1
Page 8
Discrete Fourier Transform (DFT)
Evaluating polynomial at n points via matrix multiply:
a is a primitive nth root of unity (an = 1) – a generator
1
1
1

a2
1 a
4
T  1 a 2
a





n 1
2 ( n 1)
1
a
a

L
L
L

L


n 1
a

a 2( n 1) 



a ( n 1)(n 1) 
1
 c0 
 m0 




  
  
c 
m 
 k 1   T   k 1 
 ck 
 0 










 0 
c 


 n 1 
Evaluate polynomial mk-1xk-1 +  + m1x + m0
at n distinct roots of unity, 1, a, a2, a3, , an-1
DFT: c  Tm
1
Inverse DFT: m  T c
FFT is fast way to compute DFT
Page 9
Galois Field
GF(23) with irreducible polynomial: x3 + x + 1
a = x is a generator
0
0
000
y0
a
x
010
y1
a2
x2
100
y2
a3
x+1
011
y3
a4
x2 + x
110
y4
a5
x2 + x
+1
111
y5
a6
x2 + 1
101
y6
a7
1
001
y7
Will use this as an example.
Page 10
DFT Example
a = x is 7th root of unity in GF(23)/x3 + x + 1
(i.e., multiplicative group, which excludes additive inverse)
Recall a = y1, a2 = y2, …
1

1
1

T  1
1

1

1
1
1
1
1
1
a a2 a3 a4 a5
a2 a4 a6
a3 a6
a4

a5
a6
1  1
 
6
a  1
 1
 
  1
 1
 
 1
 
 1
1
y1
y2
y3
y4
y6
y7
1
y12
y 22
y 32
1
y13
y 32

1
y14
1
y15
1 

6
y1 







y 67 
Should be clear that c = T  (m0,m1,…,mk-1,0,…)T
is the same as evaluating p(x) = m0 + m1x + … + mk-1xk-1
at n points y0, y1, …, yn-1.
Page 11
Decoding
Why is it hard?
 k  2s 
 possibilities and solve for
Brute Force: try 
 ks 
each.
Page 12
Cyclic Codes
A linear code is cyclic if:
(c0, c1, …, cn-1)  C  (cn-1, c0, …, cn-2)  C
Both Hamming and Reed-Solomon codes are cyclic.
Note: we might have to reorder the columns to make
the code “cyclic”.
Motivation: Cyclic codes are easier to decode than
general codes.
Page 13
Linear Code Generator and Parity Check Matrices
View message, codeword as vectors (m0, m1, …,mk-1) and (c0, c1, …,cn-1)
Generator Matrix:
A k x n matrix G such that:
C = {m  G | m  k}
Made from stacking the basis vectors
Parity Check Matrix:
A (n – k) x n matrix H such that:
C = {v  n | H  vT = 0}
Codewords are the nullspace of H
These always exist for linear codes
H  GT = 0
Page 14
RS Generator and Parity Check Polynomials
View message (m0, m1, …,mk-1) as polynomial m0 + m1x + … + mk-1xk-1,
codeword (c0, c1, …,cn-1) as polynomial c0 + c1x + … + cn-1xn-1
Generator Polynomial:
A degree (n-k) polynomial g(x) = g0 + g1x + … + gn-kxn-k
such that g | xn – 1
C = {m  g | m  m0 + m1x + … + mk-1xk-1}
Parity Check Polynomial:
A degree k polynomial h(x) = h0 + h1x + … + hkxk
such that h | xn - 1
C = {v  n [x] | h  v = 0 (mod xn –1)}
These always exist for linear cyclic codes
h  g = xn - 1
Page 15
Poly multiplication via matrix multiplication
If g(x) = g0 + g1x + … + gn-kxn-k
We can put this generator in matrix form (k x n):
k-1
n-k+1
 g0

0
G


0

g1 L g k 1 L
g0 L L L
0

L
g0
g nk
g n  k 1
0
g nk
L
L g n  2 k 1
L
0 

0 

 

L g n k 
0
k-1
Write m = m0 + m1x +…+ mk-1xk-1 as (m0, m1, …, mk-1)
Then c = mG
Page 16
g generates cyclic codes
0
0
0   g 
 g 0 g1 L g k 1 L g n k

 

g nk
0   xg 
 0 g 0 L L L g nk
G 




L


 

  k 1 
0 0 L g
 x g
L
g
L
L
g
0
n  2 k 1
nk 



Codes consist of all linear combinations of the rows:
c = (c0,c1,…,cn-1) = m0g + m1xg + m2x2g+ … + mk-1xk-1g
Claim: c’ = (cn-1,c0,c1,…,cn-2) is also a codeword.
Right shift of every row but last is next row.
Right shift of last row is xkg mod (xn –1) = gn-k,0,…,g0,g1,…,gn-k-1
Will show xkg mod (xn –1) is a linear combination of other rows.
Consider h = h0 + h1x + … + hkxk (gh = xn –1)
h0g + (h1x)g + … + (hk-1xk-1)g + (hkxk)g = xn – 1
(hkxk)g = (xn – 1) – (h0g + (h1x)g + … + (hk-1xk-1)g )
(hkxk)g mod (xn –1) = -(h0g + h1(xg) + … + hk-1(xk-1g))
xkg mod (xn –1) = -hk-1(h0g + h1(xg) + … + hk-1(xk-1g))
Therefore right cyclic shift of every row is a linear
Page 17
combination of other rows.
Viewing h as a matrix
If h = h0 + h1x + … + hkxk
we can put this parity check poly. in matrix form ((n-k) x n) :
k+1
n-k-1
0

0
H 


h
 k
h0 

0



0 L 0 
L 0
L hk
hk L h1
hk 1 L h0

L h1
h0
n-k-1
HcT = 0 (syndrome gives coefficients of xn-1 through xk in h  c,
which are the same as in h  c mod xn-1)
Page 18
Hamming Codes Revisited
The Hamming (7,4,3)2 code.
g = 1 + x + x3
1

0
G 
0

0

1 0 1 0 0 0

1 1 0 1 0 0
0 1 1 0 1 0

0 0 1 1 0 1 
h = x4 + x2 + x + 1
 0 0 1 0 1 1 1


H   0 1 0 1 1 1 0
 1 0 1 1 1 0 0


gh = x7 – 1, GHT = 0
The columns are not identical to the previous
example Hamming code. (And note that coefficients
are from GF(2), but n > 2.)
Page 19
Factors of xn -1
Intentionally left blank
Page 20
Another way to write g
Let a be a generator of GF(pr).
Let n = pr - 1 (the size of the multiplicative group)
Then we can write a generator polynomial as
g(x) = (x-a)(x-a2) … (x - an-k), h = (x- an-k+1)…(x-an)
Lemma: g | xn – 1, h | xn – 1, gh = xn – 1
(a | b means a divides b)
Proof:
– an = 1 (because of the size of the group)
 an – 1 = 0
 a root of xn – 1
 (x - a) | xn -1
– similarly for a2, a3, …, an
– therefore xn - 1 is divisible by (x - a)(x - a2) …
Page 21
Back to Reed-Solomon
Consider a generator polynomial g  GF(pr)[x], s.t. g | (xn – 1)
Recall that n – k = 2s (the degree of g is n-k, n-k+1 coefficients)
Encode (trick to make code systematic):
– m’ = m x2s
(basically shift by 2s)
– b = m’ (mod g), m’ = qg + b for some q
– c = m’ – b = (mk-1, …, m0, -b2s-1, …, -b0)
– Note that c is a cyclic code based on g
- c = m’ – b = qg
(i.e., given m we found another message q such that qg
is “systematic” for m)
Parity check:
- hc=0?
Page 22
Example
Lets consider the (7,3,5)8 Reed-Solomon code.
We use GF(23)/x3 + x + 1
a
x
010
a2
x2
100
a3
x+1
011
a4
x2 + x
110
a5
x2 + x + 1
111
a6
x2 + 1
101
a7
1
001
Page 23
Example RS (7,3,5)8
n = 7, k = 3, n-k = 2s = 4, d = 2s+1 = 5
g = (x - a)(x - a2)(x - a3)(x - a4)
a 010
4
3
3
2
3
= x + a x + x + ax + a
a2 100
h = (x - a5)(x - a6)(x - a7)
a3 011
= x3 + a3x2 + a2x + a4
a4 110
gh = x7 - 1
a5 111
Consider the message: 110 000 110
a6 101
m = (a4, 0, a4) = a4x2 + a4
a7 001
m’ = x4m = a4x6 + a4x4
= (a4 x2 + x + a3)g + (a3x3 + a6x + a6)
c = (a4, 0, a4, a3, 0, a6, a6)
ch = 0 (mod x7 –1)
= 110 000 110 011 000 101 101
Page 24
A useful theorem
Theorem: For any b, if g(b) = 0 then b2sm(b) = b(b)
Proof:
x2sm(x) = m’(x) = g(x)q(x) + b(x)
b2sm(b) = g(b)q(b) + b(b) = b(b)
Corollary: b2sm(b) = b(b) for b  {a, a2, a3,…, a2s=n-k}
Proof:
{a, a2, …, a2s} are the roots of g by definition.
Page 25
Fixing erasures
Theorem: Any k symbols from c can reconstruct c
and hence m
Proof:
We can write 2s equations involving m (cn-1, …, c2s)
and b (c2s-1, …, c0). These are
a2s m(a) = b(a)
a4s m(a2) = b(a2)
…
a2s(2s) m(a2s) = b(a2s)
We have at most 2s unknowns (erasures), so we can
solve for them. (I’m skipping showing that the
equations are linearly independent).
Page 26
Efficient Decoding
I don’t plan to go into the Reed-Solomon decoding
algorithm, other than to mention the steps.
c
Syndrome
Calculator
Error
Polynomial
Error
Locations
Error
Magnitudes
Berlekamp
Massy
Chien
Search
Forney
Algorithm
Error
Corrector
This is the hard part. CD players
use this algorithm.
(Can also use Euclid’s algorithm.)
Page 27
m