Information and Coding
Theory
Linear Block Codes.
Basic definitions and some examples.
Some bounds on code parameters. Hemming
and Golay codes.
Syndrome decoding.
Juris Viksna, 2016
Transmission over noisy channel
Transmission over noisy channel
Noisy channel
In practice channels are always noisy (sometimes this could be ignored).
There are several types of noisy channels one can consider.
We will restrict attention to binary symmetric channels.
Shannon Channel Coding Theorem
Shannon Channel Coding Theorem
Codes – how to define them?
In most cases it would be natural to use binary block codes that maps
input vectors of length k to output vectors of length n.
For example: 101101
110011
etc.


11100111
01000101
Thus we can define code as an injective mapping from vector space V
with dimension k to vector space W with dimension n.
V
W
Such definition essentially is used in the original Shanon’s theorem.
Codes – how to define them?
We can define code as an injective mapping from “vector space” V with
dimension k to “vector space” W with dimension n.
Arbitrary mappings between vector spaces are hard either to explicitly
define or to use (encode ort decode) in practice (there are almost 2n2k of
them – already around 1000000000000000000000000000000 for k=4
and n=7).
Simpler to define and use are linear codes that can be defined by
multiplication with matrix of size kn (called generator matrix).
0110

0110011
Shanon’s results hold also for linear codes.
Codes – how to define them?
Simpler to define and use are linear codes that can be defined by
multiplication with matrix of size kn (called generator matrix).
What should be the elements of
vector spaces V and W?
In principle in most cases it will be sufficient to have just 0-s and 1-s,
however, to define vector space in principle we need a field – an
algebraic system with operations “+” and “” defined and having similar
properties as we have in ordinary arithmetic (think of real numbers).
Field with just “0” and “1” may look very simple, but it turns out that to
get some real progress we will need more complicated fields, just that
elements of these fields themselves will be regarded as (most often)
binary vectors.
What are good codes?
Linear codes can be defined by their generator matrix of size kn .
Shanon’s theorem tells us that for a transmission channel with a bit error
probability p and for an arbitrary small bit error probability pb we wish to
achieve there exists codes with rates R = k/n that allows us to achieve pb
as long as R<C(p).
In general, however, the error rate could be different for different
codewords, pb being an “average” value.
We however will consider codes that are guaranteed to correct up to t
errors for any of codewords.
What are good codes?
We however will consider codes that are guaranteed to correct up to t
errors for any of codewords – this is equivalent with minimum distance
between codewords being d and t = (d1)/2.
Such codes will then be characterized by 3 parameters and will be
referred to as (n,k,d) codes.
For a given k we are thus interested:
- to minimize n
- to maximize d
In most cases for fixed values n and k the larger values of d will give us
lower bit error probability pb, although the computation of pb is not that
straightforward and depends from a particular code.
Note that one can completely “spoil” d value of good code with low pb
by including in it a vector with weight 1 
Vector spaces - definition
What we usually understand by vectors?
In principle we can say that vectors are n-tuples of the form:
(x1,x2,,xn)
and operations of vector addition and multiplication by scalar are defined
and have the following properties:
(x1,x2,,xn)+(y1,y2,,yn)=(x+y1,x+y2,,x+yn)
a(x1,x2,,xn)=(ax1,ax2,,axn)
The requirements actually are a bit stronger – elements a and xi should
come from some field F.
We might be able to live with such a definition, but then we will link a
vector space to a unique and fixed basis and often this will be technically
very inconvenient.
Vector spaces - definition
Definition
4-tuple (V,F,+,) is a vector space if (V,+) is a commutative group with
identity element 0 and for all u,vV and all a,bF:
1)
2)
3)
4)
a(u+v)=au+av
(a+b)v=av+bv
a(bv)=(ab)v
1v=v
Usually we will represent vectors as n-tuples of the form (x1,x2,,xn),
however such representations will not be unique and will depend from a
particular basis of vector space, which we will chose to use (but 0 will
always be represented as n-tuple of zeroes (0,0,,0)).
Groups - definition
Consider set G and binary operator +.
Definition
Pair (G,+) is a group, if there is eG such that for all a,b,cG:
1) a+bG
2) (a+b)+c = a+(b+c)
3) a+e = a and e+a = a
4) there exists inv(a) such that a+ inv(a)= e and inv(a)+a = e
5) if additionally a+b = b+a, group is commutative (Abelian)
If group operation is denoted by “+” then e is usually denoted by 0 and
inv(a) by a.
If group operation is denoted by “” hen e is usually denoted by 1 and
inv(a) by a1 (and ab are usually written as ab).
It is easy to show that e and inv(a) are unique.
Vector spaces – dot (scalar) product
Let V be a k-dimensional vector space over field F. Let b1,,bkV be
some basis of V. For a pair of vectors u,vV, such that u=a1b1+...+akbk
and v=c1b1+...+ckbk their dot (scalar) product is defined by:
u·v = a1·c1 +...+ ak·ck
Thus operator “” maps VV to F.
Lemma
For u,v,wV and all a,bF the following properties hold:
1) u·v = v·u.
2) (au+bv)·w = a(u·v)+b(v·w).
3) If u·v = 0 for all v in V, then u = 0.
Vector spaces – dot (scalar) product
Let V be a k-dimensional vector space over field F. Let b1,,bkV be
some basis of V. For a pair of vectors u,vV, such that u=a1b1+...+akbk
and v=c1b1+...+ckbk their dot (scalar) product is defined by:
u·v = a1·c1 +...+ ak·ck
Two vectors u and v are said to be orthogonal if u·v = 0. If C is a
subspace of V then it is easy to see that the set of all vectors in V that are
orthogonal to each vector in C is a subspace, which is called the space
orthogonal to C and denoted by C.
Linear block codes
Message
source
Encoder
Channel
x = x1,...,xk
message
c = c1,...,cn
codeword
y=c+e
received
vector
Decoder
Receiver
x'
estimate of
message
e = e1,...,en
error from
noise
Generally we will define linear codes as vector spaces – by taking C to
be a k-dimensional subspace of some n-dimensional space V.
Linear block codes
Let V be an n-dimensional vector space over a finite field F.
Definition
A code is any subset CV.
Definition
A linear (n,k) code is any k-dimensional subspace C⊑V.
Linear block codes
Let V be an n-dimensional vector space over a finite field F.
Definition
A linear (n,k) code is any k-dimensional subspace C⊑V.
Example (choices of bases for V and code C):
Basis of V (fixed):
Set of V elements:
Set of C elements:
2 alternative bases
for code C:
001,010,100
{000,001,010,011,100,101,110,111}
{000,001,010,011}
001,010
001,011
Essentially, we will be ready to consider alternative bases, but will stick
to “main one” for representation of V elements.
Linear block codes
Definition
A linear (n,k) code is any k-dimensional subspace C⊑V.
Definition
The weight wt(v) of a vector vV is a number of nonzero components of
v in its representation as a linear combination v = a1b1+...+anbn.
Definition
The distance d(v,w) between vectors v,wV is a number of distinct
components of these vectors.
Definition
The minimum weight of code C⊑V is defined as minvC,v0 wt(v).
A linear (n,k) code with minimum weight d is often referred to as (n,k,d)
code.
Linear block codes
Theorem
Linear (n,k,d) code can correct any number of errors not exceeding
t = (d1)/2.
Proof
The distance between any two codewords is at least d.
So, if the number of errors is smaller than d/2 then the closest
codeword to the received vector will be the transmitted one
However a far less obvious problem: how to find which codeword is the
closest to received vector?
Linear codes - the main problem
A good (n,k,d) code has small n, large k and large d.
The main coding theory problem is to optimize one of the parameters n,
k, d for given values of the other two.
Generator matrices
Definition
Consider (n,k) code C⊑V. G is a generator matrix of code C, if
C = {vG | vV} and all rows of G are independent.
It is easy to see that generator matrix exists for any code – take any
matrix G rows of which are vectors v1,,vk (represented as n-tuples in
the initially agreed basis of V) that form a basis of C. By definition G
will be a matrix of size kn.
Obviously there can be many different generator matrices for a given
code. For example, these are two alternative generator matrices for the
same (4,3) code:
 1 0 0 1
0 0 1 1




G1   0 1 0 1 G2   1 1 1 1 
 0 0 1 1
1 0 1 0




Equivalence of codes
Definition
Codes C1,C2⊑V. are equivalent, if a generator matrix G2 of C2 can be
obtained from a generator matrix G1 of C1 by a sequence of the following
operations:
1) permutation of rows
2) multiplication of a row by a non-zero scalar
3) addition of one row to another
4) permutation of columns
5)multiplication of a column by a non-zero scalar (not needed for binary)
Note that operations 1-3 actually doesn’t change the code C1. Applying
operations 4 and 5 C1 could be changed to a different subspace of V,
however the weight distribution of code vectors remains the same.
In particular, if C1 is (n,k,d) code so is C2. In binary case vectors of C1
and C2 would differ only by permutation of positions.
Generator matrices
Definition
A generator matrix G of (n,k) code C⊑V is said to be in standard form
if G = (I,A), where I is kk identity matrix.
Theorem
For code C⊑V there is an equivalent code C that has a generator matrix
in standard form.
Hamming code [7,4]
No errors - all pi-s correspond to di-s
Error in d1,...,d3 - a pair of wrong pi-s
Error in d4 - all pairs of pi-s are wrong
Error in pi - this will differ from error in
some of di-s
Parity bits of H(7,4)
So:
- we can correct any single error
- since this is unambiguous, we should be able to detect any 2 errors
Hamming code [7,4]
G - generator matrix
A (4 bit) message x is encoded as xG, i.e.
if x = 0110 then c = xG = 0110011.
Decoding?
- there are 16 codewords, if there are no errors, we can just find
the right one...
- also we can note that the first 4 digits of c is the same as x :)
Hamming code [7,4]
a = 0001111, b = 0110011 and c = 1010101
H - parity check matrix
Why it does work? We can check that without errors yH = 000
and that with 1 error yH gives the index of damaged bit...
General case: there always exists matrix for checking orthogonality
yH = 0. Finding of damaged bits however isn’t that simple.
Hamming codes
For simplicity we will consider codes over binary fields, although the
definition (and design idea) easily extends to codes over arbitrary finite
fields.
Definition
For a given positive integer r a Hemming code Ham(r) is code a parity
check of which as its rows contains all possible non-zero r-dimensional
binary vectors.
There are 2r  1 such vectors, thus parity check matrix has size 2r  1r
and respectively Ham(r) is (n = 2r  1,n  r) code.
Hamming codes
Definition
For a given positive integer r a Hemming code Ham(r) is code a parity
check of which as its rows contains all possible non-zero r-dimensional
binary vectors.
Example of Hamming code Ham(4):
0

0
H 
0

1

0 0 0 0 0 0 1 1 1 1 1 1 1 1

0 0 1 1 1 1 0 0 0 0 1 1 1 1
1 1 0 0 1 1 0 0 1 1 0 0 1 1

0 1 0 1 0 1 0 1 0 1 0 1 0 1
Also not required by definition, note that in this particular case columns
can be regarded as consecutive integers
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 written in binary form.
Dual codes
Definition
Consider code C⊑V. A dual or orthogonal code of C is defined as
C = {vV |  wC: vw = 0}.
It is easy check that C  ⊑V, i.e. C is a code. Note that actually this is
just a re-statement of definition of orthogonal vector spaces we have
already seen.
There are codes that are self-dual, i.e. C = C.
Dual codes - some examples
For the (n,1) -repetition code C, with the generator matrix
G = (1 1 … 1)
the dual code C is (n, n1) code with the generator matrix
G, described by:
 1 1 0 0 ... 0 


 1 0 1 0 ... 0 

G 

..


 1 0 0 0 ... 1 


Dual codes - some examples
[Adapted from V.Pless]
Dual codes - some examples
[Adapted from V.Pless]
Dual codes – parity checking matrices
Definition
Let code C⊑V and let C be its dual code. A generator matrix H of C is
called a parity checking matrix of C.
Theorem
If kn generator matrix of code C⊑V is in standard form if G = (I,A)
then (kn)n matrix H = (AT,I) is a parity checking matrix of C.
Proof
It is easy to check that any row of G is orthogonal to any row of H (each
dot product is a sum of only two non-zero scalars with opposite signs).
Since dim C + dim C = dim V, i.e. k + dim C = n we have to
conclude that H is a generator matrix of C.
Note that in binary vector spaces H = (AT,I) = (AT,I).
Dual codes – parity checking matrices
Theorem
If kn generator matrix of code C⊑V is in standard form if G = (I,A)
then (kn)n matrix H = (AT,I) is a parity checking matrix of C.
So, up to the equivalence of codes we have an easy way to obtain a
parity check matrix H from a generator matrix G in standard form and
vice versa.
Example of generator and parity check matrices in standard form:
1 0 0 1 1 0 0


G   0 1 0 0 1 1 0
0 0 1 0 0 1 1


1

1
H 
0

0

0 0 1 0 0 0

1 0 0 1 0 0
1 1 0 0 1 0

0 1 0 0 0 1 
Vector spaces and linear transformations
Definition
Let V be a vector space over field F. Function f : VV is called a linear
transformation, if for all u,vV and all aF the following hold:
1)af(u) = f(au).
2)f(u)+f(v) = f(u+v).
The kernel of f is defined as ker f ={vV | f(v) = 0}.
The range of f is defined as range f ={f(v) | vV}.
It is quite obvious property of f linearity that vector sums and scalar
products doesn't leave ker f or range f .
Thus ker f ⊑V and range f ⊑V.
Dimensions of orthogonal vector spaces
Theorem
If C is a subspace of V, then dim C + dim C = dim V.
C
C
0
Proof?
We could try to reduce this to “similarly looking” equality dim V = dim
(ker f) + dim (range f).
However how we can define a linear transformation from dot product?
Dimensions of orthogonal vector spaces
Theorem
If C is a subspace of V, then dim C + dim C = dim V.
Proof
However how we can define a linear transformation from dot product?
Let u1,,uk be some basis of C. We define transformation f as follows:
for all vV: f(v) = (vu1) u1 +  + (vuk) uk
Note that vui F, thus f(v)C. Therefore we have:
• ker f = C (this directly follows form definition of C)
• range f = C (this follows form definition of f)
Thus from rank-nullity theorem: dim C + dim C = dim V.
Dual codes and vector syndromes
Definition
Let C⊑V be an (n,k) code with a parity check matrix H. For each vV a
syndrome of v is nk dimensional vector syn(v) = vHT.
By definition of H we have syn(c) = 0 for all codewords cC. If some
errors have occurred, then instead of c we have received vector y = c + e,
where c is codeword and e is error vector. In this case
syn(c) = syn(c) + syn(e) = 0 + syn(e).
That is, syndrome is determined solely by error vector and in principle
the knowing of vector syndrome should us allow to infer which bits have
been transmitted incorrectly.
Hamming code (7,4)
We have already seen a Hemming (7,4) code
Parity bits of H(7,4)
with generator matrix (in standard form) shown
above and have proved that it can correct any single error
(or even more – that it is a perfect (7,4,3) code.
Can we generalize this to codes of different lengths/dimensions?
How simple the decoding procedure for such codes might be?
Hamming codes
For simplicity we will consider codes over binary fields, although the
definition (and design idea) easily extends to codes over arbitrary finite
fields.
Definition
For a given positive integer r a Hemming code Ham(r) is code a parity
check of which as its rows contains all possible non-zero r-dimensional
binary vectors.
There are 2r  1 such vectors, thus parity check matrix has size 2r  1r
and respectively Ham(r) is (n = 2r  1,n  r) code.
Hamming codes
Definition
For a given positive integer r a Hemming code Ham(r) is code a parity
check of which as its rows contains all possible non-zero r-dimensional
binary vectors.
Example of Hamming code Ham(4):
0

0
H 
0

1

0 0 0 0 0 0 1 1 1 1 1 1 1 1

0 0 1 1 1 1 0 0 0 0 1 1 1 1
1 1 0 0 1 1 0 0 1 1 0 0 1 1

0 1 0 1 0 1 0 1 0 1 0 1 0 1
Also not required by definition, note that in this particular case columns
can be regarded as consecutive integers
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 written in binary form.
Hamming codes
Hamming code Ham(4):
0

0
H 
0

1

0 0 0 0 0 0 1 1 1 1 1 1 1 1

0 0 1 1 1 1 0 0 0 0 1 1 1 1
1 1 0 0 1 1 0 0 1 1 0 0 1 1

0 1 0 1 0 1 0 1 0 1 0 1 0 1
Why we have defined this code in such particular way?
1) Note that if there have been no errors we have syn(y) = 0 for some
received vector y.
2) If a single error have occurred we will have syn(y) = [i], where is
binary representation of position (column) in which this error has
occurred.
Hamming codes
1) Note that if there have been no errors we have syn(y) = 0 for some
received vector y.
2) If a single error have occurred we will have syn(y) = [i], where is
binary representation of position (column) in which this error has
occurred.
Thus we are able to correct any single error. Also note that any larger
number of errors is not distinguishable from 0 or 1 error case. Thus all
(binary) Hamming codes have parameters (n = 2r  1,n  r,3).
A ternary (13,10,3) Hamming code
[Adapted from V.Pless]
Golay codes - some history
The brief history of the Golay Codes begins in 1949, when M. J. E.
Golay published his “Notes on Digital Coding” in the Proceedings of the
Institute of Electrical and Electronic Engineers”, ½ page in length. It
described the (23,12,7)2 code (although he evidently did not name it after
himself). This inspired a search for more perfect codes. After all, if there
was some series of perfect codes, or better yet an algorithm that produces
them, much of the rest of coding theory would possibly become obsolete.
For any given rate and blocklength, no code with a higher minimum
distance or average minimum distance can be constructed, so if it had
been determined that perfect codes existed with many rates and many
blocklengths, it may have been worthwhile to only search for perfect
codes. It soon appeared that such prayers fell on deaf ears, as the
existence of perfect codes was disproved in more and more general
scenarios. Finally, in 1973, when Aimo Tietäväinen disproved the
existence of perfect codes over finite fields in his “Nonexistence of
Perfect Codes over Finite Fields” in the SIAM Journal of Applied
Mathematics, January 1973.
[Adapted from www.wikipedia.org]
Golay codes
In mathematical terms, the extended binary Golay code consists of a 12dimensional subspace W of the space V=F224 of 24-bit words such that
any two distinct elements of W differ in at least eight coordinates.
Equivalently, any non-zero element of W has at least eight non-zero
coordinates.
The possible sets of non-zero coordinates as w ranges over W are called
code words. In the extended binary Golay code, all code words have
Hamming weight 0, 8, 12, 16, or 24.
Up to relabeling of coordinates W is unique.
[Adapted from www.wikipedia.org]
Golay codes
Golay codes G24 and G23 were used by Voyager I and Voyager II to
transmit color pictures of Jupiter and Saturn. Generation matrix for G24
has the form:
1

0
0

0
0

0
G 
0
0

0
0

0
0

0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 1 0 0 0 1 0

1 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 1 0 0 0 1
0 1 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 1 1 1 0 0 0

0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 1 1 0 1 1 1 0 0
0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 1 1 0 1 1 1 0 
0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 1 1 0 1 1 1

0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 1 0 1 1 0 1 1
0 0 0 0 0 0 1 0 0 0 0 1 1 1 0 0 0 1 0 1 1 0 1

0 0 0 0 0 0 0 1 0 0 0 1 1 1 1 0 0 0 1 0 1 1 0
0 0 0 0 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 1 0 1 1

0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 1 0 0 0 1 0 1
0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 0 0 0 1 0 
G24 is (24,12,8) –code and the weights of all codewords are multiples
of 4. G23 is obtained from G24 by deleting last symbols of each codeword
of G24. G23 is (23,12,7) –code.
Golay codes
Matrix G for Golay code G24 has actually a simple and regular
construction.
• The first 12 columns are formed by a unitary matrix I12, next column
has all 1’s.
• Rows of the last 11 columns are cyclic permutations of the first row
which has 1 at those positions that are squares modulo 11, that is
0, 1, 3, 4, 5, 9.
Ternary Golay code
The ternary Golay code consists of 36 = 729 codewords. Its parity check
matrix is:
Any two different codewords differ in at least 5 positions. Every ternary
word of length 11 has a Hamming distance of at most 2 from exactly one
codeword. The code can also be constructed as the quadratic residue
code of length 11 over the finite field F3.
This is a (11,6,5) code.
[Adapted from www.wikipedia.org]
Punctured codes
There are certain modifications that can be applied to codes and
sometimes allows to obtain codes with better parameters. The usual
codes that can be obtained in such a way are punctured, extended and
shortened codes.
Definition
A punctured code of a binary (n,k,d) code C with a generator matrix G
is a code C with generator matrix G obtained from G by deleting any
one of G columns.
By deleting different columns we may obtain different punctured codes.
Punctured codes may have parameters (n1,k,d), (n1,k,d1),
(n1,k1,d) or (n1,k1,d1).
Punctured codes - example
[Adapted from V.Pless]
Punctured codes - example
[Adapted from V.Pless]
Extended codes
Definition
An extended code of a binary (n,k,d) code C with a generator matrix G
is a code C with generator matrix G obtained from G by adding n+1-st
column (a1,,ak), where each ai is the sum of all elements of i-th row of
G.
Thus for any code C there is just one extended code C.
In the case when C contains vectors with odd weight code C has
parameters (n+1,k,d) or (n+1,k,d+1).
If all vectors of C has even weight code C has parameters (n+1,k,d).
Extended codes - example
[Adapted from V.Pless]
Shortened codes
Definition
A shortened code of a binary (n,k,d) code C is a code C obtained from
C by selecting all vectors with 0 in some fixed position i and removing
from these vectors elements in i-th position.
By choosing different positions i we may obtain different shortened
codes.
Shortened codes has parameters (n1,k1,d), where d d.
Shortened codes - example
[Adapted from V.Pless]
Shortened codes - example
[Adapted from V.Pless]
Hemming inequality and perfect codes
We will formulate these results in binary case, although they easily
extends to codes over any finite field F.
Theorem (Hemming inequality)
For any (n,k,d) code C from vector space V over the binary field F the
following inequality holds:
n
2     2 n
i 0  i 
t
k
where t = (d1)/2.
Hemming inequality and perfect codes
Definition
An (n,k,d) code C⊑V is perfect if balls with radius t = (d1)/2 around
vectors of C cover the whole vector space V.
Proposition
A binary (n,k,d) code C⊑V is perfect if and only if the following
equality holds:
 n
n
2     2
i 0  i 
t
k
It is easy to see that all “full-space” (n, n,1) codes and all binary
repetition (2k+1,1,2k+1) codes are perfect. These are trivial perfect
codes.
Hemming inequality and perfect codes
[Adapted from J.MacCay]
Hemming inequality and perfect codes
Do good perfect codes exist?
Theorem
The only non-trivial perfect codes are the following:
•all Hamming codes
•binary (23,12,7) Golay code
•ternary (12,6,5) Golay code.
We will not prove this result here. However it is very simple to show that
these particular codes are perfect, but the proof that these are the only
ones is difficult.
Singleton bound and MDS codes
Theorem (Singleton inequality) Named after Richard Collom Singleton
For any (n,k,d) code C the inequality n  k  d  1 holds.
Proof (version 1)
All codewords in C are distinct. If we delete first d1 components, they
are still distinct. New code has length nd+1 and still has size qk, thus
n  d +1  k.
Singleton bound and MDS codes
Theorem (Singleton inequality)
For any (n,k,d) code C the inequality n  k  d  1 holds.
Proof (version 2)
Observe that rank H = nk.
Any dependence of s columns in parity check matrix yields a codeword
of weight s (any set of non-zero components in codeword  dependence
relation in H).
Thus n  k  d  1.
Singleton bound and MDS codes
Definition
An (n,k,d) code is called maximum distance separable (or MDS code)
if n  k = d  1.
There exist only trivial binary MDS codes. However for codes over other
finite fields this is not necessarily so.
MDS conjecture:
[Adapted from G.Seroussi]
Some positive results
For simplicity we again will state just the binary case.
Theorem (Gilbert-Varshamow bound)
In n dimensional vector space V over the binary field F there exists a
code C with parameters (n,k,d) code for at least one value d  k = n  m,
if the following inequality holds:
 n  1
m

  2  1

i 1  i 
d 2
Some positive results
Theorem (Gilbert-Varshamow bound)
In n dimensional vector space V over the binary field F there exists a
code C with parameters (n,k,d) code for at least one value d  k = n  m,
if the following inequality holds:
 n  1
m

  2  1

i 1  i 
d 2
Proof
We start with d and m and attempt to construct parity check matrix, so
that no d  1 columns of it is dependent (implying that minimal weight
will be at least d). The number of columns will be n.
We start with an arbitrary column and continue to add other columns
whilst number of linear combinations from d  2 of them doesn’t
exceed number of all m-tuples.
Sizes of some known codes
[Adapted from V.Pless]
Cosets
Definition
Let C⊑V be an (n,k) code. For each aV a set a+C ={v+c | cC} is
called a coset.
Note that this is the same definition we used in proof of Lagrange
theorem, if we consider C as an additive group.
So, by the same simple arguments we can show that:
1) all cosets have the same number of elements (equal to |C|);
2) two cosets are either completely disjoint or equal;
3) every vector belongs to some coset.
Also note, that if error vector has been e, then received message
y = c + e is in coset e+C.
Cosets
Definition
Let C⊑V be an (n,k) code. For each aV a set a+C ={v+c| cC} is
called a coset.
Note, that if error vector has been e, then received message
y = c + e is in coset e+C.
Thus for all y  e+C we have syn(y) = syn(e).
How do we decode y? It seems reasonable to assume that the number of
errors has been minimal, so we could chose error e as a vector from e+C
with minimal weight (such vector can be non-unique).
A vector x  a+C with a minimal weight is called coset leader.
Syndrome decoding
Syndrome decoding:
1) precompute array of syndromes and their coset leaders,
2) if vector y is received, compute syn(y), locate its coset and coset
leader e,
3) decode y as x = y  e.
Syndrome decoding - example
[Adapted from V.Pless]
Syndrome decoding
Complexity??
“Brute force” method (precompute all codewords (2k of them) and for
the received vector find the closest codeword):
Time: ~ 2k n
Memory: ~ 2k n
Syndrome decoding:
Time (precomputing): ~ 2n (k+n)
Time (decoding):
~ (nk) k
Memory: ~ 2nk (k+n)
Depends... but for a range of parameters syndrome decoding has
advantages.