Error Correcting Codes: Combinatorics, Algorithms and Applications
(Fall 2007)
Lecture 15: Gilbert-Varshamov Bound
October 2, 2007
Lecturer: Atri Rudra
Scribe: Thanh-Nhan Nguyen
In the previous lectures, we have only seen upper bounds on the rate of a code (given a fixed
relative distance). In this lecture, we study our first (and only) lower bound on rate of a code.
1
Gilbert-Varshamov Bound
In today’s lecture we will prove the following result
Theorem 1.1. Let q ≥ 2. For every 0 ≤ δ < 1 − 1q , and 0 < ε ≤ 1 − Hq (δ), there exists a code
with rate R ≥ 1 − Hq (δ) − ε, and relative distance δ.
The above result was proved for general codes by Gilbert (Section 1.1) and for linear codes by
Varshamov (Section 1.2). Hence, the bound is called the Gilbert-Varshamov bound.
1.1
Gilbert Construction
Gilbert proved Theorem 1.1 by the following greedy construction (where d = δn)
(i) Start with the empty code: C ← ∅
(ii) If there exists a v ∈ [q]n such that ∆(v, c) ≥ d for every c ∈ C, add v to C.
We claim that the above algorithm terminates and C has distance d. The latter is true by step (ii).
For the former claim, note that, if we cannot add v at some point, we cannot add it later. The
running time of this algorithm is q O(n) (as step (ii) in the worst-case could be repeated for every
vector in [q]n , that is at most q n times). Further, we claim that after termination
[
B(c, d − 1) ⊇ [q]n ,
c∈C
because if not, then we can pick another another codeword. Therefore,
[
|
B(c, d − 1)| ≥ q n .
c∈C
It is obvious that
X
|B(c, d − 1)| ≥ |
c∈C
[
c∈C
1
B(c, d − 1)|
(1)
which by (1) implies that
X
V olq (c, d − 1) ≥ q n
c∈C
or
X
V olq (0, d − 1) ≥ q n .
c∈C
Thus, we have
qn
V olq (0, d − 1)
qn
≥ nHq (δ)
q
= q n(1−Hq (δ)) ,
|C| ≥
(2)
as desired. In the above, (2) follows from the fact that
V olq (0, d − 1) ≤ V olq (0, δn)
≤ q nHq (δ) ,
(3)
where the second inequality follows from the upper bound on the volume of a Hamming ball that
we proved in an earlier lecture.
It is worth noting that the code from Gilbert’s construction is not guaranteed to have any special
structure. In particular, even storing the code can take exponential space. We have seen that linear
codes have a much more succinct representation. Thus, a natural question is if one can show
that linear codes achieve the the r ≥ 1 − Hq (δ) tradeoff that the Gilbert construction achieves.
Varshamov showed that indeed this is the case and we look at his construction next.
1.2
Varshamov Construction
Now we turn to the result due to Varshamov who showed that a random linear code, with high
probability, lies on the GV bound.
The Varshamov construction is yet another use of the probabilistic method. For a recap of
the general idea behind the probabilistic method and other probability basics (such as the union
bound), please refer to Lecture 4 from Spring 09.
Pick a random linear code by picking a random k × n matrix G where each of kn entries is
chosen uniformly and independently at random from Fq . We are done if we can show that
For every m ∈ Fkq \ {0}, wt(mG) ≥ d.
Fix m ∈ Fkq \ {0}. Note that for a random G, mG is a uniformly random vector from Fnq . Thus,
we have
V olq (0, d − 1)
P r[wt(mG) < d] =
qn
q nHq (δ)
≤
(4)
qn
2
where (4) follows from (3). Thus, by the union bound
P r[∃m, wt(mG) < d] ≤ q k q −n(1−Hq (δ))
= q −ε·n ,
where the equality follows by choosing k = (1−Hq (δ)−ε)n. Since q −εn 1, by the probabilistic
method, there exists a linear code C with relative distance δ and rate at least 1 − Hq (δ) − ε. In fact,
the proof shows that most linear codes (with appropriate parameters) meet the Gilbert-Varshamov
bound.
3