Applied Probability Trust (1 June 2014)
CAPACITY AND ERROR EXPONENTS
OF STATIONARY POINT PROCESSES
UNDER RANDOM ADDITIVE DISPLACEMENTS
VENKAT ANANTHARAM,∗ University of California Berkeley
FRANÇOIS BACCELLI,∗∗ University of Texas at Austin
∗
Postal address:
Department of Electrical Engineering and Computer Sciences, University of
California, Berkeley, California 94720, U.S.A., ananth@eecs.berkeley.edu
∗∗ Postal address: The University of Texas at Austin, Department of Mathematics, Austin, Texas
78712-1202, U.S.A., baccelli@math.utexas.edu
1
2
Venkat Anantharam and François Baccelli
Abstract
Consider a real-valued discrete-time stationary and ergodic stochastic process,
called the noise process. For each dimension n, one can choose a stationary
point process in Rn and a translation invariant tessellation of Rn . Each point
is randomly displaced, with a displacement vector being a section of length n
of the noise process, independent from point to point. The aim is to find a
point process and a tessellation minimizing the probability of decoding error,
defined as the probability that the displaced version of the typical point does
not belong to the cell of this point. The present paper considers the Shannon
regime, in which the dimension n tends to infinity while the logarithm of the
intensity of the point processes, normalized by dimension, tends to a constant.
It is first shown that this problem exhibits a sharp threshold: if the sum of
the asymptotic normalized logarithmic intensity and of the differential entropy
rate of the noise process is positive, then the probability of error tends to 1
with n for all point processes and all tessellations. If it is negative, then there
exist point processes, and tessellations for which this probability tends to 0.
The error exponent function, which gives how quickly the probability of error
goes to zero in n is then derived using large deviations theory. If the entropy
spectrum of the noise satisfies a large deviations principle, then, below the
threshold, the error probability goes exponentially fast to 0 with an exponent
that is given in closed form in terms of the rate function of the noise entropy
spectrum. This is done for two classes of point processes: the Poisson process
and a Matérn hard core point process. New bounds on error exponents are
derived from this for Shannon’s additive noise channel in the high signal-tonoise ratio limit that hold for all stationary and ergodic noises with the above
properties and that match the best known bounds in the white Gaussian noise
case.
Keywords: Point process; random tessellations; high dimensional stochastic
geometry; information theory; entropy spectrum; large deviations theory.
2000 Mathematics Subject Classification: Primary 60G55;94A15
Secondary 60D05;60F10
Capacity and Error Exponents of Stationary Point Processes
3
1. Introduction
To study communication over an additive noise channel information theorists consider transmission via codebooks and decoding from the noise-corrupted reception. For
the purposes of this paper, think of a codeword as a sequence of real numbers (called
symbols) of a fixed length (called the block length). A codebook is a set of codewords.
The allowed codewords in the codebook are generally subject to constraints such as
power or magnitude constraints or more complicated constraints such as run length
constraints (which are constraints on the allowed patterns of symbols). This paper is
focused on just the power constraint, which is the most important case. The transmitter
chooses a codeword to transmit its data over the noisy communication medium. For
instance, if there are 2k codewords in the codebook the transmitter can convey k
bits by the choice of the codeword if the communication medium is noise-free. Only
additive noise channels are considered; this is the case where the receiver sees the
sum of the transmitted codeword and a noise vector. The receiver does not know
the transmitted codeword. The aim is to design the codebook so that the receiver’s
probability of error is small, assuming that the transmitter was a priori equally likely
to have transmitted any one of the codewords. One of the main preoccupations of
the subject of information theory, initiated by Shannon [18], is to study how to design
codes for various communication channels in the asymptotic limit as the block length
goes to infinity. While situations involving multiple transmitters and receivers are
also of great interest only the single transmitter and receiver case (this is called the
point-to-point case) is considered in this paper.
In the asymptotic analysis, one requires the error probability to be asymptotically
vanishing in the block length. Typically one can do this while having codebooks whose
cardinality grows exponentially with the block length. Communication channels are
thus characterized first of all by their Shannon capacity, which is the largest possible
such exponent. The next question of interest is then how quickly the error probability
can be made to go to zero when using codebooks with a rate (i.e. exponent of the
exponentially growing size of the codebook) that is less than the Shannon capacity.
The best possible exponent, as a function of rates below the Shannon capacity, is called
the error exponent function or reliability function of the channel. Characterizing this
4
Venkat Anantharam and François Baccelli
is largely an open problem and is one of the most challenging mathematical problems
in information theory. There are two major classes of lower bounds that can be proved
for the error exponent. One is the random coding bound, which comes, in the powerconstrained case, by considering codewords drawn uniformly at random from the sphere
of points that satisfy the power constraint [19]. The second is the expurgated bound,
which comes from refining this random coding ensemble by specifically eliminating
codeword pairs that are too close to each other, while only slightly changing the rate
of the codebook, in a way that is asymptotically negligible [8].
Our main contribution is to bring the techniques of point process theory, and more
specifically Palm theory [12, 6], to bear on this problem. Our approach is closely
related to earlier work of Poltyrev [17]. However the Palm theory viewpoint which
is brought into play here is not apparent in [17]; this allows one to go well beyond
the contribution of that work, which deals only with independent and identically
distributed (i.i.d.) Gaussian noise. In this framework, at block length n, one needs
to think about a stationary marked point process on Rn . Each realization of the
points is now thought of as a codebook. The power constraint has now vanished,
so one can think of being in the infinite signal-to-noise ratio (SNR) limit. A mark
is associated to each point. This mark is thought of as the realization of the noise
vector when that codeword (synonymous with point of the process) is “transmitted”.
The noise vectors are independent from point to point and have the law of a section
of length n of a given underlying stationary and ergodic centered real-valued discretetime stochastic process, which characterizes the communication channel. The “received
noise-corrupted codeword” is thus represented by the sum of the point (codeword) and
its mark (noise vector). The decoding problem is to figure out the mother point knowing
just the law of the noise process, the realization of the entire point process, and the
sum of the mother point and its mark (without knowing what the mother point is).
For instance, in the case of i.i.d. Gaussian noise a natural way to do this would be to
consider the Voronoi tessellation [16] of Rn associated to the point process. Note that
the Voronoi cells can be also thought of as marks of the point process. A decoding rule
is characterized by its error probability, defined as the limit over large cubes of the per
point error probability for the points in that cube (i.e. in order to compute the error
probability one assumes that the transmitter chooses one of the points within the cube
Capacity and Error Exponents of Stationary Point Processes
5
uniformly at random a priori). It is not hard to see that it suffices to consider decoding
rules that are jointly stationary with the underlying point process. This then means
that the error probability can be computed using Palm theory.
For the connection with information theory, the intensity of the underlying point
process is itself thought of as scaling exponentially in n. The logarithm of the point
process intensity on a per unit dimension basis will be called the normalized logarithmic
intensity. The first question that arises then is: for a given noise process, how large
can the asymptotic normalized logarithmic intensity be while still allowing for a choice
of the point process (this corresponds to the codebook) and choice of decoding rule for
that codebook such that the Palm error probability asymptotically vanishes? Proving
the existence of and identifying this threshold would give a point process analog of
Shannon’s capacity formula. This is the first thing we do in this paper. There are
no surprises here, since it boils down to volume counting. The threshold turns out to
be the negative of the differential entropy rate of the noise process. In honor of the
pioneering work in [17], we propose to call this threshold the Poltyrev capacity of the
associated noise process.
Much more interesting is the following question: for a given noise process and a given
asymptotic normalized logarithmic intensity that is less than the Poltyrev capacity of
the noise process, how big can one make the exponent of the rate at which the Palm
error probability can be made to go to zero? Here point process analogs of the random
coding and expurgated exponents are found. The random coding exponent comes from
considering the Poisson process, while the expurgated exponent comes from considering
a Matérn point process. Further, just as the capacity is determined by the differential
entropy rate of the noise process, the associated lower bounds on the infinite SNR error
exponent are derived from a large deviations principle (LDP) on its entropy spectrum
(the entropy spectrum for each dimension n is the asymptotic law of the information
density, which, in turn, is the random variable defined by the logarithm of the density
whose expectation, on a per symbol basis, asymptotically gives the differential entropy
rate). Identifying this connection is one of the main contributions of the point process
formulation investigated in this paper.
Finally, all these results obtained in the infinite SNR setting can be translated back
to give lower bounds on the error exponents in the orginal power-constrained additive
6
Venkat Anantharam and François Baccelli
noise channel which are new in information theory.
This Palm theory approach was first introduced in [1], where the i.i.d. Gaussian
case (called the additive white Gaussian noise (AWGN) case in the information theory
literature) was investigated and it was shown that the infinite SNR random coding
and expurgated exponents of [17] could be recovered with this viewpoint. The main
contribution of the present paper is to go beyond the i.i.d. Gaussian case to general
stationary and ergodic noise processes, subject to a mild technical condition needed to
have an LDP for the entropy spectrum.
The problem is formally set up in Section 2. In Section 3 it is proved that the
Poltyrev capacity is the threshold for the asymptotic normalized logarithmic intensity
in the sense described above. Section 4 gives representations of the error probability which will be instrumental to analyze the logarithmic asymptotics of the error
probability. Section 5 develops the infinite SNR random coding exponent, based on
the Poisson process and maximum likelihood (ML) decoding, while Section 6 develops
the expurgated exponent, based on a Matérn process and ML decoding. Section 7
is devoted to the connections between the results from the point process framework
and the motivating problem of information theory, namely how to translate the lower
bounds on the infinite SNR error exponent to those for the problem with power
constraints. Several examples of noise processes of practical interest are studied in
Section 8. In particular, the AWGN case is studied in depth. Section 9 briefly describes
how the results of the present paper generalize to the case of mismatched decoding,
which is of significant practical interest.
Throughout the paper, all logarithms are to the natural base. When discussing a
family of random variables indexed by the points of a point process, a notation such as
{Zk }k is used (this would mean that Zk is associated to the k-th point of the process).
For all basic definitions pertaining to point process theory see [6]; information theory
see [5, 9]; and large deviations theory see [7, 20].
2. Statement of the problem
Encoding; Normalized Logarithmic Intensity
Fix an integer n, and let (K =
Kn , K = Kn ) be a measurable space. Let M(K) (resp. M) denote the set of simple
7
Capacity and Error Exponents of Stationary Point Processes
marked counting measures (resp. simple counting measures) ν on Rn × K (resp. Rn ).
It is endowed with the σ-algebra M(K) (resp. M), which is generated by the events
ν(B × L) = k (resp. ν(B) = k), where B ranges over the Borel sets of Rn , L over the
measurable sets of Kn and k over the nonnegative integers (see e.g. [12]).
Each ν ∈ M has a representation of the form
ν=
X
ǫtk ,
k
with ǫx the Dirac measure at x and {tk }k the atoms of the counting measure ν.
Similarly, each ν ∈ M(K) has a representation of the form
ν=
X
ǫtk ,mk ,
k
with {(tk , mk )}k the atoms of ν, where tk ∈ Rn and mk ∈ Kn . The set {tk }k is the set
of points of ν n and the set {mk }k is its set of marks. Let M0 (K) (resp. M0 ) denote
the set of all simple marked counting measures (resp. simple counting measures) with
an atom having its first coordinate at 0.
Below, only stationary and ergodic marked point processes are considered. Thus it is
assumed that for each n ≥ 1 there exists a probability space (Ω, G, P, θt ), endowed with
an ergodic and measure preserving shift θt indexed by t ∈ Rn . A stationary marked
point process µ on Rn × Kn is a measurable map from (Ω, G) to (M(K), M(K)) such
that for all t ∈ Rn ,
µ(θt (ω)) = τt (µ(ω)) ,
where τt (µ) is the translation of µ by −t ∈ Rn : if µ(ω) =
µ(θt (ω)) =
X
k
P
k ǫTk (ω),Mk (ω) ,
then
ǫ−t+Tk (ω),Mk (ω) .
Let λn denote the intensity of µ = µn . The scaling where the normalized logarithmic
intensity approaches a limit as the dimension n goes to infinity is of particular interest.
Here Rn , the normalized logarithmic intensity of µn , is defined via λn = enRn . P0
denotes the Palm probability [6] of µ (by convention, under P0 , T0 = 0), Vk is the
Voronoi cell of point Tk with respect to the point process µ (see e.g. [16]) which is
taken here to be an open set.
As described informally in Section 1, the points of this point process are thought of
as representing the codewords used by a transmitter in a communication system in the
8
Venkat Anantharam and François Baccelli
infinite SNR limit. This connection with information theory provides the motivation
for the scaling considered here. Realizations of the noise vector, as well as decoding
regions (see below) are typical examples of the kinds of marks considered.
Decoding To each point Tk of the point process µ = µn (the superscript n is omitted
in this section), one associates the independent mark Dk , a random vector taking
values in Rn , called the displacement vector. When the point Tk of µ is thought of as a
codeword, the transmission over an additive noise channel adds to it the displacement
vector Dk , so that the received point is Yk = Tk + Dk .
Decoding is discussed in terms of a sequence of marks of µ which are measurable
sets of Rn . The mark of point Tk will be denoted by Ck . The set Ck is the decoding
region of Tk . The sets {Ck }k are required to form a tessellation of Rn , namely they are
all disjoint and the union of their closures is Rn .
The displacement sequence {Dk }k is assumed to be i.i.d. and independent of the
marked point process {Tk , Ck }k . This makes {Tk , (Dk , Ck )}k a marked point process.
The canonical example to keep in mind, which is motivated by the AWGN channel
of information theory, is when the vectors associated to the individual points of the
process are i.i.d. zero mean Gaussian random vectors each with i.i.d. coordinates and
independent of the points. Then the natural choice of decoding region of a point is its
Voronoi cell in the realization of the point process.
The most general setting concerning the noise (or displacement vectors) considered
in this paper will feature a real-valued centered stationary and ergodic stochastic
process ∆ = {∆l } and displacement vectors {Dk }k independent of the point process,
i.i.d. in k, and with a law defined by D = Dn = (∆1 , . . . , ∆n ) for all n. As it will be
seen, more elaborate though natural decoding tessellations then show up, determined
by the law of ∆.
The decoding strategy associated with the sequence of marks {Ck }k expects that
when Tk is transmitted, then the received point Yk lands in Ck . An error happens if this
is not the case. The error probability is now formally defined in a Palm theory setting.
Our eventual goal, as informally described in Section 1, is to study the exponent of
decay in n of the error probability.
Capacity and Error Exponents of Stationary Point Processes
9
Probability of Error Within the above setting, for all n, when (µn , C n ) and the
law of Dn are given, define the associated probability of error as
P
1T k ∈B n (0,W ) 11Ykn ∈C
/ kn
k1
Pn
pe (n) = lim
.
W →∞
1Tnk ∈B n (0,W )
k1
(1)
The limit in (1) exists almost surely and it is non-random. This follows from the
assumption that the marked point process µn with marks (Dkn , Ckn ) is stationary and
ergodic. The pointwise ergodic theorem implies that
/ C0n ) .
/ C0n ) = Pn0 (D0n ∈
pe (n) = Pn0 (Y0n ∈
(2)
3. Poltyrev Capacity of an Additive Noise Channel
The infinite SNR additive noise channel for dimension n is characterized by the
law of the typical displacement vector Dn , with Dn = (∆1 , . . . , ∆n ), with ∆ = {∆l }
as defined above. It will also be assumed that these displacement vectors Dn have a
density f n admitting a differential entropy rate
h(∆) = − lim
n→∞
1
E[ln f n (Dn )] .
n
(3)
We define −h(∆) to be the Poltyrev capacity of the additive noise channel with
displacement vectors defined in terms of the process ∆.
The terminology is chosen in honor of the work [17]. The justification for this
terminology comes from the following two simple theorems, which together give an
analog of Shannon’s capacity theorem for additive noise channels in information theory.
Before stating and proving these theorems, recall that for δ > 0, if one lets
1
n
n
n
n n
Aδ = x ∈ R : − ln(f (x )) − h(∆) < δ ,
n
(4)
then one has
P (Dn ∈ Anδ ) →n→∞ 1 .
(5)
This can be seen as a consequence of either one of the generalized Shannon-McMillanBreiman theorems in [4] or [13].
Theorem 1. For all point processes µn such that lim inf n Rn > −h(∆), for all choices
of decoding regions Ckn which are subsets of Rn jointly stationary with the points and
forming a tessellation of Rn , one has limn→∞ pe (n) = 1.
10
Venkat Anantharam and François Baccelli
Proof. For all stationary tessellations {Ckn }k , one has
Pn0 (D0n ∈ C0n ) ≤ En0 (1D0n ∈C0n ∩Anδ ) + En0 (1D0n ∈A
).
/ n
δ
The second term tends to 0 as n tends to infinity because of (5). The first term is
!
Z
n
n n
n
.
f (x )dx
E0
C0n ∩An
δ
It is bounded from above by e−n(h(D)−δ) En0 (Vol(C0n )), and for all translation invariant
tessellations of the Euclidean space En0 (Vol(C0n )) = e−nRn , which allows one to complete the proof.
✷
Theorem 2. Let µn be a Poisson point process of intensity λn = enRn . If lim supn Rn <
−h(∆) it is possible to choose decoding regions Ckn that are subsets of Rn jointly
stationary with the points and the displacements, forming a stationary tessellation of
Rn , such that limn→∞ pe (n) = 0.
Proof. Let {Ckn }k be the following tessellation of Rn :
Ckn
= {(Tkn + Anδ ) ∩ {∪l6=k (Tln + Anδ )}c }
[
{Vkn ∩ {∪l6=l′ [(Tln + Anδ ) ∩ (Tln′ + Anδ )]}}
[
{Vkn ∩ {∪l (Tln + Anδ )c }} ,
where Vkn denotes the Voronoi cell of Tkn . In words, Ckn contains all the locations x
which belong to the set Tkn + Anδ and to no other set of the form Tln + Anδ , all the
locations x that are ambiguous (i.e. belong to two or more such sets) and which are
closer to Tkn than to any other point, and all the locations which are uncovered (i.e.
belong to no such set) and which are closer to Tkn than to any other point. This scheme
will be referred to as typicality decoding in what follows.
Let µn! = µn − ǫ0 . Consider the bound
Pn0 (D0n ∈
/ C0n ) ≤ Pn0 (D0n ∈
/ Anδ ) + Pn0 (D0n ∈ Anδ , µn! (D0n − Anδ ) > 0).
The first term tends to 0 because of (5). For the second, Slivnyak’s theorem [6] is used
to bound it from above by
Pn (µn (D0n − Anδ ) > 0) ≤ En (µn (D0n − Anδ )) = En (µn (−Anδ )) = enRn |Anδ | .
11
Capacity and Error Exponents of Stationary Point Processes
But
1
≥
≥
Pn (D0n ∈ Anδ ) =
Z
An
δ
Z
f n (xn )dxn =
An
δ
Z
1
en n ln(f
n
(xn ))
dxn
An
δ
en(−h(D)−δ) dx = e−n(h(D)+δ) |Anδ | ,
so that |Anδ | ≤ en(h(D)+δ) , which allows one to complete the proof.
✷
Examples of stationary and ergodic noise processes are considered in Section 8. The
reader may wish to consult some of the examples at this stage for concrete instances
of the result above.
4. Maximum Likelihood Decoding
This section gives representations of the ML decoding error probability that will be
instrumental for the evaluation of error exponents in the forthcoming sections.
As in Section 3, f n denotes the density of the displacement vector Dn = (∆1 , . . . , ∆n )
which is a section of ∆ = {∆l }, a real-valued centered stationary and ergodic stochastic
process. The function
y n ∈ Rn → ℓf n (y n ) =
1
ln(f n (y n )) ∈ R,
n
is the (rescaled) log-likelihood of f n at y n . Note that ℓf n (y n ) ∈ [−∞, +∞] in general.
Below, −ℓf n is often used rather than ℓf n . The reason is that the real-valued random
variable −ℓf n (Dn ) is well kown and referred to as the normalized entropy density of
Dn [11]. Its law, denoted ρn∆ (du), is referred to as the entropy spectrum of Dn [11].
Note that the existence of a density for Dn does not imply that ρn∆ (.) admits a density.
Further, the support of ρn∆ (.) is not necessarily the whole real line.
The sets
n
S∆
(u) = {y n ∈ Rn : −ℓf n (y n ) ≤ u},
u∈R,
n
n
will be referred to as the log-likelihood level sets of Dn . The volume W∆
(u) of S∆
(u)
n
will be referred to as the log-likelihood level volume for u. The measure w∆
on R defined
by
n
w∆
(B) = Vol{y n ∈ Rn : −ℓf n (y n ) ∈ B},
12
Venkat Anantharam and François Baccelli
for all Borel sets B of the real line, will be called the log-likelihood level measure. It
n
turns out that the measures w∆
and ρn∆ are mutually absolutely continuous. Indeed,
one has
ρn∆ (B)
=
Z
1−ℓf n (xn )∈B f n (xn )dxn ,
which implies that for all bounded Borel sets B of the real line,
n
n
e−n sup(B) w∆
(B) ≤ ρn∆ (B) ≤ e−n inf(B) w∆
(B) .
(6)
n
From (6) it immediately follows that the measure w∆
is σ-finite. Also, for all u,
Z
Z
n
n
W∆
(u) =
w∆
(ds) =
ens ρn∆ (ds) .
(−∞,u]
(7)
(−∞,u]
Since µn is a point process, for all xn , Pn0 -a.s., the Rn –valued sequence {xn − Tkn }k ,
has no accumulation point. Hence, Pn0 -a.s., the set
argmaxk ℓf n (xn − Tkn )
is non empty, i.e. the supremum is achieved by at least one k. By definition, under
ML decoding, when xn is received, one returns the codeword argmaxk ℓf n (xn − Tkn ) if
the latter is uniquely defined. If there is ambiguity, i.e. if there are several solutions
to the above maximization problem then one returns any one of them.
Given that 0 = T0n is “transmitted” and that the realization of the additive noise
is xn , a sufficient condition for ML decoding to be successful is that µn have no point
Tkn other than T0n = 0 such that ℓf n (xn − Tkn ) ≥ ℓf n (xn ). But, for all xn ,
ℓf n (xn − Tkn ) < ℓf n (xn ), for all k 6= 0
if and only if
(µn − ǫ0 )(F (xn )) = 0,
(8)
with
F (xn ) = {y n ∈ Rn : ℓf n (xn − y n ) ≥ ℓf n (xn )} .
(9)
pe (n) ≤ Pn0 ((µn − ǫ0 )(F (Dn )) > 0) .
(10)
Hence
Also notice that the volume of the set F (xn ) only depends on ℓf n (xn ). If this last
quantity is equal to −u, the associated volume is
Vol {y n ∈ Rn : −ℓf n (xn − y n ) ≤ −u} = Vol {y n ∈ Rn : −ℓf n (y n ) ≤ −u} ,
13
Capacity and Error Exponents of Stationary Point Processes
i.e.
n
Vol(F (xn )) = W∆
(−ℓf n (xn ))) .
(11)
The main result of this section can now be stated:
Theorem 3. For all stationary and ergodic point processes µn and all i.i.d. displacement vectors, under ML decoding,
Z
pe (n) ≤ 1 −
Pn0 ((µn − ǫ0 )(F (xn )) = 0)f n (xn )dxn .
(12)
xn ∈Rn
If µn is such that, under Pn0 , the point process µn − ǫ0 admits an intensity bounded
from above by the function g n (.) on Rd , then
pe (n) ≤
Z
min 1,
Z
g n (y n )dy n
F (xn )
xn ∈Rn
!
f n (xn )dxn .
If µn is Poisson of intensity λn , then
Z
n
exp (−λn W∆
(u)) ρn∆ (du) ,
pe (n) ≤ 1 −
(13)
(14)
u∈R
where ρn∆ (du) is the entropy spectrum of f n .
Proof. The probability of success (given that 0 is sent and that the additive noise is
xn ) is the probability that µn has no point other than 0 in F (xn ), which proves (12).
(13) is immediate from (12) and the definition of F (xn ). (14) follows from (11), (12)
and Slivnyak’s theorem.
✷
With the preceding discussion of ML decoding in view, it is convenient to define the
(log-)likelihood cell Lnk (∆) of point Tkn as follows:
Lnk (∆)
= {xn : ℓf n (xn − Tkn ) > inf ℓf n (xn − Tln )}
l6=k
∪
(15)
{xn : ℓf n (xn − Tkn ) = ℓf n (xn − Tln ) for some l 6= k} ∩ Vkn .
It is comprised of the locations xn with a likelihood (with respect to f n ) to Tkn larger
than that to any other point; as well as the locations xn with an ambiguous loglikelihood but which are closer to Tkn for Euclidean distance than to all other points
of µn . These cells form a stationary tessellation of the Euclidean space which will
14
Venkat Anantharam and François Baccelli
be referred to as the likelihood tessellation with respect to the point process µn for
the noise ∆ (more precisely Dn or f n ). The likelihood tessellation with respect to
additive white Gaussian noise with positive variance is the Voronoi tessellation for all
dimensions n and for all point processes µn on Rn and for all k.
The resolution of ambiguity in this definition is somewhat Gaussian-centric. Any
other tessellation whose cells satisfy the conditions of Section 2 could be used in place
of the Voronoi tessellation.
5. Random Coding Exponent: The Poisson Process
Let ∆ be a stationary ergodic centered discrete-time real-valued stochastic process.
For all stationary and ergodic point processes µn of normalized logarithmic intensity
−h(∆) − ln(α), with α > 1, and decoding regions C n = {Ckn }k jointly stationary and
ergodic with µn , let
n
n
ppp
e (n, µ , C , α, ∆)
denote the probability of error associated with these data, as defined in (2). The
pp superscript is used to recall that the setting is the point process one described in
Section 2.
For a fixed family (µ, C) = (µn , C n ) of a jointly stationary and ergodic point process and decoding region for each dimension n, with normalized logarithmic intensity
−h(∆) − ln(αn ) for all dimensions n ≥ 1 and with αn → α as n → ∞, let
π(µ, C, α, ∆)
π(µ, C, α, ∆)
1
n
n
ln (ppp
e (n, µ , C , αn , ∆))
n
n
1
n
n
= lim inf − ln (ppp
e (n, µ , C , αn , ∆)) .
n
n
= lim sup −
(16)
(17)
The assumptions on the density f n on Rn of Dn = (∆1 , . . . , ∆n ) under which error
exponents will be analyzed in the point process formulation are summarized below
(where H-SEN stands for Hypothesis on Stationary Ergodic Noise):
H-SEN
1. For all n, the differential entropy of f n , h(Dn ), is well defined;
2. The differential entropy rate of ∆ = {∆l }, i.e. h(∆), as defined in (3), exists and
is finite;
Capacity and Error Exponents of Stationary Point Processes
15
3. The entropy spectrum ρn∆ (du), namely the law of the random variables {− n1 ln(f n (Dn ))},
satisfies an LDP (on the real line endowed with its Borel σ-field), with good (in
particular lower semicontinuous) and convex rate function I(x) [7, 20].
A simple sufficient condition for 3. above to hold is that the conditions of the
Gärtner-Ellis Theorem hold, namely that the limit
1 n n −θ lim
=: G(θ)
ln E (f (D ))
n→∞ n
exists as an extended real number, is finite in some neighborhood of the origin, and is
essentially smooth (see Definition 2.3.5 in [7]). From the Gärtner-Ellis Theorem, the
family of measures ρn∆ (dx) then satisfies an LDP with good and convex rate function
I(x) = sup (θx − G(θ)) .
(18)
θ
The following lemma gives the log-scale asymptotics of the log-likelihood level volumes.
Lemma 1. Suppose that the assumptions H-SEN hold. Then
sup(s − I(s)) ≤ lim inf
s<u
n→∞
1
1
n
n
ln(W∆
(u)) ≤ lim sup ln(W∆
(u)) ≤ sup(s − I(s)).
n
n→∞ n
s≤u
(19)
Further, the function
J(u) = sup(s − I(s)),
(20)
s≤u
which will be referred to as the volume exponent, is upper semicontinuous.
Proof. From (7),
n
W∆
(u) ≥
where
φ(s) =
Z


1
enφ(s) ρn∆ (ds),

−∞
if s < u
.
if s ≥ u.
Since ρn∆ (dx) satisfies an LDP and since the function φ is lower semicontinuous, the
lower bound is proved as in Lemma 4.3.4 in [7]. Similarly


s
if s ≤ u
e
φ(s) =

−∞ if s > u
Some of the results derived below do not require this convexity assumption.
16
Venkat Anantharam and François Baccelli
is upper semicontinuous and the upper bound is proved as in Lemma 4.3.6 in [7]. In
both cases, it should be noticed that the proofs in [7] actually allow for functions φ
with values in {−∞} ∪ R.
It is now shown that the upper semicontinuity of the function g(s) = s− I(s) implies
that of the function J(u) = sups≤u g(s). It has to be shown that
J(u) ≥ lim
sup
ǫ→0 s∈[u−ǫ,u+ǫ]
J(s) = lim J(u + ǫ) ,
ǫ→0
(21)
where the rightmost equality follows from the fact that J is non-decreasing. Hence,
using monotonicity again, it has to be shown that J is right-continuous.
One has
J(u + ǫ) = J(u) +
sup (g(s) − J(u))+ ,
s∈[u,u+ǫ]
with a+ = max(a, 0). So, either g(s) ≤ J(u) for all s ∈ [u, u + ǫ], in which case
J(u + ǫ) = J(u) and the right-continuity is trivially satisfied, or g(s) > J(u) for some
s ∈ [u, u + ǫ], in which case
J(u + ǫ) = sup g(s).
[u,u+ǫ]
It then follows from the upper semicontinuity of the function g(s) that
J(u) ≥ g(u) ≥ lim sup g(s) = lim J(u + ǫ),
ǫ→0 [u,u+ǫ]
ǫ→0
so that (21), and hence right-continuity, holds in this case too.
✷
Since I(h(∆)) = 0, it follows from (20) that J(h(∆)) ≥ h(∆). The concavity of
the function x → x − I(x) implies that this function is non decreasing on the interval
(−∞, h(∆)]. Hence, from (20), one has
J(h(∆)) = h(∆) .
Further, one may conclude that at all points u of continuity of J one has
lim
n→∞
1
n
ln(W∆
(u)) = J(u) .
n
The following theorem, which comes from considering the family of Poisson point
processes with ML decoding, gives the random coding exponent for the problem formulation adopted here.
17
Capacity and Error Exponents of Stationary Point Processes
Theorem 4. Assume that µn is Poisson with normalized logarithmic intensity −h(∆)−
ln(α) with α > 1 and that the decoder uses ML decoding. Suppose that assumptions
H-SEN hold. Then the associated error exponent is such that
π(Poi, L(∆), α, ∆) ≥ inf {F (u) + I(u)} ,
u
(22)
where I(u) is the rate function of ρn∆ (defined in (18)) and
F (u) = (ln(α) + h(∆) − J(u))
+
,
where J(u) = sups≤u (s − I(s)) is the volume exponent defined in Lemma 1.
Proof. From (14),
pe (n) ≤
Z
u∈R
n
(1 − exp (−λn W∆
(u))) ρn∆ (du) .
(23)
using (23) and the bound
n
n
(u))
1 − e−λn W∆ (u) ≤ min(1, λn W∆
one can write
pe (n) ≤
with
φn (u) =
Z
u
e−nφn (u) ρn∆ (du),
+
1
n
ln(α) + h(∆) − ln(W∆
(u))
.
n
In order to conclude, use Theorem 2.3 in [20]. Since the law ρn∆ (du) satisfies an
LDP with good rate function I(u), it is enough to prove that for all δ > 0, there exists
ǫ > 0 such that
+
1
+
n
≥ (ln(α) + h(∆) − J(u))) − δ.
ln(α) + h(∆) − ln(W∆ (u))
lim inf
inf
n→∞ (u−ǫ,u+ǫ)
n
n
Since the function u → W∆
(u) is non decreasing, it is enough to show that for all
δ > 0, there exists ǫ > 0 such that
+
1
+
m
lim ln(α) + h(∆) − sup
ln(W∆
(u + ǫ)) ≥ (ln(α) + h(∆) − J(u))) − δ.
n→∞
m
m≥n
There are two cases, If ln(α) + h(∆) − J(u) ≤ 0, the result is obvious. If ln(α) + h(∆) −
J(u) > 0, then one has to prove that for all δ, there exists an ǫ such that
lim sup
n→∞ m≥n
1
m
ln(W∆
(u + ǫ)) ≤ sup(s − I(s)) + δ
m
s≤u
18
Venkat Anantharam and François Baccelli
But from Lemma 1,
lim sup
n→∞ m≥n
1
m
ln(W∆
(u + ǫ)) ≤ sup (s − I(s)).
m
s≤u+ǫ
Hence it is enough to show that for all δ, there exists an ǫ such that
sup(s − I(s)) ≥ sup (s − I(s)) − δ .
s≤u
s≤u+ǫ
This follows from the fact that the function J(u) is upper semicontinuous.
✷
Notice that all terms in the final expression to be minimized, namely
(ln(α) + h(∆) − J(u))+ + I(u)
have a simple conceptual meaning. e−(ln(α)+h(∆)) is the intensity, i.e. λn ; enJ(u) is the
volume of the log-likelihood level set for level u; e−nI(u) is the value of the density of
the entropy spectrum at u; and finally, the positive part stems from the minimum of
the mean number of points in the above set and the number 1.
6. Expurgated Exponent: A Matérn Process
A Matérn I point process is created by deleting points from a Poisson process as
follows. Choose some positive radius called the exclusion radius. Any point in the
initial Poisson process that has another point within this fixed radius of it is deleted
(note that both points will be deleted since the first point will also be within the same
fixed radius of the second point). This is the simplest type of hard sphere exclusion. For
an information theorist, this is reminiscent of expurgation [8] and this term will also be
used below to describe the transformation of the Poisson into a Matérn point process.
This process and a related process called the Matérn II process were introduced in [14].
The Matérn II process will not be considered in this paper.
Mimicking this idea, a new class of Matérn point processes is introduced in order to
cope with the general stationary and ergodic noise in the present problem formulation.
Assume for simplicity that f n (xn ) = f n (−xn ). If two points S and T of the Poisson
point process µn are such that −ℓf n (T − S) < ξ , with ξ ∈ R some threshold, then
both T and S are deleted (−ℓf n may be thought of as a surrogate distance; two point
Capacity and Error Exponents of Stationary Point Processes
19
which are “too close” are discarded). The surviving points form the Matérn-∆-ξ point
process µ
bn .
Theorem 5. Under the assumptions of Theorem 3, the probability of error for the
Matérn-∆-ξ point process satisfies the bound


Z
Z
min 1, λn
1−ℓf n (yn )≥ξ 1ℓf n (xn −yn )≤ℓf n (xn ) dy n  f n (xn )dxn .
pe (n) ≤
xn ∈Rn
(24)
y n ∈Rn
b n denote the Palm probability of µ
b n , the point process µ
Proof Let P
bn . Under P
bn −ǫ0
0
0
has an intensity bounded from above by λn 1−ℓf n (yn )≥ξ at y n . The result then follows
from (13).
✷
Notice that the Matérn-AWGN-ξ model boils down to the Matérn I model for the
exclusion radius
√
rn (ξ) =
for ξ >
1
2
2nσ 2
r
ξ−
1
ln(2πσ 2 ) ,
2
(25)
ln(2πσ 2 ). Hence the following special case holds.
Theorem 6. In the AWGN case,
Z
min (1, λn Vol (B n (0, rn (ξ))c ∩ B n (xn (r), r))) gσn (r)dr
pe (n) ≤
(26)
r>0
with xn (r) = (r, 0, . . . , 0) ∈ Rn and rn (.) defined in (25).
Proof The result immediately follows from (24) and (25).
✷
In the general case, the unfortunate fact that the volume of the vulnerability set
(the set which ought to be empty of points for no error to occur) now depends on the
point xn , and not only on the value of ℓf n (xn ) can be taken care of by introducing the
following upper bound
n
M∆
(u, ξ) =
sup
xn :
−ℓf n (xn )=u
Z
1−ℓf n (yn )≥ξ 1−ℓf n (xn −yn )≤u dy n ,
(27)
y n ∈Rn
which only depends on ℓf n (xn ). This quantity will be referred to as the expurgated
log-likelihood level volume. By the same arguments as above, one gets the following
20
Venkat Anantharam and François Baccelli
result.
Corollary 1. The probability of error for the Matérn-∆-ξ point process satisfies the
bound
pe (n) ≤
Z
u∈R
n
min (1, λn M∆
(u, ξ)) ρn∆ (du) .
(28)
In Section 8 the expurgated exponent is worked out based on the Matérn-∆-ξ process
in some examples. Particular attention is paid to the AWGN case, where it is shown
that one recovers the expurgated exponent of [17].
7. The Channel with Power Constraints
In the traditional model for point-to-point communication over an additive noise
channel with power-constrained inputs, the codewords, of block length n, are subject
to the power constraint P . A codebook is thus a finite, non-empty subset, call it T ,
√
√
of points in B n (0, nP ) (the closed ball of radius nP around the origin), whose
elements are the codewords. R(T ) =
1
n
ln | T |≥ 0 is then the rate of the code. The
noise vector for block length n, Dn = (∆1 , . . . , ∆n ) is assumed to have the law of
the first n values of the centered real-valued stationary and ergodic stochastic process
∆ = {∆l }. Suppose that the assumptions H-SEN are in force, and that the marginals
of ∆ have finite variance.
The transmitter is assumed to pick a codeword to transmit uniformly at random from
the codebook. The receiver sees the sum of the codeword and the noise vector, and,
without knowing which codeword was picked, it is required to determine it from the
received noise-corrupted codeword. The optimum decision rule is maximum likelihood
decoding, i.e. to choose as the decision for the transmitted codeword one of those for
which the conditional probability of seeing the given observation is largest among all
codewords. The probability of error of the codebook, pe (T ), is defined to be the average
probability of error over all codewords, where the probability of error of a codeword
is the probability of error of the maximum likelihood decision rule, conditioned on
this codeword having been transmitted. Shannon [18, 19] proved that, asymptotically
in the block length, there is a threshold on the rate such that for rates below this
21
Capacity and Error Exponents of Stationary Point Processes
threshold it is possible to choose codebooks for which the probability of error goes
asymptotically to zero, while for rates above this threshold this is not possible. This
threshold is given by the Shannon capacity, defined by
CP (∆) = lim
n→∞
1
n T n,
sup
E(
Pn
I(T n , T n + Dn ) ,
n 2
i=1 (Ti ) )<nP
where the supremum is over all distribution functions for T n = (T1n , . . . , Tnn ) ∈ Rn such
Pn
that E( i=1 (Tin )2 ) < nP . This limit is known to exist. Here, for jointly distributed
vector valued random variables (X, Y ), the expression I(X; Y ) denotes their mutual
information [5, 9].
Note that the Shannon capacity is a characteristic of both noise process ∆ and
the power constraint P . Let σ 2 denote the variance of ∆0 . The relation between the
Shannon capacity and the Poltyrev capacity is given by the following lemma, which is
due to Shannon [18]. We give a proof, since it is illuminating.
Lemma 2. Under the foregoing assumptions,
1
1
ln(2πeP ) − h(∆) ≤ CP (∆) ≤ ln(2πe(P + σ 2 )) − h(∆) .
2
2
(29)
Proof. One has
I(T n + ∆n ; T n ) = h(T n + ∆n ) − h(T n + ∆n | T n ) = h(T n + ∆n ) − h(∆n ) .
It is well known that for all stationary sequences {Ak } one has
h(A1 , A2 , . . . , An ) ≤
n
ln(2πeVar(A1 )).
2
Hence
1
1
1
I(T n + ∆n ; T n ) ≤ ln(2πe(P + σ 2 )) − h(∆n ) .
n
2
n
For the lower bound, the inequality h(T n + ∆n ) ≥ h(T n ) is used to deduce that
I(T n + ∆n ; T n ) = h(T n + ∆n ) − h(T n + ∆n | T n ) ≥ h(T n ) − h(∆n ) .
Taking now T n Gaussian with i.i.d. N (0, P ) coordinates, one gets that
CP (∆) ≥
1
ln(2πeP ) − h(∆).
2
22
Venkat Anantharam and François Baccelli
✷
In the power-constrained scenario, one defines
E(n, R, P, ∆) = −
1
ln pe,opt (n, R, P, ∆) ,
n
with pe,opt (n, R, P, ∆) the infimum of pe (T ) over all codes in Rn of rate at least R ≥ 0
and all decoding rules, when the signal power is P and the noise is ∆. One then defines
Ē(R, P, ∆)
= lim sup E(n, R, P, ∆), and
E(R, P, ∆)
= lim inf E(n, R, P, ∆).
n
n
Assuming these are identical, one denotes this common limit by E(R, P, ∆). For fixed P
and ∆, the function R 7→ E(R, P, ∆), defined for rates less than the Shannon capacity,
is what is called the error exponent function or the reliability function in information
theory.
The following result shows how to get lower bounds on the error exponent function
for power-constrained additive noise channels from error exponents coming out of the
point process formulation (such as the random coding exponent and the expurgated
exponent developed in Sections 5 and 6 respectively).
The next theorem features a sequence of stationary point processses µn in Rn with
normalized logarithmic intensities converging to the finite limit −h(∆) − ln(α′ ), where
α > α′ > 1. The following condition will be required on this collection: for all γ > 0,
for all P > 0,
√
′
n
= o(n).
ln Pn µn (B n (0, nP )) ≥ (2πeP ) 2 e−nh(∆) e−n ln(α +γ)
(30)
This condition is satisfied e.g. by homogeneous Poisson and Matérn point processes as
both are such that, for all Borel sets B of Rn ,
E[µn (B)2 ] ≤ E[b
µn (B)2 ],
(31)
where µ
bn denotes the homogeneous Poisson point process with the same intensity as
µn . For Matérn point processes (31) follows from the evaluation of the reduced second
moment measure, which is classical. For all collections {µn } satisfying (31) for all n,
one gets (30) from Chebyshev’s inequality.
23
Capacity and Error Exponents of Stationary Point Processes
Theorem 7. Let ∆ be a centered real-valued stationary and ergodic stochastic process
and let α > 1. Let (µ, C) := (µn , C n ) be a sequence where, for each n ≥ 1, µn is
a stationary and ergodic point process in Rn with normalized logarithmic intensity
−h(∆) − ln(αn ), with αn → α′ as n → ∞, where α > α′ > 1, and the sequence {µn }
satisfies (30), and where, for each n ≥ 1, the tessellation C n is jointly stationary with
µn . Then, for all P > 0 such that
E
1
2
ln(2πeP ) > h(∆) + ln(α), one has
1
ln(2πeP ) − h(∆) − ln(α), P, ∆ ≥ π(µ, C, α′ , ∆),
2
(32)
and
σ2
1
, P, ∆ ≥ π(µ, C, α′ , ∆).
E CP (∆) − ln(α) − ln 1 +
2
P
(33)
Here π(µ, C, α′ , ∆) is the error exponent without restriction for the family (µ, C), as
defined in (17). In addition
lim inf E(CP (∆) − ln(α), P, ∆) ≥ π(µ, C, α′ , ∆).
P →∞
(34)
Proof. From the very definition of Palm probabilities, for all n,
pe,k
n
n
ppp
,
e (n, µ , C , αn , ∆) = −nh(∆) −n ln(α ) n √
n V
e
e
B ( nP )
En
P
k:Tkn ∈B n (0,
√
nP )
where pe,k denotes the probability that Tkn +Dkn does not belong to Ckn given {Tln, Cln }l .
Hence, for all γ > 0, we can write:
n
n
ppp
e (n, µ , C , αn , ∆)
En
≥
P
pe,k 1µn (B n (0,√nP ))≥(2πeP ) n2 e−nh(∆) e−n ln(α′ +γ)
√
k:Tkn ∈B n (0, nP )
√
e−nh(∆) e−n ln(αn ) VBn ( nP )
√
′
n
≥ Pn µn (B n (0, nP )) ≥ (2πeP ) 2 e−nh(∆) e−n ln(α +γ)
pe,opt (n,
n
′
(2πeP ) 2
1
ln(2πeP ) − h(∆) − ln(α′ + γ), P, ∆)e−n ln(α +γ) en ln(αn ) n √
,
2
VB ( nP )
24
Venkat Anantharam and François Baccelli
where we have used the fact that pe,opt (n, R, P, ∆) is nondecreasing in R and enR is
nondecreasing in R. If γ > 0 is sufficiently small, we can then write:
1
n
n
ln (ppp
e (n, µ , C , αn , ∆))
n
1
1
≤ − ln pe,opt (n, ln(2πeP ) − h(∆) − ln α, P, ∆)
n
2
′
n
1 n n n √
− ln P µ (B (0, nP )) ≥ (2πeP ) 2 e−nh(∆) e−n ln(α +γ)
n
!
n
1
(2πeP ) 2
′
√
− ln(αn ) + ln (α + γ) − ln
.
n
VBn ( nP )
−
When taking the limit in n, the second term of the R.H.S. tends to 0 (from (30)),
and the last term of this R.H.S. tends to 0 as well (from classical asymptotics on the
volume of the d-ball). Hence, first taking the limit as n → ∞ and then letting γ → 0,
(32) follows.
One gets (33) from (32) when using the second inequality of (29) and the fact that
the function x → E (x, P, ∆) is non-increasing.
To prove (34), pick α̃ such that α > α̃ > α′ > 1. It suffices to observe that from the
preceding proof, we have
1
σ2
E CP (∆) − ln(α̃) − ln 1 +
, P, ∆ ≥ π(µ, C, α′ , ∆).
2
P
✷
The preceding theorem can, in particular, be used with the family (µ, C) taken to be
either (Poi, L(∆)) or (Mat, L(∆)), for which π(µ, C, α′ , ∆) has been studied in detail
in this paper. An excellent survey of the known upper and lower bounds for the error
exponent function in the power-constrained AWGN case is given in [3].
8. Examples
This section contains several examples of noise processes ∆ of interest in applications
and work out the concrete instantiation of the preceding results in these cases. Consider
first the additive white noise (WN) case, i.e. when ∆ = {∆l } is an i.i.d. sequence,
focusing on the special cases of white symmetric exponential noise and white uniform
noise. Additive colored Gaussian noise (CGN), are then discussed, where {∆l } is
25
Capacity and Error Exponents of Stationary Point Processes
Gaussian sequence which is not necessarily white, and finally discuss in detail the
additive white Gaussian noise (AWGN) case, which is the case of most interest in
applications. Connections to the work in [17] in the AWGN case are made. A random
coding exponent is worked out in all examples, and an expurgated exponent is worked
out where it was possible to give a relatively clean looking result.
8.1. White Noise
The WN case is that where the displacement vector ∆ has i.i.d. coordinates. Let
D be a typical coordinate random variable. The differential entropy rate of ∆ is then
Z
h(∆) = h(D) = − f (x) ln(f (x))dx ,
R
where f (x) denotes the density of D.
From Cramér’s theorem [7, 20] one has
I(x) = sup θx − ln E f (D)−θ
θ
with D a random variable with density f .
,
Notice that the rate function I(.) is not necessarily a good rate function. A sufficient
condition is that 0 is in the interior of the set {θ : E (f (D))−θ < ∞} (see [7], Lemma
2.2.20).
8.1.1. White Symmetric Exponential Noise The differential entropy of the symmetric
√
exponential distribution of variance σ 2 is h(D) = ln( 2eσ) and
√ !!
√ θ
√
|D| 2
1
−θ
= ( 2σ) E exp θ
E f (D)
= ( 2σ)θ
, θ<1.
σ
1−θ
So
I(u)
√
= sup θu − θ ln( 2σ) + ln(1 − θ) ,
θ
that is
I(u) =


+∞
√

u − h(D) − ln(u − ln( 2σ))
√
for u ≤ ln( 2σ);
otherwise,
(35)
26
Venkat Anantharam and François Baccelli
which is a good and convex rate function.
From Lemma 1 one gets


−∞
J(u) =
√
√

ln( 2eσ(u − ln( 2σ)))
√
for u ≤ ln( 2σ)
(36)
otherwise .
It follows from the formula (35) for I and the formula (36) for J, that, in this case,
the function to minimize in (22) is
v − 1 − ln(v) + (ln(α) − ln(v))+ ,
for v > 0. So in this case the random coding exponent is the right hand side of the
inequality
π(Poi, L(∆), α, ∆) ≥


α − 1 − ln α
 1 − 2 ln 2 + ln α
if 1 ≤ α < 2
if α ≥ 2.
(37)
Consider the Matérn-∆-ξ point process, where ∆ is white symmetric exponential
noise and where the exclusion regions are L1 balls of radius
√
nσ
rn (ξ) = √ (ξ − ln( 2σ)) ,
2
√
for ξ > ln( 2σ). For the target normalized logarithmic intensity −h(∆)−α, one builds
this Matérn point process, µ
en , from a Poisson point process µn of intensity λn = enR
√
with R = − ln( 2eσα), where α > 1. The parameter ξ is chosen as follows:
√
ξ = α − ǫ + ln( 2σ) ,
nσ
√
(α −
2
n
−λn VB,1 (rn )
so that the L1 exclusion radius is rn =
en = λn e
Matérn point process is then λ
ǫ). The intensity of the associated
, with
√
(2rn )n
( 2σ(α − ǫ))n nn
n
VB,1
(rn ) =
=
n!
n!
en ≤ λn for all n and that
the volume of the L1 ball of radius rn . It is easy to see that λ
limn→∞
en
λ
λn
= 1.
It follows from (24) that
pe (n)≤
Z
min 1, λn
r>0
sup
xn :|xn |
1 =r
Vol
B1n
c
(0, rn ) ∩
B1n
n
(x , r)
!
gσn (r)dr,
(38)
27
Capacity and Error Exponents of Stationary Point Processes
where |.|1 denotes the L1 norm, B1n (x, r) the L1 ball of center x and radius r and gσn (r)
here denotes the density of the L1 norm of Dn , given by
√ !n n−1
√
2
r
n
− σ2 r
gσ (r) = e
, r ≥ 0.
σ
Γ(n)
√
Making the substitution v = ( 2r)/nσ, one gets that the R.H.S. of (38) is
!
Z
(vn)n
n
min 1, λn
sup
dv ,
W (x , v) e−vn
vΓ(n)
√
xn :|xn |1 = vσn
v>0
with
(39)
2
c
vnσ
nσ(α − ǫ)
√
∩ B1n xn , √
W (xn , v) = Vol B1n 0,
.
2
2
Let α
e = α − ǫ. If v ≤
below that if v >
α
e
2,
α
e
2,
then W (xn , v) = 0 for all xn with |xn |1 =
vσn
√ .
2
(40)
It is proved
then
1
lim
ln
n→∞ n
sup
√
xn :|xn |1 = vσn
2
n
!
W (x , v)
√
= ln( 2veσ) .
(41)
One has
sup
√
xn :|xn |1 = vσn
2
W (xn , v) ≥ W (xn (v), v)
√
with xn (v) = (vσn/ 2, 0, . . . , 0). The region R(n) on the right hand side of (40)
includes the region
(
n
X
vσn (α − ǫ)σn
2vσn
√
y = (y1 , . . . , yn ) ∈ R : y1 > √ ,
|yi | < √
< y1 +
2
2
2
i=2
n
n
)
,
which is comprised of 2n−1 copies (one for each configuration of signs of the variables
y2 , . . . , yn , see Figure 1) of the following basic region:
)
(
n
X
2vσn
vσn (α − ǫ)σn
n
n
√
yi < √
.
< y1 +
y = (y1 , . . . , yn ) ∈ R+ : y1 > √ ,
2
2
2
i=2
In Figure 1, the origin of the plane is the tagged codeword. The large ball centered
√
in 0 and passing through point A is that with radius nσ(α−ǫ)
. Point V is that with
2
√
n
coordinate x (v) = (vσn/ 2, 0, . . . , 0). The region R(n) is depicted by the union of
the dashed region and the grey one. The volume V (n) is that of the grey region.
The volume V (n) of this basic region is the same as that of
)
(
n
vσn
(α − ǫ − v)σn X
n
n
√
yi < √
,
<
y = (y1 , . . . , yn ) ∈ R+ :
2
2
i=1
28
Venkat Anantharam and François Baccelli
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
0
1
0
1
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
0
1
1
0
0
1
0
1
1
0
0
V1
0
1
1
0
0
1
0
1
1
0
0
1
0
1
0
1
0
1
0
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
1
0
0
1
0
1
0
0
1
A
Figure 1: Matérn case with white symmetric exponential noise
namely 2−n times the volume of the L1 ball of center 0 and radius
L1 ball of center 0 and radius
V (n) = 2
−n
+
(α−ǫ−v) σn
√
,
2
!
n
sup
deprived of the
that is
n
n
√
√
nn
+ nn
.
− ( 2(α − ǫ − v) σ)
( 2vσ)
n!
n!
Hence
1
ln
n
vσn
√
2
W (x , v)
√
xn :|xn |1 = vσn
2
≥
√
1
ln 2n−1 V (n) →n→∞ ln( 2veσ).
n
But from (40),
1
ln
n
sup
n
!
W (x , v)
√
xn :|xn |1 = vσn
2
≤
√
vσn
1
ln VolB1n (0, √ ) →n→∞ ln( 2veσ).
n
2
This completes the proof of (41).
The error exponent associated with this sequence of Matérn point processes thus
satisfies the bound:
π(Mat, L(∆), α, ∆) ≥ inf b(v) + a(v) ,
v>0
n
(vn)
), and
with a(v) = v − ln(v) − 1, (stemming from e−vn vΓ(n)


∞
if 0 < v < αe2
b(v) =
 (ln α
e − ln v)+
if αe2 < v .
Capacity and Error Exponents of Stationary Point Processes
29
(stemming from min 1, λn supxn :|xn|1 = vσn
W (xn , v) in (39)) For more details on this
√
2
derivation, see the long version of this paper [2] and in particular the analytical
arguments for the AWGN case. This leads to the following expurgated exponent for
symmetric exponential white noise.



α − ln(α) − 1
for α ≤ 2



π(Mat, L(∆), α, ∆) ≥ ln(α) + 1 − 2 ln(2)
for 2 ≤ α ≤ 4




 α − ln(α) − 1 + 2 ln(2) for α ≥ 4.
2
√
√
8.1.2. White Uniform Noise Let D be uniform on [− 3σ, + 3σ], which is centered
√
and with variance σ 2 . The differential entropy is h(D) = ln(2 3σ) and
√
E f (D)−θ = (2 3σ)θ ,
√
so that G(θ) = θ ln(2 3σ) and

√

∞ if u 6= ln(2 3σ)
I(u) =
√

0 if u = ln(2 3σ) ,
(42)
which is a good and convex rate function.
From Lemma 1,
J(u) =


−∞
√
for u < ln(2 3σ)
√
for u ≥ ln(2 3σ) .
√

ln(2 3σ)
(43)
Here it follows from (43) and (42) and (22) that
√
π(Poi, L(∆), α, ∆) ≥ F (ln(2 3σ)) = ln(α) .
The right hand side of the preceding equation is the random coding exponent for
white uniform noise.
8.2. Colored Gaussian Noise
The CGN case is that where {∆k } is a stationary and ergodic Gaussian process with
spectral density function g(β), i.e.
E(∆0 ∆k ) =
1
2π
Z
π
−π
eikβ g(β)dβ,
30
Venkat Anantharam and François Baccelli
for all k. It is well known (see e.g. [10]) that the differential entropy rate of such a
stationary process exists and is given by
Z π
1
h(∆) =
ln(2eπg(β))dβ.
4π −π
(44)
The conditions for the validity of the Gärtner-Ellis Theorem hold with
Z π
θ
1
θ 1
G(θ) = ln(2π) − ln(1 − θ) +
ln(g(β))dβ ,
2
2
2 2π −π
when θ < 1 and G(θ) = ∞ for θ > 1. This gives

Rπ

1
∞
if u ≤ 4π
−π ln(2πg(β))dβ;
I(u) =

u − h(∆) − 1 ln 2u − 1 R π ln(2πg(β))dβ
otherwise,
2
2π −π
(45)
with h(∆) as in (44). This is a good, convex, and continuous rate function.
From (45) and (20) one gets



−∞



Rπ
1
J(u) =
4π −π ln(2πeg(β))dβ +





Rπ
1
if u ≤ 4π
−π ln(2πg(β))dβ;
Rπ
1
1
ln
2u
−
ln(2πg(β))dβ
2
2π −π
(46)
otherwise .
This function is continuous.
Theorem 4, (45), and (46) give
(
+
Z π
1
1
ln 2u −
ln(2πg(β))dβ
u
2
2π −π
Z π
Z π
1
1
1
+u −
ln(2πeg(β))dβ − ln 2u −
ln(2πg(β))dβ
.
4π −π
2
2π −π
π(Poi, L(∆), α, ∆)
≥
ln(α) −
inf
Making the substitution
v=
s
1
2u −
2π
Z
π
ln(2πg(β))dβ ,
−π
one gets that the last infimum is
v2
1
+
inf (ln(α) − ln(v)) +
− − ln(v)
v≥0
2
2
and one hence gets the same function to optimize as in the AWGN case. So the random
coding exponent is that of formula (49).
31
Capacity and Error Exponents of Stationary Point Processes
8.3. White Gaussian Noise
The AWGN case is the special case of WN where f is Gaussian with mean zero and
variance σ 2 . In this case, the differential entropy of f is h(D) =
1
2
ln(2πeσ 2 ), and one
has
I(u) =


+∞

u −
for u ≤
1
2
ln(2eπσ 2 ) −
1
2
1
2
ln(2πσ 2 );
(47)
ln(2u − ln(2πσ 2 )) otherwise,
which is a good and convex rate function.
It immediately follows from Lemma 1 that


−∞
J(u) =

 1 ln(2πeσ 2 ) + 1 ln 2u − ln(2πσ 2 )
2
2
for u ≤
1
2
ln(2πσ 2 )
(48)
otherwise .
One therefore recovers the following result, first obtained by Poltyrev for the AWGN
case in [17] and revisited in [1].
π(Poi, L(AWGN), α, AWGN) ≥



α2
2
1
2
−
1
2
√
if 1 ≤ α < 2
.
√
if 2 ≤ α < ∞
− ln α
− ln 2 + ln α
(49)
This follows from (22), as it is now shown . Using the formula (47) for I and the
q
formula (48) for J in (22) and using the substitution v = 2(u − 12 ln(2πσ 2 )), one gets
the following equivalent optimization problem.
Minimize a(v) + b(v) over v ≥ 0, with
v2
1
a(v) =
− − ln(v)
2
2
b(v) = (ln α − ln v)+ ,
which is precisely that analyzed in [1]. This gives
√
1
2.
2 − ln 2 + ln α when α >
α2
2
−
1
2
− ln α when 1 < α <
(50)
√
2 and
the next discussion is centered on the expurgated exponent based on the Matérn I
process. Fix ǫ > 0. Consider a sequence of Matérn I processes µ
en . The point process
µ
en is built from a Poisson processes µn of rate λn = enR where R = 21 ln 2πeα1 2 σ2 for
√
α > 1, and has exclusion radius (α − ǫ)σ n. The intensity of this Matérn I point
In fact, it can be shown that this lower bound is tight, see [17].
32
Venkat Anantharam and François Baccelli
process is
en = λn e−λn VBn ((α−ǫ)σ
λ
en
λ
λn
and it is easy to see that
√
n)
en < λn for all n.
→n→∞ 1, with λ
Let π(Mat, L(AWGN), α, AWGN) denote the error exponent (17) associated with
this family of Matérn point processes. It is proves below that
π(Mat, L(AWGN), α, AWGN) ≥
α2
,
8
for all α ≥ 2.
(51)
√
Take an exclusion radius of (α − ǫ)σ n. From Formula (26),
Z
√ √ c
min 1, λn Vol B n 0, (α − ǫ)σ n) ∩ B n y n (v), (vσ n
pe (n) ≤
v∈R+
√ √
g1n (v n) ndv ,
√
with y n (v) = (vσ n, 0, . . . , 0). It is proved below that
√ √
√ c
Vol B n 0, (α − ǫ)σ n) ∩ B n y n (v), (vσ n ≤ VBn (c(v)σ n) ,
with



0

 q
c(v) =
v 2 − (v −




v
with α
e = α − ǫ. If v <
α
e
2,
so that c(v) = 0 in (52).
α
e
2
if 0 < v <
α
e2 2
2v ) )
if
α
e
2
if
<v<
α
e
√
2
(52)
α
e
2
α
e
√
2
(53)
<v .
then
√ √ eσ n)
B n y n (v), vσ n ⊂ B n 0, α
α
e
√
,
2
one has to find an upper bound on the volume of the portion of the
√
ball of radius vσ n around the point at distance vσ n from the origin (along some
√
ray) that is outside the ball B n (0, ασ n) (this is depicted by the shaded area on Figure
For
<v<
√
2). A first upper bound on this volume is the portion of the former ball cut off by the
√
hyperplane perpendicular to the ray and at a distance dσ n from it (i.e. a distance of
√
2
(v + d)σ n along this ray from the origin) where d = α2v − v by elementary geometry.
√ √
The latter portion is in turn included in a ball of radius σ n v 2 − d2 (that is depicted
by the dashed circle in Figure 2). In this figure, the large ball centered on the origin is
33
Capacity and Error Exponents of Stationary Point Processes
the exclusion ball of the Matérn construction around the tagged codeword. Its radius
√
is (α − ǫ)σ n. The point X is the location of the noise added to the tagged codeword.
√
√
Its norm is vσ n. The ball centered on X with radius vσ n is the vulnerability region
in the Poisson case. In the Matérn case, the vulnerability region is the shaded lune
α
2
< v < √α2 . The area of this lune is
p
upper bounded by that of the ball of radius c = n(v 2 − d2 )σ, with d as above. This
√
ball is represented by the dashed line disc. Hence, c(v) = v 2 − d2 . This completes
depicted on the figure. This is the case with
the proof of Equations (52)–(53).
v
1
0
1
0
0
X
c
d
Figure 2: The Matérn case with white Gaussian noise
By the same arguments as in the Poisson case (see Section 10.3 of [2]),
π(Mat, L(AWGN), α, AWGN) ≥ inf b(v) + a(v) ,
v>0
with a(v) =
v2
2
−
1
2
− ln v and





b(v) =
ln α −




1
2
∞
ln(v 2 − (v −
(ln α − ln v)+
if 0 < v <
α
e2 2
2v ) )
if
α
e
2
if
<v<
α
e
√
2
α
e
2
α
e
√
2
<v .
(54)
34
Venkat Anantharam and François Baccelli
The bound (51) follows when minimizing over v for each α
e ≥ 2 and then letting ǫ to 0.
The lower bound on η(α) given in (49) and (51), namely
π(α) ≥









α2
2
1
2
−
1
2
− ln α
− ln 2 + ln α
α2
8
√
if 1 ≤ α < 2
√
if 2 ≤ α < 2
(55)
if α ≥ 2 .
was first obtained by Poltyrev [17] (see eqns. (32) and (36) therein).
9. Mismatched Decoding
A scenario of interest in applications is that of mismatched decoding, where the
e In
decoder has been designed for some noise ∆ but the actual noise is in fact ∆.
e are real-valued centered stationary and ergodic stochastic
the next theorem, ∆ and ∆
processes.
By the same arguments as in the matched case, one gets the following result:
Theorem 8. For all stationary and ergodic point processes µn , all ∆ and actual dise k }k (independent from point to point), the probability
placement vectors governed by {∆
of error under ML decoding done under the belief that the law of the displacements is
governed by the law of ∆ satisfies
Z
Pn0 ((µn − ǫ0 )(F (xn )) > 0)fen (xn )dxn .
pe (n) ≤
(56)
xn ∈Rn
If µn is Poisson of intensity λn , then
Z
n
(1 − exp (−λn W∆
(u))) ρn∆
pe (n) ≤
e (du) ,
(57)
u∈R
1
n
en e n
where ρn∆
e (du) is the law of the random variable − n ln(f (D )) on R and W∆ is the
log-likelihood level volume for ∆.
The random coding exponent for mismatched decoding is given as follows.
Theorem 9. Assume that µn is Poisson with normalized logarithmic intensity −h(∆)−
ln(α) with α > 1 and that the decoder uses ML decoding under the belief that the law of
the displacement vectors is that of ∆ while the actual displacement vectors are governed
35
Capacity and Error Exponents of Stationary Point Processes
e k } (independent from point to point). Suppose that assumptions H-SEN hold for
by {∆
e Then the associated error exponent is bounded from below by
both ∆ and ∆.
n
o
e
inf F (u) + I(u)
,
u
e
where I(u)
is the rate function of ρn∆
e and
F (u) = (ln(α) + h(∆) − J(u))
+
(58)
,
where J(u) = sups≤u (s − I(s)) is the volume exponent for ∆.
Acknowledgements
The work of the first author was supported by NSF grants CCF-0500234, CNS0627161 and CCF-0635372, by Marvell Semiconductor, and by the University of California MICRO program during the initial stages of this project, and is current supported by the ARO MURI grant W911NF-08-1-0233, “Tools for the Analysis and
Design of Complex Multi-Scale Networks”, by the NSF grant CNS-0910702, and by
the NSF Science & Technology Center grant CCF-0939370, “Science of Information”,
This work started when the second author was a Visiting Miller Professor at UC
Berkeley.
The work of both authors was supported by the IT-SG-WN Associate Team of
the Inria@SiliconValley programme. Both authors would also like to thank the Isaac
Newton Institute for providing a wonderful environment during Spring 2010 as part of
the program “Stochastic Processes in Communications Sciences”, where much of the
final part of this work was done.
References
[1] V. Anantharam and F. Baccelli, “A Palm Theory Approach to Error Exponents”, Proceeding of
the IEEE International Symposium on Information Theory, Toronto, Canada, July 6 -11, 2008,
pp. 1768 -1772.
[2] V. Anantharam and F. Baccelli, “Information-Theoretic Capacity and Error Exponents of
Stationary Point Processes under Random Additive Displacements”, arXiv:1012.4924v1 [cs.IT]
22 Dec 2010.
36
Venkat Anantharam and François Baccelli
[3] A. E. Ashikhmin, A. Barg, and S. N. Litsyn, “A New Upper Bound on the Reliability Function
of the Gaussian Channel”. IEEE Trans. on Inform. Theory, Vol. 46, No. 6, pp. 1945 -1961, Sep.
2000.
[4] A. Barron, “The Strong Ergodic Theorem for Densities: Generalized Shannon-McMillan Breiman
Theorem”, The Annals of Probability, Vol. 13, No. 4, pp. 1292 -1303, 1985.
[5] T. M. Cover and J. A. Thomas, Elements of Information Theory, John Wiley and Sons Inc.,
New York, 1991.
[6] D. J. Daley and D. Vere-Jones An Introduction to the Theory of Point Processes, Springer, New
York, 1988.
[7] A. Dembo and O. Zeitouni, Large Deviation Techniques and Applications, Jones and Bartlett,
Boston, 1993.
[8] R. G. Gallager, Information Theory and Reliable Communication. John Wiley and Sons Inc.,
New York, 1968.
[9] A. El Gamal and Y.-H. Kim, Network Information Theory, Cambridge University Press, 2011.
[10] R. M. Gray, Toeplitz and Circulant Matrices: a Review, NOW publishers.
[11] T. S. Han, Information Spectrum Methods in Information Theory, Springer Verlag, 2003.
[12] O. Kallenberg, Random Measures, 3rd ed. Akademie-Verlag, Berlin, and Academic Press,
London, 1983.
[13] J. C. Kieffer, “A Simple Proof of the Moy-Perez Generalization of the Shannon-McMillan
Theorem”, Pacific Journal of Mathematics, Vol. 51, No. 1, pp. 203 -206, 1974.
[14] B. Matérn. Spatial Variation. Meddelanden Statens Skogforkningsinstitut, Vol. 49 (1960). Second
edition, Springer, Berlin, 1986.
[15] G. Matheron, Random Sets and Integral Geometry, Wiley, New York, 1975.
[16] J. Møller. Lectures on random Voronoi tessellations, volume 87 of Lect. Notes in Statist.
Springer-Verlag, 1994.
[17] G. Poltyrev, “On Coding Without Restrictions for the AWGN Channel”. IEEE Trans. on
Inform. Theory, Vol. 40, No. 2, pp. 409-417, Mar. 1994.
[18] C. E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal,
Vol. 27, pp. 379 -423 and pp. 623 -656, 1948.
[19] C. E. Shannon, “Probability of Error for Optimal Codes in a Gaussian Channel”. Bell System
Technical Journal, Vol. 38, No. 3, pp. 611 -656, May 1959.
[20] S.R.S. Varadhan Large Deviations and Applications, SIAM, 1984.