When is a Markov chain regenerative?
Krishna B. Athreya and Vivekananda Roy
Iowa State University
Ames, Iowa, 50011, USA
Abstract
A sequence of random variables {Xn }n≥0 is called regenerative if it can be broken up into
iid components. The problem addressed in this paper is to determine under what conditions is a
Markov chain regenerative. It is shown that an irreducible Markov chain with a countable state
space is regenerative for any initial distribution if and only if it is recurrent (null or positive).
An extension of this to the general state space case is also discussed.
1
Introduction
A sequence of random variables {Xn }n≥0 is regenerative (see formal definition below) if it can
be broken up into iid components. This makes it possible to apply the laws of large numbers for iid
random variables to such sequences. In particular, in Athreya and Roy (2012) this idea was used
to develop Monte Carlo methods for estimating integrals of functions with respect to distributions
π that may or may not be proper (that is, π(S) can be ∞ where S is the underlying state space).
A regenerative sequence {Xn }n≥0 , in general, need not be a Markov chain. In this short paper we
address the question of when a Markov chain is regenerative.
We now give a formal definition of a regenerative sequence of random variables.
Definition 1. Let (Ω, F, P ) be a probability space and (S, S) be a measurable space. A sequence
of random variables {Xn }n≥0 defined on (Ω, F, P ) with values in (S, S) is called regenerative if
there exists a sequence of (random) times 0 < T1 < T2 < . . . such that the excursions {Xn : Tj ≤
Key words and phrases. Harris recurrence, Markov chains, Monte Carlo, recurrence, regenerative sequence.
1
n < Tj+1 , τj }j≥1 are iid where τj = Tj+1 − Tj for j = 1, 2, . . . , that is,
P (τj = kj , XTj +q ∈ Aq,j , 0 ≤ q < kj , j = 1, . . . , r)
r
Y
=
P (τ1 = kj , XT1 +q ∈ Aq,j , 0 ≤ q < kj ),
j=1
for all k1 , . . . , kr ∈ N, the set of positive integers, Aq,j ∈ S, 0 ≤ q < kj , j = 1, . . . , r, and r ≥ 1.
The random times {Tn }n≥1 are called regeneration times.
From the definition of a regenerative sequence and the strong law of large numbers the next remark follows and Athreya and Roy (2012) used it as a basis for constructing Monte Carlo estimators
for integrals of functions with respect to improper distributions.
Remark 1. Let {Xn }n≥0 be a regenerative sequence with regeneration times {Tn }n≥1 . Let
π(A) = E
TX
2 −1
IA (Xj )
for A ∈ S.
(1)
j=T1
(The measure π is known as the canonical (or, occupation) measure for {Xn }n≥0 .) Let Nn = k if
Tk ≤ n < Tk+1 , k, n = 1, 2, . . . . Suppose f, g : S → R be two measurable functions such that
R
R
R
S |f |dπ < ∞ and S |g|dπ < ∞, S gdπ 6= 0. Then, as n → ∞,
(i) (regeneration estimator)
Pn
λ̂n =
(ii) (ratio estimator) and
j=0 f (Xj ) a.s.
Nn
Pn
R̂n =
Z
−→
f (Xj ) a.s.
Pj=0
−→
n
j=0 g(Xj )
f dπ,
S
R
f dπ
RS
.
S gdπ
Athreya and Roy (2012) showed that if π happens to be improper then the standard time average
P
estimator, nj=0 f (Xj )/n, based on Markov chains {Xn }n≥0 with π as its stationary distribution
will converge to zero with probability 1, and hence is not appropriate. On the other hand, the
R
R
regeneration and ratio estimators, namely λ̂n and ( S gdπ)R̂n , (assuming S gdπ is known) deR
fined above produce strongly consistent estimators of S f dπ and they work whether π is proper
or improper. Regeneration methods have been proved to be useful in a number of other statistical
applications. For example, in the case of proper target, that is when π(S) is finite, regenerative
method has been used to construct consistent estimates of asymptotic variance of Markov chain
2
Monte Carlo (MCMC) based estimates (Mykland, Tierney and Yu, 1995), to obtain quantitative
bounds on the rate of convergence of Markov chains (see e.g. Roberts and Tweedie, 1999), and to
construct bootstrap methods for Markov chains (see e.g. Bertail and Clémençon, 2006; Datta and
McCormick, 1993). The regeneration method has also been used in nonparametric estimation for
null recurrent time series (Karlsen and Tjøstheim, 2001).
2
The Results
Our first result shows that a necessary and sufficient condition for an irreducible countable state
space Markov chain to be regenerative for any initial distribution is that it is recurrent.
Theorem 1. Let {Xn }n≥0 be a time homogeneous Markov chain with a countable state space S =
{s0 , s1 , s2 , . . . }. Let {Xn }n≥0 be irreducible, that is, for all si , sj ∈ S, P (Xn = sj for some 1 ≤
n < ∞ | X0 = si ) > 0. Then for any initial distribution of X0 , the Markov chain {Xn }n≥0 is
regenerative if and only if it is recurrent, that is, for all si ∈ S, P (Xn = si for some 1 ≤ n <
∞ | X0 = si ) = 1.
The next result gives a sufficient condition for a Markov chain {Xn }n≥0 with countable state
space to be regenerative.
Theorem 2. Let {Xn }n≥0 be a time homogeneous Markov chain with a countable state space
S = {s0 , s1 , s2 , . . . }. Suppose there exists a state si0 ∈ S such that
P (Xn = si0 for some 1 ≤ n < ∞ | X0 = si0 ) = 1,
(2)
that is, si0 is a recurrent state. Then {Xn }n≥0 is regenerative for any initial distribution ν of X0 ,
such that P (Xn = si0 for some 1 ≤ n < ∞ | X0 ∼ ν) = 1.
Remark 2. It is known that a necessary and sufficient condition for (2) to hold is that
P∞
n=1 P (Xn
=
si0 | X0 = si0 ) = ∞ (see e.g. Athreya and Lahiri, 2006, Corollary 14.1.10).
Next we give a necessary condition for a Markov chain {Xn }n≥0 with countable state space to
be regenerative.
Theorem 3. Let {Xn }n≥0 be a time homogeneous Markov chain with a countable state space
S = {s0 , s1 , s2 , . . . } and initial distribution ν of X0 . Suppose {Xn }n≥0 is regenerative. Then,
3
there exists a nonempty set S0 ⊂ S such that for all si , sj ∈ S0 , P (Xn = sj for some 1 ≤ n <
∞ | X0 = si ) = 1.
Remark 3. Under the hypothesis of Theorem 3, the Markov chain {Xn }n≥0 is regenerative for any
initial distribution of X0 such that P (Xn ∈ S0 for some 1 ≤ n < ∞) = 1.
Next we discuss the case when {Xn }n≥0 is a Markov chain with a general state space S that need
not be countable. Our first result in this case provides a sufficient condition similar to Theorem 2.
Theorem 4. Let {Xn }n≥0 be a Markov chain with state space (S, S) and transition probability
function P (·, ·). Suppose there exists a singleton ∆ in S with P (Xn = ∆ for some 1 ≤ n <
∞ | X0 = ∆) = 1, i.e., ∆ is a recurrent singleton. Then {Xn }n≥0 is regenerative for any initial
distribution ν such that P (Xn = ∆ for some 1 ≤ n < ∞ | X0 ∼ ν) = 1.
The proof of Theorem 4 is similar to that of Theorem 2 and hence is omitted.
P −1
IA (Xj ) | X0 =
Remark 4. Let ∆ be a recurrent singleton as in Theorem 4 and let π(A) ≡ E( Tj=0
∆) for A ∈ S, where T = min{n : 1 ≤ n < ∞, Xn = ∆}. As shown in Athreya and Roy (2012),
the measure π is stationary for the transition function P , that is,
Z
P (x, A)π(dx) for all A ∈ S.
π(A) =
S
Further, if ∆ is accessible from everywhere in S, that is, if for all x ∈ S, P (Xn = ∆ for some 1 ≤
n < ∞ | X0 = x) = 1, then π is unique (upto multiplicative constant) in the class of all measures
π̃ on (S, S) that are subinvariant measure with respect to P , that is,
Z
π̃(A) ≥
P (x, A)π̃(dx) for all A ∈ S.
S
Remark 5. Let {Xn }n≥0 be a Harris recurrent Markov chain with state space S, that is, there exists
a σ-finite measure φ(·) on (S, S) such that P (Xn ∈ A for some 1 ≤ n < ∞ | X0 = x) = 1 for
all x ∈ S and all A such that φ(A) > 0. Assume that S is countably generated. Then there exists
n0 ≥ 1 such that the chain {Xnn0 }n≥0 , is equivalent, in a suitable sense, to a regenerative sequence.
(This has been independently shown by Athreya and Ney (1978) and Numelin (1978), see e.g. Meyn
and Tweedie (1993, Section 17.3.1).)
4
Our next result provides another sufficient condition for {Xn }n≥0 to regenerative.
Theorem 5. Let {Xn }n≥0 be a time homogeneous Markov chain with a general state space (S, S)
and Markov transition function P (·, ·). Suppose,
(i) there exists a S measurable function s(·) : S → [0, 1], and a probability measure Q(·) on
(S, S) such that
P (x, A) ≥ s(x)Q(A), for all x ∈ S and A ∈ S.
(3)
The above is known as the so-called minorisation condition.
(ii) Also, assume that
∞ Z Z
X
n=1 S
s(y)P n (x, dy) Q(dx) = ∞.
(4)
S
Then, {Xn }n≥0 is regenerative if X0 has distribution Q.
The next result gives a necessary condition for {Xn }n≥0 to be regenerative.
Theorem 6. Let {Xn }n≥0 be a time homogeneous Markov chain with a general state space (S, S).
Assume that
(i) {Xn }n≥0 is regenerative, and
(ii) for all j ≥ 0, the event {T2 − T1 > j} is measurable with respect to σ{XT1 +r : 0 ≤ r ≤ j},
the σ-algebra generated by {XT1 , XT1 +1 , · · · , XT1 +j }.
Let π be the occupation measure as defined in (1). Then π is σ-finite, and for all A ∈ S with
π(A) > 0,
P (Xn ∈ A for some 1 ≤ n < ∞ | X0 = x) = 1 for π almost all x.
(5)
Definition 2. A Markov chain {Xn }n≥0 is said to be weakly Harris recurrent if there exists a σfinite measure π on (S, S) such that (5) holds for all A with π(A) > 0.
Remark 6. The above Theorem 6 shows that any regenerative Markov chain is necessarily weakly
Harris recurrent. A natural open question is whether a weakly Harris recurrent Markov chain is
necessarily regenerative for some initial distribution of X0 . Note that Theorem 6 is a version of
Theorem 3 for general state space Markov chains.
5
Remark 7. The condition (ii) in Theorem 6 is satisfied in the usual construction of regeneration
schemes (see proof of Theorem 5 in Section 3).
3
Proofs of Theorems
Proof of Theorem 1
Proof. First we assume that the Markov chain {Xn }n≥0 is irreducible and recurrent. Fix si0 ∈ S.
Define T1 = min{n : n ≥ 1, Xn = si0 }, and for j ≥ 2, Tj = min{n : n > Tj−1 , Xn = si0 }.
Since {Xn }n≥0 is irreducible and recurrent, for any initial distribution, P (Tj < ∞) = 1 for
all j ≥ 1 (see e.g. Athreya and Lahiri, 2006, Section 14.1). Also by strong Markov property,
ηj ≡ {Xn , Tj ≤ n < Tj+1 , Tj+1 − Tj }j≥1 are iid with XTj = si0 , that is, {Xn }n≥0 is regenerative,
for any initial distribution.
Next we assume that {Xn }n≥0 is regenerative with X0 ∼ ν, regeneration times Tj ’s and occupation measure π as defined in (1). Note that π(S) = E(T2 − T1 ) > 0. So there exists si0 ∈ S
PTr+1 −1
such that π(si0 ) > 0. Define ξr (i0 ) = j=T
I{si0 } (Xj ) ≡ number of visits to si0 during the rth
r
cycle for r = 1, 2, . . . . Since ξr (i0 ), r = 1, 2, . . . are iid with E(ξr (i0 )) = π(si0 ), by SLLN, we
P
have nr=1 ξr (i0 )/n → π(si0 ) with probability 1 as n → ∞. Since π(si0 ) > 0, this implies that
P∞
r=1 ξr (i0 ) = ∞ with probability 1, which, in turn, implies that
Pν (ξr (i0 ) > 0 for infinitely many r) = 1,
where Pν is the distribution of {Xn }n≥0 with X0 ∼ ν. The above implies that Pν (Xn = si0 i. o.) =
1, that is, si0 is a recurrent state. Since {Xn }n≥0 is irreducible, it follows that all states are recurrent.
Proof of Theorem 2
Proof. If P (X0 = si0 ) = 1, then since si0 is recurrent, {Xn }n≥0 is regenerative with iid cycles
ηj , j ≥ 0 as defined in the proof of Theorem 1. Now if the initial distribution ν is such that
P (Xn = si0 for some 1 ≤ n < ∞ | X0 ∼ ν) = 1, then {Xn }n≥0 is regenerative with the same iid
cycles ηj , j ≥ 1 as above.
Proof of Theorem 3
6
Proof. Let π be as defined in (1) and let S0 ≡ {si : si ∈ S, π(si ) > 0}. Then as shown in
the second part of proof of Theorem 1, S0 is nonempty and for all si ∈ S0 , Pν (Ai ) = 1, where
Ai ≡ {Xn = si for infinitely many n}. This implies, for any si , sj ∈ S0 , Pν (Ai ∩ Aj ) = 1,
which, in turn, implies by the strong Markov property that P (Xn = sj i. o. | X0 = si ) = 1 for all
si , sj ∈ S0 .
Proof of Theorem 5
Proof. Following Athreya and Ney (1978) and Numelin (1978), we construct the so-called ‘split
chain’ {(δn , Xn )}n≥0 using (3) and the following recipe. At step n ≥ 0, given Xn = xn , generate
(δn+1 , Xn+1 ) as follows. Generate δn+1 ∼ Bernoulli (s(xn )). If δn+1 = 1, then draw Xn+1 ∼
Q(·); otherwise if δn+1 = 0, then draw Xn+1 ∼ R(xn , ·), where R(x, ·) ≡ {1 − s(x)}−1 {P (x, ·) −
s(x)Q(·)} for s(x) 6= 1, and if s(x) = 1 define R(x, ·) = 0. Note that the marginal {Xn } sequence
of the split chain {(δn , Xn )}n≥0 is a Markov chain with state space S and transition function P (·, ·).
We observe that for any 1 ≤ n < ∞, A ∈ S, P (Xn ∈ A | δn = 1, (δj , Xj ), j ≤ n − 1) = Q(A).
Let T = min{n : 1 ≤ n < ∞, δn = 1} and T = ∞ if δn 6= 1 for all 1 ≤ n < ∞. Let
θ = P (T < ∞ | X0 ∼ Q). If θ = 1, then the above construction shows that the Markov chain
{Xn }n≥0 with X0 ∼ Q is regenerative. Now we establish that θ = 1 if and only if (4) holds. (For
the proof of the theorem we only need the “if” part.)
P
Let N = ∞
n=1 δn be the total number of regenerations. Let δ0 = 1, X0 ∼ Q. Then it can be
shown that P (N = 0) = 1 − θ, and P (N = k) = (1 − θ)θk , for all k ≥ 1. Thus if θ < 1, then
E(N | X0 ∼ Q) =
∞
X
k(1 − θ)θk = θ/(1 − θ) < ∞.
k=0
On the other hand if θ = 1, then P (N = ∞) = 1. Hence θ = 1 if and only if E(N | X0 ∼ Q) =
∞. But,
E(N | X0 ∼ Q) =
∞
X
P (δn = 1) =
∞ Z Z
X
n=1 S
n=1
s(y)P n (x, dy) Q(dx).
S
So if (4) holds then θ = 1 and the Markov chain {Xn }n≥0 is regenerative when X0 ∼ Q. The proof
of Theorem 5 is thus complete.
Proof of Theorem 6
7
Proof. As noted in the proof of Theorem 1, 0 < π(S) = E(T2 − T1 ) ≤ ∞, and since T2 − T1 is an
integer valued random variable, π is a nontrivial σ-finite measure. Fix A such that π(A) > 0. Then
arguing as in the proof of Theorem 1,
Pν (ξr (A) > 0 for infinitely many r, 1 ≤ r < ∞) = 1,
where X0 ∼ ν, and ξr (A) =
PTr+1 −1
j=Tr
(6)
IA (Xj ) ≡ number of visits to A during the rth cycle for
r = 1, 2, . . . . This implies Pν (B) = 0, where B ≡ {Xn ∈ A for only finitely many n ≥ T1 }.
This, in turn, implies that for all j ≥ 0, Pν (B ∩ (T2 − T1 > j)) = 0. But,
Pν (B ∩ (T2 − T1 > j)) = Eν IB I(T2 −T1 >j) = Eν IBj I(T2 −T1 >j) ,
where Bj ≡ {Xn ∈ A for only finitely many n ≥ T1 + j} for j = 0, 1, . . . . By hypothesis (ii) and
the (strong) Markov property, we have
Eν IBj I(T2 −T1 >j)
= Eν I(T2 −T1 >j) Eν IBj | σ{XT1 +r : 0 ≤ r ≤ j}
= Eν I(T2 −T1 >j) ψ(XT1 +j ) ,
where ψ(x) = P (Xn ∈ A for only finitely many n ≥ 1 | X0 = x). Since Pν (B ∩ (T2 − T1 >
j)) = 0 for all j = 0, 1, . . . , it follows that
∞
X
Eν I(T2 −T1 >j) ψ(XT1 +j ) = 0.
j=0
But,
∞
X
Eν I(T2 −T1 >j) ψ(XT1 +j ) = Eν
j=0
Since
R
S
TX
2 −1
j=T1
Z
ψ(Xj ) =
ψ(x)π(dx).
S
ψ(x)π(dx) = 0, it implies that ψ(x) = 0 for π almost all x, that is, P (Xn ∈ A for infinitely many n ≥
1 | X0 = x) = 1 for π almost all x. This conclusion is stronger than (5).
Acknowledgments
The authors thank an anonymous reviewer and an editor for helpful comments and suggestions.
References
ATHREYA , K. B. and L AHIRI , S. N. (2006). Measure Theory and Probability Theory. Springer,
New York.
8
ATHREYA , K. B. and N EY, P. (1978). A new approach to the limit theory of recurrent Markov
chains. Transactions of the American Mathematical Society, 245 493–501.
ATHREYA , K. B. and ROY, V. (2012). Monte carlo methods for improper target distributions. Tech.
rep., Iowa State University.
B ERTAIL , P. and C L ÉMENÇON , S. (2006). Regenerative block bootstrap for Markov chains.
Bernoulli, 12 689–712.
DATTA , S. and M C C ORMICK , W. (1993). Regeneration-based bootstrap for Markov chains. The
Canadian Journal of Statistics, 21 181–193.
K ARLSEN , H. A. and T JØSTHEIM , D. (2001). Nonparametric estimation in null recurrent time
series. The Annals of Statistics, 29 372–416.
M EYN , S. P. and T WEEDIE , R. L. (1993). Markov Chains and Stochastic Stability. Springer
Verlag, London.
M YKLAND , P., T IERNEY, L. and Y U , B. (1995). Regeneration in Markov chain samplers. Journal
of the American Statistical Association, 90 233–41.
N UMELIN , E. (1978). A splitting technique for Harris recurrent Markov chains. Z. Wahrsch. Verw.
Gebiete, 43 309–318.
ROBERTS , G. and T WEEDIE , R. (1999). Bounds on regeneration times and convergence rates for
Markov chains. Stochastic Processes and their Applications, 80 221–229. Corrigendum (2001)
91 : 337–338.
9