Journal of Applied Mathematics and Stochastic Analysis
8, Number 4, 1995, 341-345
INVARIANT PROBABILITIES FOR
FELLER-MARKOV CHAINS 1
ONtSIMO HERNNDEZ-LERMA
CIN VESTA V- IPN
Departamento de Matemticas
A. Postal 1-70, 07000 Mxico D.F., Mxico
E-mail: ohernand@math.cinvestav.mx
JEAN B. LASSERRE
LAAS-CNRS
7 A v. du Colonel Roche
31077 Toulouse Cedex, France
E-mail:
lasserre@laas.fr
(Received April, 1995; Revised August, 1995)
ABSTRACT
We give necessary and sufficient conditions for the existence of invariant
probability measures for Markov chains that satisfy the Feller property.
Key words: Markov Chains, Feller Property, Invariant Measures.
AMS (MOS) subject classifications: 60J05, 93E15.
1. Introduction
The existence of invariant probabilities for Markov chains is an important issue for studying
the long-term behavior of the chains and also for analyzing Markov control processes under the
long-run expected average cost criterion. Inspired by the latter control problems, we present in
this paper, two necessary and sufficient conditions for the existence of invariant probabilities for
Markov chains that satisfy the Feller property. Our study extends previous results using stronger
assumptions, such as the strong Feller property in Bene [1], nondegeneracy assumptions (see
condition (2) in Bene [2]), and a uniform countable-additivity hypothesis in Liu and Susko [8].
As can be seen in the references, it is also worth noting that there are many reported results providing (only) sufficient conditions for the existence of invariant measures; in contrast however,
our conditions are also necessary.
The setting for this paper is specified in Section 2, and our main result is presented in Section
1This work is part of a research project jointly sponsored by CONACYT (Mxico) and CNRS
(France). The work of the first author was also partially supported by CONACYT grant 1332E9206.
Printed in the U.S.A.
()1995 by North
Atlantic Science Publishing Company
341
HERNNDEZ-LERMA and JEAN B. LASSERRE
ON]SIMO
342
2. Notation and Definitions
Let X be a a-compact metric space, and let
with time-homogeneous kernel P, i.e.,
{xt,
t
0, 1,...} be an X-valued Markov chain
P(BIx) Prob(x + 1 ( B Ixt x)W 0,1,..., x x, B e %(X),
where (X) denotes the Borel r-algebra of X. A probability measure (p.m.) # on (X)is said
to be invariant for P if
J
#(B)
P(B x)#(dx
VB
%(X).
x
Here, we give necessary and sufficient conditions for the existence of invariant p.m.’s when P satisfies the Feller property:
x--,
where
J
u(y)P(dy x)
is in
C(X) whenever
u
(1)
C(X),
C(X) denotes the space of all bounded and continuous functions on X. Our conditions use
a moment
function, defined as follows.
Definition: A nonnegative Borel-measurable function v on X is said to be a moment if, as
noc, inf {v(x) x gn}oo for some sequence of compact sets KnTX.
Moment functions have been used by several authors to study the existence of invariant measures for Markov processes (e.g., see Bene [1, 2], Hernndez-Lerma [6], Liu and Susko [8], and
Meyn and Tweedie [9]). The key feature used in these studies is the following (easily proved)
_
fact.
Lemma: Let M be
a
family
of p.m.’s
on
X.
If there
exists a moment v on X such that
a compact set K such
sup M f vd# < cx), then M is tight, i.e., for every positive e there exists
that #(K) > 1- e for all p M.
Therefore by Prohorov’s Theorem [3, 9], the family M in the lemma
i.e., every sequence in M contains
(b)
a
is relatively com___m_p_.t,
weakly convergent subsequence.
Our theorem below (see Section 3) extends a result by Bene [2] where our conditions (a) and
and, most importantly, we do not require Bene’ "nondegeneracy" condition, accord-
are new
ing to which
xlkrnpt(KIx)
0 for t
1,2,...,K compact,
pt(. ix being the t-step transition probability given the initial state x 0 x. This condition
excludes important classes of ergodic Markov chains, such as those that have a "m inorant " see
e.g., mynkin and Yushkevich [5], or condition RI in Hernndez-Lerma et al. [7]. See also
Remarks 2 and 3 (Section 3) for additional comments on related results.
with
3. The Theorem
If t, is a p.m. on X,
Theorem:
equivalent:
E(.
stands for the expectation given the "initial distribution"
If P satisfies the
Feller property, then the following conditions
(a), (b),
and
(c)
are
Invariant Probabilities
(a)
for Feller-Markov
Chains
343
There exists a p.m. v and a moment v such that
lim supJn(u
< cx,
whereJn(u)’-n-lE, t--E
There exists a p.m. u and a moment v such that
limsupVa() < c,
where
(c)
[kt=0ttv(xt)];
Va(u)"-(1-a)E u
There exists an invariant probability for P.
Proof: We will show that
(a)=(b)(c)=V(a).
(a)
[11],
implies (b): This follows from a well-known Abelian theorem (e.g., see Sznajder and Filar
Theorem 2.2), which states that
(Since a direct proof that
Remark 1 below.)
(b)
sure on
implies
(a)
(c): Suppose
X defined
lim supVa() <_ lim supJn( ).
n-,oo
al"l
implies (c) is surprisingly simple, it will also be included here; see
that
(b) holds and for each c E (0, 1), let Pa
be the probability mea-
as
(1 -a) a
p(B)
t=0
Then we may write
Va(u)as V c,(u
and, by (b),
pt(B z)(dz), B e (X).
X
f
Let
{c%}
limal,lSUpV,(u) =nli__,rnV(n(U) =n_..,oolim
,
be a sequence in
f vd#a
n
(0,1) such
that
c%T1
< oo.
By the lemma in Section 2, {#a } is tight and therefore, by Prohorov’s Theorem, {#a } contains
a weakly convergent subsequence which we denote by
lity measure # on X such that
lim
n
We claim that
# is invariant for
/ udiza
n
{p } again;
that is, there
ud# Vu C(X).
P.
To see this, first note that by the Markov property, we may write
#(B)
Hence, for
any u G
(1 ()u(B) + (
/ P(B
Vc
as
(0, 1), B (X).
C(X),
f
existsa probabi-
udp- (1-
c)J u(z)u(dz)+c// u(y)P(dY x)#a(dx),
ONISIMO HERN/NDEZ-LERMA and JEAN B. LASSERRE
344
f u(y)P(dy
)is in C(X). Thus, replac-
/ udlz //u(y)P(dy,x)Iz(dx).
(2)
and furthermore, note that by the Feller property
ing c by c% and letting ncx, we obtain
(1),
C(X) was arbitrary, we conclude from (2) that/ is invariant for P.
(c) implies (a): Let v be an invariant probability for P, and let {Kn} be an increasing sequence of compact sets such that KnTX and (K n + 1- Kn) < 1/n3, n- 1,2,
(Here we have
used the fact that every p.m. on a a-compact metric space is tight; see [3], p. 9.) Define a functionv(.)’-0onK landv(x):=nforxK n+l-Kn,nl. Then v is a moment and
Finally, since u E
n-2 <
v(x)u(dx)
limsuPJn(u
.
n=l
Remark 1: We will prove directly that (a) implies (c). Suppose that
1, 2,..., let #n be the probability measure on X defined as
(a)
holds and for every
[
t=O J
so that we may rewrite the condition in
(a) as
lim sup
/ vd#
J
n
< cx:)
Hence, by the lemma in Section 2, {#,} has a subsequence
probability measure u. We will show that (cf. (2))
/
where
Lu(x):= f u(y)P(dylx
0
{/,.}
which converges weakly to some
c c(x),
(3)
u(x),
thus showing that # is invariant for P.
Indeed, for any bounded measurable function u on X, the sequence
n--1
Mn(u)’-u(xn)-ELu(xt),
t--O
with
Mo(u
:=
U(Xo)
1,2,...,
n
is a martingale, which implies
Eu[Mn(u)]-- Ev[Mo(u)]Vn
C(X); replace n by
In [8], it is shown that
Finally, let u be in
Remark 2:
sup
t>l
ni; multiply by
1/ni;
and then let i--,c to get
f f g(y)pt(dylz)u(dz)<
(3).
(4)
for some moment g and initial p.m. v, is also a necessary and sufficient condition for existence of
invariant probabilities provided that the Markov chain satisfies the uniform countable-additivity
property
Invariant Probabilities for Feller-Markov Chains
lim sup
AIO
x
EK
P(A
345
x )- 0
(5)
for every compact set K in x.
Note that (4) is stronger than our condition (a) and that (5) implies: For every compact set
g C X, the family of p.m. ’s (P(. x)}x K is tight.
e
Remark 3." It is worth noting that the theorem still holds if we replace "lim sup" by "lim inf"
in both conditions (a) and (b). Now, (b):=(a) by a well-known Abelian theorem [11]. With
similar arguments as in Remark 1, (a):=v(c). We finally prove (c):=v(b) by exhibiting the same
moment function v and show that
E tv(xt
lim,nfVc(u -lim].nf(1- c)E t=O
v
n- < c.
v()u(d) <_
t=O
In conclusion, we mention that the theorem can be extended in the obvious way to continuous-time Markov processes, as in [2]. Conditions for uniqueness and ergodicity of invariant measures can be found, for instance, in [4, 7, 10] and references therein.
References
[1]
[4]
[5]
[7]
[9]
[10]
[11]
Bench, V.E., Existence of finite invariant measures for Markov processes, Proc. A mer.
Math. Soc. 18 (1967), 1058-1061.
Bene, V.E., Finite regular invariant measures for Feller processes, J. Appl. Prob. 5
(1968), 203-209.
Billingsley, P., Convergence of Probability Measures, Wiley, New York 1968.
Borovkov, A.A., Conditions for ergodicity of Markov chains which are not associated with
Harris irreducibility, Siberian Math. J. 32 (1992), 543-554.
Dynkin, E.B. and Yushkevich, A.A., Controlled Markov Processes, Springer-Verlag, New
York 1979.
tIernndez-Lerma,,O., Lyapunov criteria for stability of differential equations with Markov
parameters, Boletin Soc. Mat. Mexicana 24 (1979), 27-48.
Hernafidez-Lerma, O., Montes de Oca, R. and Cavazos-Cadena, R., Recurrence conditions
for Markov decision processes with Borel state space: a survey, Ann. Op. Res. 28 (1991),
29-46.
Liu, J. and Susko, E., On strict stationarity nd ergodicity of a nonlinear ARMA model, J.
Appl. Probab. 29 (1992), 363-373.
Meyn, S.P. and Tweedie, R.L., Markov Chains and Stochastic Stability, Springer-Verlag,
London 1993.
Skorokhod, A.V., Topologically recurrent Markov chains: ergodic properties, Theory Prob.
Appl. 31 (1986), 563-571.
Sznajder, R. and Filar, J.A., Some comments on a theorem of Hardy and Littlewood, J.
Optim. Theory Appl. 75 (1992), 201-208.