Classification of States
Throughout the state space is either a finite set {1, . . . , N } or the countably infinite set Z+ ≡
(0)
(m)
(1)
{1, . . P
. , n, . . . }. Further, ρi = 0, ρi for m ≥ 1 is the time of the mth return to i, ρi ≡ ρi , and
∞
Ti ≡ n=0 1{i} (Xn ) is the total time spent at i.
Lemma 1. For each i and m ≥ 0,
m
(m)
(1)
P T i > m X 0 = i = P ρi < ∞ X 0 = i = P ρi < ∞ X 0 = i .
In particular,
(2)
P Ti = ∞ X0 = i = 1 ⇐⇒ P ρi < ∞ X0 = i = 1,
(3)
P Ti > 0 X0 = j
,
E Ti X0 = j =
P ρi = ∞ X0 = i
and so P(Ti = ∞|X0 = i) = 1 ⇐⇒ E[Ti |X0 = i] = ∞.
Proof. The first equality in (1) is obvious. To prove the second one, observe that
(m+1)
ρi
P
∞
X
(m)
(m+1)
< ∞ X0 = i =
P ρi = n & ρi
< ∞ X0 = i
n=m
=
∞
X
(m)
P ρi
(m)
= n X0 = i P ρi < ∞ X0 = i = P ρi < ∞ X0 = i P ρi < ∞ X0 = i .
n=m
Given (1), we know that P(Ti > m|X0 = i) = 1 for all m ≥ 1 if P(ρi < ∞|X0 = i) = 1, and so
P(ρi < ∞|X0 = i) = 1 =⇒ P(Ti = ∞|X0 = i) = 1. Conversely, if P(ρi < ∞|X0 = i) < 1, then
∞
∞
X
X
m
E Ti X0 = i =
P Ti > m X0 = i =
P ρi < ∞ X 0 = i =
m=0
m=0
1
< ∞.
P(ρi = ∞|X0 = i)
Finally, if j 6= i,
∞
∞
X
X
E Ti X0 = j =
E Ti 1{n} (ρi ) X0 = j =
E Ti X0 = i P ρi = n X0 = j
n=1
n=1
= E T i X 0 = i P ρi < ∞ X 0 = j = E T i X 0 = i P T i > 0 X 0 = j . The state i is recurrent if P(ρi < ∞|X0 = i) = 1. Equivalently, by (2) and (3),
(4)
i is recurrent ⇐⇒ P(Ti = ∞|X0 = i) = 1 ⇐⇒ E Ti X0 = i = ∞.
We write i→j if (Pn )ij > 0 for some n ≥ 0 and i↔j if i→j and j→i.
Lemma 2. Assume that i is recurrent and that j 6= i. Then i→j if and only if P(ρj < ρi |X0 =
i) > 0. Moreover, if i→j, then P(ρk < ∞|X0 = `) = 1 for all k, ` ∈ {i, j}, and so i↔j and j is
recurrent.
Proof. Assuming that j 6= i, we have that
(m+1)
P ρi
∞
X
(m)
(m+1)
< ρj X0 = i =
P ρi = n & ρi
< ρj X0 = i
n=1
=
∞
X
(m)
P ρi
(m)
= n < ρj X0 = i P ρi < ρj X0 = i = P ρi < ρj X0 = i P ρi < ρj X0 = i ,
n=1
1
2
and so
(5)
(m)
j 6= i =⇒ P ρi
m
< ρj X0 = i = P ρi < ρj X0 = i .
In particular, if P(ρj < ρi |X0 = i) = 0, then P(ρi ≤ ρj |X0 = i) = 1, and so, because P(ρj 6=
ρi |X0 = i) ≥ P(ρi < ∞|X0 = i) = 1, P(ρi < ρj |X0 = i) = 1. But, by (5), this means that
(m)
P(ρi < ρj |X0 = i) = 1 for all m ≥ 1, which, since ρ(m) ≥ m, leads to P(ρj = ∞|X0 = i) = 1 and
contradicts i→j.
(m)
Knowing that P(ρj < ρi |X0 = i) > 0 and that P(m ≤ ρi < ∞|X0 = i) = 1 for all m ≥ 1, we
know that P(ρi < ρj |X0 = i) = 1 − P(ρj ≤ ρi |X0 = i) < 1, and therefore, by (5),
m
(m)
P ρj = ∞ X0 = i = lim P ρi < ρj X0 = i = lim P ρi < ρj X0 = i = 0.
m→∞
m→∞
That is, P(ρj < ∞|X0 = i) = 1.
To prove that P(ρi < ∞|X0 = j) = 1, note that
1 = P ρi < ∞ X0 = i = P ρi < ρj X0 = i + P ρj < ρi < ∞ X0 = i ,
and so
P ρj < ρi X0 = i = P ρj ≤ ρi X0 = i = P ρj < ρi < ∞ X0 = i
= P ρi < ∞ X0 = j P ρj < ρi X0 = i ,
which, because P(ρj < ρi |X0 = i) > 0, gives the desired conclusion. In particular, j→i and so
i↔j.
Finally, because P(ρi < ∞|X0 = j) = 1 = P(ρj < ∞|X0 = i),
P ρj < ∞ X0 = j = P ρj < ρi X0 = j + P ρi < ρj < ∞ X0 = j
= P ρj < ρi X0 = j + P ρj < ∞ X0 = i P ρi < ρj X0 = j = 1,
since P(ρi = ρj |X0 = j) ≤ P(ρi = ∞|X0 = j) = 0. Periodocity
Given any non-empty subset S is integers, we will say that c ∈ Z+ is a common divisor of S
and will write c|S if every s ∈ S satisfies s = ac for some a ∈ Z. In particular, we will say thatc
divides s ∈ Z and will write c|s if c|{s}. Notice that 1|S and that if m = min |s| : s ∈ S \ {0} ,
then c|S =⇒ 1 ≤ c ≤ m. Hence, the greatest common divisor gcd(S) ≡ max{c ∈ Z+ : c|S} of S
exists
Lemma 3. Given ∅ =
6 S ⊆ Z,
gcd(S) ≤ min |s| : s ∈ S \ {0} and equality holds if and only if {−gcd(S), gcd(S)} ∩ S 6= ∅.
N
Moreover, there always exist N ≥ 1, {ai }N
1 ⊆ Z \ {0}, and {ni }1 ⊆ S such that
gcd(S) =
N
X
ai ni .
i=1
Proof. The first assertion is trivial. To prove the second, let Ŝ be the smallest subset of Z with
the properties that Ŝ ⊇ S and a1 n1 + a2 n2 ∈ S whenever a1 , a2 ∈ Z and n1 , n2 ∈ S. Then
PN
N
s ∈ Ŝ if and only if s = i=1 ai ni for some N ≥ 1, {ai }N
1 ⊆ Z, and {ni }1 ⊆ S. In particular,
gcd(Ŝ) = d ≡ gcd(S). Finally, to see that d ∈ Ŝ, it suffices to show that m ≡ min{s : s ∈ Ŝ ∩Z+ } is
a common divisor of Ŝ. To this end, suppose that s ∈ S but that m6 |s. Without loss in generality,
we will assume that s > 0. Then, s = am + r for some a ∈ Z+ and 1 ≤ r < m. But this means
that r = s − am ∈ Ŝ, which contradicts the choice of m as the smallest element of Ŝ ∩ Z+ . 3
Theorem (A). Let ∅ =
6 S ⊆ Z+ , and assume that n1 +n2 ∈ S whenever n1 , n2 ∈ S. If d = gcd(S),
then there is an m0 ≥ 1 such that
{s ∈ S : s ≥ m0 d} = {md : m ≥ m0 }.
Proof. Because d|S, it suffices for us to show the existence of an m0 ≥ 1 such that md ∈ S for
all m ≥ m0 . Clearly, there is nothing to do if d ∈ S, since then we can take
m0 = 1. Thus, we
will assume now that d ∈
/ S. Because, under the present hypotheses, Ŝ = t − s : s, t ∈ S ∪ {0} ,
Lemma 3 plus d ∈
/ S, say that d = t − s for some choice of s, t ∈ S. Because s ∈ S but d ∈
/ S,
s = ad for some a ∈ Z+ \ {1}. Thus, for any m ∈ N and 0 ≤ r < a,
(a2 + ma + r)d = ms + as + r(t − s) = (m + a − r)s + rt ∈ S,
and so we can take m0 = a2 . Set S(i) = {n ≥ 1 : (Pn )ii > 0}, and let d(i) = gcd S(i) . The number d(i) is called the period
of the state i. The state i is said to be aperiodic if d(i) = 1.
Corollary. If i↔j, then d(i) = d(j). Moreover, for each i, there exists an m(i) ≥ 1 such that
n ≥ m(i)d(i) : (Pn )ii > 0 = {md(i) : m ≥ m(i)}.
In particular, if i is aperiodic, then (Pn )ii > 0 for all sufficient large n0 s.
Proof. Suppose that i↔j, and choose k, ` ∈ N so that (Pk )ij ∧ (P` )ji > 0. Then, for any n ≥ 0
with (Pn )jj > 0, (Pk+n+` )ii ≥ (Pk )ij (Pn )jj (P` )ji > 0, and so d(i)|(k + n + `). In particular,
d(i)|(k + `), and so d(i)|S(j). Hence, d(i) ≤ d(j). After reversing the roles of i and j, we get
d(j) ≤ d(i).
To complete the proof, it suffices to check that n1 , n2 ∈ S(i) =⇒ n1 + n2 ∈ S(i). But
(Pn1 +n2 )ii ≥ (Pn1 )ii (Pn2 )ii , and so n1 , n2 ∈ S(i) =⇒ n1 + n2 ∈ S(i). A transition probability matrix P is said to be irreducible if, for each pair (i, j), there exists an
n ≥ 0 such that (Pn )ij > 0. That is, irreducibility is the statement that i↔j for all (i, j). By the
Corollary to Theorem (A), irreducibility implies that d(i) is independent of i. Thus, we can talk
about the period of an irreducible P.
Corollary. If P is an irreducible, aperiodic transition probability matrix on a finite state space,
−1
there exists an M ≥ 1 such that (PM )ij > 0 for all (i, j). Hence, if πj = E[ρj |x0 = j]
, then
π is the unique probability vector µ satisfying µP = µ. In fact, there exists a θ ∈ [0, 1) such that
kνPn − πk1 ≤ 2θn−1 for all n ≥ 0 and all probability vectors ν.
Proof. Because of the Corollary to Theorem (A) together with Theorem (D) in the Doeblin theory,
all that we need to do here is check the first assertion. To this end, note that, because the state
space is finite and each state is aperiodic, the preceding Corollary guarantees that there is an m0
such that (Pm )jj > 0 for all m ≥ m0 and all j. Furthermore, for each (i, j), there exists an
m(i, j) ≥ 0 such that (Pm(i,j) )ij > 0, and so, for any m ≥ m0 ,
(Pm(i,j)+m )ij ≥ (Pm(i,j) )ij (Pm )jj > 0.
Thus, we can take M = m0 + max{m(i, j) : (i, j)}.