CSE 736: Markov Chains
SUNY Buffalo, Spring 2003
Lecturer: Hung Q. Ngo
Scribe: Peng Lin
Lecture 6: Convergence to equilibrium, Ergodic Theorem
1 Definitions
Definition 1.2 (Invariant measure and distribution). Let X be an irreducible recurrent markov chain
, then is called an invariant measure
with transition matrix P, if there exists some vector s.t.
with respect to for . Similarly, if it holds that
#%$&'($
"!
We first give out the following definitions:
consisting of coordinates
Definition 1.1 (Measures and distributions). A vector
,
, is called a measure on . If the sum of all coordinates is , then it is called a distribution.
, it’s called an invariant distribution.
With the above mentioned properties of invariant distribution, we can come up with the following
propositions:
2143 ) )5*,+
) )(*,+
.-/0
-/6
Proposition 1.3. If
is a Markov chain with
and is invariant with respect to , then
is also a Markov chain with
for any N. It’s also called stationary, equilibrium.
78, . By applying the property recurDistribution of (1)
1 3 ) 1 ',29 1;:< =', 1>:< @?A?A?B'
Proof. From the definition of invariant distribution, we have
sively, it is easy to see that
2%C$ )ED
F$ as$ GIQNHKJ B -MLNOP , i.e., the probability of ending up at N
is an invariant distribution.
F F
Proof. We want to show that
F $R "! F #%$
T E) UWV %C$ )ED F $ , we have
, Since by definition, S
F 9#Y$ S T )EUWV [ C )"D M\%$
X!
Z"!
S T )EUWV X! [ C )"D M\%$
S T )EUWV [ C $) 3#< D
F$
Proposition 1.4. Suppose
tends to
independent on the starting state, then
1
is
]
_` ]
^
1
_` ^
2
Figure 1: A two-state Markov chain
Example 1.5 (convergency). If a Markov chain is convergent, it must have an invariant distribution.
Let’s consider a two-state Markov chain as illustrated in Figure 1. The transition matrix
ba dcIf e e fhg
dc
f
f
i
e
, we have
Ignoring the case when e
) kj m 3l l m 3mm l
m 3l l m 3 l'n
when GoHJ .
f
From 1.3, we can see that the row components form an invariant distribution. If is larger, the final
f
p
state will most likely end up at state 1. However if e
, then will end up either at 1 or 2, and the
Markov chain will not converge.
Next, we will answer the question “when does a Markov chain have an invariant distribution”.
Theorem 1.6. If
is irreducible, then the following claims are equivalent:
1. All states are positive recurrent.
2. Some state is positive recurrent.
3.
F
has an invariant distribution .
q
q
r [ t s [.uX v : < Qy{z}| w ~
)Ewx+
Before showing the proof for the theorem, we first give out two lemmas. For a certain state , we can
find the expected time spent in state before visiting :
where
€[
q
is the first passage time to .
C[ D .
1. r [
C [ D r C [ D .
2. r
‚ r C [ D  J -MLƒ
„o .
3.
Lemma 1.8. If is irreducible,
r C[ D.
recurrent, then
Lemma 1.7. If
is irreducible and recurrent, then
is invariant measure with
[ , then I r C [ D . Moreover, if With the lemmas given above, we can show the proofs for the theorem as follows:
2
is
M†‡ M . This is obvious.
Md†ˆ M . Suppose is positive recurrent, then is recurrent. Since is positive recurrent, the
2.
8Š $‹X! r $ C D  J . Take F ŒE’ ŽY‘ , it’s
expected total number of steps before visits to state : ‰

Š
.
Z"! F . From lemma1.7, F is an invariant distribution.
easy to see
M4†K M . Take any state N , F $R Š X! F 9 YC$ )"D -ML G . And since F is a distribution, Š X! F 3.
is irreducible, i.e., “q -&” F [ -/ [ $2• , and from the definition of F $ , we have F $ • [ ,
L .N .Since
o
–‹——E˜ -5——š™ -A›A›A›œ , then is an invariant measure with [ . Hence, we have  r C D
Let
by Lemma1.8. It
v follows
v that:
[‰ $‹X! r $ C [ D ž $A"! FF $[ F [  J
• . And since F $ • -MLN , we are done.
That is, q is positive recurrent as long as F [
Proof.
1.
Next, we will look at the problem when a markov chain will converge.
First, we show that periodic is not a good property: Consider the example Markov chain given in
W¡2 a £¢2 a ¤
Ÿ
a
g.
figure 1,
g .
g . Hence the Markov chain has a
period of 2 and will not converge.
Theorem 1.9. If is irreducible, aperiodic and has an invariant distribution F , then
¥§¦§¨ 2© ªN"«x F $
)EUWV ‚)
In other words,
S ¬T )"UWV %C$ )"D F $
­ , define ® period of i p¯(° ®±‹G -/ ²C )"D • B³ . If ® =´ G , ( ´ means not divisible
Given ,
”
²C )ED i
by, means divisible), then
If ®
, we say is aperiodic.
N
\ ® $
Proposition 1.10.
1. If , are in same class, then ®
-R”Q ²C )ED • -ML G G.µ .
2. If is aperiodic, then “BG#µ
-¶T • 7” YC$ )"D -/ $¹C¸ · D • . ²C ) 3 · D • º† ® …”R G­» T .
Proof.
1. Since is irreducible, “BG
‘) 3 · 3x¼ • if ${¼ $ • £† ® $6”X½R- ® $0” G¾» T=† ® $0” ® ±‹G 5 ²C )ED • B³ then ® .t¯(° ® ¿ 4 . Note:
2. Let ¿
-¶Á¾ ¿ , then Â(ÀûªÄ Á ¿ ,L  - Ä Å 3 .
(a) If À
- -A›A›A›Q- À · ¿ , ¯(° ® À < - À ¡ -A›A›A›‹- À · W . Take À < ¿ and À < ÆW< Ç ˜ ›A›A›È [ Ç , Let
(b) “BÀ < À ¡
v
¿ < ±‹G ¿ 5 < ´ G ³2pÉ ÊB-A›A›A›Q- ¿ [ ±‹G ¿ 5 [ ´ G ³'É Ê
-A›A›A›Q- 'Å˔
?A?A?
-A›A›A›Q- • ,  [ 3#< -A›A›A›Q-  · Æ . Define
3. “Â <
Ä Â < À <  » · ?A?A? »Ì [  À < [ À Í< » cR [ 3#»t<  À · [ À3#· < » ?A?A ? . » Say cR  · < À · »p [ . Obviously,
- ÄcÎ ¿ , we can
Ä
¿ - ÏEÄ>ct ¿ - ÏEÄdcºÏ ¿ . In general, we have qÄdcªq to qÄ ¿ . If q is larger
infer that ÏEÄ
• Ä ), all consecutive numbers are covered in the range.
enough (i.e., q
3
.-/
-/
€ i G\) Ñ#be±‹G Markov(
” ¾) Ѓ),) Ð ³ ). be Markov(F ), and they are indeÅ Àƒ) -¶Á ) , Å ) )(*,+ is Markov( F -Ò ) with state space ¾´o . We have
1. Define )
2©%Å + Í /-¬N(¬«.i2© Àƒ+ i
Ó«2© Á + ªNE«x' F $
Ò C Ô $ D C [ Ô Õ D t [ $¹Õ
×ÖEØ Ä © €  J «x .
We want to show
Ò is irreducible and positive recurrent. (proof omitted.)
2.
Ö"Ø Ä © € [œ[  J « -ML q o‚† Ù ÖEØ Ä © €  J «x 3. From Lemma1.7, we know Ù
-¶Á
-¶Á
4. To show that Àƒ) ) have the “same” distribution after T. Note that À.) ) have the same transition
Now we show the proof to theorem 1.9.
Proof. We use a coupling argument. Let
pendent. Let T be a reference point:
matrix
. We have
2© Àƒ) t ¹- € ž G «k ) A$ "! 2© Àƒ) t
¹- À · ªNE- € tT…«
·w<
) 2© À · ªNÚ- € tT…«¾© À,) i
d” À · ª N"«
· w < $A"!
) $A"! 2© Á · ªNE- € tT…«2© Á ) t
”XÁ · Î"N «
·¾w © Á < ) t
/- € ž G «
2© À ) i
Ó«Û 2© À ) t
¹- € ž G « » ¾© À ) i
/- € • G «
2© Á ) t
/- € ž G « » 2© Àƒ) t
/- € • G «
ž 2© Á ) t
Ó« » 2© À ) t
/- € • G «
(a)
(b)
¾© Á ) Ü Ó« ž 2© À ) t
Ó« » 2© Á ) t ¹- € • G «
” 2© À,) t
Ó« c 2© Á ) i
Ó«M” ž 2© Àƒ) t
¹- € • G « » ¾ © Á ) t
/- € • G «
Based on the same arguments, we have
Hence,
Summing up on both sides,
” 2© À ) t « c 2 © Á ) i
Ó«M” ž Ï 2 © € • G «
X!
Taking limits as n tends to infinity, the second term on the left tends to
right hand side tends to 0. Hence, we have
¥§¦§¨ 2 © t
Ó«Ý F )EUWV ,À )
4
F , the term on the
where
Þ ß G ='Š )· :wx< + Q y{á‹â w ~ .
O)
2©Þ ß G H ² as GHàJ «x G
‰
Theorem 1.11 (Ergodic Theorem). Let
be irreducible and let
5
be Markov(
.-/
), then