Branching Processes and Random Walks
Adam Bowditch
April 8, 2014
Contents
1 Processes
2
2 The Fundamental Matrix
8
3 Branching Processes
13
4 Random Walks
15
5 Limiting Behaviour for Markov Chains
16
6 Continuous Time Processes
16
7 Invariant Measures For Irreducible Finite Markov Chains
19
8 Reversibility
20
1
1
Processes
A discrete state space Markov chain is said to have stationary transitions if
pi,j = P(Xn+1 = j|Xn = i) = P(X1 = j|X0 = i)
∀n ∈ T , i, j ∈ S.
We then write the transition matrix as P = (pi,j )i,j∈S , the n-step transition probability as
(n)
(n)
pi,j = P(Xm+n = j|Xm = i) and the n-step transition matrix as P (n) = (pi,j )i,j∈S
A matrix is called stochastic if all entries are positive and rows sum to 1 and sub-stochastic if all
entries are positive and row sums are less than or equal to 1.
Theorem 1.1. The distribution of a discrete state space Markov chain with stationary transition
properties is fully specified by
1. Its initial state X0 or the distribution of the initial state where random.
2. Its one-step transition matrix
Proof. Consider any path {Xi = ki }m
i=0 over an initial time segment i ∈ Z|[0,m]
This path has probability
!
!
m
m−1
m−1
\
\
\
P
{Xi = ki } = P Xm = km {Xi = ki } P
{Xi = ki }
i=0
i=0
i=0
m−1
\
= P(Xm = km |Xm−1 = km−1 )P
!
{Xi = ki }
by Markov Property
i=0
=
m
Y
P(Xi = ki |Xi−1 = ki−1 )P(X0 = k0 )
i=1
Which depends only on the one-step transition matrix and the initial state.
Theorem 1.2. Chapman-Kolmogorov Equations
If X is a discrete state space Markov chain with stationary transition properties then:
P (n+m) = P (n) P (m)
or equivalently
(n+m)
pi,j
=
X
(n) (m)
pi,k pk,j
k∈S
Proof.
(n+m)
pi,j
= P(Xn+m = j|X0 = i)
!
=P
[
{Xn = k} ∩ {Xn+m = j}|{X0 = i}
k∈S
=
X
P({Xn = k} ∩ {Xn+m = j}|{X0 = i})
since disjoint
k∈S
=
X
P({Xn+m = j}|{X0 = i} ∩ {Xn = k})P(Xn = k|X0 = i)
k∈S
=
X
P(Xn+m = j|Xn = k)P(Xn = k|X0 = i)
k∈S
=
X
(n) (m)
pi,k pk,j
k∈S
2
by Markov property
(n)
Proposition 1.1. If we denote the row vector a(n) := (ani )i∈S where ai
= P(Xn = i) then
a(n) = a(0) p(n)
Proof.
(n)
ai
= P(Xn = i)
X
=
P(Xn = i|X0 = k)P(X0 = k)
k∈S
=
X
(n) (0)
pk,i ak
k∈S
= (a(0) p(n) )i
Suppose i, j ∈ S then:
(n)
• If ∃n > 0 s.t. pi,j > 0 then i → j;
• i, j intercommunicate ( i ↔ j ) if i → j and j → i;
• i, j are in the same communicating class ( i ∼ j ) if either: i = j or i ↔ j;
Proposition 1.2. ∼ is an equivalence relation
Proof.
• Reflexive:
i ∼ i since i = i
• Symmetric:
i ∼ j =⇒ i ∼ i by definition
• Transitive:
Suppose i ∼ j j ∼ k
i ∼ k is immediate if i = j or j = k so assume not.
i → j, j → k =⇒
(n) (m)
∃n, m > 0 s.t. pi,j , pj,k > 0 so
P
(n+m)
(n) (m)
pi,j
= k∈S pi,k pk,j =⇒
(n+m)
(n) (m)
pi,k
≥ pi,j pj,k > 0
hence
(l)
∃l = n + m > s.t. pi,k > hence i → k
The reverse argument obviously holds for k → i
The state i ∈ S is essential if ∀j ∈ S s.t. i → j we must have that j → i.
A transition matrix is irreducible if all states intercommunicate.
Lemma 1.1. If i → k and i is essential then i ∼ k and k is essential.
Proof. i → k, i essential =⇒ k → i
hence i ↔ k and i ∼ k.
Suppose k → j 6= i
then i → k → j hence j → i since i is essential.
So, j → i → k hence k is essential.
If a communicating class contains a single essential class then every state in the class is essential so
essential and inessential are class properties.
The period of a state i is
(
undef ined
i9i
per(i) =
(n)
gcd{n > 0 : pi,i > 0} i → i
3
Lemma 1.2. If i → i and i has period d then
∃N > 0 s.t. k ≥ N =⇒
(kd)
pi,i
>0
Theorem 1.3. Periodicity is a class property
Proof. Suppose i ↔ j and i has period d
(n) (m) (r)
∃n, m, r > 0 s.t. pi,j , pj,i , pj,j > 0
(n+m+r)
(n) (r) (m)
pi,i
≥ pi,j pj,j pj,i > 0
and
(n+m)
(n) (m)
pi,i
≥ pi,j pj,i > 0
(r)
so we have that d|n + m + r and d|n + m so d|r which is true for any r s.t pj,j > 0 hence since this
argument can be applied to the states reversed we must have that j has period d as well.
If C is a communicating class with period d then for any i ∈ C and all integers r > 0
(kd+r)
Gr (i) = {i ∈ C : pi,j
> 0}
is called a cyclically moving subclass.
Gr (i) = Gr+d (i) moreover {Gr (i)}d−1
r=0 forms a partition of C
The first passage time to state j is Tj = inf {n ≥ 1 : Xn = j} moreover the probability j is first
(n)
reached in n steps is denoted fi,j = P(Tj = n|X0 = i) and the probability of reaching j in a finite
number of steps is
∞
X
(n)
(∗)
fi,j
fi,j =
n=1
A state i is recurrent if
(∗)
fi,i
= 1 and transient otherwise.
Lemma 1.3.
(∗)
fi,j = pi,j +
X
(∗)
pi,k fk,j
k∈S
Proof.
(1)
fi,j = pi,j
(n+1)
fi,j
= P(Tj = n + 1|X0 = i)
X
=
P({Tj = n + 1} ∩ {X1 = k}|X0 = i)
k∈S
=
X
P(Tj = n + 1|{X0 = i} ∩ {X1 = k})P(X1 = k|X0 = i)
k∈S
=
X
P(Tj = n + 1|X1 = k)P(X1 = k|X0 = i)
Markov property
k∈S
=
X
P(Tj = n|X0 = k)pi,k
stationary transitions
k∈S
=
X
(n)
fk,j pi,k
k∈S
=⇒
(∗)
fi,j = pi,j +
X
(∗)
pi,k fk,j
k∈S
4
Lemma 1.4.
(n)
pi,j =
n
X
(m) (n−m)
fi,j pj,j
m=1
Proof.
(n)
pi,j = P(Xn = j|X0 = i)
=
=
=
=
n
X
m=1
n
X
m=1
n
X
m=1
n
X
P({Xn = j} ∩ {Tj = m}|X0 = i)
P(Xn = j|Tj = m)P(Tj = m|X0 = i)
P(Xn = j|Xm = j)P(Tj = m|X0 = i)
(n−m) (m)
fi,j
pj,j
m=1
We define the first-step decomposition generating functions as
Pi,j (s) =
Fi,j (s) =
∞
X
n=0
∞
X
(n)
pi,j sn
(n)
fi,j sn
n=1
Lemma 1.5.
(0)
Pi,j (s) = pi,j + Pj,j (s)Fi,j (s)
Proof.
(0)
Pi,j (s) = pi,j +
∞
X
(n)
pi,j sn
n=1
(0)
∞ n−1
X
X
(0)
n=1 r=0
∞ X
∞
X
= pi,j +
= pi,j +
(n−r) n−r (r) r
s
pj,j s
fi,j
(n−r) n−r (r) r
s
pj,j s
fi,j
r=1 n=r+1
=
(0)
pi,j
+
∞
X
(r)
pj,j sr
r=1
=
(0)
pi,j
+
∞
X
∞
X
!
(n−r)
fi,j sn−r
n=r+1
!
(r)
pj,j sr
r=1
∞
X
n=1
(0)
= pi,j + Pj,j (s)Fi,j (s)
5
!
(n)
fi,j sn
Lemma 1.6. The expected number of visits to state j given we start in state j is
∞
X
∞
X
(n)
pj,j =
n=0
n=0
∞
X
=
P(Xn = j|X0 = j)
E[1{Xn =j} |X0 = j]
n=0
∞
hX
i
=E
1{Xn =j} |X0 = j
n=0
==
1
(∗)
1 − fj,j
Theorem 1.4. The state j is recurrent precisely when
∞
X
(n)
pj,j = ∞
n=0
(∗)
Proof. j recurrent ⇐⇒ fj,j = 1 ⇐⇒ limz→1 Fj,j (z) = 1
Pj,j (s) = 1 + Fj,j (s)Pj,j (s)
=⇒
Fj,j (s) =
Pj,j (s) − 1
Pj,j (s)
so
lim Fj,j (z) = 1 ⇐⇒ lim Pj,j (z) = ∞
z→1
z→1
which occurs iff
∞
X
(n)
pj,j = ∞
n=0
Corollary 1.1. If j is transient then
(n)
lim p
n→∞ i,j
=0
∀i ∈ S
P∞ (n)
(n)
Proof. Suppose i = j then j transient implies that n=0 pj,j < ∞ which ensures that limn→∞ pi,j = 0
If i 6= j then the generating function relationship is
Pi,j (s) = Pj,j (s)Fi,j (s)
so we have that
∞
X
n=0
(n)
pi,j = lim Pi,j (z)
z→1
= lim Pj,j (z)Fi,j (z)
z→1
= Pj,j (1)Fi,j (1)
j is transient so Pj,j (1) < ∞ and Fi,j (1) ≤ 1 since it is a probability generating function of an r.v
hence
have that
P∞ we
(n)
n=0 pi,j < ∞ hence
(n)
limn→∞ pi,j = 0
6
Theorem 1.5. Recurrence and transience are class properties.
Proof. Suppose i ↔ j then
(n) (m)
∃n, m > 0 s.t. pi,j , pj,i > 0
hence
(n+m+r)
(n) (r) (m)
pj,j
≥ pi,j pi,i pj,i > 0
which implies
∞
X
(k)
pj,j ≥
k=0
P∞
(r)
r=0 pi,i
If i is recurrent then
∞
X
(n+m+r)
pj,j
(n) (m)
≥ pi,j pj,i
r=0
= ∞ =⇒
∞
X
(r)
pi,i
r=0
P∞
(k)
k=0 pj,j
= ∞ hence j is recurrent.
(∗)
(∗)
Theorem 1.6. If i is recurrent and i → j then i ↔ j and fi,j = fj,i = 1
(∗)
Proof. Since i is recurrent we have that fi,i = 1
(n)
i → j =⇒
∃n > 0 s.t. pi,j > 0
then we have
X
(∗)
(∗)
fk,i = pk,i +
pk,l fl,i
l6=i
=
X
=
X
(∗)
(∗)
pk,l fl,i
since fi,i = 1
l∈S
X
pk,l
(∗)
pl,m fm,i
m∈S
l∈S
!
=
X
X
m∈S
=
pk,l pl,m
(∗)
fm,i
l∈S
(2) (∗)
pk,m fm,i
X
m∈S
=
(r)
X
(∗)
∀r ≥ 0 by induction
pk,m fm,i
m∈S
taking k = i, r = n we get
X (n) (∗)
(∗)
1 = fi,i =
pi,m fm,i
m∈S
X
0=1−
(n)
(∗)
pi,m fm,i
m∈S
=
X
(n)
(∗)
pi,m (1 − fm,i )
m∈S
(∗)
fj,i
=1
hence j → i
Lemma 1.7.
• Recurrent =⇒ Essential
• Inessential =⇒ Transient
Theorem 1.7. If C is a finite essential class then it is recurrent.
Proof. Suppose C = N|[1,m] is essential
(n)
then if i ∈ C and pi,j > 0 then j ∈ C
so for each n
7
1=
X
(n)
pi,k =
k∈S
but if C were transient we would have that
impossible.
2
X
(n)
pi,k
k∈C
(n)
limn→∞ pi,k
= 0 but since the sum is finite this is
The Fundamental Matrix
The fundamental matrix of a sub-stochastic matrix Q is
G=
∞
X
Qk
k=0
Lemma 2.1. If P is a transition matrix and G is the fundamental matrix of P then Gi,j is the
expected number of times the particle will be in j given the particle started in i
Proposition 2.1.
Gi,j

∞



= 0 (∗)


 fi,j
i → j, j recurrent
ı9j
i → j, j transient
(∗)
1−fj,j
Proof.
Gi,j = Pi,j (1)
(0)
= pi,j + Fi,j (1)Pj,j (1)
i → j, j recurrent =⇒ Fi,j (1) = 1, Pj,j (1) = ∞ hence Pi,j (1) = ∞
(0)
i 9 j =⇒ pi,j , Fi,j (1) = 0 =⇒ Pi,j (1) = 0
(0)
Pj,j (1) = pj,j + Fj,j (1)Pj,j (1)
(∗)
= 1 + fj,j Pj,j (1)
1
=
(∗)
1 − fj,j
Gi,j = Pi,j (1)
(0)
= pi,j + Fi,j (1)Pj,j (1)
= Fi,j (1)Pj,j (1)
(∗)
=
fi,j
(∗)
1 − fj,j
Proposition 2.2. If T is a transient class or the set of all transient classes and Q is the
sub-stochastic transition matrix P restricted to T then
G(P )|T = G(Q)
Proof. If i, j ∈ T k ∈
/ T then we cannot have i → k and k → j so
G(P )|T =
∞
X
n=0
8
Qn = G(Q)
Proposition 2.3.
G(Q) = (I − Q)−1
Proof.
G(Q) =
∞
X
Qn
n=0
=I +Q
∞
X
Qn
n=0
= I + QG(Q)
G(Q)(I − Q) = I
G(Q) = (I − Q)−1
If T is infinite then G = I + QG may have an infinite number of solutions.
P∞
Theorem 2.1. G = n=0 Qn is the smallest non-negative solution to
Z = I + QZ
Proof. We have already shown that G is a non-negative solution.
Suppose Z is another solution with non-negative entries, then
Z = I + QZ
= I + Q + Q2 Z
n
X
=
Qk + Qn+1 Z
k=0
Qn+1 , Z have non-negative entries hence Qn+1 Z has non-negative entries so
Z≥
n
X
Qk ∀n
k=0
hence
Z ≥ lim
n→∞
n
X
Qk = G
k=0
Theorem 2.2. Let A ⊆ S and Q by the transition matrix restricted to A then let
!
n
\
(n)
ai = P
{Xk ∈ A}X0 = i
k=1
and
(n)
ai = lim aI
n→∞
then a is the maximal solution to h = Qh with hi ∈ [0, 1] ∀i
(n)
(n)
Proof. Firstly {ai }∞
n=1 is a decreasing sequence of elements ai
9
∈ [0, 1] hence it has a limit ai ∈ [0, 1]
(n+1)
ai
X
=
(n)
pi,j aj
j∈A
X
ai =
pi,j aj
j∈A
a = Qa
so indeed a is a solution
Suppose h is a solution with hi ∈ [0, 1] ∀i
let e = (1, 1, 1, ..., 1)T be of appropriate length then h ≤ e
(1)
ai
= P(X1 ∈ A|X0 = i)
X
=
pi,j
j∈A
=⇒
a(1) = Qe
h = Qh
≤ Qe
h ≤ a(1)
h = Qh
≤ Qa(1)
= a(2)
= Qa(2)
= a(n)
h ≤ lim a(n)
n→∞
=a
so indeed a is maximal.
Lemma 2.2. Either a = 0 or supi ai = 1
Proof. if supi ai ∈ (0, 1) then write
a∗ =
a
supi ai
then a∗ solves a∗ = Qa∗ and a∗ > a which is a contradiction.
Theorem 2.3. If T is the set of all transient states then
• If T is finite then ∃1 solution to (I − Q)Y = I
and the only bounded solution to h = Qh is h = 0
• If T is infinite then the maximal solution to h = Qh with hi ∈ [0, 1] ∀i is a and either
– supi ai = 0 then the chain is certain to end in a recurrent class
– supi ai = 1 then there is positive probability that the chain will remain in the transient
classes forever.
10
Lemma 2.3. If we have a random walk with an absorbing barrier at 0 which moves left with probability
q and right with probability p where p + q = 1 then the chain is absorbed in 0 with probability:

1
q≥p
(∗)
fi,0 = q i

q<p
p
Proof. The transient states are τ = N which is infinitely large so we need to solve h = Qh with
hi ∈ [0, 1] ∀i
h1 = ph2
h2 = qh1 + ph3
..
.
hn = qhn−1 + phn+1
which gives
p(hn+1 − hn ) = q(hn − hn−1 )
q(hn − hn−1 )
p
n
q
=
h1
p
n i−1
X
q
hn = h1
p
i=1
hn+1 − hn =
• If q > p then:
lim hn = ∞
n→∞
∀h1 6= 0
hence since h1 is bounded we must have that h1 = 0
(∗)
so fi,0 = 1
• If q = p then:
hn = nh1 so
lim hn = ∞
n→∞
∀h1 6= 0
hence since h1 is bounded we must have that h1 = 0
(∗)
so fi,0 = 1
• If q < p then:
hn =
so h1 = 1 −
h1
1 − pq
q
p
guarantees hi ∈ [0, 1] ∀i and since supn hn = 1 this must be the maximal solution,
n
which gives that chain doesn’t reach zero with probability 1 − pq
(∗)
Theorem 2.4. fi,A := P(Xn ∈ A|X0 = i) is the minimal non-negative solution to
(∗)
fi,A =
X
pi,k +
X
k∈A
/
k∈A
11
(∗)
pi,k fk,A
Proof.
(∗)
fi,A =
X
P({X1 = k} ∩ {Xn ∈ A}|X0 = i)
k∈S
=
X
pi,k +
X
pi,k +
X
X
pi,k P(Xn ∈ A|Xn = k)
by Markov Property
k∈A
/
k∈A
=
pi,k P(Xn ∈ A|{X0 = i} ∩ {Xn = k})
k∈A
/
k∈A
=
X
pi,k +
X
(∗)
pi,k fk,A
by stationary transition properties
k∈A
/
k∈A
hence this is a solution, so it remains to show that it is the minimal non-negative solution.
(∗)
(∗)
(∗)
we know that fi,A ∈ [0, 1] so suppose that fˆi,A < fi,A is a solution
let j ∈ A s.t. i → j then
X
X
(∗)
(∗)
fˆi,A =
pi,k +
pi,k fˆk,A
k∈A
X
(∗)
pi,k fˆk,A
k∈A
/
k∈A
/
=⇒
X
(∗)
<
pi,k fk,A
k∈A
/
=⇒
(∗)
p̂k,j
(∗)
for some k ∈
/A
< pk,j
(∗)
but p̂k,j ∈ {0, 1}
=⇒
(∗)
p̂k,j
=0
(∗)
(∗)
so ∃k ∈
/ A s.t. k 9 A in fˆi,A but k → A in fi,A which is a contradiction.
(∗)
The absorption probability fi,C is the probability of being absorbed into a recurrent communicating
class C which it then never leaves.
(∗)
We write F C to be the column vector representing the absorption probabilities for i ∈ T
Proposition 2.4. If we let RC be the matrix of one step transition probabilities to the recurrent
communicating class C then
(∗)
F C = GRC e
Proof.
(∗)
(∗)
(F C )i = fi,C
(∗)
X
(∗)
X
fi,C −
(∗)
(∗) pi,k fk,C = (I − Q)F C
i
k∈τ
/
fi,C −
(∗)
pi,k fk,C =
k∈τ
/
X
pi,k
k∈τ
= RC e
RC e
i
(∗) i
= (I − Q)F C
GRC e =
12
(∗)
FC
i
Theorem 2.5. Suppose T is the union of all transient states and C is any union of recurrent classes
(∗)
s.t. fi,C = 1 ∀i ∈ T then the expected first passage time to C is
K C = Ge
Proof. Suppose X0 = i ∈ T then the number of visits that X makes to states in τ before leaving T is
the same as the number of visits to states in τ before reaching C since τ is transient and C is recurrent
hence X cannot return.
Gi,j is precisely the expected number of visits to j given the chain starts in i so
X
Gi,j
Ge i =
j∈τ
= E[TC |X0 = i]
3
Branching Processes
A branching process is a population model where at each time point every particle gives rise to a
random number of offspring from the same distribution but independently from all other particles at
all time points. We use Xn to denote the number of particles present at time n.
So we let Zk be distributed from the family distribution g then:
!
i
X
P(Xn+1 = j|Xn = i) = P
Zk = j
k=1
Lemma 3.1. If Z is an r.v distributed with g then the generating function for the branching process is
G(s) = E[sZ ] =
∞
X
sm g(m)
m=0
then we define
Fn (s) = E[sXn |X0 = 1]
is the probability generating function for the number of individuals in generation n given we started
with a single individual.
Theorem 3.1.
Fn+1 (s) = Fn (G(s))
= G(n+1) (s)
= G(Fn (s))
for n > 0
13
Proof.
Fn+1 (s) = E[sXn+1 |X0 = 1]
∞
X
=
sm P(Xn+1 = m|X0 = 1)
=
=
=
m=0
∞
X
m=0
∞
X
sm
sm
∞
X
k=0
∞
X
P({Xn+1 = m} ∩ {Xn = k}|X0 = 1)
P(Xn+1 = m|Xn = k)P(Xn = k|X0 = 1)
Markov property
m=0
k=0
∞ X
∞
X
m
s pk,m P(Xn = k|X0 = 1)
k=0 m=0
= P(Xn = 0|X0 = 1) +
= P(Xn = 0|X0 = 1) +
∞
X
∞
X
k=1
∞
X
m=0
s P
!
Zi = m
P(Xn = k|X0 = 1)
i=1
Pk
E[s
k
X
m
i=1
Zi
]P(Xn = k|X0 = 1)
k=1
= P(Xn = 0|X0 = 1) +
= P(Xn = 0|X0 = 1) +
∞
k
X
Y
k=1
∞
X
!
Zi
E[s ] P(Xn = k|X0 = 1)
i=1
G(s)k P(Xn = k|X0 = 1)
k=1
=
∞
X
G(s)k P(Xn = k|X0 = 1)
k=0
= Fn (G(s))
Lemma 3.2. The mean generation size is
µ = G0 (1) = E[Z] = E[X1 |X0 = 1]
and for µ < ∞
E[Xn |X0 = 1] = µn
(∗)
The extinction probability q = f1,0 is the probability that the branching process dies out at some
point.
Corollary 3.1.
∞
[
(∗)
fk,0 = P
Xn = 0|X0 = k = q k
n=1
Proposition 3.1. q is the smallest non-negative solution to s = G(s)
Proposition 3.2. If G(0) > 0 then
• µ > 1 is supercritical, extinction is uncertain
• µ = 1 is critical, extinction is certain
• µ < 1 is sub-critical, extinction is certain
14
4
Random Walks
For a, b ∈ Z, n ∈ N denote
Nn (a, b) := {(xi )ni=0 ∈ Zn+1 : x0 = a, xn = b, |xr − xr−1 | = 1}
to be the set of nearest neighbour paths from a to b of length n and
Nnz (a, b) := {(xi )ni=0 ∈ N (a, b) : ∃m ∈ N s.t. xm = z}
Theorem 4.1. Reflection Principle
If a, b ∈ N then #Nn0 (a, b) = #N (a, −b).
Theorem 4.2. For k ≥ 1
(
(n)
f0,k
=
n−k
k n+k
2 q 2
np
n!
n−k n+k
2 ! 2 !
0
n + k even
n + k odd
Proposition 4.1.
F0,1 (s) =
1−
p
1 − 4pqs2
2qs
and for k ≥ 1
F0,k (s) =
1−
p
1 − 4pqs2
2qs
!k
Proof.
F0,1 (s) =
∞
X
(n)
f0,1 sn
n=1
T1
= E[s |X0 = 0]
= sP(X1 = 1|X0 = 0) + E[sT1 |{X1 = −1} ∩ {X0 = 0}]P(X1 = −1|X0 = 0)
= ps + qE[sT1 |X1 = −1]
= ps + qE[s1+T1 |X0 = −1]
= ps + qsE[sT1 |X0 = −1]
= ps + qsE[sT0 |X0 = −1]E[sT1 |X0 = 0]
= ps + qsF0,1 (s)2
p
1 ± 1 − 4pqs2
=
2qs
p
1 − 1 − 4pqs2
=
2qs
since F0,1 (s) ∈ [0, 1]
Corollary 4.1.
(∗)
f0,k = F0,k (1)
k
1 − |1 − 2p|
=
2q
(
(p/q)k p < 1/2
=
1
p ≥ 1/2
15
5
Limiting Behaviour for Markov Chains
If i is a recurrent state then it is positive recurrent if E[Ti |X0 = i] < ∞ and null recurrent if
E[Ti |X0 = i] = ∞
Theorem 5.1. Positive and null recurrence are class properties
Proof. We know that recurrence is a class property so it sufficient to show that if i ↔ j and j is null
recurrent then i is null recurrent.
(s) (r)
suppose ∃s, r > 0 s.t. pi,j , pj,i > 0
(n+r+s)
then pj,j
(r) (n) (s)
≥ pj,i pi,i pi,j > 0
(n+r+s)
since j is null recurrent we have that limn→∞ pj,j
(r) (n) (s)
hence limn→∞ pj,i pi,i pi,j =
(n)
moreover limn→∞ pi,i = 0
=0
0
hence i is null recurrent
Theorem 5.2. A finite essential class is positive recurrent
Proof. WLOG let C = N|[1,m] be a finite essential class.
Pm (n)
fix i ∈ C then j=1 pi,j = 1
(n)
but if i were null recurrent then limn→∞ pi,j = 0 which cannot be true since m is finite.
The equilibrium distribution for a transition matrix P is a row vector π s.t.
πP = π
where πi ≥ 0 and
P
i
πi = 1
Theorem 5.3. If X is an irreducible, aperiodic, positive recurrent Markov Chain with transition
matrix P then:
• ∃1 equilibrium distribution π
• limn→∞ P(Xn = i) = πi whatever initial distribution
• πi =
6
1
E[Ti |X0 =i]
Continuous Time Processes
A random variable X is memoryless if
P(X > t + s|X > t) = P(X > s)
which is equivalent to
P(X > t + s) = P(X > t)P(X > s)
Corollary 6.1. The exponential distribution is memoryless
Proof. let X ∼ exp(λ)
P(X > t + s) = 1 − F (t + s)
= e−λ(t+s)
= e−λt e−λs
= P(X > t)P(X > s)
16
Lemma 6.1. If X > 0 is a memoryless random variable then it is exponential
Proof. let F X = 1 − FX then F X (t + s) = F X (t)F X (s) since memoryless
0
F X (t + s) − F X (t)
s
F X (t)F X (s) − F X (t)
= lim
s→0
s
F X (t)(F X (s) − 1)
= lim
s→0
s
F X (s) − 1
= F X (t) lim
s→0
s
0
= F X (t)F X (0)
F X (t) = lim
s→0
since memoryless
0
= −γF X (t)
where γ = −F X (0) ≥ 0
thus by solving the differential equation we get
F X (t) = Ae−γt
we know F X (0) = 1
−γt
FX (t) = 1 − Ae
with A = 1
hence indeed X has an exponential distribution.
An equivalent statement to X having the Markov property is that
P(Xt+s = j|Ft ) = P(Xt+s = j|Xt )
A Markov process is time inhomogeneous if
P(Xt+s = j|Xt = i) = P(Xs = j|X0 = i)
∀t
then we can write
P(Xt+s = j|Xt = i, Ft ) = ps (i, j) = p(i,j) (s)
{Pt }t≥0 is a sub-stochastic semi-group if
• P0 = I
• Pt has non-negative entries
• row sums are ≤ 1
• Pt solves the Chapman-Kolmogorov equations:
Pt+s = Pt Ps
Theorem 6.1. Let Pt = (Pt )i,j , then {Pt }t≥0 is a sub-stochastic semi-group
Proof. The first three criteria are trivial so it is sufficient to prove the fourth
let C be the finite state space
17
pi,j (t + s) = P(Xt+s = j|X0 = i)
X
=
P(Xt+s = j|{Xt = k} ∩ {X0 = i})P(Xt = k|X0 = i)
by the law of total probability
k∈C
=
X
P(Xt+s = j|Xt = k)P(Xt = k|X0 = i)
by the Markov property
P(Xs = j|X0 = k)P(Xt = k|X0 = i)
by stationary transition
k∈C
=
X
k∈C
=
X
pi,k (t)pk,j (s)
k∈C
= (Pt Ps )i,j
The semi-group (Pt )t≥0 is standard if Pt is continuous at t = 0
We define the instantaneous transition rate as
q(i, j) := lim
t→0
pt (i, j) − p0 (i, j)
t
For i = j write q(i) = −q(i, j) which is positive.
The generator of the Markov chain is Q := (q(i, j))i,j∈C which is also called a Q-matrix
Lemma 6.2. For A ⊂ C finite with x ∈
/ A we have
pt (x, x) +
X
pt (x, y) ≤
y∈A
X
pt (x, y)
y∈C
≤1
X
pt (x, y) ≤ 1 − pt (x, x)
y∈A
X
q(x, y) ≤ −q(x, x)
by differentiating
y∈A
X
q(x, y) ≤ q(x)
y∈C\{x}
We shall assume that
X
q(x, y) = q(x)
y∈C\{x}
thus Qe = 0
Theorem 6.2. Backward Kolmogorov Equations
dPt
= QPt
dt
We shall assume that
supx∈S q(x) < ∞
Theorem 6.3. Forward Kolmogorov Equations
dPt
= Pt Q
dt
18
Lemma 6.3. If
Q = BΛB −1
for Λ = diag(λ1 , ...λn ) then B := (v 1 , ...v n ) where v i is the eigenvector for eigenvalue λi then
Qk = BΛk B −1
then Λk = diag(λk1 , ...λkn ) and
∞ k k
X
t Λ
k=0
k!
∞
X
(tλ1 )k
= diag
= diag(e
k=0
tλ1
k!
, ...
∞
X
(tλn )k
k=0
!
k!
, ...etλn )
=: etΛ
then we have that
Pt = BetΛ B −1
Given Q to find pi,j (t) we do not need to calculate Pt
Instead we can use the following algorithm
• Solve det(Q − Iλ) to find the eigenvalues {λi }ni=1 of Q
Pn
• We then know that pi,j (t) = i=1 Ai eλt for some unknown sequence {Ai }ni=1
• We also have initial conditions
(
1 i=j
pi,j (0) =
0 i 6= j
dn Pt = Qn
dtn t=0
hence we can solve the sequence of simultaneous equations.
7
Invariant Measures For Irreducible Finite Markov Chains
λ is invariant for Q if
λQ = 0
Theorem 7.1. If Q is a Q matrix with jump matrix π and measure λ then the following are equivalent
• λ invariant
• µπ = µ
µi := λi qi
Proof. Firstly note that
(
qi,j = qi (πi,j − δi,j ) =
19
−qi
qi πi,j
i=j
i 6= j
hence
X
µi (πi,j − Ii,j )
µ(π − I) j =
i
=
X
µi (πi,j − δi,j )
i
=
X
λi qi (πi,j − δi,j )
i
=
X
λi qi,j
i
= (λQ)j
Theorem 7.2. If Q is irreducible with jump matrix π which has invariant measure µ then
λ=
µ
qi
is invariant for Q
Theorem 7.3. If λQ = 0 for irreducible Q then
λP (s) = λ
Proof. By the backward Kolmogorov equations
d
d
λP (s) = λ P (s)
ds
ds
dPt
=λ
dt
= λQP (s)
=0
=⇒ λP (s) constant
P (0) = I
λP (0) = λI
=λ
Theorem 7.4. If Q is irreducible with invariant probability measure λ then
lim pi,j (t) = λj
t→∞
8
Reversibility
Let T ∈ (0, ∞) be fixed and (Xt )t∈[0,T ] be the Markov process with initial distribution λ and
irreducible, non-explosive Q-matrix Qwith invariant distribution λ. Define X̂t = lims→t− XT −s to be
the time reversal of Xt .
Proposition 8.1. If X̂t is the time reversal of Xt then X̂t is also a MP with initial distribution λ and
Q matrix Q̂ solving λj q̂j,i = λi qi,j . Q̂ is also irreducible and non-explosive with invariant λ.
A Markov process X with irreducible, non-explosive Q and X0 = λ is reversible if
∀T > 0 X̂ := lims→t− X̂T −s also has Q matrix Q and initial distribution λ.
Q, λ are in detailed balance if λi qi,j = λj qj,i
20
Lemma 8.1. If Q, λ are in detailed balance then λ is invariant for Q
Proof.
(λQ)i =
X
λj qj,i
j
=
X
λi qi,j
j
= λi
X
qi,j
j
=0
Theorem 8.1. If X is a MP with irreducible, non-explosive Q and initial distribution λ then X is
reversible iff λ, Q are in detailed balance.
Proof. If X is reversible then
by definition λ is invariant for Q and Q̂ = Q hence
λi qi,j = λj q̂j,i = λj qj,i
hence Q, λ are in detailed balance
If λ, Q are in detailed balance then
by the previous lemma λ is invariant for Q hence
λj q̂j,i = λi qi,j = λj qj,i
hence Q̂ = Q so X is reversible.
A chain cannot be reversible if you can go directly from state i to state j but not directly from state j
to state i
21