advertisement

Branching Processes and Random Walks Adam Bowditch April 8, 2014 Contents 1 Processes 2 2 The Fundamental Matrix 8 3 Branching Processes 13 4 Random Walks 15 5 Limiting Behaviour for Markov Chains 16 6 Continuous Time Processes 16 7 Invariant Measures For Irreducible Finite Markov Chains 19 8 Reversibility 20 1 1 Processes A discrete state space Markov chain is said to have stationary transitions if pi,j = P(Xn+1 = j|Xn = i) = P(X1 = j|X0 = i) ∀n ∈ T , i, j ∈ S. We then write the transition matrix as P = (pi,j )i,j∈S , the n-step transition probability as (n) (n) pi,j = P(Xm+n = j|Xm = i) and the n-step transition matrix as P (n) = (pi,j )i,j∈S A matrix is called stochastic if all entries are positive and rows sum to 1 and sub-stochastic if all entries are positive and row sums are less than or equal to 1. Theorem 1.1. The distribution of a discrete state space Markov chain with stationary transition properties is fully specified by 1. Its initial state X0 or the distribution of the initial state where random. 2. Its one-step transition matrix Proof. Consider any path {Xi = ki }m i=0 over an initial time segment i ∈ Z|[0,m] This path has probability ! ! m m−1 m−1 \ \ \ P {Xi = ki } = P Xm = km {Xi = ki } P {Xi = ki } i=0 i=0 i=0 m−1 \ = P(Xm = km |Xm−1 = km−1 )P ! {Xi = ki } by Markov Property i=0 = m Y P(Xi = ki |Xi−1 = ki−1 )P(X0 = k0 ) i=1 Which depends only on the one-step transition matrix and the initial state. Theorem 1.2. Chapman-Kolmogorov Equations If X is a discrete state space Markov chain with stationary transition properties then: P (n+m) = P (n) P (m) or equivalently (n+m) pi,j = X (n) (m) pi,k pk,j k∈S Proof. (n+m) pi,j = P(Xn+m = j|X0 = i) ! =P [ {Xn = k} ∩ {Xn+m = j}|{X0 = i} k∈S = X P({Xn = k} ∩ {Xn+m = j}|{X0 = i}) since disjoint k∈S = X P({Xn+m = j}|{X0 = i} ∩ {Xn = k})P(Xn = k|X0 = i) k∈S = X P(Xn+m = j|Xn = k)P(Xn = k|X0 = i) k∈S = X (n) (m) pi,k pk,j k∈S 2 by Markov property (n) Proposition 1.1. If we denote the row vector a(n) := (ani )i∈S where ai = P(Xn = i) then a(n) = a(0) p(n) Proof. (n) ai = P(Xn = i) X = P(Xn = i|X0 = k)P(X0 = k) k∈S = X (n) (0) pk,i ak k∈S = (a(0) p(n) )i Suppose i, j ∈ S then: (n) • If ∃n > 0 s.t. pi,j > 0 then i → j; • i, j intercommunicate ( i ↔ j ) if i → j and j → i; • i, j are in the same communicating class ( i ∼ j ) if either: i = j or i ↔ j; Proposition 1.2. ∼ is an equivalence relation Proof. • Reflexive: i ∼ i since i = i • Symmetric: i ∼ j =⇒ i ∼ i by definition • Transitive: Suppose i ∼ j j ∼ k i ∼ k is immediate if i = j or j = k so assume not. i → j, j → k =⇒ (n) (m) ∃n, m > 0 s.t. pi,j , pj,k > 0 so P (n+m) (n) (m) pi,j = k∈S pi,k pk,j =⇒ (n+m) (n) (m) pi,k ≥ pi,j pj,k > 0 hence (l) ∃l = n + m > s.t. pi,k > hence i → k The reverse argument obviously holds for k → i The state i ∈ S is essential if ∀j ∈ S s.t. i → j we must have that j → i. A transition matrix is irreducible if all states intercommunicate. Lemma 1.1. If i → k and i is essential then i ∼ k and k is essential. Proof. i → k, i essential =⇒ k → i hence i ↔ k and i ∼ k. Suppose k → j 6= i then i → k → j hence j → i since i is essential. So, j → i → k hence k is essential. If a communicating class contains a single essential class then every state in the class is essential so essential and inessential are class properties. The period of a state i is ( undef ined i9i per(i) = (n) gcd{n > 0 : pi,i > 0} i → i 3 Lemma 1.2. If i → i and i has period d then ∃N > 0 s.t. k ≥ N =⇒ (kd) pi,i >0 Theorem 1.3. Periodicity is a class property Proof. Suppose i ↔ j and i has period d (n) (m) (r) ∃n, m, r > 0 s.t. pi,j , pj,i , pj,j > 0 (n+m+r) (n) (r) (m) pi,i ≥ pi,j pj,j pj,i > 0 and (n+m) (n) (m) pi,i ≥ pi,j pj,i > 0 (r) so we have that d|n + m + r and d|n + m so d|r which is true for any r s.t pj,j > 0 hence since this argument can be applied to the states reversed we must have that j has period d as well. If C is a communicating class with period d then for any i ∈ C and all integers r > 0 (kd+r) Gr (i) = {i ∈ C : pi,j > 0} is called a cyclically moving subclass. Gr (i) = Gr+d (i) moreover {Gr (i)}d−1 r=0 forms a partition of C The first passage time to state j is Tj = inf {n ≥ 1 : Xn = j} moreover the probability j is first (n) reached in n steps is denoted fi,j = P(Tj = n|X0 = i) and the probability of reaching j in a finite number of steps is ∞ X (n) (∗) fi,j fi,j = n=1 A state i is recurrent if (∗) fi,i = 1 and transient otherwise. Lemma 1.3. (∗) fi,j = pi,j + X (∗) pi,k fk,j k∈S Proof. (1) fi,j = pi,j (n+1) fi,j = P(Tj = n + 1|X0 = i) X = P({Tj = n + 1} ∩ {X1 = k}|X0 = i) k∈S = X P(Tj = n + 1|{X0 = i} ∩ {X1 = k})P(X1 = k|X0 = i) k∈S = X P(Tj = n + 1|X1 = k)P(X1 = k|X0 = i) Markov property k∈S = X P(Tj = n|X0 = k)pi,k stationary transitions k∈S = X (n) fk,j pi,k k∈S =⇒ (∗) fi,j = pi,j + X (∗) pi,k fk,j k∈S 4 Lemma 1.4. (n) pi,j = n X (m) (n−m) fi,j pj,j m=1 Proof. (n) pi,j = P(Xn = j|X0 = i) = = = = n X m=1 n X m=1 n X m=1 n X P({Xn = j} ∩ {Tj = m}|X0 = i) P(Xn = j|Tj = m)P(Tj = m|X0 = i) P(Xn = j|Xm = j)P(Tj = m|X0 = i) (n−m) (m) fi,j pj,j m=1 We define the first-step decomposition generating functions as Pi,j (s) = Fi,j (s) = ∞ X n=0 ∞ X (n) pi,j sn (n) fi,j sn n=1 Lemma 1.5. (0) Pi,j (s) = pi,j + Pj,j (s)Fi,j (s) Proof. (0) Pi,j (s) = pi,j + ∞ X (n) pi,j sn n=1 (0) ∞ n−1 X X (0) n=1 r=0 ∞ X ∞ X = pi,j + = pi,j + (n−r) n−r (r) r s pj,j s fi,j (n−r) n−r (r) r s pj,j s fi,j r=1 n=r+1 = (0) pi,j + ∞ X (r) pj,j sr r=1 = (0) pi,j + ∞ X ∞ X ! (n−r) fi,j sn−r n=r+1 ! (r) pj,j sr r=1 ∞ X n=1 (0) = pi,j + Pj,j (s)Fi,j (s) 5 ! (n) fi,j sn Lemma 1.6. The expected number of visits to state j given we start in state j is ∞ X ∞ X (n) pj,j = n=0 n=0 ∞ X = P(Xn = j|X0 = j) E[1{Xn =j} |X0 = j] n=0 ∞ hX i =E 1{Xn =j} |X0 = j n=0 == 1 (∗) 1 − fj,j Theorem 1.4. The state j is recurrent precisely when ∞ X (n) pj,j = ∞ n=0 (∗) Proof. j recurrent ⇐⇒ fj,j = 1 ⇐⇒ limz→1 Fj,j (z) = 1 Pj,j (s) = 1 + Fj,j (s)Pj,j (s) =⇒ Fj,j (s) = Pj,j (s) − 1 Pj,j (s) so lim Fj,j (z) = 1 ⇐⇒ lim Pj,j (z) = ∞ z→1 z→1 which occurs iff ∞ X (n) pj,j = ∞ n=0 Corollary 1.1. If j is transient then (n) lim p n→∞ i,j =0 ∀i ∈ S P∞ (n) (n) Proof. Suppose i = j then j transient implies that n=0 pj,j < ∞ which ensures that limn→∞ pi,j = 0 If i 6= j then the generating function relationship is Pi,j (s) = Pj,j (s)Fi,j (s) so we have that ∞ X n=0 (n) pi,j = lim Pi,j (z) z→1 = lim Pj,j (z)Fi,j (z) z→1 = Pj,j (1)Fi,j (1) j is transient so Pj,j (1) < ∞ and Fi,j (1) ≤ 1 since it is a probability generating function of an r.v hence have that P∞ we (n) n=0 pi,j < ∞ hence (n) limn→∞ pi,j = 0 6 Theorem 1.5. Recurrence and transience are class properties. Proof. Suppose i ↔ j then (n) (m) ∃n, m > 0 s.t. pi,j , pj,i > 0 hence (n+m+r) (n) (r) (m) pj,j ≥ pi,j pi,i pj,i > 0 which implies ∞ X (k) pj,j ≥ k=0 P∞ (r) r=0 pi,i If i is recurrent then ∞ X (n+m+r) pj,j (n) (m) ≥ pi,j pj,i r=0 = ∞ =⇒ ∞ X (r) pi,i r=0 P∞ (k) k=0 pj,j = ∞ hence j is recurrent. (∗) (∗) Theorem 1.6. If i is recurrent and i → j then i ↔ j and fi,j = fj,i = 1 (∗) Proof. Since i is recurrent we have that fi,i = 1 (n) i → j =⇒ ∃n > 0 s.t. pi,j > 0 then we have X (∗) (∗) fk,i = pk,i + pk,l fl,i l6=i = X = X (∗) (∗) pk,l fl,i since fi,i = 1 l∈S X pk,l (∗) pl,m fm,i m∈S l∈S ! = X X m∈S = pk,l pl,m (∗) fm,i l∈S (2) (∗) pk,m fm,i X m∈S = (r) X (∗) ∀r ≥ 0 by induction pk,m fm,i m∈S taking k = i, r = n we get X (n) (∗) (∗) 1 = fi,i = pi,m fm,i m∈S X 0=1− (n) (∗) pi,m fm,i m∈S = X (n) (∗) pi,m (1 − fm,i ) m∈S (∗) fj,i =1 hence j → i Lemma 1.7. • Recurrent =⇒ Essential • Inessential =⇒ Transient Theorem 1.7. If C is a finite essential class then it is recurrent. Proof. Suppose C = N|[1,m] is essential (n) then if i ∈ C and pi,j > 0 then j ∈ C so for each n 7 1= X (n) pi,k = k∈S but if C were transient we would have that impossible. 2 X (n) pi,k k∈C (n) limn→∞ pi,k = 0 but since the sum is finite this is The Fundamental Matrix The fundamental matrix of a sub-stochastic matrix Q is G= ∞ X Qk k=0 Lemma 2.1. If P is a transition matrix and G is the fundamental matrix of P then Gi,j is the expected number of times the particle will be in j given the particle started in i Proposition 2.1. Gi,j ∞ = 0 (∗) fi,j i → j, j recurrent ı9j i → j, j transient (∗) 1−fj,j Proof. Gi,j = Pi,j (1) (0) = pi,j + Fi,j (1)Pj,j (1) i → j, j recurrent =⇒ Fi,j (1) = 1, Pj,j (1) = ∞ hence Pi,j (1) = ∞ (0) i 9 j =⇒ pi,j , Fi,j (1) = 0 =⇒ Pi,j (1) = 0 (0) Pj,j (1) = pj,j + Fj,j (1)Pj,j (1) (∗) = 1 + fj,j Pj,j (1) 1 = (∗) 1 − fj,j Gi,j = Pi,j (1) (0) = pi,j + Fi,j (1)Pj,j (1) = Fi,j (1)Pj,j (1) (∗) = fi,j (∗) 1 − fj,j Proposition 2.2. If T is a transient class or the set of all transient classes and Q is the sub-stochastic transition matrix P restricted to T then G(P )|T = G(Q) Proof. If i, j ∈ T k ∈ / T then we cannot have i → k and k → j so G(P )|T = ∞ X n=0 8 Qn = G(Q) Proposition 2.3. G(Q) = (I − Q)−1 Proof. G(Q) = ∞ X Qn n=0 =I +Q ∞ X Qn n=0 = I + QG(Q) G(Q)(I − Q) = I G(Q) = (I − Q)−1 If T is infinite then G = I + QG may have an infinite number of solutions. P∞ Theorem 2.1. G = n=0 Qn is the smallest non-negative solution to Z = I + QZ Proof. We have already shown that G is a non-negative solution. Suppose Z is another solution with non-negative entries, then Z = I + QZ = I + Q + Q2 Z n X = Qk + Qn+1 Z k=0 Qn+1 , Z have non-negative entries hence Qn+1 Z has non-negative entries so Z≥ n X Qk ∀n k=0 hence Z ≥ lim n→∞ n X Qk = G k=0 Theorem 2.2. Let A ⊆ S and Q by the transition matrix restricted to A then let ! n \ (n) ai = P {Xk ∈ A}X0 = i k=1 and (n) ai = lim aI n→∞ then a is the maximal solution to h = Qh with hi ∈ [0, 1] ∀i (n) (n) Proof. Firstly {ai }∞ n=1 is a decreasing sequence of elements ai 9 ∈ [0, 1] hence it has a limit ai ∈ [0, 1] (n+1) ai X = (n) pi,j aj j∈A X ai = pi,j aj j∈A a = Qa so indeed a is a solution Suppose h is a solution with hi ∈ [0, 1] ∀i let e = (1, 1, 1, ..., 1)T be of appropriate length then h ≤ e (1) ai = P(X1 ∈ A|X0 = i) X = pi,j j∈A =⇒ a(1) = Qe h = Qh ≤ Qe h ≤ a(1) h = Qh ≤ Qa(1) = a(2) = Qa(2) = a(n) h ≤ lim a(n) n→∞ =a so indeed a is maximal. Lemma 2.2. Either a = 0 or supi ai = 1 Proof. if supi ai ∈ (0, 1) then write a∗ = a supi ai then a∗ solves a∗ = Qa∗ and a∗ > a which is a contradiction. Theorem 2.3. If T is the set of all transient states then • If T is finite then ∃1 solution to (I − Q)Y = I and the only bounded solution to h = Qh is h = 0 • If T is infinite then the maximal solution to h = Qh with hi ∈ [0, 1] ∀i is a and either – supi ai = 0 then the chain is certain to end in a recurrent class – supi ai = 1 then there is positive probability that the chain will remain in the transient classes forever. 10 Lemma 2.3. If we have a random walk with an absorbing barrier at 0 which moves left with probability q and right with probability p where p + q = 1 then the chain is absorbed in 0 with probability: 1 q≥p (∗) fi,0 = q i q<p p Proof. The transient states are τ = N which is infinitely large so we need to solve h = Qh with hi ∈ [0, 1] ∀i h1 = ph2 h2 = qh1 + ph3 .. . hn = qhn−1 + phn+1 which gives p(hn+1 − hn ) = q(hn − hn−1 ) q(hn − hn−1 ) p n q = h1 p n i−1 X q hn = h1 p i=1 hn+1 − hn = • If q > p then: lim hn = ∞ n→∞ ∀h1 6= 0 hence since h1 is bounded we must have that h1 = 0 (∗) so fi,0 = 1 • If q = p then: hn = nh1 so lim hn = ∞ n→∞ ∀h1 6= 0 hence since h1 is bounded we must have that h1 = 0 (∗) so fi,0 = 1 • If q < p then: hn = so h1 = 1 − h1 1 − pq q p guarantees hi ∈ [0, 1] ∀i and since supn hn = 1 this must be the maximal solution, n which gives that chain doesn’t reach zero with probability 1 − pq (∗) Theorem 2.4. fi,A := P(Xn ∈ A|X0 = i) is the minimal non-negative solution to (∗) fi,A = X pi,k + X k∈A / k∈A 11 (∗) pi,k fk,A Proof. (∗) fi,A = X P({X1 = k} ∩ {Xn ∈ A}|X0 = i) k∈S = X pi,k + X pi,k + X X pi,k P(Xn ∈ A|Xn = k) by Markov Property k∈A / k∈A = pi,k P(Xn ∈ A|{X0 = i} ∩ {Xn = k}) k∈A / k∈A = X pi,k + X (∗) pi,k fk,A by stationary transition properties k∈A / k∈A hence this is a solution, so it remains to show that it is the minimal non-negative solution. (∗) (∗) (∗) we know that fi,A ∈ [0, 1] so suppose that fˆi,A < fi,A is a solution let j ∈ A s.t. i → j then X X (∗) (∗) fˆi,A = pi,k + pi,k fˆk,A k∈A X (∗) pi,k fˆk,A k∈A / k∈A / =⇒ X (∗) < pi,k fk,A k∈A / =⇒ (∗) p̂k,j (∗) for some k ∈ /A < pk,j (∗) but p̂k,j ∈ {0, 1} =⇒ (∗) p̂k,j =0 (∗) (∗) so ∃k ∈ / A s.t. k 9 A in fˆi,A but k → A in fi,A which is a contradiction. (∗) The absorption probability fi,C is the probability of being absorbed into a recurrent communicating class C which it then never leaves. (∗) We write F C to be the column vector representing the absorption probabilities for i ∈ T Proposition 2.4. If we let RC be the matrix of one step transition probabilities to the recurrent communicating class C then (∗) F C = GRC e Proof. (∗) (∗) (F C )i = fi,C (∗) X (∗) X fi,C − (∗) (∗) pi,k fk,C = (I − Q)F C i k∈τ / fi,C − (∗) pi,k fk,C = k∈τ / X pi,k k∈τ = RC e RC e i (∗) i = (I − Q)F C GRC e = 12 (∗) FC i Theorem 2.5. Suppose T is the union of all transient states and C is any union of recurrent classes (∗) s.t. fi,C = 1 ∀i ∈ T then the expected first passage time to C is K C = Ge Proof. Suppose X0 = i ∈ T then the number of visits that X makes to states in τ before leaving T is the same as the number of visits to states in τ before reaching C since τ is transient and C is recurrent hence X cannot return. Gi,j is precisely the expected number of visits to j given the chain starts in i so X Gi,j Ge i = j∈τ = E[TC |X0 = i] 3 Branching Processes A branching process is a population model where at each time point every particle gives rise to a random number of offspring from the same distribution but independently from all other particles at all time points. We use Xn to denote the number of particles present at time n. So we let Zk be distributed from the family distribution g then: ! i X P(Xn+1 = j|Xn = i) = P Zk = j k=1 Lemma 3.1. If Z is an r.v distributed with g then the generating function for the branching process is G(s) = E[sZ ] = ∞ X sm g(m) m=0 then we define Fn (s) = E[sXn |X0 = 1] is the probability generating function for the number of individuals in generation n given we started with a single individual. Theorem 3.1. Fn+1 (s) = Fn (G(s)) = G(n+1) (s) = G(Fn (s)) for n > 0 13 Proof. Fn+1 (s) = E[sXn+1 |X0 = 1] ∞ X = sm P(Xn+1 = m|X0 = 1) = = = m=0 ∞ X m=0 ∞ X sm sm ∞ X k=0 ∞ X P({Xn+1 = m} ∩ {Xn = k}|X0 = 1) P(Xn+1 = m|Xn = k)P(Xn = k|X0 = 1) Markov property m=0 k=0 ∞ X ∞ X m s pk,m P(Xn = k|X0 = 1) k=0 m=0 = P(Xn = 0|X0 = 1) + = P(Xn = 0|X0 = 1) + ∞ X ∞ X k=1 ∞ X m=0 s P ! Zi = m P(Xn = k|X0 = 1) i=1 Pk E[s k X m i=1 Zi ]P(Xn = k|X0 = 1) k=1 = P(Xn = 0|X0 = 1) + = P(Xn = 0|X0 = 1) + ∞ k X Y k=1 ∞ X ! Zi E[s ] P(Xn = k|X0 = 1) i=1 G(s)k P(Xn = k|X0 = 1) k=1 = ∞ X G(s)k P(Xn = k|X0 = 1) k=0 = Fn (G(s)) Lemma 3.2. The mean generation size is µ = G0 (1) = E[Z] = E[X1 |X0 = 1] and for µ < ∞ E[Xn |X0 = 1] = µn (∗) The extinction probability q = f1,0 is the probability that the branching process dies out at some point. Corollary 3.1. ∞ [ (∗) fk,0 = P Xn = 0|X0 = k = q k n=1 Proposition 3.1. q is the smallest non-negative solution to s = G(s) Proposition 3.2. If G(0) > 0 then • µ > 1 is supercritical, extinction is uncertain • µ = 1 is critical, extinction is certain • µ < 1 is sub-critical, extinction is certain 14 4 Random Walks For a, b ∈ Z, n ∈ N denote Nn (a, b) := {(xi )ni=0 ∈ Zn+1 : x0 = a, xn = b, |xr − xr−1 | = 1} to be the set of nearest neighbour paths from a to b of length n and Nnz (a, b) := {(xi )ni=0 ∈ N (a, b) : ∃m ∈ N s.t. xm = z} Theorem 4.1. Reflection Principle If a, b ∈ N then #Nn0 (a, b) = #N (a, −b). Theorem 4.2. For k ≥ 1 ( (n) f0,k = n−k k n+k 2 q 2 np n! n−k n+k 2 ! 2 ! 0 n + k even n + k odd Proposition 4.1. F0,1 (s) = 1− p 1 − 4pqs2 2qs and for k ≥ 1 F0,k (s) = 1− p 1 − 4pqs2 2qs !k Proof. F0,1 (s) = ∞ X (n) f0,1 sn n=1 T1 = E[s |X0 = 0] = sP(X1 = 1|X0 = 0) + E[sT1 |{X1 = −1} ∩ {X0 = 0}]P(X1 = −1|X0 = 0) = ps + qE[sT1 |X1 = −1] = ps + qE[s1+T1 |X0 = −1] = ps + qsE[sT1 |X0 = −1] = ps + qsE[sT0 |X0 = −1]E[sT1 |X0 = 0] = ps + qsF0,1 (s)2 p 1 ± 1 − 4pqs2 = 2qs p 1 − 1 − 4pqs2 = 2qs since F0,1 (s) ∈ [0, 1] Corollary 4.1. (∗) f0,k = F0,k (1) k 1 − |1 − 2p| = 2q ( (p/q)k p < 1/2 = 1 p ≥ 1/2 15 5 Limiting Behaviour for Markov Chains If i is a recurrent state then it is positive recurrent if E[Ti |X0 = i] < ∞ and null recurrent if E[Ti |X0 = i] = ∞ Theorem 5.1. Positive and null recurrence are class properties Proof. We know that recurrence is a class property so it sufficient to show that if i ↔ j and j is null recurrent then i is null recurrent. (s) (r) suppose ∃s, r > 0 s.t. pi,j , pj,i > 0 (n+r+s) then pj,j (r) (n) (s) ≥ pj,i pi,i pi,j > 0 (n+r+s) since j is null recurrent we have that limn→∞ pj,j (r) (n) (s) hence limn→∞ pj,i pi,i pi,j = (n) moreover limn→∞ pi,i = 0 =0 0 hence i is null recurrent Theorem 5.2. A finite essential class is positive recurrent Proof. WLOG let C = N|[1,m] be a finite essential class. Pm (n) fix i ∈ C then j=1 pi,j = 1 (n) but if i were null recurrent then limn→∞ pi,j = 0 which cannot be true since m is finite. The equilibrium distribution for a transition matrix P is a row vector π s.t. πP = π where πi ≥ 0 and P i πi = 1 Theorem 5.3. If X is an irreducible, aperiodic, positive recurrent Markov Chain with transition matrix P then: • ∃1 equilibrium distribution π • limn→∞ P(Xn = i) = πi whatever initial distribution • πi = 6 1 E[Ti |X0 =i] Continuous Time Processes A random variable X is memoryless if P(X > t + s|X > t) = P(X > s) which is equivalent to P(X > t + s) = P(X > t)P(X > s) Corollary 6.1. The exponential distribution is memoryless Proof. let X ∼ exp(λ) P(X > t + s) = 1 − F (t + s) = e−λ(t+s) = e−λt e−λs = P(X > t)P(X > s) 16 Lemma 6.1. If X > 0 is a memoryless random variable then it is exponential Proof. let F X = 1 − FX then F X (t + s) = F X (t)F X (s) since memoryless 0 F X (t + s) − F X (t) s F X (t)F X (s) − F X (t) = lim s→0 s F X (t)(F X (s) − 1) = lim s→0 s F X (s) − 1 = F X (t) lim s→0 s 0 = F X (t)F X (0) F X (t) = lim s→0 since memoryless 0 = −γF X (t) where γ = −F X (0) ≥ 0 thus by solving the differential equation we get F X (t) = Ae−γt we know F X (0) = 1 −γt FX (t) = 1 − Ae with A = 1 hence indeed X has an exponential distribution. An equivalent statement to X having the Markov property is that P(Xt+s = j|Ft ) = P(Xt+s = j|Xt ) A Markov process is time inhomogeneous if P(Xt+s = j|Xt = i) = P(Xs = j|X0 = i) ∀t then we can write P(Xt+s = j|Xt = i, Ft ) = ps (i, j) = p(i,j) (s) {Pt }t≥0 is a sub-stochastic semi-group if • P0 = I • Pt has non-negative entries • row sums are ≤ 1 • Pt solves the Chapman-Kolmogorov equations: Pt+s = Pt Ps Theorem 6.1. Let Pt = (Pt )i,j , then {Pt }t≥0 is a sub-stochastic semi-group Proof. The first three criteria are trivial so it is sufficient to prove the fourth let C be the finite state space 17 pi,j (t + s) = P(Xt+s = j|X0 = i) X = P(Xt+s = j|{Xt = k} ∩ {X0 = i})P(Xt = k|X0 = i) by the law of total probability k∈C = X P(Xt+s = j|Xt = k)P(Xt = k|X0 = i) by the Markov property P(Xs = j|X0 = k)P(Xt = k|X0 = i) by stationary transition k∈C = X k∈C = X pi,k (t)pk,j (s) k∈C = (Pt Ps )i,j The semi-group (Pt )t≥0 is standard if Pt is continuous at t = 0 We define the instantaneous transition rate as q(i, j) := lim t→0 pt (i, j) − p0 (i, j) t For i = j write q(i) = −q(i, j) which is positive. The generator of the Markov chain is Q := (q(i, j))i,j∈C which is also called a Q-matrix Lemma 6.2. For A ⊂ C finite with x ∈ / A we have pt (x, x) + X pt (x, y) ≤ y∈A X pt (x, y) y∈C ≤1 X pt (x, y) ≤ 1 − pt (x, x) y∈A X q(x, y) ≤ −q(x, x) by differentiating y∈A X q(x, y) ≤ q(x) y∈C\{x} We shall assume that X q(x, y) = q(x) y∈C\{x} thus Qe = 0 Theorem 6.2. Backward Kolmogorov Equations dPt = QPt dt We shall assume that supx∈S q(x) < ∞ Theorem 6.3. Forward Kolmogorov Equations dPt = Pt Q dt 18 Lemma 6.3. If Q = BΛB −1 for Λ = diag(λ1 , ...λn ) then B := (v 1 , ...v n ) where v i is the eigenvector for eigenvalue λi then Qk = BΛk B −1 then Λk = diag(λk1 , ...λkn ) and ∞ k k X t Λ k=0 k! ∞ X (tλ1 )k = diag = diag(e k=0 tλ1 k! , ... ∞ X (tλn )k k=0 ! k! , ...etλn ) =: etΛ then we have that Pt = BetΛ B −1 Given Q to find pi,j (t) we do not need to calculate Pt Instead we can use the following algorithm • Solve det(Q − Iλ) to find the eigenvalues {λi }ni=1 of Q Pn • We then know that pi,j (t) = i=1 Ai eλt for some unknown sequence {Ai }ni=1 • We also have initial conditions ( 1 i=j pi,j (0) = 0 i 6= j dn Pt = Qn dtn t=0 hence we can solve the sequence of simultaneous equations. 7 Invariant Measures For Irreducible Finite Markov Chains λ is invariant for Q if λQ = 0 Theorem 7.1. If Q is a Q matrix with jump matrix π and measure λ then the following are equivalent • λ invariant • µπ = µ µi := λi qi Proof. Firstly note that ( qi,j = qi (πi,j − δi,j ) = 19 −qi qi πi,j i=j i 6= j hence X µi (πi,j − Ii,j ) µ(π − I) j = i = X µi (πi,j − δi,j ) i = X λi qi (πi,j − δi,j ) i = X λi qi,j i = (λQ)j Theorem 7.2. If Q is irreducible with jump matrix π which has invariant measure µ then λ= µ qi is invariant for Q Theorem 7.3. If λQ = 0 for irreducible Q then λP (s) = λ Proof. By the backward Kolmogorov equations d d λP (s) = λ P (s) ds ds dPt =λ dt = λQP (s) =0 =⇒ λP (s) constant P (0) = I λP (0) = λI =λ Theorem 7.4. If Q is irreducible with invariant probability measure λ then lim pi,j (t) = λj t→∞ 8 Reversibility Let T ∈ (0, ∞) be fixed and (Xt )t∈[0,T ] be the Markov process with initial distribution λ and irreducible, non-explosive Q-matrix Qwith invariant distribution λ. Define X̂t = lims→t− XT −s to be the time reversal of Xt . Proposition 8.1. If X̂t is the time reversal of Xt then X̂t is also a MP with initial distribution λ and Q matrix Q̂ solving λj q̂j,i = λi qi,j . Q̂ is also irreducible and non-explosive with invariant λ. A Markov process X with irreducible, non-explosive Q and X0 = λ is reversible if ∀T > 0 X̂ := lims→t− X̂T −s also has Q matrix Q and initial distribution λ. Q, λ are in detailed balance if λi qi,j = λj qj,i 20 Lemma 8.1. If Q, λ are in detailed balance then λ is invariant for Q Proof. (λQ)i = X λj qj,i j = X λi qi,j j = λi X qi,j j =0 Theorem 8.1. If X is a MP with irreducible, non-explosive Q and initial distribution λ then X is reversible iff λ, Q are in detailed balance. Proof. If X is reversible then by definition λ is invariant for Q and Q̂ = Q hence λi qi,j = λj q̂j,i = λj qj,i hence Q, λ are in detailed balance If λ, Q are in detailed balance then by the previous lemma λ is invariant for Q hence λj q̂j,i = λi qi,j = λj qj,i hence Q̂ = Q so X is reversible. A chain cannot be reversible if you can go directly from state i to state j but not directly from state j to state i 21