Markov Chains A Markov chain is characterized by: Pij Pr X n 1 j | X n i , X n 1 in 1 , , X 0 i0 Pr X n 1 j | X n i Pij 0 i , j 0 Pij 1 j 0 i 0,1,2, pi Pr X 0 i i 0,1,2, Pr X n j Pr X 0 i Pr X n j | X 0 i i 0 pi Pijn i 0 Matrix notation: The matrix of one-step transition probabilities Pij = Ρ The matrix of n-step transition probabilities Pijn P n P P P P n times p p0 , p1 , p2 , Pr X n j p P n P Q R 0 I The one-step transition probability matrix partitioned into transient, Q, and absorbing, R, states. N I Q 1 T N 1 The expected times spent in each state before being absorbed. The expected times to absorption. B NR The probabilities of absorption. Limiting probabilities: Regular chains (for some n, all elements of P n are > 0) lim Pijn j n j N k Pkj k 0 j 0,1,2, , N N j 1 j 0 Classification of states: Accessible: state j is accessible from state i if for some n 0, Pijn 0 . Notation: i j 1 Communication: States i and j communicate if each state is accessible from the other. Notation: i j Two states that communicate are in the same class. A Markov chain with only one state is irreducible. Periodicity: State i is said to have period d i if d i is the greatest common divisor of all integers n 1 for which Piin 0 . If d i =1, the chain is aperiodic. Probability of first return to state i at step n given you start in state i. f iin Pr X n i , X k i k 1,2, , n 1 | X 0 i Connection with n-step probabilities. Piin n f iik Piin k k 0 Probability that a process starting in i returns to i at some time. f ii n0 f iin lim N f iin N n0 Recurrent: state i is recurrent if and only if f ii 1 or equivalently if and only if Piin n 1 Transient: state i is transient if and only if f ii 1 or equivalently if and only if Piin n 1 Theorem: For a recurrent irreducible aperiodic Markov chain 1 1 a) lim Piin b) lim P jin lim Piin for all states j m n n n i nf iin n0 Stationary probability distribution: A set of probabilities, i for i 0 to ,that satisfy the following: j k Pkj k 0 j 0,1,2, j 1 j 0 2