Markov Chains
A Markov chain is characterized by:
Pij  Pr  X n 1  j | X n  i , X n 1  in 1 ,  , X 0  i0   Pr  X n 1  j | X n  i 
Pij  0 i , j  0

 Pij  1
j 0
i  0,1,2, 
pi  Pr  X 0  i  i  0,1,2, 
Pr  X n  j  

 Pr X 0  i  Pr  X n  j | X 0  i  
i 0

 pi Pijn
i 0
Matrix notation:
The matrix of one-step transition probabilities Pij = Ρ
The matrix of n-step transition probabilities Pijn   P n   P  P  P  P
n times
p   p0 , p1 , p2 , 
Pr  X n  j   p  P n 
P
Q R
0 I
The one-step transition probability matrix partitioned into
transient, Q, and absorbing, R, states.
N  I  Q 1
T  N 1
The expected times spent in each state before being absorbed.
The expected times to absorption.
B  NR
The probabilities of absorption.
Limiting probabilities:
Regular chains (for some n, all elements of P n  are > 0)
lim Pijn    j
n
j 
N
  k Pkj
k 0
j  0,1,2,  , N
N
 j  1
j 0
Classification of states:
Accessible: state j is accessible from state i if for some n  0, Pijn   0 . Notation: i  j
1
Communication: States i and j communicate if each state is accessible from the other.
Notation: i  j
Two states that communicate are in the same class. A Markov chain with only one state is
irreducible.
Periodicity: State i is said to have period d i if d i is the greatest common divisor of all
integers n  1 for which Piin   0 . If d i =1, the chain is aperiodic.
Probability of first return to state i at step n given you start in state i.
f iin   Pr  X n  i , X k  i k  1,2,  , n  1 | X 0  i 
Connection with n-step probabilities.
Piin  
n
 f iik Piin  k 
k 0
Probability that a process starting in i returns to i at some time.
f ii 


n0
f iin   lim
N
f iin 

N 
n0
Recurrent: state i is recurrent if and only if f ii  1

or equivalently if and only if
 Piin   
n 1
Transient: state i is transient if and only if f ii  1

or equivalently if and only if
 Piin  
n 1
Theorem: For a recurrent irreducible aperiodic Markov chain
1
1
a) lim Piin  

b) lim P jin   lim Piin  for all states j

m
n
n 
n
i
 nf iin 
n0
Stationary probability distribution: A set of probabilities,  i for i  0 to  ,that satisfy the
following:
j 

  k Pkj
k 0
j  0,1,2, 

 j  1
j 0
2