IEOR 165 Discussion 9
April 17, 2015
Stochastic Process
Definition: A stochastic process is a collection of random variables indexed by time. The
sequence {Xt , t ∈ T } is called a stochastic process.
1. Time index (T)
(a) Discrete time index: T = {0, 1, 2, ...}
(b) Continuous time index: T = [0, ∞) or T = [−∞, ∞)
2. State space (S)
(a) Countable (discrete) state space
(b) Uncountable state space
Discrete Time Markov Chain (DTMC)
Markov Property: Given the present state, the future state of the system is independent
of the past.
P (Xt+1 = jt+1 |X1 = j1 , ..., Xt = jt ) = P (Xt+1 = jt+1 |Xt = jt )
DTMC: A stochastic process is called a DTMC if
1. T is discrete
2. S is discrete
3. {Xt } has Markov Property
One Step Transition Matrix: Each of its entries is a non-negative real number representing a transition probability. Each row of P sums to 1.


p1,1 p1,2 ... p1,n
 p2,1 p2,2 ... p2,n 


P = .
.. 
..
.
.
.
 .
.
. 
.
pn,1 pn,2
1
... pn,n
where P (Xt+1 = j|Xt = i) = pi,j .
Time Homogeneous MC: The transition matrix P stays same after each step, so the
k-step transition probability is equal to the k th power of the transition matrix, P k .
Chapman Kolmogorov Equation: The k-step transition probability satisfies:
∑
Pijk =
Pirm Prjk−m ∀i, j ∈ S and 0 ≤ m ≤ k
r∈S
Define fj to be the probability that starting with state j, the process
ever return to state
∑ will
k
k
)
to be the probability of first return to j at step k.(fj = k fjj
j and fjj
Classsification of States:
• j is accessible from i: ∃k ≥ 0 such that Pijk > 0. (written as i → j)
• i and j are called communicative iff i → j and j → i. (written as i ↔ j)
• Two states that communicate are in the same class.
• MC is said to be irreducible if its state space is a single communicating class.
∑
k
• A state j is said to be transient if fj < 1 or ∞
k=1 Pjj < ∞.
∑
k
• A state j is said to be recurrent if fj = 1 or ∞
k=1 Pjj = ∞.
• Every finite state, irreducible MC is positive recurrent.
• The period of state j is defined as:
(n)
d(j) = gcd{n ≥ 1 : pjj > 0}
where gcd means greatest common divisor. If d(j) = 1, call j aperiodic. If d(j) > 1,
call j periodic with period d(j).
Question 1. Classify S = 1, 2, 3, 4 by recurrence and transience:
Solution 1.
2
State 1:
1
3
2
=
3
= 0 ∀k > 2
1
2
= f11
+ f11
=1
1
=
f11
2
f11
k
f11
f1
recurrent
State 3:
1
2
= 0 ∀k > 1
1
1
= f33
=
2
1
f33
=
k
f33
f3
transient
State 4:
1
2
= 0 ∀k > 1
1
1
= f44
=
2
1
f44
=
k
f44
f4
transient
State 2:
1
f22
= 0
2
2
f22
=
3
1 k−2 2
k
f22 =
∀k > 2
3
3
∞
∑
1 k−2 2
1 2
f2 =
=
=1
1
3
3
3
1
−
3
k=2
recurrent
Proposition: If MC is aperiodic, irreducible and positive recurrent, there exists a unique
stationary distribution π. It is called a stationary distribution for the MC with transition
matrix P if
π′ = π′P
∑
i.e., πj = r∈S πr pr,j , j ∈ S. The mean return (recurrence) time is:
µj =
3
1
πj
Question 2. Find the stationary distribution of MC with P:


0.45 0.48 0.07
P =  0.05 0.7 0.25 
0.01 0.5 0.49
Solution 2.
Sum of the stationary distribution probabilities is equal to

[
] [
] 0.45
π1 π2 π3 = π1 π2 π3  0.05
0.01
1. i.e. π1 + π2 + π3 = 1.

0.48 0.07
0.7 0.25 
0.5 0.49
π1 = 0.45π1 + 0.05π2 + 0.01π3
π2 = 0.48π1 + 0.7π2 + 0.5π3
π3 = 0.07π1 + 0.25π2 + 0.49π3
Since the row sum of P is equal to 1, one constraint is redundant. Solution is π1 = 0.07,
π2 = 0.62 and π3 = 0.31.
4