Markov Chains
A stochastic process is a collection of random variables. We have studied sequences of independent
identically-distributed random variables (IIDs)
in independent trials processes. Two principal
theorems for these processes are the Law of Large Numbers and the Central Limit Theorem. We now
consider the simplest extension of IID sequences to allow dependence between successive terms.
Definition
Let
variables
Markov condition,
be a set of states,
, where
(
. Consider a sequence (chain) of discrete random
is the state at time . is a Markov chain if it satisfies the
|
)
(
|
)
The probability distribution of
given the whole history is depends only on that of the most recent. The
next state depends only on the current state, but not anything before that. The future only depends on the
present, not the past.
Example
Swapping balls
Box 1
Box 2
Four balls, 2 black and 2 white, are put in two boxes, with 2 balls in each box. At each time step, select
one ball in each box and swap.
Let
be the number of black balls in Box 1. We have a Markov chain
(
Moreover, (
to
|
)
|
)
(
(
|
)
|
)
where the transition matrix
is the | |
(
|
)
| | matrix with entries
)
|
, i.e. the transition probability
is independent of time, and can be defined as
(
(
( )
)
from a state
Properties of
1.
( )
∑( )
2.
What is (
|
)?
(
|
∑ (
)
∑ (
) (
|
) (
|
|
|
∑
(
In general, (
|
)
)
) .
(
We are ready to study the long-term behavior of a Markov chain given a starting state, which does not
have to be a pure state. In general, is a probability distribution on , and can be represented by a
probability vector , where the th component is the probability that the system is in initially. The
probability distribution of
is
( )
Example
Swapping balls
(
( )
( )
(
( )
)(
(
)(
(
)(
)
)
(
)
)
(
)
)
Absorbing Markov Chains
Definitions
1.
2.
3.
4.
is accessible from if (
for some .
|
)
A state is absorbing if
(impossible to leave it).
A chain is absorbing if for every state , there is an absorbing state
from .
In an absorbing chain, a state is transient if it is not absorbing.
such that
is accessible
Example
Ball Swapping
(


)(
)
(
)
Every state is accessible from every state.
There are no absorbing states. The chain is not absorbing.
Example
Drunkard’s walk
Bar
Home
(


)
is not accessible from .
and are absorbing states. It is an absorbing chain because every state can reach
or
.
The first step in solving an absorbing chain is to put the transition matrix in the canonical form by
renumbering the states so that the transient states come first.
(
where
)
is the transition matrix for the transient states.
Example
Drunkard’s walk
(
)
It is straightforward to show (by matrix multiplication) that
(
where


depends on
As
and .
, ( )
has an inverse
(
|
)
Proof Let (
since
. Since (
. Then
)
(
) (
Let
,(
)
)
)
as
.
(called the fundamental matrix for ).
( )
has no non-zero solution,
)
(
)
is invertible. Note that
,
Theorem
Let
be transient.
1.
is the expected number of times a chain starting in
Proof
∑
2.
∑
visits .
∑ (
|
)
Let be the expected number of steps before the chain is absorbed given that the chain starts in
state , then
∑
Proof Follows from (1) above.
3.
Let
. Then
(absorbing).
Proof
is the probability that a chain starting in
∑
Example
∑∑
(transient) is absorbed in
∑ (∑
)
Drunkard’s walk
(
(
)
(
)
(
(
)
)
(
)
)
Note: There are many free programs online that calculate the inverse of a matrix.
1.
2.
3.
If a chain starts in
If a chain starts in
If a chain starts in
, the expected number it visits
, the expected number it visits
, the expected number it visits
If a chain starts in
, the expected number of steps before being absorbed is
If a chain starts in
, the expected number of steps before being absorbed is
If a chain starts in , the probability it is absorbed in
that it is absorbed in is
.
If a chain starts in , the probability it is absorbed in
.
before being absorbed is
before being absorbed is
before being absorbed is
is
.
.
.
, whereas the probability
= the probability that it is absorbed in
All these results make sense intuitively.
Summary
It is straightforward to implement the algorithm for solving a Markov chain. Specifically,
starting from the transition matrix in the canonical form, one computes the matrices and , whose
entries describe the long-term behavior of the chain.