Problem 1
Draw a diagram for the following Markov chains and specify the classes and determine
whether or not they are recurrent or transient.


0.1 0 0.9 0
0.1 0.2 0.7 0 
,
P =
0
1
0
0
0.2 0.3 0.3 0.2

1
0
0
0
0.1 0.2 0.3 0.4
,
P =
0
1
0
0
0.2 0.3 0.3 0.2



1 0
0
0
0 0.6 0 0.4
,
P =
0 1
0
0
0 0.3 0.3 0.4
Problem 2
For the Markov chain with the following transition matrix find the long-run proportion
of time that the process will be in each state.


0.5 0.5 0
0
0.1 0.2 0.3 0.4
,
P =
0
1
0
0
0.2 0.3 0.3 0.2
Problem 3
A man possesses n umbrellas which he uses in going from his home to office and vice
versa. If he is at home (the office) at the beginning (end) of a day, and it is raining, then
he will take an umbrella with him to the office (home), provided there is one to be taken. If
it is not raining, then he never takes an umbrella. Assume that, independent of the past, it
rains at the beginning (end) of a day with probability p.
a) Define Markov chain with n + 1 states which will help us to determine proportion of
time our man gets wet.
b) Show that the limiting probabilities are given by π0 = (1 − p)/(n + 1 − p), and
πi = 1/(n + 1 − p) for i = 1, ..., n.
c) What fraction of time our man get wet?
Problem 4
Customers arrive at a single-server station in accordance with a Poisson process with
rate λ. However, if the arrival finds n customers already in the station, then he (or she) will
enter the system with probability αn . Assuming an exponential service rate µ, set this up
as a birth and death process and determine the birth and death rates.
1