2.7 Number of Visits to a State
2.7.1 Number of visits to a state
In section 2.6.1 we calculated the probability of reaching a state. In this section we want to calculate the
expected number of visits to a state in the case where this is not infinite. As an example we use the circuit
board manufacturing example used in section 2.6.1.
Example 1. A company manufactures circuit boards. There are four steps involved.
1.
Tinning
2.
Forming
3.
Insertion
4.
Soldering
There is a 5% chance that after the forming step a board has to go back to timing. There is a 20% chance
that after the insertion step the board has to be scrapped. There is a 30% chance that after the soldering step
the board has to go back to the insertion step and a 10% chance that the board has to be scrapped. What is
the expected number of visits to each of the states in the production of one good board. If a board is
scrapped then we start over with tinning and include the visits to each of the states for the scrapped board.
We make a small modification to the transition diagram used in the previous section. This time we go from
the scrapped state (5) back to the tinning state (1).
0.05
Tinning - 1
1
Forming - 2
0.3
0.95
0.8
Insertion - 3
0.2
0.6
Solder - 4
Success - 6
0.1
Scrap - 5
1
The transition matrix is now
0
(1)
P =
 0.05
 0
 01
 0
1 0
0 0.95
0 0
0 0.3
0 0
0 0
0
0
0
0
0.8 0.2
0 0.1
0
0
0
0
0
0
0
0.6
0
1




The number of visits to any particular state in the production of a good board is a random variable, since the
number of visits will vary with the particular board being produced.
2.7.1 - 1
Definition 1. If i and j are states in a Markov chain, let
(2)
N(j) = the total number of times that the system is in state j
(3)
sij = E{N(j) | X0 = i} = expected number of visits to state j if we start in state i
(4)
(5)
S =
 ss ss
...
...
s s
11
12
21
22
m1
m2
… s1m
… s2m
… smm




ST = restriction of S to the transient states, i.e. we only include rows and columns
corresponding to the transient states
where m is the total number of states. If the system is in state j at time n = 0, then we include this in the
count of the total number of times.
N(j) is a random variable that depends on the values of X0, X1, … It is possible that N(j) =  if the system is
in state j infinitely often and hence some or all of the sij will . In Example 1 we want to find
s11 = E{N(1) | X0 = 1}, s12 = E{N(2) | X0 = 1}, s13 = E{N(3) | X0 = 1}, s14 = E{N(4) | X0 = 1} and
s15 = E{N(5) | X0 = 1}.
The sij satisfy equations similar to the ones satisfied by the Fij in section 2.6.1. In order to make the
argument more concrete, let's first concentrate on finding s11 = E{N(1) | X0 = 1}, the expected number of
times we visit the tinning state in the production of one good board. Even though we are not particularly
interested in these values, we also find
(6)
si1 = si = E{N(1) | X0 = i} = expected number of visits to the tinning state 1 if we start in state i
for i = 1, 2, 3, 4, 5 and 6. As indicated, we are mostly interested in s1. Note that s6 = 0.
In (6) and the following we will omit the second subscript 1 for convenience. With that in mind, the si
satisfy a system of equations. Note that if i  1 then
si = E{N(1) | start in state i}
= Pr{next state is 1 | start in state i}  E{N(1) | start in state 1}
+ Pr{next state is 2 | start in state i}  E{N(1) | start in state 2}
+ Pr{next state is 3 | start in state i}  E{N(1) | start in state 3}
+ Pr{next state is 4 | start in state i}  E{N(1) | start in state 4}
+ Pr{next state is 5 | start in state i}  E{N(1) | start in state 5}
+ Pr{next state is 6 | start in state i}  E{N(1) | start in state 6}
= pi1s1 + pi2s2 + pi3s3 + pi4s4 + pi5s5 + pi6s6
2.7.1 - 2
Since s6 = 0 we get
si = pi1s1 + pi2s2 + pi3s3 + pi4s4 + pi5s5
If i = 1 then we have to add one to the right side for being in state 4 at time n = 0. So
s1 = 1 + p11s1 + p12s2 + p13s3 + p14s4 + p15s5
So the set of equations for s1, s2, s3, s4 and s5 is
s1 = p11s1 + p12s2 + p13s3 + p14s4 + p15s5 + 1
s2 = p21s1 + p22s2 + p23s3 + p24s4 + p15s5
(7)
s3 = p31s1 + p32s2 + p33s3 + p34s4 + p15s5
s4 = p41s1 + p42s2 + p43s3 + p44s4 + p15s5
s5 = p51s1 + p52s2 + p53s3 + p54s4 + p55s5
This is a system of five equations which we can solve for s1, s2, s3, s4, s5. To see the structure of the
equations we can put them in vector form.
s1
s
s
s
s
2
3
4
5




=
p11 p12
21 p22
31 p32
41 p42
51 p52
p
p
p
p
p13
p23
p33
p43
p53
p14
p24
p34
p44
p54
p15
p25
p35
p45
p55
s1
s
s
s
s
2
3
4
5




1
+
0
0
0
0
or
1
s = PTs +
0
0
0
0
or
1
(8)
(I – PT)s =
s1
s

s = s
s
s
2
where
3
4
5




0
0
0
0
PT
p11 p12
21 p22
31 p32
41 p42
51 p52
p

= p
p
p
p13
p23
p33
p43
p53
p14
p24
p34
p44
p54
p15
p25
p35
p45
p55




Note that PT is just the part of P corresponding to transient states 1, 2, 3, 4 and 5. We can write
2.7.1 - 3
1
0
0
0
0
s = (I – PT)-1
= [(I – PT)-1]●,1 = 1st column of (I – PT)-1
although it is usually faster to solve (7) directly using Gaussian elimination.
In our example
0
PT =
1 0
0 0.95
0 0
0 0.3
0 0
 0.05
 0
 0
 1
0
0
0
0
0.8 0.2
0 0.1
0
0




1
I - PT =
 - 0.05
 0
 0
 -1
-1
0
0
0
1 - 0.95 0
0
0
1
- 0.8 - 0.2
0 - 0.3
1
- 0.1
0
0
0
1




So the equations (7) are
1
 - 0.05
 0
 0
 -1
-1
0
0
0
1 - 0.95 0
0
0
1
- 0.8 - 0.2
0 - 0.3
1
- 0.1
0
0
0
1
s1
s
s
s
s
2
3
4
5




1
=
0
0
0
0
or
s1
- 0.05 s1
- s2
= 1
+ s2 - 0.95 s3
= 0
s3 - 0.8 s4 - 0.2 s5 = 0
-
0.3 s3 +
s1
s4 - 0.1 s5 = 0
-
s5 = 0
Adding the first equation to the second gives 0.95s1 - 0.95s3 = 1 or
Eqn 2
s1
- s3
= 20/19
s3 - 0.8 s4 - 0.2 s5 = 0
-
0.3 s3 +
s1
s4 - 0.1 s5 = 0
-
s5 = 0
where we omitted the first equation. Adding equation 2 to the third equation and subtracting 0.3 times
equation 2 to the fourth equation gives
2.7.1 - 4
Eqn 3
s1
- 0.8 s4 - 0.2 s5 = 20/19
- 0.3s1
+
s4 - 0.1 s5 = - 6/19
s1
-
s5 = 0
where we omitted the second equation. Subtracting 0.2 times the last equation from the third and 0.1 times
the last equation from the fourth gives
Eqn 3
0.8s1
- 0.8s4 = 20/19
- 0.4s1
+
s4 = - 6/19
or
Eqn 3
s1
-
- 0.4s1
+
s4 = 100/76
s4 = - 6/19
where we omitted the last equation. Adding the last equation to equation 3 gives
Eqn 3
0.6s1
= 1
So s1 = 5/3  1.67. So the expected number of visits to the tinning state in the production of one good
2
board is 1 . This includes the tinning of the initial board and 2/3 expected revists.
3
Using the same argument, one sees that
ST = (I – PT)-1
In particular
(9)
[(I – PT)-1]ij = sij = E{N(j) | X0 = i}
= expected total number of visits to state j if one starts in state i
Note that one needs state j to be transient in order for this to be finite if one can go from i to j. If state j is
recurrent then sij =  if one can go from state i to state j.
Using software one obtains for Example 1
(10)
(I – PT)
-1

1.67 1.67
1.67
0.61
0.35
1.67
 0.67
 0.61
 0.35
 1.67
2.08
2.08
2.08
0.83
2.08
1.67
1.67
1.67
1.67
1.67
2.7.1 - 5
0.58
0.58
0.58
0.33
1.58




The first row contains the answers to the original questions in Example 1, i.e.
1.67 = Expected total number of visits to the tinning state in the production of one good board
1.67 = Expected total number of visits to the forming state in the production of one good board
2.08 = Expected total number of visits to the insertion state in the production of one good board
1.67 = Expected total number of visits to the soldering state in the production of one good board
0.58 = Expected number of scrapped boards in the production of one good board
Profits and costs. Often there are profits or costs associated with each visit to various states and one wants
to find the expected total profit or cost.
Example 2. In the context of Example 1, suppose the cost of
tinning
= $ 18 = c1
forming
= $ 15 = c2
insertion
= $ 25 = c3
soldering
= $ 20 = c4
scrap
= - $ 2 = c5 (You get $2 for a scrapped board)
What is the expected total cost C of a board?
In the notation (2) one has
C = 18N(1) + 15N(2) + 25N(3) + 20N(4) - 2N(5)
E(C | X0 = 1) = 18E(N(1) | X0 = 1) + 15E(N(2) | X0 = 1) + 25E(N(3) | X0 = 1)
+ 20E(N(4) | X0 = 1) - 2E(N(5) | X0 = 1)
= 18s11 + 15s11 + 25s11 + 20s11 - 2s11
Using the values of s1j above one has
E(C | X0 = 1) = (18)(1.67) + (15)(1.67) + (25)(2.08) + (20)(1.67) - (2)(0.58)
= 30.06 + 25.05 + 52.00 + 33.40 - 1.16
= $ 139.35
We can put this example in a general framework if we let
c1
c =
c
c
c
c
2
3
4
5




2.7.1 - 6
Then
E(C | X0 = 1) = s11c1 + s12c2 + s13c3 + s14c4 + s15c5
c1
c

) c
c
c
= (s11, s12, s13, s14, s15
2
3
4
5




= (row 1 of (I – PT)-1) c
= ((I – PT)-1c)1
In general,
E(C | X0 = i) = ((I – PT)-1c)i
So
(I – PT)-1c = vector whose components are the expected total costs beginning in the
various states
In Example 1,
139.25
(I – PT)-1c =
 121.25 
 104.51 
 65.08 
 137.25 
Formulas. There are some useful formulas connecting the Fij, sij and i. Recall from section 2.6.1 that Fij
is the probability of visiting j if one starts at i. If j = i then Fii is the probability of returning to i if one starts
at i.
Proposition 1. If j is transient then
(11)
sjj =
1
1 - Fjj
Fjj = 1 -
1
sjj
If i and j are both transient, then
(12)
sij = Fijsjj
Proof.
sjj = (1 for starting out in state j) + (expected number of future visits to j)
Expected number of future visits =
(probability of returning to j) (expected number of visits to j starting at j) = Fjjsjj
2.7.1 - 7
Combining we get sjj = 1 + Fjjsjj or (1 – Fjj)sjj = 1 which gives the first equation in (11). The second follow
from the first by solving for Fjj. The proof of (12) is similar, i.e.
sij = (probability of reaching j) (expected number of visits to j starting at j) = Fijsjj
One can give a more detailed proof of (11) as follows. One has
Pr{N(j) = n | X0 = j} = (Fjj)n-1(1 – Fjj)
So

1
1
 n (Fjj)n-1(1 – Fjj) = (1 - Fjj)2 (1 – Fjj) = 1 - Fjj
E{N(j) | X0 = j} =
n=0
where we used the formula


n=0
n=0
 nxn-1 = 1/(1 – x)2 which follows by differentiating  xn = 1/(1 – x). //
Example 2. In Example 1, we saw that s11 = 5/3. By (12) one has F11 = 1 – 1/(5/3) = 1 – 3/5 = 2/5. This is
the probability of a board having to go to the tinning stage a second time. On the other hand, by (12) the
probability of a board being scrapped is F15 = S15/S55. Using (10) this is 0.583/1.583 = 0.368. So, the
probability of a board not being scrapped is 1 – 0.368 = 0.632 which agrees with what we got in section
2.6.1.
Here is another formula for sij = E{N(j) | X0 = i}. It is more useful for theoretical purposes since computing
Pn involves finding the eigenvalues and eigenvectors of P which requires mathematical software.
Proposition 2.

 (Pn)ij
E{N(j) | X0 = i} =
(13)
n=0
1
Proof. N =  vXn where vk = 
n=0
0
if k = j

. So E{N(j) | X0 = i} =  E{vXn | X0 = i}. However, we saw
n=0
if k  j
n
n
in section 2.3.1 that E{vXn | X0 = i} =  (P )ikvk = (P )ij. Combining proves (13).
(j)

k
Expected time to reach a state. Something that is related to the expected number of visits to a state is the
expected time to reach a state. Recall from section 2.5.1 that
T(j) = first time (greater than or equal to one) that the system is in state j
In section 2.6.1 we saw how to determine the probability of reaching a state j if we start at a state i, i.e.
Pr{T(j) <  | X0 = i}. In the case where the probability of reaching a state j is 1 if we start at a state i we
sometimes want to compute the expected time to reach j if we start at i. This can be done using an argument
similar to computing the expected number of visits to j.
2.7.1 - 8
Definition 2. If i and j are states in a Markov chain, let
(14)
uij = E{T(j) | X0 = i} = expected time to reach state j if we start in state i
If i = j, then this is the expected time to return to j.
The uij satisfy equations similar to the ones satisfied by the sij. Note that
uij = E{T(j) | start in state i}
(15)
= 1 +
 Pr{next state is k | start in state i}  E{T(j) | start in state k}
kj
= 1 +
 pikukj
kj
We don't include j in the sum, since if the next state is j then we have reached j and there is no additional
time required. We just solve these equations for the uij where i  j. Then we can use this to compute ujj.
However, another way to compute ujj is given by Proposition 2 below.
Example 3. Mary likes to drink coffee, tea and milk. If her current
0.2
0.7
drink is one of these then the probability her next drink is another one of
these is given in the transition diagram at the right. The transition matrix
0.5
Coffee - 1
Tea - 2
is
0.4
P =
 0.2 0.5 0.3 
 0.1 0.7 0.2 
 0.4 0.3 0.3 
0.1
0.2
0.3
0.3
Milk - 3
Suppose Mary is drinking tea now. What is the expected number of
0.3
drinks until she drinks milk. This includes the milk, but not the tea she is drinking now.
In Example 3 we want to find u23 = E{T(3) | X0 = 2}. The equations (15) for u23 also involve,
u13 = E{T(3) | X0 = 1}. These equations are
u13 = 1 + 0.2u13 + 0.5u23
u23 = 1 + 0.1u13 + 0.7u23
or
0.8u13 - 0.5u23 = 1
- 0.1u13 + 0.3u23 = 1
Multiply the second equation by 8 and add to the first equation giving 1.9u23 = 9 or u23 = 90/19  4.74. So
an the average, Mary would drink 4.74 drinks until she drinks Milk. We can use the second equation to get
u13. It gives u13 = 3u23 – 10 = 80/19  4.21.
To get u33 we use (15) again which gives u33 = 1 + 0.4u13 + 0.3u23 = 78/19  4.11.
2.7.1 - 9
The following Theorem says that ujj and the steady probability j are reciprocals.
Theorem 3. If j is recurrent and j = lim Pr{ Xn = j | X0 = j} is the steady state probability of being in
n
state j, then
(16)
1
j = E{T(j) | X = j}
0
The proof somewhat difficult and will be omitted.
Example 3 (continued). Check that (16) holds for 3 in Example 3.
The i satisfy the equations P =  together with the fact that  is a probability vector. In Example 3, these
equations are
0.21 + 0.12
0.51 + 0.72
0.31 + 0.22
1 +
2
+ 0.43
+ 0.33
+ 0.33
+
3
=
=
=
=
1
2
3
1
The third equation is the negative of the sum of the first and second, so we omit it. Then we have
1 +
2 +
3 = 1
0.81 - 0.12 - 0.43 = 0
- 0.51 + 0.32 - 0.33 = 0
or
1 +
81 - 51 +
2 +
2 32 -
3 = 1
43 = 0
33 = 0
Subtracting 8 times the first equation from the second and adding 5 times the first equation to the third gives
92 + 123 = 8
82 + 23 = 5
Subtracting 6 times the second equation from the first gives 39 2 = 22 or 2 = 22/39  0.564. Then
3 = (5 - 82)/2 = 19/78  0.244. Finally, 1 = 1 - 2 - 3 = 15/78  0.192. Applying Theorem 3 we get
1
E{T(1) | X0 = 1} =
1
E{T(2) | X0 = 2} =
2
E{T(3) | X0 = 3} =
3
1
1
=
78
 5.2
15
=
78
 1.77
44
=
78
 4.11
19
2.7.1 - 10