Solutions Markov Chains 2 3) Given the following one-step transition matrix of a Markov chain, determine the classes of the Markov chain and whether they are recurrent. 1 4 3 4 1 P 3 3 4 1 4 1 3 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 4 1 4 0 1 2 3 1 4 3 4 Drawing the transition diagram above, we see that states 1 and 2 communicate. Once you leave state 2, there is a finite probability you will never return. Therefore, state 2 is transient. States 3 and 4 communicate with each other but not with states 0, 1, or 2. Consequently, { 0, 1 } is recurrent { 2 } is transient { 3, 4 } is recurrent 4 Solutions Markov Chains 6 4) A computer is inspected at the end of every hour. It is found to be either working (up) or failed (down). If the computer is found to be up, the probability of its remaining up for the next hour is 0.90. It it is down, the computer is repaired, which may require more than one hour. Whenever, the computer is down (regardlewss of how long it has been down), the probability of its still being down 1 hour later is 0.35. a. Construct the one-step transition probability matrix. b. Find the expected first passage time from i to j for all i, j. Soln: Let, S = 0 if computer is down = 1 if computer is up Then, P 0.35 0.65 010 . 0.90 b. Find expected first passage times 01 1 P00 01 01 (1 P01 ) 1 01 1 1 154 . 1 P00 1 .35 Solutions Markov Chains 7 4) (cont.) b. Find expected first passage times P 0.35 0.65 010 . 0.90 01 1 P00 01 01 (1 P01 ) 1 01 1 1 154 . 1 P00 1 .35 00 1 P01 10 1 .65(10) 7.5 10 1 P11 10 10 (1 P11 ) 1 10 1 1 10.0 1 P11 1 .9 11 1 P10 01 1 .1(154 . ) 1154 . Solutions Markov Chains 8 4) (cont.) Alternative solution to b. P 0.35 0.65 010 . 0.90 0 .35 0 .1 1 1 .65 0 .9 1 1 0 1 1 1 0 Substituting (4) into (1) gives .1 1 .65 0 .1(1 0 ) .65 0 0 013 . And from (4), 1 0.87 (1) (2) (3) (4) Solutions Markov Chains 4) (cont.) Alternative solution to b. P 0.35 0.65 010 . 0.90 0 013 . 1 0.87 00 1 0 1 1 7.5 .13 1 11 1154 . 1 .87 9 Solutions Markov Chains 10 5) A manufacturer has a machine that, when operational at the beginning of a day, has a probability of 0.1 of breaking down sometime during the day. When this happens, the repair is done the next day and completed at the end of that day. a. Formulate the evolution of the status of the machine as a 3 state Markov Chain. b. Fine the expected first passage times from i to j. c. Suppose the machine has gone 20 full days without a breakdown since the last repair was completed. How many days do we expect until the next breakdown/repair? Soln: a. Let, S = 0 if machine running at day’s end | running at start = 1 if machine down at day’s end | running at start = 2 if machine running at day’s end | down at start P00 P{ X 1 0 | X 0 0 } P{ run at start and end day 1| run at start and end day 0 } 0.9 P10 P{ X 1 0 | X 0 1} P{ run at start and end day 1| run at start and down at end day 0 } not possible 0.0 Solutions Markov Chains 11 5) (cont.) a. S = 0 if machine running at day’s end | running at start = 1 if machine down at day’s end | running at start = 2 if machine running at day’s end | down at start Continuing in this fashion gives the one-step transition prob. 0.9 01 . 0.0 P 0.0 0.0 10 . 0.9 01 . 0.0 b. ij 1 Pik kj k j 01 1 P00 01 P02 21 01 0 1 1 10 1 P00 1 .1 Note: This makes intuitive sense. If the machine has a 10% chance of failing on any given day, then the expected number of days between failures is 10, (01 = 10). Solutions Markov Chains 12 5) (cont.) 0.9 01 . 0.0 P 0.0 0.0 10 . 0.9 01 . 0.0 02 1 P00 02 P01 12 .1 02 1 .1 12 01 1 P00 01 P02 21 01 1 1 10 1 P00 1 .1 0 00 1 P01 10 P02 20 1 .1(10) 00 2 0 0 12 1 P10 02 P11 12 1 10 1 P11 10 P12 20 1 20 11 1 P10 01 P12 21 1 21 21 1 P20 01 P22 21 1 .9 01 1 .9(10) 10 Solutions Markov Chains 5) (cont.) 0.9 01 . 0.0 P 0.0 0.0 10 . 0.9 01 . 0.0 20 1 P21 10 P22 20 1 .1 10 1 .1(1 20 ) 20 1.222 22 1 P20 02 P21 12 1 .9(11) .1(1) 11 Back substituting for 02, 10, 11 02 110 . 10 2.22 11 110 . 13 Solutions Markov Chains 14 5) (cont.) 00 2.0 10 2.22 20 122 . 01 10.0 11 110 . 21 10.0 02 110 . 12 10 . 22 110 . c. Suppose the machine has gone 20 full days without a breakdown since the last repair was completed. How many days do we expect until the next breakdown/repair? If we read this as the expected number of days to breakdown since the last repair, we are asking for 21 in which case, 21 10.0 If we read this as the expected number of days to breakdown and subsequent repair since the last repair, we are asking for 22. 22 110 . Again, this should make intuitive sense. A machine has a 10% chance of breaking down. Therefore, the expected time between failures is 10 days. Since it takes 1 day to repair, the time from repair to repair is 10+1 = 11 days.