advertisement

STOCHASTIC MODELLING – WEEK 5 TUTORIAL STATIONARY PROBABILITIES, FIRST PASSAGES (1) (easy) Find the stationary probabilities of a the simple weather chain with S = {0, 1} (sunny, rainy) and 0.6 0.4 P = 0.2 0.8 Solution: We solve the global balance equations 0.6 0.4 (π0 , π1 ) = (π0 , π1 ) 0.2 0.8 with π0 + π1 = 1 to obtain π0 = 1/3 and π1 = 2/3. (2) (easy) A chain is called doubly P stochastic if all of its columns also sum to one (the rows already sum to one), that is i∈S pij = 1 for all j. Show that for an irreducible doubly stochastic chain with N states the stationary probabilities are πi = 1/N for all i. Solution: Just substitute πi = 1/N into the global balance equations X πi pij = i and check that P 1 X 1 pij = = πj N N i πi = 1. For an irreducible chain this is the unique stationary solution. (3) (easy) The detailed balance condition means that πi pij = πj pji for any i, P j ∈ S, that is the probability flow between any pair of states balances out (and additionally i πi = 1). Show that if the detailed balance condition is satisfied then the global balance condition is also satisfied. Solution: Sum both sides of the detailed balance condition over j to obtain the global balance condition X X πi pij = πi = πj pji j and P j πi = 1 was assumed. (4) (easy) Consider the following DTMC: 0.5 0.5 0 P = 0.3 0.1 0.6 0.2 0.4 0.4 (a) Calculate the stationary probabilities. (Hint: can you use your previous results?) (b) Show that it doesn’t satisfy detailed balance. Solution: (a) This chain is doubly stochastic, hence π = (1/3, 1/3, 1/3). (b) There cannot be detailed balance since p13 = 0, and p31 6= 0, which implies π3 = 0, and not 1/3. (5) (standard) Birth and death chains on a state space {`, ` + 1, . . . , r − 1, r} are defined by the property that they cannot move by more than one site in a single step (describing a single birth or death event or that nothing happened), that is pi,i+1 = pi , l≤i<r pi,i−1 = qi , `<i≤r pi,i, = 1 − pi − qi ` ≤ i ≤ r 2 SMO – WEEK 5 TUTORIAL with all pi , qi > 0, and all other transition probabilities are zero. Show that the stationary probabilities are p` p`+1 · · · p`+i−1 π`+i = π` q`+1 q`+2 · · · q`+i P where π` can be determined from the normalization ri=` πi = 1. (Hint: this chain has to satisfy detailed balance (why?), but you can also use the global balance equations). Solution: For any i < r state we can write down the detailed balance equation πi pi = πi+1 qi+1 , from which pi πi πi+1 = qi+1 Using this repeatedly starting from i = ` we get π`+1 = π`+2 = .. . p` q`+1 π` p` p`+1 q`+1 q`+2 π` which leads to the result. To prove it just use induction. (6) (standard) Random walk on graphs: A graph is a set of vertexes where some of them are connected by edges. The adjacency matrix A describes which vertexes are connected: aij = 1 if i and j vertexes are connected and zero otherwise. We assume P that aii = 0 for all i. The degree δi of vertex i is the number of edges connected to it, δi = j aij . Consider a random walk Xn which takes values from the vertexes, and in each time step it jumps to a randomly chosen connected vertex. (a) Write down the one-step transition probabilities pij . (b) Show that for a connected graph (why do we need this?) the stationary distribution is πi = cδi , and give the value of c. (Hint: use either detailed or global balance.) Solution: (a) aij δi (b) πi satisfies the detailed balance condition pij = πi pij = caij = caji = πj pji P and from i πi = 1 we get c = 1/ i δi . Alternatively, we can use the global balance equations X X aij X X πj = πi pij = cδi =c aij = c aji = cδj = πj δi P i i i i where we used that aij = aji be definition. (7) (standard) There are two rooms and there is a spider in the second room. A fly flies around every morning and lands in a room according to the one-step transition matrix 0.5 0.5 P = 0.6 0.4 If the fly lands in the room with the spider, then the spider immediately catches it with probability 0.7. (a) Formulate this problem as a DTMC. (Hint: we need an extra state for the fly getting caught.) (b) What is the expected time until the fly is caught by the spider, if the fly is initially in one of the rooms? (hint: condition on first step.) Solution: (a) We define the state space as S = {1, 2, D}, where 1 and 2 means the room where the fly is if it’s alive, and D corresponds to the state where the fly is dead. Then, the one SMO – WEEK 5 TUTORIAL 3 step transition probability can be written as 0.5 0.15 0.35 P = 0.6 0.12 0.28 0 0 1 (b) Let T̂ = min{n ≥ 0 : Xn = D}, and gi = E(T̂ |X0 = i), the expected number of steps starting from state i until the fly is caught. We know that gD = 0 and for the other states g1 = 1 + 0.5g1 + 0.15g2 + 0.35gD g2 = 1 + 0.6g1 + 0.12g2 + 0.28gD Solving these two equations we find that g1 = 2.94286 and g2 = 3.14286. So it’s better to start from room 2 if you are a fly. (8) (hard) A spider is hunting a fly and moves randomly between two rooms according to the following one-step transition matrix 0.3 0.7 P = 0.6 0.4 The fly, unaware of the spider, also moves according to the one-step transition matrix 0.5 0.5 P = 0.6 0.4 If spider and fly arrive into the same room, then the spider immediately catches the fly with probability 0.7. (a) Formulate this problem as a DTMC. (Hint: to describe the location of both animals we need a size four state space, but we need an extra state for the fly getting caught.) (b) What is the expected time until the fly is caught by the spider? Solution: (a) We define the state space as S = {(1, 1), (1, 2), (2, 1), (2, 2), D}. For example, Xn = (2, 1) means that the spider is in room 2 and the fly is in room 1 and alive. D corresponds to the state where the fly is dead. Then, the one step transition probability can be written as 0.045 0.15 0.35 0.105 0.35 0.054 0.12 0.42 0.084 0.332 P = 0.09 0.3 0.2 0.06 0.35 0.108 0.24 0.24 0.048 0.364 0 0 0 0 1 (b) Let again T̂ = min{n ≥ 0 : Xn = D}, and gi = E(T̂ |X0 = i), the expected number of steps starting from state i until the fly is caught. We know that gD = 0 and for the other states g11 = 1 + 0.045g11 + 0.15g12 + 0.35g21 + 0.105g22 + 0.35gD g12 = 1 + 0.054g11 + 0.12g12 + 0.42g21 + 0.084g22 + 0.35gD g21 = 1 + 0.09g11 + 0.3g12 + 0.2g21 + 0.06g22 + 0.35gD g22 = 1 + 0.108g11 + 0.24g12 + 0.24g21 + 0.048g22 + 0.35gD Solving these four equations we find that g11 = 2.89339, g12 = 2.97351, g21 = 2.90512, and g22 = 2.86067. Flies do not live long.