MATH 5050: Homework 1 (Due Monday, Feb. 1, at the beginning of lecture)
Problem 1
Consider a simple random walk with absorbing boundaries on {0, 1, 2, . . . , 10}. From a
site k that is not 0 nor 10 the walk goes equally likely to k + 1 or k − 1. 0 and 10 are
absorbing states. The payoff function is [0, 2, 4, 3, 10, 0, 6, 4, 3, 3, 0].
a) Find the optimal strategy and optimal expected gain.
b) Find the optimal strategy and optimal expected gain if there is a discount factor of 0.9.
c) Find the optimal strategy and optimal expected gain if there each move costs 2.
d) Use a computer simulation to verify your answers.
Problem 2 Consider the Markov chain on state space {1, 2, . . . , 10} with transition matrix


1
0
0
0
0
0
0
0
0
0
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1


0
0
1
0
0
0
0
0
0
0


0.4 0.1 0
0
0 0.5 0
0
0
0


0.3 0.1 0

0
0.1
0.2
0
0.1
0.1
0.1
.
P =
0.2 0.1 0.1 0 0.1 0.4 0
0 0.1 0 


0.1 0.2 0.1 0 0.2 0.2 0
0 0.1 0.1


0
0
0
0
0
0
0
1
0
0


0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
0.2 0.1 0.1 0 0.2 0.1 0 0.1 0.1 0.1
The pay off function is [2, 4, 3, 10, 0, 6, 4, 3, 3, 0]. Solve the following questions numerically.
a) Find (approximately) the optimal strategy and optimal expected gain.
b) Find (approximately) the optimal strategy and optimal expected gain if there is a discount factor of 0.6.
c) Find (approximately) the optimal strategy and optimal expected gain if the cost of each
move out of the different states is given by [∞, 2, ∞, 1, 0, 0, 1, ∞, 3, 2].
d) Can you find (approximately) the optimal strategy and optimal expected gain if the cost
of each move out of the different states is given by [∞, 2, ∞, 1, 0, 0, 1, ∞, 3, 2] AND the pay
off is discounted by a factor of 0.6 at each move?
1