MTHE/MATH 472/872
Winter 2016
Solutions to Assignment 1
Problem 1
a) Let Fβ be a σ−field of subsets of some space X for all β ∈ Γ where Γ is some set. Let
F=
\
Fβ
β∈Γ
Show that F is also a σ−field on X.
For a space X, on which a distance metric is defined, the Borel σ−field is generated by the collection
of open sets. This means that, the Borel σ−field is the smallest σ−field containing open sets, and
as such it is the intersection of all σ-fields containing open sets. On R, the Borel σ−field is the
smallest σ−field containing open intervals.
b) Is the set of rational numbers an element of the Borel σ-field on R? Is the set of irrational
numbers an element?
c) [Optional ]Let X be a countable set. On this set, let us define a metric as follows:
(
0, if x = y
d(x, y) =
1, if x 6= y
Show that, the Borel σ−field on X is generated by the individual sets {x}, x ∈ X.
d) [Optional] Consider the distance function d(x, y) defined as above. Is the σ−field generated by
open sets according to this metric the same as the Borel σ−field on R?
Solution:
a) Let {Ak } ∈ F. This implies that {Ak } ∈ Fα for all α. This implies that ∪k Ak ∈ Fα for all α
and as such ∪k Ak ∈ F.
Likewise, if A ∈ F, with the same reasoning as above, Ω − A ∈ F, and Ω itself is also in F.
Thus F is a σ−field.
b) We observed in class that ∩∞
n=1 (a − 1/n, a + 1/n) = {a} ∈ B(R). Since the rational numbers are
a countable collection of such points with each a being rational, the set ∪a∈Q {a} ∈ B(R).
Since the irrationals are R \ Q, this set is also in B(R) since the entire R is in B(R).
c-d) The Borel σ-field is generated by open sets.
Recall that for a general metric space (X, d), open sets are those of the form {x : d(x, y) < r} for
r ∈ [0, ∞) for some x ∈ X. More precisely, a set U is open, if for every x ∈ U , there exists a γ > 0
such that {y : d(x, y) < γ} ⊂ U .
Since the set is discrete; and the distance metric is either zero or one, all singletons {x} are open,
therefore, all unions of such sets are open. As such, the σ−field generated by such open sets is the
full power set.
For the uncountable space, each singleton set {x}, x ∈ X is open, and all subsets, by this definition
is open. Hence, the sigma-field general by open sets is the full power set.
Note that, the above is not the same as the σ−field generated by the individual (singleton) sets for
an uncountable space. The sigma field generated by such sets is the smallest set which is closed
under countable union, and intersection and complements of the individual sets {x} ⊂ X.
Let M be the collection of those subsets of R which contain either countably many elements or is
the complements of such sets and the empty set or the entire R. For two sets which are countable,
their union is also countable. Furthermore, the countable union of a countable collection is also
countable. Thus, M is a sigma-field. This sigma-field contains all the singletons in R. Therefore,
the sigma-field generated by individual points has to be at most M, which does not contain open
intervals. Therefore, the sigma-fields are not the same. The σ−algebra generated by individual
points is not the same as the power set on R
Problem 2
a) Let X and Y be real-valued random variables defined on a given probability space (Ω, F, P ).
Show that X + Y is also a random variable.
b) Let F be a σ−field of subsets over a set X and let A ∈ F. Prove that {A ∩ B, B ∈ F} is a
σ−field over A (that is a σ−field of subsets of A).
Hint for part a: The following holds: X + Y ≤ x = ∪r∈Q {X ≤ r, Y ≤ x − r}. To check if X + Y
is a random variable, it suffices to check if the event {X + Y ≤ x} = {ω : X(ω) + Y (ω) ≤ x} is an
element of F for every x ∈ R.
Solution:
a) We observe that for any x,
{ω : X(ω) + Y (ω) ∈ (−∞, x]} = ∪r∈Q {ω : Y (ω) ≤ r} ∩ {ω : X(ω) ≤ x − r} ∈ F.
Hence, the sum is a random variable (For measurability, it suffices to check the property that
X −1 (B) ∈ F, for B of the form B = (a, b), as opposed to all sets B in Borel σ−field on R).
b) Let M be the set {A ∩ B, B ∈ F}. If A1 , A2 ∈ M, there exist B1 , B2 ∈ F so that B1 ∩ B2 F, and
thus, A ∩ (B1 ∩ B2 ) ∈ M. This set is equal to A1 ∩ A2 . This reasoning applies for any countable
collection of sets {An }. The discussion also applies for the complements: A1 \ A2 = A1 ∩ AC
2 , and
the analysis follows.
A ∈ M since A = A ∩ X and so is ∅.
Problem 3
Consider a line of customers at a service station (such as at an airport, a grocery store, or a
communication network where customers are packets). Let Lt be the length of the line, that is the
total number of customers waiting in the line.
Let there be Mt servers, serving the customers at time t. Let there be a manager (controller) who
decides on the number of servers to be present. Let each of the servers be able to serve N customers
for every time-stage. The dynamics of the line can be expressed as follows:
Lt+1 = Lt + At − Mt N 1(Lt ≥N Mt ) ,
where 1(E) is the indicator function for event E, i.e., it is equal to zero if E does not occur and is
equal to 1 otherwise. In the equation above, At is the number of customers that have just arrived
at time t. We assume {At } to be an independent process, and to have the following distribution
P (A(t) = k) =
λk e−λ
,
k!
k ∈ {0, 1, 2, . . . , }
The manager has only access to the information vector
It = {L0 , L1 , . . . , Lt ; A0 , A1 , . . . , At−1 },
while implementing his policies. A consultant proposes a number of possible policies to be adopted
by the manager: According to Policy A, the number of servers is given by:
Mt =
Lt + Lt−1 + Lt−3
.
2
According to Policy B, the number of servers is
Mt =
Lt+1 + Lt + Lt−1
.
2
According to Policy C,
Mt = ⌈λ + 0.1⌉1t≥10 .
Finally, according to Policy D, the update is
Mt = ⌈λ + 0.1⌉
a) Which of these policies are admissible, that is measurable with respect to the σ−field generated
by It ? Which are Markov, or stationary?
b) Consider the model above. Further, suppose policy D is adopted by the manager. Consider the
induced Markov chain by the policy, that is, consider a Markov chain with the following dynamics:
Lt+1 = Lt + At − ⌈λ + 0.1⌉N 1(Lt ≥⌈λ+0.1⌉)N ,
with At as described the the earlier question.
Is this Markov chain irreducible?
Is there an absorbing set in this Markov chain? If so, what is it?
Solution:
a) Policies A, C, D are measurable with respect to σ(It ). Policy B is not measurable since
Lt+1 + Lt + Lt−1
= a} ∈
/ σ(It ),
2
is not measurable on σ(It ) and
{{Lt−1 , Lt , Lt+1 } :
since Lt+1
∀a ∈ Z+ ,
{{Lt+1 } : Lt+1 = 2a − Lt − Lt−1 } ∈
/ σ(It ).
If we expand the state space such that we define L̄t = {Lt , Lt−1 , Lt−2 , Lt−3 }, then (L̄t , Mt ) will
form a Markov chain and according to this definition, policy A is Markov. Policy C is Markov and
D is stationary.
b) The chain is irreducible since for any a, b ∈ Z+ , there exists a sequence of events and an n ∈ Z+
such that P (Lt+n = b|Lt = a) > 0. For b > N ⌈λ + 0.1⌉ and a > N ⌈λ + 0.1⌉, this is immediate.
For b < N ⌈λ + 0.1⌉, one can make the following argument. The state will eventually hit any of the
states where there are {N ⌈λ + 0.1⌉, N ⌈λ + 0.1⌉ + 1, N ⌈λ + 0.1⌉ + 2, . . . , 2N ⌈λ + 0.1⌉ − 1} customers.
Hence, the chain is irreducible on Z+ . The entire state space Z+ is absorbing.
Problem 4
Consider a service station with the following description: Consider a line where at any given
time there can be at most N customers, N ∈ N. The probability description is such that for
j = 1, 2, . . . , N − 1:
P (Lt+1 = j − 1|Lt = j) = α
P (Lt+1 = j + 1|Lt = j) = β
and
P (Lt+1 = 1|Lt = 0) = β
P (Lt+1 = 0|Lt = 0) = α
P (Lt+1 = N |Lt = N ) = β
P (Lt+1 = N − 1|Lt = N ) = α
When there are N customers (that is the line is full), no other customers can enter the line, and
the newly arriving customers are declined service.
Find the invariant probability measure for this service station.
Solution:
The Markov Chain is irreducible. As such, there exists a unique invariant distribution (by Dobrushin’s Lemma).
For sake of presentation, let N = 4. We obtain a finite-state Markov chain such that the invariant
distribution satisfies the following:

α
α

π(0) π(1) π(2) π(3) π(4) = π(0) π(1) π(2) π(3) π(4) 
0
0
0
β
0
α
0
0
0
β
0
α
0
0
0
β
0
α

0
0

0

β
β
Now, for a general N ∈ N+ , the invariant distribution satisfies the following recursions:
π(0) = απ(0) + απ(1),
leading to π(1) = αβ π(0) since α + β = 0. Likewise
π(1) = βπ(0) + απ(2),
leading to
β
π(2) = π(0)( )2 ,
α
and
π(2) = βπ(1) + απ(3),
leading to
β
π(3) = π(0)( )3 .
α
Likewise As such for all 1 ≤ k ≤ N , we have
β
π(k) = π(0)( )k .
α
For k = N , it follows that, through the equation We now compute π(0). Since
P
β k
π(0) N
k=0 ( α ) , it follows that
1 − αβ
,
π(0) =
1 − ( αβ )N +1
PN
k=0 π(i)
=1=
The probability of being declined service is equal to the probability that the line is full, that is the
probability that there are exactly N customers. This is equal to π(N ) =
β
1− α
β N
β N+1 ( α ) .
)
1−( α
(We note that, the probability of being declined is conditioned on an arrival of customers; the
arrival is independent on the number of customers, as such, the number of customers is statistically
independent of the event of customers arriving; the probability of the line being full is the same as
the probability of the line being full given that there is a new customer).
Problem 5
Consider a Markov chain evolving in a countable set X.
Recall that the period of a state i is defined as the greatest common divisor of k ∈ N+ values for
which P (xt+k = i|x0 = i) > 0. Prove that, if the chain is irreducible, then all states must have the
same period.
Hint: Irreducibility implies that, for any two states there exists a k1 ∈ N+ such that one state can
move to the other one, and k2 ∈ N+ such that moving in the other direction can happen.
Solution:
Suppose the periods for two states are different, that is for some state i the period is P1 and for
another state j the period is P2 .
Irreducibility implies that, from i to j there exists a k1 ∈ Z+ such that the chain can move from i
to state j, and k2 ∈ Z+ such that moving in the other direction can happen.
Now, from i to j and back, the chain might travel in k1 + ζ1 P2 + k2 stages and likewise from j to i
back to j the chain might travel in k2 + ζ2 P1 + k2 time states, for some ζ1 , ζ2 non-negative integers.
Since the greatest common divisor of
{k1 + k2 , k1 + ζ1 P2 + k2 }
is P1 , it must be that k1 + k2 is a multiple of P1 . Likewise, k1 + k2 is also a multiple of P2 by a
symmetric argument. Suppose that P2 > P1 . We will show that this can’t be true. Since
{k1 + k2 , k1 + ζ1 P2 + k2 }
divides P1 , and it is possible to move from state i back to state i in k1 + k2 stages, it must be that
{k1 + k2 + k1 + k2 , k1 + ζ1 P2 + k2 + k1 + k2 }
must divide P2 . Hence, all of these integers must also divide P1 and P2 . Hence P1 cannot be the
greatest common divisor if it is less than P2 . A symmetric argument leads to the conclusion that
P1 = P2 .
Problem 6
Let {xt , t ∈ N} be an irreducible Markov chain taking values in a countable space X. Let a hitting
time be defined by τA = min(t > 0 : xt ∈ A) where A ⊂ X. Let x0 ∈ A.
Show that for n > 1:
Px (τA = n) =
X
P (x, y)Py (τA = n − 1).
y ∈A
/
and for n = 1,
Px (τA = n) =
X
P (x, y).
y∈A
Solution:
It follows that for n = 1:
Px (τA = 1) = Px (x1 ∈ A) =
X
P (x, y).
y∈A
For n > 1:
Px (τA = n) =
X
y∈X
Problem 7
P (x, y)Py (τA = n − 1, y ∈
/ A) =
X
y ∈A
/
P (x, y)Py (τA = n − 1).
(1)
Consider a Markov chain {xt } with a countable state space X. For any set A ∈ X, the occupation
time ηA is the number of visits of {xt , t ∈ Z+ } to set A given by
ηA =
∞
X
1(xt ∈A) ,
t=1
where 1(E) denotes the indicator function for an event E, that is, it takes the value 1 when E takes
place, and is otherwise 0. Let τA = min{t > 0 : xt ∈ A}, be the first (entry) time that the state
visits the set A.
Prove that if Pi (τi < ∞) = 1, then Pi (ηi = ∞) = 1, where {i} ⊂ X.
Solution:
See the solution provided in class or Theorem 3.6 in the lecture notes.
Problem 8
Suppose that there are two decision makers DM1 and DM2 . Suppose that the information available
to to DM1 is a random variable y 1 and the information available to DM2 is y 2 , where these random
variables are defined on a probability space (Ω, F, P ). Suppose that y i is Yi -valued and these spaces
admit a Borel -field B(Yi ).
Suppose that the sigma-field generated by y 1 is a subset of the sigma-field generated by y 2 , that is
σ(y 1 ) ⊂ σ(y 2 ).
Further, suppose that the decision makers wish to minimize the following cost function:
E[c(ω, u)],
where c : Ω × U → R+ is a measurable cost function (on F × B(U)), where B(U) is a σ-field over U.
Here, for i = 1, 2, ui = γ i (y i ) is generated by a measurable function γ i on the sigma-field generated
by the random variable y i . Let Γi denote the space of all such policies.
Prove that
inf E[c(ω, u1 )] ≥ inf E[c(ω, u2 )].
γ 1 ∈Γ1
γ 2 ∈Γ2
Hint: Make the argument that every policy u1 = γ 1 (y 1 ) can be expressed as u2 = γ 2 (y 2 ) for some
γ 2 ∈ Γ2 . In particular, for any B ∈ B(U), γ −1 (B) ∈ B(Yi ). Furthermore, (y 1 )−1 (γ −1 (B)) ∈
σ(y 1 ) ⊂ σ(y 2 ). Following such a reasoning, the result can be obtained.
Solution:
Any policy achievable by DM1 is also achievable by DM2 , since any policy of DM1 is measurable
on σ(Y 1 ) (that is, DM 2 knows at least as much as DM 1 knows). Let Π1 be a policy for DM1 and
let J(Π1 ) = E[c(ω, Π1 (ω))]. As a result, any sequence of policies Π1n leading to a sequence J(Π1n )
which converge to the infimum, is also achievable by DM2 .