Problem sheet 2, solutions.
1(a). From the definition of covariance:
COV(X, Y ) = E[(X − E[X])(Y − E[Y ])]
= E[XY − XE[Y ] − E[X]Y + E[X]E[Y ]]
= E[XY ] − E[X]E[Y ] − E[X]E[Y ] + E[X]E[Y ]
= E[XY ] − E[X]E[Y ]
1(b). Since X and Y are independent, P (X, Y ) = P (X)P (Y ). Let X be the set
of all possible values of X and Y the set of all possible values of Y . Then:
XX
E[XY ] =
xyP (X = x, Y = y)
y∈Y x∈X
=
XX
xyP (X = x)P (Y = y)
y∈Y x∈X
=
X
X
xP (X = x) ×
x∈X
yP (Y = y)
y∈Y
= E[X]E[Y ]
Substituting into (1) above, gives COV(X, Y ) = 0.
1(c). Since Y = X 2 , E[XY ] = E[X 3 ] = 0. Also, we can see that E[X] = 0 and
therefore E[X]E[Y ] = 0. This gives COV(X, Y ) = E[XY ] − E[X]E[Y ] = 0.
2. Let p(x) represent the pdf of RV X. Then:
Z ∞
E[X] =
x p(x) dx
Z
Z0 a
x p(x) dx +
=
∞
x p(x) dx
a
0
Since p(x) is a pdf, it is everywhere non-negative, so the first term on the RHS
must be non-negative. This means:
Z ∞
x p(x) dx
E[X] ≥
a
1
Since a is the lower bound on the integral above, we can write
Z ∞
Z ∞
a p(x) dx
x p(x) dx ≥
a
a
which gives
E[X] ≥
∞
Z
a p(x) dx
a
= a
Z
∞
p(x) dx
a
= a P (X ≥ a)
from which the required result follows.
3. First, note that
P (|X − µX | ≥ a) = P ((X − µX )2 ≥ a2 )
Here, (X − µX )2 is a non-negative RV. Using the Markov inequality, we get:
P ((X − µX )2 ≥ a2 ) ≤
=
E[(X − µX )2 ]
a2
2
σX
a2
as required.
4. If θ̂n is unbiased, we can write
P (|θ̂n − θ| ≥ ) = P (|θ̂n − E[θ̂n ]| ≥ )
Applying the Chebyshev inequality to the RHS, we get:
P (|θ̂n − E[θ̂n ]| ≥ ) ≤
VAR(θ̂n )
2
From the RHS above we can see that if
lim VAR(θ̂n ) = 0
n→∞
the estimator converges in probability to θ, that is, it is consistent.
2
5. Let X̄n denote the sample mean derived from n observations. This is easily
shown to be unbiased. Using the Chebyshev inequality:
P (|X̄n − µX | ≥ ) ≤
VAR(X̄n )
2
But:
VAR(X̄n ) = VAR
=
1
(X1 + . . . + Xn )
n
2
σX
n
Therefore
2
σX
n2
P (|X̄n − µX | ≥ ) ≤
and
lim P (|X̄n − µX | ≥ ) = 0
n→∞
which means X̄n converges in probability to the true mean µX , as required.
6(a). Log-likelihood:
n
L(µ, Σ) = −
n
1X
dn
(Xi − µ)T Σ−1 (Xi − µ)
log(2π) − log(|Σ|) −
2
2
2
i=1
6(b). We proceed in two steps: we first treat Σ as fixed, and maximize L to get
a value µ̂(Σ) which maximizes L for a given matrix parameter Σ. Taking the
derivative of the L wrt vector µ, we get:
n
X
d
(Xi − µ))T
L = (Σ−1
dµ
i=1
Setting the derivative to zero, taking the transpose of both sides and pre-multiplying
by Σ, we get:
0 =
n
X
(Xi − µ)
i=1
3
Solving for µ:
n
1X
Xi
n
µ̂(Σ) =
i=1
= X̄
Since this solution does not depend on Σ, X̄ is the maximum likelihood estimator of µ for any Σ. To obtain Σ̂ we plug µ̂(Σ) = X̄ into the log-likelihood to
obtain
n
n
1X
dn
(Xi − X̄)T Σ−1 (Xi − X̄)
log(2π) − log(|Σ|) −
−
2
2
2
(1)
i=1
and maximize this function wrt Σ.
We first introduce a sample covariance matrix S defined as follows:
n
S =
1X
(Xi − X̄)(Xi − X̄)T
n
i=1
This allows us to re-write the quadratic form in (1) as a matrix trace:
n
X
(Xi − X̄)T Σ−1 (Xi − X̄) = n Tr(Σ−1 S)
i=1
where Tr(·) denotes the trace of its matrix argument.
This in turn allows us to write the derivative of (1) wrt Σ as follows:
−
n d
n d
log(|Σ|) −
Tr(Σ−1 S)
2 dΣ
2 dΣ
At this point we make use of two useful matrix derivatives (these can be found
in Appendix C of Bishop and the note “Matrix Identities” by Roweis, available on
the course website):
∂
log(|A|) = (A−1 )T
∂A
∂
Tr(X−1 A) = −X−1 AT X−1
∂X
This gives the derivative (2) in the following form (where we make use of the
fact that both Σ−1 and S are symmetric):
n
n
− Σ−1 + Σ−1 SΣ−1
2
2
4
Setting to zero and solving, we get:
Σ̂ = S
n
1X
=
(Xi − X̄)(Xi − X̄)T
n
i=1
5