S OLUTION M ANUAL F OR
PATTERN R ECOGNITION AND M ACHINE
L EARNING
E DITED B Y
ZHENGQI GAO
the State Key Lab. of ASIC and System
School of Microelectronics
Fudan University
N OV.2017
1
0.1
Introduction
Problem 1.1 Solution
We let the derivative of error function E with respect to vector w equals
E
to 0, (i.e. ∂∂w
= 0), and this will be the solution of w = {w i } which minimizes
error function E. To solve this problem, we will calculate the derivative of E
with respect to every w i , and let them equal to 0 instead. Based on (1.1) and
(1.2) we can obtain :
=>
∂E
∂w i
=>
N
X
=
{ y(xn , w) − t n } xni = 0
n=1
N
X
y(xn , w)xni =
xni t n
n=1
n=1
=>
N
X
N X
M
N
X
X
j
( w j xn )xni =
xni t n
n=1 j =0
=>
N X
M
X
n=1
( j+ i)
w j xn
=
n=1 j =0
=>
M X
N
X
N
X
xni t n
n=1
( j+ i)
xn
wj =
N
X
xni t n
j =0 n=1
n=1
i+ j
PN
PN
If we denote A i j = n=1 xn and T i = n=1 xn i t n , the equation above can
be written exactly as (1.222), Therefore the problem is solved.
Problem 1.2 Solution
This problem is similar to Prob.1.1, and the only difference is the last
term on the right side of (1.4), the penalty term. So we will do the same thing
as in Prob.1.1 :
=>
∂E
∂w i
=>
=
N
X
n=1
M X
N
X
j =0 n=1
=>
{ y(xn , w) − t n } xni + λw i = 0
( j+ i)
xn
w j + λw i =
N
X
xni t n
n=1
M X
N
N
X
X
( j+ i)
{
xn + δ ji λ}w j =
xni t n
j =0 n=1
n=1
2
where
δ ji
(
0
j 6= i
1
j=i
Problem 1.3 Solution
This problem can be solved by Bayes’ theorem. The probability of selecting
an apple P(a) :
P(a) = P(a| r)P(r) + P(a| b)P(b) + P(a| g)P(g) =
3
1
3
× 0.2 + × 0.2 + × 0.6 = 0.34
10
2
10
Based on Bayes’ theorem, the probability of an selected orange coming
from the green box P(g| o) :
P(g| o) =
P(o| g)P(g)
P(o)
We calculate the probability of selecting an orange P(o) first :
P(o) = P(o| r)P(r) + P(o| b)P(b) + P(o| g)P(g) =
4
1
3
× 0.2 + × 0.2 + × 0.6 = 0.36
10
2
10
Therefore we can get :
3
P(o| g)P(g) 10 × 0.6
P(g| o) =
=
= 0.5
P(o)
0.36
Problem 1.4 Solution
This problem needs knowledge about calculus, especially about Chain
rule. We calculate the derivative of P y (y) with respect to y, according to
(1.27) :
d p y (y)
dy
=
d(p x (g(y))| g‘ (y)|) d p x (g(y)) ‘
d | g‘ (y)|
=
| g (y)| + p x (g(y))
dy
dy
dy
(∗)
The first term in the above equation can be further simplified:
d p x (g(y)) ‘
d p x (g(y)) d g(y) ‘
| g (y)| =
| g (y)|
dy
d g(y)
dy
(∗∗)
If x̂ is the maximum of density over x, we can obtain :
d p x (x) ¯¯
=0
dx x̂
Therefore, when y = ŷ, s.t. x̂ = g( ŷ), the first term on the right side of (∗∗)
will be 0, leading the first term in (∗) equals to 0, however because of the
existence of the second term in (∗), the derivative may not equal to 0. But
3
when linear transformation is applied, the second term in (∗) will vanish,
(e.g. x = a y + b). A simple example can be shown by :
p x (x) = 2x,
x ∈ [0, 1]
=>
x̂ = 1
And given that:
x = sin(y)
Therefore, p y (y) = 2 sin(y) | cos(y)|, y ∈ [0, π2 ], which can be simplified :
π
y ∈ [0, ]
2
p y (y) = sin(2y),
=>
ŷ =
π
4
However, it is quite obvious :
x̂ 6= sin( ŷ)
Problem 1.5 Solution
This problem takes advantage of the property of expectation:
var[ f ] = E[( f (x) − E[ f (x)])2 ]
= E[ f (x)2 − 2 f (x)E[ f (x)] + E[ f (x)]2 ]
= E[ f (x)2 ] − 2E[ f (x)]2 + E[ f (x)]2
=> var[ f ] = E[ f (x)2 ] − E[ f (x)]2
Problem 1.6 Solution
Based on (1.41), we only need to prove when x and y is independent,
E x,y [x y] = E[x]E[y]. Because x and y is independent, we have :
p(x, y) = p x (x) p y (y)
Therefore:
Z Z
Z Z
x yp(x, y) dx d y =
x yp x (x)p y (y) dx d y
Z
Z
= ( xp x (x) dx)( yp y (y) d y)
=> E x,y [x y] = E[x]E[y]
Problem 1.7 Solution
We need Integration by substitution.
Z +∞ Z +∞
1
1
2
I
=
exp(− 2 x2 − 2 y2 ) dx d y
2σ
2σ
−∞ −∞
Z 2π Z +∞
1 2
=
exp(− 2 r )r dr d θ
2σ
0
0
4
Here we utilize :
x = r cos θ ,
y = r sin θ
Based on the fact :
Z +∞
r2
r 2 ¯+∞
exp(− 2 )r dr = −σ2 exp(− 2 )¯0 = −σ2 (0 − 1) = σ2
2σ
2σ
0
Therefore, I can be solved :
2
Z 2π
I =
0
σ2 d θ = 2πσ2 ,
=> I =
p
2πσ
¯
And next,we will
show that Gaussian distribution N (x¯µ, σ2 ) is normal¯
R +∞
ized, (i.e. −∞ N (x¯µ, σ2 ) dx = 1) :
Z +∞
−∞
¯
N (x¯µ, σ2 ) dx
Z +∞
1
1
exp{− 2 (x − µ)2 } dx
p
2
2σ
−∞
2πσ
Z +∞
1
1
=
exp{− 2 y2 } d y ( y = x − µ )
p
2σ
−∞
2πσ2
Z +∞
1
1
= p
exp{− 2 y2 } d y
2σ
2πσ2 −∞
= 1
=
Problem 1.8 Solution
The first question will need the result of Prob.1.7 :
Z +∞
Z +∞
¯
1
1
N (x¯µ, σ2 )x dx =
exp{− 2 (x − µ)2 } x dx
p
2σ
−∞
−∞
2πσ2
Z +∞
1
1
=
exp{− 2 y2 }(y + µ) d y (y = x − µ)
p
2
2
σ
−∞
2πσ
Z +∞
Z +∞
1
1
1 2
1
= µ
exp{− 2 y } d y +
exp{− 2 y2 } y d y
p
p
2
2
2σ
2σ
−∞
−∞
2πσ
2πσ
= µ+0 = µ
The second problem has already be given hint in the description. Given
that :
d( f g)
dg
df
= f
+g
dx
dx
dx
We differentiate both side of (1.127) with respect to σ2 , we will obtain :
Z +∞
−∞
(−
¯
(x − µ)2
1
+
)N (x¯µ, σ2 ) dx = 0
2
4
2σ
2σ
5
Provided the fact that σ 6= 0, we can get:
Z +∞
Z +∞
¯
¯
2
2
¯
(x − µ) N (x µ, σ ) dx =
σ2 N (x¯µ, σ2 ) dx = σ2
−∞
−∞
So the equation above has actually proven (1.51), according to the definition:
Z +∞
¯
var[x] =
(x − E[x])2 N (x¯µ, σ2 ) dx
−∞
Where E[x] = µ has already been proved. Therefore :
var[x] = σ2
Finally,
E[x2 ] = var[x] + E[x]2 = σ2 + µ2
Problem 1.9 Solution
Here we only focus on (1.52), because (1.52) is the general form of (1.42).
Based on the definition : The maximum of distribution is known as its mode
and (1.52), we can obtain :
¯
∂N (x¯µ, Σ)
∂x
¯
1
= − [Σ−1 + (Σ−1 )T ] (x − µ)N (x¯µ, Σ)
2
¯
= −Σ−1 (x − µ)N (x¯µ, Σ)
Where we take advantage of :
∂xT Ax
∂x
= (A + AT )x
Therefore,
only when x = µ,
and
(Σ−1 )T = Σ−1
¯
∂N (x¯µ, Σ)
∂x
=0
Note: You may also need to calculate Hessian Matrix to prove that it is
maximum. However, here we find that the first derivative only has one root.
Based on the description in the problem, this point should be maximum point.
Problem 1.10 Solution
We will solve this problem based on the definition of expectation, variation
6
and independence.
E[x + z] =
Z Z
(x + z)p(x, z) dx dz
Z Z
(x + z)p(x)p(z) dx dz
Z Z
=
xp(x)p(z) dx dz +
z p(x)p(z) dx dz
Z Z
Z Z
=
( p(z) dz)xp(x) dx + ( p(x) dx)z p(z) dz
Z
Z
=
xp(x) dx + z p(z) dz
=
Z Z
= E[x] + E[z]
Z Z
var[x + z] =
( x + z − E[x + z] )2 p(x, z) dx dz
Z Z
=
{ ( x + z )2 − 2 (x + z) E[x + z]) + E2 [x + z] } p(x, z) dx dz
Z Z
Z Z
2
(x + z) p(x, z) dx dz − 2E[x + z]
(x + z)p(x, z) dx dz + E2 [x + z]
Z Z
(x + z)2 p(x, z) dx dz − E2 [x + z]
Z Z
(x2 + 2xz + z2 ) p(x)p(z) dx dz − E2 [x + z]
Z Z
Z Z
Z Z
2
( p(z)dz) x p(x) dx +
2xz p(x)p(z) dx dz + ( p(x)dx) z2 p(z) dz − E2 [x + z]
Z Z
E[x2 ] + E[z2 ] − E2 [x + z] +
2xz p(x)p(z) dx dz
Z Z
E[x2 ] + E[z2 ] − (E[x] + E[z])2 +
2xz p(x)p(z) dx dz
Z Z
E[x2 ] − E2 [x] + E[z2 ] − E2 [z] − 2E[x] E[z] + 2
xz p(x)p(z) dx dz
Z
Z
var[x] + var[z] − 2E[x] E[z] + 2( xp(x) dx)( z p(z) dz)
=
var[x] + var[z]
=
=
=
=
=
=
=
=
Problem 1.11 Solution
Based on prior knowledge that µ ML and σ2ML will decouple. We will first
calculate µ ML :
¯
N
d(ln p(x¯ µ, σ2 ))
1 X
= 2
(xn − µ)
dµ
σ n=1
We let :
¯
d(ln p(x¯ µ, σ2 ))
dµ
=0
7
Therefore :
µ ML =
N
1 X
xn
N n=1
And because:
¯
d(ln p(x¯ µ, σ2 ))
d σ2
We let :
=
N
1 X
(
(xn − µ)2 − N σ2 )
2σ4 n=1
¯
d(ln p(x¯ µ, σ2 ))
d σ2
Therefore :
σ2ML =
=0
N
1 X
(xn − µ ML )2
N n=1
Problem 1.12 Solution
It is quite straightforward for E[µ ML ], with the prior knowledge that xn is
i.i.d. and it also obeys Gaussian distribution N (µ, σ2 ).
E[µ ML ] = E[
N
N
1 X
1 X
xn ] = E[xn ] = µ
x n ] = E[
N n=1
N n=1
For E[σ2ML ], we need to take advantage of (1.56) and what has been given
in the problem :
E[σ2ML ] = E[
N
1 X
(xn − µ ML )2 ]
N n=1
=
N
1 X
E[
(xn − µ ML )2 ]
N n=1
=
N
1 X
E[
(x2 − 2xn µ ML + µ2ML )]
N n=1 n
=
N
N
N
1 X
1 X
1 X
E[
x2n ] − E[
2xn µ ML ] + E[
µ2 ]
N n=1
N n=1
N n=1 ML
= µ2 + σ 2 −
N
N
2 X
1 X
xn (
xn )] + E[µ2ML ]
E[
N n=1
N n=1
= µ2 + σ 2 −
N
N
N
X
X
2
1 X
E
[
x
(
x
)]
+
E
[(
xn )2 ]
n
n
N
N 2 n=1
n=1
n=1
= µ2 + σ 2 −
N
N
X
X
2
1
2
E
[(
x
)
]
+
E
[(
xn )2 ]
n
N 2 n=1
N 2 n=1
N
X
1
E
[(
xn )2 ]
N 2 n=1
1
= µ2 + σ 2 − 2 [ N ( N µ2 + σ 2 ) ]
N
= µ2 + σ 2 −
8
Therefore we have:
E[σ2ML ] = (
N −1 2
)σ
N
Problem 1.13 Solution
This problem can be solved in the same method used in Prob.1.12 :
E[σ2ML ] = E[
N
1 X
(xn − µ)2 ]
N n=1
(Because here we use µ to replace µ ML )
=
N
1 X
(xn − µ)2 ]
E[
N n=1
=
N
1 X
(x2 − 2xn µ + µ2 )]
E[
N n=1 n
=
N
N
N
1 X
1 X
1 X
x2n ] − E[
2xn µ] + E[
µ2 ]
E[
N n=1
N n=1
N n=1
= µ2 + σ 2 −
N
2µ X
E[
x n ] + µ2
N n=1
= µ2 + σ2 − 2µ2 + µ2
= σ2
Note: The biggest difference between Prob.1.12 and Prob.1.13 is that the
mean of Gaussian Distribution is known previously (in Prob.1.13) or not (in
Prob.1.12). In other words, the difference can be shown by the following equations:
E[ µ 2 ] = µ 2
(µ is determined, i.e. its expectation is itself, also true for µ2 )
N
N
X
1 X
1
1
σ2
E[µ2ML ] = E[(
xn )2 ] = 2 E[(
xn )2 ] = 2 N(N µ2 + σ2 ) = µ2 +
N n=1
N
N
N
n=1
Problem 1.14 Solution
This problem is quite similar to the fact that any function f (x) can be
written into the sum of an odd function and an even function. If we let:
wSij =
w i j + w ji
2
and
w iAj =
w i j − w ji
2
It is obvious that they satisfy the constraints described in the problem,
which are :
w i j = wSij + w iAj , wSij = wSji , w iAj = −w Aji
9
To prove (1.132), we only need to simplify it :
D
D X
X
wi j xi x j
=
D
D X
X
(wSij + w iAj )x i x j
i =1 j =1
i =1 j =1
=
D X
D
X
wSij x i x j +
D X
D
X
w iAj x i x j
i =1 j =1
i =1 j =1
Therefore, we only need to prove that the second term equals to 0, and
here we use a simple trick: we will prove twice of the second term equals to 0
instead.
2
D X
D
X
w iAj x i x j
=
i =1 j =1
D X
D
X
( w iAj + w iAj )x i x j
i =1 j =1
=
D
D X
X
( w iAj − w Aji )x i x j
i =1 j =1
=
D X
D
X
w iAj x i x j −
i =1 j =1
=
D X
D
X
D X
D
X
w Aji x i x j
i =1 j =1
w iAj x i x j −
i =1 j =1
D X
D
X
w Aji x j x i
j =1 i =1
= 0
Therefore, we choose the coefficient matrix to be symmetric as described
in the problem. Considering about the symmetry, we can see that if and only
if for i = 1, 2, ..., D and i ≤ j, w i j is given, the whole matrix will be determined.
Hence, the number of independent parameters are given by :
D + D − 1 + ... + 1 =
D(D + 1)
2
Note: You can view this intuitively by considering if the upper triangular
part of a symmetric matrix is given, the whole matrix will be determined.
Problem 1.15 Solution
This problem is a more general form of Prob.1.14, so the method can also
be used here: we will find a way to use w i 1 i 2 ...i M to represent w
e i 1 i 2 ...i M .
We begin by introducing a mapping function:
F(x i1 x i2 ...x iM ) = x j1 x j2 ..., x jM
s.t.
M
[
k=1
x ik =
M
[
k=1
x jk ,
and
x j1 ≥ x j2 ≥ x j3 ... ≥ x jM
10
It is complexed to write F in mathematical form. Actually this function
does a simple work: it rearranges the element in a decreasing order based on
its subindex. Several examples are given below, when D = 5, M = 4:
F(x5 x2 x3 x2 ) = x5 x3 x2 x2
F(x1 x3 x3 x2 ) = x3 x3 x2 x1
F(x1 x4 x2 x3 ) = x4 x3 x2 x1
F(x1 x1 x5 x2 ) = x5 x2 x1 x1
After introducing F, the solution will be very simple, based on the fact
that F will not change the value of the term, but only rearrange it.
D X
D
X
...
i 1 =1 i 2 =1
where
D
X
w i 1 i 2 ...i M x i1 x i2 ...x iM =
i M =1
j1
D X
X
j 1 =1 j 2 =1
w
e j 1 j 2 ... j M =
X
...
jX
M −1
j M =1
w
e j 1 j 2 ... j M x j1 x j2 ...x jM
w
w∈Ω
¯
Ω = {w i 1 i 2 ...i M ¯ F(x i1 x i2 ...x iM ) = x j1 x j2 ...x jM , ∀ x i1 x i2 ...x iM }
By far, we have already proven (1.134). Mathematical induction will be
used to prove (1.135) and we will begin by proving D = 1, i.e. n(1, M) =
n(1, M − 1). When D = 1, (1.134) will degenerate into wx
e 1M , i.e., it only has
one term, whose coefficient is govern by w
e regardless the value of M.
Therefore, we have proven when D = 1, n(D, M) = 1. Suppose (1.135)
holds for D, let’s prove it will also hold for D + 1, and then (1.135) will be
proved based on Mathematical induction.
Let’s begin based on (1.134):
i1
DX
+1 X
i 1 =1 i 2 =1
...
iX
M −1
i M =1
w
e i 1 i 2 ...i M x i1 x i2 ...x iM
(∗)
We divide (∗) into two parts based on the first summation: the first part
is made up of i i = 1, 2, ..., D and the second part i 1 = D + 1. After division, the
first part corresponds to n(D, M), and the second part corresponds to n(D +
1, M − 1). Therefore we obtain:
n(D + 1, M) = n(D, M) + n(D + 1, M − 1)
And given the fact that (1.135) holds for D :
n(D, M) =
D
X
i =1
n(i, M − 1)
(∗∗)
11
Therefore,we substitute it into (∗∗)
n(D + 1, M) =
D
X
n(i, M − 1) + n(D + 1, M − 1) =
DX
+1
n(i, M − 1)
i =1
i =1
We will prove (1.136) in a different but simple way. We rewrite (1.136) in
Permutation and Combination view:
D
X
i =1
M
C iM+−M1−2 = C D
+ M −1
Firstly, We expand the summation.
M −1
M −1
M −1
M
CM
+ ... C D
−1 + C M
+ M −2 = C D + M −1
M
M −1
Secondly, we rewrite the first term on the left side to C M
, because C M
=
−1
M
CM
= 1. In other words, we only need to prove:
M
M −1
M −1
M
CM
+ CM
+ ... C D
+ M −2 = C D + M −1
r
r
r −1
Thirdly, we take advantage of the property : C N
= CN
+ CN
. So we
−1
−1
can recursively combine the first term and the second term on the left side,
and it will ultimately equal to the right side.
(1.137) gives the mathematical form of n(D, M), and we need all the conclusions above to prove it.
Let’s give some intuitive concepts by illustrating M = 0, 1, 2. When M = 0,
(1.134) will consist of only a constant term, which means n(D, 0) = 1. When
M = 1,it is obvious n(D, 1) = D, because in this case (1.134) will only have D
terms if we expand it. When M = 2, it degenerates to Prob.1.14, so n(D, 2) =
D (D +1)
is also obvious. Suppose (1.137) holds for M − 1, let’s prove it will also
2
hold for M.
n(D, M) =
D
X
n(i, M − 1)
( based on (1.135) )
i =1
=
=
=
=
=
D
X
C iM+−M1−2
...
=
( based on (1.137) holds for M − 1 )
i =1
M −1
M −1
M −1
M −1
CM
+ CM
−1 + C M
+1 ... + C D + M −2
M
M −1
M −1
M −1
( CM
+ CM
) + CM
+1 ... + C D + M −2
M
M −1
M −1
( CM
+1 + C M +1 )... + C D + M −2
M
M −1
CM
+2 ... + C D + M −2
M
CD
+ M −1
By far, all have been proven.
12
Problem 1.16 Solution
This problem can be solved in the same way as the one in Prob.1.15.
Firstly, we should write the expression consisted of all the independent terms
up to Mth order corresponding to N(D, M). By adding a summation regarding to M on the left side of (1.134), we obtain:
i1
M X
D X
X
...
m=0 i 1 =1 i 2 =1
iX
m−1
i m =1
w
e i 1 i 2 ...i m x i1 x i2 ...x im
(∗)
(1.138) is quite obvious if we view m as an looping variable, iterating
through all the possible orders less equal than M, and for every possible oder
m, the independent parameters are given by n(D, m).
Let’s prove (1.138) in a formal way by using Mathematical Induction.
When M = 1,(∗) will degenerate to two terms: m = 0, corresponding to n(D, 0)
and m = 1, corresponding to n(D, 1). Therefore N(D, 1) = n(D, 0) + n(D, 1).
Suppose (1.138) holds for M, we will see that it will also hold for M + 1. Let’s
begin by writing all the independent terms based on (∗) :
D
M
+1 X
X
i1
X
...
m=0 i 1 =1 i 2 =1
iX
m−1
i m =1
w
e i 1 i 2 ...i m x i1 x i2 ...x im
(∗∗)
Using the same technique as in Prob.1.15, we divide (∗∗) to two parts
based on the summation regarding to m: the first part consisted of m =
0, 1, ..., M and the second part m = M + 1. Hence, the first part will correspond to N(D, M) and the second part will correspond to n(D, M + 1). So we
obtain:
N(D, M + 1) = N(D, M) + n(D, M + 1)
Then we substitute (1.138) into the equation above :
N(D, M + 1) =
M
X
n(D, m) + n(D, M + 1)
m=0
=
M
+1
X
n(D, m)
m=0
To prove (1.139), we will also use the same technique in Prob.1.15 instead
of Mathematical Induction. We begin based on already proved (1.138):
N(D, M) =
M
X
m=0
n(D, M)
13
We then take advantage of (1.137):
N(D, M) =
=
=
=
M
X
m
CD
+ m−1
m=0
M
C 0D −1 + C 1D + C 2D +1 + ... + C D
+ M −1
M
( C 0D + C 1D ) + C 2D +1 + ... + C D
+ M −1
M
( C 1D +1 + C 2D +1 ) + ... + C D
+ M −1
= ...
=
M
CD
+M
Here as asked by the problem, we will view the growing speed of N(D, M).
We should see that in n(D, M), D and M are symmetric, meaning that we only
need to prove when D À M, it will grow like D M , and then the situation of
M À D will be solved by symmetry.
N(D, M) =
=
=
≈
=
=
(D + M)D + M
(D + M)!
≈
D! M!
DD MM
1 D+M D
(
) (D + M) M
D
MM
M D
1
[(1 + ) M ] M (D + M) M
M
D
M
e M
( ) (D + M) M
M
M
eM
(1 + ) M D M
M
D
M
M D M2
eM
[(1 + ) M ] D D M
M
D
M
M2
≈
e M+ D
MM
DM
≈
eM
MM
DM
M2
Where we use Stirling’s approximation, lim (1 + n1 )n = e and e D ≈ e0 =
n→+∞
1. According to the description in the problem, When D À M, we can actually
eM
M
view M
in this case. And by
M as a constant, so N(D, M) will grow like D
D
symmetry, N(D, M) will grow like M , when M À D.
Finally, we are asked to calculate N(10, 3) and N(100, 3):
3
N(10, 3) = C 13
= 286
3
N(100, 3) = C 103
= 176851
Problem 1.17 Solution
14
Γ(x + 1) =
Z +∞
u x e−u du
0
Z +∞
− u x de−u
0
¯+∞ Z +∞
¯
= − u x e−u ¯
−
e−u d(− u x )
0
0
Z +∞
¯+∞
x −u ¯
= −u e ¯
e−u u x−1 du
+x
0
0
¯+∞
x −u ¯
= −u e ¯
+ x Γ(x)
=
0
Where we have taken advantage of Integration by parts and according to
the equation above, we only need to prove the first term equals to 0. Given
L’Hospital’s Rule:
x!
ux
lim − u = lim − u = 0
u→+∞ e
u→+∞ e
And also when u = 0,− u x e u = 0, so we have proved Γ(x + 1) = xΓ(x). Based
on the definition of Γ(x), we can write:
Z +∞
¯+∞
¯
Γ(1) =
e−u du = − e−u ¯
= −(0 − 1) = 1
0
0
Therefore when x is an integer:
Γ(x) = (x − 1) Γ(x − 1) = (x − 1) (x − 2) Γ(x − 2) = ... = x! Γ(1) = x!
Problem 1.18 Solution
Based on (1.124) and (1.126) and by substituting x to
obvious to obtain :
Z +∞
p
2
e− xi dx i = π
p
2σ y, it is quite
−∞
D
Therefore, the left side of (1.42) will equal to π 2 . For the right side of
(1.42):
Z +∞
Z +∞
p
D −1
2
SD
e−r r D −1 dr = S D
e−u u 2 d u (u = r 2 )
0
0
Z
S D +∞ −u D −1
=
e u 2 du
2 0
SD D
=
Γ( )
2
2
Hence, we obtain:
D
π2 =
SD D
Γ( )
2
2
D
=>
SD =
2π 2
Γ( D2 )
15
S D has given the expression of the surface area with radius 1 in dimension D, we can further expand the conclusion: the surface area with radius r
in dimension D will equal to S D · r D −1 , and when r = 1, it will reduce to S D .
This conclusion is naive, if you find that the surface area of different sphere
in dimension D is proportion to the D − 1th power of radius, i.e. r D −1 . Considering the relationship between V and S of a sphere with arbitrary radius
in dimension D: dV
dr = S, we can obtain :
Z
V =
Z
S dr =
S D r D −1 dr =
SD D
r
D
The equation above gives the expression of the volume of a sphere with
radius r in dimension D, so we let r = 1 :
VD =
SD
D
For D = 2 and D = 3 :
V2 =
S2
1 2π
= ·
=π
2
2 Γ(1)
3
3
S3
1 2π 2
1 2π 2
4
V3 =
= · 3 = · p = π
π
3
3 Γ( 2 )
3
3
2
Problem 1.19 Solution
We have already given a hint in the solution of Prob.1.18, and here we
will make it more clearly: the volume of a sphere with radius r is VD · r D .
This is quite similar with the conclusion we obtained in Prob.1.18 about the
surface area except that it is proportion to Dth power of its radius, i.e. r D not
r D −1 .
D
volume of sphere
VD aD
SD
π2
=
= D =
(∗)
volume of cube
(2a)D
2 D
2D −1 D Γ( D )
2
Where we have used the result of (1.143). And when D → +∞, we will use
a simple method to show that (∗) will converge to 0. We rewrite it :
(∗) =
2 π D
1
·( ) 2 · D
D 4
Γ( )
2
Hence, it is now quite obvious, all the three terms will converge to 0 when
D → +∞. Therefore their product will also converge to 0. The last problem is
quite simple :
p
p
center to one corner
a2 · D p
=
= D and
lim
D = +∞
D →+∞
center to one side
a
Problem 1.20 Solution
16
The density of probability in a thin shell with radius r and thickness ²
can be viewed as a constant. And considering that a sphere in dimension D
with radius r has surface area S D r D −1 , which has already been proved in
Prob.1.19 :
2
Z
Z
shell
p(x) dx = p(x)
shell
dx =
Thus we denote :
p(r) =
exp(− 2rσ2 )
D
(2πσ2 ) 2
2
· V (shell) =
S D r D −1
exp(−
D
exp(− 2rσ2 )
D
(2πσ2 ) 2
S D r D −1 ²
r2
)
2σ 2
(2πσ2 ) 2
We calculate the derivative of (1.148) with respect to r :
d p(r)
SD
r2
r2
D −2
=
r
exp(
−
)
(D
−
1
−
)
D
dr
2σ 2
σ2
(2πσ2 ) 2
(∗)
We let p
the derivative equal to 0, we will obtain its unique root( stationary
point) r̂ = D − 1 σ, because r ∈ [0, +∞]. When r < r̂, the derivative is large
than 0, p(r) will increase as r ↑, and when r > r̂, the derivative is less than 0,
p(r) will decrease as r ↑. Therefore
r̂ will be the only maximum point. And it
p
is obvious when D À 1, r̂ ≈ D σ.
p( r̂ + ² )
p( r̂)
2
=
( r̂ + ²)D −1 exp(− (r̂2+σ²2) )
2
r̂ D −1 exp(− 2r̂σ2 )
²
2² r̂ + ²2
= (1 + )D −1 exp(−
)
r̂
2σ 2
2² r̂ + ²2
²
= exp( −
+ (D − 1)ln(1 + ) )
2
r̂
2σ
We process for the exponential term by using Taylor Theorems.
−
²
²
2² r̂ + ²2
2² r̂ + ²2
²2
+
(D
−
1)
ln(1
+
)
≈
−
+
(D
−
1)
(
−
)
r̂
r̂ 2 r̂ 2
2σ2
2σ 2
2² r̂ + ²2 2 r̂ ² − ²2
= −
+
2σ 2
2σ2
2
²
= − 2
σ
2
Therefore, p( r̂ + ²) = p( r̂) exp(− σ² 2 ). Note: Here I draw a different conclusion compared with (1.149), but I do not think there is any mistake in
my deduction.
Finally, we see from (1.147) :
¯
1
¯
p(x)¯
=
D
x=0
(2πσ2 ) 2
17
¯
¯
p(x)¯
||x||2 = r̂
=
2
1
exp(−
D
(2πσ2 ) 2
r̂ 2
1
D
)≈
exp(− )
D
2
2
2σ
(2πσ2 ) 2
Problem 1.21 Solution
The first question is rather simple :
1
1
1
1
(ab) 2 − a = a 2 (b 2 − a 2 ) ≥ 0
Where we have taken advantage of b ≥ a ≥ 0. And based on (1.78):
p(mistake) =
p(x ∈ R 1 , C 2 ) + p(x ∈ R 2 , C 1 )
Z
Z
=
p(x, C 2 ) dx +
p(x, C 1 ) dx
R1
R2
Recall that the decision rule which can minimize misclassification is that
if p(x, C 1 ) > p(x, C 2 ), for a given value of x, we will assign that x to class
C 1 . We can see that in decision area R 1 , it should satisfy p(x, C 1 ) > p(x, C 2 ).
Therefore, using what we have proved, we can obtain :
Z
Z
1
{ p(x, C 1 ) p(x, C 2 ) } 2 dx
p(x, C 2 ) dx ≤
R1
R1
It is the same for decision area R 2 . Therefore we can obtain:
Z
1
p(mistake) ≤ { p(x, C 1 ) p(x, C 2 ) } 2 dx
Problem 1.22 Solution
We need to deeply understand (1.81). When L k j = 1 − I k j :
X
X
¯
¯
¯
p(C k ¯x) − p(C j ¯x)
L k j p(C k ¯x) =
k
k
Given a specific x, the first term on the right side is a constant, which
equals to 1, no matter which class C j we assign
x to. Therefore if we want to
¯
¯
minimize the loss, we will maximize p(C j x). Hence, we will
¯ assign x to class
C j , which can give the biggest posterior probability p(C j ¯x).
The explanation of the loss matrix is quite simple. If we label correctly,
there is no loss. Otherwise, we will incur a loss, in the same degree whichever
class we label it to. The loss matrix is given below to give you an intuitive
view:


0 1 1 ... 1


1 0 1 . . . 1
. . .
.
 . . . ..

. .. 
. . .
1 1 1 ...
Problem 1.23 Solution
0
18
E[L] =
XXZ
k
j
Rj
L k j p(x, C k ) dx =
XXZ
k
j
Rj
¯
L k j p(C k )p(x¯C k ) dx
If we denote a new loss matrix by L?jk = L jk p(C k ), we can obtain a new
equation :
XXZ
¯
¯
E[L] =
L?
k j p(x C k ) dx
j
k
Rj
Problem 1.24 Solution
This description of the problem is a little confusing, and what it really
mean is that λ is the parameter governing the loss, just like θ governing the
posterior probability p(C k |x) when we introduce the reject option. Therefore
the reject option can be written in a new way when we view it from the view
of λ and the loss:

class C j min Pk L kl p(C k | x) < λ
l
choice
reject
else
Where C j is the class that can obtain the minimum. If L k j = 1 − I k j ,
according to what we have proved in Prob.1.22 :
X
X
¯
¯
¯
¯
p(C k ¯x) − p(C j ¯x) = 1 − p(C j ¯x)
L k j p(C k ¯x) =
k
k
Therefore, the reject criterion from the view of λ above is actually equivalent to the largest posterior probability is larger than 1 − λ :
min
l
X
L kl p(C k | x) < λ
<=>
max p(C l | x) > 1 − λ
k
l
And from the view of θ and posterior probability, we label a class for x (i.e.
we do not reject) is given by the constrain :
max p(C l | x) > θ
l
Hence from the two different views, we can see that λ and θ are correlated
with:
λ+θ =1
Problem 1.25 Solution
We can prove this informally by dealing with one dimension once a time
just as the same process in (1.87) - (1.89) until all has been done, due to the
fact that the total loss E can be divided to the summation of loss on every
19
dimension, and what’s more they are independent. Here, we will use a more
informal way to prove this. In this case, the expected loss can be written :
Z Z
E[L] =
{y(x) − t}2 p(x, t) dt dx
Therefore, just as the same process in (1.87) - (1.89):
Z
∂E[L]
= 2 {y(x) − t} p(x, t) dt = 0
∂ y(x)
R
tp(x, t) dt
=> y(x) =
= Et [t|x]
p(x)
Problem 1.26 Solution
The process is identical as the deduction we conduct for (1.90). We will
not repeat here. And what we should emphasize is that E[t|x] is a function of
x, not t. Thus the integral over t and x can be simplified based on Integration
by parts and that is how we obtain (1.90).
Note: There is a mistake in (1.90), i.e. the second term on the right side
is wrong. You can view (3.37) on P148 for reference. It should be :
Z
Z
¯
¯
E[L] = { y(x) − E[t¯x]}2 p(x) dx + {E[t¯x − t]}2 p(x, t) dx dt
Moreover, this mistake has already been revised in the errata.
Problem 1.27 Solution
We deal with this problem based on Calculus of Variations.
Z
∂E[L q ]
= q [y(x − t)] q−1 si gn(y(x) − t) p(x, t) dt = 0
∂ y(x)
Z +∞
Z y(x)
=>
[y(x) − t] q−1 p(x, t) dt =
[y(x) − t] q−1 p(x, t) dt
=>
−∞
y(x)
Z y(x)
Z +∞
−∞
[y(x) − t] q−1 p(t|x) dt =
y(x)
[y(x) − t] q−1 p(t|x) dt
Where we take advantage of p(x, t) = p(t|x) p(x) and the property of sign
function. Hence, when q = 1, the equation above will reduce to :
Z y(x)
Z +∞
p(t|x) dt =
p(t|x) dt
−∞
y(x)
In other words, when q = 1, the optimal y(x) will be given by conditional
median. When q = 0, it is non-trivial. We need to rewrite (1.91) :
Z nZ
o
E[L q ] =
| y(x) − t| q p(t|x)p(x) dt dx
Z n
Z
o
=
p(x) | y(x) − t| q p(t|x) dt dx (∗)
20
If we want to minimize E[L q ], we only need to minimize the integrand of
(∗):
Z
| y(x) − t| q p(t|x) dt
(∗∗)
When q = 0, | y(x) − t| q is close to 1 everywhere except in the neighborhood
around t = y(x) (This can be seen from Fig1.29). Therefore:
Z
Z
Z
Z
p(t|x) dt − p(t|x) dt
p(t|x) dt − (1 − | y(x) − t| q ) p(t|x) dt ≈
(∗∗) ≈
U
U
²
²
Where ² means the small neighborhood,U means the whole space x lies
in. Note that y(x) has no correlation with the first term, but the second term
(because how to choose y(x) will affect the location of ²). Hence we will put ²
at the location where p(t|x) achieve its largest value, i.e. the mode, because
in this way we can obtain the largest reduction. Therefore, it is natural we
choose y(x) equals to t that maximize p(t|x) for every x.
Problem 1.28 Solution
Basically this problem is focused on the definition of Information Content,
i.e.h(x). We will rewrite the problem more precisely. In Information Theory,
h(·) is also called Information Content and denoted as I(·). Here we will still
use h(·) for consistency. The whole problem is about the property of h(x).
Based on our knowledge that h(·) is a monotonic function of the probability
p(x), we can obtain:
h(x) = f (p(x))
The equation above means that the Information we obtain for a specific
value of a random variable x is correlated with its occurring probability p(x),
and its relationship is given by a mapping function f (·). Suppose C is the
intersection of two independent event A and B, then the information of event
C occurring is the compound message of both independent events A and B
occurring:
h(C) = h(A ∩ B) = h(A) + h(B)
(∗)
Because A and B is independent:
P(C) = P(A) · P(B)
We apply function f (·) to both side:
f (P(C)) = f (P(A) · P(B))
(∗∗)
Moreover, the left side of (∗) and (∗∗) are equivalent by definition, so we
can obtain:
h(A) + h(B) = f (P(A) · P(B))
=>
f (p(A)) + f (p(B)) = f (P(A) · P(B))
21
We obtain an important property of function f (·): f (x · y) = f (x) + f (y).
Note: In problem (1.28), what it really wants us to prove is about the form
and property of function f in our formulation, because there is one sentence
in the description of the problem : "In this exercise, we derive the relation
between h and p in the form of a function h(p)", (i.e. f (·) in our formulation
is equivalent to h(p) in the description).
At present, what we know is the property of function f (·):
f (x y) = f (x) + f (y)
(∗)
Firstly, we choose x = y, and then it is obvious : f (x2 ) = 2 f (x). Secondly, it
is obvious f (x n ) = n f (x) , n ∈ N is true for n = 1, n = 2. Suppose it is also true
for n, we will prove it is true for n + 1:
f (x n+1 ) = f (x n ) + f (x) = n f (x) + f (x) = (n + 1) f (x)
Therefore, f (x n ) = n f (x) , n ∈ N has been proved. For an integer m, we
n m
rewrite x n as (x m ) , and take advantage of what we have proved, we will
obtain:
n m
n
f (x n ) = f ((x m ) ) = m f (x m )
n
Because f (x n ) also equals to n f (x), therefore n f (x) = m f (x m ). We simplify the equation and obtain:
n
f (x m ) =
n
f (x)
m
For an arbitrary positive x , x ∈ R+ , we can find two positive rational array
{ yn } and { z n }, which satisfy:
y1 < y2 < ... < yN < x
and
z1 > z2 > ... > z N > x,
and
lim yN = x
N →+∞
lim z N = x
N →+∞
We take advantage of function f (·) is monotonic:
yN f (p) = f (p yN ) ≤ f (p x ) ≤ f (p z N ) = z N f (p)
And when N → +∞, we will obtain: f (p x ) = x f (p) , x ∈ R+ . We let p = e,
it can be rewritten as : f (e x ) = x f (e). Finally, We denote y = e x :
f (y) = ln(y) f (e)
Where f (e) is a constant once function f (·) is decided. Therefore f (x) ∝
ln(x).
Problem 1.29 Solution
22
This problem is a little bit tricky. The entropy for a M-state discrete random variable x can be written as :
H[x] = −
M
X
λ i ln(λ i )
i
Where λ i is the probability that x choose state i. Here we choose a concave
function f (·) = ln(·), we rewrite Jensen’s inequality, i.e.(1.115):
ln(
M
X
λi xi ) ≥
i =1
M
X
λ i ln(x i )
i =1
We choose x i = λ1i and simplify the equation above, we will obtain :
lnM ≥ −
M
X
λ i ln(λ i ) = H[x]
i =1
Problem 1.30 Solution
Based on definition :
ln{
p(x)
} =
q(x)
=
1
s
1
ln( ) − [ 2 (x − µ)2 − 2 (x − m)2 ]
σ
2σ
2s
s
1
1
µ
m
µ2
m2
ln( ) − [( 2 − 2 )x2 − ( 2 − 2 )x + ( 2 − 2 )]
σ
2σ
2s
σ
s
2σ
2s
We will take advantage of the following equations to solve this problem.
Z
E[x2 ] = x2 N (x|µ, σ2 ) dx = µ2 + σ2
E[x] =
Z
Z
x N (x|µ, σ2 ) dx = µ
N (x|µ, σ2 ) dx = 1
Given the equations above, it is easy to see :
Z
q(x)
K L(p|| q) = − p(x)ln{
} dx
p(x)
Z
p(x)
=
N (x|µ, σ) ln{
} dx
q(x)
=
=
s
1
1
µ
m
µ2
m2
ln( ) − ( 2 − 2 )(µ2 + σ2 ) + ( 2 − 2 )µ − ( 2 − 2 )
σ
2σ
2s
σ
s
2σ
2s
s
σ2 + (µ − m)2 1
ln( ) +
−
σ
2
2s2
23
We will discuss this result in more detail. Firstly, if K L distance is defined
in Information Theory, the first term of the result will be log 2 ( σs ) instead of
ln( σs ). Secondly, if we denote x = σs , K L distance can be rewritten as :
K L(p|| q) = ln(x) +
1
1
− + a,
2
2
2x
where a =
(µ − m)2
2s2
We calculate the derivative of K L with respect to x, and let it equal to 0:
d(K L)
1
= − x−3 = 0
dx
x
=>
x = 1 ( ∵ s, σ > 0 )
When x < 1 the derivative is less than 0, and when x > 1, it is greater than
0, which makes x = 1 the global minimum. When x = 1, K L(p|| q) = a. What’s
more, when µ = m, a will achieve its minimum 0. In this way, we have shown
that the K L distance between two Gaussian Distributions is not less than 0,
and only when the two Gaussian Distributions are identical, i.e. having same
mean and variance, K L distance will equal to 0.
Problem 1.31 Solution
We evaluate H[x] + H[y] − H[x, y] by definition. Firstly, let’s calculate
H[x, y] :
Z Z
H[x, y] = −
p(x, y) lnp(x, y) dx dy
Z Z
Z Z
= −
p(x, y) lnp(x) dx dy −
p(x, y) lnp(y|x) dx dy
Z
Z Z
= − p(x) lnp(x) dx −
p(x, y) lnp(y|x) dx dy
=
H[x] + H[y|x]
R
Where we take advantage of p(x, y) = p(x)p(y|x), p(x, y) dy = p(x) and
(1.111). Therefore, we have actually solved Prob.1.37 here. We will continue
our proof for this problem, based on what we have proved:
H[x] + H[y] − H[x, y] =
=
=
=
=
H[y] − H[y|x]
Z
Z Z
− p(y)lnp(y) dy +
p(x, y) lnp(y|x) dx dy
Z Z
Z Z
−
p(x, y) lnp(y) dx dy +
p(x, y) lnp(y|x) dx dy
Z Z
¡ p(x)p(y) ¢
−
p(x, y)ln
dxdy
p(x, y)
K L( p(x, y)|| p(x)p(y) ) = I(x, y) ≥ 0
Where we take advantage of the following properties:
Z
p(y) = p(x, y) dx
24
p(y)
p(x)p(y)
=
p(y|x)
p(x, y)
Moreover, it is straightforward that if and only if x and y is statistically
independent, the equality holds, due to the property of KL distance. You can
also view this result by :
Z Z
H[x, y] = −
p(x, y) lnp(x, y) dx dy
Z Z
Z Z
= −
p(x, y) lnp(x) dx dy −
p(x, y) lnp(y) dx dy
Z
Z Z
= − p(x) lnp(x) dx −
p(y) lnp(y) dy
=
H[x] + H[y]
Problem 1.32 Solution
It is straightforward based on definition and note that if we want to
change variable in integral, we have to introduce a redundant term called
Jacobian Determinant.
Z
H[y] = − p(y) lnp(y) dy
Z
p(x) p(x) ∂y
ln
| | dx
= −
|A|
|A| ∂x
Z
p(x)
= − p(x) ln
dx
|A|
Z
Z
1
= − p(x) lnp(x) dx − p(x) ln
dx
|A|
= H[x] + ln|A|
Where we have taken advantage of the following equations:
∂y
∂x
=A
p(x) = p(y) |
and
∂y
∂x
| = p(y) |A|
Z
p(x) dx = 1
Problem 1.33 Solution
Based on the definition of Entropy, we write:
XX
H[y| x] = −
p(x i , y j ) lnp(y j | x i )
xi y j
Considering the property of probability, we can obtain that 0 ≤ p(y j | x i ) ≤
1, 0 ≤ p(x i , y j ) ≤ 1. Therefore, we can see that − p(x i , y j ) lnp(y j | x i ) ≥ 0 when
0 < p(y j | x i ) ≤ 1. And when p(y j | x i ) = 0, provided with the fact that lim plnp =
p →0
25
0, we can see that − p(x i , y j ) lnp(y j | x i ) = − p(x i )p(y j | x i ) lnp(y j | x i ) ≈ 0, (here
we view p(x) as a constant). Hence for an arbitrary term in the equation
above, we have proved that it can not be less than 0. In other words, if and
only if every term of H[y| x] equals to 0, H[y| x] will equal to 0.
Therefore, for each possible value of random variable x, denoted as x i :
X
− p(x i , y j ) lnp(y j | x i ) = 0
(∗)
yj
If there are more than one possible value of random variable y given
x = x i , denoted as y j , such that p(y j | x i ) 6= 0 (Because x i , y j are both "possible", p(x i , y j ) will also not equal to 0), constrained by 0 ≤ p(y j | x i ) ≤ 1 and
P
j p(y j | x i ) = 1, there should be at least two value of y satisfied 0 < p(y j | x i ) <
1, which ultimately leads to (∗) > 0.
Therefore, for each possible value of x, there will only be one y such that
p(y| x) 6= 0. In other words, y is determined by x. Note: This result is quite
straightforward. If y is a function of x, we can obtain the value of y as soon
as observing a x. Therefore we will obtain no additional information when
observing a y j given an already observed x.
Problem 1.34 Solution
This problem is complicated. We will explain it in detail. According to
Appenddix D, we can obtain the relation,i.e. (D.3) :
Z
∂F
F[y(x) + ²η(x)] = F[y(x)] +
²η(x) dx
(∗∗)
∂y
Where y(x) can be viewed as an operator that for any input x it will give
an output value y, and equivalently, F[y(x)] can be viewed as an functional
operator that for any input value y(x), it will give an ouput value F[y(x)].
Then we consider a functional operator:
Z
I[p(x)] = p(x) f (x) dx
Under a small variation p(x) → p(x) + ²η(x), we will obtain :
Z
Z
I[p(x) + ²η(x)] = p(x) f (x) dx + ²η(x) f (x) dx
Comparing the equation above and (∗), we can draw a conclusion :
∂I
∂ p(x)
= f (x)
Similarly, let’s consider another functional operator:
Z
J[p(x)] = p(x)lnp(x) dx
26
Then under a small variation p(x) → p(x) + ²η(x):
Z
J[p(x) + ²η(x)] =
( p(x) + ²η(x) ) ln( p(x) + ²η(x) ) dx
Z
Z
=
p(x) ln( p(x) + ²η(x) ) dx + ²η(x) ln( p(x) + ²η(x) ) dx
Note that ²η(x) is much smaller than p(x), we will write its Taylor Theorems at point p(x):
ln( p(x) + ²η(x) ) = lnp(x) +
²η(x)
p(x)
+ O(²η(x)2 )
Therefore, we substitute the equation above into J[p(x) + ²η(x)]:
Z
Z
J[p(x) + ²η(x)] = p(x) lnp(x) dx + ²η(x) (lnp(x) + 1) dx + O(²2 )
Therefore, we also obtain :
∂J
∂ p(x)
= lnp(x) + 1
Now we can go back to (1.108). Based on ∂ ∂pJ( x) and ∂ p∂(Ix) , we can calculate
the derivative of the expression just before (1.108) and let it equal to 0:
− lnp(x) − 1 + λ1 + λ2 x + λ3 (x − µ)2 = 0
Hence we rearrange it and obtain (1.108). From (1.108) we can see that
p(x) should take the form of a Gaussian distribution. So we rewrite it into
Gaussian form and then compare it to a Gaussian distribution with mean µ
and variance σ2 , it is straightforward:
exp(−1 + λ1 ) =
1
1
(2πσ2 ) 2
,
exp(λ2 x + λ3 (x − µ)2 ) = exp{
(x − µ)2
}
2σ 2
Finally, we obtain :
λ1 = 1 − ln(2πσ2 )
λ2 = 0
1
2σ 2
Note that there is a typo in the official solution manual about λ3 . Moreover, in the following parts, we will substitute p(x) back into the three constraints and analytically prove that p(x) is Gaussian. You can skip the following part. (The writer would especially thank Dr.Spyridon Chavlis from
IMBB,FORTH for this analysis)
λ3 = −
27
We already know:
p(x) = exp(−1 + λ1 + λ2 x + λ3 (x − µ)2 )
Where the exponent is equal to:
−1 + λ1 + λ2 x + λ3 (x − µ)2 = λ3 x2 + (λ2 − 2λ3 µ)x + (λ3 µ2 + λ1 − 1)
Completing the square, we can obtain that:
ax2 + bx + c = a(x − d)2 + f , d = −
b2
b
, f = c−
2a
4a
Using this quadratic form, the constraints can be written as
R∞
R∞
2
1. −∞ p(x)dx = −∞ e[a( x−d ) + f ] dx = 1
R∞
[ a ( x − d )2 + f ]
dx = µ
−∞ xp(x)dx = −∞ xe
2.
R∞
3.
R∞
R∞
2
2 [ a ( x − d )2 + f ]
dx = σ2
−∞ (x − µ) p(x)dx = −∞ (x − µ) e
The first constraint can be written as:
Z ∞
Z ∞
2
2
e a( x−d ) dx
e[a( x−d ) + f ] dx = e f
I1 =
−∞
−∞
Let u = x − d, which gives du = dx, and thus:
Z ∞
2
e au du
I1 = e f
−∞
p
p
Let −w2 = au2 ⇒ w = −au ⇒ dw = −adu, and thus:
ef
I1 = p
−a
Z ∞
2
e−w dw
−∞
2
As e− x is an even function the integral is written as:
2e f
I1 = p
−a
2
Let w = t ⇒ w =
2e f
I1 = p
−a
Z ∞
t
0
p
Z ∞
2
e−w dw
0
1
t ⇒ dw = p
dt, and thus:
2 t
2e f
e dt = p
−a
− 21 − t
Z ∞
0
1 1 −1 − t
ef
1
t 2 e dt = p Γ( ) = e f
2
−a 2
r
Here the Gamma function is used. Gamma function is defined as
Z ∞
Γ(z) =
t z−1 e− t dt
0
π
−a
28
where for non-negative integer values of n, we have:
1
(2n)! p
π
Γ( + n) = n
2
4 n!
Thus, the first constraint can be rewritten as:
r
π
f
e
=1
−a
(∗)
The second constraint can be written as:
Z ∞
Z ∞
2
2
I2 =
xe[a( x−d ) + f ] dx = e f
xe a( x−d ) dx
−∞
−∞
Let u = x − d ⇒ x = u + d ⇒ du = dx, and thus:
Z ∞
2
I2 = e f
(u + d)e au du
−∞
Using integral additivity, we have:
Z ∞
Z ∞
2
2
ue au du + e f
de au du
I2 = e f
−∞
−∞
We first deal with the first term on the right hand side. Here we denote it
as I 21 :
µZ
¶
Z
Z
I 21 = e f
∞
−∞
0
2
ue au du = e f
−∞
∞
2
ue au du +
2
ue au du
0
Swapping the integration limits, we obtain:
¶
µ Z −∞
Z ∞
f
au2
au2
I 21 = e −
ue du +
ue du
0
0
µZ −∞
¶
Z ∞
f
a (− u )2
au2
= e
(− u)e
du +
ue du
0
¶
µ 0Z ∞
Z ∞
f
a (− u )2
au2
ue du = 0
(− u)e
(− du) +
= e −
0
0
Then we deal with the second term I 22 :
Z ∞
2
I 22 = e f
de au du
−∞
Let −w2 = au2 ⇒ w =
p
p
−au ⇒ dw = −adu, and thus:
ef d
I 22 = p
−a
Z ∞
2
e−w dw
−∞
2
As e− x is an even function the integral is written as:
2e f d
I 22 = p
−a
Z ∞
0
2
e−w dw
29
Let w2 = t ⇒ w =
2e f d
I 22 = p
−a
Z ∞
t
p
t ⇒ dw =
1
p
dt, and thus:
2 t
2e f d
e dt = p
−a
− 21 − t
0
Z ∞
0
1 1 −1 − t
ef d 1
t 2 e dt = p Γ( ) = e f d
2
−a 2
Thus, the second constraint can be rewritten
r
π
ef d
=µ
−a
r
π
−a
(∗∗)
Combining (∗) and (∗∗), we can obtain that d = µ. Recall that:
d=−
λ2 − 2λ3 µ
b
= µ ⇒ λ2 − 2λ3 µ = −2λ3 µ ⇒ λ2 = 0
=−
2a
2λ3
So far, we have:
b = −2λ3 µ
And
2 2
4λ µ
b2
= λ3 µ2 + λ1 − 1 − 3 = λ1 − 1
4a
4λ3
Finally, we deal with the third also the last constraint. Substituting λ2 = 0
into the last constraint we have:
Z ∞
Z ∞
2
2
(x − µ)2 eλ3 ( x−µ) dx
(x − µ)2 e[λ3 ( x−µ) +λ1 −1] dx = eλ1 −1
I3 =
f = c−
−∞
−∞
Let u = x − µ ⇒ du = dx, and thus:
Z ∞
2
λ1 −1
u2 eλ3 u dx
I3 = e
−∞
2
2
p
p
Let −w = λ3 u ⇒ w = −λ3 u ⇒ dw = −λ3 du, and thus:
Z
Z ∞
2
dw
eλ1 −1 ∞ 2 −w2
1
w e
=
dw
I 3 = eλ1 −1
− w2 e − w p
3
−∞
−∞ λ3
2
−λ3
−λ3
Because it is an even function, we can further obtain:
Z
eλ1 −1 ∞ 2 −w2
I3 = 2
w e
dw
3
−λ32 0
p
1
Let w2 = t ⇒ w = t ⇒ dw = p
dt, and thus:
2 t
I3
= 2
=
e
λ1 −1 Z ∞
−λ3
0
eλ1 −1
Z ∞
3
2
−λ3
=
te
3
2
eλ1 −1
−t
1
eλ1 −1
p dt =
3
2 (t)
−λ 2
3
3
t 2 −1 e− t dt
0
3
eλ1 −1 π
Γ( ) =
3
2
2
−λ3
−λ32
3
2
Z ∞
0
1
t1− 2 e− t dt
30
Thus, the third constraint can be rewritten
p
eλ1 −1 π
= σ2
3
2
−λ32
(∗ ∗ ∗)
Rewriting (∗) with f = λ1 − 1, d = µ and a = λ3 , we obtain the following
equation
r
π
eλ1 −1
=1
(∗ ∗ ∗∗)
−λ3
Substituting the equation above back into (∗ ∗ ∗), we obtain
s
p
1
1
−λ3 1
π
= σ2 ⇔ −
= 2σ2 ⇔ λ3 = − 2
3
π
2
λ3
2σ
−λ32
Substituting λ3 back into (∗ ∗ ∗∗), we obtain:
s
r
1
π
π
1
eλ1 −1
= 1 ⇔ eλ1 −1
= 1 ⇔ eλ1 −1 = p
⇔ λ1 − 1 = ln( p
)
1
2
2
−λ3
2
πσ
2
πσ
2
2σ
Thus, we obtain:
1
ln(2πσ2 )
2
So far, we have obtainde λ i , where i = 1, 2, 3. We substitute them back
into p(x), yielding:
µ
¶
1
1
2
2
p(x) = exp −1 + 1 − ln(2πσ ) − 2 (x − µ)
2
2σ
µ
¶
µ
¶
1
1
2
2
= exp − ln(2πσ ) exp − 2 (x − µ)
2
2σ
¶
µ µ
¶¶
µ
1
1
2
= exp ln p
exp − 2 (x − µ)
2σ
2πσ2
λ1 = 1 −
Thus,
1
µ
¶
1
2
p(x) = p
exp − 2 (x − µ)
2σ
2πσ2
Just as required.
Problem 1.35 Solution
If p(x) = N (µ, σ2 ), we write its entropy:
Z
H[x] = − p(x)lnp(x) dx
Z
Z
1
(x − µ)2
= − p(x) ln{
}
dx
−
p(x)
{−
} dx
2πσ2
2σ 2
1
σ2
= − ln{
}
+
2πσ2
2σ2
1
{ 1 + ln(2πσ2 ) }
=
2
31
Where we have taken advantage of the following properties of a Gaussian
distribution:
Z
Z
p(x) dx = 1 and (x − µ)2 p(x) dx = σ2
Problem 1.36 Solution
Here we should make it clear that if the second derivative is strictly positive, the function must be strictly convex. However, the converse may not be
true. For example f (x) = x4 , g(x) = x2 , x ∈ R are both strictly convex by definition, but their second derivatives at x = 0 are both indeed 0 (See keyword
convex function on Wikipedia or Page 71 of the book Convex Optimization
written by Boyd, Vandenberghe for more details). Hence, here more precisely
we will prove that a convex function is equivalent to its second derivative is
non-negative by first considering Taylor Theorems:
f (x + ²) = f (x) +
f 0 (x)
f 00 (x) 2 f 000 (x) 3
²+
² +
² + ...
1!
2!
3!
f (x − ²) = f (x) −
f 0 (x)
f 00 (x) 2 f 000 (x) 3
²+
² −
² + ...
1!
2!
3!
Then we can obtain the expression of f 00 (x):
f (x + ²) + f (x − ²) − 2 f (x)
² →0
²2
f 00 (x) = lim
Where O(²4 ) is neglected and if f (x) is convex, we can obtain:
1
1
1
1
f (x) = f ( (x + ²) + (x − ²)) ≤ f (x + ²) + f (x − ²)
2
2
2
2
Hence f 00 (x) ≥ 0. The converse situation is a little bit complex, we will use
Lagrange form of Taylor Theorems to rewrite the Taylor Series Expansion
above :
f 00 (x? )
f (x) = f (x0 ) + f 0 (x0 )(x − x0 ) +
(x − x0 )
2
Where x? lies between x and x0 . By hypothesis, f 00 (x) ≥ 0, the last term is
non-negative for all x. We let x0 = λ x1 + (1 − λ)x2 , and x = x1 :
f (x1 ) ≥ f (x0 ) + (1 − λ)(x1 − x2 ) f 0 (x0 )
(∗)
And then, we let x = x2 :
f (x2 ) ≥ f (x0 ) + λ(x2 − x1 ) f 0 (x0 )
(∗∗)
We multiply (∗) by λ, (∗∗) by 1 − λ and then add them together, we will
see :
λ f (x1 ) + (1 − λ) f (x2 ) ≥ f (λ x1 + (1 − λ)x2 )
32
Problem 1.37 Solution
See Prob.1.31.
Problem 1.38 Solution
When M = 2, (1.115) will reduce to (1.114). We suppose (1.115) holds for
M, we will prove that it will also hold for M + 1.
f(
M
X
λm xm )
=
f (λ M +1 x M +1 + (1 − λ M +1 )
m=1
M
X
m=1 1 − λ M +1
≤ λ M +1 f (x M +1 ) + (1 − λ M +1 ) f (
≤ λ M +1 f (x M +1 ) + (1 − λ M +1 )
≤
M
+1
X
λm
M
X
xm )
λm
m=1 1 − λ M +1
xm )
M
X
λm
f (xm )
1
−
λ M +1
m=1
λm f (xm )
m=1
Hence, Jensen’s Inequality, i.e. (1.115), has been proved.
Problem 1.39 Solution
It is quite straightforward based on definition.
2 2 1 1
p(x i )lnp(x i ) = − ln − ln = 0.6365
3 3 3 3
i
H[x] = −
X
H[y] = −
X
H[x, y] = −
H[x| y] = −
H[y| x] = −
2 2 1 1
p(yi )lnp(yi ) = − ln − ln = 0.6365
3 3 3 3
i
1 1
p(x i , y j )lnp(x i , y j ) = −3 · ln − 0 = 1.0986
3 3
i, j
X
1
1 1 1 1
p(x i , y j )lnp(x i | y j ) = − ln1 − ln − ln = 0.4621
3
3 2 3 2
i, j
X
1 1
1 1 1
p(x i , y j )lnp(y j | x i ) = − ln − − ln − ln1 = 0.4621
3
2
3 2 3
i, j
X
I[x, y] = −
X
p(x i , y j )ln
i, j
p(x i )p(y j )
p(x i , y j )
2 1
3·3
2 2
1 2
1
1 3·3
1 3·3
= − ln
− ln
− ln
= 0.1744
3
1/3
3
1/3
3
1/3
Their relations are given below, diagrams omitted.
I[x, y] = H[x] − H[x| y] = H[y] − H[y| x]
33
H[x, y] = H[y| x] + H[x] = H[x| y] + H[y]
Problem 1.40 Solution
f (x) = lnx is actually a strict concave function, therefore we take advantage of Jensen’s Inequality to obtain:
f(
M
X
λm xm ) ≥
i =1
M
X
λm f (xm )
i =1
1
We let λm = M
, m = 1, 2, ..., M. Hence we will obtain:
ln(
x1 + x2 + ... + xm
1
1
)≥
[ ln(x1 ) + ln(x2 ) + ... + ln(x M ) ] =
ln(x1 x2 ...x M )
M
M
M
We take advantage of the fact that f (x) = lnx is strictly increasing and
then obtain :
x1 + x2 + ... + xm
p
≥ M x1 x2 ...x M
M
Problem 1.41 Solution
Based on definition of I[x, y], i.e.(1.120), we obtain:
Z Z
p(x)p(y)
dx dy
I[x, y] = −
p(x, y)ln
p(x, y)
Z Z
p(x)
= −
p(x, y)ln
dx dy
p(x|y)
Z Z
Z Z
= −
p(x, y)lnp(x) dx dy +
p(x, y)lnp(x|y) dx dy
Z Z
Z Z
= −
p(x)lnp(x) dx +
p(x, y)lnp(x|y) dx dy
=
H[x] − H[x|y]
Where we have taken advantage of the fact: p(x, y) = p(y)p(x|y), and
p(x, y) dy = p(x). The same process can be used for proving I[x, y] = H[y] −
H[y|x], if we substitute p(x, y) with p(x)p(y|x) in the second step.
R
0.2
Probability Distribution
Problem 2.1 Solution
Based on definition, we can obtain :
X
p(x i ) = µ + (1 − µ) = 1
x i =0,1
E[x] =
X
x i =0,1
x i p(x i ) = 0 · (1 − µ) + 1 · µ = µ
34
var[x] =
(x i − E[x])2 p(x i )
X
x i =0,1
= (0 − µ)2 (1 − µ) + (1 − µ)2 · µ
= µ(1 − µ)
H[x] = −
X
p(x i ) lnp(x i ) = − µ lnµ − (1 − µ) ln(1 − µ)
x i =0,1
Problem 2.2 Solution
The proof in Prob.2.1. can also be used here.
X
p(x i ) =
x i =−1,1
E[x] =
X
1−µ 1+µ
+
=1
2
2
x i · p(x i ) = −1 ·
x i =−1,1
var[x] =
X
1−µ
1+µ
+ 1·
=µ
2
2
(x i − E[x])2 · p(x i )
x i =−1,1
= (−1 − µ)2 ·
1−µ
1+µ
+ (1 − µ)2 ·
2
2
= 1 − µ2
H[x] = −
X
p(x i ) · ln p(x i ) = −
x i =−1,1
1−µ
1−µ 1+µ
1+µ
· ln
−
· ln
2
2
2
2
Problem 2.3 Solution
(2.262) is an important property of Combinations, which we have used
m
before, such as in Prob.1.15. We will use the ’old fashioned’ denotation C N
to
represent choose m objects from a total of N. With the prior knowledge:
m
CN
=
N!
m! (N − m)!
We evaluate the left side of (2.262) :
m
m−1
CN
+ CN
=
=
=
N!
N!
+
m! (N − m)! (m − 1)! (N − (m − 1))!
N!
1
1
( +
)
(m − 1)! (N − m)! m N − m + 1
(N + 1)!
m
= CN
+1
m! (N + 1 − m)!
To proof (2.263), here we will proof a more general form:
(x + y) N =
N
X
m=0
m m N −m
CN
x y
(∗)
35
If we let y = 1, (∗) will reduce to (2.263). We will proof it by induction.
First, it is obvious when N = 1, (∗) holds. We assume that it holds for N, we
will proof that it also holds for N + 1.
(x + y) N +1
= (x + y)
N
X
m=0
=
x
N
X
m=0
=
N
X
m=0
=
NX
+1
m=1
=
N
X
m=1
=
N
X
m=1
=
NX
+1
m=0
m m N −m
CN
x y
m m N −m
CN
x y
+y
m m+1 N − m
CN
x
y
+
N
X
m=0
N
X
m=0
m−1 m N +1− m
CN
x y
+
m m N −m
CN
x y
m m N +1− m
CN
x y
N
X
m=0
m m N +1− m
CN
x y
m−1
m m N +1− m
(C N
+ CN
)x y
+ x N +1 + y N +1
m
m N +1− m
+ x N +1 + y N +1
CN
+1 x y
m
m N +1− m
CN
+1 x y
By far, we have proved (∗). Therefore, if we let y = 1 in (∗), (2.263) has
been proved. If we let x = µ and y = 1 − µ, (2.264) has been proved.
Problem 2.4 Solution
Solution has already been given in the problem, but we will solve it in a
36
more intuitive way, beginning by definition:
E[m] =
N
X
m=0
=
N
X
m=1
m m
mC N
µ (1 − µ) N −m
m m
mC N
µ (1 − µ) N −m
N
X
=
N!
µm (1 − µ) N −m
(m
−
1)!(N
−
m)!
m=1
=
N ·µ
=
N ·µ
N
X
(N − 1)!
µm−1 (1 − µ) N −m
(m
−
1)!(N
−
m)!
m=1
N
X
m=1
=
N ·µ
NX
−1
k=0
=
m−1 m−1
CN
(1 − µ) N −m
−1 µ
k
k
N −1− k
CN
−1 µ (1 − µ)
N · µ [ µ + (1 − µ) ] N −1 = N µ
Some details should be explained here. We note that m = 0 actually
doesn’t affect the Expectation, so we let the summation begin from m = 1,
i.e. (what we have done from the first step to the second step). Moreover, in
the second last step, we rewrite the subindex of the summation, and what we
actually do is let k = m − 1. And in the last step, we have taken advantage of
(2.264). Variance is straightforward once Expectation has been calculated.
var[m] = E[m2 ] − E[m]2
N
X
m m
=
m2 C N
µ (1 − µ) N −m − E[m] · E[m]
m=0
=
N
X
m=0
=
N
X
m=1
=
N
X
m=1
=
Nµ
m m
m2 C N
µ (1 − µ) N −m − (N µ) ·
m m
m2 C N
µ (1 − µ) N −m − N µ ·
m
Nµ
m=0
N
X
m=1
m m
mC N
µ (1 − µ) N −m
m m
mC N
µ (1 − µ) N −m
N
X
N!
m m
µm (1 − µ) N −m − (N µ) ·
mC N
µ (1 − µ) N −m
(m − 1)!(N − m)!
m=1
N
X
m=1
=
N
X
N
X
m=1
m
N
X
(N − 1)!
m m
µm−1 (1 − µ) N −m − N µ ·
mC N
µ (1 − µ) N −m
(m − 1)!(N − m)!
m=1
m−1
m
mµm−1 (1 − µ) N −m ( C N
−1 − µC N )
Here we will use a little tick, −µ = −1 + (1 − µ) and then take advantage
37
m−1
m
m
.
+ CN
= CN
of the property, C N
−1
−1
var[m] =
Nµ
N
X
m=1
=
Nµ
N
X
m=1
=
Nµ
N
X
m=1
=
Nµ
£ m−1
¤
m
m
mµm−1 (1 − µ) N −m C N
−1 − C N + (1 − µ)C N
£
¤
m
m−1
m
mµm−1 (1 − µ) N −m (1 − µ)C N
+ CN
−1 − C N
£
¤
m
m
mµm−1 (1 − µ) N −m (1 − µ)C N
− CN
−1
nX
N
m=1
=
=
m
mµm−1 (1 − µ) N −m+1 C N
−
N
X
m=1
m
mµm−1 (1 − µ) N −m C N
−1
n
N µ · N(1 − µ)[µ + (1 − µ)] N −1 − (N − 1)(1 − µ)[µ + (1 − µ)] N −2
n
o
N µ N(1 − µ) − (N − 1)(1 − µ) = N µ(1 − µ)
o
o
Problem 2.5 Solution
Hints have already been given in the description, and let’s make a little
improvement by introducing t = y + x and x = tµ at the same time, i.e. we will
do following changes:
(
½
t = x+ y
x = tµ
and
µ = x+x y
y = t(1 − µ)
Note t ∈ [0, +∞], µ ∈ (0, 1), and that when we change variables in integral,
we will introduce a redundant term called Jacobian Determinant.
¯ ∂x ∂x ¯ ¯
¯
∂(x, y) ¯¯ ∂µ ∂ t ¯¯ ¯¯ t
µ ¯¯
=t
= ¯ ∂y ∂y ¯ =
∂(µ, t) ¯ ∂µ ∂ t ¯ ¯ − t 1 − µ ¯
Now we can calculate the integral.
Z +∞
Z +∞
exp(− x) xa−1 dx
exp(− y) yb−1 d y
Γ(a)Γ(b) =
0
0
Z +∞ Z +∞
a−1
exp(− y) yb−1 d y dx
=
exp(− x) x
0
0
Z +∞ Z +∞
=
exp(− x − y) xa−1 yb−1 d y dx
0
0
Z 1 Z +∞
=
exp(− t) (tµ)a−1 (t(1 − µ))b−1 t dt d µ
0 0
Z +∞
Z 1
a+ b−1
=
exp(− t) t
dt ·
µa−1 (1 − µ)b−1 d µ
0
0
Z 1
= Γ(a + b) ·
µa−1 (1 − µ)b−1 d µ
0
38
Therefore, we have obtained :
Z 1
0
µa−1 (1 − µ)b−1 d µ =
Γ(a)Γ(b)
Γ(a + b)
Problem 2.6 Solution
We will solve this problem based on definition.
E[µ] =
Z 1
µ Beta(µ|a, b) d µ
0
Γ(a + b) a
µ (1 − µ)b−1 d µ
0 Γ(a) Γ(b)
Z
Γ(a + b)Γ(a + 1) 1 Γ(a + 1 + b) a
µ (1 − µ)b−1 d µ
Γ(a + 1 + b)Γ(a) 0 Γ(a + 1) Γ(b)
Z
Γ(a + b)Γ(a + 1) 1
Beta(µ|a + 1, b) d µ
Γ(a + 1 + b)Γ(a) 0
Γ(a + 1)
Γ(a + b)
·
Γ(a + 1 + b)
Γ(a)
a
a+b
Z 1
=
=
=
=
=
Where we have taken advantage of the property: Γ(z + 1) = zΓ(z). For
variance, it is quite similar. We first evaluate E[µ2 ].
E[ µ 2 ] =
Z 1
0
µ2 Beta(µ|a, b) d µ
Γ(a + b) a+1
(1 − µ)b−1 d µ
µ
Γ
(a)
Γ
(b)
0
Z
Γ(a + b)Γ(a + 2) 1 Γ(a + 2 + b) a+1
µ
(1 − µ)b−1 d µ
Γ(a + 2 + b)Γ(a) 0 Γ(a + 2) Γ(b)
Z
Γ(a + b)Γ(a + 2) 1
Beta(µ|a + 2, b) d µ
Γ(a + 2 + b)Γ(a) 0
Γ(a + b)
Γ(a + 2)
·
Γ(a + 2 + b)
Γ(a)
a(a + 1)
(a + b)(a + b + 1)
Z 1
=
=
=
=
=
Then we use the formula: var[µ] = E[µ2 ] − E[µ]2 .
var[µ] =
=
Problem 2.7 Solution
a(a + 1)
a 2
−(
)
(a + b)(a + b + 1)
a+b
ab
2
(a + b) (a + b + 1)
39
The maximum likelihood estimation for µ, i.e. (2.8), can be written as :
µ ML =
m
m+l
Where m represents how many times we observe ’head’, l represents how
many times we observe ’tail’. And the prior mean of µ is given by (2.15), the
posterior mean value of x is given by (2.20). Therefore, we will prove that
(m + a) / (m + a + l + b) lies between m / (m + l), a / (a + b). Given the fact that :
λ
m
m+a
a+b
a
+ (1 − λ)
=
where λ =
a+b
m+l
m+a+l +b
m+l +a+b
We have solved problem. Note : you can also solve it in a more simple way
by prove that :
(
m+a
a
m+a
m
−
)·(
−
)≤0
m+a+l +b a+b
m+a+l +b m+l
The expression above can be proved by reduction of fractions to a common
denominator.
Problem 2.8 Solution
We solve it base on definition.
E y [E x [x| y]]] =
=
=
=
=
Z
E x [x| y]p(y) d y
Z Z
( x p(x| y) dx)p(y) d y
Z Z
x p(x| y) p(y) dx d y
Z Z
x p(x, y) dx d y
Z
x p(x) dx = E[x]
(2.271) is complicated and we will calculate every term separately.
Z
E y [var x [x| y]] =
var x [x| y]p(y) d y
Z Z
=
( (x − E x [x| y])2 p(x| y) dx ) p(y) d y
Z Z
=
(x − E x [x| y])2 p(x, y) dx d y
Z Z
=
(x2 − 2 x E x [x| y] + E x [x| y]2 ) p(x, y) dx d y
Z Z
Z Z
Z Z
=
x2 p(x) dx −
2 x E x [x| y] p(x, y) dx d y +
(E x [x| y]2 ) p(y) d y
40
About the second term in the equation above, we further simplify it :
Z Z
Z
Z
2 x E x [x| y] p(x, y) dx d y = 2 E x [x| y] ( xp(x, y) dx ) d y
Z
Z
= 2 E x [x| y] p(y) ( xp(x| y) dx ) d y
Z
= 2 E x [x| y]2 p(y) d y
Therefore, we obtain the simple expression for the first term on the right
side of (2.271) :
Z Z
Z Z
E y [var x [x| y]] =
x2 p(x) dx −
E x [x| y]2 p(y) d y
(∗)
Then we process for the second term.
Z
var y [E x [x| y]] =
(E x [x| y] − E y [E x [x| y]])2 p(y) d y
Z
=
(E x [x| y] − E[x])2 p(y) d y
Z
Z
Z
=
E x [x| y]2 p(y) d y − 2 E[x]E x [x| y]p(y) d y + E[x]2 p(y) d y
Z
Z
2
=
E x [x| y] p(y) d y − 2E[x] E x [x| y]p(y) d y + E[x]2
Then following the same procedure, we deal with the second term of the
equation above.
Z
2E[x] · E x [x| y]p(y) d y = 2E[x] · E y [E x [x| y]]] = 2E[x]2
Therefore, we obtain the simple expression for the second term on the
right side of (2.271) :
Z
var y [E x [x| y]] = E x [x| y]2 p(y) d y − E[x]2
(∗∗)
Finally, we add (∗) and (∗∗), and then we will obtain:
E y [var x [x| y]] + var y [E x [x| y]] = E[x2 ] − E[x]2 = var[x]
Problem 2.9 Solution
This problem is complexed, but hints have already been given in the description. Let’s begin by performing integral of (2.272) over µ M −1 . (Note :
41
by integral over µ M −1 , we actually obtain Dirichlet distribution with M − 1
variables.)
Z 1−µ−m − ... −µM −2
p M −1 (µ, m, ..., µ M −2 ) =
CM
0
k=1
CM
=
M
−1
Y
M
−2
Y
k=1
α −1
µk k
α −1
µk k
(1 −
M
−1
X
µ j )αM −1 d µ M −1
j =1
Z 1−µ−m − ... −µM −2
0
α
µ MM−−11
−1
(1 −
M
−1
X
µ j )αM −1 d µ M −1
j =1
We change variable by :
t=
µ M −1
1 − µ − m − ... − µ M −2
The reason we do so is that µ M −1 ∈ [0, 1 − µ − m − ... − µ M −2 ], by making this
changing of variable, we can see that t ∈ [0, 1]. Then we can further simplify
the expression.
p M −1
=
CM
M
−2
Y
k=1
=
CM
M
−2
Y
k=1
=
CM
M
−2
Y
k=1
α −1
µk k (1 −
α −1
µk k
α −1
µk k
(1 −
M
−2
X
µ j)
α M −1 +α M −1
j =1
M
−2
X
µ j )αM −1 +αM −1
j =1
(1 −
M
−2
X
j =1
µ j )αM −1 +αM −1
Z 1
0
Z 1
0
α
µ MM−−11
−1
(1 −
P M −1
j =1
µ j )αM −1
(1 − µ − m − ... − µ M −2 )αM −1 +αM −2
tαM −1 −1 (1 − t)αM −1 dt
Γ(α M −1 − 1)Γ(α M )
Γ(α M −1 + α M )
Comparing the expression above with a normalized Dirichlet Distribution
with M −1 variables, and supposing that (2.272) holds for M −1, we can obtain
that:
Γ(α M −1 )Γ(α M )
Γ(α1 + α2 + ... + α M )
CM
=
Γ(α M −1 + α M )
Γ(α1 )Γ(α2 )...Γ(α M −1 + α M )
Therefore, we obtain
CM =
as required.
Problem 2.10 Solution
Γ(α1 + α2 + ... + α M )
Γ(α1 )Γ(α2 )...Γ(α M −1 )Γ(α M )
dt
42
Based on definition of Expectation and (2.38), we can write:
Z
E[µ j ] =
µ j D ir(µ|α) d µ
K
Y
Γ(α0 )
α −1
µ k dµ
Γ(α1 )Γ(α2 )...Γ(αK ) k=1 k
Z
K
Y
Γ(α0 )
α −1
µj
µk k d µ
Γ(α1 )Γ(α2 )...Γ(αK )
k=1
Z
=
=
=
=
µj
Γ(α1 )Γ(α2 )...Γ(α j−1 )Γ(α j + 1)Γ(α j+1 )...Γ(αK )
Γ(α0 )
Γ(α1 )Γ(α2 )...Γ(αK )
Γ(α0 + 1)
αj
Γ(α0 )Γ(α j + 1)
=
Γ(α j )Γ(α0 + 1) α0
It is quite the same for variance, let’s begin by calculating E[µ2j ].
E[µ2j ] =
=
=
=
Z
µ2j D ir(µ|α) d µ
Γ(α0 )
Γ(α1 )Γ(α2 )...Γ(αK )
Z
µ2j
K
Y
k=1
α −1
µk k
dµ
Γ(α1 )Γ(α2 )...Γ(α j−1 )Γ(α j + 2)Γ(α j+1 )...Γ(αK )
Γ(α0 )
Γ(α1 )Γ(α2 )...Γ(αK )
Γ(α0 + 2)
Γ(α0 )Γ(α j + 2)
α j (α j + 1)
=
Γ(α j )Γ(α0 + 2) α0 (α0 + 1)
Hence, we obtain :
var[µ j ] = E[µ2j ] − E[µ j ]2 =
α j (α j + 1)
α0 (α0 + 1)
−(
αj 2
α j (α0 − α j )
) = 2
α0 (α0 + 1)
α0
It is the same for covariance.
Z
cov[µ j µl ] =
(µ j − E[µ j ])(µl − E[µl ]) D ir(µ|α) d µ
Z
=
(µ j µl − E[µ j ]µl − E[µl ]µ j + E[µ j ]E[µl ]) D ir(µ|α) d µ
=
=
=
=
Γ(α0 )Γ(α j + 1)Γ(αl + 1)
Γ(α j )Γ(αl )Γ(α0 + 2)
α j αl
α0 (α0 + 1)
α j αl
− E[µ j ]E[µl ]
−
α0 (α0 + 1)
α j αl
− 2
α0 (α0 + 1)
α j αl
α20
( j 6= l )
− 2E[µ j ]E[µl ] + E[µ j ]E[µl ]
43
Note : when j = l, cov[µ j µl ] will actually reduce to var[µ j ], however we
cannot simply replace l with j in the expression of cov[µ j µl ] to get the right
R
R
result and that is because µ j µl D ir(µ|α) d α will reduce to µ2j D ir(µ|α) d α
in this case.
Problem 2.11 Solution
Based on definition of Expectation and (2.38), we first denote :
Γ(α0 )
= K(α)
Γ(α1 )Γ(α2 )...Γ(αK )
Then we can write :
∂D ir(µ|α)
∂α j
= ∂(K(α)
K
Y
α −1
µi i
i =1
K
∂K(α) Y
=
∂α j
i =1
K
∂K(α) Y
=
∂α j
i =1
) / ∂α j
α −1
µ i i + K(α)
α −1
µi i
∂
α i −1
i =1 µ i
QK
∂α j
+ lnµ j · D ir(µ|α)
Then let us perform integral to both sides:
Z
∂D ir(µ|α)
∂α j
dµ =
Z
K
∂K(α) Y
∂α j
i =1
α −1
µi i dµ +
Z
lnµ j · D ir(µ|α) d µ
The left side can be further simplified as :
R
∂ D ir(µ|α) d µ
∂1
left side =
=
=0
∂α j
∂α j
The right side can be further simplified as :
right side =
=
=
∂K(α)
∂α j
∂K(α)
Z Y
K
i =1
α −1
µi i
d µ + E[lnµ j ]
1
+ E[lnµ j ]
∂α j K(α)
∂ lnK(α)
+ E[lnµ j ]
∂α j
44
Therefore, we obtain :
E[lnµ j ] = −
= −
=
=
=
∂ lnK(α)
∂α j
n
o
P
∂ lnΓ(α0 ) − K
i =1 lnΓ(α i )
∂α j
∂ lnΓ(α j )
∂α j
∂ lnΓ(α j )
∂α j
∂ lnΓ(α j )
−
−
−
∂ lnΓ(α0 )
∂α j
∂ lnΓ(α0 ) ∂α0
∂α0
∂α j
∂ lnΓ(α0 )
∂α j
∂α0
= ψ(α j ) − ψ(α0 )
Therefore, the problem has been solved.
Problem 2.12 Solution
Since we have :
Z b
a
1
dx = 1
b−a
It is straightforward that it is normalized. Then we calculate its mean :
E[x] =
Z b
x
a
x2 ¯¯b a + b
1
dx =
¯ =
b−a
2(b − a) a
2
Then we calculate its variance.
Z b 2
x
x3 ¯¯b
a+b 2
a+b 2
2
2
var[x] = E[x ] − E[x] =
dx − (
) =
)
¯ −(
2
3(b − a) a
2
a b−a
Hence we obtain:
var[x] =
(b − a)2
12
Problem 2.13 Solution
This problem is an extension of Prob.1.30. We can follow the same procep ( x)
dure to solve it. Let’s begin by calculating ln q(( x) :
ln(
p(x)
) =
q(x)
1
|L|
1
1
ln (
) + (x − m)T L−1 (x − m) − (x − µ)T Σ−1 (x − µ)
2
|Σ|
2
2
If x ∼ p(x) = N (µ|Σ), we then take advantage of the following properties.
Z
p(x) dx = 1
45
E[x] =
Z
x p(x) dx = µ
E[(x − a)T A(x − a)] = tr(A Σ) + (µ − a)T A(µ − a)
We obtain :
Z n
o
1 |L| 1
1
KL =
ln
− (x − µ)T Σ−1 (x − µ) + (x − m)T L−1 (x − m) p(x) dx
2 |Σ| 2
2
1 |L| 1
1
=
ln
− E[(x − µ)Σ−1 (x − µ)T ] + E[(x − m)T L−1 (x − m)]
2 | Σ| 2
2
1
1
1 |L| 1
ln
− tr { I D } + (µ − m)T L−1 (µ − m) + tr{L−1 Σ}
=
2 | Σ| 2
2
2
|L|
1
[ln
− D + tr{L−1 Σ} + (m − µ)T L−1 (m − µ)]
=
2
|Σ|
Problem 2.14 Solution
The hint given in the problem is straightforward, however it is a little bit
difficult to calculate, and here we will use a more simple method to solve this
problem, taking advantage of the property of Kullback—Leibler Distance. Let
g(x) be a Gaussian PDF with mean µ and variance Σ, and f (x) an arbitrary
PDF with the same mean and variance.
½
¾
Z
Z
g(x)
0 ≤ K L( f || g) = − f (x)ln
dx = − H( f ) − f (x)lng(x) dx
(∗)
f (x)
Let’s calculate the second term of the equation above.
¾
½
Z
Z
£ 1
¤
1
1
T −1
exp
−
(x
−
µ
)
Σ
(x
−
µ
)
dx
f (x)lng(x) dx =
f (x)ln
2
(2π)D /2 |Σ|1/2
½
¾
Z
Z
¤
£ 1
1
1
T −1
=
f (x)ln
Σ
(x
−
µ
)
dx
dx
+
f
(x)
−
(x
−
µ
)
2
(2π)D /2 |Σ|1/2
½
¾
¤
1
1
1 £
T −1
= ln
−
E
(x
−
µ
)
Σ
(x
−
µ
)
2
(2π)D /2 |Σ|1/2
½
¾
1
1
1
= ln
− tr{ I D }
D
/2
1/2
2
(2π)
|Σ|
¾
½
D
1
ln|Σ| + (1 + ln(2π))
= −
2
2
= − H(g)
We take advantage of two properties of PDF f (x), with mean µ and variance Σ, as listed below. What’s more, we also use the result of Prob.2.15,
which we will proof later.
Z
f (x) dx = 1
E[(x − a)T A(x − a)] = tr(A Σ) + (µ − a)T A(µ − a)
46
Now we can further simplify (∗) to obtain:
H(g) ≥ H( f )
In other words, we have proved that an arbitrary PDF f (x) with the same
mean and variance as a Gaussian PDF g(x), its entropy cannot be greater
than that of Gaussian PDF.
Problem 2.15 Solution
We have already used the result of this problem to solve Prob.2.14, and
now we will prove it. Suppose x ∼ p(x) = N (µ|Σ) :
Z
H[x] = − p(x)lnp(x) dx
¾
½
Z
¤
£ 1
1
1
T −1
= − p(x)ln
exp − (x − µ) Σ (x − µ) dx
2
(2π)D /2 |Σ|1/2
½
¾
Z
Z
£ 1
¤
1
1
= − p(x)ln
dx
−
f (x) − (x − µ)T Σ−1 (x − µ) dx
D
/2
1/2
2
(2π)
|Σ|
¾
½
¤
1
1 £
1
+ E (x − µ)T Σ−1 (x − µ)
= − ln
D
/2
1/2
2
(2π)
|Σ|
½
¾
1
1
1
= − ln
+ tr{ I D }
2
(2π)D /2 |Σ|1/2
1
D
=
ln|Σ| + (1 + ln(2π))
2
2
Where we have taken advantage of :
Z
p(x) dx = 1
E[(x − a)T A(x − a)] = tr(A Σ) + (µ − a)T A(µ − a)
Note : Actually in Prob.2.14, we have already solved this problem, you can
intuitively view it by replacing the integrand f (x)lng(x) with g(x)lng(x), and
R
the same procedure in Prob.2.14 still holds to calculate g(x)lng(x) dx.
Problem 2.16 Solution
Let us consider a more general conclusion about the Probability Density
Function (PDF) of the summation of two independent random variables. We
denote two random variables X and Y . Their summation Z = X + Y , is still a
random variable. We also denote f (·) as PDF, and F(·) as Cumulative Distribution Function (CDF). We can obtain :
Ï
F Z (z) = P(Z < z) =
f X ,Y (x, y) dxd y
x+ y≤ z
47
Where z represents an arbitrary real number. We rewrite the double integral into iterated integral :
¸
Z +∞ ·Z z− y
f X ,Y (x, y) dx d y
F Z (z) =
−∞
−∞
We fix z and y, and then make a change of variable x = u− y to the integral.
¸
¸
Z +∞ ·Z z
Z +∞ ·Z z− y
f X ,Y (u − x, y) du d y
f X ,Y (x, y) dx d y =
F Z (z) =
−∞
−∞
−∞
−∞
Note: f X ,Y (·) is the joint PDF of X and Y , and then we rearrange the
order, we will obtain :
¸
Z z ·Z +∞
f X ,Y (u − y, y) d y du
F Z (z) =
−∞
−∞
Compare the equation above with th definition of CDF :
Z z
f Z (u) du
F Z (z) =
−∞
We can obtain :
Z +∞
f Z (u) =
−∞
f X ,Y (u − y, y) d y
And if X and Y are independent, which means f X ,Y (x, y) = f X (x) f Y (y), we
can simplify f Z (z) :
Z +∞
f X (u − y) f Y (y) d y i.e. f Z = f X ∗ f Y
f Z (u) =
−∞
Until now we have proved that the PDF of the summation of two independent random variable is the convolution of the PDF of them. Hence it is
straightforward to see that in this problem, where random variable x is the
summation of random variable x1 and x2 , the PDF of x should be the convolution of the PDF of x1 and x2 . To find the entropy of x, we will use a simple
method, taking advantage of (2.113)-(2.117). With the knowledge :
1
p(x2 ) = N (µ2 , τ−
2 )
1
p(x| x2 ) = N (µ1 + x2 , τ−
1 )
We make analogies : x2 in this problem to x in (2.113), x in this problem to
y in (2.114). Hence by using (2.115), we can obtain p(x) is still a normal distribution, and since the entropy of a Gaussian is fully decided by its variance,
there is no need to calculate the mean. Still by using (2.115), the variance of
1
−1
x is τ−
1 + τ2 , which finally gives its entropy :
H[x] =
¤
1£
1
−1
1 + ln2π(τ−
1 + τ2 )
2
48
Problem 2.17 Solution
This is an extension of Prob.1.14. The same procedure can be used here.
We suppose an arbitrary precision matrix Λ can be written as ΛS + Λ A , where
they satisfy :
Λ i j + Λ ji
Λ i j − Λ ji
ΛSij =
, Λ iAj =
2
2
Hence it is straightforward that ΛSij = ΛSji , and Λ iAj = − Λ Aji . If we expand
the quadratic form of exponent, we will obtain :
(x − µ)T Λ(x − µ) =
D X
D
X
(x i − µ i )Λ i j (x j − µ j )
(∗)
i =1 j =1
It is straightforward then :
(∗) =
D X
D
X
(x i − µ i )ΛSij (x j − µ j ) +
i =1 j =1
=
D X
D
X
D X
D
X
(x i − µ i )Λ iAj (x j − µ j )
i =1 j =1
(x i − µ i )ΛSij (x j − µ j )
i =1 j =1
Therefore, we can assume precision matrix is symmetric, and so is covariance matrix.
Problem 2.18 Solution
We will just follow the hint given in the problem. Firstly, we take complex
conjugate on both sides of (2.45) :
Σu i = λi u i
=>
Σu i = λi u i
Where we have taken advantage of the fact that Σ is a real matrix, i.e.,
Σ = Σ. Then using that Σ is a symmetric, i.e., ΣT = Σ :
u i T Σu i = u i T ( Σu i ) = u i T ( λi u i ) = λi u i T u i
T
u i T Σ u i = ( Σ u i )T u i = ( λ i u i )T u i = λ i u i T u i
T
Since u i 6= 0, we have u i T u i 6= 0. Thus λTi = λ i , which means λ i is real.
Next we will proof that two eigenvectors corresponding to different eigenvalues are orthogonal.
λ i < u i , u j > = < λ i u i , u j > = < Σ u i , u j > = < u i , ΣT u j > = λ j < u i , u j >
Where we have taken advantage of ΣT = Σ and for arbitrary real matrix
A and vector x, y, we have :
< Ax, y > = < x, A T y >
49
Provided λ i 6= λ j , we have < u i , u j > = 0, i.e., u i and u j are orthogonal. And then if we perform normalization on every eigenvector to force its
Euclidean norm to equal to 1, (2.46) is straightforward. By performing normalization, I mean multiplying the eigenvector by a real number a to let its
Euclidean norm (length) to equal to 1, meanwhile we should also divide its
corresponding eigenvalue by a.
Problem 2.19 Solution
For every N × N real symmetric matrix, the eigenvalues are real and the
eigenvectors can be chosen such that they are orthogonal to each other. Thus
a real symmetric matrix Σ can be decomposed as Σ = U ΛU T ,where U is an
orthogonal matrix, and Λ is a diagonal matrix whose entries are the eigenvalues of A. Hence for an arbitrary vector x, we have:



u 1T x
λ1 u 1T x
D
X
 . 


..
Σ x = U ΛU T x = U Λ  ..  = U 
λk u k u T
=(
.
k )x

uT
x
D
λD u T
x
D
k=1
And since Σ−1 = U Λ−1U T , the same procedure can be used to prove
(2.49).
Problem 2.20 Solution
Since u 1 , u 2 , ..., u D can constitute a basis for RD , we can make projection
for a :
a = a 1 u 1 + a 2 u 2 + ... + a D u D
We substitute the expression above into aT Σa, taking advantage of the
property: u i u j = 1 only if i = j, otherwise 0, we will obtain :
aT Σa = (a 1 u 1 + a 2 u 2 + ... + a D u D )T Σ(a 1 u 1 + a 2 u 2 + ... + a D u D )
= (a 1 u 1 T + a 2 u 2 T + ... + a D u D T )Σ(a 1 u 1 + a 2 u 2 + ... + a D u D )
= (a 1 u 1 T + a 2 u 2 T + ... + a D u D T )(a 1 λ1 u 1 + a 2 λ2 u 2 + ... + a D λD u D )
= λ1 a 1 2 + λ2 a 2 2 + ... + λD a D 2
Since a is real,the expression above will be strictly positive for any nonzero a, if all eigenvalues are strictly positive. It is also clear that if an eigenvalue, λ i , is zero or negative, there will exist a vector a (e.g. a = u i ), for
which this expression will be no greater than 0. Thus, that a real symmetric matrix has eigenvectors which are all strictly positive is a sufficient and
necessary condition for the matrix to be positive definite.
Problem 2.21 Solution
It is straightforward. For a symmetric matrix Λ of size D × D, when the
lower triangular part is decided, the whole matrix will be decided due to
50
symmetry. Hence the number of independent parameters is D + (D − 1) + ... +
1, which equals to D(D + 1)/2.
Problem 2.22 Solution
Suppose A is a symmetric matrix, and we need to prove that A −1 is also
symmetric, i.e., A −1 = (A −1 )T . Since identity matrix I is also symmetric, we
have :
A A −1 = (A A −1 )T
And since AB T = B T A T holds for arbitrary matrix A and B, we will
obtain :
T
A A −1 = (A −1 ) A T
Since A = A T , we substitute the right side:
T
A A −1 = (A −1 ) A
And note that A A −1 = A −1 A = I, we rearrange the order of the left side :
T
A −1 A = (A −1 ) A
Finally, by multiplying A −1 to both sides, we can obtain:
T
A −1 A A −1 = (A −1 ) A A −1
Using A A −1 = I, we will get what we are asked :
A −1 = (A −1 )
T
Problem 2.23 Solution
Let’s reformulate the problem. What the problem wants us to prove is
that if (x − µ)T Σ−1 (x − µ) = r 2 , where r 2 is a constant, we will have the
volume of the hyperellipsoid decided by the equation above will equal to
VD |Σ|1/2 r D . Note that the center of this hyperellipsoid locates at µ, and a
translation operation won’t change its volume, thus we only need to prove
that the volume of a hyperellipsoid decided by xT Σ−1 x = r 2 , whose center
locates at 0 equals to VD |Σ|1/2 r D .
This problem can be viewed as two parts. Firstly, let’s discuss about VD ,
the volume of a unit sphere in dimension D. The expression of VD has already
be given in the solution procedure of Prob.1.18, i.e., (1.144) :
VD =
SD
2πD /2
=
D
Γ( D + 1)
2
And also in the procedure, we show that a D dimensional sphere with
radius r, i.e., xT x = r 2 , has volume V (r) = VD r D . We move a step forward: we
51
perform a linear transform using matrix Σ1/2 , i.e., yT y = r 2 , where y = Σ1/2 x.
After the linear transformation, we actually get a hyperellipsoid whose center
locates at 0, and its volume is given by multiplying V (r) with the determinant
of the transformation matrix, which gives |Σ|1/2 VD r D , just as required.
Problem 2.24 Solution
We just following the hint, and firstly let’s calculate :
·
A
C
¸ ·
×
B
D
M
−D −1 CM
− MBD −1
−1
D + D −1 CMBD −1
¸
The result can also be partitioned into four blocks. The block located at
left top equals to :
AM − BD −1 CM = (A − BD −1 C)(A − BD −1 C)−1 = I
Where we have taken advantage of (2.77). And the right top equals to :
− AMBD −1 + BD −1 + BD −1 CMBD −1 = (I − AM + BD −1 CM)BD −1 = 0
Where we have used the result of the left top block. And the left bottom
equals to :
CM − DD −1 CM = 0
And the right bottom equals to :
−CMBD −1 + DD −1 + DD −1 CMDD −1 = I
we have proved what we are asked. Note: if you want to be more precise,
you should also multiply the block matrix on the right side of (2.76) and then
prove that it will equal to a identity matrix. However, the procedure above
can be also used there, so we omit the proof and what’s more, if two arbitrary
square matrix X and Y satisfied X Y = I, it can be shown that Y X = I also
holds.
Problem 2.25 Solution
We will take advantage of the result of (2.94)-(2.98). Let’s first begin by
grouping xa and x b together, and then we rewrite what has been given as :
µ
x=
xa,b
xc
¶
µ=
µ
µa,b
µc
¶
Σ=
·
Σ(a,b)(a,b)
Σ(a,b) c
Then we take advantage of (2.98), we can obtain :
p(xa,b ) = N (xa,b |µa,b , Σ(a,b)(a,b) )
Σ(a,b) c
Σ cc
¸
52
Where we have defined:
µ
¶
µa
µa,b =
µb
Σ(a,b)(a,b) =
·
Σaa
Σba
Σab
Σbb
¸
Since now we have obtained the joint contribution of xa and x b , we will
take advantage of (2.96) (2.97) to obtain conditional distribution, which gives:
1
p(xa | x b ) = N (x|µa|b , Λ−
aa )
Where we have defined
1
µa|b = µa − Λ−
aa Λab (x b − µ b )
1
And the expression of Λ−
aa and Λab can be given by using (2.76) and (2.77)
once we notice that the following relation exits:
·
Λaa
Λba
Λab
Λbb
¸
·
=
Σaa
Σba
Σab
Σbb
¸−1
Problem 2.26 Solution
This problem is quite straightforward, if we just follow the hint.
¡
¢
(A + BCD) A −1 − A −1 B(C −1 + D A −1 B)−1 D A −1
= A A −1 − A A −1 B(C −1 + D A −1 B)−1 D A −1 + BCD A −1 − BCD A −1 B(C −1 + D A −1 B)−1 D A −1
= I − B(C −1 + D A −1 B)−1 D A −1 + BCD A −1 + B(C −1 + D A −1 B)−1 D A −1 − BCD A −1
=I
Where we have taken advantage of
− BCD A −1 B(C −1 + D A −1 B)−1 D A −1
= −BC(−C −1 + C −1 + D A −1 B)(C −1 + D A −1 B)−1 D A −1
= (−BC)(−C −1 )(C −1 + D A −1 B)−1 D A −1 + (−BC)(C −1 + D A −1 B)(C −1 + D A −1 B)−1 D A −1
= B(C −1 + D A −1 B)−1 D A −1 − BCD A −1
Here we will also directly calculate the inverse matrix instead to give
another solution. Let’s first begin by introducing two useful formulas.
(I + P)−1
= (I + P)−1 (I + P − P)
=
I − (I + P)−1 P
And since
P + PQP = P(I + QP) = (I + PQ)P
53
The second formula is :
(I + PQ)−1 P = P(I + QP)−1
And now let’s directly calculate (A + BCD)−1 :
(A + BCD)−1
= [A(I + A −1 BCD)]−1
= (I + A −1 BCD)−1 A −1
£
¤
= I − (I + A −1 BCD)−1 A −1 BCD A −1
=
A −1 − (I + A −1 BCD)−1 A −1 BCD A −1
Where we have assumed that A is invertible and also used the first formula we introduced. Then we also assume that C is invertible and recursively use the second formula :
(A + BCD)−1
=
A −1 − (I + A −1 BCD)−1 A −1 BCD A −1
=
A −1 − A −1 (I + BCD A −1 )−1 BCD A −1
=
=
A −1 − A −1 B(I + CD A −1 B)−1 CD A −1
£
¤−1
CD A −1
A −1 − A −1 B C(C −1 + D A −1 B)
=
A −1 − A −1 B(C −1 + D A −1 B)−1 C −1 CD A −1
=
A −1 − A −1 B(C −1 + D A −1 B)−1 D A −1
Just as required.
Problem 2.27 Solution
The same procedure used in Prob.1.10 can be used here similarly.
Z Z
E[x + z] =
(x + z)p(x, z) dx dz
Z Z
=
(x + z)p(x)p(z) dx dz
Z Z
Z Z
=
xp(x)p(z) dx dz +
z p(x)p(z) dx dz
Z Z
Z Z
=
( p(z) dz)xp(x) dx + ( p(x) dx)z p(z) dz
Z
Z
=
xp(x) dx + z p(z) dz
= E[x] + E[z]
And for covariance matrix, we will use matrix integral :
Z Z
cov[x + z] =
( x + z − E[x + z] )( x + z − E[x + z] )T p(x, z) dx dz
Also the same procedure can be used here. We omit the proof for simplicity.
54
Problem 2.28 Solution
It is quite straightforward when we compare the problem with (2.94)(2.98). We treat x in (2.94) as z in this problem, xa in (2.94) as x in this
problem, x b in (2.94) as y in this problem. In other words, we rewrite the
problem in the form of (2.94)-(2.98), which gives :
µ
z=
x
y
¶
E(z) =
µ
µ
Aµ + b
¶
·
cov(z) =
Λ−1 A T
−1
L + A Λ−1 A T
Λ−1
A Λ−1
¸
By using (2.98), we can obtain:
p(x) = N (x|µ, Λ−1 )
And by using (2.96) and (2.97), we can obtain :
p(y| x) = N (y|µ y| x , Λ−yy1 )
Where Λ yy can be obtained by the right bottom part of (2.104),which gives
Λ yy = L−1 , and you can also calculate it using (2.105) combined with (2.78)
and (2.79). Finally the conditional mean is given by (2.97) :
µ y| x = A µ + L − L−1 (−LA)(x − µ) = Ax + L
Problem 2.29 Solution
It is straightforward. Firstly, we calculate the left top block :
h
i−1
left top = (Λ + A T LA) − (− A T L)(L−1 )(−LA)
= Λ−1
And then the right top block :
right top = −Λ−1 (− A T L)L−1 = Λ−1 A T
And then the left bottom block :
left bottom = −L−1 (−LA)Λ−1 = A Λ−1
Finally the right bottom block :
right bottom = L−1 + L−1 (−LA)Λ−1 (− A T L)L−1 = L−1 + A Λ−1 A T
Problem 2.30 Solution
It is straightforward by multiplying (2.105) and (2.107), which gives :
µ
Λ−1
A Λ−1
Λ−1 A T
L−1 + A Λ−1 A T
Just as required in the problem.
¶µ
Λµ − A T Lb
Lb
¶
µ
=
µ
Aµ + b
¶
55
Problem 2.31 Solution
According to the problem, we can write two expressions :
p(x) = N (x|µ x , Σ x ) ,
p(y| x) = N (y|µ z + x, Σ z )
By comparing the expression above and (2.113)-(2.117), we can write the
expression of p(y) :
p(y) = N (y|µ x + µ z , Σ x + Σ z )
Problem 2.32 Solution
Let’s make this problem more clear. The deduction in the main text, i.e.,
(2.101-2.110), firstly denote a new random variable z corresponding to the
joint distribution, and then by completing square according to z,i.e.,(2.103),
obtain the precision matrix R by comparing (2.103) with the PDF of a multivariate Gaussian Distribution, and then it takes the inverse of precision
matrix to obtain covariance matrix, and finally it obtains the linear term i.e.,
(2.106) to calculate the mean.
In this problem, we are asked to solve the problem from another perspective: we need to write the joint distribution p(x, y) and then perform integration over x to obtain marginal distribution p(y). Let’s begin by write the
quadratic form in the exponential of p(x, y) :
1
1
− (x − µ)T Λ(x − µ) − (y − Ax − b)T L(y − Ax − b)
2
2
We extract those terms involving x :
1
= − xT (Λ + A T LA) x + xT [Λµ + A T L(y − b) ] + const
2
1
1
= − (x − m)T (Λ + A T LA) (x − m) + mT (Λ + A T LA)m + const
2
2
Where we have defined :
m = (Λ + A T LA)−1 [Λµ + A T L(y − b) ]
Now if we perform integration over x, we will see that the first term vanish to a constant, and we extract the terms including y from the remaining
parts, we can obtain :
i
1 h
= − yT L − LA (Λ + A T LA)−1 A T L y
2 n
£
¤
+ yT L − LA (Λ + A T LA)−1 A T L b
o
+LA(Λ + A T LA)−1 Λµ
We firstly view the quadratic term to obtain the precision matrix, and
then we take advantage of (2.289), we will obtain (2.110). Finally, using the
56
linear term combined with the already known covariance matrix, we can obtain (2.109).
Problem 2.33 Solution
p( x,y)
According to Bayesian Formula, we can write p(x| y) = p( y) , where we
have already known the joint distribution p(x, y) in (2.105) and (2.108), and
the marginal distribution p(y) in Prob.2.32., we can follow the same procedure in Prob.2.32., i.e. firstly obtain the covariance matrix from the quadratic
term and then obtain the mean from the linear term. The details are omitted
here.
Problem 2.34 Solution
Let’s follow the hint by firstly calculating the derivative of (2.118) with
respect to Σ and let it equal to 0 :
−
N
N ∂
1 ∂ X
ln|Σ| −
(xn − µ)T Σ−1 (xn − µ) = 0
2 ∂Σ
2 ∂Σ n=1
By using (C.28), the first term can be reduced to :
−
N ∂
N
N
ln|Σ| = − (Σ−1 )T = − Σ−1
2 ∂Σ
2
2
Provided with the result that the optimal covariance matrix is the sample
covariance, we denote sample matrix S as :
S=
N
1 X
(xn − µ)(xn − µ)T
N n=1
We rewrite the second term :
second term = −
N
1 ∂ X
(xn − µ)T Σ−1 (xn − µ)
2 ∂Σ n=1
N ∂
T r[Σ−1 S]
2 ∂Σ
N −1
Σ S Σ−1
2
= −
=
Where we have taken advantage of the following property, combined with
the fact that S and Σ is symmetric. (Note : this property can be found in The
Matrix Cookbook.)
∂
∂X
T r(A X −1 B) = −(X −1 BA X −1 )T = −(X −1 )T A T B T (X −1 )T
Thus we obtain :
−
N −1 N −1
Σ + Σ S Σ−1 = 0
2
2
57
Obviously, we obtain Σ = S, just as required.
Problem 2.35 Solution
The proof of (2.62) is quite clear in the main text, i.e., from page 82 to
page 83 and hence we won’t repeat it here. Let’s prove (2.124). We first begin
by proving (2.123) :
E[µ ML ] =
N
1 X
1
xn ] =
E[
· Nµ = µ
N n=1
N
Where we have taken advantage of the fact that xn is independently and
identically distributed (i.i.d).
Then we use the expression in (2.122) :
E[Σ ML ] =
N
1 X
E[ (xn − µ ML )(xn − µ ML )T ]
N n=1
=
N
1 X
E[(xn − µ ML )(xn − µ ML )T ]
N n=1
=
N
1 X
E[(xn − µ ML )(xn − µ ML )T ]
N n=1
=
N
1 X
E[xn xn T − 2µ ML xn T + µ ML µT
ML ]
N n=1
=
N
N
N
1 X
1 X
1 X
E[xn xn T ] − 2
E[µ ML xn T ] +
E[µ ML µT
ML ]
N n=1
N n=1
N n=1
By using (2.291), the first term will equal to :
first term =
1
· N(µµT + Σ) = µµT + Σ
N
The second term will equal to :
second term = −2
N
1 X
E[µ ML xn T ]
N n=1
= −2
N
N
1 X
1 X
E[ (
xm )xn T ]
N n=1 N m=1
= −2
N X
N
1 X
E[xm xn T ]
N 2 n=1 m=1
N X
N
1 X
(µµT + I nm Σ)
N 2 n=1 m=1
1
= −2 2 (N 2 µµT + N Σ)
N
1
= −2(µµT + Σ)
N
= −2
58
Similarly, the third term will equal to :
third term =
N
1 X
E[µ ML µT
ML ]
N n=1
=
N
N
N
1 X
1 X
1 X
E[(
x j) · (
x i )]
N n=1 N j=1
N i=1
=
N
N
N
X
X
1 X
E
[(
x
)
·
(
x i )]
j
N 3 n=1 j=1
i =1
N
1 X
(N 2 µµT + N Σ)
N 3 n=1
1
= µµT + Σ
N
=
Finally, we combine those three terms, which gives:
E[Σ ML ] =
N −1
Σ
N
Note: the same procedure from (2.59) to (2.62) can be carried out to prove
(2.291) and the only difference is that we need to introduce index m and n
to represent the samples. (2.291) is quite straightforward if we see it in this
way: If m = n, which means xn and xm are actually the same sample, (2.291)
will reduce to (2.262) (i.e. the correlation between different dimensions exists) and if m 6= n, which means xn and xm are different samples, also i.i.d,
then no correlation should exist, we can guess E[xn xm T ] = µµT in this case.
Problem 2.36 Solution
Let’s follow the hint. However, firstly we will find the sequential expression based on definition, which will make the latter process on finding coefficient a N −1 more easily. Suppose we have N observations in total, and then
we can write:
N)
σ2(
ML
=
=
=
N
1 X
N) 2
(xn − µ(ML
)
N n=1
#
"
−1
1 NX
N) 2
N) 2
(xn − µ(ML
) + (x N − µ(ML
)
N n=1
−1
N − 1 1 NX
1
N) 2
N) 2
(xn − µ(ML
) + (x N − µ(ML
)
N N − 1 n=1
N
N − 1 2( N −1)
1
N) 2
σ ML
+ (x N − µ(ML
)
N
N
h
i
1
N −1)
(N ) 2
2( N −1)
= σ2(
+
(x
−
µ
)
−
σ
N
ML
ML
ML
N
=
59
And then let us write the expression for σ ML .
(
)
¯
N
∂
1 X
¯
lnp(x
|
µ
,
σ
)
=0
¯
n
2
σ ML
N n=1
∂σ
By exchanging the summation and the derivative, and letting N → +∞,
we can obtain :
·
¸
N
1 X
∂
∂
lim
lnp(x
|
µ
,
σ
)
=
E
lnp(x
|
µ
,
σ
)
n
x
n
N →+∞ N n=1 ∂σ2
∂σ2
Comparing it with (2.127), we can obtain the sequential formula to estimate σ ML :
N)
σ2(
ML
N −1)
= σ2(
+ a N −1
ML
∂
N −1)
∂σ2(
ML
"
N −1)
+ a N −1 −
= σ2(
ML
N)
N −1)
lnp(x N |µ(ML
, σ(ML
)
1
N −1)
2σ2(
ML
+
N) 2
(x N − µ(ML
)
(∗)
#
N −1)
2σ4(
ML
N)
to represent the Nth estimation of σ2ML , i.e., the estiWhere we use σ2(
ML
mation of σ2ML after the Nth observation. What’s more, if we choose :
a N −1 =
N −1)
2σ4(
ML
N
Then we will obtain :
N −1)
N)
+
= σ2(
σ2(
ML
ML
i
1 h 2( N −1)
N) 2
)
+ (x N − µ(ML
−σ ML
N
We can see that the results are the same. An important thing should be
N)
noticed : In maximum likelihood, when estimating variance σ2(
, we will
ML
N)
N)
first estimate mean µ(ML
, and then we we will calculate variance σ2(
.
ML
In other words, they are decoupled. It is the same in sequential method.
For instance, if we want to estimate both mean and variance sequentially,
N −1)
after observing the Nth sample (i.e., x N ), firstly we can use µ(ML
together
N)
with (2.126) to estimate µ(ML
and then use the conclusion in this problem
N)
N)
N −1)
to obtain σ(ML
. That is why in (∗) we write lnp(x N |µ(ML
, σ(ML
) instead of
N −1) ( N −1)
lnp(x N |µ(ML
, σ ML ).
Problem 2.37 Solution (Wait for revising)
We follow the same procedure in Prob.2.36 to solve this problem. Firstly,
60
we can obtain the sequential formula based on definition.
N)
Σ(ML
N
1 X
N) T
N)
)
)(xn − µ(ML
(xn − µ(ML
N n=1
#
"
−1
1 NX
(N ) T
(N )
(N ) T
(N )
(xn − µ ML )(xn − µ ML ) + (x N − µ ML )(x N − µ ML )
N n=1
=
=
N − 1 ( N −1) 1
N) T
N)
)
)(x N − µ(ML
Σ ML + (x N − µ(ML
N
N
i
h
1
N −1)
N −1)
N) T
N)
+
= Σ(ML
) − Σ(ML
)(x N − µ(ML
(x N − µ(ML
N
=
If we use Robbins-Monro sequential estimation formula, i.e., (2.135), we
can obtain :
N)
Σ(ML
N −1)
+ a N −1
= Σ(ML
N −1)
+ a N −1
= Σ(ML
∂
N −1)
∂Σ(ML
∂
N −1)
∂Σ(ML
N −1)
N)
)
, Σ(ML
lnp(x N |µ(ML
N −1)
N)
)
, Σ(ML
lnp(x N |µ(ML
·
¸
1
1
N −1)
N −1) −1
N −1) T
N −1)
N −1) −1
N −1) −1
+ a N −1 − [Σ(ML
= Σ(ML
]
) [Σ(ML
)(xn − µ(ML
] + [Σ(ML
] (xn − µ(ML
2
2
Where we have taken advantage of the procedure we carried out in Prob.2.34
to calculate the derivative, and if we choose :
a N −1 =
2 2( N −1)
Σ
N ML
We can see that the equation above will be identical with our previous
conclusion based on definition.
Problem 2.38 Solution
It is straightforward. Based on (2.137), (2.138) and (2.139), we focus on
the exponential term of the posterior distribution p(µ| X ), which gives :
−
N
1 X
1
1
(xn − µ)2 − 2 (µ − µ0 )2 = − 2 (µ − µ N )2
2
2σ n=1
2σ 0
2σ N
We rewrite the left side regarding to µ.
quadratic term = −(
PN
linear term = (
N
1
+
) µ2
2
2σ
2σ20
µ0
n=1 x n
+ 2)µ
σ2
σ0
61
We also rewrite the right side regarding to µ, and hence we will obtain :
N
1
1
−( 2 +
) µ2 = − 2 µ2 , (
2
2σ
2σ 0
2σ N
Then we will obtain :
1
1
=
σ2N
σ20
µN
=
= (
1
N
σ2
n=1 x n = N · µ ML , we can write :
µ0
n=1 x n
+ 2)
σ2
σ0
N −1 N µ ML
µ0
) ·(
+ 2)
2
2
σ
σ
σ0
σ20 σ2
=
+
PN
+
σ20
µ0
µN
n=1 x n
+ 2)µ = 2 µ
2
σ
σ0
σN
PN
And with the prior knowledge that
σ2N · (
PN
·
2
N µ ML σ20 + µ0 σ2
σσ20
σ2 + N σ0
σ2
=
N σ20 + σ2
µ0 +
N σ20
N σ20 + σ2
µ ML
Problem 2.39 Solution
Let’s follow the hint.
1
σ2N
=
1
σ20
+
N
N −1
1
1
1
1
= 2 +
+ 2 = 2
+ 2
2
2
σ
σ
σ
σ
σ0
σ N −1
However, it is complicated to derive a sequential formula for µ N directly.
Based on (2.142), we see that the denominator in (2.141) can be eliminated if
we multiply 1/σ2N on both side of (2.141). Therefore we will derive a sequential formula for µ N /σ2N instead.
µN
σ2N
=
=
=
=
=
σ2 + N σ20
σ20 σ2
σ2 + N σ20
σ20 σ2
µ0
σ20
µ0
+
(
(
σ2
N σ20 + σ2
σ2
N σ20 + σ2
N)
N µ(ML
σ2
P N −1
=
µ0
σ20
µ0 +
µ0 +
PN
+
xn
xN
+ n=12
+ 2
2
σ
σ
σ0
µ N −1
xN
+ 2
σ
σ2N −1
N σ20
N σ20 + σ2
N σ20
N σ20 + σ2
n=1 x n
σ2
N)
µ(ML
)
N)
µ(ML
)
62
Another possible solution is also given in the problem. We solve it by
completing the square.
−
1
1
1
(x N − µ)2 − 2 (µ − µ N −1 )2 = − 2 (µ − µ N )2
2
2σ
2σ N −1
2σ N
By comparing the quadratic and linear term regarding to µ, we can obtain:
1
1
1
= 2 + 2
2
σ
σN
σ N −1
And :
µN
σ2N
=
µ N −1
xN
+ 2
2
σ
σ N −1
It is the same as previous result. Note: after obtaining the Nth observation, we will firstly use the sequential formula to calculate σ2N , and then µ N .
This is because the sequential formula for µ N is dependent on σ2N .
Problem 2.40 Solution
Based on Bayes Theorem, we can write :
p(µ| X ) ∝ p(X |µ)p(µ)
We focus on the exponential term on the right side and then rearrange it
regarding to µ.
"
#
N
X
1
1
T −1
right =
− (xn − µ) Σ (xn − µ) − (µ − µ0 )T Σ0 −1 (µ − µ0 )
2
2
n=1
"
#
N
X
1
1
T −1
=
− (xn − µ) Σ (xn − µ) − (µ − µ0 )T Σ0 −1 (µ − µ0 )
2
n=1 2
N
X
1
1
−1
= − µ (Σ0 −1 + N Σ−1 ) µ + µT (Σ−
xn ) + const
0 µ0 + Σ
2
n=1
Where ’const’ represents all the constant terms independent of µ. According to the quadratic term, we can obtain the posterior covariance matrix.
Σ−N1 = Σ0 −1 + N Σ−1
Then using the linear term, we can obtain :
1
−1
Σ−N1 µ N = (Σ−
0 µ0 + Σ
N
X
xn )
n=1
Finally we obtain posterior mean :
1
−1
µ N = (Σ0 −1 + N Σ−1 )−1 (Σ−
0 µ0 + Σ
N
X
n=1
xn )
63
Which can also be written as :
µ N = (Σ0 −1 + N Σ−1 )−1 (Σ0 −1 µ0 + Σ−1 N µ ML )
Problem 2.41 Solution
Let’s compute the integral of (2.146) over λ.
Z +∞
Z +∞
1 a a−1
ba
b λ
exp(− bλ) d λ =
λa−1 exp(− bλ) d λ
Γ(a)
Γ(a) 0
0
Z +∞
u
1
ba
( )a−1 exp(− u) du
=
Γ(a) 0
b
b
Z +∞
1
=
u a−1 exp(− u) du
Γ(a) 0
1
=
· Γ(a) = 1
Γ(a)
Where we first perform change of variable bλ = u, and then take advantage of the definition of gamma function:
Z +∞
u x−1 e−u du
Γ(x) =
0
Problem 2.42 Solution
We first calculate its mean.
Z +∞
1 a a−1
λ
b λ
exp(− bλ) d λ =
Γ
(a)
0
=
=
=
Z +∞
ba
λa exp(− bλ) d λ
Γ(a) 0
Z +∞
ba
u
1
( )a exp(− u) du
Γ(a) 0
b
b
Z +∞
1
u a exp(− u) du
Γ(a) · b 0
1
a
· Γ(a + 1) =
Γ(a) · b
b
Where we have taken advantage of the property Γ(a + 1) = aΓ(a). Then
we calculate E[λ2 ].
Z +∞
Z +∞
ba
2 1
a a−1
b λ
exp(− bλ) d λ =
λ
λa+1 exp(− bλ) d λ
Γ(a)
Γ(a) 0
0
Z +∞
ba
1
u
=
( )a+1 exp(− u) du
Γ(a) 0
b
b
Z +∞
1
a+1
=
u
exp(− u) du
Γ(a) · b2 0
a(a + 1)
1
· Γ(a + 2) =
=
Γ(a) · b2
b2
64
Therefore, according to var[λ] = E[λ2 ] − E[λ]2 , we can obtain :
var[λ] = E[λ2 ] − E[λ]2 =
a
a(a + 1)
a 2
−( ) = 2
2
b
b
b
For the mode of a gamma distribution, we need to find where the maximum of the PDF occurs, and hence we will calculate the derivative of the
gamma distribution with respect to λ.
·
¸
d
1 a a−1
1 a a−2
b λ
exp(− bλ)
= [(a − 1) − bλ]
b λ
exp(− bλ)
d λ Γ(a)
Γ(a)
It is obvious that Gam(λ|a, b) has its maximum at λ = (a − 1)/b. In other
words, the gamma distribution Gam(λ|a, b) has mode (a − 1)/b.
Problem 2.43 Solution
Let’s firstly calculate the following integral.
Z +∞
Z +∞
| x| q
xq
exp(− 2 ) dx = 2
exp(− 2 ) dx
2σ
2σ
−∞
−∞
1
Z +∞
(2σ2 ) q 1q −1
exp(− u)
= 2
u
du
q
0
1 Z
1
(2σ2 ) q +∞
−1
exp(− u)u q dx
= 2
q
0
1
(2σ2 ) q 1
= 2
Γ( )
q
q
And then it is obvious that (2.293) is normalized. Next, we consider about
the log likelihood function. Since ² = t − y(x, w) and ² ∼ p(²|σ2 , q), we can
write:
ln p(t| X , w, σ2 ) =
N
X
¡
¢
ln p y(xn , w) − t n |σ2 , q
n=1
= −
·
¸
N
q
1 X
q
+
N
·
ln
|
y(x
,
w)
−
t
|
n
n
2σ2 n=1
2(2σ2 )1/ q Γ(1/q)
= −
N
1 X
N
| y(xn , w) − t n | q − ln(2σ2 ) + const
2
q
2σ n=1
Problem 2.44 Solution
Here we use a simple method to solve this problem by taking advantage
of (2.152) and (2.153). By writing the prior distribution in the form of (2.153),
i.e., p(µ, λ|β, c, d), we can easily obtain the posterior distribution.
p(µ, λ| X ) ∝
p(X |µ, λ) · p(µ, λ)
"
#
·
¸ N +β
N
N x2
X
X
λµ2
n
1/2
∝
λ exp(−
)
exp (c +
xn )λµ − (d +
)λ
2
n=1
n=1 2
65
Therefore, we can see that the posterior distribution has parameters: β0 =
P
P
x2
β + N, c0 = c + nN=1 xn , d 0 = d + nN=1 2n . And since the prior distribution is
actually the product of a Gaussian distribution and a Gamma distribution:
£
¤
p(µ, λ|µ0 , β, a, b) = N µ|µ0 , (βλ)−1 Gam(λ|a, b)
Where µ0 = c/β, a = 1 + β/2, b = d − c2 /2β. Hence the posterior distribution can also be written as the product of a Gaussian distribution and a
Gamma distribution.
£
¤
p(µ, λ| X ) = N µ|µ00 , (β0 λ)−1 Gam(λ|a0 , b0 )
Where we have defined:
N
X
µ00 = c0 /β0 = (c +
±
xn ) (N + β)
n=1
±
a0 = 1 + β0 /2 = 1 + (N + β) 2
2
b0 = d 0 − c0 /2β0 = d +
N x2
X
n
n=1 2
− (c +
N
X
±
xn )2 (2(β + N))
n=1
Problem 2.45 Solution
Let’s begin by writing down the dependency of the prior distribution W (Λ|W, v)
and the likelihood function p(X |µ, Λ) on Λ.
p(X |µ, Λ) ∝ |Λ| N /2 exp
And if we denote
S=
N
£X
¤
1
− (xn − µ)T Λ(xn − µ)
n=1 2
N
1 X
(xn − µ)(xn − µ)T
N n=1
Then we can rewrite the equation above as:
£ 1
¤
p(X |µ, Λ) ∝ |Λ| N /2 exp − Tr(S Λ)
2
Just as what we have done in Prob.2.34, and comparing this problem with
Prob.2.34, one important thing should be noticed: since S and Λ are both
¡
¢
symmetric, we have: Tr(S Λ) = Tr (S Λ)T = Tr(ΛT S T ) = Tr(ΛS). And we
can also write down the prior distribution as:
£ 1
¤
W (Λ|W, v) ∝ |Λ|(v−D −1)/2 exp − Tr(W −1 Λ)
2
Therefore, the posterior distribution can be obtained:
p(Λ| X ,W, v) ∝
p(X |µ, Λ) · W (Λ|W, v)
© 1 £
¤ª
∝ |Λ|( N +v−D −1)/2 exp − Tr (W −1 + S)Λ
2
66
Therefore, p(Λ| X ,W, v) is also a Wishart distribution, with parameters:
vN = N + v
W N = (W −1 + S)−1
Problem 2.46 Solution
It is quite straightforward.
Z ∞
p(x|µ, a, b) =
N (x|µ, τ−1 )Gam(τ|a, b) d τ
0
n τ
o
b a exp(− bτ)τa−1 τ 1/2
( ) exp − (x − µ)2 d τ
Γ(a)
2π
2
0
Z ∞
n
o
a
1
τ
b
( )1/2
τa−1/2 exp − bτ − (x − µ)2 d τ
Γ(a) 2π
2
0
Z ∞
=
=
And if we make change of variable: z = τ[b + (x − µ)2 /2], the integral above
can be written as:
Z
n
o
b a 1 1/2 ∞ a−1/2
τ
p(x|µ, a, b) =
τ
exp − bτ − (x − µ)2 d τ
( )
Γ(a) 2π
2
0
¸a−1/2
Z ∞·
a
1
b
z
1
( )1/2
dz
=
exp {− z}
2 /2
Γ(a) 2π
b
+
(x
−
µ
)
b
+
(x
− µ)2 /2
0
¸a+1/2 Z ∞
·
b a 1 1/2
1
z a−1/2 exp {− z} dz
=
( )
Γ(a) 2π
b + (x − µ)2 /2
0
¸−a−1/2
·
b a 1 1/2
(x − µ)2
=
Γ(a + 1/2)
( )
b+
Γ(a) 2π
2
And if we substitute a = v/2 and b = v/2λ, we will obtain (2.159).
Problem 2.47 Solution
We focus on the dependency of (2.159) on x.
·
¸−v/2−1/2
λ(x − µ)2
St(x|µ, λ, v) ∝
1+
v
·
¸
−v − 1
λ(x − µ)2
∝ exp
ln(1 +
)
2
v
·
¸
−v − 1 λ(x − µ)2
−2
∝ exp
(
+ O(v ))
2
v
·
¸
λ(x − µ)2
≈ exp −
(v → ∞)
2
Where we have used Taylor Expansion: ln(1 + ²) = ² + O(²2 ). We see that
this, up to an overall constant, is a Gaussian distribution with mean µ and
precision λ.
67
Problem 2.48 Solution
The same steps in Prob.2.46 can be used here.
Z +∞
¯
¯
¯v v
¯
St(x µ, Λ, v) =
N (x ¯ µ, (ηΛ)−1 ) · Gam(η ¯ , ) d η
2 2
0
¾
½
Z +∞
1
1
1
vη
v v/2
1/2
T
=
|ηΛ| exp − (x − µ) (ηΛ)(x − µ) −
( ) ηv/2−1 d η
D
/2
2
2 Γ(v/2) 2
(2π)
0
½
¾
v/2
1/2 Z +∞
(v/2) |Λ|
1
vη D /2+v/2−1
T
=
exp
−
(x
−
µ
)
(
η
Λ
)(x
−
µ
)
−
η
dη
2
2
(2π)D /2 Γ(v/2) 0
Where we have taken advantage of the property: |ηΛ| = ηD |Λ|, and if we
denote:
η
∆2 = (x − µ)T Λ(x − µ) and z = (∆2 + v)
2
The expression above can be reduced to :
Z
¯
2z D /2+v/2−1
(v/2)v/2 |Λ|1/2 +∞
2
exp(
−
z)(
St(x ¯ µ, Λ, v) =
)
· 2
dz
2
(2π)D /2 Γ(v/2) 0
∆ +v
∆ +v
D /2+v/2 Z +∞
(v/2)v/2 |Λ|1/2
2
exp(− z) · z D /2+v/2−1 dz
)
=
(
(2π)D /2 Γ(v/2) ∆2 + v
0
D /2+v/2
(v/2)v/2 |Λ|1/2
2
=
(
)
Γ(D/2 + v/2)
(2π)D /2 Γ(v/2) ∆2 + v
And if we rearrange the expression above, we will obtain (2.162) just as
required.
Problem 2.49 Solution
Firstly, we notice that if and only if x = µ, ∆2 equals to 0, so that St(x|µ, Λ, v)
achieves its maximum. In other words, the mode of St(x|µ, Λ, v) is µ. Then
we consider about its mean E[x].
Z
St(x|µ, Λ, v) · x dx
E[x] =
x∈RD
·Z +∞
¸
Z
¯
¯v v
−1
¯
¯
=
N (x µ, (ηΛ) ) · Gam(η , ) d η x dx
2 2
x∈RD
0
Z
Z +∞
¯
¯
v v
=
xN (x ¯ µ, (ηΛ)−1 ) · Gam(η ¯ , ) d η dx
D
2 2
x∈R
0
¸
Z +∞ ·Z
¯
¯v v
−1
¯
¯
=
xN (x µ, (ηΛ) ) dx · Gam(η , ) d η
2 2
0
x∈RD
Z +∞ h
¯v v i
=
µ · Gam(η ¯ , ) d η
2 2
0
Z +∞
¯v v
= µ
Gam(η ¯ , ) d η = µ
2 2
0
Where we have taken the following property:
Z
¯
xN (x ¯ µ, (ηΛ)−1 ) dx = E[x] = µ
x∈RD
68
Then we calculate E[xxT ]. The steps above can also be used here.
Z
T
E[xx ] =
St(x|µ, Λ, v) · xxT dx
x∈RD
·Z +∞
¸
Z
¯
¯v v
−1
T
¯
¯
N (x µ, (ηΛ) ) · Gam(η , ) d η xx dx
=
2 2
0
x∈RD
Z
Z +∞
¯
¯v v
xxT N (x ¯ µ, (ηΛ)−1 ) · Gam(η ¯ , ) d η dx
=
2 2
x∈RD 0
¸
Z +∞ ·Z
¯
¯v v
T
−1
¯
¯
xx N (x µ, (ηΛ) ) dx · Gam(η , ) d η
=
2 2
x∈RD
0
Z +∞ h
i
¯
v
v
=
E[µµT ] · Gam(η ¯ , ) d η
2 2
0
Z +∞ h
i
¯v v
=
µµT + (ηΛ)−1 Gam(η ¯ , ) d η
2 2
0
Z +∞
¯
v
v
(ηΛ)−1 · Gam(η ¯ , ) d η
= µµT +
2 2
0
Z +∞
1
v
v v/2 v/2−1
−1
T
(ηΛ) ·
exp(− η) d η
= µµ +
( ) η
Γ
(v/2)
2
2
0
Z
v
1
v v/2 +∞ v/2−2
T
−1
η
exp(− η) d η
= µµ + Λ
( )
Γ(v/2) 2
2
0
vη
If we denote: z = 2 , the equation above can be reduced to :
T
E[xx ] =
=
=
=
=
Z
2
1
v v/2 +∞ 2z v/2−2
µµ + Λ
( )
exp(− z) dz
( )
Γ(v/2) 2
v
v
0
Z +∞
1
v
µµT + Λ−1
z v/2−2 exp(− z) dz
·
Γ(v/2) 2 0
Γ(v/2 − 1) v
µµT + Λ−1
·
Γ(v/2)
2
1 v
µµT + Λ−1
v/2 − 1 2
v
µµT +
Λ−1
v−2
T
−1
Where we have taken advantage of the property: Γ(x + 1) = xΓ(x), and
£
¤
since we have cov[x] = E (x − E[x])(x − E[x])T , together with E[x] = µ, we
can obtain:
v
cov[x] =
Λ−1
v−2
Problem 2.50 Solution
69
The same steps in Prob.2.47 can be used here.
¸−D /2−v/2
·
∆2
St(x|µ, Λ, v) ∝
1+
v
·
¸
∆2
∝ exp (−D/2 − v/2) · ln(1 +
)
v
¸
·
2
D+v ∆
−2
·(
+ O(v ))
∝ exp −
2
v
∆2
≈ exp(−
) (v → ∞)
2
Where we have used Taylor Expansion: ln(1 + ²) = ² + O(²2 ). And since
∆2 = (x−µ)T Λ(x−µ), we see that this, up to an overall constant, is a Gaussian
distribution with mean µ and precision Λ.
Problem 2.51 Solution
We first prove (2.177). Since we have exp(i A)· exp(− i A) = 1, and exp(i A) =
cosA + isinA. We can obtain:
(cosA + isinA) · (cosA − isinA) = 1
Which gives cos2 A + sin2 A = 1. And then we prove (2.178) using the hint.
cos(A − B) = ℜ[exp(i(A − B))]
±
= ℜ[exp(i A) exp(iB)]
cosA + isinA
= ℜ[
]
cosB + isinB
(cosA + isinA)(cosB − isinB)
= ℜ[
]
(cosB + isinB)(cosB − isinB)
= ℜ[(cosA + isinA)(cosB − isinB)]
=
cosAcosB + sinAsinB
It is quite similar for (2.183).
sin(A − B) = ℑ[exp(i(A − B))]
= ℑ[(cosA + isinA)(cosB − isinB)]
=
sinAcosB − cosAsinB
Problem 2.52 Solution
Let’s follow the hint. We first derive an approximation for exp[mcos(θ −
70
θ0 )].
¸¾
·
(θ − θ0 )2
4
+ O((θ − θ0 ) )
exp m 1 −
2
½
¾
(θ − θ0 )2
exp m − m
− mO((θ − θ0 )4 )
2
¾
½
©
ª
(θ − θ0 )2
· exp − mO((θ − θ0 )4 )
exp(m) · exp − m
2
½
exp { mcos(θ − θ0 )} =
=
=
It is same for exp(mcosθ ) :
exp { mcosθ } =
exp(m) · exp(− m
θ2
2
©
ª
) · exp − mO(θ 4 )
Now we rearrange (2.179):
p(θ |θ0 , m) =
=
=
=
1
exp { mcos(θ − θ0 )}
2π I 0 (m)
1
exp { mcos(θ − θ0 )}
R 2π
0 exp { mcosθ } d θ
n
o
©
ª
2
exp(m) · exp − m (θ−2θ0 ) · exp − mO((θ − θ0 )4 )
©
ª
R 2π
θ2
4
0 exp(m) · exp(− m 2 ) · exp − mO(θ ) d θ
½
¾
(θ − θ 0 )2
1
exp − m
R 2π
2
2
exp(− m θ ) d θ
0
2
Where we have taken advantage of the following fact:
ª
©
ª
©
(when m → ∞)
exp − mO((θ − θ0 )4 ) ≈ exp − mO(θ 4 )
Therefore, it is straightforward that when m → ∞, (2.179) reduces to a
Gaussian Distribution with mean θ0 and precision m.
Problem 2.53 Solution
Let’s rearrange (2.182) according to (2.183).
N
X
sin(θ − θ0 ) =
n=1
N
X
(sinθn cosθ0 − cosθn sinθ0 )
n=1
=
cosθ0
N
X
sinθn − sinθ0
n=1
N
X
cosθn
n=1
Where we have used (2.183), and then together with (2.182), we can obtain :
N
N
X
X
cosθ0
sinθn − sinθ0
cosθn = 0
n=1
n=1
71
Which gives:
θ0ML = tan−1
¾
½P
n sinθ n
P
n cosθ n
Problem 2.54 Solution
We calculate the first and second derivative of (2.179) with respect to θ .
p(θ |θ0 , m)0 =
p(θ |θ0 , m)00 =
©
ª
1
[− msin(θ − θ0 )] exp mcos(θ − θ0 )
2π I 0 (m)
£
¤
©
ª
1
− mcos(θ − θ0 ) + (− msin(θ − θ0 ))2 exp mcos(θ − θ0 )
2π I 0 (m)
If we let p(θ |θ0 , m)0 equals to 0, we will obtain its root:
θ = θ0 + k π
(k ∈ Z)
When k ≡ 0 (mod2), i.e. θ ≡ θ0 (mod2π), we have:
p(θ |θ0 , m)00 =
− m exp(m)
<0
2π I 0 (m)
Therefore, when θ = θ0 , (2.179) obtains its maximum. And when k ≡
1 (mod2), i.e. θ ≡ θ0 + π (mod2π), we have:
p(θ |θ0 , m)00 =
m exp(− m)
>0
2π I 0 (m)
Therefore, when θ = θ0 + π (mod2π), (2.179) obtains its minimum.
Problem 2.55 Solution
According to (2.185), we have :
A(m ML ) =
N
1 X
cos(θn − θ0ML )
N n=1
By using (2.178), we can write :
N
1 X
cos(θn − θ0ML )
N n=1
´
N ³
1 X
=
cosθn cosθ0ML + sinθn sinθ0ML
N n=1
!
Ã
!
Ã
N
N
X
1
1 X
=
cosθ N cosθ0ML +
sinθ N sinθ0ML
N n=1
N n=1
A(m ML ) =
By using (2.168), we can further derive:
Ã
!
Ã
!
N
N
1 X
1 X
ML
A(m ML ) =
cosθ N cosθ0 +
sinθ N sinθ0ML
N n=1
N n=1
=
r̄cosθ̄ · cosθ0ML + r̄sinθ̄ · sinθ0ML
=
r̄cos(θ̄ − θ0ML )
72
And then by using (2.169) and (2.184), it is obvious that θ̄ = θ0ML , and
hence A(m ML ) = r̄.
Problem 2.56 Solution
Recall that the distributions belonging to the exponential family have the
form:
p(x|η) = h(x) g(η) exp(ηT u(x))
And according to (2.13), the beta distribution can be written as:
Beta(x|a, b) =
=
=
Γ(a + b) a−1
x (1 − x)b−1
Γ(a)Γ(b)
Γ(a + b)
exp [(a − 1)lnx + (b − 1)ln(1 − x)]
Γ(a)Γ(b)
Γ(a + b) exp [alnx + bln(1 − x)]
Γ(a)Γ(b)
x(1 − x)
Comparing it with the standard form of exponential family, we can obtain:

η = [a, b]T




 u(x) = [lnx, ln(1 − x)]T
±£
¤


 g(η) = Γ±(η 1 + η 2 ) Γ(η 1 )Γ(η 2 )


h(x) = 1 (x(1 − x))
Where η 1 means the first element of η, i.e. η 1 = a − 1, and η 2 means the
second element of η, i.e. η 2 = b − 1. According to (2.146), Gamma distribution
can be written as:
Gam(x|a, b) =
1 a a−1
b x
exp(− bx)
Γ(a)
Comparing it with the standard form of exponential family, we can obtain:

T


η = [a, b]


 u(x) = [0, − x]
.
η1

g(
η
)
=
η
Γ(η 1 )

2



 h(x) = xη1 −1
According to (2.179), the von Mises distribution can be written as:
p(x|θ0 , m) =
=
1
exp(mcos(x − θ0 ))
2π I 0 (m)
1
exp [m(cosxcosθ0 + sinxsinθ0 )]
2π I 0 (m)
73
Comparing it with the standard form of exponential family, we can obtain:


η = [mcosθ0 , msinθ0 ]T




 u(x) = [cosx, sinx]
q
.

g(
η
)
=
1
2
π
I
(
η21 + η22 )

0




h(x) = 1
Note : a given distribution can be written into the exponential family in
several ways with different natural parameters.
Problem 2.57 Solution
Recall that the distributions belonging to the exponential family have the
form:
p(x|η) = h(x) g(η) exp(ηT u(x))
And the multivariate Gaussian Distribution has the form:
¾
½
1
1
1
T −1
N (x|µ, Σ) =
Σ
(x
−
µ
)
exp
−
(x
−
µ
)
2
(2π)D /2 |Σ|1 /2
We expand the exponential term with respect to µ.
½
¾
1
1
1 T −1
T −1
−1
N (x|µ, Σ) =
exp
−
(x
Σ
x
−
2
µ
Σ
x
+
µ
Σ
µ
)
2
(2π)D /2 |Σ|1/2
½
¾
½
¾
1
1 T −1
1
1
T −1
−1
exp
−
=
x
Σ
x
+
µ
Σ
x
exp
−
µ
Σ
µ
)
2
2
(2π)D /2 |Σ|1/2
Comparing it with the standard form of exponential family, we can obtain:

η = [Σ−1 µ, − 12 vec(Σ−1 ) ]T




 u(x) = [x, vec(xxT ) ]

g(η) = exp( 41 η 1 T η 2 −1 η 1 ) + | − 2η 2 |1/2




h(x) = (2π)−D /2
Where we have used η1 to denote the first element of η, and η2 to denote
the second element of η. And we also take advantage of the vectorizing operator, i.e.vec(·). The vectorization of a matrix is a linear transformation which
converts the matrix into a column vector. This can be viewed in an example :
·
¸
a b
A=
=> vec(A) = [a, c, b, d]T
c d
Note: By introducing vectorizing operator, we actually have vec(Σ−1 ) ·
vec(xxT ) = xT Σ−1 x
Problem 2.58 Solution
74
Based on (2.226), we have already obtained:
Z
−∇ ln g(η) = g(η) h(x) exp{ηT u(x)} u(x)dx
Then we calculate the derivative of both sides of the equation above with
respect to η using the Chain rule of Calculus:
Z
Z
−∇∇ ln g(η) = ∇ g(η) h(x) exp{ηT u(x)} u(x)T dx + g(η) h(x) exp{ηT u(x)} u(x)u(x)T dx
One thing needs to be addressed here: please pay attention to the transpose operation, and −∇∇ ln g(η) should be a matrix. Notice the relationship
∇ ln g(η) = ∇ g(η)/g(η), the first term on the right hand side of the above equation can be simplified as:
Z
(first term on the right) = ∇ ln g(η) · g(η) h(x) exp{ηT u(x)} u(x)T dx
= ∇ ln g(η) · E[u(x)T ]
= −E[u(x)] · E[u(x)T ]
Based on the definition, the second term on the right hand side is E[u(x)u(x)T ].
Therefore, combining these two terms, we obtain:
−∇∇ ln g(η) = −E[u(x)] · E[u(x)T ] + E[u(x)u(x)T ] = cov[u(x)]
Problem 2.59 Solution
It is straightforward.
Z
1 x
f ( ) dx
σ σ
Z
1
f (u)σ du
=
σ
Z
p(x|σ) dx =
Z
f (u) du = 1
=
Where we have denoted u = x/σ.
Problem 2.60 Solution
Firstly, we write down the log likelihood function.
N
X
n=1
lnp(xn ) =
M
X
n i ln(h i )
i =1
Some details should be explained here. If xn falls into region ∆ i , then
p(xn ) will equal to h i , and since we have already been given that among
75
all the N observations, there are n i samples fall into region ∆ i , we can easily
write down the likelihood function just as the equation above, and note we use
M to denote the number of different regions. Therefore, an implicit equation
should hold:
M
X
ni = N
i =1
We now need to take account of the constraint that p(x) must integrate
P
to unity, which can be written as M
j =1 h j ∆ j = 1. We introduce a Lagrange
multiplier to the expression, and then we need to minimize:
M
X
M
X
n i ln(h i ) + λ( h j ∆ j − 1)
i =1
j
We calculate its derivative with respect to h i and let it equal to 0.
ni
+ λ∆ i = 0
hi
Multiplying both sides by h i , performing summation over i and then using the constraint, we can obtain:
N +λ=0
In other words, λ = − N. Then we substitute the result into the likelihood
function, which gives:
ni 1
hi =
N ∆i
Problem 2.61 Solution
It is straightforward. In K nearest neighbours (KNN), when we want to
estimate probability density at a point x i , we will consider a small sphere
centered on x i and then allow the radius to grow until it contains K data
points, and then p(x i ) will equal to K/(NVi ), where N is total observations
and Vi is the volume of the sphere centered on x i . We can assume that Vi is
small enough that p(x i ) is roughly constant in it. In this way, We can write
down the integral:
Z
p(x) dx ≈
N
X
i =1
p(x i ) · Vi =
N K
X
· Vi = K 6= 1
i =1 NVi
We also see that if we use "1NN" (K = 1), the probability density will be
well normalized. Note that if and only if the volume of all the spheres are
small enough and N is large enough, the equation above will hold. Fortunately, these two conditions can be satisfied in KNN.
76
0.3
Probability Distribution
Problem 3.1 Solution
Based on (3.6), we can write :
2σ(2a) − 1 =
2
1 − exp(−2a)
exp(a) − exp(−a)
−1=
=
1 + exp(−2a)
1 + exp(−2a)
exp(a) + exp(−a)
Which is exactly tanh(a). Then we will find the relation between µ i , w i
in (3.101) and (3.102). Let’s start from (3.101).
y(x, w) =
w0 +
M
X
w j σ(
x −µj
s
j =1
=
w0 +
M
X
x−µ
wj
tanh( 2s j ) + 1
2
j =1
=
w0 +
)
M
M w
X
x −µj
1X
j
wj +
tanh(
)
2 j=1
2s
j =1 2
Hence the relation is given by :
µ0 = w0 +
M
1X
wj
2 j=1
and µ j =
wj
2
Note: there is a typo in (3.102), the denominator should be 2s instead of
s, or alternatively you can view it as a new s0 , which equals to 2s.
Problem 3.2 Solution
To begin with, if we denote v? = (ΦT Φ)−1 ΦT v. Then we have:
Φ(ΦT Φ)−1 ΦT v = Φv?
(1)
By definition, Φv? is in the column space of Φ. In other words, we have prove
Φ(ΦT Φ)−1 ΦT can project a vector v into the column space of Φ. Next, we are
required to prove the ‘residue’ of the projection shown in Fig. 3.2 of the main
text (i.e., y − t is orthogonal to the column space of Φ). Since we have:
(y − t)T Φ = (Φw ML − t)T
= (Φ(ΦT Φ)−1 ΦT t − t)T Φ
= tT (Φ(ΦT Φ)−1 ΦT − I)T Φ
= tT (Φ(ΦT Φ)−1 ΦT − I)Φ
= tT (Φ − Φ)
=0
(2)
77
which means (y − t) is in the left null space of Φ, and it is also orthogonal to
the column space of Φ.
Problem 3.3 Solution
Let’s calculate the derivative of (3.104) with respect to w.
∇E D (w) =
N
X
n
o
r n t n − wT Φ(xn ) Φ(xn )T
n=1
We set the derivative equal to 0.
0=
N
X
Ã
r n t n Φ(xn )T − wT
n=1
If we denote
p
r n Φ(xn )Φ(xn )T
n=1
r n φ(xn ) = φ0 (xn ) and
0=
!
N
X
N
X
p
Ã
t0n Φ0 (xn )T − wT
n=1
r n t n = t0n , we can obtain:
N
X
!
0
0
Φ (xn )Φ (xn )
T
n=1
Taking advantage of (3.11) – (3.17), we can derive a similar result, i.e.
w ML = (ΦT Φ)−1 ΦT t. But here, we define t as:
t=
£p
r 1 t1 ,
p
¤T
p
r 2 t 2 , ... , r N t N
p
We also define Φ as a N × M matrix, with element Φ(i, j) = r i φ j (x i ).
The interpretation is two folds: (1) Examining Eq (3.10)-(3.12), we see that if
we substitute β−1 by r n ·β−1 in the summation term, Eq (3.12) will become the
expression in exercise 3.3. (2) r n can also be viewed as the effective number
of observation of (xn , t n ). Alternatively speaking, you can treat (xn , t n ) as
repeatedly occurring r n times.
Problem 3.4 Solution
Firstly, we rearrange E D (w).
E D (w) =
=
=
=
(
)2
N
D
X
£
¤
1 X
w0 +
w i (x i + ² i ) − t n
2 n=1
i =1
(
)2
N
D
D
X
X
¡
¢
1 X
w0 +
wi xi − t n +
wi ²i
2 n=1
i =1
i =1
(
)2
N
D
X
1 X
y(xn , w) − t n +
wi ²i
2 n=1
i =1
)
(
N
D
D
¢¡
¢
¡
¢2 ¡ X
¢2
¡X
1 X
y(xn , w) − t n +
wi ²i + 2
w i ² i y(xn , w) − t n
2 n=1
i =1
i =1
78
Where we have used y(xn , w) to denote the output of the linear model
when input variable is xn , without noise added. For the second term in the
equation above, we can obtain :
E² [(
D
X
w i ² i )2 ] = E² [
D X
D
X
wi w j ²i ² j ] =
w i w j E² [² i ² j ] = σ2
i =1 j =1
i =1 j =1
i =1
D X
D
X
D X
D
X
wi w j δi j
i =1 j =1
Which gives
E² [(
D
X
w i ² i )2 ] = σ2
i =1
D
X
w2i
i =1
For the third term, we can obtain:
D
D
X
¡X
¢¡
¢
¡
¢
w i ² i y(xn , w) − t n ] = 2 y(xn , w) − t n E² [ w i ² i ]
E² [2
i =1
i =1
¡
= 2 y(xn , w) − t n
D
¢X
E² [w i ² i ]
i =1
= 0
Therefore, if we calculate the expectation of E D (w) with respect to ², we
can obtain:
N ¡
D
¢2 σ2 X
1 X
E² [E D (w)] =
y(xn , w) − t n +
w2
2 n=1
2 i=1 i
Problem 3.5 Solution
We can firstly rewrite the constraint (3.30) as :
Ã
!
M
1 X
q
|w j | − η ≤ 0
2 j=1
Where we deliberately introduce scaling factor 1/2 for convenience.Then
it is straightforward to obtain the Lagrange function.
Ã
!
o2 λ X
N n
M
1 X
T
q
L(w, λ) =
t n − w φ(xn ) +
|w j | − η
2 n=1
2 j=1
It is obvious that L(w, λ) and (3.29) has the same dependence on w. Meanwhile, if we denote the optimal w that can minimize L(w, λ) as w? (λ), we can
see that
M
X
η=
|w?j | q
j =1
Problem 3.6 Solution
Firstly, we write down the log likelihood function.
lnp(T | X ,W, β) = −
N £
¤T
£
¤
N
1 X
ln|Σ| −
t n − W T φ(xn ) Σ−1 t n − W T φ(xn )
2
2 n=1
79
Where we have already omitted the constant term. We set the derivative
of the equation above with respect to W equals to zero.
0=−
N
X
¤
Σ−1 [t n − W T φ(xn ) φ(xn )T
n=1
Therefore, we can obtain similar result for W as (3.15). For Σ, comparing
with (2.118) – (2.124), we can easily write down a similar result :
Σ=
N
¤
¤T
1 X
T
[t n − W T
ML φ(x n ) [t n − W ML φ(x n )
N n=1
We can see that the solutions for W and Σ are also decoupled.
Problem 3.7 Solution
Let’s begin by writing down the prior distribution p(w) and likelihood
function p(t| X , w, β).
p(w) = N (w| m0 , S 0 ) ,
p(t| X , w, β) =
N
Y
N (t n |wT φ(xn ), β−1 )
n=1
Since the posterior PDF equals to the product of the prior PDF and likelihood function, up to a normalized constant. We mainly focus on the exponential term of the product.
exponential term
=
=
=
N n
β X
1
1
− (w − m0 )T S −
0 (w − m0 )
2
o 1
N n
β X
1
−
t2n − 2t n wT φ(xn ) + wT φ(xn )φ(xn )T w − (w − m0 )T S −
0 (w − m0 )
2 n=1
2
"
#
N
1 T X
T
−1
− w
βφ(xn )φ(xn ) + S 0 w
2
n=1
"
#
N
X
1
1
− −2m0T S −
2β t n φ(xn )T w
0 −
2
n=1
−
2 n=1
t n − wT φ(xn )
o2
+ const
Hence, by comparing the quadratic term with standard Gaussian Distri1
−1
T
bution, we can obtain: S −
N = S 0 + βΦ Φ. And then comparing the linear
term, we can obtain :
1
−2m N T S N −1 = −2m0T S −
0 −
N
X
2β t n φ(xn )T
n=1
If we multiply −0.5 on both sides, and then transpose both sides, we can
easily see that m N = S N (S 0 −1 m 0 + βΦT t)
80
Problem 3.8 Solution
Firstly, we write down the prior :
p(w) = N (m N , S N )
Where m N , S N are given by (3.50) and (3.51). And if now we observe
another sample (X N +1 , t N +1 ), we can write down the likelihood function :
p(t N +1 | x N +1 , w) = N (t N +1 | y(x N +1 , w), β−1 )
Since the posterior equals to the production of likelihood function and the
prior, up to a constant, we focus on the exponential term.
exponential term
=
=
1
T
2
(w − m N )T S −
N (w − m N ) + β(t N +1 − w φ(x N +1 ))
£
¤
wT S N −1 + β φ(x N +1 ) φ(x N +1 )T w
£ 1
¤
−2wT S −
N m N + β φ(x N +1 ) t N +1
+const
Therefore, after observing (X N +1 , t N +1 ), we have p(w) = N (m N +1 , S N +1 ),
where we have defined:
1
−1
T
S−
N +1 = S N + β φ(x N +1 ) φ(x N +1 )
And
¡ 1
¢
m N +1 = S N +1 S −
N m N + β φ(x N +1 ) t N +1
Problem 3.9 Solution
We know that the prior p(w) can be written as:
p(w) = N (m N , S N )
And the likelihood function p(t N +1 | x N +1 , w) can be written as:
p(t N +1 | x N +1 , w) = N (t N +1 | y(x N +1 , w), β−1 )
According to the fact that y(x N +1 , w) = wT φ(x N +1 ) = φ(x N +1 )T w, the
likelihood can be further written as:
p(t N +1 | x N +1 , w) = N (t N +1 |(φ(x N +1 )T w, β−1 )
Then we take advantage of (2.113), (2.114) and (2.116), which gives:
©
ª
p(w| x N +1 , t N +1 ) = N (Σ φ(x N +1 )β t N +1 + S N −1 m N , Σ)
Where Σ = (S N −1 + φ(x N +1 )βφ(x N +1 )T )−1 , and we can see that the result
is exactly the same as the one we obtained in the previous problem.
81
Problem 3.10 Solution
We have already known:
p(t|w, β) = N (t| y(x, w), β−1 )
And
p(w|t, α, β) = N (w| m N , S N )
Where m N , S N are given by (3.53) and (3.54). As what we do in previous
problem, we can rewrite p(t|w, β) as:
p(t|w, β) = N (t|φ(x)T w, β−1 )
And then we take advantage of (2.113), (2.114) and (2.115), we can obtain:
p(t|t, α, β) = N (φ(x)T m N , β−1 + φ(x)T S N φ(x))
Which is exactly the same as (3.58), if we notice that
φ(x)T m N = m N T φ(x)
Problem 3.11 Solution
We need to use the result obtained in Prob.3.8. In Prob.3.8, we have de1
rived a formula for S −
:
N +1
1
−1
T
S−
N +1 = S N + β φ(x N +1 ) φ(x N +1 )
And then using (3.110), we can obtain :
£
¤−1
S N +1 = S N −1 + β φ(x N +1 ) φ(x N +1 )T
p
p
£
¤−1
= S N −1 + βφ(x N +1 ) βφ(x N +1 )T
p
p
S N ( βφ(x N +1 ))( βφ(x N +1 )T )S N
= SN −
p
p
1 + ( βφ(x N +1 )T )S N ( βφ(x N +1 ))
=
SN −
βS N φ(x N +1 )φ(x N +1 )T S N
1 + βφ(x N +1 )T S N φ(x N +1 )
Now we calculate σ2N (x) − σ2N +1 (x) according to (3.59).
σ2N (x) − σ2N +1 (x)
= φ(x)T (S N − S N +1 )φ(x)
= φ(x)T
=
=
βS N φ(x N +1 )φ(x N +1 )T S N
1 + βφ(x N +1 )T S N φ(x N +1 )
φ(x)
φ(x)T S N φ(x N +1 )φ(x N +1 )T S N φ(x)
1/β + φ(x N +1 )T S N φ(x N +1 )
£
¤2
φ(x)T S N φ(x N +1 )
(∗)
1/β + φ(x N +1 )T S N φ(x N +1 )
82
And since S N is positive definite, (∗) is larger than 0. Therefore, we have
proved that σ2N (x) − σ2N +1 (x) ≥ 0
Problem 3.12 Solution
Let’s begin by writing down the prior PDF p(w, β):
p(w, β)
N (w| m 0 , β−1 S 0 ) Gam(β|a 0 , b 0 ) (∗)
β 2
1
a
∝ (
) exp(− (w − m 0 )T βS 0−1 (w − m 0 )) b 0 0 βa0 −1 exp(− b 0 β)
|S 0 |
2
=
And then we write down the likelihood function p(t|X, w, β) :
p(t|X, w, β)
=
N
Y
N (t n |wT φ(xn ), β−1 )
n=1
∝
N
Y
£ β
¤
β1/2 exp − (t n − wT φ(xn ))2
2
n=1
(∗∗)
According to Bayesian Inference, we have p(w, β|t) ∝ p(t|X, w, β)× p(w, β).
We first focus on the quadratic term with regard to w in the exponent.
N
X
β
β
quadratic term = − wT S 0 −1 w +
− wT φ(xn )φ(xn )T w
2
n=1 2
N
X
£
¤
β
= − wT S 0 −1 +
φ(xn )φ(xn )T w
2
n=1
Where the first term is generated by (∗), and the second by (∗∗). By now,
we know that:
N
X
φ(xn )φ(xn )T
S N −1 = S 0 −1 +
n=1
We then focus on the linear term with regard to w in the exponent.
linear term = β m 0 T S 0 −1 w +
N
X
β t n φ(xn )T w
n=1
£
= β m 0 T S 0 −1 +
N
X
¤
t n φ(xn )T w
n=1
Again, the first term is generated by (∗), and the second by (∗∗). We can
also obtain:
N
X
m N T S N −1 = m 0 T S 0 −1 +
t n φ(xn )T
n=1
Which gives:
N
X
£
¤
m N = S N S 0 −1 m 0 +
t n φ(xn )
n=1
83
Then we focus on the constant term with regard to w in the exponent.
N
β X
β
t2
constant term = (− m 0 T S 0 −1 m 0 − b 0 β) −
2
2 n=1 n
N
£1
¤
1 X
= −β m 0 T S 0 −1 m 0 + b 0 +
t2n
2
2 n=1
Therefore, we can obtain:
N
1
1 X
1
m N T S N −1 m N + b N = m 0 T S 0 −1 m 0 + b 0 +
t2
2
2
2 n=1 n
Which gives :
bN =
N
1 X
1
1
t2 − m N T S N −1 m N
m 0 T S 0 −1 m 0 + b 0 +
2
2 n=1 n 2
Finally, we focus on the exponential term whose base is β.
exponent term = (2 + a 0 − 1) +
N
2
Which gives:
2 + a N − 1 = (2 + a 0 − 1) +
N
2
Hence,
a N = a0 +
N
2
Problem 3.13 Solution
Similar to (3.57), we write down the expression of the predictive distribution p(t|X, t):
Z Z
p(t|X, t) =
p(t|w, β) p(w, β|X, t) dw d β
(∗)
We know that:
p(t|w, β) = N (t| y(x, w), β−1 ) = N (t|φ(x)T w, β−1 )
and that:
p(w, β|X, t) = N (w| m N , β−1 S N ) Gam(β|a N , b N )
We go back to (∗), and first deal with the integral with regard to w:
Z Z
£
¤
p(t|X, t) =
N (t|φ(x)T w, β−1 ) N (w| m N , β−1 S N ) dw Gam(β|a N , b N ) d β
Z
=
N (t|φ(x)T m N , β−1 + φ(x)T β−1 S N φ(x)) Gam(β|a N , b N ) d β
Z
£
¤
=
N t|φ(x)T m N , β−1 (1 + φ(x)T S N φ(x)) Gam(β|a N , b N ) d β
84
Where we have used (2.113), (2.114) and (2.115). Then, we follow (2.158)(2.160), we can see that p(t|X, t) = St(t|µ, λ, v), where we have defined:
µ = φ(x)T m N ,
λ=
¤−1
aN £
· 1 + φ(x)T S N φ(x) ,
bN
v = 2a N
Problem 3.14 Solution(Wait for updating)
Firstly, according to (3.16), if we use the new orthonormal basis set specified in the problem to construct Φ, we can obtain an important property:
ΦT Φ = I. Hence, if α = 0, together with (3.54), we know that SN = 1/β.
Finally, according to (3.62), we can obtain:
k(x, x0 ) = βψ(x)T SN ψ(x0 ) = ψ(x)T ψ(x0 )
Problem 3.15 Solution
It is quite obvious if we substitute (3.92) and (3.95) into (3.82), which
gives,
β
α
N −γ γ
N
E(m N ) = ||t − Φ m N ||2 + m N T m N =
+ =
2
2
2
2
2
Problem 3.16 Solution(Waiting for update)
We know that
p(t|w, β) =
N
Y
N (φ(xn )T w, β−1 ) ∝ N (Φw, β−1 I)
n=1
And
p(w|α) = N (0, α−1 I)
Comparing them with (2.113), (2.114) and (2.115), we can obtain:
p(t|α, β) = N (0, β−1 I + α−1 ΦΦT )
Problem 3.17 Solution
We know that:
p(t|w, β) =
N
Y
N (φ(xn )T w, β−1 )
n=1
=
N
Y
©
ª
1
T
2
exp
−
(t
−
φ
(x
)
w)
n
n
−1 1/2
2β−1
n=1 (2πβ )
N
©X
ª
β
− (t n − φ(xn )T w)2
2π
n=1 2
© β
ª
β N /2
= ( ) exp − ||t − Φw||2
2π
2
= (
β
1
) N /2 exp
85
And that:
p(w|α) = N (0, α−1 I)
=
α M /2
© α
ª
2
exp
−
||
w
||
2
(2π) M /2
If we substitute the expressions above into (3.77), we can obtain (3.78)
just as required.
Problem 3.18 Solution
We expand (3.79) as follows:
E(w) =
=
=
β
α
||t − Φw||2 + wT w
2
2
α
β T
(t t − 2tT Φw + wT ΦT Φw) + wT w
2
2
¤
1£ T
T
T
w (βΦ Φ + αI)w − 2βt Φw + βtT t
2
Observing the equation above, we see that E(w) contains the following
term :
1
(w − m N )T A(w − m N )
(∗)
2
Now, we need to solve A and m N . We expand (∗) and obtain:
(∗) =
1 T
(w Aw − 2m N T Aw + m N T Am N )
2
We firstly compare the quadratic term, which gives:
A = βΦT Φ + αI
And then we compare the linear term, which gives:
m N T A = βtT Φ
Noticing that A = AT , which implies A−1 is also symmetric, we first transpose and then multiply A−1 on both sides, which gives:
m N = βA−1 ΦT t
86
Now we rewrite E(w):
E(w) =
=
=
=
=
=
=
¤
1£ T
w (βΦT Φ + αI)w − 2βtT Φw + βtT t
2
¤
1£
(w − m N )T A(w − m N ) + βtT t − m N T Am N
2
1
1
(w − m N )T A(w − m N ) + (βtT t − m N T Am N )
2
2
1
1
(w − m N )T A(w − m N ) + (βtT t − 2m N T Am N + m N T Am N )
2
2
1
1
T
(w − m N ) A(w − m N ) + (βtT t − 2m N T Am N + m N T (βΦT Φ + αI)m N )
2
2
¤ α
1
1£
(w − m N )T A(w − m N ) + βtT t − 2βtT Φ m N + m N T (βΦT Φ)m N + m N T m N
2
2
2
1
β
α
T
2
T
(w − m N ) A(w − m N ) + ||t − Φ m N || + m N m N
2
2
2
Just as required.
Problem 3.19 Solution
Based on the standard form of a multivariate normal distribution, we
know that
Z
© 1
ª
1
1
T
exp
−
(w
−
m
)
A(w
−
m
)
dw = 1
N
N
2
(2π) M /2 |A|1/2
Hence,
Z
© 1
ª
exp − (w − m N )T A(w − m N ) dw = (2π) M /2 |A|1/2
2
And since E(m N ) doesn’t depend on w, (3.85) is quite obvious. Then we
substitute (3.85) into (3.78), which will immediately gives (3.86).
Problem 3.20 Solution
You can just follow the steps from (3.87) to (3.92), which is already very
clear.
Problem 3.21 Solution
Let’s first prove (3.117). According to (C.47) and (C.48), we know that if A
is a M × M real symmetric matrix, with eigenvalues λ i , i = 1, 2, ..., M, |A| and
Tr(A) can be written as:
|A| =
M
Y
i =1
λi ,
Tr(A) =
M
X
λi
i =1
Back to this problem, according to section 3.5.2, we know that A has
eigenvalues α + λ i , i = 1, 2, ..., M. Hence the left side of (3.117) equals to:
left side =
M
M d
M
X
£Y
¤ X
d
1
ln
(α + λ i ) =
ln(α + λ i ) =
dα
i =1
i =1 d α
i =1 α + λ i
87
And according to (3.81), we can obtain:
A−1
d
A = A−1 I = A−1
dα
For the symmetric matrix A, its inverse A−1 has eigenvalues 1 / (α+λ i ) , i =
1, 2, ..., M. Therefore,
M
X
d
1
Tr(A−1
A) =
dα
i =1 α + λ i
Hence there are the same, and (3.92) is quite obvious.
Problem 3.22 Solution
Let’s derive (3.86) with regard to β. The first term dependent on β in
(3.86) is :
d N
N
( lnβ) =
dβ 2
2β
The second term is :
d
E(m N ) =
dβ
1
β d
d α
||t − Φ m N ||2 +
||t − Φ m N ||2 +
mN T mN
2
2 dβ
dβ 2
The last two terms in the equation above can be further written as:
β d
2 dβ
||t − Φ m N ||2 +
d α
mN T mN
dβ 2
=
=
=
=
=
=
©β
ª dm N
d
d α
||t − Φ m N ||2 +
mN T mN ·
2 dm N
dm N 2
dβ
©β
ª
α
dm
N
[−2ΦT (t − Φ m N )] + 2m N ·
2
2
dβ
©
ª dm N
− βΦT (t − Φ m N ) + α m N ·
dβ
©
ª
dm N
− βΦT t + (αI + βΦT Φ)m N ·
dβ
©
ª dm N
− βΦT t + Am N ·
dβ
0
Where we have taken advantage of (3.83) and (3.84). Hence
N
d
1
1 X
E(m N ) = ||t − Φ m N ||2 =
(t n − m N T φ(xn ))2
dβ
2
2 n=1
The last term dependent on β in (3.86) is:
d 1
γ
( ln|A|) =
dβ 2
2β
Therefore, if we combine all those expressions together, we will obtain
(3.94). And then if we rearrange it, we will obtain (3.95).
88
Problem 3.23 Solution
First, according to (3.10), we know that p(t|X, w, β) can be further written
as p(t|X, w, β) = N (t|Φw, β−1 I), and given that p(w|β) = N (m 0 , β−1 S0 ) and
p(β) = Gam(β|a 0 , b 0 ). Therefore, we just follow the hint in the problem.
Z Z
p(t) =
p(t|X, w, β) p(w|β) dw p(β) d β
Z Z
© β
ª
β
=
( ) N /2 exp − (t − Φw)T (t − Φw) ·
2π
2
©
ª
β M /2
β
( ) |S0 |−1/2 exp − (w − m 0 )T S0 −1 (w − m 0 ) dw
2π
2
Γ(a 0 )−1 b 0a0 βa0 −1 exp(− b 0 β) d β
Z Z
a
b0 0
ª
© β
=
exp − (t − Φw)T (t − Φw)
(
M
+
N
)/2
1/2
2
(2π)
|S0 |
ª
© β
exp − (w − m 0 )T S0 −1 (w − m 0 ) dw
2
βa0 −1+ N /2+ M /2 exp(− b 0 β) d β
Z Z
a
b0 0
© β
ª
=
exp − (w − m N )T SN −1 (w − m N ) dw
(
M
+
N
)/2
1/2
2
(2π)
|S0 |
© β T
ª
exp − (t t + m 0 T S0 −1 m 0 − m N T SN −1 m N )
2
a N −1+ M /2
β
exp(− b 0 β) d β
Where we have defined
m N = SN (S0 −1 m 0 + ΦT t)
S N −1 = S0 −1 + ΦT Φ
a N = a0 +
N
2
N
X
1
b N = b 0 + (m 0 T S0 −1 m 0 − m N T SN −1 m N +
t2n )
2
n=1
Which are exactly the same as those in Prob.3.12, and then we evaluate
the integral, taking advantage of the normalized property of multivariate
Gaussian Distribution and Gamma Distribution.
a
p(t) =
=
=
b0 0
Z
2π M /2
1/2
( ) | SN |
βa N −1+ M /2 exp(− b N β) d β
(2π)( M + N )/2 |S0 |1/2 β
Z
a
b0 0
M /2
1/2
(2π) |SN |
βa N −1 exp(− b N β) d β
(2π)( M + N )/2 |S0 |1/2
a0
1
|SN |1/2 b 0 Γ(a N )
a
(2π) N /2 |S0 |1/2 b NN Γ(b N )
89
Just as required.
Problem 3.24 Solution
Let’s just follow the hint and we begin by writing down expression for the
likelihood, prior and posterior PDF. We know that p(t|w, β) = N (t|Φw, β−1 I).
What’s more, the form of the prior and posterior are quite similar:
p(w, β) = N (w|m0 , β−1 S0 ) Gam(β|a 0 , b 0 )
And
p(w, β|t) = N (w|mN , β−1 SN ) Gam(β|a N , b N )
Where the relationships among those parameters are shown in Prob.3.12,
Prob.3.23. Now according to (3.119), we can write:
p(t) = N (t|Φw, β−1 I)
N (w|m0 , β−1 S0 ) Gam(β|a 0 , b 0 )
N (w|mN , β−1 SN ) Gam(β|a N , b N )
a
−1
= N (t|Φw, β
0 a −1
N (w|m0 , β−1 S0 ) b 0 β 0 exp(− b 0 β) / Γ(a 0 )
I)
N (w|mN , β−1 SN ) b aNN βa N −1 exp(− b N β) / Γ(a N )
a
= N (t|Φw, β−1 I)
0
©
ª
N (w|m0 , β−1 S0 ) b 0 Γ(a N ) a0 −a N
β
exp − (b 0 − b N )β
aN
−
1
N (w|mN , β SN ) b N Γ(a 0 )
= N (t|Φw, β−1 I)
©
ª b 0 0 Γ(a N ) − N /2
N (w|m0 , β−1 S0 )
exp
−
(b
−
b
)
β
β
0
N
a
N (w|mN , β−1 SN )
b NN Γ(a 0 )
a
Where we have used a N = a 0 + N
2 . Now we deal with the terms expressed
in the form of Gaussian Distribution:
N (w|m0 , β−1 S0 )
N (w|mN , β−1 SN )
© β
ª
β
= ( ) N /2 exp − (t − Φw)T (t − Φw) ·
2π
2
©
ª
−1
1/2 exp − β (w − m )T S −1 (w − m )
| β SN |
0
0
0
2
©
ª
|β−1 S0 |1/2 exp − β (w − mN )T SN −1 (w − mN )
2
Gaussian terms = N (t|Φw, β−1 I)
β
|SN |1/2
© β
ª
T
exp
−
(t
−
Φ
w)
(t
−
Φ
w)
·
2π
2
|S0 |1/2
© β
ª
exp − 2 (w − m0 )T S0 −1 (w − m0 )
© β
ª
exp − 2 (w − mN )T SN −1 (w − mN )
= (
) N /2
We look back to the previous problem and we notice that at the last step
in the deduction of p(t), we complete the square according to w. And if we
carefully compare the left and right side at the last step, we can obtain :
ª
© β
ª
© β
exp − (t − Φw)T (t − Φw) exp − (w − m 0 )T S0 −1 (w − m 0 )
2
2
© β
ª
©
ª
= exp − (w − m N )T SN −1 (w − m N ) exp − (b N − b 0 )β
2
90
Hence, we go back to deal with the Gaussian terms:
Gaussian terms = (
β
2π
) N /2
|SN |1/2
|S0 |
©
ª
exp
−
(b
−
b
)
β
0
N
1/2
If we substitute the expressions above into p(t), we will obtain (3.118)
immediately.
0.4
Linear Models Classification
Problem 4.1 Solution
If the convex hull of {xn } and {yn } intersects, we know that there will be a
P
P
point z which can be written as z = n αn xn and also z = n βn yn . Hence we
can obtain:
b T z + w0
w
b T(
= w
= (
X
n
=
X
n
X
α n xn ) + w 0
n
X
b T xn ) + ( αn )w0
αn w
n
T
b xn + w 0 )
αn (w
(∗)
P
Where we have used n αn = 1. And if {xn } and {yn } are linearly separab T xn + w0 > 0 and w
b T yn + w0 < 0, for ∀xn , yn . Together with
ble, we have w
T
b z + w0 > 0. And if we calculate w
b T z + w0
αn ≥ 0 and (∗), we know that w
from the perspective of {yn } following the same procedure, we can obtain
b T z + w0 < 0. Hence contradictory occurs. In other words, they are not linw
early separable if their convex hulls intersect.
We have already proved the first statement, i.e., "convex hulls intersect"
gives "not linearly separable", and what the second part wants us to prove
is that "linearly separable" gives "convex hulls do not intersect". This can be
done simply by contrapositive.
The true converse of the first statement should be if their convex hulls do
not intersect, the data sets should be linearly separable. This is exactly what
Hyperplane Separation Theorem shows us.
Problem 4.2 Solution
e on w0 explicitly:
Let’s make the dependency of E D (W)
e =
E D (W)
ª
1 ©
Tr (XW + 1w0 T − T)T (XW + 1w0 T − T)
2
e with respect to w0 :
Then we calculate the derivative of E D (W)
e
∂E D (W)
∂w0
= 2Nw0 + 2(XW − T)T 1
91
Where we have used the property:
∂
∂X
£
¤
Tr (AXB + C)(AXB + C)T = 2AT (AXB + C)BT
We set the derivative equals to 0, which gives:
w0 = −
1
(XW − T)T 1 = t̄ − WT x̄
N
Where we have denoted:
t̄ =
1 T
T 1,
N
and x̄ =
1 T
X 1
N
e we can obtain:
If we substitute the equations above into E D (W),
©
ª
e = 1 Tr (XW + T̄ − X̄W − T)T (XW + T̄ − X̄W − T)
E D (W)
2
Where we further denote
T̄ = 1t̄T ,
and X̄ = 1x̄T
e with regard to W to 0, which gives:
Then we set the derivative of E D (W)
b †T
b
W=X
Where we have defined:
b = X − X̄ ,
X
b = T − T̄
and T
Now consider the prediction for a new given x, we have:
y(x) = WT x + w0
= WT x + t̄ − WT x̄
= t̄ + WT (x − x̄)
If we know that aT tn + b = 0 holds for some a and b, we can obtain:
aT t̄ =
N
1 X
1 T T
a T 1=
aT tn = − b
N
N n=1
Therefore,
£
¤
aT y(x) = aT t̄ + WT (x − x̄)
= aT t̄ + aT WT (x − x̄)
b T (X
b † )T (x − x̄)
= − b + aT T
= −b
92
Where we have used:
bT
aT T
= aT (T − T̄)T = aT (T −
= aT TT −
1 T T
11 T)
N
1 T T T
a T 11 = − b1T + b1T
N
= 0T
Problem 4.3 Solution
Suppose there are Q constraints in total. We can write aq T tn + b q = 0 , q =
1, 2, ...,Q for all the target vector tn , n = 1, 2..., N. Or alternatively, we can
group them together:
A T tn + b = 0
Where A is a Q × Q matrix, and the qth column of A is aq , and meanwhile b is a Q × 1 column vector, and the qth element is bq . for every pair
of {aq , b q } we can follow the same procedure in the previous problem to show
that aq y(x) + b q = 0. In other words, the proofs will not affect each other.
Therefore, it is obvious :
AT y(x) + b = 0
Problem 4.4 Solution
We use Lagrange multiplier to enforce the constraint wT w = 1. We now
need to maximize :
L(λ, w) = wT (m2 − m1 ) + λ(wT w − 1)
We calculate the derivatives:
∂L(λ, w)
∂λ
And
∂L(λ, w)
∂w
= wT w − 1
= m2 − m1 + 2λw
We set the derivatives above equals to 0, which gives:
w=−
1
(m2 − m1 ) ∝ (m2 − m1 )
2λ
Problem 4.5 Solution
We expand (4.25) using (4.22), (4.23) and (4.24).
J(w) =
=
(m 2 − m 1 )2
s21 + s22
P
||wT (m2 − m1 )||2
P
T
2
T
2
n∈C 1 (w xn − m 1 ) + n∈C 2 (w xn − m 2 )
93
The numerator can be further written as:
£
¤£
¤T
numerator = wT (m2 − m1 ) wT (m2 − m1 ) = wT SB w
Where we have defined:
SB = (m2 − m1 )(m2 − m1 )T
And ti is the same for the denominator:
X
X
denominator =
[wT (xn − m1 )]2 +
[wT (xn − m2 )]2
n∈C 1
n∈C 2
T
T
= w Sw1 w + w Sw2 w
= w T Sw w
Where we have defined:
X
X
Sw =
(xn − m1 )(xn − m1 )T +
(xn − m2 )(xn − m2 )T
n∈C 1
n∈C 2
Just as required.
Problem 4.6 Solution
Let’s follow the hint, beginning by expanding (4.33).
(4.33) =
N
X
w T xn xn + w 0
n=1
=
N
X
N
X
n=1
xn xn T w − w T m
n=1
=
N
X
xn −
N
X
N
X
N
X
n=1
xn xn T w − NwT mm − N(
N
X
X
t n xn +
n∈C 1
xn xn T w − wT m · (Nm) − (
n=1
=
X
xn − (
n=1
=
t n xn
n=1
X N
X −N
xn +
xn )
n ∈ C 1 N1
n ∈ C 2 N2
X 1
1
xn −
xn )
n ∈ C 2 N2
n ∈ C 1 N1
X
xn xn T w − NmmT w − N(m1 − m2 )
n=1
= [
N
X
(xn xn T ) − NmmT ]w − N(m1 − m2 )
n=1
If we let the derivative equal to 0, we will see that:
[
N
X
(xn xn T ) − NmmT ]w = N(m1 − m2 )
n=1
Therefore, now we need to prove:
N
X
n=1
t n xn )
n∈C 2
(xn xn T ) − NmmT = Sw +
N1 N2
SB
N
94
Let’s expand the left side of the equation above:
left =
N
X
xn xn T − N(
n=1
=
N
X
xn xn T − N(
n=1
=
N
X
xn xn T −
n=1
=
N
X
N1
N2
m1 +
m2 )2
N
N
N12
N22
N
N2
N12
N
||m1 ||2 +
2
||m1 ||2 −
xn xn T + (N1 +
n=1
=
N
X
N22
N
||m2 ||2 + 2
||m2 ||2 − 2
N1 N2
m1 m2 T )
N2
N1 N2
m1 m2 T
N
N1 N2
N1 N2
N1 N2
− 2N1 )||m1 ||2 + (N2 +
− 2N2 )||m2 ||2 − 2
m1 m2 T
N
N
N
xn xn T + (N1 − 2N1 )||m1 ||2 + (N2 − 2N2 )||m2 ||2 +
n=1
=
N
X
N1 N2
||m1 − m2 ||2
N
xn xn T + N1 ||m1 ||2 − 2m1 · (N1 m1 T ) + N2 ||m2 ||2 − 2m2 · (N2 m2 T ) +
n=1
=
N
X
xn xn T + N1 ||m1 ||2 − 2m1
n=1
=
X
n∈C 1
T
2
xn xn + N1 ||m1 || − 2m1
n∈C 1
+
=
X
xnT +
n∈C 2
(xn xn T + ||m1 ||2 − 2m1 xnT ) +
n∈C 1
X
n∈C 2
xnT +
N1 N2
SB
N
xnT
X
xn xn T + N2 ||m2 ||2 − 2m2
n∈C 2
=
X
n∈C 1
X
X
xnT + N2 ||m2 ||2 − 2m2
X
N1 N2
SB
N
N1 N2
SB
N
(xn xn T + ||m2 ||2 − 2m2 xn T ) +
X
n∈C 2
||xn − m1 ||2 +
n∈C 1
= Sw +
X
||xn − m2 ||2 +
n∈C 2
N1 N2
SB
N
N1 N2
SB
N
N1 N2
SB
N
Just as required.
Problem 4.7 Solution
This problem is quite simple. We can solve it by definition. We know that
logistic sigmoid function has the form:
σ(a) =
1
1 + exp(−a)
Therefore, we can obtain:
σ(a) + σ(−a)
=
=
=
1
1
+
1 + exp(−a) 1 + exp(a)
2 + exp(a) + exp(−a)
[1 + exp(−a)][1 + exp(a)]
2 + exp(a) + exp(−a)
=1
2 + exp(a) + exp(−a)
95
Next we exchange the dependent and independent variables to obtain its
inverse.
1
a=
1 + exp(− y)
We first rearrange the equation above, which gives:
exp(− y) =
1−a
a
Then we calculate the logarithm for both sides, which gives:
y = ln(
a
)
1−a
Just as required.
Problem 4.8 Solution
According to (4.58) and (4.64), we can write:
a = ln
p(x|C 1 )p(C 1 )
p(x|C 2 )p(C 2 )
p(C 1 )
p(C 2 )
1
p(C 1 )
1
= − (x − µ1 )T Σ−1 (x − µ1 ) + (x − µ2 )T Σ−1 (x − µ2 ) + ln
2
2
p(C 2 )
1
1
p(C
)
1
= Σ−1 (µ1 − µ2 )x − µ1 T Σ−1 µ1 + µ2 T Σ−1 µ2 + ln
2
2
p(C 2 )
= ln p(x|C 1 ) − ln p(x|C 2 ) + ln
= w T x + w0
Where in the last second step, we rearrange the term according to x, i.e.,
its quadratic, linear, constant term. We have also defined :
w = Σ−1 (µ1 − µ2 )
And
1
1
p(C 1 )
w0 = − µ1 T Σ−1 µ1 + µ2 T Σ−1 µ2 + ln
2
2
p(C 2 )
Finally, since p(C 1 |x) = σ(a) as stated in (4.57), we have p(C 1 |x) = σ(wT x+
w0 ) just as required.
Problem 4.9 Solution
We begin by writing down the likelihood function.
p({φn , t n }|π1 , π2 , ..., πK ) =
N Y
K
Y
[p(φn |C k ) p(C k )] t nk
n=1 k=1
=
N Y
K
Y
[πk p(φn |C k )] t nk
n=1 k=1
96
Hence we can obtain the expression for the logarithm likelihood:
ln p =
K
N X
X
K
N X
X
£
¤
t nk ln πk
t nk ln πk + ln p(φn |C k ) ∝
n=1 k=1
n=1 k=1
Since there is a constraint on πk , so we need to add a Lagrange Multiplier
to the expression, which becomes:
L=
N X
K
X
t nk ln πk + λ(
n=1 k=1
K
X
πk − 1)
k=1
We calculate the derivative of the expression above with regard to πk :
∂L
=
∂πk
N t
X
nk
n=1 π k
+λ
And if we set the derivative equal to 0, we can obtain:
πk = − (
N
X
t nk ) / λ = −
n=1
Nk
λ
(∗)
And if we preform summation on both sides with regard to k, we can see
that:
K
X
N
1 = −(
Nk ) / λ = −
λ
k=1
Which gives λ = − N, and substitute it into (∗), we can obtain πk = Nk / N.
Problem 4.10 Solution
This time, we focus on the term which dependent on µk and Σ in the
logarithm likelihood.
ln p =
N X
K
X
N X
K
X
£
¤
t nk ln πk + ln p(φn |C k ) ∝
t nk ln p(φn |C k )
n=1 k=1
n=1 k=1
Provided p(φ|C k ) = N (φ|µk , Σ), we can further derive:
ln p ∝
N X
K
X
£ 1
¤
1
t nk − ln |Σ| − (φn − µk )Σ−1 (φn − µk )T
2
2
n=1 k=1
We first calculate the derivative of the expression above with regard to
µk :
∂ ln p
∂µk
=
N
X
t nk Σ−1 (φn − µk )
n=1
We set the derivative equals to 0, which gives:
N
X
n=1
t nk Σ−1 φn =
N
X
n=1
t nk Σ−1 µk = Nk Σ−1 µk
97
Therefore, if we multiply both sides by Σ / Nk , we will obtain (4.161). Now
let’s calculate the derivative of ln p with regard to Σ, which gives:
∂ ln p
∂Σ
=
=
K
N X
X
K
N X
1
1 ∂ X
t nk (− Σ−1 ) −
t nk (φn − µk )Σ−1 (φn − µk )T
2
2
∂
Σ
n=1 k=1
n=1 k=1
K
N X
X
−
n=1 k=1
=
N
K X
t nk −1 1 ∂ X
t nk (φn − µk )Σ−1 (φn − µk )T
Σ −
2
2 ∂Σ k=1 n=1
N
X
K
1
1 ∂ X
− Σ−1 −
Nk Tr(Σ−1 Sk )
2 ∂Σ k=1
n=1 2
= −
K
N −1 1 X
Nk Σ−1 Sk Σ−1
Σ +
2
2 k=1
Where we have denoted
Sk =
N
1 X
t nk (φn − µk )(φn − µk )T
Nk n=1
Now we set the derivative equals to 0, and rearrange the equation, which
gives:
K N
X
k
Σ=
Sk
N
k=1
Problem 4.11 Solution
Based on definition, we can write down
p(φ|C k ) =
L
M Y
Y
m=1 l =1
φ
ml
µkml
Note that here only one of the value among φm1 , φm2 , ... φmL is 1, and the
others are all 0 because we have used a 1 − of − L binary coding scheme, and
also we have taken advantage of the assumption that the M components of
φ are independent conditioned on the class C k . We substitute the expression
above into (4.63), which gives:
ak =
M X
L
X
φml · ln µkml + ln p(C k )
m=1 l =1
Hence it is obvious that a k is a linear function of the components of φ.
Problem 4.12 Solution
Based on definition, i.e., (4.59), we know that logistic sigmoid has the
form:
1
σ(a) =
1 + exp(−a)
98
Now, we calculate its derivative with regard to a.
exp(a)
exp(a)
1
d σ(a)
=
=
·
= [ 1 − σ(a) ] · σ(a)
2
da
1 + exp(−a) 1 + exp(−a)
[ 1 + exp(−a) ]
Just as required.
Problem 4.13 Solution
Let’s follow the hint.
∇E(w) = −∇
N
X
{ t n ln yn + (1 − t n ) ln(1 − yn ) }
n=1
N
X
= −
∇{ t n ln yn + (1 − t n ) ln(1 − yn ) }
n=1
N d { t ln y + (1 − t ) ln(1 − y ) } d y da
X
n
n
n
n
n
n
= −
n=1
= −
= −
d yn
da n dw
N t
X
1 − tn
n
( −
) · yn (1 − yn ) · φn
y
1
− yn
n=1 n
N
X
t n − yn
· yn (1 − yn ) · φn
n=1 yn (1 − yn )
N
X
= −
(t n − yn )φn
n=1
=
N
X
(yn − t n )φn
n=1
Where we have used yn = σ(a n ), a n = wT φn , the chain rules and (4.88).
Problem 4.14 Solution
According to definition, we know that if a dataset is linearly separable,
we can find w, for some points xn , we have wT φ(xn ) > 0, and the others
wT φ(xm ) < 0. Then the boundary is given by wT φ(x) = 0. Note that for any
point x0 in the dataset, the value of wT φ(x0 ) should either be positive or
negative, but it can not equal to 0.
Therefore, the maximum likelihood solution for logistic regression is trivial. We suppose for those points xn belonging to class C 1 , we have wT φ(xn ) >
0 and wT φ(xm ) < 0 for those belonging to class C 2 . According to (4.87), if
|w| → ∞, we have
p(C 1 |φ(xn )) = σ(wT φ(xn )) → 1
Where we have used wT φ(xn ) → +∞. And since wT φ(xm ) → −∞, we can
also obtain:
p(C 2 |φ(xm )) = 1 − p(C 1 |φ(xm )) = 1 − σ(wT φ(xm )) → 1
99
In other words, for the likelihood function, i.e.,(4.89), if we have |w| → ∞,
and also we label all the points lying on one side of the boundary as class C 1 ,
and those on the other side as class C 2 , the every term in (4.89) can achieve
its maximum value, i.e., 1, finally leading to the maximum of the likelihood.
Hence, for a linearly separable dataset, the learning process may prefer
to make |w| → ∞ and use the linear boundary to label the datasets, which
can cause severe over-fitting problem.
Problem 4.15 Solution(Waiting for update)
Since yn is the output of the logistic sigmoid function, we know that 0 <
yn < 1 and hence yn (1 − yn ) > 0. Then we use (4.97), for an arbitrary non-zero
real vector a 6= 0, we have:
aT Ha = aT
N
£X
¤
yn (1 − yn )φn φT
n a
n=1
N
X
=
T
T
yn (1 − yn ) (φT
n a) (φ n a)
n=1
N
X
=
yn (1 − yn ) b2n
n=1
Where we have denoted b n = φT
n a. What’s more, there should be at least
one of { b 1 , b 2 , ..., b N } not equal to zero and then we can see that the expression
above is larger than 0 and hence H is positive definite.
Otherwise, if all the b n = 0, a = [a 1 , a 2 , ..., a M ]T will locate in the null
space of matrix Φ N × M . However, with regard to the rank-nullity theorem,
we know that Rank(Φ) + Nullity(Φ) = M, and we have already assumed that
those M features are independent, i.e., Rank(Φ) = M, which means there is
only 0 in its null space. Therefore contradictory occurs.
Problem 4.16 Solution
We still denote yn = p(t = 1|φn ), and then we can write down the log
likelihood by replacing t n with πn in (4.89) and (4.90).
ln p(t|w) =
N
X
{ πn ln yn + (1 − πn ) ln(1 − yn ) }
n=1
Problem 4.17 Solution
We should discuss in two situations separately, namely j = k and j 6= k.
When j 6= k, we have:
∂ yk
∂a j
=
− exp(a k ) · exp(a j )
= − yk · y j
P
[ j exp(a j ) ]2
And when j = k, we have:
P
exp(a k ) j exp(a j ) − exp(a k ) exp(a k )
∂ yk
=
= yk − yk2 = yk (1 − yk )
P
∂a k
[ j exp(a j ) ]2
100
Therefore, we can obtain:
∂ yk
∂a j
= yk (I k j − y j )
Where I k j is the elements of the indentity matrix.
Problem 4.18 Solution
We derive every term t nk ln ynk with regard to a j .
∂ t nk ln ynk
∂ t nk ln ynk ∂ ynk ∂a j
=
∂wj
∂ ynk
∂a j ∂wj
1
t nk
· ynk (I k j − yn j ) · φn
ynk
t nk (I k j − yn j ) φn
=
=
Where we have used (4.105) and (4.106). Next we perform summation
over n and k.
∇wj E
= −
N X
K
X
t nk (I k j − yn j ) φn
n=1 k=1
=
N X
K
X
t nk yn j φn −
n=1 k=1
=
N
X
n=1
k=1
N
X
yn j φn −
n=1
=
t nk I k j φn
n=1 k=1
N £ X
K
N
X
¤ X
(
t nk )yn j φn −
t n j φn
n=1
=
N X
K
X
N
X
t n j φn
n=1
(yn j − t n j ) φn
n=1
Where we have used the fact that for arbitrary n, we have
Problem 4.19 Solution
We write down the log likelihood.
ln p(t|w) =
N ©
X
t n ln yn + (1 − t n ) ln(1 − yn )
n=1
Therefore, we can obtain:
∇w ln p
=
=
=
∂ ln p ∂ yn ∂a n
·
·
∂ yn ∂a n ∂w
N t
X
1 − tn 0
n
)Φ (a n )φn
( −
1 − yn
n=1 yn
N
X
yn − t n
Φ0 (a n )φn
y
(1
−
y
)
n
n
n=1
ª
PK
k=1 t nk = 1.
101
Where we have used y = p(t = 1|a) = Φ(a) and a n = wT φn . According to
(4.114), we can obtain:
¯
1
1
Φ0 (a) = N (θ |0, 1)¯θ=a = p exp(− a2 )
2
2π
Hence, we can obtain:
a2
N
X
yn − t n exp(− 2n )
∇w ln p =
φn
p
2π
n=1 yn (1 − yn )
To calculate the Hessian Matrix, we need to first evaluate several derivatives.
∂
yn − t n
} =
∂w yn (1 − yn )
{
=
∂
yn − t n
∂ yn ∂a n
}·
·
∂ yn yn (1 − yn ) ∂a n ∂w
yn (1 − yn ) − (yn − t n )(1 − 2yn ) 0
Φ (a n )φn
[ yn (1 − yn ) ]2
{
a2
=
yn2 + t n − 2yn t n exp(− 2n )
φn
p
yn2 (1 − yn )2
2π
And
a2
∂
∂
a2
exp(− n ) ∂a n
{ p 2 }
∂a n
∂w
2π
2
a
an
= − p exp(− n )φn
2
2π
exp(− n )
{ p 2 } =
∂w
2π
Therefore, using the chain rule, we can obtain:
a2
∂
yn − t n exp(− 2n )
{
} =
p
∂w yn (1 − yn )
2π
a2
a2
a2
a2
yn − t n exp(− 2n )
yn − t n ∂ exp(− 2n )
{
} p
+
{ p
}
∂w yn (1 − yn )
yn (1 − yn ) ∂w
2π
2π
∂
£ yn2 + t n − 2yn t n exp(− 2n )
¤ exp(− 2n )
− a n (yn − t n ) p
φn
=
p
yn (1 − yn )
2π
2π yn (1 − yn )
Finally if we perform summation over n, we can obtain the Hessian Matrix:
H = ∇∇w ln p
a2
=
N ∂
X
yn − t n exp(− 2n )
{
} · φn
p
2π
n=1 ∂w yn (1 − yn )
=
N £ y2 + t − 2y t exp(− n )
X
¤ exp(− 2n )
n
n n
n
2
− a n (yn − t n ) p
φn φn T
p
y
(1
−
y
)
2
π
2
π
y
(1
−
y
)
n
n
n=1
n
n
a2
a2
102
Problem 4.20 Solution(waiting for update)
We know that the Hessian Matrix is of size MK × MK, and the ( j, k) th
block with size M × M is given by (4.110), where j, k = 1, 2, ..., K. Therefore,
we can obtain:
K X
K
X
uT
(∗)
uT Hu =
j Hj,k uk
j =1 k=1
Where we use uk to denote the k th block vector of u with size M × 1, and
Hj,k to denote the ( j, k) th block matrix of H with size M × M. Then based on
(4.110), we further expand (4.110):
(∗) =
K X
K
X
uT
j {−
j =1 k=1
=
N
K X
K X
X
j =1 k=1 n=1
=
N
K X
K X
X
j =1 k=1 n=1
=
K X
N
X
k=1 n=1
N
X
ynk (I k j − yn j ) φn φn T }uk
n=1
T
uT
j {− ynk (I k j − yn j ) φ n φ n }uk
T
uT
j {− ynk I k j φ n φ n }uk +
T
uT
k {− ynk φ n φ n }uk +
N
K X
K X
X
j =1 k=1 n=1
K X
K X
N
X
j =1 k=1 n=1
T
uT
j { ynk yn j φ n φ n }uk
T
yn j uT
j { φ n φ n } ynk uk
Problem 4.21 Solution
It is quite obvious.
Φ(a) =
=
=
=
=
=
=
Z a
N (θ |0, 1) d θ
Z a
1
N (θ |0, 1) d θ
+
2
Z0 a
1
+
N (θ |0, 1) d θ
2
0
Z a
1
1
+p
exp(−θ 2 /2) d θ
2
2π 0
p Z a
1
1
π
2
+p
p exp(−θ 2 /2) d θ
2
π
2π 2 0
Z a
1
1
2
(1 + p
p exp(−θ 2 /2) d θ )
2
π
2 0
ª
©
1
1
1 + p er f (a)
2
2
−∞
Where we have used
Z 0
−∞
N (θ |0, 1) d θ =
1
2
103
Problem 4.22 Solution
If we denote f (θ ) = p(D |θ )p(θ ), we can write:
Z
Z
p(D) =
p(D |θ )p(θ ) d θ = f (θ ) d θ
(2π) M /2
=
f (θ M AP )
=
p(D |θ M AP ) p(θ M AP )
|A|1/2
(2π) M /2
|A|1/2
Where θ M AP is the value of θ at the mode of f (θ ), A is the Hessian Matrix
of − ln f (θ ) and we have also used (4.135). Therefore,
ln p(D) = ln p(D |θ M AP ) + ln p(θ M AP ) +
1
M
ln 2π − ln |A|
2
2
Just as required.
Problem 4.23 Solution
According to (4.137), we can write:
M
1
ln 2π − ln |A|
2
2
M
1
1
= ln p(D |θ M AP ) −
ln 2π − ln |V0 | − (θ M AP − m)T V0 −1 (θ M AP − m)
2
2
2
M
1
+
ln 2π − ln |A|
2
2
1
1
1
= ln p(D |θ M AP ) − ln |V0 | − (θ M AP − m)T V0 −1 (θ M AP − m) − ln |A|
2
2
2
ln p(D) = ln p(D |θ M AP ) + ln p(θ M AP ) +
Where we have used the definition of the multivariate Gaussian Distribution. Then, from (4.138), we can write:
A = −∇∇ ln p(D |θ M AP )p(θ M AP )
= −∇∇ ln p(D |θ M AP ) − ∇∇ ln p(θ M AP )
ª
© 1
= H − ∇∇ − (θ M AP − m)T V0 −1 (θ M AP − m)
2
© −1
ª
= H + ∇ V0 (θ M AP − m)
= H + V0 −1
Where we have denoted H = −∇∇ ln p(D |θ M AP ). Therefore, the equation
104
above becomes:
ª
1 ©
1
1
(θ M AP − m)T V0 −1 (θ M AP − m) − ln |V0 | · |H + V−
0 |
2
2
ª
1
1 ©
= ln p(D |θ M AP ) − (θ M AP − m)T V0 −1 (θ M AP − m) − ln |V0 H + I|
2
2
1
1
1
−1
T
≈ ln p(D |θ M AP ) − (θ M AP − m) V0 (θ M AP − m) − ln |V0 | − ln |H|
2
2
2
1
1
≈ ln p(D |θ M AP ) − (θ M AP − m)T V0 −1 (θ M AP − m) − ln |H| + const
2
2
ln p(D) = ln p(D |θ M AP ) −
Where we have used the property of determinant: |A|·|B| = |AB|, and the
fact that the prior is board, i.e. I can be neglected with regard to V0 H. What’s
more, since the prior is pre-given, we can view V0 as constant. And if the data
is large, we can write:
N
X
b
Hn = N H
H=
n=1
b = 1/N
Where H
PN
n=1 Hn , and then
1
1
(θ M AP − m)T V0 −1 (θ M AP − m) − ln |H| + const
2
2
1
1
b | + const
≈ ln p(D |θ M AP ) − (θ M AP − m)T V0 −1 (θ M AP − m) − ln | N H
2
2
1
M
1
b | + const
≈ ln p(D |θ M AP ) − (θ M AP − m)T V0 −1 (θ M AP − m) −
ln N − ln |H
2
2
2
M
≈ ln p(D |θ M AP ) −
ln N
2
ln p(D) ≈ ln p(D |θ M AP ) −
This is because when N >> 1, other terms can be neglected.
Problem 4.24 Solution(Waiting for updating)
Problem 4.25 Solution
We first need to obtain the expression for the first derivative of probit
function Φ(λa) with regard to a. According to (4.114), we can write down:
d
Φ(λa) =
da
=
d Φ(λa) d λa
·
d(λa)
da
© 1
ª
λ
p exp − (λa)2
2
2π
Which further gives:
¯
λ
d
¯
= p
Φ(λa)¯
a=0
da
2π
And for logistic sigmoid function, according to (4.88), we have
dσ
1
= σ (1 − σ) = 0.5 × 0.5 =
da
4
105
Where we have used σ(0) = 0.5. Let their derivatives at origin equals, we
have:
1
λ
=
p
4
2π
p ±
±
2
i.e., λ = 2π 4. And hence λ = π 8 is obvious.
Problem 4.26 Solution
We will prove (4.152) in a more simple and intuitive way. But firstly, we
need to prove a trivial yet useful statement: Suppose we have a random variable satisfied normal distribution denoted as X ∼ N (X |µ, σ2 ), the probability
x−µ
of X ≤ x is P(X ≤ x) = Φ( σ ), and here x is a given real number. We can see
this by writing down the integral:
Z x
£
¤
1
1
P(X ≤ x) =
exp − 2 (X − µ)2 d X
p
2σ
−∞ 2πσ2
Z x−µ
σ
1
1
=
exp(− γ2 ) σ d γ
p
2
2
−∞
2πσ
Z x−µ
σ
1
1
=
p exp(− γ2 ) d γ
2
−∞
2π
x−µ
= Φ(
)
σ
Where we have changed the variable X = µ + σγ. Now consider two random variables X ∼ N (0, λ−2 ) and Y ∼ N (µ, σ2 ). We first calculate the conditional probability P(X ≤ Y | Y = a):
a−0
) = Φ(λa)
λ−1
Together with Bayesian Formula, we can obtain:
Z +∞
P(X ≤ Y ) =
P(X ≤ Y | Y = a) pd f (Y = a) dY
−∞
Z +∞
Φ(λa) N (a|µ, σ2 ) da
=
P(X ≤ Y | Y = a) = P(X ≤ a) = Φ(
−∞
Where pd f (·) denotes the probability density function and we have also
used pd f (Y ) = N (µ, σ2 ). What’s more, we know that X − Y should also satisfy normal distribution, with:
E[X − Y ] = E[X ] − E[Y ] = 0 − µ = −µ
And
var[X − Y ] = var[X ] + var[Y ] = λ−2 + σ2
Therefore, X − Y ∼ N (−µ, λ−2 + σ2 ) and it follows that:
0 − (−µ)
µ
P(X − Y ≤ 0) = Φ( p
) = Φ( p
)
λ−2 + σ2
λ−2 + σ2
Since P(X ≤ Y ) = P(X − Y ≤ 0), we obtain what have been required.
106
0.5
Neural Networks
Problem 5.1 Solution
Based on definition of tanh(·), we can obtain:
e a − e −a
e a + e −a
2e a
= −1 + a
e + e −a
1
= −1 + 2
1 + e−2a
= 2σ(2a) − 1
tanh(a) =
s)
s)
s)
s)
If we have parameters w(1
, w(1
and w(2
, w(2
for a network whose
ji
j0
kj
k0
t)
t)
hidden units use logistic sigmoid function as activation and w(1
, w(1
and
ji
j0
t)
t)
w(2
, w(2
for another one using tanh(·), for the network using tanh(·) as
kj
k0
activation, we can write down the following expression by using (5.4):
a(kt)
=
M
X
j =1
=
M
X
j =1
=
M
X
j =1
t)
t)
w(2
tanh(a(jt) ) + w(2
kj
k0
t)
t)
w(2
[ 2σ(2a(jt) ) − 1 ] + w(2
kj
k0
M
£ X
t)
t)
t) ¤
2 w(2
σ(2a(jt) ) + −
w(2
+ w(2
kj
kj
k0
j =1
What’s more, we also have :
a(ks) =
M
X
j =1
s)
s)
w(2
σ(a(js) ) + w(2
kj
k0
To make the two networks equivalent, i.e., a(ks) = a(kt) , we should make
sure:


a(s) = 2a(jt)

 j
s)
t)
= 2w(2
w(2
kj
kj


P
w(2s) = − M w(2 t) + w(2 t)
k0
j =1
kj
k0
Note that the first condition can be achieved by simply enforcing:
s)
t)
w(1
= 2w(1
,
ji
ji
and
s)
t)
w(1
= 2w(1
j0
j0
Therefore, these two networks are equivalent under a linear transformation.
Problem 5.2 Solution
107
It is obvious. We write down the likelihood.
N
Y
p(T|X, w) =
N (tn |y(xn , w), β−1 I)
n=1
Taking the negative logarithm, we can obtain:
E(w, β) = − ln p(T|X, w) =
N £
β X
¤ NK
( y(xn , w)−tn )T ( y(xn , w)−tn ) −
ln β+const
2 n=1
2
Here we have used const to denote the term independent of both w and
β. Note that here we have used the definition of the multivariate Gaussian
Distribution. What’s more, we see that the covariance matrix β−1 I and the
weight parameter w have decoupled, which is distinct from the next problem. We can first solve wML by minimizing the first term on the right of the
equation above or equivalently (5.11), i.e., imaging β is fixed. Then according
to the derivative of E(w, β) with regard to β, we can obtain (5.17) and hence
β ML .
Problem 5.3 Solution
Following the process in the previous question, we first write down the
negative logarithm of the likelihood function.
E(w, Σ) =
N ©
ª N
1 X
[ y(xn , w) − tn ]T Σ−1 [ y(xn , w) − tn ] +
ln |Σ| + const (∗)
2 n=1
2
Note here we have assumed Σ is unknown and const denotes the term
independent of both w and Σ. In the first situation, if Σ is fixed and known,
the equation above will reduce to:
E(w) =
N ©
ª
1 X
[ y(xn , w) − tn ]T Σ−1 [ y(xn , w) − tn ] + const
2 n=1
We can simply solve wML by minimizing it. If Σ is unknown, since Σ is
in the first term on the right of (∗), solving wML will involve Σ. Note that in
the previous problem, the main reason that they can decouple is due to the
independent assumption, i.e., Σ reduces to β−1 I, so that we can bring β to the
front and view it as a fixed multiplying factor when solving wML .
Problem 5.4 Solution
Based on (5.20), the current conditional distribution of targets, considering mislabel, given input x and weight w is:
p(t = 1|x, w) = (1 − ²) · p(t r = 1|x, w) + ² · p(t r = 0|x, w)
108
Note that here we use t to denote the observed target label, t r to denote
its real label, and that our network is aimed to predict the real label t r not t,
i.e., p(t r = 1|x, w) = y(x, w), hence we see that:
£
¤
p(t = 1|x, w) = (1 − ²) · y(x, w) + ² · 1 − y(x, w)
(∗)
Also, it is the same for p(t = 0|x, w):
£
¤
p(t = 0|x, w) = (1 − ²) · 1 − y(x, w) + ² · y(x, w)
(∗∗)
Combing (∗) and (∗∗), we can obtain:
p(t|x, w) = (1 − ²) · y t (1 − y)1− t + ² · (1 − y) t y1− t
Where y is short for y(x, w). Therefore, taking the negative logarithm, we
can obtain the error function:
E(w) = −
N
X
©
1− t ª
t
ln (1 − ²) · ynn (1 − yn )1− t n + ² · (1 − yn ) t n yn n
n=1
When ² = 0, it is obvious that the equation above will reduce to (5.21).
Problem 5.5 Solution
It is obvious by using (5.22).
E(w) = − ln
N
Y
p(t|xn , w)
n=1
= − ln
K
N Y
Y
£
¤1− t nk
yk (xn , w) t nk 1 − yk (xn , w)
n=1 k=1
= −
N X
K
X
©
£
¤1− t nk ª
ln yk (xn , w) t nk 1 − yk (xn , w)
n=1 k=1
= −
N X
K
X
n=1 k=1
= −
£ t nk
¤
ln ynk
( 1 − ynk )1− t nk
N X
K ©
X
ª
t nk ln ynk + (1 − t nk ) ln( 1 − ynk )
n=1 k=1
Where we have denoted
ynk = yk (xn , w)
Problem 5.6 Solution
We know that yk = σ(a k ), where σ(·) represents the logistic sigmoid function. Moreover,
dσ
= σ(1 − σ)
da
109
dE(w)
da k
¤
¤
1£
1 £
yk (1 − yk ) + (1 − t k )
yk (1 − yk )
yk
1 − yk
£
¤ £ 1 − tk tk ¤
=
yk (1 − yk )
−
1 − yk yk
= (1 − t k )yk − t k (1 − yk )
= −t k
=
yk − t k
Just as required.
Problem 5.7 Solution
It is similar to the previous problem. First we denote ykn = yk (xn , w). If
we use softmax function as activation for the output unit, according to (4.106),
we have:
d ykn
= ykn (I k j − y jn )
da j
Therefore,
dE(w)
da j
=
K
N X
ª
d © X
t kn ln yk (xn , w)
−
da k
n=1 k=1
= −
= −
K
N X
X
ª
d ©
t kn ln ykn
n=1 k=1 da j
K
N X
X
t kn
n=1 k=1
= −
K
N X
X
¤
1 £
ykn (I k j − y jn )
ykn
(t kn I k j − t kn y jn )
n=1 k=1
= −
K
N X
X
N
X
n=1
=
N
X
t jn +
t kn y jn
n=1 k=1
n=1 k=1
= −
K
N X
X
t kn I k j +
N
X
y jn
n=1
(y jn − t jn )
n=1
Where we have used the fact that only when k = j, I k j = 1 6= 0 and that
k=1 t kn = 1.
PK
Problem 5.8 Solution
It is obvious based on definition of ’tanh’, i.e., (5.59).
(e a + e−a )(e a + e−a ) − (e a − e−a )(e a − e−a )
(e a + e−a )2
(e a − e−a )2
= 1− a
(e + e−a )2
= 1 − tanh(a)2
d
tanh(a) =
da
110
Problem 5.9 Solution
We know that the logistic sigmoid function σ(a) ∈ [0, 1], therefore if we
perform a linear transformation h(a) = 2σ(a) − 1, we can find a mapping function h(a) from (−∞, +∞) to [−1, 1]. In this case, the conditional distribution
of targets given inputs can be similarly written as:
p(t|x, w) =
£ 1 + y(x, w) ¤(1+ t)/2 £ 1 − y(x, w) ¤(1− t)/2
2
2
£
¤
Where 1 + y(x, w) /2 represents the conditional probability p(C 1 | x). Since
now y(x, w) ∈ [−1, 1], we also need to perform the linear transformation to
make it satisfy the constraint for probability.Then we can further obtain:
E(w) = −
N ©1+ t
X
n
n=1
= −
N ©
1 X
2 n=1
2
ln
1 + yn 1 − t n 1 − yn ª
+
ln
2
2
2
ª
(1 + t n ) ln(1 + yn ) + (1 − t n ) ln(1 − yn ) + N ln 2
Problem 5.10 Solution
It is obvious. Suppose H is positive definite, i.e., (5.37) holds. We set v
equals to the eigenvector of H, i.e., v = ui which gives:
vT Hv = vT (Hv) = ui T λ i ui = λ i ||ui ||2
Therefore, every λ i should be positive. On the other hand, If all the eigenvalues λ i are positive, from (5.38) and (5.39), we see that H is positive definite.
Problem 5.11 Solution
It is obvious. We follow (5.35) and then write the error function in the
form of (5.36). To obtain the contour, we enforce E(w) to equal to a constant
C.
1X
E(w) = E(w∗ ) +
λ i α2i = C
2 i
We rearrange the equation above, and then obtain:
X
λ i α2i = B
i
Where B = 2C − 2E(w∗ ) is a constant. Therefore, the contours of constant error are ellipses whose axes are aligned with the eigenvector ui of
the Hessian Matrix H. The length for the jth axis is given by setting all
α i = 0, s.t.i 6= j:
s
B
αj =
λj
111
In other words, the length is inversely proportional to the square root of
the corresponding eigenvalue λ j .
Problem 5.12 Solution
If H is positive definite, we know the second term on the right side of
(5.32) will be positive for arbitrary w. Therefore, E(w∗ ) is a local minimum.
On the other hand, if w∗ is a local minimum, we have
1
E(w∗ ) − E(w) = − (w − w∗ )T H(w − w∗ ) < 0
2
In other words, for arbitrary w, (w − w∗ )T H(w − w∗ ) > 0, according to the
previous problem, we know that this means H is positive definite.
Problem 5.13 Solution
It is obvious. Suppose that there are W adaptive parameters in the network. Therefore, b has W independent parameters. Since H is symmetric,
there should be W(W + 1)/2 independent parameters in it. Therefore, there
are W + W(W + 1)/2 = W(W + 3)/2 parameters in total.
Problem 5.14 Solution
It is obvious. Since we have
E n (w ji + ²) = E n (w ji ) + ²E 0n (w ji ) +
And
²2
2
E 00n (w ji ) + O(²3 )
²2
E 00 (w ji ) + O(²3 )
2 n
We combine those two equations, which gives,
E n (w ji − ²) = E n (w ji ) − ²E 0n (w ji ) +
E n (w ji + ²) − E n (w ji − ²) = 2²E 0n (w ji ) + O(²3 )
Rearrange the equation above, we obtain what has been required.
Problem 5.15 Solution
It is obvious. The back propagation formalism starts from performing
summation near the input, as shown in (5.73). By symmetry, the forward
propagation formalism should start near the output.
Jki =
∂ yk
∂xi
=
∂ h(a k )
= h0 (a k )
∂xi
∂a k
∂xi
(∗)
Where h(·) is the activation function at the output node a k . Considering
all the units j, which have links to unit k:
∂a k
∂xi
=
X ∂a k ∂a j
j
∂a j ∂ x i
=
X
j
wk j h0 (a j )
∂a j
∂xi
(∗∗)
112
Where we have used:
ak =
X
wk j z j ,
z j = h(a j )
j
It is similar for ∂a j /∂ x i . In this way we have obtained a recursive formula
starting from the input node:
(
wl i , if there is a link from input unit i to l
∂a l
=
∂xi
0, if there isn’t a link from input unit i to l
Using recursive formula (∗∗) and then (∗), we can obtain the Jacobian
Matrix.
Problem 5.16 Solution
It is obvious. We begin by writing down the error function.
E=
N
N X
M
1 X
1 X
||yn − tn ||2 =
(yn,m − t n,m )2
2 n=1
2 n=1 m=1
Where the subscript m denotes the mthe element of the vector. Then we
can write down the Hessian Matrix as before.
H = ∇∇E =
N X
M
X
∇yn,m ∇yn,m +
n=1 m=1
N X
M
X
(yn,m − t n,m )∇∇yn,m
n=1 m=1
Similarly, we now know that the Hessian Matrix can be approximated as:
H≃
N X
M
X
bn,m bT
n,m
n=1 m=1
Where we have defined:
bn,m = ∇ yn,m
Problem 5.17 Solution
It is obvious.
∂2 E
∂wr ∂ws
=
=
∂ 1
Z Z
2(y − t)
∂wr 2
Z Z
£
(y − t)
Since we know that
Z Z
∂ y2
(y − t)
p(x, t)dxdt
∂wr ∂ws
∂y
p(x, t)dxdt
∂ws
∂ y2
∂wr ∂ws
+
∂y ∂y ¤
∂ws ∂wr
Z Z
(y − t)
=
Z
=
= 0
∂ y2
∂wr ∂ws
∂ y2
p(x, t)dxdt
p(t|x)p(x)dxdt
∂wr ∂ws
Z
©
ª
(y − t)p(t|x)dt p(x)dx
113
Note that in the last step, we have used y =
tute it into the second derivative, which gives,
∂2 E
Z Z
=
∂wr ∂ws
Z
=
R
tp(t|x) dt. Then we substi-
∂y ∂y
p(x, t)dxdt
∂ws ∂wr
∂y ∂y
p(x)dx
∂ws ∂wr
Problem 5.18 Solution
skip
By analogy with section 5.3.2, we denote wki as those parameters corresponding to skip-layer connections, i.e., it connects the input unit i with the
output unit k. Note that the discussion in section 5.3.2 is still correct and
now we only need to obtain the derivative of the error function with respect
skip
to the additional parameters wki .
∂E n
=
skip
∂wki
∂E n ∂a k
∂a k ∂wskip
= δk x i
ki
Where we have used a k = yk due to linear activation at the output unit
and:
M
X
X skip
w(2)
z
+
wki x i
yk =
j
kj
j =0
i
Where the first term on the right side corresponds to those information
conveying from the hidden unit to the output and the second term corresponds to the information conveying directly from the input to output.
Problem 5.19 Solution
The error function is given by (5.21). Therefore, we can obtain:
∇E(w) =
N ∂E
X
n=1 ∂a n
= −
= −
N
X
∇a n
∂ £
n=1 ∂a n
N © ∂(t ln y ) ∂ y
X
n
n
n
∂ yn
n=1
= −
N £t
X
n
n=1
= −
¤
t n ln yn + (1 − t n ) ln(1 − yn ) ∇a n
N £
X
yn
=
n=1
∂(1 − t n ) ln(1 − yn ) ∂ yn ª
∇a n
∂ yn
∂a n
· yn (1 − yn ) + (1 − t n )
¤
−1
· yn (1 − yn ) ∇a n
1 − yn
¤
t n (1 − yn ) − (1 − t n )yn ∇a n
n=1
N
X
∂a n
+
(yn − t n )∇a n
114
Where we have used the conclusion of problem 5.6. Now we calculate the
second derivative.
∇∇E(w) =
N ©
X
yn (1 − yn )∇a n ∇a n + (yn − t n )∇∇a n
ª
n=1
Similarly, we can drop the last term, which gives exactly what has been
asked.
Problem 5.20 Solution(waiting for update)
We begin by writing down the error function.
E(w) = −
N X
K
X
t nk ln ynk
n=1 k=1
Here we assume that the output of the network has K units in total and
there are W weights parameters in the network. WE first calculate the first
derivative:
∇E
=
N dE
X
n=1 da n
= −
=
· ∇an
N £ d
K
X
X
¤
t nk ln ynk ) · ∇an
(
n=1 da n k=1
N
X
cn · ∇an
n=1
Note that here cn = − dE/dan is a vector with size K × 1, ∇an is a matrix
with size K × W. Moreover, the operator · means inner product, which gives
∇E as a vector with size 1 × W. According to (4.106), we can obtain the jth
element of cn :
c n, j
= −
= −
= −
= −
∂
(
K
X
∂a j k=1
t nk ln ynk )
K
X
∂
(t nk ln ynk )
k=1 ∂a j
K t
X
nk
k=1 ynk
K
X
ynk (I k j − yn j )
K
X
t nk I k j +
k=1
= − t n j + yn j (
K
X
k=1
=
yn j − t n j
t nk yn j
k=1
t nk )
115
Now we calculate the second derivative:
∇∇E
N dc
X
n
(
∇an ) · ∇an + cn ∇∇an
da
n
n=1
=
Here dcn /dan is a matrix with size K × K. Therefore, the second term can
be neglected as before, which gives:
H=
N dc
X
n
∇an ) · ∇an
(
da
n
n=1
Problem 5.21 Solution
We first write down the expression of Hessian Matrix in the case of K
outputs.
K
N X
X
bn,k bT
H N,K =
n,k
n=1 k=1
Where bn,k = ∇w an,k . Therefore, we have:
H N +1,K = H N,K +
K
X
k=1
T
b N +1,k bT
N +1,k = H N,K + B N +1 B N +1
Where B N +1 = [b N +1,1 , b N +1,2 , ..., b N +1,K ] is a matrix with size W × K,
and here W is the total number of the parameters in the network. By analogy
with (5.88)-(5.89), we can obtain:
1
−1
H−
N +1,K = H N,K −
1
H−
B
BT H−1
N,K N +1 N +1 N,K
1 + BT
H−1 B
N +1 N,K N +1
(∗)
Furthermore, similarly, we have:
H N +1,K +1 = H N +1,K +
NX
+1
n=1
T
bn,K +1 bT
n,K +1 = H N +1,K + BK +1 BK +1
Where BK +1 = [b1,K +1 , b2,K +1 , ..., b N +1,K +1 ] is a matrix with size W ×(N +
1). Also, we can obtain:
1
−1
H−
N +1,K +1 = H N +1,K −
1
H−
B
BT H−1
N +1,K K +1 K +1 N +1,K
1 + BT
H−1
B
K +1 N +1,K K +1
1
Where H−
is defined by (∗). If we substitute (∗) into the expression
N +1,K
1
1
above, we can obtain the relationship between H−
and H−
.
N +1,K +1
N,K
Problem 5.22 Solution
116
We begin by handling the first case.
∂2 E n
∂
(
(2)
=
∂w(2)
∂w(2)
kj
k0 j 0
∂E n
∂wk j ∂w(2)
k0 j 0
∂
=
∂w(2)
kj
(
∂
)
∂ E n ∂ a k0
)
∂a k0 ∂w(2)0 0
k j
P
∂ E n ∂ j 0 w k0 j 0 z j 0
(
∂ a k0
∂w(2)
kj
=
∂
=
∂w(2)
kj
∂
=
∂w(2)
kj
(
(
∂E n
∂ a k0
∂E n
∂a
k0
∂w(2)
k0 j 0
)
z j0 )
)z j0 +
∂E n ∂ z j0
∂a k0 ∂w(2)
kj
∂E n ∂a k
(
)
z j0 + 0
∂a k ∂a k0 ∂w(2)
kj
∂
=
∂
=
(
∂E n
∂ a k ∂ a k0
z j z j0 M kk0
=
)z j z j0
Then we focus on the second case, and if here j 6= j 0
∂2 E n
∂w(1)
∂w(1)
ji
j0 i0
=
∂
∂w(1)
ji
(
∂E n
∂w(1)
j0 i0
)
=
X ∂ E n ∂ a k0
(
)
(1)
∂w ji k0 ∂a k0 ∂w(1)
j0 i0
=
X
=
X
∂
∂
(1)
k0 ∂w ji
(
∂E n
∂ a k0
h0 (a j0 )x i0
k0
=
X
h0 (a j0 )x i0
k0
w(2)
h0 (a j0 )x i0 )
k0 j 0
∂
∂w(1)
ji
(
∂E n
∂ a k0
w(2)
)
k0 j 0
X ∂ ∂E n (2) ∂a k
(
wk0 j0 ) (1)
∂w
k ∂ a k ∂ a k0
ji
X ∂ ∂E n (2)
h0 (a j )x i )
=
h0 (a j0 )x i0
(
wk0 j0 ) · (w(2)
kj
0
∂
a
∂
a
0
k
k
k
k
X 0
X
=
h (a j0 )x i0 M kk0 w(2)
· w(2)
h0 (a j )x i
k0 j 0
kj
X
k0
=
0
k
0
x i0 x i h (a j0 )h (a j )
XX
k0
k
w(2)
· w(2)
M kk0
k0 j 0
kj
117
When j = j 0 , similarly we have:
∂2 E n
∂w(1)
∂w(1)
ji
ji 0
=
X
k0
∂
∂E n
∂w ji
∂ a k0
(
(1)
X
∂
w(2)
h0 (a j )x i0 )
k0 j
∂E n
=
x i0
=
x i0 x i h0 (a j )h0 (a j )
(
(1)
k0 ∂w ji
∂ a k0
w(2)
)h0 (a j ) + x i0
k0 j
ji
=
X ∂E n (2) ∂ h0 (a j )
(2)
0
0
w(2)
·
w
M
+
x
(
w k0 j )
i
kk
k0 j
kj
∂w(1)
k
k0 ∂ a k0
XX
k0
=
X ∂E n (2) ∂ h0 (a j )
(
w k0 j )
∂w(1)
k0 ∂ a k0
ji
X ∂E n (2) 00
(2)
0 + x i0
x i0 x i h0 (a j )h0 (a j )
w(2)
·
w
M
(
wk0 j )h (a j )x i
0
kk
k j
kj
k0 k
k0 ∂ a k0
X X (2)
X
x i0 x i h0 (a j )h0 (a j )
wk0 j · w(2)
M kk0 + h00 (a j )x i x i0 δk0 w(2)
kj
k0 j
XX
k0 k
k0
It seems that what we have obtained is slightly different from (5.94) when
j = j 0 . However this is not the case, since the summation over k0 in the second
term of our formulation and the summation over k in the first term of (5.94) is
actually the same (i.e., they both represent the summation over all the output
units). Combining the situation when j = j 0 and j 6= j 0 , we can obtain (5.94)
just as required. Finally, we deal with the third case. Similarly we first focus
on j 6= j 0 :
∂2 E n
∂w(1)
∂w(2)
ji
k j0
=
=
=
=
=
=
=
∂
∂w(1)
ji
(
∂
(
(1)
∂w ji
∂
∂E n
∂w(2)
k j0
)
∂E n ∂a k
)
∂a k ∂w(2)0
kj
P
∂E n ∂ j0 wk j0 z j0
(
∂a k
∂w(1)
ji
∂
∂w(1)
ji
(
∂w(2)
k j0
∂E n
∂a k
)
z j0 )
X ∂ ∂ E n ∂ a k0
(
) (1)
k0 ∂a k0 ∂a k ∂w ji
X
z j0 M kk0 w(2)
h0 (a j )x i
k0 j
z j0
k0
0
x i h (a j )z j0
X
k0
M kk0 w(2)
k0 j
Note that in (5.95), there are two typos: (i)H kk0 should be M kk0 . (ii) j should
118
exchange position with j 0 in the right side of (5.95). When j = j 0 , we have:
∂2 E n
∂w(1)
∂w(2)
ji
kj
=
=
=
=
=
=
∂
(
(1)
∂E n
∂w ji ∂w(2)
kj
∂
∂w(1)
ji
(
∂
)
∂E n ∂a k
)
∂a k ∂w(2)
kj
P
∂E n ∂ j wk j z j
(
∂a k
∂w(1)
ji
∂
∂w(1)
ji
∂
∂w(1)
ji
(
(
∂w(2)
kj
∂E n
z j)
∂a k
∂E n
)z j +
∂a k
x i h0 (a j )z j
=
x i h (a j )z j
∂E n ∂ z j
∂a k w(1)
ji
X
k0
0
)
X
k0
M kk0 w(2)
+
k0 j
∂E n ∂ z j
∂a k w(1)
ji
(2)
0
M kk0 wk0 j + δk h (a j )x i
Combing these two situations, we obtain (5.95) just as required.
Problem 5.23 Solution
It is similar to the previous problem.
∂2 E n
∂ w k0 i 0 ∂ w k j
=
=
=
=
∂
(
∂E n
)
∂ w k0 i 0 ∂ w k j
∂E n
∂
(
z j)
0
0
∂wk i ∂a k
∂ w k0 i 0 ∂ ∂ E n
zj
(
)
∂ a k0 ∂ a k0 ∂ a k
z j x i0 M kk0
119
And
∂2 E n
∂
X ∂E n ∂a k
)
(
∂wk0 i0 k ∂a k ∂w ji
X ∂E n
∂
(
wk j h0 (a j )x i )
∂ w k0 i 0 k ∂ a k
X 0
∂
∂E n
h (a j )x i wk j
(
)
∂ w k0 i 0 ∂ a k
k
X 0
∂ ∂ E n a k0
h (a j )x i wk j
(
)
∂ a k0 ∂ a k w k0 i 0
k
X 0
h (a j )x i wk j M kk0 x i0
=
∂wk0 i0 ∂w ji
=
=
=
=
k
x i x i0 h0 (a j )
=
X
wk j M kk0
k
Finally, we have
∂2 E n
=
∂wk0 i0 wki
=
=
=
∂
∂E n
(
)
∂wk0 i0 ∂wki
∂
∂E n
xi )
(
∂ w k0 i 0 ∂ a k
∂ ∂ E n ∂ a k0
xi
(
)
∂ a k0 ∂ a k w k0 i 0
x i x i0 M kk0
Problem 5.24 Solution
It is obvious. According to (5.113), we have:
X
ae j =
w
e ji e
xi + w
e j0
i
=
X1
a
i
=
X
w ji · (ax i + b) + w j0 −
bX
w ji
a i
w ji x i + w j0 = a j
i
Where we have used (5.115), (5.116) and (5.117). Currently, we have
proved that under the transformation the hidden unit a j is unchanged. If
the activation function at the hidden unit is also unchanged, we have e
z j = z j.
Now we deal with the output unit e
yk :
X
e
yk =
w
ek j e
zj + w
e k0
j
=
X
cwk j · z j + cwk0 + d
j
X£
¤
w k j · z j + w k0 + d
=
c
=
c yk + d
j
120
Where we have used (5.114), (5.119) and (5.120). To be more specific,
here we have proved that the linear transformation between e
yk and yk can
be achieved by making transformation (5.119) and (5.120).
Problem 5.25 Solution
Since we know the gradient of the error function with respect to w is:
∇E = H(w − w∗ )
Together with (5.196), we can obtain:
w(τ)
= w(τ−1) − ρ ∇E
= w(τ−1) − ρ H(w(τ−1) − w∗ )
Multiplying both sides by uTj , using w j = wT u j , we can obtain:
w(jτ)
£
¤
= uTj w(τ−1) − ρ H(w(τ−1) − w∗ )
=
w(jτ−1) − ρ uTj H(w(τ−1) − w∗ )
=
w(jτ−1) − ρη j uTj (w(τ−1) − w∗ )
=
w(jτ−1) − ρη j (w(jτ−1) − w∗j )
= (1 − ρη j )w(jτ−1) + ρη j w∗j
Where we have used (5.198). Then we use mathematical deduction to
:
prove (5.197), beginning by calculating w(1)
j
w(1)
j
= (1 − ρη j )w(0)
+ ρη j w∗j
j
= ρη j w∗j
£
¤
=
1 − (1 − ρη j ) w∗j
Suppose (5.197) holds for τ, we now prove that it also holds for τ + 1.
w(jτ+1)
= (1 − ρη j w(jτ) + ρη j w∗j
£
¤
= (1 − ρη j ) 1 − (1 − ρη j )τ w∗j + ρη j w∗j
©
£
¤
ª
= (1 − ρη j ) 1 − (1 − ρη j )τ + ρη j w∗j
£
¤
=
1 − (1 − ρη j )τ+1 w∗j
Hence (5.197) holds for τ = 1, 2, .... Provided |1 − ρη j | < 1, we have (1 −
ρη j )τ → 0 as τ → ∞ ans thus w(τ) = w∗ . If τ is finite and η j >> (ρτ)−1 , the
above argument still holds since τ is still relatively large. Conversely, when
η j << (ρτ)−1 , we expand the expression above:
£
¤
|w(jτ) | = | 1 − (1 − ρη j )τ w∗j | ≈ |τρη j w∗j | << |w∗j |
121
We can see that (ρτ)−1 works as the regularization parameter α in section
3.5.3.
Problem 5.26 Solution
Based on definition or by analogy with (5.128), we have:
Ωn
1 X ∂ ynk ¯¯
(
)2
2 k ∂ξ ξ=0
1 X X ∂ ynk ∂ x i ¯¯
(
)2
2 k i ∂ x i ∂ξ ξ=0
∂
1X X
( τi
ynk )2
2 k i
∂xi
=
=
=
Where we have denoted
τi =
∂ x i ¯¯
∂ξ ξ=0
And this is exactly the form given in (5.201) and (5.202) if the nth observation ynk is denoted as yk in short. Firstly, we define α j and β j as (5.205)
shows, where z j and a j are given by (5.203). Then we will prove (5.204) holds:
αj
=
X
=
X
τi
i
τi
∂z j
∂xi
=
X
τi
∂ h(a j )
i
∂xi
∂ h(a j ) ∂a j
∂a j ∂ x i
X
∂
h0 (a j ) τ i
a j = h0 (a j )β j
∂
x
i
i
i
=
Moreover,
βj
=
X
=
=
X
τi
∂
P
i 0 w ji 0 z i 0
∂xi
X
X X
∂ z i0
τi
=
τ i w ji0
∂xi
∂xi
i
i
i0
i0
X
X
X ∂ z i0
=
w ji0 α i0
w ji0 τ i
∂xi
i
i0
i0
i
=
τi
∂a j
∂xi
i
X ∂w ji0 z i0
So far we have proved that (5.204) holds and now we aim to find a forward
propagation formula to calculate Ωn . We firstly begin by evaluating {β j } at
the input units, and then use the first equation in (5.204) to obtain {α j } at the
input units, and then the second equation to evaluate {β j } at the first hidden
layer, and again the first equation to evaluate {α j } at the first hidden layer.
We repeatedly evaluate {β j } and {α j } in this way until reaching the output
122
layer. Then we deal with (5.206):
∂Ωn
∂wrs
=
ª 1 X ∂(G yk )2
∂ ©1 X
(G yk )2 =
∂wrs 2 k
2 k ∂wrs
∂G yk
1 X ∂(G yk )2 ∂(G yk ) X
=
G yk
2 k ∂(G yk ) ∂wrs
∂wrs
k
X
£ ∂ yk ¤ X
£ ∂ yk ∂a r ¤
=
G yk G
=
αk G
∂wrs
∂a r ∂wrs
k
k
X
£
¤ X ©
ª
=
αk G δkr z s =
αk G [δkr ]z s + G [z s ]δkr
=
k
=
X
k
©
ª
αk φkr z s + αs δkr
k
Provided with the idea in section 5.3, the backward propagation formula
is easy to derive. We can simply replace E n with yk to obtain a backward
equation, so we omit it here.
Problem 5.27 Solution
Following the procedure in section 5.5.5, we can obtain:
Z
1
(τT ∇ y(x))2 p(x)dx
Ω=
2
±
Since we have τ = ∂s(x, ξ) ∂ξ and s = x + ξ, so we have τ = I. Therefore,
substituting τ into the equation above, we can obtain:
Z
1
(∇ y(x))2 p(x)dx
Ω=
2
Just as required.
Problem 5.28 Solution
The modifications only affect derivatives with respect to the weights in
the convolutional layer. The units within a feature map (indexed m) have
different inputs, but all share a common weight vector, w(m) . Therefore, we
can write:
( m)
X ∂E n ∂a j
X ( m) ( m)
∂E n
=
=
δ j z ji
( m)
( m)
∂w(im)
j ∂a j ∂w i
j
Here a(jm) denotes the activation of the jth unit in th mth feature map,
whereas w(im) denotes the ith element of the corresponding feature vector
)
and finally z(im
denotes the ith input for the jth unit in the mth feature map.
j
Note that δ(jm) can be computed recursively from the units in the following
layer.
Problem 5.29 Solution
123
It is obvious. Firstly, we know that:
ª
wi − µ j
π j N (w i |µ j , σ2j ) = −π j
N (w i |µ j , σ2j )
∂w i
σ2j
∂ ©
We now derive the error function with respect to w i :
e
∂E
∂w i
=
=
=
=
=
=
=
=
∂E
∂w i
∂E
∂w i
∂E
∂w i
∂E
∂w i
∂E
∂w i
∂E
∂w i
+
∂λΩ(w)
−λ
−λ
∂w i
∂
(
X
∂w i
i
(
∂
∂w i
Ã
ln
M
X
!)
π j N (w i |µ j , σ2j )
j =1
Ã
ln
− λ PM
!)
M
X
π j N (w i |µ j , σ2j )
j =1
∂
1
2 ∂w
i
j =1 π j N (w i |µ j , σ j )
+ λ PM
+λ
(
1
M
X
(
M
X
)
π j N (w i |µ j , σ2j )
j =1
πj
2
j =1 π j N (w i |µ j , σ j ) j =1
PM
w i −µ j
2
j =1 π j σ2 N (w i |µ j , σ j )
wi − µ j
σ2j
)
N (w i |µ j , σ2j )
j
2
k π k N (w i |µ k , σ k )
M
π j N (w i |µ j , σ2j ) w i − µ j
X
∂E
+λ
P
2
∂w i
σ2j
j =1 k π k N (w i |µ k , σ k )
∂E
∂w i
+λ
P
M
X
j =1
γ j (w i )
wi − µ j
σ2j
Where we have used (5.138) and defined (5.140).
Problem 5.30 Solution
Is is similar to the previous problem. Since we know that:
ª
wi − µ j
∂ ©
π j N (w i |µ j , σ2j ) = π j
N (w i |µ j , σ2j )
∂µ j
σ2j
124
We can derive:
e
∂E
∂µ j
=
∂λΩ(w)
∂µ j
= −λ
∂
(
X
∂µ j
i
X ∂
= −λ
i ∂µ j
= −λ
!)
π j N (w i |µ j , σ2j )
j =1
(
ln
Ã
M
X
!)
π j N (w i |µ j , σ2j )
j =1
∂
2 ∂µ
j
j =1 π j N (w i |µ j , σ j )
PM
(
M
X
)
π j N (w i |µ j , σ2j )
j =1
wi − µ j
N (w i |µ j , σ2j )
πj
2
2
σ
π
N
(w
|
µ
,
σ
)
j
i
j
i
j
j =1
j
2
π j N (w i |µ j , σ j )
X
X
µ j − wi
µ j − wi
= λ γ j (w i )
λ PK
2
2
σj
σ2j
i
i
k=1 π k N (w i |µ k , σ k )
= −λ
=
ln
M
X
1
X
i
Ã
1
X
PM
Note that there is a typo in (5.142). The numerator should be µ j − w i
instead of µ i − w j . This can be easily seen through the fact that the mean and
variance of the Gaussian Distribution should have the same subindex and
since σ j is in the denominator, µ j should occur in the numerator instead of
µi .
Problem 5.31 Solution
It is similar to the previous problem. Since we know that:
Ã
!
ª
1 (w i − µ j )2
∂ ©
2
π j N (w i |µ j , σ2j )
π j N (w i |µ j , σ j ) = −
+
∂σ j
σj
σ3j
125
We can derive:
e
∂E
∂σ j
=
∂λΩ(w)
∂σ j
= −λ
∂
(
X
∂σ j
i
X ∂
= −λ
i ∂σ j
= −λ
=
=
=
=
ln
M
X
!)
π j N (w i |µ j , σ2j )
j =1
(
Ã
ln
M
X
!)
π j N (w i |µ j , σ2j )
j =1
1
X
i
Ã
∂
2 ∂σ
j
j =1 π j N (w i |µ j , σ j )
PM
(
M
X
)
π j N (w i |µ j , σ2j )
j =1
o
X
∂ n
1
π j N (w i |µ j , σ2j )
−λ P M
2 ∂σ
j
i
j =1 π j N (w i |µ j , σ j )
Ã
!
X
1
1 (w i − µ j )2
λ PM
−
π j N (w i |µ j , σ2j )
3
2) σ
σ
π
N
(w
|
µ
,
σ
j
i j
i
j
j =1 j
j
!
Ã
2
π j N (w i |µ j , σ j )
X
1 (w i − µ j )2
λ PM
−
2) σ
σ3j
π
N
(w
|
µ
,
σ
j
i
i
k
k
k=1
k
Ã
!
X
1 (w i − µ j )2
−
λ γ j (w i )
σj
σ3j
i
Just as required.
Problem 5.32 Solution
It is trivial. We begin by verifying (5.208) when j 6= k.
½
¾
∂πk
∂
exp(η k )
=
P
∂η j
∂η j
k exp(η k )
− exp(η k )exp(η j )
=
£P
¤2
k exp(η k )
= −π j πk
And if now we have j = k:
¾
½
∂πk
∂
exp(η k )
=
P
∂η k
∂η k
k exp(η k )
¤
£P
exp(η k ) k exp(η k ) − exp(η k )exp(η k )
=
¤2
£P
k exp(η k )
= πk − πk πk
If we combine these two cases, we can easily see that (5.208) holds. Now
126
we prove (5.147).
e
∂E
∂η j
= λ
∂Ω(w)
= −λ
∂η j
∂
(
X
∂η j
i
X ∂
= −λ
i ∂η j
= −λ
=
π j N (w i |µ j , σ2j )
j =1
(
ln
M
X
))
π j N (w i |µ j , σ2j )
j =1
∂
2 ∂η
j
j =1 π j N (w i |µ j , σ j )
PM
(
M
X
)
πk N (w i |µk , σ2k )
k=1
M ∂ ©
X
ª
πk N (w i |µk , σ2k )
π j N (w i |µ j , σ2 ) k=1 ∂η j
1
PM
j =1
j
M
X
ª ∂πk
∂ ©
πk N (w i |µk , σ2k )
∂η j
π j N (w i |µ j , σ2 ) k=1 ∂πk
1
PM
j =1
j
M
X
1
N (w i |µk , σ2k )(δ jk π j − π j πk )
2)
π
N
(w
|
µ
,
σ
i j
i
j =1 j
j k=1
(
)
M
X
X
1
2
2
πk N (w i |µk , σk ))
−λ P M
π j N (w i |µ j , σ j ) − π j
2
i
k=1
j =1 π j N (w i |µ j , σ j )
)
(
P
π j N (w i |µ j , σ2j )
X
π j kM=1 πk N (w i |µk , σ2k ))
− PM
−λ
PM
2
2
i
j =1 π j N (w i |µ j , σ j )
j =1 π j N (w i |µ j , σ j )
X©
X
ª
©
ª
−λ
γ j (w i ) − π j = λ
π j − γ j (w i )
= −λ
=
))
1
X
i
=
ln
(
X
i
= −λ
M
X
X
i
= −λ
(
X
PM
i
i
Just as required.
Problem 5.33 Solution
It is trivial. We set the attachment point of the lower arm with the ground
as the origin of the coordinate. We first aim to find the vertical distance from
the origin to the target point, and this is also the value of x2 .
x2
=
L 1 sin(π − θ1 ) + L 2 sin(θ2 − (π − θ1 ))
=
L 1 sin θ1 − L 2 sin(θ1 + θ2 )
Similarly, we calculate the horizontal distance from the origin to the target point.
x1
= −L 1 cos(π − θ1 ) + L 2 cos(θ2 − (π − θ1 ))
=
L 1 cos θ1 − L 2 cos(θ1 + θ2 )
From these two equations, we can clearly see the ’forward kinematics’ of
the robot arm.
127
Problem 5.34 Solution
By analogy with (5.208), we can write:
∂πk (x)
∂aπj
= δ jk π j (x) − π j (x)πk (x)
Using (5.153), we can see that:
)
(
K
X
2
πk N (tn |µk , σk )
E n = − ln
k=1
Therefore, we can derive:
)
(
K
X
∂
∂E n
2
πk N (tn |µk , σk )
= − π ln
∂aπj
∂a j
k=1
(
)
K
X
1
∂
2
= − PK
πk N (tn |µk , σk )
π
πk N (tn |µk , σ2 ) ∂a j k=1
k=1
= − PK
k
1
K ∂π
X
k
2
π N (t n |µ k , σ k )
2
∂a j
k=1 π k N (t n |µ k , σ k ) k=1
= − PK
1
K £
X
¤
δ jk π j (xn ) − π j (xn )πk (xn ) N (tn |µk , σ2k )
2
k=1 π k N (t n |µ k , σ k ) k=1
(
K
X
)
πk (xn )N (tn |µk , σ2k )
π j (xn )N (tn |µ j , σ2j ) − π j (xn )
2)
π
N
(t
|
µ
,
σ
n
k
k
=
1
k
k=1
k
(
)
K
X
1
2
2
−π j (xn )N (tn |µ j , σ j ) + π j (xn )
πk (xn )N (tn |µk , σk )
PK
2)
π
N
(t
|
µ
,
σ
n
k
k
=
1
k
k=1
k
= − PK
=
1
And if we denoted (5.154), we will have:
∂E n
∂aπj
= −γ j + π j
Note that our result is slightly different from (5.155) by the subindex. But
there are actually the same if we substitute index j by index k in the final
expression.
Problem 5.35 Solution
We deal with the derivative of error function with respect to µk instead,
which will give a vector as result. Furthermore, the lth element of this vector
will be what we have been required. Since we know that:
ª tn − µk
∂ ©
πk N (tn |µk , σ2k ) =
πk N (tn |µk , σ2k )
2
∂µk
σ
k
128
One thing worthy noticing is that here we focus on the isotropic case as
stated in page 273 of the textbook. To be more precise, N (tn |µk , σ2k ) should
be N (tn |µk , σ2k I). Provided with the equation above, we can further obtain:
∂E n
∂µk
=
(
∂
− ln
∂µk
K
X
)
πk N (tn |µk , σ2k )
k=1
= − PK
(
∂
1
2 ∂µ
k
k=1 π k N (t n |µ k , σ k )
1
= − PK
2
k=1 π k N (t n |µ k , σ k )
= −γk
·
K
X
k=1
tn − µk
σ2k
)
πk N (tn |µk , σ2k )
πk N (tn |µk , σ2k )
tn − µk
σ2k
Hence noticing (5.152), the lth element of the result above is what we are
required.
µkl − tl
∂E n
∂E n
= γk
µ =
∂µkl
σ2k
∂a
kl
Problem 5.36 Solution
Similarly, we know that:
ª
∂ ©
πk N (tn |µk , σ2k ) =
∂σk
(
)
D ||tn − µk ||2
−
+
πk N (tn |µk , σ2k )
3
σk
σk
Therefore, we can obtain:
(
)
K
X
∂
∂E n
2
=
− ln
πk N (tn |µk , σk )
∂σk
∂σk
k=1
(
)
K
X
1
∂
= − PK
πk N (tn |µk , σ2k )
2 ) ∂σ
π
N
(t
|
µ
,
σ
k
n
k
k
=
1
k
k=1
k
)
(
1
D ||tn − µk ||2
= − PK
+
πk N (tn |µk , σ2k )
· −
3
2
σ
σk
k
k=1 π k N (t n |µ k , σ k )
(
)
2
D ||tn − µk ||
= −γk −
+
σk
σ3k
Note that there is a typo in (5.157) and the underlying reason is that:
|σ2k ID ×D | = (σ2k )D
Problem 5.37 Solution
First we know two properties for the Gaussian distribution N (t|µ, σ2 I):
Z
E[t] = tN (t|µ, σ2 I) dt = µ
129
And
E[||t||2 ] =
Z
||t||2 N (t|µ, σ2 I) dt = Lσ2 + ||µ||2
Where we have used E[tT At] = Tr[Aσ2 I] + µT Aµ by setting A = I. This
property can be found in Matrixcookbook eq(378). Here L is the dimension of
t. Noticing (5.148), we can write:
Z
E[t|x] =
tp(t|x) dt
Z
t
=
K
X
πk N (t|µk , σ2k ) dt
k=1
K
X
=
πk
Z
tN (t|µk , σ2k ) dt
k=1
K
X
=
πk µk
k=1
Then we prove (5.160).
¡
¢
s2 (x) = E[||t − E[t|x]||2 |x] = E[ t2 − 2tE[t|x] + E[t|x]2 |x]
= E[t2 |x] − E[2tE[t|x]|x] + E[t|x]2 = E[t2 |x] − E[t|x]2
Z
K
K
X
X
=
||t||2
πk N (µk , σ2k ) dt − ||
πl µl ||2
=
K
X
πk
k=1
l =1
Z
K
X
||t||2 N (µk , σ2k ) dt − ||
k=1
=
K
X
l =1
πk (Lσ2k + ||µk ||2 ) − ||
k=1
=
L
=
=
=
L
L
L
L
πk σ2k +
πl µl ||2
K
X
K
X
πk ||µk ||2 − ||
k=1
πk σ2k +
K
X
k=1
k=1
K
X
K
X
πk σ2k +
K
X
πl µl ||2
l =1
πk ||µk ||2 − 2 × ||
K
X
πl µl ||2 + 1 × ||
l =1
2
πk ||µk || − 2(
K
X
πl µl )(
K
X
πl µl ||2
l =1
K
X
Ã
πk µk ) +
K
X
k=1
k=1
l =1
k=1
k=1
K
X
K
X
K
X
K
X
K
X
πk σ2k +
k=1
k=1
K
X
K
X
πk σ2k +
k=1
=
K
X
l =1
K
X
k=1
=
πl µl ||2
K
X
k=1
πk ||µk ||2 − 2(
πl µl )(
l =1
πk ||µk −
k=1
Ã
πk Lσ2k + ||µk −
K
X
k=1
πk µk ) +
k=1
!
πk ||
πk ||
K
X
πl µl ||2
l =1
K
X
πl µl ||2
l =1
πl µl ||2
l =1
K
X
!
πl µl ||2
l =1
Note that there is a typo in (5.160), i.e., the coefficient L in front of σ2k is
missing.
130
Problem 5.38 Solution
From (5.167) and (5.171), we can write down the expression for the predictive distribution:
Z
p(t|x, D, α, β) =
p(w|D, α, β)p(t|x, w, β) dw
Z
≈
q(w|D)p(t|x, w, β) dw
Z
=
N (w|wMAP , A−1 )N (t|gT w − gT wMAP + y(x, wMAP ), β−1 ) dw
Note here p(t|x, w, β) is given by (5.171) and q(w|D) is the approximation
to the posterior p(w|D, α, β), which is given by (5.167). Then by analogy with
(2.115), we first deal with the mean of the predictive distribution:
mean = gT w − gT wMAP + y(x, wMAP )|w = wMAP
=
y(x, wMAP )
Then we deal with the covariance matrix:
Covariance matrix = β−1 + gT A−1 g
Just as required.
Problem 5.39 Solution
Using Laplace Approximation, we can obtain:
p(D |w, β)p(w|α) =
n
o
p(D |wMAP , β)p(wMAP |α) exp −(w − wMAP )T A(w − wMAP )
Then using (5.174), (5.162) and (5.163), we can obtain:
Z
p(D |α, β) =
p(D |w, β)p(w, α) dw
Z
n
o
=
p(D |wMAP , β)p(wMAP |α) exp −(w − wMAP )T A(w − wMAP ) dw
=
=
p(D |wMAP , β)p(wMAP |α)
N
Y
(2π)W /2
|A|1/2
N (t n | y(xn , wMAP ), β−1 )N (wMAP |0, α−1 I)
n=1
(2π)W /2
|A|1/2
If we take logarithm of both sides, we will obtain (5.175) just as required.
Problem 5.40 Solution
For a k-class classification problem, we need to use softmax activation
function and also the error function is now given by (5.24). Therefore, the
131
Hessian matrix should be derived from (5.24) and the cross entropy in (5.184)
will also be replaced by (5.24).
Problem 5.41 Solution
By analogy to Prob.5.39, we can write:
p(D |α) = p(D |wMAP )p(wMAP |α)
(2π)W /2
|A|1/2
Since we know that the prior p(w|α) follows a Gaussian distribution, i.e.,
(5.162), as stated in the text. Therefore we can obtain:
1
ln |A| + const
2
α
W
1
= ln p(D |wMAP ) − wT w +
ln α − ln |A| + const
2
2
2
1
W
ln α − ln |A| + const
= −E(wMAP ) +
2
2
ln p(D |α) = ln p(D |wMAP ) + ln p(wMAP |α) −
Just as required.
0.6
Kernel Methods
Problem 6.1 Solution
Recall that in section.6.1, a n can be written as (6.4). We can derive:
an
1
= − {wT φ(xn ) − t n }
λ
1
= − {w1 φ1 (xn ) + w2 φ2 (xn ) + ... + w M φ M (xn ) − t n }
λ
w1
w2
wM
tn
= −
φ1 (xn ) −
φ2 (xn ) − ... −
φ M (xn ) +
λ
λ
λ
λ
w1
w2
wM
= (c n −
)φ1 (xn ) + (c n −
)φ2 (xn ) + ... + (c n −
)φ M (xn )
λ
λ
λ
Here we have defined:
cn =
t n /λ
φ1 (xn ) + φ2 (xn ) + ... + φ M (xn )
From what we have derived above, we can see that a n is a linear combination of φ(xn ). What’s more, we first substitute K = ΦΦT into (6.7), and
then we will obtain (6.5). Next we substitute (6.3) into (6.5) we will obtain
(6.2) just as required.
Problem 6.2 Solution
132
If we set w(0) = 0 in (4.55), we can obtain:
w(τ+1) =
N
X
η c n t n φn
n=1
where N is the total number of samples and c n is the times that t n φn has
been added from step 0 to step τ + 1. Therefore, it is obvious that we have:
w=
N
X
αn t n φn
n=1
We further substitute the expression above into (4.55), which gives:
N
X
α(nτ+1) t n φn =
N
X
α(nτ) t n φn + η t n φn
n=1
n=1
In other words, the update process is to add learning rate η to the coefficient αn corresponding to the misclassified pattern xn , i.e.,
α(nτ+1) = α(nτ) + η
Now we similarly substitute it into (4.52):
y(x) =
=
f ( wT φ(x) )
N
X
f(
α n t n φT
n φ(x) )
n=1
=
f(
N
X
αn t n k(xn , x) )
n=1
Problem 6.3 Solution
We begin by expanding the Euclidean metric.
||x − xn ||2
= (x − xn )T (x − xn )
= (xT − xT
n )(x − x n )
T
= xT x − 2xT
n x + xn xn
Similar to (6.24)-(6.26), we use a nonlinear kernel k(xn , x) to replace xT
n x,
which gives a general nonlinear nearest-neighbor classifier with cost function
defined as:
k(x, x) + k(xn , xn ) − 2k(xn , x)
Problem 6.4 Solution
To construct such a matrix, let us suppose the two eigenvalues are 1 and
2, and the matrix has form:
·
¸
a b
c d
133
Therefore, based on the definition of eigenvalue, we have two equations:
½
(a − 2)(d − 2) = bc (1)
(a − 1)(d − 1) = bc (2)
(2)-(1), yielding:
a+d =3
Therefore, we set a = 4 and d = −1. Then we substitute them into (1), and
thus we see:
bc = −6
Finally, we choose b = 3 and c = −2. The constructed matrix is:
·
¸
4
3
−2 −1
Problem 6.5 Solution
Since k 1 (x, x0 ) is a valid kernel, it can be written as:
k 1 (x, x0 ) = φ(x)T φ(x0 )
We can obtain:
k(x, x0 ) = ck 1 (x, x0 ) =
£p
cφ(x)
¤T £p
¤
cφ(x0 )
Therefore, (6.13) is a valid kernel. It is similar for (6.14):
£
¤T £
¤
k(x, x0 ) = f (x)k 1 (x, x0 ) f (x0 ) = f (x)φ(x)
f (x0 )φ(x0 )
Just as required.
Problem 6.6 Solution
We suppose q(x) can be written as:
q(x) = a n x n + a n−1 x n−1 + ... + a 1 x + a 0
We now obtain:
n
k(x, x0 ) = a n k 1 (x, x0 ) + a n−1 k 1 (x, x0 )
n−1
+ ... + a 1 k 1 (x, x0 ) + a 0
By repeatedly using (6.13), (6.17) and (6.18), we can easily verify k(x, x0 )
is a valid kernel. For (6.16), we can use Taylor expansion, and since the
coefficients of Taylor expansion are all positive, we can similarly prove its
validity.
Problem 6.7 Solution
To prove (6.17), we will use the property stated below (6.12). Since we
know k 1 (x, x0 ) and k 2 (x, x0 ) are valid kernels, their Gram matrix K1 and K2
134
are both positive semidefinite. Given the relation (6.12), it can be easily
shown K = K1 + K2 is also positive semidefinite and thus k(x, x0 ) is also a
valid kernel.
To prove (6.18), we assume the map function for kernel k 1 (x, x0 ) is φ(1) (x),
and similarly φ(2) (x) for k 2 (x, x0 ). Moreover, we further assume the dimension
of φ(1) (x) is M, and φ(2) (x) is N. We expand k(x, x0 ) based on (6.18):
k(x, x0 ) =
k 1 (x, x0 )k 2 (x, x0 )
= φ(1) (x)T φ(1) (x0 )φ(2) (x)T φ(2) (x0 )
N
M
X
X
(1) 0
φ(2)
(x)φ(2)
(x0 )
φ(1)
(x)
φ
(x
)
=
j
j
i
i
j =1
i =1
=
ih
i
M X
N h
X
(2)
(1) 0 (2) 0
φ(1)
(x)
φ
(x)
φ
(x
)
φ
(x
)
i
j
i
j
i =1 j =1
=
MN
X
φk (x)φk (x0 ) = φ(x)T φ(x0 )
k=1
where φ(1)
(x) is the ith element of φ(1) (x), and φ(2)
(x) is the jth element
i
j
of φ(2) (x). To be more specific, we have proved that k(x, x0 ) can be written as
φ(x)T φ(x0 ). Here φ(x) is a MN ×1 column vector, and the kth (k = 1, 2, ..., MN)
element is given by φ(1)
(x) × φ(2)
(x). What’s more, we can also express i, j in
i
j
terms of k:
i = (k − 1) ® N + 1 and j = (k − 1) ¯ N + 1
where ® and ¯ means integer division and remainder, respectively.
Problem 6.8 Solution
For (6.19) we suppose k 3 (x, x0 ) = g(x)T g(x0 ), and thus we have:
k(x, x0 ) = k 3 (φ(x), φ(x0 )) = g(φ(x))T g(φ(x0 )) = f (x)T f (x0 )
where we have denoted g(φ(x)) = f (x) and now it is obvious that (6.19)
holds. To prove (6.20), we suppose x is a N × 1 column vector and A is a N × N
symmetric positive semidefinite matrix. We know that A can be decomposed
to QBQT . Here Q is a N × Northogonal matrix, and B is a N × N diagonal
matrix whose elements are no less than 0. Now we can derive:
k(x, x0 ) = xT Ax0 = xT QBQT x0 = (QT x)T B(QT x0 ) = yT By0
N
N p
p
X
X
0
0
=
B ii yi yi =
( B ii yi )( B ii yi ) = φ(x)T φ(x0 )
i =1
i =1
To be more specific, we have proved that k(x, x0 ) = φ(x)T φ(x0 ), and here
φ
vector, whose ith (i = 1, 2, ..., N) element is given by
p(x) is a N ×p1 column
B ii yi , i.e., B ii (QT x) i .
135
Problem 6.9 Solution
To prove (6.21), let’s first expand the expression:
0
k(x, x0 ) =
0
k a (xa , xa ) + k b (xb , xb )
M
X
=
i =1
MX
+N
=
0
φ(ia) (xa )φ(ia) (xa ) +
N
X
0
j =1
φ(ib) (xb )φ(ib) (xb )
0
φk (x)φk (x ) = φ(x)T φ(x0 )
k=1
where we have assumed the dimension of xa is M and the dimension of
xb is N. The mapping function φ(x) is a (M + N) × 1 column vector, whose kth
(k = 1, 2, ..., M + N) element φk (x) is:
(
φ(ka) (x)
1≤k≤M
φk (x) =
φ(kb−) M (xa ) M + 1 ≤ k ≤ M + N
(6.22) is quite similar to (6.18). We follow the same procedure:
k(x, x0 ) =
=
0
M
X
i =1
=
0
φ(ia) (xa )φ(ia) (xa )
M X
N h
X
i =1 j =1
=
0
k a (xa , xa )k b (xb , xb )
MN
X
N
X
j =1
0
φ(jb) (xb )φ(jb) (xb )
φ(ia) (xa )φ(jb) (xb )
ih
i
0
0
φ(ia) (xa )φ(jb) (xb )
φk (x)φk (x0 ) = φ(x)T φ(x0 )
k=1
By analogy to (6.18), the mapping function φ(x) is a MN ×1 column vector,
whose kth (k = 1, 2, ..., MN) element φk (x) is:
φk (x) = φ(ia) (xa ) × φ(jb) (xb )
To be more specific, xa is the sub-vector of x made up of the first M element of x, and xb is the sub-vector of x made up of the last N element of x.
What’s more, we can also express i, j in terms of k:
i = (k − 1) ® N + 1
and
j = (k − 1) ¯ N + 1
where ® and ¯ means integer division and remainder, respectively.
Problem 6.10 Solution
According to (6.9), we have:
T
−1
y(x) = k(x) (K + λI N )
T
t = k(x) a =
N
X
n=1
"
f (xn ) · f (x) · a n =
N
X
n=1
#
f (xn ) · ·a n f (x)
136
We see that if we choose k(x, x0 ) = f (x) f (x0 ) we will always find a solution
y(x) proportional to f (x).
Problem 6.11 Solution
We follow the hint.
k(x, x0 ) = exp(−xT x/2σ2 ) · exp(xT x0 /σ2 ) · exp(−(x0 )T x0 /2σ2 )


xT x0 2
T 0
(
)
x
x
2
= exp(−xT x/2σ2 ) · 1 + 2 + σ
+ · · · · exp(−(x0 )T x0 /2σ2 )
2!
σ
= φ(x)T φ(x0 )
where φ(x) is a column vector with infinite dimension. To be more specific, (6.12) gives a simple example on how to decompose (xT x0 )2 . In our case,
we can also decompose (xT x0 )k , k = 1, 2, ..., ∞ in the similar way. However,
since k → ∞, i.e., the decomposition will consist monomials with infinite degree. Thus, there will be infinite terms in the decomposition and the feature
mapping function φ(x) will have infinite dimension.
Problem 6.12 Solution
First, let’s explain the problem a little bit. According to (6.27), what we
need to prove here is:
k(A 1 , A 2 ) = 2| A 1 ∩ A 2 | = φ(A 1 )T φ(A 2 )
The biggest difference from the previous problem is that φ(A) is a 2|D | × 1
column vector and instead of indexed by 1, 2, ..., 2|D | here we index it by {U |U ⊆
D } (Note that {U |U ⊆ D } is all the possible subsets of D and thus there are
2|D | elements in total). Therefore, according to (6.95), we can obtain:
φ(A 1 )T φ(A 2 ) =
X
φU (A 1 )φU (A 2 )
U ⊆D
By using the summation, we actually iterate through all the possible subsets of D. If and only if the current iterating subset U is a subset of both A 1
and A 2 simultaneously, the current adding term equals to 1. Therefore, we
actually count how many subsets of D is in the intersection of A 1 and A 2 .
Moreover, since A 1 and A 2 are both defined in the subset space of D, what
we have deduced above can be written as:
φ(A 1 )T φ(A 2 ) = 2| A 1 ∩ A 2 |
Just as required.
Problem 6.13 SolutionWait for update
Problem 6.14 Solution
137
Since the covariance matrix S is fixed, according to (6.32) we can obtain:
µ
¶
∂
1
T −1
g(µ, x) = ∇µ ln p(x|µ) =
− (x − µ) S (x − µ) = S−1 (x − µ)
∂µ
2
Therefore, according to (6.34), we can obtain:
h
i
h
i
F = Ex g(µ, x)g(µ, x)T = S−1 Ex (x − µ)(x − µ)T S−1
Since x ∼ N (x|µ, S), we have:
h
i
Ex (x − µ)(x − µ)T = S
So we obtain F = S−1 and then according to (6.33), we have:
k(x, x0 ) = g(µ, x)T F−1 g(µ, x0 ) = (x − µ)T S−1 (x0 − µ)
Problem 6.15 Solution
We rewrite the problem. What we are required to prove is that the Gram
matrix K:
·
¸
k 11 k 12
K=
,
k 21 k 22
where k i j (i, j = 1, 2) is short for k(x i , x j ), should be positive semidefinite. A
positive semidefinite matrix should have positive determinant, i.e.,
k 12 k 21 ≤ k 11 k 22 .
Using the symmetric property of kernel, i.e.,k 12 = k 21 , we obtain what
has been required.
Problem 6.16 Solution
Based on the total derivative of function f , we have:
³
´ X
N
f (w + ∆w)T φ1 , (w + ∆w)T φ2 , ..., (w + ∆w)T φ N =
∂f
T
n=1 ∂(w φ n )
· ∆wT φn
Which can be further written as:
³
T
T
T
´
f (w + ∆w) φ1 , (w + ∆w) φ2 , ..., (w + ∆w) φ N =
"
N
X
#
∂f
n=1 ∂(w
Tφ
n)
· φT
n
∆w
Note that here φn is short for φ(xn ). Based on the equation above, we can
obtain:
N
X
∂f
∇w f =
· φT
n
Tφ )
∂
(w
n=1
n
138
Now we focus on the derivative of function g with respect to w:
∇w g =
∂g
∂(wT w)
· 2wT
In order to find the optimal w, we set the derivative of J with respect to
w equal to 0, yielding:
∇w J = ∇w f + ∇w g =
N
X
∂f
n=1
∂(wT φ
n)
· φT
n +
∂g
∂(wT w)
· 2wT = 0
Rearranging the equation above, we can obtain:
N
1 X
∂f
·φ
2a n=1 ∂(wT φn ) n
w=
∂g
Where we have defined: a = 1 ÷ ∂(wT w) , and since g is a monotonically
increasing function, we have a > 0.
Problem 6.17 Solution
We consider a variation in the function y(x) of the form:
y(x) → y(x) + ²η(x)
Substituting it into (6.39) yields:
E[y + ²η] =
N
1 X
2 n=1
Z
N
1 X
2 n=1
Z
=
=
E[y] + ²
©
y + ²η − t n
ª2
v(ξ)d ξ
©
ª
(y − t n )2 + 2 · (²η) · (y − t n ) + (²η)2 v(ξ)d ξ
N Z
X
{ y − t n } ηvd ξ + O(²2 )
n=1
Note that here y is short for y(xn + ξ), η is short for η(xn + ξ) and v is short
for v(ξ) respectively. Several clarifications must be made here. What we have
done is that we vary the function y by a little bit (i.e., ²η) and then we expand
the corresponding error with respect to the small variation ². The coefficient
before ² is actually the first derivative of the error E[y + ²η] with respect to
² at ² = 0. Since we know that y is the optimal function that can make E
the smallest, the first derivative of the error E[y + ²η] should equal to zero at
² = 0, which gives:
N Z
X
n=1
{ y(xn + ξ) − t n } η(xn + ξ)v(ξ)d ξ = 0
139
Now we are required to find a function y that can satisfy the equation
above no matter what η is. We choose:
η(x) = δ(x − z)
This allows us to evaluate the integral:
N Z
X
{ y(xn + ξ) − t n } η(xn + ξ)v(ξ)d ξ =
N
X
{ y(z) − t n } v(z − xn )
n=1
n=1
We set it to zero and rearrange it, which finally gives (6.40) just as required.
Problem 6.18 Solution
According to the main text below Eq (6.48), we know that f (x, t), i.e., f (z),
follows a zero-mean isotropic Gaussian:
f (z) = N (z|0, σ2 I)
Then f (x − xm , t − t m ), i.e., f (z − zm ) should also satisfy a Gaussian distribution:
f (z − zm ) = N (z|zm , σ2 I)
Where we have defined:
zm = (xm , t m )
R
The integral f (z − zm )dt corresponds to the marginal distribution with
respect to the remaining variable x and, thus, we obtain:
Z
f (z − zm ) dt = N (x| xm , σ2 )
We substitute all the expressions into Eq (6.48), which gives:
P
2
p(t, x)
n N (z|z m , σ I)
p(t| x) = R
= P
2
p(t, x) dt
m N (x| x m , σ )
¡ 1
¢
P 1
T 2 −1
n 2πσ2 exp − 2 (z − z n ) (σ I) (z − z n )
³
´
=
P
1
1
2
(x
−
x
)
exp
−
m
m (2πσ2 )1/2
2σ 2
³
´
³
´
P 1
1
1
2
2
n 2πσ2 exp − 2σ2 (x − x n ) exp − 2σ2 (t − t n )
³
´
=
P
1
1
2
exp
−
(x
−
x
)
m
m (2πσ2 )1/2
2σ2
³
´
µ
¶
p 1
exp − 2σ1 2 (x − xn )2
X
1
1
2πσ2
2
³
´
·
exp
−
(t
−
t
)
=
p
n
P
1
1
2σ 2
2
n
2πσ2
m (2πσ2 )1/2 exp − 2σ2 (x − x m )
X
=
πn · N (t| t n , σ2 )
n
140
Where we have defined:
³
´
exp − 2σ1 2 (x − xn )2
³
´
πn = P
1
2
exp
−
(x
−
x
)
m
m
2σ 2
We also observe that:
X
πn = 1
n
Therefore, the conditional distribution p(t| x) is given by a Gaussian Mixture. Similarly, we attempt to find a specific form for Eq (6.46):
R
f (x − xn , t) dt
k(x, xn ) = P R
m f (x − x m , t) dt
=
P
N (x| xn , σ2 )
2
m N (x| x m , σ )
= πn
In other words, the conditional distribution can be more precisely written
as:
p(t| x) =
X
k(x, xn ) · N (t| t n , σ2 )
n
Thus its mean is given by:
E[t| x] =
X
k(x, xn ) · t n
n
Its variance is given by:
var[t| x] = E[(t| x)2 ] − E[t| x]2
µ
¶2
X
X
=
k(x, xn ) · (t2n + σ2 ) −
k(x, xn ) · t n
n
n
Problem 6.19 Solution
Similar to Prob.6.17, it is straightforward to show that:
y(x) =
X
t n k(x, xn )
n
Where we have defined:
g(xn − x)
k(x, xn ) = P
n g(x n − x)
Problem 6.20 Solution
Since we know that t N +1 = (t 1 , t 2 , ..., t N , t N +1 )T follows a Gaussian distribution, i.e., t N +1 ∼ N (t N +1 |0, C N +1 ) given in Eq (6.64), if we rearrange its
141
order by putting the last element (i.e., t N +1 ) to the first position, denoted as
t̄ N +1 , it should also satisfy a Gaussian distribution:
t̄ N +1 = (t N +1 , t 1 , ..., t 2 , t N )T ∼ N (t̄ N +1 |0, C̄ N +1 )
Where we have defined:
µ
C̄ N +1 =
c kT
k CN
¶
Where k and c have been given in the main text below Eq (6.65). The
conditional distribution p(t N +1 |t N ) should also be a Gaussian. By analogy to
Eq (2.94)-(2.98), we can simply treat t N +1 as xa , t N as xb , c as Σaa , k as Σba ,
kT as Σab and C N as Σbb . Substituting them into Eq (2.79) and Eq (2.80)
yields:
Λaa = (c − kT C−N1 k)−1
And:
Λab = −(c − kT C−N1 k)−1 kT C−N1
Then we substitute them into Eq (2.96) and (2.97), yields:
1
p(t N +1 |t N ) = N (µa|b , Λ−
aa )
For its mean µa|b , we have:
³
´ h
i
1
T −1 −1 T −1
µa|b = 0 − c − kT C−
k
·
−
(c
−
k
C
k)
k
C
N
N · (t N − 0)
N
1
= kT C−
N t N = m(x N +1 )
1
Similarly, for its variance Λ−
aa (Note that here since t N +1 is a scalar, the
mean and the covariance matrix actually degenerate to one dimension case),
we have:
1
T −1
2
Λ−
aa = c − k C N k = σ (x N +1 )
Problem 6.21 Solution
We follow the hint beginning by verifying the mean. We write Eq (6.62)
in a matrix form:
1
C N = ΦΦT + β−1 I N
α
Where we have used Eq (6.54). Here Φ is the design matrix defined below
Eq (6.51) and I N is an identity matrix. Before we use Eq (6.66), we need to
obtain k:
k = [k(x1 , x N +1 ), k(x2 , x N +1 ), ..., k(x N , x N +1 )]T
1
=
[φ(x1 )T φ(x N +1 ), φ(x2 )T φ(x N +1 ), ..., φ(xn )T φ(x N +1 )]T
α
1
=
Φφ(x N +1 )T
α
142
Now we substitute all the expressions into Eq (6.66), yielding:
h
i−1
m(x N +1 ) = α−1 φ(x N +1 )T ΦT α−1 ΦΦT + β−1 I N
t
Next using matrix identity (C.6), we obtain:
h
i−1
h
i−1
ΦT α−1 ΦΦT + β−1 I N
= αβ βΦT Φ + αI M
ΦT = αβS N ΦT
Where we have used Eq (3.54). Substituting it into m(x N +1 ), we obtain:
m(x N +1 ) = βφ(x N +1 )T S N ΦT t = < φ(x N +1 )T , βS N ΦT t >
Where < ·, · > represents the inner product. Comparing the result above
with Eq (3.58), (3.54) and (3.53), we conclude that the means are equal. It is
similar for the variance. We substitute c, k and C N into Eq (6.67). Then we
simplify the expression using matrix identity (C.7). Finally, we will observe
that it is equal to Eq (3.59).
Problem 6.22 Solution
Based on Eq (6.64) and (6.65), We first write down the joint distribution
for t N +L = [t 1 (x), t 2 (x), ..., t N +L (x)]T :
p(t N +L ) = N (t N +L |0, C N +L )
Where C N +L is similarly given by:
µ
C1,N
C N +L =
KT
K
¶
C N +1,N +L
The expression above has already implicitly divided the vector t N +L into
two parts. Similar to Prob.6.20, for later simplicity we rearrange the order
of t N +L denoted as t̄ N +L = [t N +1 , ..., t N +L , t 1 , ..., t N ]T . Moreover, t̄ N +L should
also follows a Gaussian distribution:
p(t̄ N +L ) = N (t̄ N +L |0, C̄ N +L )
Where we have defined:
µ
C̄ N +L =
C N +1,N +L
K
KT
C1,N
¶
Now we use Eq (2.94)-(2.98) and Eq (2.79)-(2.80) to derive the conditional
distribution, beginning by calculate Λaa :
1
−1
Λaa = (C N +1,N +L − KT · C−
1,N · K)
and Λab :
1
−1
1
Λab = − (C N +1,N +L − KT · C−
· KT · C−
1,N · K)
1,N
143
Now we can obtain:
1
p(t N +1 , ..., t N +L |t N ) = N (µa|b , Λ−
aa )
Where we have defined:
1
T
−1
µa|b = 0 + KT · C−
1,N · t N = K · C1,N · t N
If now we want to find the conditional distribution p(t j |t N ), where N + 1 ≤
j ≤ N + L, we only need to find the corresponding entry in the mean (i.e., the
( j − N)-th entry) and covariance matrix (i.e., the ( j − N)-th diagonal entry) of
p(t N +1 , ..., t N +L |t N ). In this case, it will degenerate to Eq (6.66) and (6.67)
just as required.
Problem 6.24 Solution
By definition, we only need to prove that for arbitrary vector x 6= 0, xT Wx
is positive. Here suppose that W is a M × M matrix. We expand the multiplication:
M X
M
M
X
X
xT Wx =
Wi j · x i · x j =
Wii · x2i
i =1 j =1
i =1
where we have used the fact that W is a diagonal matrix. Since Wii > 0,
we obtain xT Wx > 0 just as required. Suppose we have two positive definite
matrix, denoted as A1 and A2 , i.e., for arbitrary vector x, we have xT A1 x > 0
and xT A2 x > 0. Therefore, we can obtain:
xT (A1 + A2 )x = xT A1 x + xT A2 x > 0
Just as required.
Problem 6.25 Solution
Based on Newton-Raphson formula, Eq(6.81) and Eq(6.82), we have:
anew
N
1 −1
−1
= a N − (−W N − C−
N ) (t N − σ N − C N a N )
1 −1
−1
= a N + (W N + C−
N ) (t N − σ N − C N a N )
£
¤
1 −1
1
−1
= (W N + C−
(W N + C−
N )
N )a N + t N − σ N − C N a N
1
−1 −1
= C N C−
N (W N + C N ) (t N − σ N + W N a N )
= C N (C N W N + I)−1 (t N − σ N + W N a N )
Just as required.
Problem 6.26 Solution
Using Eq(6.77), (6.78) and (6.86), we can obtain:
Z
p(a N +1 |t N ) =
p(a N +1 |a N )p(a N |t N )da N
Z
1
T −1
?
−1
=
N(a N +1 |kT C−
N a N , c − k C N k) · N(a N |a N , H ) da N
144
By analogy to Eq (2.115), i.e.,
Z
p(y) =
p(y|x)p(x) dx
We can obtain:
p(a N +1 |t N ) = N(Aµ + b, L−1 + AΛ−1 AT )
(∗)
Where we have defined:
1
−1
1
A = kT C−
= c − kT C−
N , b = 0, L
N k
And
µ = a?
N, Λ = H
Therefore, the mean is given by:
1 ?
T −1
T
Aµ + b = kT C−
N a N = k C N C N (t N − σ N ) = k (t N − σ N )
Where we have used Eq (6.84). The covariance matrix is given by:
L−1 + AΛ−1 AT
=
1
T −1 −1 T −1 T
c − kT C−
N k + k C N H (k C N )
=
1
1 −1 −1
c − kT (C−
− C−
N H C N )k
³ N
´
1
−1
−1 −1 −1
c − kT C−
−
C
(W
+
C
)
C
N
N
N
N
N k
³
´
1
−1 −1
c − kT C−
−
(C
W
C
+
C
)
k
N N N
N
N
=
=
Where we have used Eq (6.85) and the fact that C N is symmetric. Then we
use matrix identity (C.7) to further reduce the expression, which will finally
give Eq (6.88).
Problem 6.27 Solution(Wait for update) This problem is really complicated.
What’s more, I find that Eq (6.91) seems not right.
0.7
Sparse Kernel Machines
Problem 7.1 Solution
By analogy to Eq (2.249), we can obtain:
p(x| t) =

+1

1 NX
1


· k(x, xn )

 N+1
Zk
t = +1

−1

1 NX
1


· k(x, xn )

N−1 n=1 Z k
t = −1
n=1
145
where N+1 represents the number of samples with label t = +1 and it is
the same for N−1 . Z k is a normalization constant representing the volume of
the hypercube. Since we have equal prior for the class, i.e.,
(
0.5 t = +1
p(t) =
0.5 t = −1
Based on Bayes’ Theorem, we have p(t|x) ∝ p(x| t) · p(t), yielding:

+1

1
1 NX


·
· k(x, xn ) t = +1

 Z N+1
n=1
p(t|x) =

−1

1 NX
1


· k(x, xn ) t = −1
 ·
Z N−1 n=1
Where 1/Z is a normalization constant to guarantee the integration of the
posterior equal to 1. To classify a new sample x? , we try to find the value t?
that can maximize p(t|x). Therefore, we can obtain:

+1
−1

1 NX
1 NX


·
k(x,
x
)
≥
+
1
if
· k(x, xn )

n

N+1 n=1
N−1 n=1
?
t =
(∗)

+1
−1

1 NX
1 NX


· k(x, xn ) ≤
· k(x, xn )
−1 if
N+1 n=1
N−1 n=1
If we now choose the kernel function as k(x, x0 ) = xT x0 ,we have:
+1
+1
1 NX
1 NX
k(x, xn ) =
xT xn = xT x̃+1
N+1 n=1
N+1 n=1
Where we have denoted:
x̃+1 =
+1
1 NX
xn
N+1 n=1
and similarly for x̃−1 . Therefore, the classification criterion (∗) can be
written as:
(
+1 if x̃+1 ≥ x̃−1
?
t =
−1 if x̃+1 ≤ x̃−1
When we choose the kernel function as k(x, x0 ) = φ(x)T φ(x0 ), we can similarly obtain the classification criterion:
(
+1 if φ̃(x+1 ) ≥ φ̃(x−1 )
?
t =
−1 if φ̃(x+1 ) ≤ φ̃(x−1 )
Where we have defined:
φ̃(x+1 ) =
+1
1 NX
φ(xn )
N+1 n=1
146
Problem 7.2 Solution
Suppose we have find w0 and b 0 , which can let all points satisfy Eq (7.5)
and simultaneously minimize Eq (7.3). This hyperlane decided by w0 and
b 0 is the optimal classification margin. Now if the constraint in Eq (7.5)
becomes:
t n (wT φ(xn ) + b) ≥ γ
We can conclude that if we perform change of variables: w0 − > γw0 and
b− > γ b, the constraint will still satisfy and Eq (7.3) will be minimize. In
other words, if the right side of the constraint changes from 1 to γ, The new
hyperlane decided by γw0 and γ b 0 is the optimal classification margin. However, the minimum distance from the points to the classification margin is
still the same.
Problem 7.3 Solution
Suppose we have x1 belongs to class one and we denote its target value
t 1 = 1, and similarly x2 belongs to class two and we denote its target value
t 2 = −1. Since we only have two points, they must have t i · y(x i ) = 1 as shown
in Fig. 7.1. Therefore, we have an equality constrained optimization problem:
1
minimize ||w||2
2
s.t.
( T
w φ(x1 ) + b = 1
wT φ(x2 ) + b = −1
This is an convex optimization problem and it has been proved that global
optimal exists.
Problem 7.4 Solution
Since we know that
ρ=
1
||w||
Therefore, we have:
1
= ||w||2
ρ2
In other words, we only need to prove that
||w||2 =
N
X
an
n=1
When we find th optimal solution, the second term on the right hand side
of Eq (7.7) vanishes. Based on Eq (7.8) and Eq (7.10), we also observe that its
dual is given by:
N
X
1
L̃(a) =
a n − ||w||2
2
n=1
147
Therefore, we have:
N
X
1
1
a n − ||w||2
||w||2 = L(a) = L̃(a) =
2
2
n=1
Rearranging it, we will obtain what we are required.
Problem 7.5 Solution
We have already proved this problem in the previous one.
Problem 7.6 Solution
If the target variable can only choose from {−1, 1}, and we know that
p(t = 1| y) = σ(y)
We can obtain:
p(t = −1| y) = 1 − p(t = 1| y) = 1 − σ(y) = σ(− y)
Therefore, combining these two situations, we can derive:
p(t| y) = σ(yt)
Consequently, we can obtain the negative log likelihood:
− ln p(D) = − ln
N
Y
n=1
σ(yn t n ) = −
N
X
n=1
ln σ(yn t n ) =
N
X
E LR (yn t n )
n=1
Here D represents the dataset, i.e.,D = {(xn , t n ); n = 1, 2, ..., N }, and E LR (yt)
is given by Eq (7.48). With the addition of a quadratic regularization, we obtain exactly Eq (7.47).
Problem 7.7 Solution
The derivatives are easy to obtain. Our main task is to derive Eq (7.61)
148
using Eq (7.57)-(7.60).
L
=
C
−
N
X
1
(µ n ξ n + µ
bn ξbn )
(ξn + ξbn ) + ||w||2 −
2
n=1
n=1
N
X
N
X
a n (² + ξn + yn − t n ) −
n=1
n=1
=
C
−
C
−
abn (² + ξbn + yn − t n )
N
X
N
N
X
X
1
(ξn + ξbn ) + ||w||2 −
(a n + µn )ξn −
(abn + µ
bn )ξbn
2
n=1
n=1
n=1
N
X
a n (² + yn − t n ) −
N
X
n=1
n=1
=
N
X
abn (² + yn − t n )
N
X
N
N
X
X
1
(ξn + ξbn ) + ||w||2 −
C ξbn
C ξn −
2
n=1
n=1
n=1
N
X
n=1
(a n + abn )² −
N
X
n=1
(a n − abn )(yn − t n )
=
N
N
X
X
1
||w||2 −
(a n + abn )² −
(a n − abn )(yn − t n )
2
n=1
n=1
=
N
N
N
X
X
X
1
||w||2 −
(a n − abn )(wT φ(xn ) + b − t n ) −
(a n + abn )² +
2
n=1
n=1
n=1
=
N
N
N
X
X
X
1
||w||2 −
(a n − abn )(wT φ(xn ) + b) −
(a n + abn )² +
(a n − abn )t n
2
n=1
n=1
n=1
=
N
N
N
X
X
X
1
(a n − abn )t n
(a n + abn )² +
(a n − abn )wT φ(xn ) −
||w||2 −
2
n=1
n=1
n=1
=
N
N
X
X
1
||w||2 − ||w||2 −
(a n + abn )² +
(a n − abn )t n
2
n=1
n=1
N
N
X
X
1
(a n − abn )t n
(a n + abn )² +
= − ||w||2 −
2
n=1
n=1
Just as required.
Problem 7.8 Solution
This obviously follows from the KKT condition, described in Eq (7.67) and
(7.68).
Problem 7.9 Solution
The prior is given by Eq (7.80).
p(w|α) =
M
Y
1
−1
N (0, α−
i ) = N (w|0, A )
i =1
Where we have defined:
A = diag(α i )
149
The likelihood is given by Eq (7.79).
N
Y
p(t|X, w, β) =
p(t n |xn , w, β−1 )
n=1
N
Y
=
N (t n |wT φ(xn ), β−1 )
n=1
= N (t|Φw, β−1 I)
Where we have defined:
Φ = [φ(x1 ), φ(x2 ), ..., φ(xn )]T
Our definitions of Φ and A as consistent with the main text. Therefore,
according to Eq (2.113)-Eq (2.117), we have:
p(w|t, X, α, β) = N (m, Σ)
Where we have defined:
Σ = (A + βΦT Φ)−1
And
m = βΣΦT t
Just as required.
Problem 7.10&7.11 Solution
It is quite similar to the previous problem. We begin by writting down the
prior:
p(w|α) =
M
Y
1
−1
N (0, α−
i ) = N (w|0, A )
i =1
Then we write down the likelihood:
N
Y
p(t|X, w, β) =
p(t n |xn , w, β−1 )
n=1
N
Y
=
N (t n |wT φ(xn ), β−1 )
n=1
= N (t|Φw, β−1 I)
Since we know that:
p(t|X, α, β) =
Z
p(t|X, w, β)p(w|α) dw
150
First as required by Prob.7.10, we will solve it by completing the square.
We begin by write down the expression for p(t|X, w, β):
Z
p(t|X, α, β) =
N (w|0, A−1 )N (t|Φw, β−1 I) dw
= (
β
2π
) N /2 ·
1
M
Y
(2π)
m=1
·
M /2
α1/2
i ·
Z
exp{−E(w)} dw
Where we have defined:
1 T
β
w Aw + ||t − Φw||2
2
2
We expand E(w) with respect to w:
o
1n T
E(w) =
w (A + βΦT Φ)w − 2βtT (Φw) + βtT t
2
o
1 n T −1
w Σ w − 2mT Σ−1 w + βtT t
=
2
o
1n
=
(w − m)T Σ−1 (w − m) + βtT t − mT Σ−1 m
2
Where we have used Eq (7.82) and Eq (7.83). Substituting E(w) into the
integral, we will obtain:
Z
M
Y
β N /2
1
1/2
p(t|X, α, β) = ( )
·
·
α · exp{−E(w)} dw
2π
(2π) M /2 m=1 i
n 1
o
M
Y
β
1
1/2
M /2
1/2
T
T −1
·
(2
π
)
·
|
Σ
|
exp
−
α
= ( ) N /2 ·
·
t
−
m
Σ
m)
(
β
t
2π
2
(2π) M /2 m=1 i
n
o
M
Y
β
1
T
T −1
= ( ) N /2 · |Σ|1/2 ·
α1/2
·
exp
−
(
β
t
t
−
m
Σ
m)
i
2π
2
m=1
n
o
M
Y
β
α1/2
= ( ) N /2 · |Σ|1/2 ·
i · exp − E(t)
2π
m=1
E(w) =
We further expand E(t):
E(t) =
=
=
=
=
=
=
1
(βtT t − mT Σ−1 m)
2
1
(βtT t − (βΣΦT t)T Σ−1 (βΣΦT t))
2
1
(βtT t − β2 tT ΦΣΣ−1 ΣΦT t)
2
1
(βtT t − β2 tT ΦΣΦT t)
2
1 T
t (βI − β2 ΦΣΦT )t
2
i
1 Th
t βI − βΦ(A + βΦT Φ)−1 ΦT β t
2
1 T −1
1
t (β I + ΦA−1 ΦT )−1 t = tT C−1 t
2
2
151
Note that in the last step we have used matrix identity Eq (C.7). Therefore, as we know that the pdf is Gaussian and the exponential term has been
given by E(t), we can easily write down Eq (7.85) considering those normalization constant.
What’s more, as required by Prob.7.11, the evaluation of the integral can
be easily performed using Eq(2.113)- Eq(2.117).
Problem 7.12 Solution
According to the previous problem, we can explicitly write down the log
marginal likelihood in an alternative form:
ln p(t|X, α, β) =
M
N
N
1
1X
ln β − ln 2π + ln |Σ| +
ln α i − E(t)
2
2
2
2 i=1
We first derive:
dE(t)
dαi
1 d
(mT Σ−1 m)
2 dαi
1 d
−
(β2 tT ΦΣΣ−1 ΣΦT t)
2 dαi
1 d
−
(β2 tT ΦΣΦT t)
2 dαi
1 £ d
d Σ−1
2 T
T
− Tr
(
β
t
ΦΣΦ
t)
·
]
2
dαi
d Σ−1
1
1 2 £
β T r Σ(ΦT t)(ΦT t)T Σ · I i ] = m2ii
2
2
= −
=
=
=
=
In the last step, we have utilized the following equation:
d
T r(AX−1 B) = −X−T AT BT X−T
dX
Moreover, here I i is a matrix with all elements equal to zero, expect the
i-th diagonal element, and the i-th diagonal element equals to 1. Then we
utilize matrix identity Eq (C.22) to derive:
d ln |Σ|
dαi
d ln |Σ−1 |
dαi
h d
i
(A + βΦT Φ)
= −T r Σ
dαi
= −Σ ii
= −
Therefore, we can obtain:
d ln p
1
1
1
=
− m2 − Σ ii
dαi
2α i 2 i 2
152
Set it to zero and obtain:
αi =
1 − α i Σ ii
γi
= 2
mi
mi
Then we calculate the derivatives of ln p with respect to β beginning by:
d ln |Σ|
dβ
d ln |Σ−1 |
dβ
i
h d
(A + βΦT Φ)
= −T r Σ
dβ
h
i
= −T r ΣΦT Φ
= −
Then we continue:
dE(t)
dβ
=
=
=
=
=
=
=
=
=
=
=
1 T
1 d
t t−
(mT Σ−1 m)
2
2 dβ
1 T
1 d 2 T
t t−
(β t ΦΣΣ−1 ΣΦT t)
2
2 dβ
1 T
1 d 2 T
t t−
(β t ΦΣΦT t)
2
2 dβ
1 T
1
d T
t t − βtT ΦΣΦT t − β2
(t ΦΣΦT t)
2
2 dβ
o
1n T
d T
t t − 2βtT ΦΣΦT t − β2
(t ΦΣΦT t)
2
dβ
o
n
1 T
d T
t t − 2tT (Φm) − β2
(t ΦΣΦT t)
2
dβ
n
£ d
1 T
d Σ−1 ¤o
T
T
(t
ΦΣΦ
t)
·
t t − 2tT (Φm) − β2 T r
2
dβ
d Σ−1
n
£
¤o
1 T
t t − 2tT (Φm) + β2 T r Σ(ΦT t)(ΦT t)T Σ · ΦT Φ
2
o
£
1n T
t t − 2tT (Φm) + T r mmT · ΦT Φ]
2
o
£
1n T
t t − 2tT (Φm) + T r ΦmmT · ΦT ]
2
1
||t − Φm||2
2
Therefore, we have obtained:
´
d ln p
1³ N
=
− ||t − Φm||2 − T r[ΣΦT Φ]
dβ
2 β
153
Using Eq (7.83), we can obtain:
ΣΦT Φ = ΣΦT Φ + β−1 ΣA − β−1 ΣA
= Σ(βΦT Φ + A)β−1 − β−1 ΣA
= Iβ−1 − β−1 ΣA
= (I − ΣA)β−1
Setting the derivative equal to zero, we can obtain:
β−1 =
||t − Φm||2
||t − Φm||2
=
P
N − T r(I − ΣA)
N − i γi
Just as required.
Problem 7.13 Solution
This problem is quite confusing. In my point of view, the posterior should
be denoted as p(w|t, X, {a i , b i }, a β , b β ), where a β , b β controls the Gamma distribution of β, and a i , b i controls the Gamma distribution of α i . What we
should do is to maximize the marginal likelihood p(t|X, {a i , b i }, a β , b β ) with
respect to {a i , b i }, a β , b β . Now we do not have a point estimation for the hyperparameters β and α i . We have a distribution (controled by the hyper priors,
i.e., {a i , b i }, a β , b β ) instead.
Problem 7.14 Solution
We begin by writing down p(t|x, w, β∗ ). Using Eq (7.76) and Eq (7.77), we
can obtain:
p(t|x, w, β∗ ) = N (t|wT φ(x), (β∗ )−1 )
Then we write down p(w|X, t, α∗ , β∗ ). Using Eq (7.81), (7.82) and (7.83),
we can obtain:
p(w|X, t, α∗ , β∗ ) = N (w|m, Σ)
Where m and Σ are evaluated using Eq (7.82) and (7.83) given α = α∗
and β = β∗ . Then we utilize Eq (7.90) and obtain:
Z
∗ ∗
p(t|x, X, t, α , β ) =
N (t|wT φ(x), (β∗ )−1 )N (w|m, Σ) d w
Z
=
N (t|φ(x)T w, (β∗ )−1 )N (w|m, Σ) d w
Using Eq (2.113)-(2.117), we can obtain:
p(t|x, X, t, α∗ , β∗ ) = N (µ, σ2 )
Where we have defined:
µ = mT φ(x)
154
And
σ2 = (β∗ )−1 + φ(x)T Σφ(x)
Just as required.
Problem 7.15 Solution
We just follow the hint.
1
L(α) = − { N ln 2π + ln |C| + tT C−1 t}
2
1n
1 T −1
= − N ln 2π + ln |C− i | + ln |1 + α−
i ϕ i C− i ϕ i |
2
1
C−
ϕ ϕT C−1 o
−i i i −i
1
)t
+tT (C−
−
−i
α i + ϕT
C−1 ϕ
i −i i
=
=
=
=
L(α− i ) −
−1
T −1
1
1 T C− i ϕ i ϕ i C− i
1
ln |1 + α−i 1 ϕTi C−
ϕ
|
+
t
t
−i i
1ϕ
2
2 α i + ϕTi C−
−i i
2
1 qi
1
−1
L(α− i ) − ln |1 + α i s i | +
2
2 αi + s i
2
1 αi + s i 1 q i
+
L(α− i ) − ln
2
αi
2 αi + s i
h
q2i i
1
L(α− i ) + ln α i − ln(α i + s i ) +
= L(α− i ) + λ(α i )
2
αi + s i
Where we have defined λ(α i ), s i and q i as shown in Eq (7.97)-(7.99).
Problem 7.16 Solution
We first calculate the first derivative of Eq(7.97) with respect to α i :
q2i
1 1
1
= [ −
−
]
∂α i
2 α i α i + s i (α i + s i )2
∂λ
Then we calculate the second derivative:
2q2i
1
1
1
=
[
−
+
+
]
2 α2i (α i + s i )2 (α i + s i )3
∂α2i
∂2 λ
Next we aim to prove that when α i is given by Eq (7.101), i.e., setting the
first derivative equal to 0, the second derivative (i.e., the expression above) is
negative. First we can obtain:
αi + s i =
s2i
q2i − s i
+ si =
s i q2i
q2i − s i
155
Therefore, substituting α i + s i and α i into the second derivative, we can
obtain:
∂2 λ
∂α2i
2
2
2
2
2
2
3
=
(q i − s i )
2q i (q i − s i )
1 (q i − s i )
[−
+
+
]
2
s4i
s2i q4i
s3i q6i
=
s (q − s i )
2s i (q i − s i )
1 q i (q i − s i )
+ i i4 4
+
]
[−
4
4
2
qi si
si qi
s4i q4i
=
1 (q i − s i )
[− q4i + s2i + 2s i (q2i − s i )]
2 q4i s4i
=
1 (q i − s i )
[−(q2i − s i )2 ]
2 q4i s4i
4
2
2
2
2
2
2
= −
2
2
2
2
2
3
4
1 (q i − s i )
<0
2 q4i s4i
Just as required.
Problem 7.17 Solution
We just follow the hint. According to Eq (7.102), Eq (7.86) and matrix
identity (C.7), we have:
Qi
−1
= ϕT
i C t
−1
−1 T −1
= ϕT
i (β I + ΦA Φ ) t
T
−1 T
= ϕT
i (βI − βIΦ(A + Φ βIΦ) Φ βI)t
2
T
−1 T
= ϕT
i (β − β Φ(A + βΦ Φ) Φ )t
2
T
= ϕT
i (β − β ΦΣΦ )t
2 T
T
= βϕT
i t − β ϕ i ΦΣΦ t
Similarly, we can obtain:
Si
−1
= ϕT
i C ϕi
2
T
= ϕT
i (β − β ΦΣΦ )ϕ i
2 T
T
= βϕT
i ϕ i − β ϕ i ΦΣΦ ϕ i
Just as required.
Problem 7.18 Solution
We begin by deriving the first term in Eq (7.109) with respect to w. This
can be easily evaluate based on Eq (4.90)-(4.91).
N
∂ nX
∂w
n=1
o X
N
t n ln yn + (1 − t n ) ln(1 − yn ) =
(t n − yn )φn = ΦT (t − y)
n=1
156
Since the derivative of the second term in Eq (7.109) with respect to w
is rather simple to obtain. Therefore, The first derivative of Eq (7.109) with
respect to w is:
∂ ln p
= ΦT (t − y) − Aw
∂w
For the Hessian matrix, we can first obtain:
∂ n
∂w
ΦT (t − y)
o
o
N ∂ n
X
(t n − yn )φn
n=1 ∂w
o
N ∂ n
X
= −
yn · φn
n=1 ∂w
=
= −
N ∂σ(wT φ )
X
n
n=1
= −
∂w
· φT
n
N ∂σ(a) ∂a
X
·
· φT
∂w n
n=1 ∂a
Where we have defined a = wT φn . Then we can utilize Eq (4.88) to derive:
o
N
X
T
ΦT (t − y) = −
σ(1 − σ) · φn · φT
n = −Φ BΦ
∂ n
∂w
n=1
Where B is a diagonal N × N matrix with elements b n = yn (1 − yn ). Therefore, we can obtain the Hessian matrix:
∂ n ∂ ln p o
H=
= −(ΦT BΦ + A)
∂w ∂w
Just as required.
Problem 7.19 Solution
We begin from Eq (7.114).
p(t|w∗ )p(w∗ |α)(2π) M /2 |Σ|1/2
¯
hY
ih Y
i
N
M
1
M /2
1/2 ¯
=
p(t n | xn , w)
N (w i |0, α−
)
(2
π
)
|
Σ
|
¯
i
p(t|α) =
n=1
=
hY
N
n=1
i =1
¯
i
¯
p(t n | xn , w) · N (w|0, A) · (2π) M /2 |Σ|1/2 ¯
w = w∗
w = w∗
We further take logarithm for both sides.
ln p(t|α) =
hX
N
ln p(t n | xn , w) + ln N (w|0, A) +
n=1
hX
N £
i¯
1
M
¯
ln 2π + ln |Σ| ¯
w = w∗
2
2
i¯
¤ 1
1
1
¯
t n ln yn + (1 − t n ) ln(1 − yn ) − wT Aw − ln |A| + ln |Σ| + const ¯
w = w∗
2
2
2
n=1
hX
i h1
i¯
N £
¤ 1
1
¯
=
t n ln yn + (1 − t n ) ln(1 − yn ) − wT Aw + ln |Σ| − ln |A| + const ¯
w = w∗
2
2
2
n=1
=
157
Using the Chain rule, we can obtain:
∂ ln p(t|α) ¯¯
∂ ln p(t|α) ∂w ¯¯
=
¯
¯
w = w∗
∂α i
∂w
∂α i w = w∗
Observing Eq (7.109), (7.110) and that (7.110) will equal 0 at w∗ , we can
conclude that the first term on the right hand side of ln p(t|α) will have zero
derivative with respect to w at w∗ . Therefore, we only need to focus on the
second term:
·
¸¯
∂ ln p(t|α) ¯¯
1
∂ 1
¯
ln |Σ| − ln |A| ¯
=
¯
w = w∗
w = w∗
∂α i
∂α i 2
2
It is rather easy to obtain:
¤
1
1 ∂ £X
1
[− ln |A|] = −
ln α−i 1 =
∂α i 2
2 ∂α i i
2α i
∂
Then we follow the same procedure as in Prob.7.12, we can obtain:
∂
1
1
[ ln |Σ|] = − Σ ii
∂α i 2
2
Therefore, we obtain:
∂ ln p(t|α)
∂α i
=
1
1
− Σ ii
2α i 2
Note: here I draw a different conclusion as the main text. I have also
verified my result in another way. You can write the prior as the product of
1
N (w i |0, α−
) instead of N (w|0, A). In this form, since we know that:
i
M
∂ X
∂α i i=1
ln N (w i |0, α−i 1 ) =
∂
1
αi
1
1
( ln α i − w2i ) =
− (w∗i )2
∂α i 2
2
2α i 2
The above expression can be used to replace the derivative of −1/2wT Aw−
1/2 ln |A|. Since the derivative of the likelihood with respect to α i is not zero
at w∗ , (7.115) seems not right anyway.
0.8
Graphical Models
Problem 8.1 Solution
We are required to prove:
Z
x
p(x) dx =
Z Y
K
x k=1
p(xk | pa k ) dx = 1
158
Here we adopt the same assumption as in the main text: No arrows lead
from a higher numbered node to a According to Eq(8.5), we can write:
Z
Z Y
K
p(xk | pa k ) dx
p(x) dx =
x
x k=1
Z
=
x
p(xK | pa K )
Z
=
h
[ x1 ,x2 ,...,xK −1 ] xK
=
p(xK | pa K )
[ x1 ,x2 ,...,xK −1 ]
Z
p(xk | pa k )
xK
k=1
h KY
−1
[ x1 ,x2 ,...,xK −1 ]
i
p(xk | pa k )dxK dx1 dx2 , ...dxK −1
i
p(xK | pa K )dxK dx1 dx2 , ...dxK −1
i
p(xk | pa k ) dx1 dx2 , ...dxK −1
k=1
KY
−1
Z
KY
−1
k=1
h KY
−1
Z
=
p(xk | pa k ) dx
k=1
Z
Z
=
KY
−1
[ x1 ,x2 ,...,xK −1 ] k=1
p(xk | pa k ) dx1 dx2 , ...dxK −1
Note that from the third line to the fourth line, we have used the fact
that x1 , x2 , ...xK −1 do not depend on xK , and thus the product from k = 1 to
K − 1 can be moved to the outside of the integral with respect to xK , and that
we have used the fact that the conditional probability is correctly normalized
from the fourth line to the fifth line. The aforementioned procedure will be
repeated for K times until all the variables have been integrated out.
Problem 8.2 Solution
This statement is obvious. Suppose that there exists an ordered numbering of the nodes such that for each node there are no links going to a
lower-numbered node, and that there is a directed cycle in the graph:
a 1 → a 2 → ... → a N
To make it a real cycle, we also require a N → a 1 . According to the assumption, we have a 1 ≤ a 2 ≤ ... ≤ a N . Therefore, the last link a N → a 1 is
invalid since a N ≥ a 1 .
Problem 8.3 Solution
Based on definition, we can obtain:


 0.336, if a = 0, b = 0

 0.264, if a = 0, b = 1
p(a, b) = p(a, b, c = 0) + p(a, b, c = 1) =
 0.256, if a = 1, b = 0


 0.144, if a = 1, b = 1
Similarly, we can obtain:
½
p(a) = p(a, b = 0) + p(a, b = 1) =
0.6, if a = 0
0.4, if a = 1
159
And
½
p(b) = p(a = 0, b) + p(a = 1, b) =
0.592, if b = 0
0.408, if b = 1
Therefore, we conclude that p(a, b) 6= p(a)p(b). For instance, we have
p(a = 1, b = 1) = 0.144, p(a = 1) = 0.4 and p(b = 1) = 0.408. It is obvious
that:
0.144 = p(a = 1, b = 1) 6= p(a = 1)p(b = 1) = 0.4 × 0.408
To prove the conditional dependency, we first calculate p(c):
p(c) =
X
a,b = 0,1
½
p(a, b, c) =
0.480, if c = 0
0.520, if c = 1
According to Bayes’ Theorem, we have:


0.400, if a = 0, b = 0, c = 0




0.277, if a = 0, b = 0, c = 1





0.100, if a = 0, b = 1, c = 0


p(a, b, c)  0.415, if a = 0, b = 1, c = 1
p(a, b| c) =
=

0.400, if a = 1, b = 0, c = 0
p(c)




0.123, if a = 1, b = 0, c = 1





0.100, if a = 1, b = 1, c = 0


 0.185, if a = 1, b = 1, c = 1
Similarly, we also have:


0.240/0.480 = 0.500, if a = 0, c = 0


p(a, c)  0.360/0.520 = 0.692, if a = 0, c = 1
=
p(a| c) =

0.240/0.480 = 0.500, if a = 1, c = 0
p(c)


 0.160/0.520 = 0.308, if a = 1, c = 1
Where we have used p(a, c) = p(a, b = 0, c) + p(a, b = 1, c). Similarly, we
can obtain:


0.384/0.480 = 0.800, if b = 0, c = 0



p(b, c)
0.208/0.520 = 0.400, if b = 0, c = 1
p(b| c) =
=

0.096/0.480 = 0.200, if b = 1, c = 0
p(c)


 0.312/0.520 = 0.600, if b = 1, c = 1
Now we can easily verify the statement p(a, b| c) = p(a| c)p(b| c). For instance, we have:
0.1 = p(a = 1, b = 1| c = 0) = p(a = 1| c = 0)p(b = 1| c = 0) = 0.5 × 0.2 = 0.1
Problem 8.4 Solution
160
This problem follows the previous one. We have already calculated p(a)
and p(b| c), we rewrite it here.
½
p(a) = p(a, b = 0) + p(a, b = 1) =
0.6, if a = 0
0.4, if a = 1
And


0.384/0.480 = 0.800, if b = 0, c = 0


p(b, c)  0.208/0.520 = 0.400, if b = 0, c = 1
p(b| c) =
=

0.096/0.480 = 0.200, if b = 1, c = 0
p(c)


 0.312/0.520 = 0.600, if b = 1, c = 1
We can also obtain p(c|a):


0.24/0.6 = 0.4, if a = 0, c = 0


p(a, c)  0.36/0.6 = 0.6, if a = 0, c = 1
p(c|a) =
=

0.24/0.4 = 0.6, if a = 1, c = 0
p(a)


 0.16/0.4 = 0.4, if a = 1, c = 1
Now we can easily verify the statement that p(a, b, c) = p(a)p(c|a)p(b| c)
given Table 8.2. The directed graph looks like:
a→c→b
Problem 8.5 Solution
It looks quite like Figure 8.6. The difference is that we introduce α i for
each w i , where i = 1, 2, ..., M.
Figure 1: probabilistic graphical model corresponding to the RVM described
in (7.79) and (7.80).
Problem 8.6 Solution(Wait for update)
Problem 8.7 Solution
Let’s just follow the hint. We begin by calculating the mean µ.
E[x1 ] = b 1
161
According to Eq (8.15), we can obtain:
E[x2 ] =
X
w2 j E[x j ] + b 2 = w21 b 1 + b 2
j ∈ pa 2
Then we can obtain:
E[x3 ] =
w32 E[x2 ] + b 3
=
w32 (w21 b 1 + b 2 ) + b 3
=
w32 w21 b 1 + w32 b 2 + b 3
Therefore, we obtain Eq (8.17) just as required. Next, we deal with the
covariance matrix.
cov[x1 , x1 ] = v1
Then we can obtain:
cov[x1 , x2 ] =
X
w2k cov[x1 , xk ] + I 12 v2 = w21 cov[x1 , x1 ] = w21 v1
k=1
And also cov[x2 , x1 ] = cov[x1 , x2 ] = w21 v1 . Hence, we can obtain:
X
cov[x2 , x2 ] =
2
w2k cov[x2 , xk ] + I 22 v2 = w21
v1 + v2
k=1
Next, we can obtain:
X
cov[x1 , x3 ] =
w3k cov[x1 , xk ] + I 31 v1 = w32 w21 v1
k=2
Then, we can obtain:
X
cov[x2 , x3 ] =
2
w3k cov[x2 , xk ] + I 23 v3 = w32 (v2 + w21
v1 )
k=2
Finally, we can obtain:
cov[x3 , x3 ] =
X
w3k cov[x3 , xk ] + I 33 v3
k=2
=
h
i
2
w32 w32 (v2 + w21
v1 ) + v3
Where we have used the fact that cov[x3 , x2 ] = cov[x2 , x3 ]. By now, we
have obtained Eq (8.18) just as required.
Problem 8.8 Solution
According to the definition, we can write:
p(a, b, c| d) = p(a| d) p(b, c| d)
162
We marginalize both sides with respect to c, yielding:
p(a, b| d) = p(a| d) p(b| d)
Just as required.
Problem 8.9 Solution
This statement is easy to see but a little bit difficult to prove. We put Fig
8.26 here to give a better illustration.
Figure 2: Markov blanket of a node x i
Markov blanket Φ of node x i is made up of three kinds of nodes:(i) the
set Φ1 containing all the parents of node x i (x1 and x2 in Fig.2), (ii) the set
Φ2 containing all the children of node x i (x5 and x6 in Fig.2), and (iii) the set
Φ3 containing all the co-parents of node x i (x3 and x4 in Fig.2). According to
the d-separation criterion, we need to show that all the paths from node x i to
an arbitrary node x̂ ∉ Φ = {Φ1 ∪ Φ2 ∪ Φ3 } are blocked given that the Markov
blanket Φ are observed.
It is obvious that x̂ can only connect to the target node x i via two kinds of
node: Φ1 ,Φ2 . First, suppose that x̂ connects to x i via some node x? ∈ Φ1 . The
arrows definitely meet head-to-tail or tail-to-tail at node x? because the link
from a parent node x? to x i has its tail connected to the parent node x? , and
since x? is in Φ1 ⊆ Φ, we see that this path is blocked.
In the second case, suppose that x̂ connects to x i via some node x? ∈ Φ2 .
We need to further divide this situation. If the path from x̂ to x i also goes
through a node x?? from Φ3 (e.g., in Fig.2, some node x̂ connects to node x3 ,
and in this example x?? = x3 , x? = x5 ), it is clearly that the arrows meet
head-to-tail or tail-to-tail at the node x?? ∈ Φ3 ⊆ Φ, this path is blocked.
In the final case, suppose that x̂ connects to x i via some node x? ∈ Φ2 and
the path doesn’t go through any node from Φ3 . An important observation is
that the arrows cannot meet head-to-head at node x? (otherwise, this path
will go through a node from Φ3 ). Thus, the arrows must meet either headto-tail or tail-to-tail at node x? ∈ Φ2 ⊆ Φ. Therefore, the path is also blocked.
163
Problem 8.10 Solution
By examining Fig.8.54, we can obtain:
p(a, b, c, d) = p(a) p(b) p(c|a, b) p(d | c)
Next we performing summation on both sides with respect to c and d, we
can obtain:
XX
p(a, b) = p(a) p(b)
p(c|a, b) p(d | c)
=
=
c
d
p(a) p(b)
X
hX
i
p(c|a, b)
p(d | c)
c
d
p(a) p(b)
X
p(c|a, b) × 1
c
=
p(a) p(b) × 1
=
p(a) p(b)
If we want to prove that a and b are dependent conditioned on d, we only
need to prove:
p(a, b| d) = p(a| d) p(b| d)
We multiply both sides by p(d) and use Bayes’ Theorem, yielding:
p(a, b, d) = p(a) p(b| d)
(∗)
In other words, we can equivalently prove the expression above instead.
Recall that we have:
p(a, b, c, d) = p(a) p(b) p(c|a, b) p(d | c)
We perform summation on both sides with respect to c, yielding:
X
p(a, b, d) = p(a) p(b) p(c|a, b) p(d | c)
c
Combining with (∗), we only need to prove:
X
p(b| d) = p(b) p(c|a, b) p(d | c)
c
However, we can see that the value of the right hand side depends on a,b
and d, while the left hand side only depends on b and d. In general, this
expression will not hold, and, thus, a and b are not dependent conditioned on
d.
Problem 8.11 Solution
This problem is quite straightforward, but it needs some patience. According to the Bayes’ Theorem, we have:
p(F = 0|D = 0) =
p(D = 0|F = 0) p(F = 0)
p(D = 0)
(∗)
164
We will calculate each of the term on the right hand side. Let’s begin from
the numerator p(D = 0). According to the sum rule, we have:
p(D = 0) =
=
p(D = 0,G = 0) + p(D = 0,G = 1)
p(D = 0|G = 0)p(G = 0) + p(D = 0|G = 1)p(G = 1)
= 0.9 × 0.315 + (1 − 0.9) × (1 − 0.315)
= 0.352
Where we have used Eq(8.30), Eq(8.105) and Eq(8.106). Note that the
second term in the denominator, i.e., p(F = 0), equals 0.1, which can be easily
derived from the main test above Eq(8.30). We now only need to calculate
p(D = 0|F = 0). Similarly, according to the sum rule, we have:
p(D = 0|F = 0) =
X
p(D = 0,G |F = 0)
G =0,1
=
X
p(D = 0|G, F = 0) p(G |F = 0)
G =0,1
=
X
p(D = 0|G) p(G |F = 0)
G =0,1
= 0.9 × 0.81 + (1 − 0.9) × (1 − 0.81)
= 0.748
Several clarifications must be made here. First, from the second line
to the third line, we simply eliminate the dependence on F = 0 because we
know that D only depends on G according to Eq(8.105) and Eq(8.106). Second,from the third line to the fourth line, we have used Eq(8.31), Eq(8.105)
and Eq(8.106). Now, we substitute all of them back to (∗), yielding:
p(F = 0|D = 0) =
0.748 × 0.1
p(D = 0|F = 0) p(F = 0)
=
= 0.2125
p(D = 0)
0.352
Next, we are required to calculate the probability conditioned on both
D = 0 and B = 0. Similarly, we can write:
p(F = 0|D = 0, B = 0) =
=
=
=
p(D = 0, B = 0, F = 0)
p(D = 0, B = 0)
P
G p(D = 0, B = 0, F = 0,G)
P
p(D = 0, B = 0,G)
P G
p(B
= 0, F = 0,G) p(D = 0|B = 0, F = 0,G)
G
P
G p(B = 0,G) p(D = 0|B = 0,G)
P
G p(B = 0, F = 0,G) p(D = 0|G)
(∗∗)
P
G p(B = 0,G) p(D = 0|G)
We need to calculate p(B = 0, F = 0,G) and p(B = 0,G), where G = 0, 1.
165
We begin by calculating p(B = 0, F = 0,G = 0):
p(B = 0, F = 0,G = 0) =
=
p(G = 0|B = 0, F = 0) × p(B = 0, F = 0)
p(G = 0|B = 0, F = 0) × p(B = 0) × p(F = 0)
= (1 − 0.1) × (1 − 0.9) × (1 − 0.9)
= 0.009
Similarly, we can obtain p(B = 0, F = 0,G = 1) = 0.001. Next we calculate
p(B = 0,G):
p(B = 0,G = 0) =
X
p(B = 0,G = 0, F)
F =0,1
=
X
p(G = 0|B = 0, F) × p(B = 0, F)
F =0,1
=
X
p(G = 0|B = 0, F) × p(B = 0) × p(F)
F =0,1
= (1 − 0.1) × (1 − 0.9) × (1 − 0.9) + (1 − 0.2) × (1 − 0.9) × 0.9
= 0.081
Similarly, we can obtain p(B = 0,G = 1) = 0.019. We substitute them back
into (∗∗), yielding:
P
G p(B = 0, F = 0,G) p(D = 0|G)
p(F = 0|D = 0, B = 0) =
P
G p(B = 0,G) p(D = 0|G)
0.009 × 0.9 + 0.001 × (1 − 0.9)
=
0.081 × 0.9 + 0.019 × (1 − 0.9)
= 0.1096
Just as required. The intuition behind this result coincides with the common sense. Moreover, by analogy to Fig.8.54, the node a and b in Fig.8.54
represents B and F in our case. Node c represents G, while node d represents
D. You can use d-separation criterion to verify the conditional properties.
Problem 8.12 Solution
An intuitive solution is that we construct a matrix A with size of M × M.
If there is a link from node i to node j, the entry on the i-th row and j-th
column of matrix A, i.e., A i, j , will equal to 1. Otherwise, it will equal to 0.
Since the graph is undirected, the matrix A will be symmetric. What’s more,
the element on the diagonal is 0 by definition. For a undirected graph, we can
use a matrix A to represent it. It is also a one-to-one mapping.
In other words, we equivalently count the number of possible matrix A
satisfying the following criteria: (i) each of the entry is either 0 or 1, (ii) it is
symmetric, and (iii) all of the entries on the diagonal are already determined
(i.e., they all equal 0).
166
Using the property of symmetry, we only need to count the free variables
on the lower triangle of the matrix. In the first column, there are (M − 1) free
variables. In the second column, there are (M − 2) free variables. Therefore,
the total free variables are given by:
(M − 1) + (M − 2) + ... + 0 =
M(M − 1)
2
Each value of these free variables has two choices, i.e., 1 or 0. Therefore,
the total number of such matrix is 2 M ( M −1)/2 . In the case of M = 3, there are
8 possible undirected graphs:
Figure 3: the undirected graph when M = 3
Problem 8.13 Solution
It is straightforward. Suppose that xk is the target variable whose state
may be {−1.1} while all other variables are fixed. According to Eq (8.42), we
can obtain:
E(x, y) =
h
X
X
xi − β
i 6= k
+ h xk − β
i, j 6= k
X
X
xi x j − η
x i yi
i 6= k
xk xm − η xk yk
m
Note that we write down the dependence of E(x, y) on xk explicitly, which
is expressed via the second line. Moreover, the x i x j term in the first line
doesn’t include the pairs { x i , x j }, which one of them is xk . These terms are
considered by xk xm in the secone line. To be more specific, here xm represents
the neighbor of xk . Noticing that the first line doesn’t depend on xk , we can
obtain:
X
E(x, y)| xk = 1 − E(x, y)| xk = −1 = 2h − 2β xm − 2η yk
m
Obviously, the difference depends locally on xk , implied by h, the neighbors xm and its observed value yk .
Problem 8.14 Solution
167
It is quite obvious. When h = 0, β = 0, the energy function reduces to
X
E(x, y) = −η x i yi
i
If there exists some index j which satisfies x j 6= y j , considering that x j , y j ∈
{−1.1}, then x j y j will equal to −1. By changing the sign of x j , we can always
increase the value of x j y j from −1 to 1, and, thus, decrease the energy function E(x, y).
Therefore, given the observed binary pixels yi ∈ {−1.1}, where i = 1, 2, ..., D,
in order to obtain the minimum of energy, the optimal choice for x i is to set it
equal to yi .
Problem 8.15 Solution
This problem can be solved by analogy to Eq (8.49) - Eq(8.54). We begin
by noticing:
X X X X
p(xn−1 , xn ) = ...
... p(x)
x1
xn−2 xn+1
xN
We also have:
p(x) =
1
ψ1,2 (x1 , x2 ) ψ2,3 (x2 , x3 ) ... ψ N −1,N (x N −1 , x N )
Z
By analogy to Eq(8.52), we can obtain:
·
hX
hX
ii ¸
1 X
ψn−2,n−1 (xn−2 , xn−1 )...
p(xn−1 , xn ) =
ψ2,3 (x2 , x3 )
ψ1,2 (x1 , x2 ) ...
Z xn−2
x2
x1
× ψn−1,n (xn−1,xn )
·
hX
i ¸
X
×
ψn,n+1 (xn , xn+1 )...
ψ N −1,N (x N −1 , x N ) ...
xn+1
=
xN
1
× µα (xn−1 ) × ψn−1,n (xn−1 , xn ) × µβ (xn )
Z
Just as required.
Problem 8.16 Solution
We can simply obtain p(x N ) using Eq(8.52) and Eq(8.54):
p(x N ) =
1
µα (x N )
Z
(∗)
According to Bayes’ Theorem, we have:
p(xn | x N ) =
p(xn , x N )
p(x N )
Therefore, now we only need to derive an expression for p(xn , x N ), where
n = 1, 2, ..., N − 1. We follow the same procedure as in the previous problem.
Since we know that:
X X X
X
p(xn , x N ) = ...
...
p(x)
x1
xn−1 xn+1
x N −1
168
We can obtain:
·
hX
hX
ii ¸
1 X
ψn−1,n (xn−1 , xn )...
ψ2,3 (x2 , x3 )
ψ1,2 (x1 , x2 ) ...
p(xn , x N ) =
Z xn−1
x2
x1
·
hX
i ¸
X
×
ψn,n+1 (xn , xn+1 )...
ψ N −2,N −1 (x N −2 , x N −1 )ψ N −1,N (x N −1 , x N ) ...
xn+1
x N −1
Note that in the second line, the summation term with respect to x N −1 is
the product of ψ N −2,N −1 (x N −2 , x N −1 ) and ψ N −1,N (x N −1 , x N ). So here we can
actually draw an undirected graph with N − 1 nodes, and adopt the proposed
algorithm to solve p(xn , x N ). If we use x?
n to represent the new node, then the
joint distribution can be written as:
1 ? ? ? ? ? ?
?
?
ψ (x , x ) ψ (x , x ) ... ψ?
N −2,N −1 (x N −2 , x N −1 )
Z ? 1,2 1 2 2,3 2 3
? ?
Where ψ?
n,n+1 (x n , x n+1 ) is defined as:
½
ψn,n+1 (xn , xn+1 ),
n = 1, 2, ..., N − 3
?
? ?
ψn,n+1 (xn , xn+1 ) =
ψ N −2,N −1 (x N −2 , x N −1 )ψ N −1,N (x N −1 , x N ),
n = N −2
p(x? ) =
In other words, we have combined the original node x N −1 and x N . Moreover, we have the relationship:
1 ? ? ? ?
µ (x )µ (x ) n = 1, 2, ..., N − 1
Z? α n β n
By adopting the proposed algorithm to the new undirected graph, p(x?
n)
can be easily evaluated, and so is p(xn , x N ).
p(xn , x N ) = p(x?
n) =
Problem 8.17 Solution
It is straightforward to see that for every path connecting node x2 and x5
in Fig.8.38, it must pass through node x3 . Therefore, all paths are blocked
and the conditional property holds. For more details, you should read section
8.3.1. According to Bayes’ Theorem, we can obtain:
p(x2 | x3 , x5 ) =
p(x2 , x3 , x5 )
p(x2 )
Using the proposed algorithm in section 8.4.1, we can obtain:
P P
p(x2 , x3 , x5 )
x
x p(x)
p(x2 | x3 , x5 ) =
= P P1 P4
p(x3 , x5 )
x1 x2 x4 p(x)
P P
x
x ψ1,2 ψ2,3 ψ3,4 ψ4,5
= P P1 P4
x1 x2 x4 ψ1,2 ψ2,3 ψ3,4 ψ4,5
³P
´
³P
´
x1 ψ1,2 · ψ2,3 ·
x4 ψ3,4 ψ4,5
´
i ³P
´
= P h³ P
ψ
ψ
·
ψ
ψ
1
,
2
2
,
3
3
,
4
4
,
5
x2
x1
x4
³P
´
x1 ψ1,2 · ψ2,3
´
i
= P h³ P
ψ
ψ
2,3
x2
x1 1,2
169
It is obvious that the right hand side doesn’t depend on x5 .
Problem 8.18 Solution
First, the distribution represented by a directed tree can be trivially be
written as an equivalent distribution over an undirected tree by moralization.
You can find more details in section 8.4.2.
Alternatively, now we want to represent a distribution, which is given by
a directed graph, via a directed graph. For example, the distribution defined
by the undirected tree in Fig.4 can be written as:
p(x) =
1
ψ1,3 (x1 , x3 ) ψ2,3 (x2 , x3 ) ψ3,4 (x3 , x4 ) ψ4,5 (x4 , x5 )
Z
We simply choose x4 as the root and the corresponding directed tree is
well defined by working outwards. In this case, the distribution defined by
the directed tree is:
p(x) = p(x4 ) p(x5 | x4 ) p(x3 | x4 ) p(x1 | x3 ) p(x2 | x3 )
Thus it is not difficult to change an undirected tree to a directed on if
performing:
p(x4 )p(x5 | x4 ) ∝ ψ5,4 , p(x3 | x4 ) ∝ ψ3,4 , p(x2 | x3 ) ∝ ψ2,3 , p(x1 | x3 ) ∝ ψ1,3 ,
Figure 4: Example of changing an undirected tree to a directed one x i
The symbol ∝ is used to represent a normalization term, which is used
to guarantee the integral of PDF equal to 1. In summary, in the particular
case of an undirected tree, there is only one path between any pair of nodes,
and thus the maximal clique is given by a pair of two nodes in an undirected
tree. This is because if we choose any three nodes x1 , x2 , x3 , according to the
definition there cannot exist a loop. Otherwise there are two paths between
x1 and x3 : (i) x1 − > x3 and (ii) x1 − > x2 − > x3 . In the directed tree, each node
170
only depends on only one node (except the root), i.e., its parent. Thus we can
easily change a undirected tree to a directed one by matching the potential
function with the corresponding conditional PDF, as shown in the example.
Moreover, we can choose any node in the undirected tree to be the root and
then work outwards to obtain a directed tree. Therefore, in an undirected tree
with n nodes, there is n corresponding directed trees in total.
Problem 8.19-8.29 Solution (Waiting for update)
I am quite confused by the deduction in Eq(8.66). I do not understand the
sum-prodcut algorithm and the max-sum algorithm very well.
0.9
Mixture Models and EM
Problem 9.1 Solution
For each r nk when n is fixed and k = 1, 2, ..., K, only one of them equals
1 and others are all 0. Therefore, there are K possible choices. When N
data are given, there are K N possible assignments for { r nk ; n = 1, 2, ..., N; k =
1, 2, ..., K }. For each assignments, the optimal {µk ; k = 1, 2, ..., K } are well determined by Eq (9.4).
As discussed in the main text, by iteratively performing E-step and Mstep, the distortion measure in Eq (9.1) is gradually minimized. The worst
case is that we find the optimal assignment and {µk } in the last iteration. In
other words, K N iterations are required. However, it is guaranteed to converge because the assignments are finite and the optimal {µk } is determined
once the assignment is given.
Problem 9.2 Solution
By analogy to Eq (9.1), we can write down:
JN = JN −1 +
K
X
r N k ||x N − µk ||2
k=1
In the E-step, we still assign the N-th data x N to the closet center and
suppose that this cloest center is µm . Therefore, the expression above will
reduce to:
JN = JN −1 + ||xn − µm ||2
In the M-step, we set the derivative of JN with respect to µk to 0, where
k = 1, 2, ..., K. We can observe that for those µk , k 6= m, we have:
∂ JN
∂µk
=
∂ JN −1
∂µk
171
In other words, we will only update µm in the M-step by setting the
derivative of JN equal to 0. Utilizing Eq (9.4), we can obtain:
µ(mN )
P N −1
n = 1 r nk x n + x N
P N −1
n = 1 r nk + 1
=
P N −1
x
n = 1 r nk x n
P
+ PN −1N
N −1
r
nk
n=1
n = 1 r nk
=
1 + PN −11
n = 1 r nk
x
µ(mN −1) + PN −1N
r
n=1
=
nk
1
1 + PN −1
n = 1 r nk
= µ(mN −1) +
x
P N −1N
n = 1 r nk
µ( N −1)
n = 1 r nk
− PNm−1
1 + PN −11
n = 1 r nk
x N − µ(mN −1)
µ(mN −1) +
P
1 + nN=−11 r nk
=
So far we have obtained a sequential on-line update formula just as required.
Problem 9.3 Solution
We simply follow the hint.
p(x) =
X
p(z) p(x|z)
z
=
i zk
K h
XY
(πk N (x|µk , Σk )
z k=1
Note that we have used 1-of-K coding scheme for z = [z1 , z2 , ..., zK ]T . To
be more specific, only one of z1 , z2 , ..., zK will be 1 and all others will equal 0.
Therefore, the summation over z actually consists of K terms and the k-th
term corresponds to z k equal to 1 and others 0. Moreover, for the k-th term,
the product will reduce to πk N (x|µk , Σk ). Therefore, we can obtain:
p(x) =
K h
XY
z k=1
(πk N (x|µk , Σk )
i zk
=
K
X
k=1
Just as required.
Problem 9.4 Solution
According to Bayes’ Theorem, we can write:
p(θ |X) ∝ p(X|θ )p(θ )
πk N (x|µk , Σk )
172
Taking logarithm on both sides, we can write:
ln p(θ |X) ∝ ln p(X|θ ) + ln p(θ )
Further utilizing Eq (9.29), we can obtain:
nX
o
ln p(θ |X) ∝ ln
p(X, Z|θ ) + ln p(θ )
Z
=
ln
n£ X
ln
nX
o
¤
p(X, Z|θ ) · p(θ )
Z
=
p(X, Z|θ )p(θ )
o
Z
In other words, in thise case, the only modification is that the term p(X, Z|θ )
in Eq (9.29) will be replaced by p(X, Z|θ )p(θ ). Therefore, in the E-step, we still
need to calculate the posterior p(Z|X, θ old ) and then in the M-step, we are re0
quired to maximize Q (θ , θ old ). In this case, by analogy to Eq (9.30), we can
0
write down Q (θ , θ old ):
h
i
X
0
Q (θ , θ old ) =
p(Z|X, θ old ) ln p(X, Z|θ )p(θ )
Z
=
X
=
X
=
X
=
X
h
i
p(Z|X, θ old ) ln p(X, Z|θ ) + ln p(θ )
Z
p(Z|X, θ old ) ln p(X, Z|θ ) +
Z
X
p(Z|X, θ old ) ln p(θ )
Z
p(Z|X, θ
old
p(Z|X, θ
old
) ln p(X, Z|θ ) + ln p(θ ) ·
Z
X
p(Z|X, θ old )
Z
) ln p(X, Z|θ ) + ln p(θ )
Z
= Q(θ , θ old ) + ln p(θ )
Just as required.
Problem 9.5 Solution
Notice that the condition on µ, Σ and π can be omitted here, and we only
need to prove p(Z|X) can be written as the product of p(zn |xn ). Correspondingly, the small dots representing µ, Σ and π can also be omitted in Fig 9.6.
Observing Fig 9.6 and based on definition, we can write :
p(X, Z) = p(x1 , z1 )p(z1 )...p(x N , z N )p(z N ) = p(x1 , z1 )...p(x N , z N )
Moreover, since there is no link from zm to zn , from xm to xn , and from
zm to xn (m 6= n), we can obtain:
p(Z) = p(z1 )...p(z N ),
p(X) = p(x1 )...p(x N )
173
These can also be verified by calculating the marginal distribution from
p(X, Z), for example:
p(Z) =
X
X
p(X, Z) =
p(x1 , z1 )...p(x N , z N ) = p(z1 )...p(z N )
x1 ,...,x N
X
According to Bayes’ Theorem, we have
p(Z|X) =
=
=
p(X|Z)p(Z)
p(X)
hQ
ih Q
i
N
N
n=1 p(x n |z n )
n=1 p(z n )
QN
n=1 p(x n )
N p(x |z )p(z )
Y
n n
n
p(xn )
n=1
=
N
Y
p(zn |xn )
n=1
Just as required. The essence behind the problem is that in the directed
graph, there are only links from zn to xn . The deeper reason is that (i) the
mixture model is given by Fig 9.4, and (ii) we assume the data {xn } is i.i.d,
and thus there is no link from xm to xn .
Problem 9.6 Solution
By analogy to Eq (9.19), we calculate the derivative of Eq (9.14) with
respect to Σ:
N
N 1 ∂a
X
∂ X
∂ ln p
n
=
{
ln a n } =
(∗)
∂Σ
∂Σ n=1
n=1 a n ∂Σ
Where we have defined:
an =
K
X
πk N (xn |µk , Σ)
k=1
Recall that in Prob.2.34, we have proved:
∂ ln N (xn |µk , Σ)
∂Σ
1
1
= − Σ−1 + Σ−1 Snk Σ−1
2
2
Where we have defined:
Snk = (xn − µk )(xn − µk )T
174
Therefore, we can obtain:
∂a n
∂Σ
=
=
=
=
=
K
∂ nX
∂Σ
o
πk N (xn |µk , Σ)
k=1
o
K ∂ n
X
πk N (xn |µk , Σ)
k=1 ∂Σ
K
X
£
¤o
∂ n
πk
exp ln N (xn |µk , Σ)
∂Σ
k=1
i
K
X
£
¤ ∂ h
πk · exp ln N (xn |µk , Σ) ·
ln N (xn |µk , Σ)
∂Σ
k=1
K
X
1
1
πk · N (xn |µk , Σ) · (− Σ−1 + Σ−1 Snk Σ−1 )
2
2
k=1
Substitute the equation above into (∗), we can obtain:
∂ ln p
∂Σ
=
=
=
=
N 1 ∂a
X
n
a
∂
Σ
n
n=1
PK
1 −1
N
X
+ Σ−1 Snk Σ−1 )
k=1 π k · N (x n |µ k , Σ) · (− 2 Σ
PK
n=1
j =1 π j N (x n |µ j , Σ)
N X
K
X
1
1
γ(z nk ) · (− Σ−1 + Σ−1 Snk Σ−1 )
2
2
n=1 k=1
o
o
nX
N X
K
N X
K
1n X
1
−
γ(z nk ) Σ−1 + Σ−1
γ(z nk )Snk Σ−1
2 n=1 k=1
2
n=1 k=1
If we set the derivative equal to 0, we can obtain:
P N PK
n=1 k=1 γ(z nk )S nk
Σ = PN P
K
n=1 k=1 γ(z nk )
Problem 9.7 Solution
We begin by calculating the derivative of Eq (9.36) with respect to µk :
∂ ln p
∂µk
=
N X
K
∂ n X
∂µk
z nk [ ln πk + ln N (xn |µk , Σk ) ]
n=1 k=1
o
N
∂ n X
=
z nk [ ln πk + ln N (xn |µk , Σk ) ]
∂µk n=1
o
N
X
∂ n
=
z nk ln N (xn |µk , Σk )
n=1 ∂µ k
o
X ∂ n
ln N (xn |µk , Σk )
=
xn ∈C k ∂µ k
o
175
Where we have used xn ∈ C k to represent the data point xn which are
assigned to the k-th cluster. Therefore, µk is given by the mean of those
xn ∈ C k just as the case of a single Gaussian. It is exactly the same for the
covariance. Next, we maximize Eq (9.36) with respect to πk by enforcing a
Lagrange multiplier:
K
X
πk − 1)
L = ln p + λ(
k=1
We calculate the derivative of L with respect to πk and set it to 0:
∂L
∂πk
=
N z
X
nk
n=1 π k
+λ = 0
We multiply both sides by πk and sum over k making use of the constraint
Eq (9.9), yielding λ = − N. Substituting it back into the expression, we can
obtain:
N
1 X
z nk
πk =
N n=1
Just as required.
Problem 9.8 Solution
Since γ(z nk ) is fixed, the only dependency of Eq (9.40) on µk occurs in the
Gaussian, yielding:
∂E z [ln p]
∂µk
N
∂ nX
=
∂µk
N
X
=
o
γ(z nk ) ln N (xn |µk , Σk )
n=1
γ(z nk ) ·
∂ ln N (xn |µk , Σk )
∂µk
n=1
N
X
=
n=1
h
i
1
γ(z nk ) · − Σ−
k (x n − µ k )
Setting the derivative equal to 0, we obtain exactly Eq (9.16), and consequently Eq (9.17) just as required. Note that there is a typo in Eq (9.16), Σk
1
shoule be Σ−
.
k
Problem 9.9 Solution
We first calculate the derivative of Eq (9.40) with respect to Σk :
∂E z
∂Σk
=
=
N
∂ nX
∂Σk
N
X
n=1
=
N
X
γ(z nk ) ln N (xn |µk , Σk )
o
n=1
γ(z nk )
∂ ln N (xn |µk , Σk )
∂Σk
h 1
i
1 −1
1
−1
γ(z nk ) · − Σ−
+
Σ
S
Σ
nk
k
2 k
2 k
n=1
176
As in Prob 9.6, we have defined:
Snk = (xn − µk )(xn − µk )T
Setting the derivative equal to 0 and rearranging it, we obtain:
PN
PN
γ(z nk ) Snk
n=1 γ(z nk ) S nk
Σk = P N
= n=1
Nk
n=1 γ(z nk )
Where Nk is given by Eq (9.18). So now we have obtained Eq (9.19) just
as required. Next to maximize Eq (9.40) with respect to πk , we still need to
introduce Lagrange multiplier to enforce the summation of pi k over k equal
to 1, as in Prob 9.7:
K
X
πk − 1)
L = E z + λ(
k=1
We calculate the derivative of L with respect to πk and set it to 0:
∂L
∂πk
=
N γ(z )
X
nk
n=1
πk
+λ = 0
We multiply both sides by πk and sum over k making use of the constraint
Eq (9.9), yielding λ = − N (you can see Eq (9.20)- Eq (9.22) for more details).
Substituting it back into the expression, we can obtain:
πk =
N
1 X
Nk
γ(z nk ) =
N n=1
N
Just as Eq (9.22).
Problem 9.10 Solution
According to the property of PDF, we know that:
p(xb |xa ) =
K
X
p(xa , xb )
p(x)
πk
=
=
· p(x| k)
p(xa )
p(xa ) k=1 p(xa )
Note that here p(xa ) can be viewed as a normalization constant used to
guarantee that the integration of p(xb |xa ) equal to 1. Moreover, similarly, we
can also obtain:
K
X
πk
p(xa |xb ) =
· p(x| k)
k=1 p(x b )
Problem 9.11 Solution
According to the problem description, the expectation, i.e., Eq(9.40), can
now be written as:
E z [ln p] =
N X
K
X
n=1 k=1
n
o
γ(z nk ) ln πk + ln N (xn |µk , ²I)
177
In the M-step, we are required to maximize the expression above with
respect to µk and πk . In Prob.9.8, we have already proved that µk should be
given by Eq (9.17):
N
1 X
µk =
γ(z nk )xn
(∗)
Nk n=1
Where Nk is given by Eq (9.18). Moreover, in this case, by analogy to Eq
(9.16), γ(z nk ) is slightly different:
γ(z nk )
πk N (xn |µk , ²I)
P
j π j N (x n |µ j , ²I)
=
When ² → 0, we can obtain:
X
π j N (xn |µ j , ²I) ≈ πm N (xn |µm , ²I),
where
m = argmin j ||xn − µ j ||2
j
To be more clear, the summation is dominated by the max of π j N (xn |µ j , ²I),
and this term is further determined by the exponent, i.e., −||xn − µ j ||2 . Therefore, γ(z nk ) is given by exactly Eq (9.2), i.e., we have γ(z nk ) = r nk . Combining
with (∗), we can obtain exactly Eq (9.4). Next, according to Prob.9.9, πk is
given by Eq(9.22):
PN
γ(z nk )
r nk
Nk
= n=
=
πk =
N
N
N
In other words, πk equals the fraction of the data points assigned to the
k-th cluster.
Problem 9.12 Solution
First we calculate the mean µk :
Z
µk =
xp(x) dx
Z
=
x
K
X
πk p(x| k) dx
k=1
=
K
X
πk
Z
x p(x| k) dx
k=1
=
K
X
πk µk
k=1
Then we deal with the covariance matrix. For an arbitrary random variable x, according to Eq (2.63) we have:
cov[x] = E[(x − E[x])(x − E[x])T ]
= E[xxT ] − E[x]E[x]T
178
Since E[x] is already obtained, we only need to solve E[xxT ]. First we only
focus on the k-th component and rearrange the expression above, yielding:
Ek [xxT ] = covk [x] + Ek [x]Ek [x]T = Σk + µk µT
k
We further use Eq (2.62), yielding:
E[xxT ] =
Z
xxT
K
X
πk p(x| k) dx
k=1
=
K
X
πk
Z
xxT p(x| k) dx
k=1
=
K
X
πk Ek [xxT ]
k=1
=
K
X
πk (µk µT
k + Σk )
k=1
Therefore, we obtain Eq (9.50) just as required.
Problem 9.13 Solution
First, let’s make this problem more clear. In a mixture of Bernoulli distribution, whose complete-data log likelihood is given by Eq (9.54) and whose
model parameters are πk and µk . If we want to obtain those parameters, we
can adopt EM algorithm. In the E-step, we calculate γ(z nk ) as shown in Eq
(9.56). In the M-step, we update πk and µk according to Eq (9.59) and Eq
(9.60), where Nk and x¯k are defined in Eq (9.57) and Eq (9.58). Now let’s
back to this problem. The expectation of x is given by Eq (9.49):
E[x] =
K
X
k=1
( opt)
Here πk
( opt)
and µk
( opt) ( opt)
µk
πk
are the parameters obtained when EM is converged.
179
Using Eq (9.58) and Eq(9.59), we can obtain:
E[x] =
K
X
k=1
=
K
X
k=1
( opt) ( opt)
µk
πk
( opt)
NK
( opt)
=
K N
X
k
=
=
K 1 X
N
X
k=1 N n=1
=
N
X
( opt)
NK
n=1
γ(z nk )( opt) xn
γ(z nk )( opt) xn
K γ(z )( opt) x
N X
X
n
nk
N
n=1 k=1
=
γ(z nk )( opt) xn
n=1
1
N
k=1
N
X
1
( opt)
πk
N x
X
n
K
X
n=1 N k=1
γ(z nk )( opt)
N
1 X
xn = x̄
N n=1
If we set all µk equal to µ
b in initialization, in the first E-step, we can
obtain:
π(0)
π(0)
p(xn |µk = µ
b)
k
(1)
= P k (0) = π(0)
γ(z nk ) = P
k
(0)
K
K
b)
j =1 π j p(x n |µ j = µ
j =1 π j
Note that here µ
b and π(0)
are the initial values. In the subsequent M-step,
k
according to Eq (9.57)-(9.60), we can obtain:
µ(1)
=
k
1
N
X
Nk(1) n=1
(1)
γ(z nk )
And
π(1)
=
k
Nk(1)
PN
(1)
n=1 γ(z nk ) x n
=
PN
(1)
n=1 γ(z nk )
xn =
PN
n=1 γ(z nk )
PN
(0)
xn
n=1 π k x n
= n=1
PN
(0)
N
n=1 π k
PN
(0)
n=1 π k
PN
(1)
= π(0)
k
N
N
N
In other words, in this case, after the first EM iteration, we find that the
new µ(1)
are all identical, which are all given by x̄. Moreover, the new π(1)
are
k
k
=
=
identical to their corresponding initial value π(0)
. Therefore, in the second
k
EM iteration, we can similarly conclude that:
µ(2)
= µ(1)
= x̄ ,
k
k
π(2)
= π(1)
= π(0)
k
k
k
In other words, the EM algorithm actually stops after the first iteration.
Problem 9.14 Solution
180
Let’s follow the hint.
p(x, z|µ, π) =
p(x|z, µ) · p(z|π)
K
K
Y
Y
z
πkk
p(x|µk ) zk ·
=
k=1
k=1
K h
Y
πk p(x|µk )
=
i zk
k=1
Then we marginalize over z, yielding:
p(x|µ) =
X
p(x, z|µ, π) =
z
i zk
K h
XY
πk p(x|µk )
z k=1
The summation over z is made up of K terms and the k-th term corresponds to z k = 1 and other z j , where j 6= k, equals 0. Therefore, the k-th term
will simply reduce to πk p(x|µk ). Hence, performing the summation over z
will finally give Eq (9.47) just as required. To be more clear, we summarize
the aforementioned statement:
p(x|µ) =
K h
XY
z k=1
=
i zk
πk p(x|µk )
i zk ¯
K h
Y
¯
πk p(x|µk ) ¯
k=1
z1 = 1
+ ... +
i zk ¯
K h
Y
¯
πk p(x|µk ) ¯
k=1
zK = 1
= π1 p(x|µ1 ) + ... + πK p(x|µK )
=
K
X
πk p(x|µk )
k=1
Problem 9.15 Solution
Noticing that πk doesn’t depend on any µki , we can omit the first term in
the open brace when calculating the derivative of Eq (9.55) with respect to
µki :
∂E z [ln p]
∂µki
=
=
=
=
=
∂
D £
N X
K n
X
X
¤o
γ(z nk )
xni ln µki + (1 − xni ) ln(1 − µki )
∂µki n=1 k=1
∂
i =1
N X
K X
D n
X
£
¤o
γ(z nk ) xni ln µki + (1 − xni ) ln(1 − µki )
∂µki n=1 k=1 i=1
N
X
£
¤o
∂ n
γ(z nk ) xni ln µki + (1 − xni ) ln(1 − µki )
n=1 ∂µ ki
µ
¶
N
X
xni 1 − xni
γ(z nk )
−
µki 1 − µki
n=1
N
X
n=1
γ(z nk )
xni − µki
µki (1 − µki )
181
Setting the derivative equal to 0, we can obtain:
PN
n=1 γ(z nk )x ni
µki =
PN
n=1 γ(z nk )
=
N
1 X
γ(z nk )xni
Nk n=1
Where Nk is defined as Eq (9.57). If we group all the µki as a column
vector, i.e., µk = [µk1 , µk2 , ..., µkD ]T , we will obtain Eq (9.59) just as required.
Problem 9.16 Solution
We follow the hint beginning by introducing a Lagrange multiplier:
L = E z [ln p(X, Z|µ, π)] + λ(
K
X
πk − 1)
k=1
We calculate the derivative of L with respect to πk and then set it equal
to 0:
N γ(z )
X
∂L
nk
=
+λ = 0
(∗)
∂πk
π
k
n=1
Here E z [ln p] is given by Eq (9.55). We first multiply both sides of the
expression by πk and then adopt summation with respect to k, which gives:
K
N X
X
γ(z nk ) +
n=1 k=1
Noticing that
K
X
λπk = 0
k=1
PK
k=1 π k equals 1, we can obtain:
λ=−
N X
K
X
γ(z nk )
n=1 k=1
Finally, substituting it back into (∗) and rearranging it, we can obtain:
PK
k=1 γ(z nk )
πk = −
λ
PK
γ(z nk )
Nk
=1
= P N kP
=
K
N
n=1 k=1 γ(z nk )
Where Nk is defined by Eq (9.57) and N is the summation of Nk over k,
and also equal to the number of data points.
Problem 9.17 Solution
The incomplete-data log likelihood is given by Eq (9.51), and p(xn |µk ) lies
in the interval [0, 1], which can be easily verified by its definition, i.e., Eq
(9.44). Therefore, we can obtain:
ln p(X|µ, π) =
N
X
n=1
ln
K
©X
k=1
N
K
N
ª X
©X
ª X
πk p(xn |µk ) ≤
ln
πk × 1 ≤
ln 1 = 0
n=1
k=1
n=1
182
Where we have used the fact that the logarithm is monotonic increasing,
and that the summation of πk over k equals 1. Moreover, if we want to achieve
the equality, we need p(xn |µk ) equal to 1 for all n = 1, 2, ..., N. However, this
is hardly possible.
To illustrate this, suppose that p(xn |µk ) equals 1 for all data points. Without loss of generality, consider two data points x1 = [x11 , x12 , ..., x1D ]T and
x2 = [x21 , x22 , ..., x2D ]T , whose i-th entries are different. We further assume
x1 i = 1 and x2 i = 0 since x i is a binary variable. According to Eq (9.44), if we
want p(x1 |µk ) = 1, we must have µ i = 1 (otherwise it muse be less than 1).
However, this will lead p(x2 |µk ) equal to 0 since there is a term 1 − µ i = 0 in
the product shown in Eq (9.44).
Therefore, when the data set is pathological, we will achieve this singularity point by adopting EM. Note that in the main text, the author states
that the condition should be pathological initialization. This is also true. For
instance, in the extreme case, when the data set is not pathological, if we
initialize one πk equal to 1 and others all 0, and some of µ i to 1 and others 0,
we may also achieve the singularity.
Problem 9.18 Solution
In Prob.9.4, we have proved that if we want to maximize the posterior
by EM, the only modification is that in the M-step, we need to maximize
0
Q (θ , θ old ) = Q(θ , θ old ) + ln p(θ ). Here Q(θ , θ old ) has already been given by
E z [ln p], i.e., Eq (9.55). Therefore, we derive for ln p(θ ). Note that ln p(θ ) is
made up of two parts:(i) the prior for µk and (ii) the prior for π, we begin by
dealing with the first part. Here we assume the Beta prior for µki , where k is
fixed, is the same, i.e.,:
p(µki |a k , b k ) =
Γ(a k + b k ) a k −1
µ
(1 − µki )b k −1 ,
Γ(a k )Γ(b k ) ki
i = 1, 2, ..., D
Therefore, the contribution of this Beta prior to ln p(θ ) should be given by:
K X
D
X
(a i − 1) ln µki + (b i − 1) ln(1 − µki )
k=1 i =1
0
One thing worthy mentioned is that since we will maximize Q (θ , θ old )
with respect to π, µk , we can omit the terms which do not depend on π, µk ,
±
such as Γ(a k + b k ) Γ(a k )Γ(b k ). Then we deal with the second part. According
to Eq (2.38), we can obtain:
p(π|α) =
K
Y
Γ(α0 )
α −1
π k
Γ(α1 )...Γ(αK ) k=1 k
Therefore, the contribution of the Dirichlet prior to ln p(θ ) should be given
by:
K
X
k=1
(αk − 1) ln πk
183
0
Therefore, now Q (θ , θ old ) can be written as:
0
Q (θ , θ old ) = E z [ln p] +
i X
K
D h
K X
X
(a i − 1) ln µki + (b i − 1) ln(1 − µki ) + (αk − 1) ln πk
k=1 i =1
k=1
0
Similarly, we calculate the derivative of Q (θ , θ old ) with respect to µki .
This can be simplified by reusing the deduction in Prob.9.15:
∂Q
0
∂µki
=
=
=
=
∂E z [ln p]
∂µki
N
X
+
ai − 1 bi − 1
−
µki
1 − µki
xni 1 − xni
ai − 1 bi − 1
−
)+
−
µki 1 − µki
µki
1 − µki
n=1
PN
PN
(1 − xni )γ(z nk ) + b i − 1
n=1 x ni · γ(z nk ) + a i − 1
− n=1
µki
1 − µki
Nk x̄ki + a i − 1 Nk − Nk x̄ki + b i − 1
−
µki
1 − µki
γ(z nk )(
Note that here x̄ki is defined as the i-th entry of x̄k defined in Eq (9.58).
To be more clear, we have used Eq (9.57) and Eq (9.58) in the last step:
N
X
xni · γ(z nk ) = Nk ·
n=1
h 1 X
N
Nk n=1
i
xni · γ(z nk ) = Nk · x̄ki
Setting the derivative equal to 0 and rearranging it, we can obtain:
µki =
Nk x̄ki + a i − 1
Nk + a i − 1 + b i − 1
0
Next we maximize Q (θ , θ old ) with respect to π. By analogy to Prob.9.16,
we introduce Lagrange multiplier:
L ∝ Ez +
K
X
(αk − 1) ln πk + λ(
k=1
K
X
πk − 1)
k=1
0
Note that the second term on the right hand side of Q in its definition has
been omitted, since that term can be viewed as a constant with regard to π.
We then calculate the derivative of L with respect to πk by taking advantage
of Prob.9.16:
N γ(z )
X
∂L
αk − 1
nk
=
+
+λ = 0
∂πk
πk
n=1 π k
Similarly, We first multiply both sides of the expression by πk and then adopt
summation with respect to k, which gives:
K X
N
X
k=1 n=1
γ(z nk ) +
K
X
k=1
(αk − 1) +
K
X
k=1
λπk = 0
184
Noticing that
PK
k=1 π k equals 1, we can obtain:
K
X
λ=−
Nk −
k=1
K
X
(αk − 1) = − N − α0 + K
k=1
Here we have used Eq (2.39). Substituting it back into the derivative, we
can obtain:
PN
Nk + αk − 1
n=1 γ(z nk ) + α k − 1
πk =
=
−λ
N + α0 − K
It is not difficult to show that if N is large, the update formula for π and
µ in this case (MAP), will reduce to the results given in the main text (MLE).
Problem 9.19 Solution
We first introduce a latent variable z = [z1 , z2 , ..., zK ]T , only one of which
equals 1 and others all 0. The conditional distribution of x is given by:
p(x|z, µ) =
K
Y
p(x|µk ) zk
k=1
The distribution of the latent variable is given by:
p(z|π) =
K
Y
k=1
z
πkk
If we follow the same procedure as in Prob.9.14, we can show that Eq
(9.84) holds. In other words, the introduction of the latent variable is valid.
Therefore, according to Bayes’ Theorem, we can obtain:
p(X, Z|µ, π) =
N
Y
N Y
K h
Y
p(zn |π)p(xn |zn , µ) =
n=1
πk p(x|µ)
i znk
n=1 k=1
We further use Eq (9.85), which gives:
ln p(X, Z|µ, π) =
N X
K
X
n=1 k=1
=
N X
K
X
n=1 k=1
h Y
i
D Y
M
xni j
z nk ln πk
µki j
d =1 j =1
h
z nk ln πk +
D X
M
X
xni j ln µki j
i
d =1 j =1
Similarly, in the E-step, the responsibilities are evaluated using Bayes’
theorem, which gives:
πk p(xn |µk )
γ(z nk ) = E[z nk ] = PK
j =1 π j p(x n |µ j )
185
Next, in the M-step, we are required to maximize E z [ln p(X, Z|µ, π)] with
respect to π and µk , where E z [ln p(X, Z|µ, π)] is given by:
E z [ln p(X, Z|µ, π)] =
K
N X
X
i
h
D X
M
X
xni j ln µki j
γ(z nk ) ln πk +
n=1 k=1
i =1 j =1
Notice that there exists two constraints: (i) the summation of πk over k
equals 1, and (ii) the summation of µki j over j equals 1 for any k and i, we
need to introduce Lagrange multiplier:
K
X
L = E z [ln p] + λ(
πk − 1) +
D
K X
X
η ki (
k=1 i =1
k=1
M
X
µki j − 1)
j =1
First we maximize L with respect to πk . This is actually identical to the
case in the main text. To be more clear, we calculate the derivative of L with
respect to πk :
N γ(z )
X
∂L
nk
=
+λ
∂πk
π
k
n=1
As in Prob.9.16, we can obtain:
πk =
Nk
N
Where Nk is defined as:
N
X
Nk =
γ(z nk )
n=1
N is the summation of Nk over k, and also equals the number of data
points. Then we calculate the derivative of L with respect to µki j :
∂L
∂µki j
=
N γ(z )x
X
nk ni j
n=1
µki j
+ η ki
We set it to 0 and multiply both sides by µki j , which gives:
N
X
γ(z nk ) xni j + η ki µki j = 0
n=1
By analogy to deriving πk , an intuitive idea is to perform summation for
P
the above expression over j and hence we can use the constraint j µki j = 1.
η ki = −
M X
N
X
j =1 n=1
γ(z nk ) xni j = −
N
X
n=1
hX
i
M
N
X
γ(z nk )
xni j = −
γ(z nk ) = − N k
j =1
n=1
186
Where we have used the fact that
derivative, we can obtain:
P
j x ni j = 1. Substituting back into the
PN
µki j = −
n=1 γ(z nk ) x ni j
η ki
=
N
1 X
γ(z nk ) xni j
Nk n=1
Problem 9.20 Solution
We first calculate the derivative of Eq (9.62) with respect to α and set it
to 0:
∂E[ln p]
M 1 2π E[wT w]
=
−
=0
∂α
2 2π α
2
We rearrange the equation above, which gives:
α=
M
(∗)
E[wT w]
Therefore, we now need to calculate the expectation E[wT w]. Notice that
the posterior has already been given by Eq (3.49):
p(w|t) = N (m N , S N )
To calculate E[wT w], here we write down an property for a Gaussian random variable: if x ∼ N (m, Σ), we have:
E[xT Ax] = Tr[AΣ] + mT Am
This property has been shown in Eq(378) in ’the Matrix Cookbook’. Utilizing this property, we can obtain:
E[wT w] = Tr[S N ] + mT
N mN
Substituting it back into (∗), we obtain what is required.
Problem 9.21 Solution
We calculate the derivative of Eq (9.62) with respect to β and set it equal
to 0:
N
N 1 2π 1 X
∂ ln p
=
−
E[(t n − wT φn )2 ] = 0
∂β
2 2π β 2 n=1
Rearranging it, we obtain:
β = PN
N
n=1 E[(t n − w
Tφ
2
n) ]
Therefore, we are required to calculate the expectation. To be more clear,
this expectation is with respect to the posterior defined by Eq (3.49):
p(w|t) = N (m N , S N )
187
We expand the expectation:
E[(t n − wT φn )2 ] = E[t2n − 2t n · wT φn + wT φn φT
n w]
= E[t2n ] − E[2t n · wT φn ] + E[wT (φn φT
n ) w]
=
T
T
T
t2n − 2t n · E[φT
n w] + Tr[φ n φ n S N ] + m N φ n φ n m N
=
T
T
T
t2n − 2t n φT
n · E[w] + Tr[φ n φ n S N ] + m N φ n φ n m N
=
T
T
T
t2n − 2t n φT
n m N + Tr[φ n φ n S N ] + m N φ n φ n m N
2
T
= (t n − mT
N φ N ) + Tr[φ n φ n S N ]
Substituting it back into the derivative, we can obtain:
1
β
o
N n
1 X
2
T
(t n − mT
φ
)
+
Tr[
φ
φ
S
]
n n N
N N
N n=1
o
1n
||t − Φm N ||2 + Tr[ΦT ΦS N ]
N
=
=
Note that in the last step, we have performed vectorization. Here the j-th
row of Φ is given by φ j , identical to the definition given in Chapter 3.
Problem 9.22 Solution
First let’s expand the complete-data log likelihood using Eq (7.79), Eq
(7.80) and Eq (7.76).
ln p(t|X, w, β)p(w|α) = ln p(t|X, w, β) + ln p(w|α)
N
M
X
X
=
ln p(t n | xn , w, β−1 ) +
ln N (w i |0, α−i 1 )
n=1
=
N
X
i =1
ln N (t n |wT φn , β−1 ) +
n=1
=
M
X
ln N (w i |0, α−i 1 )
i =1
N
M
M α
1X
αi X
N
β
β X
i 2
(t n − wT φn )2 +
ln
ln
−
−
w
2
2π 2 n=1
2 i=1 2π i=1 2 i
Therefore, the expectation of the complete-data log likelihood with respect
to the posterior of w equals:
Ew [ln p] =
N
M
M α
β
β X
N
1X
αi X
i
ln
−
−
Ew [w2i ]
Ew [(t n − wT φn )2 ] +
ln
2
2π 2 n=1
2 i=1 2π i=1 2
We calculate the derivative of Ew [ln p] with respect to α i and set it to 0:
∂Ew [ln p]
∂α i
=
1 1 2π 1
− Ew [w2i ] = 0
2 2π α i 2
Rearranging it, we can obtain:
αi =
1
Ew [w2i ]
=
1
Ew [wwT ] ( i,i)
188
Here the subscript (i, i) represents the entry on the i-th row and i-th
column of the matrix Ew [wwT ]. So now, we are required to calculate the
expectation. To be more clear, this expectation is with respect to the posterior
defined by Eq (7.81):
p(w|t, X, α, β) = N (m, Σ)
Here we use Eq (377) described in ’the Matrix Cookbook’. We restate it
here: if w ∼ N (m, Σ), we have:
E[wwT ] = Σ + mmT
According to this equation, we can obtain:
αi =
1
Ew
[wwT ]
=
( i,i )
1
(Σ + mmT )
=
( i,i )
1
Σ ii + m2i
Now We calculate the derivative of Ew [ln p] with respect to β and set it to
0:
∂Ew [ln p]
∂β
=
N
N 1 2π 1 X
Ew [(t n − wT φn )2 ] = 0
−
2 2π β 2 n=1
Rearranging it, we obtain:
β(new) = P N
N
n=1 Ew [(t n − w
Tφ
2
n) ]
Therefore, we are required to calculate the expectation. By analogy to the
deduction in Prob.9.21, we can obtain:
1
β(new)
=
=
o
N n
1 X
(t n − mT φ N )2 + Tr[φn φT
Σ
]
n
N n=1
o
1n
||t − Φm||2 + Tr[ΦT ΦΣ]
N
To make it consistent with Eq (9.68), let’s first prove a statement:
(β−1 A + ΦT Φ)Σ = β−1 I
This can be easily shown by substituting Σ, i.e., Eq(7.83), back into the
expression:
(β−1 A + ΦT Φ) Σ = (β−1 A + ΦT Φ) (A + βΦT Φ)−1 = β−1 I
Now we start from this statement and rearrange it, which gives:
ΦT ΦΣ = β−1 I − β−1 AΣ = β−1 (I − AΣ)
189
Substituting back into the expression for β(new) :
1
β(new)
=
=
=
=
=
o
1n
||t − Φm||2 + Tr[ΦT ΦΣ]
N
o
1n
||t − Φm||2 + Tr[β−1 (I − AΣ)]
N
o
1n
||t − Φm||2 + β−1 Tr[I − AΣ]
N
o
X
1n
||t − Φm||2 + β−1 (1 − α i Σ ii )
N
i
P
||t − Φm||2 + β−1 i γ i
N
Here we have defined γ i = 1 − α i Σ ii as in Eq (7.89). Note that there is a
typo in Eq (9.68), m N should be m.
Problem 9.23 Solution
Some clarifications must be made here, Eq (7.87)-(7.88) only gives the
same stationary points, i.e., the same α? and β? , as those given by Eq (9.67)(9.68). However, the hyper-parameters estimated at some specific iteration
may not be the same by those two different methods.
When convergence is reached, Eq (7.87) can be written as:
α? =
1 − α? Σ ii
m2i
Rearranging it, we can obtain:
α? =
1
m2i + Σ ii
This is identical to Eq (9.67). When convergence is reached, Eq (9.68) can
be written as:
P
||t − Φm||2 + (β? )−1 i γ i
(β? )−1 =
N
Rearranging it, we can obtain:
(β? )−1 =
This is identical to Eq (7.88).
Problem 9.24 Solution
||t − Φm||2
P
N − i γi
190
We substitute Eq (9.71) and Eq (9.72) into Eq (9.70):
n p(X, Z|θ )
p(Z|X, θ ) o
q(Z) ln
− ln
q(Z)
q(Z)
Z
n p(X, Z|θ ) o
X
=
q(Z) ln
p(Z|X, θ )
Z
X
=
q(Z) ln p(X|θ )
L(q, θ ) + KL (q|| p) =
X
Z
= ln p(X|θ )
Note that in the last step, we have used the fact that ln p(X|θ ) doesn’t
depend on Z, and that the summation of q(Z) over Z equal to 1 because q(Z)
is a PDF.
Problem 9.25 Solution
We calculate the derivative of Eq (9.71) with respect to θ , given q(Z) =
p(Z|X, θ (old) ):
∂L(q, θ )
∂θ
=
=
=
=
=
=
=
∂ nX
∂θ
p(Z|X, θ (old) ) ln
p(X, Z|θ ) o
p(Z|X, θ (old) )
o
X
∂ nX
p(Z|X, θ (old) ) ln p(X, Z|θ ) − p(Z|X, θ (old) ) ln p(Z|X, θ (old) )
∂θ Z
Z
o
∂ nX
p(Z|X, θ (old) ) ln p(X, Z|θ )
∂θ Z
X
∂ ln p(X, Z|θ )
p(Z|X, θ (old) )
∂θ
Z
X
∂ p(X, Z|θ )
1
p(Z|X, θ (old) )
p(X,
Z
|
θ
)
∂θ
Z
X
1
∂ p(X|θ ) · p(Z|X, θ )
p(Z|X, θ (old) )
p(X, Z|θ )
∂θ
Z
h
(old)
X p(Z|X, θ
)
∂ p(Z|X, θ )
∂ p(X|θ ) i
p(X|θ )
+ p(Z|X, θ )
p(X, Z|θ )
∂θ
∂θ
Z
Z
191
We evaluate this derivative at θ = θ old :
n X p(Z|X, θ (old) ) h
∂ p(Z|X, θ )
∂ p(X|θ ) i o¯¯
∂L(q, θ ) ¯¯
p(X|θ )
+ p(Z|X, θ )
¯ old =
¯ old
θ
θ
∂θ
p(X, Z|θ )
∂θ
∂θ
Z
¯
¯
i
X p(Z|X, θ (old) ) h
(old) ∂ p(Z|X, θ ) ¯
(old) ∂ p(X|θ ) ¯
p(X
|
θ
)
=
+
p(Z
|
X,
θ
)
¯
¯
(old)
(old)
(old)
θ
θ
∂θ
∂θ
)
Z p(X, Z|θ
¯
¯
h
i
X
1
(old) ∂ p(Z|X, θ ) ¯
(old) ∂ p(X|θ ) ¯
=
p(X
|
θ
)
+
p(Z
|
X,
θ
)
¯
¯
(old)
θ (old)
θ (old)
∂θ
∂θ
)
Z p(X|θ
X ∂ p(Z|X, θ ) ¯¯
X p(Z|X, θ (old) ) ∂ p(X|θ ) ¯¯
·
=
¯ (old) +
¯ (old)
(old)
θ
θ
∂θ
∂θ
)
Z
Z p(X|θ
¯
¯
X ∂ p(Z|X, θ ) ¯
∂ p(X|θ ) ¯
1
·
=
¯ (old) +
¯ (old)
(old)
θ
θ
∂θ
∂θ
p(X|θ
)
Z
¯
X ∂ p(Z|X, θ ) ¯¯
∂ ln p(X|θ ) ¯
=
¯ (old) +
¯ (old)
θ
θ
∂θ
∂θ
Z
n ∂ X
o¯
∂ ln p(X|θ ) ¯¯
¯
=
p(Z|X, θ ) ¯ (old) +
¯ (old)
θ
θ
∂θ Z
∂θ
¯
¯
∂ ln p(X|θ ) ¯
∂1 ¯
=
¯ (old) +
¯ (old)
θ
θ
∂θ
∂θ
¯
∂ ln p(X|θ ) ¯
=
¯ (old)
θ
∂θ
This problem can be much easier to prove if we view it from the perspective of KL divergence. Note that when q(Z) = p(Z|X, θ (old) ), the KL divergence
vanishes, and that in general KL divergence is less or equal to zero. Therefore, we must have:
∂K L(q|| p) ¯¯
¯ (old) = 0
θ
∂θ
Otherwise, there exists a point θ in the neighborhood near θ (old) which
leads the KL divergence less than 0. Then using Eq (9.70), it is trivial to
prove.
Problem 9.26 Solution
From Eq (9.18), we have:
Nkold =
X old
γ (z nk )
n
If now we just re-evaluate the responsibilities for one data point xm , we can
obtain:
X old
Nknew =
γ (z nk ) + γnew (z mk )
n6= m
X old
=
γ (z nk ) + γnew (z mk ) − γold (z mk )
n
=
Nkold + γnew (z mk ) − γold (z mk )
192
Similarly, according to Eq (9.17), we can obtain:
µnew
k
=
=
=
=
1
Nknew n6=m
=
γold (z nk )xn +
γnew (z mk )xm
Nknew
X old
γnew (z mk )xm γold (z mk )xm
γ
(z
)x
+
−
nk n
Nknew n
Nknew
Nknew
1
Nkold
γnew (z mk )xm γold (z mk )xm
1 X old
γ
(z
)x
+
−
n
nk
Nknew N old n
Nknew
Nknew
k
Nkold
h
i x
m
old
new
old
µ
+
γ
(z
)
−
γ
(z
)
mk
mk
new k
new
Nk
Nk
Nknew − Nkold
h
i x
m
old
new
old
µ
+
γ
(z
)
−
γ
(z
)
mk
mk
k
Nknew
Nknew
i x
γnew (z mk ) − γold (z mk ) old h new
m
(z mk ) − γold (z mk )
µk + γ
µold
k −
new
Nk
Nknew
´
γnew (z mk ) − γold (z mk ) ³
old
µold
+
·
x
−
µ
m
k
k
Nknew
= µold
k −
=
X
Just as required.
Problem 9.27 Solution
By analogy to the previous problem, we use Eq (9.24)-Eq(9.27), beginning
by first deriving an update formula for mixing coefficients πk :
πnew
k
Nknew
o
1 n old
Nk + γnew (z mk ) − γold (z mk )
N
N
new
γ
(z mk ) − γold (z mk )
= πold
+
k
N
=
=
Here we have used the conclusion (the update formula for Nknew ) in the
previous problem. Next we deal with the covariance matrix Σ. By analogy to
193
the previous problem, we can obtain:
Σnew
k
=
1
new T
γold (z nk ) (xn − µnew
k ) (x n − µ k )
Nknew n6=m
1
new T
γnew (z mk ) (xm − µnew
k ) (x m − µ k )
Nknew
1 X old
old T
γ (z nk ) (xn − µold
k ) (x n − µ k )
Nknew n6=m
+
≈
X
1
new T
γnew (z mk ) (xm − µnew
k ) (x m − µ k )
Nknew
1 X old
old T
=
γ (z nk ) (xn − µold
k ) (x n − µ k )
Nknew n
1
new T
+ new γnew (z mk ) (xm − µnew
k ) (x m − µ k )
Nk
1
old T
− new γold (z mk ) (xm − µold
k ) (x m − µ k )
Nk
1
1
old old
new T
=
γnew (z mk ) (xm − µnew
k ) (x m − µ k )
new N k Σ k +
Nk
Nknew
1
old T
− new γold (z mk ) (xm − µold
k ) (x m − µ k )
Nk
+
=
n
N old − N new o old
Σk
1 + k newk
Nk
1
new T
γnew (z mk ) (xm − µnew
k ) (x m − µ k )
Nknew
1
old T
− new γold (z mk ) (xm − µold
k ) (x m − µ k )
Nk
n
γold (z mk ) − γnew (z mk ) o old
=
1+
Σk
Nknew
+
+
−
γnew (z mk )
Nknew
γold (z mk )
Nknew
new T
(xm − µnew
k ) (x m − µ k )
old T
(xm − µold
k ) (x m − µ k )
= Σold
k
o
γnew (z mk ) n
new T
old
(xm − µnew
k ) (x m − µ k ) − Σ
new
Nk
o
γold (z mk ) n
old
old T
old
−
(x
−
µ
)
(x
−
µ
)
−
Σ
m
m
k
k
k
Nknew
+
One important thing worthy mentioned is that in the second step, there
is an approximate equal sign. Note that in the previous problem, we have
194
shown that if we only recompute the data point xm , all the center µk will
also change from µold
to µnew
, and the update formula is given by Eq (9.78).
k
k
However, for the convenience of computing, we have made an approximation
here. Other approximation methods can also be applied here. For instance,
you can replace µnew
with µold
whenever it occurs.
k
k
The complete solution should be given by substituting Eq (9.78) into the
right side of the first equal sign and then rearranging it, in order to construct
a relation between Σnew
and Σold
. However, this is too complicated.
k
k
0.10
Variational Inference
Problem 10.1 Solution
This problem is very similar to Prob.9.24. We substitute Eq (10.3) and Eq
(10.4) into Eq (10.2):
Z
n p(X, Z)
p(Z|X) o
q(Z) ln
L(q) + KL (q|| p) =
− ln
dZ
q(Z)
q(Z)
Z
Z
n p(X, Z) o
q(Z) ln
=
dZ
p(Z|X)
ZZ
q(Z) ln p(X)dZ
=
Z
= ln p(X)
Note that in the last step, we have used the fact that ln p(X) doesn’t depend on Z, and that the integration of q(Z) over Z equal to 1 because q(Z) is
a PDF.
Problem 10.2 Solution
To be more clear, we are required to solve:
½
1
m 1 = µ 1 − Λ−
11 Λ12 (m 2 − µ2 )
1
m 2 = µ 2 − Λ−
22 Λ21 (m 1 − µ1 )
To obtain the equation above, we need to substitute E[z i ] = m i , where
i = 1, 2, into Eq (10.13) and Eq (10.14). Here the unknown parameters are
m 1 and m 2 . It is trivial to notice that m i = µ i is a solution for the equation
above.
Let’s solve this equation from another perspective. Firstly, if any (or both)
1
−1
of Λ−
11 and Λ22 equals 0, we can obtain m i = µ i directly from Eq (10.13)1
−1
(10.14). When none of Λ−
11 and Λ22 equals 0, we substitute m 1 , i.e., the first
195
line, into the second line:
m2
1
= µ2 − Λ−
22 Λ21 (m 1 − µ1 )
h
i
1
−1
= µ 2 − Λ−
22 Λ21 µ1 − Λ11 Λ12 (m 2 − µ2 ) − µ1
1
−1
−1
−1
= µ 2 − Λ−
22 Λ21 µ1 + Λ22 Λ21 Λ11 Λ12 (m 2 − µ2 ) + Λ22 Λ21 µ1
1
−1
−1
−1
= (1 − Λ−
22 Λ21 Λ11 Λ12 ) µ2 + Λ22 Λ21 Λ11 Λ12 m 2
We rearrange the expression above, yielding:
1
−1
(1 − Λ−
22 Λ21 Λ11 Λ12 ) (m 2 − µ2 ) = 0
The first term at the left hand side will equal 0 only when the distribution is singular, i.e., the determinant of the precision matrix Λ (i.e., Λ11 Λ22 −
Λ12 Λ21 ) is 0. Therefore, if the distribution is nonsingular, we must have
m 2 = µ2 . Substituting it back into the first line, we obtain m 1 = µ1 .
Problem 10.3 Solution
Let’s start from the definition of KL divergence given in Eq (10.16).
Z
hX
i
M
K L(p|| q) = − p(Z)
ln q i (Z i ) dZ + const
i =1
Z
= −
h
i
X
p(Z) ln q j (Z j ) + ln q i (Z i ) dZ + const
i 6= j
Z
p(Z) ln q j (Z j ) dZ + const
Z hZ
i
Y
= −
p(Z) dZ i ln q j (Z j ) dZ j + const
= −
i 6= j
Z
= −
P(Z j ) ln q j (Z j ) dZ j + const
Note that in the third step, since all the factors q i (Z i ), where i 6= j, are
fixed, they can be absorbed into the ’Const’ variable. In the last step, we have
denoted the marginal distribution:
Z
Y
p(Z j ) = p(Z) dZ i
i 6= j
We introduce the Lagrange multiplier to enforce q j (Z j ) integrate to 1.
Z
Z
L = − P(Z j ) ln q j (Z j ) dZ j + λ ( q j (Z j ) dZ j − 1)
Using the functional derivative (for more details, you can refer to Appendix D or Prob.1.34), we calculate the functional derivative of L with respect to q j (Z j ) and set it to 0:
−
p(Z j )
q j (Z j )
+λ = 0
196
Rearranging it, we can obtain:
λ q j (Z j ) = p(Z j )
Integrating both sides with respect to Z j , we see that λ = 1. Substituting
it back into the derivative, we can obtain the optimal q j (Z j ):
q?j (Z j ) = p(Z j )
Notice that actually we should also enforce q j (Z j ) > 0 in the Lagrange
multiplier, however as we can see that when we only enforce q j (Z j ) integrate
to 1 and obtain the final close expression, q j (Z j ) is definitely larger than 0 at
all Z j because p(Z j ) is a PDF. Therefore, there is no need to introduce this
inequality constraint in the Lagrange multiplier.
Problem 10.4 Solution
We begin by writing down the KL divergence.
Z
n q(x) o
dx
KL (p|| q) = − p(x) ln
p(x)
Z
= − p(x) ln q(x) dx + const
Z
h D
i
1
1
= − p(x) − ln 2π − ln |Σ| − (x − µ)T Σ−1 (x − µ) dx + const
2
2
2
Z
h1
i
1
=
p(x) ln |Σ| + (x − µ)T Σ−1 (x − µ) dx + const
2
2
Z
h1
i
1
ln |Σ| + p(x) (x − µ)T Σ−1 (x − µ) dx + const
=
2
2
Z
i
1
1 h T −1
=
ln |Σ| + p(x) x Σ x − 2µT Σ−1 x + µT Σ−1 µ dx + const
2
2
Z
1
1
1
=
ln |Σ| +
p(x) Tr[Σ−1 (xxT )] dx − µT Σ−1 E[x] + µT Σ−1 µ + const
2
2
2
1
1
1 T −1
−1
T
T −1
=
ln |Σ| + Tr[Σ E(xx )] − µ Σ E[x] + µ Σ µ + const
2
2
2
Here D is the dimension of x. We first calculate the derivative of KL (p|| q)
with respect to µ and set it to 0:
∂KL
∂µ
= −Σ−1 E[x] + Σ−1 µ = 0
Therefore, we can obtain µ = E[x]. When µ = E[x] is satisfied, KL divergence reduces to:
KL (p|| q) =
1
1
1
ln |Σ| + Tr[Σ−1 E(xxT )] − µT Σ−1 µ + const
2
2
2
197
Then we calculate the derivative of KL (p|| q) with respect to Σ and set it
to 0:
∂KL
∂Σ
=
1 −1 1 −1
1
Σ − Σ E[xxT ]Σ−1 + Σ−1 µµT Σ−1 = 0
2
2
2
Note that here we have used and Eq (61) and Eq (124) in ’MatrixCookBook’, and that Σ, E[xxT ] are both symmetric. We rewrite those equations
here for your reference:
∂aT X−1 b
∂X
= −X−T abT X−T
and
∂Tr(AX−1 B)
∂X
= −X−T AT BT X−T
Rearranging the derivative, we can obtain:
Σ = E[xxT ] − µµT = E[xxT ] − E[x]E[x]T = cov[x]
Problem 10.5 Solution
We introduce a property of Dirac function:
Z
δ(θ − θ 0 ) f (θ ) d θ = f (θ 0 )
We first calculate the optimal q(z, θ ) by fixing q θ (θ ). This is achieved by
minimizing the KL divergence given in Eq (10.4):
Z Z
n p(Z|X) o
KL(q|| p) = −
q(Z) ln
dZ
q(Z)
Z Z
n p(z, θ |X) o
= −
q z (z)q θ (θ ) ln
dz d θ
q z (z)q θ (θ )
Z Z
Z
n p(z, θ |X) o
= −
q z (z)q θ (θ ) ln
dz d θ + q θ (θ ) ln q θ (θ ) d θ
q z (z)
Z Z
n p(z, θ |X) o
= −
q z (z)q θ (θ ) ln
dz d θ + const
q z (z)
Z
nZ
n p(z, θ |X) o o
q z (z) ln
= − q θ (θ )
dz d θ + const
q z (z)
Z
n p(z, θ |X) o
0
= − q z (z) ln
dz + const
q z (z)
Z
n p(z|θ , X)p(θ |X) o
0
0
dz + const
= − q z (z) ln
q z (z)
Z
n p(z|θ , X) o
0
= − q z (z) ln
dz + const
q z (z)
Here the ’Const’ denotes the terms independent of q z (z). Note that we
will show at the end of this problem, here ’Const’ actually is −∞ due to the
existence of the entropy of Dirac function:
Z
q θ (θ ) ln q θ (θ ) d θ
198
Now it is clear that when q z (z) equals p(z|θ 0 , X), the KL divergence is
minimized. This corresponds to the E-step. Next, we calculate the optimal
q θ (θ ), i.e., θ 0 , by maximizing L(q) given in Eq (10.3), but fixing q θ (θ ):
Z Z
n p(X, Z) o
L(q) =
q(Z) ln
dZ
q(Z)
Z Z
n p(X, z, θ ) o
=
q z (z)q θ (θ ) ln
dz d θ
q z (z)q θ (θ )
Z
Z Z
n p(X, z, θ ) o
dz d θ − q θ (θ ) ln q θ (θ ) d θ
=
q z (z)q θ (θ ) ln
q z (z)
Z Z
Z
n
o
=
q z (z)q θ (θ ) ln p(X, z, θ ) dz d θ − q θ (θ ) ln q θ (θ ) d θ + const
Z
Z
=
q θ (θ ) E qz [ln p(X, z, θ )] d θ − q θ (θ ) ln q θ (θ ) d θ + const
Z
= E qz (z) [ln p(X, z, θ 0 )] − q θ (θ ) ln q θ (θ ) d θ + const
The second term is actually the entropy of a Dirac function, which is −∞
and independent of the value of θ 0 . Not strictly speaking, we only need to
maximize the first term. This is exactly the M-step.
One important thing needs to be clarified here. You may ask no matter
how we set θ 0 , L(q) will always be −∞. Actually, this is an intrinsic problem
as long as we use a point estimate q θ (θ ). This will even occur when we derive
the optimal q z (z) by minimizing the KL divergence at the first step. Therefore, the ’Maximizing’ and ’Minimizing’ is a general meaning in this problem
where we neglect the −∞ term.
Problem 10.6 Solution
199
Let’s use the hint by first enforcing α → 1.
Z
´
4 ³
(1+α)/2 (1−α)/2
1
−
p
q
dx
D α (p|| q) =
1 − α2
Z
£
4 n
p
1−α
1−α 2 ¤ o
=
1
−
1
+
ln
q
+
O(
) dx
2
2
1 − α2
p(1−α)/2
Z
o
1 + 1−2α ln q + O( 1−2α )2
4 n
=
1
−
p
·
dx
1 − α2
1 + 1−2α ln p + O( 1−2α )2
Z
o
1 + 1−2α ln q
4 n
1
−
p
·
≈
dx
1 − α2
1 + 1−2α ln p
Z
o
£ 1 + 1−2α ln q
¤
4 n
−
p
·
−
1
dx
=
1 − α2
1 + 1−2α ln p
Z
1−α
1−α
o
4 n
2 ln q − 2 ln p
=
−
p
·
dx
1 − α2
1 + 1−2α ln p
Z
n
o
2
ln q − ln p
=
− p·
dx
1+α
1 + 1−2α ln p
Z
Z
q
≈ − p · (ln q − ln p) dx = − p · ln dx
p
Here p and q is short for p(x) and q(x). It is similar when α → −1. One
important thing worthy mentioning is that if we directly approximate p(1+α)/2
by p instead of p/p(1−α)/2 in the first step, we won’t get the desired result.
Problem 10.7 Solution
Let’s begin from Eq (10.25).
o
N
X
E[ τ ] n
λ0 (µ − µ0 )2 +
(xn − µ)2 + const
2
n=1
o
N
N
X
X
E[ τ ] n
= −
λ0 µ2 − 2 λ0 µ0 µ + λ0 µ20 + N µ2 − 2 (
xn ) µ +
x2n + const
2
n=1
n=1
n
o
N
N
X
X
E[ τ ]
= −
(λ0 + N) µ2 − 2 ( λ0 µ0 +
xn ) µ + (λ0 µ20 +
x2n ) + const
2
n=1
n=1
PN
2 PN
n
λ0 µ0 + n=1 x2n o
λ0 µ0 + n=1 xn
E[τ] (λ0 + N) 2
= −
µ −2
µ+
+ const
2
λ0 + N
λ0 + N
ln q?
µ (µ) = −
From this expression, we see that q?
µ (µ) should be a Gaussian. Suppose
?
−1
that is has form: q µ (µ) ∼ N (µ|µ N , λ N ), then its logarithm can be written as:
ln q?
µ (µ) =
1 λN λN
ln
−
(µ − µ N )2
2 2π
2
200
We match the terms related to µ (the quadratic term and linear term),
yielding:
P
λ0 µ0 + nN=1 xn
λ N = E[τ] · (λ0 + N) , and λ N µ N = E[τ] · (λ0 + N) ·
λ0 + N
Therefore, we obtain:
µN =
λ0 µ0 + N x̄
λ0 + N
Where x̄ is the mean of xn , i.e.,
N
1 X
xn
N n=1
x̄ =
Then we deal with the other factor q τ (τ). Note that there is a typo in
Eq (10.28), the coefficient ahead of ln τ should be N2+1 . Let’s verify this by
considering the terms introducing ln τ. The first term inside the expectation,
i.e., ln p(D |µ, τ), gives N
2 ln τ, and the second term inside the expectation, i.e.,
1
ln p(µ|τ), gives 2 ln τ. Finally the last term ln p(τ) gives (a 0 − 1) ln τ. Therefore,
Eq (10.29), Eq (10.31) and Eq (10.33) will also change consequently. The right
forms of these equations will be given in this and following problems.
Now suppose that q τ (τ) is a Gamma distribution, i.e., q τ (τ) ∼ Gam(τ|a N , b N ),
we have:
ln q τ (τ) = − ln Γ(a N ) + a N ln b N + (a N − 1) ln τ − b N τ
Comparing it with Eq (10.28) and matching the coefficients ahead of τ
and ln τ, we can obtain:
a N − 1 = a0 − 1 +
N +1
2
a N = a0 +
⇒
N +1
2
And similarly
i
N
1 hX
b N = b 0 + Eµ
(xn − µ)2 + λ0 (µ − µ0 )2
2
n=1
Just as required.
Problem 10.8 Solution
According to Eq (B.27), we have:
E[τ] =
≈
=
=
a 0 + (N + 1)/2
b 0 + 21 Eµ
hP
N
2
2
n=1 (x n − µ) + λ0 (µ − µ0 )
N/2
1
2 Eµ
hP
N
2
n=1 (x n − µ)
i
N
Eµ
hP
n1
N
N
2
n=1 (x n − µ)
· Eµ
hX
N
n=1
i
(xn − µ)2
io−1
i
201
According to Eq (B.28), we have:
var[τ] =
≈
³
a 0 + (N + 1)/2
£
¤´2
P
N
2 + λ (µ − µ )2
(x
−
µ
)
b 0 + 12 Eµ
n
0
0
n=1
N/2
1
4 Eµ
£ PN
2
n=1 (x n − µ)
¤2 ≈ 0
Just as required.
Problem 10.9 Solution
The underlying assumption of this problem is a 0 = b 0 = λ0 = 0. According to Eq (10.26), Eq (10.27) and the definition of variance, we can obtain:
1
2
E[µ2 ] = λ−
N + E[ µ ] =
=
1
λ0 µ0 + N x 2
+(
)
(λ0 + N)E[τ]
λ0 + N
1
+ x2
N E[τ]
Note that since there is a typo in Eq (10.29) as stated in the previous
problem, i.e., missing a term 21 . E[τ]−1 actually equals:
hP
i
1
N
2
2
b
+
E
(x
−
µ
)
+
λ
(
µ
−
µ
)
0
0
0
n=1 n
bN
1
2 µ
=
=
E[τ]
aN
a 0 + (N + 1)/2
hP
i
1
N
2
E
(x
−
µ
)
n=1 n
2 µ
=
(N + 1)/2
N
1 X
(xn − µ)2 ]
= Eµ [
N + 1 n=1
=
=
=
=
=
N
N
1 X
Eµ [
(xn − µ)2 ]
N +1
N n=1
o
N n 2
x − 2xE[µ] + E[µ2 ]
N +1
o
N n 2
1
x − 2x2 +
+ x2
N +1
N E[τ]
1 o
N n 2
x − x2 +
N +1
N E[τ]
o
N n 2
1
2
x −x +
N +1
(N + 1)E[τ]
Rearranging it, we can obtain:
N
1
1 X
= (x2 − x2 ) =
(xn − x)2
E[τ]
N n=1
Actually it is still a biased estimator.
202
Problem 10.10 Solution
We substitute L m , i.e., Eq (10.35), back into the right hand side of Eq
(10.34), yielding:
n
XX
p(Z, X, m)
p(Z, m|X) o
(right) =
q(Z| m)q(m) ln
− ln
q(Z| m)q(m)
q(Z| m)q(m)
m Z
n
o
XX
p(Z, X, m)
=
q(Z| m)q(m) ln
p(Z, m|X)
m Z
XX
=
q(Z, m) ln p(X)
m Z
= ln p(X)
Just as required.
Problem 10.11 Solution
We introduce the Lagrange Multiplier:
n p(Z, X, m) o
nX
o
XX
L =
q(Z| m)q(m) ln
−λ
q(m) − 1
q(Z| m)q(m)
m Z
m
o
n
o XX
nX
XX
q(m) − 1
=
q(Z| m)q(m) ln p(Z, X, m) − q(Z| m) −
q(Z| m)q(m) ln q(m) − λ
m Z
=
X
m Z
q(m) · C −
m
X
nX
o
nX
o
q(Z| m)
q(m) ln q(m) − λ
q(m) − 1
m
Z
m
Where we have defined:
n
o
X
C=
q(Z| m) ln p(Z, X, m) − q(Z| m)
Z
According to Calculus of Variations given in Appendix D, and also Prob.1.34,
we can obtain the derivative of L with respect to q(m) and set it to 0:
h
i
X
∂L
= C + q(Z| m) ln q(m) + 1 − λ
∂ q(m)
Z
n
o X
X
=
q(Z| m) ln p(Z, X, m) − q(Z| m) + q(Z| m) ln q(m) + 1 − λ
Z
=
X
Z
Z
q(Z| m) ln
n p(Z, X, m) o
q(Z| m)q(m)
+1−λ = 0
We multiply both sides by q(m) and then perform summation over m,
yielding:
n p(Z, X, m) o
XX
X
q(Z| m)q(m) ln
+ (1 − λ) q(m) = 0
q(Z| m)q(m)
m Z
m
Notice that the first term is actually L m defined in Eq (10.35) and that
the summation of q(m) over m equals 1, we can obtain:
λ = Lm + 1
m
203
We substitute λ back into the derivative, yielding:
X
q(Z| m) ln
n p(Z, X, m) o
q(Z| m)q(m)
Z
− Lm = 0
(∗)
One important thing must be clarified here, there is a typo in Eq (10.36),
00
L m in Eq (10.36) should be L , which is defined as:
00
L =
X
q(Z| m) ln
n p(Z, X| m) o
q(Z| m)
Z
00
Now with the definition of L , we expand (∗):
(∗) =
X
q(Z| m) ln
Z
=
X
q(Z| m) ln
Z
=
=
=
q(Z| m)q(m)
− Lm
n p(Z, X| m)p(m) o
q(Z| m)q(m)
− Lm
p(m)
− Lm
q(m)
Z
n p(Z, X, m) o
00
p(m) X X
−
q(Z| m)q(m) ln
L + ln
q(m) m Z
q(Z| m)q(m)
n
X
X
00
p(m)
p(Z, X| m)p(m) o
L + ln
q(Z| m) ln
− q(m)
q(m) m
q(Z| m)q(m)
Z
n
X
00
00
p(m) X
p(m) o
L + ln
− q(m) L + q(Z| m) ln
q(m) m
q(m)
Z
n
o
X
00
00
p(m)
p(m)
− q(m) L + ln
L + ln
q(m) m
q(m)
00
= L +
=
n p(Z, X, m) o
X
q(Z| m) ln
00
00
p(m) exp(L ) X
p(m) exp(L )
= ln
− q(m) ln
=0
q(m)
q(m)
m
The solution is given by:
q(m) =
00
1
· p(m) exp(L )
A
Where A1 is a normalization constant, used to guarantee the summation
of q(m) over m equals 1. More specific, it is given by:
X
00
A=
p(m) exp(L )
Z
Therefore, it is obvious that A does not depend on the value of Z. You
can verify the result of q(m) by substituting it back into the last line of (∗),
yielding:
00
00
X
p(m) exp(L ) X
p(m) exp(L )
ln
− q(m) ln
= ln A − q(m) · ln A = 0
q(m)
q(m)
m
m
204
One last thing worthy mentioning is that you can directly start from L m
given in Eq (10.35), without enforcing Lagrange Multiplier, to obtain q(m).
In this way, we can actually obtain:
00
p(m) exp(L )
Lm =
q(m) ln
q(m)
m
X
00
It is actually the KL divergence between q(m) and p(m) exp(L ). Note
00
00
that p(m) exp(L ) is not normalized, we cannot let q(m) equal to p(m) exp(L )
to achieve the minimum of a KL distance, i.e., 0, since q(m) is a probability
distribution and should sum to 1 over m.
Therefore, we can guess that the optimal q(m) is given by the normal00
ized p(m) exp(L ). In this way, the constraint, i.e., summation of q(m) over
m equals 1, is implicitly guaranteed. The more strict proof using Lagrange
Multiplier has been shown above.
Problem 10.12 Solution
The solution procedure has already been given in Eq (10.43) - (10.49), so
here we explain it in more details, starting from Eq (10.43):
ln q? (Z) = Eπ,µ,Λ [ln p(X, Z, π, µ, Λ)] + const
= Eπ [ln p(Z|π)] + Eµ,Λ [ln p(X|Z, µ, Λ)] + const
= const + Eπ [
N X
K
X
z nk ln πk ]
n=1 k=1
+Eµ,Λ [
N X
K
X
z nk {
n=1 k=1
= const +
N X
K
X
D
1
1
ln |Λk | − ln 2π − (xn − µk )T Λk (xn − µk ) } ]
2
2
2
z nk Eπ [ln πk ]
n=1 k=1
+
N X
K
X
z nk Eµ,Λ [{
n=1 k=1
=
N X
K
X
D
1
1
ln |Λk | − ln 2π − (xn − µk )T Λk (xn − µk ) } ]
2
2
2
z nk ln ρ nk + const
n=1 k=1
Where we have substituted used Eq (10.37) and Eq (10.38), and D is the
dimension of xn . Here ln ρ nk is defined as:
ln ρ nk
D
1
1
ln |Λk | − ln 2π − (xn − µk )T Λk (xn − µk ) } ]
2
2
2
1
D
1
= Eπ [ln πk ] + Eµ,Λ [ ln |Λk |] − ln 2π − Eµk ,Λk [ (xn − µk )T Λk (xn − µk ) ]
2
2
2
= Eπ [ln πk ] + Eµ,Λ [{
Taking exponential of both sides, we can obtain:
q? (Z) ∝
N Y
K
Y
n=1 k=1
z
nk
ρ nk
205
Because q? (Z) should be correctly normalized, we are required to find
the normalization constant. In this problem, we find that directly calculate
the normalization constant by performing summation of q? (Z) over Z is non
trivial. Therefore, we will proof that Eq (10.49) is the correct normalization
Q
z1 k
by mathematical induction. When N = 1, q? (Z) will reduce to: K
k=1 ρ 1 k , and
it is easy to see that the normalization constant is given by:
A=
K
XY
z1 k=1
K
X
z
ρ 11kk =
ρ1 j
j =1
Here we have used 1-of-K coding scheme for z1 = [z11 , z12 , ..., z1K ]T , i.e.,
only one of { z11 , z12 , ..., z1K } will be 1 and others all 0. Therefore the summation over z1 is made up of K terms, and the j-th term corresponds to z1 j = 1
and other z1 i equals 0. In this case, we have obtained:
q? (Z) =
´ z1 k
K
K ³
Y
1 Y
ρ 1k
z
ρ 11kk =
PK
A k=1
k=1
j =1 ln ρ 1 j
It is exactly the same as Eq (10.48) and Eq (10.49). Suppose now we have
proved that for N − 1, the normalized q? (Z) is given by Eq (10.48) and Eq
(10.49). For N, we have:
X
q? (Z) =
X
=
Xn
K
N Y
Y
z1 ,...,z N n=1 k=1
Z
zN
=
=
K
£Y
z1 ,...,z N −1 k=1
k=1
K
X n£ Y
zN
=
X
K
X n£ Y
zN
K
N Y
Y
z1 ,...,z N −1 n=1 k=1
Xn
zN
=
X
z
nk
r nk
k=1
K
XY
z N k=1
z
o
o
K
NY
−1 Y
z nk ¤
z ¤ £
r nk
r NNkk ·
n=1 k=1
zN k ¤
rNk ·
z
nk
r nk
X
NY
−1 Y
K
z1 ,...,z N −1 n=1 k=1
z
nk
r nk
o
o
z ¤
r NNkk · 1
r NNkk =
K
X
rNk = 1
k=1
The proof of the final step is exactly the same as that for N = 1. So now,
with the assumption Eq (10.48) and Eq (10.49) are right for N − 1, we have
shown that they are also correct for N. The proof is complete.
Problem 10.13 Solution
206
Let’s start from Eq (10.54).
ln q? (π, µ, Λ) ∝ ln p(π) +
K
X
ln p(µk , Λk ) + E[ln p(Z|π)] +
k=1 n=1
k=1
=
ln C(α0 ) +
N
K X
X
K
X
1
E[z nk ] ln N (xn |µk , Λ−
k )
(α0 − 1) ln πk
k=1
+
K
X
ln N (µk |m0 , (β0 Λk )−1 ) +
ln W (Λk |W0 , v0 )
k=1
k=1
+
K
X
K
N X
X
ln πk E[z nk ] +
N
K X
X
k=1 n=1
n=1 k=1
1
E[z nk ] ln N (xn |µk , Λ−
k )
It is easy to observe that the equation above can be decomposed into a
sum of terms involving only π together with those only involving µ and Λ. In
other words, q(π, µ, Λ) can be factorized into the product of q(π) and q(µ, Λ).
We first extract those terms depend on π.
ln q? (π) ∝ (α0 − 1)
K
X
ln πk +
(α0 − 1)
K
X
ln πk +
K
X
K X
N
X
r nk ln πk
k=1 n=1
k=1
=
ln πk E[z nk ]
k=1 n=1
k=1
=
K X
N
X
h
i
N
X
ln πk · α0 − 1 +
r nk
n=1
k=1
Comparing it to the standard form of a Dirichlet distribution, we can conclude that q? (π) = Dir(π|α), where the k-th entry of α, i.e., αk is given by:
N
X
αk = α0 +
r nk = α0 + Nk
n=1
Next we gather all the terms dependent on µ = {µk } and Λ = {Λk }:
ln q? (µ, Λ)
=
K
X
ln N (µk |m0 , (β0 Λk )−1 ) +
k=1
K
X
ln W (Λk |W0 , v0 ) +
k=1
K X
N
X
k=1 n=1
1
E[z nk ] ln N (xn |µk , Λ−
k )
K n1
X
o
1
∝
ln |β0 Λk | − (µk − m0 )T β0 Λk (µk − m0 )
2
k=1 2
n
o
K
X v0 − D − 1
1
1
+
ln |Λk | − Tr(W−
Λ
)
k
0
2
2
k=1
n1
o
K X
N
X
1
+
r nk ln |Λk | − (xn − µk )T Λk (xn − µk )
2
2
k=1 n=1
With the knowledge that the optimal q? (µ, Λ) can be written as:
q? (µ, Λ) =
K
Y
k=1
q? (µk |Λk )q? (Λk ) =
K
Y
k=1
N (µk |mk , (βk Λk )−1 )W (Λk |Wk , vk ) (∗)
207
We first complete square with respect to µk . The quadratic term is given
by:
K X
N
X
1
1
1 T
− µT
(
β
Λ
)
µ
−
r nk µT
0
k
k
k
k Λ k µ k = − µ k (β0 Λ k + N k Λ k ) µ k
2
2
2
k=1 n=1
Therefore, comparing with (∗), we can obtain:
βk = β0 + N k
Next, we write down the linear term with respect to µk :
µT
k (β0 Λ k m0 ) +
N
X
n=1
T
r nk · µT
k (Λ k x n ) = µ k (β0 Λ k m0 +
= µT
k Λ k (β0 m0 +
N
X
r nk Λk xn )
n=1
N
X
r nk xn )
n=1
= µT
k Λ k (β0 m0 + N k x̄ k )
Where we have defined:
x̄k = P N
1
N
X
n=1 r nk n=1
r nk xn =
N
1 X
r nk xn
Nk n=1
Comparing to the standard form, we can obtain:
mk =
1
βk
(β0 m0 + Nk x̄k )
Now we have obtained q? (µk |Λk ) = N (µk |mk , (βk Λk )−1 ), using the relation:
ln q? (Λk ) = ln q? (µk , Λk ) − ln q? (µk |Λk )
208
And focusing only on the terms dependent on Λk , we can obtain:
o
n1
1
ln |β0 Λk | − (µk − m0 )T β0 Λk (µk − m0 )
ln q? (Λk ) ∝
2
2
nv −D −1
o
1
0
1
+
ln |Λk | − Tr(W−
Λ
)
k
0
2
2
n1
o
N
X
1
r nk ln |Λk | − (xn − µk )T Λk (xn − µk )
+
2
2
n=1
n1
o
1
− ln |βk Λk | − (µk − mk )T βk Λk (µk − mk )
2
2
i
n1
1 h
∝
ln |Λk | − Tr β0 (µk − m0 )(µk − m0 )T · Λk
2
2
nv −D −1
o
1
0
1
+
ln |Λk | − Tr(W−
Λ
)
k
0
2
2
nN
h
i
N
X
1
k
+
ln |Λk | − Tr
r nk (xn − µk )T (xn − µk ) · Λk
2
2
n=1
i
n1
1 h
− ln |Λk | − Tr βk (µk − mk )T (µk − mk ) · Λk
2
2
v0 − D − 1 + N k
1
=
ln |Λk | − Tr[T · Λk ]
2
2
Where we have defined:
1
T = β0 (µk −m0 )(µk −m0 )T +W−
0 +
N
X
r nk (xn −µk )T (xn −µk )−βk (µk −mk )T (µk −mk )
n=1
By matching the coefficient ahead of ln |Λk |, we can obtain:
v k = v0 + N k
Next, by matching the coefficient in the Trace, we see that:
1
W−
k =T
Let’s further simplify T, beginning by introducing a useful equation, which
will be used here and later in Prob.10.16:
N
X
r nk xn xT
n
=
n=1
N
X
r nk (xn − x̄k + x̄k )(xn − x̄k + x̄k )T
n=1
=
N
X
n=1
=
N
X
n=1
=
=
=
£
¤
T
r nk (xn − x̄k )(xn − x̄k )T + x̄k x̄T
k + 2(x n − x̄ k )x̄ k
N
N
£
¤ X
£
¤ X
£
¤
r nk (xn − x̄k )(xn − x̄k )T +
r nk x̄k x̄T
r nk 2(xn − x̄k )x̄T
k +
k
Nk Sk + Nk x̄k x̄T
k +2
n=1
N
X
£
¤
r nk (xn − x̄k )x̄T
k
n=1
£
¤
T
Nk Sk + Nk x̄k x̄k + 2 (Nk x̄k − Nk x̄k )x̄T
k
Nk Sk + Nk x̄k x̄T
k
n=1
209
Where in the last step we have used Eq (10.51). Now we are ready to
prove that T is exactly given by Eq (10.62). Let’s first consider the coefficients
ahead of the quadratic term with repsect to µk :
(quad) = β0 µk µT
k +
N
X
n=1
T
r nk µk µT
k − β k µ k µ k = (β0 +
N
X
n=1
r nk − βk )µk µT
k =0
Where the summation is actually equal to Nk and we have also used Eq
(10.60). Next we focus on the linear term:
(linear) = −2β0 m0 µT
k +
= 2(−β0 m0 +
N
X
n=1
N
X
n=1
T
2r nk xn µT
k + 2β k m k µ k
r nk xn + βk mk )µT
k =0
Finally we deal with the constant term:
1
T
(const) = W−
0 + β0 m0 m0 +
=
N
X
T
r nk xn xT
n − βk mk mk
n=1
−1
T
T
W0 + β0 m0 m0 + Nk Sk + Nk x̄k x̄T
k − βk mk mk
1 2
β m k mT
k
βk k
1
1
T
T
(β0 m0 + NK x̄k )(β0 m0 + NK x̄k )T
= W−
0 + N k S k + β0 m0 m0 + N k x̄ k x̄ k −
βk
1
T
T
= W−
0 + N k S k + β0 m0 m0 + N k x̄ k x̄ k −
1
= W−
0 + N k S k + (β0 −
=
=
β20
)m0 m0T + (Nk −
Nk2
)x̄k x̄T
k −
1
2(β0 m0 ) · (NK x̄k )T
βk
βk
βk
β
N
β
N
β0 NK
0 k
0 k
1
W−
m0 m0T +
2(m0 ) · (x̄k )T
x̄k x̄T
0 + Nk Sk +
k −
βk
βk
βk
β0 N k
1
W−
(m0 − x̄k )(m0 − x̄k )T
0 + Nk Sk +
βk
Just as required.
Problem 10.14 Solution
Let’s begin by definition.
Z Z
(xn − µk )T Λk (xn − µk ))q? (µk , Λk ) d µk d Λk
Z nZ
o
=
(xn − µk )T Λk (xn − µk ))q? (µk |Λk ) d µk q? (Λk ) d Λk
Z
=
Eµk [(µk − xn )T Λk (µk − xn )] · q? (Λk ) d Λk
Eµk ,Λk [(xn − µk )T Λk (xn − µk )] =
The inner expectation is with respect to µk , which satisfies a Gaussian
distribution. We use Eq (380) in ’MatrixCookbook’: if x ∼ N (m, Σ), we have:
0
0
0
0
E[(x − m )T A(x − m )] = (m − m )T A(m − m ) + Tr(AΣ)
210
Therefore, here we can obtain:
i
h
Eµk [(µk − xn )T Λk (µk − xn )] = (mk − xn )T Λk (mk − xn ) + Tr Λk · (βk Λk )−1
Substituting it back into the integration, we can obtain:
Z h
i
T
1
Eµk ,Λk [(xn − µk ) Λk (xn − µk )] =
(mk − xn )T Λk (mk − xn ) + D β−
· q? (Λk ) d Λk
k
h
1
= D β−
+
E
(mk − xn )T Λk (mk − xn )]
Λ
k
k
n
o
1
T
= D β−
+
E
Tr[
Λ
·
(m
−
x
)(m
−
x
)
]
Λk
n
n
k
k
k
k
n
o
1
T
= D β−
+
Tr
E
[
Λ
]
·
(m
−
x
)(m
−
x
)
Λ
n
n
k
k
k
k
k
n
o
−1
= D βk + Tr vk Wk · (mk − xn )(mk − xn )T
=
1
T
D β−
k + v k (m k − x n ) W k (m k − x n )
Just as required.
Problem 10.15 Solution
There is a typo in Eq (10.69). The numerator should be α0 + Nk . Let’s
substitute Eq (10.58) into (B.17):
α0 + N k
α0 + N k
αk
E[ π k ] = P
=
=
P
K α0 + k Nk
K α0 + N
k αk
Problem 10.16 Solution
According to Eq (10.38), we can obtain:
E[ln p(X|Z, µ, Λ)] =
N X
K
X
n=1 k=1
=
N X
K
X
n=1 k=1
=
=
1
E[z nk ln N (xn |µk , Λ−
k )]
E[z nk ] · E[−
1
1
D
ln 2π + ln |Λk | − (xn − µk )T Λk (xn − µk )]
2
2
2
n
o
N X
K
1 X
E[z nk ] · − D ln 2π + E[ln |Λk |] − E[(xn − µk )T Λk (xn − µk )]
2 n=1 k=1
n
o
N X
K
1 X
e k − D β−1 − vk (xn − mk )T Wk (xn − mk )
r nk · − D ln 2π + ln Λ
k
2 n=1 k=1
Where we have used Eq (10.50), Eq (10.64) and Eq (10.65). Then we first
deal with the first three terms inside the bracket, i.e.,
n
o
n
o
N X
K
K X
N
1 X
1X
e k − D β−1
e k − D β−1
r nk · − D ln 2π + ln Λ
=
r
·
−
D
ln
2
π
+
ln
Λ
nk
k
k
2 n=1 k=1
2 k=1 n=1
i £
K £X
N
¤
1X
e k − D β−1
r nk · − D ln 2π + ln Λ
=
k
2 k=1 n=1
=
K
£
¤
1X
e k − D β−1
Nk · − D ln 2π + ln Λ
k
2 k=1
211
Where we have used the definition of Nk . Next we deal with the last term
inside the bracket, i.e.,
n
o
K
K
N X
N X
1 X
1 X
r nk · − vk (xn − mk )T Wk (xn − mk )
= −
Tr[r nk vk · (xn − mk )(xn − mk )T · Wk ]
2 n=1 k=1
2 n=1 k=1
= −
N
K
X
1X
r nk vk · (xn − mk )(xn − mk )T · Wk ]
Tr[
2 k=1
n=1
Since we have:
N
X
r nk vk · (xn − mk )(xn − mk )T
=
vk
n=1
N
X
r nk · (x̄k − mk + xn − x̄k )(x̄k − mk + xn − x̄k )T
n=1
=
vk
N
X
r nk · (x̄k − mk )(x̄k − mk )T
n=1
+vk
N
X
r nk · (xn − x̄k )(xn − x̄k )T
n=1
+vk
N
X
r nk · 2(x̄k − mk )(xn − x̄k )T
n=1
=
vk Nk · (x̄k − mk )(x̄k − mk )T
+vk N k Sk
+vk · 2(x̄k − mk )(
N
X
r nk xn −
n=1
=
N
X
r nk x̄k )T
n=1
vk Nk · (x̄k − mk )(x̄k − mk )T + vk Nk Sk
+vk · 2(x̄k − mk )(N k x̄k − N k x̄k )T
=
vk Nk · (x̄k − mk )(x̄k − mk )T + vk Nk Sk
Therefore, the last term can be reduced to:
n
o
N X
K
K
1 X
1X
r nk · − vk (xn − mk )T Wk (xn − mk )
= −
Tr[vk Nk · (x̄k − mk )(x̄k − mk )T Wk ]
2 n=1 k=1
2 k=1
−
= −
−
K
1X
Tr[vk Nk Sk Wk ]
2 k=1
K
1X
Nk vk · (x̄k − mk )Wk (x̄k − mk )T
2 k=1
K
1X
Nk vk Tr[Sk Wk ]
2 k=1
If we combine the first three and the last term, we just obtain Eq (10.71).
Next we prove Eq (10.72). According to Eq (10.37), we have:
E[ln p(Z|π)] =
N X
K
X
n=1 k=1
E[z nk ln πk ] =
N X
K
X
n=1 k=1
r nk ln π
ek
212
Just as required.
Problem 10.17 Solution
According to Eq (10.39), we have:
E[ln p(π)] = ln C(α0 ) + (α0 − 1)
K
X
E[ln πk ]
k=1
= ln C(α0 ) + (α0 − 1)
K
X
k=1
ln π
ek
According to Eq (10.40), we have:
E[ln p(µ, Λ)] =
K
X
E[ln N (µk |m0 , (β0 Λk )−1 )] +
k=1
K
X
E[ln W (Λk |W0 , v0 )]
k=1
o
K n D
X
1
1
E − ln 2π + ln |β0 Λk | − (µk − m0 )T (β0 Λk )(µk − m0 )
2
2
2
k=1
o
K n
X
1
v0 − D − 1
1
ln |Λk | − Tr[W−
+
E ln B(W0 , v0 ) +
0 Λk ]
2
2
k=1
o
K n D
X
D
1
1
=
E − ln 2π + ln β0 + ln |Λk | − (µk − m0 )T (β0 Λk )(µk − m0 )
2
2
2
2
k=1
n
o
K
X
v0 − D − 1
1
1
+
E ln B(W0 , v0 ) +
ln |Λk | − Tr[W−
Λ
]
k
0
2
2
k=1
o
K
K n
K · D β0 1 X
1X
fk −
=
ln
+
ln Λ
E (µk − m0 )T (β0 Λk )(µk − m0 )
2
2π 2 k=1
2 k=1
o
K
K n
v0 − D − 1 X
1X
1
fk −
K · ln B(W0 , v0 ) +
ln Λ
E Tr[W−
Λ
]
k
0
2
2 k=1
k=1
=
So now we need to calculate these two expectations. Using (B.80), we can
obtain:
n
o X
n
o
K
K
K n
X
X
1
−1
−1
Λ
]
=
Tr
W
·
E
[
Λ
]
=
v
·
Tr
W
W
E Tr[W−
k
k
k
k
0
0
0
k=1
k=1
k=1
To calculate the other expectation, first we write down two properties of
the Gaussian distribution, i.e.,
E[ µ k ] = m k ,
T
−1 −1
E[µk µT
k ] = mk mk + βk Λk
213
Therefore, we can obtain:
o
o
K n
K n
X
X
E Tr[Λk · (µk − m0 )(µk − m0 )T ]
E (µk − m0 )T (β0 Λk )(µk − m0 )
= β0
k=1
k=1
= β0
K
X
n £
¤o
T
T
Eµk ,Λk Tr Λk · (µk µT
−
2
µ
m
+
m
m
)
0
k 0
0
k
k=1
= β0
K
X
n £
¤o
−1 −1
T
T
EΛk Tr Λk · (mk mT
+
β
Λ
−
2m
m
+
m
m
)
0 0
k 0
k
k
k
k=1
= β0
K
X
n £
¤o
1
T
T
T
EΛk Tr β−
k I + Λ k · (m k m k − 2m k m0 + m0 m0 )
k=1
= β0
K
X
n
£
¤o
1
T
EΛk D · β−
+
Tr
Λ
·
(m
−
m
)(m
−
m
)
0
0
k
k
k
k
k=1
=
n
o
K
X
K D β0
EΛk (mk − m0 )Λk (mk − m0 )T
+ β0
βk
k=1
=
K
X
K D β0
+ β0
(mk − m0 ) · EΛk [Λk ] · (mk − m0 )T
βk
k=1
=
K
X
K D β0
+ β0
(mk − m0 ) · (vk Wk ) · (mk − m0 )T
βk
k=1
Substituting these two expectations back, we obtain Eq (10.74) just as
required. According to Eq (10.48), we have:
E[ln q(Z)] =
N,K
X
E[z nk ] · ln r nk =
n,k=1
N,K
X
r nk · ln r nk
n,k=1
According to Eq (10.57), we have:
E[ln q(π)] = ln C(α) + (αk − 1)
K
X
E[ln πk ]
k=1
= ln C(α0 ) + (αk − 1)
K
X
k=1
ln π
ek
214
To derive Eq(10.77), we follow the same procedure as that for Eq (10.74):
E[ln q(µ, Λ)] =
K
X
E[ln N (µk |mk , (βk Λk )−1 )] +
K
X
E[ln W (Λk |Wk , vk )]
k=1
k=1
o
K n D
X
D
1
1
E − ln 2π + ln βk + ln |Λk | − (µk − mk )T (βk Λk )(µk − mk )
2
2
2
2
k=1
n
o
K
X
vk − D − 1
1
1
E ln B(Wk , vk ) +
+
ln |Λk | − Tr[W−
Λ
]
k
k
2
2
k=1
o
K
K n
K · D βk 1 X
1X
fk −
=
ln
+
ln Λ
E (µk − mk )T (βk Λk )(µk − mk )
2
2π 2 k=1
2 k=1
o
K
K n
vk − D − 1 X
1X
1
fk −
K · ln B(Wk , vk ) +
ln Λ
E Tr[W−
Λ
]
k
k
2
2 k=1
k=1
=
=
K
K · D βk 1 X
KD
fk −
ln Λ
ln
+
2
2π 2 k=1
2
K · ln B(Wk , vk ) +
=
n
o
K
K
1X
vk − D − 1 X
1
fk −
ln Λ
vk E Tr[W−
W
]
k
k
2
2 k=1
k=1
K
K · D βk 1 X
KD
fk −
ln Λ
ln
+
2
2π 2 k=1
2
K · ln B(Wk , vk ) +
K
K
1X
vk − D − 1 X
fk −
ln Λ
vk · D
2
2 k=1
k=1
It is identical to Eq (10.77).
Problem 10.18 Solution
This problem is very complicated. Let’s explain it in details. In section
10.2.1, we have obtained the update formula for all the coefficients using the
general framework of variational inference. For more details you can see
Prob.10.12 and Prob.10.13.
Moreover, in the previous problem, we have shown that L is given by
Eq (10.70)-Eq (10.77), if we have assumed the form of q, i.e., Eq (10.42), Eq
(10.48),Eq (10.55), Eq (10.57) and Eq (10.59). Note that here we do not know
the specific value of those coefficients, e.g., Eq (10.60)-Eq (10.63). In this problem, we will show that by maximizing L with respect to those coefficients,
we will obtain those formula just as in section 10.2.1.
To summarize, here we write down all the coefficients required to estimate: {βk , mk , vk , Wk , αk , r nk }. We begin by considering βk . Note that only Eq
(10.71), (10.74) and (10.77) contain βk , we calculate the derivative of L with
215
respect to βk and set it to zero:
∂L
∂βk
1
D 1 2π
1
2
−2
= ( N k D β−
)
k ) + ( D β0 β k ) − (
2
2
2 2π β k
1 −2
β · (N k D + D β0 − D βk ) = 0
=
2 k
The three brackets in the first line correspond to the derivative with respect to Eq (10.71), (10.74) and (10.77). Rearranging it, we obtain Eq (10.60).
Next we consider mk , which only occurs in the quadratic terms in Eq (10.71)
and (10.74).
∂L
∂mk
=
=
¤ £ 1
¤
Nk vk · 2Wk (x¯k − mk ) + − β0 vk · 2Wk (mk − m0 )
2
2
£
¤
vk Wk Nk · (x¯k − mk ) − β0 (mk − m0 ) = 0
£1
Similarly, the two brackets in the first line correspond to the derivative
with respect to Eq (10.71) and (10.74). Rearranging it, we obtain Eq (10.61).
Next noticing that vk and Wk are always coupled in L , e.g.,vk occurs ahead of
quadratic terms in Eq (10.71). We will deal with vk and Wk simultaneously.
Let’s first make this more clear by writing down those terms depend on vk
and Wk in L :
o
K n1
X
e k − H[q(Λk )]
(10.77) ∝
ln Λ
(∗)
k=1 2
(10.71) ∝
(10.74) ∝
=
n
o
K
1X
e k − vk · Tr[(Sk + Ak )Wk ]
Nk ln Λ
2 k=1
o v −D −1 X
K n
K
K
1X
1X
1
e k − β0 vk · Tr[Bk Wk ] + 0
ek −
ln Λ
ln Λ
vk Tr[W−
0 Wk ]
2 k=1
2
2
k=1
k=1
K
K
v0 − D X
1X
1
ek −
ln Λ
vk Tr[(β0 Bk + W−
0 )W k ]
2 k=1
2 k=1
e k is given by Eq (10.65) and Ak and Bk are given by:
Where ln Λ
Ak = (x¯k − mk )(x¯k − mk )T ,
Bk = (mk − m0 )(mk − m0 )T
Moreover, H[q(Λk )] is given by (B.82):
H[q(Λk )] = − ln B(Wk , vk ) −
vk − D − 1
v D
ek + k
ln Λ
2
2
Where ln B(Wk , vk ) can be calculated based on (B.79). Note here we only
focus on those terms dependent on vk and Wk :
ln B(Wk , vk ) ∝ −
D
X
vk
vk D
vk + 1 − i
ln |Wk | −
ln 2 −
Γ(
)
2
2
2
i =1
216
To further simplify the derivative, we now write down those terms in L
which only depends on vk and Wk with a given specific index k:
n1
o 1
n
o
e k − H[q(Λk )] + N k ln Λ
e k − vk · Tr[(Sk + Ak )Wk ]
L ∝ − ln Λ
2
2
v0 − D
1
e k − vk Tr[(β0 Bk + W−1 )Wk ]
+
ln Λ
0
2
2
1
1
e k + H[q(Λk )] − vk · Tr[(N k Sk + N k Ak + β0 Bk + W−1 )Wk ]
=
(−1 + Nk + v0 − D) ln Λ
0
2
2
1
1
e k − vk · Tr[(N k Sk + N k Ak + β0 Bk + W−1 )Wk ]
(−1 + Nk + v0 − D) ln Λ
=
0
2
2
vk − D − 1
v D
ek + k
− ln B(Wk , vk ) −
ln Λ
2
2
1
v D
1
e k − vk · Tr[Fk Wk ] + k − ln B(Wk , vk )
(Nk + v0 − vk ) ln Λ
=
2
2
2
Where we have defined:
Fk
=
1
Nk Sk + Nk Ak + β0 Bk + W−
0
Note that Eq (10.77) has a minus sign in L , the negative of (∗) has been
used in the first line. We first calculate the derivative of L with respect to vk
and set it to zero:
ek 1
e k ln Λ
∂L
1
d ln Λ
D
=
−
(Nk + v0 − vk )
− Tr[Fk Wk ] +
∂vk
2
dvk
2
2
2
D
|Wk | D ln 2 1 X 0 vk + 1 − i
Γ(
+
+
+
)
2
2
2 i=1
2
i
ek
1h
d ln Λ
=
(Nk + v0 − vk )
− Tr[Fk Wk ] + D = 0
2
dvk
e k , i.e., Eq
Where in the last step, we have used the definition of ln Λ
(10.65). Then we calculate the derivative of L with respect to Wk and set
it to zero:
∂L
∂Wk
=
=
1
vk
vk
1
1
(Nk + v0 − vk )W−
F k + W−
k −
2
2
2 k
vk
1
1
1
(Nk + v0 − vk )W−
(Fk − W−
k −
k )=0
2
2
Staring at these two derivatives long enough, we find that if the following
two conditions:
1
Nk + v0 − vk = 0 , and Fk = W−
k
are satisfied, the derivatives of L with respect to vk and Wk will all be
zero. Rearranging the first condition, we obtain Eq (10.63). Next we prove
that the second condition is exactly Eq (10.62), by simplifying Fk .
Fk
=
1
Nk Sk + Nk Ak + β0 Bk + W−
0
1
T
T
= W−
0 + N k S k + N k · (x¯k − m k )(x¯k − m k ) + β0 · (m k − m0 )(m k − m0 )
217
Comparing this with Eq (10.62), we only need to prove:
Nk ·(x¯k −mk )(x¯k −mk )T +β0 ·(mk −m0 )(mk −m0 )T =
β0 N k
β0 + N k
(x¯k −m0 )(x¯k −m0 )T
Let’s start from the left hand side.
T
T
T
T
Nk x¯k x¯k T − 2Nk x¯k mT
k + N k m k m k + β0 m k m k − 2β0 m k m0 + β0 m0 m0
β0 m0 + N k x¯k β0 m0 + N k x¯k T
β0 m0 + N k x¯k T
) + (Nk + β0 )(
)(
)
= N k x¯k x¯k T − 2N k x¯k (
β0 + N k
β0 + N k
β0 + N k
β0 m0 + N k x¯k
−2β0 (
)m0T + β0 m0 m0T
β0 + N k
(left) =
Then we complete the square with respect to x¯k , and we will see the coefficients match with the right hand side. Here as an example, we calculate
the coefficients ahead of the quadratic term x¯k x¯k T :
(quad) =
=
=
Nk − 2Nk
Nk
Nk
+ (β0 + N k )(
)2
β0 + N k
β0 + N k
Nk (β0 + Nk ) − 2Nk2 + Nk2
β0 + N k
β0 N k
β0 + N k
It is similar for the linear and the constant term, and here due to page
limit, we omit the proof. the update formula for αk , r nk are still remaining to
obtain. Noticing that only Eq (10.72), (10.73) and (10.76) depend on αk , we
now calculate the derivative of L with respect to αk :
∂L
∂αk
=
N
X
n=1
r nk
d ln π
ek
d ln π
ek £
d ln π
ek
d ln C(α) ¤
+ (α0 − 1)
− (αk − 1)
+ ln π
ek +
d αk
d αk
d αk
d αk
d ln π
ek
d ln C(α)
− ln π
ek −
d αk
d αk
£ 0
¤
£
¤ d[ ln Γ(α
0
b) − ln Γ(αk ) ]
= (N k + α0 − αk ) φ (αk ) − φ (α
b) − φ(αk ) − φ(α
b) −
d αk
£ 0
¤ £
¤
0
= (N k + α0 − αk ) φ (αk ) − φ (α
b) − φ(αk ) − φ(α
b) − [ φ(α
b) − φ(αk ) ]
£ 0
¤
0
= (N k + α0 − αk ) φ (αk ) − φ (α
b) = 0
= (N k + α0 − αk )
Where we have used (B.25), Eq (10.66). Therefore, we obtain Eq (10.58).
Finally, we are required to derive an update formula for r nk . Note that x¯k ,
Sk and Nk also contains r nk , we conclude that Eq (10.71), (10.72) and (10.75)
depend on r nk . Using the definition of Nk , i.e., Eq (10.51), we can obtain:
n
o 1X
1X
e k − D β−1 −
L ∝
r nk ln Λ
Nk vk Tr[(Sk + Ak )Wk ]
k
2 k,n
2 k
1X
1X
+
r nk ln π
ek −
r nk ln r nk
2 k,n
2 k,n
218
P
Note that constraint exists for r nk : k r nk = 1, we cannot calculate the
derivative and set it to zero. We must introduce a Lagrange Multiplier. Before
doing so, let’s simplify Sk + Ak :
Sk + Ak
=
=
=
N
1 X
r nk (xn − x¯k )(xn − x¯k )T + (x¯k − mk )(x¯k − mk )T
Nk n=1
i
N h
1 X
T
T
+ x¯k x¯k T − 2x¯k mk + mk mT
r nk xn xT
n − 2r nk x n x¯k + r nk x¯k x¯k
k
Nk n=1
PN
PN
N
¯ T
r nk x¯k x¯k T
1 X
n=1 2r nk x n x k
T
r nk xn xn −
+ n=1
+ x¯k x¯k T − 2x¯k mk + mk mT
k
Nk n=1
Nk
Nk
=
N
1 X
2Nk x¯k x¯k T Nk x¯k x¯k T
r nk xn xT
+
+ x¯k x¯k T − 2x¯k mk + mk mT
−
n
k
Nk n=1
Nk
Nk
=
N
1 X
T
r nk xn xT
n − 2x¯k m k + m k m k
Nk n=1
=
=
=
N
1 X
T
r nk xn xT
(
n − 2N k x¯k m k + N k m k m k )
Nk n=1
i
N
1 hX
T
¯
r nk (xn xT
−
2
x
m
+
m
m
)
k
k
k
n
k
Nk n=1
N
1 X
r nk (xn − mk )(xn − mk )T
Nk n=1
Therefore, we obtain:
L
∝
=
n
o X
X
1X
e k − D β−1 + r nk ln π
r nk ln Λ
e k − r nk ln r nk
k
2 k,n
k,n
k,n
1X
−
Nk vk Tr[(Sk + Ak )Wk ]
2 k
n
o X
X
1X
e k − D β−1 + r nk ln π
r nk ln Λ
e k − r nk ln r nk
k
2 k,n
k,n
k,n
−
K X
N
1X
vk r nk (xn − mk )T Wk (xn − mk )
2 k=1 n=1
Introducing Lagrange Multiplier λn , we obtain:
(Lagrange) =
n
o X
X
1X
e k − D β−1 + r nk ln π
r nk ln Λ
e k − r nk ln r nk
k
2 k,n
k,n
k,n
−
K X
N
N
X
X
1X
vk r nk (xn − mk )T Wk (xn − mk ) +
λn (1 − r nk )
2 k=1 n=1
n=1
k
Calculating the derivative with respect to λn and setting it to zero, we can
219
obtain:
∂(Lagrange)
∂ r nk
=
1
e k − D β−1 } + ln π
{ln Λ
e k − [ln r nk + 1]
k
2
1
− vk (xn − mk )T Wk (xn − mk ) + λn = 0
2
Moving ln r nk to the right side and then exponentiating both sides, we
obtain Eq (10.67), and the normalized r nk is given by Eq (10.49), (10.46), and
(10.64)-(10.66).
Problem 10.19 Solution
Let’s start from the definition, i.e., Eq (10.78).
XZ Z Z
b |X) =
b |b
p(x
p(x
z, µ, Λ)p(b
z|π)p(π, µ, Λ|X) d π d µ d Λ
b
z
=
K
XZ Z Z Y
k=1
b
z
1 b
zk
b |µk , Λ−
N (x
k ) ·
K
Y
k=1
b
z
πkk · p(π, µ, Λ|X) d π d µ d Λ
K £
XZ Z Z Y
¤ bzk
1
b |µk , Λ−
≈
N (x
· q(π, µ, Λ) d π d µ d Λ
k ) · πk
k=1
b
z
=
K Z Z Z £
X
¤
1
b |µk , Λ−
N (x
k ) · π k · q(π, µ, Λ) d π d µ d Λ
k=1
=
K Z Z Z
X
k=1
K
Y
£
¤
1
b |µk , Λ−
N (x
q(µ j , Λ j ) d π d µ d Λ
k ) · π k · q(π) ·
j =1
Where we have used the fact that z uses a one-of-k coding scheme. Recall
that µ = {µk } and Λ = {Λk }, the term inside the summation can be further
simplified. Namely, for those index j 6= k, the integration with respect to µ j
and Λ j will equal 1, i.e.,
b |X) =
p(x
=
K Z Z Z
X
k=1
K Z Z Z
X
k=1
=
K Z Z Z
X
k=1
K
Y
£
¤
1
b |µk , Λ−
N (x
)
·
π
·
q(
π
)
·
q(µ j , Λ j ) d π d µ d Λ
k
k
j =1
1
b |µk , Λ−
N (x
k ) · π k · q(π) · q(µ k , Λ k ) d π d µ k d Λ k
1
−1
b |µk , Λ−
N (x
k ) · π k · Dir(π|α) · N (µ k |m k , (β k Λ k ) )W (Λ k |W k , v k ) d π d µ k d Λ k
We notice that in the expression above, only πk · Dir(π|α) contains πk ,
and we know that the expectation of πk with respect to Dir(π|α) is αk /α
bk .
220
Therefore, we can obtain:
b |X) =
p(x
=
=
=
K Z Z α
X
k
k=1
K nZ £
X
α
b
Z
1
−1
b |µk , Λ−
N (x
k ) · N (µ k |m k , (β k Λ k ) ) · W (Λ k |W k , v k ) d µ k d Λ k
o
¤ αk
1
−1
b |µk , Λ−
N (x
)
·
N
(
µ
|
m
,
(
β
Λ
)
)
d
µ
·
W
(
Λ
|
W
,
v
)
d
Λ
·
k
k
k
k
k
k
k
k
k
k
α
b
k=1
Z
n
o
K
X
1
−1 α k
b |mk , (1 + β−
N (x
)
Λ
)
·
·
W
(
Λ
|
W
,
v
)
d
Λ
k
k k
k
k
k
α
b
k=1
K α Z
X
k
1
−1
b |mk , (1 + β−
N (x
k )Λ k ) · W (Λ k |W k , v k ) d Λ k
α
b
k=1
Notice that the Wishart distribution is a conjugate prior for the Gaussian
distribution with known mean and unknown precision. We conclude that the
1
1
b |mk , (1 + β−
product of N (x
)Λ−
) · W (Λk |Wk , vk ) is again a Wishart distribuk
k
tion without normalized, which can be verified by focusing on the dependency
on Λk :
n Tr[Λ · (x
o
b − m k )T ] 1
k b − m k )(x
−1
(product) ∝ |Λk |1/2+(vk −D −1)/2 · exp −
−
Tr[
Λ
W
]
k
k
1)
2
2(1 + β−
k
0
0
∝ W (Λ k | W , v )
Where we have defined:
0
v = vk + 1
and
0
[W ]−1 =
b − mk )(x
b − m k )T
(x
1
1 + β−
k
1
+ W−
k
Using the normalization constant of Wishart distribution, i.e., (B.79), we
can obtain:
b |X)
p(x
=
=
∝
∝
K α Z
X
k
b
k=1 α
K α
X
k
1
−1
b |mk , (1 + β−
N (x
k )Λ k ) · W (Λ k |W k , v k ) d Λ k
0
0
· B(W , v )
b
k=1 α
¯ (x
¯−(vk +1)/2
b − m k )T
¯ b − mk )(x
−1 ¯
+
W
¯
¯
k
1
1 + β−
k
¯ 1
¯−(vk +1)/2
¯
¯
T
b
b
W
(
x
−
m
)(
x
−
m
)
+
I
¯
¯
k
k
k
1
1 + β−
k
b k . Next, we use:
Here we have only considered those terms dependent on x
|I + abT | = 1 + aT b
221
The expression above can be further simplified to:
=
1
¯−(vk +1)/2
¯
T
b
b
W
(
x
−
m
)(
x
−
m
)
+
I
¯
k
k
k
−1
1 + βk
h
i−(vk +1)/2
1
T
b
b
1+
(
x
−
m
)
W
(
x
−
m
)
k
k
k
1
1 + β−
k
¯
¯
b |X) ∝ ¯
p(x
By comparing it with (B.68), we notice that it is a Student’s t distribution,
whose parameters are defined by Eq (10.81)-(10.82).
Problem 10.20 Solution
Let’s begin by dealing with q? (Λk ). When N → +∞, we know that Nk
also approaches +∞ based on Eq (10.51). Therefore, we know that [Wk ]−1 →
1
Nk Sk and vk → Nk . Using (B.80), we conclude that E[Λk ] = vk Wk → S−
.
k
If we now can prove that the entropy H[Λk ] is zero, we can conclude that
the distribution collapse to a Dirac function, i.e, the distribution is sharply
1
peaked around S−
, which is identical to the EM of Gaussian mixture given
k
by Eq (9.25). Therefore, let’s now start from ln B(Wk , vk ), i.e., (B.79).
ln B(Wk , vk )
D
X
vk
vk + 1 − i
vk D
D(D − 1)
ln Γ(
ln |Wk | −
ln 2 −
ln π −
)
2
2
4
2
i =1
=
−
→
D
X
Nk
Nk D
Nk + 1 − i
ln | Nk Sk | −
ln 2 −
ln Γ(
)
2
2
2
i =1
=
D
X
Nk
Nk − 1 − i
(D ln Nk + ln |Sk | − D ln 2) −
ln Γ(
+ 1)
2
2
i =1
≈
≈
=
=
Nk
Nk
(D ln
+ ln |Sk |)
2
2
D £1
X
Nk − 1 − i
Nk − 1 − i 1
Nk − 1 − i ¤
−
ln 2π −
+(
+ ) ln
2
2
2
2
i =1 2
D £ N
X
Nk
Nk
Nk Nk ¤
k
(D ln
+ ln |Sk |) −
−
+
ln
2
2
2
2
2
i =1
Nk
Nk
Nk D Nk D Nk
(D ln
+ ln |Sk |) +
−
ln
2
2
2
2
2
Nk
(D + ln |Sk |)
2
Where we have used Eq (1.146) to approximate the logarithm of Gamma
222
function. Next we deal with E[ln Λk ] based on (B.81):
E[ln Λk ]
D
X
=
φ(
i =1
D
X
→
ln(
i =1
D
X
≈
vk + 1 − i
) + D ln 2 + ln |Wk |
2
ln
i =1
Nk + 1 − i
) + D ln 2 − ln | Nk Sk |
2
Nk
+ D ln 2 − D ln N k − ln |Sk |
2
Nk
+ D ln 2 − D ln N k − ln |Sk |
2
− ln |Sk |
D ln
=
=
Where we have used Eq (10.241) to approximate the φ( vk +21− i ). Now we
are ready to deal with the entropy H[q(Λk )]:
H[q(Λk )]
vk − D − 1
vk D
E[ln Λk ] +
2
2
Nk
Nk
Nk D
→ −
(D + ln |Sk |) +
ln |Sk | +
=0
2
2
2
=
− ln B(Wk , vk ) −
Therefore, we can conclude that the distribution q? (Λk ) collapse to a
1
Dirac function at S−
. In other words, when N → +∞, Λk can only achieve
k
−1
one value Sk .
Next, we deal with q? (µk |Λk ). According to Eq (10.60), when N → +∞,
we conclude that βk → Nk , and thus, mk → x¯k based on Eq (10.61). Since
1
we know q? (µk |Λk ) = N (µk | mk , (βk Λk )−1 ) and βk Λk → Nk S−
is large, we
k
conclude that when N → ∞, µk also achieves only one value x¯k , which is
identical to the EM of Gaussian Mixture, i.e., Eq (9.24).
Finally, we consider q? (π) given by Eq (10.54). Since we know αk → Nk
N
based on Eq (10.58), we see that E[µk ] = αk /α
b → Nk and
var[µk ] =
αk (α
b − αk )
α
b 2 (α
b + 1)
≤
α
b·α
b
α
b3
=
1
α
b
→0
We can also conclude that pi k only achieves one value NNk , which is identical to the EM of Gaussian Mixture, i.e., Eq (9.26). Now it is trivial to see
that the predictive distribution will reduce to a Mixture of Gaussian using Eq
(10.80). Beause π, µk and Λk all reduce to a Dirac function, the integration
is easy to perform.
Problem 10.21 Solution
This can be verified directly. The total number of labeling equals assign
K labels to K object. For the first label, we have K choice, K − 1 choice for the
second label, and so on. Therefore, the total number is given by K!.
Problem 10.22 Solution
223
Let’s explain this problem in details. Suppose that now we have a mixture of Gaussian p(Z|X), which are required to approximate. Moreover, it
has K components and each of the modes is denoted as {µ1 , µ2 , ..., µK }. We
use the variational inference, i.e., Eq (10.3), to minimize the KL divergence:
KL(q|| p), and obtain an approximate distribution q s (Z) and a corresponding
lower bound L(q s ).
According to the problem description, this approximate distribution q s (Z)
will be a single mode Gaussian located at one of the modes of p(Z|X), i.e.,
q s (Z) = N (Z|µs , Σs ), where s ∈ {1, 2, ..., K }. Now, we replicate this q s for K!
times in total. Each of the copies is moved to one mode’s center.
Now we can write down the mixing distribution made up of K! Gaussian
distribution:
K!
1 X
q m (Z) =
N (Z|µC (m) , Σs )
K! m=1
Where C(m) represents the mode of the m-th component. C(m) ∈ {1, 2, ..., K }.
What the problem wants us to prove is:
L(q s ) + ln K! ≈ L(q m )
In other words, the lower bound using q m to approximate, i.e., L(q m ), is
ln K! larger than using q s , i.e., L(q s ). Based on Eq (10.3), let’s equivalently
deal with the KL divergence. According to Eq (10.4), we can obtain:
Z
p(Z|X)
KL(q m || p) = − q m (Z) ln
dZ
q m (Z)
Z
Z
= − q m (Z) ln p(Z|X) dZ + q m (Z) ln q m (Z) dZ
Z
Z
q m (Z) ln p(Z|X) dZ +
= −
Z
Z
q m (Z) ln p(Z|X) dZ +
= −
q m (Z) ln
q m (Z) ln
n 1 X
K!
K! m=1
o
N (Z|µC (m) , Σs ) dZ
nX
K!
o
1
N (Z|µC (m) , Σs ) dZ + ln
K!
m=1
Z
= − ln K! −
+
1
K!
q m (Z) ln p(Z|X) dZ
Z X
K!
m=1
N (Z|µC (m) , Σs ) ln
nX
K!
o
N (Z|µC (m) , Σs ) dZ
m=1
In order to further simplify the KL divergence, here we write down two
useful equations. First, we use the "negligible overlap" property. To be more
specific, according to the assumption that the overlap are negligible, we can
obtain:
Z
N (Z|µC (m) , Σs ) ln
nX
K!
m=1
Z
o
n
o
N (Z|µC (m) , Σs ) dZ ≈ N (Z|µC (m) , Σs ) ln N (Z|µC (m) , Σs ) dZ
224
The second equation is that for any m 1 , m 2 ∈ {1, 2, ..., K }, we have:
Z
Z
n
o
q s ln q s dZ =
N (Z|µC (m1 ) , Σs ) ln N (Z|µC (m1 ) , Σs ) dZ
Z
n
o
=
N (Z|µC (m2 ) , Σs ) ln N (Z|µC (m2 ) , Σs ) dZ
Therefore, now we can obtain:
Z
KL(q m || p) = − ln K! − q m (Z) ln p(Z|X) dZ
+
1
K!
Z X
K!
N (Z|µC (m) , Σs ) ln
nX
K!
m=1
o
N (Z|µC (m) , Σs ) dZ
m=1
Z
≈ − ln K! −
1
+
K!
q m (Z) ln p(Z|X) dZ
Z X
K!
n
o
N (Z|µC (m) , Σs ) ln N (Z|µC (m) , Σs ) dZ
m=1
Z
= − ln K! − q m (Z) ln p(Z|X) dZ
Z
n
o
+ N (Z|µC (m) , Σs ) ln N (Z|µC (m) , Σs ) dZ (∀ m ∈ {1, 2, ..., K })
Z
Z
= − ln K! − q m (Z) ln p(Z|X) dZ + q s (Z) ln q s (Z) dZ
Z nX
Z
o
K!
1
= − ln K! −
N (Z|µC (m) , Σs ) ln p(Z|X) dZ + q s (Z) ln q s (Z) dZ
K!
m=1
Z
Z
≈ − ln K! − q s (Z) ln p(Z|X) dZ + q s (Z) ln q s (Z) dZ
Z
p(Z|X)
dZ = − ln K! + KL(q s || p)
= − ln K! − q s (Z) ln
q s (Z)
To obtain the desired result, we have adopted an approximation here,
however, you should notice that this approximation is rough.
Problem 10.23 Solution
Let’s go back to Eq (10.70). If now we treat πk as a parameter without
a prior distribution, πk will only occur in the second term in Eq (10.70), i.e.,
E[ln p(Z|π)]. Therefore, we can obtain:
L ∝ E[ln p(Z|π)] =
N X
K
X
r nk ln πk
n=1 k=1
Where we have used Eq (10.72), and here since πk is a point estimate,
the expectation E[ln πk ] will reduce to ln πk . Now we introduce a Lagrange
Multiplier.
N X
K
K
X
X
Lag =
r nk ln πk + λ · (
πk − 1)
n=1 k=1
k=1
225
Calculating the derivative of the expression above with respect to πk and
setting it to zero, we obtain:
PN
Nk
n=1 r nk
+λ =
+λ = 0
(∗)
πk
πk
Multiplying both sides by πk and then adopting summation of both sides
with respect to k, we obtain
K
X
Nk + λ
k=1
K
X
πk = 0
k=1
Since we know the summation of Nk with respect to k equals N, and the
summation of πk with respect to k equals 1, we rearrange the equation above,
yielding:
λ = −N
Substituting it back into (∗), we can obtain:
πk =
N
Nk
1 X
r nk
=
N
N n=1
Just as required.
Problem 10.24 Solution
Recall that the singularity in the maximum likelihood estimation of Gaussian mixture is caused by the determinant of the covariance matrix Σk approaches 0, and thus the value in N (xn |µk , Σk ) will approach +∞. For more
details, you can read Section 9.2, especially page 434.
In this problem, an intuition is that since we have introduce a prior distribution for Λk , this singularity won’t exist when adopting MAP. Let’s verify
this statement beginning by writing down the posterior.
p(Z|X, π, µ, Λ) ∝
p(X|Z, π, µ, Λ) · p(Z, π, µ, Λ)
=
p(X|Z, π, µ, Λ) · p(Z|π, µ, Λ) · p(π|µ, Λ) · p(µ, Λ)
=
p(X|Z, µ, Λ) · p(Z|π) · p(π) · p(µ, Λ)
Note that in the first step we have used Bayes’ theorem, that in the second
step we have used the fact that p(a, b) = p(a| b) · p(b), and that in the last step
we have omitted the extra dependence based on definition, i.e., Eq (10.37)(10.40). Now let’s calculate the MAP solution for Λk .
ln p(Z|X, π, µ, Λ) ∝
=
n
o
N
1 X
z nk ln |Λk | − (xn − µk )T Λk (xn − µk )
2 n=1
o
1n
ln |Λk | − β0 (µk − m0 )T Λk (µk − m0 )
2
o
1n
1
+ (v0 − D − 1) ln |Λk | − Tr[W−
0 Λ k ] + const
2
c · ln |Λk | − Tr[BΛk ] + const
226
Where const is the term independent of Λk , and we have defined:
c=
N
X
1
z nk )
(v0 − D +
2
n=1
and
B=
o
N
1n X
1
z nk (xn − µk )(xn − µk )T + β0 (µk − m0 )(µk − m0 )T + W−
0
2 n=1
Next we calculate the derivative of ln p(Z|X, π, µ, Λ) with respect to Λk
and set it to 0, yielding:
1
c · Λ−
k −B = 0
therefore, we obtain:
1
B
c
Note that in the MAP framework, we need to solve z nk first, and then
substitute them in c and B in the expression above. Nevertheless, from the
1
won’t have zero determinant.
expression above, we can see that Λ−
k
1
Λ−
k =
Problem 10.25 Solution
We qualitatively solve this problem. As the number of mixture components grows, so does the number of variables that may be correlated, but they
are treated as independent under a variational approximation if Eq (10.5) has
been used. Therefore, the proportion of probability mass under the true distribution, p(Z, π, µ, Σ|X), that the variational approximation q(Z, π, µ, Σ) does
not capture, will grow. The consequence will be that the second term in (10.2),
the KL divergence between q(Z, π, µ, Σ) and p(Z, π, µ, Σ|X) will increase.
To answer the question whether we will underestimate or overestimate
the number of components by minimizing KL(q|| p) divergence under factorization, we only need to see Fig.10.3. It is obvious that we will underestimate
the number of components.
Problem 10.26 Solution
In this problem, we also need to consider the prior p(β) = Gam(β| c 0 , d 0 ).
To be more specific, based on the original joint distribution p(t, w, α), i.e., Eq
(10.90), the joint distribution p(t, w, α, β) now should be written as:
p(t, w, α, β) = p(t|w, β)p(w|α)p(α)p(β)
Where the first term on the right hand side is given by Eq (10.87), the
second one is given by Eq (10.88), the third one is given by Eq (10.89), and
the last one is given by Gam(β| c 0 , d 0 ). Using the variational framework, we
assume a posterior variational distribution:
q(w, α, β) = q(w)q(α)q(β)
227
It is trivial to observe that introducing a Gamma prior for β doesn’t affect
q(α) because the expectation of p(β) can be absorbed into the ’const’ term in
Eq (10.92). In other words, we still obtain Eq (10.93)-Eq(10.95).
Now we deal with q(w). By analogy to Eq (10.96)-(10.98), we can obtain:
ln q? (w) ∝ Eβ [ln p(t|w, β)] + Eα [ln p(w|α)] + const
N
Eβ [β] X
Eα [α] T
(t n − wT φn )2 −
·
w w + const
2
2
n=1
n
o
1
− wT Eβ [β] · ΦT Φ + Eα [α] · I w + Eβ [β]wT ΦT t + const
2
∝ −
=
Therefore, by analogy to Eq (10.99)-(10.101), we can conclude that q? (w)
is still Gaussian, i.e., q? (w) = N (w|m N , S N ), where we have defined:
m N = Eβ [β]S N ΦT t
and
o−1
n
S N = Eβ [β] · ΦT Φ + Eα [α] · I
Next, we deal with q(β). According to definition, we have:
ln q? (β) ∝ Ew [ln p(t|w, β)] + ln p(β) + const
N
X
N
β
∝
· ln β − · E[ (t n − wT φn )2 ] + (c 0 − 1) ln β − d 0 β
2
2
n=1
=
=
=
=
=
=
=
N
β
+ c 0 − 1) · ln β − · E[||Φw − t||2 ] − d 0 β
2
2
n1
o
N
( + c 0 − 1) · ln β − β ·
· E[||Φw − t||2 ] + d 0
2
2
n1
o
N
( + c 0 − 1) · ln β − β ·
· E[wT ΦT Φw − 2tT Φw + tT t] + d 0
2
2
n1
o
N
1
( + c 0 − 1) · ln β − β ·
· Tr[ΦT ΦE[wwT ]] − tT ΦE[w] + tT t + d 0
2
2
2
n1
o
N
1 T
T
( + c 0 − 1) · ln β − β ·
· Tr[ΦT Φ(m N mT
+
S
)]
−
t
Φ
m
+
t
t
+
d
0
N
N
N
2
2
2
n
o
N
1
1 T T
1
T
T
( + c 0 − 1) · ln β − β · Tr[Φ ΦS N ] + m N Φ Φm N − t Φm N + tT t + d 0
2
2
2
2
o
N
1n
( + c 0 − 1) · ln β − β · Tr[ΦT ΦS N ] + ||Φm N − t||2 + 2d 0
2
2
(
Therefore, we obtain q? (β) = Gam(β| c N , d N ), where we have defined:
cN =
and
d N = d0 +
N
+ c0
2
o
1n
Tr[ΦT ΦS N ] + ||Φm N − t||2
2
228
Furthermore, notice that from (B.27), the expectations in m N and S N can
be expressed in a N , b N , and c N , d N :
E[α] =
aN
bN
and E[β] =
cN
dN
We have already obtained all the update formula. Next, we calculate the
lower bound. By noticing Eq (10.107), in this case, the first term on the right
hand side of Eq (10.107) will be modified, and two more terms will be added
on the right hand side, i.e., +E[ln p(β)] and −E[ln q? (β)]. Let’s start from
calculating the adding two terms:
+E[ln p(β)] = (c 0 − 1)E[ln β] − d 0 E[β] + c 0 ln d 0 − ln Γ(c 0 )
cN
= (c 0 − 1) · (ϕ(c N ) − ln d N ) − d 0
+ c 0 ln d 0 − ln Γ(c 0 )
dN
where we have used (B.26) and (B.30). Similarly, we have:
−E[ln q? (β)] = (c N − 1) · ϕ(c N ) − c N + ln d N − ln Γ(c N )
where we have used (B.31). Finally, we deal with the modification of the
first term on the right hand side of Eq (10.107):
o
N
β
ln 2π − Ew [||Φw − t||2 ]
2
2
2
Eβ [β]
N
N
Eβ [ln β] − ln 2π −
Ew [||Φw − t||2 ]
2
2
2
N
cN
(ϕ(c N ) − ln d N − ln 2π) −
Ew [||Φw − t||2 ]
2
2d N
o
N
cN n
(ϕ(c N ) − ln d N − ln 2π) −
Tr[ΦT ΦS N ] + ||Φm N − t||2
2
2d N
Eβ,w [ln p(t|w, β)] = Eβ
=
=
=
nN
ln β −
The last question is the predictive distribution. It is not difficult to observe that the predictive distribution is still given by Eq (10.105) and Eq
(10.106), with 1/β replaced by 1/E[β].
Problem 10.27 Solution
Let’s deal with the terms in Eq(10.107) one by one. Noticing Eq (10.87),
we have:
E[ln p(t|w)]w
i
N
N
N
β hX
ln(2π) + ln β − E
(t n − wT φn )2
2
2
2 n=1
i
N
N
N
X
X
N
N
β hX
= − ln(2π) + ln β − E
t2n − 2
t n · wT φ n +
wT φ n · φT
nw
2
2
2 n=1
n=1
n=1
i
N
β h T
N
= − ln(2π) + ln β − E t t − 2wT ΦT t + wT · (ΦT Φ) · w
2
2
2
h £
i
¤
N
N
β T
= − ln(2π) + ln β − t t − βE[wT ] · ΦT t + Tr E (wwT ) · (ΦT Φ)]
2
2
2
= −
229
Where we have defined Φ = [φ1 , φ2 , ..., φ N ]T , i.e., the i-th row of Φ is
T
φ i . Then using Eq(10.99), (10.100) and (10.103), it is easy to obtain (10.108).
Next, we deal with the second term by noticing Eq (10.88):
£
¤
E ln p(w|α) w,α
= −
M
M
E[α]α
ln(2π) + E[ln α]α −
· E[wwT ]w
2
2
2
Then using Eq (10.93)-(10.95), (B.27), (B.30) and Eq (10.103), we obtain
Eq (10.109) just as required. Then we deal with the third term in Eq (10.107)
by noticing Eq (10.89):
E[ln p(α)]α
a 0 ln b 0 + (a 0 − 1)E[ln α] − b 0 E[α] − ln Γ(a 0 )
=
Similarly, using Eq (10.93)-(10.95), (B.27), (B.30), we will obtain Eq (10.110).
Notice that there is a typo in Eq (10.110). The last term in Eq (10.110) should
be ln Γ(a 0 ) instead of ln Γ(a N ).
Finally we deal with the last two terms in Eq (10.107). We notice that
these two terms are actually negative entropy of a Gaussian and a Gamma
distribution, so that using (B.31) and (B.41), we can obtain:
−E[ln q(α)]α = H[α] = ln Γ(a N ) − (a N − 1) · ϕ(a N ) − ln b N + a N
and
1
M
ln |S N | + (1 + ln(2π))
2
2
Problem 10.28 Solution(waiting for update)
−E[ln q(w)]w = H[w] =
Problem 10.29 Solution
The second derivative of f (x) is given by:
1
d 1
d2
( ) = − 2 <0
(ln x) =
2
dx x
dx
x
Therefore, f (x) = ln x is concave for 0 < x < ∞. Based on definition, i.e.,
Eq (10.133), we can obtain:
g(λ) = min{λ x − ln x}
x
We observe that:
d
1
(λ x − ln x) = λ −
dx
x
In other words, when λ ≤ 0, λ x−ln x will always decrease as x increase. On
the other hand, when λ > 0, λ x − ln x will achieve its minimum when x = 1/λ.
Therefore, we conclude that:
g(λ) = λ ·
1
1
− ln = 1 + ln λ
λ
λ
230
Substituting g(λ) back into Eq (10.132), we obtain:
f (x) = min{λ x − 1 − ln λ}
λ
We calculate the derivative:
1
d
(λ x − 1 − ln λ) = x −
dλ
λ
Therefore, when λ = 1/x, λ x − 1 − ln λ achieves minimum with respect to
λ, which yields:
1
1
f (x) = · x − 1 − ln = ln x
x
x
In other words, we have shown that Eq (10.132) indeed recovers f (x) =
ln x.
Problem 10.30 Solution
We begin by calculating the first derivative:
d f (x)
− e− x
= σ(x) · e− x
=−
dx
1 + e− x
Then we can obtain the second derivative:
d 2 f (x)
− e− x (1 + e− x ) − e− x (− e− x )
=
= −[σ(x)]2 · e− x < 0
2
dx
(1 + e− x )2
Therefore, the log logistic function f (x) is concave. Utilizing this concave
property, we can obtain:
0
f (x) ≤ f (ξ) + f (ξ) · (x − ξ)
which gives,
ln σ(x) ≤ ln σ(ξ) + σ(ξ) · e−ξ · (x − ξ)
(∗)
Comparing the expression above with Eq (10.136), we define λ = σ(ξ) · e−ξ .
Then we can obtain:
λ = σ(ξ) · e−ξ =
e−ξ
1 + e−ξ
= 1−
1
1 + e−ξ
= 1 − σ (ξ )
In other words, we have obtained σ(ξ) = 1 − λ. In order to simplify (∗),
we need to express ξ using λ and x. According to the definition of λ, we can
obtain:
ξ = ln σ(ξ) − ln λ = ln(1 − λ) − ln λ
Now (∗) can be simplified as:
ln σ(x) ≤ ln(1 − λ) + λ · (x − ξ)
= λ · x + ln(1 − λ) − λ · ξ
£
¤
= λ · x + ln(1 − λ) − λ · ln(1 − λ) − ln λ
= λ · x + (1 − λ) ln(1 − λ) + λ ln λ
= λ · x − g(λ)
231
Just as required.
Problem 10.31 Solution
We start from calculating the derivative of f (x) with respect to x:
1 x/2
e − 1 e− x/2
d f (x)
1 e x/2 − e− x/2
= − 2 x/2 2− x/2 = − · x/2
dx
2 e + e− x/2
e +e
Then the second derivative is:
1 x/2
d 2 f (x)
dx2
= −
1 (2 e
2
+ 12 e− x/2 )(e x/2 + e− x/2 ) − (e x/2 − e− x/2 )( 12 e x/2 − 12 e− x/2 )
(e x/2 + e− x/2 )2
1 4· e · e
1
= −
= − x/2
<0
x
/2
−
x
/2
2
4 (e + e
)
(e + e− x/2 )2
x/2
− x/2
Therefore, the f (x) is concave with respect to x. We denote y = x2 , and
calculate:
1 −1/2
p
p
y
ª
d ©
df
=
− ln(e y/2 + e− y/2 ) = − 4
dy
dy
e
e
p
p
y/2
p
− 14 y−1/2 e−
y/2
p
y/2 + e− y/2
Then the second derivative can be calulated as:
d2 f
d y2
p
p
1 d © −1/2 e y/2 − e− y/2 ª
y
· p
= −
p
4 dy
e y/2 + e− y/2
p
p
o
1 n 1 −3/2 e y/2 − e− y/2
y−1/2
= − − y
· p
+ y−1/2 · p
p
p
4
2
e y/2 + e− y/2
(e y/2 + e− y/2 )2
p
p
© 1
ª
1
y−1
1
· − y−1/2 · (e y/2 − e− y/2 ) + p
= − p
p
p
4 e y/2 + e− y/2
2
e y/2 + e− y/2
In order to show that the second derivative is no less than 0, we only need
to prove:
p
p
1
1
− y−1/2 · (e y/2 − e− y/2 ) + p
≤0
p
y
/2
2
e
+ e− y/2
Which is equivalent to:
(e
p
y/2
p
+ e−
y/2
)(e
p
y/2
p
− e−
y/2
) ≥ 2y1/2 ⇐⇒ e
p
y/2
p
− e−
y/2
p
≥2 y
(∗)
We construct a function g(t) = e t − e− t − 2t, where t ≥ 0. Notice that:
p
0
g (t) = e t + e− t − 2 ≥ 2 e t · e− t − 2 = 0
Therefore, we conclude that g(t) is monotonically increasing on t ≥ 0, and
the minimum of g(t) is achieved when t = 0. Substituting t = 0 into g(t), we
obtain:
e t − e− t ≥ 2t, ∀ t ≥ 0
232
By now, we have already proved (∗), and thus f (x) is convex with respect
to x2 . Utilizing the convex property of f (x) with respect to x2 , we can obtain:
− ln(e
p
y/2
+e
p
− y/2
) ≥ − ln(e
p
²/2
+e
p
− ²/2
p
p
1
e ²/2 − e− ²/2
) − ²−1/2 · p
· (y − ²)
p
4
e ²/2 + e− ²/2
Substituting y = x2 and ² = ξ2 back into the expression above, we obtain:
1
eξ/2 − e−ξ/2
− ln(e x/2 + e− x/2 ) ≥ − ln(eξ/2 + e−ξ/2 ) − ξ−1 · ξ/2
· (x2 − ξ2 )
4
e + e−ξ/2
= − ln(eξ/2 + e−ξ/2 ) − λ(ξ) · (x2 − ξ2 )
Where we have used the definition of λ(ξ), i.e., Eq (10.141). Notice that
the expression above is identical to Eq (10.143), from which we can easily
obtain Eq (10.144).
Figure 5: f (x) with respect to x and x2
Problem 10.32 Solution
By carefully reading Section 10.6.1, we can obtain an Gaussian approximation to the posterior p(w|t), i.e,. Eq (10.156)-(10.168) if variational lower
bound of logistic sigmoid function has been used, i.e., Eq (10.151). What the
1
problem asks is to obtain a sequential update for m N and S−
.
N
The intrinsic reason is that we notice that each data point {φn , t n } corresponds to one variational parameter ξn and the likelihood is given by the
233
product of p(t n |w). Let’s begin by deriving a sequential update formula for
1
S−
:
N
1
S−
N +1
1
= S−
0 +2
NX
+1
λ(ξn )φn φT
n
n=1
−1
= 2λ(ξ N +1 )φ N +1 φT
N +1 + S0 + 2
N
X
λ(ξn )φn φT
n
n=1
−1
= 2λ(ξ N +1 )φ N +1 φT
N +1 + S N
Next we derive a sequential update formula for m N :
m N +1
i
h
NX
+1
1
= S N +1 S−
(t
−
1/2)
φ
m
+
n
0
n
0
n=1
h
i
N
X
1
= S N +1 S−
(t
−
1/2)
φ
+
(t
−
1/2)
φ
m
+
n
0
N +1
n
N +1
0
n=1
i
h
N
X
¤
£ −1
1
= S N +1 S−
(t n − 1/2)φn + (t N +1 − 1/2)φ N +1
N S N S0 m0 +
n=1
i
h
1
= S N +1 S−
N m N + (t N +1 − 1/2)φ N +1
1
In conclusion, when a new data (φ N +1 , t N +1 ) arrives, we first update S−
,
N +1
and then update m N +1 based on the formula we obtained above.
Problem 10.33 Solution
To prove Eq (10.163), we only need to prove Eq (10.162), from which Eq
(10.163) can be easily derived according to the text below Eq (10.132). Therefore, in what follows, we prove that the derivative of Q(ξ, ξold ) with respect to
ξn will give Eq (10.162). We start by noticing Eq (4.88), i.e.,
d σ (ξ )
= σ(ξ) · (1 − σ(ξ))
dξ
Noticing Eq (10.150), now we can obtain:
dQ(ξ, ξold )
d ξn
0
σ (ξn )
0
1
T
2
− λ (ξ n ) · (φ T
n E[ww ]φ n − ξ n ) − λ(ξ n ) · (−2ξ n )
σ(ξn ) 2
0
1
T
2
= 1 − σ(ξn ) − + 2ξn · λ(ξn ) − λ (ξn ) · (φT
n E[ww ]φ n − ξ n )
2
0
1
1
T
2
=
− σ(ξn ) + σ(ξn ) − − λ (ξn ) · (φT
n E[ww ]φ n − ξ n )
2
2
0
T
2
= −λ (ξn ) · (φT
n E[ww ]φ n − ξ n )
=
−
Setting the derivative equal to zero, we obtain Eq (10.162),from which Eq
(10.16b3) follows.
234
Problem 10.34 Solution
First, we should clarify one thing and that is there is typos in Eq(10.164).
It is not difficult to observe these error if we notice that for q(w) = N (w|m N , S N ),
in its logarithm, i.e.,ln q(w), 21 ln |S N | should always have the same sign as
1 T −1
2 m N S N m N . This is our intuition. However, this is not the case in Eq(10.164).
Based on Eq(10.159), Eq(10.153) and the Gaussian prior p(w) = N (w|m0 , S0 ),
we can analytically obtain the correct lower bound L(ξ) (this will also be
strictly proved by the next problem):
L(ξ) =
=
1
1
1
1 T −1
−1
ln |S N | − ln |S0 | + mT
N S N m N − m0 S0 m0
2
2
2
2
N ©
X
ª
1
+
ln σ(ξn ) − ξn + λ(ξn )ξ2n
2
n=1
N ©
X
ª
1
1
1
−1
ln |S N | + mT
ln σ(ξn ) − ξn + λ(ξn )ξ2n + const
N SN mN +
2
2
2
n=1
Where const denotes the term unrelated to ξn because m0 and S0 don’t
1
depend on ξn . Moreover, noticing that S−
· m N also doesn’t depend on ξn
N
according to Eq(10.157),thus it will be convenient to define a variable: z N =
1
S−
· m N , and we can easily verify:
N
−1
−1
T −1
−1
T −1
T −1
mT
N S N m N = [S N S N m N ] S N [S N S N m N ] = [S N z N ] S N [S N z N ] = z N S N z N
Now, we can obtain:
N ©
X
ªo
d n1
1
1
∂L(ξ)
2
=
ln |S N | + zT
S
z
+
ln
σ
(
ξ
)
−
ξ
+
λ
(
ξ
)
ξ
n
n
n n
N N
∂ξn
d ξn 2
2 N
2
n=1
0
1 £ −1 ∂S N ¤ 1 £
∂S N ¤
=
Tr S N
+ Tr z N zT
+ λ (ξn )ξ2n
N·
2
∂ξn
2
∂ξn
Where we have used Eq(3.117) for the first term, and for the second term
we have used:
ª 1 d n £
¤o 1 £
d ©1 T
∂S N ¤
zN SN zN =
Tr z N zT
= Tr z N zT
N · SN
N·
d ξn 2
2 d ξn
2
∂ξn
Furthermore, for the last term, we can follow the same procedure as in
the previous problem and now our remain task is to calculate ∂S N /∂ξn . Based
on Eq(10.158) and (C.21), we can obtain:
∂S N
∂ξn
= −S N
1
∂S−
N
∂ξn
0
S N = −S N · [2λ (ξn )φn φT
n ] · SN
Substituting it back into the derivative, we can obtain:
0
∂L(ξ)
1 £ −1
∂S N ¤
=
Tr (S N + z N zT
+ λ (ξn )ξ2n
N)
∂ξn
2
∂ξn
¤
0
0
1 £
1
T
T
2
= − Tr (S−
N + z N z N )S N · [2λ (ξ n )φ n φ n ] · S N + λ (ξ n )ξ n
2
n £
¤ 2o
0
1
T
T
= −λ (ξn ) · Tr (S−
+
z
z
)
·
S
·
φ
φ
·
S
N N
N
N − ξn = 0
n n
N
235
Therefore, we can obtain:
ξ2n
£
¤
1
T
T
= Tr (S−
N + z N z N ) · S N · φn φn · S N
1
T
= (S N · φn )T · (S−
N + z N z N ) · (S N · φ n )
T
= φT
n · (S N + S N z N z N S N ) · φ n
T
= φT
n · (S N + m N m N ) · φ n
1
Where we have used the defnition of z N ,i.e., z N = S−
· m N and also reN
peatedly used the symmetry property of S N .
Problem 10.35 Solution
There is a typo in Eq (10.164), for more details you can refer to the previous problem. Let’s calculate L(ξ) based on Based on Eq(10.159), Eq(10.153)
and the Gaussian prior p(w) = N (w|m0 , S0 ):
h(w, ξ) p(w) = N (w|m0 , S0 ) ·
N
Y
n
σ(ξn ) exp wT φn tn − (wT φn + ξn )/2
n=1
o
−λ(ξn )([w φn ]2 − ξ2n )
T
=
o
o
n 1
n
N
Y
1
(w
−
m
)
σ(ξn ) · exp − (w − m0 )T S−
(2π)−W /2 · |S0 |−1/2 ·
0
0
2
n=1
n
o
N
Y
·
exp wT φn tn − (wT φn + ξn )/2 − λ(ξn )([wT φn ]2 − ξ2n )
n=1
n
N
N
N ξ
Y
X
X
¡ 1
¢o
n
2
1
σ(ξn ) · exp − m0T S−
λ
(
ξ
)
ξ
=
(2π)−W /2 · |S0 |−1/2 ·
+
m
−
n
0
n
0
2
n=1
n=1
n=1 2
n 1
³
´
³
N
N
X
X
1 ´o
1
T
T
−1
· exp − wT S−
+
2
λ
(
ξ
)
φ
φ
w
+
w
S
m
+
φ
(t
−
n
0
n
n n
n
0
0
2
2
n=1
n=1
Noticing Eq (10.157)-(10.58), we can obtain:
n
N
N ξ
N
Y
X
X
¡ 1
¢o
n
1
(2π)−W /2 · |S0 |−1/2 ·
σ(ξn ) · exp − m0T S−
+
λ(ξn )ξ2n
0 m0 −
2
n=1
n=1 2
n=1
n 1
o
1
T −1
· exp − wT S−
N w + w SN mN
2
n
N
Y
=
(2π)−W /2 · |S0 |−1/2 ·
σ(ξn )
h(w, ξ) p(w) =
n=1
N ξ
N
X
X
¡ 1
¢o
1
n
1
−1
· exp − m0T S−
+
λ(ξn )ξ2n + mT
0 m0 −
N SN mN
2
2
n=1 2
n=1
n 1
o
1
· exp − (w − m N )T S−
N (w − m N )
2
Therefore, utilizing the normalization constant of Gaussian distribution,
236
now we can obtain:
Z
n
N
Y
σ(ξn )
h(w, ξ) p(w) dw = (2π)W /2 · |S N |1/2 · (2π)−W /2 · |S0 |−1/2 ·
n=1
N ξ
X
N
X
¡ 1
¢o
1 T −1
n
1
2
· exp − m0T S−
m
−
+
λ
(
ξ
)
ξ
m
+
S
m
0
n
N
n
0
2
2 N N
n=1 2
n=1
n |S |
N
N 1/2 Y
σ (ξ n )
) ·
=
(
| S0 |
n=1
N ξ
N
X
X
¡ 1
¢o
1 T −1
n
2
1
· exp − m0T S−
λ
(
ξ
)
ξ
+
m
−
+
m
S
m
n n
0
N
0
2
2 N N
n=1 2
n=1
Therefore, L(ξ) can be written as:
Z
L(ξ) = ln h(w, ξ) p(w) dw
=
o
N n
X
1
1
1 |S N | 1 T −1
−1
ln σ(ξn ) − ξn + λ(ξn )ξ2n
ln
− m0 S0 m0 + mT
N SN mN +
2
|S0 | 2
2
2
n=1
Problem 10.36 Solution
Let’s clarify this problem. What this problem wants us to prove is that
suppose at beginning the joint distribution comprises a product of j −1 factors,
i.e.,
jY
−1
p j−1 (D, θ ) =
f j−1 (θ )
i =1
and now the joint distribution comprises a product of j factors:
p j (D, θ ) =
j
Y
f j (θ ) = p j−1 (D, θ ) · f j (θ )
i =1
Then we are asked to prove Eq (10.242). This situation corresponds to
j − 1 data points at the beginning and then one more data point is obtained.
For more details you can read the text below Eq (10.188). Based on definition,
we can write down:
Z
p j (D) =
p j (D, θ ) d θ
Z
=
p j−1 (D, θ ) · f j (θ ) d θ
Z
=
p j−1 (D) · p j−1 (θ |D) · f j (θ ) d θ
Z
= p j−1 (D) · p j−1 (θ |D) · f j (θ ) d θ
Z
≈ p j−1 (D) · q j−1 (θ ) · f j (θ ) d θ
=
p j−1 (D) · Z j
237
Where we have sequentially used Bayes’ Theorem, q j−1 (θ ) is an approximation for the posterior p j−1 (θ |D), and Eq (10.197). To further prove Eq
(10.243), we only need to recursively use the expression we have proved.
Problem 10.37 Solution
Let’s start from definition. q() will be initialized as
qinit (θ ) = fe0 (θ )
Y
fei (θ ) = f 0 (θ )
i 6=0
Y
fei (θ )
i 6=0
Where we have used fe0 (θ ) = f 0 (θ ) according to the problem description.
Then we can obtain:
Y
q(θ )
q/0 (θ ) =
=
fei (θ )
fe0 (θ ) i6=0
Next, we will obtain qnew (θ ) by matching its moments against q/0 (θ ) f 0 (θ ),
which exactly equals:
q/0 (θ ) f 0 (θ ) =
Y
q(θ )
=
fei (θ ) · f 0 (θ ) = qinit (θ )
e
f 0 (θ ) i6=0
In other words, in order to obtain qnew (θ ), we need to match its moment
against q/0 (θ ), and since qnew and qinit both belong to exponential family,
they will be identical if they have the same moment. Moreover, based on Eq
(10.206), we have:
Z
Z
/0
Z0 = q (θ ) f 0 (θ ) d θ = qinit (θ ) d θ = 1
Therefore, based on Eq(10.207), we have:
qnew (θ )
qinit (θ )
fe0 (θ ) = Z0 /0
= 1 · /0
= f 0 (θ )
q (θ )
q (θ )
Problem 10.38 Solution
Based on Eq (10.205), (10.212) and (10.213), we can obtain:
q/ j (θ )
q(θ )
N (θ |m, vI)
=
s n N (θ |mn , vn I)
fej (θ )
n
o
exp − 12 (θ − m)T (vI)−1 (θ − m)
n
o
∝
exp − 12 (θ − mn )T (vn I)−1 (θ − mn )
o
n 1
1
= exp − (θ − m)T (vI)−1 (θ − m) + (θ − mn )T (vn I)−1 (θ − mn )
2
2
n 1
o
T
T
= exp − (θ Aθ + θ · B + C)
2
=
238
Where we have completed squares over θ in the last step, and we have
defined:
h
i
A = (vI)−1 − (vn I)−1 and B = 2 · − (vI)−1 · m + (vn I)−1 · mn
Note that in order to match this to a Gaussian, we don’t actually need C,
so we omit it here. Now we match this against a Gaussian, beginning by first
considering the quadratic term, we can obtain:
1 −1
[Σ/n ]−1 = (vI)−1 − (vn I)−1 = (v−1 − v−
= [v/n ]−1 · I−1
n )I
It is identical to Eq (10.214). By matching the linear term, we can also
obtain:
h
i
−2 · [Σ/n ]−1 · (m/n ) = B = 2 · − (vI)−1 · m + (vn I)−1 · mn
Rearranging it, we can obtain:
i
h
(m/n ) = −[Σ/n ] · − (vI)−1 · m + (vn I)−1 · mn
h
i
1
·
m
= −[v/n ] · − v−1 · m + v−
n
n
v/ n
· mn
vn
=
v/n · v−1 · m −
=
1
v/n ([v/n ]−1 − v−
n )·m−
= m+
v/ n
· mn
vn
v/ n
· (m − mn )
vn
Which is identical to Eq (10.214). One important thing worthy clarified is that: for arbitrary two Gaussian random variable, their division is
not a Gaussian. You can find more details by typing "ratio distribution" in
Wikipedia. Generally speaking, the division of two Gaussian random variable follows a Cauchy distribution. Moreover, the product of two Gaussian
random variables is not a Gaussian random variable.
However, the product of two Gaussian PDF, e.g.,p(x) and p(y), can be a
Gaussian PDF because when x and y are independent, p(x, y) = p(x)p(y),
is a Gaussian PDF. In the EP framework,according to Eq (10.204), we have
already assumed that q(θ ), i.e., Eq (10.212), is given by the product of fej (θ ),
i.e.,(10.213). Therefore, their division still gives by the product of many remaining Gaussian PDF, which is still a Gaussian.
Finally, based on Eq (10.206) and (10.209), we can obtain:
Z
Zn =
q/n (θ )p(xn |θ ) d θ
Z
©
ª
=
N (θ |m/n , v/n I) · (1 − w)N (xn |θ , I) + wN (xn |0, αI) d θ
Z
Z
/n /n
= (1 − w) N (θ |m , v I)N (xn |θ , I) d θ + w N (θ |m/n , v/n I) · N (xn |0, αI) d θ
= (1 − w)N (xn |m/n , (v/n + 1)I) + wN (xn |0, αI)
239
Where we have used Eq (2.115).
Problem 10.39 Solution
This problem is really complicated, but hint has already been given in
Eq (10.244) and (10.255). Notice that in Eq (10.244), we have a quite complicated term ∇m/n ln Z n , which we know that ∇m/n ln Z n = (∇m/n Z n )/Z n based
on the Chain Rule, and since we know the exact form of Z n which has been
derived in the previous problem, we guess that we can start from dealing
with ∇m/n ln Z n to obtain Eq (10.244). Before starting, we write down a basic
formula here: for a Gaussian random variable x ∼ N (x|µ, Σ), we have:
∇µ N (x|µ, Σ) = N (x|µ, Σ) · (x − µ)Σ−1
Now we can obtain:
∇m/n ln Z n
=
=
=
=
=
=
1
· ∇ /n Z n
Zn m
Z
1
· ∇ /n q/n (θ )p(xn |θ )d θ
Zn m
Z n
o
1
·
∇m/n q/n (θ ) · p(xn |θ )d θ
Zn
Z
1
1
(θ − m/n ) · q/n (θ ) · p(xn |θ ) d θ
·
Zn
v/ n
Z
Z
o
1
1 n
/n
θ · q (θ ) · p(xn |θ ) d θ − m/n · q/n (θ ) · p(xn |θ ) d θ
· /n ·
Zn v
o
1 n
/n
·
E
[
θ
]
−
m
v/ n
Here we have used q/n (θ ) = N (θ |m/n , v/n I), and q/n (θ ) · p(xn |θ ) = Z n ·
q
(θ ). Rearranging the equation above, we obtain Eq (10.244). Then we
use Eq (10.216), yielding:
new
E[θ ] = m/n + v/n · ∇m/n ln Z n
1
1
= m/n + v/n ·
(1 − w)N (xn |m/n , (v/n + 1)I) · /n
(xn − m/n )
Zn
v +1
1
= m/n + v/n · ρ n · /n
(xn − m/n )
v +1
Where we have defined:
ρn
1
(1 − w)N (xn |m/n , (v/n + 1)I)
Zn
1
Z n − wN (xn |0, αI)
=
(1 − w) ·
Zn
1−w
w
= 1−
N (xn |0, αI)
Zn
=
240
Therefore, we have proved the mean m is given by Eq (10.217), next we
prove Eq (10.218). Similarly, we can write down:
∇v/n ln Z n
=
=
=
=
=
=
1
· ∇ /n Z n
Zn v
Z
1
· ∇v/n q/n (θ )p(xn |θ )d θ
Zn
Z n
o
1
·
∇v/n q/n (θ ) p(xn |θ )d θ
Zn
Z n
1
1
D o /n
/n
2
·
||
m
−
θ
||
−
q (θ ) · p(xn |θ )d θ
Z
2(v/n )2
2v/n
Zn
n 1
D o
/n
T
/n
(m
−
θ
)
(m
−
θ
)
−
dθ
qnew (θ ) ·
2(v/n )2
2v/n
o
1 n
D
T
/n
/n 2
E
[
θθ
]
−
2
E
[
θ
]m
+
||
m
||
− /n
/
n
2
2(v )
2v
Rearranging it, we can obtain:
E[θθ T ] = 2(v/n )2 · ∇v/n ln Z n + 2E[θ ]m/n − ||m/n ||2 + D · v/n
There is a typo in Eq (10.255), and the intrinsic reason is that when calculating ∇v/n q/n (θ ), there are two terms in q/n (θ ) dependent on v/n : one is inside
the exponential, and the other is in the fraction |v/n1I|1/2 , which is outside the
exponential. Now, we still use Eq (10.216), yielding:
∇v/n ln Z n
£
¤
1
1
D
/n 2
(1 − w)N (xn |m/n , (v/n + 1)I) ·
||
x
−
m
||
−
n
Zn
2(v/n + 1)2
2(v/n + 1)
¤
£
D
1
||xn − m/n ||2 −
= ρn ·
/
n
2
/
2(v + 1)
2(v n + 1)
=
Finally, using the definition of variance, we obtain:
vI = E[θθ T ] − E[θ ]E[θ T ]
Therefore, taking the trace, we obtain:
v =
=
=
=
o
o
1 n
1 n
· E[θ T θ ] − E[θ T ]E[θ ] =
· E[θ T θ ] − ||E[θ ]||2
D
D
o
1 n /n 2
· 2(v ) · ∇v/n ln Z n + 2E[θ ]m/n − ||m/n ||2 + D · v/n − ||E[θ ]||2
D
o
1 n /n 2
· 2(v ) · ∇v/n ln Z n − ||E[θ ] − m/n ||2 + D · v/n
D
o
1
1 n /n 2
· 2(v ) · ∇v/n ln Z n − ||v/n · ρ n · /n
(xn − m/n )||2 + D · v/n
D
v +1
If we substitute ∇v/n ln Z n into the expression above, we will just obtain
Eq (10.215) as required.
241
0.11
Sampling Methods
Problem 11.1 Solution
Based on definition, we can write down:
E[ fb] = E[
L
1X
f (z(l ) )]
L l =1
=
L
1X
E[ f (z(l ) )]
L l =1
=
1
· L · E[ f ] = E[ f ]
L
Where we have used the fact that the expectation and the summation can
exchange order because all the z(l ) are independent, and that E[ f (z(l ) )] = E[ f ]
because all the z(l ) are drawn from p(z). Next, we deal with the variance:
var[ fb] = E[( fb − E[ fb])2 ] = E[ fb2 ] − E[ fb]2 = E[ fb2 ] − E[ f ]2
L
1X
= E[(
f (z(l ) ))2 ] − E[ f ]2
L l =1
=
L
X
1
E
[(
f (z(l ) ))2 ] − E[ f ]2
L2 l =1
=
L
L
X
1 X
2 (l )
E
[
f
(z
)
+
f (z( i) ) f (z( j) )] − E[ f ]2
L2 l =1
i, j =1,i 6= j
=
L
L2 − L
1 X
2 (l )
(z
)]
+
E
[
f
E[ f ]2 − E[ f ]2
L2 l =1
L2
=
=
=
L
1 X
1
E[ f 2 (z(l ) )] − E[ f ]2
L
L2 l =1
1
1
· L · E[ f 2 ] − E[ f ] 2
2
L
L
1
1
1
2
E[ f ] − E[ f ]2 = E[( f − E[ f ])2 ]
L
L
L
Just as required.
Problem 11.2 Solution
What this problem wants us to prove is that if we use y = h−1 (z) to transform the value of z to y, where z satisfies a uniform distribution over [0, 1]
and h(·) is defined by Eq(11.6), we can enforce y to satisfy a specific desired
distribution p(y). Let’s prove it beginning by Eq (11.1):
Z y
0
dz
d
p? (y) = p(z) · |
| = 1 · h (y) =
p( b
y)d b
y = p(y)
dy
d y −∞
242
Just as required.
Problem 11.3 Solution
We use what we have obtained in the previous problem.
Z y
h(y) =
p( b
y) d b
y
Z−∞
y 1
1
=
db
y
y2
−∞ π 1 + b
= tan−1 (y)
Therefore, since we know that z = h(y) = tan−1 (y), we can obtain the
transformation from z to y: y = tan(z).
Problem 11.4 Solution
First, I believe there is a typo in Eq (11.10) and (11.11). Both ln z1 and
ln z2 should be ln(z12 + z22 ). In the following, we will solve the problem under
this assumption.
We only need to calculate the Jacobian matrix. First, based on Eq (11.10)(11.11), it is not difficult to observe that z1 only depends on y1 , and z2 only
depends on y2 , which means that ∂ z1 /∂ y2 = 0 and ∂ z2 /∂ y1 = 0. To obtain
the diagonal terms of the Jacobian matrix, i.e., ∂ z1 /∂ y1 and ∂ z2 /∂ y2 . To deal
with the problem associated with a circle, it is always convenient to use polar
coordinate:
z1 = r cos θ , and z2 = r sin θ
It is easily to obtain:
∂(z1 , z2 )
·
∂ z 1 /∂ r
=
∂ z2 /∂ r
∂(r, θ )
¸ ·
cos θ
∂ z1 /∂θ
=
sin θ
∂ z2 /∂θ
− r sin θ
r cos θ
¸
Therefore, we can obtain:
|
∂(z1 , z2 )
∂(r, θ )
| = r(cos2 θ + sin2 θ ) = r
Then we substitute r and θ into Eq (11.10), yielding:
−2 ln r 2 1/2
) = cos θ (−2 ln r 2 )1/2
r2
(∗)
y2 = sin θ (−2 ln r 2 )1/2
(∗∗)
y1 = r cos θ (
Similarly, we also have:
It is easily to obtain:
·
¸ ·
∂(y1 , y2 )
∂ y1 /∂ r ∂ y1 /∂θ
−2 cos θ (−2 ln r 2 )−1/2 · r −1
=
=
∂ y2 /∂ r ∂ y2 /∂θ
−2 sin θ (−2 ln r 2 )−1/2 · r −1
∂(r, θ )
− sin θ (−2 ln r 2 )1/2
cos θ (−2 ln r 2 )1/2
¸
243
Therefore, we can obtain:
|
∂(y1 , y2 )
∂(r, θ )
| = (−2r −1 (cos2 θ + sin2 θ )) = −2r −1
Next, we need to use the property of Jacobian Matrix:
|
∂(z1 , z2 )
∂(y1 , y2 )
| = |
=
=
=
∂(z1 , z2 )
·
∂(r, θ )
|
∂(r, θ ) ∂(y1 , y2 )
∂(r, θ )
∂(z1 , z2 )
|·|
|
|
∂(r, θ )
∂(y1 , y2 )
∂(z1 , z2 ) ∂(y1 , y2 ) −1
|
|·|
|
∂(r, θ )
∂(r, θ )
r · (−2r −1 )−1 = −
r2
2
By squaring both sides of (∗) and (∗∗) and adding them together, we can
obtain:
n y2 + y2 o
2
y12 + y22 = −2 ln r 2 => r 2 = exp 1
−2
Finally, we can obtain:
p(y1 , y2 ) = p(z1 , z2 )|
∂(z1 , z2 )
∂(y1 , y2 )
2
|=
2
ny +y o
1
r2
1 2
1
2
·|− | =
r =
exp 1
π
2
2π
2π
−2
Just as required.
Problem 11.5 Solution
This is a linear transformation of z, we still obtain a Gaussian random
variable y. We only need to match its moments (mean and variance). We
know that z ∼ N (0, I), Σ = LLT , and y = µ + Lz. Now, using E[z] = 0, we
obtain:
E[y] = E[µ + Lz]
= µ + L · E[z]
= µ
Moreover, using cov[z] = E[zzT ] − E[z]E[zT ] = E[zzT ] = I, we can obtain:
cov[y] = E[yyT ] − E[y]E[yT ]
= E[(µ + Lz) · (µ + Lz)T ] − µµT
= E[µµT + 2µ · (Lz)T + (Lz) · (Lz)T ] − µµT
= 2µ · E[zT ] · LT + E[LzzT LT ]
= L · E[zzT ] · LT = L · I · LT
= Σ
244
Just as required.
Problem 11.6 Solution
This problem is all about definition. According to the description of rejection sampling, we know that: for a specific value z0 (drawn from q(z)), we
will generate a random variable u 0 , which satisfies a uniform distribution in
the interval [0, kq(z0 )], and if the generated value of u 0 is less than pe(z0 ), we
will accept this value. Therefore, we obtain:
P[accept|z0 ] =
pe(z0 )
kq(z0 )
Since we know z0 is drawn from q(z), we can obtain the total acceptance
rate by integral:
Z
Z
pe(z0 )
dz0
P[accept] = P[accept|z0 ] · q(z0 ) dz0 =
k
It is identical to Eq (11.14). We substitute Eq (11.13) into the expression
above, yielding:
Zp
P[accept] =
k
We define a very small vector ², and we can obtain:
P[x0 ∈ (x, x + ²)] =
=
=
P[z0 ∈ (x, x + ²)|accept]
P[accept, z0 ∈ (x, x + ²)]
P[accept]
R
(x,x+²) q(z0 )P[accept|z0 ] dz0
Z p /k
k
· q(z0 ) · p(accept|z0 ) dz0
Z
(x,x+²) p
Z
pe(z0 )
k
· q(z0 ) ·
dz0
=
kq(z0 )
(x,x+²) Z p
Z
1
=
·p
e(z0 ) dz0
(x,x+²) Z p
Z
=
p(z0 ) dz0
Z
=
(x,x+²)
Just as required. Several clarifications must be made here:(1)we
have used P[A] to represent the probability of event A occurs, and p(z) or q(z)
to represent the Probability Density Function (PDF). (2) Please be careful
with P[x0 ∈ (x, x + ²)] = P[z0 ∈ (x, x + ²)|accept], and this is the key point of
this problem.
Problem 11.7 Solution
245
Notice that the symbols used in the main text is different from those in
the problem description. in the following, we will use those in the main text.
Namely, y satisfies a uniform distribution on interval [0, 1], and z = b tan y+ c.
Then we aims to prove Eq (11.16). Since we know that:
q(z) = p(y) · |
dy
|
dz
and that:
y = arctan
z−c
b
=>
dy
1
1
= ·
dz
b 1 + [(z − c)/b]2
Substituting it back, we obtain:
q(z) = 1 ·
1
1
·
b 1 + [(z − c)/b]2
In my point of view, Eq (11.16) is an expression for the comparison function kq(z), not the proposal function q(z). If we wish to use Eq (11.16) to
express the proposal function, the numerator in Eq (11.16) should be 1/b instead of k. Because the proposal function q(z) is a PDF, it should integrate
to 1. However, in rejection sampling, the comparison function is what we
actually care about.
Problem 11.8 Solution
There is a typo in Eq (11.17), which is not difficult to observe, if we carefully examine Fig.11.6. The correct form should be:
q i (z) = k i λ i exp{−λ i (z − z i )},
e
z i−1,i < z ≤ e
z i,i+1 ,
where i = 1, 2, ..., N
Here we use e
z i,i+1 to represent the intersection point of the i-th and i + 1th envelope, q i (z) to represent the comparison function of the i-th envelope,
and N is the total number of the envelopes.Notice that e
z0,1 and e
z N,N +1 could
be −∞ and ∞ correspondingly.
First, from Fig.11.6, we see that: q(z i ) = pe(z i ), substituting the expression above into the equation and yielding:
k i λ i = pe(z i )
(∗)
One important thing should be made clear is that we can only evaluate
pe(z) at specific point z, but not the normalized PDF p(z). This is the assumption of rejection sampling. For more details, please refer to section 11.1.2.
Notice that q i (z) and q i+1 (z) should have the same value at e
z i,i+1 , we
obtain:
k i λ i exp{−λ i ( e
z i,i+1 − z i )} = k i+1 λ i+1 exp{−λ i+1 ( e
z i,i+1 − z i+1 )}
246
After several rearrangement, we obtain:
e
z i,i+1 =
n
o
k i λi
1
ln
+ λ i z i − λ i+1 z i+1
λ i − λ i+1
k i+1 λ i+1
(∗∗)
Before moving on, we should make some clarifications: the adaptive rejection sampling begins with several grid points, e.g., z1 , z2 , ..., z N , and then
we evaluate the derivative of pe(z) at those points, i.e., λ1 , λ2 , ..., λ N . Then we
can easily obtain k i based on (∗), and next the intersection points e
z i,i+1 based
on (∗∗).
Problem 11.9 Solution
In this problem, we will still use the same notation as in the previous one.
First, we need to know the probability of sampling from each segment. Notice
that Eq (11.17) is not correctly normalized, we first calculate its normalization constant Z q :
Zq
=
=
=
Z ez N,N +1
e
z 0, 1
N Z e
z i,i+1
X
z i−1,i
i =1 e
N
X
i =1
=
q(z) dz =
N
X
i =1
N Z e
z i,i+1
X
z i−1,i
i =1 e
q i (z i ) dz i
k i λ i exp{−λ i (z − z i )} dz i
¯ ez i,i+1
¯
− k i exp{−λ i (z − z i )}¯
e
z i−1,i
h
i X
N
bi
− k i exp{−λ i ( e
z i,i+1 − z i )} − exp{−λ i ( e
z i−1,i − z i )} =
k
Where we have defined:
h
i
b i = − k i exp{−λ i ( e
k
z i,i+1 − z i )} − exp{−λ i ( e
z i−1,i − z i )}
i =1
(∗)
From this derivation, we know that the probability of sampling from the
b i /Z q , where Z q = P N k
b
i-th segment is given by k
i =1 i . Therefore, now we define
an auxiliary random variable η, which is uniform in interval [0, 1], and then
define:
−1
j
X
1 jX
bm , 1
b m ], j = 1, 2, ..., N
i = j if η ∈ [
k
k
(∗∗)
Z q m=0
Z q m=0
b0 = 0 for convenience. Until now, we have decide
Where we have defined k
the chosen i-th segment. Next, we should sample from the i-th exponential
distribution using the technique in section 11.1.1.. According to Eq (11.6), we
247
can write down:
Z z
h i (z) =
=
=
=
=
q i (z i )
dz i
bi
e
z i−1,i
k
Z z
1
·
k λ exp{−λ i (z − z i )} dz i
b i ez i−1,i i i
k
¯z
−k i
¯
· exp{−λ i (z − z i )}¯
bi
e
z i−1,i
k
h
i
−k i
· exp{−λ i (z − z i )} − exp{−λ i ( e
z i−1,i − z i )}
bi
k
h
i
ki
· exp(λ i z i ) exp{−λ i e
z i−1,i } − exp{−λ i z}
bi
k
b i is the correct
Notice that q i (z) is not correctly normalized, and q i (z)/ k
normalized form. With several rearrangement, we can obtain:
h−i 1 (ξ) =
=
i
h
ξ
1
· ln exp{−λ i e
z i−1,i } − k
i
−λ i
b i · exp(λ i z i )
k
h
i
ln
exp
{−
λ
e
z
}
i
i
−
1
,i
1
·
bi ξ
k
−λ i
ln
k i ·exp(λ i z i )
=
e
z i−1,i
ln ξ + ln kk ii − λ i z i
b
In conclusion, we first generate a random variable η, which is uniform
in interval [0, 1], and obtain the value i according to (∗∗), and then we generate a random variable ξ, which is also uniform in interval [0, 1], and then
transform it to z using z = h−i 1 (ξ).
Notice that here, λ i , e
z i,i+1 and k i can be obtained once the grid points
z1 , z2 , ..., z N are given. For more details, please refer to the previous problem.
b i can also be determined using (∗), and
After these variables are obtained, k
−1
thus h i (ξ) can be determined.
Problem 11.10 Solution
Based on definition and Eq (11.34)-(11.36), we can write down:
Eτ [(z(τ) )2 ] = 0.5 · Eτ−1 [(zτ−1 )2 ] + 0.25 · Eτ−1 [(zτ−1 + 1)2 ] + 0.25 · Eτ−1 [(zτ−1 − 1)2 ]
= Eτ−1 [(zτ−1 )2 ] + 0.5
If the initial state is z(0) = 0 (there is a typo in the line below Eq (11.36)),
we can obtain Eτ [(z(τ) )2 ] = τ/2 just as required.
Problem 11.11 Solution
248
This problem requires you to know the definition of detailed balance, i.e.,
Eq (11.40):
p? (z)T(z, z0 ) = p? (z0 )T(z0 , z)
Note that here z and z0 are the sampled values of [z1 , z2 , ..., z M ]T in two
consecutive Gibbs Sampling step. Without loss of generality, we assume that
we are now updating zτj to zτj +1 in step τ:
p? (z)T(z, z0 ) =
p(z1τ , z2τ , ..., zτM ) · p(zτj +1 |zτ/ j )
=
p(zτj |zτ/ j ) · p(zτ/ j ) · p(zτj +1 |zτ/ j )
=
p(zτj |zτ/ j+1 ) · p(zτ/ j+1 ) · p(zτj +1 |zτ/ j+1 )
=
p(zτj |zτ/ j+1 ) · p(z1τ+1 , z2τ+1 , ..., zτM+1 )
=
T(z0 , z) · p? (z)
To be more specific, we write down the first line based on Gibbs sampling,
where zτ/ j denotes all the entries in vector zτ except zτj . In the second line,
we use the conditional property, i.e, p(a, b) = p(a| b)p(b) for the first term.
In the third line, we use the fact that zτ/ j = zτ/ j+1 . Then we reversely use
the conditional property for the last two terms in the fourth line, and finally
obtain what has been asked.
Problem 11.12 Solution
Obviously, Gibbs Sampling is not ergodic for this specific distribution, and
the quick reason is that neither the projection of the two shaded region on z1
axis nor z2 axis overlaps. For instance, we denote the left down shaded region
as region 1. If the initial sample falls into this region, no matter how many
steps have been carried out, all the generated samples will be in region 1. It
is the same for the right up region.
Problem 11.13 Solution
Let’s begin by definition.
p(µ| x, τ, µ0 , s 0 ) ∝
p(x|µ, τ, µ0 , s 0 ) · p(µ|τ, µ0 , s 0 )
=
p(x|µ, τ) · p(µ|µ0 , s 0 )
=
N (x|µ, τ−1 ) · N (µ|µ0 , s 0 )
Where in the first line, we have used Bayes’ Theorem:
p(µ| x, c) ∝ p(x|µ, c) · p(µ| c)
Now we use Eq (2.113)-Eq (2.117), we can obtain: p(µ| x, τ, µ0 , s 0 ) = N (µ|µ? , s? ),
where we have defined:
1
[s? ]−1 = s−
0 +τ
,
1
µ? = s? · (τ · x + s−
0 µ0 )
249
It is similar for p(τ| x, µ, a, b):
p(τ| x, µ, a, b) ∝
p(x|τ, µ, a, b) · p(τ|µ, a, b)
=
p(x|µ, τ) · p(τ|a, b)
=
N (x|µ, τ−1 ) · Gam(τ|a, b)
Based on Section 2.3.6, especially Eq (2.150)-(2.151), we can obtain p(τ| x, µ, a, b) =
Gam(τ|a? , b? ), where we have defined:
a? = a + 0.5 ,
b? = b + 0.5 · (x − µ)2
Problem 11.14 Solution
Based on definition, we can write down:
E[z0i ] = E[µ i + α(z i − µ i ) + σ i (1 − α2i )1/2 v]
= µ i + E[α(z i − µ i )] + E[σ i (1 − α2i )1/2 v]
= µ i + α · E[z i − µ i ] + [σ i (1 − α2i )1/2 ] · E[v]
= µi
Where we have used the fact that the mean of z i is µ i , i.e., E[z i ] = µ i , and
that the mean of v is 0, i.e., E[v] = 0. Then we deal with the variance:
var[z0i ] = E[(z0i − µ i )2 ]
= E[(α(z i − µ i ) + σ i (1 − α2i )1/2 v)2 ]
= E[α2 (z i − µ i )2 ] + E[σ2i (1 − α2i )v2 ] + E[2α(z i − µ i ) · σ i (1 − α2i )1/2 v]
= α2 · E[(z i − µ i )2 ] + σ2i (1 − α2i ) · E[v2 ] + 2α · σ i (1 − α2i )1/2 · E[(z i − µ i )v]
= α2 · var[z i ] + σ2i (1 − α2i ) · (var[v] + E[v]2 ) + 2α · σ i (1 − α2i )1/2 · E[(z i − µ i )] · E[v]
= α2 · σ2i + σ2i (1 − α2i ) · 1 + 0
= σ2i
Where we have used the fact that z i and v are independent and thus
E[(z i − µ i )v] = E[z i − µ i ] · E[v] = 0
Problem 11.15 Solution
Using Eq (11.57), we can write down:
∂H
∂r i
=
∂K
∂r i
= ri
Comparing this with Eq (11.53), we obtain Eq (11.58). Similarly, still
using Eq (11.57), we can obtain:
∂H
∂zi
=
∂E
∂zi
250
Comparing this with Eq (11.55), we obtain Eq (11.59).
Problem 11.16 Solution
According to Bayes’ Theorem and Eq (11.54), (11.63), we have:
p(r|z) =
Zp
p(z, r)
1/Z H · exp(− H(z, r))
=
=
· exp(−K(r))
p(z)
1/Z p · exp(−E(z))
ZH
where we have used Eq (11.57). Moreover, by noticing Eq (11.56), we
conclude that p(r|z) should satisfy a Gaussian distribution.
Problem 11.17 Solution
There are typos in Eq (11.68) and (11.69). The signs in the exponential of
the second argument of the min function is not right. To be more specific, Eq
(11.68) should be:
1
1
exp(− H(R))δV min{1, exp(H(R) − H(R 0 ))}
ZH
2
(∗)
and Eq (11.69) is given by:
1
1
exp(− H(R 0 ))δV min{1, exp(H(R 0 ) − H(R))}
ZH
2
(∗∗)
When H(R) = H(R 0 ), they are clearly equal. When H(R) > H(R 0 ), (∗) will
reduce to:
1
1
exp(− H(R))δV
ZH
2
Because the min function will give 1, and in this case (∗∗) will give:
1
1
1
1
exp(− H(R 0 ))δV exp(H(R 0 ) − H(R))} =
exp(− H(R))δV
ZH
2
ZH
2
Therefore, they are identical, and it is similar when H(R) < H(R 0 ).
0.12
Continuous Latent Variables
Problem 12.1 Solution
By analogy to Eq (12.2), we can conclude that the projected data with respect to a vector u M +1 should have variance given by uT
Su M +1 . Moreover,
M +1
there are two constraints for u M +1 : (1) it should be correctly normalized, i.e.,
uT
u
= 1, and (2) it should be orthogonal to all the previous M chosen
M +1 M +1
eigenvectors {u1 , u2 , ..., u M }. We aim to maximize the variance with respect
to u M +1 satisfying these two constraints. This can be done by enforcing the
Lagrange Multiplier:
T
L = uT
M +1 Su M +1 + λ(1 − u M +1 u M +1 ) +
M
X
m=1
η m uT
m u M +1
(∗)
251
Therefore, we can further calculate its derivative with respect to u M +1 :
∂L
∂u M +1
= 2Su M +1 − 2λu M +1 +
M
X
η m um = 0
m=1
We left multiply uT
m , yielding:
T
T
2uT
m Su M +1 − 2λu m u M +1 + u m
M
X
η m um
= 2uT
m Su M +1 − 0 + η m
m=1
= 2uT
M +1 Su m + η m
= 2uT
M +1 λ m u m + η m
= ηm
where we have used the property of orthogonality and in the second line
we have transpose the first term and use the property that S is symmetric. So
now we obtain η m = 0. This will directly lead (∗) reduce to the form as shown
in Eq (12.4), and thus consequently we now need to choose a eigenvector of S
among those not chosen, which has the largest eigenvalue.
Problem 12.2 Solution(Waiting for update)
Problem 12.3 Solution
According to Eq (12.30), we can obtain:
uTi u i =
1 T
v XXT v i
N λi i
We left multiply vTi on both sides of Eq (12.28), yielding:
1 T
v XXT v i = λ i vTi v i = λ i ||v i ||2 = λ i
N i
Here we have used the fact that v i has unit length. Substituting it back
into uTi u i , we can obtain:
uTi u i = 1
Just as required.
Problem 12.4 Solution
We know p(z) = N (z|m, Σ), and p(x|z) = N (x|Wz + µ, σ2 I). According to
Eq (2.113)-(2.115), we have:
bW
b T)
p(x) = N (x|Wm + µ, σ2 I + WΣWT ) = N (x|µ
b , σ2 I + W
where we have defined:
µ
b = Wm + µ
252
and
b = WΣ1/2
W
Therefore, in the general case, the final form of p(x) can still be written
as Eq (12.35).
Problem 12.5 Solution