50 Pen and Paper Exercises on Machine Learning
(with Solutions)
Tanujit Chakraborty (Blog)
2024
Topics to be covered (Exercises and Solutions):
1. Mathematics for Machine learning - Optimization and Probability
2. Linear Models
3. Decision Trees and Nearest Neighbors
4. Model Selection and Cross-Validation
5. Kernel machines (Support Vector Machines)
6. Probabilistic Machine Learning
7. EM Algorithm
8. Generative models
9. Unsupervised Learning
10. Neural Networks
Acknowledgment: These problems and their solutions are largely based on some extraordinary books on Machine Learning and several outstanding university courses available
online, with some minor additions by the author. A non-exhaustive list is given below.
Happy Learning!
References (Books):
1. Deisenroth, Marc Peter, A. Aldo Faisal, and Cheng Soon Ong. Mathematics for
machine learning. Cambridge University Press, 2020.
2. Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition and machine learning. Vol. 4. No. 4. New York: Springer, 2006.
3. Murphy, Kevin P. Machine learning: a probabilistic perspective. MIT Press, 2012.
References (Online Resources):
1. CS229: Machine Learning Course at Stanford University.
2. CS771A: Introduction to Machine Learning Course at IIT Kanpur.
3. 10-701 Machine Learning Course at CMU, USA.
MATH 370 - Machine Learning
By Tanujit Chakraborty
Machine Learning - Pen and Paper Exercises (with Solutions)
Problem 1. (Placement Woes)
Tanujit wants to set up a shop at a point in R2 . The consumer density is such that if he opens shop at a point (x, y) ∈ R2 , his
daily income will be 2x+4y. However, since his supplier is situated at (2, −1), he will incur a daily cost of (x−2)2 +(y +1)2
for transporting goods from his supplier to his shop. Where should Tanujit open his shop to get the maximum daily profit,
and what is that maximum profit value?
Solution:
We cast the above problem as an optimization problem. Let a = (2, 4), b = (2, −1) ∈ R2 . We wish to minimize
f (x) = ∥x − b∥22 − a⊤ x (minimizing loss is the same as maximizing profit). We have ∇f (x) = 2(x − b) − a and
∇2 f (x) = 2I > 0, i.e., f is a convex differentiable function. Using first-order optimality to get the global minimum gives
us xopt = b + a/2 = (3, 1) and the profit earned by opening up a shop at xopt is −f (xopt ) = 5 units.
Problem 2. (Bayes Rule)
Suppose that 1% of Olympic athletes use performance-enhancing drugs and that a particular drug test has a 5% false
positive rate and a 2% false negative rate.
(a) Athlete A tests positive for drug use. What is the probability that Athlete A is not using drugs?
(b) Athlete B tests negative for drug use. What is the probability that Athlete B is using drugs?
Solution:
Some calculations about joint probabilities.
P (D = 0) = 1 − 0.01 = 0.99
P (D = 1) = 0.01
P (D = 0, T = 0) = 0.99 × (1 − 0.05) = 0.9405
P (D = 0, T = 1) = 0.99 × 0.05 = 0.0495
P (D = 1, T = 0) = 0.01 × 0.02 = 0.0002
P (D = 1, T = 1) = 0.01 × (1 − 0.02) = 0.0098
P (D=0,T =1)
0.0495
(a) P (D = 0 | T = 1) = P (D=0,T
=1)+P (D=1,T =1) = 0.0495+0.0098 ≈ 0.8347
P (D=1,T =0)
0.0002
(b) P (D = 1 | T = 0) = P (D=0,T
=0)+P (D=1,T =0) = 0.9405+0.0002 ≈ 0.0002
Problem 3. (Derivatives)
Let a ∈ Rd be a constant
vector and b ∈ R be a constant scalar. For x ∈ Rd , let us define the function f (x) =
⊤
ln 1 + exp −b · a x (where ln is the natural logarithm). Find ∇f (x) and briefly show all major steps in your derivation.
Solution:
We have f (t) = ln(t) where t(s) = 1 + exp(−s) where s(x) = b · a⊤ x. We have f ′ (t) = t1′ , t′ (s) = − exp(−s), and
∇s(x) = b · a. Thus, applying the chain rule gives us
−b · exp −b · a⊤ x
∇f (x) =
· a.
1 + exp (−b · a⊤ x)
Page 1
Problem 4. (Optimization Problem)
Consider the optimization problem below.
min
1 2
2x
s.t.
x≥2
x∈R
Write down the Lagrangian of the problem. Then write down the dual of the problem. Eliminate the primal variable and
write down the simplified dual problem with the primal variable eliminated. Then solve the dual problem and write down
the optimal values of the primal and dual variables that you have obtained.
Solution:
If we introduce a dual variable α, the Lagrangian is
L(x, α) =
1 2
x + α(2 − x).
2
The dual problem is
1
max min x2 + α(2 − x).
α≥0 x 2
Using first-order optimality gives us x − α = 0 or in other words, x = α. Substituting this into the dual problem gives us
maxα≥0 2α − 21 α2 as the simplified dual problem.
If suppose we ignore the constraint α ≥ 0 for a moment, then applying the first order optimality condition tells us that
the function 2α − 21 α2 achieves its maximum where 2 − α = 0 or in other words at α = 2. However, this point does satisfy
α ≥ 0 which means α = 2 is the maximum for 2α − 12 α2 even with the constraint α ≥ 0 and thus, the solution of the dual
problem. Using x = α and strong duality tells us that x = 2 is the solution to the primal problem.
Problem 5. (Sub-differential)
Let a ∈ Rd be a constant vector and b ∈ R be a constant scalar. Let f : Rd → R be a function defined as f (x) =
max a⊤ x + b, 0 . Find an expression for subdifferential of f at any x ∈ Rd . Show all the main steps of your derivation
briefly in the space provided.
Solution:
We have f (x) = max{g(x), h(x)} where g(x) = a⊤ x + b and h(x) = 0. Note that both g(x) and h(x) are differentiable
functions. Thus, applying the max rule for subgradients gives us

a if a⊤ x + b > 0

0 if a⊤ x + b < 0

λ · a if a⊤ x + b = 0
where λ ∈ [0, 1] and 0 ∈ Rd is the d-dimensional all zeros vector. Note that in the second case, the subgradient is the zero
vector, not the number/ scalar zero.
Problem 6. (Chain Rule)
Let x = [1, −1]⊤ , y = [−1, 1]⊤ ∈ R2 . Define the
f : R2 → R2 as f (z) = z1 · x + z2 · y for any z = [z1 , z2 ] ∈ R2 .
function
2
2
Define another function g : R → R as g(r) = r, r where r ∈ R. Let h : R → R2 be defined as h(r) = f (g(r)). Derive a
dh
general expression for dh
dr using the chain rule giving major steps of derivation and then evaluate dr at r = 3.
Solution:
1 −1
∈ R2 so that we have f (z) = Az which gives us J f = ∇f = A (to see that the answer is
−1 1
⊤
3
indeed A and not
A , think of a hypothetical example where we have z = [z1 , z2 , z3 ] ∈ R and f (z) = z1 · x + z2 · y + z3 · p
⊤
for p = [1, 1] .
Let A = x⊤ , y⊤ =
Page 2
Next, we calculate J g = [1, 2r]⊤ (notice that this is a column vector since this is not a gradient of a real-valued function
but rather the Jacobian of a vector-valued function). Thus, we have
dh
1 −1
1
1 − 2r
= Jh = Jf · Jg =
=
.
−1 1
2r
2r − 1
dr
Note the dimensionality of J h which fits our
convention of Jacobians being of dimensionality o/p dims ×i/p dims since
−5
2
h
h : R → R . At r = 3 we have J =
.
5
Problem 7. (Squared Hinge Loss)
Let a ∈ Rd and b ∈ R be constants. For x ∈ Rd , define the function f (x) =
2
1 − b · a⊤ x + . Find ∇f (x).
Solution:
2
The function g(t) = ([1 − t]+ ) is actually differentiable. The function is clearly differentiable for t ̸= 1 with
0
t>1
′
g (t) =
2(t − 1) t < 1
′
′
However, it is clear
that limt→1+ g (t) = limt→1− g (t) = 0 which shows that g(t) is differentiable. Now, we simply have
⊤
f (x) = g b · a x and thus, on applying chain rule we get
0
b · a⊤ x ≥ 1
∇f (x) = g ′ b · a⊤ x · b · a =
⊤
2b b · a x − 1 · a b · a⊤ x < 1
Note that the 0 above is the zero vector and not the zero real number.
Problem 8. (Model Exfiltration)
Doraemon has a secret function f : R4 → R. We know that f thresholds one of the 4 coordinates of the input at a value,
i.e., for all x ∈ R4 , f (x) = xj − c where j ∈ {1, 2, 3, 4} and c ∈ R. E.g. if j = 2, c = 1.5, then for x = (1, 5, 2, 2), f (x) =
5 − 1.5 = 3.5. I want to steal Doraemon’s model and find out what value
of j, c is Doraemon using. I can send Doraemon
any number of inputs x1 , x2 , . . . ∈ R4 and Doraemon will return f x1 , f x2 , . . . ∈ R back to me. Design an algorithm
below that asks Doraemon function values on one or more 4D vectors and uses Doraemon’s responses to find the value
of j and c being used. You must give explicit descriptions of the 4D vectors you are querying Doraemon (e.g., you may
say that you wish to query only three vectors (1, −0.5, 1, −3), (8, −0.1, 1, 5) and (9, 1, 5, 6)). The fewer vectors you query
Doraemon to correctly find j, c, the better your answer.
Solution:
Step 1: Query x1 = (0, 0, 0, 0). Note that f x1 = −c so c = −f x1 .
Step 2: Query x2 = (1, 2, 3, 4). Note that f x2 = x2j − c = x2j + f x1 so x2j = f x2 − f x1 . Since we have
deliberately set x2i = i for all i ∈ {1, 2, 3, 4}, we get j = f x2 − f x1 .
Step 3: Output c = −f x1 , j = f x2 − f x1 .
Note that the only thing required in step 2 is for x2 to have different values for all its four coordinates. This helps us
figure out which coordinate is being thresholded. Setting x2i = i merely makes the algorithm more aesthetically pleasing.
Also note that the above scheme uses two queries to figure out the model completely. This is, in general, optimal, i.e.,
there is no way an algorithm can figure out both j, c using just one query. Suppose an algorithm makes a single query on
a vector x = (p, q, r, s) to see why. Upon receiving the response, such an algorithm will still be unsure whether j = 1 and
c = p − f (x) or whether j = 2 and c = q − f (x) or j = 3 and c = r − f (x) or j = 4 and c = s − f (x). The algorithm will
necessarily need to make another query to zero in on one of the above 4 cases.
Page 3
Problem 9. (Normal Random Variable)
Let X be a Normal random variable with mean µ ∈ R and variance τ 2 > 0, i.e. X ∼ N µ, τ 2 . Recall that the probability
density of X is given by
2
2
1
fX (x) = √
e−(x−µ) /2τ , −∞ < x < ∞.
2πτ
Furthermore, the random variable Y given X = x is normally distributed with mean x and variance σ 2 , i.e.,
Y | X = x ∼ N x, σ 2 .
(a) Derive the marginal distribution of Y .
(b) Use Bayes’ theorem to derive the conditional distribution of X given Y = y.
[Hint: For both tasks derive the density up to a constant factor and use this to identify the distribution.]
Solution:
Before starting calculations, it is good to mention that one can easily compute the following integral for a > 0 by creating
complete squares:
"
#!
2
Z
Z
b
b2 − ac
−(ax2 +2bx+c)
e
dx =
exp −a x +
−
dx
a
a2
R
R
2 !
2
Z
b − ac
1 x + ab
= exp
·
exp −
dx
a
2 1/2a
R
2
b − ac p
= exp
π/a
a
As a prelude to both (a) and (b) we consider the joint density function fX,Y (x, y) of X and Y

fX,Y (x, y) = fY |X (y | X = x)fX (x) =

 1 (x − µ)2
1
(y − x)2 


exp −
+
.
2πστ
τ2
σ2
 2

|
{z
}
(A)
For brevity, let us define
σ2 + τ 2
,
2σ 2 τ 2
σ2 µ + τ 2 y
b := −
,
2σ 2 τ 2
σ 2 µ2 + τ 2 y 2
c :=
.
2σ 2 τ 2
Using simple algebraic operations, we obtain that (A) = ax2 + 2bx + c.
a :=
(a) The marginal density of Y is given by
fY (y) =
Z
Z
R
fX,Y (x, y)dx =
R
fY |X (y | X = x)fX (x)dx.
Using the formula discussed at the beginning of the solution, we can compute this integral by just putting in the
values of a, b, and c:
Z
fY (y) =
Z
R
fX,Y (x, y)dx
1 −(ax2 +2bx+c)
e
dx
R 2πστ
2
1
b − ac p
=
exp
π/a
2πστ
a
2
b − ac
∝ exp
(a does not depend on y)
a
=
Page 4
Now we try to write b2 − ac /a as a complete square:
(
)
2
σ 2 + τ 2 σ 2 µ2 + τ 2 y 2
b2 − ac
1
σ2 µ + τ 2 y
=
−
2
a
a
2σ 2 τ 2
(2σ 2 τ 2 )
1
1
=− ·
· σ 2 τ 2 y 2 − 2τ 2 σ 2 µy + σ 2 τ 2 µ2
2
2
2
a (2σ τ )
1
σ2 τ 2
=− ·
· (y − µ)2 + · · ·
2
a (2σ 2 τ 2 )
1
1
· (y − µ)2 + · · ·
=−
2
2
2 (σ + τ )
Putting everything together yields
1 (y − µ)2
fY (y) ∝ exp −
2 (σ 2 + τ 2 )
meaning that Y has a Gaussian distribution with mean µ and variance σ 2 + τ 2 .
(b) The conditional density of X given Y = y is proportional to the joint density function, i.e.
fX|Y (x | Y = y) =
fX,Y (x, y)
∝ fX,Y (x, y).
fY (y)
By the discussion at the beginning of the solution, fX,Y (x, y) ∝ exp − ax2 + 2bx + c . Since c does not depend
on x (and y is considered as fixed/given), we can say:
2 !
1 x + ab
fX|Y (x | Y = y) ∝ exp −
2 1/2a
So the mean would be −b/a, and the variance would be 1/2a. Concretely:
mean = −
b
σ2 µ + τ 2 y
σ2
τ2
=
=
µ
+
y
a
σ2 + τ 2
σ2 + τ 2
σ2 + τ 2
Note that the mean is a convex combination of µ and the observation y. Also
variance =
1
σ2 τ 2
= 2
.
2a
σ + τ2
Problem 10. (Bivariate Normal Random Variable)
Let X
Normal random variable (taking on values in R2 ) with mean µ = (1, 1) and covariance matrix
be a bivariate
3 1
Σ=
. The density of X is then given by
1 2
1
1
T −1
exp − (x − µ) Σ (x − µ) .
fX (x) = p
2
(2π)2 det(Σ)
Find the conditional distribution of Y = X1 + X2 given Z = X1 − X2 = 0.
Solution:
We present two approaches for solving this exercise:
Approach 1. Note that Z = 0 implies X1 = X2 . Furthermore by the definition of Y , we have X1 = X2 = Y /2 given
Z = 0. Hence the marginal density of Y given Z = 0 is proportional to
fY,Z (y, 0)
y/2
∝ fY,Z (y, 0) ∝ fX
.
fY |Z (y | Z = 0) =
y/2
fZ (0)
Page 5
We then have
fX
y/2
y/2
1
∝ exp −
2
y
2 −1
y
2 −1
T 3 1
1 2
y
T 1
1
2
2 −1
= exp −
y
−
1
−1
2
5
2
1 (y − 2)2
.
= exp −
20
2
3
!
−
1
2
y
2 −1
y
!
−1
−
1
2
y
3
2 −1
−1 y
Clearly, the conditional distribution of Y given Z = 0 is hence Normal with mean 2 and variance 20
3 .
Approach 2. We define the random variable R as
Y
1 1
R=
=
X.
Z
1 −1
|
{z
}
=A
By linearity of expectation, the mean µR of R is
E[R] = AE[X] = Aµ =
2
0
.
The covariance matrix ΣR of R is given by
ΣR = E (R − E[R])(R − E[R])T = E A(X − E[X])(X − E[X])T AT
= AE (X − E[X])(X − E[X])T AT = AΣAT
1 1
3 1
1 1
=
1 −1
1 2
1 −1
4 3
1 1
=
2 −1
1 −1
7 1
=
1 3
Since X is multivariate Gaussian and R is an affine transformation of X, R is a bivariate Normal random variable with
mean µR and covariance matrix ΣR .1
The conditional density of Y given Z = 0 is then given by
fY,Z (y, 0)
∝ fY,Z (y, 0)
fZ (0)
T −1 !
1
y−2
7 1
y−2
∝ exp −
0
1 3
0
2
T
!
1
1
y−2
3 −1
y−2
= exp −
0
−1 7
0
2
20
1 (y − 2)2
= exp −
.
20
2
3
fY |Z (y | Z = 0) =
Clearly, the conditional distribution of Y given Z = 0 is hence Normal with mean 2 and variance 20
3 .
1 This result can be easily derived from the characteristic function of the multivariate Normal distribution. R is bivariate Normal if and only
if for any t ∈ R2 .
[ T ]
T
T
E eit R = eit µR −t ΣR t/2 .
This holds since the corresponding property holds for X with s = tT A, i.e.
[ T ]
[ T
]
[ T ]
T
T
T
T
T
T
T
E eit R = E eit AX = E eis X = eis µ−s Σs/2 = eit Aµ−t AΣA t/2 = eit µR −t ΣR t/2 .
Page 6
Problem 11. (Linear Regression and Ridge Regression)
Let D = {(x1 , y1 ) , (x2 , y2 ) , . . . (xn , yn )} where xi ∈ Rd and yi ∈ R be the training data that you are given. As you have
to predict a continuous variable, one of the simplest possible models is linear regression, i.e., to predict y as wT x for some
parameter vector w ∈ Rd 2 . We thus suggest minimizing the following loss
argminR̂(w) = argmin
w
w
n
X
yi − w T x i
2
(1)
.
i=1
Let us introduce the n × d matrix X ∈ Rn×d with the xi as rows, and the vector y ∈ Rn consisting of the scalars yi . Then,
Eqn. (1) can be equivalently re-written as
argmin∥Xw − y∥2 .
w
∗
We refer to any w that attains the above minimum as a solution to the problem.
(a) Show that if XT X is invertible, then there is a unique w∗ that can be computed as w∗ = XT X
−1
XT y.
(b) Show for n < d that Eqn. (1) does not admit a unique solution. Intuitively explain why this is the case.
(c) Consider the case n ≥ d. Under what assumptions on X does Eqn. (1) admit a unique solution w∗ ? Give an
example with n = 3 and d = 2 where these assumptions do not hold.
The ridge regression optimization problem with parameter λ > 0 is given by
argminR̂Ridge (w) = argmin
w
" n
X
w
yi − w T x i
2
#
+ λwT w .
(2)
i=1
(d) Show that R̂Ridge (w) is convex with regards to w. You can use the fact that a twice differentiable function is convex
if and only if it’s Hessian H ∈ Rd×d satisfies wT Hw ≥ 0 for all w ∈ Rd (is positive semi-definite).
−1 T
∗
(e) Derive the closed form solution wRidge
= XT X + λId
X y to Eqn. (2) where Id denotes the identity matrix of
size d × d.
∗
(f) Show that Eqn. (2) admits the unique solution wRidge
for any matrix X. Show that this even holds for the cases
in (b) and (c) where Eqn. (1) does not admit a unique solution w∗ .
∗
(g) What is the role of the term λwT w in R̂Ridge (w)? What happens to wRidge
as λ → 0 and λ → ∞?
Solution:
(a) Note that
R̂(w) = ∥Xw − y∥2 = (Xw − y)T (Xw − y) = wT XT Xw − 2wT XT y + yT y.
The gradient of this function is equal to (see Lemma 1 in Supplementary Material)
∇R̂(w) = 2XT Xw − 2XT y.
Because R̂(w) is convex (formally proven in (d)), its optima are exactly those points that have a zero gradient,
i.e. those w∗ that satisfy XT Xw∗ = XT y. Under the given assumption, the unique minimizer is indeed equal to
−1 T
w ∗ = XT X
X y.
(b) Consider the singular value decomposition X = UΣVT where U is a unitary n × n matrix, V is a unitary d × d
matrix and Σ is a diagonal n × d matrix with the singular values of X on the diagonal. We then have
argminR̂(w) = argmin wT VΣ2 VT w − 2yT UΣVT w
w
2 Without loss of generality, we assume that both x
w
i and yi are centered, i.e., they have zero empirical means.
otherwise necessary bias term b.
Page 7
Hence we can neglect the
Since V is unitary (and hence it is a bijection), we may rotate w using V to z = VT w and formulate the optimization
problem in terms of z, i.e.
d
X
2 2
argmin zT Σ2 z − 2yT UΣz = argmin
zi σ i − 2 U T y i zi σ i
z
z
i=1
where σi is the i entry in the diagonal of Σ. Note that this problem decomposes into d independent optimization
problems of the form
zi = argmin z 2 σi2 − 2 UT y i zσi
z
for i = 1, 2, . . . , d. Since each problem is quadratic with a positive coefficient and thus convex, we may obtain the
solution by finding the root of the first derivative. For i = 1, 2, . . . d we require that zi satisfies
zi σi2 − Ut y i σi = 0.
For all i = 1, 2, . . . d such that σi ̸= 0, the solution zi is thus given by
zi =
(Ut y)i
.
σi
For the case n < d, however, X has at most rank n as it is a n × d matrix and hence at most n of its singular values
are nonzero. This means that there is at least one index j such that σj = 0 and hence any zj ∈ R is a solution to the
optimization problem. As a result the set of optimal solutions for z is a linear subspace of at least one dimension.
By rotating this subspace back using V, i.e., w = Vz, it is evident that the optimal solution to the optimization
problem in terms of w is also a linear subspace of at least one dimension and that thus no unique solution exists.
Furthermore, since X has at most rank n, XT X is not of full rank (see Lemma 2 of Supplementary Section). As a
−1
result XT X
does not exist and w∗ is ill-defined.
The intuition behind these results is that the “linear system” Xw ≈ y is under-determined as there are fewer data
points than parameters that we want to estimate.
(c) We showed in (b) that the optimization problem admits a unique solution only if all the singular values of X are
nonzero. For n ≥ d, this is the case if and only if X is of full rank, i.e., all the columns of X are linearly independent.
As an example of a matrix not satisfying these assumptions, any matrix with linearly dependent suffices, e.g.


1 −2
0 .
Xdegenerate =  0
−2 4
(d) Because convex functions are closed under addition, we will show that each term in the objective is convex, from
2
which the claim will follow. Each data term yi − wT xi has a Hessian xi xTi , which is positive semi-definite because
2
for any w ∈ Rd we have wT xi xTi w = xTi wi ≥ 0 (note that xTi w = wT xi are scalars). The regularizer λwT w
has the identity matrix λId as a Hessian, which is also positive semi-definite because for any w ∈ Rd we have
wT λId w = λ∥w∥2 ≥ 0, and this completes the proof.
(e) The gradient of R̂Ridge (w) with respect to w is given by
∇R̂Ridge (w) = 2XT (Xw − y) + 2λw.
∗
Similar to (a), because R̂Ridge (w) is convex, we only have to find a point wRidge
such that
∗
∇R̂Ridge wRidge
This is equivalent to
∗
∗
= 2XT XwRidge
− y + 2λwRidge
= 0.
∗
XT X + λId wRidge
= XT y
which implies the required result
∗
wRidge
= XT X + λId
Page 8
−1
XT y
Pn
2
(f) Note that XT X is a positive semi-definite matrix3 since ∀w ∈ Rd : wT XT Xw = i=1 [(Xw)i ] ≥ 0, which implies
that it has non-negative eigenvalues. But then, XT X + λId has eigenvalues bounded from below by λ > 0, which
means that it is invertible and thus the optimum is uniquely defined.
Note. Since XT X is symmetric, all of its eigenvalues are real, and it is clear that µ is an eigenvalue of XT X if and
only if µ + λ is an eigenvalue of XT X + λI. Also note that if a linear function is injective, its kernel is {0}, meaning
it does not have a zero eigenvalue. The converse is also true.
(g) The term λwT w “biases” the solution towards the origin, i.e., there is a quadratic penalty for solutions w that is far
from the origin. The parameter λ determines the extend of this effect: As λ → 0, R̂Ridge (w) converges to R̂(w). As
∗
a result the optimal solution wRidge
approaches the solution of (1). As λ → ∞, only the quadratic penalty wT w is
∗
relevant and wRidge hence approaches the null vector (0, 0, . . . , 0).
One can also pose this interesting question: Assume n < d (as the situation discussed in (b)). Then w* for linear
regression is not unique. Denote by wλ∗ the unique solution to the Ridge regression problem for λ > 0. Does the
limit limλ→0 wλ∗ exist? If yes, because of the completeness of Rd , the limit point should fall inside the space of
solutions to linear regression problems. What is this solution?
Supplementary Material:
Lemma 1. Let A ∈ Rn×n
be a real matrix and define f (x) = x⊤ Ax to be the quadratic form defined via A. Then we
have ∇f (x) = A + A⊤ x. Moreover, if A is symmetric, then ∇f (x) = 2Ax.
Proof. Let us compute the derivative of f at point x. We know
f (x + h) − f (x) = h⊤ Ah + h⊤ Ax + x⊤ Ah = h⊤ A + x⊤ A⊤ + x⊤ A h.
By taking the limit ∥h∥ → 0, the linear operator x⊤ A⊤ + x⊤ A would be the derivative. So the gradient would be
∇f (x) = A + A⊤ x
Lemma 2. Let A ∈ Rm×n and B ∈ Rn×k be two matrices. Then
rank(AB) ≤ rank(A)
Proof. If we denote the columns of B by b1 , . . . , bk , then we can write AB = [Ab1 , . . . , Abk ]. Now Abi is a linear
combination of columns of A, so the columns of AB are all linear combinations of columns of A. It follows that the
subspace spanned by the columns of AB is included in the span of columns of A. Hence we will have the desired
inequality.
Problem 12. (Multi-output Regression with Reduced Number of Parameters)
Consider the multi-output regression in which each output y n ∈ RM in a real-valued vector rather than a scalar. Assuming
a linear model, we can model the outputs as Y = XW, where X is the N × D feature matrix and Y is N × M response
matrix with row n being y ⊤
n (note that each column of Y denotes one of the M responses), and W is the D × M weight
matrix, with its M columns containing the M weight vectors w1 , w2 , . . . , wM . Let’s define a squared error loss function
2
PN PM
⊤
n=1
m=1 ynm − w m xn , which is just the usual squared error but
summed ⊤over all the M outputs. Firstly, verify
that this can also be written in a more compact notation as TRACE (Y − XW) (Y − XW) .
Note: Here we will assume that the weight matrix W can be written as a product of two matrices, i.e., W = BS where
B is D × K and S is K × M (assume K < min{D, M } ). Note that there is a benefit of modeling W this way since now
we need to learn only K × (D + M ) parameters as opposed to D × M parameters. If K is small, this can significantly
reduce the number of parameters (in fact, reducing the effective number of parameters to be learned is another way of
regularizing a machine learning model). Note (you can verify) that each wm can be written as a linear combination of K
columns of B in this formulation.
3 An equivalent notion for a matrix A being positive semi-definite is that for all x ∈ Rn we have x⊤ Ax ≥ 0.
Page 9
With the proposed representation of W, the new objective will be TRACE (Y − XBS)⊤ (Y − XBS) and you need to
learn both B and S. While it is certainly possible to learn both (one way to do it is using a procedure called “alternating
optimization”), for now, let’s keep it simple and assume that the matrix B is known and only S needs to be estimated.
So your problem is reduced to
Ŝ = arg min TRACE (Y − XBS)⊤ (Y − XBS)
S
Derive the expression for Ŝ. Briefly explain how the form of the solution is identical to the solution of standard multioutput regression but using a transformed version of the inputs X.
Solution:
Consider the multi-output regression in which each output yn ∈ Rm is a real-valued vector rather than a scalar. Assuming
a linear model, we can model the outputs as Y = XW , where X is the usual input data matrix, Y is the response matrix
of dimensions N × M, W is the weight matrix to be learned of dimensions D × M . The least squares objective in this
setting can be stated as
N X
M
X
2
T
yn m − w m
xn
L(W ) =
n=1 m=1
∴ L(W ) =
∴ L(W ) =
M
X
N
X
m=1
n=1
M
X
N
X
m=1
n=1
2
T
yn m − w m
xn
!
!
T
(ym − Xwm ) (ym − Xwm )
∴ L(W ) = T RACE (Y − XW )T (Y − XW )
Here, we will assume that the weight matrix W can be written as a product of two matrices, i.e., W = BS where B is
D × K and S is K × M (assume K < min{D, M } ). We further assume that B is known and we would like to estimate
S. Thus, estimating S can be stated as an optimization problem as follows:
Ŝ = arg min TRACE (Y − XW )T (Y − XW ) .
S
The loss function can be stated as a function of S as follows:
L(S) = tr (Y − XBS)T (Y − XBS) = tr Y T Y − tr (XBS)T Y − tr Y T XBS + tr (XBS)T XBS
Now, we need to compute ∂L(S)
and equate it to 0 to find the optimal solution for S.
∂S
∂ tr Y T Y
∂ tr (XBS)T Y
∂ tr Y T XB S
∂ tr (XBS)T XBS
∂L(S)
=
−
−
+
∂S
∂S
∂S ∂S ∂S
T
∂ tr S B T X T Y
∂ tr Y T XB S
∂ tr S T B T X T XB S
∂L(S)
∴
=0−
−
+
∂S
∂S
∂S
∂S
T
T
∂L(S)
T
T
T
T
T
T
T
∴
= −B X Y − Y XB + B X XB S + B X XB S
∂S
∴ 0 = 2B T X T Y + 2 B T X T XB S
∴ B T X T XB S = B T X T Y
−1 T T
∴ S = B T X T XB
B X Y
−1
−1
T
T
∴ S = (XB) XB
(XB) Y = X T X
XY
where X̄ = XB i.e., the input matrix transformed by the product with B. Thus, the form of the solution is identical to
the solution of standard multi-output regression but uses a transformed version of the inputs X.
Problem 13. (Feature-Specific ℓ2 Regularization)
2
PN
The usual ℓ2 regularized least squares regression objective can be written as L(w) = n=1 yn − w⊤ x + λ2 w⊤ w. In
this case, the extent of regularization is the same on all the features (and is controlled by λ ). Propose an alternative that
Page 10
still uses ℓ2 regularization on w, but the extent of regularization is different for each entry wd . Write down the objective
function and derive the closed-form expression for the weight vector w.
Solution:
The usual l2 regularized least squares regression objective can be written as
N
X
L(w) =
yn − w T x n
2
+
n=1
λ T
w w
2
Now, the first term can be rewritten in terms of the input data matrix and output label vector. Also, considering different
hyperparameters for each feature, say λi for feature i, the objective function can be re-stated as
L(w) = (y − Xw)T (y − Xw) +
N
X
λi
2
i=1
wi2
We need to compute ∂L(w)
∂w . We already know the derivative of the first term from the analysis of least squares regression.
For the second term, w.r.t the feature i, we have
P
N λ1 2
∂
i=1 2 wi
= 2 ∗ (λi /2) wi = λi wi .
∂wi
Thus, the gradient w.r.t w can be computed:
P
N λ1 2
∂
i=1 2 wi
= λ 1 w 1 λ2 w 2 λ 3 w 3
∂w

P
λ1 0 0 . . .
N λ1 2

∂
w
i
i=1 2
 0 λ2 0 . . .
∴
= .
.. .. . .
∂w
 ..
.
. .
0
0
0
...
where diag(v) is a diagonal matrix for vector v and λ̄ =
...
0
0
..
.

λ N wN


 w1

T
w2
w3
...
wN
T
= diag(λ̄)w
λN
λ1
. . . λN
0 = X T y − X T X + diag(λ̄) w
∴ X T X + diag(λ̄) w = X T y
−1 T
∴ w = X T X + diag(λ̄)
X y
Thus, we get the closed form solution as ŵ = X T X + diag(λ̄)
λ2
−1
λ3
. Now, equating ∂L(w)
∂w = 0, we get
X T y.
Problem 14. (Linear Regression meets Nearest Neighbors)
(a) Show that for the un-regularized linear regression model, where the solution ŵ = X⊤ X
a test input x∗ can be written as a weighted sum of all the training responses, i.e.,
f (x∗ ) =
N
X
−1
X⊤ y, the prediction at
w n yn .
n=1
Give the expression for the weights wn ’s in this case and briefly discuss ( < 50 words) in what way these weights
are different from the weights in a weighted version of K nearest neighbors where each wn typically is the inverse
distance of x∗ from the training input xn . Note: You do not need to give a very detailed expression for wn (if it
makes algebra messy) but must give a precise meaning as to what wn depends on and how it differs from the weights
in the weighted K nearest neighbors.
Page 11
(b) An important notion for a classifier is that of consistency. A classification algorithm is said to be consistent if,
whenever it has access to infinite amounts of training data, its error rate approaches the optimal error rate (a.k.a.
Bayes optimal). Consider the noise-free setting (i.e., every training input is labeled correctly). Here, the Bayes
optimal error rate is zero. Is the one-nearest-neighbor algorithm consistent in this setting? Briefly justify your
answer.
Solution:
−1 T
X y. For a test example x∗ , we can predict
(a) The solution to the linear regression problem is given by ŵ = X T X
the output as follows:
f (x∗ ) = ŵT x∗
−1 T T
∴ f (x∗ ) = X T X
X y x∗
T h T −1 iT
∴ f (x∗ ) = y T X T
X X
x∗
h
i
−1
T
−1
∴ f (x∗ ) = y T X X T X
x∗ = y T X X T X
x∗
N
D
E X
−1
∴ f (x∗ ) = X X T X
x∗ , y =
w n yn
n=1
−1
where wn is the n component of the term X X X
x∗ . Let A := X X T X
. Let us denote this weight
vector corresponding to linear regression as wLR and that of weighted kNN as wkN N . Our goal is to find/interpret
some sort of a relation between these two. As given, we know:
th
T
−1
wnkN N =
1
.
d (xn , x∗ )
Now, we will try to interpret what wnLR means. Note that wk N N is a sort of similarity between the input data
−1
matrix X and x∗ . Similarly, observe that wLR = X X T X
x∗ and thus we have
wnLR = xTn X T X
−1
x∗
Thus, wnLR is also a sort of similarity between xn and x∗ which is weighted by the matrix X T X
Mahalanobois matrix.
−1
much like the
(b) Claim: 1-nearest neighbor (1-NN) classification is a consistent classification method.
Justification: Suppose we consider the 1-nearest neighbor (1-NN) classification trained on infinite training examples.
Note that k − NN has an error rate not worse than twice that of Bayes optimal classifier.
Ek−N N ≤ 2EBO → 0 ⇒ Ek−N N → 0
Note that the Bayes optimal error rate goes to 0 in a noise-free setting. Thus, the error rate for 1-NN also goes to
zero. Another way to look at it is that since 1-NN considers distance from just one nearest sample point at test time,
it is prone to overfitting and gives almost zero training error. Now, for a given test example, since we assume that
the training set is infinite, it essentially implies that the test example will have some training example in very close
proximity, and thus, the error rate would be very low, close to 0. Thus, 1-NN is a consistent classification method.
Problem 15. (Feature Selection)
We covered ridge (l2 − regularised) and l1 -regularised (lasso) regression in class. A hybrid version called elastic net exists
which uses both l1 and l2 regularisation terms:
JEL = ∥y − Xw∥2 + λ1 ∥w∥1 + λ2 ∥w∥2
Defining
J2 = ∥ỹ − X̃w∥2 + cλ1 ∥w∥1
Page 12
where c = (1 + λ2 )
−1/2
and
X̃ = c
√X
λ 2 Id
,
ỹ =
y
0d×1
.
Show that
arg min JEL (w) = c (argminw J2 (w))
w
This implies that an elastic net problem can be solved as a lasso problem using modified data.
Solution:
JEL (w) = |y − Xw|2 + λ2 |w|2 + λ1 |w|1
cJEL (w) = c|y − Xw|2 + cλ2 |w|2 + cλ1 |w|1
= cy T y − 2cy T Xw + cwT X T Xw + cλ2 |w|2 + cλ1 |w|1
= cy T y − 2cy T Xw + c wT X T Xw + λ2 |w|2 + cλ1 |w|1
= cỹ T ỹ − 2ỹ T X̃w + c wT X T Xw + λ2 |w|2 + cλ1 |w|1
1 T T
= cỹ T ỹ − 2ỹ T X̃w +
w X̃ X̃w + cλ1 |w|1
c
Let w̃ = c−1 w
JEL (w̃) = cỹ T ỹ − 2cỹ T X̃ w̃ + c w̃T X̃ T X̃ w̃ + c2 λ1 |w̃|1
JEL (w̃) = c|Ỹ − X̃w|2 + c2 λ1 |w̃|1
Problem 16. (On Statistical Distances)
In some situations in statistics and machine learning, the objective that we are going to optimize is some distribution
(e.g., in the E-step of the EM algorithm). This motivates us to understand a bit more about the space of probability
distributions. For simplicity, let P be the set of all probability distributions over the set {1, . . . , n}, i.e.,
n
o
X
P = (p1 , . . . , pn ) |
pi = 1, pi ≥ 0 .
Usually, P is called the probability simplex, or simply the n-simplex. One can equip P with a metric, inducing a geometry
on P. Recall that a metric is a function d : P × P → R+ satisfying the following criteria:
• (Non-negativity) d(p, q) ≥ 0 for all p, q ∈ P and equality holds iff p = q,
• (Symmetry) d(p, q) = d(q, p),
• (Triangle inequality) d(p, q) + d(q, r) ≥ d(p, r).
A metric on the probability simplex is also called a statistical distance. Here we mention a few distances and some of
their properties:
(a) Total Variation Distance. For p, q ∈ P, we define their TV distance as
n
1
1X
DTV (p, q) := ∥p − q∥1 =
|pi − qi | .
2
2 i=1
Prove that TV distance is indeed a metric and equals to the largest possible difference between the probabilities
that the two probability distributions p and q can assign to the same event, i.e.
DTV (p, q) =
max
E⊆{1,...,n}
|p(E) − q(E)|.
(b) Kullback-Leibler Divergence. For p, q ∈ P, we define their KL divergence as
DKL (p∥q) = −
n
X
i=1
Page 13
pi log
qi
.
pi
(i) Prove that KL divergence satisfies the first property of a metric: it is non-negative, and it is zero if and only if
the distributions are equal.
(ii) Give an example that DKL (p∥q) ̸= DKL (q∥p).
(iii) Give a counter-example for the triangle inequality for KL divergence.
(iv) Prove the Pinsker’s Inequality:
r
DTV (p, q) ≤
1
DKL (p∥q).
2
(v) Although KL divergence fails to be a metric on P, it satisfies some convergence properties. As an example,
prove the following theorem: Let p(1) , p(2) , . . . be a sequence of probability distributions in P, such that
lim DKL p(n) ∥q = 0,
n→∞
i.e. the sequence is “converging” to q with respect to KL divergence. Prove that this sequence is actually
converging to q in Euclidean sense, i.e.
lim p(n) − q = 0.
n→∞
2
(vi) Let X and Y be two random variables with distributions pX and pY and joint distribution pX,Y . If X and Y
were independent, the we had pX,Y = pX pY . Otherwise, if one tries to give a “measure of independence” of X
and Y , one idea is to consider
DKL (pX,Y ∥pX pY )
This value is called the mutual information between X and Y , denoted by I(X, Y ). Prove that
I(X, Y ) = H(X) − H(X | Y ),
where H(X) is the entropy4 of X and H(X | Y ) is the conditional entropy of X given Y . From the Bayesian
point of view, mutual information shows how much information knowledge about Y reveals about X.
Solution:
(a) We first prove that Total Variation is a distance function. Let p, q, r ∈ P be three probability distributions over the
set {1, . . . , n}. Non-negativity follows by definition,
DTV (p, q) =
1
∥p − q∥1 ≥ 0.
2
Symmetry follows from ∥p − q∥1 = ∥q − p∥1 . Also, Triangle inequality follows from the triangle inequality for ℓ1
norm.
These three properties show that the Total Variation distance is indeed a distance function.
Now we prove the second argument. Let E = {i : pi ≥ qi } be the event that contains the elements which p gives higher
probability than q. We claim that E attains the maximum value of |p(F ) − q(F )| among all events F ⊆ {1, . . . , n}
By writing F = (F ∩ E) ∪ (F ∩ E c ) we observe that
p(F ) − q(F ) = p(F ∩ E) − q(F ∩ E) + p (F ∩ E c ) − q (F ∩ E c ) ≤ p(E) − q(E).
|
{z
}
(3)
≤0
This is true since for all elements i ∈ E c we have pi < qi , which makes p (F ∩ E c ) − q (F ∩ E c ) ≤ 0, and adding
elements in E to F ∩ E will not decrease the difference in probability, meaning p(F ∩ E) − q(F ∩ E) ≤ p(E) − q(E).
With the same argument, but for E c this time, we arrive at
q(F ) − p(F ) ≤ q (E c ) − p (E c ) .
∑
(4)
4 “Entropy of a random variable X is defined as H(X) := E [− log X] = −
X
x pX (x) log pX (x), and is a measure of “uncertainty” of X.
For example if X has the uniform distribution, it has the highest entropy. If the base of log is 2, entropy is measured with the unit “bits”,
suggesting the idea that one needs H(X) bits to encode the outcome of X with zeros and ones. Convince yourself that this definition makes
sense.
Page 14
Since p(E) − q(E) = q (E c ) − p (E c ), the upper bounds for Eqn. (3) and qn. (??) become the same, and we get
p(E) − q(E) = max |p(F ) − q(F )|.
F
Also, by definition of E, we can write
X
X
∥p − q∥1 =
(pi − qi ) +
(qj − pj ) = p(E) − q(E) + q (E c ) − p (E c ) = 2(p(E) − q(E)),
i∈E
j ∈E
/
where the last equality is because p(E) − q(E) = q (E c ) − p (E c ).
(b) The solutions for part (b) are given below:
(i) We prove that for p, q ∈ P, we have DKL (p∥q) ≥ 0. Note that the positivity of the KL Divergence is regardless
of the basis of the logarithm since for all a > 1, we have loga (x) = ln(x)/ ln(a). Hence we prove that for all
p, q ∈ P we have
n
X
qi
≥ 0.
−
pi ln
pi
i=1
A useful inequality about logarithms is ln(x) ≤ x − 1 for all x > 0, with equality iff x = 1. Using this inequality,
we have
n
n
X
X
qi
qi
−
pi ln
≥−
pi
−1
pi
pi
i=1
i=1
=−
n
X
(qi − pi ) = 0.
i=1
The equality only happens if pi = qi for all i, or equivalently when p = q.
(ii) Take p = (0.1, 0.9) and q = (0.5, 0.5). Then we have
DKL (p∥q) = 0.1 × log 0.2 + 0.9 × log 1.8 ≈ 0.531
while
DKL (q∥p) = 0.5 × log 5 + 0.5 × log
5
≈ 0.737
9
(iii) To give a counterexample for Triangle inequality, we should provide three distributions p, q, r ∈ P, such that
DKL (p∥q) + DKL (q∥r) < DKL (p∥r).
By moving the first term to the right-hand side and expanding the definition of KL divergence, we need to
have
X
X
X qi
pi
qi
pi
qi log <
pi log − log
=
pi log
r
r
q
r
i
i
i
i
i
i
i
This suggests that we take q and r, two arbitrary distributions, and find a p that makes this inequality possible.
Take q = (0.5, 0.5) and r = (0.1, 0.9). Then log rq11 ≈ 2.322 and log rq22 ≈ −0.847. Now take p = (1, 0). In this
way we have
X
qi
qi log ≈ 0.737
r
i
i
but
X
pi log
i
qi
q1
= log
≈ 2.322
ri
r1
(iv) We first prove this inequality for the case that p and q are two probability distributions over a set of two
elements, i.e., p = (p, 1 − p) and q = (q, 1 − q). In this case we have ∥p − q∥1 = 2|p − q|. We shall prove
2(p − q)2 =
?
p
1
1−p
∥p − q∥21 ≤ DKL (p∥q) = p log + (1 − p) log
.
2
q
1−q
To prove this inequality, we fix p and look at
f (q) = p log
1−p
p
+ (1 − p) log
− 2(p − q)2
q
1−q
Page 15
So it suffices to prove f (q) is always nonnegative. First, observe that at q = p, we have f (p) = 0. Taking the
derivative w.r.t. q we have
p 1−p
1
′
f (q) = − +
+ 4(p − q) = (q − p)
−4
q
1−q
q(1 − q)
Since for 0 < q < 1 we know q(1 − q) ≤ 14 , it follows immediately that the sign of the derivative is the same
as q − p. This concludes that the point p is the minimum of f , with a value of 0. For the more general case,
we try to reduce the problem to the case we already solved. Before doing this, let us state (without proof; the
reader can carry out the proof) a useful and important lemma:
Lemma 3. (Log-Sum inequality) Let a1 , . . . , an and b1 , . . . , bn be nonnegative numbers. Then
n
X
where a =
P
ai and b =
P
ai log
i=1
ai
a
≥ a log
bi
b
bi .
NowP
let p and q be two
P probability distributions over a set of n elements. Let A = {i : pi ≥ qi }. Define
p̃ = i∈A pi and q̃ = i∈A qi and take the distributions p̃ = (p̃, 1 − p̃) and q̃ = (q̃, 1 − q̃). We show that the
Pinsker’s inequality for p and q is reduced to the Pinsker inequality for p̃ and q̃, which we have already proved.
To show this reduction, we first show that DTV (p, q) = DTV (p̃, q̃). This follows from
1X
DTV (p, q) =
|pi − qi |
2 i=1
1X
1X
=
(pi − qi ) +
(qi − pi )
2
2
n
i∈A
/
i∈A
= p̃ − q̃ = DTV (p̃, q̃)
Next, we show that DKL (p∥q) ≥ DKL (p̃∥q̃)
n
X
pi
qi
i=1
X
pi X
pi
=
pi log +
pi log
qi
qi
DKL (p∥q) =
pi log
i∈A
/
i∈A
p̃
1 − p̃
≥ p̃ log + (1 − p̃) log
= DKL (p̃∥q̃)
q̃
1 − q̃
where the inequality follows from the Log-Sum inequality. Putting it all together we have
DKL (p∥q) ≥ DKL (p̃∥q̃) ≥ 2DTV (p̃, q̃)2 = 2DTV (p, q)2
(v) By the Pinsker’s inequality, we know that if DKL p(n) ∥q → 0, then DTV p(n) , q → 0, meaning that
p(n) − q 1 → 0. But since in finite dimensions, all norms are equivalent, meaning that there is a constant C
such that ∥p − q∥2 ≤ C∥p − q∥1 for all p, q, then this implies that p(n) − q 2 → 0.
(vi) By definition we have
X
pX,Y (i, j)
I(X, Y ) = DKL (pX,Y ∥pX pY ) =
pX,Y (i, j) log
pX (i)pY (j)
i,j
Using the chain rule for probabilities, we have pX,Y (i, j) = pX|Y (i | j)pY (j). Hence
X
pX|Y (i | j)pY (j)
pX (i)pY (j)
i,j
X
X
=−
pX,Y (i, j) log pX (i) +
pX,Y (i, j) log pX|Y (i | j).
I(X, Y ) =
pX,Y (i, j) log
i,j
i,j
In the first sum, the summation on j changes pX,Y (i, j) to pX (i), and the sum becomes H(X). The second
sum is, by definition, minus the conditional entropy. So we have
I(X, Y ) = H(X) − H(X | Y )
Page 16
Problem 17. (Entropy for Classification Tree)
(a) Let X be a discrete random variable with P (X = xi ) = pi for i ∈ 1, 2, . . . , n. The entropy H[X] of the random
variable X is a measure of its uncertainty. It is defined as:
H[X] = −
n
X
pi log pi ,
i=1
where log denotes the natural logarithm. Show that the entropy H[X] is maximized when pi = n1 for all i. You
should
Pdo this by computing the gradient with respect to pi and using Lagrange multipliers to enforce the constraint
that i pi = 1 (In the course, we use similar calculations in classification trees).
(b) The joint entropy of n discrete random variables (X1 , X2 , . . . , Xn ) is defined as:
XX
X
H (X1 , X2 , . . . , Xn ) = −
...
P (x1 , x2 , . . . , xn ) log P (x1 , x2 , . . . , xn )
x1
x2
xn
where the sums range over all possible instantiations of (X1 , X2 , . . . , Xn ). Show that if the variables Xi are independent, then their joint entropy is the sum of their individual entropies: namely,
P (x1 , x2 , . . . , xn ) =
n
Y
implies
P (xi )
H (X1 , X2 , . . . , Xn ) =
i=1
n
X
H (Xi ) .
i=1
Solution:
(a) We want to solve
max
p1 ,p2 ,...,pn
H[X] = −
n
X
pi log pi , s.t.
i=1
n
X
pi = 1
i=1
Firstly, we write down the Lagrange function as
L (p1 , p2 . . . , pn , λ) = −
n
X
pi log pi − λ ·
i=1
Secondly, we let
n
X
!
pi − 1 .
i=1
∂L (p1 , p2 , . . . , pn , λ)
1
= − log pi + pi ·
− λ · 1 = −1 − log pi − λ = 0
∂pi
pi
pi = e−1−λ .
Then, we rewrite the Lagrange function as
L(λ) = −n · e−1−λ · (1 − λ) − λ · n · e−1−λ − 1 = λ − n · e−1−λ .
When optimal occurs, we have
∂L(λ)
= 1 − n · (−1) · e−1−λ = 0
∂λ
1
⇒ e−1−λ =
n
1
⇒ pi = .
n
Page 17
(b)
H (X1 , X2 , . . . , Xn ) = −
XX
x1
=−
x2
xn
XX
X
x1
=−
...
X
...
"
P (x1 , x2 , . . . , xn )
xn
!#
P (xi )
i=1
" n
X X
x2
n
Y
P (x1 , x2 , . . . , xn ) log
xn
XX
x1
...
x2
"
n
X
#
log P (xi )
i=1
#
[P (x1 , x2 , . . . , xn ) log P (xi )]
i=1
We can reorder the above summation formula and sum them through log P (xi ). Note that xi was determined before
the square brackets below.


n X
n
X
X
X
log P (xi ) ·
H (X1 , X2 , . . . , Xn ) = −
P (x1 , x2 , . . . , xn )
i=1 xi
j=1,j̸=i xj
From above, we can get H (Xi ) by accumulating all other xj , j ∈ {1, . . . , n}, j ̸= i, that is
H (X1 , X2 , . . . , Xn ) = −
n X
X
[log P (xi ) · P (xi )]
i=1 xi
=
=
n
X
i=1
n
X
"
−
X
#
P (xi ) log P (xi )
xi
H (Xi ) .
i=1
Problem 18. (Misclassification Rate Vs. Information Gain)
Consider a binary classification data set consisting of 400 data points from class 0 and 400 data points from class 1.
Suppose that a decision tree model A splits these into (300, 100) at the first leaf node and (100, 300) at the second leaf
node. (Here, (n, m) denotes that n points are assigned to class 0 and m points are assigned to class 1.) Similarly, suppose
that a second decision tree model B splits them into (200, 400) and (200, 0). (See the figure below.)
Figure 1: Two Decision Tree Models
(a) Compute the training data misclassification rate (i.e., what fraction of training examples will be misclassified) for
the two trees: are they equal or not?
(b) Evaluate the information gain for the two trees and use these to compare the trees.
Page 18
(c) Do you get different answers for (a) and (b)? Does this make sense?
Solution:
We will introduce the notations and state the definitions that will be needed for the problem.
- TA : Tree A, TB : Tree B
- IG(T ) : Information gain of tree T
- M R(T ) : Misclassification rate of tree T
- D : original training dataset, Dt : training subset at node t
- N : total number of training samples, Nt : number of training samples at node t
- H(S) : Entropy of S ⊆ D
- Let p denote the parent node, l denote the left node and r the right node
Nc
- pc (t) := Ntt : probability of a sample belonging to class c at node t
The formal definitions of misclassification rate and information gain are given by:
X
Nt
M R(T ) :=
1 − max {pc (t)}
N
c∈{0,1}
t∈ leaves (T )
X
Nr
Nl
H (Dl ) +
H (Dr ) where H (Dt ) = −
IG(T ) := H (Dp ) −
N
N
pc (t) log2 (pc (t))
c∈ classes
(a) Misclassification rate: For TA , substituting N = 800, Nl = Nr = 400, and observing
p0 (l) = 300/400 = 3/4, p1 (l) = 100/400 = 1/4
p0 (r) = 100/400 = 1/4, p1 (r) = 300/400 = 3/4
Now, we substitute these values in the above formulae of MR(T) and get
MR (TA ) = (1/2)[1/4 + 1/4] = 1/4 = 0.25
Similarly, working out for TB yields
M R (TB ) = (3/4)[1/3 + 0] = 1/4 = 0.25
M R (TA ) = 0.25,
M R (TB ) = 0.25
Thus, we observe that the misclassification rate for both trees is the same. Thus, the two trees are incomparable in
terms of the misclassification rate metric.
(b) Information gain: First, we compute H (Dp ) , H (Dl ) and H (Dr ) using above formulae of IG(T) for tree TA .
1
1
1
1
H (Dp ) = − log2
− log2
=1
2
2
2
2
1
3
3
1
− log2
= 0.81125
H (Dl ) = − log2
4
4
4
4
1
1
3
3
H (Dr ) = − log2
− log2
= 0.81125
4
4
4
4
Further, using the equation for information gain above, we compute the information gain for tree TA
400
400
0.81125 −
0.81125
IG (TA ) = 1 −
800
800
∴ IG (TA ) = 0.18875
Similarly, working out for tree TB , we get
∴ IG (TB ) = 0.3115
Thus, we note that IG (TB ) > IG (TA ) indicating that data split in the latter results in a better decision tree in
terms of the metric (information gain) based on entropy.
Page 19
Figure 2: Classification
(c) Comparison: We get different answers in (a) and (b). This indicates that entropy-based information gain is a better
metric than the misclassification rate since it is evident that the split is purer in the tree TB relative to that of tree
TA , and the information gain reflects the same. This, however, is not the case with the misclassification rate, which
assigns the same values to both trees.
Problem 19. (Classification)
1
Consider the data set plotted in Fig. 2. Show that a = ∥w∥
. How would L2 regularization on w affect the margin around
T
w x = 0?
Solution:
|b|
First, we use the fact that the minimal Euclidean distance from any point on a plane wT x = b to the origin is ∥w∥
.
We know that our other vectors are wT x + b = 1 and wT x + b = −1. This is re-written as wT x = 1 − b and wT x = −1 − b.
2
1
The distance between these two vectors is then ∥w∥
. Therefore, a is ∥w∥
.
We see that the margin around wT x = 0 is maximized when ∥w∥ is minimized.
Problem 20. (Model selection and Cross-validation)
n
Suppose we are given a noise-free set of points X = {xi }i=1 ⊂ (−1, 1), Y = {sin (xi )}, which we want to fit with a
k
polynomial, but we do not know which degree to choose. Suppose our candidate polynomial families are Pk = {P2i+1 }i=0 ,
where P2i+1 denotes the family of polynomials with real-valued coefficients of maximum degree 2i + 1. We want to find
the optimal hyperparameter value k̂ ∈ {1, . . . , k}.
Given a family of polynomials P2ℓ+1 and a training set, suppose we have an oracle (i.e., an exact algorithm) that is
able to find the polynomial p̂ ∈ P2ℓ+1 with optimal coefficients with respect to the square loss objective
Page 20
L(X, Y, p) =
n
X
2
(yi − p (xi )) ,
p ∈ P2ℓ+1
i=1
(a) Show that when the optimization is performed on each family in Pk , the lowest score is achieved when p̂ ∈
P2k+1 \P2k−1 (i.e., p̂ will be of degree 2k + 1 ).
(b) What potential issue with using cross-validation does this demonstrate?
(c) Suppose we widen the boundaries of X to (−2π, 2π). Write a short script to simulate samples Xi , Ỹi with different
values of σi2 and use 10-fold cross-validation to find corresponding optimal values k̂i . How do L Xi , Ŷi , p and k̂i
behave as k and σ 2 increase?
Solution:
(a) Note: We should consider polynomials of the following type, not the odd-ordered polynomials stated in the question
previously
P1 = w 1 x
P3 = w 1 x − w 2 x3
P5 = w 1 x − w 2 x3 + w 3 x5
P7 = w 1 x − w 2 x3 + w 3 x5 − w 3 x7
···
We also remember the Taylor-series approximation of sin(x)
sin(x) = x −
x3
x5
x7
+
−
···
3!
5!
7!
The loss can be calculated as:
L(X, Y, p, k) =
=
=
n
X
i=1
n
X
i=1
n
X
i=1
2
(yi − p (xi )) ,
p ∈ P2ℓ+1 .
(sin (xi ) − p2k+1 (xi ))
O
n
X
2k+3 2
xi
=
O
i=1
2
all terms in Taylor series expansion will be eliminated up to 2k + 1
x4k+6
i
>
n
X
O
i=1
4(k+1)+6
xi
=
n
X
O x4k+10
i
i=1
(b) When fitting a model, we can still choose overly complex models even when using cross-validation. In this case, it
occurred because of a special relationship between our data and the family of functions we were using to approximate
it.
(c) As σ 2 increases, we will no longer have a special relation between our learned function and sin(x). This means that
we will no longer be guaranteed to get the more complex model when using cross-validation (also see Fig. 3).
Problem 21. (Kernel function)
Kernel functions implicitly define some mapping function ϕ(·) that transforms an input instance x ∈ Rd to high dimensional
space Q by giving the form of the dot product in Q : K (xi , xj ) ≡ ⟨ϕ (xi ) , ϕ (xj )⟩.
(a) Prove that the kernel is symmetric, i.e. K (xi , xj ) = K (xj , xi ).
2
(b) Assume we use radial basis kernel function K (xi , xj ) = exp − 12 ∥xi − xj ∥ . Thus there is some implicit unknown
mapping function ϕ(x). Prove that for any two input instances xi and xj , the squared Euclidean distance of their
2
corresponding points in the feature space Q is less than 2, i.e., prove that ∥ϕ (xi ) − ϕ (xj )∥ ≤ 2.
Page 21
Figure 3: Model selection and Cross-validation
Solution:
(a) We have K (xi , xj ) = ⟨ϕ (xi ) , ϕ (xj )⟩ = ⟨ϕ (xj ) , ϕ (xi )⟩ = K (xj , xi ).
(b) We have
2
∥ϕ (xi ) − ϕ (xj )∥
= ⟨ϕ (xi ) , ϕ (xi )⟩ + ⟨ϕ (xj ) , ϕ (xj )⟩ − 2 · ⟨ϕ (xi ) , ϕ (xj )⟩
=K (xi , xi ) + K (xj , xj ) − 2 · K (xi , xj )
1
2
=1 + 1 − 2 exp − ∥xi − xj ∥ < 2
2
Problem 22. (Support vector machine)
With the help of a kernel function, SVM attempts to construct a hyper-plane in the feature space Q that maximizes the
Page 22
margin between two classes. The classification decision of any x is made on the basis of the sign of
X
⟨ŵ, ϕ(x)⟩ + ŵ0 =
yi αi K (xi , x) + ŵ0 = f (x; α, ŵ0 ) ,
i∈SV
where ŵ and ŵ0 are parameters for the classification hyper-plane in the feature space Q, SV is the set of support vectors,
and αi is the coefficient for the i-th support vector. Again we use the radial basis kernel function. Assume that the
training instances are linearly separable in the feature space Q, and assume that the SVM finds a margin that perfectly
separates the points.
If we choose
a test point xf ar which is far away from any training instance xi (distance here is measured in the original
space Rd , prove that f (xfar ; α, ŵ0 ) ≈ ŵ0 .
Solution:
∥xf ar − xi ∥ ≫ 0 ∀i ∈ SV
⇒K (xf ar , xi ) ≈ 0 ∀i ∈ SV
X
⇒
yi αi K (xi , x) ≈ 0
i∈SV
⇒f (x; α, ŵ0 ) ≈ ŵ0
Problem 23. (Perceptron/ SVM)
(a) How does the perceptron algorithm relate to stochastic gradient descent?
(b) How does the perceptron objective relate to the support vector machine objective?
(c) Write down the training objective for the SVM and derive the gradient updates using stochastic gradient descent.
Assume a minibatch size of B.
(d) The perceptron, in its original formulation, uses a 0/1 loss function (shown below, solid). A surrogate loss function
lp (w; x, y) = max 0, −ywT x is instead used in optimisation (dashed). We see that this surrogate loss is a poor
match for the 0/1 loss near zero. Suppose we try (shown in a dotted line):
(
0,
for sign wT x = y
ls (w; x, y) = p
−ywT x, for sign wT x ̸= y
(i) Show that f (x) =
√
x is not convex.
(ii) Show that f (x) = x is convex for all p ∈ N>0 and x ∈ [0, ∞). (Hint: use properties of derivatives of convex
functions.)
p
Solution:
Page 23
(a) Perceptron is SGD on the perception loss function
∇lp (wi , x, y) =
(b) Perceptron
min
X
0 if y = sign wT x −yx if y ̸= sign wT x
max 0, −yi αT ki
i
SVM
min
X
max 0, 1 − yi αT ki + λ∥w∥22
i
Difference is essentially an L2 penalty
(c)
1X
∇gi (x)
n i
∇gi (w) = ∇ max 0, 1 − yi wT xi + ∇λ∥w∥22
B
∇G(w) =
So looking at these separately:
∇λ∥w∥2 = 2λwk
0 if yi wT xi ≥ 1
∇ max 0, 1 − yi wT xi =
−yi xi otherwise
So from this, we get:
w = w + η(−2λw) if yi wT xi ≥ 1
w = w + η (yi xi − 2λw) otherwise
(d) From the given graph we have:
(i) Can show that −f (x) is convex
p
√
√
tx1 + (1 − t)x2 > t x1 + (1 − t) x2
√
tx1 + (1 − t)x2 > t2 x1 + (1 − t)2 x2 + t2(1 − t) x1 x2
√
x1 + x2 > 2 x1 x2
√
√ 2
( x1 − x2 ) > 0
(ii) Note that x is restricted to be positive, so one can easily check a few examples of p to gain some intuition.
f ′′ = p(p − 1)xp−2
p(p − 1) > 0
Since p is positive and x positive, the second derivated will always be positive
Problem 24. (Kernels)
2
(a) For x, x′ ∈ Rd , and K (x, x′ ) = xT x′ + 1 , find a feature map ϕ(x), such that k (x, x′ ) = ϕ(x)⊤ ϕ (x′ ).
(b) For the dataset X = {xi }i=1,2 = {(−3, 4), (1, 0)} and the feature map ϕ(x) = x(1) , x(2) , ∥x∥ , calculate the Gram
matrix (for a vector x ∈ R2 we denote by x(1) , x(2) its components).
Solution:
Page 24
(a)
  ′ 
x1
x1
 x2   x′2 



 ·   · 
′
′
′ 2



k
 ·   ·  = (1 + x1 x1 + x2 x2 + . . . + xn xn )



 ·   · 
xn
x′n
2
2
= 1 + (x1 x′1 ) + . . . + (xn x′n ) + 2x1 x′1 + . . . + 2xn x′n + 2x1 x′1 x2 x′2 + . . . + 2x1 x′1 xn x′n + . . .

= ϕ(x)T ϕ(x′)
√
√
√
√
√
√
√
Thus, ϕ(x) = 1, 2x1 , . . . , 2xn , 2x2 x1 , 2xn−1 x1 , . . . , 2xn−1 xn−2 , 2xn x1 , , . . . 2xn xn−1 , x21 , . . . , x2n .
(b)
k(x, x) = ϕ(x)T ϕ (x′ )
x(1) x(1) + x(2) x(2) + ∥x∥2
k(x, x) =
(1) ′(1)
x x
+ x(2) x′(2) + ∥x∥ ∥x′ ∥
50 2
k(x, x) =
2 2
x(1) x′(1) + x(2) x′(2) + ∥x∥ ∥x′ ∥
x′(1) x′(1) + x′(2) x′(2) + ∥x′∥2
Problem 25. (The Equivalence)
Consider a constrained version of least squares linear regression where we constrain the ℓ2 norm of w to be less than or
2
PN
equal to some c > 0 : ŵ = arg minw n=1 yn − w⊤ xn , s.t. ∥w∥ ≤ c.
Show, formally, that it is possible to have an ℓ2 regularized least squares linear regression model that will give the exact
same solution as the solution to the above-constrained optimization problem.
Solution:
Consider the following constrained version of least squares linear regression:
ŵ = arg min
w
N
X
yn − w T x n
2
subject to ∥w∥ ≤ c
n=1
Note that the constraint ∥w∥ ≤ c is equivalent to ∥w∥2 ≤ c2 since ∥w∥ ≥ 0. The Lagrangian function for the modified
constrained problem can be stated as follows:
L(w, α) =
N
X
yn − w T x n
2
+ α ∥w∥2 − c2
n=1
Ignoring the constant and re-writing,
L(w, α) =
N
X
yn − w T x n
2
+ αwT w = (y − Xw)T (y − Xw) + αwT w
n=1
Solving using the dual variable technique (We can do it since both the objective and constraint are convex),
∂L
= −2XT y + 2 XT X + αI w = 0
∂w
−1 T
w = XT X + αI
X y
∴ LD (α) = yT y − wT XT y − yT Xw + wT XT X + αI w
∴ LD (α) = yT y − wT XT y − yT Xw + wT XT y = yT y − yT Xw
Page 25
Ignoring the term yT y, we get
T
−1 T
LD (α) = −yT Xw = − XT y
XT X + αI
X y = −ZT M−1
α Z
T −1 ∂ Z Mα Z ∂Mα
∂L
T
−1 −1
=−
= M−1
I
α ZZ Mα
∂α
∂Mα
∂α
∂L
T
T
−1 −1
∴
= M−1
α X yy XMα
∂α
Note that MTα = Mα and thus we have
∴
T −1
∂L T
−1 T
= M−1
X
y
M
X
y
α
α
∂α
Problem 26. (Arbitrary Choice?)
Show that changing the condition yn w⊤ xn + b ≥ 1 in SVM to a different condition yn w⊤ xn + b ≥ m does not change
the effective separating hyperplane that the SVM learns. Assume the hard-margin SVM for simplicity.
Solution:
The original hard-margin SVM optimization problem can be stated as
∥w∥2
w,b
2
T
yn w xn + b ≥ 1,
arg min
∀n = 1, 2, . . . , N
The modified version of SVM involves changing the inequalities as yn wT xn + b ≥ m,
[α1 , α2 , .., αN ] be the Lagrange variables. The Lagrangian can be stated as
subject to
∀n = 1, 2, .., N . Let α =
N
∥w∥2 X
+
α n m − yn w T x n + b
L(w, b, α) =
2
n=1
Using the dual formulation to solve the constrained optimization problem, we have
Now, Substituting w =
PN
N
N
X
X
∂L(w, α)
∂L(w, α)
α n yn x n ,
α n yn = 0
=0⇒w=
=0⇒
∂w
∂b
n=1
n=1
n=1 αn yn xn in Lagrangian, we get the dual problem as
LD (α) = wT w +
N
X
(αn m − yn b) −
n=1
N
X
yn w T x n =
n=1
N
X
1 X
αl αn yl yn xTl xn
2
N
αn m −
n=1
n,l=1
Now, the objective can be stated in a compact form as
1
max m αT 1 − αT Gα
α≥0
2
where G is an N × N matrix with Gln = yl yn xTl xn , and 1 is a vector of ones. Note that m simply turns out to be a
multiplicative constant and thus does not affect the solution of the optimization problem. Thus, the solution to modified
SVM effectively remains the same as that of the original one.
Problem 27. (Learning SVM via Co-ordinate Ascent)
Consider the soft-margin linear SVM problem
arg max f (α),
0≤α≤C
where f (α) = α⊤ 1 − 12 α⊤ Gα, G is an N × N matrix such that Gnm = yn ym x⊤
xm and α = [α1 , α2 , . . . , αN ] are the
PN n
Lagrange multipliers. Given the optimal α, the SVM weight vector is w = n=1 αn yn xn .
Page 26
Your goal is to derive a co-ordinate ascent procedure for the vector α, such that each iteration updates a uniformly
randomly chosen entry αn of the vector α. However, instead of updating α via standard co-ordinate descent as αn =
αn + ηgn where gn denotes the n-th entry of the gradient vector ∇α f (α), we will update it as αn = αn + δ∗ where
δ∗ = arg maxδ f (α + δen ) and en denotes a vector of all zeros except a 1 at entry n.
Essentially, this will give the new αn that guarantees the maximum increase in f , with all other αn ’s fixed at their current
value. Derive the expression for δ∗ and give a sketch of the overall co-ordinate ascent algorithm.
Note that your expression for δ∗ should be such that the constraint 0 ≤ αn ≤ C is maintained.
Solution:
The soft-margin linear SVM dual objective to compute weights can be stated as
arg max f (α),
0≤α≤C
where f (α) = α
T
.1 − 12 αT Gα. The gradient-ascent step that we consider for αn is:
αn = αn + δ∗ ; where δ∗ := arg max f (α + δen )
δ
First, we will have to compute δ∗ in order to apply the ascent step.
1
T
(α + δen ) G (α + δen )
2
1
1
1
1
∴ f (α + δen ) = αT .1 + δeTn .1 − αT Gα − αT Gδen − δeTn Gα − δenT Gδen
2
2
2
2
1 2 T
T
T
f (α + δen ) = f (α) + δ en .1 − α Gen − δ en Gen
2
Taking partial derivative w.r.t. δ, and equating it to 0 , we get
T
f (α + δen ) = (α + δen ) .1 −
∂f (α + δen )
= eTn · 1 − αT Gen − δeTn Gen
∂δ
⇒ δ∗ =
eTn · 1 − αT Gen
eTn Gen
(5)
(t)
Note that we want to ensure that 0 ≤ αn ≤ C, thus if the updated value for αn becomes ≥ C, we will just set it to C
and if it goes less 0, we will just set it to 0. Thus, the update for αn will become as follows:

(t−1)

+ δ∗ < 0
if αn
0,
(t−1)
(t)
αn = C,
(6)
+ δ∗ > C
if αn

 (t−1)
αn
+ δ∗ otherwise
Coordinate ascent algorithm sketch
h
i
(0)
(0)
1. Inititalize α = α(0) = α1 , . . . , αN
2. Randomly choose n ∈ {1, 2, . . . , N }. Compute δ∗ from equation 5.
(t)
(t−1)
3. Update αn in terms of αn
and δ∗ as in equation 6.
4. If not converged, go to step 2.
Problem 28. (Separating Convex Hulls)
P
Given a set of
Pdata points x1 , . . . , xN , we define the convex hull to be the set of all points x given by x = n αn xn where
αn ≥ 0 and n αn = 1 (Intuitively, the convex hull of a set of points in the solid region that they enclose). Consider a
second set of points y 1 , . . . , y M together with their corresponding convex hull. Show that the set of x and the set of y
are linearly separable if and only if the convex hulls do not intersect.
Page 27
Solution:
N
N
Let Cx and Cy be the convex hulls corresponding to points X = {xi }i=1 and Y = {yi }i=1 respectively.
(
)
N
X
X
CX := x =
αn xn : αi ≥ 0,
αi = 1
(
CY :=
y=
n=1
i
N
X
X
βn yn : βi ≥ 0,
n=1
)
βi = 1
i
Definition 1. The sets of points X and Y are said to be linearly seperable if ∃ a line (hyperplane in high dimensional
space) L := x ∈ RD : mT x + b = 0 such that mT xi + b ≥ 0 , and mT yi + b < 0 ∀ i = 1, 2, . . . , N .
We have to show that X and Y are linearly separable iff CX ∩ CY = ϕ
(⇒) Assume that X and Y are linearly separable. Suppose for contradiction that CX ∩ CY ̸= ϕ. This implies that
∃z ∈ CX ∩ CY . Thus, by definition,
z=
N
X
αn xn =
n=1
N
X
β n yn
n=1
P
P
where αi ≥ 0, i αi = 1, βi ≥ 0, i βi = 1. Since, X and Y are linearly separable, there exists a line L satisfying
mT xi + b ≥ 0, and mT yi + b < 0 ∀ i = 1, 2, . . . , N .
Now, since αi ≥ 0, ∀ i,
αi mT xi + αi b ≥ 0, ∀i = 1, 2, . . . , N
∴
N
X
T
αn m x n +
n=1
N
X
αn b =
n=1
N
X
αn mT xn + b.1 = mT z + b ≥ 0
(7)
βn mT xn + b.1 = mT z + b < 0
(8)
n=1
Similarly, since βi mT yi + βi b < 0 ∀ i = 1, 2, . . . , N ,
∴
N
X
β n mT x n +
n=1
N
X
βn b =
n=1
N
X
n=1
The equations 7 and 8 present a contradiction. Thus, we have CX ∩ CY = ϕ.
(⇐) To show the converse, assume that CX ∩ CY = ϕ. To show that X and Y are linearly separable i.e. we have to find
L satisfying definition 1 Define
d :=
min
x∈CX ,y∈CY
∥x − y∥2
Note, since CX and CY are closed and bounded, the quantity d exists, and suppose it achieves the minima at xo ∈ CX , yo ∈
CY . Also d > 0 since CX ∩ CY = ϕ. Let L be the perpendicular bisector of the line joining xo and yo . It is trivial to
observe that L is a linear separator of X and Y . Hence, we are done.
Problem 29. (A Circular Definition)
Consider a logistic regression model p (yn | wn , w) = 1+exp(−y1 n w⊤ xn ) , with a zero-mean Gaussian prior p(w) = N 0, λ−1 I .
Note that this loss function for logistic regression assumes yn ∈ {−1, +1} instead of {0, 1}. Show that the MAP estiPN
mate for w can be written as w = n=1 αn yn xn where each αn itself is a function of w. Based on the expression of
αn , you
PNwould see that it has a precise meaning. Briefly, state what αn means, and also briefly explain why the result
w = n=1 αn yn xn makes sense for this model.
Solution:
Consider a logistic regression model with Gaussian prior.
p (yn | xn , w) =
1
,
1 + exp (−yn wT xn )
Page 28
p(w) = N 0, λ−1 I
For MAP estimation, we need to solve the following maximization problem:
ŵM AP = arg min L(w) = arg min −
w
w
N
X
!
log (p (yn | xn , w)) − log(p(w))
n=1
log (p (yn | xn , w)) = − log 1 + exp −yn wT xn , log(p(w)) = −(D/2) log(2π) + log(λ) − (λ/2)wT w
∴ L(w) =
N
X
log 1 + exp −yn wT xn
n=1
+
λ T
w w
2
Computing the partial derivative w.r.t w, and equating to 0, we get
N
N
X
X
−yn xn
∂L
yn α n x n ,
=
+
λw
⇒
w
=
∂w n=1 1 + exp (−yn wT xn )
n=1
where αn (w) =
1
p (yn | xn , w)
=
.
λ (1 + exp (−yn wT xn ))
λ
Thus, the MAP estimate tries to weigh examples based on the prediction probability of each given example. It will ignore
the examples on which the predicted probability is low (p (yn | xn , w) → 0) and consider only those examples that have
this predicted probability high much like the support vectors in SVM. The hyperparameter λ will decide the amount of
regularization as usual.
Problem 30. (Softmax and Variants)
N
Consider N training examples {xn , yn }n=1 where xn ∈ RD , and yn ∈ {1, . . . , K}. Suppose we wish to use this data to
learn the multiclass logistic (or “softmax”) regression model which defines
exp w⊤
k wn
p (yn = k | xn , W) = µnk = PK
,
⊤
ℓ=1 exp w ℓ w n
where µnk is the predicted probability of yn = k. Derive the MLE solution for W = [w1 , w2 , . . . , wK ]. You would notice
that, just like logistic regression, there is no closed-form solution for W. So you will need to write down the log-likelihood,
take its derivative w.r.t. each column wk of W to compute the gradient, and derive the gradient descent (GD) update
rule for each wk . Show the basic steps of your derivation and write down the final expression for the GD update of each
wk . Assume a fixed learning rate η = 1 for simplicity.
Next, consider the stochastic gradient descent (SGD) update for the same model where each iteration takes a randomly
chosen example (xn , yn ) to update W. Write down the expressions for these SGD updates and the overall sketch of the
corresponding SGD algorithm.
Finally consider a special case of the above SGD algorithm for this model, where the predicted “soft” class probabilities
K
µnk are replaced by hard class assignments, i.e., µnk = 1 for k = arg maxℓ {µnℓ }ℓ=1 , and µnk′ = 0, ∀k ′ ̸= k. Write down
the expressions for the SGD update in this case and the overall sketch of the SGD algorithm. How are these updates
different from the previous case where you used soft probabilities µnk ?
Solution:
N
Consider N training examples {xn , yn }n=1 where xn ∈ RD , yn ∈ RK . Consider the softmax regression model for multi-class
classification:
exp wkT xn
p (yn = k | xn , W ) = µnk = PK
T
l=1 exp wl xn
The MLE objective can be stated as ŴM LE = arg minW −N LL(W ). Let us work for each column wk of W :
Page 29
N
X
−∂N LL(W )
∂ log (p (yn = k | xn , W ))
=−
∂wk
∂wk
n=1
log (p (yn = k | xn , W )) = wkT xn − log
K
X
exp
wlT xn
!
l=1
∴
∂ log (p (yn = k | xn , W ))
= xn − µnk xn
∂wk
N
−∂N LL(W ) X
∴
=
xn (µnk − 1) = X T Diagonal ([µ1k − 1, µ2k − 1, . . . , µN k − 1])
∂wk
n=1
Thus, no closed-form solution for wk can be computed from this equation. We can write the gradient descent update for
wk with η = 1 as follows:
(t+1)
wk
(t)
= wk −
N
X
(t)
xn µnk − 1
n=1
Similarly, the stochastic gradient descent update will turn out to be:
(t+1)
(t)
(t)
wk
= wk − xn µnk − 1
Consider the case of ‘hard’ assignments, i.e., let k = arg maxk µnk and µnk′ = 0, ∀ k ′ ̸= k. The SGD update will simply
be reduced to
(
(t)
wk
for p (yn = k | xn , W ) = 1
(t+1)
(t)
(t)
wk
= wk − xn µnk − 1 =
(t)
wk + xn for p (yn = k | xn , W ) = 0
Thus, we simply do NOT update the weight vector wk if we encounter an example (xn , yn ) such that the current prediction
is correct, i.e. whose label is k. Otherwise, we make the weight vector wk move in the direction of xn . The SGD algorithm
sketch for this case is presented in Algorithm 1
Algorithm 1 Multi-class Stochastic Gradient Descent
(0)
Choose initial wk = wk , ∀k ∈ 1, 2, . . . , K.
for k ∈ 1, 2, . . . , K do
Keeping wl , for l ̸= k fixed, update wk
for s iterations until convergence do
if p(yn |xn , W (t) ) = 1 then
(t+1)
(t)
wk
= wk
else
(t+1)
(t)
wk
= w k + xn
end if
end for
end for
Problem 31. (Estimating a Gaussian when Data is Missing)
Suppose we have collected N observations {x1 , . . . , xN } using a sensor. Let us assume each x ∈ RD as generated
from a Gaussian distribution N (µ, Σ). We would like to estimate the mean and covariance of this Gaussian. However,
suppose
and each xn could only have part of it as observed (think of a blacked-out image). Denote
the sensor
was faulty
miss
miss
xn = xobs
where xobs
denote the observed and missing parts, respectively, of xn . We only get to see
n , xn
n and xn
obs
xn . Note that different observations could have different parts as missing (e.g., different images may have different sets
of pixels as missing), so the indices of the observed/missing entries of the vector xn may be different for different n.
Your goal is to develop an EM algorithm that gives maximum likelihood estimates of µ and Σ given this partially observed
data. In particular, in the EM setting, you will treat each xmiss
as a latent variable and estimate its conditional distribution
n
Page 30
p xmiss
| xobs
n
n , µ, Σ , given the current estimates µ and Σ of the parameters. In the M step, you will re-estimate µ and Σ
and will alternate between E and M steps until convergence.
Solve the following:
1. The expression for p xmiss
| xobs
n
n , µ, Σ
2. The expected CLL for this model
3. The update equations for µ and Σ.
Also clearly write down all the steps of the EM algorithm in this case, with appropriate equations.
Hint: For this problem, you may find it useful to use the result that if x = [xa , xb ] is Gaussian then p (xa | xb ) is also
Gaussian.
Solution:
Let X = {x1 , x2 , .., xN } , Θ = {µ, Σ} and
p (xn | Θ) = N (xn | Θ)
Part 1: Estimating the conditional given current parameter estimates
Let’s say we have Θ as our current parameter estimates. Let us try to estimate the missing part of data for a point xn .
We have
p
obs miss miss
xn , x n
| Θ = N xobs
|Θ
n , xn
Based on the missing and observed part of this data point, let us denote the parameters as follows:
µmiss
µ1
Σ11 Σ12
µ=
=
,Σ =
µobs
µ2
Σ21 Σ22
Note that this miss-observed split may be different for different examples. This also makes intuitive sense, for example, if
for the first example the first two features are missing, then its estimate should be based on the means of these features
of other examples and also covariances with other features. For an example with a different set of missing features, the
estimates will be different. Since a conditional of a Gaussian is a Gaussian, we have
miss
where
p xmiss
| xobs
| xobs
n
n , Θ = N xn
n , µmiss|obs , Σmiss|obs
−1
obs
µmiss|obs = µ1 + Σ12 Σ22 xn − µ2
Σmiss |obs = Σ11 − Σ12 Σ−1
22 Σ21
Part 2: Expected CLL
First, I will compute the CLL as follows and then take the expectation:
CLL =
N
X
log p (xn | Θ = {µ, Σ})
n=1
CLL =
N
X
log p
obs miss xn , xn
| Θ = {µ, Σ}
n=1
E[CLL] =
N
X
miss
E log p xobs
| µ, Σ
n , xn
n=1
E[CLL] = constant +
S=
N
X
1
N
log |Σ|−1 − Trace Σ−1 E[S]
2
2
(xn − µ) (xn − µ)
n=1
Page 31
T
We will need to compute the expectation of S over the posterior. Note that we will have the following expectation values:
xobs
E xobs
n
n E [xn ] =
=
(9)
E xmiss
E xmiss
n
n
!
miss T
(obs)T
xobs
xobs
n xn
n E xn
T
h
i
obs T
T
E xn xn =
(10)
E xmiss
xn
E xmiss
xmiss
n
n
n
E[S] =
N n
o
X
T
E xn xTn + µµT − 2µE [xn ]
(11)
n=1
Note that from part 1, we have got the conditional distribution for xmiss
, thus we will have the mean E xmiss
n
n = µmiss obs .
The expected CLL computation is over the conditional distribution. Thus, we can plug the value of E xmiss
in the above
n
equations to obtain the expected CLL. The only computation that will remain is
h
i
miss T
miss T
miss T
E xmiss
x
= E xmiss
E xn
+ cov xmiss
= E xmiss
E xn
+ Σ11
n
n
n
n
n
Part 3: Estimation of parameters: M step
Let us denote E[CLL] = L(µ, Σ).
to obtain the updates for these paramteres, we will set the partial derivatives
In order
to 0 treating terms involving E xmiss
as constant.
n
T ∂E[S]
∂L
1
= − Σ−1
∂µ
2
∂µ


T
N 

∂
µE
[x
]
T
X
n
∂E[S]
∂µµ
=
−2
 ∂µ

∂µ
∂µ
n=1
h
miss T i
obs(T )
N ∂
µE
xn
µx
T
X
n
∂µµ
∂E[S]
=N
−2
∂µ
∂µ
∂µ
n=1
∂L
N |Σ| ∂|Σ|−1
1 ∂ Trace Σ−1 E[S]
=
−
∂Σ
2
∂Σ
2
∂Σ
EM Algorithm steps:
1. Initialize Θ = Θ(0) = µ(0) , Σ(0) , set t = 1
2. E step: Now, assuming the values of Θ(t−1) , compute the expected CLL using expression for E[S] of equation 11
which in turn needs the equations 9 and 10.
3. M step: Now, update the values for µ(t) and Σ(t) using the equations in part 3. Update t = t + 1
4. Repeat steps 2-4 until convergence.
Problem 32. (Semi-supervised Classification)
Consider learning a generative classification model for K-class classification with Gaussian class-conditionals N (x | µk , Σk ),
k = 1, . . . , K with class marginals p(y = k) = πk . However, unlike traditional generative classification, in this setting, we
N
are given N labeled examples {(xn , yn )}n=1 and an additional M unlabeled examples {xN +1 , . . . , xN +M }. Design an EM
algorithm to estimate all the unknowns of this model and clearly write down the expressions required in each step of the
EM algorithm.
Solution:
N
N +M
N +M
Let us fix the following notation: X1 = {(xn , yn )}n=1 , X2 = {xm }m=N +1 and let Z2 = {zm }m=N +1 be the latent variables.
K
The set of parameters is Θ = {µk , Σk , πk }k=1 . We assume that all the examples, whether with known labels or not are
i.i.d sampled. We have the conditional distribution as follows:
Page 32
p (xm | zm = k, Θ) p (zm = k | Θ)
p (xm | zm = k, Θ) p (zm | Θ)
= PK
p (xm | Θ)
l=1 p (xm , zm = l | Θ)
p (zm = k | xm , X1 , Θ) = p (zm = k | xm , Θ) =
πk N (xm | µk , Σk )
Let γmk = p (zm = k | xm , X1 , Θ) = PK
l=1 πl N (xm | µl , Σl )
γmk =
πk N (xm | µk , Σk )
P
N ( l πl µ l , )
(12)
Expected CLL
Let us fix the notation to be (as usual): zmk = 1 iff zm = k and ynk = 1 iff yn = k. Now, the CLL will be computed as
follows, note the assumption of IID.
CLL = log p (X1 , (X2 , Z2 ) | Θ) = log p (X2 , Z2 | X1 , Θ) + log p (X1 | Θ) = log p (X2 , Z2 | Θ) + log p (X1 | Θ)
Now, we have
N X
K
X
log p (X1 | Θ) =
ynk (log πk + log N (xn | yn = k, µk , Σk ))
n=1 k=1
NX
+M
log p (X2 , Z2 | Θ) =
K
X
zmk (log πk + log N (xm | zm = k, µk , Σk ))
m=N +1 k=1
Now, notice that for computing the expected CLL, we will need the expectation over the posterior: E [znk ] = 1∗p (znk = 1)+
0 ∗ p (znk = 0) = p (znk = 1) = γmk .
E[CLL] =
K
N X
X
NX
+M
ynk (log πk + log N (xn | yn = k, µk , Σk )) +
n=1 k=1
K
X
γmk (log πk + log N (xm | zm = k, µk , Σk )) (13)
m=N +1 k=1
Updating the paramters
We have the following objective.
Θ̂ = arg max E[CLL]
Θ
It basically boils down to solving the problem of Gaussian generative classification with known labels:
E[CLL] =
NX
+M X
K
tnk (log πk + log N (xn | µk , Σk )) ,
n=1 k=1
where tnk = ynk , ∀n ∈ {1, 2, . . . , N } and tnk = γnk , ∀n ∈ {N + 1, N + 2, . . . , N + M }. Thus the parameters will become:
PN +M
π̂k =
tnk
=
N +M
PN
n=1
PN +M
m=N +1 γnk
N +M
=
N k + Mk
N +M
(14)
NX
+M
1
tnk xn
Nk + Mk n=1
(15)
NX
+M
1
T
tnk (xn − µ̂k ) (xn − µ̂k )
Nk + Mk n=1
(16)
µ̂k =
Σ̂k =
n=1 ynk +
The EM algorithm
n
o
(0)
(0)
(0)
1. Initialize Θ = Θ(0) = πk , µk , Σk
, set t = 1
k
Page 33
2. E step: Now, assuming the values of Θ(t−1) , compute the expected CLL from equation 13 using expression for
E [zmk ] = γmk (See equation 12).
(t)
(t)
(t)
3. M step: Now, update the values for πk , µk and Σk using the equations 14, 15, and 16 respectively. Update
t = t + 1.
4. Repeat steps 2-4 until convergence.
Problem 33. (Latent Variable Models for Supervised Learning)
N
Consider learning a regression model given training data {(xn , yn )}n=1 , with xn ∈ RD and yn ∈ R. Let us give a
small twist to the standard probabilistic linear model for regression that we have seen in class. In particular, we will be
introducing a latent variable z n with each training example (xn , yn ). The generative story would now be as follows
zn ∼ multinoulli (π1 , . . . , πK )
−1
yn ∼ N w ⊤
z n xn , β
Note that the model for the responses yn is still discriminative since the inputs are not being modeled.
The latent variables are Z = (z1 , . . . , zN ) and the global parameters are Θ = {(w1 , . . . , wK ) , (π1 , . . . , πK )}.
1. Give a brief explanation (max. 5 sentences) of what the above model is doing and why you might want
to use it
instead of the standard probabilistic linear model which models each response as yn ∼ N w⊤ xn , β −1 .
2. Derive an ALT-OPT algorithm to estimate Z and (MLE of) Θ, and clearly write down each step’s update equations.
For Z, you must give the update equation for each individual latent variable zn , n = 1, . . . , N . Likewise, for Θ, you
must give the update equation for each wk , k = 1, . . . , K, and πk , k = 1, . . . , K. Also, what will be the update of
each zn if πk = 1/K, ∀k. Give a brief intuitive explanation (max 1-2 sentences) as to what this update does.
3. Derive an expectation-maximization (EM) algorithm to estimate Z and (MLE of) Θ, and clearly write down each
step’s update equations. Also show that, as β → ∞, the EM algorithm reduces to ALT-OPT.
Solution:
N
Let the observed data be {(xn , yn )}n=1 and latent variables Z = {z1 , z2 , .., zK }. The global set of parameters be given by
Θ = {(w1 , . . . , wK ) , (π1 , . . . , πK )}, we have
zn ∼ multinoulli (π1 , .., πK )
yn ∼ N wzTn xn , β −1
Part 1: Mixture of regression models
The model is basically fitting multiple regression models wzn on the input data. It is similar to doing multi-output
regression. In simple probabilistic linear regression, we assume that all of the data points are sampled, i.i.d. from a linear
space. However, as shown in Figure 4, for such kind of data, a mixture of multiple regression models can help.
Part 2: ALT-OPT Algorithm
Estimating the latent variables fixing the parameters
Suppose we have the optimal value of Θ = Θ̂, then we can estimate zn as follows:
p yn , zn = k | xn , Θ̂
ẑn = arg max p zn = k | yn , xn , Θ̂ = arg max
k
k
p yn | xn , Θ̂
p yn | zn = k, xn , Θ̂ p zn = k | Θ̂
πk N wkT xn , β −1
= arg max PK
= arg max P
T
K
−1
k
k
l=1 πl N wl xn , β
l=1 p yn | zn = l, xn , Θ̂ p zn = l | Θ̂
Page 34
Figure 4: If the data looks somewhat like this, using multiple mixtures of linear regressions would be a better idea.
Let us denote the term
πk N wkT xn , β −1
PK
l=1 πl N
wlT xn , β −1
= γnk .
Notice that the denominator does not depend on k and can be dropped, but I am keeping it since the expression takes
the form of probability. It will not matter for the maximization. Also, let’s use the notation that znk = 1 iff zn = k.
Estimating the parameters fixing the latent variables
Once we pick ẑn ’s that maximize γnk ’s, we can now keep them fixed and estimate parameters.
Θ̂ = arg max L (Θ, {ẑn }) = arg max
Θ
Θ
Θ̂ = arg max
Θ
N X
K
X
N
X
log p (yn , ẑn | xn , Θ)
n=1
ẑnk log πk + log N wkT xn , β −1
n=1 k=1
P
Note that we will also have to incorporate the constraint k πk = 1. We can then solve this by constrained optimization.
For πk , since the term involving it is not coupled with any of the other parameters, we can solve for it independently. The
problem for finding πk is exactly the same as that in generative classification with known labels thus we have
Nk
1 X
ẑnk =
N n=1
N
N
π̂k =
Now, to estimate the weights wk ’s, we have
K
{ŵk }k=1 = arg
K
X
K
{ŵk }k=1 = arg max
w1 ,.,wK
max
w1 ,...,wK
N X
K
X
ẑnk wkT xn − yn
2
n=1 k=1
T
(y − Xwk ) Zk (y − Xwk ) where Zk = Diag (z1k , .., zN k )
k=1
Note that these terms can be maximized independently since they do not depend upon each other. Thus, we get
T
ŵk = arg max (y − Xwk ) Zk (y − Xwk )
wk
This is the same as solving a linear regression problem with examples being weighted by {znk }’s. Thus, the solution will
be as follows:
ŵk = XT Zk X XT Zk y
A Special case
Page 35
For the case when πk = (1/K), ∀k = 1, 2, ..K implies that
N wkT xn , β −1
ẑn = arg max δnk = arg max PK
k
k
l=1 N
wlT xn , β −1
This kind of update ignores the number of points that can be explained by a single model wk . Thus, it will only consider
the probability of xn explained by these different models giving them equal weightage.
Part 3: EM Algorithm
A. Computing the posterior
We have already computed the conditional distribution for the latent variables in Part 2 as:
πk N wkT xn , β −1
p (zn = k | yn , xn , Θ) = γnk = PK
T
−1
l=1 πl N wl xn , β
From the discrete distribution, we have the expectation as simply:
E [zn ] =
K
X
kp (zn = k | yn , xn , Θ) =
k=1
K
X
kγnk
k=1
E [znk ] = 1 ∗ p (znk = 1) + 0 ∗ p (znk = 0) = p (zn = k) = γnk
(17)
B. Computing the expected CLL
Now, let us compute the expected CLL.
CLL = log p(Y, Z | X, Θ) =
N
X
log p (yn , zn | xn , Θ) =
n=1
E[CLL] =
N X
K
X
znk log πk + log N wkT xn , β −1
n=1 k=1
N X
K
X
E [znk ] log πk + log N wkT xn , β −1
(18)
n=1 k=1
We can get E [znk ] from equation 17, and thus the expected CLL. The rest of the procedure for maximizing the expected
CLL will remain the same as in the previous part, except that instead of znk , we will be using γnk . Thus, we have
1 X
π̂k =
γnk ,
N n=1
N
ŵk = XT Γk X XT Γk y,
(19)
where Γk = Diag (γ1k , γ2k , .., γnk ).
The Algorithm
n
o
(0)
(0)
1. Initialize Θ = Θ(0) = πk , wk , set t = 1
2. E step: Now, assuming the values of Θ(t−1) , compute the expected CLL from equation 18 using expression for
E [znk ] = γnk .
(t)
(t)
3. M step: Now, update the values for πk , wk using the equation 19. Update t = t + 1.
4. Repeat steps 2-4 until convergence.
The Particular case
We have to show that when β → ∞, the EM algorithm boils down to the ALT-OPT algorithm. Basically, we want to
show the following:
Page 36
πk N wkT xn , β −1
γnk = 1 iff k∗ = arg max PK
l=1 πl N
k
wlT xn , β −1
We have
i
h
2
πk exp −β 2 yn − wkT xn
h
i
γnk (β) = P
2
K
2 y − wT x
n
l=1 πl exp −β
l n
⇒ γnk (β) =
1+
P
πl
l̸=k πk exp
h
−β 2
1
yn − wlT xn
2
− yn − wkT xn
2
2
i
2
Now, consider the case when k ̸= k∗ as defined above, the term yn − wlT xn − yn − wkT xn < 0, ∀l = k∗ , thus the
exponent becomes ∞ making γnk → 0. For k = k∗ , clearly, all other terms will go to 0, and thus γnk∗ = 1. Thus, the
claim is proved, and hence we will have
E [znk ] = 1 only when ẑnk = 1
This implies that the expectation step becomes the same as that of ALT-OPT, and so does maximization since the
expected likelihood will be the same as the parameter update step in ALT-OPT.
Problem 34. (Probabilistic Formulation of Matrix Factorization with Side Information)
Consider an N × M rating matrix X, where the rows represent the N users, and the columns represent the M items. We
are also given some side information: for each user n, a feature vector an ∈ RDU , and for each item m,
a feature vector
−1
bm ∈ RDI . Let’s model each entry of X as p (Xnm | un , v m , θn , ϕm ) = N Xnm | θn + ϕm + u⊤
.
n v m , λx
In this model, un ∈ RK and v m ∈ RK represent the user n and item m latent factors, respectively. In addition, for each
user n, we have a user-specific bias θn ∈ R (bias regardless of the item being rated), and for each item m, we have an
item-specific bias ϕm ∈ R (“popularity” of the item, regardless is who has rated this item).
Assume Gaussian priors
on the latent factors un ∈ RK and v m ∈ RK : p (un ) = N un | Wu an , λ−1
u IK and p (v m ) =
N v m | Wv bm , λ−1
v IK . Note that the mean of the priors of the latent factors un and v m depends on the given user
and item features (an and bm , respectively) via a linear model with regression parameters matrices Wu ∈ RK×DU and
Wv ∈ RK×DI , respectively. You don’t need to assume any priors on the bias parameters θn , ϕm , and the regression
parameters Wu , Wv .
Assume Ω = {(n, m)} to denote the set of indices of the observed entries of X, Ωrn to be the set of items rated by user
N
M
n, and Ωcm to be the set of users who rated item m. Your goal is to estimate {un , θn }n=1 , {v n , ϕm }m=1 , Wu , and Wv .
Assume all other parameters to be known.
Write down the loss function for this model (this will be the negative of the MAP objective for the model) and use the
ALT-OPT algorithm to derive the update equations for all the unknowns. The expressions must be in closed form (it is
possible for this model).
Solution:
Consider an N × M rating matrix X, where the rows represent the N users, and the columns represent the M items. We
are also given some side information: for each user n, a feature vector an ∈ RDU , and for each item m, a feature vector
bm ∈ RDI . Let un , vm ∈ RK represent the latent factors for user n and item m respectively. Let θn be user bias and ϕm
be the popularity of the item. We have
p (Xnm | un , θn , vm , ϕm ) = N Xnm | θn + ϕm + uTn vm , λ−1
p (un ) = N un | Wu an , λ−1
u IK
p (vm ) = N vm | Wv bm , λ−1
v IK
Page 37
Assume Ω = {(n, m)} to denote the set of indices of the observed entries of X, Ωrn to be the set of items rated by user n,
and Ωcm to be the set of users who rated item m. Let Θ := {{(un , θn )} , {vm , ϕm } , Wu , Wv }
The MAP objective can be written as follows:
Θ̂M AP = arg max{log(p(X | Θ)) + log(p(Θ))}
Θ
X
log(p(X | Θ)) =
log (p (Xnm | Θ)) = −
(n,m)∈Ω
log(p(Θ)) =
N
X
M
X
log (p (un )) +
n=1
log (p (vm )) = −
λ
2
X
2
(n,m)∈Ω
N
λu X
2 n=1
m=1
Xnm − θn + ϕm + uTn vm
2
∥un − Wu an ∥ −
M
λv X
2
∥vm − Wv bm ∥
2 m=1
Thus, the consolidated loss can be written as
L(Θ) = λ2
P
(n,m)∈Ω
Xnm − θn + ϕm + uTn vm
2
+ λ2u
PN
2
2
λv PM
n=1 ∥un − Wu an ∥ + 2
m=1 ∥vm − Wv bm ∥ .
Optimizing using ALT-OPT
nn
o
o
• Estimating latent variables and parameters for users: Let us keep
v̂m , ϕ̂m , Ŵv fixed and estimate the
remaining variables. We will only consider relevant terms in the loss function.
λ
2
L(Θ) =
X
Xnm − θn + ϕm + uTn vm
2
λu X
2
∥un − Wu an ∥
2 n=1
N
+
(n,m)∈Ω
1. Estimating un keeping θn , Wu fixed: Note that this will turn out to be simple a least-squares problem with
the prior on un being non-zero mean. Thus, we can write the solution in closed form as follows:

X
un = 
−1 
T
vm
v m + λ u IK 
X
λu Wu an + λ
m∈Ωrn

(Xnm − θn − ϕm ) vm 
m∈Ωrn
2. Estimating θn keeping un , Wu fixed: Now, we can simply set the derivative to 0 and obtain θn as follows:
P
θn =
m∈Ωrn
Xnm − ϕm − uTn vm
P
m∈Ωrn 1
3. Estimating Wu keeping θn , un , ∀n fixed: This time we get the loss function which resembles the loss of a
multi-output linear regression problem. Thus, we can write the solution as follows -
Wu =
N
X
!
un aTn
n=1
N
X
!−1
an aTn
n=1
• Estimating latent variables and parameters for items: Similar procedure (as for estimating user variables
and parameters) can be followed for items.
1. Estimating vm keeping ϕm , Wv fixed:

vm = 
X
−1 
uTn un + λv IK 
λ v W v b m + λ
X

(Xnm − θn − ϕm ) un 
n∈Ωcm
n∈Ωcm
2. Estimating ϕm keeping vm , Wv fixed:
P
ϕm =
n∈Ωcm
Xnm − θn − uTn vm
P
n∈Ωcm 1
Page 38
3. Estimating Wv keeping ϕm , vm , ∀m fixed:
Wv =
M
X
!
!−1
M
X
vm bTm
m=1
bm bTm
m=1
Problem 35. (EM for Naïve Bayes)
Assume that you want to train a naïve Bayes model on data with missing class labels. Specifically, there are k binary
variables X1 , . . . Xk corresponding to the features, and a variable Y taking on values in {1, 2, . . . , m} denoting the class.
Let us denote the set of model parameters as P (Xi = 1 | Y = y) = θi|y and P (Y = y) = θy . You are given n data points
D = {(x1 , y1 ) , . . . , (xn , yn )} where xi ∈ {0, 1}k and yi ∈ {1, 2, . . . , m, ×}. The value x means that the label of the data
point is missing.
• Write down the log-likelihood ℓ(θ) of the data as a function of the parameters θ.
• Recall that the E-step of the EM algorithm computes the posterior over the unknown variables when we fix the
parameters θ. Compute these probabilities γj (xi ) = P (Y = j | xi ; θ) for j s.t. yi = x.
• P
Once P
we have the quantities γj (·), we can compute the M-step update, which is computed as the maximizer θ∗ of
n
m
∗
∗
j=1 γj (xi ) log P (xi , yi = j; θ). Show how to compute θ . Note that there are constraints on θ to make sure
i=1
that the distributions are valid (non-negative and sum up to 1 ).
Solution:
The log-likelihood is equal to
ℓ(θ) = log P (D)
n
n
X
X
=
log P (xi ; θ) +
log P (xi , yi ; θ)
i=1
yi =x
=
n
X
i=1
yi ̸=x
log
i=1
yi =x
=
=
n
X
m
X
j=1
log
m
X
i=1
yi =x
j=1
n
X
m
X
i=1
yi =x
P (xi , Y = j; θ) +
log
j=1
n
X
log P (xi , yi ; θ)
i=1
yi ̸=x
P (xi | Y = j; θ) P (Y = j; θ) +
n
X
log P (xi , yi ; θ)
i=1
yi ̸=x
k
n
k
Y
Y
1−xi,l X
1−xi,l
xi,l
x
θj
θl|j 1 − θl|j
+
log θyi
θl|yi,li 1 − θl|yi
.
i=1
yi ̸=x
l=1
l=1
To compute the requested posterior probabilities, note that by Bayes’ rule
γj (xi ) = P (yi = j | xi ; θ)
P (xi | yi = j; θ) P (yi = j; θ)
P (xi ; θ)
1
= P (xi | yi = j; θ) P (yi = j; θ)
Z
k
1−xi,l
1 Y xi,l
= θj
θl|j 1 − θl|j
.
Z
l=1
Pm
We then have to compute the normalizer Z so that j=1 γj (xi ) = 1. Note that for those data points xi for which we are
given the labels yi we set the γj (xi ) to be a deterministic distribution, i.e. γj (xi ) = [j = yi ].
=
To compute the M-step update we have to optimize the following quantity
Page 39
n X
m
X
γj (xi ) log P (xi , yi = j; θ) =
i=1 j=1
n X
m
X
"
γj (xi ) log θj +
i=1 j=1
k
X
x
log θl|ji,l
1 − θl|j
1−xi,l
#
l=1
with respect to the parameters θ. We form the Lagrangian by adding a multiplier λ to make sure that


"
#
n X
m
k
m
X
X
X
1−xi,l
x
L(θ, λ) =
γj (xi ) log θj +
log θl|ji,l 1 − θl|j
+ λ
θj − 1 .
i=1 j=1
Pm
j=1 θj = 1:
j=1
l=1
By setting the derivatives to zero we obtain:
∂
∂θl|j
L(θ, λ) =
n
X
γj (xi ) /θl|j +
i=1
xi,l =1
n
X
Pn
γj (xi ) / θl|j − 1 = 0 =⇒ θl|j =
i=1
xi,l =0
[x = 1] γj (xi )
i=1
Pni,l
i=1 γj (xi )
Pn
n
X
γj (xi )
∂
L(θ, λ) =
γj (xi ) /θj + λ = 0 =⇒ θj = − i=1
∂ θj
λ
i=1
Pn Pm
Pm
From the constraint j=1 θj = 1 we find the correct multiplier to be λ = − i=1 j=1 γj (xi ) = −n.
Problem 36. (EM for a 1D Laplacian Mixture Model)
In this problem, you will derive the EM algorithm for a one-dimensional Laplacian mixture model. You are given n
observations x1 , . . . , xn ∈ R and we want to fit a mixture of m Laplacians, which has the following density
f (x) =
m
X
πj fL (x; µj , βj ) ,
j=1
− 1 |x−µj |
where fL (x; µj , βj ) = 2β1 j e βj
, and the mixture weights πj are a convex combination, i.e. πj ≥ 0 and
For simplicity, assume that the scale parameters βj > 0 are known beforehand and thus fixed.
Pm
j=1 πj = 1.
• Introduce latent variables so that we can apply the EM procedure.
• Analogously to the previous question, write down the steps of the EM procedure for this model. If some updates
cannot be written analytically, give an approach on how to compute them.
(Hint: Recall a property of functions that makes them easy to optimize.)
Solution:
For each data point xi , we introduce a latent variable Yi ∈ {1, 2, . . . , m} denoting the component that point belongs to.
For the E-step, we compute the posterior over the classes similarly to the previous problem, i.e.
γj (xi ) = P (yi = j | xi ) ∝ P (xi | yi = j) P (yi = j) = πj fL (xi ; µj , βj ) .
Again, we have to normalize so that the final posterior is equal to
πj fL (xi ; µj , βj )
γj (xi ) = Pm
l=1 πl fL (xi ; µl , βl )
In the M-step, we optimize
n X
m
X
i=1 j=1
γj (xi ) log P (xi , yi = j) =
n X
m
X
γj (xi ) log πj fL (xi ; µj , βj )
i=1 j=1
n X
m
X
1
=
|xi − µj | + const.
γj (xi ) log πj −
βj
i=1 j=1
Page 40
(20)
We add a Lagrange multiplier λ to make sure that
Pm
j=1 πj = 1 and obtain the Lagrangian


m
X
1
L(π, µ, λ) =
|xi − µj | + λ 
γj (xi ) log πj −
πj − 1
β
j
i=1 j=1
j=1
n X
m
X
Exactly as in the previous problem, by setting the gradient with respect to πj to zero, we obtain
Pn
n
X
γj (xi )
∂
γj (xi ) /πj + λ = 0 =⇒ πj = i=1
L(π, µ, λ) =
.
∂ πj
−λ
i=1
The multiplier is again equal to λ = −n, and we arrive at the same equation as in the last example. If we want to
maximize Eqn. (20) with respect to the variables µj , we have to solve m separate optimization problems, one for each µj .
These m problems have the following form
maximize −
µj
n
X
γj (xi )
i=1
βj
|xi − µj |
These are one-dimensional convex optimization problems (the negative of the objective is easily seen to be convex). While
one can try solving this via an iterative process like subgradient descent, a direct approach is also possible if we observe
that the function is piecewise linear. The breakpoints are x1 , x2 , . . . , xn . Hence, the optimum must be attained at one of
these n points, and we can simply set µj to the point xi with the largest objective value.
Problem 37. (A different perspective on EM)
In this question, you will show that EM can be seen as iteratively maximizing a lower bound on the log-likelihood. We
will treat any general model P (X, Z) with observed variables X and latent variables Z. For the sake of simplicity, we
will assume that Z is discrete and takes on values in {1, 2, . . . , m}. If we observe X = x, the goal is to maximize the
log-likelihood
ℓ(θ) = log P (x; θ) = log
m
X
P (x, z; θ)
z=1
with respect to the parameter vector θ. In what follows, we will denote any distribution over the latent variables by Q(Z).
• Show that if Q(z) > 0 when P (x, z) > 0, then it holds that (Hint: Consider using Jensen’s inequality)
ℓ(θ) ≥ EQ [log P (X, Z)] −
m
X
Q(z) log Q(z).
z=1
Hence, we have a bound on the log-likelihood parametrized by a distribution Q(Z) over the latent variables.
• Show that for a fixed θ, the lower bound is maximized for Q∗ (Z) = P (Z | X; θ). Moreover, it shows that the bound
is exact (holds with equality) for this specific distribution Q∗ (Z).
(Hint: Do not forget to add Lagrange multipliers to make sure that Q∗ is a valid distribution.)
• Show that if we optimize with respect to Q and θ in an alternating manner, this corresponds to the EM procedure.
Discuss what this implies for the convergence properties of EM.
Solution:
For the first part, note that
ℓ(θ) = log P (x; θ) = log
≥ EZ∼Q log
m
X
z=1
P (x, z; θ) = log
m
X
P (x, z; θ)
z=1
Q(z)
Q(z) = log EZ∼Q
m
X
P (x, z; θ)
= EZ∼Q [log P (x, z; θ)] −
Q(z) log Q(z),
Q(z)
z=1
Page 41
P (x, z; θ)
Q(z)
where for the inequality, we have used Jensen’s inequality. Now, assume that we want to maximize the above with respect
to Q, and let us add a multiplier λ to make sure that Q sums up to 1. Then, we have the following Lagrangian
!
m
m
m
X
X
X
L(Q, λ) =
Q(z) log P (x, z; θ) −
Q(z) log Q(z) + λ
Q(z) − 1 .
z=1
z=1
z=1
By setting the derivative of the Lagrangian with respect to Q(z) to zero, we have
∂
L(Q, λ) = log P (x, z; θ) − 1 − log Q(z) + λ = 0 =⇒ Q(z) = eλ−1 P (x, z; θ)
∂Q(z)
Hence, we have that Q(z) ∝ P (x, z; θ) and this is exactly the posterior P (Z | x; θ), which we had to show. It is also easy
to see that the bound is tight, as
X
m
m
P (x, z; θ) X
P (z | x; θ)P (x; θ)
P (x, z; θ)
EZ∼Q log
=
Q(z) log
=
P (z | x; θ) log
= log P (x; θ)
Q(z)
Q(z)
P (z | x; θ)
z=1
z=1
Then we can easily see the EM algorithm as optimizing the lower bound with respect to Q(·) and θ in an alternating
manner. Specifically, if we optimize with respect to Q, we have shown that the optimal Q is posterior, and this is exactly
the E-step. Optimizing with respect to θ for fixed Q is clearly equivalent to the M-step. The EM algorithm has to converge
as the lower bound is monotonically increased at every step.
Problem 38. (EM for Censored Linear Regression)
Suppose you are trying to learn a model that can predict how long a program will take to run for different settings. In
some situations, when the program is taking too long, you abort the program and note down the time at which you abort
it. These values are lower bounds for the actual running time of the program. We call this type of data right-censored.
Concretely, all you know is that the running time yi ≥ ci , where ci is the censoring time. Written in another way, one
can say yi = min {zi , ci } where zi is the true running time. Our goal is to derive an EM algorithm for fitting a linear
regression model to right-censored data.
(a) Let zi = µi + σεi , where εi ∼ N (0, 1). Suppose that we do not observe zi , but we observe the fact that it is higher
than some threshold. Namely, we observe the event E = I (zi ≥ ci ). Show that
c i − µi
E [zi | zi ≥ ci ] = µi + σR
σ
and
E zi2 | zi ≥ ci = µ2i + σ 2 + σ (ci + µi ) R
where we have defined
R(x) :=
c i − µi
σ
,
ϕ(x)
.
1 − Φ(x)
Here, ϕ(x) is the pdf of the standard Gaussian, and Φ(x) is its cdf.
(b) Derive the EM algorithm for fitting a linear regression model to right-censored data. Describe completely the E-step
and M-step.
Solution:
ci −µi
i ,E)
(a) First note that p (εi | E) = p(ε
p(E) . Also for brevity, define ai :=
σ . Then we have E = I (zi ≥ ci ) = I (εi ≥ ai ).
So we can write
Z
Z
p (εi , E)
zi p (εi | E) dεi =
zi
dεi
p(E)
R
R
Z ∞
Z ∞
σ
1
(µi + σεi ) p (εi ) dεi = µi +
εi p (εi ) dεi
=
p(E) ai
p(E) ai
E [zi | zi ≥ ci ] =
Page 42
The equality follows from the fact that p(E) =
distribution density ϕ(x), we have
R∞
ai
p (εi ) dεi = 1 − Φ (ai ). Now observe that for the Standard Normal
d
ϕ(x) = −xϕ(x),
dx
implying that
Z ∞
εi p (εi ) dεi = ϕ (ai ) − ϕ(+∞) = ϕ (ai ) .
ai
Putting it all together we get
E [zi | zi ≥ ci ] = µi + σ
ϕ (ai )
= µi + σR
1 − Φ (ai )
c i − µi
σ
.
To compute E zi2 | zi ≥ ci , we first note that
d2
ϕ(x) = −ϕ(x) + x2 ϕ(x),
dx2
implying that
Z b
x2 ϕ(x)dx = Φ(b) − Φ(a) + aϕ(a) − bϕ(b).
(21)
a
Now we have
E zi2 | zi ≥ ci =
1
p(E)
Z ∞
ai
2µi σ
= µ2i +
p(E)
µ2i + 2µi σεi + σ 2 ε2i p (εi ) dεi
Z ∞
ai
σ2
εi p (εi ) dεi +
p(E)
Z ∞
ε2i p (εi ) dεi
ai
2µi σ
σ2
ϕ (ai ) +
(1 − Φ (ai ) + ai ϕ (ai ))
p(E)
p(E)
σ2
2µi σ
ϕ (ai ) +
(1 − Φ (ai ) + ai ϕ (ai ))
= µ2i +
1 − Φ (ai )
1 − Φ (ai )
= µ2i + σ 2 + 2µi σ + ai σ 2 R (ai )
= µ2i +
= µ2i + σ 2 + σ (µi + ci ) R (ai )
(b) The model we have for linear regression is zi ∼ N w⊤ xi , σ where zi is missing. Our observed variable is yi =
min {zi , ci }. For ease of notation, let
(
1 if zi ≤ ci
di =
0 if zi > ci
to be the censoring indicator, i.e., it is 1 if the observation is not censored, and is 0 otherwise. We denote by z the
set of all zi ’s, by X the set of all xi ’s, by y the set of all yi ’s, by c the set of all ci ’s, and by d the set of all di ’s.
The complete-data log-likelihood would be
log p (zi | w) = −
2
1
zi − w⊤ xi + const.
2
2σ
For the first step, we need to find the posterior of the missing data given the observed data and parameters. We
have


if di = 1
 δ (zi − y⊤i )
N (zi |w xi ,σ )
p(zi | xi , yi , ci , di , w) =
(
)
if di = 0 ,
| {z }

 1−Φ ci −wσ⊤ xi
observed
Page 43
ci −µi
σ
in which δ(·) is the dirac delta function, and 1 − Φ
is the probability that zi > ci .
Now we should compute the expected value of the complete-data log-likelihood w.r.t the posterior p (zi | xi , yi , ci , di , w′ ).
This can be computed as
Z
R
log p (zi | w) · p (zi | xi , yi , ci , di , w′ ) dzi .
2
Note that if di = 1, the integral is evaluated as − 2σ1 2 yi − w⊤ xi , and if di = 0, we can use part (a) to compute
the expectation. For ease of notation, we call µi := w⊤ xi and µ′i := w′⊤ xi and ai =
ci −µ′i
σ . We then have
1 2
µi + E zi2 | zi > ci − 2µi E [zi | zi > ci ]
2
2σ 


1  2
2
= − 2 µi + µ′i + R (ai ) − 2µi ((µ′i ) + σ 2 + σ (µ′i + ci ) R (ai )) .
{z
}
|
2σ 
E [log p (zi | w) | zi > ci ] = −
:=bi
Adding the evaluated expectation for all data and removing the terms that are not dependent on w, we get
Q (w, w′ ) = −
n
1 X
2
(yi − µi ) · di + µ2i − 2bi µi · (1 − di )
2
2σ i=1
1 X 2
.
=− 2
µi − 2yi µi · di + µ2i − 2bi µi · (1 − di )
2σ i=1
n
=−
n
1 X 2
µ − 2µi (yi di + bi (1 − di ))
{z
}
|
2σ 2 i=1 i
:=ei
1
= − 2 w⊤ X⊤ Xw − 2w⊤ X⊤ e
2σ
The maximizer for Q (w, w′ ) would be
w⋆ = −
−1
1
X⊤ e X⊤ X
,
2
which sums up the M-step.
Problem 39. (Yet another perspective on EM)
The EM algorithm is a general technique for finding maximum likelihood solutions for probabilistic models having latent
variables. Take a probabilistic model in which we denote all of the observed variables as X and all of the hidden variables
as Z (here we assume Z is discrete, for the sake of simplicity). Let us assume that the joint distribution is p(X, Z | θ),
where θ is the set of all parameters describing this distribution (e.g. for a Gaussian distribution, θ = (µ, Σ)). The goal is
to maximize the likelihood function
X
p(X | θ) =
p(X, Z | θ).
Z
(a) For an arbitrary distribution q(Z) over the latent variables, show that the following decomposition holds:
ln p(X | θ) = L(q, θ) + DKL (q∥ppost )
(22)
where ppost = p(Z | X, θ) is the posterior distribution. Also find the formulation of L(q, θ).
(b) Verify that L(q, θ) ≤ ln p(X | θ), and that equality holds if and only if q(Z) = p(Z | X, θ).
(c) Suppose that the current value of the parameters is θcurr . Verify that in the E-step, the lower bound L (q, θ curr ) is
maximized with respect to the distribution q(Z), while keeping θ curr fixed. Since the lefthand-side of Eqn. (22) does
not depend on q(Z), maximizing L (q, θ curr ) will result in minimizing the KL divergence between q and ppost , which
happens at q ⋆ = ppost .
Page 44
(d) Verify that in the M-step, the lower bound L(q, θ) is maximized with respect to θ while keeping q(Z) fixed, resulting
in a new value of parameters θnew . This step will increase the left-hand side of Eqn. (22) (if it is not already in a
local maximum).
(e) Substitute q(Z) = p (Z | X, θ curr ) in Eqn. (22), and observe that
L(q, θ) = Eq [complete-data log likelihood ] − H(q).
In other words, in the M-step, we are maximizing the expectation of the complete-data log-likelihood (p(X, Z|θ))
since the entropy term is independent of θ. Compare this result with the EM for Gaussian mixture models.
(f) Show that the lower bound L(q, θ), where q(Z) = q ⋆ (Z) = p (Z | X, θ curr ), has the same gradient w.r.t. θ as the
log likelihood function p(X | θ) at the point θ = θ curr . This shows that the lower bound becomes tangent to the
log-likelihood function at the end of the E-step.
Solution:
(a) By computing the KL Divergence of q to ppost we get
DKL (q∥ppost ) =
X
q(Z) log
Z
q(Z)
.
p(Z | X, θ)
Knowing that p(Z | X, θ) = p(X,Z|θ)
p(X|θ) , we get
DKL (q∥ppost ) =
X
q(Z) log
Z
q(Z)p(X | θ) X
q(Z)
=
+ log p(X | θ).
q(Z) log
p(Z, X | θ)
p(Z, X | θ)
Z
This implies that
log p(X | θ) = DKL (q∥ppost ) + L(q, θ),
where
L(q, θ) =
X
q(Z) log
Z
p(Z, X | θ)
.
q(Z)
(23)
(b) Since KL divergence is always nonnegative and is zero only if the distributions are the same, we have
L(q, θ) ≤ log p(X | θ).
with equality iff q = ppost .
(c) This is for you to verify. As seen in the examples in the slides of the course, in some situations, the E-step is just
computing the posterior, which is equivalent to minimizing the KL divergence of q to the posterior.
(d) Another thing to verify by yourself. Look at GMMs as an example and try to relate the variables defined in here
and the parameters and variables there.
(e) Putting q = ppost in Eqn. (23) we get
X
p(Z, X | θ)
p(Z | X, θ)
Z
X
X
=
p(Z | X, θ) log p(Z, X | θ) −
p(Z | X, θ) log p(Z | X, θ)
L (ppost , θ) =
p (Z | X, θ curr ) log
Z
Z
= Eq [complete-data log likelihood] + H(q)
Page 45
(f) We have
∇θ L(q, θ)|θ=θcurr = ∇θ
X
q(Z) log
Z
=
X
q(Z)
Z
=
p(Z, X | θ)
q(Z)
∇θ p(Z, X | θ)|θ=θcurr
p (Z, X | θ curr )
X ∇θ p(Z, X | θ)|θ=θ
curr
p (X | θ curr )
Z
∇θ p(X | θ)|θ=θcurr
=
= ∇θ log p(X | θ)|θ=θcurr
p (X | θ curr )
Problem 40. (Generative meets Discriminative)
Consider a generative classification model for binary classification. Assume the class-marginal distribution to be defined
as p(y = 1) = π and assume each class-conditional distribution to be defined as a product of D Bernoulli distributions,
QD
QD
i.e., p(x | y = 1) = d=1 p (xd | y = 1) where p (xd | y = 1) = Bernoulli (xd | µd,1 ), and p(x | y = 0) = d=1 p (xd | y = 0)
where p (xd | y = 0) = Bernoulli (xd | µd,0 ). Note that this makes use of the naïve Bayes assumption.
Show that this model is equivalent (in its mathematical form) to a probabilistic discriminative classifier. In particular,
derive the expression for p(y = 1 | x), and state what type of decision boundary this model will learn - linear, quadratic,
or something else (looking at the expression of p(y = 1 | x) should reveal that)? Clearly write down the expressions for the
parameters of the equivalent probabilistic discriminative model in terms of the generative model parameters (π, µd,0 , µd,1 ).
Note that you do not have to estimate the parameters π, µd,0 , µd,1 (but you may try that for practice if you want).
Solution:
Consider a generative classification model for binary classification with the Naive Bayes assumption of feature independence. Let the class marginal and class conditional distributions be as follows:
p(y = 1) = π,
p(x | y = j) =
D
Y
Bernoulli (xd | µd,j ) =
d=1
D
Y
d
µxd,j
(1 − µd,j )
1−xd
,
j = 0, 1
d=1
For convenience, let us consider simplification of only p(y = 1 | x).
p(x | y = 1)p(y = 1)
p(x | y = 1)p(y = 1)
=
p(x)
p(x | y = 1)p(y = 1) + p(x | y = 0)p(y = 0)
QD
π d=1 B (xd , µd,1 )
∴ p(y = 1 | x) = QD
;
where B denotes Bernoulli
QD
π d=1 B (xd , µd,1 ) + (1 − π) d=1 B (xd , µd,0 )
1
1
∏D
∴ p(y = 1 | x) =
=
QD µd0 xd 1−µd0 1−xd
1−π ∏d=1 B(xd ,µd,1 )
1−π
1+ π
D
1+ π
d=1 µd1
1−µd1
d=1 B(xd ,µd,0 )
p(y = 1 | x) =
1−π
d0
Now, let sd = µµd0
, rd = 1−µ
1−µd1 and C = π . On further simplification, we get
d1
p(y = 1 | x) =
1+C
QD
1
xd 1−xd
d=1 sd rd
=
1 + exp log(C) +
PD
1
d=1 xd log (sd ) + (1 − xd ) log (rd )
1
,
1 + exp (wT x + b)
Q
T
D
r
.
where w = [log (s1 ) − log (r1 ) , . . . , log (sD ) − log (rD )] and b = log C
d
d=1
∴ p(y = 1 | x) =
Thus, this corresponds to the discriminative logistic regression model for binary classification with labels ∈ {0, 1}. The
decision boundary is linear.
Page 46
Problem 41. (K-means convergence)
In the K-means clustering algorithm, you are given a set of n points xi ∈ Rd , i ∈ {1, . . . , n} and you want to find the
centers of k clusters µ = (µ1 , . . . , µk ) by minimizing the average distance from the points to the closest cluster center.
Formally, you want to minimize the following loss function
L(µ) =
n
X
i=1
2
min
j∈{1,...,k}
∥xi − µj ∥2 .
2
To approximate the solution, we introduce new assignment variables zi ∈ arg minj∈{1,...,k} ∥xi − µj ∥2 for each data point
xi . The K-means
P algorithm iterates between updating the variables zi (assignment step) and updating the centers
µj = |{i:zi1=j}| i:zi =j xi (refitting step). The algorithm stops when no change occurs during the assignment step. Show
that K-means are guaranteed to converge (to a local optimum).
Hint: You need to prove that the loss function is guaranteed to decrease monotonically in each iteration until convergence.
Prove this separately for the assignment step and the refitting step.
Solution:
To prove the convergence of the K-means algorithm, we show that the loss function is guaranteed to decrease monotonically
in each iteration until convergence for the assignment step and for the refitting step. Since the loss function is non-negative,
the algorithm will eventually converge when the loss function reaches its (local) minimum.
Let z = (z1 , . . . , zn ) denote the cluster assignments for the n points.
(i) Assignment step
We can write down the original loss function L(µ) as follows:
L(µ, z) =
n
X
2
∥xi − µzi ∥2
i=1
Let us consider a data point xi , and let zi be the assignment from the previous iteration and zi∗ be the new assignment
obtained as:
2
zi∗ ∈ arg min ∥xi − µj ∥2
j∈{1,...,k}
Let z ∗ denote the new cluster assignments for all the n points. The change in loss function after this assignment
step is then given by:
n X
2
2
xi − µzi∗ 2 − ∥xi − µzi ∥2 ≤ 0
L (µ, z ∗ ) − L(µ, z) =
i=1
The inequality holds by the rule zi∗ is determined, i.e. to assign xi to the nearest cluster.
(ii) Refitting step
We can write down the original loss function L(µ) as follows:


k
X
X
2

∥xi − µj ∥2 
L(µ, z) =
j=1
i:zi =j
Let us consider the j th cluster, and let µj be the cluster center from the previous iteration and µ∗j be the new cluster
center obtained as:
X
1
µ∗j =
xi
|{i : zi = j}| i:z =j
i
Let µ∗ denote the new cluster centers for all the k clusters. The change in loss function after this refitting step is
then given by:

 

k
X
X
X
2
2

L (µ∗ , z) − L(µ, z) =
xi − µ∗j 2  − 
∥xi − µj ∥2  ≤ 0
j=1
i:zi =j
i:zi =j
The inequality holds because the update rule of µ∗j essentially minimizes this quantity.
Page 47
Problem 42. (K-medians clustering)
In this exercise, you are asked to derive a new clustering algorithm that would use a different loss function given by
L(µ) =
n
X
i=1
min
j∈{1,...,k}
∥xi − µj ∥1 .
(i) Find the update steps for both zi and for µj in this case.
(ii) What can you say about the convergence of your algorithm?
(iii) In which situation would you prefer to use K-medians clustering instead of K-means clustering?
Solution:
(i) As in the K-means algorithm, let’s again introduce hidden variables zi = arg minj∈1,...,k ∥xi − µj ∥1 for each data
point xi . Then the initial problem
n
X
µ = arg min
min ∥xi − µj ∥1
µ
j∈1,...,k
i=1
can be rewritten in a different form (because we know where exactly the minimum is achieved):
µ = arg min
µ
n
X
∥xi − µzi ∥1
i=1
In order to find the solution with respect to µj with fixed zi , let’s leave only the data points that correspond to the
j th component:
d
X X
X
|xi,q − µj,q |
∥xi − µj ∥1 µj = arg min
µj = arg min
µj
µj
i:zi =j
i:zi =j q=1
This can again be separated component-wise:
X
µj,q = arg min
µj,q
|xi,q − µj,q |
i:zi =j
Again, as in the K-means algorithm, we find the derivative of the function and set it to zero. In order to get rid of
the L1 norm, we also separate the functional into the sum over those xi,q that are smaller than µj,q and those that
are larger:
P
P
P
P
i:zi =j,xi,q ≤µj,q |xi,q − µj,q |+
i:zi =j,xi,q >µj,q |xi,q − µj,q | =
i:zi =j,xi,q ≤µj,q (µj,q − xi,q )+
i:zi =j,xi,q >µj,q (xi,q − µj,q )
The derivative of every bracket in the sum is either +1 or −1, and the number of +1’s is exactly | {i : zi =
j, xi,q ≤ µj,q } |. Therefore, we need to set
|{i : zi = j, xi,q ≤ µj,q }| − |{i : zi = j, xi,q > µj,q }| = 0
This means that µj,q is nothing but the median of all the numbers xi,q , i : zi = j.
The resulting algorithm then iterates between two steps:
• zi = arg minj∈1,...,k ∥xi − µj ∥1
• µj,q = median (xi,q , i : zi = j) , ∀j = 1, . . . , k; ∀q = 1, . . . , d.
(ii) You can prove the same convergence properties for K-medians as for K-means.
(iii) In comparison with K-means, K-medians clustering is particularly robust to outliers. Thus, if we expect our input
data to have many outliers, it is preferable to use K-medians clustering.
Page 48
Problem 43. (PCA)
Suppose we have a dataset with 4 points:
D = {(1, 5), (0, 6), (−7, 0), (−6, −1)}
(a) Plot the dataset and try to guess two principal components (k = 2).
(b) Compute the empirical covariance matrix, its eigenvalues and eigenvectors. Do the eigenvectors correspond to your
guess of principal components? Please do not forget the assumptions of PCA. (The dataset should be centered and
we want unit eigenvectors.)
Solution:
(a) Plot of the original dataset:
(b) We first need to center the data by subtracting from it its mean (−3, 2.5)T , obtaining
x1 = (4, 2.5)T ,
x2 = (3, 3.5)T ,
x3 = (−4, −2.5)T ,
x4 = (−3, −3.5)T .
The plot of the centered dataset:
For the empirical covariance matrix, we obtain
1X
1
xi xTi = ·
n i=1
4
n
Σ=
50
41
41
37
=
12.5 10.25
10.25 9.25
The unit-length eigenvectors of Σ are v1 = (0.76045416, 0.64939162)T and v2 = (−0.64939162, 0.76045416)T with
eigenvalues w1 = 21.25301161 and w2 = 0.49698839, respectively.
Plot of the centered dataset with principal components 1 (red) and 2 (blue):
Problem 44. (Clustering - Within and Across)
Suppose we wish to cluster some data by learning a function f such that fn = f (xn ) is the cluster assignment for point
xn . Show that finding f by minimizing LW , which is defined as the sum of squared distances between all pairs of points
that are within the same cluster, i.e.,
Page 49
arg min LW = arg min
f
f
X
2
I [fn = fm ] ∥xn − xm ∥
n,m
implicitly also maximizes the sum of squared distances between all pairs of points that are in different clusters. (Note:
You can also show that the above is equivalent to the K-means objective!)
Solution:
Let fˆ be the function learned by minimizing LW , which is defined as the sum of squared distances between all pairs of
points that are within the same cluster, i.e.,
X
2
fˆ = arg min
I (fn = fm ) ∥xn − xm ∥
f
n,m
Note that we can write the indicator function as follows:
I (fn = fm ) = 1 − I (fn ̸= fm )
Thus, replacing the loss function, we obtain
"
#
X
X
2
2
ˆ
f = arg min
∥xn − xm ∥ −
I (fn ̸= fm ) ∥xn − xm ∥
f
n,m
n,m
We can rewrite it as follows since the first term does not involve the argument for minimization. Essentially, it remains
to be a constant w.r.t the minimization.
X
X
2
2
∴ fˆ =
∥xn − xm ∥ − arg min
I (fn ̸= fm ) ∥xn − xm ∥
f
n,m
∴ fˆ = − arg min
f
∴ fˆ = arg max
f
X
n,m
2
I (fn ̸= fm ) ∥xn − xm ∥
n,m
X
2
I (fn ̸= fm ) ∥xn − xm ∥
n,m
Hence, it turns out to be equivalent to maximizing the inter-cluster distances - the distance between points of different
clusters.
Problem 45. (Eigenchangers!)
Suppose we wish to do PCA for an N × D matrix X and assume D > N . The traditional way to do PCA is to compute
the eigenvectors of the covariance matrix S = N1 X⊤ X (assuming centered data). Show that, if someone instead gives you
an eigenvector v ∈ RN of the matrix N1 XX⊤ , you can use it to get an eigenvector u ∈ RD of S. What is the advantage
of this way of obtaining the eigenvectors of S?
Solution:
Page 50
Let X be the N × D data matrix. The covariance matrix is given by (assuming centered data) S = N1 X T X. Let
T = N1 XX T . Suppose λ is an eigenvalue and v is an eigenvector of T , then my claim is that λ is also an eigenvalue of S
with corresponding eigenvector being X T v. The proof follows:
T v = λv
1
∴ XX T v = λv
N
1 T
∴ X XX T v = λX T v
N
∴ S XT v = λ XT v
Let u := X T v. We have Su = λu. Therefore, u turns out to be the eigenvector for S corresponding to the eigenvalue λ.
Thus, if we know the eigenvectors of matrix T , we can find the eigenvectors of matrix S by simple matrix multiplication,
which is O(N D). The advantage of using this approach to obtain the eigenvectors is that we will need to diagonalize the
N × N matrix T instead of the D × D matrix S for getting eigenvectors. Note that we have been given N < D. Thus,
obtaining eigenvectors in this manner is computationally cheaper when D > N . Also, note that we can kernelize the
matrix T and then conduct eigendecomposition of the kernel matrix enabling us to do non-linear PCA.
Problem 46. (Soft k-means, Revisited)
(a) Consider the following optimization problem:
max
c∈Rk
k
X
vi log (ci )
s.t.
ci > 0,
i=1
k
X
ci = 1,
i=1
where v ∈ Rk+ is a vector of non-negative weights. Check that the M-step of soft k-means includes solving such an
optimization problem.
(b) Let c⋆ = ∑1 vi v. Verify that c⋆ is a probability vector.
i
(c) Show that the optimization problem is equivalent to the following problem:
min DKL (c ∥c)
⋆
c∈Rk
s.t.
ci > 0,
k
X
ci = 1.
i=1
(d) Using the properties of KL divergence, prove that c⋆ is indeed the solution to the optimization problem.
Solution:
(a) Check class notes.
(b) The components of c⋆ are non-negative (since v is non-negative), and add up to 1.
(c) Since the optimization is over c, it makes no difference if we divide it by a positive number or add/subtract terms
P
Pk
that are not dependent on c. We first divide the objective by i vi and then subtract from the sum i=1 c⋆i log c⋆i .
We get
k
X
i=1
c⋆i log (ci ) −
k
X
c⋆i log (c⋆i ) =
i=1
k
X
i=1
c⋆i log
ci
= −DKL (c⋆ ∥c) .
c⋆i
Thus maximizing the objective is equivalent to minimizing DKL (c⋆ ∥c).
(d) Since KL Divergence is always non-negative and is zero if and only if the two distributions are equal, we get that
the optimal solution to the optimization problem is indeed c = c⋆ .
Page 51
Problem 47. (Sigmoidal function)
Let Y ∈ {0, 1} denote a binary random variable that depends on k other random variables Xi as:
!
k
X
1
P (Y = 1 | X1 = x1 , X2 = x2 , . . . , Xk = xk ) = σ
w i xi
where σ(z) =
.
1 + e−z
i=1
The real-valued parameters wi in this CPT are known as weights. The so-called sigmoid function σ(z) arises in many
contexts. In neural networks, it models the probability that a neuron Y fires given its input from other neurons Xi ; the
weights wi describe the connections between neurons. In statistics, the sigmoid function appears in models of logistic
regression. Sketch the function σ(z), and verify the following properties:
(a) σ ′ (z) = σ(z)σ(−z).
(b) σ(−z) + σ(z) = 1.
(c) L(σ(z)) = z, where L(p) = log
p
1−p
is the log-odds function.
(d) wi = L (pi ), where pi = P (Y = 1 | Xi = 1, Xj = 0 for all j ̸= i).
Solution:
(a)
σ ′ (z) =
0 + 1 · −e−z
−z 2
=
1
−e−z
·
−z
1+e
1 + e−z
(1 + e )
1
1
=
·
= σ(z) · σ(−z)
1 + e−z ez + 1
(b)
σ(−z) + σ(z) =
(c)
L(σ(z)) = log
(d)
L (pi ) = log
1
1
1 + e−z + 1 + ez
+
=
=1
z
−z
1+e
1+e
1 + ez + e−z + e0
σ(z)
1 − σ(z)
= log
1
1+e−z
−z
e
1+e−z
!
1
= log
e−z
P (Y = 1 | Xi = 1, Xj = 0, ∀j ̸= i)
1 − P (Y = 1 | Xi = 1, Xj = 0, ∀j ̸= i)
= log
= log (ez ) = z
σ (wi )
1 − σ (wi )
= L (σ (wi ))
According to the conclusion from (c), L (σ (wi )) = wi .
Problem 48. (A General Activation Function)
1
Consider the following activation function: h(x) = xσ(βx) where σ denotes the sigmoid function σ(z) = 1+exp(−z)
. Show
that, for appropriately chosen values of β, this activation function can approximate (1) the linear activation function and
(2) the ReLU activation function.
Solution:
Given h(x) = xσ(βx) where σ is the usual sigmoid activation function.
h(x) =
x
1 + exp(−βx)
Case 1: Linear approximation
Take β = 0, we will have
Page 52
h(x) =
x
2
[Linear]
Case 2: Approximating ReLU
Take β → ∞, we will have exp(−βx) → ∞ for all x < 0 and exp(−βx) → 0 for x ≥ 0. Thus, we get
(
0 ∀x < 0
h(x) =
x ∀x ≥ 0
Hence, we can approximate ReLU using the given activation function h(x).
Problem 49. (Recurrent Neural Networks)
We saw feedforward artificial neural networks, which do not contain any cycles and for which the nodes do not maintain a
persistent state over several runs. This exercise considers artificial neural networks with nodes that maintain a persistent
state that can be updated. This kind of neural network is called a recurrent neural network (RNN). As an example,
consider the following RNN with
yt = W x t + V s t
st+1 = yt
from some initial state s0 , where t denotes the t th call of the RNN, i.e., xt is the tth input.
(a) What is the recurrent state in the RNN from Figure 1? Name one example that can be more naturally modeled
with RNNs than with feedforward neural networks.
(b) As the state of an RNN changes over different runs of the RNN, the loss functions that we use for feedforward neural
networks do not yield consistent results. For the given dataset X, please propose a loss function ( based on the
mean square loss function) for RNNs and justify why you chose this loss function.
k
(c) For a dataset X := (xt , yt )1 (for some k ∈ N ), show how information is propagated by drawing a feedforward neural
network that corresponds to the RNN from Figure 1 for k = 3. Recall that a feedforward neural network does not
contain nodes with a persistent state. (Hint: unfold the RNN.)
Solution:
(a) The recurrent state is denoted s. In this case, it coincides with the output. Recurrent models are used to model
data with temporal structure, e.g., time series, speech, and sound.
(b) We have a data X = {(xt , yt )}, where we assume that the data is ordered temporily. Thus, we define the loss
PT
function to be L (U, W, s0 ) = t=1 (y(t) − f (xt , st−1 (U, W ), U, W ) , where st is the previous reccurent state. The
initial state s0 needs to be specified, and the problem also depends on it.
(c) Check Figure 2.
Page 53
Problem 50. (Mixtures meet Neural Nets!)
N
Consider modeling some data {(xn , yn )}n=1 , xn ∈ RD , yn ∈ {0, 1}, using a mixture of logistic regression models, where we
model each binary label yn by first picking one of the K logistic regression models, based on the value of a latent variable
zn ∼ multinoulli (π1 , . . . , πK ), and then generating yn conditioned on zn as yn ∼ Bernoulli σ w⊤
zn xn .
Now, consider the marginal probability of the label yn = 1, given xn , i.e., p (yn = 1 | xn ), and show that this can also be
thought of as the output of a neural network. Clearly specify the input layer, hidden layer(s), activations, output layer,
and connection weights of this neural network.
Solution:
N
We have been given data {(xn , yn )}n=1 , xn ∈ R2 , yn ∈ {0, 1}. We are given
zn ∼ multinoulli (π1 , π2 , .., πK )
yn ∼ Bernoulli σ wzTn xn
We want to estimate p (yn = 1 | xn ):
p (yn = 1 | xn ) =
PK
k=1 p (yn = 1, zn = k | xn ) =
PK
k=1 p (yn = 1 | zn = k, xn ) p (zn = k) =
p (yn = 1 | xn ) =
K
X
PK
k=1 σ
wkT xn πk
σ wkT xn πk
k=1
We can think of this as a neural network in the following way:
• Input layer: (x1 , x2 , .., xD ) , x ∈ RD is the input example.
• Hidden layer: We consider a single hidden layer of size K with each hidden node
output σ wkT x . The
T k having
T
weight matrix will be W = [wij ] , i ∈ {1, 2, . . . , D}, j ∈ {1, 2, . . . , K} i.e. W = w1 w2T . . . wK
• Final layer: The output layer will consist of only a single node whose output will be p(y = 1 | x). The weight vector
T
U = π1 π2 . . . π K
.
Page 54