18.657 PS 2 SOLUTIONS
1. Problem 1
(1) Let Y1 , . . . , Yn be ghost copies of X1 , . . . , Xn . Then we have
#
n
n
#
"
"
1 X
1 X
σi [f (Xi ) − E[f (X)]] = E sup E sup σi [f (Xi ) − E[f (Yi )]]
f ∈F n i=1
f ∈F n i=1
#
"
n
1 X
σi [f (Xi ) − f (Yi )] ,
≤ E sup n
f ∈F
i=1
by Jensen’s inequality,
n
#
1 X
[f (Xi ) − f (Yi )] ,
= E sup f ∈F n i=1
"
as σi [f (Xi ) − f (Yi )] and f (Xi ) − f (Yi ) have the same distribution,
#
"
n
1 X
= E sup [(f (Xi ) − E[f (Xi )]) − (f (Yi ) − E[f (Yi )])]
f ∈F n i=1
#
n
"
1 X
[f (Xi ) − E[f (Xi )]] ,
≤ 2E sup n
f ∈F
i=1
by the triangle inequality.
(2) The distribution of gi is the same as that of |gi |σi , so we can write
"
#
"
"
##
n
n
X
X
E sup
gi f (Xi ) = EXi ,σi Egi sup
|gi |σi f (Xi ) | Xi , σi
f ∈F i=1
"
≥ EXi ,σi sup
f ∈F i=1
n
X
f ∈F i=1
E[|gi |]σi f (Xi ) ,
by Jensen’s inequality,
r
=
"
#
n
X
2
E sup
σi f (Xi ) ,
π
f ∈F i=1
using the first absolute moment of the standard Gaussian.
Date: Fall 2015.
1
#
2
18.657 PS 2 SOLUTIONS
(3) We compute:
#
n
1 X
Rn (F + h) = E sup σi (f (Xi ) + h(Xi ))
n
f ∈F
#
n
i=1
"
#
"
n
1 X
1 X
h(Xi ) ,
≤ E sup σi f (Xi ) + E
n
f ∈F n "
i=1
i=1
by the triangle inequality,
#
n
1 X
h(Xi ) ,
= Rn (F) + E
n i=1
v 
u
!2 
n
u
1u  X
σi h(Xi ) ,
≤ Rn (F) + tE
n
i=1
"
by Jensen’s inequality,
v
uX
n
X
1u
= Rn (F) + t
E[h(Xi )2 ] + 2
E[σi σj h(Xi )h(Xj )]
n i=1
i<j
v
u n
u
1 X
= Rn (F) + t
E[h(Xi )2 ],
n i=1
by symmetry,
1p
nkhk2∞
n
khk∞
= Rn (F) +
.
n
≤ Rn (F) +
(4) By the triangle inequality, we have

n
k
X
X
1
Rn (F1 + . . . + Fk ) = E  sup
σi
fj (Xi )
fj ∈Fj n i=1
j=1
#
"
k
n
X
1 X
≤
E sup
σi fj (Xi )
fj ∈Fj n 
j=1
=
k
X
j=1
i=1
Rn (Fj ).
18.657 PS 2 SOLUTIONS
3
(5) The supremum over different choices of fj is at least the supremum over a single repeated
choice:


X
k
X
n
1

Rn (F + . . . + F ) = E sup σi
fj (Xi )
|
{z
}
fj ∈F n i=1
j=1
k


n
k
1 X X

σi
f (Xi )
≥ E sup f ∈F n i=1
j=1
n
#
"
1 X
= kE sup σi f (Xi )
f ∈F n i=1
= kRn (F),
which provides the reverse bound to that of (4), establishing equality in this case.
4
18.657 PS 2 SOLUTIONS
2. Problem 2
(1) Let A be a maximum 2ε-packing of (T, d); let B be a minimum ε-covering. As B is an εcovering, for each a ∈ A there exists some choice of b(a) ∈ B such that d(a, b(a)) ≤ ε. This
map b : A → B is an injection: by the triangle inequality, no point in B can be within ε of two
points of A, as these are at least 2ε apart. Thus |A| ≤ |B|, so that D(T, d, 2ε) ≤ N (T, d, ε).
Let C be maximum ε-packing. Note that C is in fact an ε-covering: if some point of T
had distance greater than ε to each point of C, then we could add it to C to produce a larger
ε-packing, contradicting maximality. It follows that N (T, d, ε) ≤ D(T, d, ε).
(2) Let B n denote the Euclidean ball in Rn . Let u ∈ B n and v ∈ B m . Then we can write
u = u∗ + εu , v = v∗ + εv , where u∗ ∈ Nn , v∗ ∈ Nm , εu ∈ 14 B n , and εv ∈ 14 B m . We compute:
kM k =
u> M v
sup
u∈B n ,v∈B m
=
sup
u∗ ,v∗ ,εu ,εv
>
u∗> M v∗ + ε>
u M v∗ + u M εv
≤
sup
u∗ ∈Nn ,v∗ ∈Nm
≤
max
u∗ ∈Nn ,v∗ ∈Nm
u∗> M v∗
u>
∗ M v∗
!
+
sup
εu ∈ 14 B n ,v∗ ∈Nm
ε>
u M v∗
!
+
sup
>
u M εv
u∈B n ,εv ∈ 14 B m
1
1
+ kM k + kM k.
4
4
Rearranging, we have
kM k ≤ 2
max
u∗ ∈Nn ,v∗ ∈Nm
u>
∗ M v∗
as desired.
(3) By independence and Hoeffding’s lemma, we have, for all s > 0,
Y
E[exp(s u> M v)] =
E[exp(s ui Mij vj )]
i,j
Y 1 2 2 2
≤
E exp
s ui vj
2
i,j



X
1
= E exp  s2
u2 v 2 
2 i,j i j
≤ exp(s2 /2),
whenever u ∈ B n and v ∈ B m .
d
Recall from the notes that B d has covering number at most 3ε , so that we are maximizing over |Nn × Nm | ≤ 12n+m points. It follows from the standard maximal inequality
for subgaussian random variables, or else by explicitly using Jensen with log and exp and
replacing a maximum by a sum, that
p
√
√
EkM k ≤ 2 2 log(12m+n ) ≤ C( m + n).
18.657 PS 2 SOLUTIONS
5
3. Problem 3
1
(1) Let A be an ε-net for [0, 1]; we can construct one with size at most 2ε
+ 1. Let Xpre denote
the set of non-decreasing functions from {x1 , . . . , xn , } to A, and let X be the set of functions
[0, 1] → [0, 1] defined by piecewise linear extension of functions in Xpre (and constant extension
x
on [0, x1 ] and [xn , 1]). It is straightforward to see that X is an ε-net for (F, d∞
): given f ∈ F ,
we define g ∈ Xpre by taking g(xi ) to be the least point of A lying within ε of f (xi ), and then
the function in X defined by extension of g is within ε of f .
1
It remains to count X. Let a1 < . . . < ak be the elements of A, where k ≤ 2ε
+1. Functions
f ∈ X are uniquely defined by the count for each 1 ≤ i ≤ k of how many xj satisfy f (xj ) = ai .
A naive count of the possibilities for these values yields
N (F, dx∞ , ε) ≤ |X| ≤ (n + 1)1+1/2ε ≤ n2ε ,
valid for n ≥ 3.
(2) As N (F, dx2 , ε) ≤ N (F, dx∞ , ε), and |f | ≤ 1 for all f ∈ F , the chaining bound yields
Z 1q
12
Rn ≤ inf 4ε + √
log N (F, dx2 , t) dt
ε>0
n ε
Z 1p
12
≤ inf 4ε + √
t−1 log n dt
ε>0
n ε
r
√
log n
= inf 4ε + 24
(1 − ε)
ε>0
n
r
√
log n
≤ lim 4ε + 24
(1 − ε)
ε→0
n
r
log n
= 24
.
n
(3) We bound N (F, dx1 , ε) ≤ N (F, dx∞ , ε) ≤ n2/ε . The theorem in Section 5.2.1 yields
r
2 log(2N (F, dx1 , ε))
Rn ≤ inf ε +
ε
n
r
−1
2ε log n + 2 log 2
≤ inf ε +
.
ε
n
Setting the two terms equal, to optimize over ε (in asymptotics), we have
r
r
2 log n + 2ε log 2
2 log n
3/2
ε
=
≈
,
n
n
for a bound of
1/3
log n
Rn .
,
n
which is strictly weaker than the chaining bound.
6
18.657 PS 2 SOLUTIONS
4. Problem 4
(1) We adapt the proof from class to the case of a penalized, rather than constrained, norm.
¯ n be the span of the functions k(xi , −). We can decompose any function g uniquely
Let W
¯ n and g ⊥ ⊥ W
¯ n . As in class, g ⊥ (xi ) = hg ⊥ , k(xi , −)i = 0, so
as g = gn + g ⊥ , where gn ∈ W
that g(xi ) = gn (xi ).
Plugging in to the objective function:
ψ(Y − g) + µkgk2W = ψ(Y − gn ) + µkgn k2W + µkg ⊥ k2W .
For any fixed gn , this is a constant plus µkg ⊥ k2W , which is minimized uniquely at g ⊥ = 0.
¯ n , as desired.
Thus (unfixing gn ) any minimizer fˆ must lie in W
Pn
>
ˆ
ˆ
ˆ2
¯
^
(2) As f ∈ Wn , write f =
i=1 θi k(xi , −), so that f = Kθ and kf kW = θ Kθ. Then the
first-order optimality conditions read
0 = ∇ φ(Y − ^
f) + µkfˆk2W
= ∇ Y> Σ−1/2 Y − 2Y> Σ−1/2 Kθ + θ> KΣ−1/2 Kθ + µθ> Kθ
= −2KΣ−1/2 Y + 2KΣ−1/2 Kθ + 2µKθ.
Rearranging, and recalling that Kθ = ^
f, we have
(KΣ−1/2 + µIn )^
f = KΣ−1/2 Y,
as desired.
(3) As above, it suffices to consider f , g ∈ W̄n , so that we can write f = Kθf , g = Kθg . From
the definition of fˆ, we have
ψ(f + ξ − ^
f) + µkfˆk2 ≤ ψ(f + ξ − g) + µkgk2 .
W
W
Separating out the contribution of ξ, we obtain
ψ(f − ^
f) + φ(ξ) + ξ > Σ−1/2 (f − ^
f) + µkfˆk2W ≤ ψ(f − g) + ψ(ξ) + ξ > Σ−1/2 (f − g) + µkgk2W .
Rearranging,
ψ(f − ^
f) ≤ −ξ > Σ−1/2 (f − ^
f) − µkfˆk2W + ψ(f − g) + ξ > Σ−1/2 (f − g) + µkgk2W
^ − g) − µkfˆk2W − µkgk2W
= ψ(f − g) + 2µkgk2W + ξ > Σ−1/2 (f
^ − g) − µ kfˆ − gk2W ,
≤ ψ(f − g) + 2µkgk2W + ξ > Σ−1/2 (f
2
2
2
2
by the inequality ka − bkW ≤ 2kakW + 2kbkW . Let Z = Σ−1/2 ξ, which is distributed as
N (0, I). Continuing the line of algebra,
µ
ψ(f − ^
f) ≤ ψ(f − g) + 2µkgk2W + Z > (^
f − g) − kfˆ − gk2W
2
µ
2
>
= ψ(f − g) + 2µkgkW + Z K(θfˆ − θg ) − (θfˆ − θg )> K(θfˆ − θg )
2
1 >
2
≤ ψ(f − g) + 2µkgkW + Z KZ,
µ
by the inequality
µ
1
a> P b ≤ a> P a + b> P b
2
µ
for any PSD matrix P . Now since
n
n
X
X
Z > KZ =
Zi hk(xi , −), k(xj , −)iW = k
Zi k(xi , −)k2W ,
i,j =1
i=1
¯ n , which equals the infimum over g ∈ W .
we conclude by taking the infimum over g ∈ W
18.657 PS 2 SOLUTIONS
7
(4) This follows from the previous part together with the observation


2
n
n
X
X
X
E[Zi2 ]Kii = Tr(K).
Zi k(xi , −) = E 
Zi Zj Kij  =
E
i=1
W
i,j
i=1
¯ n , since only its evaluations at the design points
(5) It is sufficient to prove the bound for f ∈ W
matter. For the Gaussian kernel we have k(x, x) = 1, so that Tr(K) = n. Applying the
previous part, and taking the case g = f as an upper bound on the minimizer, we have
^)] ≤ ψ(f − f) + 2µkf k2W + n = 2µkf k2W + n .
E[ψ(f − f
µ
µ
p
Taking µ = kf kW n/2 gives the result.
MIT OpenCourseWare
http://ocw.mit.edu
18.657 Mathematics of Machine Learning
Fall 2015
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.