Practice Problems: 10/36-702
1. Let (X1 , Y1 ), . . . , (Xn , Yn ) be iid. Suppose that X1 , . . . , Xn ∼ Unif(0, 1) and that
Yi = m(Xi ) + i
where 1 , . . . , n are iid with mean 0 and variance σ 2 and are independent of the Xi ’s.
Assume that m, m0 , m00 , m000 are bounded and continous functions. Let x ∈ (0, 1) and
define
n
x − Xi
1X 1
Yi K
m(x)
b
=
n i=1
h
h
where K is a smooth, symmetric, kernel with bounded support. Show that
E[m(x)]
b
= m(x) + Ch2 + O(h3 )
for some C > 0.
2. Let (X1 , Y1 ), . . . , (Xn , Yn ) be iid. Suppose that Yi ∈ {0, 1} and also suppose that
Xi ∈ {1, 2, . . . , k} is discrete. Assume that px ≡ P (X = x) > 0 for each x. Let b
h be
the plug-in classifier defined by
(
1 if m(x)
b
≥ 1/2
b
h(x) =
0 if m(x)
b
< 1/2
where
pb(Y = 1, X = x)
,
pb(X = x)
P
P
pb(Y = 1, X = x) = n1 ni=1 I(Yi = 1, Xi = x) and pb(X = x) = n1 ni=1 I(Xi = x). (We
define m(x)
b
= 0 if pb(X = x) = 0.)
m(x)
b
= pb(Y = 1|X = x) =
P
(a) Show that maxx |m(x)
b
− m(x)| → 0.
(b) Show that
P
P (Y 6= b
h(X)) − P (Y 6= h∗ (X)) → 0
where h∗ is the Bayes rule.
3. Let Y1 , . . . , Yn ∼ p where p is a density on [0, 1]. Let k > 2 be a fixed integer and let
B1 , . . . , Bk be define by:
B1 = [0, 1/k), B2 = [1/k, 2/k), . . . , Bk = [1 − 1/k, 1].
Let P be the set of densities that are constant over each √
Bj . Let x ∈ (0, 1) and let
θ = p(x). Show that the minimax rate of convergence is 1/ n, under usual Euclidean
distance in R, i.e. d(θ1 , θ2 ) = |θ1 − θ2 |.
1
p
|f − g| ≤ 2K(f, g).
R
R
R
Hint: Let A = {f ≥ g}. Let p = A f and q = A g. Show that A f log(f /g) ≥
p log(p/q). Next show that
1−p
≡ H(p, q).
K(f, g) ≥ p log(p/q) + (1 − p) log
1−q
4. Let f, g be densities. Show that
R
Bonus: Now write p = q + r and use a Taylor series to show that H ≥ 2r2 .
5. Recall that X is sub-Gaussian if E(X) = 0 and
log ψX (t) ≤
t2 v
2
for some v, where ψX is the moment generating function. Suppose that X is subGaussian.
(a) Show that, for every t > 0,
2 /(2v)
P (X > t) ∨ P (−X > t) ≤ e−t
(b) Show that, for every integer q ≥ 1,
E[X 2q ] ≤ q!(4v)q .
2
.