5
Regression
5.1
Derivation of the conditional mean
The conditional mean of y is defined as
R
yf (y, x)dy
.
f (x)
E[Y |x = x] =
Replacing unknown quantities with estimates we can see that the integral in the numerator is
Z
Z
n
X
1
y
−
y
i
dy.
y fb(y, x)dy =
Kh (xi , x) yk
nhy |h| i=1
hy
Using the change of variable u = (y − yi )/hy , yielding du = dy/hy and y = hy u + yi , we have
n
1 X
Kh (xi , x)
n |h| i=1
Z
yk
y − yi
hy
n
1 X
dy =
Kh (xi , x)
n |h| i=1
Z
(hy u + yi )k (u) du
Z
Z
n
1 X
=
Kh (xi , x) hy uk (u) du + yi k(u)du
n |h| i=1
n
1 X
=
Kh (xi , x) [hy · 0 + yi ]
n |h| i=1
n
1 X
=
Kh (xi , x) yi .
n |h| i=1
Using this we have that our nonparametric estimator of the conditional mean is
1
n|h|
b |x = x] = m(x)
E[Y
b
=
5.2
n
P
Kh (xi , x) yi
i=1
n
P
1
n|h|
.
(30)
Kh (xi , x)
i=1
Asymptotic Bias of the Local-Constant Estimator
To investigate the bias and variance of this estimator we first rewrite the estimator as
n
P
(n |h|)−1
Kh (xi , x) yi
m(x)
b
= i=1
n
P
i=1
=
Kh (xi , x)
n
P
Kh (xi , x) yi
gb(x)
i=1
=
.
n
P
b(x)
f
−1
(n |h|)
Kh (xi , x)
i=1
64
(31)
Next, we rewrite (31) as
"
#−1
fb(x) − f (x)
gb(x)
gb(x)
m(x)
b
=
=
· 1+
.
f (x)
f (x)
fb(x)
(32)
Using what we know about the bias and variance of the multivariate kernel estimator, the
b
(x)
term f (x)−f
is asymptotically negligible and so can be ignored. This means that m(x)
b
≈
f (x)
gb(x)/f (x) so that E[m(x)]
b
= E[b
g (x)]/f (x).
Now, to derive the expected value of gb(x) we start with its definition.
"
E[b
g (x)] =E (n |h|)−1
n
X
#
Kh (xi , x) yi
i=1
"
=E Ex (n |h|)−1
"
=E Ex (n |h|)−1
n
X
i=1
n
X
!#
Kh (xi , x) yi
!#
Kh (xi , x) (m(xi ) + ui )
i=1
"
=E (n |h|)−1
n
X
#
"
Kh (xi , x) m(xi ) + E Ex (n |h|)−1
i=1
n
X
!#
Kh (xi , x) ui )
. (33)
i=1
The second term has conditional expectation zero (Our first two assumptions become ui ∼
D(0, σ 2 ) and xi is independent of the ui s). Continuing we have
"
−1
E[b
g (x)] =E (n |h|)
n
X
#
Kh (xi , x) m(xi ) = (n |h|)
i=1
−1
= |h|
−1
n
X
E [Kh (x1 , x) m(x1 )]
i=1
Z
Kh (x1 , x) m(x1 )f (x1 )dx1
Using the change of variable t = v1
Z
= K(t)m(x + ht)f (x + ht)dt.
(34)
The notation r(x + ht) is taken to mean r(x1 + h1 · t1 , x2 + h2 · t2 , · · · , xd + hd · td ). To
further simplify this expression we assume that both m(·) and f (·) are twice continuously
differentiable in a neighborhood of x. This will allow us to take Taylor expansions of order
two for both m(·) and f (·) around the point x. Remember that for a multivariate Taylor
2
P
expansion the appropriate formula, for order two, is: r(x + a) =
(j!)−1 aj ∇jx0 r(x0 ) |x0 =x
j=0
65
+o(a3 ). Moving on, and ignoring the o(a3 ) terms, we have
Z
E[b
g (x)] =
K(t)
" 2
X
(j!)−1 (ht)j ∇jx0 m(x0 ) |x0 =x
#" 2
X
j=0
#
(j!)−1 (ht)j ∇jx0 f (x0 ) |x0 =x dt
j=0
1
2 2
= K(t) m(x) f (x) + (ht)∇f (x) + (ht) ∇ f (x) dt
2
Z
1
2 2
+ K(t) (ht)∇m(x) f (x) + (ht)∇f (x) + (ht) ∇ f (x) dt
2
Z
1
1
2 2
2 2
+ K(t) (ht) ∇ m(x) f (x) + (ht)∇f (x) + (ht) ∇ f (x) dt.
2
2
Z
(35)
Assuming the following properties of our kernel function, K,
(i)
R
K(t)dt = 1,
(ii)
R
tK(t)dt = 0,
(iii)
R
t2 K(t)dt = κ2 < ∞.
R
R
R
Note that (iii) can be written as t21 k(t1 ) t22 k(t2 ) · · · t2d k(td )dtd · · · dt2 dt1 = κ2 < ∞
using the definition of the product kernel. This fact along with (ii) imply that for s 6= r,
R
ts tr k(ts )dts = 0. Using these properties we can further simplify our expression as follows:
E[b
g (x)] =m(x) f (x) +
+
d
κ2 X
2 s=1
!
h2s fss (x)
+ κ2
d
X
h2s ms (x)fs (x)
s=1
d
X
κ2
h2s mss (x)f (x)
2 s=1
d
κ2 X 2
=m(x)f (x) +
h Ds (x).
2 s=1 s
(36)
where Ds (x) = m(x)fss (x) + 2ms (x)fs (x) + mss (x)f (x). Now this is only the expected value
of gb(x) and to determine the bias of m(x)
b
we need to determine E[m(x)]
b
− m(x). Using the
same logic as above we note that
"
#−1
gb(x) − m(x)fb(x)
fb(x) − f (x)
gb(x) − m(x)fb(x)
m(x)
b
− m(x) =
=
· 1+
,
f (x)
f (x)
fb(x)
66
(37)
so the bias of the estimator of m(·) is roughly equivalent to E[(b
g (x) − m(x)fb(x)]/f (x).
Using our knowledge of the expected value of the multivariate kernel density estimate we
can completely solve for the bias of the LCLS estimator. Recall that
d
κ2 X 2
b
E[f (x)] = f (x) +
h fss (x).
2 s=1 s
(38)
Combining this with our derivation for E[b
g (x)], the bias of the LCLS estimator is
Bias(m(x))
b
=E[m(x)]
b
− m(x) ≈
m(x)f (x) + κ22
=
d
P
s=1
E[b
g (x)] − m(x)E[fb(x)]
f (x)
h2s Ds (x) − m(x)(f (x) + κ22
d
P
h2s fss (x))
s=1
f (x)
d
=
κ2 X 2
h Bs (x),
2f (x) s=1 s
(39)
where Bs (x) = 2ms (x)fs (x) + mss (x)f (x).
5.3
Asymptotic Variance of the Local-Constant Estimator
From equation (37) we have that
V ar(m(x))
b
≈
V ar(b
g (x)) + m(x)2 V ar(fb(x)) − 2m(x)Cov(b
g (x), fb(x))
.
f (x)2
(40)
From our derivations for multivariate kernel density estimators, we know that the variance
of our density estimate is
V ar(fb(x)) = (n |h|)−1 f (x)
67
Z
K(t)2 dt,
(41)
so all that remains is to determine the variance of gb(x) and the covariance between this and
our density estimate. Lets derive the variance first.
V ar(b
g (x)) =E[V arx (b
g (x))] + V ar(Ex [b
g (x)])
"
#
!
n
n
X
X
=E (n |h|)−2
K(vi )2 V arx (ui ) + V ar (n |h|)−1
K(vi )m(xi )
i=1
2
i=1
n
X
σ
2
=
+ (n |h|)−2 V ar
E
K(v
)
i
(n |h|)2 i=1
2
nσ
2
+ (n |h|)−2 V ar
E
K(v
)
=
1
(n |h|)2
n
X
n
X
!
K(vi )m(xi )
i=1
!
K(vi )m(xi ) .
(42)
i=1
We will consider the two terms in (42) separately. The first term is simplified as
Z
Z
nσ 2
nσ 2
σ2
2
2
E K(v1 ) =
K(v1 ) f (x1 )dx1 =
K(t)2 f (x + ht)dt
2
2
(n |h|)
(n |h|)
(n |h|)
Z
2
σ
1
2
2 2
K(t) f (x) + (ht)∇f (x) + (ht) ∇ f (x) dt
=
(n |h|)
2
2 Z
2 Z
f (x)σ
σ
=
K(t)2 dt +
K(t)2 dt · t∇f (x) + ht2 ∇2 f (x)
(n |h|)
n
Z
2 Z
1
f (x)σ
2 2
2
2
2
K(t) dt + · σ
K(t) dt · t∇f (x) + ht ∇ f (x)
=
(n |h|)
n
Z
1
f (x)σ 2
2
K(t) dt + O
.
(43)
=
(n |h|)
n
We immediately notice that the first term decreases with the sample size and increases as
the bandwidth decreases.
Now we consider the second term:
" n
!
n
X
X (n |h|)−2 V ar
K(vi )m(xi ) =(n |h|)−2
E K(v1 )2 m(x1 )2
i=1
i=1
n X
X
+
E [K(v1 )m(x1 )K(v2 )m(x2 )]
i=1 j6=i
−n2 (E [K(v1 )m(x1 )])2 .
(44)
The last of these terms we have already calculated above, see the arguments leading up to
equation (36). Focusing attention for the moment on the first two summands we can derive
68
the variance for the first term as
(n |h|)
−2
n
X
E K(v1 )2 m(x1 )2 =n(n |h|)−2 E K(v1 )2 m(x1 )2
i=1
−2
=n(n |h|)
−1
Z
K(v1 )2 m(x1 )2 f (x1 )dx1
Z
K(t)2 m(x + ht)2 f (x + ht)dt
" 2
#2
Z
X
j
=(n |h|)−1 K(t)2
(j!)−1 (ht)j ∇x0 m(x0 ) |x0 =x
=(n |h|)
j=0
·
" 2
X
#
(j!)−1 (ht)j ∇jx0 f (x0 ) |x0 =x dt.
(45)
j=0
This simplifies further to
−2
(n |h|)
n
X
2
2
E K(v1 ) m(x1 )
−1
=(n |h|)
Z
K(t)2 m(x)2 + 2htm(x)∇m(x) + (ht)2 (∇m(x))2
i=1
+ 2(ht)3 ∇m(x)∇2 m(x) + 2(ht)2 m(x)∇2 m(x)
+(ht)4 (∇2 m(x))2 · f (x) + ht∇f (x) + (ht)2 ∇2 f (x) dt.
Z
m(x)2 · f (x)
1
2
=
K(t) dt + O
.
(46)
(n |h|)
n
The terms of order inverse n arise from the fact that every other term from the two Taylor
expansions have bandwidths in the numerator while n is in the denominator. Given that
typical assumptions are the sample size growing to infinity and the bandwidths decreasing
to zero we see that all the other terms are dominated by n. Moving to the second term we
69
see that
−2
(n |h|)
n X
X
i=1 j6=i
n(n − 1)
(E [m(x1 )K(v1 )])2
(n |h|)2
Z
2
n(n − 1)
=
K(v1 )m(x1 )f (x1 )dx1
(n |h|)2
Z
2
n(n − 1)
K(t)m(x + ht)f (x + ht)dt
=
n2
Z
2
1
K(t)m(x + ht)f (x + ht)dt .
= 1−
n
(47)
E [K(v1 )m(x1 )K(v2 )m(x2 )] =
Combining this with our previously known results we have that the variance of gb(x) is
Z
Z
m(x)2 · f (x)
f (x)σ 2
2
K(t) dt +
K(t)2 dt
V ar(b
g (x)) =
(n |h|)
(n |h|)
Z
2
1
1−
K(t)m(x + ht)f (x + ht)dt
n
Z
2
1
−
K(t)m(x + ht)f (x + ht)dt + O
n
2
2 Z
1
f (x)(σ + m(x) )
K(t)2 dt + O
=
,
(n |h|)
n
(48)
where the integral has disappeared into the O n1 part due to the fact that the integral
represented the expectation of g(x) which did not depend on n and hence is dominated by
the term.
Finally, the covariance. Deriving this will be almost identical to the derivations of the
bias and the variance.
h
i
h
i
Cov fb(x), gb(x) =E fb(x)b
g (x) − E fb(x) · E [b
g (x)]
Z
1
−1
2
=(n |h|)
K(t) m(x + ht)f (x + ht)dt + O
.
n
70
(49)
We can again take a second-order Taylor expansion around x and simplify our covariance as
Z
−1
b
Cov f (x), gb(x) =(n |h|)
K(t)2 m(x + ht)f (x + ht)dt + O n−1
Z
1
2 2
−1
2
=(n |h|)
K(t) m(x) + ht∇m(x) + (ht) ∇ m(x)
2
1
· f (x) + ht∇f (x) + (ht)2 ∇2 f (x) dt + O n−1
2
Z
m(x)f (x)
=
K(t)2 dt + O n−1 .
(n |h|)
(50)
where all of the remaining terms from the Taylor expansions only have h in the numerator
but always have an n in the denominator which makes them of order n.
Referring back to equation (40) we have our variance as
V ar(b
g (x)) + m(x)2 V ar(fb(x)) − 2m(x)Cov(b
g (x), fb(x))
f (x)2
Z
Z
m(x)2 f (x)
f (x)(σ 2 + m(x)2 )
2
K(t) dt +
K(t)2 dt
=
2
2
f (x) (n |h|)
f (x) (n |h|)
Z
2
2m(x) )f (x)
−
K(t)2 dt
f (x)2 (n |h|)
2
Z
σ + m(x)2
m(x)2
2m(x)2
=
+
−
K(t)2 dt
f (x)(n |h|) f (x)(n |h|) f (x)(n |h|)
Z
σ2
=
K(t)2 dt.
f (x)(n |h|)
V ar(m(x))
b
≈
71
(51)