Hindawi Publishing Corporation
Advances in Decision Sciences
Volume 2011, Article ID 485974, 22 pages
doi:10.1155/2011/485974
Research Article
Some Asymptotic Theory for Functional
Regression and Classification
Frits Ruymgaart, Jing Wang, Shih-Hsuan Wei, and Li Yu
Department of Mathematics and Statistics, Texas Tech University, Lubbock, TX 79409, USA
Correspondence should be addressed to Frits Ruymgaart, h.ruymgaart@ttu.edu
Received 10 October 2011; Accepted 2 November 2011
Academic Editor: Wing Keung Wong
Copyright q 2011 Frits Ruymgaart et al. This is an open access article distributed under the
Creative Commons Attribution License, which permits unrestricted use, distribution, and
reproduction in any medium, provided the original work is properly cited.
Exploiting an expansion for analytic functions of operators, the asymptotic distribution of an estimator of the functional regression parameter is obtained in a rather simple way; the result is applied to testing linear hypotheses. The expansion is also used to obtain a quick proof for the asymptotic optimality of a functional classification rule, given Gaussian populations.
1. Introduction
Certain functions of the covariance operator such as the square root of a regularized inverse
are important components of many statistics employed for functional data analysis. If Σ is a
a sample analogue of this operator, and ϕ a funccovariances operator on a Hilbert space, Σ
tion on the complex plane, which is analytic on a domain containing a contour around the
and ϕΣ by means of
spectrum of Σ, a tool of generic importance is the comparison of ϕΣ
a Taylor expansion:
− Σ remainder.
ϕΣ ϕ̇Σ Σ
ϕ Σ
1.1
It should be noted that ϕ̇Σ is not in general equal to ϕ Σ, where ϕ is the numerical derivative of ϕ; see also Section 3. In this paper, two further applications of the approximation in
1.1 will be given, both related to functional regression.
The first application Section 4 concerns the functional regression estimator itself.
Hall and Horowitz 1 have shown that the IMSE of their estimator, based on a Tikhonov
type regularized inverse, is rate optimal. In this paper, as a complementary result, the general
asymptotic distribution is obtained, with potential application to testing linear hypotheses of
2
Advances in Decision Sciences
arbitrary finite dimension, mentioned in Cardot et al. 2 as an open problem: these authors
concentrate on testing a simple null hypotheses. Cardot et al. 3 establish convergence in
probability and almost sure convergence of their estimator which is based on spectral cutoff
regularization of the inverse of the sample covariance operator. In the present paper, the
covariance structure of the Gaussian limit will be completely specified. The proof turns out to
− ϕΣ, which is almost immediate from 1.1.
be routine thanks to a “delta-method” for ϕΣ
The second application Section 5 concerns functional classification, according to a
slight modification of a method by Hastie et al. 4, exploiting penalized functional regression. It will be shown that this method is asymptotically optimal Bayes when the two
populations are represented by equivalent Gaussian distributions with the same covariance
− ϕΣ, which
operator. The simple proof is based on an upper bound for the norm of ϕΣ
follows at once from 1.1.
Let us conclude this section with some comments and further references. The expansion in 1.1 can be found in Gilliam et al. 5, and the ensuing delta method is derived and
applied to regularized canonical correlation in Cupidon et al. 6. For functional canonical
correlation see also Eubank and Hsing 7, He et al. 8, and Leurgans et al. 9. When the
− Σ in the present case commutes with Σ the expansion 1.1 can already be
perturbation Σ
found in Dunford & Schwartz 10, Chapter VII, and the derivative does indeed reduce to the
numerical derivative. This condition is fulfilled only in very special cases, for instance, when
the random function, whose covariance operator is Σ, is a second order stationary process on
the unit interval. In this situation, the eigenfunctions are known and only the eigenvalues
are to be estimated. This special case, that will not be considered here, is discussed in
Johannes 11 who in particular deals with regression function estimators and their IMSE is
Sobolev norms, when the regression is such a stationary process. General information about
functional data analysis can be found in the monographs by Ramsay and Sliverman 12 and
Ferraty and Vieu 13. Functional time series are considered in Bosq 14; see also Mas 15.
2. Preliminaries and Introduction of the Models
2.1. Preliminaries
As will be seen in the examples below, it is expedient to consider functional data as elements
in an abstract Hilbert space H of infinite dimension, separable, and over the real numbers.
Inner product and norm in H will be denoted by ·, · and · , respectively. Let Ω, F, P be a
probability space, X : Ω → H a Hilbert space valued random variable i.e., measurable with
respect to the σ-field of Borel sets BH in H, and η : Ω → R a real valued random variable.
For all that follows it will be sufficient to assume that
EX4 < ∞,
Eη2 < ∞.
2.1
The mean and covariance operator of X will be denoted by
EX μX ,
E X − μX ⊗ X − μX ΣX,X ,
2.2
Advances in Decision Sciences
3
respectively, where a ⊗ b is the tensor product in H. The Riesz representation theorem guarantees that these quantities are uniquely determined by the relations
Ea, X a, μX , ∀a ∈ H,
E a, X − μX X − μX , b a, ΣX,X b, ∀a ∈ H ∀b ∈ H,
2.3
see Laha & Rohatgi 16. Throughout ΣX,X is assumed to be one-to-one.
Let L denote the Banach space of all bounded linear operators T : H → H equipped
with the norm · L . An operator U ∈ L is called Hilbert-Schmidt if
∞
Uek 2 < ∞,
2.4
k1
for any orthonormal basis e1 , e2 , . . . of H. The number in 2.4 is in fact independent of the
choice of basis. The subspace LHS ⊂ L of all Hilbert-Schmidt operators is a Hilbert space in
its own right with the inner product
U, V HS ∞
Uek , V ek ,
2.5
k1
again independent of the choice of basis. This inner product yields the norm
U2HS ∞
Uek 2 ,
2.6
k1
which is the number in 2.4. The tensor product for elements a, b ∈ H will be denoted by
a ⊗ b, and that for elements U, V ∈ LHS by U⊗HS V .
The two problems to be considered in this paper both deal with cases where the best
linear predictor of η in terms of X is linear:
E η | X α X, f ,
α ∈ R, f ∈ H.
2.7
Just as in the univariate case Rao 17, Section 4g, we have the relation
ΣX,X f E η − μη X − ηX ΣX,η .
2.8
It should be noted that if ΣX,X is one-to-one and ΣX,η in its range, we can solve 2.8 and
obtain
f Σ−1
X,X ΣX,η .
2.9
Since the underlying distribution is arbitrary, the empirical distribution, given a sample X1 , η1 , . . . , Xn , ηn of independent copies of X, η, can be substituted for it. The
4
Advances in Decision Sciences
minimization property is now the least squares property, the same formulas are obtained
with μX , ΣX,X , μη , and ΣX,η replaced with their estimators
μ
X n
1
Xi X,
n i1
n X,X 1
Xi − X ⊗ Xi − X ,
Σ
n i1
μ
η n
1
ηi η,
n i1
n
X,η 1
Σ
ηi − η × X i − X .
n i1
2.10
2.11
2.12
2.13
Let us next specify the two problems.
2.2. Functional Regression Estimation
The model here is
η α X, f ε,
2.14
where ε is a real valued error variable and the following assumption is satisfied.
Assumption 2.1. The error variable has a finite second moment, and
ε ⊥⊥ X,
Eε 0,
Var ε v2 < ∞.
2.15
Example 2.2. The functional regression model in Hall and Horowitz 1 is essentially obtained
by choosing H L2 0, 1, so that the generic observation is given by
η α
1
Xtft dt ε.
2.16
0
Example 2.3. Mas and Pumo 18 argue that in the situation of Example 2.2, the derivative
X of X may contain important information and should therefore be included. Hence, these
authors suggest to choose for H the Sobolev space W 2,1 0, 1 in which case the generic
observation satisfies
η α
1
0
Xtftdt 1
X tf tdt ε.
2.17
0
Example 2.4. Just as in the univariate case, we have that the model
2
η α X, f ε,
2.18
Advances in Decision Sciences
5
quadratic in the inner product of H, is in fact linear in the inner product of LHS , because
X, f2 X ⊗ X, f ⊗ f HS .
2.19
We will not pursue this example here.
X,X cannot be one-to-one, and in order to estimate f
In the infinite dimensional case Σ
from the sample version of 2.9, a regularized inverse of Tikhonov type will be used, as in
Hall & Horowitz 1. Thus, we arrive at the estimator see also 2.11 and 2.13
−1
X,X
fδ δI Σ
X,X
δI Σ
−1
n
1
ηi Xi − X
n i1
X,η ,
Σ
2.20
for some δ > 0.
Let us also introduce
fδ δI Σ−1 Σf,
f ∈ H.
2.21
In Section 4, the asymptotic distribution of this estimator will be obtained, and the result will
be applied to testing.
2.3. Functional Classification
The method discussed here is essentially the one in Hastie et al. 4 and Hastie et al. 19,
Sections 4.2 and 4.3. Let P1 and P2 be two probability distributions on H, BH with means
μ1 and μ2 and common covariance operator Σ. Consider a random element I, X : Ω →
{1, 2} × H with distribution determined by
P X ∈ B | I j Pj B, B ∈ BH ,
P I j πj ≥ 0, π1 π2 1.
2.22
In this case, the distribution of X is π1 P1 π2 P2 , with mean
μX π1 μ1 π2 μ2 ,
2.23
ΣX,X Σ π1 π2 μ1 − μ2 ⊗ μ1 − μ2 .
2.24
and covariance operator
6
Advances in Decision Sciences
Hastie et al. 19 now introduce the indicator response variables ηj 1{j} I, j 1, 2,
and assume that the ηj satisfies 2.7 for αj ∈ R and fj ∈ H. Note that
ΣX,ηj
μηj Eηj πj ,
⎧
⎨π1 π2 μ1 − μ2 ,
Eηj X − μX ⎩π π μ − μ ,
1 2
2
1
2.25
j 1,
j 2.
2.26
Since ηj is Bernoulli, we have, of course, Eηj | X P{I j | X}. Precisely as for matrices
Rao & Toutenburg 20, Theorem A.18, the inverse of the operator in 2.24 equals
−1
−1
Σ−1
X,X Σ − γ · Σ
μ1 − μ2 ⊗ μ1 − μ2 Σ−1 ,
2.27
where γ π1 π2 /1 π1 π2 μ1 − μ2 , Σ−1 μ1 − μ2 , provided that the following assumption is
satisfied.
Assumption 2.5. The vector μ1 − μ2 lies in the range of Σ, that is,
Σ−1 μ1 − μ2 is well defined.
2.28
It will also be assumed that
π1 π2 1
.
2
2.29
Assuming 2.28, 2.8 can be solved and yields after some algebra
fj Σ−1
X,X ΣX,ηj
⎧
⎨γΣ−1 μ1 − μ2 ,
⎩γΣ−1 μ − μ ,
2
1
j 1,
j 2.
2.30
If only X, and not I, is observed, the rule in Hastie et al. 19 assigns X to P1 if and only if
E η1 | X > E η2 | X ,
2.31
⇐⇒
π2 − π1 .
X − μX , Σ−1 μ1 − μ2 >
2γ
2.32
Because of assumption 2.29, the rule reduces to
1
X − μ1 μ2 , Σ−1 μ1 − μ2
2
> 0.
2.33
Advances in Decision Sciences
7
Hastie et al. 19 claim that in the finite dimensional case, their rule reduces to Fisher’s
linear discriminant rule and to the usual rule when the distributions are normal. This remains
in fact true in the present infinite dimensional case. Let us assume that
Pj G μj , Σ ,
j 1, 2,
2.34
where Gμ, Σ denotes a Gaussian distribution with mean μ and covariance operator Σ. It is
well known 21–23 that under Assumption 2.5 these Gaussian distributions are equivalent.
This is important since there is no “Lebesgue measure” on H 24. However, now the densities
of P1 and P2 with respect to P1 can be considered; it is well known that
dP1
−1
x e−x−1/2μ1 μ2 , Σ μ1 −μ2 ,
dP2
x ∈ H.
2.35
This leads at once to 2.33 as an optimal Bayes rule, equal in appearance to the one for the
finite dimensional case.
In most practical situations, μ1 , μ2 , and Σ are not known, but a training sample
I1 , X1 , . . . , In , Xn of independent copies of I, X is given. Let
Xj Jj i : Ii j ,
1 Xi ,
nj j∈Jj
#Jj nj ,
2.36
2 1
Xi − X j ⊗ Xi − X j ,
Σ
n j1 i∈J
2.37
X,X Σ
n1 n2 X 1 − X 2 ⊗ X 1 − X 2 .
Σ
n
2.38
j
and we have cf. 2.24
and Σ
X,X for that matter cannot be one-to-one. In order to
Once again the operator Σ
obtain an empirical analogue of the rule 2.32, Hastie et al. 4 employ penalized regression,
and Hastie et al. 19 also suggest to use a regularized inverse. The methods are related.
Here the latter method will be used and X will be assigned to P1 if and only if
−1 1
X−
> 0.
X 1 X 2 , δI Σ
X1 − X2
2
2.39
Section 5 is devoted to showing that this rule is asymptotically optimal when Assumption
2.5 is fulfilled.
3. A Review of Some Relevant Operator Theory
It is well known 16 that the covariance operator Σ is nonnegative, Hermitian, of finite trace,
and hence Hilbert-Schmidt and therefore compact. The assumption that Σ is one-to-one is
8
Advances in Decision Sciences
i
σ12
−δ −(1/2)δ 0
σ12 + 1
Ω
−i
Γ
D
Figure 1
equivalent to assuming that Σ is strictly positive. Consequently, Σ has eigenvalues σ12 > σ22 >
· · · ↓ 0, all of finite multiplicity. If we let P1 , P2 , . . . be the corresponding eigenprojections, so
that ΣPk σk2 Pk , we have the spectral representation:
Σ
∞
σk2 Pk ,
with
k1
∞
Pk I.
3.1
k1
The spectrum of Σ equals σΣ {0, σ12 , σ22 , . . .} ⊂ 0, σ12 . Let us introduce a rectangular
contour Γ around the spectrum as in Figure 1, where δ > 0 is the regularization parameter in
2.20, and let Ω be the open region enclosed by Γ. Furthermore, let D ⊃ Ω ∪ Γ Ω be an
open neighborhood of Ω and suppose that
ϕ : D −→ C is analytic.
3.2
ϕΣΠ, where Π ∈ L is a perturbation.
We are interested in approximations of ϕΣ
Σ
− Σ and yields an approximation of
The application we have in mind arises for Π Π
see also Watson 25 for the matrix case. Therefore, we will not in general assume that
ϕΣ;
Π and Σ will commute. In the special case where X is stationary, as considered in Johannes
11, there exists a simpler estimator Σ of Σ, such that Σ and Π do commute, which results in
a simpler theory; see also Remark 4.1.
The resolvent of Σ,
Rz zI − Σ−1 ,
z ∈ ρΣ σΣc ,
3.3
is analytic on the resolvent set ρΣ, and the operator
1
ϕΣ 2πi
Γ
ϕzRz dz
3.4
Advances in Decision Sciences
9
is well defined. For the present operator Σ, as given in 3.1, the resolvent equals more explicitly
Rz ∞
1
P .
2 k
k1 z − σk
3.5
Substitution of 3.5 in 3.4 and application of the Cauchy integral formula yields
ϕΣ ∞ ϕ σk2 Pk .
3.6
k1
Example 3.1. The two functions
ϕ1 z 1
,
δz
z
,
δz
ϕ2 z z ∈ C \ {−δ},
3.7
are analytic on their domain that satisfies the conditions. With the help of these functions we
may write cf. 2.20 and 2.21
fδ − fδ ϕ1 Σ
n
1
εi Xi − X
n i1
− ϕ2 Σ f,
ϕ2 Σ
f ∈ H,
3.8
and cf. 2.39
δI Σ
−1 X1 − X2 .
X 1 − X 2 ϕ1 Σ
3.9
Regarding the following brief summary and slight extension of some of the results in
5, we also refer to Dunford & Schwartz 10, Kato 26, and Watson 25. Henceforth, we
will assume that
ΠL ≤
δ
.
4
3.10
σΣ Π ⊂ Ω, so that the resolvent set of Σ
satisfies
For such perturbations, we have σΣ
ρΣ Π ⊃ Ωc ⊃ Γ.
ρ Σ
3.11
It should also be noted that
2
1
≤
RzL sup 2
δ
k∈N z − σk
∀z ∈ Ωc .
3.12
10
Advances in Decision Sciences
The basic expansion similar to Watson 25
∞
−1
Rz RzΠRzk ,
Rz
zI − Σ
z ∈ Ωc ,
3.13
k1
can be written as
Rz
Rz RzΠRzI − ΠRz−1 ,
3.14
useful for analyzing the error probability for δ → 0, as n → ∞, and also as
Rz
Rz RzΠRz RzΠRz2 I − ΠRz−1 ,
3.15
useful for analyzing the convergence in distribution of the estimators.
Let us decompose the contour Γ into two parts Γ0 {−1/2δ iy : −1 ≤ y ≤ 1} and
Γ1 Γ \ Γ0 , write Mϕ maxz∈Γ |ϕz|, and observe that 3.10 and 3.12 entail that
I − ΠRz−1 ≤ 2,
L
z ∈ Ωc .
3.16
We now have
1
k
−1
2πi ϕzRzΠRz I − ΠRz dz
Γ
≤
1
Mϕ ΠkL
π
1
≤ Mϕ ΠkL
π
L
Γ
Rzk1
L dz
1 −1
1 2
δ y2
4
−1/2k1
k
2
4
k
≤ Mϕ ΠL
5 2ΣL ,
π
δ
dy 1dz
Γ1
3.17
k ∈ N.
Multiplying both sides by ϕz, taking 1/2πi Γ , and using 0 < C < ∞ as a generic constant
that does not depend on Π or δ, 3.14 and 3.15 yield the following.
Theorem 3.2. Provided that 3.10 is fulfilled, one has
ϕΣ Π − ϕΣ ≤ CMϕ ΠL ,
L
δ
3.18
2
ϕΣ Π − ϕΣ − ϕ̇Σ Π ≤ CMϕ ΠL ,
L
δ
3.19
Advances in Decision Sciences
11
where ϕ̇Σ : L → L is a bounded operator, given by
2
2
ϕ σk − ϕ σj
ϕ̇Σ Π ϕ σk2 Pk ΠPk Pj ΠPk .
σk2 − σj2
k
j/
k
3.20
Remark 3.3. If Σ and Π commute, so will Pk and Π, and Rz and Π, and the expressions
simplify considerably. In particular, 3.20 reduces to
2
ϕ σk Pk Π,
ϕ̇Σ Π 3.21
k
that is, ϕ̇Σ ϕ Σ, where ϕ is the numerical derivative of ϕ, and ϕ Σ should be understood
in the sense of “functional calculus” as in 3.6. For the commuting case, see Dunford &
Schwartz 10.
Let us now present some basic facts that are useful to subsequent statistical applications. Dauxois et al. 27 have shown that there exists a Gaussian random element GΣ :
√ d
Ω → LHS , such that nΣ
− Σ → GΣ , as n → ∞, in LHS . Because the identity map from LHS
in L is continuous, we may state
d
√ − Σ −→
n Σ
GΣ ,
as n −→ ∞, in LHS ⇒ in L,
3.22
by the continuous mapping theorem. This entails that
1
Π Σ − Σ Op √ ,
L
L
n
as n −→ ∞,
3.23
so that condition 3.10 is fulfilled with arbitrary high probability for n sufficiently large.
replaced with
Expansions 3.14 and 3.15 and the resulting inequalities hold true for Σ
L ≤ δ/4}.
Σω
Σ Πω
for ω ∈ {Π
Example 3.4. Application to asymptotic distribution theory. In this application δ > 0 will be
kept fixed: see also Section 4.1. It is based on the delta method for functions of operators 6
which follows easily from 3.19. In conjunction with 3.22 this yields
d
√ − ϕ2 Σ −→
n ϕ2 Σ
ϕ̇2,Σ GΣ ,
as n −→ ∞, in L.
3.24
In turn this yields
d √ − ϕ2 Σ f −→
n ϕ2 Σ
ϕ̇2,Σ GΣ f,
as n −→ ∞, in H,
3.25
for any f ∈ H, by the continuous mapping theorem. This result will be used in Section 4.
12
Advances in Decision Sciences
Example 3.5. Application to classification. Here we will let δ δn ↓ 0, as n → ∞, and write
ϕ1,n z 1/δn z to stress this dependence on sample sizes. Since
maxϕ1,n z ≤
z∈Γ
1
,
δn
3.26
it is immediate from 3.18 that
ϕ1,n Σ − ϕ1,n Σ Op
L
1
√ ,
δ2 n n
as n −→ ∞,
3.27
a result that will be used in Section 5.
4. Asymptotics for the Regression Estimator
4.1. The Asymptotic Distribution
The central limit theorem in Hilbert spaces entails at once
n
d
1 εi Xi − μ −→ G0 ,
√
n i1
as n −→ ∞, in H,
4.1
where G0 is a zero mean Gaussian random variable in H, and
n
d
1 Xi − μ ⊗ Xi − μ − Σ −→ GΣ ,
√
n i1
as n −→ ∞, in LHS ,
4.2
where GΣ is a zero mean random variable in LHS . These convergence results remain true with
μ replaced by X and, because ε ⊥⊥ X by assumption 2.15, we also have that jointly
⎡
n
⎤
1 εi Xi − X ⎥ d & G '
⎢√
0
⎢ n i1
⎥
⎣ √ ⎦ −→ GΣ ,
−Σ
n Σ
as n −→ ∞, in H × LHS ,
4.3
where GΣ is the same in 3.22, and
G0 ⊥⊥ GΣ .
4.4
Advances in Decision Sciences
13
Because the limiting variables are generated by the sums of iid variables on the left in
4.1 and 4.2 we have
EG0 ⊗ G0 E ε X − μ ⊗ ε X − μ
Eε2 E X − μ ⊗ X − μ
4.5
v Σ,
EGΣ ⊗HS GΣ E X − μ ⊗ X − μ − Σ ⊗HS X − μ ⊗ X − μ − Σ ,
2
4.6
for the respective covariance structures. These are important to further specify the limiting
distribution of the regression estimator as will be seen in Section 4.2.
Let us write, for brevity,
fδ − fδ Un Vn ,
4.7
where, according to 2.20 and 2.21,
n
1
εi Xi − X
,
n i1
− ϕ2 Σ f.
Vn ϕ2 Σ
Un ϕ1 Σ
4.8
Note that ϕ1 and ϕ2 depend on δ.
With statistical applications in mind, it would be interesting if there would exist numbers an ↑ ∞ and δn ↓ 0, as n → ∞, such that
d
an fδn − f −→ H,
as n −→ ∞, in H,
4.9
where H is a nondegenerate random vector. It has been shown in Cardot et al. 28, however,
that such a convergence in distribution when we center at f is not in general possible.
Theorem 4.1. For fixed δ > 0, one has
d
√ n fδ − fδ −→ H H1 H2 ,
as n −→ ∞, in H,
4.10
where H1 ϕ1 ΣG0 and H2 ϕ̇2,Σ GΣ f are zero mean Gaussian random elements, and H1 ⊥⊥ H2 .
Further information about the structure of the covariance operator of the random
vector H on the right in 4.10 will be needed in order to be able to exploit the theorem
for statistical inference. This will be addressed in the next section.
14
Advances in Decision Sciences
4.2. Further Specification of the Limiting Distribution
It follows from 4.5 that G0 has a Karhunen-Loève expansion
G0 ∞
vσj Zj pj ,
4.11
i1
where the real valued random variables
Zj j ∈ N are iid N0, 1.
4.12
Accordingly H1 in 4.10 can be further specified as
H1 ϕ1 ΣG0 v
∞
σj
j1
δ σj2
Zj pj .
4.13
The Gaussian operator in 4.2 has been investigated in Dauxois et al. 27, and here we
will briefly summarize some of their results in our notation. By evaluating the inner product
in LHS in the basis p1 , p2 . . . it follows from 4.6 that
E pj ⊗ pk , GΣ HS GΣ , pα ⊗ pβ HS
E pj , X − μ, pk X − μ − σk2 pk × pα , X − μ, pβ X − μ − σβ2 pβ
E X − μ, pj X − μ, pk X − μ, pα X − μ, pβ − δj,k δα,β σk2 σβ2 .
4.14
This last expression does not in general further simplify. However, if we assume that
the regressor X satisfies
d
X Gaussian μ, Σ ,
4.15
d X − μ, pj N 0, σj2 are independent,
4.16
it can be easily seen that the
so that the expression in 4.14 equals zero if j, k /
α, β. As in Dauxois et al. 27, we obtain
in this case
E pj ⊗ pk , GΣ HS GΣ , pα ⊗ pβ HS 0,
2
,
vj,k
j, k /
α, β ,
j, k α, β ,
4.17
where
2
vj,k
4
2σj ,
j k,
·
j/
k.
σj2
σk2 ,
4.18
Advances in Decision Sciences
15
Consequently the pj ⊗ pk j ∈ N, k ∈ N are an orthonormal basis of eigenvectors of the
2
. Hence GΣ has the Karhunen-Loève expansion
covariance operator of GΣ with eigenvalues vj,k
in LHS GΣ ∞
∞ vj,k Zj,k pj ⊗ pk ,
4.19
j1 k1
where the random variables
Zj,k j ∈ N, k ∈ N are iid N0, 1.
4.20
Let us, for brevity, write see 3.7 for ϕ2 wj,k
⎧ 2
⎪
σj2
σ
−
ϕ
ϕ
2
2
⎪
k
⎪
⎨
, j
/k
σk2 − σj2
⎪
⎪
⎪
⎩
ϕ2 σj2 , j k
⎫
⎪
⎪
⎪
⎬
⎪
⎪
⎪
⎭
δ
σj2
δ
δ σk2
,
∀j, k ∈ N,
4.21
H2 ϕ̇2,Σ G f
∞
∞ j1 k1
∞
∞ wj,k Pj GΣ Pk f
⎛
⎞
∞
∞ wj,k Pj ⎝
vα,β Zα,β pα ⊗ pβ ⎠Pk f
j1 k1
∞
∞ 4.22
α1 β1
wj,k vj,k Zj,k f, pk pj .
j1 k1
Summarizing, we have the following result.
Theorem 4.2. The random element H1 , on the right in 4.10 can be represented by 4.13. If one
d
assumes that the regressor X Gaussianμ, Σ, the random element H2 on the right in 4.10 can be
represented by 4.22, where the Zj,k in 4.20 are stochastically independent of the Zj in 4.12.
4.3. Asymptotics under the Null Hypothesis
Let us recall that fδ is related to f according to 2.21 so that the equivalence
H0 : fδ 0 ⇐⇒ f 0,
where again δ > 0 is fixed, holds true. The following is immediate from Theorem 4.1.
4.23
16
Advances in Decision Sciences
Theorem 4.3. Under the null hypothesis in 4.23, one has
2
d
nfδ −→ H2 H1 2 ,
as n −→ ∞,
4.24
where
d
H1 2 v2
∞
j1
σj2
δ
2
σj2
2 Zj .
4.25
An immediate generalization of the hypothesis in 4.23 is
H0 : fδ fδ,0 ϕ2 Σ f0 ⇐⇒ f f0 ,
4.26
for some given f0 ∈ H. This hypothesis is in principle of interest for confidence sets. Of course,
testing 4.26 can be reduced to testing 4.23 by replacing the ηi with
ηi ηi − Xi , fδ,0 α Xi , f − fδ,0 εi ,
4.27
and then using the estimator
fδ
δI Σ
−1
n
1
η Xi − X
.
n i1 i
4.28
Since f − fδ,0 0 the following is immediate.
Theorem 4.4. Assuming 4.27, one has
2
d
nfδ −→ H1 2 ,
as n −→ ∞,
4.29
where H1 2 has the same distribution as in 4.25. Related results can be found in Cardot et al. [2].
Another generalization of the hypothesis in 4.23 is
0
1
H0 : fδ ∈ M q1 , . . . , qM ,
4.30
where q1 , . . . , qM are orthonormal vectors in H. Let Q and Q⊥ denote the orthogonal projection
onto M and M⊥ respectively, and note that fδ ∈ M if and only if Q⊥ fδ 0. A test statistic
might be based on Q⊥ fδ 2 and we have the following.
Theorem 4.5. Under H0 in 4.30, one has
2
2
d nQ⊥ fδ −→ Q⊥ H ,
as n −→ ∞.
4.31
Advances in Decision Sciences
17
The distribution on the right in 4.31 is rather complicated if q1 , . . . , qM remain arbitrary. But if we are willing to assume 4.20, it follows from 4.27 that
⊥ 2 ⊥ 2 ⊥ 2
Q H Q H1 Q H2 2 Q⊥ H1 , Q⊥ H2
v2
σk
⊥
⊥
Q
Z
Z
p
,
Q
p
j
k
j
k
2
2
j1 k1 δ σj δ σk
∞
∞ σj
∞ ∞ ∞
∞ wj,k wα,β vj,k vα,β Zj,k Zα,β f, pk f, pβ Q⊥ pj , Q⊥ pα
4.32
j1 k1 α1 β1
2v
∞
σj
j1
Z
2 j
δ σj
∞
∞ wα,β vα,β Zα,β f, pβ Q⊥ pj , Q⊥ pα .
α1 β1
A simplification is possible if we are willing to modify the hypothesis in 4.30 and
use a so-called neighborhood hypothesis. This notion has a rather long history and has been
investigated by Hodges & Lehmann 29 for certain parametric models. Dette & Munk 30
have rekindled the interest in it by an application in nonparametric regression. In the present
context we might replace 4.30 with the neighborhood hypothesis
2
H0,ε : Q⊥ fδ ≤ ε2 ,
for some ε > 0.
4.33
It is known from the literature that the advantage of using a neighborhood hypothesis is
not only that such a hypothesis might be more realistic and that the asymptotics are much
simpler, but also that without extra complication we might interchange null hypothesis and
alternative. This means in the current situation that we might as well test the null hypothesis
2
: Q⊥ fδ ≥ ε2 ,
H0,ε
for some ε > 0,
4.34
which could be more suitable, in particular in goodness-of-fit problems.
The functional g → Q⊥ g2 , g ∈ H, has a Fréchet derivative at fδ given by the functional g → 2g, Q⊥ fδ , g ∈ H. Therefore, the delta method in conjunction with Theorem 4.1
entails the following result.
Theorem 4.6. One has
√
2
3
d
⊥ 2 ⊥ 2
n Q fδ − Q fδ −→ 2 H, Q⊥ fδ ,
as n −→ ∞.
4.35
18
Advances in Decision Sciences
The limiting distribution on the right in 4.35 is normal with mean zero and complicated variance
2
Δ2 4E H, Q⊥ fδ
2 2
2 3
⊥
⊥
4 E H1 , Q fδ E H2 , Q fδ
4v2
∞
j1
σj2
δ σj2
2
⊥
2 pj , Q fδ
⎛
⎞
52
4
∞
∞
∞ ∞ ⊥
4E
wj,k Pj ⎝
vα,β Zα,β pα ⊗ pβ ⎠Pk f, Q fδ
j1 k1
4v2
∞
j1
σj2
δ σj2
4.36
α1 β1
2
⊥
2 pj , Q fδ
∞
∞ 2
2 2
2
pj , Q⊥ fδ .
4
wj,k
vj,k
f, pk
j1 k1
Remark 4.7. As we see from the expressions in 4.24, 4.32, and 4.36, the limiting distributions depend on infinitely many parameters that must be suitably estimated in order to be in a
position to use the statistics for actual testing. Estimators for the individual parameters are not
too hard to obtain. The eigenvalues σj2 and eigenvectors pj of Σ, for instance, can in principle
Although in any practical situation only a
be estimated by the corresponding quantities of Σ.
finite number of these parameters can be estimated, theoretically this number must increase
with the sample size and some kind of uniform consistency will be needed for a suitable
approximation of the limiting distribution. This interesting question of uniform consistency
seems to require quite some technicalities and will not be addressed in this paper.
Remark 4.8. In this paper we have dealt with the situation where Σ is entirely unknown. It has
been observed in Johannes 11 that if X is a stationary process on the unit interval, the eigenfunctions pj of the covariance operator are always the same, known system of trigonometric
functions, and only its eigenvalues σj2 are unknown. Knowing the pj leads to several simplifications. In the first place, Σ can now be estimated by the expression on the right in 3.1 with
is this estimator, it is clear that Σ and Π
Σ
−Σ
only the σk2 replaced with estimators. If Σ
commute, so that the derivative ϕ̇2,Σ now simplifies considerably see Remark 3.3. Secondly,
we might consider the special case of H0 in 4.30, where qj pj , j 1, . . . , M. We now have
0
1
fδ ∈ M p1 , . . . , pM ⇐⇒ f ∈ M,
4.37
so that even for fixed δ we can test the actual regression function. In the third place, under the
null hypothesis in 4.37, the number of unknown parameters in 4.32 reduces considerably
because now Q⊥ pj 0 for j 1, . . . , M. When the pj are known, in addition to all the changes
Considering all
mentioned above, also the limiting distribution of Σ differs from that of Σ.
these modifications that are needed, it seems better not to include this important special case
in this paper.
Advances in Decision Sciences
19
4.4. Asymptotics under Local Alternatives
Again we assume that X is Gaussian. Suppose that
1
f fn f √ g,
n
g ∈ H.
for f,
4.38
For such fn only minor changes in the asymptotics are required, because the conditions on
the Xi and εi are still the same and do not change with n. Let us write
1
fδ fn,δ fδ √ gδ ,
n
4.39
gδ δI Σ−1 Σg. The following is immediate from a minor modiwhere fδ δI Σ−1 f,
fication of Theorem 4.1.
Theorem 4.9. For fδ fn,δ as in 4.39, one has
d
√ 62 ,
n fδ − fδ −→ gδ H1 H
4.40
62 is obtained from H2 in 4.22 by replacing f with
where H1 ϕ1 ΣG0 is the same as in 4.13, H
6
f, and H1 ⊥⊥ H2 .
By way of an example, let us apply this result to testing the neighborhood hypothesis
and consider the asymptotics of the test statistics in 4.35 under the local alternatives fn,δ in
4.39 with
⊥ 2
Q fδ ε2 ,
gδ ⊥ M, fδ , gδ > 0.
4.41
Note that under such alternatives, fδn is still a consistent estimator of f for any δn ↓
0, as n → ∞, and so is fδ for fδ . The following is immediate from Theorem 4.9. To conclude
this section, let us first assume that parameters can be suitably estimated. We then arrive
at the limiting distribution of a test statistics that allows the construction of an asymptotic
level-α test whose asymptotic power can be computed.
Theorem 4.10. For fδ fn,δ , as in 4.40 and 4.41, one has
⎛ ⎞
⊥
√ ,
Q
2
g
f
δ
δ
2
n ⊥
d
⎟
⎜
, 1⎠,
Tn Q fδ − ε2 −→ N ⎝
Δ
Δ
as n −→ ∞,
4.42
is a consistent estimator of Δ in 4.36. Note that the limiting distribution is N0, 1
assuming that Δ
under H0 (i.e., gδ 0).
20
Advances in Decision Sciences
5. Asymptotic Optimality of the Classification Rule
In addition to Assumption 2.5 and 2.34, it will be assumed that the smoothness parameter
δ δn in 2.39 satisfies
δn −→ 0,
δn n−1/4 ,
as n −→ ∞.
5.1
We will also assume that the sizes of the training samples n1 and n2 see 2.36 are deterministic and satisfy n n1 n2 0 < lim inf
n→∞
nj
nj
≤ lim sup
< 1.
n n→∞
n
5.2
Let us recall from 3.7 that ϕ1 z ϕ1,n z 1/{δn z}, z /
− δn.
Under these conditions the probability of misclassification equals P{X − 1/2X 1 d
X 2 , ϕ1,n ΣX
1 − X 2 > 0 | X Gμ2 , Σ}. Let us note that
1
X − 1 X 1 X 2 , ϕ1,n Σ
X1 − X2
−
X
−
μ
−
μ
μ
,
ϕ
μ
Σ
1
2
1,n
1
2
2
2
:
9
1 ≤
X 1 − μ1 X 2 − μ2 ϕ1,n ΣL μ1 − μ2 2
;
1
× − ϕ1,n Σ
X
−
X
X
−
X
Σ
ϕ
X
1
2
1,n
1
2
L
2
:
9
<
X 1 − μ1 X 2 − μ2 ϕ1,n ΣL .
5.3
Since X j − μj Op n−1/2 , ϕ1,n ΣL Oδ−1 n, and, according to 3.21,
ϕ1,n Σ − ϕ1,n Σ Op
L
1
√ ,
δ2 n n
5.4
it follows from 5.1 that the limit of the misclassification probability equals
2
3
1
lim P X − μ2 − μ1 − μ2 , ϕ1,n Σ μ1 − μ2
>0
n→∞
2
1
,
μ1 − μ2 , Σ−1 μ1 − μ2
1−Φ
2
5.5
where Φ is the standard normal cdf.
For 5.5 we have used the well-known property of regularized inverses that δ Σ−1 Σf − f → 0, as δ → 0, for all f ∈ H, and the fact that we may choose f Σ−1 μ1 − μ2 by
Assumption 2.5. Since rule 2.33 is optimal when parameters are known, we have obtained
the following result.
d
Theorem 5.1. Let Pj Gμj , Σ and let Assumption 2.5 and 5.1 be satisfied. Then the classification
rule 2.39 is asymptotically optimal.
Advances in Decision Sciences
21
References
1 P. Hall and J. L. Horowitz, “Methodology and convergence rates for functional linear regression,” The
Annals of Statistics, vol. 35, no. 1, pp. 70–91, 2007.
2 H. Cardot, F. Ferraty, A. Mas, and P. Sarda, “Testing hypotheses in the functional linear model,”
Scandinavian Journal of Statistics, vol. 30, no. 1, pp. 241–255, 2003.
3 H. Cardot, F. Ferraty, and P. Sarda, “Functional linear model,” Statistics & Probability Letters, vol. 45,
no. 1, pp. 11–22, 1999.
4 T. Hastie, A. Buja, and R. Tibshirani, “Penalized discriminant analysis,” The Annals of Statistics, vol.
23, no. 1, pp. 73–102, 1995.
5 D. S. Gilliam, T. Hohage, X. Ji, and F. Ruymgaart, “The Fréchet derivative of an analytic function
of a bounded operator with some applications,” International Journal of Mathematics and Mathematical
Sciences, vol. 2009, Article ID 239025, 17 pages, 2009.
6 J. Cupidon, D. S. Gilliam, R. Eubank, and F. Ruymgaart, “The delta method for analytic functions of
random operators with application to functional data,” Bernoulli, vol. 13, no. 4, pp. 1179–1194, 2007.
7 R. L. Eubank and T. Hsing, “Canonical correlation for stochastic processes,” Stochastic Processes and
their Applications, vol. 118, no. 9, pp. 1634–1661, 2008.
8 G. He, H.-G. Miiller, and J.-L. Wang, “Functional canonical analysis for square integrable stochastic
processes,” Journal of Multivariate Analysis, vol. 85, no. 1, pp. 54–77, 2003.
9 S. E. Leurgans, R. A. Moyeed, and B. W. Silverman, “Canonical correlation analysis when the data are
curves,” Journal of the Royal Statistical Society. Series B, vol. 55, no. 3, pp. 725–740, 1993.
10 N. Dunford and J. T. Schwartz, Linear Operators, Part I: General Theory, Interscience, New York, NY,
USA, 1957.
11 J. Johannes, “Privileged communication,” 2008.
12 J. O. Ramsay and B. W. Silverman, Functional Data Analysis, Springer Series in Statistics, Springer,
New York, NY, USA, 2nd edition, 2005.
13 F. Ferraty and P. Vieu, Nonparametric Functional Data Analysis, Springer Series in Statistics, Springer,
New York, NY, USA, 2006.
14 D. Bosq, Linear Processes in Function Spaces, vol. 149 of Lecture Notes in Statistics, Springer-Verlag, New
York, NY, USA, 2000.
15 A. Mas, Estimation d’opérateurs de corrélation de processus fonctionnels: Lois limites, tests, déviations
modérées, Ph.D. thesis, Université Paris VI, 2000.
16 R. G. Laha and V. K. Rohatgi, Probability Theory, John Wiley & Sons, New York, NY, USA, 1979, Wiley
Series in Probability and Mathematical Statistic.
17 C. R. Rao, Linear Statistical Inference and Its Applications, John Wiley & Sons, New York, NY, USA, 1965.
18 A. Mas and B. Pumo, “Functional linear regression with derivatives,” Tech. Rep., Institut de
Modélisation Mathématique de Montpellier, 2006.
19 T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, Springer Series in
Statistics, Springer-Verlag, New York, NY, USA, 2001.
20 C. R. Rao and H. Toutenburg, Linear Models, Springer Series in Statistics, Springer, New York, NY,
USA, 1995.
21 J. Feldman, “Equivalence and perpendicularity of Gaussian processes,” Pacific Journal of Mathematics,
vol. 8, pp. 699–708, 1958.
22 Y. Hájek, “On a property of normal distribution of any stochastic process,” Czechoslovak Mathematical
Journal, vol. 8, no. 83, pp. 610–617, 1958.
23 U. Grenander, Abstract Inference, Wiley Series in Probability and Mathematical Statistic, John Wiley &
Sons, New York, NY, USA, 1981.
24 A. V. Skorohod, Integration in Hilbert Space, Springer, New York, NY, USA, 1974.
25 G. S. Watson, Statistics on Spheres, John Wiley & Sons, New York, NY, USA, 1983.
26 T. Kato, Perturbation Theory for Linear Operators, Die Grundlehren der mathematischen Wissenschaften, Band 132, Springer-Verlag, New York, NY, USA, 1966.
27 J. Dauxois, A. Pousse, and Y. Romain, “Asymptotic theory for the principal component analysis of
a vector random function: some applications to statistical inference,” Journal of Multivariate Analysis,
vol. 12, no. 1, pp. 136–154, 1982.
28 H. Cardot, A. Mas, and P. Sarda, “CLT in functional linear regression models,” Probability Theory and
Related Fields, vol. 138, no. 3-4, pp. 325–361, 2007.
22
Advances in Decision Sciences
29 J. L. Hodges, Jr. and E. L. Lehmann, “Testing the approximate validity of statistical hypotheses,”
Journal of the Royal Statistical Society. Series B, vol. 16, pp. 261–268, 1954.
30 H. Dette and A. Munk, “Validation of linear regression models,” The Annals of Statistics, vol. 26, no. 2,
pp. 778–800, 1998.
Advances in
Operations Research
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Advances in
Decision Sciences
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Mathematical Problems
in Engineering
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Journal of
Algebra
Hindawi Publishing Corporation
http://www.hindawi.com
Probability and Statistics
Volume 2014
The Scientific
World Journal
Hindawi Publishing Corporation
http://www.hindawi.com
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
International Journal of
Differential Equations
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Volume 2014
Submit your manuscripts at
http://www.hindawi.com
International Journal of
Advances in
Combinatorics
Hindawi Publishing Corporation
http://www.hindawi.com
Mathematical Physics
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Journal of
Complex Analysis
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
International
Journal of
Mathematics and
Mathematical
Sciences
Journal of
Hindawi Publishing Corporation
http://www.hindawi.com
Stochastic Analysis
Abstract and
Applied Analysis
Hindawi Publishing Corporation
http://www.hindawi.com
Hindawi Publishing Corporation
http://www.hindawi.com
International Journal of
Mathematics
Volume 2014
Volume 2014
Discrete Dynamics in
Nature and Society
Volume 2014
Volume 2014
Journal of
Journal of
Discrete Mathematics
Journal of
Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com
Applied Mathematics
Journal of
Function Spaces
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Optimization
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014