Empirical likelihood confidence regions for
comparison distributions and ROC curves
Gerda CLAESKENS1 , Bing-Yi JING2 , Liang PENG3 and Wang ZHOU2
Abstract
Empirical likelihood confidence regions are constructed for the comparison distribution of two random samples whose distributions are to be tested for equality. A second,
though related, application is to receiver operating characteristic curves, where the data
are measurements of a diagnostic test from two populations. The authors investigate
the smoothed empirical likelihood method for estimation in this context, and empirical
likelihood based confidence intervals are obtained via Wilks’s theorem. A bootstrap
approach allows for the construction of confidence bands. Empirical likelihood methods
are appealing for this purpose since they do not require any pre-specified assumptions
about the shape of the confidence regions. The method is illustrated via data analysis
and a simulation study.
Key words and phrases: Bootstrap; Comparison distribution; Confidence region;
Coverage accuracy; Empirical likelihood; ROC curve; Smoothing.
MSC 2000: Primary 62E20; secondary 62G15.
1
Department of Statistics, Texas A&M University, College Station, Texas 77843, U.S.A.
gerda@stat.tamu.edu
2 Department of Mathematics, Hong Kong University of Science and Technology, Hong Kong
majing@ust.hk, wangzhou@ust.hk
3
School of Mathematics, Georgia Institute of Technology, Georgia 30332, U.S.A.
peng@math.gatech.edu
1
1. INTRODUCTION
To compare two populations with distribution functions F1 and F2 , the comparison distribution (Parzen, 1998) is defined as D(t) = F1 (F2−1 (t)), for 0 ≤ t ≤ 1. Testing the null
hypothesis of equality of these distributions is equivalent to testing the comparison distribution D for uniformity. The graph of D is known as the percentile-percentile (PP) plot. A
graphical way of performing the test is to construct the PP-plot and check whether it lies on
the 45-degree line (Holmgren, 1995). A plot strictly above or below this line indicates that
one of the distributions is stochastically larger than the other. By constructing a confidence
band for the comparison distribution, a formal two-sample test can be performed. We will
use the method of smoothed empirical likelihood for this purpose. Kim (2000) constructs a
goodness of fit test based on the corresponding comparison density function (Parzen, 1979),
by testing whether this density function is equal to one.
The comparison distribution is also the starting point for control median tests, where
1 − D(0.5) gives the survival probability in population F1 , beyond the median in population
F2 . A confidence interval for 1−D(0.5) will be of particular interest here. For other treatments
of control median tests, refer to, for example, Chakraborti & Mukerjee (1989), Li, Tiwari &
Wells (1996) and references therein.
A second motivation for constructing confidence regions in this setting, is the receiver
operating characteristic (ROC) curve, which is an important tool to summarize the performance of a medical diagnostic test for determining whether or not a patient has a disease. In
a medical test resulting in a continuous measurement T , the disease is diagnosed if T > t, for
a given threshold t. Suppose the distribution function of T is F1 conditional on disease and
F2 conditional on non-disease. The ROC curve is defined as the graph (1 − F2 (t), 1 − F1 (t))
for various values of the threshold t, or in other words, sensitivity versus 1 - specificity, power
versus size for a test with critical region {T > t}. This enables one to summarize a test’s
performance or to compare two diagnostic tests. For more information about ROC curves and
their use, we refer to Swets & Pickett (1982), Pepe (1997) and Metz, Herman & Shen (1998).
An alternative definition (see Pepe, 1997) is R(t) = 1−F1 {F2−1 (1−t)} for 0 ≤ t ≤ 1. Indeed,
as also noted by Lloyd (1998), the ROC curve is the distribution function of F 0 = 1 − F2 (X1 ).
Distribution F0 can not be estimated directly since we only observe data from F1 and F2
separately.
To estimate the comparison distribution and ROC curve, there are several approaches.
One way is to model F1 and F2 parametrically (see Tosteston & Begg, 1988 and Goddard &
Hinberg, 1990). Hsieh & Turnbull (1996) estimate this curve empirically; denote by F 1n1 the
−1
empirical distribution function of a sample of size n1 from the disease population and let F2n
2
be the empirical quantile function of a sample of size n2 from the non-disease population. The
−1
(1 − t)}.
empirical or nonparametric estimator for R(t) is defined as R̂n (t) = 1 − F1n1 {F2n
2
For a semi-parametric approach to estimate the curves D and R, we refer to Li, Tiwari &
Wells (1999). Zou, Hall & Shapiro (1997) and Lloyd (1998) construct a smooth estimator
−1
e
(1 − t)} where the Feini ’s are smooth versions of
of R(t) by considering R(t)
= 1 − Fe1n1 {Fe2n
2
the Fini ’s, for example kernel distribution estimators. Lloyd & Yong (1999) show that the
e
kernel estimator R(t)
for R(t) is better than the empirical estimator R̂n (t) in terms of having
smaller mean squared error. In this paper we define a smooth estimator of both D and R by
2
using empirical likelihood methods.
The main motivation to employ empirical likelihood (Owen, 1988, 1990) above other
existing estimation methods, is the construction of confidence regions, which is the focus
point of this paper. Empirical likelihood methods are very appealing for this purpose since
they involve no pre-specified assumptions about the shape of the confidence regions, which is
determined automatically by the sample, as pointed out by Owen (1990). Empirical likelihood
regions are Bartlett correctable in most cases; refer to Hall & La Scala (1990) and DiCiccio,
Hall & Romano (1991). As an interesting alternative to the symmetric confidence regions
imposed by a normal approximation method, it also avoids the need to estimate the variance
term which contains both the unknown curve of interest and its first derivative. Confidence
intervals based on likelihood ratios also have the additional advantage of respecting the range
of the parameter. A recent overview of empirical likelihood methods and applications is given
by Owen (2001).
We extend the method of empirical likelihood to obtain a nonparametric estimator of
the comparison distribution function, which is a combination of a distribution function and a
quantile function. Chen & Hall (1993) introduce the method of smoothed empirical likelihood
for the construction of confidence intervals for quantiles. Likelihood ratio based confidence
intervals and bands for survival and quantile functions are constructed by Li (1995), Murphy
(1995), Li, Hollander, McKeague & Yang (1996), and Hollander, McKeague & Yang (1997)
amongst others. Einmahl & McKeague (1999) construct confidence tubes for quantile-quantile
(QQ) plots, which in essence involves a study of the function F1−1 (F2 (t)). PP-plots are to
be preferred above QQ-plots when outliers might be present, and as an additional advantage
they allow for scale invariant treatment comparisons (Holmgren, 1995).
The structure of this paper is as follows. Section 2 defines and studies the smoothed
empirical likelihood estimator for the comparison distribution and ROC curve. In Section 3
the empirical likelihood based confidence interval is presented, Section 4 applies a bootstrap
method to construct confidence bands. A simulation study is conducted in Section 5.1 to
compare the empirical likelihood method and the normal approximation method. In Section 5.2 we apply the smoothed empirical likelihood method to a dataset on the detection of
pancreatic cancer. All proofs are deferred until the appendix.
2. ESTIMATION OF THE COMPARISON DISTRIBUTION AND ROC CURVE
This section is devoted to point estimation. The smoothed empirical likelihood function and
its score vector for this two sample setting are introduced and defined in Section 2.1. Estimation of the comparison distribution and values of the ROC curve is the topic of Section 2.2.
2.1 Notation and definitions
Suppose that X11 , . . . , X1n1 and X21 , . . . , X2n2 are two random samples from populations
with (unknown) distribution functions F1 and F2 , respectively. We wish to construct a point
or interval estimator for the ROC value R(t) = 1 − F1 (F2−1 (1 − t)), where 0 ≤ t ≤ 1. Note
that the comparison distribution function is related to the ROC function in the following way:
D(t) = 1 − R(1 − t), 0 ≤ t ≤ 1. Therefore, the results below will be presented only for the
ROC function R.
3
In order to construct the empirical likelihood estimator of R(t), for j = 1, 2, let G j be a
smoothed version of the distribution function represented by the indicator function I{t > 0}.
More specifically, for KRj a compactly supported r-th order kernel (r ≥ 2, see condition C3
below), define Gj (t) = u≤t Kj (u)du. If we take Kj a density function and r = 2, Gj is a well
defined distribution function. For future use, define Ghj (t) = Gj (t/hj ) where hj = hj (nj ) > 0
is a bandwidth sequence, converging to zero as nj grows to infinity.
Let p = (p1 , · · · , pn1 ) and q = (q1 , · · · , qn2 ) be two probability vectors, that is, consisting
of nonnegative numbers adding to one. Define the estimators
F̂h1 ,p (η) =
n1
X
i=1
pi Gh1 (η − X1i ) and F̂h2 ,q (η) =
n2
X
j=1
qj Gh2 (η − X2j ).
For ROC curve estimation, where we denote θ = R(t), there clearly exists a value η such that
η = F1−1 (1 − θ) = F2−1 (1 − t).
(1)
For this setting, we define the smoothed empirical likelihood for θ, as
L(θ) =
sup
(p,q,η)
Ãn
1
Y
i=1

! n
2
Y
p i  qj  ,
j=1
subject to the following constraints
F̂h1 ,p (η) = 1 − θ and F̂h2 ,q (η) = 1 − t.
(2)
2
Note that without constraints (2), the maximum of L(θ) would be n1−n1 n−n
. To better
2
appreciate this definition, consider the second constraint which is used in the non-disease
population F2 only, to estimate the quantile η = F2−1 (1 − t) via empirical likelihood. For
ROC curve estimation, however, we add the first constraint, which determines the estimation
of the 1−θ quantile in the disease population F1 . A challenging aspect of this approach is that
the parameter of interest is not this quantile, but instead R(t) = θ. The link between both
populations, disease and non-disease is through the receiver operator characteristic relation
(1). A similar motivation holds for the comparison distribution function D.
Chen & Hall (1993) introduced smoothed empirical likelihood for the construction of
confidence intervals for quantiles in one population only. The current setting requires a nontrivial extension of their results. Einmahl & McKeague (1999) define an empirical likelihood
approach for the estimation of multiple quantile-quantile plots from independent samples. A
quantile function Q differs from the comparison distribution function used as the basis of a PPplot in that the order of the quantile and distribution function is reversed; more specifically,
Q(t) = F1−1 (F2 (t)), while D(t) = F1 (F2−1 (t)). For this reason, a different theoretical treatment
is required.
We now define the empirical log likelihood ratio (multiplied by minus two) as
`(θ) = −2
n1
³X
ln (n1 pi ) +
n2
X
j=1
i=1
4
´
ln (n2 qj ) .
To make this definition more explicit, we define the weight functions
w1 (η, X1i ) = Gh1 (η − X1i ) − (1 − θ), and w2 (η, X2j ) = Gh2 (η − X2j ) − (1 − t).
By the method of Lagrange multipliers, also taking the probability contraints into account,
P 1
P 2
that is, ni=1
pi = 1, nj=1
qj = 1 and pi , qj ≥ 0, it easily follows that
1
n1
1
=
n2
pi =
qj
1
,
1 + λ1 w1 (η, X1i )
1
·
,
1 + λ2 w2 (η, X2j )
·
i = 1, . . . , n1 ,
j = 1, . . . , n2 ,
where (λ1 , λ2 , η) are solutions to the following three score equations
nj
X
∂
wj (η, Xji )
= 0 (j = 1, 2),
`(θ) =
∂λj
i=1 1 + λj wj (η, Xji )
(3)
nj
2
X
X
wj0 (η, Xji )
∂
`(θ) =
= 0,
λj
∂η
j=1
i=1 1 + λj wj (η, Xji )
with
wj0 (η, z)
∂wi (η, z)
= h−1
=
j Kj
∂η
Ã
η−z
hj
!
(4)
for j = 1, 2.
The empirical log likelihood ratio can now be rewritten as
`(θ) = 2
nj
2 X
X
ln (1 + λj wj (η, Xji )) .
j=1 i=1
To ease interpretation, in the above notation we suppressed the dependence of (λ1 , λ2 , η) on
θ, which takes place via the function w1 .
2.2 Point estimation
Although the focus of this paper is on the construction of confidence regions, we wish to
mention the point estimator itself.
The maximum smoothed empirical likelihood estimator is that value θ̂ for which the
smoothed empirical likelihood function is maximized:
θ̂ = arg max L(θ).
θ
We can obtain an estimator of θ by minimizing the empirical log likelihood ratio l(θ) given
above.
Throughout, we assume that the following conditions hold for j = 1, 2.
(r−1)
(C1) For each density function fj = Fj0 , there exists an integer r ≥ 2 such that fj
in a neighborhood of η and is continuous at η. Furthermore, f1 (η)f2 (η) > 0.
5
exists
(C2) As min(n1 , n2 ) → ∞, nj /(n1 + n2 ) → γj , where 0 < γj < 1.
(C3) Let Kj (·) denote an rth-order (r ≥ 2) kernel satisfying
Z


 1,
k = 0,
1 ≤ k ≤ r − 1,
u Kj (u)du =  0,

c 6= 0, k = r.
k
Let us explain these conditions some more. Condition (C1) requires that the densities
fj (j = 1, 2) be sufficiently smooth only in a neighborhood of η. The requirement that the
densities are positive at η assures the correct order (n1 + n2 )−1 for the asymptotic variance
of the estimator. Condition (C2) is the balancing condition between the two sample sizes,
stating that their growth rate should be of equal order. Clearly, γ 1 + γ2 = 1. The kernel
functions given in (C3) are of the type commonly used in nonparametric density or regression.
The case r = 2 is the most common, where we can take Kj to be a symmetric probability
density function.
Theorem 1. With C representing either the ROC function R(·) = 1 − F1 {F2−1 (1 − ·)} or the
comparison distribution D(·) = F1 {F2−1 (·)}, the following asymptotic normality result holds.
Assume regularity conditions (C1)–(C3), if n1 h2r
1 → 0, the smoothed empirical likelihood
estimator Ĉ(t) satisfies
√
n1 (Ĉ(t) − C(t)) →D N (0, C(t){1 − C(t)} + {C 0 (t)}2 t(1 − t)γ1 /γ2 ).
In the above, “→D ” denotes convergence in distribution. Notice that the smoothed empirical likelihood estimator has the same asymptotic behavior as the kernel estimator studied
by Lloyd & Yong (1999). The reason is that there is no extra information involved in the
empirical likelihood function. The proof of this theorem requires Taylor series expansions
around θ, similar to those in the proof of Theorem 2, and is similar to asymptotic normality results obtained by Qin and Lawless (1994). Hence we do not include the proof here.
The leading terms of the asymptotic variance of this estimator also coincide with the leading
−1
(·)}. Note that for estimation
asymptotic variance terms of the empirical estimator F1n1 {F2n
2
of the comparison distribution function, the variance features the squared comparison density
function (Parzen, 1979, 1998).
Under the condition that n1 h2r
1 → 0 there is no bias in the above asymptotic normality
result. A bandwidth sequence tending to zero at a slower rate would introduce a bias originating from the smoothing procedures. Optimal bandwidth choice is an interesting topic, but
is beyond the scope of this paper.
The point estimator R(·) can be used to construct summary values such as the area under
the ROC curve (Hanley & McNeil, 1982). This is an important quantity which summarizes the
accuracy of a test. The area under the R(·) curve is equal to P (X1 − X2 > 0), and is related
to the Wilcoxon statistic used for testing whether P (X1 − X2 > 0) = 0.5. RAn estimator
of this probability can be obtained by numerical evaluation of the integral 01 R̂(t)dt. An
6
alternative is to estimate first the distribution of X1 − X2 via
F̂j are estimators of the distribution functions Fj .
R∞
−∞
F̂1 (s + t)dF̂2 (t) where the
3. WILKS’S THEOREM FOR CONFIDENCE INTERVAL CONSTRUCTION
In this section we construct an empirical likelihood based confidence interval for the ROC
value R(t) and for the value of the comparison distribution D(t). The same notation as
introduced in Section 2 is adapted throughout.
The idea behind confidence interval construction based on (empirical) likelihood ratios is
to include in the interval all those values ϑ for which the (empirical) likelihood ratio test for
the null hypothesis H0 : R(t) = ϑ, or H0 : D(t) = ϑ, respectively, cannot be rejected. By
definition, see Section 2.1, large values of the empirical log likelihood ratio `(ϑ) provide evidence in favor of the (two-sided) alternative hypothesis. This motivates defining a smoothed
empirical likelihood confidence interval I for either R(t) or D(t) by
Ic = {ϑ : `(ϑ) ≤ c}.
The value of the constant c > 0 determines the coverage probability of the interval I c . Its
value will be determined via Wilks’s theorem, below.
For the construction of a smoothed empirical likelihood ratio confidence interval, the
explicit maximization of the likelihood function can be avoided. To construct the empirical
likelihood function as described in Section 2.1, only equations (3)–(4) need to be solved,
solutions of which may, for practical matters, be obtained in two stages. In the first stage,
we fix η and solve for λ1 = λ1 (η) and λ2 = λ2 (η) from the set of equations (3). In the second
stage, we obtain η̂ as a solution to equation (4) where now λj = λj (η). It is straightforward
to obtain that this provides identical solutions to a direct “one-stage” solution to (3)–(4).
In addition to regularity conditions (C1)–(C3), presented in Section 2.2, we assume condition (C4), which dictates the rates at which the bandwidths tend to zero, and ensures that
this convergence is fast enough.
2r
(C4) For i = 1, 2: ni h4r
i → 0, and ni hi / ln ni → ∞ as ni → ∞.
First, we state two lemmas which will facilitate obtaining the main result in Theorem 2.
Lemma 1. Assume that conditions (C1) − (C4) hold. Then, with probability one, there exists
a root ηe of equation (4), such that |ηe − η| < δ a.s. where δ = O(hr1 + hr2 ).
Lemma 2. Assume that conditions (C1) − (C4) hold. Then, for ηe given in Lemma 1,
γ1 f1 (η)λ1 (ηe) = −γ2 f2 (η)λ2 (ηe) + op (n−1/2 )
³
´
√
N (ηe − η) →D N 0, d−1 θ(1 − θ)t(1 − t)
Ã
!
√
γ2 f2 (η)2
N λ1 (ηe) →D N 0,
,
γ1 d
!
Ã
√
γ1 f1 (η)2
N λ2 (ηe) →D N 0,
,
γ2 d
7
where d = γ1 t(1 − t)f12 (η) + γ2 θ(1 − θ)f22 (η) and N = n1 + n2 .
Both proofs, as well as the proof of the following theorem are given in the Appendix.
Theorem 2. Under conditions (C1) − (C4), the empirical log likelihood ratio statistic `(θ)
converges in distribution to a χ21 distributed random variable.
Theorem 2 is the nonparametric version of Wilks’s theorem in the ROC and comparison
distribution context. Note that the limit distribution of `(θ) does not depend on unknown
quantities, and so is asymptotically pivotal. Based on Theorem 2, an effective approach to
construct a (1 − α)-level confidence interval for θ is
Ic1−α = {ϑ : `(ϑ) ≤ c1−α },
where c1−α is the 1 − α quantile of a χ21 distribution. This will give a confidence interval for
θ with asymptotically correct coverage probability (1 − α), as stated in the next corollary.
Corollary 1. Assume that conditions (C1) − (C4) hold. Then it follows that
³
´
P θ ∈ Ic1−α = (1 − α) + o (1) .
Empirical likelihood based confidence intervals not only have as an advantage that their
shape is determined by the data, the method also avoids searching for a transformation
resulting in less skewness of the distributions, a requirement for the normal approximation to
perform well. See, Zhou, Hall & Shapiro (1997) who apply a logit transformation to 1 − F 1 (t)
and 1 − F2 (t).
4. BOOTSTRAP CONFIDENCE BANDS
In order to compare two diagnostic tests via their ROC curve, or to compare two distributions
via their comparison distribution function, pointwise confidence intervals are not optimal.
Instead of developing asymptotic properties for global curve estimation, which would be one
way to construct a confidence region for {R(t) : a ≤ t ≤ b} and {D(t) : a ≤ t ≤ b}, a
bootstrap algorithm provides an interesting alternative. In the context of empirical likelihood
estimation for density functions, Hall & Owen (1993) explain that very large sample sizes are
needed to get accurate confidence regions based on an infinite-parameter version of Wilks’s
theorem. In the current context, if anything, this is not expected to be better. The main
reason for the large sample sizes is the known slow convergence of normal extremes (see for
example Hall, 1979).
Lloyd (1998) uses percentiles of a nonparametric bootstrap to construct pointwise confidence bands for the ROC curve, this as an alternative to the pointwise normal approximation.
Li, Tiwari & Wells (1996, 1999) construct a confidence band with global coverage 1-α by determining the appropriate percentile via bootstrap resampling and construct the band symmetrically around the estimated curve. Our construction provides a bootstrap confidence region
8
without losing the advantages of automated shape-determination by the empirical likelihood
method. The level of the confidence bands is set by means of the bootstrap. This method is
similar to one presented by Hall & Owen (1993) in the context of density estimation.
The bootstrap method proceeds as follows. We nonparametrically draw a resample from
∗
∗
∗
∗
each of the two samples separately to obtain {X11
, . . . , X1n
} and {X21
, . . . , X2n
}. Using
1
2
these bootstrap data, we form the bootstrap smoothed empirical log-likelihood ratio
`∗ (θ) = 2
nj
2 X
X
j=1 i=1
³
´
ln 1 + λ∗j wj∗ (η ∗ , Xji∗ ) ,
where, for the ROC curve R(t) = θ,
∗
∗
∗
∗
w1∗ (η ∗ , X1i
) = Gh1 (η ∗ − X1i
) − (1 − θ), and w2∗ (η ∗ , X2j
) = Gh2 (η ∗ − X2j
) − (1 − t).
Analogously, for the comparison distribution D(t) = θ,
∗
∗
∗
∗
w1∗ (η ∗ , X1i
) = Gh1 (η ∗ − X1i
) − θ, and w2∗ (η ∗ , X2j
) = Gh2 (η ∗ − X2j
) − t.
The bootstrap values λ∗1 , λ∗2 and η ∗ are solutions to
∂ ∗
∂ ∗
` (θ) = 0,
` (θ) = 0 (j = 1, 2).
∂λj
∂η
To construct a simultaneous confidence region for C(·), with C representing either D or R,
over the interval (a, b), first we calculate the appropriate bootstrap critical value c ∗ such that
P {`∗ (Ĉ(t)) ≤ c∗ for a ≤ t ≤ b|Xji , j = 1, 2; i = 1, . . . , nj } = 1 − α,
where Ĉ(t) is the smoothed empirical likelihood estimator derived in Section 2.2. Hence the
bootstrap confidence band C consists of those curves C(·) such that the corresponding log
likelihood ratio statistic stays below c∗ over the interval (a, b), that is,
C = {C(·) : `(C(t)) ≤ c∗ for a ≤ t ≤ b}.
This bootstrap method is applied in Section 5.2 to the dataset on pancreatic cancer.
5. SIMULATION STUDY AND DATA ANALYSIS
In this section we apply the smoothed empirical likelihood method. First, in Section 5.1 in
the setting of ROC curves, we simulate data, and compare the simulated pointwise coverage probabilities of the smoothed empirical likelihood method to those obtained by normal
approximations to kernel and empirical estimators, as well as by using bootstrap percentile
techniques. In Section 5.2 we construct simultaneous confidence regions for the ROC curve
and comparison distributions for a dataset from a study on pancreatic cancer.
5.1 Simulation study
9
In this section, we report results of a simulation study to investigate the performance of the
smoothed empirical likelihood method for the construction of confidence intervals for the ROC
values. In particular, we conducted a Monte Carlo study to compare the coverage accuracy
of confidence intervals by the empirical likelihood method, by normal approximations and by
using bootstrap percentiles.
Confidence intervals by means of normal approximations based on the empirical estimator
R̂n (t), see Introduction, can be obtained as follows. It can be verified that the asymptotic
variance of R̂n (t) is given by
(
R(t)(1 − R(t))
f1 (F2−1 (1 − t))
V (t) =
+
n1
f2 (F2−1 (1 − t))
)2
t(1 − t)
.
n2
A consistent estimator of V (t), say V̂ (t), can be obtained by replacing R(t), the Fi (·)’s and
fi (·)’s in the above formula by their empirical versions (or smoothed ones when appropriate).
Then, from Theorem 2 of Lloyd & Yong (1999), a two-sided confidence interval based on the
normal approximation of R̂(t) is constructed via
(N )
I1−α
·
q
q
¸
= R̂n (t) − zα/2 V̂ (t), R̂n (t) + zα/2 V̂ (t) ,
where zα/2 is the 1 − α/2 quantile of a standard normal distribution. In order to avoid
additional bandwidth choices for estimating unknown densities involved in estimation of the
variance, we draw 300 bootstrap resamples of size n1 from X11 , · · · , X1n1 and 300 bootstrap
resamples of size n2 from X21 , · · · , X2n2 , and then construct a bootstrap variance estimator for
estimation of V (t). Similarly, we obtain a confidence interval by normal approximation, based
P 1
−1
Gh1 (x − X1i )
on the kernel estimator R̃(t) = 1 − F̃1n1 {F̃2n
(1 − p)}, where F̃1n1 (x) = n11 ni=1
2
1 P n2
and F̃2n2 (x) = n2 i=1 Gh2 (x − X2i ). The bootstrap percentile method based on either kernel
estimator or empirical estimator was conducted by drawing 300 bootstrap resamples of size
n1 from X11 , · · · , X1n1 and 300 bootstrap resamples of size n2 from X21 , · · · , X2n2 .
We generated 10,000 pseudorandom samples of various realistic sample sizes from F1 =
N (1, 1) and F2 = N (0, 1). We used the biquadratic kernel throughout, that is K(t) =
−3/20
−3/20
15
(1 − t2 )2 I{|t| ≤ 1}, and we employed bandwidths h1 = n1
and h2 = n2
. It is easy to
16
check that conditions (C1) − (C4) are satisfied. Confidence intervals are calculated for values
of t equal to 0.1, 0.3, 0.5, 0.7 and 0.9. Table 1 presents the results of this simulation study for
a nominal coverage level equal to 90%, results for 95% nominal coverage are given in Table 2.
From Tables 1 and 2, we make the following observations. The smoothed empirical likelihood (E.L.) method outperforms the normal approximation method and bootstrap percentile
method by showing simulated coverage probabilities closer to the nominal value, for both
90% and 95% nominal coverage probabilities. The bootstrap percentile method is better than
normal approximations. For almost all cases the normal approximation method based on empirical estimators has better coverage accuracy than the normal approximation method based
on kernel estimators. The relatively poor performance for the case t = 0.9 may be explained
by the fact that R(0.9) is very close to 99%. Therefore, neither method can be expected to
work very well in this case. In summary, the smoothed empirical likelihood based confidence
intervals for the ROC curve have better coverage probabilities than confidence intervals based
10
Table 1: Coverage accuracy of confidence intervals with nominal level 0.90. Results from a
simulation study. E.L.: smoothed empirical likelihood; N.A.E.: normal approximation based
on empirical estimators; N.A.K.: normal approximation based on kernel estimators; B.P.E.:
bootstrap percentile based on empirical estimators; B.P.K.: bootstrap percentile based on
kernel estimators.
n1 = n2 = 50
n1 = n2 = 70
n1 = n2 = 100
n1 = 50, n2 = 70
n1 = 70, n2 = 50
Method t = 0.1
E.L.
0.903
N.A.E.
0.885
N.A.K.
0.871
B.P.E.
0.909
B.P.K.
0.893
E.L.
0.899
N.A.E.
0.885
N.A.K.
0.874
B.P.E.
0.909
B.P.K.
0.894
E.L.
0.898
N.A.E.
0.882
N.A.K.
0.872
B.P.E.
0.896
B.P.K.
0.888
E.L.
0.896
N.A.E.
0.888
N.A.K.
0.875
B.P.E.
0.914
B.P.K.
0.897
E.L.
0.896
N.A.E.
0.882
N.A.K.
0.871
B.P.E.
0.907
B.P.K.
0.896
t = 0.3
0.900
0.883
0.879
0.917
0.895
0.897
0.888
0.884
0.911
0.894
0.897
0.892
0.883
0.912
0.890
0.891
0.880
0.881
0.918
0.891
0.898
0.883
0.883
0.909
0.899
t = 0.5
0.896
0.881
0.882
0.918
0.892
0.896
0.883
0.886
0.908
0.893
0.893
0.887
0.884
0.908
0.886
0.891
0.884
0.884
0.918
0.891
0.894
0.888
0.887
0.910
0.896
t = 0.7
0.899
0.885
0.872
0.910
0.887
0.895
0.895
0.880
0.917
0.892
0.896
0.889
0.881
0.910
0.884
0.892
0.878
0.874
0.902
0.887
0.895
0.903
0.886
0.923
0.897
t = 0.9
0.826
0.798
0.748
0.748
0.759
0.826
0.807
0.766
0.790
0.783
0.835
0.824
0.800
0.833
0.821
0.805
0.755
0.715
0.707
0.739
0.841
0.848
0.794
0.818
0.807
on normal approximations and bootstrap percentile methods. This result is consistent with
applications of empirical likelihood based confidence intervals for other population values.
5.2 Data analysis
We apply our method in Section 4 to the data set analyzed by Wieand et al. (1989) on the
accuracy of a carbohydrate antigenic determinant (CA19-9) in detecting pancreatic cancer.
Concentrations of CA19-9 in sera (mL) from n2 =51 ‘control’ patients with pancreatitis and
n1 =90 ‘case’ pancreatic cancer were collected. We are interested in obtaining a simultaneous
confidence band for the ROC curve of CA19-9 in distinguishing between a control patient
11
Table 2: Coverage accuracy of confidence intervals with nominal level 0.95. Results from a
simulation study. E.L.: smoothed empirical likelihood; N.A.E.: normal approximation based
on empirical estimators; N.A.K.: normal approximation based on kernel estimators; B.P.E.:
bootstrap percentile based on empirical estimators; B.P.K.: bootstrap percentile based on
kernel estimators.
n1 = n2 = 50
n1 = n2 = 70
n1 = n2 = 100
n1 = 50, n2 = 70
n1 = 70, n2 = 50
Method t = 0.1
E.L.
0.948
N.A.E.
0.930
N.A.K.
0.923
B.P.E.:
0.949
B.P.K.:
0.945
E.L.
0.951
N.A.E.
0.934
N.A.K.
0.927
B.P.E.
0.952
B.P.K.
0.943
E.L.
0.945
N.A.E.
0.932
N.A.K.
0.926
B.P.E.
0.949
B.P.K.
0.939
E.L.
0.947
N.A.E.
0.937
N.A.K.
0.929
B.P.E.
0.953
B.P.K.
0.945
E.L.
0.946
N.A.E
0.930
N.A.K.
0.923
B.P.E.
0.946
B.P.K.
0.943
t = 0.3
0.950
0.934
0.931
0.957
0.943
0.949
0.937
0.934
0.953
0.942
0.948
0.939
0.936
0.957
0.942
0.946
0.938
0.932
0.959
0.943
0.947
0.935
0.932
0.957
0.946
t = 0.5
0.952
0.928
0.929
0.959
0.944
0.946
0.930
0.933
0.951
0.941
0.945
0.935
0.932
0.954
0.938
0.945
0.929
0.931
0.958
0.940
0.945
0.935
0.935
0.956
0.945
t = 0.7
0.945
0.931
0.913
0.954
0.939
0.950
0.939
0.925
0.958
0.941
0.946
0.938
0.929
0.954
0.937
0.938
0.926
0.916
0.951
0.936
0.949
0.942
0.926
0.961
0.943
t = 0.9
0.880
0.824
0.775
0.801
0.817
0.887
0.836
0.794
0.842
0.842
0.892
0.856
0.831
0.877
0.871
0.867
0.779
0.744
0.757
0.794
0.896
0.873
0.825
0.873
0.864
and case patient, and for this purpose apply the bootstrap method as given in Section 4.
The same kernel function as in Section 5.1 was used and a total of B = 10, 000 bootstrap
resamples were taken. In Figure 1, confidence bands for the ROC curve R(p) with confidence
levels 0.90 and 0.95 are plotted against p ranging from 0.20 to 0.99 by employing bandwidths
h1 = 1.0, h2 = 0.2 (upper panel) and h1 = 10.0, h2 = 0.2 (lower panel), respectively. The
reason of choosing large h1 is the sparseness of disease data. Also the empirical estimator and
the kernel estimator are plotted in Figure 1.
A graphical way of performing a test for the equality of disease and non-disease distribution
functions is the following. If the 100(1 − α)% confidence band contains the bisector, a test
12
0.7
0.6
R(p)
0.8
0.9
1.0
h1=1.0, h2=0.2
0.4
0.5
Empirical estimator
Kernel estimator
90% confidence band
95% confidence band
0.2
0.4
0.6
0.8
1.0
p
0.7
0.6
R(p)
0.8
0.9
1.0
h1=10.0, h2=0.2
0.4
0.5
Empirical estimator
Kernel estimator
90% confidence band
95% confidence band
0.2
0.4
0.6
0.8
1.0
p
Figure 1: Confidence bands, kernel estimator and empirical estimator for the ROC curve.
Confidence bands with level 90% and 95%, and kernel estimator and empirical estimator are
calculated for bandwidths h1 = 1 (upper panel), h1 = 10 (lower panel) and h2 =0.2.
for H0 : F1 = F2 cannot be rejected at a significance level α. In our data example, it is clear
that in this case there is evidence to conclude that the disease and non-disease population
distributions are significantly different at the 5% level.
13
APPENDIX
First we introduce a few lemmas, which will be useful in proving the main theorem. Throughout the proofs, we let c denote a generic constant, which may assume a different value at each
occurrence. Denote n = min{n1 , n2 }, N = n1 + n2 , h = max{h1 , h2 }, and for j = 1, 2,
wj1 (η) =
nj
1 X
wj (η, Xji ),
nj i=1
wj2 (η) =
nj
1 X
w2 (η, Xji ).
nj i=1 j
Also, we shall use δ = δn = O(hr ) to denote a nonrandom nonnegative sequence. Proofs
are given in the context of ROC estimation, it is straightforward to formulate the proofs for
estimation of a comparison distribution D.
A.1 Technical lemmas.
Lemma 3.Assume that conditions (C1) and (C3) hold. For each fixed η, we have that, for
i = 1, 2 and j = 1, 2,
E{wi (η, Xj1 )} = Fi (η) − Fi (η0 ) + o(hrj ),
Var{wi (η, Xj1 )} = Fi (η){1 − Fi (η)} + O(hj ).
Proof. Without loss of generality, let i = j = 1. For the expected value of wi (η, Xj1 ) we
calculate
Z ∞
η − X11
[F1 (η − h1 u) − F1 (η)] K1 (u)du
)} − F1 (η) =
E{G1 (
h1
−∞
·
¸
Z ∞
1 0
1 (r−1)
2
r
r
f1 (η)(−h1 u) + f1 (η)(−h1 u) + . . . + f1
=
(η)(−h1 u) + o(h1 ) K1 (u)du = o(hr1 ).
2
r!
−∞
And for the second moment,
E
½
G21
µ
η − X11
h1
= F1 (η)
Z
∞
−∞
¶¾
=2
dG21 (u)
= F1 (η) + O(h1 ).
Z
∞
(F1 (η) − h1 uf1 (η 0 )) G1 (u)dG1 (u)
−∞
Z ∞
−
−∞
h1 uf1 (η 0 )G1 (u)dG1 (u)
Therefore, Var{w1 (η, X11 )} = F1 (η) − F12 (η) + O(h1 ), which ends the proof.
Lemma 4. Assume that conditions (C1), (C3) and (C4) hold. Then uniformly for η ∈ {η :
|η − η0 | < δ}, for j = 1, 2,
−1/2
wj1 (η) = o(hrj ) + Op (δ + nj
)
(5)
−1/2
wj2 (η) = Fj (η0 ) (1 − Fj (η0 )) + Op (δ + hj + nj
14
)
(6)
Proof. We shall only prove the case j = 1. By Taylor expansion,
1
w11 (η) = w 11 (η0 ) + w011 (η0 )(η − η0 ) + w0011 (η 0 )(η0 − η0 )2 ,
2
(7)
where η 0 is between η0 and η. From the central limit theorem and Lemma 3, it is obtained
that
−1/2
w11 (η0 ) = E{w1 (η0 , X11 )} + Op (n1
−1/2
) = o(hr1 ) + Op (n1
).
The strong law of large numbers implies that w 011 (η0 ) = f1 (η0 )+O(hr1 ). From the boundedness
of the kernel by condition (C3) and the bandwidth condition in (C4), it follows that there
exists a constant c such that
¯
µ
¶¯
¯ 0 η − X1i ¯
¯K
¯ ≤ c,
¯ 1
¯
and
h1
h−2
1 δ ≤ c,
which, in turn, implies that
¯
¯
¯ 00
¯
¯w 11 (η 0 )(η0 − η0 )2 ¯ ≤
Ã
n1
η 0 − X1i
δ2 X
0
K
h21 n1 i=1 1
h1
!
= O(δ).
(8)
The result of (8), together with (7) proves (5). Similarly, we can prove (6).
Lemma 5. Assume that conditions (C1), (C3) and (C4) hold. Then for j = 1, 2,
³
−1/2
wj1 (η0 + δ) = o(hrj ) + O δ + nj
(ln nj )1/2
³
´
a.s.
−1/2
wj2 (η0 + δ) = Fj (η0 ) (1 − Fj (η0 )) + O δ + hj + nj
(ln nj )1/2
´
a.s.
Proof. We shall only prove the case j = 1. An application of Bernstein’s inequality (see, e.g.,
Serfling, 1980, p.95), and the Borel-Cantelli lemma imply that .
−1/2
|w1k (η0 + δ) − Ew 1k (η0 + δ)| = O(n1
(ln n1 )1/2 ),
a.s.,
k = 1, 2.
Combining this with Lemma 3, yields
³
−1/2
w11 (η0 + δ) = o(hr1 ) + O δ + n1
(ln n1 )1/2
³
´
a.s.
−1/2
w12 (η0 + δ) = F1 (η0 ) (1 − Fj (η0 )) + O δ + h1 + n1
which completes the proof.
(ln n1 )1/2
´
a.s.,
Lemma 6. Assume that conditions (C1) − (C4) hold. Then uniformly for η ∈ {η : |η − η 0 | <
δ}, for j = 1, 2,
³
−1/2
λj (η) = o(hrj ) + Op δ + nj
15
´
;
(9)
furthermore, on the boundary points, we have
³
−1/2
λj (η0 ± δ) = o(hrj ) + O δ + nj
(ln nj )1/2
´
a.s.
(10)
Proof. We only prove the case j = 1. Rewrite (3) as
n1
1 X
w12 (η, X1i )
w11 (η) − λ1 (η)
= 0,
n1 i=1 1 + λ1 (η)w1 (η, X1i )
(11)
and define Zn = max1≤i≤n |w1 (η, X1i )|. Clearly, there exists a constant c > 0 such that
0 ≤ Zn ≤ c. An application of Lemma 4 to (11), gives the following result,
n1
w12 (η, X1i )
1 X
−1/2
= o(hr1 ) + Op (δ + n1 )
|λ1 (η)| ·
n1 i=1 1 + λ1 (η)w1 (η, X1i )
|λ1 (η)|
|λ1 (η)|
≥
· [t0 (1 − t0 ) + op (1)]
· w12 (η) =
1 + |λ1 (η)|Zn
1 + c|λ1 (η)|
³
−1/2
This implies that λ1 (η) = o(hr1 ) + Op δ + n1
Now let us prove (10). From Lemma 5,
³
´
, which proves (9).
−1/2
w11 (η0 + δ) = o(hr1 ) + O δ + n1
³
(ln n1 )1/2
´
−1/2
w12 (η0 + δ) = t0 (1 − t0 ) + O δ + h1 + n1
a.s.
(ln n1 )1/2
Following similar steps as those to obtain (12), it can be shown that
³
−1/2
λ1 (η0 + δ) = o(hr1 ) + O δ + n1
³
−1/2
Similarly, λ1 (η0 − δ) = o(hr1 ) + O δ + n1
A.2 Proof of Lemma 1.
(12)
(ln n1 )1/2
´
´
(13)
´
a.s.
(14)
a.s.
(ln n1 )1/2 a.s.
−1/2
For simplicity, write ∆δ = δ + hr1 + n1 (ln n1 )1/2 . Also denote H(θ0 , η) = n1 H1 (θ0 , η) +
n2 H2 (θ0 , η), where
nj
1 X
Hj (θ0 , η) =
ln (1 + λj (η)wj (η, Xji )) .
nj i=1
By a Taylor expansion and (10), it follows that
1
H1 (θ0 , η0 + δ) = λ1 (η0 + δ) w11 (η0 + δ) − λ21 (η0 + δ) w12 (η0 + δ) + O(∆3δ ) a.s.
2
(15)
From (11), it follows that
0 = w11 (η0 + δ) − λ1 (η0 + δ)w12 (η0 + δ) + λ21 (η0 + δ)
16
n1
1 X
w13 (η0 + δ, X1i )
.
n1 i=1 1 + λ1 (η0 + δ)w1 (η0 + δ, X1i )
Combining this with (14) and (10), yields
2
λ1 (η0 + δ) = w 11 (η0 + δ)w−1
12 (η0 + δ) + O(∆δ ) a.s.
(16)
Substituting (16) into (15) and using (13), for η0 ≤ η 0 ≤ η0 + δ,
³
´
1
2 −1 2
+ O(∆3δ ) a.s.
H1 (θ0 , η0 + δ) w−1
w
(η
)
+
δf
(η
)
+
O(δ
h
(η
+
δ)
)
11
0
1
0
0
1
2 12
´
´2
³
³
1
−1/2
)
=
+ O(∆3δ ) a.s., (17)
o(hr1 ) + O n1 (ln n1 )1/2 + δf1 (η0 ) + O(δ 2 h−1
1
2t0 (1 − t0 )
where we have used the following facts: w 011 (η0 ) → f1 (η0 ) a.s. and for η0 ≤ η ∗ ≤ η 0 ≤ η0 + δ,
|w011 (η 0 ) − w011 (η0 )| = |w0011 (η ∗ )(η 0 − η0 )|
¯
¶¯
µ
2δ ¯¯ 0 η ∗ − X1i ¯¯
−1
≤ 2 E ¯K1
¯ ≤ cδh1
h1
h1
a.s.
Letting δ = 0 in (17),
H1 (θ0 , η0 ) =
´´2
³
³
³ ´
1
−1/2
o(hr1 ) + O n1 (ln n1 )1/2
+ O ∆30
2t0 (1 − t0 )
a.s.
For δ = O(hr1 ), then under condition (C4), we have that H1 (θ0 , η0 + δ) ≥ H1 (θ0 , η0 ) a.s.
Similarly, we can show that H1 (θ0 , η0 − δ) ≥ H1 (θ0 , η0 ) a.s. and that H2 (θ0 , η0 ± δ) ≥
H2 (θ0 , η0 ) a.s. Therefore, also H (θ0 , η0 ± δ) ≥ H (θ0 , η0 ) a.s. Since H (θ0 , η) is continuous in {η : |η − η0 | ≤ δ}, thus with probability one H (θ0 , η) achieves its minimum in the
interior of the region {η : |η − η0 | ≤ δ}, say at ηe, such that ηe is a root of ∂H (θ0 , ηe) /∂η = 0,
or equivalently ηe is a root of (4).
A.3 Proof of Lemma 2.
e = λ (ηe), and the components of the score vector
For j = 1, 2, denote λj = λj (η), λ
j
j
nj
wj (η, Xji )
1 X
Qjn (η, λ1 , λ2 ) =
,
N i=1 1 + λj wj (η, Xji )
³
nj
2
X
wj0 (η, Xji )
1 X
.
Q3n (η, λ1 , λ2 ) =
λj
N j=1 i=1 1 + λj wj (η, Xji )
´
e ,λ
e = 0, for i=1,2,3. By a Taylor expansion and Lemma 6,
From Lemma 1, Qin ηe, λ
1
2
0 = Qin (η0 , 0, 0) +
³
∂Qin
∂Qin
e + ∂Qin (η , 0, 0)λ
e
(η0 , 0, 0) (ηe − η0 ) +
(η0 , 0, 0)λ
1
0
2
∂η
∂λ1
∂λ2
+ o(hr ) + Op (δ + n−1/2 )
´2
.
³
Note that δ = O(hr ). Then, from the assumption (C4), we have o(hr ) + Op (δ + n−1/2 )
op (n−1/2 ). It is easy to show that, for j = 1, 2,
∂Q3n
∂Qjn
(η0 , 0, 0) =
(η0 , 0, 0) → γj fj (η0 ) a.s.,
∂η
∂λj
17
´2
=
∂Q1n
∂Q2n
(η0 , 0, 0) → −γ1 θ0 (1 − θ0 ) a.s.,
(η0 , 0, 0) = −γ2 t0 (1 − t0 ) a.s.
∂λ1
∂λ2
All other partial derivatives of components of the score vector with respect to the components
of the parameter vector, evaluated at (η0 , 0, 0) are zero. Therefore,




e
Q1n (η0 , 0, 0)
λ
1


−1/2
−1 
e
Q2n (η0 , 0, 0) 
),
=
−S
 + op (n


 λ
2
0
ηe − η0
where the limiting hessian matrix


−γ1 θ0 (1 − θ0 )
0
γ1 f1 (η0 )
0
−γ
t
(1
−
t
)
γ2 f2 (η0 ) 
S=

.
2 0
0
γ1 f1 (η0 )
γ2 f2 (η0 )
0
The inverse of S is obtained as
S −1


−γ22 f22 (η0 )
γ1 γ2 f1 (η0 )f2 (η0 )
γ1 γ2 t0 (1 − t0 )f1 (η0 )
1 
2 2
γ1 γ2 θ0 (1 − θ0 )f2 (η0 ) 
−γ1 f1 (η0 )
=  γ1 γ2 f1 (η0 )f2 (η0 )

c0
γ1 γ2 t0 (1 − t0 )f1 (η0 ) γ1 γ2 θ0 (1 − θ0 )f2 (η0 ) γ1 γ2 t0 (1 − t0 )θ0 (1 − θ0 )
where c0 = det(S) = γ1 γ2 {γ1 f12 (η0 )t0 (1 − t0 ) + γ2 f22 (η0 )θ0 (1 − θ0 )}. Hence,
−1/2
ηe − η0 = −c−1
)
0 γ1 γ2 [t0 (1 − t0 )f1 (η0 )Q1n (η0 , 0, 0) + θ0 (1 − θ0 )f2 (η0 )Q2n (η0 , 0, 0)] + op (n
−1/2
−1
e
λ
)
1 = c0 γ2 f2 (η0 ) [γ2 f2 (η0 )Q1n (η0 , 0, 0) − γ1 f1 (η0 )Q2n (η0 , 0, 0)] + op (n
−1
−1/2
e
λ
).
2 = −c0 γ1 f1 (η0 ) [γ2 f2 (η0 )Q1n (η0 , 0, 0) − γ1 f1 (η0 )Q2n (η0 , 0, 0)] + op (n
Therefore, Lemma 2 follows directly from the above and the fact that
√
N
Ã
Q1n (η0 , 0, 0)
Q2n (η0 , 0, 0)
!
→D N
(Ã
0
0
! Ã
,
γ1 θ0 (1 − θ0 )
0
0
γ2 t0 (1 − t0 )
A.4 Proof of Theorem 2.
From Lemma 2 and a Taylor expansion, we have
n1 2
λ (ηe)w12 (ηe)
2 1
n2
−n2 λ2 (ηe)w21 (ηe) + λ22 (ηe)w22 (ηe) + op (1).
2
l(θ0 ) = −n1 λ1 (ηe)w11 (ηe) +
Choosing η = ηe in (11) and using Lemma 2, we get
w11 (ηe) = λ1 (ηe)w12 (ηe) − λ21 (ηe)
n1
w13 (ηe, X1i )
1 X
n1 i=1 1 + λ1 (ηe)w1 (ηe, X1i )
= λ1 (ηe)w12 (ηe) + op (n−1/2 ).
18
!)
.
Similarly, w 21 (ηe) = λ2 (ηe)w22 (ηe) + op (n−1/2 ).
Using w12 (ηe) = t0 (1 − t0 ) + op (1), w22 (ηe) = θ0 (1 − θ0 ) + op (1), and Lemma 2, we have
l(θ0 )
=
=
=
=
=
n1 λ21 (ηe)w12 (ηe) + n2 λ22 (ηe)w22 (ηe) + op (1)
n1 λ21 (ηe)t0 (1 − t0 ) + n2 λ22 (ηe)θ0 (1 − θ0 ) + op (1)
N λ21 (ηe)γ1 t0 (1 − t0 ) + N λ22 (ηe)γ2 θ0 (1 − θ0 ) + op (1)

Ã
γ1 f1 (η0 )
N λ21 (ηe) γ1 t0 (1 − t0 ) +
γ2 f2 (η0 )
!
Ã
√
2
√
c0 γ1
N λ1 (ηe) √
+ op (1)
γ 2 f2 (η0 )
!2

γ2 θ0 (1 − θ0 )
+
(1)
op
→D χ21 .
ACKNOWLEDGEMENTS
The research of G. Claeskens was partially funded by the National Cancer Institute (CA–
74552). The authors thank the reviewers of the manuscript for their helpful comments.
REFERENCES
S. Chakraborti & R. Mukerjee (1989). A confidence interval for a measure associated with the
comparison of a treatment with a control. South African Statistical Journal, 23, 219–230.
S. Chen & P. Hall (1993). Smoothed empirical likelihood confidence intervals for quantiles. The
Annals of Statistics, 21, 1166 – 1181.
T. J. DiCiccio, P. Hall & J. P. Romano (1991). Empirical likelihood is Bartlett-correctable. The
Annals of Statistics, 19, 1053–1061.
J. H. J. Einmahl & I. W. McKeague (1999). Confidence tubes for multiple quantile plots via
empirical likelihood. The Annals of Statistics, 27, 1348–1367.
M. J. Goddard & I. Hinberg (1990). Receiver operating characteristics (ROC) curves and non
normal data: an empirical study. Statistics in Medicine, 9, 325–337.
P. Hall (1979). On the rate of convergence of normal extremes. Journal of Applied Probability, 16,
433–439.
P. Hall & B. La Scala (1990). Methodology and algorithms of empirical likelihood. International
Statistical Review, 58, 109–127.
P. Hall & A. B. Owen (1993). Empirical likelihood confidence bands in density estimation. Journal
of Computational and Graphical Statistics, 2, 273–289.
J. A. Hanley & B. J. McNeil (1982). The meaning and use of the area under a receiver operating
characteristic (ROC) curve. Radiology, 143, 29–36.
M. Hollander, I. W. McKeague & J. Yang (1997). Likelihood-ratio based confidence bands for
survival function. Journal of the American Statistical Association, 92, 215–226.
19
E. C. Holmgren (1995). The P − P plot as a method of comparing treatment effects. Journal of
the American Statistical Association, 90, 360–365.
F. Hsieh & B. W. Turnbull (1996). Non-parametric and semi-parametric estimation of the receiver
operating characteristic curve. The Annals of Statistics, 24, 25–40.
J. T. Kim (2000). An order selection criterion for testing goodness of fit. Journal of the American
Statistical Association, 95, 829–835.
G. Li (1995). On nonparametric likelihood ratio estimation of survival probabilities for censored
data. Statistics and Probability Letters, 25, 95–104.
G. Li, M. Hollander, I. W. McKeague & J. Yang (1996). Nonparametric likelihood ratio confidence
bands for quantile functions from incomplete survival data. The Annals of Statistics, 24,
628–640.
G. Li, R. C. Tiwari & M. T. Wells (1996). Quantile comparison functions in two-sample problems:
with applications to comparisons of diagnostic markers. Journal of the American Statistical
Association, 91, 689–698.
G. Li, R. C. Tiwari & M. T. Wells (1999). Semiparametric inference for a quantile comparison
function with applications to receiver operating characteristic curves. Biometrika, 86, 487–502.
C. J. Lloyd (1998). The use of smoothed ROC curves to summarize and compare diagnostic
systems. Journal of the American Statistical Association, 93, 1356–1364.
C. J. Lloyd & Z. Yong (1999). Kernel estimators for the ROC curve are better than empirical.
Statistics and Probability Letters, 44, 221–228.
C. E. Metz, B. A. Herman & J. -H. Shen (1998). Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data Statistics in Medicine,
17, 1033–1053.
S. A. Murphy (1995). Likelihood ratio-based confidence intervals in survival analysis. Journal of
the American Statistical Association, 90, 1399–1405.
A. B. Owen (1988). Empirical likelihood ratio confidence intervals for a single functional. Biometrika,
75, 237–249.
A. B. Owen (1990). Empirical likelihood ratio confidence regions. The Annals of Statistics, 18,
90–120.
A. B. Owen (2001). Empirical likelihood. Boca Raton: Chapman & Hall/CRC
E. Parzen (1979) Nonparametric statistical data modeling. Journal of the American Statistical
Association, 74, 105–131.
E. Parzen (1998). Statistical methods mining, two sample data analysis, comparison distributions
and quantile limit theorems. Asymptotic Methods in Probability and Statistics, B. Szyszkowicz
(Ed.), 611–617.
M. S. Pepe (1997). A regression modelling framework for receiver operating characteristic curves
in medical diagnostic testing. Biometrika, 84, 595–608.
20
J. Qin & J. Lawless (1994). Empirical likelihood and general estimating equations. The Annals of
Statistics, 22, 300–325.
R. J. Serfling (1980). Approximation theorems of mathematical statistics. New York: Wiley.
I. A. Swets & R. M. Pickett (1982). Evaluation of Diagnostic Systems: Methods from Signal
Detection Theory. New York: Academic Press.
A. N. Tosteston & C. B. Begg (1988). A general regression methodology for ROC curve estimation.
Medical Decision Making, 8, 205–215.
S. Wieand, M. H. Gail, B. R. James & K. L. James (1989). A family of nonparametric statistics
for comparing diagnostic markers with paired and unpaired data. Biometrika, 76, 585–592.
K. H. Zou, W. H. Hall & D. E. Shapiro (1997). Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Statistics in Medicine, 16, 2143–2156.
21