Here we are detailing our concerns on the validity of the bootstrap method
proposed in BWC. Let Aijk denote the psudovalue for the ROC curve area of the
modality i, reader j, and case sample k, and let Yijkt denote the value of the
decision variable for the modality i, reader j, case k, and the true disease
state t. Then, we can write Aijk as a function of Yij11 , , Yijn1 , Yij12 , , Yijm 2 ,
Aijk  hi (Yij11 ,
, Yijm 2 ) .
, Yijn1 , Yij12 ,
Because Yij11 ,
, Yijn1 , Yij12 ,
, Yijm 2 are not independent, to derive valid bootstrap
estimates we have to modify the standard bootstrap method in the i.i.d case to
account for the correlation structure of the data. To understand the issues
associated with the BWC’s bootstrap method, we first need to understand what is
the definition of a bootstrap estimator for the variance of Aijk when
Yij11 ,
, Yijm 2 are correlated. (For a more detailed description on this,
, Yijn1 , Yij12 ,
see Shao and Tu (1995)).
Let Fij ( y11,
, yn1 , y12 , ym 2 ) be the joint distribution of Yij11 ,
estimator for Fij ( y11,
,Yijn1 ,Yij12 ,
, yn1 , y12 , ym 2 ) using the data Yij11 ,
, Yijm 2 , and F n ,m be an
, Yijn1 , Yij12 ,
, Yijm 2 .
Then, the bootstrap estimator for the variance of Aijk is given by the following
formula:
VarBOOT  Var*[hi (Yij*11 ,
where
{Yij*11 ,
, Yijn* 1, Yij*12 ,
, Yijn* 1 , Yij*12 ,
variance given Yij11 ,
*
, Yijm
2 )],
(3)
*
, Yijm
2 } is a sample from F n , m , and Var* is the conditional
, Yijm 2 . Since we cannot directly calculate the
, Yijn1 , Yij12 ,
bootstrap variance estimator given in (3), we need to use a Monte Carlo method
for approximating VarBOOT . If we can generate B independent samples
{Yij*11b ,
, Yijn*b1, Yij*12b ,
Am*b,n  hi (Yij*11b ,
*b
, Yijm
2 } ,b=1,…,B, from the estimated F n , m , we calculate
, Yijn*b1 , Yij*12b ,
*b
, Yijm
2 )], and approximate VarBOOT by
B
B
b 1
b 1
B
VarBOOT
 (1/ B) ( An*,bm  (1/ B) An*,bm ) 2 .
From this definition for the bootstrap variance estimator, we see that
the validity of the bootstrap variance estimator depends on two closely related
assumptions: (1) we can have a consistent estimator F n , m for the joint
distribution Fij ( y11,
, yn1 , y12 ,
, ym 2 ) , and (2) we can effectively sample from the
estimated joint distribution F n , m .
In the case where Yij11 ,
Fij ( y11,
, yn1 , y12 ,
, Yijn1 , Yij12 ,
, ym 2 )  Fij ( y11 )
, Yijm 2 are i.i.d., we can write
Fij ( ym 2 ) and estimate the univariate distribution
function Fij(.) by its empirical distribution. We can also easily generate a
bootstrap sample
{Yij*11 ,
replacement from Yij11 ,
, Yijn* 1 , Yij*12 ,
, Yijn1 , Yij12 ,
*
, Yijm
2 } by a simple random sampling with
, Yijm 2 . However, when Yij11 ,
, Yijn1 , Yij12 ,
, Yijm 2 are
correlated, how to deal with these two issues is not obvious due to the
multivariate nature of Fij ( y11, , yn1 , y12 , , ym 2 ) . The method taken by BWC for
estimating this multivariate distribution and generating a bootstrap sample from
it was to treat reader and cases as either random or fixed. However, it is not
clear to us what is the property of the resulting bootstrap procedure.
In addition, the proposed re-sampling method seems to treat diseased and nondiseased patients in the same way. However, as we know the distribution of
diseased patients is different from that of a non-diseased patient, they should
be treated differently.
Finally, we must point out that, despite our concerns, the BWC methods seem to
have undergone extensive development, testing and use without serious problems
arising. Thus the method appears to be robust enough that the concerns raised
in our book and in this letter do not seem to constitute "fatal flaws".
References:
1.
S. V. Beiden, R. F. Wagner, G. Campbell. Components-of-variance models and multiplebootstrap experiments. Acad Radiol 2000; 7: 341-349.
2.
J. Shao and D. Tu (1995). The Jackknife and Bootstrap. Springer, New
York.