ROBUST BIAS ESTIMATION FOR KAPLAN–MEIER
SURVIVAL ESTIMATOR WITH JACKKNIFING
By Md Hasinur Rahaman Khan
ISRT, University of Dhaka
Email: hasinur@isrt.ac.bd
and
By J. Ewart H. Shaw
Department of Statistics, University of Warwick
Email: Ewart.Shaw@warwick.ac.uk
For studying or reducing the bias of functionals of the Kaplan–
Meier survival estimator, the jackknifing approach of Stute & Wang
(1994) is natural. We have studied the behavior of the jackknife estimate of bias under different configurations of the censoring level,
sample size, and the censoring and survival time distributions. The
empirical research reveals some new findings about robust calculation of the bias, particularly for higher censoring levels. We have extended their jackknifing approach to cover the case where the largest
observation is censored, using the imputation methods for the largest
observations proposed in Khan & Shaw (2013). This modification to
the existing formula reduces the number of conditions for creating
jackknife bias estimates to one from the original two, and also avoids
the problem that the Kaplan–Meier estimator can be badly underestimated by the existing jackknife formula.
1. Introduction. Suppose that there is a random sample of n individuals. Let Ti and Ci be the random variables that represent the lifetime and
censoring time for the ith individual. We also assume Ti has unknown distribution function F . The Kaplan-Meier (K–M) estimator, F̂ KM (Kaplan &
Meier, 1958) is then defined by
(1.1)
1 − F̂ KM (t) =
∏ (
y(i) ≤y
n − i )δ(i)
,
n−i+1
where Y(1) ≤ · · · ≤ Y(n) are the ordered observations (censored and uncensored lifetimes), δ(i) = 1 if Y(i) is observed and δ(i) = 0 if Y(i) is censored,
ties between censoring times are treated as if the former precede the latter,
and other ties are ordered arbitrarily. Suppose that S is a given statistical
Keywords and phrases: Bias, Censoring, Jackknifing, Kaplan–Meier (K–M)
1
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
2
KHAN, MHR AND SHAW, JEH
function so that S(F ) is the parameter of interest. It follows from Stute
(1994) that if S is nonlinear then the K–M based estimator, S(F KM ), is
biased. Stute (1994) also discussed the situation where the bias arises even
for linear S when the data of interest are partially observable. Now for any
F -integrable function φ, the corresponding estimator
of the parameter of
∫
KM
interest, S(F̂
) is defined by the K–M integral φ(Y(i) ) dF̂ KM .
The K–M estimator is well known to be unbiased if there is no random censorship but it becomes biased under censorship. Gill (1980) was
the first to bound the bias of F̂ KM : −F H ≤ E(F̂ KM ) − F ≤ 0, where
H is the distribution function of Y . Mauro (1985) extended this result to
arbitrary K–M integrals with non-negative integrands. Zhou (1988) proved
that the bias of the K–M estimator functional decreases at an exponential
rate, and ∫always underestimates
the true value. He established the lower
∫
bound: − φ H F (dt) ≤ bias(
φ dF̂ KM ) ≤ 0. Stute (1994) derived the ex∫
act formula for the bias of φ dF̂ KM for a general Borel-measurable function, φ. He also
discussed the effect of light, medium or heavy censoring
∫
on the bias of φ dF̂ KM . Stute & Wang∫(1994) derived an explicit formula
for the jackknife estimate of the bias of φ(Y(i) ) dF̂ KM . They also showed
that jackknifing can lead to a considerable reduction of the bias. Four years
later, Shen
(1998) proposed another explicit formula for jackknife estimate of
∫
∗ ) dF̂ KM . He used delete-2 jackknifing where two observations
bias of φ(T(i)
are deleted. It follows from Shen (1998) that the formula based on delete-2
doesn’t show any further improvement on the delete-1∫formula. Stute (1996)
also proposed a jackknife estimate of the variance of φ(Y(i) ) dF̂ KM .
As mentioned in Stute & Wang (1994), under random censorship the
estimator S(F̂ KM ) becomes the K–M integral
(1.2)
S(F̂
KM
)=
n
∑
wi φ(Y(i) ) ≡ ŜφKM ,
i = 1, · · · , n
i=1
where the the K–M weights wi are the sizes of the jumps by which the K–M
estimator of F changes at the uncensored points Y(i) , given by
(1.3)
w1 =
δ(1)
,
n
wi =
i−1
∏ ( n − j )δ(j)
δ(i)
,
n − i + 1 j=1 n − j + 1
i = 2, · · · , n.
A detailed study of the wi ’s in connection with the strong law of large
numbers under censoring has been carried out in Stute & Wang (1993).
The jackknife estimate of bias for the K–M integral (Equation 1.2) is given
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING
3
by
(1.4)
Bias (ŜφKM ) = −
n−2
∏ ( n − 1 − j )δ(j)
n−1
φ(Y(n) ) δ(n) (1 − δ(n−1) )
.
n
n−j
j=1
The associated bias corrected jackknife estimator is therefore given by
(1.5)
S̃φKM = ŜφKM − Bias (ŜφKM ).
2. Modified Jackknife Bias for K–M Lifetime Estimator. When
no censoring is present, F̂ KM reduces to the usual sample distribution estimator F̂ that assign weight n1 to each observation. With censoring, the
+
weighting method (1.3) gives zero weight to the censored observations Y(.)
,
causing particular problems if the largest datum is censored (i.e. δ(n) = 0).
As a first step one may apply Efron’s (1967) tail correction approach: reclassify δ(n) = 0 as δ(n) = 1. In order to reduce estimation bias and inefficiency,
Khan & Shaw (2013) proposed five alternatives to Efron’s approach, that
can lead to more efficient and less biased estimates. The approaches are
summarised in Table 1. The first four approaches are based on the underTable 1
The imputation approaches from Khan & Shaw (2013).
Wτm : Adding the Conditional Mean
Wτmd : Adding the Conditional Median
∗ : Adding the Resampling-based Conditional Mean
Wτm
∗ : Adding the Resampling-based Conditional Median
Wτmd
Wν : Adding the Predicted Difference Quantity
lying regression assumption relating lifetimes and covariates (e.g., the AFT
model), and the fifth approach Wν , is based on only the random censorship
assumption.
The jackknife bias in Equation (1.4) is non-zero if and only if the largest
datum is uncensored, δ(n) = 1, and the second largest datum is censored,
δ(n−1) = 0. Stute & Wang (1994) state that if δ(n) = 0, then the corresponding observation doesn’t contain enough information about F to make
a change of ŜφKM desirable. This inability to estimate bias if δ(n) = 0 is a
major limitation of the jackknife bias formula.
If (δ(n−1) = 0, δ(n) = 0), then we can obtain a modified jackknife estimate of bias by imputing the largest datum, for example using any of the
approaches given in Table 1. From Equation (1.2) this gives the modified
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
4
KHAN, MHR AND SHAW, JEH
estimator
KM
(2.1) Sˆφ∗
≡
n−1
∑
wi φ(Y(i) ) + ẃn φ(Ỹ(n) ),
i = 1, · · · , n − 1,
i=1
where Ỹ(n) is the imputed largest observation, and ẃn is the corresponding
adjusted K–M weight
∏ ( n − 1 − j )δ(j)
n − 1 n−2
ẃn = wn +
n j=1
n−j
as suggested in Stute & Wang (1994) for the pair (δ(n−1) = 0, δ(n) = 1).
The modified estimator (2.1) is also obtained when imputing in the situation
(δ(n−1) = 1, δ(n) = 0). In this case the K–M weight to Ỹ(n) is not adjusted
and we arrive at the estimator
KM
Sˆφ∗
≡
n−1
∑
wi φ(Y(i) ) + wn φ(Ỹ(n) ),
i = 1, · · · , n − 1.
i=1
So unlike the actual jackknife formula the modified approach doesn’t impose any condition on the censoring status of Y(i) . The modified estimate of
bias is given by
n−2
∏ ( n − 1 − j )δ(j)
n−1
KM
∗
φ(Ỹ(n) ) δ(n)
(1 − δ(n−1) )
,
(2.2) Bias (Sˆφ∗
)=−
n
n−j
j=1
∗ is the modified censoring indicator for Ỹ
where δ(n)
(n) . With the above ap∗
proach, δ(n) is always 1. It follows from Equation (2.2) the larger bias quan-
tity because Ỹ(n) > Y(n) . The modified bias corrected jackknife estimator is
then defined by
(2.3)
KM
KM
KM
S˜φ∗
= Sˆφ∗
− Bias (Sˆφ∗
).
The K–M estimates under both approaches for the four pairs are summarized
in Table 2.
We investigate below the effect of censoring on the K–M estimator S(F̂ KM )
based on both the actual and the modified jackknife bias formula. For computational simplicity we look only at the K–M mean lifetime estimator,
obtained by replacing φ(y) by y in Equation (1.2). Note that researchers
in reliability are very often interested in estimating the mean lifetime of a
component, and that the K–M mean lifetime estimate also has an important
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING
5
Table 2
K–M lifetime estimates by censoring indicators for the last two observations.
K–M estimate
KM
Sˆφ∗
+
n−1
n
∗
φ(Ỹ(n) ) δ(n)
(1 − δ(n−1) )
KM
Sˆφ∗
ŜφKM
ŜφKM +
n−1
n
φ(Ỹ(n) ) δ(n) (1 − δ(n−1) )
∏n−2 ( n−1−j )δ(j)
j=1
n−j
∏n−2 ( n−1−j )δ(j)
j=1
n−j
δ(n−1)
δ(n)
0
0
1
1
0
1
0
1
role in Health Economics, for example, in a “QTWIST” analysis (Glasziou
et al. 1990). Obviously the behaviour of the K–M mean lifetime estimator
depends on the nature of the distribution being estimated and the degree
of censoring, although the true distribution of censored data is generally
unknown. We therefore conducted simulation studies to demonstrate the
behavior of the K–M mean lifetime estimator in the presence of right censoring. We assume that the lifetimes and censoring times have independent
distributions.
Note that the mean survival time can be defined as the area under the
survival curve, S(t) (Kaplan & Meier, 1958). A nonparametric estimate of
the mean survival time can also be obtained by substituting
the K–M mean
∫
estimator for the unknown survival function µ̂ = 0∞ Ŝ(t) dt. Stute (1994)
proposed a bias corrected jackknife estimator for the K–M mean lifetime.
Various estimation procedures for mean lifetime are discussed by Young
(1977), Susarla and Van Ryzin (1980), Gill (1983), Kumazawa (1987), and
elsewhere. When the observations are subject to right censoring, the usual
mean estimator of the mean lifetime is not appropriate (Datta, 2005). The
reason is that the censoring leads to an inconsistent estimator that underestimates the true mean and the bias worsens as the censoring increases.
3. Simulation Study. This section reports on three simulation based
examples. The first example extends the Koziol-Green model simulations of
Stute & Wang (1994). The second example considers various skewed distributions for survival times and corresponding distributions for the associated
censored times. The third example uses a log-normal AFT model where the
event times are assumed to be associated with several covariates.
3.1. Koziol-Green Model based Example. This extends the simulations
of the Koziol-Green proportional hazards model from Stute & Wang (1994).
Under this model both T and C were exponentially distributed: T ∼ Exp (1)
and C ∼ Exp (λ), with varying λ’s. Four different sample sizes n = 30, 50,
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
6
KHAN, MHR AND SHAW, JEH
100, 150 are used. For each sample, 100, 000 simulation runs are drawn
KM
and the bias and variance of both the mean lifetime estimators Ŝmean
and
KM
S̃mean are computed. The bias and its variance are shown in Table 3 and
7 (the first sub-table for both tables) respectively, are displayed in Figure 1
(1st and 2nd graph of each panel).
Table 3
KM
KM
Simulation results for bias of the four K−M mean lifetime estimators Ŝmean
, S̃mean
,
KM
KM
Sˆ∗ mean and S˜∗ mean based on the Koziol−Green model.
P%
n=30
n=50 n=100
KM
Bias of Ŝmean
n=150
n=30
n=50 n=100
KM
Bias of S̃mean
n=150
10
20
30
40
50
60
70
80
90
-0.155
-0.197
-0.250
-0.304
-0.364
-0.409
-0.430
-0.402
-0.280
-0.114
-0.157
-0.205
-0.265
-0.327
-0.389
-0.426
-0.417
-0.304
Bias of
-0.073
-0.107
-0.151
-0.209
-0.278
-0.349
-0.413
-0.428
-0.335
KM
Sˆ∗ mean
-0.055
-0.085
-0.126
-0.178
-0.248
-0.328
-0.396
-0.428
-0.346
-0.154
-0.191
-0.233
-0.267
-0.295
-0.287
-0.224
-0.082
0.161
-0.114
-0.155
-0.195
-0.239
-0.268
-0.281
-0.234
-0.097
0.178
Bias of
-0.073
-0.107
-0.146
-0.193
-0.237
-0.263
-0.246
-0.127
0.171
KM
S˜∗ mean
-0.056
-0.086
-0.123
-0.164
-0.215
-0.255
-0.245
-0.141
0.164
10
20
30
40
50
60
70
80
90
-0.208
-0.259
-0.326
-0.391
-0.465
-0.511
-0.518
-0.463
-0.304
-0.147
-0.202
-0.261
-0.335
-0.407
-0.481
-0.512
-0.481
-0.331
-0.090
-0.132
-0.186
-0.260
-0.343
-0.426
-0.495
-0.496
-0.367
-0.067
-0.104
-0.155
-0.218
-0.304
-0.400
-0.475
-0.498
-0.380
-0.207
-0.252
-0.309
-0.354
-0.396
-0.389
-0.312
-0.162
0.151
-0.147
-0.200
-0.251
-0.310
-0.349
-0.372
-0.320
-0.162
0.151
-0.090
-0.132
-0.181
-0.243
-0.303
-0.341
-0.328
-0.195
0.139
-0.068
-0.104
-0.152
-0.205
-0.271
-0.327
-0.325
-0.210
0.129
The results show that, for both estimators, the bias increases as censoring
increases until a particular censoring level, then declines. That particular
censoring level falls in the range 60 to 80. Above that censoring level the
bias decreases as censoring increases, and decreases much more rapidly for
the corrected estimator than for the K–M estimator. In addition, the bias
for the corrected estimator at P% = 90 censoring is positive for all sample
sizes. This behaviour at high censoring levels does not appear in Stute &
Wang (1994) who investigated the bias up to only P% = 66.7, but it is easily
seen from Table 2 that if censoring is 100%, then δ(n) = 0, so the bias is 0. A
similar trend is observed for the variance of the bias of the two estimators.
We have computed also the bias of the jackknife estimate and its variance
KM
KM
based on both the modified estimators Sˆ∗ mean and S˜∗ mean . The modifi-
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING
20 40 60 80
20 40 60 80
0.1
0.0
Bias
0
Censoring percentage
N 30
N 50
N 100
N 150
−0.2
Bias
0
Censoring percentage
Modified corr. estimator
N 30
N 50
N 100
N 150
−0.4
0.1
0.0
Bias
0
Modified estimator
N 30
N 50
N 100
N 150
−0.3 −0.2 −0.1
−0.1
−0.2
−0.4
−0.3
Bias
Corrected estimator
N 30
N 50
N 100
N 150
−0.5 −0.4 −0.3 −0.2 −0.1
Estimator
7
20 40 60 80
0
20 40 60 80
Censoring percentage
Censoring percentage
Modified estimator
Modified corr. estimator
20 40 60 80
Censoring percentage
0
20 40 60 80
Censoring percentage
0.15
0.10
Variance of bias
0.08
N 30
N 50
N 100
N 150
0.00
0.05
0.12
N 30
N 50
N 100
N 150
0.04
Variance of bias
N 30
N 50
N 100
N 150
0.00
Variance of bias
0.03
0.02
0.01
0
0.00 0.02 0.04 0.06 0.08 0.10
Corrected estimator
N 30
N 50
N 100
N 150
0.00
Variance of bias
0.04
Estimator
0.20
(a) Bias
0
20 40 60 80
Censoring percentage
0
20 40 60 80
Censoring percentage
(b) Variance of bias
KM
KM
Fig 1. Bias and its variance for four K−M mean lifetime estimators Ŝmean
, S̃mean
,
KM
KM
∗
∗
ˆ
˜
S mean and S mean under Koziol−Green model at different censoring points. Lowess
smooths are superimposed.
cation is based on the predicted difference quantity approach where Ỹ(n) is
replaced by Y(n) + ν (Wν in Table 1), as discussed in Khan & Shaw (2013).
The bias and its variance are shown in Table 3 and 7 respectively (the second sub-table for both tables) and are displayed in Figure 1 (the third and
fourth graph of each panel). The results demonstrate that under the modified approach, slightly larger bias and variance estimates are obtained. Their
overall trends are similar to those of the original estimators.
3.2. Second Simulation Study. In the second simulation, survival times
are generated from four skewed distributions as shown in Figure 2, and
censoring times independently from other specified distributions, as listed in
Table 4. Datasets are generated randomly subject to the restriction δ(n−1) =
0, and, for the original jackknife formula, with the additional restriction
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
8
KHAN, MHR AND SHAW, JEH
δ(n) = 1.
Table 4
The failure time distributions with their corresponding censoring distributions.
Survival time distributions
Censoring time distributions
Uniform: U (a, 2a)
Exponential: Exp (λ)
Uniform: U (a, 2a)
Uniform: U (a, 2a)
0.4
2
Log-normal (1.1, 1): √12π exp (−(logtt−1.1) /2)
Exponential (0.2): 15 exp (− 5t )
1
Gamma (4, 1): Γ(4)
t3 exp (−t)
3
t3
Weibull (3.39, 3): 38.96
t2 exp (− 38.96
)
0.2
0.0
0.1
Probability density
0.3
Log−normal
Exponential
Gamma
Weibull
0
5
10
15
Failure time
Fig 2. The four failure time distributions used in the simulation study
In the case when T ∼ Exp (0.2) and C ∼ Exp (λ) for a chosen level
of censoring percentage P% , it follows that Y and δ are independent with
P% /100 = pr (δ = 0) = λ/(0.2 + λ) (see Appendix for details for the associated a values). For censoring time the Uniform distribution over the range
[a, 2a] is chosen because it makes easy to derive suitable values for a for a
specified value of P% (see Appendix).
We use four samples n = 30, 50, 100, 150. The jackknife estimate of bias
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING
9
and its variance for all four estimators from 10, 000 simulated datasets are
shown in Figures 3 and 5 respectively. Because of space constraints, Table 5
KM
KM
shows the bias for only two estimators, Ŝmean
and Sˆ∗ mean for three samples n = 50, 100, 150. The associated modification is here carried out using
method Wν of Table 1), described fully in (Khan & Shaw, 2013).
Figures 3(a), 3(d) and 5(a), 5(d) reveal similar results to our large simulation based Koziol–Green model example. For example, given the modification, the bias estimate is bound to be higher. This seems to be true also for
the variance estimate. In addition, we find that for both actual and modified
estimators the trend in bias differs for different censoring levels, but they
behave similarly under different lifetime distributions (see Figure 3). The
relationship between bias and censoring level varies substantially between
the distributions and the sample sizes. For a log-normal distribution, the
bias for the estimators except for the corrected estimators tends to increase
as P% increases until 50. The maximum bias for the other distributions investigated occurs between 60% and 80% censoring. Under the Exponential
lifetime distribution the bias behaves very similarly to that of the Koziol–
Green proportional hazards model. Given that the estimators are original
or modified the corrected estimators seem to be overestimated in the higher
censoring points (i.e., the bias becomes positive in higher censoring).
The variance (Figure 5) of bias for estimators also differs according to
sample sizes and censoring level. The variance generally reaches a maximum
at some censoring level between 50% and 70%, then declines. However, for
the corrected estimators under a log-normal distribution the variance decreases consistently as censoring increases (see Figure 5(a)).
3.3. Third Simulation Study. This simulation study is conducted to investigate how the modified estimators behave relative to the original estimators when lifetimes are modeled as an AFT model that has the form
(3.1) Zi = α + XiT β + σεi , i = 1, · · · , n
εi ∼ N (0, 1) for i = 1, · · · , n
where Zi = log (Ti ), X is the covariate vector, α is the intercept term, β is
the unknown p × 1 vector of true regression coefficients. The logarithm of
the true survival time is generated from the true model (3.1). The logarithm
of censoring time is assumed to be distributed as U(a, 2a) where a is chosen
analytically in the same way as done in the previous example. We consider
five covariates X = (X1 , X2 , X3 , X4 , X5 ) each of which is generated using
U(0, 1), seven P% points, and three samples n = 30, 50 and 100. The coefficients of the covariates are chosen as βj = j + 1 where j = 1, · · · , 5 and
σ = 1. Of the five proposed imputation approaches of Table 1 and Khan
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
KHAN, MHR AND SHAW, JEH
10
n=50
-0.856
-1.161
-1.350
-1.464
-1.506
-1.482
-1.381
-1.209
-0.921
n=100
-0.845
-1.157
-1.362
-1.481
-1.522
-1.491
-1.394
-1.216
-0.924
-0.827
-1.141
-1.348
-1.470
-1.513
-1.484
-1.390
-1.213
-0.923
n=150
-0.745
-1.000
-1.339
-1.707
-2.133
-2.477
-2.710
-2.563
-1.766
-0.578
-0.783
-1.055
-1.356
-1.700
-2.017
-2.267
-2.229
-1.639
n=50
-0.461
-0.668
-0.943
-1.317
-1.754
-2.215
-2.560
-2.561
-1.878
-0.371
-0.540
-0.765
-1.060
-1.414
-1.812
-2.145
-2.221
-1.726
n=100
n=150
n=50
KM
Bias of Ŝmean
-0.280
-0.364
-0.427
-0.561
-0.631
-0.781
-0.912
-0.991
-1.261
-1.194
-1.655
-1.375
-2.039
-1.495
-2.212
-1.534
-1.780
-1.388
KM
Bias of Sˆ∗ mean
-0.340
-0.384
-0.518
-0.580
-0.771
-0.800
-1.120
-1.008
-1.550
-1.209
-2.012
-1.387
-2.450
-1.504
-2.570
-1.539
-1.950
-1.390
-0.282
-0.481
-0.704
-0.936
-1.165
-1.356
-1.489
-1.535
-1.388
n=100
-0.256
-0.453
-0.691
-0.930
-1.158
-1.358
-1.500
-1.538
-1.392
-0.246
-0.442
-0.679
-0.918
-1.147
-1.349
-1.493
-1.534
-1.390
n=150
-0.221
-0.336
-0.484
-0.679
-0.923
-1.183
-1.445
-1.645
-1.653
-0.208
-0.322
-0.470
-0.665
-0.909
-1.171
-1.435
-1.639
-1.650
n=50
-0.142
-0.238
-0.376
-0.578
-0.830
-1.117
-1.403
-1.626
-1.642
-0.135
-0.230
-0.367
-0.569
-0.820
-1.108
-1.395
-1.621
-1.640
n=100
-0.109
-0.194
-0.333
-0.533
-0.793
-1.100
-1.397
-1.617
-1.640
-0.104
-0.188
-0.326
-0.525
-0.784
-1.092
-1.390
-1.613
-1.638
n=150
Weibull
-0.295
-0.495
-0.718
-0.949
-1.177
-1.366
-1.496
-1.540
-1.389
Gamma
KM
KM
Table 5. Simulation results for bias of the K−M mean lifetime estimators Ŝmean
and Sˆ∗ mean under four lifetime distributions. Results
are based on three samples n = 50, 100, 150 and ten censoring levels.
P%
-0.940
-1.204
-1.379
-1.476
-1.510
-1.471
-1.380
-1.205
-0.920
-0.878
-1.175
-1.365
-1.476
-1.516
-1.489
-1.386
-1.211
-0.922
Exponential
10
20
30
40
50
60
70
80
90
-0.969
-1.228
-1.399
-1.492
-1.522
-1.480
-1.386
-1.208
-0.921
Log-normal
10
20
30
40
50
60
70
80
90
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING 11
∗ ) is
& Shaw (2013), the resampling based conditional mean approach (Wτm
∗ from 10, 000 simulation
found to have the least bias, and the results for Wτm
runs are shown in Figure (4).
4. Discussion. The behavior of bias for the K–M lifetime estimators
is influenced by many factors in practice. For example, the nature of the
distributions to be used for lifetimes, the censoring rate, the sample size,
whether the lifetimes are modeled with the covariates and so on. To explore
the behaviour of the jackknife bias for K–M estimators under various conditions (in particular, censoring levels) a large simulation is required. Our
simulation studies go beyond the small simulation study in Stute & Wang
(1994) and show clear differences from many of their results. In particular,
the bias (Equation (1.4) and (2.2)) will be 0 at 0% censoring and increases
as the censoring level increases. However, the bias will also tend to 0 as the
censoring level tends to 100% (because the bias is 0 when either δ(n−1) or
δ(n) is 0). Therefore, as shown in the figures, the bias increases up to a particular censoring level (typically 50% − 80%) but then reduces. The variance
of the bias shows similar behaviour. Note also that the bias for the corrected
estimators tends to be overestimated at the higher censoring level (90%).
We propose the modified K–M survival estimator, the modified jackknife
estimate of bias for K–M estimator and the modified bias corrected K–
M estimator. The modification allows one pair of observations (δ(n) = 0,
δ(n−1) = 0) to contribute to the bias calculation. So our modifications reduce
the original conditions needed for jackknife estimation of bias (δ(n−1) = 0,
δ(n) = 1) to the single condition δ(n−1) = 0. The modified jackknife estimate
also prevents the K–M estimator from being badly underestimated by the
jackknife estimate when the largest observation is censored. For implementing the existing and the modified jackknifing approach, we have provided a
publicly available package jackknifeKME (Khan & Shaw, 2013) implemented
in the R programming system.
Acknowledgements. The first author is grateful to the CRiSM, department of Statistics, University of Warwick, UK for offering research funding for his PhD study.
References.
Datta, S. (2005). Estimating the mean life time using right censored data.
Statistical Methodology 2 65-69.
Efron, B. (1967). The two sample problem with censored data. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and
Probability, 4 831-853. New York: Prentice Hall.
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
12
KHAN, MHR AND SHAW, JEH
Gill, R. (1980). Censoring and Stochastic Integrals. Mathematical Centre
Tracts 124. Amsterdam: Mathematisch Centrum.
Gill, R. (1983). Large Sample Behavior of the Product−Limit Estimator
on the Whole line. The Annals of Statistics 11 49-58.
Glasziou, P., Simes, R. and Gelber, R. (1990). Quality Adjusted Survival Analysis. Statistics in Medicine 9 1259-1276.
Kaplan, E. L. and Meier, P. (1958). Nonparametric estimation from
incomplete observations. J. Amer. Stat. Assoc. 53 457-581.
Khan, M. H. R. and Shaw, J. E. H. (2013a). On Dealing with Censored
Largest Observations under Weighted Least Squares. CRiSM working
paper, No. 13-07, Department of Statistics, University of Warwick.
Khan, M. H. R. and Shaw, J. E. H. (2013b). jackknifeKME: Jackknife
estimates of Kaplan-Meier estimators or integrals. R package version
1.0.
Kumazawa, Y. (1987). A note on an Estimator of Life Expectancy with
Random Censorship. Biometrika 74 655-658.
Mauro, D. (1985). A combinatoric approach to the Kaplan-Meier estimator. Annals of Statistics 13 142-149.
Shen, P. S. (1998). Problems arising from jackknifing the estimate of a
Kaplan-Meier integral. Statistics & Probability Letters 40 353-361.
Stute, W. (1994). The Bias of Kaplan-Meier Integrals. Scandinavian Journal of Statistics 21 475-484.
Stute, W. (1996). The jackknife estimate of variance of A Kaplan-Meier
integral. The Annals of Statistics 24 2679-2704.
Stute, W. and Wang, J. L. (1993). The strong law under random censorship. Annals of Statistics 21 1591-1607.
Stute, W. and Wang, J. (1994). The jackknife estimate of a Kaplan-Meier
integral. Biometrika 81 602-606.
Susarla, V. and Van Ryzin, J. (1980). Large Sample Theory for an Estimator of the Mean Survival Time from Censored Samples. The Annals
of Statistics 8 1002-1016.
Yang, G. (1977). Life Expectancy under Random Censorship. Stochastic
Processes and Their Applications 6 33-39.
Zhou, M. (1988). Two-Sided Bias Bound of the Kaplan-Meier Estimator.
Probability Theory and Related Fields 79 165-173.
APPENDIX A: GENERATING CENSORED DATA
A.1. When Censoring Time has Exponential Distribution. Let
the censoring time C and the survival time T have the distribution function
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING 13
G and F respectively. Then for any censoring percentage P%
P(T > C) =
∫
(A.1)
c
P%
100
F (c) g(c) dc = 1 − q,
P%
d
where g(c) = dc
G(c) and q = 100
. If C ∼ Exp(λ) i.e. g(c) = λ e−λ c ; c ≥ 0
then the equation (A.1) becomes
∫
∞
(A.2)
0
λ e−λ c F (c) dc = 1 − q.
If T ∼ Exp (0.2) i.e. f (t) = 0.2 e−0.2t ; t ≥ 0 so that F (c) = 1 − e−0.2c . Then
the equation (A.2) becomes
∫
∞
λe
−λ c
0
dc −
∫
0
∞
λ e−(0.2+λ) c dc = 1 − q
⇒q=
λ
.
0.2 + λ
A.2. When Censoring Time has Uniform Distribution. If C ∼
U (a, 2a) i.e. g(c) = a1 , a ≤ c ≤ 2a then equation (A.1) follows that
∫
2a 1
(A.3)
a
a
F (c) dc = 1 − q.
d
Now for any given survival time distribution f (t) = dt
F (t) and censoring
percentage P% , a can be determined using equation (A.3). For illustration,
(4, t)
1
if T ∼ Gamma (4, 1) i.e. f (t; 4, 1) = Γ(4)
t3 e−t ; t ≥ 0 so that F (t) = γΓ(4)
where γ (4, t) is the lower incomplete gamma function. Then the equation
(A.3) becomes
∫
2a
a
γ (4, c)
dc = 1 − q,
Γ(4)
which can be solved now for any given q, say if P% = 30 this equation
gives a = 2.75. The derivation of a for the remaining lifetime distributions
log-normal (1.1, 1) and Weibull (3.39, 3) also follows the similar procedure.
The calculated values of a for U(a, 2a) are given in Table 6.
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
14
KHAN, MHR AND SHAW, JEH
Table 6
The parameter estimation of censoring time distributions U (a, 2a) and Exp (λ) obtained
for different lifetime distributions across different censoring points.
Distribution
10
20
30
40
log-normal (1.1, 1)
Gamma (4, 1)
Weibull (3.39, 3)
7.534
4.119
3.288
4.810
3.273
2.802
3.480
2.750
2.482
2.640
2.356
2.227
Exp (0.2)
Exp (1)
0.022
0.111
0.050
0.250
0.086
0.429
0.133
0.667
P%
50
a
2.039
2.027
2.003
λ
0.200
1.000
60
70
80
90
1.575
1.734
1.791
1.195
1.457
1.576
0.865
1.176
1.338
0.552
0.856
1.036
0.300
1.500
0.467
2.333
0.800
4.000
1.800
9.000
Table 7
Simulation results for variance of bias for the four K−M mean lifetime estimators
KM
KM
KM
KM
Ŝmean
, S̃mean
, Sˆ∗ mean and S˜∗ mean based on the Koziol−Green model.
P%
n=30 n=50 n=100 n=150
KM
Variance of bias of Ŝmean
n=30 n=50 n=100 n=150
KM
Variance of bias of S̃mean
10
20
30
40
50
60
70
80
90
0.004 0.002
0.001
0.000
0.008 0.006
0.003
0.002
0.016 0.012
0.006
0.004
0.024 0.019
0.012
0.009
0.034 0.028
0.021
0.016
0.041 0.037
0.029
0.025
0.040 0.038
0.034
0.032
0.030 0.030
0.029
0.029
0.011 0.011
0.013
0.013
KM
∗
ˆ
Variance of bias of S mean
0.010 0.005
0.002
0.001
0.019 0.013
0.006
0.004
0.037 0.027
0.014
0.010
0.056 0.045
0.028
0.021
0.082 0.064
0.049
0.037
0.096 0.088
0.067
0.058
0.092 0.090
0.081
0.074
0.071 0.074
0.069
0.071
0.034 0.032
0.034
0.035
KM
∗
˜
Variance of bias of S mean
10
20
30
40
50
60
70
80
90
0.021
0.031
0.056
0.078
0.116
0.117
0.101
0.063
0.018
0.034
0.053
0.095
0.135
0.201
0.209
0.183
0.121
0.045
0.008
0.019
0.035
0.056
0.077
0.100
0.092
0.064
0.019
0.003
0.007
0.015
0.031
0.054
0.073
0.082
0.061
0.022
0.001
0.004
0.011
0.022
0.039
0.059
0.073
0.063
0.023
0.014
0.032
0.061
0.099
0.136
0.181
0.171
0.128
0.045
0.004
0.012
0.027
0.057
0.098
0.132
0.151
0.118
0.049
0.002
0.008
0.020
0.039
0.070
0.108
0.135
0.123
0.051
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING 15
20 40 60 80
0
20 40 60 80
0.5
−0.5
0
Censoring percentage
N 30
N 50
N 100
N 150
2.5
Bias
Bias
−1.3
−1.5
0.5
−0.5
−1.5
0
Censoring percentage
Modified corr. estimator
N 30
N 50
N 100
N 150
−0.9
2.5
Bias
1.5
−0.9
−1.1
Bias
−1.3
Modified estimator
N 30
N 50
N 100
N 150
1.5
Corrected estimator
N 30
N 50
N 100
N 150
−1.1
Estimator
20 40 60 80
0
Censoring percentage
20 40 60 80
Censoring percentage
(a) For T ∼ LN (1.1, 1) & C ∼ U (a, 2a).
−1
−2.5
20 40 60 80
−2
−1.5
0
0
Censoring percentage
N 30
N 50
N 100
N 150
1
Bias
−1.5
Bias
Bias
−0.5
−1.5
Modified corr. estimator
N 30
N 50
N 100
N 150
0
−0.5
Modified estimator
N 30
N 50
N 100
N 150
0.5
−1.0
−0.5
N 30
N 50
N 100
N 150
−2.0
Bias
Corrected estimator
1.5
Estimator
20 40 60 80
0
Censoring percentage
20 40 60 80
0
Censoring percentage
20 40 60 80
Censoring percentage
(b) For T ∼ EX (0.2) & C ∼ EX (λ).
Modified estimator
0.2
0.0
−0.4
Bias
−0.8
−1.6
0
20 40 60 80
0
Censoring percentage
20 40 60 80
0
Censoring percentage
N 30
N 50
N 100
N 150
−0.8
−0.6
−1.2
Bias
Modified corr. estimator
N 30
N 50
N 100
N 150
−0.4
N 30
N 50
N 100
N 150
0.2
Bias
−0.6
−1.0
−1.4
Bias
Corrected estimator
N 30
N 50
N 100
N 150
−0.2
−0.2
Estimator
20 40 60 80
0
Censoring percentage
20 40 60 80
Censoring percentage
(c) For T ∼ G (4, 1) & C ∼ U (a, 2a).
20 40 60 80
Censoring percentage
20 40 60 80
Censoring percentage
Bias
−0.6
−0.2
N 30
N 50
N 100
N 150
−1.0
−1.5
0
Modified corr. estimator
N 30
N 50
N 100
N 150
−0.5
Bias
−0.2
Bias
−1.0
0
Modified estimator
N 30
N 50
N 100
N 150
−0.6
−0.5
−1.0
−1.5
Bias
Corrected estimator
N 30
N 50
N 100
N 150
−1.0
Estimator
0
20 40 60 80
Censoring percentage
0
20 40 60 80
Censoring percentage
(d) For T ∼ WB (3.39, 3) & C ∼ U (a, 2a).
KM
KM
KM
KM
Fig 3. The bias of the K–M mean lifetime estimators Ŝmean
, S̃mean
, Sˆ∗ mean and S˜∗ mean
in 10000 simulation runs. The value of a and λ for the censoring time distribution are
shown in the appendix.
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
16
KHAN, MHR AND SHAW, JEH
20
40
N 30
N 50
N 100
−8
−6
N 30
N 50
N 100
60
80
20
Censoring percentage
40
60
80
−2
−4
40
−6
Bias
N 30
N 50
N 100
20
Censoring percentage
Modified corr. estimator
−8
−5
−6
Bias
−4
Bias
−3
−4
Bias
−2
−2
Modified estimator
−6 −5 −4 −3 −2 −1
Corrected estimator
−1
Estimator
60
80
N 30
N 50
N 100
20
40
60
80
Censoring percentage
Censoring percentage
Modified estimator
Modified corr. estimator
(a) Bias
40
60
80
Censoring percentage
20
40
60
80
Censoring percentage
2.0
3.0
N 30
N 50
N 100
0.0
1.0
Variance of bias
2.0
1.0
Variance of bias
N 30
N 50
N 100
0.0
Variance of bias
1.5
1.0
0.5
20
N 30
N 50
N 100
0.0 0.5 1.0 1.5 2.0 2.5
2.0
N 30
N 50
N 100
3.0
Corrected estimator
0.0
Variance of bias
2.5
Estimator
20
40
60
80
Censoring percentage
20
40
60
80
Censoring percentage
(b) Variance of bias
Fig 4. Simulation results for the third simulated example for all four K−M mean lifetime
KM
KM
KM
KM
estimators Ŝmean
, S̃mean
, Sˆ∗ mean and S˜∗ mean under the log-normal AFT model at
different censoring points. Lowess smooths are superimposed.
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism
BIAS CALCULATION FOR THE K–M ESTIMATOR WITH JACKKNIFING 17
20 40 60 80
0
20 40 60 80
0.20
0.00
0
Censoring percentage
0.10
Variance of bias
0.08
Variance of bias
N 30
N 50
N 100
N 150
0.00
0.00
0
Censoring percentage
Modified corr. estimator
N 30
N 50
N 100
N 150
0.04
0.20
N 30
N 50
N 100
N 150
0.10
Variance of bias
0.08
0.04
Modified estimator
0.12
Corrected estimator
N 30
N 50
N 100
N 150
0.00
Variance of bias
0.12
Estimator
20 40 60 80
0
Censoring percentage
20 40 60 80
Censoring percentage
(a) For T ∼ LN (1.1, 1) & C ∼ U (a, 2a).
20 40 60 80
6
4
5
N 30
N 50
N 100
N 150
3
Variance of bias
3.0
2.0
1
1.0
Variance of bias
0
Censoring percentage
Modified corr. estimator
N 30
N 50
N 100
N 150
20 40 60 80
0
0.0
Variance of bias
0
Modified estimator
N 30
N 50
N 100
N 150
2
Corrected estimator
0.0 0.5 1.0 1.5 2.0 2.5
Variance of bias
0.0 0.2 0.4 0.6 0.8 1.0 1.2
Estimator
N 30
N 50
N 100
N 150
0
Censoring percentage
20 40 60 80
0
Censoring percentage
20 40 60 80
Censoring percentage
(b) For T ∼ EX (0.2) & C ∼ EX (λ).
Corrected estimator
Modified estimator
Modified corr. estimator
0
20 40 60 80
0
Censoring percentage
20 40 60 80
0
Censoring percentage
0.10
0.06
Variance of bias
0.10
N 30
N 50
N 100
N 150
0.02
N 30
N 50
N 100
N 150
0.02
0.02
N 30
N 50
N 100
N 150
0.06
Variance of bias
0.10
0.06
Variance of bias
0.06
N 30
N 50
N 100
N 150
0.02
Variance of bias
0.10
0.14
Estimator
20 40 60 80
0
Censoring percentage
20 40 60 80
Censoring percentage
(c) For T ∼ G (4, 1) & C ∼ U (a, 2a).
20 40 60 80
Censoring percentage
20 40 60 80
Censoring percentage
0.14
0.10
N 30
N 50
N 100
N 150
0.02
0.06
Variance of bias
0.12
0.08
Variance of bias
0.00
0
Modified corr. estimator
N 30
N 50
N 100
N 150
0.04
0.10
0.02
0
Modified estimator
N 30
N 50
N 100
N 150
0.06
Variance of bias
0.12
0.08
0.04
0.00
Variance of bias
Corrected estimator
0.14
Estimator
N 30
N 50
N 100
N 150
0
20 40 60 80
Censoring percentage
0
20 40 60 80
Censoring percentage
(d) For T ∼ WB (3.39, 3) & C ∼ U (a, 2a).
KM
KM
Fig 5. The variance of the bias of the K−M mean lifetime estimators Ŝmean
, S̃mean
,
KM
KM
∗
∗
Sˆ mean and S˜ mean in 10000 simulation runs.
CRiSM Paper No. 13-09, www.warwick.ac.uk/go/crism