External validation and updating of a prognostic survival
model 1
Patrick Royston & Mahesh K B Parmar
Hub for Trials Methodology Research
MRC Clinical Trials Unit and University College London
222 Euston Road,
LONDON NW1 2DA
UK.
Douglas G. Altman
Centre for Statistics in Medicine, University of Oxford
17 March 2010
Summary
We propose the view that external validation of a prognostic survival model means evaluating its performance in an independent (secondary) dataset obtained separately from the data
in the original (primary) study, with a minimum of change or reconstruction of the model. The
primary and secondary datasets should consist of patients with a set of prognostically relevant
predictors in common, comparably de…ned time-to-event outcomes with similar follow-up times,
and the same clinical condition observed in similar settings. We suggest graphical and analytical techniques for evaluating the …t of a proposed model in the secondary dataset. The analyses
involve evaluating by how much minor changes to the model in the secondary dataset improve
the …t there. The ‘ideal’ model does not require any such modi…cation. Other assessments
of performance we discuss include a goodness of …t test and a measure of explained variation.
We note that full validation of a model requires the model to provide a complete probabilistic
description of the data, su¢ cient to predict the survival probabilities at any relevant timepoint
and for any combination of values of the prognostic factors. The Cox proportional hazards
model was designed to estimate the e¤ects of covariates on the hazard function, but not to
estimate or predict survival probabilities; by de…nition, it does not satisfy our full validation
requirement. Instead, we recommend use of adequately ‡exible parametric survival models,
both for estimation and validation. We focus on the families of models proposed by Royston &
Parmar. We show how the techniques may be used in the analysis of two datasets in cancer.
1
Introduction
Techniques for validation of regression models may be classi…ed as internal or external. The key
element of internal validation is that only one dataset (the primary dataset) is used. Internal
1
Research report No. 307, Department of Statistical Science, University College London. Date: March 2010.
1
validation always involves some form of data re-use. One older technique is to split the data
at random, develop a model on one portion and evaluate it on the other. More recent and
arguably more appropriate techniques include k-fold cross-validation and bootstrap resampling
[1]. While internal validation can throw light on possible fragilities (such as ‘over-optimism’) of
the model, it can never be an adequate substitute for validation on independent data (external
validation). Unvalidated models are likely to be “quickly forgotten” [2].
In external validation, two independent datasets (primary and secondary) are required.
Typically, the primary dataset is collected …rst and a prognostic model is derived from it.
Later, further relevant data are assembled to form the secondary dataset and the performance
of the model is evaluated. In this paper, we propose the view that external validation of a
prognostic survival model derived on the primary dataset means evaluating its performance in
the secondary dataset with a minimum of change or reconstruction of the model. Prerequisites
of such analyses are that the primary and secondary datasets comprise patients having a set of
prognostically relevant predictors in common, comparably de…ned time-to-event outcomes with
roughly similar follow-up times, and the same clinical condition observed in similar settings.
The acceptable degree of di¤erence between the populations from which the samples were drawn
is a matter of debate [3]; for example, it may be unreasonable to expect a model derived on a
sample of adults to perform well in children. However, we do not address that issue here. The
populations and samples under study are taken as given.
External validation of prognostic models with a binary outcome (usually, logistic regression
models) has become quite a well-established practice in the medical literature. An example is
reference [4]. However, the task there is much simpler than for survival models, since predictions
are essentially estimates of the patient- or group-speci…c probability of the event of interest,
conditional on covariates, at a prede…ned, …xed point in time. Estimating and comparing
survival probabilities across the whole epoch since study inception are avoided. By contrast,
there is little statistical literature on formal techniques for external validation of models for timeto-event data. Authors of medical papers tend to con…ne themselves to creating a prediction
score or nomogram (a graphical version of the score) on the primary data (or using an existing
score), generating prognostic groups from the score, and plotting Kaplan-Meier survival curves
for the prognostic groups in the secondary data (for example, references [5], [6]). A measure
of discrimination (see below) of the score, such as Harrell’s concordance or c-index [7], may be
evaluated on the primary and secondary datasets, and compared. Sometimes, a signi…cance
test, based on the logrank test or a Cox model, of the relative hazards among the prognostic
groups in the secondary dataset is performed. However, these approaches do not adequately
address the question of how well the model works in the secondary dataset, the main topic of
the present paper.
A prognostic survival model has a systematic component, typically several predictors relevant to the disease in question whose e¤ects are combined linearly into a risk score or prognostic index, and a stochastic part describing the probabilistic element, almost always represented
2
through the (unspeci…ed) hazard function in a Cox proportional hazards (PH) model. We agree
with van Houwelingen, who states “Generally speaking, validation of well-speci…ed (quantitative) models is better de…ned than validation of vague (qualitative) models that only produce
subgroups.” [8] More precisely, we regard the characteristic aim of a survival model, which
typically operates with right-censored time-to-event data, to be to estimate or predict survival
probabilities at given times after the temporal origin at either the individual or the group
level, conditional on covariates or prognostic factors. The terms ‘covariates’, ‘predictors’ and
‘prognostic factors’ are used interchangeably in the present context. While in our validation
activities we largely focus on this aspect of model performance, we also look at global measures
of explained variation and goodness of …t.
The predictors used in the model for the primary dataset must somehow be selected from
a set of candidate variables. The process can be done in many ways (e.g. use prespeci…ed
predictors based on literature evidence and/or medical knowledge, use all available predictors,
select variables by criteria of statistical importance, etc.) We do not consider in any detail the
method of selecting predictors in the primary dataset. We concentrate on external validation,
interpreted as the model’s performance in a separate, secondary dataset.
The concept of validation appears at face value to be straightforward: either a model ‘validates’or it does not. The dichotomy is deceptive. In practice, assessment of model performance
in new data does not give a simple ‘yes/no’answer. Many performance measures are available,
and each of them is a value on a continuous scale. Subjective judgment of performance is
therefore always needed, guided by clinical aims of the model [9]. Traditionally, performance
measures are classi…ed in terms of ‘calibration’or ‘discrimination’. Calibration relates to bias
in predictions, and discrimination to the ability of a model to distinguish between outcomes of
patients with putatively di¤erent risks of an event. However, the attributes are not necessarily independent. Further, measures of calibration or discrimination do not directly re‡ect the
goodness of …t of a model.
Validation is becoming an increasingly important issue in prognostic studies, as interest in
prognostic and ‘predictive’molecular and genetic markers grows, e.g. references [10], [11]. (A
predictive marker is one that predicts response to treatment. For example, in primary breast
cancer, oestrogen receptor status of the tumour predicts whether a patient’s cancer responds
to the hormonal drug tamoxifen.) Practical guidance on the assessment of such markers is
however very limited.
The Cox proportional hazards model is a near-universal tool in the analysis of prognostic
models in clinical medicine [12]. However, it was not formulated with our concept of full external
validation in mind. The main issue is that a parametric estimate of the baseline survival function
is not available (note that ‘baseline’refers to zero values of the covariates, not to time t = 0).
Prediction in new data of survival probabilities from a Cox model is therefore problematic.
Further, covariate e¤ects must conform to the proportional hazards assumption, which is quite
3
limiting, particularly in data with medium- or long-term follow-up. We also work in some detail
with three families of ‡exible parametric survival (‘Royston-Parmar’) models [13]. The three
families assume proportional (cumulative) hazards, proportional (cumulative) odds and probitscale covariate e¤ects, respectively. By the ‘scale’ of a model we mean the transformation of
the survival function on which covariate e¤ects are assumed linear. For example, covariates in
the PH model are assumed to act linearly on the log-minus-log transformation of the survival
function, referred to as hazards scaling. In Royston-Parmar models, the scale transformation
(or link function) of the baseline survival function is approximated parametrically by a restricted
cubic spline function (see section 2.2). In the simplest cases, the spline function reduces to a
linear function of log survival time. The models then simplify to the Weibull, log-logistic and
lognormal distributions, respectively, and so are seen as generalizations of these more familiar
survival models.
The goal of the paper is to propose and demonstrate a practical approach to full validation
of a model for time-to-event data. To keep things relatively simple, we do not consider models
with time-dependent covariate e¤ects— either time-varying e¤ects of static covariates, or (a
much less common case) dynamic covariates whose values change over time. We aim to show
that ‡exible parametric modelling is necessary if validation of both the systematic and the
stochastic components of a model is required (as it should be). We propose a technique which
teases out three aspects of possible lack of …t of a model on a secondary dataset: the location
of the baseline survival function, its shape, and calibration of the linear predictor or prognostic
index.
The plan of the paper is as follows. Section 2 outlines the Cox and Royston-Parmar models.
Section 3 describes the methodology, discussing model assessment, goodness of …t, validation,
prediction of survival probabilities, graphical display of results and explained variation. Section
4 describes two cancer datasets and prognostic models used to predict time-to-event outcomes
in them. Section 5 presents analyses of the datasets using the performance measures and
approaches to validation described in section 3. Section 6 is a discussion.
2
2.1
Models
Cox model
In a Cox model with covariate vector x and parameter vector , the stochastic component is
the (hypothetical) baseline hazard function, h0 (t) = h (t; 0), and the systematic part is the
prognostic index, (i.e. eta) = x . The baseline hazard function is ‘hypothetical’ in the
sense of depending on how the covariates x are coded and/or centered; there is no guarantee or
necessity that any patient in the sample has x = 0. Estimation of by maximizing the partial
log likelihood does not require an explicit formulation of h0 (t), which is not itself estimated,
4
hence the description of the model as ‘semi-parametric’. See section 3.5 for the implications of
the semi-parametric nature of the Cox model for validation.
Hazards-scaled Royston-Parmar models may be seen as parametric extensions of the Cox
model that retain the stochastic/systematic decomposition. Writing the Cox model for the
hazard function h (t; x) as
h (t; x) = h0 (t) exp (x ) ;
and integrating with respect to time, gives the equivalent model
H (t; x) = H0 (t) exp (x ) ;
a proportional cumulative hazards model. Royston-Parmar models include a parametric representation of H0 (t), whereas in the Cox model H0 (t) is left unspeci…ed. See section 2.2 for a
general de…nition of Royston-Parmar models.
2.2
Royston-Parmar survival models
Royston & Parmar [13] described families of ‡exible parametric survival models resembling
generalized linear models with various link functions. The models rely on transformation of the
survival function by a link function g (:),
g [S (t; x)] = g [S0 (t)] + x ;
(1)
where S0 (t) = S (t; 0) is the baseline survival function and is a vector of parameters to be
estimated for covariate vectors x. In earlier work, Younes & Lachin [14] used the parameterised
link function g (z; ) = ln z
1 = proposed by Aranda-Ordaz [15], in which = 1 corresponds to the proportional odds model and ! 0 to the proportional hazards (PH) model.
Royston & Parmar [13] investigated these two models and also considered the probit model for
1 (s), where
1 (s) is the inverse standard Normal distribution function.
which g (s) =
By letting
! 0 and exponentiating (1), proportional cumulative hazards models are
obtained, namely
H (t; x) = H0 (t) exp (x )
(2)
where H (t; x) is the cumulative hazard function and H0 (t) = H (t; 0) is the baseline cumulative
hazard function. This is also a PH model. Royston & Parmar [13] approximated ln H0 (t) by
a restricted cubic spline function of log time with two ‘boundary’knots and m interior knots,
namely
ln H0 (t) = 0 + 1 ln t + 2 v1 (ln t) + : : : + m+1 vm (ln t)
(3)
where v (ln t) = ln t; v1 (ln t) ; : : : ; vm (ln t) is a vector of spline basis functions (see Royston &
Parmar [13] for details). Ignoring the constant term 0 , a spline with m knots has m+1 degrees
5
of freedom (d.f.). The PH model obtained by substituting (3) for H0 (t) in (2) is fully speci…ed
parametrically and may be written
ln H (t; x) =
0
+ v (ln t)
+x
(4)
If we write the prognostic index as = x then model (4) may be extended slightly or ‘recalibrated’as
ln H (t; x) = 0 + v (ln t) +
(5)
Since the term v (ln t) in (5) has not changed, the shape of the baseline cumulative hazard
function is unaltered, but its location changes through 0 +
when 0 6= 0 or
6= 1.
Estimation of
is relevant in validation studies where recalibration is of interest (see reference
[8], for example). Regression on the prognostic index in the primary (derivation) data yields
b
1. In the secondary (validation) data, a value of b much less than 1 may indicate a
model with poor performance. If b = 0 the model is probably useless.
3
3.1
Methods
Preliminary investigation
As a …rst step before any modelling starts, it is worthwhile plotting the overall Kaplan-Meier
survival curves by dataset. If survival varies noticeably across the datasets, inaccurate prediction of survival probabilities from the primary to the secondary datasets may result. If
di¤erences in the shapes of the survival curves are present, systematic di¤erences in the study
populations may be indicated.
Among possible exploratory analyses, it is also worth comparing the distributions of the
covariates between the primary and secondary datasets. Substantial di¤erences here may also
indicate important di¤erences in the study populations.
3.2
Components of parametric survival models
The prognostic index, , is of central importance in validation and assessment of model performance. The spread or variance of across patients indicates the degree of discrimination,
predictive ability or variation explained by the model [16]. A large variance suggests that a
model has a (relatively) good ability to discriminate between patients who have di¤erent risks
of an event. The role of the prognostic index has been widely recognised, but that of the
baseline survival function, S0 (t), has often been ignored, presumably because it is not part of
Cox model. However, S0 (t) is arguably of equal importance, since it determines the pattern
of absolute survival probabilities over time, whereas the prognostic index expresses the (log)
relative risk (hazard, etc) of one individual in the sample relative to another, or relative to a
6
hypothetical baseline individual for whom all covariate values are zero. An appraisal of a survival model is incomplete without considering both components. For a model to be clinically
useful, it must estimate actual probabilities.
We present an approach to assessing model performance in independent data that takes into
account both the prognostic index and the baseline distribution function. In the proportional
cumulative hazards model (4), the baseline distribution function is the baseline log cumulative
hazard function. Two aspects of the baseline distribution function, represented by the parameters 0 and in (4), may be distinguished. The …rst, 0 , is a type of intercept and indicates the
position in ‘log relative risk space’of the baseline distribution function. A higher value of 0
indicates generally higher risk and hence lower survival. As already mentioned, the parameter
vector controls the shape of the baseline distribution function and hence of the predicted survival and hazard functions. The prognostic index raises (when > 0) or lowers (when < 0)
the risk for an individual relative to 0 . In a validation study, interest lies in whether values
of 0 , or from the primary dataset still hold good in the secondary dataset. Sometimes a
simple recalibration
of markedly improves model …t in the secondary dataset.
3.3
Model validation and updating
In the secondary dataset, the prognostic index is calculated using values of the parameter
vector b estimated on the primary dataset and applied to the covariate values of the secondary
dataset. The components of are not re-estimated on the secondary dataset. Similarly, the
baseline survival function is calculated in the secondary dataset using the parameter vector
(b0 ; b ) estimated on the primary dataset, together with the (log) time values in the secondary
dataset and the set of spline knots used in the primary dataset.
It is possible that the predicted survival probabilities, derived from b0 ; b and b , are suf…ciently accurate in the secondary dataset to remove the need to ‘tune’ any of the model
parameters. This is the ideal scenario regarding validation; we refer to it as Case 1. We now
describe six additional cases, which comprise a structured, ordered set of models allowing one
to assess and if necessary improve the …t of the original model in the secondary dataset. The
adjustments provide a route to updating a given model in a simple way, thus avoiding complete
reconstruction (see van Houwelingen [8] for a related approach). Of course, in some situations
reconstruction may be unavoidable. In any case, an updated model should be re-validated in
further independent data.
‘Case 1’(see section 3.2) is the situation that the parameters of model (4) estimated on the
primary dataset also provide an adequate …t to the secondary dataset. Going beyond that, we
consider four general and parsimonious ways to update the model on the primary dataset using
data from the secondary dataset. These are summarized in Table 1. [TABLE 1 NEAR HERE]
The need for such updating, if detected, indicates di¤erent types of lack of …t of the original
7
Baseline
distribution
Not re-estimated
0 re-estimated
re-estimated
re-estimated
0 and
1
Estimated
Case 1
Case 2a
Case 2b
Case 3
Case 4
Case 5a
[Case 5b ]
Case 6
Table 1: De…nition of Cases 1-6 in structured updating of a ‡exible parametric survival model
on a secondary dataset.
Infeasible (see text)
model:
1. Re-estimate 0 (Case 2a). Altering 0 moves the entire baseline distribution function
up or down. The shift is sometimes referred to as ‘recalibration in the large’ [1]. A
situation in which 0 changes is where the susceptibility of the patients to an event di¤ers
systematically between the primary and secondary datasets.
2. Re-estimate (but not 0 ) (Case 2b). This is a more radical revision. Changing
the shape of the baseline distribution, but not its general level.
3. Re-estimate
changed.
0
and
alters
(Case 3). Similar to Case 2b, except that the general level is also
4. Regarding as a continuous covariate with a linear e¤ect in the secondary dataset, estimate its slope,
(Cases 4, 5 and 6). This idea, which was proposed for the logistic
model by Miller [17], posits that the e¤ect of is linear but may be mis-calibrated (i.e.
6= 1). Since b is often < 1,
is sometimes described as a shrinkage factor.
Table 1 shows how Cases 1-6 relate to the types of model revision listed above. [TABLE 1
NEAR HERE] The reason why there is no Case 5b is that estimating
and re-estimating
with 0 …xed is infeasible. Multiplying by
a¤ects the value of 0 .
A nice feature of this schema is that almost all of the pairs of models moving down each
column and across each row of Table 1 are nested. The exception is that Case 2a is not nested
in Case 2b, although Cases 2a and 2b nest Case 1 and are nested in Case 3. The deviances
(minus twice the maximised log likelihood) of the 7 models may be tabulated and compared.
The d.f. of the 7 models corresponding to Cases 1-6 are 0, 1, 1 + m, 2 + m, 1, 2, 3 + m,
respectively. Likelihood ratio tests based on 2 approximations may be performed to compare
pairs of nested models.
These simple adjustments to the original model may produce major or minor changes in the
goodness of …t (deviance). Such information may guide the interpretation of the results, and
where appropriate, the choice of a ‘…nal’updated model for the secondary dataset.
8
3.4
Goodness of …t
Attention should also be paid to goodness of …t of the model in the primary and secondary
datasets. A common cause of poor …t of a survival model is failure of the proportionality
assumption (e.g. non-proportional hazards). A simple ‘global’or ‘average’assessment of nonproportionality of the prognostic index on the chosen scale of a Royston-Parmar model is to
test the interaction between the spline basis functions and the prognostic index. Such tests are
discussed in principle in section 4.1 of Royston & Parmar [13].
The …rst step is to estimate b on the primary dataset using model (4). In the next step, (4)
is extended to include the interaction between b and the spline basis functions v (ln t), giving
(for cumulative hazards scaling) the following model:
ln H (t; x) =
0
+ v (ln t)
+
b + interaction
(6)
where interaction = y1 b ln t + y2 bv1 (ln t) + : : : + ym+1 bvm (ln t). The interaction induces nonproportionality of cumulative hazards by allowing the shape of the distribution function to vary
with b. All the parameters in (6) are estimated by maximum likelihood in the primary dataset.
To test for lack of …t, the di¤erence in deviance between (4) and (4) is compared with a 2
distribution on m + 1 d.f.
Given b from the primary dataset (now evaluated on the secondary dataset), the same
approach to testing an interaction is applied in the secondary dataset. Since the parameters of
(6) are re-estimated in the secondary dataset, …rst without and then with the interaction term,
the two models that are compared resemble Case 6 (see Table 1). Thus there is recalibration
of the prognostic index and re-estimation of the distribution function. The goodness of …t
assessment in the secondary dataset, therefore, is of a revised model.
Since categorisation of the prognostic index is avoided, the approach is parsimonious, which
should confer good power. However, the properties of the test await detailed evaluation.
In the Cox model, a test of non-proportional hazards in the same spirit may be performed
by applying Grambsch-Therneau tests [18] of standardized Schoenfeld residuals for in the
primary and secondary datasets. Other possibilities are described by Sauerbrei et al [19].
3.5
Validation of a Cox model
Principles similar to those just described for parametric survival models may be applied to
validating a Cox PH model, but with restrictions. As already mentioned, the Cox model does
not include a transportable estimate of the baseline survival function, making prediction of
the latter from the primary to the secondary dataset infeasible. Two updating options remain
from the list given in Table 1, equivalent to Cases 3 and 6 in section 3.3: either
= 1 (Case
3, no recalibration) or
is estimated (Case 6, recalibration is performed). In each case, the
9
(possibly recalibrated) prognostic index is applied to the baseline survival function, estimated
by a standard method (such as Lawless [20]) at = 0 in the secondary dataset. To test the
hypothesis
= 1, i.e. to assess the need for recalibration, the deviance di¤erence between
Cases 3 and 6, based on the partial likelihood, may be compared with a 2 distribution on 1
d.f. Further investigation of possible model weaknesses of the type available with a parametric
model cannot be done with the Cox model.
An alternative approach to validating a Cox model is to regard a Royston-Parmar PH model
as a Cox model with a parametric baseline distribution function. Validation is done exactly as
described in section 3.3. The steps of the analysis are
1. Fit the Cox model to the primary data and estimate the prognostic index, ;
2. In Royston-Parmar PH models for the primary data with o¤set, search for a parsimonious spline function for the baseline cumulative hazard function;
3. Apply the methods of section 3.3 to the secondary dataset.
In fact, since Cox and Royston-Parmar PH models yield almost identical parameter estimates [13], one could dispense with …tting the Cox model and regard validating a RoystonParmar model as equivalent to validating a Cox model with the same covariates.
Assessment of the discrimination of
discussed in section 3.7.
3.6
in a secondary dataset is straightforward, and is
Prediction of survival probabilities and graphical display
A cardinal strength of parametric survival models is their ability to predict survival probabilities
for individuals and groups in the primary and secondary datasets. Survival probabilities in
groups may be estimated by averaging the survival curves for individuals at the each of the
observed failure and censoring times. The group survival probabilities may then be compared
with Kaplan-Meier curves.
Risk groups must …rst be created by categorising the (continuous) prognostic index, .
Clearly many such schemes are possible. For the purpose of validation (but not necessarily
more generally), we suggest forming three groups using cutpoints at the …rst quartile and third
quartile of the distribution of across the individuals in the primary dataset. (If validation
of more extreme predicted risks is important, the cutpoints can be adjusted accordingly.) The
predicted survival curves are averaged over all individuals in a given group at the observed
failure and censoring times. Predicted and observed (Kaplan-Meier) curves may be compared
graphically. As seen in the examples to come, the resulting graph, with all groups displayed
in the same plot, indicates the predictive performance of the model over the entire time range.
10
We strongly recommend producing such plots, which with parametric models may be done for
Cases 1-6.
For the Cox model, similar plots may be produced, but as discussed in section 3.5, they are
limited to Cases 3 and 6. The predicted survival probabilities are again obtained by averaged
individual survival curves in risk groups. The predicted survival curve for an individual with
prognostic index is S0 (t)exp( ) , where S0 (t) is the baseline survival distribution estimated in
the secondary dataset.
Similar graphical comparisons of observed and predicted survival probabilities may also be
produced for the model for the primary dataset. They are quite helpful in assessing the accuracy
of the predicted survival probabilities over time. Comparison of the plots between the primary
and secondary datasets is also informative. For example, the plots may reveal lack of model …t
on the primary or secondary dataset (or both), or that model …t is worse (or indeed better) in
the secondary than the primary dataset.
3.7
Measure of explained variation suitable for validation studies
The explained variation statistic R2 for the linear regression model y
may be written as
var ( )
R2 = 2
+ var ( )
N
;
2
with
=x
(7)
where var( ) is the the variance of the distribution of across the sample (see eqn. (9) of
Helland [21]). Several authors have proposed versions of explained variation statistics for use
with censored survival data. These measures have not necessarily been extensions of the linear
regression case. For example, Graf et al [22] based their proposed measure on the Brier score
for survival data (a measure of predictive accuracy at the individual level), while others (e.g.
Schemper [23]) have worked with the survival curves estimated from a model.
2 , a transformation of their DRoyston & Sauerbrei [16] proposed a measure called RD
measure of discrimination, for use with the Cox model and other fairly general models for
2 is de…ned analogously to R2 in (7) as
censored survival data. RD
2
RD
=
D2 = 2
2 + D2 =
2
(8)
where 2 = 8= and D is the slope of the regression of the survival outcome on scaled Normal
order statistics corresponding to . See Royston & Sauerbrei [16] for justi…cation and for
explanation of the motivation, interpretation and properties of D. Under the assumption of
normality of the distribution of across the sample, D2 = 2 is an estimate of var( ) that is
nevertheless reasonably robust to departures from normality of . Furthermore, D and hence
2 are insensitive to outliers in
RD
when the underlying distribution of is normal but possibly
contaminated by a small number of extreme values.
11
2 has the advantages that it was designed for use with Cox models
For present purposes, RD
and Royston-Parmar models of all three types (hazard, odds and probit scaling), and that it is
suitable for assessing explained variation in a secondary dataset (albeit in a ‘Case 6’scenario,
since the outcome must be regressed on in the secondary dataset). Although D changes
2 by altering the value of 2 . As a
according to the scaling, the scaling is allowed for in RD
2 does not usually change much across di¤erent scalings. See Appendix A of Royston
result, RD
& Sauerbrei [16] for further details, including the values of 2 that are used in (8) with each
2 is available [24].
type of model. Stata software to compute D and RD
4
4.1
Example datasets and models
Breast cancer
From July 1984 to December 1989, the German Breast Cancer Study Group recruited 720
patients with primary node positive breast cancer into a 2 2 trial investigating the e¤ectiveness
of three versus six cycles of chemotherapy and of additional hormonal treatment with tamoxifen.
We analysed the recurrence-free survival time of the 686 patients who had complete data for the
prognostic factors age (x1), menopausal status (x2), tumour size (x3), tumour grade (2 dummy
variables x4a, x4b), number of positive lymph nodes (x5), progesterone (x6) and oestrogen (x7)
receptor status. With updated follow-up [25], 398 patients had had an event recorded. This is
the primary dataset. The secondary dataset is from a trial of 303 patients (158 events) with
identical inclusion criteria, procedures and recorded prognostic factors, but studying a di¤erent
intervention. For comparability, we truncated follow-up time at 9 years in both datasets, thus
censoring 17 (3:1%) of the total of 556 events.
All Cox models for the primary dataset we investigated had signi…cant non-proportional
hazards (see also section 5.1). Using multivariable fractional polynomial (MFP) methodology
[26], which combines backward elimination of weakly in‡uential predictors with selection of fractional polynomial functions for continuous predictors, we developed a Royston-Parmar model
with probit scaling and m = 1 knot (see section 2.2). It included the following predictors: x1 1 ,
x1 0:5 , x3, x4a, log x5, log (x6+1). This model was validated (i.e. evaluated) on the secondary
dataset. We also evaluated the Cox model III reported in Table 4 of reference [26].
Figure 1 shows Kaplan-Meier curves for the two studies (datasets). [FIGURE 1 NEAR
HERE] The survival curves are extremely similar up to about 3 years and not much di¤erent
after that.
12
1.00
0.75
S(t)
0.50
0.25
0.00
Prim ary dataset
Secondary dataset
0
3
6
Recurrence-free survival tim e, yr
Num ber at risk (number of events)
Prim ary
686
(235)
407
(115)
243
Secondary
303
(98)
189
(37)
112
9
(35)
(19)
126
51
Figure 1: Breast cancer datasets. Kaplan-Meier curves for recurrence-free survival by dataset
4.2
Bladder cancer
Over the past two decades, the European Organisation for the Research and Treatment of
Cancer (EORTC) and the UK Medical Research Council (MRC) Working Party on Super…cial
Bladder Cancer have performed several phase III randomised controlled trials for the prophylactic treatment of stage T1 /Ta super…cial bladder cancer following transurethral resection.
Further clinical details are given by Royston et al [27] and references therein. Royston et al
[27] developed a Cox model for the progression-free survival of the 3441 patients (721 events)
in all 9 trials, adjusting for study di¤erences. We combined the seven EORTC trials to form
the primary dataset and the two MRC trials to constitute the secondary dataset. Since the aim
of the analysis was to illustrate the methodology of validation, the small overall di¤erences in
progression-free survival between the studies within each combined dataset were ignored. We
truncated the follow-up at 10 years, thus censoring 21 (2:9%) events overall.We took the prognostic model (10 terms) described in Table 3 of reference [27], …tted it to the primary dataset
and evaluated it on the secondary dataset. A Royston-Parmar proportional cumulative hazards
model with m = 1 (i.e. one interior knot) and a Cox model were used.
Figure 2 shows Kaplan-Meier curves for the two sets of trials (datasets). [FIGURE 2 NEAR
HERE] The survival curves di¤er in shape, with a much higher hazard in the early follow-up
13
1.00
0.75
S(t)
0.50
0.25
0.00
Prim ary dataset
Secondary dataset
0
2.5
5
7.5
Progression-free survival tim e, yr
Num ber at risk (number of events)
Prim ary
2610 (280) 1934 (167) 1352 (86)
731
Secondary
831
(32)
749
(33)
655
(44)
362
10
(29)
(29)
322
157
Figure 2: Bladder cancer data. Kaplan-Meier curves for progression-free survival by dataset
period in the primary dataset. Notice also the high attrition rates in each dataset, with many
patients being lost to follow up at all stages of each study. Attrition has the potential to induce
bias if it is not at random with respect to the survival outcome.
5
5.1
Results
Breast cancer example
Column (a) of Table 2 shows the results of the validation analysis for the breast cancer datasets.
[TABLE 2 NEAR HERE] Note that the Royston-Parmar models had probit link functions,
whereas the Cox model assumed proportional hazards.
The Royston-Parmar models corresponding to Cases 1-6 all …t about equally well in the
secondary dataset, suggesting that the model estimated on the primary dataset may be used
to predict in the secondary dataset without the need to modify the baseline distribution or the
prognostic index (Case 1). This is the benchmark or ideal validation scenario. The goodness of
…t test for the prognostic index of the probit model is not signi…cant at the 5% level in either the
primary or the secondary dataset. Figure 3 compares the observed and predicted recurrence-free
14
Model
(a)
Breast cancer
(b)
Bladder cancer
Royston-Parmar
Case 1
Case 2a
Case 2b
Case 3
Case 4
Case 5a
Case 6
Calibration slope b (SE)
2 (SE) primary data
RD
2 (SE) secondary data
RD
GOF primary data (P -value)
GOF secondary data (P -value)
651:03
650:09
647:96
647:95
650:08
650:08
647:94
0:99 (0:13)
0:23 (0:03)
0:22 (0:05)
0:24
0:45
900:80
889:52
864:18
850:00
891:73
887:85
846:30
1:26 (0:13)
0:23 (0:02)
0:30 (0:05)
0:44
0:0009
Cox
Case 3
Case 6
Calibration slope b (SE)
2 (SE) primary data
RD
2 (SE) secondary data
RD
GOF primary data (P -value)
GOF secondary data (P -value)
1576:00
1575:87
0:95 (0:14)
0:20 (0:03)
0:17 (0:04)
0:004
0:11
1648:63
1644:54
1:27 (0:14)
0:23 (0:02)
0:31 (0:05)
0:22
0:003
Table 2: Model deviances, calibration constants, goodness of …t and explained variation for
Cases 1-6 for (a) breast and (b) bladder cancer datasets. Probit and proportional-hazards
Royston-Parmar models were …tted to the breast and bladder cancer datasets, respectively.
GOF denotes goodness of …t
Deviances for Cases 1-6 are based on maximized likelihoods
Deviances for Cases 3 and 6 are based on maximized partial likelihoods
15
survival probabilities for the primary data and Cases 1-6. [FIGURE 3 NEAR HERE] Although
3
6
9
1
0 .25 .5 .75
0 .25 .5 .75
0 .25 .5 .75
0
0
3
6
9
6
9
6
9
6
9
1
0 .25 .5 .75
1
3
3
Case 4
0 .25 .5 .75
0 .25 .5 .75
0
0
Case 3
1
Case 2b
0
3
6
9
6
9
0
3
0 .25 .5 .75
1
Case 6
1
Case 5a
0 .25 .5 .75
Survival probability
Case 2a
1
Case 1
1
Primary data
0
3
6
9
0
3
Recurrence-free survival time, yr
Figure 3: Breast cancer data. Plots of observed and predicted progression-free survival probabilities from a Royston-Parmar probit model in 3 prognostic groups for the primary data and
Cases 1-6 for the secondary data (see text for details). Dashed lines, predicted probabilities;
solid lines and shaded areas, Kaplan-Meier estimates with their 95% pointwise con…dence intervals. Note that CIs for the central group sometimes overlap and obscure those for the outer
groups
the predicted survival probabilities are not a perfect …t in the poor prognosis group in the
secondary dataset, the number of events beyond 4 years is very small. The results for Case 6 (4
additional parameters …tted) are similar to those for Case 1 (no additional parameters …tted).
2 values are similar between the two datasets. The value of b is near 1.
The RD
The results from the Cox model are similar to those from the Royston-Parmar probit model
(see Table 2). However, the PH assumption for the prognostic index is violated in the primary
dataset (P = 0:004, Grambsch-Therneau test). Figure 4 shows the observed and predicted
survival probabilities. [FIGURE 4 NEAR HERE] Slight lack of …t is visible in the extreme
groups, the pattern being similar between the primary and secondary datasets.
Finally, as described in section 3.5, we validated a Cox model with the same covariates as the
probit model through the ‘equivalent’Royston-Parmar PH model with 2 df (one interior knot).
16
.75
.5
.25
0
0
0
3
6
9
6
9
0
3
6
9
.5
.75
1
Case 6
0
.25
Survival probability
.25
.5
.75
1
Case 3
1
Primary data
0
3
Recurrence-free survival time, yr
Figure 4: Breast cancer data. Plots of observed and predicted recurrence-free survival probabilities from a Cox model in 3 prognostic groups for the primary dataset and for Cases 3 and 6
in the secondary dataset (see text for details). Dashed lines, predicted probabilities; solid lines
and shaded areas, Kaplan-Meier estimates with their 95% pointwise con…dence intervals
17
The comparison of Cases 1-6 was essentially as given in Table 2, except that the deviances of
all the models were around 660, compared with around 650 for the probit models in Table 2.
This shows that the probit models are a rather better …t.
5.2
Bladder cancer example
Column (b) of Table 2 shows the results for Cases 1-6 for the bladder cancer datasets and
Cases 3 and 6 for the Cox model. The pattern of deviance di¤erences suggests that estimating
both 0 and is required in the Royston-Parmar model, pointing to a di¤erence in shape and
position of the baseline distributions between the datasets. The …t of the Case 3 model is not
signi…cantly worse than that of Case 6, although Case 6 is marginally better (P = 0:054). The
calibration slope in Case 6 is 1:26 (SE 0:13).
Figure 5 shows the observed and predicted survival probabilities in 3 prognostic groups
for the primary data and Cases 1-6 for the secondary data. [FIGURE 5 NEAR HERE] The
2
4
6
8
10
1
.8
.4
.6
.8
.4
.6
.8
.6
.4
0
0
2
4
6
8
10
4
6
8
10
4
6
8
10
8
10
1
.4
.6
.8
1
.8
.4
2
2
Case 4
.6
.8
.6
.4
0
0
Case 3
1
Case 2b
0
2
4
6
8
10
8
10
0
2
4
6
.6
.4
.4
.6
.8
1
Case 6
1
Case 5a
.8
Survival probability
Case 2a
1
Case 1
1
Primary data
0
2
4
6
8
10
0
2
4
6
Progression-free survival time, yr
Figure 5: Bladder cancer data. Plots of observed and predicted progression-free survival probabilities from a Royston-Parmar proportional hazards model in 3 prognostic groups for the
primary data and Cases 1-6 for the secondary data (see text for details). Dashed lines, predicted probabilities; solid lines and shaded areas, Kaplan-Meier estimates with 95% pointwise
con…dence intervals. Note that the vertical axes begin at 0.4
18
observed (Kaplan-Meier) and predicted probabilities agree well in the primary data but rather
poorly in all cases in the secondary data. The goodness of …t tests align with this observation,
the P -values being 0:4 and 0:0009 in the primary and secondary data, respectively (Table 2).
2 ) is actually higher in the secondary data (0:31 vs. 0:23).
Explained variation (RD
The prognostic index was split at its 25th and 75th centiles in the primary dataset. In
the secondary dataset, the risk groups de…ned using the same cutpoints contain approximately
39=48=13 percent of patients. The case mix is di¤erent, with about half as many poor-prognosis
patients (see also Figure 2).
Results for the Cox model are also given in Table 2. Where available, they mirror closely
the results for the Royston-Parmar model. Figure 6 shows the observed and predicted survival
probabilities. [FIGURE 6 NEAR HERE] Apart from the absence of smoothness of the predicted
.8
.6
.4
0
2
4
6
8
10
6
8
10
0
2
4
6
8
10
.6
.8
1
Case 6
.4
Survival probability
.4
.6
.8
1
Case 3
1
Primary data
0
2
4
Progression-free survival time, yr
Figure 6: Bladder cancer data. Plots of observed and predicted progression-free survival probabilities from a Cox model. For details, see Figure 4
survival functions, the results are similar to those in Figure 5.
The main conclusion from this example is that the prognostic index of the prognostic model
validates well, but that the baseline survival distributions are substantially di¤erent between the
primary and secondary datasets. A possible explanation could be that the patient populations
in the two trials datasets were somewhat dissimilar. The MRC patients had not yet had any
19
recurrence of their cancer, whereas a substantial proportion of the EORTC patients had had
recurrences. This may explain the higher hazard of early disease progression seen among the
EORTC patients. However, the shape of the survival distribution in the EORTC (primary)
patients is similar between patients who had and had not experienced a recurrence (data not
shown).
6
Discussion
In our view, validation is not about re…tting the predictors in a postulated model on the
secondary dataset and comparing the estimated ’s, as some have done. It is about predicting
relevant quantities (such as survival distributions) from a model fully speci…ed on the primary
dataset to a secondary dataset (our Case 1), and examining its accuracy. Only parametric
models support this approach for time-to-event data. Since the Cox model does not estimate
the baseline distribution, it cannot be fully validated. To throw further light on possible lack
of …t, we suggest modifying the model in minor ways corresponding to our Cases 2a-6. Tests
related to these models may point to di¤erences in model …t between the primary and secondary
datasets. We describe the di¤erences in terms of the parameters 0 ,
and
. It is also
informative to apply a simple goodness of …t test; we suggest a proportionality test based on
the prognostic index, . A model for which Case 1 …tted as well as Cases 2a-6 and which passed
the goodness of …t test would validate as well as it possibly could, within the limitations of our
method. That is so with the breast cancer example, but not with the bladder cancer example.
Most prognostic survival models in the (medical) literature are reported as b ’s with SE or CI
from a Cox model. Kaplan-Meier survival curves in groups obtained from the prognostic index
are sometimes presented, but no attempt is made to report formal estimates of the baseline
survival distribution. We propose that the baseline survival distribution and the prognostic
index are equally important components of a survival model. Without the baseline distribution,
model speci…cation is incomplete, so both should be estimated and reported. Many parametric
survival models are available (e.g. exponential, Weibull, etc), but they are little used in clinical
research. Presumably the reason is lack of ‡exibility, and the concern that the …t may be
poor. Due to the incorporation of spline modelling of the baseline cumulative distribution, the
Royston-Parmar family can successfully model arbitrarily complex distributions. We suggest
that they have potential value in prognostic research, particularly in better reporting and
validation of survival models.
Due to the pre-eminence of the Cox model, models with other link functions are not used
very often. The Royston-Parmar family o¤ers three di¤erent link functions. In the breast
cancer data, which has moderately long follow-up, we found that a probit model provided a
much better …t than a proportional hazards model (see also [28]), and also validated well.
We have concentrated here on external validation of survival models. Internal validation
20
is no substitute, of course, but our methods could be used, for example, with internal datasplitting. With the latter, the data are split (usually unequally) at random into a primary
and secondary dataset. Model development is done on the primary data and validation on the
secondary data, once testing and ‘tuning’of the model on the primary data has been completed.
Although our approach is formal in the sense that it provides estimates and signi…cance
tests relevant to the validation task, we regard the graphical component as equally important.
The plots of observed and predicted survival probabilities, in both the primary and secondary
datasets, are a useful tool for illustrating the amount of discrimination o¤ered by the model in
each dataset and the nature of the survival distributions, and may point to areas of lack of …t.
Varying prognostic classi…cation schemes could be used, depending on the clinical context. For
example, particular categorisations of the prognostic index in the secondary dataset could be
used to study predictions at the extremes of the risk distribution.
Perhaps the most challenging questions in validation are what do we expect to see, and what
is it reasonable to expect? A suggested benchmark of ‘optimal’performance is for a model to
give satisfactory performance in Case 1 (i.e. b0 and b not requiring revision), to have b near
2 values and to pass a goodness of …t test. The
1, and, in both datasets, to have comparable RD
probit model in the breast cancer example conforms to the benchmark. The bladder cancer
model does not; the predicted survival probabilities do not match the observed probabilities
very closely and Case 6 is a signi…cantly better …t than Case 1, both indicating lack of …t.
2 statistic, is actually better in
Nevertheless, the model’s discrimination, according to the RD
the secondary dataset. The clinical aim(s) are paramount. If the aim is accurate prediction of
survival probabilities, the bladder cancer model does not perform very well. If the aim is to …nd
prognostic factors leading to a risk score that discriminates well, the opposite is the case. There
is no absolute standard of performance here. Nevertheless, our proposed benchmark may be
helpful in that it supplies a practical framework in which di¤erent aspects of model validation
may be assessed systematically.
A situation we have not considered here is whether it is possible to validate (in our sense) a
published model for which the original (primary) data are not available. With the Cox model,
the answer is clearly ‘no’, since the baseline survival function is not available. With parametric
models, the answer is a provisional ‘yes’, depending on publication of the full speci…cation of
the model and its estimated parameters. For Royston-Parmar models, for example, one would
need to know the scale (hazards, odds or probit) of the model, the knot position(s) on t, and
the estimates of 0 ,
and . Precisely how parametric survival models should be reported
with possible validation (and other uses) in mind is a topic for further research.
Do we recommend the routine use of (‡exible) parametric survival models, both for modelling and for validation? The answer is ‘yes’— provided the chosen model …ts the data. In many
datasets (but by no means all) this rules out too-simple models, such as the exponential and
Weibull. Although the Cox model is convenient, its strength lies mainly in estimating relative
21
hazards for covariate e¤ects. It is ‘semi-parametric’in nature and was never intended for the
task of comprehensive prediction of survival probabilities and other important quantities. Although we favour Royston-Parmar models, other ‡exible parametric models exist, for example
those based on the Poisson distribution (e.g. reference [29]). The original implementation of
Royston-Parmar models is through the stpm user-written command for Stata [30]; an improved
version, stpm2, is available [31]. An undocumented feature of the mfp command in version 11
of Stata [32] allows users to develop MFP models [26] using stpm or stpm2. Features of mfp include elimination of weakly in‡uential variables by backward elimination and determination of
functional form for continuous predictors using fractional polynomial functions. Further Stata
software is available on request from the …rst author to carry out the computations described
in the paper.
References
[1] F. E. Harrell. Regression modeling strategies, with applications to linear models, logistic
regression, and survival analysis. Springer, New York, 2001.
[2] J.C. Wyatt and D. G. Altman. Prognostic models: clinically useful or quickly forgotten?
British Medical Journal, 311:1539–1541, 1995.
[3] K. G.M. Moons, D. G. Altman, P. Royston, and Y. T. Vergouwe. Generalisability, impact
and application of prediction models in clinical practice. 2007. In preparation.
[4] Y. Vergouwe, E. W. Steyerberg, R. S. Foster, J. D. F. Habbema, and J. P. Donohue.
Validation of a prediction model and its predictors for the histology of residual masses in
nonseminomatous testicular cancer. Journal of Urology, 165:84–88, 2001.
[5] J. Hermans, A. D. G. Krol, K. van Groningen, P. M. Kluin, J. C. Kluin-Nelemans, M. H. H.
Kramer, E. M. Noordijk, F. Ong, and P. W. Wijermans. International Prognostic Index
for aggressive non-HodgkinŠs lymphoma is valid for all malignancy grades. Blood, 86:1460–
1463, 1995.
[6] V. Ficarra, G. Martignoni, C. Lohse, G. Novara, M. Pea, S. Cavalleri, and W. Artibani.
External validation of the Mayo Clinic stage, size, grade and necrosis (SSIGN) score to
predict cancer speci…c survival using a European series of conventional renal cell carcinoma.
Journal of Urology, 175:1235–1239, 2006.
[7] F. E. Harrell, K. L. Lee, and D. B. Mark. Multivariable prognostic models: issues in
developing models, evaluating assumptions and accuracy, and measuring and reducing
errors. Statistics in Medicine, 15:361–387, 1996.
[8] H. C. van Houwelingen. Validation, calibration, revision and combination of prognostic
survival models. Statistics in Medicine, 19:3401–3415, 2000.
22
[9] D. G. Altman and P. Royston. What do we mean by validating a prognostic model?
Statistics in Medicine, 19:453–473, 2000.
[10] S. Paik, S. Shak, G. Tang, C. Kim, J. Baker, M. Cronin, F. L. Baehner, M. G. Walker,
D. Watson, T. Park, W. Hiller, E. R. Fisher, D. L. Wickerham, J. Bryant, and N. Wolmark.
A multigene assay to predict recurrence of tamoxifen-treated, node-negative breast cancer.
New England Journal of Medicine, 351:2817–2826, 2004.
[11] M. Buyse, S. Loi, L. van’t Veer, G. Viale, M. Delorenzi, A. M. Glas, M. Saghatchian
d’Assignies, J. Bergh, R. Lidereau, P. Ellis, A. Harris, J. Bogaerts, P. Therasse, A. Floore,
M. Amakrane, F. Piette, E. Rutgers, C. Sotiriou, F. Cardoso, and M. J. Piccart on behalf
of the TRANSBIG Consortium. Validation and clinical utility of a 70-gene prognostic
signature for women with node-negative breast cancer. Journal of the National Cancer
Institute, 98:1183–1192, 2006.
[12] A. Burton and D. G. Altman. Missing covariate data within cancer prognostic studies: a
review of current reporting and proposed guidelines. British Journal of Cancer, 91:4–8,
2004.
[13] P. Royston and M. K. B. Parmar. Flexible proportional-hazards and proportional-odds
models for censored survival data, with application to prognostic modelling and estimation
of treatment e¤ects. Statistics in Medicine, 21:2175–2197, 2002.
[14] N. Younes and J. Lachin. Link-based models for survival data with interval and continuous
time censoring. Biometrics, 53:1199–1211, 1997.
[15] F. J. Aranda-Ordaz. On two families of transformations to additivity for binary response
data. Biometrika, 68:357–363, 1981.
[16] P. Royston and W. Sauerbrei. A new measure of prognostic separation in survival data.
Statistics in Medicine, 23:723–748, 2004.
[17] M. E. Miller and S. L. Hui. Validation techniques for logistic regression models. Statistics
in Medicine, 10:1213–1226, 1991.
[18] P. M. Grambsch and T. M. Therneau. Proportional hazards tests and diagnostics based
on weighted residuals. Biometrika, 81:515–526, 1994.
[19] W. Sauerbrei, P. Royston, and M. Look. A new proposal for multivariable modelling of
time-varying e¤ects in survival data based on fractional polynomial time-transformation.
Biometrical Journal, 49:453–473, 2007.
[20] J. F. Lawless. Statistical models and methods for lifetime data. Wiley, New York, 1982.
[21] I. Helland. On the interpretation and use of R2 in regression analysis. Biometrics, 43:61–69,
1987.
23
[22] E. Graf, C. Schmoor, W. Sauerbrei, and M. Schumacher. Assessment and comparison of
prognostic classi…cation schemes for survival data. Statistics in Medicine, 18:2529–2545,
1999.
[23] M. Schemper. The explained variation in proportional hazards regression. Biometrika,
77:216–218, 1990.
[24] P. Royston. Explained variation for survival models. Stata Journal, 6:83–96, 2006.
[25] W. Sauerbrei, G. Bastert, H. Bojar, C. Beyerle, R. L. A. Neumann, C. Schmoor, and
M. Schumacher for the German Breast Cancer Study Group. Randomized 2 2 trial
evaluating hormonal treatment and the duration of chemotherapy in node-positive breast
cancer patients: An update based on 10 years’ follow-up. Journal of Clinical Oncology,
49:453–473, 2000.
[26] W. Sauerbrei and P. Royston. Building multivariable prognostic and diagnostic models:
transformation of the predictors by using fractional polynomials. Journal of the Royal
Statistical Society (Series A), 162:71–94, 1999.
[27] P. Royston, M. K. B. Parmar, and R. Sylvester. Construction and validation of a prognostic
model across several studies, with an application in super…cial bladder cancer. Statistics
in Medicine, 23:907–926, 2004.
[28] P. Royston. The lognormal distribution as a model for survival time in cancer, with an
emphasis on prognostic factors. Statistica Neerlandica, 55:89–104, 2001.
[29] P. W. Dickman, A. Sloggett, M. Hills, and T. Hakulinen. Regression models for relative
survival. Statistics in Medicine, 23:51–64, 2004.
[30] P. Royston. Flexible parametric alternatives to the Cox model, and more. Stata Journal,
1:1–28, 2001.
[31] P. C. Lambert and P. Royston. Further development of ‡exible parametric models for
survival analysis. The Stata Journal, 9:265–290, 2009.
[32] StataCorp. Stata Reference Manual, version 11. Stata Press, 2009.
24