WORK IN PROGRESS: PLEASE DO NOT CITE WITHOUT AUTHORS’ PERMISSION
Using survival analysis to improve estimates of life year gains in policy evaluations
Mark Harrison, Rachel Meacock, Matt Sutton
Manchester Centre for Health Economics, University of Manchester
Correspondence to: mark.harrison@manchester.ac.uk
Abstract
Background: Policy evaluations commonly convert estimated changes in short-term mortality to
expected life year gains using published estimates of life expectancy for the general population.
Examples include measuring NHS productivity, estimating the NICE decision threshold, and
evaluating pay-for-performance (P4P) programmes. Life expectancy of the affected patients may
differ from the general population. In trials, survival models are commonly used to extrapolate life
year gains from observed data.
Aim: To demonstrate the feasibility of using survival models to estimate future life year gains from
administrative data and assess the materiality of using different methods to estimate the life year
gains from Advancing Quality (AQ); a hospital P4P programme introduced in the North West (NW) of
England in October 2008.
Data: Patient-level data from Hospital Episode Statistics for patients admitted for pneumonia
between 1st April 2007 and 31st March 2010 linked to ONS death records for the period 1st April 2007
to 30th November 2011.
Methods: We apply three methods for estimating the life expectancy of the cohort of patients
admitted between 1st April 2007 and 31st March 2008 and compare the results to the actual survival
of this cohort observed until 30th November 2011. We then use the most accurate method to
estimate risk-adjusted life expectancy separately for four patient groups admitted to: (i) non-NW
hospitals in 2007/8; (ii) NW hospitals in 2007/8; (iii) non-NW hospitals in 2009/10; and (iv) NW
hospitals in 2009/10. We compare these results to a difference-in-differences estimate of the impact
of AQ on life expectancy from a risk-adjusted pooled survival model.
Results: Application of published life expectancy estimates for the general population to patients
admitted for pneumonia over-estimate their survival two-fold. Life expectancy for patients admitted
to non-NW hospitals decreased by 0.29 years, from 6.95 years in 2007/8 to 6.66 years in 2009/10.
Life expectancy increased by 0.83 years for patients admitted to NW hospitals between the same
two periods, from 6.16 years to 6.99 years. The risk-adjusted difference-in-difference estimate from
the pooled survival model suggests that life expectancy increased by a factor of 1.21 in the NW
compared to the rest of England.
Implications: Even with only short follow-up periods, survival analysis on patient-level administrative
data improves the accuracy of estimates of life expectancy gains considerably compared to the use
of general population figures.
Paper presented to the Health Economists’ Study Group, Glasgow June 2014
1
Introduction
As for health care treatments, the effects of health care policies and programmes should be
evaluated in terms of their impact on health outcomes. This impact can be comprised of effects on
both the quality and length of life. In this paper we focus on methods for estimating impacts on
length of life.
Policy evaluations commonly convert estimated changes in short-term mortality to projected gains
in life years using published estimates of life expectancy for the general population. Examples
include measuring NHS productivity (Castelli et al., 2007; Dawson et al., 2005), estimating the
National Institute for Health and Care Excellence (NICE) decision threshold (Claxton et al., 2013), and
cost-effectiveness analysis of pay-for-performance (P4P) programmes (Meacock et al., 2014).
The approach often taken in previous work has been to estimate the impact of a programme in
terms of changes in the probability of mortality within 30 days, assessed as a binary outcome
(Castelli et al., 2007; Dawson et al., 2005; Meacock et al., 2014). Estimated reductions in the
mortality rate are then translated into life years gained. Patients dying within 30 days are effectively
assumed to die instantly and attributed no survival days, whilst those surviving past 30 days are
assigned the remaining age-gender specific life expectancy of the general population.
These published estimates of life expectancy at particular ages are calculated from mortality rates in
the general population. Although they appear to be projections, life expectancy is in fact a summary
statistic of cross-sectional age-specific mortality rates. Life expectancy is the average length of life of
a hypothetical cohort of individuals exposed for each of their remaining years to the age-specific
annual mortality rates experienced by the general population who were alive at the start of a
reference period. Life expectancy is positive at each age and the implied length of life (years lived so
far plus remaining life expectancy) increases with age. Thus, while life expectancy at birth is 82 for
females in England, life expectancy for those who survive to age 82 is 8 years (Office for National
Statistics, 2011).
The length of life of patients affected by health care policies and programmes is, however, likely to
differ from that of the general population. This may lead to over-estimates of the effect on life years
of reducing the mortality rate. Even with the minimal available data of one financial year, it is
possible to observe the majority of patients for longer than the standard period of 30 days, with the
exception of those entering treatment during the last month of the data set. This enables
observation of patients for an additional 1-334 days depending upon when in the year they entered
treatment.
In clinical trials, survival models are commonly used to extrapolate gains in life expectancy from the
observed trial data. In this paper we consider whether the additional information on survival
available within administrative data sets, albeit censored, can be used to improve the accuracy of
estimated life years gained in policy evaluations. We investigate the feasibility and materiality of
using survival analysis models commonly employed in clinical trial analysis to extrapolate future
survival.
2
We use our previous analysis of the AQ programme as a motivating application. We estimated that
the programme reduced the 30-day mortality rate for pneumonia patients by 1.6 percentage points
(95% CI; -2.4,-0.8). In that analysis we considered all patients admitted over a three-year period,
including 18 months before the programme was introduced and the first 18 months of its operation.
In this paper we consider a more typical situation in which data on dates of admission and death are
available for one financial year prior to the introduction of the programme and one financial year
following its implementation. We examine whether it is possible to obtain accurate estimates of life
year gains with such short follow-up periods. We then compare these results to the observed
survival information now available with a longer follow-up.
Data
We use individual patient-level data on admission dates and patient characteristics from national
Hospital Episode Statistics for patients admitted between 1st April 2007 and 31st March 2010. These
were linked to Office of National Statistics (ONS) death records for a period up until 30th November
2011, the latest date on which the death records were complete at the time of the data extract.
We restrict the analysis to patients admitted in an emergency with pneumonia using ICD-10 codes
for the rules specified for the AQ scheme1. Secondary ICD-10 diagnosis codes were used to calculate
the Elixhauser algorithm (Quan et al., 2005), which was used to risk-adjust out estimates in
conjunction with the location from which a patient was admitted (own home or institution) and the
type of admission (emergency or transfer).
Methods
Development of methods
We first use the cohort of patients admitted during the period 1st April 2007 to 31st March 2008 to
demonstrate and compare three potential methods, using up to one year of survival information for
estimating expected remaining life years. We compare these to the observed survival of the cohort
to 30th November 2011. The purpose of this initial analysis is to identify the most accurate method
for estimating the remaining life years for a given patient population, which will later be used to
evaluate the impact of the programme in terms of life years gained using DiD.
The first method is that used in our original analysis of the programme (Meacock et al., 2014), in
which mortality occurring within 30 days of admission is defined as a binary outcome. Sex-specific
life expectancy estimates at each single year of age from 18-100 are then taken from the 2008-10
Interim Life Tables from the ONS (Office for National Statistics, 2011), and attached to patients
surviving beyond this 30 day period to estimate their remaining life expectancy. This method
implicitly assumes that individuals surviving beyond 30 days after admission survive on average the
life expectancy of the general population of the same age and gender. This method will be an
1
Primary diagnosis of J13, J14, J15, J16.0, J16.8, J18.0, J18.1, J18.2, J18.8 or J18.9, or a primary diagnosis of
A40.0, A40.1, A40.2, A40.3, A40.8, A40.9, A41.0, A41.1, A41.2, A41.3, A41.4, A41.5, A41.8, A41.9, J96.0 or J96.2
with a secondary diagnosis from the list of primary pneumonia diagnoses
3
inaccurate estimate of actual life expectancy for two reasons; because a) the period of survival
within 30 days is not incorporated into the estimate, and b) it assumes that the life expectancy of
individuals that survive past 30 days of admission will be equal to that of the general population of
their age and sex. Moreover, this method effectively ignores potential data on observed survival. As
at least one year of data is always likely to be available, it is possible to calculate the actual days
survived within the year. It is important to note that this method does correctly estimate the impact
of a programme upon 30 day mortality. It is the translation of these changes in mortality into life
years that is questioned here.
Method two uses all of the available information on mortality within the year of data (1st April 2007
– 31st March 2008) to observe mortality within that year, following patients for between 1 and 365
days depending upon their admission date. For those who did die during the period, the number of
days survived between the date of admission and the date of death is used. Age- and sex-specific
estimates of life expectancy are again applied to all patients who remain alive at the end of the
observed data period of one year. This improves upon method one by eliminating problem a) and
reducing but not eliminating inaccuracies due to b).
Method three uses parametric survival regression models on the observed one-year data and
extrapolates survival estimates of remaining years of life expectancy beyond this follow-up period.
The total number of years survived by the cohort is calculated as a sum of the survival time in the
observed period plus the predicted survival of those alive at the end of the observation period from
the parametric survival model.
These three methods were compared to what can be observed now with prolonged follow-up, using
the data now available to observe patients from the financial year 2007/8 until 30th November 2011.
This offers a maximum of 4.7 years of follow-up. We make comparisons to the observed survival
days during this period, and the proportion of patients surviving until 30th November 2011. We also
estimated the remaining life expectancy from 30th November 2011 onwards using the parametric
survival method described under method 3. These are our best estimates of the total life years for
the initial cohort until they all die, and we therefore treat this as the gold standard.
The estimated survival of the cohort using the three different methods were summarised as the total
number of years contributed by the cohort as a whole and the mean years contributed by each
person within the cohort. We compared methods two (one year follow up and life expectancy based
on life tables) and three (one year follow up and extrapolation of the survival function), using
observed follow up data from the ‘gold standard’ scenario. The comparison tested the plausibility of
the expected mean survival estimates of those living beyond the observation period, by calculating
the contribution to the overall mean that people surviving longer than 5 years would have to make
to observe the estimated cohort mean survival. The proportion of cohort members surviving a given
period of observation (<30 days, >30 days & <1year, >1 year & <5 years), and their contribution to
the population mean was calculated. The life expectancy required of those surviving beyond the
observation period was then calculated by dividing the difference between the estimated mean
survival and the contribution to the mean survival of the observed periods, divided by the
proportion of the cohort surviving the observation period.
4
When developing the parametric survival model, we compared six different distributions:
exponential, Weibull, Gompertz, lognormal, log-logistic, and generalised gamma. Visual comparisons
of Cox-Snell residual plots were made, where the optimal function would plot residuals on a 45° line.
Log-likelihood and Akaike Information Criteria (AIC) were also calculated, with lower values
indicating better model fit. The models included interactions of age (categorised into broadly
similarly-populated age bands: 0/49, 50/59, 60/69, 70/79, 80/89, 90/100) and sex. The predictions
were restricted to generate a maximum survival equivalent to reaching 100 years of age based on
the midpoint of each age group.
Application to a difference-in-differences programme evaluation
Having identified which method produces the most accurate life expectancy results, we consider two
ways in which this can be used in an applied programme evaluation. We consider a dichotomous
difference-in-differences design, in which outcomes are observed for treated and control units
before and after the introduction of the programme.
Our motivating example is the AQ programme (see (Meacock et al., 2014; Sutton et al., 2012) for a
full description of the policy). The mean and total years of follow-up contributed were estimated for
patients admitted to the NW and non-NW Trusts in a financial year prior to the introduction of AQ
(1st April 2007 to 31st March 2008) and a financial year following the introduction (1st April 2009 to
31st March 2010). We compare two financial years as a typical scenario for evaluation, though our
original evaluation considered 18 month periods pre and post the intervention. The four values on
mean life expectancy can be used to calculate a raw difference–in-differences estimate.
In the first approach, we estimate separate survival models for the four groups of patients described
above. This has the advantage of allowing the survival function to vary both between regions and
time periods. However, it risks generating more inaccurate estimates because it may be appropriate
to pool the data and estimate a common survival model. Moreover, it does not allow for adjustment
for variations in underlying risks across groups. Therefore, we next pooled the data from the four
groups and estimate a common survival model with the patient-level risk adjusters, a region dummy,
a time dummy and the difference-in-differences interaction term. To calculate the impact of the
introduction of AQ we predicted the mean and total life years contributed for patients admitted to
NW hospitals in the post-AQ period with the difference-in-differences term set to 0 (absence of the
policy) and 1 (presence of the policy).
Results
Observed mortality data
Figure 1 shows the Kaplan-Meier survival curve for the 121,056 patients admitted for pneumonia
between 1st April 2007 and 31st March 2008. Their survival can be followed up until 30th November
2011, with a maximum of 1,705 days (4.7 years) follow-up for those admitted on 1st April 2007. 63%
of patients had died by the end of this observation period. The cohort as a whole contributed
238,870 years of survival before 30th November 2011. The mean number of observed life years per
person in this period was 2.0 years.
5
Figure 1: Proportion of cohort still alive on each day of the follow-up period
0.00
0.25
0.50
0.75
1.00
Kaplan-Meier survival estimate
0
500
1000
analysis time
1500
2000
Table 1 shows the observed proportions of the cohort still alive at certain intervals, the mean
survival (in years) of those who died within these intervals, and their contribution to the mean
survival in years for the overall cohort. This contribution to mean survival of the overall cohort is the
product of the proportion in each interval (a) and the mean survival for each category (b). 39% of the
cohort died within the one year follow-up period. The contribution to the population mean survival
of the 63% of patients dying within the full follow-up period was just 0.454 years.
Table 1: Summary of observed data on survival times
Length of life category
Contribution to mean
years survived by
whole cohort
(a)
27%
Mean years
survived by
those dying
within interval
(b)
0.024
12%
0.256
0.031
24%
1.739
0.417
Death before 30/11/2011
63%
0.722
0.454
>30/11/2011
37%
Not known
Not known
Death within 30 days of
admission
Death after 30 days of
admission but within the
financial year
Death after the end of
the financial year but
before 30/11/2011
Proportion
of initial
cohort
(a)*(b)
0.006
6
Comparison of methods
Table 2 summarises the performance of the six functional forms for the survival model estimated on
one year of data (1st April 2007 to 31st March 2008). The generalised gamma function gave the best
fit as judged by the log-likelihood and the AIC. The superior performance of this specification was
confirmed by visual inspection of the Cox-Snell plots (Appendix 1). This functional form was
therefore adopted for the parametric survival models.
Table 2: Comparison of survival functions in the empty and full model
Statistic
Initial
AIC
Log-L
Model
AIC
Log-L
Exponential Weibull
Gompertz
Lognormal
Loglogistic
Generalised
gamma
395835
-197917
337577
-168787
341178
-170587
329890
-164942
334132
-167064
322099
-161046
369970
-184973
317580
-158777
321030
-160502
311869
-155922
314132
-157053
311027
-155450
Table 3 provides estimates of the total life years remaining for the cohort, comprising of the life
years during the observation period and the extrapolated life expectancy until they all die. The
results produced using methods 1-3 are shown, along with a comparison against the gold standard
produced with the up to 4.7 years of follow-up data now available. The first row shows the effect of
extrapolating life expectancy from a survival model using all of the now data available. The cohort
contributed 238,870 life years between the financial year 2007/8 and 30th November 2011. This is
just one quarter of the estimated total life years (961,607 years) contributed by the cohort when
total life expectancy is extrapolated from the survival model.
The estimates of life years for the cohort as a whole vary widely between methods; method 1 yields
a total of almost 1.6m years (mean 13.2 years per person), whereas method 3 estimates a much
lower total of 640,000 years (mean 5.3 years per person). The estimates vary by a factor of 2.5
between the traditional method of estimating 30 day mortality as a binary outcome and attaching
general population life expectancy estimates (method 1) and the method that uses observed survival
during the year and estimated life expectancy based on a parametric survival model (method 3). As
the mean age of the cohort at time of admission is 72 years, method 1 implies that on average the
cohort would survive until the age of 85, whilst method 3 estimates them to live to an average age
of 77.
A direct comparison of the estimates from methods 1 and 2 emphasises the value of using the
additional data available within the financial year. An additional 12% of the cohort were observed to
die after the 30 day period of usual follow-up but before the end of the financial year in question.
Taking this supplementary information into account, the estimated mean life years remaining per
person in the cohort reduces from 13.2 (method 1) to 11.6 (method 2). The use of the observed data
from the follow up period of the financial year overcomes the inaccuracies caused by failing to utilise
the period of observed survival in the estimate and reduces the impact of overestimating the life
expectancy of those surviving beyond a 30 day period but dying before the end of the financial year.
7
The wide variation between estimates remains even when the follow-up period for observing deaths
is identical and only the method of estimating remaining life expectancy varies, indicating that the
majority of the inaccuracies in the estimates are a result of assuming that the life of expectancy of
individuals is equal to that of the general population rather than failing to incorporate the time
survived within the observed period. Attaching general population life estimates to those surviving
at the end of the year in question (method 2) provides estimates of total years contributed by the
cohort which are higher than estimates produced using survival models (method 3) by a factor of
2.2. Estimates of the remaining life expectancy using method 3 are closer (-321,015 years) to the
gold standard estimates than the estimates from method 2 (+447,971 years).
Table 3: Estimated mean and total years contributed by the cohort using different methods
Method
Observed period
1st April 2007 to 30th
November 2011
1. 1-30 Days
2. 1-365 Days
3. 1-365 Days
Extrapolation method
Survival analysis
estimated on data to 30
November 2011
LE of general population
LE of general population
Survival analysis
estimated on data for 1365 days
Number (%) alive
at end of
observed followup period
Mean
life
years
Total life
years of
cohort
44,333 (37%)
7.9
961,607
88,356 (73%)
74,215 (61%)
74,215 (61%)
13.2
11.6
5.3
1,597,571
1,409,578
640,592
We can derive the implied estimated life expectancy of the individuals who survived to 30 th
November 2011 from methods 2 and 3 by reconciling the results in Table 1 and Table 3. Table 1
shows that the 63% of individuals who died before 30th November 2011 contributed 0.454 years to
the overall cohort mean number of years survived. Method 2 proposes that the overall cohort has a
mean survival time of 12.4 years. Method 2 therefore implies that the 37% who survived to 30 th
November 2011 would need to have a mean life expectancy of 32.3 years (=(12.4-0.454)/0.37).
Method 3 proposes that the overall cohort has a mean survival time of 5.3 years, and therefore
requires the 37% who survived to 30th November 2011 to have a mean life expectancy of 13.1 years
(=(5.3-0.454)/0.37). In the context of the high mortality rate of patients during the observed follow
up period and average age of the cohort at admission (72 years), the estimate from method 2
appears highly implausible and suggests that method 3 is the superior method for estimating life
expectancy.
Application to a difference-in-differences programme evaluation
19,249 (16%) of the 121,056 patients admitted for pneumonia between 1st April 2007 and 31st March
2008 were admitted to hospitals that would later participate in the AQ programme in the North
West of England (Table 4). Patients admitted to NW hospitals in the pre-AQ period were slightly
younger than those admitted to hospitals in the rest of England (mean 71.7 years versus 72.1 years,
p=0.011). However, the NW patients were likely sicker than the non-NW patients as the proportion
8
of patients alive at the end of the financial year of observation was lower in NW than non-NW
hospitals (60% versus 62%, p<0.001).
148,695 patients were admitted for pneumonia in the 2009/10 financial year (1st April 2009 and 31st
March 2010) following the introduction of AQ. 16% (23,727) of these patients were admitted to
hospitals that were participating in the AQ scheme. As in the pre-AQ period, patients admitted to
the NW hospitals were younger (mean 72.2 years versus 73.0 years, p<0.001). By the end of the
period, there was no difference in the proportion of patients still alive (63%, p=0.906), indicating a
positive effect of the programme.
The estimated mean years contributed by each of the four cohorts using method 3 are also shown in
Table 4. Compared with non-AQ hospitals, the mean years contributed by patients admitted to AQ
hospitals were lower in the pre-AQ period (6.16 years versus 6.95 years) but higher in the post-AQ
period (6.99 years versus 6.66 years). The raw difference-in-differences estimate (without riskadjustment) based on separate models for each group was an increase in mean years contributed of
1.12 years, a relative increase of approximately 18% on the baseline life expectancy estimate
(=1.12/6.16).
Table 4: Patient characteristics and estimates of mean life expectancy for each group
Pre AQ
Post AQ
Number of admitted patients
Age, mean upon admission (SD)
Alive at end of the year, n (%)
Mean years contributed
Total years contributed
Number of admitted patients
Age, mean upon admission (SD)
Alive at end of the year, n (%)
Mean years contributed
Total years contributed
NW
19,249
71.7 (17.0)
11,458 (60%)
6.16
118,550
23,727
72.2 (16.7)
14,880 (63%)
6.99
165,959
Non-NW
101,807
72.1 (17.6)
62,757 (62%)
6.95
707,950
124,968
73.0 (17.1)
78,321 (63%)
6.66
831,708
The difference-in-differences model obtained by pooling the four groups and including patient-level
risk adjusters produces a coefficient of 0.191 using one-year follow up and 0.188 using the full
follow-up. Taking the exponential of these shows an increase in relative survival by a factor of 1.21 (
9
Table 5, full analysis results in Appendix 3).
10
Table 5: Difference-in-differences estimates from common survival data models, adjusted for
patient-level risk adjusters*
AQ (FY2007/8,FY2009/10)
Full data
1 year
Coeff.
p
Coeff.
p
Age & Sex interaction
Male, 50/59
-1.565
<0.001
-1.963 <0.001
Male, 60/69
-2.338
<0.001
-3.007 <0.001
Male, 70/79
-3.095
<0.001
-3.962 <0.001
Male, 80/89
-3.869
<0.001
-4.930 <0.001
Male, 90/100
-4.501
<0.001
-5.720 <0.001
Female, 0/49
-0.032
0.582
0.123
0.022
Female, 50/59
-1.452
<0.001
-1.760 <0.001
Female, 60/69
-2.180
<0.001
-2.702 <0.001
Female, 70/79
-3.043
<0.001
-3.813 <0.001
Female, 80/89
-3.923
<0.001
-4.892 <0.001
Female, 90/100
-4.586
<0.001
-5.759 <0.001
Post-AQ period
0.118
<0.001
0.155 <0.001
AQ Trust
-0.256
<0.001
-0.287 <0.001
DiD
0.191
<0.001
0.188 <0.001
Constant
9.123
11.093
Log-sigma
1.035
<0.001
1.030 <0.001
Kappa
-0.082
<0.001
0.161 <0.001
Sigma
2.816
2.802
* Adjusted for Elixhauser morbidities, admission method, admission source, primary diagnosis
11
Table 6 shows how these coefficients convert to estimated changes in life expectancy for the cohort
of patients treated in NW hospitals following the introduction of AQ. AQ was estimated to have
increased mean life years contributed for the 23,727 patients in the AQ region in the post-AQ period
(after adjustment for patient-level risk adjusters) from 5.42 to 5.70 years, increasing the total life
years contributed by the cohort by 6459 years (5.0%). Using all available observed data on survival
until 30th November 2011 increased the expected mean life years for the cohort both in the
presence and absence of AQ, but reduced the difference in total years contributed attributable to
AQ to 5592 years (3.7%).
12
Table 6: Estimates of mean and total life years from models in Table 5
Follow-up
AQ (FY2007/8, FY2009/10)
Mean years
contributed
No AQ AQ
5.42
5.70
Total years contributed
No AQ
128,668
AQ
135,127
AQ-No AQ
6459 (+5.0%)
Financial
year
to 30
6.34
6.57
150,383 155,975 5592 (+3.7%)
November
2011
* Adjusted for Elixhauser morbidities, admission method, admission source, primary diagnosis
Discussion
It is common practice in policy evaluations to convert estimated changes in short-term mortality to
projected gains in life years using general population estimates of life expectancy. The length of life
enjoyed by patients affected by such health care programmes is, however, likely to differ from that
of the general population, leading to over-estimates of the effect of the policy in question on life
years. We developed a method to improve these estimates of life years on a one-year cohort of
patients, making use of available data on observed survival beyond the usual period of 30 days,
which has previously been ignored. We proposed a new approach to extrapolate gains in life
expectancy beyond the observation period using parametric survival models commonly employed in
clinical trials analysis. The application of this new method was then demonstrated using our previous
evaluation of the AQ P4P programme as a motivating example to illustrate how this technique could
be applied to a DiD policy evaluation in practice. Even with only short follow-up periods of one
financial year, survival analysis on patient-level administrative data improves the accuracy of
estimates of life expectancy gains considerably compared to the use of general population figures.
In a cohort of patients admitted during one financial year, 27% died within 30 days, and a further
12% were observed to have died within the financial year in question. These deaths would not have
been taken into account under the traditional method of attaching life expectancy estimates to 30
day mortality reductions. Accounting for the additional deaths occurring within the observed
financial year reduced the estimates of mean life years remaining for each member of the cohort
from 13.2 to 11.6 years. Although this signifies an improvement in accuracy as compared to the
traditional method, there were still large discrepancies between these results and those obtained
under the gold standard using up to 4.7 years of follow-up data now available. This indicated that
the majority of the inaccuracies in the estimates are a result of the method used for estimating
remaining life years beyond the period that can be observed rather than the failure to take into all of
the data available on mortality during this observed period. Extending our method further by
estimating life expectancy using parametric survival models on all of the follow-up data available
during the financial year led to a reduction in the estimate of total life years remaining for the cohort
as a whole by a factor of 2.5 as compared to those obtained using the traditional method.
Comparisons of these estimates with those achieved under the gold standard, along with the age of
the patient cohort in question, suggest that this method produces more plausible estimates of the
remaining life years of the cohort.
13
The work here represents work in progress at a relatively early stage, and we therefore acknowledge
its various limitations. To judge with certainty the accuracy of the estimates obtained through each
method would require complete follow-up data until every member of the cohort has died. The best
that we can do at this time is compare to the gold standard method presented, which uses up to 4.7
years of follow-up. An alternative approach would be to compare the mortality rates by age in the
treated cohort to those of the general population. If, during the period of observation, the agespecific mortality rates return to those of the general population, the life expectancy figures for the
general population would be appropriate to use.
In developing this work further we have a number of future plans. In particular, the application to a
difference-in-differences programme evaluation framework is still at the preliminary stage. We plan
to formally test whether it is appropriate to pool data from patients treated before and after the
implementation of the policy, or those treated in the two regions. For example, if AQ resulted in a
shift in the existing survival function, but not a change in its functional form and shape, then
estimating survival models on the pre-AQ population within a region and following them up during
the post-AQ period would allow for greater accuracy in the estimation of these survival models as
the follow-up period increases. These survival models could then be applied to patients admitted
post-AQ to extrapolate predicted survival more accurately. We will also apply the developed method
to the two remaining conditions examined in our original evaluation (acute myocardial infarction
and heart failure). We will again test the fit of the six functional forms of the survival model to
investigate whether the patterns of survival differ substantially between conditions. We will then
perform the 18 months pre- and post-AQ DiD comparison using the methods developed here and
compare the impact which this has on our original evaluation results.
14
References
Castelli, A., Dawson, D., Gravelle, H., Jacobs, R., Kind, P., Loveridge, P., Martin, S., O’Mahony, M.,
Stevens, P.A., Stokes, L., Street, A., Weale, M., 2007. A new approach to measuring health
system output and Productivity. Natl. Inst. Econ. Rev. 200, 105–117.
doi:10.1177/0027950107080395
Claxton, K., Martin, S., Soares, M., Rice, N., Spackman, E., Hinde, S., Devlin, N., Smith, P., Sculpher,
M., 2013. Methods for the Estimation of the NICE Cost Effectiveness Threshold (No. 81), CHE
Research Papers. The University of York, York.
Dawson, D., Gravelle, H., O’Mahony, M., Steet, A., Weale, M., Castelli, A., Jacobs, R., Kind, P.,
Loveridge, P., Martin, S., Stevens, P., Stokes, L., 2005. Developing New Approaches to
Measuring NHS Outputs and Activity (No. 6), CHE Research Papers. The University of York.
Meacock, R., Kristensen, S.R., Sutton, M., 2014. The Cost-Effectiveness of Using Financial Incentives
to Improve Provider Quality: A Framework and Application. Health Econ. 23, 1–13.
doi:10.1002/hec.2978
Office for National Statistics, 2011. England, Iterim Life Tables, 1980-82 to 2008-10 [Online].
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L.D., Beck, C.A.,
Feasby, T.E., Ghali, W.A., 2005. Coding algorithms for defining comorbidities in ICD-9-CM
and ICD-10 administrative data. Med. Care 43, 1130–1139.
Sutton, M., Nikolova, S., Boaden, R., Lester, H., McDonald, R., Roland, M., 2012. Reduced Mortality
with Hospital Pay for Performance in England. N. Engl. J. Med. 367, 1821–1828.
doi:10.1056/NEJMsa1114951
15
Appendix 1: Plots of the Cox-Snell residuals for the alternative survival functions
Exponential
Full follow up
0
1
2
3
4
One-year follow up
0
1
2
Cox-Snell residual
exp_H
3
4
Cox-Snell residual
Weibull
Full follow up
0
0
.5
.2
1
.4
.6
1.5
2
.8
One-year follow up
0
.2
.4
Cox-Snell residual
w_H
.6
.8
0
Cox-Snell residual
.5
1
Cox-Snell residual
weibull_H
1.5
2
Cox-Snell residual
lognormal
Full follow up
0
0
.2
.5
.4
1
.6
.8
1.5
One-year follow up
0
.2
.4
Cox-Snell residual
ln_H
One-year follow up
.6
.8
0
.5
1
lnormal_H
Cox-Snell residual
1.5
Cox-Snell residual
Cox-Snell residual
log-logistic
Full follow up
16
1.5
.8
1
.6
.5
.4
0
.2
0
0
.2
.4
Cox-Snell residual
ll_H
.6
0
.8
.5
1
1.5
Cox-Snell residual
loglogistic_H
Cox-Snell residual
Cox-Snell residual
Gompertz
Full follow up
0
0
.5
.2
1
.4
1.5
.6
One-year follow up
0
.2
.4
.6
0
Cox-Snell residual
g_H
.5
1
1.5
Cox-Snell residual
Cox-Snell residual
gompertz_H
Cox-Snell residual
Generalised gamma
Full follow up
0
0
.2
.5
.4
1
.6
1.5
One-year follow up
0
.2
.4
Cox-Snell residual
gamma_H
Cox-Snell residual
.6
0
.5
1
1.5
Cox-Snell residual
gamma_H
Cox-Snell residual
17
Appendix 2: Unadjusted difference-in-differences estimates from pooled survival data models
1 year
Coef.
Age & Sex interaction
Male, 50/59
Male, 60/69
Male, 70/79
Male, 80/89
Male, 90/100
Female, 0/49
Female, 50/59
Female, 60/69
Female, 70/79
Female, 80/89
Female, 90/100
Post-AQ period
AQ Trust
DiD
Constant
Log-sigma
Kappa
Sigma
-1.792
-2.563
-3.266
-4.026
-4.666
0.004
-1.641
-2.249
-3.064
-3.966
-4.643
0.164
-0.175
0.114
8.045
1.218
-0.677
3.381
Full data
Coef.
p
p
0.000
0.000
0.000
0.000
0.000
0.932
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.003
0.000
0.000
0.000
-2.357
-3.516
-4.529
-5.559
-6.403
0.174
-2.075
-3.031
-4.166
-5.330
-6.252
0.172
-0.256
0.136
10.247
1.154
0.027
3.170
0.000
0.000
0.000
0.000
0.000
0.001
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.005
18
Appendix 3: Difference-in-differences estimates from pooled survival models, adjusted for patientlevel risk adjusters – full analysis results
1 year
Coef.
Full data
Coef.
p
p
Age & Sex interaction
Male, 50/59
Male, 60/69
Male, 70/79
Male, 80/89
Male, 90/100
Female, 0/49
Female, 50/59
Female, 60/69
Female, 70/79
Female, 80/89
Female, 90/100
-1.565
-2.338
-3.095
-3.869
-4.501
-0.032
-1.452
-2.180
-3.043
-3.923
-4.586
0.118
-0.256
0.191
0.000
0.000
0.000
0.000
0.000
0.582
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
-1.963
-3.007
-3.962
-4.930
-5.720
0.123
-1.760
-2.702
-3.813
-4.892
-5.759
0.155
-0.287
0.188
0.000
0.000
0.000
0.000
0.000
0.022
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
A401
A402
A403
A408
A409
A410
A411
A412
A413
A414
A415
A418
A419
J13X
J14X
J150
J151
J152
J153
J154
J155
J156
J157
J158
0.601
-1.376
0.051
0.352
-0.889
-0.377
-0.446
-0.057
-0.829
1.394
0.129
-0.427
-1.606
1.578
1.900
0.928
0.687
0.527
1.507
1.532
1.082
1.084
2.004
0.657
0.476
0.218
0.926
0.566
0.159
0.451
0.431
0.929
0.486
0.188
0.792
0.425
0.001
0.001
0.000
0.058
0.155
0.277
0.022
0.002
0.036
0.038
0.000
0.197
0.120
-2.370
-0.509
-0.249
-1.315
-1.237
-1.231
-0.724
-1.367
0.329
-0.583
-0.993
-2.371
0.883
0.894
0.035
-0.391
-0.379
0.877
0.788
0.024
0.031
1.513
-0.110
0.885
0.035
0.355
0.683
0.037
0.014
0.029
0.249
0.246
0.741
0.237
0.064
0.000
0.066
0.067
0.944
0.420
0.437
0.170
0.109
0.962
0.953
0.002
0.828
Post-AQ period
AQ Trust
DiD
Primary diagnosis
19
J159
J168
J180
J181
J182
J188
J189
J960
Admission method
(Ref: Elective)
Emergency via A&E
Transfer from other hospital
Admission source
(Ref: Usual residence)
NHS other hospital provider
LA residential accommodation
Elixhauser
Congestive heart failure
Cardiac arrhythmia
Valvular disease
Pulmonary circulation disorders
Peripheral vascular disease
Hypertension uncomplicated
Hypertension complicated
Paralysis
Other neurologic disorders
Chronic pulmonary disease
Diabetes uncomplicated
Diabetes complicated
Hypothyroidism
Renal failure
Liver disease
Peptic ulcer disease excl. bleeding
AIDS/HIV
Lymphoma
Metastatic cancer
Solid tumour without metastasis
Rheumatoid arthritis
Coagulopathy
Obesity
Weight loss
Fluid and electrolyte disorders
Blood loss anaemia
Deficiency anaemia
Alcohol abuse
1.733
-0.684
-1.410
1.122
0.113
0.958
0.559
-0.725
0.001
0.199
0.003
0.018
0.834
0.048
0.238
0.137
0.854
-1.317
-2.146
0.364
-0.807
0.197
-0.122
-1.343
0.088
0.013
0.000
0.447
0.134
0.686
0.799
0.006
-0.787
-0.923
0.000
0.000
-0.529
-0.825
0.000
0.000
0.018
-0.876
0.538
0.000
0.060
-1.016
0.030
0.000
-0.587
-0.204
0.063
-0.518
-0.408
0.331
0.322
-0.634
-0.704
0.155
0.066
-0.231
0.161
-0.656
-1.432
-0.076
0.471
-0.912
-1.739
-1.429
-0.034
-0.733
-0.205
-0.833
-0.919
-0.127
0.187
-0.608
0.000
0.000
0.079
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.469
0.152
0.000
0.000
0.000
0.344
0.000
0.002
0.000
0.000
0.670
0.000
0.000
-0.669
-0.250
0.004
-0.604
-0.518
0.389
0.360
-0.791
-0.923
-0.031
0.006
-0.489
0.113
-0.764
-1.549
-0.140
0.634
-1.288
-2.088
-1.718
-0.132
-0.756
-0.147
-0.939
-0.920
-0.125
0.047
-0.720
0.000
0.000
0.913
0.000
0.000
0.000
0.000
0.000
0.000
0.022
0.735
0.000
0.000
0.000
0.000
0.155
0.028
0.000
0.000
0.000
0.000
0.000
0.019
0.000
0.000
0.659
0.189
0.000
20
Drug abuse
Psychoses
Depression
Constant
Log-sigma
Kappa
Sigma
0.436
-0.251
0.089
9.123
1.035
-0.082
2.816
0.000
0.000
0.033
0.000
0.000
0.000
0.106
-0.399
-0.020
11.093
1.030
0.161
2.802
0.291
0.000
0.602
0.000
0.000
0.000
21