Assessment of the benefits of Discrete
Conditional Survival Models in modelling
ambulance response times
Karen J. Cairns, Adele H. Marshall
Centre for Statistical Science and Operational Research (CenSSOR),
Queen’s University Belfast.
Keywords: Time-to-event data; Distribution-fitting; Data-mining; Ambulance
response.
Acknowledgements: KJC is supported through an Engineering & Physical
Sciences Research Council (EPSRC) RCUK Academic Fellowship.
Corresponding Author:
Dr Karen Joanne Cairns
Centre for Statistical Science and Operational Research (CenSSOR)
Sir David Bates Building, Room 01.008
Queen’s University Belfast
University Road
Belfast BT7 1NN
Email: k.cairns@qub.ac.uk
Telephone: +44 (0)28 9097 6058
Fax: +44 (0)28 9097 6061
Assessment of the benefits of Discrete Conditional Survival
Models in modelling ambulance response times
Abstract: Many of the challenges faced in health care delivery can be informed
through building models. In particular, Discrete Conditional Survival (DCS)
models, recently under development, can provide policymakers with a flexible
tool to assess time-to-event data. The DCS model is capable of modelling the
survival curve based on various underlying distribution types and is capable of
clustering or grouping observations (based on other covariate information)
external to the distribution fits. The flexibility of the model comes through the
choice of data mining techniques that are available in ascertaining the different
subsets and also in the choice of distribution types available in modelling these
informed subsets. This paper presents an illustrated example of the Discrete
Conditional Survival model being deployed to represent ambulance responsetimes by a fully parameterised model. This model is contrasted against use of a
parametric accelerated failure-time model, illustrating the strength and usefulness
of Discrete Conditional Survival models.
1
Introduction
Policy makers and health care providers must determine how to provide the most
effective health care to citizens using the limited resources available to them.
They need effective methods for planning, prioritisation, and decision making, as
1
well as effective methods for management and improvement of health care
systems (Brandeau et al 2004). Operational Research (OR) techniques can inform
these processes with a wide range of health OR illustrated in the literature (Davies
and Bensley 2005, Brailsford and Harper 2007, Baker et al 2008, Royston 2009).
Methodologies considered vary with the problems being addressed and range
from ‘soft’ OR techniques to more quantitative approaches such as mathematical
modelling, simulation, queuing theory and system dynamics.
Emergency response issues are amongst the problems being addressed, with much
of the earlier research focusing on location planning issues particularly in urban
areas (Simpson and Hancock 2009). Part of this research area, originating in the
research of Kolesar and Blum (1973), has examined the relationship between
emergency response times and distance (the ‘square-root law’). The derivation of
such relationships has informed and aided understanding, and has formed the
cornerstone of further analytic models developed (Green and Kolesar 2004,
Budge et al 2010). Indeed, Erkut et al (2008) indicates that it is generally more
useful to know the entire response time distribution, rather than considering just
specific quantiles of it – something which many performance measures typically
correspond to (e.g. the percentage of urgent emergency incidents reached within 8
minutes is a performance measure used within the National Health Service in the
United Kingdom (Department of Health 2009)).
This paper presents the family of Discrete Conditional Survival (DCS) models
recently under development. This toolkit of models aims to aid understanding of
2
time-to-event data, by modelling the entire time-to-event distribution through a
fully parameterised model, and should be flexible to modelling many forms of
such data within health care. In particular this paper presents the application of the
DCS model to a particular emergency response example – modelling ambulance
response times.
The paper first presents some background information on the data being analysed.
An outline of Discrete Conditional Survival models is then presented, together
with information on the choice of techniques deployed from the toolkit for this
particular example. In order to assess the usefulness of this model, a comparison
has then been made with results from one of the most common regression-based
techniques used to fully parameterise survival data, namely the parametric
accelerated failure-time model (Kalbfleisch and Prentice 1980).
2
Data Set of Ambulance Response Times
Ambulance response time data has been examined from a region of the United
Kingdom (Northern Ireland), where the Northern Ireland Ambulance Service
(NIAS) is responsible for providing emergency medical response. NIAS currently
responds to over 115,000 emergency calls in a year, through a fleet of over 300
ambulances, operating from 52 ambulance stations and sub-stations. It serves a
population of over 1.7 million, with an operational area of 14 000 square
kilometres (Northern Ireland Ambulance Service, 2009a).
3
The data set considered contains dispatch event details (for example, the date and
time of an event, its geographical position, and the perceived severity
categorisation of the emergency call) together with response time information
(e.g. response time(s), type of emergency vehicle(s) responding) for all
emergency calls in Northern Ireland in the year 2003.
The illustrated example considered in this paper considers the sub-set of all
emergency response activations where an ambulance response to the incident was
achieved (i.e. excludes cancellations, where the ambulance never reached the
scene) and considers only the best response time (in the case where multiple
vehicles are dispatched). In 2003, there were 75,774 such responses to emergency
incidents across the entire Northern Ireland region. Removal of records where
either the response time has not been collected, or geographical location
information is incomplete reduces the number of observations to 73,190 (96.6%).
For the purposes of assessing how well the DCS and parametric accelerated
failure-time models perform, the data was separated into training (50%) and test
sets. It was ensured that training and test sets contained observations across the
Northern Ireland region, at locations both close and far away from ambulance
stations.
3
Discrete Conditional Survival (DCS) Model
Discrete Conditional Survival (DCS) models are a family of models capable of
representing a skewed survival distribution as a Process Component preceded by a
4
set of related variables that determine the clustering or grouping of entities (or
observations) into distinct classes (the discrete classes), that may be referred to as
the Conditional Component. The models possess the following characteristics:

The Conditional Component comprises a structure that captures the nature of
the data by representing the various inter-relationships between variables, and
thus can categorise observations into a number of discrete classes.

The Process Component represents the skewed survival distribution of each
discrete class by an appropriate distribution form.
Figure 1 illustrates the general form of the DCS model comprising these two
components. This figure illustrates that many kinds of data-mining techniques
could represent the Conditional Component, with the illustrated example in this
paper utilising multinomial logistic regression. The figure also highlights that a
number of survival distribution forms can be considered for the Process
Component, with the DCS model incorporating the assessment of the most
appropriate fit.
This model expands previous research which had led to the development of the
Conditional Phase-type (C-Ph) model, which describes duration until an event
occurs in terms of a process consisting of a sequence of latent phases (the Process
Component) which are conditioned on a set of inter-related variables represented
by a Bayesian network (the Conditional Component) (Marshall and McClean
2003). Previous research fitted the C-Ph model, a special type of DCS model, by
considering the model structure as having one entity for which the likelihood
5
value was calculated. This led to very cumbersome and difficult calculations for
the likelihood value of every possible Conditional Component structure along
with every possible survival distribution fit. To ease this process, the flexible
nature of the DCS model allows for the two components in the model to be fitted
separately combining the result in an overall likelihood. To do this, requires the
separate inspection of each combination of component variables and how they
relate to survival. As a result this will ultimately reduce the complexity in model
fitting.
3.1
Conditional Component
The Conditional Component of the DCS model categorises observations into a
number of discrete classes, with the aim that the survival of entities in each
discrete class differ (and so the resulting survival distributions of the discrete
classes will be distinguishable). To achieve this, various data-mining techniques
(see Figure 1) can be used to consider the influence of covariates on survival or,
has in previous research, on a correlated intermediate variable (Marshall and
Burns 2007).
3.1.1 Multinomial Logistic Regression
In this illustrated example multinomial logistic regression is used with the aim to
accurately predict the most probable response time-band for each emergency
incident (through the consideration of the influence of other covariates). In
multinomial logistic regression, a special case of the discrete choice model
6
introduced by McFadden (1974), the probability a response, Y, belongs to the ith
of k+1 classes satisfies the following relationship:
 Pr(Y  i | x) 
'
log 
  i  β i x
 Pr(Y  k  1| x) 
i  1,
,k
(1)
where Y is the discrete response of an entity (or observation) taking one of k+1
possible values (discrete classes), x is the vector of explanatory variables for the
entity, 1 ,
,  k are the k intercept parameters, and β '1 ,
, β ' k are k vectors of
parameters.
The fitting of multinomial logistic regression models is possible in a number of
software packages. This work was performed in SAS (version 9.2), using the
PROC LOGISTIC procedure.
3.1.2 Application to Ambulance Response Time Data
To fit such a model the continuous ambulance response time variable had to be
converted into a discrete response, Y. The discrete response, Y, considered was
directed by the target and performance measures of NIAS. Over the last number
of years performance has been monitored and targets set by considering the
proportion of incidents responded to within 8 minutes and again within 18
minutes (Northern Ireland Ambulance Service, 2004 and Northern Ireland
Ambulance Service, 2009b). However, rather than limit the response, Y, to just
three response time-bands: [0, 8); [8, 18); and [18, ], the following five response
time-bands: [0, 5.5); [5.5, 8); [8, 11.5); [11.5, 18); and [18, ∞) were considered.
7
The reason for further subdividing was to enhance the quality of the resulting
multinomial logistic regression model (bearing in mind the large volume of data
available).
Within the ambulance response time data set there are a number of covariates
available that could be incorporated into the vector of explanatory variables x .
These covariates may provide information on either the geographical location of
an incident, temporal information on when an incident occurred, or information
relating to the response deployment. The functional form utilised for these
covariates within the model may improve the goodness of fit. Table 1 provides
details of the different covariates that have been considered for inclusion in
different multinomial logistic regression model fits. Notice in the case of some
pairs of the categorical covariates listed (e.g. u and , h and g), the covariates
actually correspond to different levels of sub-grouping of categorical variables,
and thus only one is potentially selected in any given set of explanatory variables
x . Similarly, for highly correlated variables (e.g. the geographical location
information r2 and s2) only one is potentially selected in any given set of
explanatory variables x .
Over 100 different sets of covariates have been
considered for inclusion in different multinomial logistic regression model fits.
All of these sets included either r1 or s1, a measure of the distance between the
incident and the closest ambulance station, given its strong influence on response
time (Kolesar and Blum 1973). Some of these sets also considered the effects of
interaction between different covariates e.g. the interaction between u and r1. Fits
8
were also performed based on utilising backward elimination and forward
selection techniques.
The optimal choice of model (and the explanatory variables to be retained) was
selected using Schwarz’s Bayesian Criterion (SBC) (Schwarz 1978), calculated as
follows:
SBC  model with set of explanatory variables i   pi ln(n)  2ln( Li )
(2)
where pi is the number of parameters to be estimated for the model with a set of
explanatory variables i, n is the number of observations, and Li is the maximised
value of the likelihood function (in this case based on a generalised logit model
with a set of explanatory variables i). The model with the set of explanatory
variables corresponding to the lowest SBC value was selected. The SBC tends to
penalise overly complex models (more than the Akaike’s Information Criterion
(AIC) (Akaike 1974)) and is useful for finding the simplest model that still
represents the data accordingly.
The multinomial logistic regression model has been fitted to the training data from
each of the 26 Local Government Districts (LGDs), with optimal model fits
determined in each case using SBC. Examination of these optimal models
suggests that the explanatory variables (influencing the prediction of response
time-band) vary across the 26 LGDs. For 4 of the 26 LGDs only the radial
distance r1 is suggested by the optimal model to influence the prediction of the
9
response time-band. In other LGDs however a number of covariates are found to
influence the prediction of the response time-band.
Ards Local Government District
For example, in the case of Ards LGD (LGD=2), in the east of Northern Ireland,
the optimal model is found include interaction terms between the radial distance r1
and the urban/rural indicator variable, u, such that:
1i  3 i

 
 r2  1 4i
 Pr(Y  i | x)  e i 2 i  r1  1




 Pr(Y  k  1| x)   ei  r1  1 1i  r2  1 4 i
if urban
i  1,
,k
(3)
if rural
This model suggests that within this LGD changes in response time-band
predicted may occur at comparably shorter radial distances in urban areas in
comparison to that of rural areas. The model also suggests the predicted response
time-band is also influenced by proximity to the second closest ambulance station.
A few illustrative situations for the model of this LGD are considered in Figure 2.
For example, consider an incident occurring in an urban region, where there are
two ambulance stations relatively close (r1≤r2=2.4 miles). In this case the
response is likely to be quick, with the predicted response time-band being either
band 1 or 2 (i.e. predicted below the 8 minutes target), except for values of
r1>2.14 miles where the response time-band changes to band 4 (i.e. predicted
below the 18 minutes target). The second sub-plot of the figure illustrates the
prediction for an incident occurring in an urban region where the second closest
ambulance station would be considered at a large distance away (r2=7.4 miles).
10
Here the predicted response is below the 8 minutes target provided r1<1.5 miles
(not as large as when there are two ambulance stations relatively close), otherwise
it is predicted below the 18 minutes target.
In contrast, consider an incident occurring in a rural region of Ards LGD, where
there are two ambulance stations relatively close (r1≤r2=3.8 miles). The predicted
response is below the 8 minutes target for r1<2.45 miles, and below the 18
minutes target otherwise. Also if an incident occurs in a rural region with the
second closest ambulance station being relatively far away (r2=15.5 miles), the
response is likely to be very short (band 1) if the closest ambulance station is
nearby (r1<3.42 miles), or very long (band 5) otherwise.
Such interpretations of the models predicted by the Conditional Component can
aid decision-makers in understanding and identifying combinations of covariate
values that often lead to longer response times. Indeed, plotting such information
on colour-coded choropleth maps may also be beneficial.
Magherafelt Local Government District
As an additional example, results are also summarised for the Local Government
District of Magherafelt (LGD=20). In this case the optimal multinomial logistic
regression model fit was also found to include interaction terms between the radial
distance r1 and the urban/rural indicator variable, u, with relative probabilities
satisfying the following:
11
1i  3 i
 
 Pr(Y  i | x)  e i 2 i  r1  1



 Pr(Y  k  1| x)   ei  r1  1 1i
if urban
i  1,
,k
(4)
if rural
Figure 3 illustrates how the most probable response time-band varies with r1 in
urban/rural regions. The model suggests that in the urban regions the response is
likely to be below the 8 minutes target when r1<1.46 miles, otherwise it is likely
to be well below the 18 minutes target. In the rural regions the response is likely
to be below the 8 minutes target when r1<1.9 miles, otherwise it is likely to be
below the 18 minutes target. No responses were predicted within response timeband 5.
3.1.3 The Discrete Classes
Using the model to predict the response time-band of each observation in the
training set, it is then possible to visualise the survival distribution of observations
from each of the discrete classes. Figure 4 illustrates the distribution of actual
response times in the five discrete classes (or response time-bands) for Ards LGD.
Each of these exhibit a skewed survival distribution, with the peak in the
probability density function clearly shifting to larger response times as one moves
through the 5 discrete classes. Notice also the range of the distributions appear to
spread as one moves through the 5 discrete classes – that is, the Conditional
Component has been much less successful at classifying observations as the
response time-band increases.
12
A similar trend is found on examining the response times from the four discrete
classes in the case of Magherafelt LGD (see Figure 5).
3.2
Process Component
Once the training data had been separated into the discrete classes by the
Conditional Component of the DCS model, the Process Component of the DCS
model requires the skewed survival distribution of each discrete class to be
represented by an appropriate distribution form.
3.2.1 Survival Distributions
A number of forms of survival distribution can be considered for the Process
component. In the case of the illustrated example in this paper, these include the
log-logistic distribution; a two-term log-logistic distribution; a 2-phase Coxian
phase-type distribution (Cox 1955), and the log-normal distribution. In particular,
the probability density function for the log-logistic distribution has the following
form:
f ( x) 
 x 1
  1    x 
2
(5)
,
where  ,   >0 are the parameters of the distribution. The two-term log-logistic
distribution has the following form:
f ( x)  p
1 x 1
1
1 1  1 x
1
1
1

2
 (1  p)
 2 x
2 1
 2 1   2  x
2
13
2
2

2
(6)
where 1 ,  2 , 1 , 2  >0 and 0  p  1 are the parameters of the distribution.
Parameter estimates relevant to each distributional form were estimated via
Maximum Likelihood Estimation (MLE). Schwarz’s Bayesian Criterion (SBC)
(Schwarz 1978) was again utilised to measure the goodness of fit of each
distribution form and determine the optimal form. This aspect of the work was
carried out in Matlab (version 7.8.0.347), using the FMINSEARCH procedure.
3.2.2 Application to Ambulance Response Time Data
Figures 4 and 5 illustrate the various distribution fits to each of the discrete classes
or (response time-bands) considered in the case of Ards and Magherafelt LGD.
Using SBC, the optimal distribution forms are found to be either the log-logistic
or the two-term log-logistic. In the case of Ards LGD, response time-bands 1, 2, 4
and 5 were found to be best represented by two-term log-logistic distributions,
while time-band 3 is best represented by a log-logistic distribution. In the case of
Magherafelt LGD, response time-bands 1 and 3 were found to be best represented
by two-term log-logistic distributions, while time-bands 2 and 4 are best
represented by a log-logistic distribution.
3.3
Simulating Data from the DCS Model
The DCS model represents ambulance response times, where the Conditional
Component consists of a multinomial logistic regression model, which categorises
the responses as belonging to one of 5 response time-bands, based on covariate
information. The Conditional Component acts as a filter variable that can be used
14
to create 5 different streams of ambulance response time distribution, which are
then fed into the second component of the DCS model. In the Process Component
of the DCS model, the distributions of the response times in each discrete class (or
response time-band) are represented by either a log-logistic or a two-term mixed
log-logistic distributional form.
Simulated data, corresponding to observations in the test set, can be generated
with the DCS model. For each observation in the test set, the multinomial logistic
regression model is first applied to determine the most probable response timeband (based on the covariate information relating to the observation). Then,
depending on the response time-band predicted, data can then be simulated from
the appropriate log-logistic/two-term log-logistic distribution.
Figure 6 illustrates how data simulated from the DCS model compares to the
observed response times in the test set, for observations in Ards and Magherafelt
LGD. Here cumulative response time distributions have been determined for both
observed and simulated data, by first separating observations in the test set
according to how close the nearest ambulance station is - observations in each
LGD were first divided into four groups, depending on the r1 value, then the
cumulative response time distributions were determined.
In each case, the cumulative response time distribution obtained from the DCS
model simulated data, agrees reasonably well with that observed in the test set
(see also Appendix 1 for results for all 26 LGDs). Notice however that the
simulated data demonstrates that there may be some uncertainty in the position of
15
the cumulative response time distribution determined by the DCS model,
particularly at larger response times. For example, consider observations from
Magherafelt LGD with r1 values in the first quartile. In this case there is little
spread in cumulative response time distributions determined across the
simulations for response times below around 6 minutes. Beyond this time
however, the cumulative response time distribution determined for each
simulation shows more variability in its position, and so this is reflected in a
spread in the 95% confidence interval for the cumulative response time
distribution (beyond t=6 minutes).
4
Assessing the DCS Model
The DCS model is a recently developed approach that attempts to utilise the
benefits of previous modelling conventions to aid understanding of time-to-event
data. Most importantly, the DCS model models the entire time-to-event
distribution through a fully parameterised model.
In order to assess the usefulness of the DCS model, a comparison has been made
with results from one of the most common regression-based techniques used to
fully parameterise such data, namely the parametric accelerated failure-time
model (Kalbfleisch and Prentice 1980).
16
4.1
Modelling ambulance response with the parametric accelerated
failure-time model
A parametric accelerated failure-time model has previously been utilised to model
ambulance response times in a sub-region of Northern Ireland (Cairns et al 2010).
This model fully parameterises the ambulance response-time distribution and
accounts for the effect of multiple covariates on the response.
In the parametric accelerated failure-time model the response time T satisfies:


T  exp xcβc T0
(7)
where x c is the vector of covariates (not including the intercept term), β c is a
vector of unknown parameters, and T0 is a response time sampled from the
baseline distribution. Numerous baseline distributions can be considered (e.g.
exponential, generalized gamma, log-normal, Weibull, log-logistic), depending on
the software package used, though are usually limited to standard distributions. In
this work, fits to the parametric accelerated failure-time model were performed in
SAS (version 9.2), using the PROC LIFEREG procedure.
In order to be able to compare the results from the parametric accelerated failuretime model with that from the DCS model, this work has involved using the same
training set of data (as was used in determining the DCS model) to fit the
parametric accelerated failure-time model. As in the DCS case, separate
parametric accelerated failure-time models were produced for each of the 26
LGDs. In each case, the same sets of covariates (see Table 1 for list of covariates)
17
were considered as those used when generating potential model fits for the DCS
model - resulting in over 100 potential models fits for each LGD. The optimal
choice of parametric accelerated failure-time model in each LGD was then
selected, again based on using SBC. Note that in each model fit the baseline
distribution was chosen to be log-logistic. This choice was based on previous
analysis on sub-regions of Northern Ireland (Marshall et al 2006) and evidence
from some test calculations run as part of this work – which suggest this form of
baseline distribution consistently provides better model fits compared to those fits
obtained when other standard distributions are used (e.g. exponential, Weibull
etc.).
In the case of Magherafelt LGD, the optimal parametric accelerated failure-time
model was found to depend on the radial distance r1, such that:
T  (r1  1)0.6028 T0
(8)
In terms of the dependence on radial distance r1, the findings of this optimal
model are in agreement with Kolesar et al (1975) – being between the square root
relation they found for short trips and the linear relation they found for long trips.
Note however more covariates were found to influence response within the
Conditional Component of the DCS model for this LGD (see Equation 4).
In the case of Ards LGD, the optimal parametric accelerated failure-time model
was found to depend on a number of covariates, such that:
18
T  (r1  1)0.6812 T0
if in  group 1
 1
1.151 if in  group 2
1.108 if hour = 0-7

1.030 if in  group 3 0.946 if hour = 8-9


1.666
if
in

group
4

0.975 if hour = 10-13
1.666 if in  group 5 1
if hour = 14-23

1.034 if in  group 6
1.081 if unclassified
1.095 if rural

 1.037 if not high priority  
if urban
 1
1
if high priority

(9)
Thus the optimal parametric accelerated failure-time model in this LGD (Equation
9) would suggest more covariates influence response (e.g. incidents occurring in
the early morning (hours 0-7) are likely to take longer, as too unclassified/low
priority calls) , than was found via the Conditional Component of the DCS model
(see Equation 3). However, it is important to note that this optimal parametric
accelerated failure-time model was based on the underlying assumption of a loglogistic baseline distribution.
Such an assumption may be inappropriate,
particularly given the influence of performance targets on response, and so could
have resulted in additional covariates appearing influential. Also, as will be
demonstrated in Section 4.2, one also needs to consider how well each model
captures the entire time-to-event distribution and assess whether simulated data
from this model would be comparable to that observed in unseen test data.
19
4.2
Comparing with the parametric accelerated failure-time model
In order to compare the DCS model and the parametric accelerated failure-time
model, simulated response times were also generated using the parametric
accelerated failure-time model. The simulated data was based on using the
covariate information for each of the observations in the test set. For each
observation, data was first simulated from the appropriate baseline log-logistic
distribution (depending on LGD) and then scaled according to its covariate
information (e.g. using Equations 8/9).
Figure 6 illustrates how data simulated from the parametric accelerated failuretime model compares to the observed response times in the test set, for
observations in Ards and Magherafelt LGD. In each LGD the parametric
accelerated failure-time model appears to fail in capturing the cumulative
response time distribution for observations from the lowest quartile of r1 values.
Therefore while the results from the DCS model display more uncertainty (seen in
their large 95% confidence interval), the overall placement of its cumulative
response time distribution appears more appropriate than that obtained through the
parametric accelerated failure-time model.
5
Conclusions
This paper considers the Discrete Conditional Survival model used to model
ambulance response times, and can potentially identify emergency incidents at
risk of having long response times.
As well as identifying risk, this fully
20
parameterised model enables simulation-based techniques to be deployed to
compare ambulance response times to the response of other organisations (such as
the First Responders in Cairns et al (2010)).
Use of such methods can aid
policymakers in their decision making process.
Whilst other models exist to produce fully parameterised models of time-to-event
data, this toolkit approach provides the user with a choice of unlimited datamining techniques to be used for the Conditional Component of the model, and
the choice of unlimited distribution forms to represent the skewed time-to-event
data within each discrete class (the Process Component).
The Conditional Component of the DCS model aims to aid understanding of what
influences the time-to-event through inclusion of appropriate covariates. Any
inadequacies in the Conditional Component of the model will be reflected in the
spread of the time-to-event data in each of the discrete classes. Even if
inadequacies exist however, the DCS model is still able to fully parameterise the
data due to the Process Component reflecting any inadequacies.
In the illustrative example of this work, results from the DCS model were
compared to results from the parametric accelerated failure-time model.
Differences were found between these models in what covariates were found to
influence ambulance response times in the various LGDs. These differences in
influential covariates could be attributed to the potentially inappropriate
assumption within the parametric accelerated failure-time model of an underlying
log-logistic distribution.
21
Within the illustrative example, a comparison was also made between observed
data (on which models were not trained) and data simulated from the DCS and the
parametric accelerated failure-time models. These results suggest that the DCS
model is a tool capable of modelling data on which it has not been trained, while
the parametric accelerated failure-time model may not – particularly in situations
where response is likely to be influenced by performance targets.
In general, the DCS toolkit of models aims to aid understanding of time-to-event
data and should be flexible to modelling many forms of such data within health
care. In particular it is likely to be very useful in modelling health care data with a
non-standard underlying distribution (such the Coxian phase-type distribution –
found as the most appropriate distribution for total wait times in Accident and
Emergency in the case of admitted patients (Burns 2007)). Such health care data
could not be readily modelled by the likes of the parametric accelerated failuretime model using most standard software packages – the option of such a baseline
distribution would be unavailable. The DCS toolkit however is flexible to
modelling such data.
22
References
Akaike, H. (1974), A new look at the statistical model identification, IEEE
Transactions on Automatic Control, 19 (6): 716–723.
Baker, R.D., Chaussalet, T.J. and Utley, M. (eds.) (2008), Special issue on IMA
conference, Health Care Mngt Sci, 11: 87-213.
Brailsford, S. and Harper, P. (eds.) (2007), Special issue: Operational research in
health, J Opl Res Soc, 58: 147-270.
Brandeau, M.L., Sainfort, F. and Pierskalla, W.P. (2004), Operations research and
health care: a handbook of methods and applications, Kluwer Academic
Publishers, Massachusetts.
Budge, S., Ingolfsson, A. and Zerom, D. (2010), Empirical Analysis of
Ambulance Travel Times: The Case of Calgary Emergency Medical
Services, Mngt Sci, 56 (4): 716-723.
Burns, M.L. (2007), MPhil Thesis: Modelling Waiting Times for Accident and
Emergency Patients, Queen’s University Belfast, Belfast.
Cairns, K.J., Marshall, A.H. and Kee, F. (2010), Using Simulation to assess a
Novel Public Health Scheme with Stochastic and Spatial Complexities,
accepted by J Opl Res Soc, doi:10.1057/jors.2010.27.
Cox, D. (1955), A use of complex probabilities in the theory of stochastic
processes, Proceedings of the Camb Phil Soc, 51: 313-319.
23
Davies, R. and Bensley, D. (eds.) (2005), Special Issue: Meeting health challenges
with OR, J Opl Res Soc, 56: 123-233.
Department of Health (2009), NHS Performance Framework: Implementation
guidance,
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsP
olicyAndGuidance/DH_098525, accessed 31/07/2010.
Erkut, E., Ingolfsson, A. and Erdogan, G. (2008), Ambulance deployment for
maximum survival, Naval Res Logist, 55 (1): 42-58.
Green, L.V. and Kolesar, P.J. (2004), Improving Emergency Responsiveness with
Management Science, Mngt Sci, 50 (8): 1001-1014.
Kalbfleisch, J.D. and Prentice, R.L. (1980), The Statistical Analysis of Failure
Time Data, John Wiley & Sons, New York, NY.
Kolesar, P. and Blum, E.H. (1973), Square root laws for fire engine response
distances, Mngt Sci, 19 (12): 1368-1378.
Kolesar, P., Walker, W. and Hausner, J. (1975), Determining the relation between
fire engine travel times and travel distances in New York City, Oper Res, 23
(4): 614-627.
Marshall, A.H. and Burns, M.L. (2007), A Bayesian Network Hybrid Model for
Representing Accident and Emergency Waiting Times, Proceedings of the
20th IEEE Symposium on Computer Based Medical Systems (CBMS),
Maribor, Slovenia: 91-96.
24
Marshall, A.H., Cairns, K.J., Kee, F., Moore, M.J., Hamilton, A.J. and Adgey,
A.A.J. (2006), A Monte Carlo Simulation Model to Assess Volunteer
Response Times in a Public Access Defibrillation Scheme in Northern
Ireland, Proceedings of the 19th IEEE International Symposium on
Computer-Based Medical Systems (CBMS), Salt Lake City: 783 - 788.
Marshall, A.H. and McClean, S.I. (2003), Conditional Phase-Type Distributions
for Modelling Patient Length of Stay in Hospital, International Transactions
in Operational Research, 10 (6): 565-576.
McFadden, D. (1974), Conditional Logit Analysis of Qualitative Choice
Behaviour. In: Zarembka, P. (ed.) Frontiers in Econometrics, Academic
Press, New York.
Northern Ireland Ambulance Service, (2004), Northern Ireland Ambulance
Service Annual Report 2003/04,
http://www.niamb.co.uk/docs/documents/annual_reports/annual_report_03_0
4.pdf, accessed 31 July 2010.
Northern Ireland Ambulance Service, (2009a), Northern Ireland Ambulance
Service Annual Report 2008/09,
http://www.niamb.co.uk/docs/documents/08833%20NIAS%20Annual%20Re
port.pdf, accessed 31 July 2010.
Northern Ireland Ambulance Service, (2009b), Emergency Services and Support,
http://www.niamb.co.uk/docs/our_services_index.html, accessed 31 July
2010.
25
Royston, G. (2009), One hundred years of Operational Research in Health – UK
1948-2048, J Opl Res Soc, 60: S169-S179.
Schwarz, G.E. (1978), Estimating the dimension of a model, Annals of Statistics,
6 (2): 461-464.
Simpson, N.C. and Hancock, P.G. (2009), Fifty years of operational research and
emergency response, J Opl Res Soc, 60: S126-S139.
26
Naïve Bayes
Neural Networks
Classification Trees
Gamma
Weibull
Log-normal
Coxian Phase-Type
CONDITIONAL
COMPONENT
PROCESS
COMPONENT
Outcome
Log-logistic
Exponential
Pearson
Erlang
Bayesian Networks
Logistic Regression
Clustering
Figure 1: Schematic diagram illustrating the key components of the Discrete
Conditional Survival (DCS) model.
27
Ratio of Probability in Response Timeband
i to Probability in Response Timeband 5
10
Urban
100002
9000
7000
10
6000
10
3000
2000
10000
4000
10
2000
10000
Urban
9000
0
0.5
08000
2
3
4
Radial distance r1 (miles)
2.5
5
0
6
0
8000
6000
10
4000
3000
4
4000
10
2000
10
3 1000
10
10
10
1
2
3
4
Radial distance r1 (miles)
5
6
1
0
-1
0
8
6
Response Timeband 1
Response Timeband 2
Response Timeband 3
Response Timeband 4
Response Timeband 5
4
3
1000
0
2
7
5
Ards Local Government District (LGD=2)
Rural: Radial distance r2=15.5 miles
5
3000
2000
2
3
4
5
6
Radial distance r1 (miles)
2
3
4
Radial distance r1 (miles)
5000
Response Timeband 1
Response Timeband 2
Response Timeband 3
Response Timeband 4
Response Timeband 5
5
1
1
7000
6000
5000
Ratio of Probability in Response Timeband
i to Probability in Response Timeband 5
2
Urban
-2
1000
10
9000
Ards Local Government District (LGD=2)
Rural: Radial distance r2=3.8 miles
7000
6
1
1.5
Radial distance r1 (miles)
1
10
Response Timeband 1
Response Timeband 2
Response Timeband 3
Response Timeband 4
Response Timeband 5
-1
3000
1000-1
10
0
5000
Response Timeband 1
Response Timeband 2
Response Timeband 3
Response Timeband 4
Response Timeband 5
40000
10
10
6000
5000
10
1
8000
70001
10
10
9000
8000
10
Ards Local Government District (LGD=2)
Urban: Radial distance r2=7.4 miles
2
Urban
10000
10
Ratio of Probability in Response Timeband
i to Probability in Response Timeband 5
Ratio of Probability in Response Timeband
i to Probability in Response Timeband 5
10
Ards Local Government District (LGD=2)
Urban: Radial distance r2=2.4 miles
3
1
2
3
Radial distance r1 (miles)
4
10
2
0
10
10
10
1
2
3
4
Radial distance r1 (miles)
5
1
0
-1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Radial distance r1 (miles)
Figure 2 Visualisations based on the optimal multinomial logistic regression
model (see Equation 3) found for Ards Local Government District. The model
predicts the relative probability an observation lies in one of 5 response timebands depending on the radial distances to the two closest ambulance stations (r1
28
6
and r2), and whether an observation occurs at an urban/rural location. The red
shaded regions indicate values of r1 not applicable/not observed, given the values
of r2 and the urban/rural indicator variable.
29
Urban
10
Urban
10000
10000
9000
9000
8000
8000
Magherafelt Local Government District (LGD=20)
7000
Urban
2
Magherafelt Local Government District (LGD=20)
Rural
7000
10
4
6000
6000
4000
10
5000
Response Timeband 1
Response Timeband 2
Response Timeband 3
Response Timeband 4
Response Timeband 5
1
3000
2000
1000
10
0
0
10
10
1
2
3
4
Radial distance r1 (miles)
5
6
-1
-2
0
1
2
3
4
Radial distance r1 (miles)
5
6
Ratio of Probability in Response Timeband
i to Probability in Response Timeband 5
Ratio of Probability in Response Timeband
i to Probability in Response Timeband 5
5000
10
Response Timeband 1
Response Timeband 2
Response Timeband 3
Response Timeband 4
Response Timeband 5
3
4000
3000
10
2
2000
10
1000
1
0
10
10
10
1
2
3
4
Radial distance r1 (miles)
-1
-2
0
5
10
Radial distance r1 (miles)
model (see Equation 4) found for Magherafelt Local Government District (LGD).
The model predicts the relative probability an observation lies in one of 5
response time-bands depending on the radial distance to the closest ambulance
station (r1), and whether an observation occurs at an urban/rural location. The
red shaded regions indicate values of r1 not observed in the urban/rural areas of
30
6
0
Figure 3 Visualisations based on the optimal multinomial logistic regression
the LGD.
5
15
Ards Local Government District (LGD=2)
Proportion of Incidents
0.2
0.1
0
0
0.2
0.1
0
0
0.2
Timeband 1
5
10
15
5
10
15
20
5
10
15
20
0.05
0
0
30
25
30
Timeband 4
0.1
0
0
0.1
25
Timeband 3
0.1
0
0
0.2
20
Timeband 2
data
log-logistic
Coxian252 phase
30
two-term log-logistic
log-normal
5
10
15
20
25
30
15
20
Response Times (minutes)
25
30
Timeband 5
5
10
Figure 4: Distribution of observed response times within the training set of Ards
Local Government District, where the data has been separated into five discrete
classes (or response time-bands) using the Conditional Component of the DCS
model. The plot illustrates possible distribution fits to these, based on various
distribution forms (log-logistic, 2-phase Coxian, two-term log-logistic and lognormal).
31
Magherafelt Local Government District (LGD=20)
Timeband 1
0.1
0.05
Proportion of Incidents
0
0
0.1
5
10
15
20
Timeband 2
data
log-logistic
Coxian 2 phase
25log-logistic
two-term
log-normal
30
0.05
0
0
5
10
15
25
30
25
30
Timeband 3
0.1
0.05
0
0
0.1
20
5
10
15
20
Timeband 4
0.05
0
0
5
10
15
20
Response Times (minutes)
25
Figure 5: Distribution of observed response times within the training set of
Magherafelt Local Government District, where the data has been separated into
four discrete classes (or response time-bands) using the Conditional Component
of the DCS model. The plot illustrates possible distribution fits to these, based on
various distribution forms (log-logistic, 2-phase Coxian, two-term log-logistic and
log-normal).
32
30
LGD=2 Q1<r1<=Q2
LGD=2 r1>Q3
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
1
0.9
Proportion
1
0.9
0.3
0
LGD=20 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=20 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
8
16
24
Response time (mins)
0.1
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=20 r1>Q3
1
0.5
0
LGD=20 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
LGD=2 Q2<r1<=Q3
1
0.9
Proportion
Proportion
LGD=2 r1<=Q1
1
0.9
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
Figure 6: This figure compares data simulated from the Discrete Conditional
Survival (DCS) model and from the parametric accelerated failure-time model to
that observed in the test set for the Local Government Districts (LGDs) of Ards
(LGD=2) and Magherafelt (LGD=20) (where simulated response times are based
on using each of the fully parameterised models together with covariate
information from each of the observations in the test set). The cumulative
response time distributions of observed/simulated data are determined and plotted
separately for the different r1 quartiles of each Local Government District.
33
Variable
Description
Geographical Location Information
Relative to ambulance stations:
r1
radial distance between incident and closest ambulance station
r2
radial distance between incident and second closest ambulance station

Based on quartiles of r1 and r2 (6 categories):
s1
1. r1≤Q1
2. Q1<r1≤Q2
3. Q2< r1≤Q3 and r2≤ Q2
4. Q2< r1≤Q3 and r2> Q2
5. r1>Q3 and {r2≤Q1 or r2> Q3}
6. r1>Q3 and Q1<r2≤Q3
road distance between incident and closest ambulance station
s2
road distance between incident and second closest ambulance station
Location of incident: Depending on the Census Output Area that the incident
occurred in, the location is classified as...
u
an urban/rural area (2 categories)

one of 8 classification band categories ranging from open countryside
to metropolitan urban area
Temporal Information
h
Incident hour (24 categories)
g
Incident hour group (4 categories: 0-7;8-9;10-13;14-23)
w
Day of week: 7 categories
Response Deployment Information
Ω
β
Categorical variable indicating whether an incident was
1. classified as high priority
2. not classified as high priority
3. unclassified
Binary variable indicating if observation corresponds to a Rapid
Response Vehicle response
Table 1: Variables considered for inclusion in the Discrete Conditional Survival
(DCS) model and the parametric accelerated failure-time model.
34
Appendix 1: The cumulative response time distribution observed in the test set,
where observations in the different r1 quartiles of each Local Government
District (LGD) are plotted separately. These have been compared to results
simulated from the Discrete Conditional Survival (DCS) model and from the
parametric accelerated failure-time model, where in each case simulated
response times are based on covariate information from each of the observations
in the test set.
35
LGD=1 Q1<r1<=Q2
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0
LGD=2 r1<=Q1
0.5
0.4
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=2 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
LGD=3 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=3 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
LGD=4 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=4 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
8
16
24
Response time (mins)
0.1
36
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=4 r1>Q3
1
0.9
0.4
0
LGD=4 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=3 r1>Q3
1
0.9
0.4
0
LGD=3 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=2 r1>Q3
1
0.5
0
LGD=2 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
0.6
0.3
0.2
32
Proportion
0.9
Proportion
1
0.2
Proportion
LGD=1 r1>Q3
1
0.3
Proportion
LGD=1 Q2<r1<=Q3
1
Proportion
Proportion
LGD=1 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
LGD=5 Q1<r1<=Q2
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0
LGD=6 r1<=Q1
0.5
0.4
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=6 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
LGD=7 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=7 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
LGD=8 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=8 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
8
16
24
Response time (mins)
0.1
37
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=8 r1>Q3
1
0.9
0.4
0
LGD=8 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=7 r1>Q3
1
0.9
0.4
0
LGD=7 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=6 r1>Q3
1
0.5
0
LGD=6 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
0.6
0.3
0.2
32
Proportion
0.9
Proportion
1
0.2
Proportion
LGD=5 r1>Q3
1
0.3
Proportion
LGD=5 Q2<r1<=Q3
1
Proportion
Proportion
LGD=5 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
LGD=9 Q1<r1<=Q2
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0
LGD=10 r1<=Q1
0.5
0.4
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=10 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
LGD=11 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=11 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
LGD=12 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=12 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
8
16
24
Response time (mins)
0.1
38
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=12 r1>Q3
1
0.9
0.4
0
LGD=12 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=11 r1>Q3
1
0.9
0.4
0
LGD=11 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=10 r1>Q3
1
0.5
0
LGD=10 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
0.6
0.3
0.2
32
Proportion
0.9
Proportion
1
0.2
Proportion
LGD=9 r1>Q3
1
0.3
Proportion
LGD=9 Q2<r1<=Q3
1
Proportion
Proportion
LGD=9 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
LGD=13 Q1<r1<=Q2
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0
LGD=14 r1<=Q1
0.5
0.4
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=14 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
LGD=15 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=15 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
LGD=16 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=16 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
8
16
24
Response time (mins)
0.1
39
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=16 r1>Q3
1
0.9
0.4
0
LGD=16 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=15 r1>Q3
1
0.9
0.4
0
LGD=15 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=14 r1>Q3
1
0.5
0
LGD=14 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
0.6
0.3
0.2
32
Proportion
0.9
Proportion
1
0.2
Proportion
LGD=13 r1>Q3
1
0.3
Proportion
LGD=13 Q2<r1<=Q3
1
Proportion
Proportion
LGD=13 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
LGD=17 Q1<r1<=Q2
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0
LGD=18 r1<=Q1
0.5
0.4
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=18 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
LGD=19 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=19 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
LGD=20 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=20 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
8
16
24
Response time (mins)
0.1
40
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=20 r1>Q3
1
0.9
0.4
0
LGD=20 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=19 r1>Q3
1
0.9
0.4
0
LGD=19 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=18 r1>Q3
1
0.5
0
LGD=18 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
0.6
0.3
0.2
32
Proportion
0.9
Proportion
1
0.2
Proportion
LGD=17 r1>Q3
1
0.3
Proportion
LGD=17 Q2<r1<=Q3
1
Proportion
Proportion
LGD=17 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
LGD=21 Q1<r1<=Q2
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0
LGD=22 r1<=Q1
0.5
0.4
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=22 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
LGD=23 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=23 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
LGD=24 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.6
0.5
0.4
0
LGD=24 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.8
Proportion
1
0.9
Proportion
1
0.9
0.6
0
8
16
24
Response time (mins)
0.1
41
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=24 r1>Q3
1
0.9
0.4
0
LGD=24 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=23 r1>Q3
1
0.9
0.4
0
LGD=23 Q2<r1<=Q3
1
0.5
Observed
DCS Model
Failure-time model
0.2
0.9
0.6
32
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=22 r1>Q3
1
0.5
0
LGD=22 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
0.6
0.3
0.2
32
Proportion
0.9
Proportion
1
0.2
Proportion
LGD=21 r1>Q3
1
0.3
Proportion
LGD=21 Q2<r1<=Q3
1
Proportion
Proportion
LGD=21 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32
LGD=25 Q1<r1<=Q2
LGD=25 r1>Q3
0.9
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.4
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.6
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
1
Proportion
1
0.3
0
LGD=26 r1<=Q1
8
16
24
Response time (mins)
0.1
0
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
0.6
0
LGD=26 Q1<r1<=Q2
8
16
24
Response time (mins)
0.1
0
32
0.9
0.9
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
0.3
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
0.1
0
0.5
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.1
0
0.6
0.5
0.4
0.3
0.2
32
Proportion
0.9
Proportion
1
0.6
0
8
16
24
Response time (mins)
0.1
42
0
32
0.6
0.5
0.4
0.3
Observed
DCS Model
Failure-time model
0.2
32
8
16
24
Response time (mins)
LGD=26 r1>Q3
1
0.5
0
LGD=26 Q2<r1<=Q3
1
0.6
Observed
DCS Model
Failure-time model
0.2
1
Proportion
Proportion
LGD=25 Q2<r1<=Q3
1
Proportion
Proportion
LGD=25 r1<=Q1
1
0
8
16
24
Response time (mins)
Observed
DCS Model
Failure-time model
0.2
0.1
32
0
0
8
16
24
Response time (mins)
32