Markov Chain Population Models in
Medical Decision Making
Gordon B. Hazen
Min Huang
July 2011
1
Abstract
Markov models are widely employed in cost-effectiveness analysis of healthcare
interventions. When modeling populations of non-interacting individuals, it is typical to
formulate Markov models at the individual level and scale up the results to a fixed cohort
or population by weighting by population size. However, we argue that this may not
adequately capture population dynamics, as for example when an intervention extends
life and thereby increases population size.
We examine population-level Markov models of non-interacting individuals using
techniques from the literature on stochastic networks. A key construct from this literature
is the notion of population equilibrium. We also examine potentially relevant measures
of health-related outcome, exploring how they differ from each other and from
individual-level measures such as QALYs. As we illustrate, population modeling can
provide a more refined picture of outcome. For instance, a beneficial intervention might
increase population size but result on average in less healthy individuals. Further
interesting consequences are first, that in an equilibrium analysis, the common procedure
of weighting QALYs by a representative population size will usually be inappropriate; and
second, that in contrast to common practice, there is no need for time discounting in an
equilibrium analysis.
Key words: Cost-effectiveness, Markov models, Equilibrium, Stochastic networks,
Population-level Markov models, Population measures.
1 Introduction
Markov models1,2,3 have long been utilized in medical decision making and health
economics4, 5. They are designed to describe the course of a disease experienced by a
patient in terms of mutually exclusive ‘health states’ and the transitions among them.
They are widely employed in disease modeling, prognosis analyses, and in costeffectiveness analysis of healthcare interventions. A May 2011 PubMed search on
“Markov” AND “Cost-Effective” for the preceding 10 years resulted in 1134 citations.
To date, medical Markov models are usually formulated at the individual level. That is,
the model follows progress of a given patient from an initial health state for either a fixed
period of time or until death, both with and without the intervention in question. The
analyst calculates both incremental cost and incremental individual benefit, the latter
often in terms of health-related outcome measures6 such as life expectancy or qualityadjusted life years (QALYs)7,8,9. Often these results are scaled up to the level of a relevant
population size by simply multiplying by a population size estimate, or choosing a
convenient population level (e.g., 10,000 or 100,000). Equivalently, the Markov model
may be applied to a cohort of independent patients of an appropriate representative size10.
It is recognized that this naïve treatment of population dynamics is not adequate for
infectious disease modeling.10 The position we take in the current paper is that it also
may not adequately capture important population dynamics even for populations of noninteracting individuals. In particular, the impact of an intervention on the size of a
relevant population may be overlooked, as for example when an intervention extends
longevity and thereby increases population count.
2
Mathematical tools for analyzing populations of individuals are available in the
operations research literature11,12,13 under the names stochastic networks and queuing
networks. In this paper we adapt techniques from this literature to model populations of
non-interacting individuals each of whose health progresses as a Markov chain. A key
notion we adapt from this literature is the notion of long-run population equilibrium.
Once one shifts the focus from an individual to a population, it is natural to ask what the
relevant measures of health-related outcome are, how they relate to individual measures
such as QALYs, and whether individual measures and population measures may conflict
in evaluating an intervention. That is the second focus of this paper. An interesting
result from this analysis is that for a population at equilibrium, the naive procedure of
weighting QALYs by population size may give inappropriate results, and in fact the most
natural measures of ongoing population health benefit and cost involve no time
discounting whatsoever. This is in clear conflict with current practice for individual-level
models.
We begin in the following Section 2 by giving a synopsis of individual-level Markov
cohort models. In Section 3 we introduce Markov population models, and in Section 4
we treat measures of health-related outcome. In Section 5 we re-analyze from a
population perspective a previously published Markov cohort cost-effectiveness effort.
2 Individual-Level Markov Models
Individual-level Markov models, also known as cohort models, have become quite
popular approaches to assessments of cost-effectiveness and comparative effectiveness.
In Figure 1 we give a simple example of such a model, with arrows denoting possible
transitions between health states. Arrows are labeled by transition rates. More
sophisticated versions of the model in Figure 1 have been used to evaluate glycoprotein
2b/3a antagonists for acute coronary syndrome.14
Figure 1. An acute coronary individual-level Markov model depicting health risks associated with
ischemic heart disease (IHD). Risks include myocardial infarction (MI) occurring at annual rate ,
and death, occurring at annual rate 0 from the IHD state and at a greater rate 1 from the Post MI
state.
In a time-homogeneous Markov model such as this one, a transition rate from health state
j to health state k can in general be denoted by a parameter λjk . An annual rate jk of
transition from j to k induces10 an approximate probability jkdt of transition from state j
to state k in a short time interval of length dt.1
Individual-level Markov models are solved by mathematically following a cohort of
individuals through the model’s health states for a fixed period of time. One postulates a
1
The exact formula for arbitrary intervals t is pjk = 1  exp(jkt).
3
initial proportion pj(0) of the cohort in state j at time t = 0. Calculating forward in time, if
one knows the proportions pj(t) for all j for some time epoch t, then one may calculate the
proportions pj(t+dt) in the next time epoch t+dt via the rate equations:
pj(t+dt) = pj(t) + Rate into j  Rate out of j
= pj(t) +
i pi(t)  probability of transition
ij
 k pj(t)  probability of transition jk
= pj(t) +

i
pi (t )ij dt 

k
p j (t ) jk dt .
all states j
(1)
In the limit as dt  0, these become a system of differential equations
d
p j (t )   i pi (t )ij   k p jk (t ) jk
dt
all states j.
It is not widely recognized in the medical community that the equations (1) implement
the most basic method for solution of ordinary differential equations, the Euler method.15
For our acute coronary model of Figure 1, if one starts 100% of the cohort in state IHD at
t = 0, then this solution method gives the proportions pj(t) for j = IHD and j = PostMI as
shown in Figure 2(a). If quality coefficients Qj for each health state j are available, then
one may calculate quality-adjusted life years (QALYs) per patient. For instance, when
QIHD = 0.9 and QPostMI = 0.7, the result is 6.62 QALYs per patient (undiscounted). In
general, the relevant formula, including discounting, is
QALY =


0
e rt  j Q j p j (t )dt .
Here QALYs accrued at time t are

j
Q j p j (t )dt , and these are discounted to the present
by the factor ert corresponding to discount rate r. The result is summed by integrating
from t = 0 to . Usually this integral would be approximated by a finite sum over a finite
duration, but for mathematical simplicity in our development we use the integral form.2
2
In addition to continuously accrued quality as described here, is sometimes desired to include one-time
quality tolls. We cover this in Appendix 1.
4
(a)
(b)
1
1
0.8
0.8
p IHD( t )
p IHD( t )
0.6
0.6
p PostM I( t )0.4
p PostM I( t )0.4
0.2
0.2
0
0
0
5
10
15
20
0
5
10
t
t
(yr)
(yr)
15
20
Figure 2. The proportions pj(t) of a starting cohort in the Markov model of Figure 1 who at time t are
in state j = IHD or j = PostMI, shown at times t from 0 to 20 yr, obtained via the rate equations (1).
Input parameters were 0 = 0.04/yr,  = 0.12/yr, 1 = 0.4/yr for (a), and in (b), the Post-MI mortality
rate 1 = 0.1 is decreased by 75% and the MI incidence rate  is decreased by 50%.
Now consider, in our acute coronary model, a hypothetical intervention (such as
glycoprotein 2b/3a antagonists) that not only lowers the mortality rate 1 post MI by 75%,
but also lowers the incidence of MI by 50%. The results of a cohort analysis are shown
in Figure 2(b). Lowering incidence and mortality seems obviously beneficial, and one
may quantify this benefit by calculating improvement in QALYs under the intervention.
The result, calculated in the same way, is 10.28 QALYs per patient, an improvement of
3.66 QALYs.
The usual method for estimating the effect of an intervention on a population is to scale
up cohort results by multiplying by population size. For instance, the 3.66 QALY/patient
net benefit just discussed would allegedly scale up to a benefit of 366,000 QALYs in a
population of 100,000. However, this scaling-up procedure does not capture dynamical
population effects. An actual population will renew itself continually as individuals enter
and later depart, and an intervention may change these dynamics. For example, in the
model of Figure 1, individuals may enter by developing IHD, and depart by death. How
can we estimate the effect of an intervention on such dynamic populations?
One approach that suggests itself intuitively is to assume that a renewing population is
equivalent to a cohort of individuals, each of whom re-enter the population immediately
after death. We call this the closed routing process corresponding to the original Markov
model. For the model of Figure 1, the corresponding closed routing process is shown in
Figure 3.
Figure 3. The closed routing process corresponding to the cohort model of Figure 1. Departures by
death from either state IHD or Post MI immediately re-enter the IHD state.
Population dynamics for Figure 3 can be calculated using the same rate equations (1) as
before, but with transition rates suitably modified to include re-entry transitions. The
5
results with and without intervention are shown in Figure 4. In each case, the population
reaches equilibrium. The standard method1 for calculating the equilibrium probabilities
j = lim p j (t ) is to set Rate into j  Rate out of j = 0 in equations (1) and solve, that is,
t 
solve the so called balance equations
    
  = 1.
i
i ij
j
j
k
j
jk
=0
for all states j
(2)
The results in our acute coronary model are IHD = 1PostMI = 77% without intervention,
and IHD = 1PostMI = 62.5% with intervention.
(a)
(b)
1
1
0.8
p IHD( t )
0.8
p IHD( t )
0.6
0.6
p PostM I( t )0.4
p PostM I( t )0.4
0.2
0.2
0
0
5
10
15
20
0
0
5
10
t
t
(yr)
(yr)
15
20
Figure 4. Dynamics for the closed routing process described in Figure 3. Parameter values from
Figure 2 are used. The population without intervention is shown in (a) and with intervention in (b).
These population results are unsettling: A clearly beneficial intervention appears to have
made the population worse off in equilibrium. The equilibrium proportion of the
population in the undesirable Post MI state is 23% without intervention and 37.5% with
intervention. This is because the intervention extends the lifetime of Post MI patients,
increasing their prevalence in the population in spite of the lower incidence of MI.
Incorporating the quality coefficients QIHD = 0.9, QPostMI = 0.7, one may calculate the
average QALYs accrued per year at equilibrium using
AQ/yr =

j
 jQ j .
The results are 0.854 per patient per year pre-intervention and 0.825 post-intervention, a
decline of 0.029 QALYs per patient per year.
Is the re-entry assumption we used to construct the closed routing process the source of
this anomaly, that is, does the closed routing process not adequately represent population
dynamics? We shall see below that on the contrary, formulating a closed routing process
is a valid approach. The problem is rather that AQ/yr is not the “right” measure of benefit
for a population. In section 4 below we shall address what we think the right measures
should be, and discuss implications. We first turn, however, in the next section to the
existing theory on Markov population processes. Along the way we show that this theory
justifies the re-entry assumption for constructing the closed routing process.
6
3 Markov Population Models
The theory for treating Markov chain population models has been available in the
stochastic networks literature for some time.11,13 We review that theory here and add our
own result on closed routing processes.
We assume we have an individual-level Markov model with transition rates jk in the
model from health state j to health state k. Because individuals may enter a population as
well as depart it, we need to include arrival rates j to each health state j, representing
individuals who arrive in that health state from outside of the population. For example,
in the individual level model of Figure 1, we add entrance arrows with rates 0 and 1 to
the states IHD and Post MI respectively, representing individual who arrive with
ischemic heart disease, and individuals who arrive already having a myocardial infarction.
The result is the diagram in Figure 5, which is referred to as the open routing process.
We have omitted the Dead states, which represent exit from the population, and therefore
are not explicitly considered, although the individual exit rates j remain.
0
1
IHD
0

(MI)
Post MI
1
Figure 5. The model of Figure 1 augmented by arrival rates 0,1 to the two possible health states.
The result is called the open routing process for the individual-level Markov model of Figure 1.
To summarize, the open routing process has the same transition rates jk from j to k as the
individual-level model, has exit rates j from state j equal to the mortality rates in the
individual-level model, and in addition has arrival rates j to each state j. Just as we can
solve for the equilibrium probabilities in the closed routing process of Figure 3 using
balance equations (2), we can also set up and solve balance equations for the open routing
process. These equations are based on the same principle Rate into j  Rate out of j = 0
as in (2). However, instead of equilibrium probabilities j, the new balance equations
involve the mean number j of individuals in health state j. The revised balance
equations are as follows.
 j  i i ij    j   k  jk   j = 0.
for all states j
(3)
For instance, suppose in the model of Figure 5 that we take 0 = 1,000/yr, that is, one
thousand new cases of ischemic heart disease per year; and 1 = 0, that is, no arrivals to
the population already having MI. At equilibrium by solving (3), the mean numbers
0,1in states IHD, Post MI respectively are (0,1) = (6,250, 1,875) without
intervention, and (0,1) = (10,000, 6,000) with intervention. The proportions in the two
health states are the same as calculated from the balance equations (2), but the
equilibrium total number of individuals alive has increased on average from 0+1 =
8,125 without intervention to 0+1 = 16,000 with intervention, a net increase of 7,875.
Using the same quality coefficients Q0 = 0.9, Q1 = 0.7, we may calculate population total
QALYs accrued per year at equilibrium via the equation
7
TQ/yr =

j
 jQ j .
The result is 6,937.5/yr accrued without intervention and 13,200/yr accrued with
intervention, an increase of 6,262.5/yr. This result confirms our intuition that the
intervention has to be beneficial. The result is in this sense is consistent with the cohort
analysis presented in Section 2, a fact we shall comment on below.
We have yet to formally define what we mean in general by Markov population model.
By this we follow the stochastic networks literature and formulate a Markov model
whose states are vectors n = (nj), where nj is a count of the number of individuals in the
individual-level health state j at a given time. In this model, transitions correspond to the
possible transitions of the individual-level model, with additional transitions added to
represent individuals entering the population. For example, for the individual-level
model of Figure 1, the corresponding Markov population model would have states
consisting of the infinite number of possible vectors n = (nIHD,nPostMI), where nIHD is the
number of individuals in health state IHD, and nPostMI is the number of individuals in
health state Post MI. Deaths in either health state would be represented by transitions
that decrease either nIHD or nPostMI by one. A transition from IHD to Post MI would
induce a population transition that decreases nIHD by one and increases nPostMI by one. In
addition, individuals entering the population would increase either nIHD or nPostMI by one.
Such population models are dealt with in the general theory of stochastic networks,11,13 to
which the reader is referred for details. In particular, what we have described are known
as Jackson networks, which constitute an important branch of queuing theory.12 The
models we construct here are specific kinds of Jackson networks that in queuing
terminology would be known as networks of M/M/ queues. The following is a basic
result from this literature (e.g., see Serfozo11 Example 1.29), and provides a justification
for our assertion above that the solutions j to the balance equations (3) are equal to the
equilibrium mean number of occupants of state j.
Theorem 1: In a Markov population model with health states j, the counts nj of
individuals in health states j are, at equilibrium, independent Poisson variables with
means j given by the solution to the balance equations (3), that is, at equilibrium
P(nj = b) =
 bj
b!
e
 j
b = 0,1,2,…
We now turn to the validity of the re-entry assumption introduced for the closed routing
process discussed in the previous section (Figure 3). Given an open routing process such
as Figure 5, we construct a corresponding closed routing process by routing all departing
transitions back to state j with probability proportional to the entry ratej to state j. In
other words, letting  =  j j be the total arrival rate, we define the modified transition
rates
8
 jk   jk   j
j

and let the closed routing process be the process with transition rates  jk . Note that if
entries to the population occur only at a particular state, say state j = 0, it then follows
that 0 = , all other j are zero, and the only modification to the transition rates is that all
departures are routed to state 0 at rate  j 0   j 0   j . This is precisely how the closed
routing process of Figure 3 was constructed.
Theorem 2: Consider a Markov population model with a specified open routing process
and a corresponding closed routing process as described above. Let (j) be the
equilibrium means of the population model calculated by solving the balance equations (3)
of the open routing process, and let (j) be the equilibrium probabilities for the closed
routing process calculated by solving the balance equations (2) with transition rates  jk .
Then
(a) the equilibrium expected proportion of the population in state j is
(b)  j 
j
;
 j j
(c)  j   j
where  =
j
;
 j j

j


 j  j is the equilibrium average departure rate.

We are unaware of any previous version of this result, and provide a proof in the
appendix. Taken together, parts (a) and (b) of the theorem indicate that the re-entry
assumption used to construct the closed routing process is valid, because the solution (j)
to the corresponding balance equations (2) do in fact constitute the equilibrium expected
proportion of the population in state j. Parts (b) and (c) of the theorem indicate how one
can move between the equilibrium of the closed routing process and the equilibrium of
the open routing process. One can solve either balance equation (2) or (3) and then
directly recover the solution to the other.
4 Measures of Benefit
Suppose as above that we have an individual-level Markov model with quality
coefficients Qj for health state j, whose solution by cohort analysis yields quality-adjusted
life years QALYj for individuals beginning in state j. Suppose also that we formulate and
solve a Markov population model to obtain equilibrium mean numbers j of individuals
in health state j, equilibrium mean total population size  = jj and equilibrium
9
proportions j = j/ of the population in health state j. In this situation, what would be
reasonable measures of benefit, and how do such measures compare?
As we have noted, the conventional approach to this question is to scale up the cohort
analysis benefit QALY0 beginning in some distinguished initial health state j = 0 by
multiplying by a population size estimate ̂ . If we use the equilibrium population size 
as the estimate, then we obtain the measure
Population weighted QALY:
PWQ = QALY0.
One may object that at equilibrium there may be many individuals in health states j other
than j = 0. It would therefore be more reasonable to weight the mean number j of such
individuals by their specific QALY beginning in state j, thereby obtaining
Total lifetime QALY:
TLQ =

j
 j QALY j .
The corresponding average across the equilibrium population would be
Average Lifetime QALY:
ALQ =

j
 j QALY j .
One may also examine not lifetime QALY, but only expected QALY accrued in one year
at equilibrium, thereby obtaining
Total QALY per year:
TQ/yr =

j
 jQ j .
The corresponding average across the equilibrium population would be
Average QALY per year:
AQ/yr =

j
 jQ j .
Note that because j = j, the average and total measures are proportional:
TLQ = ALQ
TQ/yr = AQ/yr.
All of these measures assume a population at equilibrium, but one may consider nonequilibrium measures as well. In particular, suppose we can compute the expected
number Enj(t) of individuals in health state j at every future time t. If we discount future
QALYs at rate r, then total infinite-horizon discounted expected QALY would be given as
follows.
Discounted total population QALY:

DTQ   e rt   j Q j En j (t )dt .
0
10
Please note that we list these measures only as ones that might seem reasonable at first
glance. In fact, some may not be so reasonable after all. We know, for instance, from the
example in Section 2 that AQ/yr does not serve by itself as a reasonable measure of
benefit.
Note also that if instead of quality coefficients Qj we had cost rates Cj for each health
state j, then we could define measures for population cost analogously. That is, we could
define TLC, ALC, TC/yr, AC/yr, DTC just as above. The discussion below would apply
to these measures as well.
Of course, the listed measures may conflict in ranking interventions. The “average”
measures can be particularly misleading, as they do not account for intervention-induced
increases or decreases in population size, as we saw in Section 3 for AQ/yr. It is not
difficult to construct examples in which ALQ yields inappropriate conclusions as well.
We therefore restrict our attention to the remaining “total” measures.
Among the total measures, DTQ seems the most complete, as it accounts for impacts on
the current and all future generations and not on just one generation, like TLQ, or in one
year like TQ/yr. Unfortunately, DTQ may be more bothersome to compute. This
difficulty is ameliorated by the following result.
Theorem 3: DTQ and TQ/yr are proportional and therefore essentially equivalent when
the population is initially in equilibrium. Specifically, if E n j (t )   j for all t ≥ 0, then
DTQ =
1
TQ / yr .
r
In this result the reciprocal 1/r of the discount rate is equal to the discounted value


0
e rt dt of an infinite-horizon lifetime. The result therefore takes the form
DTQ = (discounted number of years)(total QALYs per year).
This relationship to DTQ gives TQ/yr a conceptual advantage as a measure of benefit
over the other total measure TLQ. Another advantage obtains due to the following result.
Theorem 4: In a Markov population model with entry rates j into health states j we have
TQ/yr =
  QALY
j
j
j
where QALYj is calculated with zero discounting.
The import of this result obtains when entries to the population can only happen into a
distinguished state j = 0. Then all other j are zero, and we have
11
TQ/yr = 0QALY0.
In this case, suppose the intervention in question alters QALY0 and not the entry rate 0
(as would usually be true unless the individual level were not adequate for modeling the
intervention). Then the equilibrium population measure TQ/yr and the individual-level
measure QALY0 (undiscounted) would evaluate the intervention identically (either both
would indicate it is desirable or neither would). Note, however, that population weighted
QALY might conflict with this evaluation, since it is given by PWQ = QALY0 and an
intervention might change both QALY0 and the equilibrium mean population size .
Similarly, total lifetime QALY, given by TLQ =  j  j QALY j , might conflict with TQ/yr.
However, an intervention that alters only the quality coefficients Qj would leave the
equilibrium population mean  unaltered, so PWQ, QALY0 and TQ/yr would evaluate this
intervention identically. We summarize as follows.
Corollary: Consider a Markov population model with only one entry state j = 0. Then:
(a) The desirability of an intervention that does not alter the entry rate 0 is evaluated
identically by TQ/yr and by undiscounted QALY0.
(b) The desirability of an intervention that alters only the quality coefficients Qj is
evaluated identically by TQ/yr, by undiscounted PWQ, and by undiscounted
QALY0.
On the other hand, it is quite easy to devise a simple example where TQ/yr conflicts with
TLQ and PWQ, even with the latter two undiscounted.
Example: Consider a Markov population model with only one state 0 having quality
coefficient Q0, entry rate  and departure rate . It is easy to check that with discount
rate r,
QALY0 =
Q0
r
 = / 
TQ/yr = Q0 = Q0/
TLQ = PWQ = QALY0 =
 Q0
.
 r
Consider an intervention that extends life () at reduced quality (Qo), leaving Q0/
constant. Then TQ/yr remains unchanged, but TLQ and PWQ both increase. By altering
the changes in Q0 and  slightly, we could make TQ/yr decrease slightly while TLQ and
PWQ still both increase. 
To summarize the results of this section, suppose it is desired to assess the benefit of an
intervention on an existing population, not considering costs for the time being. For
12
simplicity, suppose entry to this population occurs at rate 0 (not influenced by the
intervention) only at a distinguished state 0. In our view, the “right” approach would be
to adopt an equilibrium population perspective as captured by DTQ or its more easily
computed relative TQ/yr. However, we can imagine analysts unaware of our perspective
proceeding in alternate ways. Suppose in particular that:
1. Analyst A computes increase in discounted QALY0, perhaps scaling it up to a
representative population size.
2. Analyst B understands that the intervention may change equilibrium population
size . She estimates population entry rate 0, and computes increase in PWQ =
QALY0.
3. Analyst C forgets to think about population issues, and doesn’t believe in
discounting health benefits. He computes increase in QALY0 (undiscounted).
Which analyst is guaranteed to always produce the “right” recommendation concerning
the desirability of the intervention? If “right” means consistent with TQ/yr, then
ironically only the naïve Analyst C will do so (Corollary (a) above). The take-home
point is that seemingly reasonable ways to extend individual-level analyses to the
population level may in fact yield worse results than doing nothing at all. Of course,
should there be more than one entry point to the population, then none of the analysts
above are guaranteed to get the “right” answer, even analyst B who might be smart
enough to replace PWQ by TLQ.
Cost-Effectiveness
As already noted, all of the mentioned measures have cost analogs, and it is interesting to
consider how cost-effectiveness analysis might be accomplished in this setting. Suppose
we want to compare incremental discounted total QALYs, which we denote DTQ with
its analog incremental discounted total cost, or DTC. In a population initially at
equilibrium, we would have by Theorem 3,
DTQ =
1
TQ / yr
rQ
DTC =
1
TC / yr ,
rC
where rC and rQ are the discount rates for costs and life years. The cost-effectiveness
ratio would be given by
DTC 1 rC TC / yr
.

DTQ 1 rQ TQ / yr
Therefore, if costs and QALYs are discounted at the same rate, as is often
recommended,16 then the cost-effectiveness ratio reduces to TC/yr / TQ/yr, a figure
that is independent of the common discount rate. Therefore, for populations initially at
equilibrium, the common discount rate for costs and life-years is irrelevant – all one
needs to do is compare TC/yr with TQ/yr.
13
For the other commonly used assumption that life-years are not discounted at all (rQ = 0),
then DTQ is infinite and DTQ is not even defined, so equilibrium cost-effectiveness
analysis is not possible except in a limiting sense if one adopts the implausible
assumption that both costs and life years are discounted at a common vanishing rate.
5 An Illustrative Example
In this section we apply the population methods of this paper to a decision analysis by
Col and colleagues17 of tamoxifen use to prevent breast cancer. This section also
illustrates how one may calculate the short-term effect of an intervention on an existing
equilibrium population, a point we have not discussed above.
Tamoxifen, an estrogen agonist/antagonist, is an effective therapy against established
breast cancer. There is also evidence that it can reduce breast cancer incidence, but
unfortunately, its use can produce life-threatening side effects including endometrial
cancer and vascular events. The question is whether the benefit of its prophylactic use in
healthy women is worth the associated risks.
For simplicity here, we examine only benefits and not costs, and the only side effect we
include is endometrial cancer. We modify the Col model by using cure rate models18,19
for breast cancer survival (parameters pb, b) and endometrial cancer survival (parameters
pe, e). These are shown in Figure 6, formulated as stochastic trees.20,21,22 These two
factors are assumed to be independent. For background mortality, we use a constant
mortality rate μ0 equal to the reciprocal of life expectancy of a 50-year-old woman.
(Compared to a more accurate Gompertz mortality model, the bias for total population
counts in equilibrium is 2% and the bias for the outcome measure TQ/yr is 3%  see
Huang's Ph.D. thesis for details23).
No Disease
No Disease
b
pb
Cured
Breast Ca
(a)
1-pb
e
b
pe
Cured
Endometrial Ca
Death
Not Cured
e
Death
Not Cured
(b)
1pe
Figure 6: Cure models depicting incidence and survival for breast cancer (a) and endometrial cancer
(b). Unlike all previous diagrams, this diagram has instantaneous states, namely breast cancer, and
endometrial cancer. Straight arrows emanating from these state represent chance events and
parameters associated with them are probabilities. Continuous transitions in this diagram are shown
by wavy arrows, and the parameters adjacent to the wavy arrows are transition rates. See Hazen.20,
21,22
In each of the factors of Figure 6, there are only three nonfatal states that persist in time,
namely ‘No disease’, ‘Cured’ and ‘Not Cured’ (the ‘Breast Cancer’ and ‘Endometrial
Cancer’ states are mathematically instantaneous and do not affect the analysis). This
produces 3× 3 = 9 nonfatal state combinations. Using Cykert24 and Jones25 we added
quality coefficients Qjk for each of these 9 states to the Col analysis. These are presented
in Table 1.
14
Table 1. The nine nonfatal state combinations in our replication of the Col et al analysis, with quality
coefficients Qjk specified for each state combination j,k.
Endo Ca
Qjk
No Ca
Cured
Not Cured
No Ca
1
0.81
0.39
Cured
0.81
0.656
0.316
Not Cured
0.39
0.316
0.152
Br Ca
We conduct our analysis for women in only one of Col’s categories, namely, those who
are at fourfold higher and twofold higher risk for developing breast cancer and
endometrial cancer, respectively, as might be expected for women with family histories
of breast cancer, who constitute an estimated 4.8% of women aged 35-64.26 We assume
this population of women begins tamoxifen treatment in the vicinity of age 50. The
values of parameters are listed in Table 2. Following Col, we set the incidence rate of
breast cancer without tamoxifen to be 0.0086/year, corresponding to a five-year risk of
approximately 4.19%, four times higher than average. For simplicity here we depart
from Col and presume the effects of tamoxifen on breast and endometrial cancer persist
through the patient’s lifetime.
Women enter this population at rate 0 in distinguished state 00 = (No Diseaase, No
Disease). To give an estimate of 0, we observe that in the year 2010 in the U.S., there
were 2.3 million women of age 49 who would enter the 50+ age group in 201127, of
whom approximately 4.8% or 110,000 have a family history of breast cancer, and would
hypothetically begin tamoxifen. For expositional simplicity, and recognizing the
roughness of this estimate, we round to0 = 100,000/yr.
Table 2: Risk rate, risk ratio, probability and mortality rate parameters in the Col et al. model17. The
risk rates and risk ratios are estimated from the studies by Fisher et al. 28 that Col et al have cited.
The probabilities and mortality rates are estimated based on SEER data. The parameter μ0 is
estimated such that the average survival time of 50-year-old women is 1/ μ0.
Variable
λb0
RRb
λe0
RRe
pb
μb
pe
μe
μ0
Description
incidence rate of breast cancer without tamoxifen
risk ratio of breast cancer with tamoxifen
incidence rate of endometrial cancer without tamoxifen
risk ratio of endometrial cancer with tamoxifen
probability of cure for breast cancer
mortality rate of breast cancer if not cured
probability of cure for endometrial cancer
mortality rate of endometrial cancer if not cured
Background mortality
Value
0.0086/yr
0.4936
0.00152/yr
4.0132
0.5631
0.0996/yr
0.9019
0.3159/yr
0.03118/yr
Results
We conduct our analyses without discounting future life years. Table 3 displays the
individual-level QALY results with and without tamoxifen treatment. Here QALYjk is
expected QALYs with or without tamoxifen for an individual beginning in state
combination j,k. The key figures that would be used in a cohort analysis are QALY00 with
and without tamoxifen (29.19 yr and 28.79 yr) and the incremental QALY00 = 0.405 yr =
4.9 mo. (This compares to the 3.8 mo. gain in pure life expectancy reported by Col.
15
Table 4 displays equilibrium population counts jk in state combination j,k as well as
equilibrium total population count  = jkjk, both with and without tamoxifen. As
expected, the equilibrium incremental effect of tamoxifen is to decrease breast cancer
prevalence and increase endometrial cancer prevalence. The overall increase in
equilibrium population size ( = 46.4 thousand) is achieved not because the number
never acquiring cancer increases (it actually decreases by 14.4 thousand), but because
cases of cured endometrial cancer increase more than cases of cured breast cancer
decrease (293.4 thousand versus 239 thousand). It is true, however, that the equilibrium
number of cancer-free women (no cancer or cured) does increase by 72.2 thousand.
These ambivalent statistics are reflected in the corresponding equilibrium population
measures in Table 5, where we see that TQ/yr increases by 40.5 thousand, whereas AQ/yr
actually goes down slightly. Much as in the example of Section 2, the intervention keeps
more individuals alive but they tend to be in less healthy states. Recall that according to
the corollary to Theorem 4, incremental TQ/yr must agree in sign with incremental
(undiscounted) QALY00, as it does here. We also include TLQ and ALQ in Table 5.
These incremental values are in this case consistent with incremental TQ/yr, although in
general they need not be.
Table 3. Individual-level QALY benefits of tamoxifen. QALYjk is the expected QALYs accrued by an
individual beginning in state combination j,k.
No Tamoxifen
Endo Ca
QALYjk No Ca
Br Ca
Tamoxifen
Endo Ca
QALYjk No Ca
Cured
Not Cured
23.58
1.00
No Ca 29.19
24.65
1.00
No Ca 0.405
1.07
0.004
Cured 25.66
21.04
0.81
Cured 24.87
21.04
0.81
Cured -0.791
0
0
2.73
0.32
2.73
0.32
Not Cured -0.032
0
0
3.36
Not Cured
3.33
Cured Not Cured
QALYjk No Ca
No Ca 28.79
Not Cured
Cured Not Cured
Incremental
Endo Ca
Table 4. Equilibrium population levels with and without tamoxifen. ji is the equilibrium mean
population count (in 1000s) in state combination j,k.
 jk
No Tamoxifen
Endo Ca
No Ca 2423.7
84.4
Not
Cured
0.833 2508.9
Cured
419.4
33.9
0.16 453.5
Not Cured
60.1
2.8
0.022 62.9
2903.2
121.1
1.02 3025
(1000s) No Ca
Br Ca
Cured
Tamoxifen
Endo Ca
 jk
No Ca 2409.3
377.8
Not
Cured
3.4 2790.5
Cured
180.5
66.0
0.279 246.8
Not Cured
28.4
6.0
2618.2
449.8
(1000s) No Ca
Cured
Incremental
Endo Ca
 jk
(1000s) No Ca
Cured
Not
Cured
2.567 281.6
No Ca -14.4
293.4
Cured -238.9
32.1
0.119 -206.7
0.043 34.4
Not Cured -31.7
3.2
0.0206 -28.5
3.72 3072
-285
328.7
2.71
46.4
Table 5. Comparison of equilibrium population measures for tamoxifen intervention.
No
Tamoxifen Tamoxifen
TQ/yr (1000s)
AQ/yr
TLQ (1000s)
ALQ
2878.7
0.9515
83448.6
27.48
2919.2
0.9504
85637.3
27.88
Increase
40.5
-0.0012
2188.7
0.40
Short-term results
The results in Table 5 are long-term equilibrium effects, but it could also be useful to ask
what the immediate effects of this intervention would be, that is, the effect on the current
population. It is reasonable to expect that the current population is likely to be in
equilibrium, so that the no-tamoxifen equilibrium population counts in Table 4 match the
16
current population of high-risk women not using tamoxifen prophylactically. One can
therefore estimate what the short-term effect would be of putting these women on
tamoxifen by combining the tamoxifen and no-tamoxifen QALY estimates from Table 3
with these counts to form TLQ = jkjkQALYjk and its counterpart ALQ = TLQ/.
These results are shown in Table 6, and indicate a beneficial effect. Notice that the
incremental values of the measures TQ/yr = jkjkQjk and AQ/yr = (TQ/yr)/ are zero
because we are fixing the mean counts jk at the current population level, and the
tamoxifen intervention does not change the quality coefficients Qjk.
Table 6. Short-term effects of prophylactic tamoxifen intervention on the no-tamoxifen equilibrium
population in Table 4.
No
Tamoxifen Tamoxifen
TLQ (1000s)
ALQ
83448.6
27.58
84186.9
27.83
Increase
7.4
0.24
4 Conclusion
In this paper we have presented population-level Markov models for medical applications,
and introduced population measures for cost-effectiveness analyses of health
interventions. Compared with the individual-based analyses and cohort analyses currently
used, we contend that population methods can provide useful adjuncts for populationbased medical decision making. One interesting consequence of our analysis is that the
common procedure of weighting QALYs by population size may be inappropriate for
equilibrium analysis.
In this paper, we focus on time-homogeneous models, which assume constant transition
rates, or exponential transition times, between all states. The assumption can be
unrealistic since some transitions such as human mortalities are usually time/age
dependent. However, the exponential function with constant mortality rate as the
reciprocal of life expectancy has been commonly employed in disease models to
approximate human mortality since the introduction of the convenient approximation to
life expectancy29 by Beck and colleagues in the 1980's. Although constant survival rates
can be over simplistic and lead to inaccuracies in estimate of life expectancy30,31,32
especially when the time horizon is long and disease-specific mortality rates are low, they
offer reasonable approximations in some decision scenarios. Finally, by expanding the set
of Markov states, constant transition rate models can approximate human mortality
reasonably well.20,33 For example, in our reconstruction of the Col et al analysis, we
employed a cure-rate model18, which has constant transition rates and provides a
reasonable approximation to cancer mortality.
Finally, we note that the results we present here apply only to populations of noninteracting individuals, which is nevertheless the most common assumption (albeit made
implicitly) used in these kinds of analyses. Our results would not apply to populations of
interacting individuals such as infectious disease models.
17
Appendix
Appendix 1: Individual QALYs
With continuous discounting at rate r > 0, the total expected quality-adjusted life year
accrued beginning in state i is expressed as


QALYi  E[  e rt QX (t ) dt ] =  e rt  j Q j p j (t )dt
0
0
where it is assumed Qj = 0 for mortality states j. In some cases, transition from state j to
state k produces a quality toll Q(j,k) ≥ 0. The usual practice in discrete-time models is
to incorporate such tolls into the quality coefficient Qj for state j for one cycle only. In
the continuous-time case we consider, it is more convenient to attach the toll to the
transition j  k. The total expected QALY due to transitions, beginning in state i is

QALY 'i  E[  e rt  Q( j, k )dX j ,k (t )] ,
j ,k
0
where dXjk(t) denotes the number of transitions from state j to k at the infinitesimal time
interval (t, t+dt).
For time-homogeneous Markov models, these values are convenient to represent in terms
of matrix algebra. Suppose we augment the quality coefficients Qj and the overall QALYs
by terms accounting for transition QALYs:
Qj  Qj +

jk
Q( j , k )
k
QALYj  QALYj + QALY j
Then Qj accounts for both quality accrual and expected transition QALY while in state j,
and similarly QALYj now accounts for both types of QALY accrual. It has been derived
by Kulkarni 1 that
QALY  (rI  M ) 1 Q,
(4)
where M is the submatrix of the rate matrix corresponding to all nonfatal states, and
QALY  (QALY0 , QALY1 ,..., QALYJ )T , Q  (Q0 , Q1 ,..., QJ )T . 
Appendix 2: Proof of Theorem 2
Proof. (a) From Theorem 1 and properties of independent Poisson variables, the
equilibrium distribution of the population model n conditional on total population size |n|
is a multinomial distribution with parameters (| n |,
0
,
J
1
 
j 0
j
,...,
J
j 0
j
J
).
J

j 0
j
18
Therefore the expected proportion of the population in each health state j is
j
, j  0,..., J .
J

j 0
j
(b). Equilibrium population means  j , j  0,..., J are given by the balance equations (3),
which are
J
J
k 0
k j
k 0
k j
 j (  j    jk )   j    k kj , j  0,..., J
By summing up all the above equations, we get
J
J
j 0
j 0
 j  j   j  
(5)
On the other hand,  j , j  0,..., J are given by balance equations (2), which are
J
J
J
k 0
k 0
j 0
 j   ' jk    k  'kj , j  0,..., J , and   j  1
Or,
J
J
k 0
k j
k 0
k j
 j  ( jk   j k / )    k (kj   k j / ) , j  0,..., J .
Which is equivalent to
J
J
J
k 0
k j
k 0
k j
k 0
 j (  j    jk )    k kj  (  k  k ) j / , j  0,..., J .
By using equation (5), it is easy to see that  j , j  0,..., J satisfies the above system of
balance equations, since
J
J
k 0
k j
k 0
k j
 j (  j    jk )    k kj  j
J
J
k 0
k j
k 0
   k kj  (  k  k ) j / ,
J
J
J
k 0
k j
k 0
k j
k 0
j  0,..., J
However,  j (  j    jk )    k kj  (  k  k ) j / , j  0,..., J is a system of
homogenous linear equations, its solutions only differ by a multiplier.
19
J
Since

j 0
j
 1 , we have  j 
j
, j  0,..., J .
J

j
j 0
J
j
J
(c). From (5), we have   j .
j 0
j 0

j 0
Since  j 
j

j 0
J
J
J
j 0
j 0
J

j 0
j
  /  .Therefore
j
 j   j  j   j
j 0
j
 j .  j  j   , or
, we have
J
 j  .
J

.

Appendix 3: Proof of Theorem 3

Proof. By definition DTQ   e rt  j Q j En j (t )dt . If the population is in equilibrium from
0
the current time t=0, then En j (t )   j for all t. Thus

DTQ   e rt dt  j Q j j 
0
1
 Q j j .
r j
Therefore
1
DTQ  TQ / yr
r
as claimed. 
Appendix 4: Proof of Theorem 4

Proof: 1) From Equation (4) we have QALY   M 1 Q . We also have     M 1 if
writing equation (3) in matrix form. Here M is a submatrix of the rate matrix of the
routing process, and also a submatrix of the rate matrix of the underlying individual
model corresponding to all nonfatal states, and   ( 0 , 1 ,...,  J ),
  ( 0 , 1 ,..., J ) .Therefore,

TQ/yr=   Q    M 1  Q    QALY   j j QALY j

20
References
1
Kulkarni VG. Modeling and analysis of stochastic systems. Champman& Hall 1996.
2
Beck JR and Pauker SG. The Markov process in medical prognosis. Med Decis Making
3 (1983) 419-58.
Sonnenberg JB and Beck JR. Markov models in medical decision making: a practical
guide. Med Decis Making 13 (1993) 322-38.
3
4
Ludbrook A. A cost-effectiveness analysis of the treatment of chronic renal failure.
Appl Econ 1981; 13: 337-50.
Simon DG . A cost-effectiveness analysis of cyclosporine in cadaveric kidney
transpalanation. Med Decis Making 1986; 6; 199-207.
5
Deverill M, Brazier J, Green C, Booth A. The use of QALY and Non-QALY measures
of health-related quality of life. Pharmacoeconomics 1998; 13: 411-420.
6
7
Pliskin JS, Shepard DS and Weinstein MC, Utility functions for life years and health
status, Operations Research 28 (1980), 206-224.
Mehrez A, Gafni A. “Quality adjusted life years, utility theory, and healthy-years
equivalents”, Med Decis Making 1989; 9: 142-149.
8
9
Hunink M, Glasziou P, Siegel J, Weeks J, Pliskin J, Elstein A, Weinstein M, Decision
Making in Health and Medicine: Integrating Evidence and Values, Cambridge University
Press, 2001.
Briggs A, Sculpher M, Claxton K. Decision Modeling for Health Economic Evaluation.
Oxford University Press 2006.
10
11
Serfozo R. Introduction to Stochastic Networks. Springer 1999.
Gross D, Shortle JF, Thompson JM, Harris CM. Fundamentals of Queuing Theory.
John Wiley and Sons 2008.
12
Chen H, Yao DD. Fundamentals of Queuing Networks: Performance, Asymptotics, and
Optimization. Springer-Verlag 2001.
13
Palmer S, Sculpher M, Phillips Z, Robinson M, Ginnelly L, Bakhai A et al.
Management of non-ST elevation acute coronary syndrome: how cost-effective are
glycoprotein IIb/IIIa antagonists in the U.K. National Health Service?. International J
Cardiology 100 (2005) 229-40.
14
Ascher UM, Petzold LR. Computer methods for ordinary differential equations and
differential-algebraic equations. 1998. SIAM. ISBN 0898714125.
15
Gold MR, Siegel JE, Russell LB, Weinstein MC, Cost-Effectiveness in Health and
Medicine, Oxford University Press, 1996.
16
Col NF, Orr RK, Fortin JM. Survival impact of tamoxifen use for breast cancer risk
reduction: projections from a patient-specific Markov model. Med Decis Making 2002;
22: 386-393.
17
21
18
Ibrahim JG, Chen M-H, Sinha D. Bayesian Survival Analysis. Springer, 2001.
19
Hazen GB, Huang M. Parametric Sensitivity Analysis Using Large-Sample
Approximate Bayesian Posterior Distributions Analysis. Decision Analysis, 2006:3,208219.
20
Hazen GB. Stochastic trees: A new technique for temporal medical decision modeling.
Med Decis Making 1992; 12:163-178.
21
Hazen GB. Factored stochastic trees: a tool for solving complex temporal medical
decision models. Med Decis Making 1993; 13:227-236.
22
Hazen GB. Stochastic trees and the StoTree modeling environment: models and
software for medical decision analysis”, Journal of Medical Systems 2002; 26:399-413.
Huang M, Hazen GB., Markov Chain Population Models in Medical Decision Making,
IEMS, Northwestern Univ. 2007
23
24
Cykert S, Phifer N, Hansen C. Tamoxifen for Breast Cancer Prevention: A Framework
for Clinical Decisions. The American College of Obstetricians and Gynecologists. 2004:
104: 433-442.
25
Jones L, Hawkins N, Westwood M, Wright K, Richardson G, Riemsma R. Systematic
review of the clinical effectiveness and cost effectiveness of capecitabine (Xeloda (R))
for locally advanced and/or metastatic breast cancer. Health Technology Assessment
2004;8: 1-176.
Johnson N, Lancaster T, Fuller A, Hodgson SV. 287-9.The prevalence of a family
history of cancer in general practice. Fam Pract. 1995 Sep;12(3):287-9.
26
27
U.S. Census Bureau, 2010 Census Summary File 1.
28
Fisher B, Costantino JP,Wickerham DL, et al. Tamoxifen for prevention of breast
cancer: report of the National Surgical Adjuvant Breast and Bowel Project P-1 Study. J
Natl Cancer Inst. 1998; 90:1371–88.
Beck JR, Kassirer JP, Pauker SG. A convenient approximation of life expectancy (the
DEALE"). I. Validation of the model. Am J Med. 1982;73:883-8.
29
Robert R. Holland, Charles A. Ellis, Berta M. Geller, Dennis A. Plante and Roger H.
Secker-Walker. Life Expectancy Estimation with Breast Cancer: Bias of the Declining
Exponential Function and an Alternative to Its Use. Med Decis Making 1999; 19; 385.
30
31
Jochanan Benbassat, Gershom Zajicek, Gerrit J. Van Oortmarssen, Isachar Ben-Dov
and Mark H. Eckman Inaccuracies in Estimates of Life Expectancies of Patients with
Bronchial Cancer in Clinical Decision Making Med Decis Making 1993; 13; 237.
32
J. Robert Beck and Stephen G. Pauker Does DEALE-ing Stack the Deck? Med Decis
Making 1999; 19; 503.
van den Hout WB. The GAME estimate of reduced life expectancy. Med Dec Making
2004; 24: 80-88.
33
22