Markov versus Medical Markov
Modeling – Contrasts and Refinements
Gordon Hazen
February 2012
Medical Markov Modeling
• We think of Markov chain models as the province of
operations research analysts
• However …
• The number of publications in medical journals
– using Markov models
– to address medical cost-effectiveness
– approaches 300 per year!
2
Medical Markov Modeling
• Why the large buy-in from
the medical community?
– Easy-to-use software that
combines decision trees
and Markov models (Data,
TreeAge)
– Simplicity of models
• Discrete time
• Transient
3
Overview of this talk
1. Background on medical Markov modeling
2. Population modeling versus individual-level
modeling
3. Product structure in medical Markov models
Overview
1. Background on medical Markov modeling
2. Population modeling versus individual-level
modeling
3. Product structure in medical Markov models.
Medical Markov Modeling
• The kind of modeling that is typical
IHD = Ischemic heart disease
MI = Myocardial infarction (heart attack)
A simplification of: Palmer S, Sculpher M, Phillips Z, Robinson M, Ginnelly L,
Bakhai A et al. Management of non-ST elevation acute coronary syndrome:
how cost-effective are glycoprotein IIb/IIIa antagonists in the U.K. National
Health Service?. International J Cardiology 100 (2005) 229-40.
6
The kind of modeling that is typical
• Cohort analysis
0.9
IHD
0.7
0
Post MI Dead
1000
1
0
0.9867
986.67
0.01
10
0
Alive
QALY
1000
1
900
0.9
0.0033 996.67
3.3333
0.9967 0.895
895
0.9735 19.533
973.51
0.0195 6.9556
0.007 993.04
0.993 889.83
0.8898
…
…
…
…
7
…
Our preference: Continuous-time
pMI = lDt
p0 = m0Dt
p1 = m1Dt
• Cohort analysis in continuous time
𝑝𝑗 𝑡 = 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑠𝑡𝑎𝑡𝑒 𝑗 𝑖𝑠 𝑜𝑐𝑐𝑢𝑝𝑖𝑒𝑑 𝑎𝑡 𝑡𝑖𝑚𝑒 𝑡
8
Our preference – continuous time
• Discounted expected quality-adjusted life years:
9
Cohort analysis in continuous time
• Intervention: Post-MI mortality rate m1 = 0.1/yr is decreased by 75%
and the MI incidence rate l = 0.12/yr is decreased by 50%.
p IHD( t )
1
1
0.8
0.8
p IHD( t )
0.6
0.6
p P ostMI ( t )0.4
p P ostMI ( t )0.4
0.2
0.2
0
0
0
5
10
15
20
0
5
10
15
t
t
(yr)
(yr)
6.62 QALY/patient
10.28 QALY/patient
10
20
Continuous-time version of cohort analysis
𝑝𝑗 𝑡 + 𝑑𝑡 = 𝑝𝑗 𝑡 +
𝑖
𝑝𝑖 𝑡 𝜆𝑖𝑗 𝑑𝑡 −
𝑘
𝑝𝑗 (𝑡)𝜆𝑗𝑘 𝑑𝑡
• Let dt  0 to obtain
… the Kolmogorov differential equations.
• The cohort analysis procedure
is merely the Euler method for solving the
Kolmogorov equations.
11
Overview
1. Background on medical Markov modeling
2. Population modeling versus individual-level
modeling
3. Product structure in medical Markov models.
Question: How to incorporate population issues?
• An intuitive approach: Restart following death
Open routing process
Closed routing process
• Then compute steady-state probabilities in the
resulting irreducible chain.
13
Question: How to incorporate population issues?
• Balance equations for steady-state probabilities:
• Intervention assumptions:
– Post-MI mortality rate m1 = 0.1/yr is decreased by 75%
– MI incidence rate l = 0.12/yr is decreased by 50%.
• Results: PostMI increases from 23.0% to 38.5%
• The population is less healthy!
• So what is wrong here?
14
Population issues: A more rigorous approach
• Observation: A population of non-interacting individuals is
equivalent to a Jackson network of infinite-server queues.
15
Equilibrium results
• Jackson network balance equations
𝛼𝑗 = 𝑚𝑒𝑎𝑛 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑛𝑑𝑖𝑣𝑖𝑑𝑢𝑎𝑙𝑠 𝑖𝑛 ℎ𝑒𝑎𝑙𝑡ℎ 𝑠𝑡𝑎𝑡𝑒 𝑎𝑡 𝑠𝑡𝑎𝑡𝑖𝑜𝑛 𝑗
• Theorem (e.g. Serfozo 1999): The counts nj of individuals in
health state j are, at equilibrium, independent Poisson
variables with means j given by the solution to the balance
equations.
16
Equilibrium results
• Solve balance equations with entrance rate n =
1000/yr
More survivors
under
intervention!
mean (sd)
IHD
Status Quo 6250 (79)
Intervention 10000 (100)
Post MI
1875 (43)
6000 (77)
8125 (90)
16000 (126)
The closed routing process again
• Convert the open routing process to a closed one in
the following way
Open
Closed
𝜈
𝜆′𝑖𝑗 = 𝜆𝑖𝑗 + 𝜇𝑖 𝜈𝑗
𝜈=
𝑗
𝜈𝑗
18
.
Open versus closed routing
Equilibrium means j
Steady-state probabilities j
• Theorem (Hazen and Huang 2011): One may obtain
equilibrium means from steady state probabilities, and vice
versa:
j
n
j j
m
j 

j
j
• where 𝜈 = 𝑗 𝜈𝑗 is the total arrival rate, and 𝜇 =
the equilibrium departure rate .
𝑗 𝜇𝑗 𝜋𝑗
19
is
.
Open versus closed results
Equilibrium means j

IHD
Status Quo 6250
Intervention 10000
MI
1875
6000
Total
8125
16000
Steady-state probabilities j

IHD
MI
Status Quo 77.0% 23.0%
Intervention 62.5% 37.5%
20
Example: Re-analysis of the preventive use of tamoxifen
• Original analysis: Col et al 2002
• Tamoxifen
– an estrogen agonist/antagonist
– an effective therapy against established breast cancer
• Evidence that it can reduce breast cancer incidence
• But life-threatening side effects
– endometrial cancer
– vascular events.
• Would the benefit of its prophylactic use in healthy
women be worth the associated risks?
21
Preventive use of tamoxifen: Our model
• Cure rate models for breast and endometrial cancer treatment
– Mortality decreases in time survived after cancer diagnosis.
– This cannot be directly modeled as a Markov model – it is semi-Markov.
– Cure rate model with unobserved states Cured/ Not Cured allows implicit
mortality to decrease over time survived.
No Disease
No Disease
lb
pb
Cured
Breast Ca
1-pb
le
mb
Death
pe
Endometrial Ca
Not Cured
Breast cancer incidence
and treatment
Cured
me
Not Cured
1-pe
Endometrial cancer
incidence and treatment
22
Death
Preventive use of tamoxifen: Our model
• Overall model is the Cartesian product of the two factors
below and a third Background Mortality factor.
• More on this later …
No Disease
No Disease
lb
pb
Cured
Breast Ca
1-pb
le
mb
Death
pe
Endometrial Ca
Not Cured
Breast cancer incidence
and treatment
Cured
me
Not Cured
1-pe
Endometrial cancer
incidence and treatment
23
Death
Preventive use of tamoxifen: Our model
• Estimated parameters (max likelihood estimates)
No Disease
No Disease
lb
pb
Cured
Breast Ca
1-pb
mb
pb
μb
Death
pe
Cured
Endometrial Ca
Not Cured
me
Death
Not Cured
1-pe
Variable Description
incidence rate of br ca without
λb0
tamoxifen (high risk)
RRb
le
Value
0.0086/yr
risk ratio of br ca with
tamoxifen
probability of cure for br ca
0.4936
mortality rate of br ca if not
cured
0.0996/yr
0.5631
Variable Description
Value
incidence rate of endo ca
0.00152/yr
λe0
without tamoxifen
risk ratio of endo ca with
4.0132
RRe
tamoxifen
probability of cure for endo ca
0.9019
pe
mortality rate of endo ca if not 0.3159/yr
μe
cured
Variable Description
Background mortality (age 50)
μ0
Value
0.03118/yr
24
Preventive use of tamoxifen: Our model
• Product structure for quality of life
– Q jk = Qbj Qek
– more on this later
Endo Ca
Qjk
Br Ca
No Ca
Cured
Not Cured
No Ca
1
0.81
0.39
Cured
0.81
0.656
0.316
Not Cured
0.39
0.316
0.152
• Model entry rate n0 = 110,000/yr
– 2.3 M women reaching age 50 each year x 4.8% at
high risk for breast cancer
25
Preventive use of tamoxifen: Results
Incremental QALYs/woman
Incremental equilibrium
probabilities
Incremental
Endo Ca
DQALYjk No Ca Cured Not Cured
Endo Ca
None
Cured
Not
Cured
No Ca 0.405
1.07
0.004
Djk
Cured -0.791
0
0
None
-0.02
0.10
0.00
Not Cured -0.032
0
0
Br Ca Cured
-0.08
0.01
0.00
Not Cured
-0.01
0.00
0.00
-0.11
0.11
0.00
Br Ca
0.08
-0.07
-0.01
Incremental equilibrium means
D jk
Incremental
Endo Ca
(1000s) No Ca
No Ca -15.84
Br Ca
Not
Cured
322.74 2.824 309.7
Cured
Cured -262.79 35.31 0.131 -227.3
Not
-34.87
3.52 0.023 -31.3
Cured
-313.5 361.57 2.98 51.0
• A more nuanced picture
of the effects of this
intervention than just
incremental QALYs .
26
Overview
1. Background on medical Markov modeling
2. Population modeling versus individual-level
modeling
3. Product structure in medical Markov models.
Markov models with product structure
• Product structure is relatively
common in medical Markov
models
Roach PJ, Fleming C, Hagen MD, Pauker SG.
Prostatic cancer in a patient with asymptomatic
HIV infection: are some lives more equal than
others? Med Decis Making. 1988 AprJun;8(2):132-44.
28
Markov models with product structure
• Much simpler depiction of model
structure: Independent factors
29
Product structure is relatively common
Schousboe JT, Nyman JA, Kane RL, Ensrud KE. Cost-effectiveness of alendronate therapy
for osteopenic postmenopausal women. Ann Intern Med. 2005 May 3;142(9):734-41.
• Schousboe et al. considered 5 different types of fractures:
–
–
–
–
–
hip fracture
clinical vertebral (Cv) fracture
radiographic vertebral (Rv) fracture
distal forearm (Df) fracture
other fracture
• In principle this should allow 25 = 32 state combinations corresponding to
5 factors each at 2 possible levels.
• What their model actually did: 6 states
– 5 states corresponding to a single fracture type
– 1 other state corresponding to the combination of the worst two possible
fracture types
– Disadvantage: Such a model “forgets” past fractures when a new fracture
occurs, which the 32-state model would not do.
30
Advantages of explicitly accounting for product
structure
• Model formulation: Simpler to merely consider one
factor at a time
• Model presentation: Simple factors easier to
understand and critique.
– Model is less likely to be perceived as a “black box”
• Computational advantages as well when factors are
independent.
31
Computation of QALYs under product structure
• Product structure
– Health state 𝐱 = 𝑥1 , … , 𝑥𝑛
– Quality coefficient 𝑣 𝐱 =
𝑖 𝑣𝑖 (𝑥𝑖 )
• The latter assumption on 𝑣 𝐱 is equivalent to
standard gamble independence (Hazen 2003)
• Standard gamble independence:
1-p
p
(yi*,zi,t)
~
(yi*,zi,0)
(yi,zi,t)
True for one 𝑧𝑖 implies true for
all 𝑧𝑖 - the indifference does not
depend on what 𝑧𝑖 is.
32
Computation of QALYs under product structure
• Product structure
– Health state 𝐱 = 𝑥1 , … , 𝑥𝑛
– Quality coefficient 𝑣 𝐱 =
𝑖 𝑣𝑖 (𝑥𝑖 )
• Theorem (Hazen and Li 2010): One may calculate mean QALYs
as follows.
∞
E[QALY/person] = 0 𝑒 −𝑟𝑡 E[𝑄𝑅 𝑡 ]𝑑𝑡
E 𝑄𝑅 𝑡
=
E 𝑄𝑅𝑖 𝑡
=
𝑖
E[𝑄𝑅𝑖 𝑡 ]
𝑥𝑖
𝑣𝑖 (𝑥𝑖 )𝑃𝑥𝑖 (𝑡)
𝑃𝑥𝑖 𝑡 = P(𝐻𝑒𝑎𝑙𝑡ℎ 𝑠𝑡𝑎𝑡𝑒 𝑖𝑛 𝑓𝑎𝑐𝑡𝑜𝑟 𝑖 𝑎𝑡 𝑡𝑖𝑚𝑒 𝑡 𝑖𝑠 𝑥𝑖 )
(cohort analysis in factor i)
33
Computation of QALYs under product structure
Cancer
Health
quality
Year
0
1
2
3
4
5
6
7
8
9
10
38
39
40
1
0.6
0
1
E[QALY
No
Rate]/
Cancer Cancer Death person
10000
0
0
1.0000
9680
320
0
0.9872
9371
545
85
0.9697
9071
700
229
0.9491
8781
804
415
0.9264
8500
872
628
0.9023
8228
912
859
0.8776
7965
934
1101 0.8525
7711
941
1349 0.8275
7464
938
1598 0.8027
7225
928
1847 0.7782
2908
2815
2725
Bkgd
Mortality
AIDS
399
386
374
6693
6799
6901
No
AIDS
10000
9048
8187
7408
6703
6065
5488
4966
4493
4066
3679
0.3148
0.3047
0.2950
224
202
183
0.5
0
E[QALY
Rate]/
AIDS Death person
0
0
1.0000
952
0
0.9524
1212 601
0.8793
1226 1366 0.8021
1157 2140 0.7282
1064 2870 0.6598
970 3542 0.5973
880 4154 0.5406
797 4710 0.4892
721 5213 0.4426
653 5668 0.4005
40
36
33
9737
9762
9784
0.0244
0.0220
0.0199
Mortality
Rate
Survival
/10,000
Prob.
1.0000
99
0.9901
107
0.9796
116
0.9682
126
0.9561
136
0.9431
148
0.9293
160
0.9146
173
0.8988
188
0.8821
203
0.8644
1894
2051
2221
0.0952
0.0775
0.0621
Total
QALY
Rate/
person
0.5
0.9038
0.7873
0.6745
0.5730
0.4843
0.4080
0.3427
0.2872
0.2402
0.2005
0.0002
0.0002
0.0001
6.3331
𝑃𝑥𝑖 𝑡 = P(𝐻𝑒𝑎𝑙𝑡ℎ 𝑠𝑡𝑎𝑡𝑒 𝑖𝑛 𝑓𝑎𝑐𝑡𝑜𝑟 𝑖 𝑎𝑡 𝑡𝑖𝑚𝑒 𝑡 𝑖𝑠 𝑥𝑖 )
E 𝑄𝑅𝑖 𝑡
=
E 𝑄𝑅 𝑡
=
𝑥𝑖
𝑖
𝑣𝑖 (𝑥𝑖 )𝑃𝑥𝑖 (𝑡)
E[𝑄𝑅𝑖 𝑡 ]
∞
E[QALY/person] = 0 𝑒 −𝑟𝑡 E[𝑄𝑅 𝑡 ]𝑑𝑡
34
Computation of expected cost under product
structure
• Assumptions
– Costs accrue in each factor as long as survival is maintained in other
factors
– 𝑆𝑖 𝑡 = P(Survival through time t in factor i)
– Costs add across factors
• Theorem (Hazen and Li 2010): Computational formulas are
E[Cost/person for factor i] =
E 𝐶𝑅𝑖 𝑡
=
𝑥𝑖
∞ −𝑟𝑡
𝑒 E[𝐶𝑅𝑖
0
𝑡 ]
𝑗≠𝑖 𝑆𝑗
𝑡 𝑑𝑡
𝑐(𝑥𝑖 )𝑃𝑥𝑖 (𝑡)
𝑃𝑥𝑖 𝑡 = P(𝐻𝑒𝑎𝑙𝑡ℎ 𝑠𝑡𝑎𝑡𝑒 𝑎𝑡 𝑡𝑖𝑚𝑒 𝑡 𝑖𝑠 𝑥𝑖 )
(cohort analysis in factor i)
35
Advantages of factored computation
• Computational work in cohort analysis is proportional to
the number of state transitions
• Suppose the number of transitions in a factor with s nondeath states is roughly s also.
• Then assuming s states in each factor and f factors,
– sf transitions in the overall model under naïve cohort analysis
– sf transitions in cohort decomposition
– Big advantage for large f
• Caveat: s and f are not usually large.
36
Cohort decomposition issues
• How often are factors independent?
– Ans: More often not probabilistically independent.
– But one factor is almost always probabilistically independent:
Background mortality.
• How reasonable is the product form for the quality
coefficient v(x)?
– Empirical support for product form in HUI literature – additive
decomposition is not supported.
– Often only one factor carries quality adjustments, in which case
product form holds by default.
37
Summary
• These are just the basics
– Population modeling
• Population model  Jackson network
• One can get at equilibrium population issues by solving the usual balance
equations for steady-state probabilities and scaling them up appropriately.
– Product structure
• Common feature of medical Markov models
• Recognizing it can assist in model formulation and presentation, as well as
computation.
– Drawbacks for continuous-time models
• Medical researchers don’t “get” the models
• Software not widely available
• There is more to do here …
Questions?