Measuring change in health outcomes over time:
a comparison of three methods
Paul Campbell Erwin, MD, DrPH1; Glen P. Mays, PhD, MPH2;
Mary F. Evans, PhD3
1University
of Tennessee, Center for Public Health
of Arkansas for Medical Sciences
McKenna College
2University
3Claremont
Introduction
‡ “Performance measurement in the
public health system must be able to
measure inputs, processes, outputs,
and outcomes in ways that allow for
changes (emphasis added) in one to
be linked with another”
B. Turnock, 1997
‡ Dissertation focus: association of changes
in Local Health Department inputs with
changes in state-level health outcomes
Introduction
‡
‡
‡
Changes in LHD inputs from NACCHO surveys of
in 1993 and 2005, aggregated to the state level
Changes in Health Outcomes available from
America’s Health Rankings, for each year, 19932005
Special situation of having periodic/episodic data
for independent variables while having annual
data for dependent variables (Health Outcomes)
How do I make use of all available data for
Health Outcomes?
Cancer Mortality, Tennessee
230
225
220
Rate per 100,000
‡
Challenge: How best to show changes in Health Outcomes when
there are multiple data points, not just a first and last data point
215
210
205
200
195
190
185
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Year
...especially when the low and high values are not the first and last values
...Or in the extreme case, when the first and last
values are the same yet there are obvious changes
between these data points
Research Objective
‡
215
210
205
200
195
190
185
180
175
170
165
160
The Question:
„
What methods are appropriate, understandable, and
useful in measuring change when there are multiple
data points (i.e., not just a first and last data point) for
one set of variables only?
Cancer
‡
The Objective:
„
1990 1992 1994 1996 1998 2000 2002 2004
Compare two standard methods for measuring change –
relative and absolute change, which ignore intermediate
variability – with a two-step regression method, which
considers all data points
1
Study Design
‡
‡
‡
‡
‡
Within a Retrospective cohort analysis
Sources of data: NACCHO and AHR (United
Health Foundation)
The changes in LHD inputs, aggregated to the
state level, served as independent variables, and
changes in health outcomes served as dependent
variables
Determination of change in LHD inputs – by
relative and absolute change (only two data
points available)
Determination of change in Health Outcomes:
„
„
„
‡
Relative Change =
‡
Absolute Change =
(2005 figure- 1993 figure) x 100
1993 figure
2005 figure – 1993 figure
Relative change
Absolute change
Time Trend Analysis
Study Design
‡
Study Design
Time Trend Analysis
„
Step 1: Produce time trend (beta)
coefficients for each health outcome and
each state
„
Step 2: use these coefficients as new
dependent variables in a second
regression model
Study Design
Time Trend Analysis, Formal Model:
Step 1: Cancert = α + β ⋅ t + ε t
where Cancert is the Cancer Mortality rate in the
state in year t , α and β are parameters to be
estimated, ε t is an error and t takes the value of
1, 2, 3,…,T for the first, second, third,…,final year
in the data. The state-level time trend model will
produce an estimate (and standard error) of β ,
which will become a new dependent variable,
denoted CancerTrend for each state.
Study Design
CancerTrend i = δ + λ ⋅ ExpCap + ε i
225
220
R ate p er 100,000
Time Trend Analysis, Formal Model:
Step 2: The second step involves estimating the
main model of interest using the “new” dataset.
The basic model, with changes in Expenditures
per Capita (ExpCap), as the only independent
variable is:
Cancer Mortality, Tennessee
230
215
210
205
200
195
190
185
where i denotes the state.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Year
2
Cancer Mortality, Tennessee
230
225
225
220
220
215
215
R ate p er 100,000
R ate p er 100,000
Cancer Mortality, Tennessee
230
210
205
200
195
210
205
Slope of line in Step 1
becomes new dependent variable
for each state in Step 2
200
195
190
190
185
185
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
1
2
3
4
5
6
7
8
9
Year
10
11
12
13
14
15
16
17
18
19
Year
Results
Infectious Disease Morbidity
Infectious Diseases Morbidity
Changes in LHD Expenditures with Changes in ID morbidity
50.00
45.00
Relative Change
Absolute Change
Time Trend
Dependent variable
Percent change in
Infectious Disease
cases, 1993-2005
Absolute change in
Infectious Disease
cases, 1993-2005
(1) State and timespecific Infectious
Disease cases
(2) β coefficient from
step 1
Independent variable
Percent Change in
Expenditures per capita
Absolute Change in
Expenditures per
capita
Percent Change in
Expenditures per
capita
C ase R ate per 100,000
40.00
35.00
30.00
25.00
20.00
15.00
β coefficient
10.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
-.3561153
-.0234901
.0491225
.1375864
.006322
-2.88
-2.59
-3.72
p
0.007
0.015
0.001
F
4.41
4.01
5.05
19
Year
-.1416378
SE
t
5.00
Prob > F
0.0018
0.0033
0.0007
Adj R-squared
0.3921
0.3626
0.4339
Results
Cardiovascular Disease Mortality
Cardiovascular Disease Mortality
400
Changes in LHD Expenditures with Changes in CVD mortality
350
Relative Change
Absolute Change
Time Trend
Dependent variable
Percent change in
CVD mortality,
1993-2005
Absolute change in
CVD mortality,
1993-2005
(1) State and timespecific CVD
mortality
(2) β coefficient from
step 1
Independent variable
Percent Change in
Expenditures per capita
Absolute Change in
Expenditures per
capita
Percent Change in
Expenditures per
capita
β coefficient
.005494
-.0212522
C V D death s per 100,000
300
250
200
150
100
.0177797
.1256267
.0070548
0.31
-0.17
-0.31
p
0
1
2
3
4
5
6
7
8
Year
9
10
11
12
13
14
-.0021793
SE
t
50
0.759
0.867
0.760
F
3.70
2.56
2.26
Prob > F
0.0054
0.0343
0.0563
Adj R-squared
0.3379
0.2276
0.1930
3
Conclusions
‡
‡
‡
‡
‡
Three methods of measuring change produced similar
findings re: significance and direction of effect
Time Trend Analysis produced smaller β-coefficients
Time Trend Analysis has the advantage of considering all
available data, particularly changes that take place between
the first and last measures, and may be a useful method
given the types of data challenges we face in PHSSR
Time Trend Analysis may be more appropriate for
measuring change when there is less difference between a
first and last value and more variability year-to-year
Time Trend Analysis may be a better method of arriving at
a monotonic estimate of change, but still assumes change
in only one direction
Measuring change in health outcomes over time:
a comparison of three methods
Paul Campbell Erwin, MD, DrPH1; Glen P. Mays, PhD, MPH2;
Mary F. Evans, PhD3
1University
of Tennessee, Center for Public Health
of Arkansas for Medical Sciences
McKenna College
2University
3Claremont
4