ACRM-ASNR Pre-Conference Institute
Quality Measures for Rehabilitation: Policy,
Provider and Patient Perspectives
Measuring Clinical Change:
Quality Indicators
Kenneth J. Ottenbacher
University of Texas Medical Branch
Disclosures
National Institutes of Health
Grants: R24 HD065702; R01 AG017638; K12 HD055929;
T32 HD007539
PI: Kenneth J. Ottenbacher
National Center for Medical Rehabilitation Research
(NICHD), National Institute on Neurological Disorders
and Stroke, and National Institute on Aging
Department of Education
Grants: H133G080163; H133P110012
PI: Kenneth J. Ottenbacher
National Institute on Disability and Rehabilitation
Research
Learning Objectives
• Describe three approaches to
measuring clinical change
• Discuss the difference between
clinical change at the individual level
and facility level
• Identify challenges in measuring
clinical change
Measuring clinically important
change is complex. Multiple
methods are available with various
advantages and disadvantages.
How NOT to measure clinically
important change:
p<.05
Statistical Significance versus
Clinical Importance
Design sensitivity:
statistical power for
experimental research
Mark W. Lipsey
The Cult of Statistical
Significance
Stephen T. Ziliak & Deirdre
N. McCloskey
Investigators and clinicians have a long
history of interpreting statistically
significant findings as also being
clinically significant.
Example:
Smith reports a rehabilitation outcome
study with statistically significant
results. Jones conducts a replication
study with (null) non-significant
results.
Assume that both trials are
well-designed and their
internal validity is good.
Results:
Smith, t = 2.21 (df = 78, p < 0.05)
Jones, t = 1.06 (df = 18, p = ns)
The interpretation is that Smith found
a statistically significant difference
(change) in the outcome measure and
Jones did not find a statistically
significant difference (change).
And… that the treatment produced a
significant change in the Smith study,
but not the Jones study.
An alternative way to look at the
results of the Smith and Jones
studies is to examine the degree
to which the null hypothesis is
false in both investigations.
Cohen (1988) refers to this as the
“effect size.”
If we compute an effect size
for each study, we will find the
following:
Formula: d = 2(t)/  df
Smith d = 0.50
Jones d = 0.50
The effect size is the same for the two
trials - the degree to which the null
hypothesis is false is identical for both
the Smith and Jones study.
The amount of change associated with
the outcome is the same in both studies.
Why are the findings in one study
statistically significant and the other
study non-significant?
Things are not
always what they
appear to be.
Power for the Jones
trial = 0.18
The chance of a Type 2
error is 0.82 (82%)
From: Day, R.H. (1972). Science, Vol. 175, p.1335.
Reading made
Don Quixote a
gentleman, but
believing what
he read made
him insane.
G. B. Shaw
Measuring Clinical Change
Who is the Target:
1. Assessment of individual change
2. Assessment of facility change
3. Assessment of population change
Approaches to Assessing
(individual) Change
• Comparison of change in raw scores or
ratings
• Effect size (standardized mean
difference)
• Reliable change index (Minimal
detectable change)
• Minimal clinically important difference
• Standardized regression-based change
scores
There are two approaches to measuring
(individual) change.
• Distribution-based approaches that rely
on expressing change scores in terms
of an underlying sampling distribution.
• Anchor-based approaches that require
an external, independent standard to
‘anchor’ the meaning of clinical
importance.
Examples:
Reliability Change Index (RCI)
Need the following information: a) baseline
score, b) follow-up score, and c) the standard
error of measurement (SEM) for the test. (SEM
= S1 1-r) where S1 = standard deviation, r =
reliability coefficient.
RCI = (X2 – X1) / SEM
The RCI provides an estimate of whether a
change is greater than would be expected by
chance (measurement error).
Examples:
Minimal Clinically Important Difference (MCID)
Requires identifying a clinically important
external standard. For example, 5 motor FIM
points. 5 FIM points represents ~ 5% of total
scale range on the motor domain. This
standard can be based on clinical experience,
research literature, patient goals, or some
combination.
Use of UDS burden of care values to
determine external anchor.
Mean Decrease in Minutes
of Help Per Day per FIM Point Change
TM
Minutes of Help
6
5
4
3
2
1
0
13 - 26
26 - 39
39 - 52
52 - 65
65 - 78
78 - 91
Motor FIM Score Ranges
Minutes per FIM Point
Spinal Cord Injury Study, 1990-91 Hamilton et. al.
UDSMR
Converting individual measures of clinical
change to facility or population measures
of clinical change (quality indicators).
Examples:
Percent of persons in a Case Mix Group
achieving Minimal Detectable Change or
Reliability Change Index criteria.
Proportion of persons in a Case Mix Group
achieving a Minimal Clinically Important
Difference.
Examples (continued):
Percent of persons in a Case Mix Group
achieving above benchmark expectations.
Me
Admission
Discharge
Benchmark
Ea
Gr
Ps
Ba
Si
Polar Graph
DU
Ex
DL
Co
To
St
Bl
WC
Bo
TuT
ToT Ch T
UDSMR
Challenges:
1. How to risk Adjust?
2. How to make the process
patient centered?
3. What are the unintended
consequences?
Doing outcomes
research is a lot like
raising children… you
always think you are
going to do a better job
next time.
The
End
http://rehabsciences.utmb.edu/R24/ACRM.asp