Do Good Nursing Homes Achieve Good
Outcomes? Measurement Considerations
Associated with Public Reporting
Vincent Mor, Ph.D.
Brown University
Purpose
 Review Long Term Care Assessments
 Using Nursing Home as Example, Review Long Term
Care Outcome Domains and Measures
 Present Empirical Data Addressing Select Nursing
Home Quality Measurement Issues
 How Correlated are Quality Measures?
 How Stable are Quality Measures?
 How Variable are Quality Measures?
 Implications for Public Reporting
Do Good Nursing Homes Achieve Good
Outcomes?
2
Background
 Long Term Care, while relatively deficient in
resources, had head start with uniform clinical
assessment tools in Rehab Hospitals, nursing
homes and home health agencies
 FIM for Rehab Hospitals, MDS for nursing homes
and OASIS for home health emerged from
acknowledged need to improve care quality.
 MDS and OASIS implemented and applied to
policies ranging from case mix reimbursement to
quality review
Do Good Nursing Homes Achieve Good
Outcomes?
3
The Nursing Home Resident
Assessment Instrument (RAI)
 1986 Institute of Medicine Report on Nursing Home
Quality Recommended a Uniform Resident Assessment
Instrument to Guide Care Planning
 OBRA ‘87 Contained Nursing Home Reform Act
Including RAI Requirement
 A 300 Item, Multi-Dimensional RAI – MDS was Tested
for 2 Years
 Mandated Implementation in 1991
 Fully Computerized in 1998
 Case Mix Reimbursement in 1999
 Quality Reporting in 2002
The Outcome and Assessment
Information Set (OASIS)
 Initially designed as an outcome
measurement tool for home health agency
staff to track how well patients progressing
 Transformed into a multi-purpose tool for:





patient assessment and care planning for individual adult patients
agency-level case mix reports
internal HHA performance improvement
Case-Mix Reimbursement mandated by BBA
National Home Health Quality Compare 2003
Do Good Nursing Homes Achieve Good
Outcomes?
5
Functional Impact Measure (FIM)
 Initiated as a way to document functional
progress during rehabilitation
 Widely used in Rehab Hospitals around the
country
 Measures Mobility, ADL dependence and
Cognition
 Summary scores created and differences
between admission & discharge calculated
 Some provider performance profiling
underway
Do Good Nursing Homes Achieve Good
Outcomes?
6
Nursing Home Measurement Issues
 What does quality mean and to whom?
 Hard to know how much inter-state variation
is due to measurement vs. real processes
 Differentiating between providers; how much
of a difference is important?
 Inter-relationship among the domains and
measures; implications of composite scores
 Stability of measures over time
Do Good Nursing Homes Achieve Good
Outcomes?
7
Domains of Nursing Home Quality
Measures
 Regulatory Compliance
 Deficiencies, Complaints, etc.
 Staffing
 Hours/Resident day, Skill Mix, contractors, etc.
 Clinical Processes
 Restraints, Psychotropics, etc.
 Clinical Outcomes
 ADL decline, Unmet needs, hospitalization
 Quality of Life
 Mood, Food, Activity, Social Interaction
 Satisfaction
 Service meets Expectations? Getting what you want? etc.
Do Good Nursing Homes Achieve Good
Outcomes?
8
Regulatory Based Quality Measures
 State surveyors conduct annual inspections
following national protocol to assess
compliance with Conditions of Participation
 Substantial inter & intra-state variation in
number, type, severity and distribution of
regulatory infractions
 Factors associated with deficiencies also differ
somewhat by state
 Termination of Medicare/Medicaid due to
regulatory violations rare
Do Good Nursing Homes Achieve Good
Outcomes?
9
State Average Total Number of Deficiencies Per Nursing Home, 2004
WA
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
MA
RI
CT
NY
WI
MI
WY
NJ
PA
IA
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
TX
AL
GA
LA
10.8 - 19.4
8.9 - 10.2
7.0 - 8.6
4.7 - 6.5
FL
Do Good Nursing Homes Achieve Good
Outcomes?
Source: OSCAR 2004.
10
Percentage of NHs Cited for Causing Actual Harm or
Immediate Jeopardy to Redisents in Each State, 2004
WA
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
MA
RI
CT
NY
WI
MI
WY
NJ
PA
IA
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
TX
AL
GA
LA
13.3% - 35%
10.1% - 13.2%
7.7% - 10.1%
4.7% - 7.5%
FL
Do Good Nursing Homes Achieve Good
Outcomes?
Source: OSCAR 2004.
11
0
20
%
40
60
Trends and Inter-State Variations in the Percentage of Facilities
Cited for Actual Harm or Immediate Jeopardy to Residents, 1995-2004
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
Source: OSCAR.
Do Good Nursing Homes Achieve Good
Outcomes?
12
Staffing as Structural Quality Measures
 Historically, most measures of nursing home
quality revolved around staffing levels
 The mix of staffing has also been considered,
particularly the ratio of RNs to practical
nurses and to nursing assistants
 The availability of Medical Directors and
Nurse Practitioners is a more recent focus
 Data Quality poor since reported by home;
we apply longitudinal cleaning algorithms
Do Good Nursing Homes Achieve Good
Outcomes?
13
Trends in Total Staffing Levels
100%
Hartford Panel
(4.44+ HPRD)
90%
80%
CMS Optimum
(3.90+ HPRD)
70%
60%
CMS Preferred
(3.00+ HPRD)
50%
40%
CMS Minimum
(2.75+ HPRD)
30%
20%
Below Minimum
(< 2.75 HPRD)
10%
0%
1996
1997
1998
1999
2000
2001
2002
2003
Do Good Nursing Homes Achieve Good
Outcomes?
2004
14
Percent Nursing Homes Exceeding CMS Preferred
Total Direct Care Staffing Level (3.0+ HPRD)
WA
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
NY
WI
MI
CT
WY
PA
IA
MA
RI
NJ
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
78.7 - 96.5
66.8 - 78.6
56.8 - 66.7
32.3 - 56.7
22.4 - 32.2
TX
AL
GA
LA
FL
Do Good Nursing Homes Achieve Good
Source: OSCAR 2004.
Outcomes?
15
Percent Nursing Homes Exceeding Hartford Recommended
Total Direct Care Staffing Level (4.44+ HPRD)
WA
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
NY
WI
MI
CT
WY
PA
IA
MA
RI
NJ
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
11.7 - 30.0
8.5 - 11.6
5.1 - 8.4
3.6 - 5.0
0.0 - 3.5
TX
AL
GA
LA
FL
Do Good Nursing Homes Achieve Good
Source: OSCAR 2004.
Outcomes?
16
Proportion of facilities using 5% or more
contract RNs or LPNs, 1992-2001
14
12
10
%
8
6
4
2
0
1992
1993
1994
1995
1996
RN
1997
1998
1999
2000
2001
LPN
Do Good Nursing Homes Achieve Good
Outcomes?
17
Mean Staffing HPPD, 1999-2004
3.15
3.10
3.10
3.05
3.04
3.02
HPPD
3.00
2.95
2.95
2.93
2.91
2.90
2.97
2.96
2.88 2.88
2.90
2.88
2.85
2.80
2.75
1999
2000
Unadjusted
2001
2002
2003
2004
RUG-NCMI Adjusted
Note: Difference between unadjusted & adjusted means is statistically significant (at p<.001) by paired T-test for all
years 2000-2004.
18
MDS Resident Assessment Data
Quality Issues
 Although inter-rater reliability studies show
good results, some studies suggest data MDS
not consistently collected across providers.
 Inconsistency of data collection undermines
comparability of quality measures
 Analyses of largest reliability trial done reveals
large inter-state differences in the “direction”
of disagreements; this is related to inter-state
variation in quality indicators
Do Good Nursing Homes Achieve Good
Outcomes?
19
Process Measures of Quality
 Calculating the rate at which treatments
that should be delivered to certain patients
are and the rate at which certain things
are done that shouldn’t be done




Restraints
Anti-psychotics
Advanced Directives in Cognitively Impaired
Feeding Tubes in the Cognitively Impaired
Do Good Nursing Homes Achieve Good
Outcomes?
20
State Averaged Quality Measure, 2004:
Prevalence of Restraint Use
WA
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
NY
WI
MI
CT
WY
PA
IA
MA
RI
NJ
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
TX
0.070
0.050
0.036
0.015
-
0.141
0.064
0.047
0.035
AL
GA
LA
FL
Do Good Nursing Homes Achieve Good
Outcomes?
21
Changing Rate of restraint use in
facilities 1999-2004
0
.1
.2
.3
Restraints
0 1 2 3
Do Good Nursing Homes Achieve Good
4 5 6 7 8 Outcomes?
9 10 11 12 13 14 15 16 17 18 19 20 21
22
Outcome Measures of Quality
 Clinical measures related to “sentinel”
health events; rare but should be near “0”
 Functional decline rates calculated as
changes between assessments
 Incidence of clinical events that are not
desired but hard to prevent - hospitalization
 “Unmet” need for care
 Measures of Satisfaction from family or
resident
Do Good Nursing Homes Achieve Good
Outcomes?
23
State Averaged Quality Measure, 2004:
Incidence of Late-loss ADL Worsening
WA
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
NY
WI
MI
CT
WY
PA
IA
MA
RI
NJ
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
TX
0.177
0.154
0.140
0.126
-
0.244
0.177
0.153
0.140
AL
GA
LA
FL
Double the rate between lowest & highest state.
Do Good Nursing Homes Achieve Good
Outcomes?
24
Hospitalization Rate within 6 Months in the Long-Stay NH Population
per State in 2000
30
20
15
10
5
0
LA
MS
NJ
OK
TX
KY
AR
WV
GA
IL
FL
MO
AL
TN
MD
OH
PA
IN
MI
NY
IA
SD
CA
VA
MA
NC
SC
RI
NV
DE
KS
NE
MN
WY
CT
WI
AZ
CO
WA
ND
MT
VT
ID
OR
NH
ME
NM
UT
% Hospitalized
25
Source: MDS 2000; Medicare inpatient claims 2000.
Do Good Nursing Homes Achieve Good
Outcomes?
25
How have clinical quality measures
changed over the last half decade?
 Pressure Ulcer Prevalence remained stable
 Anti-psychotic medication use increased
 Restraint use declined
 HOWEVER
 Lots of variation;
 Facilities in the Top Quintile in 1999 converged to
the middle; and vice versa
 Restraint Improvement resulted from reductions
in restraint use among highest users
Do Good Nursing Homes Achieve Good
Outcomes?
26
Changing Rate of ADL decline in nursing
homes 1999-2004
0
.1
.2
.3
ADL Decline
0 1 2 3 4 Do
5 Good
6 7 Nursing
8 9 10
11 12
13 14Good
15 16 17 18 19 20 21
Homes
Achieve
Outcomes?
27
Change in the Rate of Restraint Use: Stratified by Quintile Ranking at Baseline (1999Q3)
0.300
QUINTILE 5
0.200
QUINTILE 4
0.150
QUINTILE 3
QUINTILE 2
0.100
QUINTILE 1
0.050
Do Good Nursing Homes Achieve Good
Outcomes?
2004Q4
2004Q3
2004Q2
2004Q1
2003Q4
2003Q3
2003Q2
2003Q1
2002Q4
2002Q3
2002Q2
2002Q1
2001Q4
2001Q3
2001Q2
2001Q1
2000Q4
2000Q3
2000Q2
2000Q1
1999Q4
0.000
1999Q3
Mean Rate
0.250
28
Do Nursing Homes that start out with
good clinical outcomes remain good?
 Split all US nursing homes into quintiles in
1999 based upon their clinical quality
 Tracked facilities’ clinical quality measures
each quarter over next 5 years
 Observed substantial “regression to the
mean”
 Most homes beginning in the top quintile
ended up very similar to those beginning
at the bottom
Do Good Nursing Homes Achieve Good
Outcomes?
29
Change in the Rate of ADL Decline: Stratified by Quintile Ranking at Baseline (1999Q3)
0.250
0.200
0.150
QUINTILE 4
QUINTILE 3
0.100
QUINTILE 2
QUINTILE 1
0.050
Do Good Nursing Homes Achieve Good
Outcomes?
2004Q4
2004Q3
2004Q2
2004Q1
2003Q4
2003Q3
2003Q2
2003Q1
2002Q4
2002Q3
2002Q2
2002Q1
2001Q4
2001Q3
2001Q2
2001Q1
2000Q4
2000Q3
2000Q2
2000Q1
1999Q4
0.000
1999Q3
Mean Rate
QUINTILE 5
30
Measuring Differences between Providers
 Variation is reduced after aggregating
measures to the level of the provider
 Number of residents per facility means little
power in standard statistical tests
 Many measures are skewed
 Comparing averages versus rates proportion
meeting a threshold; basis for setting
thresholds?
 Are we seeing differences when there are
none?
Do Good Nursing Homes Achieve Good
Outcomes?
31
4
2
0
Density
6
8
Facility ADL Decline Rates: 1999-2005
0
.2
.4
.6
Doqi:Good
Nursing
Homes
Achieve Good
adl(28)
decline
4 points--adj
rate
Outcomes?
.8
32
0
5
Density
10
15
Facility Restraint Rates: 1999-2005
0
.2
.4
.6 Good
Do Good Nursing
Homes Achieve
qi: restraints
--unadj
rate
Outcomes?
.8
1
33
60% of customers satisfied
In homes in bottom 20%
Do Good Nursing Homes Achieve Good
Outcomes?
34
Facility level distribution of average
resident satisfaction scores: RI 2006
.5 range out of
5 points
Do Good Nursing Homes Achieve Good
Outcomes?
35
Facility level distribution of average
family satisfaction scores: RI 2006
.5 range out of
5 points
Do Good Nursing Homes Achieve Good
Outcomes?
36
How do Quality Measures Inter-relate?
 Nearly as many providers with low rates of
physical restraints have high levels of
pressure ulcers as have low levels;
 The average correlation among MDS based
measures of quality across all US nursing
homes is low;
 Structural and Regulatory based measures
of quality are only minimally related to
outcome based measures of quality
Do Good Nursing Homes Achieve Good
Outcomes?
37
Average within-MSA rank-order correlation
coefficients between Clinical & Reputation
Quality in 2004 (weighted by # NHs in MSA):
OSCAR “Reputational” Measures
[A]
[B]
[C]
[D]
MDS Clinical Measures
[E]
[F]
G]
[A] OCCUPANCY
1.000
[B] % PRIVATE PAY
0.126
1.000
[C] DEFICIENCIES
-0.133
-0.138
1.000
[D] TOTAL NURSE HPPD
-0.079
0.165
-0.125
1.000
[E] ADL DECLINE
-0.053
0.082
0.063
0.031
1.000
[F] RESTRAINTS
-0.021
-0.055
0.094
-0.052
-0.012
1.000
[G] PU WORSENING
-0.120
0.045
0.097
0.025
0.213
0.045
1.000
[H] PERSISTENT PAIN
-0.081
-0.082
0.104
-0.038
0.017
0.060
0.065
[H]
N
(MSAs)
296
296
296
296
Do Good Nursing Homes Achieve Good
Outcomes?
296
296
296
1.000
296
38
How well do the “best” homes perform?
 “Best” in terms of the fewest regulatory
deficiencies, most staff, etc.
 Classified facilities into thirds on
Deficiencies, Staffing, etc.
 Then, compared them on clinical quality
measures over multiple years
Do Good Nursing Homes Achieve Good
Outcomes?
39
0.160
0.140
0.120
0.100
0.080
0.060
0.040
0.020
0.000
1999Q3
1999Q4
2000Q1
2000Q2
2000Q3
2000Q4
2001Q1
2001Q2
2001Q3
2001Q4
2002Q1
2002Q2
2002Q3
2002Q4
2003Q1
2003Q2
2003Q3
2003Q4
2004Q1
2004Q2
2004Q3
2004Q4
Mean Rate
Change in Rate of ADL Decline, Stratified by Baseline
(1999) Ranking of Total Deficiencies (LOW/MIXED/HIGH)
LOW
MIXED
Do Good Nursing Homes Achieve Good
Outcomes?
HIGH
40
0.160
0.140
0.120
0.100
0.080
0.060
0.040
0.020
0.000
1999Q3
1999Q4
2000Q1
2000Q2
2000Q3
2000Q4
2001Q1
2001Q2
2001Q3
2001Q4
2002Q1
2002Q2
2002Q3
2002Q4
2003Q1
2003Q2
2003Q3
2003Q4
2004Q1
2004Q2
2004Q3
2004Q4
Mean Rate
Change in Rate of ADL Decline, Stratified by Baseline
(1999) Ranking of Total Nurse HPPD (LOW/MIXED/HIGH)
LOW
MIXED
Do Good Nursing Homes Achieve Good
Outcomes?
HIGH
41
Does Clinical Quality Predict Nursing
Home Termination?
 NHs with fewest deficiencies less likely to be
terminated
 NHs with lowest ADL decline and restraint
rate less likely to be terminated
 BUT, pressure ulcer worsening or persistent
pain performance not related to future
termination & staffing barely predictive
Do Good Nursing Homes Achieve Good
Outcomes?
42
Quality Predicting Nursing Home
Closure w/in 2 years
0.6
0.5
0.4
Staffing
Deficiencies
ADL Decline
Restraints
Percent
0.3
Closed
0.2
Very Rare Event
0.1
0
Worst
Quality
Mixed
Quality
Best
Quality
Do Good Nursing Homes Achieve Good
Outcomes?
43
Implications for Public Reporting and
Pay for Performance
Cross-sectional Comparisons are suspect
Comparisons across state lines suspect
Volatility in measures within facility
Measures aren’t correlated; P4P needs a
priori value based weighting
 Multi-dimensional quality measures mean
consumers will need support in using the
data to inform decisions




Do Good Nursing Homes Achieve Good
Outcomes?
44
Implications for Quality Measurement
 Improving Data Quality is essential;
applies to regulatory, staffing, MDS based
data and patient/family surveys
 Addressing Methodological issues essential
before applying these data to P4P
 Findings are likely applicable to Home
Health Agencies and Rehab Hospitals
Do Good Nursing Homes Achieve Good
Outcomes?
45