Center for Gerontology and Health Care Research, Brown University
Purpose
Do Good Nursing Homes Achieve Good
Outcomes? Measurement Considerations
Associated with Public Reporting
† Review Long Term Care Assessments
† Using Nursing Home as Example, Review Long Term
Care Outcome Domains and Measures
† Present Empirical Data Addressing Select Nursing
Home Quality Measurement Issues
„ How Correlated are Quality Measures?
„ How Stable are Quality Measures?
„ How Variable are Quality Measures?
Vincent Mor, Ph.D.
Brown University
† Implications for Public Reporting
Do Good Nursing Homes Achieve Good
Outcomes?
The Nursing Home Resident
Assessment Instrument (RAI)
Background
† 1986 Institute of Medicine Report on Nursing Home
Quality Recommended a Uniform Resident Assessment
Instrument to Guide Care Planning
† OBRA ‘87 Contained Nursing Home Reform Act
Including RAI Requirement
† A 300 Item, Multi-Dimensional RAI – MDS was Tested
for 2 Years
† Mandated Implementation in 1991
† Fully Computerized in 1998
† Case Mix Reimbursement in 1999
† Quality Reporting in 2002
† Long Term Care, while relatively deficient in
resources, had head start with uniform clinical
assessment tools in Rehab Hospitals, nursing
homes and home health agencies
† FIM for Rehab Hospitals, MDS for nursing homes
and OASIS for home health emerged from
acknowledged need to improve care quality.
† MDS and OASIS implemented and applied to
policies ranging from case mix reimbursement to
quality review
Do Good Nursing Homes Achieve Good
Outcomes?
3
The Outcome and Assessment
Information Set (OASIS)
Functional Impact Measure (FIM)
† Initiated as a way to document functional
progress during rehabilitation
† Widely used in Rehab Hospitals around the
country
† Measures Mobility, ADL dependence and
Cognition
† Summary scores created and differences
between admission & discharge calculated
† Some provider performance profiling
underway
† Initially designed as an outcome
measurement tool for home health agency
staff to track how well patients progressing
† Transformed into a multi-purpose tool for:
„
„
„
„
„
2
patient assessment and care planning for individual adult patients
agency-level case mix reports
internal HHA performance improvement
Case-Mix Reimbursement mandated by BBA
National Home Health Quality Compare 2003
Do Good Nursing Homes Achieve Good
Outcomes?
5
Do Good Nursing Homes Achieve Good
Outcomes?
6
1
1
Center for Gerontology and Health Care Research, Brown University
Domains of Nursing Home Quality
Measures
Nursing Home Measurement Issues
† What does quality mean and to whom?
† Hard to know how much inter-state variation
is due to measurement vs. real processes
† Differentiating between providers; how much
of a difference is important?
† Inter-relationship among the domains and
measures; implications of composite scores
† Stability of measures over time
Do Good Nursing Homes Achieve Good
Outcomes?
† Regulatory Compliance
„ Deficiencies, Complaints, etc.
† Staffing
„ Hours/Resident day, Skill Mix, contractors, etc.
† Clinical Processes
„ Restraints, Psychotropics, etc.
† Clinical Outcomes
„ ADL decline, Unmet needs, hospitalization
† Quality of Life
„ Mood, Food, Activity, Social Interaction
† Satisfaction
„ Service meets Expectations? Getting what you want? etc.
7
Do Good Nursing Homes Achieve Good
Outcomes?
8
State Average Total Number of Deficiencies Per Nursing Home, 2004
Regulatory Based Quality Measures
WA
ME
† State surveyors conduct annual inspections
following national protocol to assess
compliance with Conditions of Participation
† Substantial inter & intra-state variation in
number, type, severity and distribution of
regulatory infractions
† Factors associated with deficiencies also differ
somewhat by state
† Termination of Medicare/Medicaid due to
regulatory violations rare
Do Good Nursing Homes Achieve Good
Outcomes?
MT
MI
MN
VT NH
ID
MA
RI
NY
WI
SD
MI
CT
WY
NJ
PA
IA
NE
NV
OH
UT
MD DE
IN
IL
CA
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
AL
MS
TX
GA
LA
10.8 - 19.4
8.9 - 10.2
7.0 - 8.6
4.7 - 6.5
FL
9
Percentage of NHs Cited for Causing Actual Harm or
Immediate Jeopardy to Redisents in Each State, 2004
ND
OR
Do Good Nursing Homes Achieve Good
Outcomes?
Source: OSCAR 2004.
10
T rends and Inter-State Variations in the Percent age of Facilities
C ited for Actual Harm or Imm ediate Jeopardy to Residents, 1995-2004
WA
60
ME
MT
ND
MI
MN
VT NH
ID
MA
RI
NY
WI
SD
MI
CT
WY
NJ
PA
IA
40
OR
NE
OH
UT
IL
MD DE
IN
WV
CO
KS
%
NV
CA
VA
MO
KY
NC
20
TN
OK
SC
AR
NM
MS
TX
AL
GA
LA
13.3% - 35%
10.1% - 13.2%
7.7% - 10.1%
4.7% - 7.5%
FL
0
AZ
199 5
1 996
1 997
19 98
19 99
20 00
200 1
2002
2 003
20 04
Sourc e: O SC AR.
Do Good Nursing Homes Achieve Good
Outcomes?
Source: OSCAR 2004.
11
Do Good Nursing Homes Achieve Good
Outcomes?
12
2
2
Center for Gerontology and Health Care Research, Brown University
Trends in Total Staffing Levels
Staffing as Structural Quality Measures
† Historically, most measures of nursing home
quality revolved around staffing levels
† The mix of staffing has also been considered,
particularly the ratio of RNs to practical
nurses and to nursing assistants
† The availability of Medical Directors and
Nurse Practitioners is a more recent focus
† Data Quality poor since reported by home;
we apply longitudinal cleaning algorithms
Do Good Nursing Homes Achieve Good
Outcomes?
100%
Ha rtford P a ne l
(4.44+ HP RD)
90%
80%
CM S O ptim um
(3.90+ HP RD)
70%
60%
CM S P re fe rre d
(3.00+ HP RD)
50%
40%
CM S M inim um
(2.75+ HP RD)
30%
20%
Be low M inim um
(< 2.75 HP RD)
10%
0%
1996
1997
1998
1999
13
2000
2001
2002
2003
2004
Do Good Nursing Homes Achieve Good
Outcomes?
14
Percent Nursing Homes Exceeding Hartford Recommended
Total Direct Care Staffing Level (4.44+ HPRD)
Percent Nursing Homes Exceeding CMS Preferred
Total Direct Care Staffing Level (3.0+ HPRD)
WA
WA
ME
ME
MT
MT
ND
OR
MI
MN
SD
NY
WI
MI
CT
WY
PA
IA
MI
MN
VT NH
ID
MA
RI
SD
NY
WI
MI
CT
WY
NJ
NV
NV
OH
UT
IL
UT
WV
KS
OH
MD DE
IN
CO
MD DE
IN
IL
CA
WV
CO
VA
MO
KS
VA
MO
KY
KY
NC
NC
TN
TN
AZ
OK
MS
TX
AZ
SC
AR
NM
78.7 - 96.5
66.8 - 78.6
56.8 - 66.7
32.3 - 56.7
22.4 - 32.2
MA
RI
NJ
PA
IA
NE
NE
CA
ND
OR
VT NH
ID
AL
OK
SC
AR
NM
GA
MS
FL
Do Good Nursing Homes Achieve Good
Source: OSCAR 2004.
Outcomes?
TX
11.7 - 30.0
8.5 - 11.6
5.1 - 8.4
3.6 - 5.0
0.0 - 3.5
LA
15
AL
GA
LA
FL
Do Good Nursing Homes Achieve Good
Source: OSCAR 2004.
Outcomes?
16
M ean Staffing HPPD, 1999-2004
Proportion of facilities using 5% or more
contract RNs or LPNs, 1992-2001
3.15
3.10
3.10
14
3.05
10
3.00
HPPD
12
%
8
3.04
2.93
2.91
6
2.90
4
2.85
2
2.80
2.97
2.96
2.95
2.95
3.02
2.88 2.88
2.90
2.88
2.75
0
1992
1993
1994
1995
1996
RN
1997
1998
1999
2000
1999
2001
Do Good Nursing Homes Achieve Good
Outcomes?
2000
Unadjusted
LPN
17
2001
2002
2003
2004
RUG -NCM I Adjusted
Note: Difference between unadjusted & adjusted means is statistically significant (at p<.001) by paired T-test for all
years 2000-2004.
18
3
3
Center for Gerontology and Health Care Research, Brown University
MDS Resident Assessment Data
Quality Issues
Process Measures of Quality
† Although inter-rater reliability studies show
good results, some studies suggest data MDS
not consistently collected across providers.
† Inconsistency of data collection undermines
comparability of quality measures
† Analyses of largest reliability trial done reveals
large inter-state differences in the “direction”
of disagreements; this is related to inter-state
variation in quality indicators
Do Good Nursing Homes Achieve Good
Outcomes?
† Calculating the rate at which treatments
that should be delivered to certain patients
are and the rate at which certain things
are done that shouldn’t be done
„
„
„
„
Restraints
Anti-psychotics
Advanced Directives in Cognitively Impaired
Feeding Tubes in the Cognitively Impaired
19
Do Good Nursing Homes Achieve Good
Outcomes?
20
Changing Rate of restraint use in
facilities 1999-2004
State Averaged Quality Measure, 2004:
Prevalence of Restraint Use
WA
Restraints
ME
MT
ND
MI
MN
VT NH
ID
SD
NY
WI
MI
CT
WY
PA
IA
.3
OR
MA
RI
NJ
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
VA
MO
.2
KS
KY
NC
TN
SC
AR
NM
MS
TX
-
0.141
0.064
0.047
0.035
GA
LA
FL
0
0.070
0.050
0.036
0.015
AL
.1
AZ
OK
Do Good Nursing Homes Achieve Good
Outcomes?
21
Do Good Nursing Homes Achieve Good
22
0 1 2 3 4 5 6 7 8 Outcomes?
9 10 11 12 13 14 15 16 17 18 19 20 21
State Averaged Quality Measure, 2004:
Incidence of Late-loss ADL Worsening
Outcome Measures of Quality
WA
† Clinical measures related to “sentinel”
health events; rare but should be near “0”
† Functional decline rates calculated as
changes between assessments
† Incidence of clinical events that are not
desired but hard to prevent - hospitalization
† “Unmet” need for care
† Measures of Satisfaction from family or
resident
Do Good Nursing Homes Achieve Good
Outcomes?
23
ME
MT
ND
OR
MI
MN
VT NH
ID
SD
NY
WI
MI
CT
WY
PA
IA
MA
RI
NJ
NE
NV
OH
UT
IL
CA
MD DE
IN
WV
CO
KS
VA
MO
KY
NC
TN
AZ
OK
SC
AR
NM
MS
TX
0.177
0.154
0.140
0.126
-
0.244
0.177
0.153
0.140
AL
GA
LA
FL
Double the rate between lowest & highest state.
Do Good Nursing Homes Achieve Good
Outcomes?
24
4
4
Center for Gerontology and Health Care Research, Brown University
How have clinical quality measures
changed over the last half decade?
Hospitalization Rate w ithin 6 M onths in the Long-Stay NH Population
per State in 2000
† Pressure Ulcer Prevalence remained stable
† Anti-psychotic medication use increased
† Restraint use declined
† HOWEVER
† Lots of variation;
† Facilities in the Top Quintile in 1999 converged to
the middle; and vice versa
† Restraint Improvement resulted from reductions
in restraint use among highest users
30
% Hospitalized
25
20
15
10
0
LA
MS
NJ
OK
TX
KY
AR
WV
GA
IL
FL
MO
AL
TN
MD
OH
PA
IN
MI
NY
IA
SD
CA
VA
MA
NC
SC
RI
NV
DE
KS
NE
MN
WY
CT
WI
AZ
CO
WA
ND
MT
VT
ID
OR
NH
ME
NM
UT
5
Source: MDS 2000; Medicare inpatient claim s 2000.
Do Good Nursing Homes Achieve Good
Outcomes?
25
Do Good Nursing Homes Achieve Good
Outcomes?
Changing Rate of ADL decline in nursing
homes 1999-2004
26
Change in the Rate of Restraint Use: Stratified by Quintile Ranking at Baseline (1999Q3)
0.300
.3
ADL Decline
.2
Mean Rate
0.250
QUINTILE 5
0.200
QUINTILE 4
0.150
QUINTILE 3
QUINTILE 2
0.100
QUINTILE 1
.1
0.050
2004Q4
2004Q3
2004Q2
2004Q1
2003Q4
2003Q3
2003Q2
2003Q1
2002Q4
2002Q3
2002Q2
2002Q1
2001Q4
2001Q3
2001Q2
2001Q1
2000Q4
2000Q3
2000Q2
2000Q1
1999Q4
0
1999Q3
0.000
27
Do Good Nursing Homes Achieve Good
Outcomes?
Do Nursing Homes that start out with
good clinical outcomes remain good?
29
0.250
0.200
QUINTILE 5
0.150
QUINTILE 4
QUINTILE 3
QUINTILE 2
0.100
QUINTILE 1
0.050
Do Good Nursing Homes Achieve Good
Outcomes?
2004Q4
2004Q2
2004Q3
2004Q1
2003Q3
2003Q4
2003Q2
2003Q1
2002Q3
2002Q4
2002Q2
2002Q1
2001Q3
2001Q4
2001Q2
2000Q4
2001Q1
2000Q3
0.000
2000Q2
Mean Rate
† Split all US nursing homes into quintiles in
1999 based upon their clinical quality
† Tracked facilities’ clinical quality measures
each quarter over next 5 years
† Observed substantial “regression to the
mean”
† Most homes beginning in the top quintile
ended up very similar to those beginning
at the bottom
Do Good Nursing Homes Achieve Good
Outcomes?
28
Change in the Rate of ADL Decline: Stratified by Quintile Ranking at Baseline (1999Q3)
1999Q4
2000Q1
Outcomes?
1999Q3
0 1 2 3 4 5
6 7 Nursing
8 9 10
11 12
13 14Good
15 16 17 18 19 20 21
Do Good
Homes
Achieve
30
5
5
Center for Gerontology and Health Care Research, Brown University
Do Good Nursing Homes Achieve Good
Outcomes?
31
2
Density
4
6
8
Facility ADL Decline Rates: 1999-2005
0
Measuring Differences between Providers
† Variation is reduced after aggregating
measures to the level of the provider
† Number of residents per facility means little
power in standard statistical tests
† Many measures are skewed
† Comparing averages versus rates proportion
meeting a threshold; basis for setting
thresholds?
† Are we seeing differences when there are
none?
0
.2
.4
.6
Do
Nursing
Homes
Achieve Good
qi:Good
adl(28)
decline
4 points--adj
rate
Outcomes?
.8
32
Density
10
15
Facility Restraint Rates: 1999-2005
0
5
60% of customers satisfied
In homes in bottom 20%
0
.2
.4
.6 Good
Do Good Nursing
Homes Achieve
qi: restraints
--unadj rate
Outcomes?
.8
1
33
Facility level distribution of average
resident satisfaction scores: RI 2006
Do Good Nursing Homes Achieve Good
Outcomes?
Facility level distribution of average
family satisfaction scores: RI 2006
.5 range out of
5 points
.5 range out of
5 points
Do Good Nursing Homes Achieve Good
Outcomes?
34
35
Do Good Nursing Homes Achieve Good
Outcomes?
36
6
6
Center for Gerontology and Health Care Research, Brown University
Average within-MSA rank-order correlation
coefficients between Clinical & Reputation
Quality in 2004 (weighted by # NHs in MSA):
How do Quality Measures Inter-relate?
† Nearly as many providers with low rates of
physical restraints have high levels of
pressure ulcers as have low levels;
† The average correlation among MDS based
measures of quality across all US nursing
homes is low;
† Structural and Regulatory based measures
of quality are only minimally related to
outcome based measures of quality
Do Good Nursing Homes Achieve Good
Outcomes?
OSCAR “Reputational” Measures
[A]
[C]
[A] OCCUPANCY
1.000
[B] % PRIVATE PAY
0.126
1.000
-0.133
-0.138
1.000
[C] DEFICIENCIES
N
(MSAs)
296
296
296
-0.079
0.165
-0.125
1.000
0.082
0.063
0.031
1.000
[F] RESTRAINTS
-0.021
-0.055
0.094
-0.052
-0.012
1.000
[G] PU WORSENING
-0.120
0.045
0.097
0.025
0.213
0.045
1.000
[H] PERSISTENT PAIN
-0.081
-0.082
0.104
-0.038
0.017
0.060
0.065
296
296
296
1.000
Do Good Nursing Homes Achieve Good
Outcomes?
296
38
Change in Rate of ADL Decline, Stratified by Baseline
(1999) Ranking of Total Deficiencies (LOW/MIXED/HIGH)
0.160
0.140
0.120
0.100
0.080
0.060
0.040
0.020
0.000
1999Q3
1999Q4
2000Q1
2000Q2
2000Q3
2000Q4
2001Q1
2001Q2
2001Q3
2001Q4
2002Q1
2002Q2
2002Q3
2002Q4
2003Q1
2003Q2
2003Q3
2003Q4
2004Q1
2004Q2
2004Q3
2004Q4
1999Q3
1999Q4
2000Q1
2000Q2
2000Q3
2000Q4
2001Q1
2001Q2
2001Q3
2001Q4
2002Q1
2002Q2
2002Q3
2002Q4
2003Q1
2003Q2
2003Q3
2003Q4
2004Q1
2004Q2
2004Q3
2004Q4
Mean Rate
[H]
MIXED
HIGH
Do Good Nursing Homes Achieve Good
Outcomes?
40
Does Clinical Quality Predict Nursing
Home Termination?
0.160
0.140
0.120
0.100
0.080
0.060
0.040
0.020
0.000
Do Good Nursing Homes Achieve Good
Outcomes?
G]
-0.053
Mean Rate
39
Change in Rate of ADL Decline, Stratified by Baseline
(1999) Ranking of Total Nurse HPPD (LOW/MIXED/HIGH)
MIXED
[F]
296
LOW
LOW
MDS Clinical Measures
[E]
[D] TOTAL NURSE HPPD
How well do the “best” homes perform?
Do Good Nursing Homes Achieve Good
Outcomes?
[D]
[E] ADL DECLINE
37
† “Best” in terms of the fewest regulatory
deficiencies, most staff, etc.
† Classified facilities into thirds on
Deficiencies, Staffing, etc.
† Then, compared them on clinical quality
measures over multiple years
[B]
HIGH
41
† NHs with fewest deficiencies less likely to be
terminated
† NHs with lowest ADL decline and restraint
rate less likely to be terminated
† BUT, pressure ulcer worsening or persistent
pain performance not related to future
termination & staffing barely predictive
Do Good Nursing Homes Achieve Good
Outcomes?
42
7
7
Center for Gerontology and Health Care Research, Brown University
Quality Predicting Nursing Home
Closure w/in 2 years
Implications for Public Reporting and
Pay for Performance
0.6
0.5
0.4
Staffing
Deficiencies
ADL Decline
Restraints
Percent
0.3
Closed
0.2
Very Rare Event
0.1
0
Worst
Quality
Mixed
Quality
Best
Quality
Do Good Nursing Homes Achieve Good
Outcomes?
43
Cross-sectional Comparisons are suspect
Comparisons across state lines suspect
Volatility in measures within facility
Measures aren’t correlated; P4P needs a
priori value based weighting
† Multi-dimensional quality measures mean
consumers will need support in using the
data to inform decisions
†
†
†
†
Do Good Nursing Homes Achieve Good
Outcomes?
44
Implications for Quality Measurement
† Improving Data Quality is essential;
applies to regulatory, staffing, MDS based
data and patient/family surveys
† Addressing Methodological issues essential
before applying these data to P4P
† Findings are likely applicable to Home
Health Agencies and Rehab Hospitals
Do Good Nursing Homes Achieve Good
Outcomes?
45
8
8