Measuring Quality Outcomes
John N. Morris, PhD, MSW
Director Emeritus
Institute for Aging Research
Hebrew SeniorLife, Boston
November 2011
Key Question:
How does care at your nursing
facility stack up against external
standards?
Agenda
• interRAI and interRAI Suite
• Application of interRAI data – the QIs
• Conclusions
Agenda
• interRAI and interRAI Suite
• Application of interRAI data – the QIs
• Conclusions
Who are We?
• A non-profit, international group of researchers and
clinicians
• Key interests:
– Science (e.g., cross-national comparisons)
– Assessment instrument development
– Support of implementation in countries across the globe
• Holds copyright to RAI assessment instruments
• Grants royalty-free licenses to governments and care
providers
• License software vendors around the world
• www.interrai.org
interRAI Suite of Assessment
Instruments
• interRAI has created a multi-domain suite of assessment
instruments, providing a common language to describe
persons across sites of care
• The instruments support assessment and care planning in
aged care, mental health, and disability services
• Each assessment instrument has:
– Items common to other instruments – e.g., ADLs, cognitive function,
mood, pain, falls,
– As well as specialized items that are exclusive to that setting __ e.g.,
suicide and criminality items in the mental health tool
InterRAI Assessment Instruments
• First what we do not
do
• interRAI instruments
are more than just a
cobbled together set of
measures
Why are interRAI Assessments
Different?
• Developed by international panel of experts:
geriatrics, gerontology, assessment, and health
services research
• Carefully tested psychometric properties
• New design: assessment drives decision-making at
all levels, from clinical to policy
– Collect data once, use many ways
• Develop compatible systems across health care
sectors
Why are interRAI Assessments
Different?
• cover all relevant domains
– individuals’ strengths and weaknesses
– include items on caregivers, environment
– tradeoff of breadth and length
• not only self-report
– use all possible sources of information
• include full definitions, time delimiters,
examples, exclusions
• training manuals
Examples of Domains Covered
Intake/Initial History
Cognition
Communication/Vision
Mood/Behavior
Psychosocial Well-being
Functional Status
Continence
Disease Diagnoses
Health Conditions
Oral/Nutritional Status
Skin Condition
Medications
Treatment/Procedures
Responsibility
Social Supports
Environmental Assessment
Discharge Potential/Status
Discharge
HC
F
C
LT
AL
Core
Items
PAC
CMH
PC
CHA
MH
Field Test – Can we trust the
items
• interRAI has looked at user acceptability –
the results are most encouraging
• Completed reliability in many countries
• Used Kappa statistic – below .4 poor, .41 to
60 moderate, higher scores considered
substantial agreement
Average Weighted Kappa, by Instrument
and Type of Item
Good
Acceptable
Agenda
• interRAI and interRAI Suite
• Applications of interRAI data – the QIs
• Conclusions
It’s not enough just to measure…..
Applications of interRAI Data
Care Planning
Policy
Case-Mix
ASSESSMENT
Screening
Quality
Nursing Home Quality
Indicators:
So What is a “Good” Home?
Different Ways to See Quality
• Tender loving care -- a focus on daily life, social
engagement, and happiness
– Focus on an ideal image of what happens in a home
setting.
• Standards set by the service industry
– Nature of facility -- Friendliness of staff, clean and
comfortable rooms, the quality of the food
– Qualifications of staff
– Staffing levels
– Adherence to standard care guidelines
• Resident satisfaction self reports
• Maximize resident outcomes–“Quality Indicators”
ALL ARE IMPORTANT!
• Unfortunately, the correlation among
these measures is low
• This holds for:
– Survey results generated by government
agencies
– Person-based satisfaction surveys
– Staffing levels
– Person change measures
Different Types of Quality
Indicators -- Examples
• Problem prevalence
• Proportion of residents physically restrained
• Proportion with daily pain
• Problem Incidence – “Occur over time”
• Falls over prior 90 days
• Urinary tract infection
• Change in Status over time
• Rate of Decline in Physical Functioning
• Rate of Improvement in Mood
Defining Quality
• Process standards
– Tender loving care – a mirror of life at home
– Adherence to specified care protocols
• Environmental standards
–
–
–
–
Cleanliness, quality of food
Caregiver credentials
Caregiver staffing levels
Physical environment
• Person’s satisfaction
• Person’s status and how changes over time - Quality
Indicators
Thus We Have To Make a Decision on
How to Assess Quality
• For interRAI how the person changes
over time matters – what happens with
respect to functional, clinical status,
and social aspects of daily life
• Such outcomes are measured with
program-based Quality Indicators – QIs
Our Goal is Clear
• Once in such a program the person has
the right to expect staff to take every
step possible to:
– Solve immediate problems
– Maximize the person’s functional
potential
– Enhance the person’s quality of life
But the Challenges are Many
• What specific quality measures should
we use – what are the most appropriate
yardsticks?
• What can we reasonably expect a
facility or program to achieve?
• Looking at nursing homes, for
example, we have to comprehend the
person’s underlying potential for
change
High Decline Rates for Persons in
NHs Are Not Inevitable!
35
30
Percent
25
20
15
10
5
0
Overall
Hygiene
Dressing
Locomotion
Transfer
Nature of ADL Change Over 90 Days
DECLINE
IMPROVE
Toileting
Bed Mobility
Eating
Inter-Program Variability
• Inter-program variability in outcome
rates can be enormous
• The next two graphs depict mean (or
average) mobility change rates for
nursing facilities in Canada and the
United States
Technical Issues
• Access the computerized interRAI longitudinal
assessments of the program participants
• Transform these person-level data into program level
QIs --
– introduce risk adjustment
– decide how “different” is different”
– Ultimately identify the “bad” and “good”
providers
• Establish the relationship among quality indicators
• Create a summary Quality Composite
Risk Adjustment of QIs
• Necessary to compare “apples to
apples”
• Account for bias due to variations in
program admission practices
• Three ways to adjust
– Stratification
– Exclusions
– Regression-based adjustment
• Individual risk factors (covariates)
• Direct stratification
ADL Adjustment Example
• The country is Japan
• The comparison is across three types of
facilities
• The outcome is the proportion of residents
who decline in “late loss” ADLs over a 90day period
Example Risk Adjustment : ADL Decline in the
Three Types of Japanese LTC facilities
15.0%
Hospitals
HFE
Nursing homes
10.0%
5.0%
0.0%
RAW
ADJUSTED
History of QIs
• Development goes back two decades
• We have worked to define what matters – it
is something the industry in each country
must face
• In our cross-national work, interRAI created
standardized versions of 80+ NF QIs
• Through analysis and deliberations, the set
has been reduced to 19 measures
Functional QI Items
QI Measure
ADLs
Locomotion
Falls
Cognition
Communication
X
X
X
X
X
Clinical QI Items
QI Measure
Continence
Behavior
Infection
Pain
Mood
Behavior
Pressure Ulcer
X
X
X
X
X
X
X
Utilization QI Items
QI Measure
Catheter
Feeding tube
Restraint
X
X
X
Let us Look at Rates of Change
In Nursing Facilities
• First we will look at the individual QIs
• We contrast Michigan facilities against the
interRAI cross-national standard
• With exceptions – Detroit facilities look
pretty good
id
Co
mm
Co
gn
it iv
e
un
ica
t io
n
ne
De
cli
ne
De
cli
ne
De
cli
ne
De
c li
ne
De
cli
ne
De
cli
ne
ad
de
r
Bo
we
l
Bl
ot
ion
DL
De
c li
dl
De
c li
ne
AD
L
Lo
ss
A
Lo
ss
Lo
ss
A
Lo
co
m
Ea
rly
M
La
te
Percent
FUNCTIONAL DECLINE QIs -- High is Bad
25
20
15
Reference
10
Detroit
Other Mich
5
0
id
Co
mm
ad
de
r
un
ica
t io
n
Co
gn
it iv
e
Bo
we
l
Bl
ot
ion
DL
pr
ov
e
pr
ov
e
pr
ov
e
Im
pr
ov
e
Im
pr
ov
e
Im
Im
Im
pr
ov
e
Im
pr
ov
e
pr
ov
e
Im
dl
Im
AD
L
Lo
ss
A
Lo
ss
Lo
ss
A
Lo
co
m
Ea
rly
M
La
te
Percent
FUNCTIONAL IMPROVEMENT QIs -- High is Good
30
25
20
Reference
15
Detroit
10
Other Mich
5
0
Us
e
Inf
ec
ti o
n
Tu
be
es
t ra
i nt
s
24
all
s
Be
ha
vi o
r
of
R
ce
rS
tag
e
Re
ce
nt
F
Inf
ec
tio
ns
Tr
ac
k
Ul
ina
ry
Pr
es
su
re
Ur
Fe
ed
i ng
ath
ete
r
W
or
se
nin
g
De
cl i
ne
Ind
we
l li n
gC
Pa
in
Mo
od
Percent
CLINICAL DECLINE QIs -- High is Bad
18
16
14
12
10
Reference
8
Detroit
6
4
Other Mich
2
0
Multi-Dimensionality of QIs
– Facilities that perform well in one dimension
may perform at a more average or even poor
level in other selected dimensions of quality
– How do we identify the “best” facilities or
programs?
• If a facility does poorly in any one dimension is
the facility poor?
• If it does superbly in multiple dimensions, is it
best?
A NH Example
• Facilities with the lowest rates of falls are
no more likely than other facilities to have
low rates UTIs, antipsychotic use, cognitive
change, and pain change.
• On the other hand, they are somewhat more
likely to experience an improvement in
early loss ADLs and incontinence
Creating The QI Composite
• 8 functional change QIs -- employing both
decline and improvement variants)
• 10 clinical complexity QIs – e.g., incontinence,
pain
• 80% of composite score from functional measures
• 20% of composite score is from clinical
complexity measures
Distributional Properties of
NH QI Composite
• The next table is based on data from
2 Canadian provinces and 5 US states
• Data are weighted
NH QI Composite - USA and Canada
30
25
Percent
20
USA
15
CANADA
10
5
0
-0
1
2
3
4
5
6
NH QI Composite Score - 10 Best, 4 Median
7
8
9
10
An International Standard
• Given the cross-country relative
equivalency in rates, interRAI has
adopted this North American
Distribution as its International Standard
• And we show that the distribution
differs from one jurisdiction to another
NH QI Composite - International Standard
30
25
Percent
20
15
10
5
0
-0
1
2
3
4
5
6
NH QI Composite - 10 Best, 4 Median
International Standard
7
8
9
10
NH QI COMPOSITE - TWO STATE COMPARISON
35
30
25
Percent
20
15
10
5
0
-0
1
2
3
4
5
6
7
NH QI COMPOSITE - 10 BEST, 4 MEDIAN
International Standard
Massachusetts
Illinois
8
9
10
Several Use Examples for
Quality Indicators
• The type of report provided to a facility
• Assessing inter-state variability
• Measuring change in facility
performance over time – an example
from Massachusetts
Key points
• The position of your QI score – are you better, worse, or
about average of other facilities in the comparison group
(e.g., a state)
• The range of scores across facilities in the state – some are
in similar ranges others are quite diverse
• If you are doing poorly in an area – is this a key issue that
must be reversed?
• If you are doing well in an area – do you understand why,
does it make sense? If it does not, how can you expect to
maintain this QI area as positive over time?
50
Example: Mid-Loss ADL Improvement
TECHNICAL DEFINITION
•
Percent of residents who improve in Mid-Loss ADL functioning (transfer,
locomotion) or remain completely independent in these areas
NUMERATOR – two types of persons:
• Residents at the follow-up assessment who are more independent than they
were at the prior assessment
• Residents who were independent at BOTH the baseline and follow-up
assessments
DENOMINATOR:
• All residents with a valid follow-up assessment and a valid prior baseline
assessment
51
Explanation of Example
• Higher scores on this measure imply better quality of
care.
• The adjusted QI score over the reporting period is
trending downward. The amount of change is trivial.
– Regardless, performance is suffering over the
reporting period.
• Performance is above the reference sample median at
the end of the period.
52
0.47
.95
0.34
.75
0.25
.50
0.16
.25
0.03
.05
2010Q3
2010Q4
2011Q1
Percentile Rank
Adjusted QI Score
Proportion of residents who improve in Mid-Loss ADL
functioning (transfer, locomotion) or remain completely
independent in Mid-Loss ADLs – A HIGH Score is Good
2011Q2
Quarter
53
More Detailed Data – Gives you the count
of the number of residents who experience
the condition referenced by the QIs
Quarter
2010 Q3
2010 Q4
2011 Q1
2011 Q2
Adjusted
QI Score
0.37
0.38
0.41
0.33
Raw QI
Score
0.40
0.39
0.42
0.35
# of
# of
Residents Residents
in
at risk
Numerator
38
35
41
35
94
94
97
99
54
Inter-State Comparison
Summary NH Quality Indicator Index – States and Provinces
Summary QI - State/Provincial Distribution
40.0
35.0
CC
%
30.0
CH
25.0
IL
20.0
MA
15.0
OH
PA
10.0
Reference
5.0
0.0
Worse
-4
-3
-2
-1
0
1
QI Summaey Score
2
3
4
Best
Changes in Quality Over Time
• A 21 quarter comparison of quality in
Massachusetts Nursing Homes – there
is real improvement
57
QI Composite Score –
21 Quarter Trend in Massachusetts NHs
4.2
4
QI Composite Score
3.8
3.6
Mean QI
3.4
3.2
3
2.8
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1
04 04 04 04 05 05 05 05 06 06 06 06 07 07 07 07 08 08 08 08 09
QI Composite - 21 Quarters For Massachusetts Nursing
Homes
60
50
Percent
40
30
% QI Low -- 0-2
% QI Mid -- 3-4
% QI High -- 5-10
20
10
0
Q1 Q2 Q3 Q4 Q1
04 04 04 04 05
Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1
05 05 05 06 06 06 06 07 07 07 07 08 08 08 08 09
Agenda
• interRAI and interRAI Suite
• Applications of interRAI data -- QIs
• Conclusions
Final Comments
• interRAI Quality Indicators represent
– A large set of measures
– They cross multiple dimensions
– They challenge providers to improve functional
outcomes and relevant clinical complexities
• Choices have been made and as one considers the
local situation on the ground you can select from
among these measures – setting local standards
and comparisons to international standards
Final Comment
• Choices have been made and one can select
from among these measures to assess
program performance – setting local
standards and comparisons to international
standards