INVOLVING THE PUBLIC IN
ASSESSING HEALTH QUALITY
MEASURES
Shoshanna Sofaer, Presenter
Kirsten Firminger, Research
Associate
PRESENTATION OVERVIEW
•
•
•
•
•
•
Project
Research Team
Methods
Findings
Implications
The Sequel
December 2 2009
School of Public Affairs Faculty Colloquium
2
THE PROJECT
•
•
•
“Developing and testing nursing quality measures
with consumers and patients”
Funded by the Interdisciplinary Nursing Quality
Research Initiative (INQRI) of the Robert Wood
Johnson Foundation
Project had three aims; we will focus on the first:
Gauge public response to 15 nursing sensitive
quality measures endorsed by the National Quality
Forum (NQF)
December 2 2009
School of Public Affairs Faculty Colloquium
3
The Interdisciplinary Research Team
•
•
•
Shoshanna Sofaer, DrPH, Baruch College School
of Public Affairs (Baruch), Principal Investigator
Jean Johnson, PhD, RN, FAAN, George
Washington University School of Medicine and
Health Sciences (GW), Co-Principal Investigator
Ellen Dawson, PhD, RN, ANP (GW), CoPrincipal Investigator
December 2 2009
School of Public Affairs Faculty Colloquium
4
The Interdisciplinary Research Team
•
•
•
•
Kirsten Firminger, doctoral candidate,
(Baruch)
Christine Pintz, PhD, RNC(GW)
Andrea Brassard, DNSc, MPH, ANP(GW)
Kirsten and Christine actually ran the focus
groups we describe, coded the transcripts,
and analyzed the quantitative and
qualitative data
December 2 2009
School of Public Affairs Faculty Colloquium
5
Background/Motivation
•
•
•
Public reporting of quality measures is on the rise,
but many public reports are not looked at, not
understood and not used by the intended audience
We know that unless quality measures resonate
with the public, are easy to understand and apply,
they will not influence behavior
Yet many measures that meet technical criteria
have not been tested with the public
December 2 2009
School of Public Affairs Faculty Colloquium
6
Background/Motivation
•
•
•
•
The National Quality Forum is a non-profit membership
organization governed by multiple health care stakeholders
They utilize a complex process to reach consensus across
stakeholders on health care quality measures that meet
certain criteria (next slide)
When consensus is reached, the measures are “endorsed”
Many major sponsors of health care quality reporting will
only publicly report NQF endorsed measures, including the
Centers for Medicare & Medicaid Services
December 2 2009
School of Public Affairs Faculty Colloquium
7
Background/Motivation
•
The criteria used by NQF are:
– Importance to Measure and Report to make significant
gains in healthcare quality and improve health
outcomes
– Scientific Acceptability of Measure Properties, i.e.
validity and reliability
– Usability by intended audiences (e.g. consumers,
purchasers, providers, policy makers) i.e. can
understand and would find useful in decision making
– Feasibility of data collection without undue burden
December 2 2009
School of Public Affairs Faculty Colloquium
8
Background/Motivation
•
•
However, while extensive documentation is
required about the scientific reliability and validity
of measures, and feasibility of data collection, no
documentation is required to demonstration the
usability of measures to a major audience for
public quality reports: consumers
Instead, NQF relies on the views of members who
are considered to be “representing” consumers,
such as the AARP, the AFL-CIO, etc.
December 2 2009
School of Public Affairs Faculty Colloquium
9
Background/Motivation
•
•
•
This research was designed to investigate the
hypothesis that at least some endorsed measures
are, when actually tested with consumers, less
than usable
We hoped that if our hunch proved accurate, we
could use our results to influence the process by
which NQF endorses measures going forward
The focus of our work was a set of 15 measures
that were endorsed by NQF as being “nursing
sensitive” hospital quality measures: the NQF 15
December 2 2009
School of Public Affairs Faculty Colloquium
10
Research Methods
•
We conducted 9 focus groups with recently
hospitalized patients to determine if participants:
– Could understand and interpret the NQF 15
– Found these measures significant in the light of
their hospital experiences
– Thought the measures related to the roles,
responsibilities and functions of nurses in
particular
December 2 2009
School of Public Affairs Faculty Colloquium
11
Research Methods
•
Sampling:
– Three groups each in DC, Chicago and
Phoenix
– Three groups each with people who used
OB, medical and surgical services
– Participants recruited to vary in age,
gender, race/ethnicity, education level
December 2 2009
School of Public Affairs Faculty Colloquium
12
Research Methods
•
•
•
To maximize understanding of measures,
we developed more user friendly language
for the NQF 15 (see handout)
To ensure comprehension of the task, we
presented slides on quality measures
To gauge understanding, we asked
participants to explain the measure in their
own words
December 2 2009
School of Public Affairs Faculty Colloquium
13
Research Methods
•
Participants silently rated each measure on
their:
– Importance
– Likelihood of variation in performance across
nurses and hospitals
– Whether a very high or low score would lead
them to reconsider their hospital choice
•
After each rating, participants discussed
their responses and reasons for their ratings
in depth
December 2 2009
School of Public Affairs Faculty Colloquium
14
Findings
•
On silent ratings, at least 80% of participants
found the following measures very important:
– Failure to rescue
– Pressure ulcers
– UTI among catheterized patients
– Central line infections
– Pneumonia among patients on ventilators
– Positive nursing work environment
December 2 2009
School of Public Affairs Faculty Colloquium
15
Findings
•
•
Ratings showed participants were least responsive
to three measures about whether nurses advised
patients to stop smoking
Group discussion generated these reasons:
–
–
–
–
Nurses have better things to do with their time
Patients already know they shouldn’t smoke
Smoking is the patient’s responsibility
Nurses who are obviously not models of healthy
behavior are not credible as messengers to adopt them
December 2 2009
School of Public Affairs Faculty Colloquium
16
Findings
•
Many participants were ambivalent about the
measures relating to falls and restraints
– Who is responsible when patients fall, the nurse
or the patient?
– Don’t some patients need to be restrained?
– Do nurses sometimes restrain patients as
“retribution”?
– Won’t some patients fall if they aren’t
restrained?
December 2 2009
School of Public Affairs Faculty Colloquium
17
Findings
•
•
•
•
Measures of nurse-patient staffing ratios, nursing
mix and turnover rates, confused many
Nursing mix: What’s good? Measure doesn’t
“point to” the answer
Nurse-patient ratio: Of great interest, but the
measure was just not clear enough to consumers
“Rate at which nursing staff voluntarily end their
employment” – does this mean the turnover rate?
the number of vacancies?
December 2 2009
School of Public Affairs Faculty Colloquium
18
Findings
•
•
•
•
More broadly, these patients are fully aware of the
centrality of nurses to their hospital experience
They realize that in many cases, nurses share
responsibility with others, including physicians
and patients
More than anything, people want the nurse to “be
there” and to “be competent”
We can’t assume consumers know the “evidence
base” underlying the measures, for example that
restraints actually increase rather than reduce falls
December 2 2009
School of Public Affairs Faculty Colloquium
19
Implications
•
•
•
Does this lack of expertise mean we
shouldn’t bother asking the public their
views? NO!
The public has much to contribute to
assessing quality measures
Yet at the time of the study, NQF endorsed
measures even if there was no evidence of
the public’s response to them
December 2 2009
School of Public Affairs Faculty Colloquium
20
Implications
•
Getting public input before endorsement has
these benefits:
– Professional judgments about the absolute and
relative importance of individual measures will
be either reinforced or seriously questioned
– Issues of attribution of responsibility will be
surfaced
– Problems in measure specification that lead to
potential confusion can be resolved
December 2 2009
School of Public Affairs Faculty Colloquium
21
Implications
•
We can get guidance about
– When we need to explain the evidence
base to “legitimate” a measure
– What kind of “contextual” information
will be needed if measures are to be
publicly reported
– We can learn what else the public cares
about with respect to a particular issue
December 2 2009
School of Public Affairs Faculty Colloquium
22
The Sequel
•
•
•
NQF always takes measures through a periodic
review process; this took place in 2008-09 with
the NQF 15
We were able to present our findings in detail to
NQF staff, to the Hospital Quality Alliance, as
well as to many nursing and health services
researcher
As a result of this, and the review process, the
smoking measures are gone, the measures the
public liked most remain, and other measures are
being tweaked
December 2 2009
School of Public Affairs Faculty Colloquium
23
The Sequel
•
•
•
•
NQF has said they are moving toward changing their
requirements for measures
It remains to be seen if this will happen and what it will
look like
The good news: people involved with the review
welcomed getting the kind of information we provided,
and were surprised at how much it revealed
The remaining question: no one questions spending
money on statistical field testing of measures; will they
find the resources to do careful consumer testing as well?
December 2 2009
School of Public Affairs Faculty Colloquium
24