Translating Emerging Evidence:
Lessons from the MATCH Study
(Methods for Developing Actionable Evidence for Consumers of
Health Services Research)
AcademyHealth Annual Research Meeting
27 June 2006
Research Team
John Hsu, MD, MBA, MSCE
Thomas Rundall, PhD
Mark Gibson
Pam Curtis, MS
Laura Arroyo
Ilana Graetz
Estee Neuwirth, PhD
Julie Schmittdiel, PhD
Peter Martelli, PhD Candidate
Rodney McCurdy, MHA, PhD Candidate
Kaiser Permanente Division of Research, Institute for Health Policy, and Care
Management Institute
University of California, Berkeley, Center for Health Research
Oregon Health Services University, Center for Evidence Based Policy
Funding Source: Agency for Healthcare Research and Quality
Background
• Despite the potential value of using research
evidence for organizational decisions, decisionmakers rarely use evidence
• Information needed on the reasons for the gap
between research generation and use
Study Overview
• Objectives: To identify methods for closing the
gap between the creation and use of evidence
– Determine the evidence needs of consumers of
health services research and effective methods for
communicating evidence
– Develop tools to help organizational decision-makers
Study Methods
• Qualitative study
• Dynamic, semi-structured focus groups
– Four focus groups with 31 senior health care managers from a range of
organizations (private and public) and regions (West, Midwest, East)
– Full day sessions with moderator
• Discussion focus:
–
–
–
–
–
High-priority strategic decisions
Current levels of research evidence use
Defined characteristics of useful evidence
Organizational barriers & facilitators of using evidence
Suggestions for improving the use of evidence
• Content analysis used to synthesize responses
– Codes
– Themes
Limited Use of Evidence
• Use of research evidence is limited
• Evidence does not meet decision-makers’ needs, e.g. no actionable
findings or discussion of applicable situations
“For the vast majority of questions I have, I have found the answers are
not out there. Research is not being done in a way that answers my
questions.”
“What people really want is not the evidence, they want the answer.”
• Range of organizational barriers to using evidence cited:
– Culture (values, beliefs, priorities, incentives)
– Resources (money, time, technology, data)
– Capacity (lack of experience and expertise for assessing evidence)
Definitions of Evidence
•
Broad definition of evidence:
– Colloquial, experience-based, and tacit knowledge
– Pilot and case studies, and qualitative assessments of operational or strategic
decisions by experts
– Internal measurements
– Traditional scientific evidence
•
Criteria for useful evidence:
–
–
–
–
•
Perceptions that research evidence often not useful
–
–
–
–
–
•
Accurate
Applicable
Actionable
Accessible
Not the right question for a decision
Different set of standards, e.g. bias vs. certainty, timeliness
Not applicable to the organizational context and environment
Not actionable findings
Limited accessibility
Disconnect between definitions and evaluation of evidence
Recommendations
• Presentation of evidence addresses four criteria:
– Accuracy (e.g., establishes causal relationship, credible
sources)
– Applicability (e.g., relevant to management decision, setting
and situation)
– Actionable items (e.g., information on what needs to be done
and implications, fits into time frame)
– Accessibility (e.g., easy to obtain - “at our fingertips”)
• Characteristics of effective communication:
– Emphasizes main points (4 A’s)
– Uses multiple dissemination approaches, e.g. websites and
conferences
– Linked to a trusted information source, e.g. a clearinghouse of
vetted evidence
Decision-maker Comments
“One of the things we struggle with is having concise information.
[Something that is] not 300 pages. Formatting too. Ease of use is
really important.”
“There’s an issue of transparency [for research and knowledge
brokers, e.g. consultants] - both in the evidence and in the way it is
gathered. Most people don’t have the resources to do this on their
own.”
“We want a deeper understanding of the sources of evidence and
vetting that evidence. Management research is more empirical. Can
we get evidence in a timeframe that is actionable?”
“If you don’t make evidence available, easily available, in the face of
the person, at the moment that they need it, they don’t get it. They’re
too rushed.”