Research Team
Evidence and Organizational
Decision Making:
Never the Twain Shall Meet?
Thomas Rundall, PhD
Academy Health, Annual Research Meeting
June 27, 2006
Introduction
There is a substantial gap between health services
decision makers and the research community
Studies of health organization decision makers in
Canada, the United Kingdom, and the United States
report that decision makers rarely use research evidence
The research community needs to understand the
reasons for this “gap”
gap” and work with managers and
policy makers to close it
John Hsu, MD, MBA, MSCE
Estee Neuwirth, PhD
Julie Schmittdiel, PhD
Laura Arroyo
Ilana Graetz
Kaiser Permanente Division of Research and Institute for Health Policy
Kaiser Permanente Care Management Institute
Thomas Rundall, PhD
Peter Martelli,
Martelli, PhD Candidate
Rodney McCurdy, MHA, PhD Candidate
U. C. Berkeley, Center for Health Research
Mark Gibson
Pam Curtis, MS
Oregon Health Sciences University, Center for Evidence Based Policy
Policy
Financial Support: Agency for Healthcare Research and Quality
Rationale for the Study
Source: Kovner and Rundall, EvidenceEvidence-based Management Reconsidered.
Frontiers of Health Services Management, Spring 2006.
Study Design
Qualitative
Four peerpeer-toto-peer discussion groups (full day)
Questions designed to elicit perceptions of
HighHigh-priority decisions
Current levels of research evidence use
Characteristics and definitions of useful evidence
Barriers and facilitators to using evidence
Recommended methods for communicating evidence
Using research evidence when making decisions about
the organization, financing and delivery of health care
has great appeal
Decision makers, however, do not appear to use this
information
We queried health organization decision makers about
their definition and use of evidence when making
strategic decisions
The findings of this study will help inform the
construction of an evidenceevidence-based management toolbox
Methods
Focus group discussions were held in San Francisco,
Oakland, Chicago, and New York
Purposive sampling of health care senior managers
26 organizations of various types
Private provider groups
Health insurance plans and insurers
Public health systems
Employers and employer groups
Regulatory agencies
Organizations with limited resources and wealthier ones
5-10 managers in each focus group (31 total)
1
Methods
Moderator posed questions and recorded major themes
Two researchers recorded all comments and compared
notes with meeting audiotapes
Using content analysis techniques, two researchers
independently coded the text
Initial 72.1% agreement on codes; discrepant codes
resolved
Investigators categorized codes into themes
Coded themes compared with moderators’
moderators’ themes
Results
Broad definition of acceptable evidence
Useful evidence must be
Internal data/measurements
Colloquial evidence
Externally dictated standards
Scientific evidence
Accurate
Applicable
Actionable
Accessible
Research (scientific) evidence often not useful
Disconnect between definitions and evaluation of
evidence
Results
Broad list of highhigh-priority decisions
Purchasing to improve quality and reduce costs
Organization of patient care
Organizational sustainability
Benefit design and coverage
Workforce recruitment development, retention
Limited use of research evidence in decision making
Difficulty distinguishing between evidenceevidence-based
medicine and evidenceevidence-based management
Results
Tools need to be concise and focused
Need for processes to help decision makers
appraise evidence
Desire for evidence that is “vetted”
vetted”
Lack of experience, resources, and incentives for assessing
evidence
Desire for tools and mechanisms for validating evidence
Conclusions
There is a substantial gap between health
services decision makers and the research
community
Decision makers rarely report using research
evidence and instead rely on other types of
information such as peers with experience.
Decision makers perceive that existing research
does not meet their needs, both in content and
accessibility.
Next Steps?
Build an evidenceevidence-based decision making tool box for
managers
Continue to foster collaborative research between health
services researchers and managers (e.g. The work of the
Center for Health Management Research)
Encourage national organizations to work with
appropriate information technology companies and
experts to improve access to relevant research
Training and rere-training of health services researchers to
improve their ability to conduct and communicate
research that will be used by health services managers
and policymakers
2
Next Steps?
Work with health organizations to build cultures,
processes, and structures that support the use of
research evidence in decision making
Recognize the prevalence of subsub-optimal management
decisions
Conceptualize, measure, and assess managerial decision
making
How about a new private/public partnership program:
TRIM (Translating Research into Management)
3
0
