Center for the Study of
Healthcare Provider Behavior
Synthesizing Quality Improvement Research:
Methodological & Empirical Challenges & Solutions
Brian S. Mittman, PhD
Center for the Study of Healthcare Provider Behavior
VA Greater Los Angeles Healthcare System
AcademyHealth Annual Research Meeting
June 27, 2005
Co-authors

Marjorie Pearson, PhD
RAND

Patricia Parkerton, PhD
UCLA School of Public Health

Nancy Takahashi, MPH
VA Greater Los Angeles Healthcare System

Nina Smith, MPH
VA Greater Los Angeles Healthcare System
Research synthesis in EBM

Quantitative research synthesis (meta-analysis) is a
foundation of evidence-based medicine (EBM)
- combine, reconcile seemingly inconsistent findings
- leverage small studies with limited power

Although originating in education research, metaanalysis proved extremely useful–and has been
extensively refined—in clinical research
Research synthesis in other fields

The value and power of meta-analysis in clinical
research have prompted its application to healthcare
quality improvement research, and renewed use in
education, criminal justice and other areas of
management, policy and behavioral/social research

The core assumptions and requirements for metaanalysis are often adequately met in clinical research,
but rarely in other fields

Three key questions:
- Which requirements are not met in QI research?
- How serious are the gaps?
- How should we respond?
Framework and key source
“Challenges of Summarizing Better Information for Better
Health: The Evidence-based Practice Center Experience”
Annals of Internal Medicine 142(12) Pt 2, June 21, 2005
Challenges in Systematic Reviews of …









Complementary and Alternative Medicine
Diagnostic Technologies
Delivery, Organization and Financing of Health Care
Drug Efficacy or Effectiveness
Economic Analyses
Educational Intervention Studies
Treatment Harms
Therapeutic Devices and Procedures
Nonrandomized Studies
Identifying and gathering evidence

Publication bias
− likelihood of preparation of manuscript
− selective reporting of outcomes
− likelihood of acceptance

Types of evidence
− dearth of RCTs
− questionable value of observational evidence
− limited, variable documentation of process, impact
outcomes
Variable reporting and coding

Intervention (main effect)
− insufficient specification of intervention
− intervention flexibility (customization)
− variability (skill dependence)

Contextual factors (effect modifiers)
− lack of recognition (attention) within protocol
− lack of measurement
− limited reporting even if measured (space
constraints, exclusion from analysis)
Heterogeneity of studies

Equivalence of quality problems (diagnoses treated),
i.e., comparability of “therapeutic aims”

Intervention (main effect)
− intervention flexibility (customization)
− variability (skill dependence)
− efficacy vs. effectiveness studies of QI programs

Prevalence of intervention packages
− separability of components; additive vs. interactive
effects
− comparability of unique packages
Quality of evidence

Threats to validity
− contamination (patient-, clinician-, clinic-level
randomization)
− Hawthorne effects, expectation effects
− short-term follow-up
− limited or no blinding
− PI role in intervention; intervention/evaluation team
overlap
− non-validated instruments
− variability in assessment methods
− other data/measurement flaws
Quality of evidence (cont.)

Use of non-RCT evidence
− evolving quality standards for observational evidence
− evolving standards for qualitative data

Site selection, representativeness
− Voluntary vs. random selection
− QI site selection representativeness paradox:
random selection = less representativeness
Key problems

Lack of clearly-defined, universally accepted concepts,
terms for interventions, contextual factors, effects
(process, outcome)

Lack of quality standards and guidance for study
designs, methods

Lack of guidance in identifying (and measuring) key
contextual factors

Lack of standards for consistent reporting

Evolving, developing methods for qualitative synthesis
Short-term solutions:
QUERI Research Synthesis Project

Quantitative and qualitative synthesis of 54 VA-funded
QI studies

Data sources include public documents, confidential
documents, in-depth interviews

Evidence tables (and data collection forms) cover a
wide range of intervention, contextual and outcome
variables
Long-term solutions

Consensus efforts to establish:
− shared definitions of key concepts and terms
− quality standards and guidance for study designs,
methods
− list of key contextual factors
− consistent measures of key contextual factors
− reporting standards and templates

Continued development of methods for qualitative
synthesis