C2 Training: May 9 – 10, 2011
Introduction to Systematic Reviews
The Campbell Collaboration
www.campbellcollaboration.org
Systematic Review Methods
• Systematic reviews are a form of research
– secondary observations
– in which studies are the unit of analysis
• Follow basic steps in the research process
• Aim to minimize bias and error
– But SRs are not immune to bias and error (not a
panacea)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Stages of a research synthesis (Cooper, 1982)
The seminal article outlining five stages of a research
review is by Harris Cooper:
• Cooper, H.M. (1982). Scientific guidelines for
conducting integrative research reviews. Review of
Educational Research, 52, 291-302.
• Cooper, H. M. (2009). Research synthesis and metaanalysis: A step-by-step approach. Thousand Oaks,
CA: Sage.
3
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Stages of a research synthesis (Cooper, 1982)
• Problem formulation
– Clarifying your questions and writing a protocol
– Set explicit inclusion/exclusion criteria
• Data collection
– Literature search
– Information-gathering from studies
• Data evaluation
– Criteria for including and excluding studies
– Assessing study quality
4
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Stages of a research synthesis (Cooper, 1982)
• Data analysis and interpretation
– Integrating the effects from collected studies
– Interpreting analysis results
• Report preparation
– Narrative, statistics, graphs, tables
5
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
What’s required?
• A team with
– Substantive expertise
– Methodological expertise
– Statistical expertise
– Information retrieval expertise
• Time and money
– SRs are labor intensive
– $50-$150K depending on scope, complexity, and number
of studies
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Protocols for SRs
• A detailed protocol (plan) for the SR should be developed
and made available to readers (Higgins & Green, 2008;
Moher et al., 2009)
– Protocols increase transparency, limit ad hoc
decisions
• The review process is iterative and plans may change
during the process
– The final report should document and explain changes
made (deviations from the protocol)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Stages of a research synthesis
• Most of the work involved in conducting a review is not spent
in statistical analysis.
• The scientific contribution of the final product is dependent
on all stages of the review and not just the statistical analysis
stage.
8
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
An example of the stages of a review
We will illustrate each of the stages by considering the C2
systematic review on Multisystemic Therapy (MST):
Littell, J.H., Popa, M., & Forsythe, B. (2005). Multisystemic
Therapy for social, emotional, and behavioral problems in
youth aged 10 – 17. (Campbell version)
9
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Stage 1
Problem Formulation
10
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Problem formulations
• Conceptual issues arise in attempts to combine results of
studies that vary (to some degree) in their methods,
treatments, samples, outcome measures
– Apples, oranges, and other fruits
– Parallel problems in studies of individuals (no 2 people
are identical and there is no “average” person)
• Given variation in primary research
– What should be included in a synthesis?
– How should we synthesize results?
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
SRs vary in scope
• Specific, narrow questions
– Useful for testing effects of specific treatments
• Broad, global questions
– Useful for generating new knowledge
• Identify common elements of effective programs
(Lipsey, 2008)
• build better intervention theories to guide program
development and evaluation design (Lipsey, 1997)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Scope of Systematic Reviews
• Not limited to questions about effects
– Can address trends, epidemiology, accuracy of diagnostic
and prognostic tests
• Not limited to randomized controlled trials or quantitative
data
– Qualitative synthesis (e.g., meta-ethnography, narrative
analysis of qualitative research reports)
– Mixed/multiple methods synthesis (e.g., Thomas, Harden,
et al. on progams to combat childhood obesity)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
What kinds of research questions can be
asked in a systematic review?
• Questions about intervention effects:
– What are the effects of x intervention on y
outcomes for z populations/problem?
– Variations on this theme (e.g., differences in
effects of interventions x1 vs x2)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Kinds of research questions (continued)
• Questions about associations
– How does x1 relate to x2 for population z? (direction and
strength of correlation)
– Variations on this theme (e.g., differences in relation of x1
and x2 between populations z1 and z2 )
• Diagnostic/Prognostic questions
– Which test (A vs. B) is a better predictor of y?
– Which test (A vs. B) is a better predictor of y for z1 vs. z2
populations?
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Problem formulation
• To assess the impacts of MST on out-of-home living
arrangements, crime and delinquency, and other behavioral
and psychosocial outcomes for youth and families.
16
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Problem formulation
• What research evidence will be relevant to the problem or
hypothesis of interest in the synthesis?
– Studies where youth (age 10-17) with social, emotional, and/or
behavioral problems were randomised to licensed MST programs
or other conditions (usual services or alternative treatments).
17
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Problem formulation
• Define the (a) variables and (b) relationships of
interest so that relevant and irrelevant studies can be
distinguished.
Multisystemic Therapy (MST) is a multi-faceted, short-term, home- and
community-based intervention for families of youth with severe
psychosocial and behavioral problems. Based on social ecological
and family systems theories, and on research on the causes and
correlates ofserious antisocial behavior in youth (Henggeler 1998,
Henggeler 2002a), MST is designed to address complex
psychosocial problems and provide alternatives to out-of-home
placement of children and youth.
18
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Problem formulation (continued)
• Measures of behavioral, psychosocial, and family outcomes
were examined.
– Behavioral outcomes included antisocial behavior (as measured by arrest or
conviction of a criminal offense), drug use (self-reports and drug tests), and
school attendance.
– Psychosocial outcomes included measures of psychiatric symptoms (on
standardized scales),school performance (teacher reports), peer relations
(self-reports and parent or teacher reports),and self esteem.
– To assess the impacts of MST on out-of-home living arrangements, crime
and delinquency, and other behavioral and psychosocial outcomes for youth
and families.
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Step two: Establish study inclusion and exclusion
criteria
• The PICOS framework:
– Population/Participants (problems/conditions)
– Interventions (if applicable)
– Comparison group (e.g., absolute vs. relative effects,
counterfactual conditions)
– Outcomes (primary and secondary outcomes, acceptable
outcome measures)
– Study Design (should be fit for purpose)
Geographic area, time, language, other criteria
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Participants: Who is included in the
sample of subjects?
• Why 10 - 17 only?
• Exclude special populations?
• Exclude studies with only males or females?
• Include families of these children?
• What is the rationale for the choice?
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Interventions: How do we define MST?
Comparisons: What is the comparison?
• What kinds of programs are considered MST?
• What types of treatments are the “right” comparisons to
MST?
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
What are the Outcomes?
• Research question states:
– Placement in out-of-home living arrangements
– Crime and delinquency
– Behavioral outcomes
– Psychosocial outcomes
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Study Designs: What types of study
designs are relevant for our review?
• If the research question is the assessment of the impacts of
MST on out-of-home living arrangements, crime and
delinquency, and other behavioral and social outcomes for
youth and families, what designs are most appropriate?
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
What studies are relevant?
• Will we attempt to generalize to all studies done?
• Will we look only at “high quality” studies?
• Will we look at both published and unpublished?
• Will we set a time period for the studies?
• Will we look at only randomized controlled trials or will we
include quasi-experimental studies? Why?
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Study eligibility criteria
• Usually focus on inclusion criteria, with few
exclusion criteria
• Studies should not be excluded based on:
– Sample size (or statistical power
considerations)
– Publication status
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
What kinds of studies should be included?
Garbage in, garbage out
• Synthesis of invalid studies produces invalid
conclusions
• What constitutes credible evidence?
– Depends on the question
• Need to set clear standards
– Based on methodological principles
– Relevant for topic and context
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Small group exercise: Problem formulation
• Form a working group of 2 – 5 people and develop a
preliminary research question for a comprehensive review
• Elect a recorder for the group who will report back to the
whole
• Each group will share the research question, and any
challenges that they will face in studying this question
• See the first exercise on problem formulation
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org