Campbell Methods Group
Protocol and Review Guidelines
Campbell Methods Group
Terri D. Pigott, Editor
Emily Tanner-Smith, Associate Editor
Joshua R. Polanin, Managing Editor
Constituted May 2012
Guidelines for Methods Section of Protocols and Reviews
At a minimum, the methods section of a protocol should include the following information:
(Hold control to use hyperlinks.)
I.
Inclusion criteria
a. Types of study designs
b. Types of participants
c. Types of interventions
d. Types of outcome measures
e. Any exclusion criteria
II. Search Strategy
III. Description of methods used in primary research
IV. Criteria for determination of independent findings
V.
Details of study coding categories
a. Study quality or risk of bias coding categories
VI. Proposed Statistical procedures
VII. Treatment of qualitative research
At a minimum, the methods section of a review should include the following information:
I.
Inclusion criteria
a. Types of study designs
b. Types of participants
c. Types of interventions
d. Types of outcome measures
e. Any exclusion criteria
II. Search Strategy
III. Criteria for determination of independent findings
IV. Details of study coding categories
a. Study quality or risk of bias coding categories
V.
Statistical procedures
VI. Treatment of qualitative research
Other Important Topics:
A. Advanced Methodological Topics
B. Helpful Online Resources
Inclusion Criteria
Methods
This section should clearly describe the criteria that will be used to determine whether identified
studies are eligible for inclusion in the systematic review. This includes but is not limited to
inclusion/exclusion criteria related to eligible populations, interventions, comparisons,
outcomes, and other study context characteristics. When editors of the protocol or review
examine the inclusion criteria they will be critically evaluating the reviewers’ notions on how
concept or intervention definitions correspond with research operations and outcomes.
Considerable debate about the conclusions of a particular review may focus on these decisions
by the reviewers. Some editors (and readers of the completed review) may find that the inclusion
criteria are too broad – for example, they may believe that some intervention realizations and/or
outcome measures are unrelated or irrelevant to the goals of an intervention. The reviewers
should anticipate these concerns and propose to employ distinctions in outcomes as a potential
moderator of study results (see below). Other editors may find the intervention and outcome
definitions are too narrow. This may lead them to examine excluded studies to determine if they
should, in fact, be included in the review. When evaluating this section, peer reviewers will
answer such questions as:







What characteristics of studies will lead to exclusion?
Are the inclusion/exclusion criteria excessively broad or narrow for the given
research question?
Do the authors provide justification for inclusion and exclusion criteria?
Will eligibility decisions be based on a reading of report titles? Abstracts? Full
reports?
Who will make the eligibility decisions? How will the authors handle disagreements
about inclusion and exclusion?
How will the reliability of relevance decisions be assessed?
Do the authors provide an example of a study that is eligible for inclusion, and one
that is not eligible for inclusion?
Resources for Authors:
C2 training materials on problem formulation
Cochrane Handbook - Chapter 5: Defining the review question and developing criteria for
including studies
Petrosino, A.J. (1995). Specifying Inclusion Criteria for Meta-Analysis. Evaluation Review, 19,
274-295.
Top
Search Strategy
This section should present the details of the proposed or completed search for studies. This
should include a listing and description of the search strategy to be used to retrieve studies (e. g.,
reference databases, personal contacts, hand searches of journals). Further, the protocol/review
should include a rationale for the choice of literature sources, especially with regard to how
different sources will be used to complement one another in order to reduce potential
differences in the outcome of retrieved studies versus those that may go undiscovered. The
reviewers need to report the years covered and the keywords that will guide the search for
reference databases and bibliographies. Information on the sources, keywords and the keyword
searching strategies, and years covered by the literature search is a crucial aspect of the review
methods. It gives the editors the best indication of the thoroughness and potential biases of the
search and, therefore, how much credibility could be placed in the conclusions of the proposed
review. Reviewers also need to describe the mechanisms they intend to use to retrieve
potentially relevant documents, especially ones that are unpublished. In terms of future
attempts at replication, it is the description of the literature search that will be examined first
when scholars attempt to understand why different reviews on the same topic area have come to
similar or conflicting conclusions. As far as it is possible, reviewers should take an international
perspective when searching the literature. The evidence in a review should not be restricted by
the nationality of investigators or samples or by language unless good justification exists. Peer
reviewers will answer such questions as:







Were all relevant electronic databases searched?
Do the authors provide complete search terms for at least one electronic database?
Does the search allow adequate combinations of search terms for specific databases?
Does the search strategy target specific journals, if appropriate?
Does the search strategy include a time frame, including the month and year last
searched for all databases?
Does the search strategy adequately attempt to locate unpublished and “grey”
literature?
Does the search strategy include international data sources, or a reasonable
justification otherwise?
Resources for Authors:
C2 training materials on searching
C2 information retrieval policy brief
Cochrane Handbook - Chapter 6: Searching for studies
Campbell Collaboration Online Webcast: Literature Searching
Top
Methods Used in Primary Research (Protocols Only)
This section of a protocol should describe methodologies often found in the primary research
covered by the review. This section focuses more on participant sampling procedures, research
design, and measurement techniques than on conceptual intervention realizations. The
reviewers should choose a few studies that exemplify the methods used by many studies and
present the details of these investigations. This is important because it helps to clarify the
justification for inclusion. Often peer reviewers or editors will not be familiar with the
substantive area and this section will enumerate the typical methods utilized in these areas.
Resources for Authors:
C2 research design policy brief
Top
Criteria for Determination of Independent Findings
This section should describe how the reviewers intend to handle circumstances in which a single
evaluation of effectiveness provides data on multiple outcome measures. This can happen
because several types of outcomes are measured within the same study (e.g., recidivism and
school attendance within a study of intervention effects on juvenile delinquency) and/or because
the same outcome is measured at multiple points in time. In such cases, the outcome measures
are taken on the same sample of participants, and therefore, are not independent estimates of
intervention or treatment effect. An explanation of the criteria used to determine whether
multiple outcomes from the same or related evaluations are independent data points should be
carefully spelled out.
Resources for Authors:
Finding What Works in Health Care: Standards for Systematic Reviews (Page 180)
Top
Coding Procedures
This section should describe the characteristics of the included studies that will be retrieved and
retained for further examination. The study outcomes of interest also should be described, and if
some outcome measures are excluded a rationale should be given for this decision. All retrieved
characteristics should be mentioned even if some of these characteristics eventually are not
formally tested or discussed in the final paper. Authors should describe any information they
plan to code to assess study quality in this subsection. This will alert the editors to
characteristics reviewers might have omitted but believe are important to include. This
subsection should also include information on how coding reliability will be established and
monitored. The peer reviewers will answer such questions as:





Is a codebook included in the review?
Does the codebook include relevant codes for all general study, study context,
sample, outcome, design, and effect size level variables?
Does the review plan to code items to assess the quality or risk of bias of studies?
(This might include information on design, unit of assignment, unit of analysis,
attrition, fidelity of intervention, etc.)
Does the review describe the number of coders at both the eligibility screening and
full text coding stages?
Does the review describe how reliability between coders was assessed at the eligibility
screening and full text coding stages? Does it describe how any disagreements or
problems with coder reliability are handled?
Resources for Authors:
C2 training materials on coding
More C2 training materials on coding
Dave Wilson’s materials on database structure
Cochrane Handbook - Chapter 7: Selecting studies and collecting data
Cochrane Handbook - Chapter 8: Assessing risk of bias in included studies
Cochrane training materials on collecting data from relevant studies
Campbell Collaboration Online Webcast: Coding
Top
Statistical Procedures
This section should outline the procedures and conventions the reviewers will use
to carry out the quantitative analysis of results. This important section details how the reviewers
synthesize primary study results. Many conventions remain available to the reviewer and make
it necessary for deliberate description. Greater clarity and description will help the editors and
peer reviewers understand how conclusions are formed. Future reviewers will dissect this
section for appropriate comparisons. Further, reviewers should include a discussion on the
effect size metric and weighting procedures, heterogeneity statistics, and multivariate statistics.
The reviewers should also enumerate explicitly the procedures of moderator and metaregression analyses. Any a prior hypotheses should be detailed and the procedures for
conducting these analyses should be examined. Finally, the reviewer will describe the software
package of choice. The peer reviewers will answer questions such as:












What effect size metric(s) was used? Did the review provide justification for the
chosen effect size?
What weights were used? Does the author provide convincing rationale for using
fixed or random effects weights?
Does the review outline clearly any conversions that were needed to calculate any
effect sizes?
What techniques will be used to combine results of separate tests? If no
quantitative synthesis of results will be carried out, what was the rationale for the use
of the alternative approach?
What techniques were used to assess and then analyze the variability in findings
across tests?
Were any adjustments to effect sizes used to remove bias?
How was missing handled?
How were outlier cases handled?
What sensitivity analyses (i. e., tests of the impact of such decisions on the results of
the review) will be carried out and how?
What publication bias analysis will be carried out?
How did the reviewers handle any dependent effect sizes (e.g., multiple effect sizes
on similar measures from the same study)?
What software package was used to conduct the analyses?
Resources for Authors:
Computing effect sizes:
C2 training materials on computing effect sizes
More C2 training materials on computing effect sizes
More C2 training materials on computing effect sizes
Dave Wilson’s materials on computing effect sizes
Dave Wilson’s Effect Size Calculator
ES – Effect size calculator
Campbell Collaboration Online Webcast: Effect Size Calculation
Campbell Collaboration Online Webcast: Calculating Effect Sizes, Advanced
Campbell Collaboration Online Webcast: Effect Size Cluster Adjustments
Top
Statistical Analyses (Completed Reviews Only)


















What software package was used to conduct the analyses?
What effect size metric(s) was used? Did the review provide justification for the
chosen effect size?
What weights were used? Does the author provide convincing rationale for using
fixed or random effects weights?
Does the review outline clearly any conversions that were needed to calculate any
effect sizes?
What techniques will be used to combine results of separate tests? If no
quantitative synthesis of results will be carried out, what was the rationale for the use
of the alternative approach?
What techniques were used to assess and then analyze the variability in findings
across tests?
Were any adjustments to effect sizes used to remove bias?
How was missing handled?
How were outlier cases handled?
What sensitivity analyses (i. e., tests of the impact of such decisions on the results of
the review) will be carried out and how?
What publication bias analysis will be carried out?
How did the reviewers handle any dependent effect sizes (e.g., multiple effect sizes
on similar measures from the same study)?
Are confidence intervals provided along with all point estimates?
Are graphical displays of study effects presented when appropriate?
Do the authors appropriately examine and/or model heterogeneity in effect sizes,
when appropriate?
Do the authors provide justification for using statistical moderator analysis?
Do the authors use the appropriate publication bias methods for their data?
Do the authors discuss the quality and/or risk of bias in studies?
Resources for Authors:
Synthesizing effect sizes:
C2 training materials on combining effect sizes
Cochrane Handbook - Chapter 9: Analysing data and undertaking meta-analyses
Cochrane training materials on summary statistics for dichotomous outcome data
Cochrane training materials on combining studies
Campbell Collaboration Online Webcast: Fixed vs. Random Effects
Software resources:
Comprehensive Meta-analysis
SPSS macros for synthesizing effect sizes
Stata macros for synthesizing effect sizes
SAS macros for synthesizing effect sizes
Stata ado files for synthesizing effect sizes
R package for synthesizing effect sizes
R package for synthesizing effect sizes
Macros for robust standard error estimation
MIX software for Excel
Top
Advanced Methodological Topics
Although in the past rarely used, advanced methodological topics are increasingly becoming
popular among meta-analysts. As such, if the review will use these techniques, the reviewer
should take pains to describe these issues with advanced clarity. Often these techniques are not
well-known to the editors or peer reviewers and therefore should be described with little
assumptions of prior knowledge. Greater clarity will provide the editors and peer reviewers the
opportunity to critique the review properly. The peer reviewer will answer such questions as:


Does the review include advanced synthesis techniques (e.g., Bayesian meta-analysis,
diagnostic/prognostic test accuracy, individual participant data, multivariate metaanalysis, etc.)?
Do the authors provide justification for the use of these advanced techniques?
Campbell Collaboration Online Webcast: Using Robust Standard Errors
Top
Treatment of Qualitative Research
In the context of a Campbell review, qualitative studies in the relevant field can (a) contribute to
the development of a more robust intervention by helping to define an intervention more
precisely, (b) assist in the choice of outcome measures and assist in the development of valid
research questions, and (c) help to understand heterogeneous results from studies of effect.
When a review contains descriptions of qualitative research relevant to the topic of interest, the
reviewers should operationally describe the (a) criteria for inclusion and exclusion of studies, (b)
methods used in primary research, (c) criteria for determining independent findings and (d)
characteristics of included studies in the same detail as they do for quantitative research.
Resources for Authors:
Finding what works in health care: Standards for Systematic Reviews (Page 173)
Top
Other General Methods Resources for Authors:
C2 training materials on advanced topics
C2 methods policy briefs
Centre for Reviews and Dissemination guidelines for undertaking systematic reviews
Single subject design research methods links
Meta-analysis in Stata
Meta-analysis in R
Joanna Briggs Systematic Review Software
Research Synthesis Methods Journal
Systematic Reviews Journal
Top