Evidence-Based Decision Making:
The Contribution of Systematic
Reviews in Synthesizing Evidence
General introduction to systematic reviews
1. What is a systematic review?
2. Why do/fund one?
3. For which policy questions is SR useful?
4. What are the main international bodies?
Systematic reviews as defined by the experts
• “A systematic, up-to-date summary of reliable
evidence of the benefits and risks of an
intervention” (The Cochrane Collaboration)
• Reviews “sum up the best available research on
a specific question” (The Campbell Collaboration)
• Defines a process, not a product. The process is
a systematic and transparent approach to all
phases of a literature review (Jeff Valentine).
Systematic reviews are research
• Systematic reviews are a form of research
– Secondary observations
– In which studies are the unit of analysis
• Follow basic steps in the research process
• Aim to minimize bias and error
Example: microcredit
• Evidence used to justify development policy is often
anecdotal:
“Small loans enable people in poverty to earn an income
and provide for their families… Each successful business
feeds a family, employs more people and eventually helps
empower a whole community” (Opportunity International)
• Where evidence is used it is often based on single
studies:
“Impact studies done on the Grameen Bank by independent
researchers find that 5% of borrowers come out of poverty
every year…the status of women has been enhanced”
(Yunus, 2005)
• Where a range of studies are used, not clear how
representative:
“Women participants in microcredit programs often
experience important self-empowerment… there is a strong
indication from borrowers that microcredit improves their
lives” (Microfinance Gateway)
So what do reviews synthesizing all the existing high
quality evidence say?
Duvendack et al. (2011):
‘All impact evaluations of microfinance suffer from weak methodologies
and inadequate data [which] can lead to misconceptions about the actual
effects of a microfinance programme’
‘It remains unclear under what circumstances, and for whom,
microfinance has been and could be of real, rather than imagined,
benefit to poor people’
Vaessen et al. (2012):
‘From those studies deemed comparable and of minimum acceptable
quality, we can conclude that overall the effect of microcredit on
women’s control over household spending is weak’.
Why do we need systematic reviews?
• Sheer amount and flow of
information/ research
• Variable quality of research
outputs
• Need to ‘separate the
wheat from the chaff’
• Problems of publication bias
• Limitations of single studies
What makes a systematic review ‘systematic’?
1. Scoping: defining answerable question, methods set
out in study protocol
2. Rigorous search to identify published and unpublished
sources, in any language
3. Application of study inclusion criteria (PICOS)
4. Critical appraisal of study quality, to assess how
reliable is the evidence
5. Data extraction and organisation
6. Synthesis of evidence (outcomes along causal chain)
7. Interpreting results (policy and practice, research
recommendations)
8. Improving and updating reviews as new evidence
emerges
Types of (answerable) questions
Policy area
Example question
Effectiveness: what ‘difference’ does an intervention
make?
What are the impacts of daycare on child health,
nutrition and cognitive development?
What is the comparative effectiveness of multiple
interventions in attaining an outcome?
What is the effectiveness of water, sanitation and
hygiene interventions in terms of reducing child
diarrhoea in LMICs?
Drivers of change: determinants of compliance or
adherence
What are the barriers and facilitators of patient
adherence to tuberculosis treatment in Africa?
What are the barriers to discontinuation of female
genital mutilation/ cutting in Africa?
Views: What are stakeholders’ experiences or
preferences?
What is the willingness to pay for clean water in
developing countries?
Cost-effectiveness or cost-benefit
What is the effectiveness and cost-effectiveness of
conditional and unconditional cash transfers in
improving education outcomes?
Risk factors associated with a condition or problem?
What are the predictive factors for youth
involvement in gang violence?
Resources and international bodies
• Cochrane Collaboration; 1993; www.cochrane.org
– Producing high quality information about the effectiveness of health
care (> 5000 published online – Cochrane library)
• Campbell Collaboration; 2000; www.campbellcollaboration.org
– Producing systematic reviews of the effects of social interventions
(>200 published online – Campbell library)
• International Development Coordinating Group (IDCG); 2010
www.campbellcollaboration.org/international_development
– Producing systematic reviews of high policy-relevance focusing on
social and economic development interventions in LMICs
• International Initiative for Impact Evaluation, 3ie; 2008
www.3ieimpact.org/en/evidence/systematic-reviews
• EPPI Centre - An Institute of Education centre focusing on
systematic reviews in education, health and social policy
• Collaboration for Environmental Evidence producing
systematic reviews for environmental management