The complex evaluation
framework
Simple projects, complicated programs and
complex development interventions
Large,
complex
Complex
interventions
Complicated
programs
Small,
simple
Simple projects
• country-led planning and evaluation
• Non linear
• Many components or services
• Often covers whole country
• Multiple and broad objectives
• May provide budget support with no clear
definition of scope or services
• multiple donors and agencies
• context is critical
• May include a number of projects and wider scope
• Often involves several blueprint approaches
• Defined objectives but often broader and less
precise and harder to measure
• Often not time-bound
• Context important
• multiple donors and national agencies
•
•
•
•
•
•
“blue print” producing standardized product
relatively linear
Limited number of services
Time-bound
defined and often small target population
Defined objectives
2
The Special Challenges of Assessing Outcomes for
Complex Programs
1. Most conventional impact evaluation designs cannot
be applied to evaluating complex programs
2. No clearly defined activities or objectives
– General budget and technical support integrated into
broader government programs
– Multiple activities
– Target populations not clearly defined
– Time-lines may not be clearly defined
3
Special challenges continued
3. Multiple actors
4. No baseline data
5. Difficult to define a conventional comparison
group
Alternative approaches for defining the counterfactual for complex interventions
1. Theory driven evaluation
3. Quantitative approaches
4. Qualitative approaches
5. Mixed method designs
6. Rating scales
7. Integrated strategies for strengthening the evaluation
designs
5
STRATEGIES FOR EVALUATING COMPLEX PROGRAMS
Counterfactual designs
• Attribution analysis
• Contribution analysis
• Substitution analysis
Theorybased
approaches
Qualitative
approaches
Quantitative
approaches
Mixed
method
designs
Rating
scales
strengthening alternative
counterfactuals
• “Unpacking complex
programs”
• Portfolio analysis
•Reconstructing baseline data
• Creative use of secondary
data
• Secondary data
•Triangulation
Estimating
impacts
The valueadded of
agency X
Net increase in
resources for a
program