Evaluation – Principles and
methods
Matt Barnard
Head of Evaluation
NSPCC
Purpose of evaluation
• Definition
“Examine how a policy or intervention was
designed and carried out and with what results.”
(Magenta Book)
• Asks objective questions
–
–
–
–
What were the impacts?
How was it delivered?
What were barriers and facilitators?
Did it deliver value for money?
• Aims to provide
– ‘Scientific’ basis for policy making
Evaluation Design Process
Evaluation
Design
Intervention
Design
Evaluation
implementation
Evaluation
findings
Logic
model
Intervention
implementation
planning
Intervention
implementation
Logic models
• Characteristics
–
–
–
–
Mechanisms not processes
Key steps not every step
Explanatory not descriptive
Reflects theoretical assumptions
• Benefits
– Sense check
– Identifies realistic outcomes
– Facilitates evaluation design
Strength
Strengthof
ofdesign
design matrix
Weak design
(Poor/ no
counterfactual)
Strong design
(Realistic
counterfactual)
Low power
(Small numbers/ effect
size)
Unlikely to detect
difference/
Low confidence in
attribution
Unlikely to detect
difference/
High confidence in
attribution
High power
(Large numbers/ effect
size)
Likely to detect
difference/
Low confidence in
attribution
Likely to detect
difference/
High confidence in
attribution
Strength – Evaluation design
Randomized controlled trial
Quasi-experimental design
Before and after measures
Types of design
• RCT
– Individual randomization
– Cluster randomization/ roll out
– BAU/waiting list/alternative services
• Quasi-experimental designs
–
–
–
–
Matched area/ groups
Matched individual
Interrupted time series
Regression discontinuity
Factors influencing
methodology
• Intervention stage of development
– Early exploration
– Defined and established but not proven
– Transferability
• Potential Costs and benefits
– Resources
– Timescales
Key principles
• Clarity about key question
– Avoid ‘default’ questions
• Methods matched to question
– Ensure methods match desired questions
• Claims match evidence
– Avoid over-claiming
• Have a coherent story to tell