DEFINE what is to be evaluated
Simon Hearn, ODI, London
Part 2 of 8 AEA Coffee Break Webinars 2013
The Rainbow Framework
Why do we need to
start with a clear
definition?
Source: Hobbies on a Budget / Flickr
Three evaluation tasks in DEFINING
Identify
potential
unintended
results
Develop
program
theory or
logic
model
Develop initial
description
1.
Develop initial description
Peak
Experience
Thumbnail
description
Aims &
Objectives
Vision &
Mission
2.
Develop program theory
or logic model
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
INPUTS
• Apples
• People at risk of
poor health
IMMEDIATE
RESULTS
SHORT-TERM
RESULTS
LONGER-TERM
RESULTS
• Apples eaten
• Improved
nutritional status
• Improved health
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
RESULTS AREA
INDICATOR
MEANS OF
VERIFICATION
ASSUMPTIONS/
RISKS
GOAL:
Decreased
sick days
School and work
records
Good status largely
due to nutrition
Adequate
levels of
Vitamin C
Sample blood tests
Apples retain
Vitamin C
Apples
replaced by
apple cores
Visual inspection
Apples eaten on
location
Required
numbers
delivered
Delivery receipts
Required
numbers
bought
Purchase receipts
Visual inspection
Improved health
PURPOSE:
Improved nutritional
status
OUTPUT:
Apples eaten
ACTIVITIES:
Deliver apples
ACTIVITIES:
Buy apples
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
Outcomes chain showing possible
alternative causal paths
Improved health
Improved
vitamin C levels
Improved
quercetin levels
Decreased Body
Mass Index
Apples eaten
(whole or juice)
Red apple skin
eaten
Apples eaten
instead of junk
food snacks
Apples delivered
to schools
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
Context
Mechanism
Outcome
Healthy people at high risk of
gastric cancer
Quercetin from red apple skin
(or red onion) protect cells from
free radicals
Reduced risk of cancer
Smokers or people with regular
exposure to radiation
Chemicals in cigarette smoke or
radiation damages DNA in
healthy cells
Continued increased risk of
cancer
Individuals with vitamin
deficiency through malnutrition
Sufficient vitamin C absorbed
when juice from apples (or
oranges) consumed
Stronger immune system
Individuals with vitamin
deficiency through excessive
alcohol
Alcohol destroys vitamin C
No change in vitamin C level
Individuals with obesity due to
excessive snacking on high fat,
high calorie foods
Decreased calorie intake due to
substitution of apples (or carrot
sticks)
Reduced obesity and related
conditions
Decrease sugar intake through
substitution of apples
No change in obesity but lower
risk of diabetes
Individuals with obesity due to
genetic condition
Context
Mechanism
Outcome
Healthy people at high risk of
gastric cancer
Quercetin from red apple skin
(or red onion) protect cells from
free radicals
Reduced risk of cancer
Smokers or people with regular
exposure to radiation
Chemicals in cigarette smoke or
radiation damages DNA in
healthy cells
Continued increased risk of
cancer
Individuals with vitamin
deficiency through malnutrition
Sufficient vitamin C absorbed
when juice from apples (or
oranges) consumed
Stronger immune system
Individuals with vitamin
deficiency through excessive
alcohol
Alcohol destroys vitamin C
No change in vitamin C level
Individuals with obesity due to
excessive snacking on high fat,
high calorie foods
Decreased calorie intake due to
substitution of apples (or carrot
sticks)
Reduced obesity and related
conditions
Decrease sugar intake through
substitution of apples
No change in obesity but lower
risk of diabetes
Individuals with obesity due to
genetic condition
http://betterevaluation.org/resource/example/rubrics-oakden
3.
Identify potential
unintended or
negative results
Source: www.smarttoolkit.net
Negative
program
theory
Key
informant
interviews
Six Hats
Thinking
Risk
assessment
Defining – Framing iteration
Initial
description
Agree
scope of
evaluation
DEFINE
Discuss
evaluation
purpose
FRAME
FRAME
Develop
logic model
Refine
description
DEFINE
DEFINE
Define
evaluation
questions
FRAME
http://betterevaluation.org/plan/define
facebook.com/betterevaluation
@bettereval
Simon Hearn
Research Fellow, ODI
s.hearn@odi.org.uk