FRAME what is to be evaluated
Patricia Rogers, RMIT University, Melbourne
Part 3 of 8 AEA Coffee Break Webinars 2013

The Rainbow Framework

Why do we need to start with a clear definition?
Make Decision
Source: Hobbies on a Budget / Flickr

Why do we need to start with a clear definition?
Frame Decision Make Decision
Source: Hobbies on a Budget / Flickr

Why do we need to start with a clear definition?
Frame Decision Make Decision
Design Evaluation
Source: Hobbies on a Budget / Flickr

Why do we need to start with a clear definition?
Frame Decision Make Decision
Frame Evaluation Design Evaluation
Source: Hobbies on a Budget / Flickr

Four evaluation tasks in FRAMING
Identify primary intended users
Decide purpose(s)
(intended uses)
Specify key evaluation questions
Determine what
‘success’ looks like

1.

Possible primary intended users
Image source: IN 157s01 – Curt Carnemark/ World Bank
Families
Teachers
Principal
School
Council
Education
Department
Other schools

2.

Purposes
(intended uses)
Image source: CK-CO138 - Charlotte Kesl / World Bank
Formative – improve it
Summative – continue or stop it
Broader evidence base
Lobby and advocate

Purposes
(intended uses)
Provide voice
Build trust and legitimacy
Accountability

Interconnection between intended users and intended uses
FRAME
Identify primary intended users
Decide purpose(s)
(intended uses)

3.

Descriptive:
How many children attend?
What learning tools are used?
Has learning improved?
Key evaluation questions
Image source: ML030S09 - Curt Carnemark / World Bank

Causal:
Has the program contributed to improved learning?
Key evaluation questions

Synthesis:
Has the program been a success?
Is it Value For Money compared to alternatives?
Key evaluation questions

Action:
How can the program be improved? Should it continue?
Key evaluation questions

Options for answering different types of questions
MANAGE
DEFINE
FRAME
DESCRIBE Descriptive Questions-
What were the activities, changes, context?
UNDERSTAND
CAUSES
Causal questions –What caused or contributed to the identified changes?
SYNTHESIZE Synthesis questions –
Overall was it good? Value for money?
REPORT &
SUPPORT USE
Action questions-
What should we do?

4.

What does success look like?
Image source: LQ-CN7499 World Bank
Processes:
Are students fairly treated?

What does success look like?
Outcomes:
Do students learn to read?

What does success look like?
Distribution of costs and benefits:
Who benefits from the program? Whose needs are met? Who is disadvantaged by it?

What does success look like?
Criteria and standards:
Reading levels – better than before, or better than national average, or national benchmark?

You can read about this case here : http://betterevaluation.org/blog
/outcome_harvesting_bionet

Further resources http://betterevaluation.org/plan/ frame facebook.com/betterevaluation
@bettereval
Patricia Rogers
RMIT University patricia.rogers@rmit.edu.au