WHAT IS THE EVALUATION RESOURCE PACK?
The evaluation resource pack is a comprehensive and user-friendly slide-show
which provides step by step evaluation guidance and resources. It was produced
by Justice Analytical Services at the Scottish Government to help evaluators and
funders conduct better quality evaluations to assess the value of their interventions.
The main propose of the full version evaluation resource pack is to,
•
Emphasise the importance of using the evidence-base to design interventions.
•
Promote a rigorous 4 STEP approach to evaluation which interventions of ALL SIZES
and at ALL STAGES OF DEVELOPMENT can conduct.
•
Help interventions carry out a realistic and rigorous alternative to impact evaluations
which are very difficult to do in Scotland
•
Describe how to structure an evaluation report to increase consistency and quality in
report writing
•
To provide advice to funders on how to judge the merit of interventions in Scotland.
WHO IS THE EVALUATION RESOURCE PACK
FOR?
The evaluation resource pack is designed to support anyone
commissioning or evaluating criminal justice interventions. In particular it
should help,
•
Contractors and practitioners to
• Evaluate criminal justice interventions using a robust 4 step method
• Structure an evaluation report
•
Funders to
• Commission evaluations
• Judge the strength of evaluation reports
• Assess the value of interventions
WHY DO WE NEED AN EVALUATION RESOURCE PACK? THE
BACKGROUND TO THE 4-STEP METHOD
Assessing the impact of funded interventions in Scotland is difficult due to
methodological constraints but the pressure to show that interventions are
effective has led to some poor and at times dubious evaluations.
To see whether your intervention had an impact and made a real difference to
users, you need a to compare users with a randomised or matched control group
have the same risk of reoffending as the users. You also need large sample sizes
and use statistical testing on the outcomes.
Very few evaluations are able to use this method so we devised an alternative
which mitigates the lack of impact information by emphasising the need to
embed robust evaluations from elsewhere and evaluating the
extent to which an intervention is evidence-based as part of the evaluation itself.
The evaluation then collects data to test whether the intervention was implemented
as intended and whether short and medium term outcomes materialised.
This pack was devised to support evaluators conduct this type of evaluation
which can be described in 4 steps:
THE 4 STEP APPROACH TO EVALUATION
Review the evidence
Interventions should be clearly structured and designed
using robust evidence so it is important to be familiar with
the results from the ‘what works’ and desistance literature.
This knowledge should be used to evaluate the extent to
which the intervention is grounded in strong and consistent
evidence. You could also be explcit about how much it cost
and how the funds were spent.
Draw a logic
model
Draw a logic model describing how your intervention works in
practice by describing the links between inputs, outputs and
outcomes. The logic model forms the basis for evaluating the
whole intervention so this may provide better clues as to why
an intervention acheived it’s outcomes or why it did not.
Identify Indicators
and collect monitoring
data
Evaluate logic model
Use this logic model to identify indicators for inputs,
outputs and outcomes and collect data using relevent
methods
Then analyse the data (and collect more if necessary) to
find out the extent to which your intervention was
evidence-based and if it worked as the logic model
predicted it would. Put as much emphasis on describing
and evaluating inputs as well as outputs and outcomes
CONTENTS OF THE FULL VERSION
The purpose of the pack……………………………………………………………………..
2
Background: Why a 4 step approach?........................................................................
5
The challenges of measuring impact in Scotland
Impact evaluations – why are they so hard to do?
Control and comparison groups – which characteristics need to be similar?
So if measuring impact is tricky, what can we do?
The 4 step approach to evaluation
The 4 step approach………………………………………………………………………….
4 step approach to evaluation -overview
11
Step 1: Review the evidence.........................................................................................
13
What does the evidence say?
Evidence summaries
An example – an evidence-based justification for a fictitious intervention
Step 2: Draw a logic model showing how the service or intervention works………
What are logic models
What logic models can do
A very simple evidence-based model
A logic model template to use
Logic model column content– a quick guide
The importance of designing a structured intervention
An evidence-based logic model – reducing reoffending
An example - the Reducing Reoffending evidence model
An example -A simple supervised bail logic model
Step 3: Identifying indicators and collecting monitoring data………………………..
Use the Logic model of identify indicators
Use the Logic model to set evaluation questions and guide the collection of data
Example: Indicators for outputs/activities and outcomes
Data collection
Quantitative and Qualitative data –uses, benefits and limitations.
Data capture and analysis
An example data collection framework for a criminal justice intervention
30
40
6
CONTENTS CONTINUED...
Step 4: Evaluate logic model…………………………………………………………………..
Test the logic model
Measuring and reporting outcomes
Measuring and reporting impact
Caveats to measuring impact
Subjective measures of impact
A note on Cost Benefit analysis
Evaluation Report Structure .............................................................................................
Structure and content
Judging the worth of an intervention………………………………………………………….
Assessing an evaluation report
Example judging criteria matrix for a reducing reoffending intervention
Features and advantages of a scoring system
54
61
65
Advantages and disadvantages of the 4 step logic model approach to evaluation......
69
Helpful Resources…………………………………………………………………………………..
70
/71
7