Case Study: The Evaluation Process
Tanya Fosdick
The Issue
• 16-24 years represent 13% of local population but
33% of all casualties
• Also comprise 25% of KSI casualties (largest age
group)
• More male casualties
• 68% are driver casualties
• Group E represented highest numbers and above
average risk
Lifestyle Summary
Persona
The Result
• an interactive presentation which contains several
short video clips
• the presentation also contains interviews with the
mother of a local victim and offender
• a crash car simulator which also includes a storyline
using young local actors
• creation of a microsite and ongoing development
of a social media presence
Evaluation Process
• Consultation in January 2014 between RSA and
Leicestershire FRS
• LFRS provided details of the intervention so we
could understand
• Delivery
• Management
• Structure
• Setting of Aims and Objectives
Aim
To increase the knowledge of presentation
attendees in relation to the risks of the ‘Fatal
4’ and to encourage attendees to agree to
behave in a responsible way when they drive
Objectives
1.
25% of attendees who before attending did not exhibit knowledge of road safety risks in each
of the following areas, did exhibit this knowledge afterwards (mobile phone use, speeding,
failing to wear seatbelt, drink & drug driving)
2.
15% of attendees who stated before attending that they would make calls or texts on their
mobile phone while driving, agreed afterwards that they would not now do so
3.
10% of attendees who stated before attending that driving at 40mph in a 30mph limit was not
unsafe, agreed afterwards that it was unsafe
4.
15% of attendees who stated before attending that they would not always wear their seatbelt,
agreed afterwards that they would now do so
5.
25% of attendees who stated before attending they thought they could handle a drink or two
while still able to drive safely, agreed afterwards that they did not now think so
6.
25% of attendees who stated before attending they thought they could use cannabis and still
be able to drive safety, agreed afterwards that they did not now think they could.
How to measure against the
objectives
• Example involving 1,000 respondents in pre & post
• Question asked if they would switch off phone
before driving
• Before, 149 agreed and 851 disagreed
• After, 261 agreed – 112 more than before
• 112 reported a positive change, representing 13.2% of
851 (not 1,000)
Logic Model
Evaluation Design
• Pre- and post-intervention design with comparison group
• Paper questionnaires, inputted by LFRS into Prometheus
online tool
• Pre-questionnaires identical for intervention and
comparison groups
• Baseline questions to establish attitudes and knowledge
• Demographic questions
• Post-questionnaires – same as pre- for comparison group
with additional questions about No More Lives Wasted
Roadshow for intervention group
• URNs could be created from DOB, gender, postcode and
school questions
Evaluation Process
• Data collection started in April 2014 and closed
January 2015
• Questionnaires delivered before intervention &
collected, on average, 6 weeks later (Average 4
weeks for comparison group)
• Approximately 10,000 young people attended No
More Lives Wasted Roadshow in that time
• Needed a sample size of 370 (based on 5% margin
of error, 95% confidence level & 50% response
distribution)
Results?
• At the report writing stage
• Sample sizes:
•
•
•
•
•
•
1,096 completed pre-surveys for intervention group
1,071 completed post-surveys for intervention group
757 matched pre and post (exceeding target sample)
204 completed pre-surveys for comparison group
214 completed post-surveys for comparison group
171 matched pre and post
Process
• Takes time! 15 months…..
• Consultation
• Evaluation design
• Aims and Objectives setting
• Logic model
• Questionnaire design and testing
• Data collection and intervention delivery
• Analysis and report writing