An introduction to Evidence & Insight
Two random control trials in search of
context
Prof. Betsy Stanko
Dr. Paul Dawson
Evidence & Insight
Mayor’s Office for Policing And Crime
Paul.dawson2@met.police.uk
Betsy.stanko@mopac.london.gov.uk
Our experience
captured in print
http://www.amazon.co.uk/Police-ResearchEvidence-RecommendationsSpringerBriefs/dp/3319206478
An Introduction to Evidence and Insight
•
E&I is the largest dedicated civilian research unit in UK policing & Crime and justice in London.
•
Approximately (20) specialist staff, such as researchers, analysts, criminologists & psychologists.
•
Based in the MPS for previous 9 years. Moved to MOPAC April 2014 with a broader scope.
•
Conduct wide variety of performance reports, problem profiles and research across crime and
justice in London (Large scale surveys e.g.,14,000 Londoners) to experimental evaluations).
•
Working to drive change and improvements that benefit London (within policing, crime and
justice…) from the inside (really is unique).
•
Evidence Based decision making – becoming a requirement – not a luxury – taking hold of
policing at present. A Fantastic opportunity to push through improvements and promote evidence
based working…
•
How much of policing is based on evidence of ‘what works’ as opposed to police instinct or
experience? How much of EBP is embedded into the heart of the organisation or at periphery?
An Introduction to our research – variety of research data
• A quasi-experimental evaluation of a 11 million pound initiative to reduce crime that found no
impact. Importance of programme integrity, implementation, learning cultures and
responding to negative findings. Thick skin…
• Large scale public perception surveys – e.g., public confidence and victim satisfaction. Developed
statistical models and data in public domain.
• Process evaluation around tagging alcohol offenders (performance, staff and offender views).
• Longitudinal surveys of Police graduate recruits to track their journey through the organisation…
• Large scale problem profiles on issues combining performance, external and internal research.
• The impact of a predictive burglary mapping system? Is it even being used on the ground?
• A Randomised Control Trial of providing Body Worn Video to response team officers… will this
lead to better outcomes (e.g., less complaints, Stop and Search outcomes)?
• A test to see whether writing to offenders can influence their offending behaviour (e.g., arrest).
http://www.college.police.uk/News/Collegenews/Documents/BWV%20report%20Nov%2
02015.pdf
Body Worn Video (BWV) Evaluation
Aims of the Evaluation
• Commenced April
2014 - MOPAC, the MPS and the College of Policing (CoP)
developed the largest urban Randomised Control Trial (RCT) of Body Worn Video
(BWV) to date.
• Innovative, high profile and ambitious. Potential wide ranging impacts for policing –
professionalism, reputation & productivity.
• MOPAC – have a key role in enabling innovation.
• Wanted to explore 3 broad aspects:
- to examine the MPS process of implementing the technology, capturing
learning/challenges
- to assess the impact(s) of the technology
- to examine the cost-benefit of the technology
Previous Evaluations
• Landmark U.S. study in Rialto (California) demonstrated benefits in terms of reduction in
complaints (from 24% to 3%) and officer use of force (from 61% to 25%). But…
- Small numbers
- US based
- Contamination
- Not a full range of outcomes
• Others trials have include gathering footage for training material, CJ outcomes and
domestic abuse (DA) responses.
• Not all pilots have been evaluated or evaluations have been limited by the design or size
of the sample. MOPAC research will go a long way to help address an evidence gap on the
holistic impact of BWV on numerous outcomes.
¹The Rialto police Department’s Body Worn Video Camera Experiment: Operation Candid Camera (2013)
Owens, C., Mann, D., Mckenna, R. (2014). The Essex Body Worn Video Trial: The impact of Body Worn Video on criminal justice outcomes of domestic abuse
incidents.
Body Worn Video (BWV) Evaluation
Evaluation Design: RCT
• A Randomised Control Trial (RCT): gold standard in
understanding if ‘something’ has worked.
• A RCT - greater certainty any change in outcome is a direct
result of the use of BWVs; and makes measurement of
outcomes at the beginning unnecessary (as the two groups
should be the same).
• Random allocation is crucial to ensure the groups are identical
- randomly selecting if an intervention is received or not, (in this
case the usage of BWVs) and monitoring a set of predetermined
theoretically linked outcomes.
Evaluation Design: Borough
Selection
• 10
boroughs - selection was analytically driven, based on some
general assumptions e.g. 500 available cameras.
• Further exclusion and inclusion criteria:
Exclusion criteria: those currently use BWV cameras; boroughs with
a consistently low complaint rate; boroughs whose inclusion would
otherwise adversely affect the analysis due to the relatively large shift
team sizes or number of officers on personal shift patterns.
Maryland scale
Boroughs: Brent, Barnet, Bexley, Bromley, Ealing, Camden, Croydon, Hillingdon, Havering, Lewisham
Inclusion criteria: boroughs with a relatively high rate of stop and
search; boroughs with a relatively high crime rate.
Evaluation Design: Officer Type & Team selection
A within borough design chosen – as variation within one borough was less than between two different boroughs.
• As contamination between the treatment (those who get cameras) and control (those who do not get cameras)
needed to be prevented/minimised entire teams were selected.
• Emergency response teams - chosen as they deployed to a wide variety of situations; will make the admin and
back office functions more standardised/cheaper; and will ensure a level of control over usage.
•Teams were randomly selected - 2 per borough as treatment and 3 in the control group (not individual officers
as per many prior BWVs).
Evaluation Design: Mixed Methodology
A range of social research methods used to attempt to understand the holistic picture….
Usage of BWV
Stops database
Officer surveys – process x3
Officer surveys – attitudinal x2
CRIS records (e.g., arrests and
charges)
14,000 victims of crime (USS)
DPS complaints database
Officer Interviews
12,000 thousand Londoners
(PAS)
Court Ethnography
Statistical modelling
Outcomes: What does success look like?
Body Worn Video has been linked to many different outcomes. The pilot was to take a holistic overview:
- What impact will it have on Criminal Justice outcomes? (e.g. arrests and convictions)
- What impact will it have on officer interactions with the public? (e.g. stop and search and complaints)
- Will it change officer attitudes and self reported behaviour?
- Will it save any time/money?
- Are there any unintended consequences?
• A range of data sources (CRIS, Stops database, DPS complaints, PAS, USS, survey data), many at an individual officer level have
been combined to provide an understanding of what has happened.
• Each outcome statistically tested at an individual borough and then a combined level to assess impact.
• Analysis further complicated – by the fact that some
officers in the treatment group never received cameras AND
some officers received them, but chose not to use them!!
Video Recordings
Officers are given guidance any time they are likely to make 'pocket book notes', they should think about
switching the camera on.
• There are incidents considered 'mandatory' for recording e.g. stop and search & incidents of domestic abuse.
• Anything recorded is kept on the cloud for 31 days before an automatic data cull. Anything marked as 'evidential'
will be stored in line with MOPI guidelines.
• When the 'on' button is pushed, the footage will record (video only, no audio) from 30 seconds before it was
pressed.
BWV implementing innovation
• Implementing anything is difficult – implementing innovation more so…
• Found numerous teething problems in the set-up and implementation
• Issues in the training (i.e., lack on ongoing support, lack of refresher training)
• Delays in installing infrastructure
• Shortfalls in camera numbers
• Frequent movement of staff
• Usage: 42% of officers with a camera recorded 10 or more clips per active month,
26% recorded less than 5 clips per active month, and 6% uploaded nothing.
Nudging offenders...
A test of personalisation and
offending behaviours
TOTAL POLICING
Background to Nudge and criminal behaviour
• The rise of behavioral insights – aka, Nudge
• Many diverse topics (from male urination to organ donation)
• Recent interest aimed at crime - although no clear evidence on offenders
• Personalisation a potential area from wider nudge research
• Bumblebee 2015 an opportunity (albeit small)
• A scene set for innovation...
• PS – our first attempt did not work. In 2012 there was organisational support, yet
Borough Commanders were not keen as many had their ‘own’ letters that were sent
to offenders (something we called parochial stubborness - many different designs,
questionable quality).
TOTAL POLICING
The letters......
Letters and enhanced personalisation
Same text, names, help-line – half with photo/ half without
Letter 2
Total of 176 letters (80 Photo; 96 no photo)
Letters delivered Nov-Dec 2014 (pre bumblebee peak)
Outcome a focus on offending
20 Boroughs:
Camden, Westminster, Hackney,
Letter 1
Waltham forest, Tower Hamlets,
Havering, Newham, Barking and Dagenham,
Lambeth, Bromley, Harrow, Bexley,
Richmond, Hounslow, Merton,
Wandsworth, Ealing, Hillingdon, Haringey,
Croydon
Selection of offenders – key criteria
Challenges in implementation
TOTAL POLICING
A comparison to similar offenders...
•
Boroughs randomly allocated to 'letter' conditions (e.g., letter 1 or 2).
•
Controls (no letter) were drawn from eligible nominal list within corresponding boroughs.
•
Boroughs sought to identify eligible offenders to receive a letter. Criteria included recent criminality. However
boroughs also identified other individuals to send the letter to...
•
We then developed an experimental group matched to these letter offenders on a number of relevant variables:
age, gender, offending in previous 6 and 18 months.
•
A strong match overall between groups....
Age
Avg. 32
Letter Offender
(n=176)
Range 1761
Avg. 31
Comparison
(n=183)
Range 1759
Gender
Avg offences 18
month before
Any Offence
18 month
before
Avg offences
6 month
before
Any Offence
6 month
before
3.10
67.6%
1.15
41.5%
3.34
68.3%
1.03
43.7%
91.5%
Male
91.3%
Male
TOTAL POLICING
Is there evidence that receiving a letter impacted upon committing any offence
Committed ANY offence 6 months subsequent
Any offence in the six months
subsequent to the letter.
7 percentage point difference
between receiving any letter
versus no letter (39.8% vs.
46.4%) and any offence*.
(*p=0.16, x2=1.92)
70
Total offences 6m pre post
60
Letter: -12% decrease
Control: -16% decrease
50
40
Any Letter
30
No Letter
Avg offences 6 months post
Letter: 1.03
Control: 0.86
(p= .42)
20
10
0
-9
-8
-7
-6
-5
-4
-3
-2
-1
1
2
3
4
5
6
Of those that reoffended, our letter offenders were quicker to commit any offence
•
In terms of the binary 'any offence' measure – no difference in proportion that offended in the first month
subsequent to the letter (Control 35%, Any letter 34%, L1 32%, L2 36%)
•
Subsequently we see our letter groups are quicker to commit any offence (more so Letter 2)
…But a short term impact on the average number of offences?
38% of all control group offending took part in that first month (bumblebee peak?)
Far less within our letter offenders (12% for letter one and 19% for letter 2).
In terms of average offences – for those to reoffend, again control group offenders committed an average of 2
offences in the first month, compared to 1.3 for our Letter group.
Average number of offences subsequent to letter….
0.35
Any letter
In month 1
Letter 1
0.3
Letter 2
No Letter
0.25
0.2
Letter offenders
fewer average
offences
(Any letter .16,
Controls .32, p=.07)
0.15
0.1
Letter offenders
significantly more in
later months
0.05
0
1
2
3
4
5
6
Wider influence on offending – burglary and home boroughs
Any letter group less likely to commit any burglary offence in the six months subsequent (15% Vs 21%)
(X2= 2.51, p=.113, phi=.084). Again letter groups appeared to offend less in month 1. Letter group more
average burglary’s (.16 v .3) (p=0.55).
Both comparable reduction in Burglary: Letter (75 – 54) (-28%) v Control (61 – 45) (-26%.)
Home and Away
Letter groups offended more than controls within delivery borough in general, but offending here
decreased by 5% compared to a 3% increase in controls.
Controls appeared to offend more outside their delivery borough (based on home borough data),
remaining steady pre and post intervention.
Whereas letter groups offending outside delivery borough increased by around 8%.
Offended outside delivery borough
20
15
Proportion of group (%)
Proportion of group (%)
20.81
19.79
17.5 17.7
13.29
11.25
10
5
0
Any Letter
Letter 1
Letter 2
No letter
Post
40
Post
21.8621.86
25
Pre
Offended within delivery borough
Pre
35
30
35.42
33.53
28.9
31.25
30
27.5
25.14
25
22.4
20
15
10
5
0
Any Letter
Letter 1
Letter 2
No Letter
Implications
• Small scale test of innovation
• Implementation issues
• Rolled out again in a improved manner (e.g., on all prison releases – timely)
• Mixed picture in terms of results
• Weak evidence that receiving any letter can impact upon committing any offence
• Of those that do reoffend, letter offenders appear to do this quicker
• Some change in that first month regarding average offences… albeit later peaks
• Home and away
• Differential impact – badge of honour? Delivery method? Frequency?
Wrap up: final thoughts
• The largest PCC research team in the country – decade of experience in the MPS
• Blending performance and social research into Evidence Based Policing
• Wide range of methodologies – steering a PCC from within to be evidence based
• Continued learning about the need for strong implementation
• Better use of police data for non police researchers - the oblique challenge
• How much of EBP is embedded into the heart of the organisation or at periphery
• Receptivity challenges (how to embed evidence, is there a willing culture)
• What does it really mean to be evidence based? Central movement and vision
Our experience
captured in print
http://www.amazon.co.uk/Police-ResearchEvidence-RecommendationsSpringerBriefs/dp/3319206478