Evaluating methodological
quality in the criminal justice
system literature
Dr Amanda Perry
Centre for Criminal Justice
Economics and Psychology,
University of York.
Overview of the session:
•Background to RCT in criminal justice
•Quality assessment
•The Maryland Scientific Methods
Scale (SMS)
•Problems associated with the SMS
Where next…….
Background
Few RCT conducted in the UK
Systematic review revealed 125
conducted between 1957-2005
(Farrington & Welsh, 2005)
Concurrent findings from UK systematic
review between 1990-2002
(Perry, McDougall & Farrington, 2005)
Resistance to RCT…
•Historical resistance
•Ethical and moral reasons
•Practical difficulties
BUT………... New era
Campbell Collaboration (2000)
Department of Health/Home Office (2000)
New age of RCT…
Feasibility studies
(Farrington & Joliffe, 2002)
(Farrington et al., 2002)
Matrix (2006) – Criteria for assessing feasibility
Current RCTs
•Evaluation of Cognitive behavioural skills
programmes (McDougall, Bowles, Perry & Clarbour, ongoing).
•Evaluation of Restorative Justice Programmes
(Strang & Sherman, 2006)
Quality assessment…
•Reviews of quality assessment tools: The
medical field
•Moher et al. (1995) identified 25 scales
devised up to 1993.
•Juni et al (1999) compared 25 scales for
purpose of inter-rater reliability (r=.72).
Quality assessment…
Reviews of quality assessment tools: The
Social Sciences
•Gibbs (1989) – social work
•STROBE:
STrengthening the Reporting of OBservational studies
in Epidemiology http://www.strobe-statement.org/index.html
•Maryland Scientific Methods Scale – Criminal
Justice (Sherman et al., 2002)
Quality assessment…
•The Maryland Scientific Methods
Scale (SMS)
(Sherman et al., 2002, based on work by Cook &
Campbell, 1979)
Purpose of the SMS:
•
Simple (measuring internal validity)
•
To provide policy makers with information
about the evidence
•
Aim to classify all programmes into 1 of 4
categories
The SMS quality assessment..
Five point scale:
Rating 1-5
Rating 1:
Correlational study
Rating 2:
Pre and post test study
Rating 3:
Observational cohort with
comparable group
Rating 4:
Quasi-experimental/controlled trial
Rating 5:
Randomised controlled trial
The SMS quality assessment..
Statistical conclusion validity
•Was the statistical analysis appropriate?
•Did the study have low statistical power to detect effects
because of small samples?
•Was there a low response rate or differential attrition?
Construct validity
•What was the reliability and validity of measurement of
the outcome?
Evidence and use of the SMS:
What Works: Evidence from 2 or more
studies reporting positive results scoring 3
or above on the SMS showing statistical
significance and desirable effects and the
preponderance of all available evidence
showing effectiveness.
What Doesn’t Work: Evidence from 2 or
more studies reporting negative results
scoring 3 or above on the SMS…
Evidence and use of the SMS:
What ‘s Promising: Reporting evidence
from 1 study reporting positive results
scoring 3 or above on the SMS….
What Unknown: Evidence from 1 study
with a negative or inconclusive result
scoring 3 or above on the SMS…..
Use of the SMS:
‘What works – an example’
CCTV in car parks
(evidence from Welsh and Farrington, 2002)
Street lighting
(evidence from Painter & Farrington, 1997; 1999b;2001b;
Farrington & Welsh, 2002)
Burglary reduction schemes
(evidence from Ekblom, 1996a; 1996b)
Limitations of the SMS…
•Not fully assessing all threats to the
validity of a study
•Categorising study designs
•Does not take into consideration at
What cost?
Limitations of the SMS…
•Is designed to apply equally to all
experimental units
•Does not embrace all study designs
•Method of drawing conclusions on
what works based on statistical
significance rather than effect size
Improving the SMS….
•Farrington (2003)
•Based on five key criteria:
•Internal validity
•Descriptive validity
•Statistical conclusion/validity
•Construct validity
•External validity
Improving the SMS …
Information for policy makers
3 five point scales
•Design
(internal validity)
•Execution
(construct validity/statistical conclusion validity /sampling
elements of external validity)
•Reporting of the trial
Beyond the SMS…..
•Assessing the cost of an intervention
•Adequate and standardised follow-up
periods for outcome measures
(e.g., reconviction rates)
•Encouraging journal editors to use a new
scale/standard similar to CONSORT
statement
Beyond the SMS…..
•Development of a specific quality
assessment tool evaluating RCT in the
criminal justice
•To incorporate all elements of validity
•To include an assessment of cost/cost
effectiveness/cost-benefits of an intervention
•To include guidance on adequate and
standardised follow-up periods for outcome
measures
(e.g., reconviction rates)
SUMMARY
•RCT rarely used in UK criminal justice
system
•Use of the SMS as quality measure
•Room for improvement
•Development of a new scale….
CONTACT:
Amanda Perry
Centre for Criminal Justice Economics and
Psychology
University of York
Email: aep4@york.ac.uk