Promising Results from the National
Evaluation of the Gang Resistance
Education and Training (G.R.E.A.T.)
Program
 Finn Esbensen
 E. Desmond Lee Professor of Youth Crime & Violence
 Chair, Department of Criminology and Criminal
Justice
 University of Missouri-St. Louis
Presentation Overview
 Why is there interest in youth violence and gang
prevention?
 How do we know what works?
 What do terms like “evidence-based”, “model programs”
and “effective” really mean?
 Use the G.R.E.A.T. program as an example of how
question of program effect is addressed.
NYGS: Jurisdictions Reporting Youth Gang Problems in One or More Years, 19992001(N=>1400)
Gang Violence
 Widely documented finding that gang members are
responsible for a disproportionate amount of crime.
 Thornberry and Burch (1997), for example, reported
that gang members accounted for 86 percent of all
serious offenses in the Rochester Youth Development
Study.
 Youths have higher rates of offending during active
gang membership than they do either before or after
gang involvement
How do we know what works?
 Anecdotal evidence is NOT sufficient
 Good intentions are NOT sufficient
 Saying that something works or that something is
effective, no matter how many times you say it, does
NOT make it so
 Rigorous evaluation is required
Promising Programs for
Violence Prevention
 The Blue Prints Program : Center for the Study
and Prevention of Violence at CU-Boulder
 Reviewed over 600 programs
• 11 were identified as model programs

proven scientifically to be effective in reducing youth
aggression, violence, other delinquent behavior, and substance
abuse;
 An additional 21 designated as “promising”
 That means 570 were not effective or not
evaluated
Model Program Designation
 Sound program or theoretical model
 Random assignment
 Sustained program effect (12 month post treatment)
 Replicated in at least one study
What is known about responses to
gangs?
Very little
 No shortage of programs - lack of evaluations of existing
programs or lack of positive outcomes
 G.R.E.A.T. as example
What is G.R.E.A.T.?
 13 week in-school general prevention program
 Taught by uniformed law enforcement officer
 Original program developed in 1991 by Phoenix area
officers – 9 lessons and mostly didactic
 Several efforts to review the curriculum
 null findings from longitudinal study
 Curriculum review conducted from 1999 – 2000
What is G.R.E.A.T.? - 2
 Program revision 2001 – piloted in 2002
 Seattle Social Development Model - classroom
management, cooperative learning, interactive teaching
 Life Skills Training – self-management and social skills
development
 Interactive and skills building focus
 Especially important that program be taught as designed –
lessons build on each other
 Important to assess program fidelity
 Implementation of Revised G.R.E.A.T. in 2003
G.R.E.A.T. LESSONS
 Lesson 1 – Welcome to G.R.E.A.T. - acts as the introduction to the




program and introduces the relationship between gangs, violence,
drug abuse, and crime.
Lesson 2 – What’s the Real Deal - consists of message analysis skills
and “facts and fictions” about gangs.
Lesson 3 – It’s About Us - focuses on different communities and how
students are a part of these, including their responsibilities to their
community or communities.
Lesson 4 – Where Do We Go From Here - introduces students to
the concept of goals and how to set realistic and achievable
aspirations.
Lesson 5 – Decisions, Decisions, Decisions - focuses on decisionmaking, in which students learn the G.R.E.A.T. decision making
model and the impact their decisions have on their goals; students are
able to practice making positive decisions.
G.R.E.A.T. LESSONS - II
 Lesson 6 – Do You Hear What I’m Saying? - teaches the importance




of listening to others and the difference between verbal and nonverbal communication.
Lesson 7 – Walk in Someone Else’s Shoes - instructs students in
active listening skills and how to identify others’ emotional states
through empathy-building techniques.
Lesson 8 – Say It Like You Mean It - teaches refusal skills so
students may resist peer-pressure to engage in deviant or delinquent
acts; this includes learning about body language and tone of voice.
Lesson 9 – Getting Along Without Going Along - consists of
recognizing peer pressure and other influences that may push
students into delinquency.
Lesson 10 – Keeping Your Cool - teaches students to keep calm in
the face of anger with anger management tips and practicing the
“cooling-off” technique.
G.R.E.A.T. LESSONS - III
 Lesson 11 – Keeping It Together - consists of recognizing anger in
others and learning to calm them.
 Lesson 12 – Working It Out - teaches students to work through
problems without fighting and provides tips for conflict resolution,
practice of such, and information about where to go for help in their
communities.
 Lesson 13 – Looking Back - consists of a program review and the
presentation and discussion of student final projects.
 http://www.great-online.org/
Program Goals
 Measuring Explicit Goals of Program:
 Avoid gang membership
 Do you consider your group of friends to be a gang?
 Are you now in a gang?
 Prevent violence and criminal activity
 Self-reported delinquency measures
 Develop positive relationships w/law enforcement
 “Global” attitudinal scale
 G.R.E.A.T.-specific attitudes
Lesson Specific Outcomes
 Impulsivity and risk
taking
 Anger management
 Self centeredness
 Peer pressure
 Negative peer
commitment
 Peer delinquency
 Neutralization (hitting)
 Conflict resolution
•Refusal skills
•Empathy
•Active listening
•Problem solving
•Collective efficacy
•Attitudes about gangs
•Altruism
Risk Factors
 Analogy with medical field
 Genetic and lifestyle factors associated with likelihood
of particular illness
 Presence of risk factors does not mean that person
will be ill; only increased probability
 Adopt this approach with violent offending and gang
membership
 Risk factors in 5 domains: community, school, family,
peer, & individual
Risk factors
 The more risk factors and in multiple domains; the
greater the risk of gang joining.
 No silver bullet – not all youth living in high-risk
neighborhoods join gangs.
 Addressing even one of these risk factors reduces the
odds of gang joining and violent offending.
Cumulative Effect of Multiple Risk
Factors
70.00%
60.00%
50.00%
40.00%
Non-violent
30.00%
Gang Membership
20.00%
10.00%
0.00%
0
1-5
6-10
Number of Risk Factors
11+
Does the program work?
 Is the program implemented with fidelity?
 Can the program be adopted in a variety of settings?
 Is the program effective?
This research was made possible, in part, by the support and
participation of seven school districts, including the School District of
Philadelphia. This project was supported by Award No. 2006-JV-FX-0011
awarded by the National Institute of Justice, Office of Justice Programs,
U.S. Department of Justice. The opinions, findings, and conclusions or
recommendations expressed in this presentation are those of the authors
and do not necessarily reflect the views of the Department of Justice or of
the seven participating school districts.
Finn Esbensen, Ph.D., Principal Investigator
University of Missouri-St. Louis
Terrance J. Taylor, Ph.D., Investigator
University of Missouri-St. Louis
Dana Peterson, Ph.D., Investigator
University at Albany
Wayne Osgood, Ph.D., Investigator
Pennsylvania State University
Kristy Matsuda, Ph.D., Research Associate
University of Missouri-St. Louis
Adrienne Freng, Ph.D., Site Coordinator
University of Wyoming
Dena Carson, Site Coordinator
University of Missouri-St. Louis
Site Selection
 Site selection:






Geographic and demographic diversity
Nature of community gang problems
Law enforcement agency and Program size
G.R.E.A.T. taught at least one year
No program saturation
Obtained agreements from law enforcement and public school
districts
 Process:
 Created preliminary list of 56 cities (identified by at least one RA, IIR
or BJS staff member)
 Contacted agencies about G.R.E.A.T. program
 Resulted in final list of 7 – hoped to recruit 6
 Albuquerque, NM ; Chicago, IL; DFW area city, TX; Greeley, CO;
Nashville, TN; Philadelphia, PA; Portland, OR
Process Evaluation
 To what extent is G.R.E.A.T. delivered as intended?
 If effect found, can it be attributed to the program?
 If no effect found, is it a case of program failure or failed program?
 Triangulation
 GOT observations
 Classroom observations
 Officer & Supervisor interviews
 G.R.E.A.T. Officer survey
 School personnel questionnaires
GOT Observations
 Observations of Training
 Observed nine GOTs (1 week and 2 week sessions)
 Purpose:


Learn new curriculum as officers are learning it;
Assess quality and consistency of training within and across
GOTs
 Well-designed and implemented
 Reasonable to expect that officers are prepared to teach the
program with fidelity
Classroom Observations
(2006-2007 school year)
 Purpose:
 Assess program delivery regarding:



Dosage
Adherence to intended lessons
Quality of instruction
 How?
 502 in-class observations of program delivery
 108 non-G.R.E.A.T. classes observed
Control classrooms and G.R.E.A.T. when program not taught
 33 different officers
 Each lesson observed between 27 and 49 times – allows us to assess
lesson quality in addition to officer implementation fidelity

Findings: Implementation
 We were able to provide feedback on some important
lesson-related issues (e.g., time management, student
interest, role of teacher, lesson content)
 Importantly, of 33 total officers, 27 implemented the
lessons in average/above average manner
 3 officers were judged not to have delivered the program
 Evidence of program fidelity so if effects are found, they
can be reasonably attributed to the program.
Outcome Evaluation
 To what extent is G.R.E.A.T. achieving its stated goals?
 Determine both short- and long-term effects
 Compare students who receive program to students who do not, in
an experimental design
 Multiple, diverse locations
Outcome Evaluation
 Seven cities
 Albuquerque, Chicago, DFW area, Greeley, Nashville, Philadelphia,
Portland
 31 schools total
 4 schools in five cities, 5 in Portland, 6 in Chicago
 195 classes total (102 GREAT, 93 Control)
 24 - 35 classes per city
 3 - 12 classes in each school (half receive G.R.E.A.T.)
 3,820 students
 457 - 614 students per city
 Between 59 and 186 students per school
Outcome Evaluation
 Random Assignment at classroom level
 Equivalent comparison groups
 No bias in assignment
 Active Parental Consent
 Students are the program target, so we need their
assessments. How do we do this?
 Confidential self-report questionnaires
 Questionnaires read aloud in group-administered setting
 Researchers ensure privacy, answer questions
 Pre- and post-tests
 4 annual follow-up surveys
Outcome – Student Surveys
 Site selection
 Aug and Sept 2006
 Active consent process
 Sept – Oct 2007; Jan & Feb 2008 (two added schools)
 Survey administration
Pre-tests - Sept 2006 – Feb 2007; Feb & March 2008
Post-tests - Nov 2006 – May 2007; May 2008
Wave 3 – Oct 2007 – March 2008; Dec 2008 – Feb 2009
Wave 4 – Oct 2008 – March 2009; Nov 2009 – Feb 2010
Wave 5 – Oct 2009 – March 2010; Nov 2010 – March 2011
Wave 6 – Oct 2010 – March 2011; October 2011
 Data entry (and verification)
 Preparation of analysis file and outcome analyses






Completion Rates
 Active consent rate = 78% (N=3,820)
 Pretest completion = 98%
 Post-tests = 95%
 Wave 3 = 87%
 Wave 4 = 83%
 Wave 5 = 75%
 Wave 6 = 73%
 Wave 5 surveyed students in 216 difference schools
 66 different schools in Philly
Outcome Results - I
 Compare G.R.E.A.T. students with non-G.R.E.A.T.
students
 Wave 2 (post-test) and
 Wave 3 (one year after program)
 A number of significant differences were found:
 11 of 33 significant at p<.05
 3 significant at p<.10
Program Goals - Outcomes
 Avoid gang membership
 39% lower odds of gang membership
 Develop positive relationships w/law enforcement
 More positive attitudes to police (ES=.076)
 Teaching specific role of police (ES=.204)
 Prevent violence and criminal activity
 No significant differences
Proximate outcomes
 Favorable G.R.E.A.T. outcomes p<.05
 Less positive attitudes about gangs (ES=.114)
 More us of refusal skills (ES=0.90)
 Resistance to peer pressure (ES=.079)
 Higher collective efficacy (ES=.125)
 Less hitting neutralizations (ES=.105)
 Fewer delinquent peers (ES=.083)
 Less self centeredness (ES=.054)
 Less anger (ES=.057)
Proximate outcomes
 Favorable G.R.E.A.T. outcomes p<.10
 Less use of lie neutralization
 More pro-social peers
 More pro-social involvement
 So, why are these proximate outcomes important?
 Remember, reducing one or two risk factors reduces
odds of gang joining
Bottom Line
 At 12 month post treatment, 2 of the three program goals
were met (reduction in gang joining and improvement in
attitudes towards police)
 A number of proximate factors suggested G.R.E.A.T. has a
positive effect
 But, effect sizes are modest & no effect on delinquency
 Will these effects be sustained across time?
 Are these findings replicated across cities?
 Are there differential effects based on initial level of risk?
Reports
 Website: http://www.umsl.edu/~ccj/html_files/great_evaluation.html
 Reports to Schools & Communities:
 2007: Evaluation Overview.
 2008: Anti-Social Norms among a Sample of Middle-School Students.
 2009: Program Implementation and Preliminary Outcome Results.
 2010 Report to Schools and Communities: School Safety and Victimization.
 Results from Surveys and Interviews with G.R.E.A.T.-trained Officers.
 Observing the Implementer: Description of Findings, Recommendations, and
Conclusions from GREAT Program Implementation Observations.
 School Personnel Survey Report.
 G.R.E.A.T. Officer Training (GOT) Report.
That’s all, folks
 Thanks for listening
 Any questions?