Teaching Pro-Social Skills to Anti-Social Youth
CIMH Development Teams
Aggregate Evaluation Dashboard Report
October 2008
This aggregate dashboard evaluation report describes youth for whom data were submitted in
April and May of 2008 that participated in Aggression Replacement Training (ART®) groups
through the CIMH-sponsored Teaching Pro-Social Skills to Anti-Social Youth (TPS)
Development Teams, reflecting youth served through the end of March 2008. Each dashboard
report reflects only those sites that submitted data at the specified data submission date; it does
not reflect all youth served through the Development Teams, nor does it reflect youth served for
whom data was submitted at an earlier data collection date. In other words, each report
includes only those sites who have submitted data at the most recent bi-annual data
submission, and reflects a historical record of their TPS implementation.
Twenty-four counties and 30 private-provider agencies have staff that participated in expert
clinical training in running ART® groups; most have been successfully implementing TPS
programs in a variety of settings. All counties and agencies have been offered site-specific data
workbooks for tracking their own implementation data, and have had access to ongoing
technical assistance to support evaluation. County participation in TPS Development Teams
has been based on strong collaboration between Juvenile Justice and Mental Health agencies.
Nine counties (38%) and seven private-provider agencies (23%) submitted data to CIMH
in April and May of 2008, reflecting youth served in their TPS programs through March of
2008. This report presents data from Marin, Placer, Riverside, San Bernardino, San
Francisco, Santa Clara, Stanislaus, Sutter-Yuba, and Washoe counties; and data from
Aldea Children and Family Services, Five Acres, Inland Empire Residential Center,
Martin’s Achievement Place, Pacific Educational Services, Penny Lane, and Victor
Regional Youth Services-Stockton.
Counties and agencies should be commended for their dedication to this aspect of the
Development Team, as we often find program evaluation activities to be difficult to initiate and
maintain over time.
This dashboard report reflects a total of 2,552 youth served through the TPS programs in these
nine counties and seven private-provider agencies. Youth participated in ART® groups in a
variety of settings, ranging from diversion/prevention in outpatient and education settings, to
intervention in locked treatment facilities, Camps and Ranches. TPS settings have been
condensed into four groups:
1. Juvenile Hall;
2. Other Secure Juvenile Justice Setting (i.e., Camp and Ranch);
3. Group Home/Residential Treatment Center; and,
4. Other Outpatient Setting (i.e., Court & Community School, Day Treatment, Drug Court,
Outpatient, and Regular Education.)
1
Table 1. TPS Client Demographics – Aggregate Data
Setting
Aggregate
Data
(N=2552)
Juvenile Hall
56.8%
(n=1450)
Other Secure
Juvenile
Justice
Setting
14.9%
(n=381)
Group Home/
Residential
Treatment
Center
10.7%
(n=273)
Other
Outpatient
Setting
17.4%
(n=443)
16.4
(n=2465)
16.7
(n=1444)
17.2
(n=354)
15.9
(n=270)
16.2
(n=392)
Female
(n)
21.4%
(n=545)
21.4%
(n=311)
7.9%
(n=30)
28.9%
(n=79)
27.8%
(n=123)
Male
(n)
77.4%
(n=1974)
78.6%
(n=1139)
91.3%
(n=348)
70.7%
(n=193)
65.7%
(n=291)
AfricanAmerican
(n)
19.6%
(n=500)
16.1%
(n=234)
13.9%
(n=53)
29.7%
(n=81)
29.8%
(n=132)
Asian/Pacific
Islander
(n)
3.3%
(n=85)
3.4%
(n=50)
5.2%
(n=20)
1.5%
(n=4)
2.5%
(n=11)
Caucasian
(n)
30.3%
(n=774)
35.9%
(n=520)
18.4%
(n=70)
31.9%
(n=87)
21.7%
(n=96)
Hispanic/Latino
(n)
41.4%
(n=1057)
42.3%
(n=614)
53.8%
(n=205)
33.0%
(n=90)
32.5%
(n=144)
NativeAmerican
(n)
0.6%
(n=16)
.6%
(n=8)
.8%
(n=3)
1.5%
(n=4)
.2%
(n=1)
Other
(n)
2.2%
(n=56)
1.4%
(n=21)
1.0%
(n=4)
2.2%
(n=6)
5.6%
(n=25)
Age
Average Age in
Years*
(n)
Gender
Ethnicity
Note: Percentages may not total 100 due to missing data.
* Age presented in this report is an estimate based on each youth’s year of birth.
2
Table 2. TPS Service Delivery Information – Aggregate Data
Setting
Juvenile Hall
56.8%
(n=1450)
Other Secure
Juvenile
Justice
Setting
14.9%
(n=381)
Group Home/
Residential
Treatment
Center
10.7%
(n=273)
Other
Outpatient
Setting
17.4%
(n=443)
Range: 1 – 32
Avg: 7.27
(+4.1)
Range: 1 – 28
Avg: 6.28
(+3.9)
Range: 1 – 11
Avg: 6.63
(+3.1)
Range: 5 – 13
Avg: 9.8
(+1.2)
Range: 3 – 32
Avg: 8.85
(+6.1)
(n=709)
(n=366)
(n=136)
(n=127)
(n=80)
Range: 1 – 18
Avg: 8.20
(+3.4)
Range: 1 – 18
Avg: 6.67
(+3.2)
Range: 2 – 11
Avg: 8.30
(+2.5)
Range: 5 – 15
Avg: 10.06
(+1.2)
Range: 1 – 18
Avg: 12.44
(+4.2)
(n=520)
(n=262)
(n=111)
(n=99)
(n=48)
Range: 1 – 12
Avg: 6.52
(+3.3)
Range: 1 – 11
Avg: 5.75
(+3.3)
Range: 1 – 11
Avg: 6.11
(+3.0)
Range: 4 – 11
Avg: 7.44
(+3.1)
Range: 3 – 12
Avg: 8.80
(+3.5)
(n=421)
(n=179)
(n=107)
(n=91)
(n=44)
Aggregate
Data
(N=2552)
TPS Service Delivery Information
Range and
Average # of
Skill
Streaming
Groups
(n)
Range and
Average # of
Anger Control
Groups
(n)
Range and
Average # of
Moral
Reasoning
Groups
(n)
Youth Satisfaction with Group Participation*
Skill
Streaming
(n)
3.77
(n=232)
4.16
(n=72)
3.59
(n=86)
3.64
(n=43)
3.56
(n=31)
Anger Control
(n)
3.86
(n=184)
4.12
(n=83)
3.83
(n=29)
3.66
(n=52)
3.34
(n=20)
Moral
Reasoning
(n)
3.85
(n=123)
4.15
(n=46)
3.86
(n=22)
3.62
(n=41)
3.50
(n=14)
Note1: Service delivery information is presented for youth who completed the ART group component as
defined by one or more of the following: a variable indicating that the group phase was completed, the
presence of at least one outcome measure, or the presence of a youth satisfaction questionnaire.
Note2: It is noteworthy that the average # of sessions is less than 8 for many settings and group
components.
* Satisfaction data reflect the average score across six satisfaction questions for each group component.
Possible range of scores is 1 – 5.
3
Table 3. Average Percent Pre/Post Improvement Identified in Outcome Measures – Aggregate TPS
Data
Setting
Aggregate
Juvenile Hall
Data
56.8%
(N=2552)
(n=1450)
Targeted Skill Streaming Group Outcomes
SS – Youth
(n)
SS – Parent
(n)
SS – Teacher/
Staff
(n)
Other Secure
Juvenile
Justice
Setting
14.9%
(n=381)
Group Home/
Residential
Treatment
Center
10.7%
(n=273)
Other
Outpatient
Setting
17.4%
(n=443)
12%*
8%*
19%*
18%*
2%
(n=389)
(n=132)
(n=171)
(n=56)
(n=30)
n too small
n/a
n/a
n too small
n/a
30%*
14%*
34%*
38%*
(n=204)
(n=39)
(n=117)
(n=45)
n too small
Generalized Skill Streaming Group Outcomes
SS – Youth
(n)
1.8%
.6%
(n=209)
(n=117)
SS – Parent
n too small
(n)
SS – Teacher/
n too small
Staff
(n)
Anger Control Group Outcomes
AQ - Youth
(n)
10%*
HIT – Youth
(n)
6%*
(n=584)
Moral Reasoning Group Outcomes
1.7%
4%
(n=33)
(n=45)
n/a
n too small
n/a
n too small
n/a
n/a
n/a
n too small
11%*
7%*
5%
18%*
(n=361)
(n=90)
(n=95)
(n=38)
4%
4%
12%*
4%
(n=59)
(n=68)
(n=54)
(n=27)
12%
18%
(n=81)
(n=41)
n/a
n too small
(n=208)
TPS Program Outcomes
YOQ – Youth
(n)
n too small
7%
(n=29)
YOQ – Parent/
n too small
n/a
n/a
n too small
n/a
Caregiver
(n)
Note1: All pre/post outcome measures are described in Appendix A; and all scores are reported in terms
of improvement (i.e., increased targeted use of pro-social skills, increased generalized use of pro-social
skills, better management of aggression, more mature and constructive thought processes, and
improvements in general mental health functioning).
Note2: Minimum n for aggregate pre/post analysis is 20 for any given outcome measure.
Note3: Follow-up analyses do not indicate any differences in any outcomes by gender or ethnicity.
n/a = not available
* A statistically significant improvement, p < .01.
4
Appendix A. Description of TPS Program Components and Pre/Post Outcome Measures
There are three components of TPS programs, corresponding to the three components of the
ART® curriculum: Skill Streaming, Anger Control, and Moral Reasoning. The ART® curriculum
affords sites a great deal of flexibility, as model-adherence can be achieved and maintained
within each of the three components individually. TPS Development Team implementation
varies across settings in which ART® is being delivered, with some sites running one type of
group, some sites running two, and some sites running all three. Minimum model-adherence is
achieved by presenting the curriculum, as designed, in group format, with two trained cofacilitators leading each group through component-specific tasks.
Skill Streaming groups teach the behavioral component of ART ®, in other words, what “to do” in
challenging situations. Anger Control groups teach the emotional component of ART®,
modeling and practicing techniques that help youth with affective and emotional regulation.
Moral Reasoning is the “values” component, using vignettes to guide group discussions of
morally challenging situations.
Each of the three types of ART® groups offered within the TPS Development Team has a groupspecific measure to assess the targets of the intervention. These measures are administered
just prior to and immediately after participation in a course of sessions for particular component
(pre and post). Thus, practitioners can assess the impact of group participation.
The targets of the Skill Streaming groups are assessed using a Skill Streaming Checklist.
Developed to assess the frequency with which youth use pro-social skills specifically targeted
by the Skill Streaming curriculum, there are separate versions for completion by youth,
teachers/staff, and parents. Responses on the Skill Streaming Checklist range from 1 (this skill
is almost never used) to 5 (this skill is almost always used.) Higher scores indicate better prosocial skills.
There are two ways in which Skill Streaming Checklists are used: Targeted and
Generalized. Targeted assessments focus on only those skills that are a part of a site’s
Skill Streaming curriculum. Generalized assessments measure all 50 potential skills,
although youth are only taught a smaller subset of these skills.
The targets of the Anger Control groups are assessed using the Aggression Questionnaire
(AQ). This standardized, self-report measure assesses a youth’s aggressive responses, and
his/her ability to channel those responses in a safe and constructive manner. Lower scores
indicate better management of aggression.
The targets of the Moral Reasoning groups are assessed using the How I Think Questionnaire
(HIT). This standardized, self-report measure assesses youth’s use of cognitive distortions, or
thinking errors. Lower scores indicate more mature and constructive thought processes.
The overall impact of participation in TPS programs is assessed using the Youth Outcome
Questionnaire (YOQ). This standardized self-report and parent/caregiver-report measure
assesses youth’s global mental health functioning. Lower scores indicate more normative,
non-clinical, aspects of general mental health functioning.
All outcomes reported in this dashboard reflect change in youth performance in each domain.
Absolute change in scores are not reported; rather, outcomes reflect the percent improvement
in youth pro-social skill use (targeted and generalized), management of aggression, mature and
constructive thought processes, and aspects of general mental health functioning.
5