Full Circle of Lake County, Inc
EVALUATION REPORT
July 1, 2013-June 30, 2014
Program
Parenting
DTBY kids
Empowerment
Emp kids
Mentoring
HS TOTAL
OLC
Girls Circle 8th – 12th
Boys council 8th
CIC
K-4 Prevention
JUNTOS students
TOTALS
Adults
Youth
Unduplicated Adult
Unduplicated Youth
Hispanic Mixed
Nat Am
Anglo
Total
male
female
17
3
3
16
39
3
36
46
3
5
25
79
41
38
33
33
33
2
2
2
53
2
5
60
30
30
94
2
1
22
119
66
53
58
1
2
3
64
28
36
53
1
3
57
57
10
10
10
61
42
103
16
87
22
9
31
12
19
11
21
27
5
32
476
11
12
130
629
219
410
111
3
3
58
175
19
156
365
8
9
72
454
200
254
108
348
ACTIVITIES
Mentoring
Boys Council
OLC
Girls Circle
103
23
102
183
HS
Empowerment
DARE to be You
Active Parenting
Workshops/trainings
236
40
40
12
4
TOTAL PROGRAM
DAYS
743
Male:
Female:
48%
52%
Hispanic:
Anglo:
Native
American
Mixed
80%
17%
1%
2%
1
Full Circle of Lake County, Inc. provides opportunities, education and
caring relationships to help Lake County become a place where youth
make healthy choices, families thrive and the community is united.
Full Circle accomplishes this through a variety of research-based programs
developmentally appropriate for all age groups. The logic model for Full Circle
includes three goals: Reduce alcohol, tobacco and other drug use, decrease violence
among youth, and increase protective factors using a positive youth development
strategy. Programs include: Mentoring, Leadership, After-school/summer
prevention programs, DUI prevention, Sexual Violence Prevention, Girls Circle/Boys
Council, Post-secondary school enrollment support, Parent Education and Health
Equity.
Positive Youth Development outcomes mean that youth are: Well Educated, Safe,
Connected, Contributing, and Physically and Mentally Healthy. Full Circle indicators
measure Connection, Contribution, Safety and Healthy (attitudes and beliefs).1
INDICATORS
Youth are Safe: Resilience Scale (Girls Circle/Boys Council, Outdoor Leadership)
Youth are Connected: School Attachment, Adult Bonding, Parent/Child Affective
Relationship Scale, Sense of Self scale (Mentoring, Parenting)
Youth are Contributing: Community Attachment, Cultural Appreciation, Community
Involvement (Mentoring, Empowerment)
Youth are Healthy: National Outcome Measures: Perceived Risk of Harm,
Attitudes Toward Use of Substance, 30 Day Use of Substances, Safe Driving,
Resilience ( High School DUI Prevention, Safe Dates, Boys Council, Outdoor
Leadership)
The results from FY14 are listed by program below. Statistical Significance is the
probability (p-value) that the observed difference between pre- and post-test average
scores occurred by pure chance. Tests yielding a p-value of less than .o5 (i.e., there was a
less than 5% likelihood that a pre-post difference was due to chance) were considered
significant. Statistical significance can only be determined with a group size over 30.
Another approach to determine effectiveness is to look at Effect size. Effect size is the
size of the difference between pre- and post-test average scores. Using Cohen’s d,
provides an indication of the amount of pre-to post change regardless of significance.
Effect size benchmarks (<,20 = small, .21-.79 = moderate, .80-1 = large effect.)
1
http://co9to25.org/who-we-are/about-us/ Retrieved 9/24/2013
2
MENTORING Funded in part by the Lake County Sheriff’s Office by the Division of
Criminal Justice & Office of Behavioral Health , State of Colorado.
Research identifies mentoring as an effective prevention strategy for substance abuse and
violence (e.g. mentored youth were 46% less likely to start using drugs, 27% less likely
to start using alcohol, and 33% less likely to hit someone)2. Full Circle managed 28
active matches in FY 2014 and showed significant increases in four measures of
relationship-induced growth in social or cognitive development and gains in youth’s
connection with significant others. Full Circle youth are more connected to positive
adults, are more connected to their school, have a greater appreciation for culture and are
more comfortable using community resources. These “protective factors” mitigate
negative behaviors in children. Youth showed statistically significant increases in
measures that protect against violence and substance use (Bonding, School
Attachment, Use of Community Resources).
Youth enrolled in the mentoring program are given the Full Circle Youth Profile upon
enrollment and at the end of each program year. The Full Circle Youth Profile has the
following scales:
Pre
Post
Statistical Significance
Adult Bonding
3.68 3.77
.12^
Cultural Appreciation
2.87 2.96
.62
School Bonding
3.49 3.61
.005**
Community
2.31 3.283 .000**
Attachment
2
*Mean difference is significant at p<.05 ** Mean difference is significant at p<.01 ^
approaching significance
In addition to current year data, this same scale as seen over time (2006-2014) showed
statistically significant increases in adult bonding, cultural appreciation, school bonding
and community bonding. Mentoring and bonding are considered research-proven
protective factors in the prevention of substance abuse.
4
3.5
3
2.5
2
1.5
1
0.5
0
2010/2011
2011/2012
2012/2013
2013/2014
Pre Post Pre Post Pre Post Pre Post
Bonding
Culture
School
Community
2
Tierney, J.P., Grossman, J.B., & Resch, N.L. (1995) Making a difference. An impact study of Big
Brothers/Big Sisters. Philadelphia: Public/Private Ventures.
3
Full Circle also finished a three year grant implementing a paid mentoring program for
youth on Diversion or Probation. This is a challenging population, but our outcome
objectives have been positive. The Alcohol Juvenile Arrests in Lake County have
dropped dramatically since the inception of this grant. It is difficult to ascertain direct
causal impact, but the downward trend since we started the paid mentoring program is
significant as seen by the table below:
Juvenile alcohol arrests
20
15
10
Juvenile alcohol arrests
5
0
2009
2010
2011
2012
2013
2014
OUTDOOR LEADERSHIP: Funded by Tony Grampsas Youth Services, Coors
Foundation, Anschutz Family Fund, Colorado Office of Behavioral Health, Safeway
Corporation.
Full Circle uses a combination of “established” and “developing” programming. Girls’
Circle3 is an established program designated by the Office of Juvenile Justice
Delinquency Prevention Model Program Guide as a promising program. The Outdoor
Leadership Club is a “developing” evidence-based program based on seven years Full
Circle evaluation data. Research shows that Girls’ Circle addresses the risk factors of:
Peer alcohol, tobacco and/or other drug use, Low school bonding, Early onset of
aggression and/or Violence4, the very issues that most confront youth in Lake County.
Girls’ Circle/Boys Council increase the following protective factors: Involvement with
positive peer group activities, Pro-social opportunities, Social competency, Resiliency,
Self-efficacy. Full Circle has shown statistically significant increase in positive youth
development (Belief in Moral Order, Self-efficacy, Life Skills, Resiliency) over the past
seven years.5
Youth enrolled in the Outdoor Leadership Club (grades 7 & 8) are given the Resilience
Survey to measure life skills. The Resilience Scale was used by Full Circle to measure
participating kids’ self-reported level of resilience, a construct made up of several
characteristics such as perseverance, self-reliance, and a sense of life purpose. Using this
3
Office of Juvenile Justice Delinquency Prevention, OJJDP Model Programs Guide
www.ojjdp.gov/mpg/mpgProgramDetails.aspx?ID=464 .
4
Ceres Policy Research, Roa J, Irvine A, Cervantz K, Girls Circle National Research Project: Evaluation
Results Proceedings of Persistently Safe Schools: the 2007 National Conference on Safe Schools and
Communities.
5
Full Circle evaluation results 2004-2013.
4
instrument, the matched sample size was 29 kids. Significant pretest to post-test change
was observed in the overall group (mean change = 0.25; p < 0.05). In other words, when
kids’ pre- and post-test survey scores were compared, the results indicated a significant
change in their perceptions about their own level of resilience. An analysis of the most atrisk kids (those who scored lowest at pretest; N=9) also yielded a mean pre- to post-test
change in overall resilience (mean change = 0.40; p < .05). The size of this effect was
considered large by social science standards (effect size = 0.99). Separate analyses sought
to examine any differences in boys’ versus girls’ scores. Interestingly, though they
experienced positive change, boys’ perceived resilience did not significantly increase
from pretest to post-test but girls’ did (mean change = 0.29; p < 0.05). In summary,
statistically significant improvements in resilience were observed in the total sample,
the risk group, and also specifically among girls. Results indicate that all kids were
able to benefit from Full Circle programming and report an increase in self-reported
resilience, and that girls may have experienced more positive effects overall.6
HIGH SCHOOL Funded by the Department of Human Services LEAF grant &
Sexual Violence Prevention (Colorado Dept. of Public Health and Environment)
DUI Prevention Program:
Youth in the DUI PREVENTION: (W(rec)k-less take the Safe Driving Survey and the
National Outcome Measures (NOMs). The desired direction of the NOMs is an increase
in mean scores, which indicates higher perceived risk of harm and more disapproving
attitudes. The desired direction of change for the Safe Driving Measures is a decrease in
the mean from pre- to post-test, which indicates that participants reported fewer instances
of that behavior or perceived behavior (like driving after drinking or using marijuana,
driving without a seatbelt, speeding and while talking on a cell phone).
* Mean difference is significant at p<.05
Pre
Post
Desired
SigDirection
nificance
2.5
2.90
y
NOMs
Perception of Harm
3.1
3.10
y
NOMs
Attitudes toward Use of
Substances
y
.3
.2
Safe Drive
Self as driver Safe Driving
.6
.7
no
Safe Drive
Peer as driver
.4
.2
y
Safe Drive
Self as Passenger
1.1
.6
y
*
Safe Drive
Peer as Passenger
Additional analysis of Full Circle data shows that 20% of youth participants
stopped drinking or using marijuana as a result of this program. At the end of the
program year 100% of youth enrolled in the program indicated they had zero use of
substances in the last 30 days.
6
Full Circle of Lake County, Inc. Individual Level TGYS Data Results: Fiscal Year 2013-2014 Reported
by the Colorado State University Evaluation Team.
5
Full Circle compares favorably to aggregate NOMs data (all LEAF funded programs in
Colorado, and all other programs tracking NOMs). State-wide only 9% of youth enrolled
in DUI Prevention programs stopped using alcohol and 7% stopped using marijuana,
while Full Circle youth showed 10% of youth stopped using both alcohol and marijuana
bringing our 30 day use to zero for alcohol.
Youth also completed Perceived Risk of Harm (PRH) Associated with Substance Use
and Attitudes Toward Use of Substances. Youth showed an increase in perception of
harm of all substances (alcohol, marijuana). Youth also completed the Attitudes
Toward Use (disapproval of use) scale. Full Circle youth did not show increase in
disapproval of use overall (pre: 3.1, post: 3.10), but it is interesting to look at trend data.
Over the past four years, youth disapproval of use has increased. This could be an
indicator of overall population level changes in attitudes toward use of substances and
would seem to explain the lack of significant increase this year due to already high levels
of disapproval of use.
3.4
3.2
3
Attitudes Pre
2.8
Attitudes Post
2.6
2.4
2009/10
2010/11
2011/12
2012/13
2013/14
The Self Driving measure was completed by 20 youth. Self as Driver 30-day negative
behavior decreased to less than .2 in the past 30 days. This is comparable to the Colorado
Aggregate results. The Perceptions of Peer Driving Behavior in the past 30 days
decreased from .90 to .86, while the Colorado Aggregate increased (pre: 1.15, post: 1.23).
The Peer as Passenger scale did show statistically significant decrease in being a
passenger while the driver drank (pre: 1.1, post: .6). Qualitative remarks from youth
offer an explanation. “Now that I have my driver’s license I am the designated driver for
my parents.”
Full Circle High School Programs show reduction in alcohol use, marijuana use,
cigarette use and increases in perception of harm and attitudes around use of
substances. This is a proven prevention program for youth at-risk.
HIGH SCHOOL SEXUAL VIOLENCE PREVENTION
High School youth are referred to enroll in Boys Council7 and Safe Dates to address
healthy relationships and reduce sexual violence.
7
Ceres Policy Research, Roa J, Irvine A, Cervantz K, Girls Circle National Research Project: Evaluation
Results Proceedings of Persistently Safe Schools: the 2007 National Conference on Safe Schools and
Communities
6
Statistically significant results were seen on the Life Skills, Gender Roles, Antisocial
and Delinquent Behaviors, Healthy Sexuality and Negative Assertion sub scales. The
Life Skills, Gender Roles and Antisocial and Delinquent Behaviors scales showed these
changes going in the desirable direction indicating positive improvements. The Healthy
Sexuality and Negative Assertion are significant in the reverse direction, but the numbers
are small and this result might be due to chance. The positive change in the Life Skills
scale is stronger than the SVP sample as a whole and given the strength of the mean
difference, is likely to be a true finding. The Gender Roles scale also showed a large
change at the individual level and is also likely to be real despite the small sample size.8
“The Life Skills mean of 3.75 is the mean change at the individual youth level and this is
a highly significant result. The chance of seeing this result is less than 1 in 10,000 if this
were due to chance alone. This is strong evidence of a real effect of the program.” ~
Cheryl Beseler CSU Evaluator.
PARENTING: Funded by the Office of Behavioral Health, Temple Hoyne Buell
Foundation, A.V. Hunter Trust
The Full Circle Family Support project utilizes the DARE to be You Model curriculum.9
DARE to be You (DTBY) is a multilevel prevention program that serves high-risk
families with children 2 to 5 years old. Program objectives focus on children's
developmental attainments and aspects of parenting that contribute to youth resilience to
later substance abuse, including parental self-efficacy, effective child rearing, social
support, and problem-solving skills. Families engage in parent-child workshops that
focus on developing the parents' sense of competence and satisfaction with the parent
role, providing knowledge of appropriate child management strategies, improving
parents' and children's relationships with their families and peers, and contributing to
child developmental advancement. Full Circle also utilized the Active Parenting model
program Active Parenting of Teens: Families in Action is a community-based
intervention for middle school-aged youth designed to increase protective factors that
prevent and reduce alcohol, tobacco, and other drug use; irresponsible sexual behavior;
and violence. Family, school, and peer bonding are important objectives. The program
includes a parent and teen component. This curriculum is based on Adlerian parenting
theory, which advocates mutual respect among family members, parental guidance, and
use of an authoritative (or democratic) style of parental leadership that facilitates
behavioral correction.10
8
Full Circle of Lake County Individual Level Sexual Violence Prevention Data Results, Fiscal Year
2013/2014, reported by Colorado State University Evaluation Team, 11/5/2014
9
Miller-Heyl, J., MacPhee, D., & Fritz, J. J. (1998). DARE to be You: A family-support, early prevention
program. Journal of Primary Prevention, 18(3), 257-285. 9/24/2013 Retrieved from
http://www.nrepp.samhsa.gov/ViewIntervention.aspx?id=65
10
Abbey, A., Pilgrim, C., Hendrickson, P., & Buresh, S. (2000). Evaluation of a family-based substance
abuse prevention program targeted for the middle school years. Journal of Drug Education, 30(2), 213228.
Retrieved from http://www.nrepp.samhsa.gov/ViewIntervention.aspx?id=168 9/24/2013
7
Scale
pre
post
t-test significance
Stress Total
22.37
20.71
**
Positive Attitude total
34.91
37.85
**
Autonomy total
22.95
24.06
Communication total
17.95
18.66
Limit Setting Total
21.64
24.6
**
Positive Parent Style Total (n=57)
5.21
5.74
*
* Mean difference is significant at (p<.05); ** Mean difference is significant at p<.01.
Full Circle parents showed significant improvement across all scales (Stress, Positive
Attitude, Communication, Limit Setting, Positive Parenting, and Autonomy). DARE to
be You continues to show positive indicators of behavior change in parents completing
the class. 29 parents completed class this year. Parents are less stressed, are setting
clear limits, communicate better and are using more effective (high warmth/low
criticism) parenting approaches.
40
35
30
25
20
15
10
5
0
5.8
5.6
Weight pre
Weight post
5.4
5.2
5
4.8
Pos. Parenting pre
Pos. Parenting post
*statistically significant p<.05
The Empowerment Group met weekly throughout the year. This project for
acculturating immigrant women continues to be successful. We instituted an evaluation
plan for this group this year to measure community involvement and sense of self. On
the Community Involvement scale participants showed statistically significant increases
from pre- to post-test (pre: 2.18, post: 2.93). The Sense of Self scaled also showed
statistically significant increases from pre 2.67 to post 2.94.
4
3
2
Pre
1
Post
0
Community Involvement
Sense of Self
8