Evidence2Success:
COMMUNITY PREVENTION SYSTEMS, VERSION 3.0
SARAH M. CHILENSKI
PREVENTION SEMINAR
APRIL 3, 2013
Background
2
 Evidence-based Programs (EBPs)
 Many exist
 Cross several developmental domains
 Challenges in using EBPs
 Programs to the right people
 Maintaining fidelity
 Achieving public health impact
The Answer? Collaboration!
3
 Congruent with US ideal of participatory democracy
 Theoretically, collaborations
 Use resources efficiently
 Bring expertise of diverse stakeholders
 Shared responsibility => more sustainable
Coalitions…Collaborations
4
21st
Century
Learning Centers
Strategic
Prevention
Framework (SPF)
Safe Schools,
Healthy Students
Drug Free
Communities
(DFC)
Weed & Seed
Action
Communities for
Health, Innovation,
and EnVironmental
ChangE (ACHIEVE)
Communities That
Care (CTC)
Teen Pregnancy
Prevention
Partnerships /
Coalitions
Fighting Back
Initiative
Suicide Prevention
Coalitions
Getting To
Outcomes (GTO)
PROmoting
School-universitycommunity
Partnerships to
Enhance
Resilience
(PROSPER)
Comparison: Ownership & Leadership
5
v1.0: CTC
v2.0: PROSPER
• Effort housed within
whatever makes sense in
each community
• Effort housed within the
Cooperative Extension
System
• Resources & technical
assistance gained through
CTC Trainers and
community-level leaders
• 3-tiered model which links
communities to technical
assistance, and also
resources at state-level
• Leader(s) of effort not predetermined
• Leader of effort comes from
the Cooperative Extension
System; Co-led by school
Comparison: Implementation Organization
6
v1.0: CTC
v2.0: PROSPER
• Programs implemented in
community department or
organization that fits best
• Programs implemented
through Extension System &
school
• Key Leader Board
• No Key Leader Board
• Community Board
• Community Prevention
Team
• Implementation guided by
structured phases with
specific milestones &
benchmarks within phases
• Implementation guided by a
general phased approach
linked to the academic
school-year
Comparison: Program & Monitoring Details
7
v1.0: CTC
v2.0: PROSPER
• Select programs from broad
menu
• Select programs from
narrow menu
• Target age range varies (018)
• Focused target age range
(11-14)
• Program mediators &
outcomes highly variable
• Program mediators &
outcomes more narrow in
scope
• Ongoing assessment with
youth survey
• No ongoing assessment with
youth survey
• Program monitoring part of
the model
• Program monitoring part of
the model
Comparison: Training & Technical Assistance
8
v1.0: CTC
• Multiple structured
trainings (6 in 12 months)
v2.0: PROSPER
• 1 annual meeting
• Regular smaller ‘learning
communities’ for leaders w/
TA providers
• TA Providers attend
meetings
• Team Leader has regular,
proactive contact with TA
provider
Comparison: Research Evidence
9
v1.0: CTC
v2.0: PROSPER
• Randomized trial
• Randomized trial
 14,646 residents (1,578 to
 19,100 residents (6,975 to
40,787)
44,610)
 89% White (74% to 98%)
 96% White (88% to 99%)
 37% Low Income (21% to
 29% Low Income (10% to
66%)
• Quasi-experimental design
 Community demographics
more variable
48%)
Challenges of Urban Settings
10
Size
Neighborhoods
can be distinct,
considered
different,
separate from
each other
Turnover of
leadership,
especially in
schools
Fractured
relationships with
schools
Lack of
communication,
collaboration
between public
systems – services
& funding
Distrust of civic
leadership
Many
organizations,
competition
Overwhelming
needs
Stressful
environment:
poverty, crime
Politics
More diversity
within community
residents
Distrust among
community
residents
The Attempted Answer? v3.0
11
Evidence2Success
Funded and Lead by Annie E. Casey Foundation
Project Development and Implementation Team:
Annie E Casey Foundation
Social Development Research Group, University of Washington
Mainspring Consulting
Center for the Study of Social Policy
Social Research Unit, Dartington, UK
Intervention/Outcome Research Team: U. of Wash (SDRG- Catalano, PI)
Implementation Evaluation Research Team
Investigators
Sarah Chilenski (PI)
Mark Feinberg
Brian Bumbarger
Mark Greenberg
Emilie Smith
Francisco Villarruel (MSU)
Research Staff
Mona Ostrowski (SRI)
Ali Chrisler (MSU)
Nicole Summers
Luke Ding
Project Management / Data
Collection
Diana Crum (SRC)
-Jason Padilla
-Maria Islas-Lopez
-Joan Gorman
-Jane Sullivan
Evidence2Success: Outcomes
13
Behavior
Relationships
Children’s
health and
development
Physical
Health
Educational
Achievement
Emotional
Wellbeing
Evidence2Success: Strategies
14
WORK TOGETHER.
Build capacity to work in
new ways to get results.
CREATE LASTING CHANGE.
Schools
GOAL:
Improve
Children’s
Community
Health &
Development
Public Systems
Maximize investments by changing
how programs are funded.
Use the best information to
choose programs that work.
Child/family
INVEST SMARTER FOR GREATER RETURNS.
PROVEN MEANS POSSIBLE.
Create a shared vision
through partnership.
Work together to create a shared vision
15
WORK TOGETHER.
Create a shared vision
through partnership.
Child/family
Schools
GOAL:
Improve
Children’s
Community
Health &
Development
Public Systems
Citywide & Community Partnerships
16
CITYWIDE PARTNERSHIP
Elected officials
Community
Partnership
#1
School superintendent
Public systems leaders
Representatives of
community partnership
Community
Partnership
#2
COMMUNITY
PARTNERSHIP
•
•
•
•
•
Business Leaders
Civic Leaders
Faith Leaders
Resident Leaders
Service Leaders (includes
schools and systems)
Use the best information
to choose programs that work
17
Public Systems
PROVEN MEANS POSSIBLE.
Schools
GOAL:
Improve
Children’s
Community
Health &
Development
Use the best information to
choose programs that work.
Child/family
Priorities Connect to Proven Programs
18
Absence of
Enduring
Negative
Behavior
Functional
Family Therapy
Does Not
Participate
in Crime or
Violence
Does Not
Use Illicit
Substances


Life Skills
Training


Multidimensional
Treatment Foster
Care





Nurse Family
Partnership
Absence of
Teen/
Unintended
Pregnancy
Pro-social
Behavior


Raising Healthy
Children



Strong African
American
Families



Together
Learning
Choices
Has Not
Tried Illicit
Substances
Absence of
Risky
Sexual
Behavior








Maximize investments by
changing how programs are funded
19
Child/family
Schools
GOAL:
Improve
Children’s
Community
Health &
Development
Public Systems
Maximize investments by changing
how programs are funded.
INVEST SMARTER FOR GREATER RETURNS.
Partners will map current spending on
programs for children and families
20
Sample Overview of Community Fund Map
Work Assistance
Other Programs Supplemental
Mental Health
3.2%
2.4%
Nutrition
0.8%
Child Care
Assistance
After School
1.6% Child Protective
6.8%
Programs
Services
1.0%
6.7%
Public High
Public Health
School
0.5%
16.5%
Foster Care
0.6%
Medicaid/PeachC
are
Public
10.4%
Elementary and
Title I
Population
15,500
Middle School
2%
Children
4,100
42.5%
Expenditure
$54,890,000
Pre-K/Headstart
4.7%
Shifting just over 1 percent of funds
21
Age
Group
2-4
years
5-10
years
10-14
years
10-16
years
14-19
years
#
Youth
Target Group and
Outcomes
Program
864
All children at risk of behavior
problems c. 30% = 250
Improved behavior, academics,
delinquency
Incredible Years
BASIC
1,360
ALL
Improved behavior,
academics, emotional
regulation
Promoting
Alternative
Thinking Strategies
840
ALL
Reduced substance abuse,
violence, risky driving
Life Skills Training
1,400
Young people at risk of
detention = 100
Reduced substance abuse,
recidivism, improved mental
health
Functional Family
Therapy (FFT)
650
Pregnant girls and young
women = 25
Improved prenatal health.
Fewer childhood injuries,
improved school readiness
Nurse Family
Partnership (NFP)
Unit Cost
Total
Investment
(per year)
Return on
Investment (per
dollar spent)
$127,386
$4.20
$50,773
(for 3 years)
$13.04
$14,280
$42.13
$287,100
$11.86
$103,631
(for 2 years)
$3.23
$2,022
Aiming to serve
25% of target
group (N=63)
$112
Aiming to serve
100% of target
group (N=1360)
$34
Aiming to serve
50% of target
group (N=420)
$3,190
Aiming to serve
90% of target
group (N=90)
$9,42
Aiming to serve
88% of target
group (N=22)
Build capacity to create lasting change for
children and youth
Build capacity to work in
new ways to get results.
CREATE LASTING CHANGE.
22
Child/family
Schools
GOAL:
Improve
Children’s
Community
Health &
Development
Public Systems
How capacity building will be achieved in
Evidence2Success
23
Coaching: Customized support to individuals and groups for a
limited period of time to help them build their capacity to improve
children’s health and development
Training: An organized activity that brings together a group of
people to help them to develop knowledge and skills to improve
their ability to improve children’s health and development
Technical Assistance (TA): Targeted support on contentspecific practices to build capacity or find a solution to a problem
• Individual or group
• Proactive or responsive
Proposed TA Structure
Citywide
Partnership
LEAD
Person
Community
Partnerships
LEAD
Person
School
District
LEAD
Person
Child
Welfare
LEAD
Person
Collecting/Analyzing
Child Data
TBD
TBD
TBD
TBD
Collecting/Analyzing
Finance Data
TBD
TBD
TBD
TBD
Utilizing Finance Data
TBD
TBD
TBD
TBD
Utilizing Child Wellbeing Data
TBD
TBD
TBD
TBD
Setting priorities
TBD
TBD
TBD
TBD
Assessing Program
Gaps
TBD
TBD
TBD
TBD
Selecting Programs
TBD
TBD
TBD
TBD
Implementing
Programs
TBD
TBD
TBD
TBD
Specialized TA
Support
TBD
TBD
TBD
TBD
TA Leadership
Team
Phases of Implementation
25
1
2
3
4
5
FORM
PARTNERSHIPS
BUILD
CAPACITY
KNOW THE
FACTS AND SET
PRIORITIES
SELECT AND PLAN
FOR PROVEN
PROGRAMS
TAKE ACTION,
LEARN, AND
ADAPT
Engage city, public
system, school and
community leaders
Expand citywide and
community
partnerships
Produce a “big
picture” view of child
well-being
Select proven
programs
Finalize plans and train
providers
Designate a
citywide coordinator
Build all partners’
capacity to
engage in joint
decision-making
Develop short- and
long-term action
plans
Engage the community
in recruitment
Select two
neighborhoods
Begin mapping current
program funding
Educate and inform
community and
systems
Conduct well-being
surveys
Pinpoint opportunities
to shift funding
Establish formal
partnerships
Prepare partnerships
to interpret data
Develop broad
strategy focused on
risk, protection and
priority outcomes
Develop financing
strategies to support
proven programs
Implement proven
programs and monitor
performance
Celebrate successes
Share action plan
Conduct outreach
Identify opportunities
to transform program
and service delivery
Identify service
providers
Track progress
and make
changes as needed
Penn State’s Role
26
 Implementation Evaluation
 Document activities
 Provide feedback & suggestions
 In real time
 For future model refinement
Evaluation & Monitoring Tools
27
1.
Interviews re early site
engagement
2.
Milestone & Benchmark 7. Partnership & City-level
Interviews
Ratings
3.
Training Observations
4.
5.
Training Participant
Evaluation Surveys
TA Record
6.
Coach-CoordinatorSystem Monthly Update
8. Partnership Meeting
Observations
9.
Capacity Assessment
Feedback Loop Processes
28
1.
Monthly review w/TA Team
2.
Direct communication with Site lead
3.
Debrief with Training Development Team
4.
Record deviations on Milestones & Benchmarks
5.
Overview Reports
6.
Structure (full project) meetings so that they capture
important information and create action steps
Challenges
29
 Very large project team
 What is the role of public health?
 Timing
 Start-up / early site engagement takes awhile
 City-level leadership, partnership members
 Collection of student survey data
 Community selection, coordinator, partnership members
 Delegation of responsibilities
 Program implementation
Challenges
30
 Conflicts of Interest
 Is this my job, or is this a volunteer effort?
 Is permanent systems change realistic?
 Hurry up….wait!
Thank you!
 For more information, contact:
Sarah Chilenski
sem268@psu.edu
314-599-3737
 This research was funded by the Annie E Casey Foundation. We
thank them for their support but acknowledge that the findings and
conclusions presented in this report are those of the author(s)
alone, and do not necessarily reflect the opinions of the Foundation.
 This research was also supported by the Penn State Clinical &
Transitional Research Institute, Pennsylvania State University
CTSA, NIH/NCATS Grant Number UL1 TR000127.