Sustainability in a Large-Scale State-wide Evidence-based Program Initiative
Brittany L. Rhoades, Brian K. Bumbarger, & Julia E. Moore
The EPISCenter, Prevention Research Center, Penn State University
Sustainability Research
Sustainability of EBPs is an important step towards achieving public
health impact, yet there is only limited empirical research.
•The overwhelming majority of prevention efforts in the U.S. are funded by time-limited grants
•Current theory suggests four broad dimensions associated with program sustainability:
1) Implementation Readiness (motivation for change, capacity to implement change, &
support of influential leaders)
2) Connection to Coalition
3) Program Support System (implementer characteristics & skills, training and technical
assistance)
4) Sustainability Planning
•Limitations of current research:
-mostly theoretical & anecdotal
-over generalization (differences by program type and context aren’t considered)
-little knowledge of what happens in real-world contexts
Background: The PA EBP Initiative
Goal: Identify factors associated with program
sustainability and examine variation by program type.
•In 2001, PCCD formed a partnership with Penn State’s Prevention Research Center to
provide technical assistance to grantees and study the process of program dissemination,
leading to the creation of the Evidence-based Prevention and Intervention Support Center
(EPISCenter) in 2008
Family Tx
ORGANIZATIONAL CAPACITY & SUPPORT
PARTICIPANTS
Functioning
ns
*
*
ns
•N = 77 respondents with sustainability information as of 2009
**
Financial Support
ns
ns
ns
ns
ns
ns
ns
ns
ns
ns
***
*
ns
ns
*
ANNUAL WEB-BASED SURVEY
Instrumental Support
•Annual survey completed by PCCD-funded grantees for 4 years of grant &
after PCCD funds end
Stakeholder Outreach
•Information collected on: Coalition Connection, Functioning & Support;
Implementation Readiness; Stakeholder Buy-in & Support; Training & TA;
Implementer Characteristics & Skills; Fidelity Monitoring; Sustainability
Planning; Local Evaluation
Participant Recruitment
•Sustainability was defined as maintaining program functioning 2 or more
years following the end of the initial grant funding.
IMPLEMENTATION BARRIERS (expected to favor the unsustained programs)
*
*
*
+
*
ns
*
ns
ns
ns
ns
ns
ns
ns
ns
ns
ns
ns
ns
ns
+
ns
ns
ns
ns
ns
ns
ns
ns
*
*
ns
ns
ns
ns
**
ns
ns
ns
**
*
*
ns
ns
+
ns
ns
*
ns
**
***
ns
**
**
ns
*
***
Participant Engagement
Inadequate Staff
ns
Lack of Resources
Competing Demands
•T-tests were used to examine mean differences on the aforementioned
scales/variables for sustained vs. unsustained programs
100%
80%
70%
9
16
12
16
60%
20%
6
8
5
5
10%
Staff Resistance
Requests for Program
Changes from Participants
Sustained Programs by Program Type
The majority
(69%) of EBPs
were able to
sustain
functioning at
least 2 years
beyond the
initial 4 years of
PCCD grant
funding. There
were no
differences
across program
type.
Lack of Admin Support
IMPLEMENTER CHARACTERISTICS
Communication with Trainer
Knowledge of Logic Model
SUSTAINABILITY PLANNING
Financial Planning
Alignment Planning
ns
ns
Note. Green = variable/scale mean favored sustained programs; Red = variable/scale mean favored unsustained programs
+p<.10, *p<.05, **p<.01, ***p<.001
Conclusions
0%
Classroom-Based
(n=24)
A TEST-BED FOR TYPE 2 TRANSLATIONAL RESEARCH
Family
Prevention
ns
30%
•Grantee must demonstrate ongoing relationship with local collaborative board and collect
outcome and implementation data to use for impact assessment and quality improvement
Comm/
Mentoring
ns
•Funded by the PA Commission on Crime and Delinquency (PCCD) to disseminate evidencebased delinquency & violence prevention programs
•PCCD provides 4 years of funding, with a 25% local match in Year 3 and 50% match in Year
4, to implement the selected EBP
Classroombased
ns
40%
•Utilizing the Communities That Care model, community coalitions conduct a local risk and
resource assessment to identify and prioritize risk and protective factors, then select EBPs to
address these identified targets
Overall
ns
PENNSYLVANIA’S EVIDENCE-BASED PROGRAMS (EBP) INITIATIVE
HOW DOES IT WORK?
Variable/Scale
ns
50%
•Since 1998, over $60 million in grants to 120+ PA communities to fund nearly 200 replications
of EBPs throughout the state
Significant Correlates of Sustainability: Overall & By Program Type
Connection
90%
The Pennsylvania Evidence-based Program initiative provides an ideal
context for research aimed at exploring the factors associated with realworld program sustainability.
Results
The Current Study
Overall Sustainability Rate
Community/Mentoring
(n=21)
Not Sustained
Family Prevention
(n=15)
Family Tx (n=17)
•Although programs associated with the PA EBP Initiative appear to
have had substantial success in maintaining functioning beyond their
initial grants, a minority of those continued at the same level
Sustained
Level of Functioning for Sustained Programs
Of those
sustained, the
majority were
operating at a
lower level of
functioning as
compared to
during PCCD
grant funding .
•Universal predictors of sustainability included: connection to high
functioning coalition, outreach to community stakeholders, knowledge
of the program’s logical model, communication with trainer, and
alignment & financial sustainability planning
21%
21%
58%
Reduced
Same
Higher
•The proportion of programs that were implementing at the same or a higher
level in comparison to a reduced level differed by program type (χ2 = 8.07, p <
0.05): 75% of family treatment programs, 44% of family prevention, 31% of
classroom-based programs, and 25% of community/mentoring programs
sustained program delivery at the same or higher level.
•Problems with participant recruitment and engagement, requests for
changes to the program from participants, and lack of administrator
support interfere with program sustainability
•Overall, it appears that predictors of sustainability vary substantially by
program type and therefore should be taken into consideration in future
research and when giving sustainability guidance to specific evidencebased programs
For more information email: blr162@psu.edu
The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University, and is funded by the Pennsylvania Commission on Crime
and Delinquency and the Pennsylvania Department of Public Welfare as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.