Testing the Getting To
Outcomes implementation
support intervention in
prevention-oriented,
community-based settings
A 32-site RCT
MATTHEW CHINMAN, PHD
December 14,
2015
8th Annual Conference on the Science of
Dissemination and Implementation
GTO model supports high quality
program implementation in many
domains
4
3
2
Find existing
programs and
best practices
worth copying.
Identify goals,
target population,
and desired
outcomes.
Modify the program
or best practices
to fit your needs.
5
Assess capacity
(staff, financing,
etc.) to implement
the program.
Steps 1-6
PLANNING
6
1
Choose which
problem(s) to
focus on.
DELIVERING PROGRAMS
Steps 7-10
EVALUATING AND
IMPROVING
10
Consider how to
keep the program
going if it is
successful.
9
Make a plan for
Continuous
Quality
Improvement.
Make a plan
for getting started:
who, what, when,
where, and how.
7
Evaluate
planning and
implementation.
How did it go?
8
Evaluate
program’s success
in achieving
desired results.
GTO uses multiple
implementation strategies to
build capacityERIC* (Expert
GTO Components
Recommendations for
Implementing Change)
Manual
Technical
Assistance
PRIMARY
Training
SECONDARY
4
3
Find existing
programs and
best practices
worth copying.
Modify the program
or best practices
to fit your needs.
5
2
Identify goals,
target population,
and desired
outcomes.
Assess capacity
(staff, financing,
etc.) to implement
the program.
Steps 1-6
PLANNING
6
1
Choose which
problem(s) to
focus on.
Make a plan
for getting started:
who, what, when,
where, and how.
DELIVERING PROGRAMS
Steps 7-10
EVALUATING AND IMPROVING
7
1
0
Consider how to
keep the program
going if it is
successful.
Evaluate
planning and
implementation.
How did it go?
9
Make a plan for
Continuous
Quality
Improvement.
8
Evaluate
program’s success
in achieving
desired results.
Train and Educate
Stakeholders
Provide Technical Assistance
Multiple evaluative and
iterative strategies
Adapt and Tailor to the
Context
Support
clinicians/practitioners
Stakeholder relationships
Change infrastructure
Usually not done
*Powel, Waltz, Chinman, Damschroder, Smith, Matthieu, Proctor, Kirchner,
GTO grounded in social
cognitive and implementation
theories
GTO Components
Social cognitive theories of
behavior change
Manual
Technical
Assistance
Knowledgebeliefsbehaviors
Consolidated Framework for
Implementation Research (CFIR)*
Intervention characteristics
Training
Characteristics of the
individuals involved
4
3
Find existing
programs and
best practices
worth copying.
Modify the program
or best practices
to fit your needs.
5
Identify goals,
target population,
and desired
outcomes.
Assess capacity
(staff, financing,
etc.) to implement
the program.
6
1
Make a plan
for getting started:
who, what, when,
where, and how.
DELIVERING PROGRAMS
Steps 7-10
EVALUATING AND IMPROVING
7
1
0
Consider how to
keep the program
going if it is
successful.
Evaluate
planning and
implementation.
How did it go?
9
Make a plan for
Continuous
Quality
Improvement.
8
Evaluate
program’s success
in achieving
desired results.
GTO is proactive and helps organizations
change
Inner setting
Steps 1-6
PLANNING
Choose which
problem(s) to
focus on.
GTO builds knowledge & skills to plan,
implement, and evaluate interventions
Implementation process
2
GTO helps practitioners digest evidence
based interventions and plan & evaluate
them
GTO helps practitioners adjust their
organizational context, provides
accountability
Outer
setting et al 2009
*Damschroder
Enhancing Quality of Interventions
Promoting Healthy Sexuality (EQUIPS) –
RCT similar to large scale role outs
MPC is an 8 session EBP-promotes condom
Making sex
Making
use/less
Proud
Proud
Choices
VS.
(16 Boys
&Girls Club
sites)
Implemented
Choices
+ GTO
(16 Boys
&Girls Club
sites)
MPC twice in all sites in 2011-
2013
Compared GTO and MPC-only sites on:
Performance
of key program implementation tasks
specified by GTO (e.g., goal setting, planning,
evaluation) (interview)
Enhancing Quality Interventions Promoting
Funded by a grant from the Eunice Kennedy Shriver National Institute Of Child Health & Human Development,
Healthy Sexuality (5R01HD069427)
EQUIPS is a team effort
RAND evaluators
Technical Assistance providers
Joie Acosta, Patricia Ebener, Patrick Malone,
Mary Slaughter, Lynn Polite, RAND Survey
Research Group
Georgia Campaign for Adolescent Power &
Potential
Jennifer Driver, Cody Sigel, Kim Nolte
Alabama Campaign to Prevent Teen Pregnancy
Jamie Keith
Boys & Girls Clubs delivering MPC
Two year GTO training and TA
process
•GTO
training
(Steps 1-3)
•MPC
training
Work w/ TA staff
to
set Desired
Outcomes on Goals
tool
65 HOURS
OF TA
Work w/ TA staff to
complete Fit,
Capacity, & Plan
tools
• GTO
training
(Steps 46)
Year
2
Work w/ TA staff
to revise plans
for second cycle
• Implement
MPC
Data collection
Fidelity
Outcomes
• GTO
Evaluation
and CQI
workshop
(Steps 7-9)
REPEAT
•GTO
training
(Step 10)
GTO logic model links support
to outcomes
Staff
Survey
Baseline
TIMING
TA Monitoring
Form
Ongoing during
TA
Performance
Interview
2X, after each
Year (13% double
coded)
Fidelity
monitoring,
Attendance
records
Ongoing during
GTO; Double
coding:
Youth Survey
Baseline, Post,
6 Month at each
site, Year
36% Yr 1
25% Yr 2
EBP
attitudes
& support
DOMAIN
Capacity
measures
(Efficacy/
Behaviors)
Implementatio Performance
n Support
Fidelity
Outcomes
Data collection instruments
assess many domains
Staff
Survey
Baseline
TA Monitoring
Form
Ongoing during
TA
TIMING
DOMAIN
Performance
Interview
2X, after each
Year (13% double
coded)
EBP
attitudes
& support
Implementatio Performance in
areas targeted
n Support
Capacity
measures
(Efficacy/
Behaviors)
Total
By GTO Step
Hours of
Technical
Assistance
Fidelity
monitoring,
Attendance
records
by GTO
(High=5 to
Low=1)
Goals
Fit
Capacity
Planning
Process eval
Outcome eval
CQI
Ongoing during
GTO; Double
coding:
36% Yr 1
25% Yr 2
Fidelity
Adherence (% MPC
activities
completed fully,
in part, none)
Quality of
delivery (1-7 on
class control,
teacher
enthusiasm,
student
Youth Survey
Baseline, Post,
6 Month at each
site, Year
Outcomes
(follows
Jemmott
et al.)
Behavior
(sex,
condoms)
Beliefs (sex,
condoms)
MPC+GTO sites had better
performance in Years 1 and 2
5
MPC+GTO (Y1)
MPC+GTO (Y2)
MPC only (Y1)
MPC only (Y2)
4.5
4
3.5
3
2.5
2
1.5
1
Goals
***^^
Fit*
*
Capacity
***^^
Plan***
Proc
Eval^
Out Eval
***^^
CQI*
Sustain*
^^
TOTAL
***^^
*p<.05, **p<.01 ***p<.001 Linear mixed effects regression models, ^^ Year 1 to Year 2 (MPC+GTO), ^Year 1 to
MPC+GTO sites had better
adherence in Years 1 and 2
MPC Only
MPC+GTO
12%
4%
32%
39%
56%
Year 1
Completely
Partially
Not at all
MPC Only
9%
57%
MPC+GTO
7%
36%
Year 2
55%
92%
1%
MPC+GTO sites had better
classroom delivery in Year 2
7.0
6.5
Year 2: 1MPC+GTO > MPC only, p = .016 to <.0001
Year 1-2: ^MPC+GTO > MPC only, p = .01 to .04
YearxGroup: ¥MPC+GTO > MPC only, p = .04
6.0
6.0
5.5
5.6
5.5
5.4
5.0
4.5
4.0
5.1
4.7
5.2
4.9
4.8
4.8
5.0
4.9
4.7
4.7
MPC+GTO
MPC only
Y1
Y2Y1
Y2
5.1
5.0
MPC+GTO
MPC only
Y1
Y2Y1
Y2
MPC+GTO
MPC only
Y1
Y2Y1
Y2
MPC+GTO
MPC only
Y1
Y2Y1
Y2
3.5
3.0
Classroom control1
Student interest1^¥
Teacher enthusiasm1^¥
Objectives met1
MPC+GTO sites had better Year 2
attendance rates, but not
statistically
100%
Linear mixed effects regression, NS
Percent sessions attended
90%
80%
78%
74%
73%
70%
62%
60%
50%
40%
30%
20%
10%
0%
Year 1
Year 2
MPC+GTO
Year 1
Year 2
MPC only
Compared MPC+GTO vs MPC-only on
youth measures
Abstinence
(4 scales on beliefs, link to goals,
intentions)
Condoms
Knowledge
(8 scales on beliefs, intentions,
availability, skills, efficacy, impulse control)
Multivariate four-level linear regression modeling,
parameterized as a 2 (Condition, between site) X 3 (Time, within
site) factorial model
Sex
(2 scales on HIV/STD, Condoms)
(sex ever, use of condoms)
Intercourse and unprotected/condom intercourse outcomes logistic regression
Days of intercourse and unprotected intercourse outcomes linear regression
Adjusted for baseline
Covariates
(Grade; Social desirability; Race;
In Year 2 (n=419), MPC+GTO sites
improved more than MPC only sites on
mediators
Within Group:* = significant improvement
* = significant decline
Percent of Knowledge Qs Correct
GTO+MPC
MPC only
GTO+MPC
0.3
0.25
*
*
0.2
0.15
*
0.1
*
*
*
0.05
0
Pre
Condom Attitudes/Beliefs
Post
Between group: GTO+MPC > MPC
6 mo
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
MPC only
*
*
*
Pre
*
Post
*
6 mo
GTO+MPC > MPC
• Response rates: 80% at post, 63% at 6 months
• Condoms & Knowledge: both groups improve, GTO group improves
more
• No improvement on Abstinence
Conclusions
GTO sites had better
performance/fidelity, more in Year 2
Highly
structured EBP gets modest fidelity, need
more support for high fidelity (only 65 hours over
2 years)
Performance
Youth outcomes
Fidelity
Mediators
Yr1: Similar improvement across all sites
Yr2: Improvement across all sites, GTO sites improve more (better than
Jemmott)
Sex:
low base rates make comparisons difficult
Implications
The GTO doable on a large scale: 1 TA
provider helps one site for about 32
hours per year
Training and manuals/tools were helpful,
but proactive, ongoing TA was key
Back and forth between sites and TA staff
on tools was critical to ensure good
decisions, accountability
Quality improvement step in between Years
1 and 2 very important to improved
For More Information
http://www.rand.org/gto
Matthew Chinman
RAND Corporation
4570 5th Avenue
Pittsburgh, PA 15213
(412) 683-2300 x
4287
chinman@rand.org
GTO References
Chinman M, Acosta J, Ebener P, Malone PS, Slaughter M. (2015). A novel test of the GTO implementation support intervention
in low resource settings: Year 1 findings and challenges. Implementation Science, 10(Suppl 1):A34.
Smelson DA, Chinman M, McCarthy S, Hannah G, Sawh L, Glickman M. (2015). A cluster-randomized hybrid III trial testing an
implementation support strategy to facilitate the use of an evidence-based practice in VA homeless programs.
Implementation Science, 10, 79.
Chinman M, Ebener P, Burkhart Q, Osilla KC, Imm P, Paddock SM, Wright PA (2014). Evaluating the impact of Getting To
Outcomes-Underage Drinking on prevention capacity and alcohol merchant attitudes and selling behaviors. Prevention
Science, 15, 485–496.
Acosta, J., Chinman, M., Ebener, P., et al., (2013). An intervention to improve program implementation: Findings from a
two-year cluster randomized trial of Assets-Getting To Outcomes. Implementation Science, 8, 87.
Chinman, M., Acosta, J., Ebener, P., et al., (2013). Intervening with practitioners to improve the quality of prevention:
One year findings from a randomized controlled trial of the Assets-Getting To Outcomes intervention. Journal of Primary
Prevention, 34, 173–191. PMC3703481
Chinman M, Acosta J, Ebener P, et al., (2013). Enhancing Quality Interventions Promoting Healthy Sexuality (EQUIPS): A
novel application of translational research methods. Clinical and Translational Science, 6, 232 -237. PMC3684979
Chinman M, Hunter S, Ebener P. (2012). Employing continuous quality improvement in community-based substance abuse
programs. International Journal of Health Care Quality Assurance, 25, 606-617.
Chinman M, Hannah G, & McCarthy S. (2012). Lessons learned from a quality improvement intervention with homeless veteran
services. Journal of Health Care for the Poor and Underserved, 23, 210-224.
Chinman, M., Acosta, J., Ebener, P., et al., (2012). Establishing and evaluating the key functions of an Interactive
Systems Framework based on Assets-Getting To Outcomes. American Journal of Community Psychology, 50, 295-310.
doi:10.1007/s10464-012-9504-z. PMC3399058.
Chinman M, Tremain B, Imm P, Wandersman A. (2009). Strengthening prevention performance using technology: A formative
evaluation of interactive Getting To OutcomesTM. The American Journal of Orthopsychiatry, 79, 469–481. doi:10.1037/723
a0016705. PMC2859836
Hunter, S.B., Chinman, M. Ebener, P., et al., (2009). Technical assistance as a prevention capacity-building tool: A
demonstration using the Getting To Outcomes framework. Health Education and Behavior, 36, 810-828.
Hunter S, Paddock S, Ebener P, et al., (2009). Promoting evidence based practices: The adoption of a Prevention Support
System in community settings. The Journal of Community Psychology, 37, 5, 579–593.
“Goals Tool” prompts
decisions, planning, and record
keeping
Behavior or Determinant
SMART Checklist
S
M
A
R
T
Intention to practice abstinence
SMART Desired Outcome Statement
Aligned with:
Recent sexual activity
Number of sexual partners
Frequency of sexual activity
Contraceptive use and/or use consistency
Sexual initiation and abstinence
Pregnancy or birth
STIs (including HIV)
At the completion of the program, 80% of participants’ will
report that they plan to abstain from sex for the next 90 days.
SMART Checklist
Specific- Plans to abstain from sex for Aligned with:
the next 90 days
Recent sexual activity
Measurable- 80% at post survey
Number of sexual partners
Achievable- Abstinence promotion is in
Frequency of sexual activity
line with program goals
Contraceptive use and/or use consistency
Realistic- Similar youth have achieved
Sexual initiation and abstinence
this Desired Outcome before
Pregnancy or birth
Time-bound- By the completion of the
program
STIs (including HIV)
An average of 65 hours of TA
per site was provided over the
two years
Sustainabil
ity, 0
Outcome
Evaluation,
0.3
Process
Evaluation,
0.7
Needs
Assessment,
1
Continuous
Quality
Improvement
(CQI), 4.8
Goals, 0.9
Sustainability,
0.4
Needs Assessment,
0.52
Goals, 0.89
Fit, 3.4
Fit, 2.1
Capacity,
3.7
Planning,
6.2
Capacity, 3.7
Year 1
21 hours per site
Continuous
Quality
Improvement
(CQI), 15.5
Planning, 6.8
Year 2
44 hours per site
Outcome
Evaluation, 0.9
Process
Evaluation, 1.3
Response rates were good,
similar across groups and years
Year 1
Omegas (.71-.91, .40
Baseline (N=484)
MPC
= 236
Ave gr=6.31
44% male
83% Afr-American
GTO+MPC
on one)
Year 2
Omegas
one)
Baseline
MPC
= 248
Ave gr=6.57
47% male
91% Afr-American
(N= 323, 65%)
MPC only
= 165 (70%)
GTO+MPC = 158 (60%)
= 222
Ave gr=6.69
47% male
88% Afr-American
Post
6 Month
6 Month
(N=419)
= 197
Ave gr=6.42
47% male
88% Afr-American
GTO+MPC
Post
(N= 391, 81%)
MPC only
= 188 (80%)
GTO+MPC = 203 (82%)
(.55 to .92, .35 on
(N= 334, 80%)
MPC only
= 151 (78%)
GTO+MPC = 180 (81%)
(N= 264, 63%)
MPC only
= 117 (60%)
GTO+MPC = 147 (66%)
Sexual behaviors similar across
groups, but low base rates
Sexual Behavior in Past 3 Months
MPC+GTO
MPC
p=, MPC+GTO
vs MPC
% who had sex
5.8 (8/147)
7.3 (8/117)
0.67
Adjusted mean frequency of sex, days
0.10
0.08
0.59
Sexually inexperienced at baseline
0.08
0.04
0.48
Sexually experienced at baseline
0.39
0.48
0.82
% Reporting consistent condom use
50 (4/8)
75 (6/8)
0.31
% Reporting unprotected sexual intercourse
2 (3/147)
4 (5/117)
0.33
Sexually inexperienced at baseline
1 (2/135)
3 (3/100)
0.43
Sexually experienced at baseline
9 (1/11)
17 (2/12)
0.60
0.03
0.08
0.19
Sexually inexperienced at baseline
0.03
0.04
0.72
Sexually experienced at baseline
0.01
0.69
0.21
Adjusted mean frequency of unprotected sexual
intercourse, days
