Conceptual framework and a
research agenda in the transition
from efficacy to effectiveness
Irwin Sandler
ASU Prevention Research Center
P30 MH068685-01
Overview
• What we learn from efficacy – New Beginnings
Program
• Effectiveness conceptualized
• Expanded conceptualization of effectiveness
• Research agenda on transition from efficacy to
effectiveness: NBP
• Research studies to bridge from efficacy to
effectiveness
– Marketing study
– Court factors study
Efficacy: New Beginnings Program
•
Efficacy trial tests whether a program does more good than harm
when delivered under optimal conditions (Flay, 1986)
New Beginnings Program Efficacy Trial (NBP)
 Selection process
• Extensive recruitment methods
• Participants agreed to accept randomization and assessment
• Residential mothers meeting 9 eligibility criteria

•
•
•
•
•
Implementation process
11 session manualized (scripted) program
Delivered at ASU Prevention Center
Implemented by 13 selected group leaders
Training (30 hours, plus 1.5 hours prior to each session)
Supervision (videotaped sessions, 1 hour supervision per wk)
Selection and Implementation
Selection
• 36% of eligible families enrolled in trial
• Higher SES; 88% non Hispanic white
• Low drop out rate from intervention (11%)
Implementation
• Very high (90% +) level of program
implementation
• Very low variability in implementation
Randomized Efficacy Trial: Reduce Multiple
Problem Outcomes Six-Years Later
 Effects on outcomes over 6 years:
– 36% reduction in 1-year prevalence of diagnosed
mental disorder on C-DISC
– Reduce total psychiatric symptoms
– Reduce Marijuana, Alcohol and Drug Use for
higher risk group
– Improve Grade Point Average
– Reduce number of sexual partners
 Effects are mediated by improved parenting
 Effects strongest for high risk group
 Demonstrates viability of concept but not
delivery in natural settings
Effectiveness Trial
(Flay, 1986)
• “…concerned with whether program does
more harm than good when delivered via a
real world program”.
• Effects of trial may be due to:
– Selection (acceptance): rate of acceptance; who
accepts (program x person interaction)
– Implementation - fidelity, adaptation
• Both acceptance and implementation are multilevel concepts that apply to the program and the
organization that delivers the program
Transition from Efficacy to
Effectiveness
•
Research agenda for making the transition involves developing
systems to study real world conditions in which it will be delivered
and design program to optimize successful implementation
•
Prevention programs often don’t have pre-existing delivery systems –
but host institutions with which they share an interest and through
which they are delivered
•
Identify Domestic Relations Court as the Organizational Context
– Access to the population
– History of innovative services
– Overlapping (though not identical) interests with prevention
Research Agenda
•
Do the courts want this collaboration?
 Current practices and factors influencing adoption
•
Identify factors influencing selection and implementation of NBP using
quality management concepts and methods – research with consumer
 Court factors
 Cultural factors
 Provider agency factors
•
Design program and system for robust delivery through courts
 Optimizing court, culture, provider factors that influence implementation
 Optimize participant selection processes
 Pilot test selection and redesigned program and receive feedback on
viability
•
Effectiveness trial to test whether the program have the desired
effects
 Design of multi-court effectiveness trial
Market Study: National Survey of Courts
(Cookston, Braver, Sandler & Genalo, 2002)
Question: What is the current practice and
readiness of county courts to adopt “evidencebased” parenting program in the court?
Background:
o 1556 Counties provide divorce parenting
services in 1998
o 11 states require divorcing families to participate
in parenting programs
o Programs are generally short (4 hours) and
evaluations of efficacy are limited
Method for Survey of Courts
• Stratified random sample drawn of 154 courts
with parenting program
• Interview with key informant who is most
knowledgeable about the program
• Questions addressed:
Description of current programs offered
Planned changes in programs
Support for parenting program like NBP
Barriers to implementation
Support and Opposition for Lengthier
Parent Education Programs
Influence Group
Support*
Oppose^
Judges
77% (N = 115)
11% ( N = 16)
Members of the bar
73% (N = 111)
14% (N = 21)
County supervisors
71% (N = 102)
17% (N = 25)
State legislators
63% (N = 89)
22% (N = 31)
Child advocacy
groups
96% (N = 148)
3% (N = 4)
Local child and
family agencies
97% (N = 147)
1% (N = 1)
State Supreme Court
87% (N = 111)
7% (N = 9)
* - percent of respondents who indicated the influence group would support lengthier parent education programs
^ - percent of respondents who indicated the influence group would not support lengthier parent education programs
and would, in fact, oppose such programs
Barriers To The Implementation Of Lengthier
Divorcing Parent Education Program
Attorney Support
Showing Value of Program
Parent Attendance
Barrier
Lack of Community Interest
Adequate Staff
Support from Court
Cooperation from Judges
Allocating Space
Finding Presenters
Funding
0%
20%
40%
60%
Percentage of counties
80%
Court Factors Affecting Adoption
and Implementation
• National Advisory Group of Court Leaders
– judges, court administrators, mediators,
court service providers, executive director
of national professional organization
(AFCC)
• Identified by core working group of
program advocates – program developers,
director of mediation services
National Advisory Board Role
•
Provide advice and guidance to “translate the
NBP ‘experimental prototype’ into a service that
fits as a service for divorcing families through
the Family Court.”
• Three stage process
a)
Understand issues in translation into a realworld service
b)
Advise on specific aims of effectiveness trial
and assessment of court readiness
c) Advise on protocol for court recruitment
Advisory Board Input on Court Level
Selection and Implementation Issues (2/03)
• Respond to program selection scenario
“You have just received an RFP from your
state government for a parenting program
to be delivered through your court. Two
proposals for parenting education have
been submitted and you must choose
between them. What factors are likely to
make you select one program over
another?”
Factors that Influence Program
Selection
• Members independently listed 65 factors
• Group discussion of factors – reduced to
24 factors
• Factors grouped into six categories by
research team
• Categories rank ordered by the members
Rank Order of Program
Selection Factors
1.
2.
3.
4.
5.
6.
Compatibility and resource investment (compatible
with court priorities, cost of implementation)
Program efficacy (broad support from stakeholders
and meet needs of parents)
Perceived program credibility (e.g., Reputation from
other courts)
Program structure and content (e.g., number of
sessions; program content)
Program delivery quality (e.g., mode of presentation;
accompanying material)
Program accessibility (avg. rank = 4.7) (e.g., customer
access; multicultura/multilingual)
Factors that Influence Quality of
Implementation
• Respond to quality of implementation
scenario.
“When you receive your state funding award
one of the stipulations for continuation of
future funding was assuring high quality of
implementation of this parenting program.
What factors are most likely to influence
quality of implementation in your court?”
Rank Order of Quality of
Implementation Factors
• Staffing (on-site coordinator; administrator –
audits, details of administration)
• Stakeholder/consumer feedback (ongoing;
evaluation of impact on consumer and the court)
• Training (Training and motivating providers)
• Expanded reach of program (expand range of
providers and participants)
• Program visibility and acceptance (Advisory
committee; publicity)
• Program facilities (appropriate space)
Summary
• Research for transition to effectiveness trial
involves study of court, provider, cultural,
selection factors and design system to optimize
selection and implementation
• Braver will present studies to evaluate
alternative systems to recruit participants
• MacKinnon will discuss issues in the
effectiveness trial – focusing on role of
mediational analysis
System for Implementation of New Beginnings
Program Through Domestic Relations Courts
Stakeholder input
Families
Training, technical assistance
and monitoring
Domestic Relations
Court
Service
Providers
Selection
Methods
New
Beginnings
Program
Families
Families