Training is Not Enough:
Intervention and Evaluation
Supplements
Christina J. Groark and Robert B. McCall
University of Pittsburgh
Office of Child Development
Overview—Intervention
• Training alone is a minimally
effective intervention
• Need:
− Hands-on technical assistance in situ
− Supportive work environment
− Continuous monitoring, positive
supervision
Overview—Evaluation
• Assessing trainee’s satisfaction and
learning is a minimum evaluation
• Need:
− Changes in trainee’s on-the-job behavior
− Changes in the behavior/outcomes of clients
of the trainees
− Analyses to show trainees behavior change
mediated the intervention's effect on clients’
behavior/outcomes
Intervention
Training alone is minimally effective
Why? Trainees often do not change
their behavior
Need:
• On-the-ground technical assistance
• Supportive work environment
• Monitoring and positive supervision
Intervention
On-the-ground technical assistance
• Observe (videotape) client on job
• Constructive, positive feedback
• Modeling with client on job
• Daily review
Intervention
Intervention
Work environment
• Supports behaviors taught in training
• Hand washing—need a sink, waterless
soap handy
• Early care and education—need small
group size, low children:caregiver
ratio, a few consistent caregivers
Chaotic Child Environment
Orderly Child Environment
9
Monitoring and positive supervision
• Sets expectations, standards, reminds,
rewards
• Continuing system, supervisor, staff
meetings
• Hand washing
• Early care and education—teachable
moments, responsiveness to childdirected initiatives
Illustrative Example I
Latin American Orphanage
Intervention (birth-6 years)
• Training—sensitive, responsive
interactions
◦ 6 one-day training sessions
• On-the-ground technical assistance
◦ 12 days more than planned because
caregivers were not implementing
BUT –
• Limited environmental change
◦ 4 primary caregivers per ward
◦ 8-10 children per caregiver
• No continuing supervision
Results—Caregiving behavior/environment
(ITERS/ECERS)
7
Mean Total ITERS/ECERS
6
5
Typical USA Early Care
4
3
2
1
Pre-
Post-
Intervention
Results—Children’s development (Battelle)
100
Typical Parent-Reared
Battelle
90
80
70
60
Post-
Pre-
Intervention
Conclusion
• Needed hands-on technical assistance
to promote implementation
• Limited supportive work environment
• No supervisory system
• Some improvement in caregivers,
children, but NOT MUCH
Illustrative Example II
St. Petersburg (Russia) Orphanage
Intervention
• More intensive, comprehensive
• All major components
• Minimum warm, sensitive, contingentlyresponsive caregiver-child interactions
• Perfunctory, business-like; little talking
• Caregiver directed
• Conformity
St. Petersburg-USA Orphanage Research Team, 2005,
2008
4/13/2015
18
• 12-14 children/ward
• 9-12 caregivers/week
• Homogeneous age, disability groups
• Periodic graduations
• 60-100 caregivers before age 19 months
St. Petersburg-USA Orphanage Research Team, 2005,
2008
4/13/2015
20
Intervention
• Training
◦ Train Trainers
◦ 12 sessions, 2-3 hrs.
◦ Warm, sensitive, responsive
◦ “Love these kids…”
• Hands-on technical assistance
Intervention—Supportive
Work Environment
• Reduced group size to 6-7
• Assigned primary, secondary
caregivers—6 vs 9 cgrs.
• Changed work schedules
• Integrated groups by age, disabilities
• Eliminated periodic graduations to
new groups
• Family hour
Intervention—Monitoring
and Positive Supervision
• Training for supervisors
• Trainer observed, coached both
caregivers and supervisors
• Supervisors met periodically to
problem solve, case reviews, mutual
support
Evaluation
Three orphanages (Baby Homes),
birth – 4 years
• T+SC/M&S—Training plus Structural
Changes, Monitoring, Supervision
• TO—Training Only
• NoI—No Intervention
Evaluation—Trainee Learning
Evaluation—Improved Caregiving
Longitudinal
Sample
T+SC
Evaluation – Children’s Development
100
Typical Parent Reared
Developmental Quotient
90
T+SC
80
TO
70
NoI
60
50
40
First
4-9 mos
Time in Intervention
9+ mos
Evaluation--Mediation Analysis
• Intervention improved caregiving
• Intervention improved children’s
development
• But was children’s developmental
improvement associated with
improved caregiving?
Evaluation—Mediation Analysis
Two-Stage Theory of Change:
T= Train Caregivers
SC=Structural Changes
T+SC vs NoI
Better
Caregiving
Environment
HOME
Better
Children’s
Development
Battelle DQ
Evaluation--Mediation Analysis
Independent Variable
T+SC vs NoI
Intervention
Effect of Indep.
Var. on Mediator
a = 3.41(.22)***
Independent Variable
T+SC vs NoI
Intervention
Total Effect
Ĉ = 14.14(2.07)***
Mediator
HOME
Direct effect
Controlling for
Mediator
Ĉ = 8.00(2.87)***
Outcome
Battelle
Effect of Mediator on
Outcome Controlling for
Indep. Variable
b = 1.80 (.59)**
Outcome
Battelle
Mediator accounts for 43% of Total Effect
Conclusion
• Intervention
◦Training alone is often minimally
effective
◦Has greater effect if accompanied by
 On-the-ground technical assistance
 Supportive work environment
 Monitoring and supervision
Conclusion
• Evaluation is improved if
◦ Some training vs. No Training
comparison
◦ Measures of
– Learning
– Work-place behavior change
– Outcome behavior of clients
◦ Mediation analysis
Evaluation Mediation Analysis:
MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating
mediated effects in prevention studies. Evaluation
Review, 17(2), 144-158.