+
Experimental Evidence on the Early Effectiveness of
Intervention in Childhood
Orla Doyle
(UCD School of Economics & UCD Geary Institute)
Conference on Irish Economic Policy
1st February 2013
+
Importance of Early Childhood
Investment

Targeted early intervention programmes effective way of reducing socioeconomic inequalities in children’s skills

Well-designed interventions can generate a return to society ranging
from $1.80 to $17.07 for each dollar spent

YET primarily US-based evidence only on the effectiveness & costeffectiveness of such interventions

Argument for early intervention in more generous welfare systems is not
rooted in strong empirical evidence
+
Irish Context

Prevention & Early Intervention Programme (The Atlantic Philanthropies)
& Government , DCYA) initiated in 2004

Aim: to support the development, implementation and evaluation of
strategies for disadvantaged children

All programmes evaluated using experimental or quasi-experimental
design

Little tradition of social experimental evaluation in Ireland

Comparing the effectiveness of multiple programmes will enhanced
resource allocation and policy design
+ Preparing for Life Programme

Community-led initiative: operated by Northside Partnership in highly
disadvantaged area of Dublin, Ireland

Bottom-up approach: community initiative involving 28 community
groups, service providers, & local representatives

Evidence of need: Children scored below the norm on cognitive & noncognitive skills on school entry (Doyle & McNamara, 2011)

Aim: Improve levels of school readiness by assisting parents in developing
skills to prepare their children for school

Evaluation Design: RCT & embedded quasi-experimental design
+
Design of Preparing for Life
+
PFL Evaluation
Impact Evaluation

Aim: To determine if the programme/service is effective

PFL data collection: Pre-intervention (baseline), 6mths, 12mths, 18mths,
24mths, 3yrs, 4yrs, school entry
Implementation Evaluation

Aim: Delve into the blackbox of programme effectiveness & evaluate
fidelity to the model

PFL data collection:
1.
Implementation data
2.
Focus groups with participants
3.
Semi-structured interviews with programme staff
+ Recruitment

Eligibility Criteria:



Recruitment:



Cohort of pregnant women residing in PFL catchment area between Jan
2008-August 2010 (32 months)
Includes nulliparous and multiparous women
Maternity hospital at first booking visit (b/w 12-26 weeks)
Within the local community
Population-based recruitment rate, based on all live births during
the recruitment phase, was 52%
+
Randomisation Procedure

Individual-level randomisation


Unconditional probability randomisation strategy
Design procedure that preserves the integrity of randomization
ie. no opportunity for recruiter to intentionally influence assignment

Randomisation process

After informed consent obtained, participant pressed key on tablet laptop to
allocate her assignment condition

Evaluation & Implementation team automatically received an email with the
assigned ID number and treatment condition

Thus no opportunity to compromise the randomisation process
+ Baseline Analysis

Determine the effectiveness of the randomisation procedure

Baseline assessment conducted post randomisation, pre intervention
PFL Baseline Assessment
Proportion of Measures Not Significantly Different at
Baseline
Topic
PFL Low – PFL High
PFL – Comparison group
Family Socio-demographics
33/33 (0%)
27/33 (82%)
Maternal Well-being
24/24 (0%)
18/24 (75%)
Maternal Health & Pregnancy
35/35 (0%)
26/35 (74%)
Parenting & Childcare
10/13 (74%)
6/13 (46%)
Social Support
17/18 (94%)
9/9 (0%)
119/123 (97%)
86/114 (75%)
Total NOT Statistically Different
 Randomisation
worked!
+ Baseline Differences in Selected SocioDemographics
Low Treatment – High Treatment
Instrument
Mother’s Age
First-time Mothers
Mother Married
Mothers with Junior Certificate
Qualification or Lower
Mothers with Primary Degree
Mothers Unemployed
Fathers Unemployed
Residing in Social Housing
In Possession of a Medical Card
N
(nLOW/ nHIGH)
MLOW
(SD)
MHIGH
(SD)
p
Effect Size
(d)
ns
.03
ns
.09
ns
.09
ns
.12
ns
.01
ns
.05
ns
.24
ns
.00
ns
.14
205
25.30
25.46
(101/104)
(5.99)
(5.85)
205
0.50
0.54
(101/104)
(0.50)
(0.50)
205
0.18
0.14
(101/104)
(0.38)
(0.35)
205
0.40
0.34
(101/104)
(0.49)
(0.47)
205
0.03
0.03
(101/104)
(0.17)
(0.17)
205
0.41
0.43
(101/104)
(0.49)
(0.50)
198
0.31
0.43
(97/101)
(0.46)
(0.50)
204
0.55
0.55
(101/103)
(0.50)
(0.50)
205
0.66
0.60
(101/104)
(0.47)
(0.49)
+ Methodology
 Estimate treatment effects at birth, 6 months, 12 months
 Permutation based hypothesis testing (Heckman et al. 2010)



Classical hypothesis tests unreliable when the sample size is small and the
data are not normally distributed
Permutation test are distribution free, thus suitable in small samples
Based on the assumption of exchangeability between treatment
conditions under the null hypothesis
 Stepdown procedure (Romano & Wolf, 2005)



Ignoring the multiplicity of tests may lead to the rejection of “too many”
null hypotheses
Test multiple hypotheses simultaneously by controlling overall error rates
for vectors of hypotheses using the family-wise error rate (FWER) as a
criterion
Less conservative & more powerful than other methods as takes account
of statistical dependencies between tests
+ Analysis of Maternity Records

Data Collection


Summer 2012 accessed maternity records for PFL participants who attended
Rotunda Hospital (85%) & NMH (15%)
283 parents gave consent to access records - available records for 272
Maternity records
Infant Outcomes
Apgar 5
Birthweight (grams)
Low BW (<2500gr)
High BW (>4000gr)
N
(nHIGH/ nLOW)
MHIGH
(SD)
MLOW
(SD)
Individual Test
p
ns
180
9.79
9.69
(95/85)
(0.48)
(0.63)
181
3281
3294
(95/86)
(664)
(614)
181
0.10
0.05
(95/86)
(0.30)
(0.23)
181
0.09
0.12
(95/86)
(0.29)
(0.33)
ns
ns
ns
Labour Outcomes
Weeks at gestation
Prematurity (<37 wks)
Labour onset
(Spontaneous=0, Induced=1)
Caesarean section
174
39.60
39.56
(92/82)
(2.51)
(2.01)
174
0.07
0.06
(92/82)
(0.26)
(0.24)
178
0.31
0.43
(93/85)
(0.46)
(0.49)
178
0.15
0.25
(93/85)
(0.35)
(0.44)
ns
ns
p=0.056
P=0.036
+
PFL Maternity Records – Summary

No programme impact on birth weight or gestational age, but
programme altered the birth experiences of the participating families

Significantly fewer mothers in the high treatment group had a c-section
compared to those in the low treatment group

Result remains when the incidence of a previous c-section and
pariouness are controlled for

Programme particularly effective in preventing high treatment mothers
from having a repeat c-section

Result possibly linked to Tip Sheets which specifically focus on preparing
the mother for the birth and being aware of the birthing process and
possible labour outcomes
+
Analysis of 6 Month Results

Test for treatment effects across 160 measures & 25 Step-down
categories
PFL 6 Month Results
PFL Low – PFL High
Proportion of Measures Significantly
Different at 6 Months
Individual
Tests
Multiple Hypothesis
Tests
Child Development
0% (0/13)
0%
Child Health
Parenting
Home Environment & Safety
10% (3/30)
23% (5/22)
36% (8/22)
0%
20%
50%
Maternal Health
5% (1/20)
25%
Social Support
38% (5/13)
0%
Childcare & Service Use
7% (1/14)
0%
Household Factors & SES
0% (0/26)
0%
14% (23/160)
12% (3/25)
Total Statistically Different
Sample: PFL High/Low treatment groups: 84/90
+ EXAMPLE OF RESULTS:
CHILD DEVELOPMENT @ 6 MONTHS
ASQ Scores & Difficult
Temperament
ASQ Gross Motor Score
ASQ Communication Score
*Difficult Temperament
ASQ Personal Social Score
*ASQ Social-Emotional
ASQ Fine Motor Score
ASQ Problem Solving
N
(nHIGH/ nLOW)
MHIGH
(SD)
MLOW
(SD)
Individual
Test
p1
Step Down
Test
p2
Effect Size
d
ns
ns
0.18
ns
ns
0.16
ns
ns
0.09
ns
ns
0.05
ns
ns
0.03
ns
ns
0.06
ns
ns
0.07
173
40.78
38.50
(83/90)
(11.93)
(12.99)
173
53.07
51.78
(83/90)
(7.84)
(8.49)
173
11.70
12.21
(83/90)
(5.71)
(5.50)
172
46.52
45.94
(82/90)
(12.09)
(13.57)
173
14.76
15.17
(83/90)
(10.68)
(13.75)
173
50.84
51.39
(83/90)
(9.46)
(10.17)
173
51.87
52.56
(83/90)
(9.39)
(9.92)
Notes: 1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed
(right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was
reverse coded for the testing procedure. ‘ns’ indicates the variable is not statistically significant.
+
Positive Treatment Effects @ 6 Months

Child Development: none

Child Health: Appropriateness & frequency of eating; Immunizations

Parenting: Parent-child dysfunctional interactions; Baby comparison score; Parental hostilereactive behaviour; Interaction with baby

Home Environment: HOME sub-scales – variety, childcare, toys & books, physical
environment, learning materials; Electrical socket covers, Framingham safety survey

Mother’s Health: Mother less hospitalisation for complications after birth

Social support: Frequently sees grandparent; Discuss programme with partner & others;
Knows children same age as baby; Meet friends frequently

Childcare & Service use: Voted in last election

Household & SES: none
+
Analysis of 12 Month Results

Test for treatment effects across 140 measures & 23 Step-down
categories
Sample: 12 Months: PFL High/Low treatment groups: 83/82
+
Positive Treatment Effects @ 12 Months

Child Development: ASQ Fine Motor skills; BITSEA Social Competence cut-off

Child Health: Chest Infections; Immunizations; Appropriateness eating

Parenting: none

Home Environment: none

Mother’s Health: Drank alcohol

Social support: Meet friends frequently; Voted in last elections

Childcare & Service use: ~

Household & SES: none

Results consistent with the home visiting literature
+
Attrition & Dosage


Attrition/disengagement

Attrition may bias evaluation results if non-random

29% of high treatment & 29% of low treatment dropped out/disengaged between
randomisation & 12 months

Few individual characteristics predict attrition – IQ more important
Dosage/engagement

Variation in dosage, may lead of variation in effect

Collected data on number/duration of home visits

On average, high treatment group received 21 home visits of ~1 hr in duration
between pregnancy & 12 months
 58% of prescribed visits delivered based on bi-weekly visits

Higher maternal IQ associated with higher engagement
+ Conclusions to date

Many of the results are in hypothesized direction

Few significant effects, yet consistent with literature



Significant findings on birthing experiences, parenting, home environment, child
health, social support
Corresponded directly to information on the PFL Tip Sheets
No significant impact on many key domains such as child development, birth weight,
breastfeeding, maternal well-being etc.

Attrition & disengagement between 6 & 12 months very low

Engagement below prescribed dosage

Participate satisfaction is relatively high (focus groups)
+ Update on PFL

Oldest PFL child is 4 years and old & youngest is 18 months

24, 36 & 48 month surveys are currently in the field

18 month results available very soon

Over 1,600 interviews conducted as part of impact evaluation

Conducted five school readiness surveys with junior infant
cohorts in 2008, 2009, 2010, 2011, 2012

Evaluation will continue until all children start school
Programme website: www.preparingforlife.com
Evaluation website: http://geary.ucd.ie/preparingforlife/