1
Supplemental Online Material for
Head Start Participation and School Readiness:
Evidence from the Early Childhood Longitudinal Study-Birth Cohort
Developmental Psychology
RaeHyuck Lee, Fuhua Zhai, Jeanne Brooks-Gunn, Wen-Jui Han, and Jane Waldfogel
Corresponding author:
Jane Waldfogel
Columbia University School of Social Work
1255 Amsterdam Avenue, New York, NY 10027
E-mail: jw205@columbia.edu
In this supplemental online material, we provide additional detailed information about
multiple imputation for missing data and propensity score matching. We also include an
appendix table showing descriptive statistics of all covariates of HS and other care groups after
matching.
Multiple Imputation for Missing Data
To see if five imputed data sets are enough to recover missing values, we calculated the
expected Relative Efficiency (RE) based on Rubin’ s (1987) equation and confirmed that the RE
ranged from 98.0 to 98.4 percent.
To obtain more accurate imputed data sets, we included outcome measures in the
multiple imputation, as recommended (Moons, Donders, Stijnen, & Harrell Jr., 2006). However,
we used the original outcome measures, not the imputed ones, when estimating the effects of HS
on outcomes.
Propensity Score Matching
Researchers commonly face selection bias issues with a non-experimental design,
because the estimated effects could be biased by the fact that children who attend HS differ from
2
children who do not participate (i.e., the poorest of the poor are likely to attend HS) (Gormley,
Phillips, Adelstein, & Shaw, 2010; Schnur, Brooks-Gunn, & Shipman, 1992). Indeed, studies
have shown that HS participants are strongly negatively selected and without adjusting for this
selection bias, effects of HS could be masked (Currie & Thomas, 1995).
We used a radius matching method with a caliper, which is beneficial in that it uses not
only the single nearest neighbor within each caliper but all of the control units within the caliper,
allowing for the use of additional comparison units when better matches are possible (Dehejia &
Wahba, 2002). Radius matching with a caliper achieved better balance than alternative matching
methods (such as nearest neighbor matching). Following Rosenbaum and Rubin (1985), we set a
caliper at 0.01, which was smaller than 0.25 times a standard deviation of the predicted
propensity scores.
Whether propensity score matching procedures are successful can be tested by comparing
the extent of balance and overlap of the propensity scores of the treatment and comparison
groups (Dehejia & Wahba, 2002; Gibson-Davis & Foster, 2006).
Initially, the propensity scores after matching in specific comparisons of HS with other
non-parental care and pre-K were unbalanced in several covariates; to address this issue, we
added various interaction terms into the propensity score matching procedure and achieved
balance in all covariates. For the comparison of HS and other non-parental care, we obtained
perfect balance by including interaction terms between whether a child was a twin and 1)
mother’s education, 2) family income at the 9-month wave, 3) family income at the 2-year wave,
and 4) region of country. For the comparison of HS with pre-K, we achieved the best balance
after including interaction terms between region of country and 1) maternal employment at the 2year wave, 2) family income at the 9-month wave, 3) family income at the 2-year wave, and 4)
3
child race/ethnicity.
We examined the distributions of propensity scores between the matched treatment and
comparison groups in each comparison to see whether both groups in each comparison had
enough common support regions. In all of the five imputed data sets, the distributions in
comparisons of HS with parental care, other non-parental care, and pre-K were roughly
symmetric. The distribution of the comparison group in the comparison of HS with other centerbased care was skewed to the left, but it also showed enough overlap (results are available from
the authors upon request).
4
REFERENCES
Currie, J., & Thomas, D. (1995). Does Head Start make a difference? American Economic
Review, 85, 341-364.
Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental
causal studies. Review of Economics and Statistics, 84(1), 151-161.
Gibson-Davis, C., & Foster, E. (2006). A cautionary tale: Using propensity scores to estimate the
effect of Food Stamps on food insecurity. Social Service Review, 80(1), 93-126.
Gormley, W. T., Phillips, D., Adelstein, S., & Shaw, C. (2010). Head Start's comparative
advantage: Myth or reality? Policy Studies Journal Volume, 38(3), 397-418.
Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate
matched sampling methods that incorporate the propensity score. American Statistician,
39(1), 33-38.
Rubin, D. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley &
Sons.
Schnur, E. S., Brooks-Gunn, J., & Shipman, V. (1992). Who attends programs serving poor
children?: The case of Head Start attendees and non-attendees. Journal of Applied
Developmental Psychology, 13, 405-421.
5
Table A
Descriptive Statistics of Head Start and Other Specific Care Arrangements after Matching
Other centerPre-K
based care
after
after
HS
matching
matching
(n ≈ 1,150)
(n ≈ 1,000)
(n ≈ 2,700)
Variable
M
SD
M
SD
M
SD
Child care arrangements
9-months
Parental care
0.51 0.50
0.50
0.50
0.47 0.50
Relative care
0.31 0.46
0.33
0.47
0.32 0.47
Non-relative care
0.10 0.30
0.11
0.31
0.13 0.34
Center-based care
0.08 0.26
0.06
0.25
0.08 0.28
2-years
Parental care
0.52 0.50
0.49
0.50
0.46 0.50
Relative care
0.24 0.43
0.28
0.45
0.26 0.44
Non-relative care
0.10 0.30
0.11
0.35
0.13 0.34
Center-based care
0.14 0.35
0.12
0.33
0.15 0.36
Child characteristics
Boys
0.50 0.50
0.51
0.50
0.48 0.50
Age in months at 9-months
10.52 1.92
10.51
1.96
10.45 1.92
Race/ethnicity at 9-months
Non-Hispanic White
0.21 0.40
0.19
0.39
0.21 0.41
Non-Hispanic Black
0.29 0.46
0.33
0.47
0.28 0.45
Hispanic
0.28 0.45
0.30
0.46
0.30 0.46
Non-Hispanic Asian
0.04 0.20
0.03
0.18
0.06 0.23
Other
0.18 0.38
0.14
0.35
0.15 0.35
Low birth weight at birth
0.25 0.44
0.26
0.44
0.26 0.44
Prematurity at birth
0.28 0.45
0.30
0.46
0.29 0.45
Multiple birth at birth
0.16 0.37
0.17
0.38
0.18 0.38
Breast-feeding at 9-months
None
0.45 0.50
0.47
0.50
0.42 0.49
Other nonparental care
after
matching
(n ≈ 600)
M
SD
Parental care
after
matching
(n ≈ 1,250)
M
SD
0.49
0.29
0.11
0.10
0.50
0.45
0.32
0.31
0.54
0.31
0.10
0.06
0.50
0.46
0.29
0.23
0.51
0.24
0.09
0.15
0.50
0.43
0.29
0.36
0.55
0.26
0.09
0.10
0.50
0.44
0.29
0.30
0.51
10.48
0.50
1.82
0.49
10.56
0.50
2.07
0.24
0.24
0.29
0.04
0.19
0.27
0.30
0.16
0.43
0.43
0.45
0.20
0.39
0.46
0.46
0.37
0.22
0.25
0.31
0.05
0.17
0.26
0.27
0.16
0.41
0.43
0.46
0.22
0.38
0.44
0.45
0.37
0.45
0.50
0.43
0.50
6
<3 months
3~6 months
7 months plus
Number of siblings at 9months
No sibling
One
Two or more
Health status at 2-years
Poor/Fair
Good
Very good
Excellent
Maternal characteristics
Age at birth
Mom lived with bio-mother until 16
Mom lived with bio-father until 16
Married at birth
Depression at 9-months
Employment status at 2-years
Not working
Full-time
Part-time
Foreign-born
English is primary home language at 2-years
Health status at 2-years
Poor/Fair
Good
Very good
Excellent
Parenting behaviors
KIDI at 9-months
Cognitively stimulating activities at 2-years
Having sleeping routines at 2-years
0.24 0.43
0.16 0.37
0.15 0.35
0.24
0.17
0.12
0.43
0.37
0.33
0.26
0.20
0.13
0.44
0.40
0.34
0.27
0.16
0.12
0.44
0.37
0.33
0.24
0.16
0.17
0.43
0.37
0.37
0.38 0.49
0.29 0.46
0.32 0.47
0.39
0.26
0.35
0.49
0.44
0.48
0.37
0.28
0.35
0.48
0.45
0.48
0.35
0.36
0.28
0.48
0.48
0.45
0.35
0.27
0.38
0.48
0.45
0.49
0.04
0.14
0.29
0.54
0.19
0.34
0.45
0.50
0.07
0.13
0.30
0.50
0.25
0.33
0.46
0.50
0.07
0.12
0.31
0.50
0.25
0.32
0.46
0.50
0.04
0.13
0.27
0.55
0.19
0.34
0.45
0.50
0.04
0.15
0.30
0.51
0.20
0.36
0.46
0.50
25.16
0.79
0.50
0.44
0.54
6.10
0.41
0.50
0.50
0.52
25.00
0.84
0.50
0.41
0.56
6.21
0.36
0.50
0.49
0.54
25.14
0.82
0.51
0.49
0.54
5.71
0.38
0.50
0.49
0.55
24.24
0.79
0.45
0.42
0.55
6.32
0.40
0.50
0.49
0.51
25.23
0.80
0.49
0.48
0.55
6.16
0.40
0.50
0.50
0.51
0.52
0.33
0.15
0.25
0.77
0.50
0.47
0.36
0.43
0.42
0.51
0.34
0.15
0.27
0.78
0.50
0.47
0.36
0.44
0.41
0.49
0.38
0.14
0.27
0.75
0.50
0.48
0.34
0.44
0.43
0.50
0.34
0.16
0.23
0.80
0.50
0.47
0.36
0.42
0.40
0.55
0.30
0.15
0.29
0.73
0.50
0.46
0.36
0.46
0.44
0.13
0.30
0.30
0.26
0.34
0.46
0.46
0.44
0.11
0.35
0.30
0.24
0.31
0.48
0.46
0.43
0.15
0.29
0.32
0.23
0.36
0.46
0.47
0.42
0.10
0.32
0.31
0.27
0.30
0.47
0.46
0.45
0.13
0.29
0.34
0.24
0.34
0.46
0.47
0.43
5.81 2.08
8.81 2.08
0.79 0.41
6.01
8.75
0.82
1.93
2.11
0.39
6.01
8.95
0.80
2.17
2.03
0.40
6.06
8.76
0.83
1.95
1.94
0.38
5.79
8.89
0.78
2.06
2.04
0.42
7
Eating dinner together per week at 2-years
5.29 2.30
5.02
2.38
5.16 2.36
5.14 2.42
5.05 2.41
No spanking at 2-years
0.58 0.49
0.55
0.50
0.56 0.50
0.55 0.50
0.62 0.49
Family characteristics
Parent's education at birth
Below high school (0-11)
0.23 0.42
0.22
0.41
0.20 0.40
0.23 0.42
0.25 0.43
High school (12)
0.38 0.49
0.38
0.49
0.42 0.49
0.33 0.49
0.37 0.48
Some college (13-15)
0.31 0.46
0.31
0.46
0.31 0.46
0.38 0.47
0.30 0.46
Above college (16+)
0.08 0.27
0.09
0.29
0.07 0.26
0.07 0.25
0.08 0.27
Parental occupational prestige at 9 months
1.66 2.07
1.75
2.12
1.88 2.08
1.74 2.14
1.57 2.06
Family income at 9-months
$0-$20000
0.49 0.50
0.46
0.50
0.46 0.50
0.46 0.50
0.50 0.50
$20001-$35000
0.31 0.46
0.31
0.46
0.31 0.46
0.31 0.46
0.28 0.45
$35001-$50000
0.16 0.37
0.18
0.39
0.17 0.38
0.16 0.38
0.16 0.37
$50001+
0.04 0.20
0.05
0.21
0.06 0.24
0.07 0.25
0.06 0.23
Family income at 2-years
$0-$20000
0.47 0.50
0.41
0.49
0.42 0.49
0.45 0.50
0.45 0.50
$20001-$35000
0.30 0.46
0.32
0.47
0.30 0.46
0.30 0.46
0.28 0.45
$35001-$50000
0.19 0.39
0.21
0.41
0.22 0.42
0.18 0.39
0.22 0.41
$50001+
0.05 0.22
0.06
0.23
0.06 0.23
0.06 0.24
0.05 0.22
Lived in urban area at 2-years
0.76 0.43
0.80
0.40
0.80 0.40
0.83 0.37
0.79 0.40
Region of country at 2-years
Northeast
0.13 0.34
0.16
0.37
0.14 0.35
0.15 0.36
0.13 0.33
Midwest
0.23 0.42
0.21
0.40
0.21 0.41
0.22 0.42
0.20 0.40
South
0.39 0.49
0.44
0.50
0.39 0.49
0.36 0.48
0.37 0.48
West
0.25 0.43
0.19
0.39
0.26 0.44
0.27 0.44
0.30 0.46
Mom or child received WIC at 2-years
0.73 0.44
0.70
0.46
0.69 0.46
0.72 0.45
0.70 0.46
Family members received FS 2-years
0.44 0.50
0.41
0.49
0.41 0.49
0.40 0.49
0.43 0.49
Family members received TANF at 2-years
0.15 0.36
0.12
0.33
0.13 0.35
0.19 0.39
0.16 0.37
Note. Descriptive statistics of other specific care arrangements after matching were reported from the best matched one of the five
imputed data sets. Sample sizes were rounded to the nearest 50, due to IES reporting rules. All numbers in the column for HS were
slightly different with those in the same column in Table 1 because these numbers were not weighted to be compared with those of
each comparison group after matching. Two-tailed t tests were conducted to compare means between HS participants and other
children who belonged to each specific type of care arrangement after propensity score matching; no significant differences were
8
found. WIC = Special Supplemental Nutrition Program for Women, Infants, and Children; FS = Food Stamp Program; TANF =
Temporary Assistance for Needy Families.