www.sciencemag.org/content/351/6278/aad8008/suppl/DC1
Supplementary Materials for
Comment on “Math at home adds up to achievement in school”
Michael C. Frank
E-mail: mcfrank@stanford.edu
Published 11 March 2015, Science 351, aad8008 (2016)
DOI: 10.1126/science.aad8008
This PDF file includes:
Methods
Other Supplementary Material for this manuscript includes the following:
(available at www.sciencemag.org/content/351/6278/aad8008/suppl/DC1)
Data File (RMD)
Berkowitz et al. (2015) Technical Comment Supplementary Materials
Michael C. Frank
2016-01-12
This document reports code for a Technical Comment on Berkowitz et al. (2015). Data are loaded from
Berkowitz et al (2015)’s supplementary materials. This PDF file was generated from an R Markdown script
(.Rmd file), which can be opened in any text editor and is also attached as part of the Supplementary
Materials for the Technical Comment. When the script is “knitted” using the knitr package, it will generate
this PDF file.
d <- read.csv("aac7427-Accessory-Data-File-S1.csv") %>%
mutate(condition = ifelse(cond.dum == 1, "Math App","Reading App"))
Exclusions and Data Preparation
We need to exclude:
• twins (twin)
• dropouts during the first year (year1dropout)
We also see that we need to include 34 children due to experimenter errors (24 fall, 16 math and 8 reading;
10 in spring, 9 math and 1 reading). This is in the filter variables (e.g., wjappliedfa13_filter). These
are binary variables that presumably could be conveying whether the children’s scores should be included
because they get basal and ceiling items. We do this after excluding twins and dropouts.
sum(d$wjappliedfa13_filter[d$twin == 0 & d$year1dropout == 0] == 0)
## [1] 24
sum(d$WJ.applied.Sp14.filter[d$twin == 0 & d$year1dropout == 0] == 0)
## [1] 11
These numbers are very close to the paper’s numbers, though we do get 11 in the spring (not 10). Let’s do
the same for reading, where there were 25 exclusions (7 in fall, 4 math and 3 reading; 18 in spring, 12 math
and 6 reading).
sum(d$WJletterwordFa13_filter[d$twin == 0 & d$year1dropout == 0] == 0)
## [1] 7
sum(d$WJ.letterword.Sp14.filter[d$twin == 0 & d$year1dropout == 0] == 0)
## [1] 19
1
Again we’re one off (18 vs. 19 in spring), but quite close. Note there is a separate issue in exclusions for
analyses that use math anxiety: where math anxiety questionnaire data is missing (~20% of cases), you have
to exclude data. I’m assuming the authors just dropped non-responding families for these analyses.
dt.excl <- d %>%
mutate(applied.w.fa13 = ifelse(wjappliedfa13_filter,
WJ.Applied.Problems.W.Score.Fa13,
NA),
applied.w.sp14 = ifelse(WJ.applied.Sp14.filter,
WJ.Applied.Problems_W.Score_Sp14,
NA),
reading.w.fa13 = ifelse(WJletterwordFa13_filter,
WJ.Letter.Word.ID.W.Score.Fa13,
NA),
reading.w.sp14 = ifelse(WJ.letterword.Sp14.filter,
WJ.LetterWord.Wscore.Sp14,
NA)) %>%
filter(!twin, !year1dropout) %>%
gather(measure, score,
applied.w.fa13, applied.w.sp14,
reading.w.fa13, reading.w.sp14) %>%
separate(measure, c("measure","time"), sep="w\\.") %>%
mutate(year = as.factor(str_sub(time, 4, 5)),
measure = ifelse(grepl( "applied", measure), "Math", "Reading"))
Let’s check the means now that we are doing exclusions. Table S1 of the original paper gives means, albeit
using a median split on math anxiety (e.g. not grand means).
Low anxious:
• math grp: 463 (19) in fall and 479 (21) in spring
• reading grp: 459 (18) in fall and 475 (19) in spring
High anxious:
• math grp: 459 (16) in fall and 474 (19) in spring
• reading grp: 458 (14) in fall and 469 (19) in spring
dt.excl %>%
filter(measure == "Math") %>%
filter(!is.na(parentMA_mediansplit)) %>%
group_by(parentMA_mediansplit, condition, year) %>%
summarise(mean = mean(score, na.rm=TRUE),
sd = sd(score, na.rm=TRUE))
##
##
##
##
##
##
##
Source: local data frame [8 x 5]
Groups: parentMA_mediansplit, condition [?]
parentMA_mediansplit
(dbl)
1
1
2
1
condition
year
mean
sd
(chr) (fctr)
(dbl)
(dbl)
Math App
3 462.7468 18.71507
Math App
4 478.4348 21.11036
2
##
##
##
##
##
##
3
4
5
6
7
8
1
1
2
2
2
2
Reading
Reading
Math
Math
Reading
Reading
App
App
App
App
App
App
3
4
3
4
3
4
458.8571
475.3134
459.0562
474.3836
457.4800
469.2157
18.08212
19.25602
15.98633
18.67978
13.56352
18.68402
We get only minor numerical differences between the data I’m using and the data reported by the authors
(e.g., <= 1 W score unit).
Difference scores.
diffs.excl <- dt.excl %>%
select(ChildID, condition, year, measure, score) %>%
mutate(measure_year = interaction(measure, year)) %>%
select(-year, -measure) %>%
spread(measure_year, score) %>%
mutate(math_gain = Math.4 - Math.3,
reading_gain = Reading.4 - Reading.3) %>%
select(-Math.3, -Math.4, -Reading.3, -Reading.4) %>%
gather(measure, score, math_gain, reading_gain)
Do the whole thing with grade equivalents.
dt.excl.ge <- d %>%
mutate(applied.w.fa13 = ifelse(wjappliedfa13_filter,
WJ.Applied.Problems.GE.Fa13,
NA),
applied.w.sp14 = ifelse(WJ.applied.Sp14.filter,
WJ.Applied.Problem_GE_Sp14,
NA),
reading.w.fa13 = ifelse(WJletterwordFa13_filter,
WJ.Letter.Word.ID.GE.Fa13,
NA),
reading.w.sp14 = ifelse(WJ.letterword.Sp14.filter,
WJ.LetterWord_GE_Sp14,
NA)) %>%
filter(!twin, !year1dropout) %>%
gather(measure, score,
applied.w.fa13, applied.w.sp14,
reading.w.fa13, reading.w.sp14) %>%
separate(measure, c("measure","time"), sep="w\\.") %>%
mutate(year = as.factor(str_sub(time, 4, 5)),
measure = ifelse(grepl( "applied", measure), "Math", "Reading"))
diffs.excl.ge <- dt.excl.ge %>%
select(ChildID, condition, year, measure, score) %>%
mutate(measure_year = interaction(measure, year)) %>%
select(-year, -measure) %>%
spread(measure_year, score) %>%
mutate(math_gain = Math.4 - Math.3,
reading_gain = Reading.4 - Reading.3) %>%
select(-Math.3, -Math.4, -Reading.3, -Reading.4) %>%
gather(measure, score, math_gain, reading_gain)
3
And join back in other measures.
diffs.full <- left_join(diffs.excl, d %>%
select(ChildID, ParentMAaverage.Fa13,
parentMA_mediansplit,
avg.use, use.012.groups)) %>%
left_join(diffs.excl.ge %>% rename(score.ge = score))
And compute means for plotting.
ms.diffs.excl <- diffs.excl %>%
group_by(condition, measure) %>%
multi_boot_standard(column = "score", na.rm=TRUE)
ms.diffs.excl.ge <- diffs.excl.ge %>%
group_by(condition, measure) %>%
multi_boot_standard(column = "score", na.rm=TRUE) %>%
ungroup %>%
mutate(measure = factor(measure,
levels = c("math_gain","reading_gain"),
labels = c("WJ Applied Problems\n(Math)",
"WJ Letter-Word\n(Reading)")),
Condition = condition)
Technical Comment Analyses
This section will now reproduce the analyses in the Technical Comment.
Figure 1.
ggplot(ms.diffs.excl.ge, aes(x = measure, y = mean,
ymin = ci_lower, ymax = ci_upper,
fill = Condition)) +
geom_bar(stat="identity", position = "dodge") +
geom_linerange(position = position_dodge(width = .9)) +
scale_fill_solarized() +
ylab("Improvement (Grade Equivalents)") +
xlab("Measure") + ylim(c(0,1))
4
Improvement (Grade Equivalents)
1.00
0.75
Condition
Math App
0.50
Reading App
0.25
0.00
WJ Applied Problems
(Math)
WJ Letter−Word
(Reading)
Measure
t-tests for Figure 1.
kable(tidy(with(diffs.excl.ge,
t.test(score[condition
measure
score[condition
measure
==
==
==
==
"Math App" &
"math_gain"],
"Reading App" &
"math_gain"], var.equal = TRUE))))
estimate1
estimate2
statistic
p.value
parameter
conf.low
conf.high
0.852973
0.77
1.061383
0.2890304
498
-0.0706193
0.2365653
kable(tidy(with(diffs.excl.ge,
t.test(score[condition
measure
score[condition
measure
==
==
==
==
"Math App" &
"reading_gain"],
"Reading App" &
"reading_gain"], var.equal = TRUE))))
estimate1
estimate2
statistic
p.value
parameter
conf.low
conf.high
0.8337731
0.8192308
0.2728889
0.7850497
507
-0.0901546
0.1192392
Main longitudinal models (for paragraph beginning “first”). GE scores and then W scores.
kable(summary(lmer(score ~ condition * time
+ (1 | ChildID)
5
+ (time | TeachID.Year1),
data = filter(dt.excl.ge,
measure == "Math")))$coefficients, digits = 3)
Estimate
Std. Error
t value
1.985
-0.075
0.842
-0.072
0.088
0.169
0.051
0.098
22.656
-0.444
16.650
-0.733
(Intercept)
conditionReading App
timesp14
conditionReading App:timesp14
kable(summary(lmer(score ~ condition * time
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl,
measure == "Math")))$coefficients, digits = 3)
(Intercept)
conditionReading App
timesp14
conditionReading App:timesp14
Estimate
Std. Error
t value
458.914
-1.518
15.368
-1.003
1.735
3.356
0.897
1.742
264.453
-0.452
17.141
-0.576
Three-way interaction models (for paragraph beginning “second”). GE and then W scores, as above.
kable(summary(lmer(score ~ condition * time * avg.use
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl.ge,
measure == "Math")))$coefficients, digits = 3)
(Intercept)
conditionReading
timesp14
avg.use
conditionReading
conditionReading
timesp14:avg.use
conditionReading
App
App:timesp14
App:avg.use
App:timesp14:avg.use
Estimate
Std. Error
t value
1.770
0.068
0.650
0.174
0.088
-0.128
0.157
-0.137
0.103
0.199
0.070
0.046
0.134
0.073
0.040
0.063
17.184
0.340
9.324
3.770
0.658
-1.756
3.968
-2.190
kable(summary(lmer(score ~ condition * time * avg.use
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl,
measure == "Math")))$coefficients, digits = 3)
6
(Intercept)
conditionReading
timesp14
avg.use
conditionReading
conditionReading
timesp14:avg.use
conditionReading
App
App:timesp14
App:avg.use
App:timesp14:avg.use
Estimate
Std. Error
t value
454.559
0.964
12.897
3.546
1.226
-2.381
2.021
-1.867
2.002
3.875
1.237
0.858
2.379
1.361
0.703
1.109
227.015
0.249
10.429
4.132
0.515
-1.749
2.876
-1.684
Figure 2.
ggplot(filter(diffs.full,
!is.na(parentMA_mediansplit),
condition == "Math App"),
aes(x = avg.use, y = score.ge,
col = factor(parentMA_mediansplit))) +
geom_point() +
geom_smooth(se=TRUE, method = "lm") +
scale_colour_solarized() +
xlab("Average Weekly App Usage") +
ylab("Improvement (Grade Equivalents)")
Improvement (Grade Equivalents)
3
2
factor(parentMA_mediansplit)
1
2
1
0
−1
0
1
2
3
4
Average Weekly App Usage
Math anxiety three- and four-way interactions (for paragraph beginning “third”).
7
kable(summary(lmer(score ~ time * condition * ParentMAaverage.Fa13
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl.ge,
measure == "Math")))$coefficients, digits = 3)
(Intercept)
timesp14
conditionReading App
ParentMAaverage.Fa13
timesp14:conditionReading App
timesp14:ParentMAaverage.Fa13
conditionReading App:ParentMAaverage.Fa13
timesp14:conditionReading App:ParentMAaverage.Fa13
Estimate
Std. Error
t value
2.281
1.036
-0.038
-0.112
0.167
-0.085
-0.020
-0.119
0.163
0.126
0.324
0.062
0.249
0.053
0.127
0.107
14.010
8.201
-0.117
-1.802
0.670
-1.611
-0.157
-1.117
Estimate
Std. Error
t value
464.190
17.718
-0.948
-1.937
3.740
-1.082
-0.280
-2.370
3.073
2.211
6.120
1.152
4.374
0.915
2.355
1.860
151.051
8.014
-0.155
-1.682
0.855
-1.183
-0.119
-1.275
kable(summary(lmer(score ~ time * condition * ParentMAaverage.Fa13
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl,
measure == "Math")))$coefficients, digits = 3)
(Intercept)
timesp14
conditionReading App
ParentMAaverage.Fa13
timesp14:conditionReading App
timesp14:ParentMAaverage.Fa13
conditionReading App:ParentMAaverage.Fa13
timesp14:conditionReading App:ParentMAaverage.Fa13
kable(summary(lmer(score ~ time * condition * ParentMAaverage.Fa13 * avg.use +
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl,
measure == "Math")))$coefficients, digits = 3)
(Intercept)
timesp14
conditionReading App
ParentMAaverage.Fa13
avg.use
timesp14:conditionReading App
timesp14:ParentMAaverage.Fa13
conditionReading App:ParentMAaverage.Fa13
timesp14:avg.use
8
Estimate
Std. Error
t value
459.443
12.277
-4.837
-1.589
3.049
10.657
0.000
2.647
3.899
4.621
3.598
8.722
1.832
2.876
6.722
1.488
3.472
2.339
99.432
3.412
-0.555
-0.867
1.061
1.585
0.000
0.762
1.667
conditionReading App:avg.use
ParentMAaverage.Fa13:avg.use
timesp14:conditionReading App:ParentMAaverage.Fa13
timesp14:conditionReading App:avg.use
timesp14:ParentMAaverage.Fa13:avg.use
conditionReading App:ParentMAaverage.Fa13:avg.use
timesp14:conditionReading App:ParentMAaverage.Fa13:avg.use
Estimate
Std. Error
t value
3.203
-0.071
-3.990
-4.899
-0.759
-2.413
1.137
4.750
1.252
2.816
3.857
1.018
2.037
1.655
0.674
-0.056
-1.417
-1.270
-0.746
-1.184
0.687
kable(summary(lmer(score ~ time * condition * ParentMAaverage.Fa13 * avg.use +
+ (1 | ChildID)
+ (time | TeachID.Year1),
data = filter(dt.excl.ge,
measure == "Math")))$coefficients, digits = 3)
(Intercept)
timesp14
conditionReading App
ParentMAaverage.Fa13
avg.use
timesp14:conditionReading App
timesp14:ParentMAaverage.Fa13
conditionReading App:ParentMAaverage.Fa13
timesp14:avg.use
conditionReading App:avg.use
ParentMAaverage.Fa13:avg.use
timesp14:conditionReading App:ParentMAaverage.Fa13
timesp14:conditionReading App:avg.use
timesp14:ParentMAaverage.Fa13:avg.use
conditionReading App:ParentMAaverage.Fa13:avg.use
timesp14:conditionReading App:ParentMAaverage.Fa13:avg.use
9
Estimate
Std. Error
t value
2.038
0.596
-0.142
-0.089
0.156
0.629
0.013
0.091
0.321
0.096
-0.008
-0.226
-0.333
-0.072
-0.093
0.077
0.247
0.205
0.466
0.099
0.155
0.382
0.085
0.187
0.134
0.256
0.068
0.161
0.220
0.058
0.110
0.095
8.249
2.910
-0.305
-0.904
1.004
1.645
0.150
0.488
2.405
0.374
-0.117
-1.410
-1.513
-1.233
-0.848
0.818