Building Evidence in Education:
Workshop for EEF evaluators
2nd June: York
6th June: London
www.educationendowmentfoundation.org.uk
The EEF by numbers
2,300
schools
participating
in projects
34
topics in
the Toolkit
14
6,000
members
of EEF
team
500,000
pupils involved
in EEF projects
heads
presented to
since launch
£210
16
m
estimated spend
over lifetime of
the EEF
10
reports
published
independent
evaluation
teams
83
evaluations
funded to
date
Session 1: Design
Adapting Design
Carole Torgerson (Durham)
David Torgerson (York Trials Unit)
Calculating effect sizes
Adetayo Kasim (Durham)
Calculating Effect Sizes for
Cluster Randomized Trials
Adetayo Kasim
Main Points
• Over estimation of effect size when CLUSTER
LEVEL ANALYSIS is used.
• Disconnection between hypothesis testing and
effect size from MULTILEVEL MODELS
Calculating Effect Size
Calculating Effect Size
• Cluster level analysis (CLA) - Two stage approach
• Summarise data to cluster level
• Calculate effect size using summarised data
• Multilevel models (MLM)
• Analyse pupils level data, but accounts for intra cluster
correlation
• Calculate effect size using WITHIN cluster variability
Calculating Effect Size
Illustration 1: Simulation study
Calculating Effect Size
Simulation 1: MEAN, SE and SD
Value
Mean
SE
SD
Method
MLM
0.41(0.4)
0.39(0.59)
0.41(0.75)
0.39(0.91)
0.4(1.11)
0.38(1.33)
CLA
0.41(0.4)
0.39(0.59)
0.41(0.75)
0.39(0.91)
0.4(1.11)
0.38(1.33)
MLM
0.43(0.06)
0.57(0.13)
0.73(0.18)
0.89(0.22)
1.07(0.27)
1.28(0.33)
CLA
0.39(0.10)
0.56(0.14)
0.73(0.18)
0.89(0.22)
1.07(0.27)
1.28(0.33)
MLM - W
1.98(0.15) 1.99(0.15)
1.99(0.15)
2.00(0.15)
1.99(0.15)
2.00(0.15)
MLM - T
2.00(0.14) 2.10(0.16)
2.23(0.20)
2.38(0.24)
2.56(0.31)
2.80(0.39)
CLA
0.62(0.15) 0.89(0.23) 1.15(0.29) 1.40(0.35)
• MEAN(SD) from 10,000 simulated data
1.69(0.43)
2.03(0.52)
Calculating Effect Size
Simulation 2: Calculating effect size assuming within
cluster variance
Effect
size
0.2
0.3
0.4
Method
0
0.1
0.2
0.3
0.4
0.5
MLM - W
0.20(0.20) 0.20(0.26) 0.21(0.30)
0.20(0.36)
0.2(0.42)
0.19(0.49)
MLM - T
0.20(0.20) 0.18(0.23) 0.17(0.26)
0.15(0.28)
0.13(0.29)
0.11(0.28)
CLA
0.64(0.69) 0.50(0.67) 0.43(0.68)
0.36(0.66)
0.31(0.67)
0.25(0.66)
MLM - W
0.30(0.20) 0.3(0.25)
0.3(0.36)
0.3(0.42)
0.31(0.49)
MLM - T
0.29(0.20) 0.28(0.23) 0.25(0.26)
0.22(0.28)
0.20(0.29)
0.17(0.29)
CLA
0.95(0.72) 0.76(0.68) 0.63(0.69)
0.52(0.67)
0.46(0.67)
0.39(0.67)
MLM - W
0.41(0.21) 0.40(0.40) 0.40(0.31)
0.40(0.36)
0.40(0.42)
0.40(0.50)
MLM - T
0.39(0.20) 0.37(0.24) 0.34(0.26)
0.30(0.28)
0.26(0.29)
0.22(0.30)
CLA
1.26(0.74) 1.00(0.72) 0.84(0.70)
0.71(0.69)
0.61(0.69)
0.52(0.67)
•
0.3(0.30)
MEAN(SD) of Hedges Effect Size from 10,000 simulated data
Calculating Effect Size
• Cluster level analysis may overestimates effect size
when between variability is negligible and there is
substantial variability within clusters
• Effect sizes based on within cluster variance and total
variance from multilevel model are comparable when
between cluster variance is negligible
• Using only within cluster variance could result in
different conclusions based on effect sizes and
hypothesis testing when there is a substantial
variability between clusters
Discussion
Cluster level analysis
OR
Within cluster Variance from multilevel model
OR
Total variance from multilevel model
?
References
Tymms P., Merrell C. and Henderson B. (1997) The first year at
school: a quantitative investigation of the attainment and
progress of pupils. Educational research and Evaluation, 3(2),
pp. 101 - 118
Larry V. Hedges (2007) Effect sizes in cluster-randomized
designs. Journal of Educational and Behavioural Statistics,
32(4), pp. 341-370
A. Brand, M.T. Bradley, L.A. Best, G. Stoica (2008) Accuracy
of effect size estimates from published psychological research.
Perceptual and Motor Skills, 106 (2) (2008), pp. 645–649