Building Evidence in Education:
Conference for EEF Evaluators
11th July: Theory
12th July: Practice
www.educationendowmentfoundation.org.uk
Panel session 1: Grand designs
Experimental design for Achieve
Together
Ellen Greaves and Luke Sibieta
© Institute for Fiscal Studies
Achieve Together
• Bring together three programmes in a school:
– Teach First
– Teaching Leaders
– Future Leaders
• Intensive human capital investment
• Original motivation was also to encourage schools to work
together and to engage the local community and organisation in
school-improvement
 Cluster-design (two clusters)
 Difficult to evaluate quantitatively
© Institute for Fiscal Studies
Achieve Together
• Two pilots:
1. Area-based design
2. School-level human capital investment
© Institute for Fiscal Studies
Achieve Together
• Two pilots:
1. Area-based design
•
One-cluster in Bournemouth
•
4 primary schools and 6 secondary schools
•
Involvement of local community/organisations
•
Process evaluation
2. School-level human capital investment
© Institute for Fiscal Studies
Achieve Together
• Two pilots:
1. Area-based design
2. School-level human capital investment
•
School-level intervention
•
No co-ordination within clusters or involvement of external
organisations
•
48 schools recruited for RCT (24 treatment, 24 control)
•
Quantitative evaluation and process evaluation
© Institute for Fiscal Studies
Power calculations
0
0.1
0.2
0.3
0.4
0.5
Model 1
0.048
0.203
0.283
0.345
0.398
0.444
Model 2
0.052
0.220
0.306
0.373
0.430
0.480
Model 3
0.044
0.186
0.259
0.315
0.363
0.406
Note: Model 1 reports the detectable effect size when the variance of the outcome unexplained by
attributes of the pupils (including prior attainment) is 60%. Model 2 reports a less optimistic
scenario (70% unexplained), whilst Model 3 is more optimistic (50% unexplained)
© Institute for Fiscal Studies
Challenges
• School-level RCT began to look clustered...
• Cluster based recruitment
• Co-ordination between schools
• Complicates and creates risks for evaluation:
1. What can we learn from the evaluation?
2. How will the power calculations be affected?
© Institute for Fiscal Studies
Challenges
• School-level RCT began to look clustered...
• Cluster based recruitment
• Co-ordination between schools
• Complicates and creates risks for evaluation:
1. What can we learn from the evaluation?
•
Is positive impact due to the human capital approach?
•
Or better co-ordination between schools?
•
Our findings would be inconclusive
2. How will the power calculations be affected?
© Institute for Fiscal Studies
Challenges
• School-level RCT began to look clustered...
• Cluster based recruitment
• Co-ordination between schools
• Complicates and creates risks for evaluation:
1. What can we learn from the evaluation?
2. How will the power calculations be affected?
•
Required treatment effect from power calculations with clustering at
the school level looked ambitious...
•
Clustering may increase the intra-cluster correlation and increase
the challenge
© Institute for Fiscal Studies
Power calculations and the risk of clustering
• The ICC may be higher when schools are close together, all else
equal (pupils in an area are more similar than pupils in different
areas)
 Minor
© Institute for Fiscal Studies
Power calculations and the risk of clustering
• The ICC may be higher when schools are close together, all else
equal (pupils in an area are more similar than pupils in different
areas)
 Minor
• The ICC may be higher when schools work together
 Less minor
 Co-ordination across schools may make the pilot more
homogeneous, so pupils’ outcomes will become more correlated
 In the extreme case, we can think of the unit of treatment as the
cluster
 Uncertain risk for detectable effect size
 Distinct from cluster-based delivery where treatment may be more
heterogeneous across schools
© Institute for Fiscal Studies
Solutions
• Discussion
• Negotiate the focus with the project team
• Highlight the importance of:
• Clear conclusions from this important evaluation
• Maximising the probability that the effect size is detectable
© Institute for Fiscal Studies
Conclusions
• Cluster-based interventions increase risk for detectable effect
size
• Fine distinction between cluster-based interventions and clusterbased delivery
• Most crucial to define a clear treatment, and for the project team
to understand the importance of this
• Negotiation with project team is essential, and can be done!
© Institute for Fiscal Studies
Panel session 1: Grand designs
Piloting regression discontinuity design to
evaluate the effectiveness of a Saturday school
programme
Carole Torgerson, Andy Wiggins, Victoria Menzies and
Kirsty Younger
Durham University
USING RANDOMISED DESIGNS TO EVALUATE WRITING
INTERVENTIONS FOR CHILDREN DURING THE
TRANSITION BETWEEN PRIMARY AND SECONDARY
SCHOOL
Conference for EEF Evaluators: Building evidence in education
11th and 12th July
Hannah Ainsworth, Research Fellow, York Trials Unit, University of York
Buckley, H., Heaps, C., Jefferson, L., Mitchell, N., Torgerson, C, and Torgerson, D.
TRIAL DESIGN

Individually Randomised (Discover)

Cluster Randomised (Calderdale)

Split Plot Design (Exeter)
BACKGROUND

EEF Transitions Round:
 Evaluate 3 different writing interventions with aim to
improve children’s writing skills during transition for
primary school to secondary school
In all 3 trials: the primary outcome measure will be the
combined score on the 2 writing tasks within the
Progress in English 11 (Long Form), GL Assessment
(PiE11LF).
 Secondary outcomes will be scores on the reading,
spelling and grammar components of the PiE11LF

DISCOVER SUMMER SCHOOL
Research question:
“What is the effectiveness of the Discover summer
writing workshop intervention compared with a
“business as usual” control group on the writing
abilities of participating children?”
DISCOVER: INTERVENTION
Intervention group
 Discover Children’s Story Centre summer writing workshop
 4 week programme
 Specifically aimed at children who are less able writers.
 The summer school programme will include:
 Poetry and literacy sessions, including opportunities to engage
with professional poets and writers
 Other ‘enrichment activities,’ including drama and sports
activities
 Trips to West End performances and visits to the Olympic Park.
Control group
 Business as Usual
 Offered opportunity to attend a Saturday writing workshop in the
autumn term.
DISCOVER: TRIAL DESIGN
 Individually
randomised controlled trial
 Aim to recruit 250 pupils, randomised on a 1:1
ratio to the intervention and control groups
Year 6 pupils will be eligible to be included if they:
1) Are predicted to achieve between level 3c and level 4c in
English by the end of Key Stage 2 (based on teacher
assessment).
2) Attend a participating primary school
3) Attend a participating secondary school
DISCOVER: TRIAL DESIGN DIAGRAM
DISCOVER: CURRENT PROGRESS
Progress to date





Recruitment: 124 pupils
Randomisation ratio changed to 3:2 to allow greater numbers in the
intervention group in order to fill more spaces in the summer school
Pupils have now been allocated to Intervention/Control
Summer school taking place 29th July to 23rd August
Post-test being delivered in first 2 weeks of autumn term.
Challenges encountered to date



Short time frame involved – parents already had plans for summer holidays
(reduced consent rates, but also expected to reduce summer school
attendance rates)
Potential contamination as other summer schools available to control group
(and intervention group) pupils
Identification of pupils challenging in transition from Year 6 to Year 7 as must
be able to complete post-test to be eligible
CALDERDALE IMPROVING WRITING QUALITY
Research question:
 What is the effectiveness of the Improving
Writing Quality programme compared with
“business as usual” on the writing skills of
participating children?
CALDERDALE: INTERVENTION
Intervention Group
 The intervention will be offered to all pupils in Year 6
 Pupils who are predicted to achieve Level 3 or an insecure
Level 4 in English (based on teacher assessment) will
continue to receive intervention in Year 7 (Autumn term).
 Memorable experiences for pupils in Year 6.
 Professional development for primary and secondary school
teachers in key elements of the writing intervention SelfRegulated Strategy Development (SRSD), including
discussion, modelling and planning.
Control Group
 Business as Usual
 Primary schools will be trained following year
CALDERDALE: TRIAL DESIGN
 Pragmatic
cluster randomised controlled trial.
 Approximately
24 schools (feeding into 3
secondary schools) randomised on a 1:1 ratio
to intervention and control groups
CALDERDALE: TRIAL DESIGN DIAGRAM
CALDERDALE: CURRENT PROGRESS
Progress to date
 Recruitment: 23 Primary Schools (possible 24); 3 secondary schools
(possible 4)
 Successful Information Meeting, to explain intervention and
evaluation, hosted by delivery partner and evaluators
 Opt Out consent for data sharing: 22 parents opted out
 Randomised Primary Schools in March 2013
 Intervention delivery on-going
 Post test December 2013
Challenges encountered to date
 Secure provision of pupil level data at appropriate time
EXETER: GRAMMAR FOR WRITING
Research questions:
1. “What is the effectiveness of the whole class Grammar for writing
intervention compared with a “business as usual” control group on
writing skills of participating children?”
2. “What is the effectiveness of the whole class Grammar for writing
intervention plus additional small group intervention compared with a
“business as usual” control group on writing skills of participating
children?”
3. “What is the effectiveness of the whole class Grammar for writing
intervention plus additional small group intervention compared with the
whole class Grammar for writing intervention only on writing skills of
participating children?”
EXETER: INTERVENTION
Intervention:
Cluster Level
 15 sequential guided writing sessions delivered during
literacy classes for the whole glass
 Writing sessions delivered using a number of techniques
(teacher led, DVD, power point, group exercises)
Individual level
 Whole class intervention plus additional guided writing
sessions delivered in a small group setting
Control
Cluster Level
 Business as usual
Individual Level
 Whole class intervention only
EXETER: TRIAL DESIGN
 Pragmatic
cluster randomised trial with a split
plot design
 Two levels of randomisation – class and
individual
Class
 Within each school, one class randomly allocated to
whole class intervention and one to business as usual
Individual
 Within intervention class, pupils expected to achieve
Level 3 or borderline Level 4 in English by the end of
Key Stage 2 (based on teacher assessment),
individually randomised to whole class plus small
group intervention or whole class only intervention
EXETER: TRIAL DESIGN DIAGRAM
EXETER: CURRENT PROGRESS
Progress to date:
 55 schools recruited & randomised
 2 schools have since dropped out
 Intervention delivered between 27th May – 5th July
 Post intervention testing between 1st July – 18th July
Challenges encountered:
 Short time frames challenging
 Numerous protocol deviations
CONCLUSIONS
Large variety of randomised designs available
 Consider specific nature and circumstances
 Consider carefully the exact research question/s
you will be able to answer with chosen design


Any Questions?