Conference - Day 1 Session 3 slides

Conference for EEF evaluators:
Building evidence in education
Session 3: Implementation
Hannah Ainsworth, York Trials Unit, University of
Professor David Torgerson, York Trials Unit,
University of York
Professor Carole Torgerson, School of Education,
Durham University
Session 3: Implementation
• Trial registration and CONSORT (HA – 10 mins.)
• Trial management (HA - 30mins)
• Model/approach (HA – 5mins)
– Discussion (5mins)
• Protocol and other tools (HA – 5mins)
– Discussion (5mins)
• Relationship with delivery partner (HA – 5mins)
– Discussion (5mins)
• Recruitment and retention (CT – 20 mins.)
Trial registration
• Register trial with Current Controlled Trials
at outset before beginning recruitment.
• You will be allocated an ISRCTN
Why is it important to register trials?
»Public knowledge
»Reduce duplication
»Increase opportunities for
»Reduce selective reporting and
over reporting
»Reduce publication bias
• Conduct and report trial to the CONSORT
• What is CONSORT?
• Why is it important?
• How can it help?
CONSORT checklist
CONSORT checklist
CONSORT flow diagram
CONSORT flow diagram cluster trials
From Campbell MK, Piaggio G, Elbourne DR, Altman DG; for the CONSORT Group. Consort
2010 statement: extension to cluster randomised trials. BMJ. 2012 Sep 4;345:e5661.
Trial management
• Trial management approach/model
• Developing a trial protocol and other trial
management tools
• Relationship with delivery partner
Trial management model/approach
• Current EEF model: Light touch approach to trial
management as delivery partner is often taking
responsibility for many of the ‘usual’ trial
management responsibilities
• Think about everything you would normally
do/take responsibility for as trial manager.
document this and share with delivery partner.
• Offer advice and guide the process
• What are YOUR experiences of trial
management within EFF evaluations?
• What are the advantages and
disadvantages of the current
• Possible solutions?
Trial Protocol and other tools
• Develop a trial protocol as evaluation team
• Discuss and develop trial protocol with
delivery partner
• Produce clear timeframes/deadlines which
both evaluation team and delivery partner
can work to
Produce evaluation diagrams
Primary Schools n = 24
Secondary Schools n = 3
Children in target group n = 288
(based on average 12 children per school)
Baseline data collection
Information on all Year 6 pupils including Key stage 2 English Teacher Assessments from Dec 2012
Primary School
Control Group
Intervention Group
Primary Schools N = 12
Primary Schools N = 12
No intervention
Intervention in Year 6
continued intervention in Year
7 in Secondary Schools.
Follow up data collection Dec 2013
Progress in English 11 (long form) (Conducted in Secondary School)
Long term follow up
Routine test results and pupil characteristics recorded in National Pupil Database
Clear timeframes
Provide clear information
• Help delivery partner develop information for
schools, parents and children
• Work with delivery partner to ensure clear
instructions are given to schools
• Help delivery partner develop school agreement
• Help develop parent consent forms/opt out forms
• Has the trial protocol been a useful shared
• What other tools have YOU used to aid
the process?
Relationship with delivery partner
• Intervention developer has a lot invested in the
• Evaluator must remain in equipoise
• Can be a challenging relationship
• Try to explain that it is important you remain
• Refrain from voicing your own opinions about
the intervention – let the research speak for itself
Relationship with delivery partner
• Manage expectations
• Be clear from outset who is responsible for what
• Be clear from outset on the data you will require,
when and the format you will require it in
• Provide clear instructions for secure data
• What are YOUR experiences of the
relationship with the delivery partner?
• How can challenges be overcome?
Recruitment and retention (CT)
• Randomisation ensures absence of
selection bias
• Selection bias can still be introduced
during recruitment or because of attrition
Recruitment bias
• Potential sources:
» Developer-led recruitment
» Timing
» Randomisation of clusters before recruitment of
» Teacher not linked to class before randomisation
• Possible solutions:
» Evaluators fully involved in recruitment process
» Randomise after recruitment of clusters and within
» Ensure teachers are linked to classes before
Attrition bias
• Attrition after randomisation can introduce
» Those who leave a trial tend to be different
from those who remain in the trial
» If there is unequal attrition this is worrying
» All efforts must be made to retain participants
after randomisation for post-tests even if they
don’t receive the intervention
Example of attrition bias
Random allocation
160 children 8 from
Each school
1 school 8 children withdrew
N = 17 children replaced
discussion with teacher
76 children allocated to control
76 allocated to intervention
• Discuss any issues you have experienced
with the developer-led recruitment.