Conference for EEF evaluators:
Building evidence in education
Session 3: Implementation
Hannah Ainsworth, York Trials Unit, University of
York
Professor David Torgerson, York Trials Unit,
University of York
Professor Carole Torgerson, School of Education,
Durham University
Session 3: Implementation
• Trial registration and CONSORT (HA – 10 mins.)
• Trial management (HA - 30mins)
• Model/approach (HA – 5mins)
– Discussion (5mins)
• Protocol and other tools (HA – 5mins)
– Discussion (5mins)
• Relationship with delivery partner (HA – 5mins)
– Discussion (5mins)
• Recruitment and retention (CT – 20 mins.)
Trial registration
• Register trial with Current Controlled Trials
at outset before beginning recruitment.
• You will be allocated an ISRCTN
http://www.controlled-trials.com/isrctn/submission/
Why is it important to register trials?
»Public knowledge
»Reduce duplication
»Increase opportunities for
collaboration
»Reduce selective reporting and
over reporting
»Reduce publication bias
CONSORT
• Conduct and report trial to the CONSORT
standards
• http://www.consort-statement.org/consortstatement/
• What is CONSORT?
• Why is it important?
• How can it help?
CONSORT checklist
CONSORT checklist
CONSORT flow diagram
CONSORT flow diagram cluster trials
From Campbell MK, Piaggio G, Elbourne DR, Altman DG; for the CONSORT Group. Consort
2010 statement: extension to cluster randomised trials. BMJ. 2012 Sep 4;345:e5661.
Trial management
• Trial management approach/model
• Developing a trial protocol and other trial
management tools
• Relationship with delivery partner
Trial management model/approach
• Current EEF model: Light touch approach to trial
management as delivery partner is often taking
responsibility for many of the ‘usual’ trial
management responsibilities
• Think about everything you would normally
do/take responsibility for as trial manager.
document this and share with delivery partner.
• Offer advice and guide the process
Discussion
• What are YOUR experiences of trial
management within EFF evaluations?
• What are the advantages and
disadvantages of the current
model/approach?
• Possible solutions?
Trial Protocol and other tools
• Develop a trial protocol as evaluation team
• Discuss and develop trial protocol with
delivery partner
• Produce clear timeframes/deadlines which
both evaluation team and delivery partner
can work to
Produce evaluation diagrams
Primary Schools n = 24
Secondary Schools n = 3
Children in target group n = 288
(based on average 12 children per school)
Baseline data collection
Information on all Year 6 pupils including Key stage 2 English Teacher Assessments from Dec 2012
Primary School
Randomisation
Control Group
Intervention Group
Primary Schools N = 12
Primary Schools N = 12
No intervention
Intervention in Year 6
continued intervention in Year
7 in Secondary Schools.
Follow up data collection Dec 2013
•
Progress in English 11 (long form) (Conducted in Secondary School)
Long term follow up
Routine test results and pupil characteristics recorded in National Pupil Database
Clear timeframes
Provide clear information
• Help delivery partner develop information for
schools, parents and children
• Work with delivery partner to ensure clear
instructions are given to schools
• Help delivery partner develop school agreement
documents
• Help develop parent consent forms/opt out forms
Discussion
• Has the trial protocol been a useful shared
document?
• What other tools have YOU used to aid
the process?
Relationship with delivery partner
• Intervention developer has a lot invested in the
intervention
• Evaluator must remain in equipoise
• Can be a challenging relationship
• Try to explain that it is important you remain
impartial
• Refrain from voicing your own opinions about
the intervention – let the research speak for itself
Relationship with delivery partner
• Manage expectations
• Be clear from outset who is responsible for what
• Be clear from outset on the data you will require,
when and the format you will require it in
• Provide clear instructions for secure data
transfer
Discussion
• What are YOUR experiences of the
relationship with the delivery partner?
• How can challenges be overcome?
Recruitment and retention (CT)
• Randomisation ensures absence of
selection bias
• Selection bias can still be introduced
during recruitment or because of attrition
Recruitment bias
• Potential sources:
» Developer-led recruitment
» Timing
» Randomisation of clusters before recruitment of
individuals
» Teacher not linked to class before randomisation
• Possible solutions:
» Evaluators fully involved in recruitment process
» Randomise after recruitment of clusters and within
clusters
» Ensure teachers are linked to classes before
randomisation
Attrition bias
• Attrition after randomisation can introduce
bias
» Those who leave a trial tend to be different
from those who remain in the trial
» If there is unequal attrition this is worrying
» All efforts must be made to retain participants
after randomisation for post-tests even if they
don’t receive the intervention
Example of attrition bias
Random allocation
160 children 8 from
Each school
1 school 8 children withdrew
N = 17 children replaced
following
discussion with teacher
76 children allocated to control
76 allocated to intervention
group
Discussion
• Discuss any issues you have experienced
with the developer-led recruitment.