Conference for EEF evaluators: Building evidence in education Session 3: Implementation Hannah Ainsworth, York Trials Unit, University of York Professor David Torgerson, York Trials Unit, University of York Professor Carole Torgerson, School of Education, Durham University Session 3: Implementation • Trial registration and CONSORT (HA – 10 mins.) • Trial management (HA - 30mins) • Model/approach (HA – 5mins) – Discussion (5mins) • Protocol and other tools (HA – 5mins) – Discussion (5mins) • Relationship with delivery partner (HA – 5mins) – Discussion (5mins) • Recruitment and retention (CT – 20 mins.) Trial registration • Register trial with Current Controlled Trials at outset before beginning recruitment. • You will be allocated an ISRCTN http://www.controlled-trials.com/isrctn/submission/ Why is it important to register trials? »Public knowledge »Reduce duplication »Increase opportunities for collaboration »Reduce selective reporting and over reporting »Reduce publication bias CONSORT • Conduct and report trial to the CONSORT standards • http://www.consort-statement.org/consortstatement/ • What is CONSORT? • Why is it important? • How can it help? CONSORT checklist CONSORT checklist CONSORT flow diagram CONSORT flow diagram cluster trials From Campbell MK, Piaggio G, Elbourne DR, Altman DG; for the CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012 Sep 4;345:e5661. Trial management • Trial management approach/model • Developing a trial protocol and other trial management tools • Relationship with delivery partner Trial management model/approach • Current EEF model: Light touch approach to trial management as delivery partner is often taking responsibility for many of the ‘usual’ trial management responsibilities • Think about everything you would normally do/take responsibility for as trial manager. document this and share with delivery partner. • Offer advice and guide the process Discussion • What are YOUR experiences of trial management within EFF evaluations? • What are the advantages and disadvantages of the current model/approach? • Possible solutions? Trial Protocol and other tools • Develop a trial protocol as evaluation team • Discuss and develop trial protocol with delivery partner • Produce clear timeframes/deadlines which both evaluation team and delivery partner can work to Produce evaluation diagrams Primary Schools n = 24 Secondary Schools n = 3 Children in target group n = 288 (based on average 12 children per school) Baseline data collection Information on all Year 6 pupils including Key stage 2 English Teacher Assessments from Dec 2012 Primary School Randomisation Control Group Intervention Group Primary Schools N = 12 Primary Schools N = 12 No intervention Intervention in Year 6 continued intervention in Year 7 in Secondary Schools. Follow up data collection Dec 2013 • Progress in English 11 (long form) (Conducted in Secondary School) Long term follow up Routine test results and pupil characteristics recorded in National Pupil Database Clear timeframes Provide clear information • Help delivery partner develop information for schools, parents and children • Work with delivery partner to ensure clear instructions are given to schools • Help delivery partner develop school agreement documents • Help develop parent consent forms/opt out forms Discussion • Has the trial protocol been a useful shared document? • What other tools have YOU used to aid the process? Relationship with delivery partner • Intervention developer has a lot invested in the intervention • Evaluator must remain in equipoise • Can be a challenging relationship • Try to explain that it is important you remain impartial • Refrain from voicing your own opinions about the intervention – let the research speak for itself Relationship with delivery partner • Manage expectations • Be clear from outset who is responsible for what • Be clear from outset on the data you will require, when and the format you will require it in • Provide clear instructions for secure data transfer Discussion • What are YOUR experiences of the relationship with the delivery partner? • How can challenges be overcome? Recruitment and retention (CT) • Randomisation ensures absence of selection bias • Selection bias can still be introduced during recruitment or because of attrition Recruitment bias • Potential sources: » Developer-led recruitment » Timing » Randomisation of clusters before recruitment of individuals » Teacher not linked to class before randomisation • Possible solutions: » Evaluators fully involved in recruitment process » Randomise after recruitment of clusters and within clusters » Ensure teachers are linked to classes before randomisation Attrition bias • Attrition after randomisation can introduce bias » Those who leave a trial tend to be different from those who remain in the trial » If there is unequal attrition this is worrying » All efforts must be made to retain participants after randomisation for post-tests even if they don’t receive the intervention Example of attrition bias Random allocation 160 children 8 from Each school 1 school 8 children withdrew N = 17 children replaced following discussion with teacher 76 children allocated to control 76 allocated to intervention group Discussion • Discuss any issues you have experienced with the developer-led recruitment.