Practice and research in
education:
How can we make both better,
and better aligned?
Robert Coe
ResearchED 2013, Dulwich College, 7 Sept 2013
@ProfCoe
Improving practice and research
 Problems with research and evidence
 Can research tell us what works?
 How can practice be improved?
–
–
–
–
Think hard about learning
∂
Invest in good CPD
Evaluate teaching quality
Evaluate impact of changes
Improving Education: A triumph of hope over experience
http://www.cem.org/attachments/publications/ImprovingEducation2013.pdf
2
Problems with research and
evidence
3
Problems with evidence
 Evidence can be found to support any
position in education
 Ofsted asks schools to produce evidence to
demonstrate that PP ∂spending has narrowed
the gap (even though some of it may not
have)
 DfE misunderstands/misuses evidence
(relative gaps as percentage difference;
small changes with small samples)
4
Problems with research






Quality varies, but a lot is not very good
Quality really matters
How do you know who or what to trust?
∂
Academic papers are inaccessible
Academic debates are (mostly) pointless
Peer review doesn’t work
5
Small positives
 Impact agenda requires ‘public benefit’
 EEF: funding for high quality evaluation
 Recurrent policy interest in Evidence-Based
∂
Education (see http://www.cem.org/evidencebased-education/introduction)
 Social media & internet gives instant critique,
debate, interaction
6
Evidence about the
effectiveness of different
strategies
7
Toolkit of Strategies to Improve Learning
∂
The Sutton Trust-EEF Teaching and Learning Toolkit
http://www.educationendowmentfoundation.org.uk/toolkit/
www.educationendowmentfoundation.org.uk/toolkit
Impact vs cost
Effect Size (months gain)
Promising
8
May be
worth it
Feedback
Meta-cognitive
Peer tutoring
Homework
(Secondary)
Collaborative
Early Years
1-1 tuition
∂
Behaviour
Small gp
Phonics
Parental
tuition
involvement
ICT
Social
Individualised Summer
schools
learning
Mentoring
Homework
(Primary)
Performance Aspirations
0
pay
£0 Ability grouping
Cost per pupil
Smaller
classes
After
school
Teaching
assistants
£1000
Not
worth it
Key messages
 Some things that are popular or widely
thought to be effective probably can’t
improve learning
– Ability grouping (setting); After-school clubs;
∂
Teaching assistants; Smaller classes;
Performance pay; Raising aspirations
 Some things look ‘promising’
– Effective feedback; Meta-cognitive and self
regulation strategies; Peer tutoring/peer‐assisted
learning strategies; Homework
Clear, simple advice:
 Choose from the top left
 Go back to school and do it
∂
For every complex problem
there is an answer that is
clear, simple, and wrong
H.L. Mencken
11
Why not?
 We have been doing some of these things for a
long time, but have generally not seen
improvement
 Research evidence is problematic
– Sometimes the existing evidence is thin
∂ reflect real life
– Research studies may not
– Context and ‘support factors’ may matter
 Implementation is problematic
– We may think we are doing it, but are we doing it right?
– We do not know how to get large groups of teachers
and schools to implement these interventions in ways
that are faithful, effective and sustainable
12
So how might practice be
improved?
13
Four steps to improvement




Think hard about learning
Invest in good professional development
Evaluate teaching ∂quality
Evaluate impact of changes
1. Think hard about
learning
www.educationendowmentfoundation.org.uk/toolkit
Impact vs cost
Effect Size (months gain)
Promising
8
May be
worth it
Feedback
Meta-cognitive
Peer tutoring
Homework
(Secondary)
Collaborative
Early Years
1-1 tuition
∂
Behaviour
Small gp
Phonics
Parental
tuition
involvement
ICT
Social
Individualised Summer
schools
learning
Mentoring
Homework
(Primary)
Performance Aspirations
0
pay
£0 Ability grouping
Cost per pupil
Smaller
classes
After
school
Teaching
assistants
£1000
Not
worth it
Poor Proxies for Learning
 Students are busy: lots of work is done
(especially written work)
 Students are engaged, interested, motivated
 Students are getting attention: feedback,
explanations
∂
 Classroom is ordered, calm, under control
 Curriculum has been ‘covered’ (ie presented to
students in some form)
 (At least some) students have supplied correct
answers (whether or not they really understood
them or could reproduce them independently)
17
A simple theory of learning
Learning happens
when people have
to think hard
∂
2. Invest in effective CPD
How do we get students to learn hard things?
Eg
 Place value
 Persuasive
writing
 Music
composition
 Balancing
chemical
equations
• Explain what they should do
• Demonstrate it
• Get them to do it (with
gradually
reducing support)
∂
• Provide feedback
• Get them to practise until it is
secure
• Assess their skill/
understanding
How do we get teachers to learn hard things?
Eg
 Using formative
assessment
 Assertive
discipline
 How to teach
algebra
• Explain what they should do
∂
What CPD helps learners?
 Intense: at least 15 contact hours, preferably 50
 Sustained: over at least two terms
 Content focused: on teachers’ knowledge of
subject content & how students learn it
∂
 Active: opportunities to try it out & discuss
 Supported: external feedback and networks to
improve and sustain
 Evidence based: promotes strategies
supported by robust evaluation evidence
3. Evaluate teaching
quality
Classroom observation:
The new Brain Gym?
 Validity evidence
– Are observation ratings really a reflection of
teaching quality?
∂
 Impact evaluation
– Does the process of observation and feedback
lead to improvement?
– In what, how much and for what cost?
24
Validity evidence

Do observation ratings correspond with other
indicators of teaching quality or effectiveness?
–
–
–
–

Student learning gains
Student ratings
Peer (teacher) perceptions
Self ratings
Are they consistent?
– Across occasions
– Across raters

∂
Are ratings influenced by spurious confounds
–
–
–
–
–
Charisma
Confidence
Subject matter
Students’ behaviour
Time of day
25
Does observation improve teaching?
 Need studies with
–
–
–
–
–
Clearly defined intervention
High quality outcome measures (student learning)
∂
Good control of counterfactual
(eg RCT)
Adequate sample
Measures of sustained impact
 Just one would be nice …
26
4. Evaluate impact of
changes
Mistaking School Improvement (1)
(Coe, 2009)
1. Wait for a bad year or choose underperforming
schools to start with. Most things self-correct or revert
to expectations (you can claim the credit for this).
2. Take on any initiative, and ask everyone who put
effort into it whether they feel it worked. No-one wants
∂
to feel their effort was wasted.
3. Define ‘improvement’ in terms of perceptions and
ratings of teachers. DO NOT conduct any proper
assessments – they may disappoint.
4. Only study schools or teachers that recognise a
problem and are prepared to take on an initiative.
They’ll probably improve whatever you do.
Mistaking School Improvement (2)
(Coe, 2009)
5. Conduct some kind of evaluation, but don’t let the
design be too good – poor quality evaluations are
much more likely to show positive results.
6. If any improvement occurs in any aspect of
∂
performance, focus attention
on that rather than on
any areas or schools that have not improved or got
worse (don’t mention them!).
7. Put some effort into marketing and presentation of
the school. Once you start to recruit better students,
things will improve.
Key elements of good evaluation
 Clear, well
defined, replicable
intervention
∂
 Good assessment
of appropriate
outcomes
 Well-matched
comparison group
Summary …
 A lot of educational research is rubbish, but
some is very good: relevant and rigorous
 Four steps to improve practice:
– Think hard about learning
– Invest in good CPD ∂
– Evaluate teaching quality (but not with dodgy
observation)
– Evaluate impact of changes
Robert.Coe@cem.dur.ac.uk
www.cem.org
@ProfCoe