Leadership for teacher
learning
Dylan Wiliam
1
www.dylanwiliam.net
Outline
2

In the UK, teacher quality is
 highly
consequential, and
 highly variable

Teacher quality can be improved by
 replacing
existing teachers with better ones
 by investing in the teachers we already have


Only investing in existing teachers produces
enough improvement to produce the changes that
we need
This is not happening systematically in most
schools, but could be
Educational Endowment Fund (EEF) toolkit
3
Intervention
Cost
Quality of
evidence
Extra months
of learning
Feedback
££

+8
Metacognition and self-regulation
££

+8
Peer tutoring
££

+6
Early years intervention
£££££

+6
One to one tuition
££££

+5
Homework (secondary)
£

+5
Collaborative learning
£

+5
Phonics
£

+4
Small group tuition
£££

+4
Behaviour interventions
£££

+4
Digital technology
££££

+4
£

+4
Social and emotional learning
Educational Endowment Fund (EEF) toolkit
4
Intervention
Cost
Quality of
evidence
Extra months
of learning
Parental involvement
£££

+3
£££££

+3
Summer schools
£££

+3
Sports participation
£££

+2
Arts participation
££

+2
Extended school time
£££

+2
Individualized instruction
£

+2
After school programmes
££££

+2
£

+2
£££

+1
£

+1
Reducing class size
Learning styles
Mentoring
Homework (primary)
Educational Endowment Fund (EEF) toolkit
5
Intervention
Cost
Quality of
evidence
Extra months
of learning
Teaching assistants
££££

0
Performance pay
££

0
Aspiration interventions
£££

0
Block scheduling
£

0
School uniform
£

0
Physical environment
££

0
Ability grouping
£

-1
Important caveats about research findings
6


Educational research can only tell us what was, not
what might be.
Moreover, in education, “What works?” is rarely
the right question, because
 everything works
somewhere, and
 nothing works everywhere, which is why
 in education, the right question is, “Under what
conditions does this work?”
An illustrative example: feedback
7


Kluger and DeNisi (1996) review of 3000 research reports
Excluding those:







without adequate controls
with poor design
with fewer than 10 participants
where performance was not measured
without details of effect sizes
left 131 reports, 607 effect sizes, involving 12652
individuals
On average, feedback increases achievement


Effect sizes highly variable
38% (50 out of 131) of effect sizes were negative
Getting feedback right is hard
Response type
Feedback indicates performance…
falls short of goal
exceeds goal
Change behavior
Increase effort
Exert less effort
Change goal
Reduce aspiration
Increase aspiration
Abandon goal
Decide goal is too hard
Decide goal is too easy
Reject feedback
Feedback is ignored
Feedback is ignored
Teacher quality
9
Teacher quality and student learning
Subject
Correlation
Woodhead
All
0*
Rivkin, Hanushek and Kain (2005)
Reading
>0.10
Rivkin, Hanushek and Kain (2005)
Mathematics
>0.11
Rockoff (2004)
Reading
0.20
Rockoff (2004)
Mathematics
0.25
Aaronson, Barrow and Sander (2007)
Mathematics
0.13
Annual growth in achievement, by age
11
1.6
annual growth (SDs)
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
5
6
7
8
9
10
11
Age
Bloom, Hill, Black, and Lipsey (2008)
12
13
14
15
16
12

Assuming that
one year’s student growth is 0.3 standard deviations
 the correlation between teacher quality and student
achievement is 0.15




Then with a good teacher (1sd above the mean)
students learn 50% more
And with an outstanding teacher (2sd above the mean)
they learn 100% more
Note also that students make some progress through
maturation so these are probably underestimates of
the true effect (Fitzpatrick, Grissmer, & Hastedt, 2011)
Teacher quality
13

The impact of teacher quality (Hanushek & Rivkin, 2006)
In the classroom of the best teacher in a group of 50
teachers, students learn twice as fast as average.
 In the classroom of the least effective teacher in a group of
50, students learn half as fast as average
 And in the classrooms of the best teachers, students from
disadvantaged backgrounds learn as much as others (Hamre
& Pianta, 2005)



Teachers make a difference
But what makes the difference in teachers?

In particular, can we predict student progress from:
Teacher qualifications?
 Value-added?
 Teacher observation?

Teacher qualifications
14
Teacher qualifications and student progress
15
Mathematics
Primary
Middle
Reading
High
Primary
Middle
—
—
+
General theory of
education courses
Teaching practice
courses
Pedagogical
content courses
Advanced
university courses
Aptitude test
scores
Harris and Sass (2007)
+
High
+
—
—
+
Teacher observations
16
Framework for teaching (Danielson 1996)
17

Four domains of professional practice
1.
2.
3.
4.

Planning and preparation
Classroom environment
Instruction
Professional responsibilities
Links with student achievement (Sartain, et al. 2011)
 Domains
1 and 4: no impact on student achievement
 Domains 2 and 3: some impact on student achievement
A framework for teaching (Danielson, 1996)

Domain 2: The classroom environment
2a: Creating an environment of respect and rapport
 2b: Establishing a culture for learning
 2c: Managing classroom procedures
 2d: Managing student behavior
 2e: Organizing physical space


Domain 3: Instruction
3a: Communicating with students
 3b: Using questioning and discussion techniques
 3c: Engaging students in learning
 3d: Using assessment in instruction
 3e: Demonstrating flexibility and responsiveness

Reading
Teacher value-added
0.5
0.4
2a
0.3
2b
0.2
2c
0.1
2d
0
2e
-0.1
3a
-0.2
3b
-0.3
3c
-0.4
3d
-0.5
3e
-0.6
Unsatisfactory
Basic
Proficient
Distinguished
Mathematics
0.6
2a
0.4
Teacher value-added
2b
0.2
2c
2d
0
2e
3a
-0.2
3b
3c
-0.4
3d
-0.6
-0.8
3e
Unsatisfactory
Basic
Proficient
Distinguished
Observations and teacher quality
21
Percentage change in rate of learning
Reading
20
15
Mathematics
So, the highest rated teachers are 30%
more productive than the lowest rated
10
5
0
-5
But the best teachers are 400% more
productive than the least effective
-10
-15
Unsatisfactory
Basic
Proficient
Distinguished
Sartain, Stoelinga, Brown, Luppescu, Matsko, Miller, Durwood, Jiang, and Glazer (2011)
Imprecision of lesson observations
22
Achieving a reliability of
0.9 in judging teacher
quality through lesson
observation is likely to
require observing a
teacher teaching 6
different classes, and for
each lesson to be judged
by 5 independent
observers.
Hill, Charalambous and Kraft (2012)
23

To recap
The highest rated teachers generate learning 30% faster
than the lowest rated teachers
 But the most effective teachers generate learning 400%
faster than the least effective teachers
 So the best observation systems we have capture less than
10% of teacher quality (in reality, less than this because
most teachers are in the middle two categories)


There is therefore a real danger that teachers will
“game the system” by aping features of observation
protocols while becoming less effective.
Teacher value-added
24
Teacher value-added
Value-added quintile:
Traditional model
Value-added quintile: Student fixed-effects model
1
2
3
4
5
1
38
22
24
16
0
2
26
28
15
20
11
3
20
20
20
24
16
4
13
24
26
13
24
5
9
5
12
28
47
Goldhaber, Goldschmidt, and Tseng (2013)
Mathematics
Quintile of teacher value-added
5th
4th
3rd
2nd
1st
Value-added (student SDs)
0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3
0
1
2
Years of teaching experience
Atteberry, Loeb and Wyckoff (2013)
3
4
Correlation of initial and later performance
27
Correlation of value-added with base year
All teachers
New teachers
0.5
Mathematics
0.4
0.3
0.2
0.1
0
Base+1
Base+2
Base+3
Base+4
Year of prediction
Base+5
Correlation of initial and later performance
28
Correlation of value-added with base year
All teachers
New teachers
0.5
Reading
0.4
0.3
0.2
0.1
0
Base+1
Base+2
Base+3
Base+4
Year of prediction
Base+5
Issues with value-added models for teachers
29



Different (reasonable) models of value-added give
very different estimates of teacher quality
(Goldhaber, Goldschmidt and Tseng, 2013)
Teacher value-added in their first year accounts for
less than 5% of the variation in teacher quality in
their fifth year of teaching (Atteberry, Loeb and
Wyckoff, 2013)
Teachers benefit students for at least three years
after they stop teaching them (Rothstein, 2010)
What makes effective teacher
learning?
30
Expertise
31

According to Berliner (1994), experts:








Excel mainly in their own domain
Often develop automaticity for the repetitive operations that
are needed to accomplish their goals
Are more sensitive to the task demands and social situation
when solving problems
Are more opportunistic and flexible in their teaching than
novices
Represent problems in qualitatively different ways than novices
Have faster and more accurate pattern recognition capabilities
Perceive meaningful patterns in the domain in which they are
experienced
Begin to solve problems slower but bring richer and more
personal sources of information to bear
The role of deliberate practice
32





Music professors at the Hochschule der Kuenst (Academy of
Music) Berlin identified 10 violin students who had the
potential for careers as international soloists (“best”
students)
The professors also identified a sample of 10 good, but not
outstanding students (“good” students)
Researchers recruited another 10 students training to be
music teachers who specialised in the violin (“Music Ed”
students)
An additional 10 middle-aged professional violinists from
two local orchestras were recruited to the study
Groups were matched in sex (7f, 3m) and for the first three
groups, age
How much do violinists practice?
33
Music Ed
Good
Best
Professionals
Hours of practie per week
35
30
25
20
15
10
5
0
4
9
14
Age
Ericsson, Krampe, & Tesch-Römer (1993)
19
Violinists’ hours of practice (cumulative)
34
Cumulative hours of practice
Music Ed
Good
Best
10
12
Age
Professionals
10000
8000
6000
4000
2000
0
4
6
8
14
16
18
20
These differences are substantial…
35
Hours of practice by age 18
Music Education students
3420
Good violin students
Best violin students
5301
7410
Professional musicians
7336


By the age of 18, the best violinists have accumulated 40%
more practice than good violinists
Since the amount of deliberate practice being undertaken
by the best students once they are adults is close to the
maximum possible, it is, essentially, impossible for the
good students to catch up to the best.
Psychomotor skills and surgical skill…


Ten junior surgical residents were tested on their ability
to perform five anastomoses on fresh porcine jenunum
Performance evaluated on:
a. Time taken in minutes
b. A “clumsiness” score (0 to 100), taking into account
“unproductive movements, fumbles and gross errors”
c. An anastomosis score found by rating on a 1 to 3 scale:






suture spacing
knots
overall tidiness
integrity of the anastomosis (tested with 10 cm head of water)
accuracy of suture in terms of extramucosal placement
Cumulative error score: a + b + 2 x (15 – c)
…are only weakly related
37

Residents also took three psychomotor tests:




Results:



Small parts manual dexterity test (Crawford & Crawford, 1981)
Spiral maze test (Gibson, 1961)
Embedded figures test (Witkin, Oltman, & Karp, 1971)
Surgical performance correlated negatively with tests of manual
dexterity, but
Improvement in surgical skill correlated positively with the
hidden figures score
The ability to create, and hold in mind, mental
representations appears to be more important than
manual dexterity in predicting surgical performance
Steele, Walder, and Herbert (1992)
Visual-spatial ability and surgical skill
38

37 junior surgical residents at the University of
Toronto given six tests of visual/spatial ability
 Snowy
pictures test (Ekstrom et al., 1976)
 Gestalt completion test (Street, 1931)
 Shape memory test (Ekstrom et al., 1976)
 Cube comparison test (Ekstrom et al., 1976)
 Form board test (Sylvester, 1881)
 Mental rotations test (Vandenberg & Kuse, 1978)
Visual-spatial ability and surgery skills

Residents were then tested on their ability to learn
and perform complex surgical procedures (two-flap
and four-flap Z–plasty on pig thighs)
Correlation of surgical skill with spatial ability
40
Total global
rating
Procedure
Try
SPT
GCT
SMT
CCT
FBT
MRT
Two-flap
1st
—
—
—
—
—
0.40
Two-flap
2nd
—
—
—
—
—
—
—
—
—
—
0.34
0.47
Four-flap
Quality of
result
Two-flap
1st
—
—
—
—
—
0.40
Two-flap
2nd
—
—
—
—
—
—
—
—
—
—
0.40
0.49
Four-flap
Wanzel, Hamstra, Anastakis, Matsumoto & Cusimano (2002)
General conclusions about expertise
41



Elite performance is the result of at least a decade
of maximal efforts to improve performance
through an optimal distribution of deliberate
practice
What distinguishes experts from others is the
commitment to deliberate practice
Deliberate practice is
 an
effortful activity that can be sustained only for a
limited time each day
 neither motivating nor enjoyable—it is instrumental in
achieving further improvement in performance
Talent is over-rated…
42
Effects of experience in teaching
43
Extra months per year o f learning
1
Mathematics
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
-5
Years of teaching experience
0
1
2
3 to 5
Rivkin, Hanushek and Kain (2005)
Reading
-5
0
Years of teaching experience
1
2
3 to 5
Implications for education systems





Pursuing a strategy of getting the “best and brightest”
into teaching is unlikely to succeed
Currently all teachers slow, and most actually stop,
improving after two or three years in the classroom
Expertise research therefore suggests that they are
only beginning to scratch the surface of what they are
capable of
What we need is to persuade those with a real passion
for working with young people to become teachers,
and to continue to improve as long as they stay in the
job.
There is no limit to what we can achieve if we support
our teachers in the right way
Supportive accountability
45

What is needed from teachers:
A
commitment to:
 The
continual improvement of practice
 Focus on those things that make a difference to students

What is needed from leaders:
A
commitment to engineer effective learning
environments for teachers by:
 Creating
expectations for continually improving practice
 Keeping the focus on the things that make a difference to
students
 Providing the time, space, dispensation, and support for
innovation
 Supporting risk-taking
References
46
Aaronson, D., Barrow, L., & Sander, W. (2007). Teachers and student
achievement in the Chicago Public High Schools. Journal of Labor
Economics, 25(1), 95-135.
Atteberry, A., Loeb, S., & Wyckoff, J. (2013). Do first impressions
matter? Improvement in early career teacher effectiveness.
Washington, DC: Center for Analysis of Longitudinal Data in
Educational Research.
Berliner, D. C. (1994). Expertise: the wonder of exemplary
performances. In J. N. Mangieri & C. C. Block (Eds.), Creating
powerful thinking in teachers and students: diverse perspectives
(pp. 161-186). Fort Worth, TX: Harcourt Brace College.
Danielson, C. (1996). Enhancing professional practice: a framework
for teaching. Alexandria, VA: ASCD.
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of
deliberate practice in the acquisition of expert performance.
Psychological Review, 100(3), 363-406.
Fitzpatrick, M. D., Grissmer, D., & Hastedt, S. (2011). What a
difference a day makes: Estimating daily learning gains during
kindergarten and first grade using a natural experiment. Economics
of Education Review, 30(2), 269-279.
Goldhaber, D. D., Goldschmidt, P., & Tseng, F. (2013). Teacher valueadded at the high-school level: Different models, different
answers? Educational Evaluation and Policy Analysis. doi:
10.3102/0162373712466938
Hamre, B. K., & Pianta, R. C. (2005). Academic and social advantages
for at-risk students placed in high quality first grade classrooms.
Child Development, 76(5), 949-967.
Hanushek, E. A., & Rivkin, S. G. (2006). Teacher quality. In E. A.
Hanushek & F. Welsh (Eds.), Handbook of the Economics of
Education (Vol. 2, pp. 1051-1078). Amsterdam, Netherlands:
Elsevier.
Harris, D. N., & Sass, T. R. (2007). Teacher training, teacher quality
and student achievement (Vol. 3). Washington, DC: National Center
for Analysis of Longitudinal Data in Education Research.
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater
reliability Is not enough: Teacher observation systems and a case
for the generalizability study. Educational Researcher, 41(2), 56-84.
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback
interventions on performance: a historical review, a meta-analysis,
and a preliminary feedback intervention theory. Psychological
Bulletin, 119(2), 254-284.
Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools
and academic achievement. Econometrica, 73(2), 417-458.
Rockoff, J. E. (2004). The impact of individual teachers on student
achievement: evidence from panel data. American Economic
Review, 94(2), 247-252.
Rothstein, J. (2010). Teacher quality in educational production:
Tracking, decay, and student achievement. Quarterly Journal of
Economics, 125(1), 175–214.
Sartain, L., Stoelinga, S. R., Brown, E. R., Luppescu, S., Matsko, K. K.,
Miller, F. K., Durwood, C.E., Jiang, J.Y., & Glazer, D. (2011).
Rethinking teacher evaluation in Chicago: Lessons learned from
classroom observations, principal-teacher conferences, and district
implementation. Chicago, IL: Consortium on Chicago School
Research.
Steele, R. J. C., Walder, C., & Herbert, M. (1992). Psychomotor testing
and the ability to perform an anastomosis in junior surgical trainees.
British Journal of Surgery, 79(10), 1065–1067.
Wanzel, K. R., Hamstra, S. J., Anastakis, D., Matsumoto, E. D., &
Cusimano, M. D. (2002). Effects of visual-spatial ability on learning
of spatially-complex surgical skills. Lancet, 359(9302), 230–231.