Evaluating progress and
impact
John MacBeath
University of Cambridge
A third space
The aim of teaching is not produce
learning but to provide the conditions for
learning to take place (James Bradburne)
Painting outside the frame (Ger Graus)
‘New skills’ – communication and
teamwork (Megan and Taylor)
Having the tools to figure it out
When fleas are captured for the flea circus they
are placed in jars and the lids are screwed on.
When the fleas jumped in the jars, they would hit
their heads on the lids. They still wanted to jump
so they learned to jump just high enough so that
they wouldn't hit their heads. The trainer then
comes back and takes the fleas out of the jars
and puts them in the circus. Even though the
fleas now have the whole sky above them, they
still do not jump past their now self-imposed
limits. Even though the fleas are now free, they
have made the limits truly theirs by refusing to
go beyond them.
The definition of insanity
To go on doing the
same thing and
expecting different
results
The peer and neighbourhood effect
‘You can take the child out of the neighborhood but
much more difficult to take the neighbourhood out of the
child’.
‘Neighbourhood, peer effect and its associated values
are hard to eradicate, but for children and their families
‘locked in neighborhoods of corrosive, concentrated
poverty expanding opportunities, enhancing social
networks and peer groups are all essential aspect of
substantive improvement in academic and social
environments and their inter-relationship.
(Smaker and Betey, 2011)
A world of learning
Ten to eleven years olds:
•
•
•
Hours in school
900
Hours at home
1,277
Hours in the virtual world 1,934
(Baroness Susan Greenfield, 2010)
Inside the black box
The effectiveness perspective
Attainment in
Attainment out
What knowledge is of most worth?
We couldn’t find a mechanism to show we
valued the things we didn’t test. That was the
problem. We always valued the other things but
we couldn’t find a way of showing it, that’s the
problem. We need to get to a situation where
there’s a way of showing how much we value
dancing, music, sport and PE; how much we
value how much improvement children make in
the widest sense and that really gets into the
public consciousness.
(Estelle Morris, Education Minister)
NESTED LIVES
Children and young people live nested lives, so that
when classrooms do not function as we want them to,
we go to work on improving them. Those classrooms
are in schools, so when we decide that those schools are
not performing appropriately, we go to work on
improving them, as well. But those young people are
also situated in families, in neighbourhoods, in peer
groups who shape attitudes and aspirations often more
powerfully than their parents or teachers.
(David Berliner, 2005)
School as an anchor
My thinking [was] that if you were
really interested in development of
the education of the child then you
had to think of schools being a kind of
anchor but an anchor orchestrating
sets of experiences beyond school.
(Sir Tim Brighouse, in Bangs, MacBeath & Galton, 2010)
Construction
sites
internet
Peer group
Home (s)
neighbourhood
media
Parents(s)
School
Teacher(s)
classroom
Extra curricular activities
Learning
destinations
Four big questions
 Does the Children’s University
make a difference?
 In what way?
 How do we know?
 What do we do about it?
Attendance
Achievement
Attitudes
Attendance
329 measures
between 2007and
2010, comparing
C.U. and non-C.U.
pupils,
On only 23 of
these measures is
there no difference
(or a difference in
favour of non-CU
pupils).
Blackpool
Canterbury
Chesterfield
Doncaster
Havant
Leicester
Somerset`
Sheffield
Suffolk
Sunderland
Warwickshire
Term
No.
pupils
Differ
auth
Differ
unauthor
Differ
total
1
5
9
1
1
9
1
5
9
5
1
5
9
5
9
1
5
9
1
9
1
5
9
5
9
277
275
269
23
98
95
11
16
9
26
158
157
155
240
239
87
87
85
60
59
58
58
58
90
43
-1.5
-0.6
-0.9
-2.8
-1.5
-1.7
-3.3
-2.2
-2.5
-2.8
-1.7
-1.8
-1.5
-1.8
-1.4
-1.2
-2.9
-3.4
-2.0
-1.8
-0.2
-1.2
-1.1
-3.1
-2.6
-1.5
-1.8
-1.4
-0.5
-0.6
-0.4
1.1
-0.3
-1.6
-0.2
-1.2
-2.1
-2.9
-0.6
-0.6
-1.3
-1.0
-1.2
-0.4
-0.6
-2.0
-1.7
-2.9
-0.4
-0.6
-3.1
-2.4
-2.3
-3.3
-2.1
-2.1
-2.2
-2.5
-4.1
-3.0
-2.9
-3.9
-4.4
-2.4
-2.1
-2.5
-3.9
-4.5
-2.4
-2.4
-2.2
-2.8
-4.0
-3.4
-2.0
KS1
No.
EST
Actual
diff
2009-10 Maths level 3+
105
19%
27%
8.0
diff by
school
8.4
2009-10 Maths level
105
2.60
2.69
0.10
0,10
2009-10 Writing level 3+
105
25%
30%
5.5
5.4
2009-10 Science level
105
2.56
2.64
0.07
0.06
2009-10 Reading level 3+
105
12%
9%
-3.5%
-2.3%
2009-10 English level
854
2.46
2.55
0.09
0.08
2009-10 Maths level 2+
854
89%
92%
3.1%
3.3%
2009-10 Maths level 2B+
854
71%
77%
5.3%
4.2%
2009-10 Maths level 3+
854
17%
19%
1.4%
1.5%
2009-10 Maths level
854
2.56
2.62
0.06
0.06
2009-10 Reading 2+ level
854
84%
89%
5.1%
5.3%
2009-10 Reading L2B+
854
71%
76%
5.6%
5.2%
2009-10 Writing 3+
854
23%
26%
2.8%
2.3%
2009-10 Writing L2+
854
80%
86%
6.6%
5.5%
2009-10 Writing
854
2.56
2.62
0.06
0.06
2009-10 Writing L2B+
854
58%
65%
6.8%
4.6%
2009-10 Reading L3+
854
10%
11%
0.9%
-0.2%
2010-11 Writing 3+
124
2.52
2.64
0.12
0.16
2010-11 Maths level 2++
124
91%
97%
5.9%
6.7%
KS2
436
84%
78%
-3.7%
-2.0%
436
4.58
4.52
-0.05
0.01
2009-10 Science level 5+ 436
38%
31%
-7.0
-3.4%
2009-10 Science level
436
4.68
4.60
-0.08
-0.02%
2009-10 English level
1031
4.43
4.48
0.01
0.05
2009-10 Maths 2 level
progress
2009-10 Maths level4 +
1031
82%
83%
2.6%
3.2%
1031
79%
83%
3.2%
3.2%
2009-10 Maths level5 +
1031
30%
33%
2.6%
2.2%
2009-10 Maths level
1031
4.49
4.60
0.06
0.06
2009-10 English 2 levels
progress
2009-10 English level
KS3
2009-10 Eng&Maths 5+
50
57%
70%
10.7%
8.6%
2009-10 Eng&Maths 6+
50
22%
28%
9.4%
9.6%
2009-10 English level
progress
2009-10 English level 5
50
31%
42%
15.5%
12.1%
50
68%
78%
8.2%
8.8%
2009-10 English level 6+
50
26%
36%
13.6%
11.1%
2009-10 English level
50
5.28
5.58
0.33
0.28
2009-10 Maths 2 levels
progress
2009-10 Maths level 5+
50
42%
58%
18.8%
18.6%
50
68%
76%
7.2%
3.7%
2009-10 Maths level 6+
50
37%
50%
11.6%
14.1%
2009-10 Maths level
50
5.48
5.85
0.35
0.29
2009-10
Science level 5+
50
67%
80%
12%
12.2%
2009-10
Science level 6+
50
28%
40%
16.1%
17.2%
2009-10
Science level
50
5.32
5.68
0.38
0.40
2009-10
373
64%
69%
5.3%
5.8%
373
29%
28%
373
29%
39%
11.5%
11.7%
2009-10
English & Maths
Level 5+
English & Maths
level 6+
English 2 levels
progress
English level 5+
373
72%
79%
8.7%
6.9%
2009-10
English level 6+
373
33%
37%
5.5%
4.8%
2009-10
English level
373
5.46
5.59
0.17
0.15
2009-10
373
52%
51%
0.9%
2,5%
2009-10
Maths level 2
progress
Maths level 5+
373
74%
76%
3.0
2.9
2009-10
Maths level 6+
373
48%
46%
-2.4
-1.9
2009-10
Maths level
373
5.76
5.77
0.02
0.04
2009-10
Science level 5+
373
74%
78%
5.4
2.2
2009-10
Science level 6+
373
37%
44%
8.1%
5.3%
2009-10
Science level
373
5.57
5.66
0.13
0.04
2009-10
2009-10
-0.2%
0.6%
Data is for dialogue
Professional accountability is based on data, not as a
final judgment but as part of the toolkit for
understanding current performance and formulating
plans for reasonable action…. not as a static
numerical accounting but as a conversation, using
data to stimulate discussion, challenge ideas, rethink
directions, and monitor progress, providing an
ongoing image of [learning] as it changes, progresses,
stalls, regroups, and moves forward again. (Earl and
Katz, 2006: 13).
“Numbers are like people. Torture them enough and
they will tell you anything” (Stephen Gorard)
 Case stories
 Interviews
 Focus groups
 Observation and participation
 Travellers’ tales
 Partners , patrons and products
 CONNECT
What do I already
know or can do?
 EXTEND
How does this
extend my
knowledge or skill?
 CHALLENGE
How does this
challenge my
knowledge or skill?
Curriculum
and
assessment
Beyond
curriculum and
Assessment
smart
 What are the connections?
 Who makes them?
 How are they made?
 How do we know?