King’s College (DEPS)
Assessment Group Seminar
Moving On from Levels – a chance
to reflect on TGAT and assessment
of the National Curriculum
Professor Paul Black
1
Education Reform Act 1988
three parallel processes
• Composing the Act, Legislation Process
- Summer 1987 to Summer 1988
• Task Group on Assessment and Testing (TGAT)
– September 1987 to Christmas 1987
– Tests at 7, 11 and 14 plus 16 (?)
• First Subject Working groups
- 1987 to July 1988
- Curriculum specifications
2
The TGAT reports
• Main report published January 1988
– Group of 11 members
– 100 pages including 44 recommendations
– Plus 65 pages containing 14 appendices
• Three Supplementary Reports March 1988
- Reactions to main report
- Subject specific issues
- Support system
3
The TGAT levels arguments A
• 35 We recommend that an individual subject
should report a small number (preferably no
more than four and never more than six) of
profile components reflecting the variety of
knowledge, skills and understanding to which the
subject gives rise.
• 100 . . scales should be used that indicate where
a pupil has reached in a profile component. We
shall use the word level to define one of a
sequence of points on a scale to be used in
describing the progress of attainment in the
profile component. The sequence of levels
represents the stages of progression.
4
The TGAT levels arguments B
• 101. . . . .For the purposes of national assessment, a
broad indication of progress is required. We propose
that the criteria defining successive levels be so
chosen that a pupil could reasonably be expected to
progress by one level in two years of work in that
profile component. Over the age range 7-16 this
would imply a need for 5 or 6 levels. At both ends of
the age range, however, some will be unable to
progress as fast as others, and some will be able to
make quicker progress. This leads us to recommend
that a total range of ten levels will need to be
defined . . .
5
TGAT’s 10 level framework
6
TGAT recommendations A
• Recommendations for age 16 will need to relate
to the criteria in the appropriate GCSE subjects.
• Subject working groups should specify, in broad
terms and for each profile component, the
appropriate tests (standardised assessment tasks)
which should be prepared, and the advice and
help which should be given to teachers about
their corresponding internal assessments.
7
TGAT recommendations B
• 14. The national assessment system should be based on a
combination of moderated teachers’ ratings and
standardised assessment tasks.
• 30. The only form in which results of national assessment
for, and identifying, a given school should be published
should be as part of a broader report by that school of its
work as a whole
• 31. Any report by a school which includes national
assessment results should include a general report for the
area, prepared by the local authority, to indicate the nature
of socio-economic and other influences which are known to
affect schools. This report should give a general indication
of the known effects of such influences on performance.
8
TGAT recommendations C
• 44. The new assessment system should be
phased in over a period adequate for the
preparation and trial of new assessment
methods, for teacher preparation, and for
pupils to benefit from extensive experience of
the new curriculum. This period needs to be
at least 5 years from the promulgation of the
relevant attainment targets.
9
Margaret Thatcher’s opinion
• Ken Baker warmly welcomed the report. Whether he
had read it properly I do not know: if he had it says
much for his stamina. Certainly I had no opportunity to
do so before agreeing to its publication . . .that it was
then welcomed by the Labour party, the National
Union of Teachers and the Times Educational
Supplement was enough to confirm for me that its
approach was suspect.
•
Pp.594-5 in THATCHER, M. (1993) The Downing Street Years (London, Harper
Collins).
10
Lessons from Research in Mathematics
Education
• Denvir, B and Brown, M. (1987) The feasibility of classadministered diagnostic assessments in primary mathematics.
Educational Research 29(2) 95-106
• Aims: to explore
1 What the child knows
2 How it fits in to an overall framework
3 How children learn
• Some children at age 16 show the same level of
attainments as the average 11-year old
11
Key to the numbers
3 - Mentally carries out two-digit
‘take away’ with re-grouping
4 -Models two-digit ‘take-away’
without re-grouping using
‘base-ten’ apparatus
14 - Models two-digit addition without regrouping using
‘base-ten’ apparatus
Denvir, B. & Brown, M. (1987)
Educational Researcher
29(2),95-107
Report by Brown & Denvir, commissioned by
education minister on feasibility of targets and
assessment in the primary phase
If it is decided, nevertheless, to go ahead with introducing a
system of national attainment targets we believe that the
scale of the programmes we have described is both realistic
and essential. If any attempt is made to cut corners, for
example by asking the Working Group to propose detailed
targets without a basis of adequate research , or by
economizing on the initial trialling or on the in-service
training required to implement the system in schools, we
believe that there is a vastly increased risk that the
undesirable effects will outweigh the desirable ones.
13
Example of a Science Attainment Target
Component 4: physical processes (1999 version)
• Level 5 Pupils demonstrate knowledge and understanding of
physical process drawn from the key stage 2 or key stage 3
programme of study. They use ideas to explain how to make a
range of changes. They use some abstract ideas in
descriptions of familiar phenomena. They use simple models
to explain effects that are caused by the movement of the
earth.
• Level 6 Pupils use and apply knowledge and understanding of
physical processes drawn for the key stage 3 programme of
study. They use abstract ideas in some descriptions and
explanations. They recognize and can give examples of the
wide application of many physical concepts. They give
explanations of phenomena in which a number of factors
14
have to be considered.
National Curriculum Attainment Targets for Forces and
Motion vs.
Progression proposals from King’s-Leeds science group
Level 5
• AT . . for example forces are balanced when an object is
stationary
King’s Leeds proposal:
• For an object at rest, pupils should be able to identify several
forces that oppose one another and are in balance. For turning
forces, they should relate balance to equality of moments.
Level 6
• AT . .for example forces determining changes in the direction or
speed of movement of a body
• King’s Leeds proposal:
• Pupils should relate changes in pressure to changes in the size of
forces and in the areas over which they act, and be able to make
simple calculations about pressure.
Report to SCAA on revision proposals from King’s-Leeds group May 1994
15
The Framework for the National Curriculum
A report by the Expert Panel for the National Curriculum
review Dec. 2011 .
• Attainment Targets should be statements of
specific learning outcomes
• Pupils have become more concerned with their
levels than with what they know
• We should stress effort : that all can succeed.
Other countries do this and have lower spread in
pupils’ performance
• Quote Carol Dweck and Assessment for Learning
16
The Framework for the National Curriculum
A report by the Expert Panel for the National Curriculum review
Dec. 2011 .
• . . .constant assessment to levels is over-burdensome,
obscures the genuine strengths and weaknesses in a
pupil’s attainment, obscures parental understanding of
the areas in which they might best support their child’s
learning, and likewise, weakens teachers’ clear
understandings and identification of pupils’ specific
weaknesses or misunderstandings. (para 8.24, p.50)
• Reporting, according to our suggested model, could be
based on a ‘ready to progress’ measure broken down
into key areas of subjects. (para. 8.26, p. 51).
17
Structure of Matter Learning
Progression
18
“Prehistory” of ECC
Atomic-Molecular Theory of Macro-Properties
6 – If a substance is made of molecules, then the molecules determine its
macro properties.
5 – All matter is made up of atoms and molecules and empty space
between them. Molecules are made up of atoms.
4 – Molecules and atoms are matter—they take up space and have
weight, but do not have macro-properties such as hardness and
temperature.
3 – Molecules and atoms are matter—they take up space, have weight
(mass), hardness, temperature, etc.
2 – Only some material things are made of molecules, e.g., solids and
liquids are made of molecules, but not gases; visible things are made of
molecules, but not invisible things, and living things are not made of
molecules.
19
1A-Molecules and atoms exist in matter (like raisins in fruitcake)
Initial Progress Map 6:
Particulate Evidence Based on Chemical Change
ECC-5 Chemical Changes Produce different molecules
ECC-4 Chemical & Physical Change – Particulate
Explanations of Differences
ECC-3 Atoms & Molecules
ECC-2 Chemical & Physical Change – Observable
Differences
ECC-1 Conservation of Matter
ECC-0 Matter not conserved
Black,P., Wilson, M. and Yao, Shih-Ying. (2011). Road Maps for Learning: A guide to the
Navigation of Learning Progressions. Measurement 9 (2-3), 71-123.
20
How to represent chances of success?
Probability of student’s success on a given item depends
on
student ability/ item difficulty
So outcomes are a set of ratios
The logarithm of this probability depends on
log of ability – log of item difficulty
So outcomes are represented by differences
Choose a function which can describe the variation of the
probability of success with this difference between
measure of ability and question difficulty
Wilson, M. (2005) Constructing Measures. Mahwah, N.J. : Lawrence Erlbaum
The Wright Map
Pupils’ overall scores
Difficulty of each question
Hard Question
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Easy question
X
|
XXX
|
XX - - - - - - - - - - - - - - - - - - | - - - - 50% succeed - - - - - - 80% succeed
XXXXXX
|
XXXXXXX
|
XXXXXX
|
XXXXXXXXXX
|
XXXXXXXXXXX
|
XXXXXXXXXXXX - - - - - - - - | - - - - 25% succeed - - - - - - 50% succeed
XXXXXXXXXXXXXXX
|
XXXXXXXXXXXXXXXXX
|
XXXXXXXXXX
|
XXXXXXXXXXX
|
XXXXXXXXX- - - - - - - - - - - | - - - - 10% succeed - - - - - - 25% succeed
XXXXXXXX
|
XXXXXX
|
XXXXXXX
|
XXXX
|
XXX
|
X
|
XX
|
22
The problem with ECC
23
Diagnosis – what could have gone
wrong ?
Four possible causes
Bad Map
Bad Items
Bad Students
Bad Idea
24
A way forward
Separate strands
• ECC-A Chemical or physical changes:
knowing that atoms and molecules do or do
not change in these two types of change
• ECC-B Chemical or physical changes: changes
in macroscopic properties
• ECC-C Molecular representations of
elements and compounds
25
New ECC Wright Map
ECC-A
ECC-B
ECC-C
26
Still some problems
• A general problem with this three-strand
structure is the dividing line between molecular
models representing differences and molecular
models representing changes.
• This difference interacts with the difference in
difficulty between items asking with pictures of
models and those asking only with verbal
descriptions.
• Given the model of three separate strands (a)
should each strand have its own levels, and (b)
are there links of dependency across the three
27
28
This function is known as the Rasch model
Let A measure of ability, and D item difficulty:
For high ability pupil:
say A is 2.0
Then if D=1.0 : probability of success will be = 0.73
But if D=2.0 : probability of success will be = 0.50
For low ability pupil:
say A is 1.0
Then if D=1.0 : probability of success will be = 0.50
But if D=2.0 : probability of success will be = 0.27
Rasch Model
Proby (X=1 | Øi, ∂i) =
e(Øi-∂i)
1+ e(Øi-∂i)
Proby (X=1 | 1.0, 1.0) = e(1.0-1.0)
1+ e(1.0-1.0)
= e(0)
= 0.5
1+ e(0)
For bright pupil:
Øi=2.0, ∂i=1.0 : proby = 0.73
For bright pupil’s 0.5: Øi=2.0, ∂i=2.0 : proby = 0.50
Initial Progress Map 6:
Particulate Evidence Based on Chemical Change
ECC-5 Chemical Changes Produce different molecules:
Atoms combine in different ways, and/or molecules break apart, to produce new molecules
ECC-4 Chemical & Physical Change – Particulate Explanations of Differences:
In a chemical change, the molecules/atoms of the products are different; in physical changes
they are the same
ECC-3 Atoms & Molecules: The smallest possible particles of a substance may either be
atoms, or combinations of different kinds of atoms: the latter are called molecules.
ECC-2 Chemical & Physical Change – Observable Differences:
Chemical produces new substances. Changes are often irreversible. Mass is conserved
ECC-1 Conservation of Matter:
In both chemical and physical changes, total volume may change, total mass unchanged.
ECC-0 Matter not conserved: In all types of substance change, the total mass may change.
31
Particulate explanations of chemical change
32
Particulate explanations of chemical change
33