USING CEM SECONDARY DATA
PRACTICAL APPLICATIONS IN SCHOOL
APRIL 2011
Bristol Conference
Geoff Davies
CEM Secondary data includes:
•
•
•
•
•
•
Baseline test data (developed ability)
‘Predictive’ data including chances graphs
Value-added data
Attitudinal data
Curriculum assessments (Insight/Sosca)
PARIS software programmes
The use of this data needs to allow us to do our
best to help every pupil to at least achieve if not
exceed their potential.
It may challenge
•
•
•
•
The culture of ‘my’ school
Accountability policy
Expectations
Staff training in use of data and ability to cope with
data (data overload)
• Integrating the data into school procedures,
storage, retrieval, distribution and access
• Roles and Responsibilities
Carol Fitz-Gibbon 2001 British Psychological Society
It gradually dawned on me that providing the data to
schools was the most important outcome of the
effort, far more important than writing research
papers…..
The provision of data to practitioners meant that they
participated in the research. Indeed they were the
only ones who knew the surrounding circumstances
for their classrooms, their department, each pupil,
each family, etc. They were the major players: the
ones who could interpret and learn from the detailed
data.
…..there is a need for teacher researcher
posts on the Senior Management team with a
brief to develop research that is useful. Given
time in the timetable thousands of teachers
could become active researchers….
Educational research should be a practical reality
contacting scientific enlightenment not a mathematical
weight lifting exercise
The sense and usefulness of what we are doing induces
creative thoughtfulness.
QUESTION
IS THIS STIFLED IN THE PRESENT
EDUCATIONAL CLIMATE?
creative thoughtfulness
PRIORITISING
Can school leaders find the time?
Important but not urgent
Is it more like? 65%-80% or 15%
Urgent but not important
Is it more like?
I keep six honest serving men
Who taught me all I know
Their names are What and Why and When
And How and Where and Who
Rudyard Kipling The Elephant’s Child
15% or 60%
QUESTIONS INSPIRED BY CEM CENTRE DATA
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
How do we improve our professional judgement?
Do MIDYIS scores tell us more than we think?
Which ability range of pupils gets the best deal from our school?
Which ability range of pupils is served best by a particular subject department?
Can we do action research in our school using historical data?
Can standardised residuals tell us more about a department than just the + or -?
Does learning support work?
What do attitudinal surveys over time give us?
Can we compare standardised scores with comments made by teachers on reports?
Drilling down from SPC charts.
Which pupils are we succeeding with?
Which pupils are we not succeeding with?
What can be done about it?
Is there a pattern?
What are the gender issues?
What are the prior learning issues?
What can we learn from plotting standardised scores over time?
Can SOSCA help with boy/girl issues?
Is key stage 3 teacher assessment sufficiently rigorous?
How can ALIS be used to inform teaching and learning?
WHAT information do I want?
WHY do I want it?
WHEN do I need it?
HOW do I collect it?
WHERE can I find it?
From WHO do I get it?
The CEM centre ticks many of these boxes for schools
A small selection of questions we will look at
1. Do Midyis scores tell us more than we think?
2. Which ability of students do most or least well in our
school (using YELLIS data)?
3. Can we review post 16 pedagogy (using ALIS data)?
4. What did SOSCA teach us about Key Stage 3
assessment?
Do Midyis scores tell us more than
we think?
Using Midyis baseline data
Using MidYIS IPRs Booklet.pdf
What do the sections of the test measure?
Vocabulary Score
The Vocabulary component of the test is generally an important element for most
subjects. For English, History and some Foreign Languages it is the best.
However the Vocabulary score is perhaps the most culturally linked of all
the scores. Those who have not been exposed to vocabulary-rich talk or a wide
variety of reading material or whose first language is not English are unlikely to
have developed as high a vocabulary score as they would have developed in a
different environment.
Maths Score
The Maths score is well correlated with most subjects but is particularly important
when predicting Maths, Statistics, ICT, Design Technology and Economics.
The Maths section has been designed with the emphasis on speed and fluency,
rather than knowledge of Maths. Like the Vocabulary score, the Maths score is a
good predictor of later academic performance.
Non-Verbal Score
The Non-Verbal score is composed of the three sub tests: CrossSections, Block Counting and Pictures. The Non-verbal score is
important when predicting Maths, Science, Design Technology
Geography, Art and Drama. It provides a measure of the pupil’s
ability in 3-D visualisation, spatial aptitude, pattern recognition and
logical thinking. It can give an insight in to the developed ability
for pupils for whom English is a second language
Skills Score
In the Proof Reading section pupils are asked to spot mistakes in the spelling,
punctuation and grammar of a passage of text. eg mis-spelling of words like
‘there’ and ‘their’.
The PSA (Perceptual speed and accuracy) section asks pupils to look for
matches between a sequence of symbols on the left and a number of possible
choices on the right. Given enough time most pupils would probably get the
answers correct but we are measuring how quickly pupils can find a correct
match. The PSA section allows speed to be demonstrated free from the
demands of memory.
The Proof Reading and PSA tests are tests for the modern world, and are
designed to measure fluency and speed. They rely on a pupil’s scanning and
skimming skills, skills that are desirable in examination situations.
Some pupils will display an IPR pattern with significant differences between one or
two components of the MidYIS Test.
These can be the most interesting and possibly the most challenging pupils for
mainstream classroom teachers.
Scenarios and anecdotal findings
It is when the IPR is placed in the hands of a teacher who knows
that pupil that it becomes a powerful tool.
Confidence Limits
The pupil scored 114 on the
Vocabulary section. The error bars
range from about 105 to 123,
about 9 points either side of the
pupil’s score. If this pupil was to
take this test afresh 100 times, we
would expect that 95 of those
times the pupil’s score would fall
within the range denoted by the
error bars
Comparing Maths and Vocabulary Scores
Are the scores significant?
Relative to the Average Performance?
The error bars for
Vocabulary and
Maths do not cross
the line at 100 ( av.
performance).
Performance in Vocabulary is significantly better than average
performance and Maths performance is significantly below average. The
error bars for the Non-verbal, Skills and Overall MidYIS scores do cross
the line at 100 and hence the pupil cannot be considered to have
performed significantly different to the average pupil overall.
A SELECTION OF MIDYIS SCORES FOR ‘WATERLOO ROAD’ !!
Why would this be a very challenging class to teach?
Vocabulary
Surname
Maths
Non Verbal
Skills
MidYIS Score
St. Score
Band
St. Score
Band
St. Score
Band
St. Score
Band
St. Score
Band
Sex
A
F
81
D
110
B
108
B
112
A
94
C
B
F
128
A
107
B
105
B
94
C
120
A
C
M
106
B
121
A
103
B
90
D
114
A
D
F
107
B
84
D
96
C
107
B
96
C
E
M
96
C
90
D
130
A
91
C
92
C
F
F
86
D
86
D
120
A
74
D
84
D
G
F
100
B
115
A
80
D
103
B
108
B
H
F
121
A
96
C
114
A
86
D
111
A
I
M
92
C
100
C
96
C
123
A
95
C
J
M
100
C
105
B
100
C
99
C
102
B
K
M
128
A
132
A
114
A
131
A
133
A
L
M
76
D
70
D
74
D
73
D
71
D
What do I need to know/do to teach this (difficult) class of twelve pupils
These are real anonymous scores from a number of schools around the UK
Vocabulary scores significantly lower than other component scores
Second language? Deprived areas? Difficulty accessing curriculum.? Targeted help does work. Seen in
nearly all schools. Worth further diagnosis
Vocabulary scores significantly higher than other component scores
Good communicators. Get on. Put Maths problems in words?
Mathematics significantly higher than other scores
From Far East? Done entrance tests? Primary experience?
Mathematics significantly lower than other scores
Primary experience. Use words and diagrams? Sometimes difficult to change attitude..
Low Mathematics scores with High Non-verbal Scores
Use diagrams. Confidence building often needed
Pupils with non-verbal scores different from others – High Non-verbal Scores
Frustration? Behaviour problems? Don’t do as well as good communicators or numerate pupils?
Pupils with non verbal scores different from others – Low Non-verbal Scores
Peak at GCSE? A level ?
Pupils with low Skills scores
Exams a difficulty after good coursework?
High Skills Scores
Do well in exams compared with classwork?
The Average Pupil
They do exist!
High scores throughout
Above a score of 130 puts the pupil in the top 2% nationally
Low scores throughout
Below a score of 70 puts the puil in the bottom 2% nationally
Sharing the MidYIS Information within School
Once you have received your MidYIS feedback you need to decide who
will be privy to which information. Some schools decide to keep the data
within the senior management team, others with Heads of Department
and/or Heads of Year, some share with all staff and what about pupils and
their parents?
Use you MIS systems to put the data where it matters
MidYIS data can be useful:
•to indicate reasons for student learning difficulties and may go some way
to explain lack of progress, flag up causes for underachievement and even
behaviour problems.
•for all teachers and support staff. It can help to support professional
judgement and give a better understanding of the progress students make
at school and their potential later performance.
•to refer to for pupil reviews, writing reports, meeting parents, monitoring
progress and interim assessments.
2. Which ability of students do most or least well in our school (using YELLIS data)
Which ability range of pupils gets the best deal from our
school?
Which ability range of pupils is served best by a particular
subject department?
Can standardised residuals tell us more about a department
than just the + or -?
Using standardised residuals in a different way
0.1
0.4
Voc Health & Social Care
-0.5
Voc Applied Science
-1.1
SC Religious Studies
-0.5
SC History
-0.3
SC Geography
1.0
SC Art & Design
-0.1
Science: GCSE Additional
-0.1
Science: GCSE
0.1
Religious Studies
-1.3
Music
0.1
Media Studies
-0.3
Mathematics
0.2
History
-0.2
Geography
-0.5
French
English Literature
-0.6
English
-1.0
Design & Technology
Art & Design
Average Standardised Residual
CONTRAST THIS
4.0
3.0
2.0
0.3
0.0
-0.9
-0.5
-2.0
-3.0
-4.0
-1.0
-2.0
-3.0
-4.0
0.0
-0.2
0.8
0.4
SC ICT
0.3
Welsh
0.2
Double Science
-0.1
Religious Studies
-0.1
Physical Education
0.6
Music
0.5
Maths
0.9
ICT
0.3
Home Economics
History
0.4
German
0.2
Geography
0.1
French
0.4
English Literature
0.0
English
1.0
Drama
Design & Technology
0.0
Business Studies
Art & Design
Average Standardised Residual
WITH THIS
4.0
3.0
2.0
1.4
0.9
As on the last two slides CEM provides value added charts using
average standardised residuals for departments
We are going to show how standardised residuals can be used in a
different way in your school
yellis exercise.doc
1. Review of post 16 pedagogy (using ALIS data)
SUBJECT A
Why the
improvement?
2008
PEDAGOGY….ALIS surveys
SUBJECT A 2005
SUBJECT A 2007
SUBJECT A 2008
DO NOT GET TOO EXCITED!
We have made a comparison of perceived teaching methods as analysed
by ALIS in 2004-5 with those in 2007-8. Some subject areas have
appeared to change their methods radically. Others have not. Though the
samples are small it is an interesting exercise to try to correlate it with the
departments statistical process charts over that period. One would like to
say that changes in the variety of teaching methods result in
improvement but the evidence is a little tenuous so far.
e
E x
x a
e m
r p
c a
i p
s e
e r
s s
SUBJECT
e
s
s
a
y
s
R
e
a
d
i
n
g
d
i
s
c
u
s
s
i
o
n
s
n
o
t
e
s
d
i
c
t
a
t
e
d
o
w
n
n
o
t
e
s
h
a
n
d
o
u
t
s
P
r
a
c
t
i
c
a
l
w
o
r
k
U
s
i
n
g
A
V
I
T
R
e
s
e
a
r
c
h
i
n
g
P
r
e
s
e
n
t
i
n
g
h
e
l
p
i
n
g
s
t
u
d
e
n
t
s
o
r
i
g
i
n
a
l
w
o
r
k
2005 to 2008
ART
BIOLOGY
BUSINESS
CHEMISTRY
COMPUTING
DRAMA
FOOD
PRODUCT DESIGN
ENGLISH
FRENCH
GEOGRAPHY
HISTORY
ICT
MATHS
MUSIC
PE
PHYSICS
RE
WELSH
YES
NO
MORE (M)
LESS (L)
SIGNIFICANT
CHANGE
YES
NO
YES
NO
YES
NO
YES
YES
NO
YES
YES
NO
YES
NO
YES
YES
YES
NO
YES
11
7
54
29
L
L
M
L
L
L
M
L
M
L
L
L
L L
M
M
M
L
L
L
M
L
L
M
M
L
M M
M
L
M
L
L
M
M
M
M
L
M M M
M
M M M
M
M M M
M
M
L M
M M
M
M
M
M M
M
4
3
M
M
M M
L
L
L
5
3
L
L
1
2
L
3
6
4
3
5
1
6
2
5
1
M M M
M
M
M M
M M
L
M
2
2
3
1
2
0
3
0
2
2
6
3
3
0
4. What have we learned from SOSCA?
GRAPH 1
AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO KEY
STAGE THREE TEACHER ASSESSMENT
GENDER ISSUES DYFFRYN TAF KEY STAGE 3
AVERAGE STANDARDISED RESIDUALS MIDYIS TO KS3
0.8
0.6
GIRLS
GIRLS
0.4
OVERALL
BOYS
0.2
OVERALL
GIRLS
0
BOYS
English
Maths
-0.2
BOYS
-0.4
SUBJECT
Science
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
GRAPH 2
AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO SOSCA
GENDER ISSUES DYFFRYN TAF SOSCA
0.4
AVERAGE STANDARDISED RESIDUALS MIDYIS TO SOSCA
BOYS
0.3
GIRLS
OVERALL
OVERALL
BOYS
0.2
GIRLS
0.1
BOYS
0
Reading
Maths
OVERALL
Science
-0.1
GIRLS
-0.2
95% CONFIDENCE
-0.3
SUBJECT
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
GRAPH 1
AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO KEY STAGE THREE TEACHER ASSESSMENT
GENDER ISSUES DYFFRYN TAF KEY STAGE 3
GIRLS
0.6
GIRLS
0.4
OVERALL
BOYS
0.2
OVERALL
GIRLS
0
BOYS
English
Maths
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
Science
-0.2
BOYS
-0.4
SUBJECT
GRAPH 2
AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO SOSCA
GENDER ISSUES DYFFRYN TAF SOSCA
0.4
BOYS
AVERAGE STANDARDISED RESIDUALS MIDYIS TO SOSCA
AVERAGE STANDARDISED RESIDUALS MIDYIS TO KS3
0.8
0.3
GIRLS
OVERALL
OVERALL
BOYS
0.2
GIRLS
0.1
BOYS
0
Reading
Maths
OVERALL
Science
-0.1
GIRLS
-0.2
95% CONFIDENCE
-0.3
SUBJECT
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
Using MIDYIS and SOSCA puts the school in a strong
position to improve its professional judgment of teacher
assessments at Key stage 3. Statutory testing disappeared
in Wales some five years ago.
Comparing the value added for MIDYIS to SOSCA and
MIDYIS to KS 3 teacher assessment shows up some
interesting data
Schools who depend on teacher assessment data only to measure value
added from Key stage 3 to Key stage 4 need to be aware of the pitfalls. The
use of SOSCA data in this exercise highlights that
see
http://www.tes.co.uk/article.aspx?storycode=6062337
Subjective views of pupils - as well as pressure from parents - make model unreliable, warns Professor Tymms.
The differences appear to relate to the types of assessment
used in the various subject areas. English and Welsh use
extended writing for teacher assessment which is more likely
to have subjective judgments.
What we have learnt from this is that despite a moderation
process built on portfolios of work for teacher assessment, it
is not sufficient in isolation.
Computer adaptive tests such as SOSCA and the resulting
value added information from MIDYIS are more informative in
a diagnostic sense than levels produced by teachers for
statutory assessment.
SOSCA has also been used to investigate any changes in
reading from baseline testing. A high correlation was found
between the London Reading score given to pupils on entry
the MIDYIS score and the SOSCA reading test.
PITFALLS
1. Tracking developed ability measures over time.
2. Looking at average standardised residuals for teaching sets.
3. Effect of one result in a small group of students
REGRESSION TOWARDS THE MEAN
DIFFERENCE SOSCA-MIDYIS MATHS
20
10
0
SOSCA-MIDYIS DIFFERENCE
50
70
90
110
130
150
170
-10
DIFFERENCE
-20
-30
-40
-50
MIDYIS MATHS STANDARDISED
Pupils with high MidYIS scores tend to have high SOSCA scores but
not quite as high. Similarly pupils with low MidYIS scores tend to have
low SOSCA scores, but not quite as low. It is a phenomenon seen in
any matched dataset of correlated and normally-distributed scores,
the classic example is a comparison of fathers' and sons'
heights. Regression lines reflect this phenomenon - if you look at the
predictions used in the SOSCA value-added you can see that for
pupils with high MidYIS scores their predicted SOSCA scores are
lower than their MidYIS scores, whereas for pupils with low MidYIS
scores their predicted SOSCA scores are higher than their MidYIS
scores.
Marksheet Name : SUBJECT REVIEW
Marksheet Group : 11S1
A
B
GCSE Standard Residual Ma
WJEC/GCSE 018403 ResGF
WJEC/GCSE 018402 ResGF
WJEC/GCSE 018401 ResGF
YELLIS GCSE PREDICTION MA
a
90019
6
b
90090
7
c
90045
6
63 A
B
B
B
B
0.10
d
90063
7
64 A
A
B
B
B
0.10
e
90166
6
48 B
B
B
C
C
0.40
f
90123
7
70 A
A
A
A
B
-0.40
g
90129
6
47 C
C
B
C
C
0.50
h
90146
6
59 B
B
A
B
A
1.40
I
90047
7
62 A
A
B
B
B
0.20
j
90115
7
67 A
A
*
A
A*
1.70
k
90004
6
46 C
B
B
C
B
1.50
l
90164
7
65 A
A
A
B
A*
1.90
m
90099
7
70 A
A
A
A
A*
1.50
n
90011
7
61 A
A
A
B
A
1.30
o
90112
7
66 A
A
B
A
A
0.80
p
90058
6
70 A
A
B
A
A
0.50
q
90150
7
72 A
A
A
A
A
0.40
r
90127
6
52 B
B
B
C
B
1.00
s
90030
6
58 B
B
B
B
B
0.50
t
90050
7
71 A
A
A
A
A
0.40
u
90016
6
69 A
A
B
A
B
-0.40
v
90174
7
74 A
A
A
A
A
0.20
w
91165
6
62 A
B
B
B
x
90109
7
63 A
B
B
B
B
0.10
y
90138
7
47 C
B
B
C
B
1.40
z
90122
7
60 A
A
*
B
A
1.30
ab
90009
7
60 A
A
A
B
A
1.30
ac
90169
7
79 A
A
*
A*
A
-0.20
ad
90153
6
56 B
B
B
B
ae
90010
7
64 A
B
B
B
A
1.00
af
90154
7
61 A
C
B
B
B
0.30
Total
62 A
PREDICTION MATHS
YELLIS MATHS BAND
Yellis Band
Yellis Score
Admission No.
Students
CLASS
REVIEW
: 04/10/2005
Maths Test K3 Wa
Export Date
B
A
B
1.20
B
B
0.20
B
0.70
1323
1868
109
105
201
156
12
190
31
30
30
30
31
30
2
29
30
Mean
42.68
62.27
3.63
3.5
6.48
5.2
6
6.55
0.70
Mean Grade
6.00
Num ber of Results
B
B
B
B
B
B
20.90
BEWARE
PITFALLS
INTERPRETATION
SUBJECT M
Sex
M
M
F
M
F
F
M
M
F
F
F
M
Score
(Band)
53 (B)
38 (C)
36 (D)
48 (C)
52 (B)
65 (A)
70 (A)
38 (C)
40 (C)
70 (A)
44 (C)
56 (B)
Predicted Grade
5.4 (B/C)
4.5 (C/D)
4.4 (C/D)
5.1 (C)
5.3 (B/C)
6.1 (B)
6.4 (A/B)
4.5 (C/D)
4.6 (C/D)
6.4 (A/B)
4.8 (C)
5.6 (B/C)
5.3 (B/C)
Achieved Grade
6 (B)
3 (E)
3 (E)
5 (C)
6 (B)
7 (A)
3 (E)
4 (D)
5 (C)
7 (A)
6 (B)
5 (C)
5.0 (C)
Raw
Residual
0.6
-1.5
-1.4
-0.1
0.7
0.9
-3.4
-0.5
0.4
0.6
1.2
-0.6
-0.3
Standard
ised
Residual
0.5
-1.1
-1.0
-0.1
0.5
0.7
-2.5
-0.4
0.3
0.4
0.9
-0.4
-0.2
REVISED
0.5
-1.1
-1.0
-0.1
0.5
0.7
-0.4
0.3
0.4
0.9
-0.4
0.0
PLEA
DON’T LET THE SYSTEM DESTROY
THIS
creative thoughtfulness
USING CEM SECONDARY DATA
PRACTICAL APPLICATIONS IN SCHOOL
APRIL 2011
Bristol Conference
Geoff Davies