Peter Hendry: CEM Consultant
Interpreting Feedback from
Baseline Tests – Predictive Data
Course: CEM Information Systems for
Beginners and New Users
Day 1 Session 3
Wednesday 17th October 2012
Peter.Hendry@cem.dur.ac.uk
The word ‘PREDICTION’:
Quite probably the most contentious term that is
used!! Concerns include:
• A prediction for GCSE at start of year 7?
• What has the baseline test got to do with my
subject?
• I know my pupils!
• The predictions are too low: not valid!!!
• And what about my professional judgement?
How is a ‘prediction’ generated?
*
** *** ******************* A*
* ** *** ******************************** A
* ** *** ********************************* *** ** * B
C
GRADE
GRADE
* ** *** ************************************ *** ** *
* ** *** ******************************** **** ** *
* ** ***************************** *** ** *
* ** *** ******************* ** * *
* ** *********** ** *
50
100
BASELINE SCORE
150
3 key points are:
• The higher the baseline score the higher
the final grade
• Any one grade is achievable from a range
of baseline scores
• From any baseline score, a range of
grades are possible
How is a ‘prediction’ generated?
Subject National
trend line
* **
(regression line)
50% on or above the
trend line
* ** *** ******************* A*
*** ******************************** A
* ** *** ********************************* *** ** * B
‘PREDICTION’ (expected grade)
* ** *** ******************************** **** ** *
* ** ***************************** *** ** *
* ** *** ******************* ** * *
50% on or below the
* ** *********** ** *
trend line
BASELINE SCORE
C
GRADE
GRADE
* ** *** ************************************ *** ** *
2008 GCSE Maths (representative sample)
9
8
7
Result
6
5
4
3
2
1
0
0
20
40
60
baseline score
80
100
‘Predictions’…...are based on
Average performance by similar
pupils in past examinations
The problem with the word ‘prediction’ is…?
An alternative is ‘expected’ grade
Predictions
6.6 points = A/B
4 points = D
Trend line
The graph below shows the middle 2/3 of some subject trend lines
MidYIS Year 7 to GCSE 2011
Value Added Analysis
Art & Design
8
Biology
7
English
GCSE Points Score
6
5
French
4
History
3
Mathematics
2
1
50
60
70
80
90
100
110
120
MidYIS Test Score
Comments?
130
140
150
Science
Some Subjects are More Equal than Others….
A
Grade
B
>1 grade
C
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
D
E
C
B
A
Average GCSE
A*
FACTORS THAT WILL INFLUENCE
RELIABILITY OF PREDICTIONS:
•
•
•
•
•
•
•
•
•
•
Knowledge of student
Parental support/home life
Peer influences/social life
Student attitude, interest, language
Expectations of staff
Department/institution ethos
Resources
Quality of teaching and learning: pace of lessons
Understanding how children learn………
And the reliability of the predictions......
Result
Correlation = 1
12
Correlation = 0
Correlation = 0.7
CORRELATIONS
SUBJECT
ART and DESIGN
BIOLOGY
ECONOMICS
ENG LIT
FRENCH
HISTORY
MATHS
MUSIC
PHYS EDUC
PSYCHOLOGY
GCSE
to A2
0.54
0.69
0.65
0.71
0.6
0.69
0.6
0.65
0.65
0.62
MIDYIS( Yr 7)
subject
to GCSE
ART
0.49
BIOL
0.56
ENG LANG
0.7
FRENCH
0.64
HISTORY
0.63
MATHS
0.74
MUSIC
0.57
A
6.8 7.2
2
F
7E 120
A
3
F
7C 110
4
F
7J
5
M
7D
1
7.2
7.5
7.0
6.3
6.7 6.4 6.1
6.4
6.5
6.3
B
5.8
6.2 5.7 5.4
5.5
5.6
5.5
101
B
5.4 5.8
5.1 4.8
4.9
4.9
4.9
82
D
4.5 4.9
3.8 3.5
3.4
3.2
3.5
M
7E 131
A
2
F
7E 120
A
3
F
7C 110
B
B
B
4
F
7J
101
B
B/C
B
C
5
M
7D
82
D
C/D
C
D
A
A
English
7.1 6.8
Biology
Science
7E 131
Mathematics
Art & Design
M
History
MidYIS Band
1
French
Form
• Look at Art and Design,
Biology and French.
Comments?
Sex
What pattern do you
notice?
student no.
• Compare the predictions
for English and
Mathematics.
MidYIS Score
Point and grade ‘predictions’ to GCSE
A
A/B A/B A/B
A
B
A A*/A
A
A/B A/B A/B
B/C B/C B/C B/C B/C
C
C
D/E D/E
C
C
E
D/E
The graph below shows the middle 2/3 of some subject trend lines
MidYIS Year 7 to GCSE 2011
Value Added Analysis
Art & Design
8
Biology
7
English
GCSE Points Score
6
5
French
4
History
3
Mathematics
2
1
50
60
70
80
90
100
110
120
MidYIS Test Score
Comments?
130
140
150
Science
Individual Chances Graph for Student 4- GCSE English
MidYIS Score 101 MidYIS Band B
Teacher's Adjustment : 0 grades / levels / points
45
39
40
35
Percent
30
Prediction/expected grade:
5.1 grade C
27
25
20
19
Most likely grade
15
9
10
5
5
0
0
U
G
1
1
0
F
E
D
Grade
C
B
A
A*
MidYIS Year 9 Test Academic Year 2011/2012
Predictions based on 2011 Regression Analysis
Predictions & Chances Graphs to GCSE subjects
to place school at 75th
percentile of VAD
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of
the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's
Adjustment
(Grades)
Display Legend
4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6
3
F
91400042
108
B
4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.4
4
F
91400043
101
B
4.2 5.0 5.6 5.7
4.8 5.7
Geography
A
French
111
English Literature
9140005K
5.1 5.9
English
F
Cust
Drama
2
CEM Id
Design & Technology
4.4 5.3 5.8 5.9
Form
Chemistry
B
Sex
Business Studies
105
Forename
Biology
9140005J
Surname
Art & Design
Additional Applied Science
F
Generate File for
MIS Import
Additional Science
MidYIS Band
1
Prior Value
added
Student
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
MidYIS Score
Upper Quartile
of Schools
5.3 5.2 5.4 5.5 5.0 5.1
5.0 5.0 5.1 5.3 4.7 4.7
MidYIS Year 9 Test Academic Year 2011/2012
Predictions based on 2011 Regression Analysis
Predictions & Chances Graphs to GCSE subjects
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of
the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's
Adjustment
(Grades)
Display Legend
4.9 6.0 6.3 6.5 5.8 6.5 5.9 5.8 6.0 6.1 5.6 5.8
3
F
91400042
108
B
4.8 5.8 6.2 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6
4
F
91400043
101
B
4.4 5.2 5.8 5.9
5.0 5.9
Geography
A
French
111
English Literature
9140005K
5.3 6.1
English
F
Cust
Drama
2
CEM Id
Design & Technology
4.6 5.5 6.0 6.1
Form
Chemistry
B
Sex
Business Studies
105
Forename
Biology
9140005J
Surname
Art & Design
Additional Applied Science
F
Generate File for
MIS Import
Additional Science
MidYIS Band
1
Prior Value
added
Student
0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2
MidYIS Score
Reset
Adjustments
5.5 5.4 5.6 5.7 5.2 5.3
5.2 5.2 5.3 5.5 4.9 4.9
MidYIS Year 9 Test Academic Year 2011/2012
Predictions based on 2011 Regression Analysis
Predictions & Chances Graphs to GCSE subjects
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of
the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's
Adjustment
(Grades)
Display Legend
4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6
3
F
91400042
108
B
4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.4
4
F
91400043
101
B
4.2 5.0 5.6 5.7
4.8 5.7
Geography
A
French
111
English Literature
9140005K
5.1 5.9
English
F
Cust
Drama
2
CEM Id
Design & Technology
4.4 5.3 5.8 5.9
Form
Chemistry
B
Sex
Business Studies
105
Forename
Biology
9140005J
Surname
Art & Design
Additional Applied Science
F
Generate File for
MIS Import
Additional Science
MidYIS Band
Or insert own values
and click adjust
1
Prior Value
added
Student
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
MidYIS Score
Upper Quartile
of Schools
5.3 5.2 5.4 5.5 5.0 5.1
5.0 5.0 5.1 5.3 4.7 4.7
MidYIS Year 9 Test Academic Year 2011/2012
Predictions based on 2011 Regression Analysis
Predictions & Chances Graphs to GCSE subjects
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of
the method used to make these predictions
Points To Grades
Adjust
GCSE Point Predictions
Teacher's
Adjustment
(Grades)
Display Legend
B
4.6 5.9 7.4 6.8 5.4 6.8 5.5 6.3 6.9 6.5 5.9 6.8
4
F
91400043
101
B
4.2 5.3 7.0 6.4
4.8 6.4
Geography
108
French
91400042
English Literature
F
5.1 6.6
English
3
Cust
Drama
4.7 6.1 7.5 7.0 5.6 7.0 5.7 6.5 7.1 6.7 6.1 7.0
CEM Id
Design & Technology
4.4 5.6 7.2 6.6
A
Form
Chemistry
B
111
Sex
Business Studies
105
9140005K
Forename
Biology
9140005J
F
Surname
Art & Design
Additional Applied Science
F
2
Generate File for
MIS Import
Additional Science
MidYIS Band
1
Reset
Adjustments
Student
0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4
MidYIS Score
Upper Quartile
of Schools
5.3 6.1 6.7 6.3 5.7 6.5
5.0 5.9 6.4 6.1 5.4 6.1
MidYIS Year 9 Test Academic Year 2011/2012
Predictions based on 2011 Regression Analysis
Predictions & Chances Graphs to GCSE subjects
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of
the method used to make these predictions
Adjust
A
A/B
B
A
B A*/A
A
B/C
A
B/C A/B
A
A/B
B
A
A/B
C
A/B
A/B
B
B/C
B
2
F
9140005K
111
A
C/D
3
F
91400042
108
B
C/D
4
F
91400043
101
B
D
Cust
B/C
A
A/B B/C
C
B
B
Geography
B/C A/B
C/D B/C
CEM Id
French
A
B
Form
English Literature
B/C
105
Sex
English
Business Studies
A
9140005J
Forename
Drama
Biology
B A*/A
F
Surname
Design & Technology
Art & Design
C
MidYIS Band
A/B
1
Generate File for
MIS Import
Student
A
MidYIS Score
Reset
Adjustments
0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4
Additional Science
Upper Quartile
of Schools
Additional Applied Science
Teacher's
Adjustment
(Grades)
GCSE Grade Predictions
Chemistry
Grades To Points
Display Legend
A/B A/B B/C A/B
Prediction/expected grade:
6.4 grade A/B
Most likely grade
MidYIS Year 9 Test Academic Year 2011/2012
Predictions based on 2011 Independent Sector Regression Analysis
Predictions & Chances Graphs to GCSE subjects
based on Year 9 Independent Sector standardised scores
Please refer to the Information Sheet for details of
the method used to make these predictions
Points To Grades
Design & Technology
C
6.1 6.8 6.7 6.0 6.7 6.1 6.4 6.4 6.5 6.4 6.1 6.5 6.2
3
F
91400042
90
C
6.0 6.7 6.6
5.9 6.6
4
F
91400043
81
D
5.5 6.5 6.1
5.4 6.1 5.6 5.8 5.9 5.9 5.9 5.4 5.8 5.6
German
94
Geography
9140005K
French
F
5.7 6.4
English Literature
2
Cust
English
5.8 6.6 6.4
CEM Id
Drama
D
Form
Chemistry
87
Sex
Business Studies
9140005J
Forename
Biology
Additional Science
F
Surname
Art & Design
MidYIS Band
1
Generate File for
MIS Import
Student
Classical Civilisation
GCSE Point Predictions
MidYIS Score
Display Legend
5.8 6.1 6.1 6.1 6.2 5.7 6.1 5.8
6.0 6.3 6.3 6.3 6.3 5.9 6.3 6.0
Independent Sector
Prediction/expected grade:
5.9 grade C
Most likely grade
Not a label for life
...just another piece of information
• The Chances graphs show that, from almost any
baseline score, students come up with almost any
grade - - -there are just different probabilities for
each grade depending on the baseline score.
• In working with students these graphs are more
useful than a single predicted or target grade
• Chances graphs show what can be achieved:
– By students of similar ability
– By students with lower baseline scores
Yellis predictive data: baseline score 103 (55%)
Yellis
Average
Subject
Predicted
Business Studies
5.4 (B/C)
English
5.7 (B/C)
French
5.4 (B/C)
Geography
5.6 (B/C)
Mathematics
5.7 (B/C)
Physical Education
5.7 (B/C)
Science: GCSE
5.6 (B/C)
Science: GCSE Additional 5.6 (B/C)
SC Religious Studies
5.3 (B/C)
Yellis
Average
Subject
Predicted
Business Studies
5.6 (B/C)*
English
5.9 (B)*
French
5.6 (B/C)*
Geography
5.8 (B)*
Mathematics
5.9 (B)*
Physical Education
6.0 (B)*
Science: GCSE
5.8 (B)*
Science: GCSE Additional 5.8 (B)*
SC Religious Studies
5.6 (B/C)*
Weighted Average
Weighted Average
5.6 (B/C)
Yellis
Average
Subject
Predicted
Business Studies
5.4 (B/C)
English
5.7 (B/C)*
French
5.4 (B/C)
Geography
5.8 (B)*
Mathematics
5.9 (B)*
Physical Education
6.7 (A/B)*
Science: GCSE
6.3 (A/B)*
Science: GCSE Additional 5.6 (B/C)
SC Religious Studies
5.7 (B/C)*
Weighted Average
5.8 (B)
5.8 (B)
Polly Bolton
B
SECURE
5
ORG
MC
E
B
UNLIKELY
3
Science
CPa
E
C
UNLIKELY
Science Additional
CPa
D
C
French
CK
C
History
KM
RS
CG
CB
Maths
B
SECURE
5
C
B
LIKELY
5
HW
D
B
POSSIBLE
4
4
ORG
D
C
POSSIBLE
4
ORG
LIKELY
4
ORG
B
C
LIKELY
4
ORG
B
LIKELY
4
C
B
LIKELY
5
A
A
SECURE
5
B
A
SECURE
5
D
B
POSSIBLE
4
C
B
POSSIBLE
4
KEY - target is:
SECURE
LIKELY
POSSIBLE
UNLIKELY
effort: 5 excellent - 4 good - 3 satisfactory - 2 poor - 1 very poor
concern: WW working well - ATT attitude - BEH behaviour
TEN attendance - PUN punctuality - HW homework
CON confidence - ORG organisation - EAL language
concern
B
English Literature
B
is:
effort
ORG
target
grade
5
year 10
exam
LIKELY
concern
concern
B
CB
is:
effort
effort
C
English
target
grade
is:
Teach
Subject
SUMMER
current
grade
target
grade
SPRING
current
grade
AUTUMN
Student 1
Student 2
Why is the ‘predicted’ grade not always equal
to the highest bar ?
Predicted (‘expected’) grade
Most likely grade
AT WHICH POINT WILL THE SEESAW BE BALANCED?
Predicted (‘expected’) grade
i.e. the lower grades ‘pull’ the prediction to the left
Student 3
Student 4
Student 4 - IPR
Vocab Item Responses
Correct
30
Response Time (s)
25
20
15
10
5
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
Question
Wrong
Basing Targets on Prior VA – One Methodology from an
Alis School
•
Discuss previous value added data with each HoD
•
Start with an agreed REALISTIC representative figure based, if available
on previous (3 years ideally) of value added data
•
add to each pupil prediction, and convert to grade (i.e. in-built value
added)
•
Discuss with students, using professional judgment and the chances
graphs, adjust target grade
•
calculate the department’s target grades from the addition of individual
pupil’s targets
theet to
tadl e
tasrg
d ello
s a
th e to ta l ta rg
g ra
ae
ret g
a ra
s fo
wre
s : a s fo llo w s :A
F ro m a n d in c lu d in g 2 0 0 2 , a ra w re s id u a l o f 2 0 .0 is e q u iva le n t to o n e gBra d e
1A
01
2B
32
C
6C
56
S U G G E S T E D T A R G E T S F O R 2 0 0 7 , b a s e d o n A L IS p re d a n d d e p D
t's v a lu e a d1Dd e d h is to ry
11
T h e ta rg e t g ra d e h a s a n in -b u ilt va lu e a d d e d o f 1 5 p o in ts (o n e g ra d e E
is 2 0 p o in ts 0
)E
00
ta rg e t
dept adj
D EPAR T M EN T : A
g ratadrg
e et
taTrg
T a rg e t S e ttin g
etaaerg
ct heet r
tn
g
dee
a sA
d eE T a dgj ra
ta0d
rgee t
S u rn aFth
m
e to
FAoeve
re
Are
ve
C fo
S ElloPwresTd: D
reRdAG
ic E
tioTn T g
Ara
R1G
S u rn a m e
oere
n ata
ml eta rg
Gara
CmS
Es a
TG
D
icA
tio n P
TA
no. of
S
td
.
3
yr.
A
v.
B
2
3
D
48
.7
2 .3
8
.3
6D
4 .3
D
4 .7
2
49
6449.3
p u p ils
R
e
s
id
S
td
re
s
id
ye a r
a v. 5G.8
C S E a v.3
T
D
A
ra
w
re
s
id
.
C
6
5
50
.8
0
.2
8C
8 .2
C
7 3 .2
8783.2
C
11 .4
1
2002
2
6 .8
44698.9
.0
29 4
.5
16D
.2
8
101A.7
.9
6 .4
19
1.4
.4
BA
0.8
0
.2
6 .8
1
.8
9
5.8
.2
0
98
.8
CB
2003
7
76 .1
46 19
18 3
.3
050E
.6
0B
0
3
5
1
0
Teacher
a dEj StaUrg
et R
R
LT
D
D
D
C
C
C
B
B
B
C
C
C
.1
3 .8
9
.8
7
2.8.8
C
B
B
.1
7
75
.8
B
B
2004
6
65 .6
535 91
15 8
.2
027.7
0C
ta8C
rg
et
T e aC
5
.53 .8 4 1 2 .8626
3±.3
04 .0 5 0 .686
6.3
1 .3
Cc h e r D
C
D
2005
12
6 .1 75±.50 .2 2 4 4 .5 3
00
±
01±
0.3.2 9
0 .6
5
g
ra
d
e
a
d
j
ta
rg
e
t
S u rn a m e
F o re n a m e A ve5G.4C S E
T5D4
A
P
re
d
ic
tio
n
T
A
R
G
E
T
R
E
S
U
L
T
.4
5 .4
4
.4
7C
8 .4
C
C
B
63
7683.4
C
B
D
.7
8 is e q u iva
9.9
.3
D.9
D
52.0
.2
3
3.3
5g44
9ra
.9d e
7C
4
C
C
C
3
549
.9
C
C
F ro m a n d in c lu d in g 2 0 0 2 , a ra w re54s.2
id u a l o f 2 03
le
n t to o n e76
.8
0
.2
C.1
C
C
65.1
5
779
98
B
B
633
.1
53.1
3.2
748
9.1
.1
9B
4
B
B
B
6
.9
4
8
9
6
.4
1
1
1
.4
A
B
B
RTASG F
EOARV2E0R
50.7
.8A L IS p re
6491d.6
49.6
A7G
51n
.7
.8a n d d e p86t's
.6v a lu e a 8dC
4d.6
S U G G E S T E D T AARVGEE
, bEa s e d4o
e d h is to ryC
0 .8
5 .82 0 p o in ts )B
C
C
T h e ta rg e t g ra d e h a s a n in -b u ilt va6lu.2e a d d e d o6f 11 5 p o in ts8(o
n e g ra d e9is
5 .1
39
5 7 .8
7 2 .8
B
ta rg eC
t
dept B
adj
5 .5
30
6 6 .3
8 1 .3
D
g ra d eC
ta rg eC
t
.4 d e s a re a
5s
4 fo llo w s6: 3 .4
7A
8 .4
C
C
B
th e to ta l ta rg e t g5ra
1
0
5 .2
33
5 9 .9
7B
4 .9
C
C
C
2
3
6 .1
53
7 9 .1
9C
4 .1
B
6B
5B
AVERAGE
5 .7
4 1 .8
6 9 .6
8D
4 .6
C
1
1
E
0
0
S u rn a m e
F o re n a m e
A ve G C S E
4 .7
5 .8
TDA
28
30
P re d ic tio n
4 9 .3
7 3 .2
TARGET
6 4 .3
8 8 .2
ta rg e t
g ra d e
D
C
Teacher
a d j ta rg e t R E S U L T
D
D
C
C
Key Questions for Target Setting
• What type of valid and reliable predictive data
should be used to set the targets?
• Should students be involved as part of the
process (ownership, empowerment etc.)?
• Should parents be informed of the process
and outcome?
Key points to consider might include:
•
•
•
•
•
•
•
Where has the data come from?
What (reliable and relevant) data should we use?
Enabling colleagues to trust the data: Training (staff)
Communication with parents and students
Challenging, NOT Demoralising, students…….
Storage and retrieval of data
Consistency of understanding what the data means
and does not mean