Peter Hendry: CEM Consultant
Using CEM Data in Practice:
Feedback from CEM Secondary
Assessments
Glasgow Conference
13th Feb 2013
Peter.Hendry@cem.dur.ac.uk
Secondary@cem.dur.ac.uk
Assessment for Excellence: 2 systems
S1/S2 Baseline Assessment (MidYIS):
A Test of Developed Ability at start of S1 or S2
•
4 sections: Vocabulary, Maths, Non-Verbal and Skills (Proof Reading,
and Perceptual speed and accuracy
•
Computer adaptive* or paper-based
•
Any time Term 1 + catch ups
•
Value Added measures to SOSCA and to Scottish Qualifications
S2 Curriculum-based Assessments (SOSCA):
A curriculum assessment of knowledge, skills and understanding at end of S2
•
Assesses what has been taught in the classroom
•
Covers, Maths, Science, Reading (English)
•
Computer adaptive*
•
Four week assessment window mid April to mid May + catch ups
•
Value Added measures from PIPs, InCAS and MidYIS and to Scottish
Qualifications
AfE: key elements
• School and students’ test performances are compared to a
nationally representative sample
• Assessment data from the tests provides school and pupil
data to inform teaching and learning
• ‘Predictive’ data indicates future potential performance to
SQ, and will to National 4 and 5, to inform target setting
and subsequent monitoring
• ‘Value-added data’ provides measures of relative
performance at SQ (and will for National 4 and 5) and
SOSCA from earlier baselines to inform self-evaluation
AfE Feedback: MidYIS and SOSCA
Assessment Feedback:
• Standardised score system: 50-150, mean 100, SD 15
• School data including component averages and band
profiles (historical and current)
• Pupil data in the form of spreadsheets and Individual Pupil
Records (IPRs)
AfE Feedback: MidYIS and SOSCA
‘Predictions’ Feedback:
• Currently to S2 SOSCA from S1/S2 baseline (MidYIS) and
to SQ from MidYIS and SOSCA
‘Value-Added’ Feedback:
• To SQ from curriculum baseline (SOSCA), and from S1/S2
baseline (MidYIS)
• To S2 curriculum assessment (SOSCA) from P7 baseline
(PIPS or InCAS) and/or S1/S2 baseline (MidYIS)
www.cemcentre.org
Detailed
Information
about the
assessment.
Well worth a
look.
Data for your
school.
Bands, percentiles, standardised scores…
25% 25%
25%
C B
25%
A
D
0
50
60
70
80
90
100
110
120
130
90
95 98
140
Standardised scores
Percentiles:
1
5
10
20
30 40 50 60 70
80
Mean 100, Standard Deviation of 15
99
150
SCHOOL
A: S1
baseline
data Year 2010/2011
YIS
Year 8/S1
Test
Academic
MidYIS Year 8/S1 Test Academic Year 2010/2011
es Standardised On A Nationally Representative Sample of Schools
Scores Standardised On A Nationally Representative Sample of Schools
e Surname
Forename Forename
CEM Id
JAMES
JAMES
COREY
COREY
JOHN
JOHN
SAMUEL
SAMUEL
DANIELLE
DANIELLEFRAZER
FRAZER
Standardised Scores
Sort Data By...
Standardised Scores
Mean Score: 100
Sort Data By...
Sort Data By...
Sort Data By...
Mean Score: 100
Standard Deviation: 15
Standard Deviation: 15
CEMForm
Id
AverageAverage93.3
93.3
St. error St. error 1.2
1.2
1.2
81
D
76
D
93
C
88
D
117
98A
D
69
D
61
C
87
D
91
A
C98
69
D
61
D
87
D
91
C
98
89 C
C
89
D
Form
Sex
M
M
M
M
F
M
Sex Cust
Cust
M
81
M
76
M
93
M
88
F
117
M
98
89.9 89.9
90.9 90.9
97.397.3
90.7
90.7
1.2
1.2 1.2
1.2 1.2
1.21.2
D
75
D
64
D
101
C
82
C
D 107
98
75
64
101
82
107
98
D
D
B
D
B
C
D
68 68
D
57
57
B
92
92
D
91
91
B
105
105
C
117
117
D
D
C
C
B
A
D
D
C
C
B
A
7272
65
65
89
89
88
88
109
10993
93
Band
Band
St. Score
St. Score
MidYIS
Score
MidYIS
Score
Band
Band
St. Score
St. Score
Band
Band
St. Score
St. Score
Non Verbal
Non Verbal Skills
Skills
Band
Band
St. Score
St. Score
Band
Band
St. Score
St. Score
Vocabulary
MathsMaths
Vocabulary
DD
D
D
D
D
D
D
B
BC
C
SCHOOL A: S1 baseline data
Intake profiles: overall and test component bands
A school with a ‘completely average’ intake would
have 25% of pupils within each band.
SCHOOL A: S1 Baseline
MidYIS Year 8/S1 Historical Band Profiles
Scores Standardised on a Nationally Representative Sample of Schools
Year
Band A
Band B
Band C
Band D
2013
2012
2011
2010
2009
5
5
6
11
8
12
18
18
20
18
17
28
30
29
38
65
49
46
41
36
Historical Band Profiles: Overall Score
Percentage of pupils falling into each MidYIS Band over time.
(Nationally Representative Standardisation)
(Band A = Top 25% of the sample, Band D = Bottom 25% of sample.)
100%
90%
80%
Percentage
70%
60%
Band D
50%
Band C
40%
Band B
30%
Band A
20%
10%
0%
2009
2010
2011
2012
2013
MidYIS Baseline Testing Year
Test Section
Overall
Show Band Percentage Values
Start Year
2009
End Year
2013
based Standardised Feedback 2012
Curriculum
2AfE
S2 Curriculum based Standardised Feedback 2012
tandardised On AfE Curriculum based Sample 2012
Scores Standardised On AfE Curriculum based Sample 2012
SCHOOL A
Scores
Standardised
Standardised Scores
Sort Data
SortBy...
Data By...
100
Mean Score:
Mean
Score: 100
15
Standard Deviation:
Standard Deviation:
15
Surname
Forename AfE Id
James
Form
Id
Forename AfE
Corey
James
John
Corey
Samuel
John
Danielle
Samuel
Frazer
Danielle
Frazer
St. error
Form
Sex
M
M
M
M
F
M
A
D
119
106
D
D
B
C
A
B
77
77
87
101
90
94
11987
10678
110
90
87
D
90
D
84
D
87
D
75
D
78
D
D110 86 B
D 90 81 D
B
D
117
92
92.5
92.5
1.3
1.3
1.3
84
75
D
86
D
81
D117
D92
D
D
88
D
66
D
50A
50C
88
D
66
D
78
D
50
D
78
D
50
D
50
D
105
B
D
96
C
A
C
105
96
B
C
100
91
1.3
1.3
1.3
D
D
89
D
91
89
63
D
53
D
63
66
53
B
90D
D
66
C 100
D
D
84
D
66
D
C50
B
B
C
90
100
C
B
98
95
D
D
100
1.3
96.9
Energy & Forces
97.6
96.9
96.1
97.6
96.9
1.3
1.3
1.3
1.3
1.3
1.3
137
107
A
B
1.3
D 104 B
91
D
57
D
74
91
104 B
D
111
74
D
57
D
73
D
90
A 139 A 111
121
B 7396 D C 9095
139
96
A
C
121
95
Band
Band
St. St.
Score
Score
Band
Overall
OverallScience
Science
Energy & Forces
Band
St. Score
St. Score
Earth & Space
Band
Earth & Space
St. Score
Band
Life
Band
96.1
96.9
84
D
84
66
D
50
D
84
D
50
D
78
D
50
D
84
98D C 78 137 D
95
C 84 107 D
C
C
St. Score
St. Score
Band
Mathematics
Living Things &
theLiving
Processes
Thingsof
&
the Processes
of
Life
Band
Band
Overall
Mathematics
Overall
92.1
92.1
1.3
78
78
D
50
St. Score
Band
St. Score
Band
92.6
92.6
St. Score
Shape, Position &
Movement
Science
Science
Shape, Position &
Movement
92.6
92.6
St. Score
Band
St. Score
Information
Handling
St. Score
Band
Information
Handling
St. Score
Band
Number, Money &
Measurement
Band
St. Score
Number, Money &
Measurement
Band
Overall Reading
1.3
1.3
D
D
B
C
A
B
St. Score
St. Score
Overall Reading
Band
Band
Passage
Comprehension
St. Score
St. Score
Comprehension
Band
Band
St. Score
Text
Passage
Comprehension
1.3
1.3
C
D
D
D
A
D
95.9
95.9
1.3
1.3
1.3
Sex Cust
94
CustM
M
71
77
C
94
M
76
77
D
71
M
81
101
D 115
F 76
D
81
M
8594
Mathematics
Mathematics
97.2
97.2
1.3
St. error
St. Score
Band
98.1
98.1
94.2
115
85
Band
St. Score
94.2
Average
Average
St. Score
Band
St. Score
Speed Reading
Speed
Reading
Text
Comprehension
ReadingReading
1.3
C
D
C
A
D
D
AA
DC
A
C
92
50
C
D
92
50
C
D
138
99
A
C
79
138
79
99
D
A
D
C
SCHOOL A: S2
curriculum
(SOSCA) data
School no.1
School no.2
STANINES
Stanine 1
2
4%
50
60
3
4
5
6
7
8
9
7% 12% 17% 20% 17% 12% 7%
70
80
90
100
110
120
Standardised Test Score
4%
130
140
150
Individual Pupil Record
Date Of Birth:
Sex:
Vocabulary
Maths
Non-Verbal
Skills
MidYIS Score
Student Number:
Form:
22/02/99
F
Band
Stanine
Percentile
Standard
Score
A
7
87
117
C
5
45
98
B
6
67
107
B
6
63
105
B
6
73
109
Nationally Standardised Scores with 95% Confidence Band
150
Nationally Standardised Scores
S1 baseline IPR
DANIELLE
140
130
120
110
100
90
80
70
60
50
Vocabulary
Maths
Non-Verbal
Skills
MidYIS Score
S2 curriculum (SOSCA) IPR
Individual Pupil Record
Date Of Birth:
22/2/1999
Sex:
F
Danielle
Student Number:
Form:
AfE Band
Stanine
Percentile
AfE Standardised
Score
Speed Reading
Text Comprehension
Passage Comprehension
Overall Reading
Number, Money &
Measurement
Information Handling
Shape, Position &
Movement
Overall Mathematics
Living Things & the
Processes of Life
Earth & Space
A
A
B
A
7
8
6
7
84
90
74
87
115
119
110
117
B
6
62
105
B
5
51
100
C
4
26
90
C
5
46
98
A
9
99.3
137
A
9
99.5
139
Energy & Forces
A
8
92
121
Overall Science
A
9
99.4
138
S2 curriculum (SOSCA) IPR
Individual Pupil Record
Date Of Birth:
22/2/1999
Sex:
F
Danielle
Student Number:
Form:
Predictions Feedback
• Currently available to Standard Grade from
AfE baselines
• In the future to National 4 & 5 Qualifications
from AfE baselines
• Also to AfE S2 curriculum (SOSCA) from S1
and/or S2 baseline (MidYIS)
• Includes Chances Graphs
How is a ‘prediction’ generated?
******************************* 2
*******************************************
‘PREDICTION’ (expected grade)
GRADE
*************************************
**************************************
********************************
*****************************
50% on or below the
trend line
*****************
BASELINE SCORE
3
4
GRADE
Trend line
(regression line)
50% on or above the
trend line
*********************** 1
Grade
Data for 2012 examinations
AfE S2 Curriculum based Assessments 2012
Predictions & Chances Graphs to Standard Grade subjects
based on 2012 AfE scores
Predictions & Chances Graphs to Standard Grade subjects
based on
2012
AfEInformation
scores
Please
refer
to the
Sheet for details of
the method used to make these predictions
Please refer to the Information Sheet for details of
the method used toPoints
make these
predictions
To Grades
Adjust
Standard Grade Point Predictions
M
Kieran Roland M
Megan
F
Danielle
F
Frazer
M
84
75
68
100
117
92
84
66
60
89
98
95
92
85 3.8
50
54 5.9
72
63 5.3
93
93 3.3
138 120 1.5
99
95 3.2
3.1
4.2
3.9
2.8
1.8
2.7
4.3
6.6
5.9
3.6
1.5
3.5
3.3
5.4
4.8
2.7
1.0
2.6
3.8
5.6
5.1
3.3
1.7
3.2
3.8
5.6
5.1
3.3
1.7
3.2
3.3
4.7
4.3
2.9
1.7
2.8
3.0
4.3
3.9
2.6
1.4
2.5
3.4
4.8
4.4
3.0
1.8
2.9
4.0
5.9
5.4
3.5
1.9
3.4
4.2
6.4
5.8
3.6
1.7
3.5
4.2
6.0
5.5
3.7
2.1
3.6
3.7
5.2
4.8
3.3
2.0
3.2
84
75
68
100
117
92
84
66
60
89
98
95
92
85
4
3
50
54
6
4
72
63 5/6 4
93
93 3/4 3
138 120 1/2 2
99
95
3 2/3
4/5
6/7
6
3/4
1/2
3/4
3/4 4
4 3/4 3 3/4 4
4
4 3/4
5/6 5/6 5/6 4/5 4/5 5
6 6/7 6
5
5
5
5 4/5 4 4/5 5/6 6 5/6 5
2/3 3/4 3/4 3 2/3 3 3/4 3/4 3/4 3/4
1 1/2 1/2 1/2 1/2 2
2 1/2 2
2
2/3 3
3
3 2/3 3 3/4 3/4 3/4 3
Overall AfE Score
Graphic Communication
M
Corey
German
James
Geography
M
French
Frazer
English
F
Drama
Danielle
Craft & Design
F
Computing Studies
Megan
Chemistry
Kieran Roland M
Business Management
M
Biology
Corey
Cust
Art & Design
M
AfE Id
Administration
James
Form
AfE Science Score
Sex
AfE Maths Score
Forename
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
AfE Reading Score
Teacher's
Adjustment
(Levels)
S
Predictions: Chances Graphs
Point Prediction =1.8
Grade Prediction = 2
Chances Graphs based on Pupil’s SOSCA Test Score
50% chance of a
grade 2 – the
most likely single
grade. 50%
chance of a
different grade
Not a label for life
...just another piece of information
• The Chances graphs show that, from almost any
baseline score, students come up with almost any
grade - - -there are just different probabilities for
each grade depending on the baseline score.
• In working with students these graphs are more useful
than a single predicted or target grade
• Chances graphs show what can be achieved:
– By students of similar ability
– By students with lower baseline scores
Latest information as at 13 Feb 2102
• Replacing SG with Nat 4 & 5 predictions:
CEM statisticians have put together a
proposal.
• This has been looked at by colleagues in
Scottish Authorities.
• The CEM Technical team have now
published the new predictive data.
• This data is now available on the CEM
secondary+ website alongside the existing
SG predictive data
The equivalence we have assumed between CfE outcomes and Standard Grade outcomes is
outlined below:
National Level
National Level 5 A,B
National Level 5 C,D
National Level 4
National Level 3
National Level 2
Standard Grade
Credit Level 1
Credit Level 2
General Level (3 and 4)
Foundation Level (5 and 6)
Standard grade level 7
For each student in each CfE subject, a predicted grade is given. The range of predicted
grades is as follows
Grade
National Level 5 A
National Level 5 B
National Level 5 C
National Level 5 D
National Level 4
National Level 3
National Level 2
CEM Predicted Grade
A
B
C
D
N4
N3
N2
Where a subject is only available at National Level 4 (e.g. Business), only grades N4, N3, N2
will be reported.
For subjects only available at National Level 5 (e.g. Accounting), the full range of grades will
be reported. Whilst we recognise that grades N4, N3, N2 are not available we have included
them in the chances graphs to aid the teacher in supporting the student in making realistic
subject choices.
AfE S2 Curriculum based Assessments 2012
New slide!!
Predictions & Chances Graphs for CfE National Qualifications
based on 2012 AfE scores
The grade 'predictions' and chances data presented here are simulated based on the relationship between AfE S2 Curriculum based,
Standard Grade and Curriculum for Excellence levels, in the absence of any actual CfE examination data being available.
They should therefore be treated with caution. The relationships will be updated once the first cohort are awarded CfE qualifications in 2014.
C
N4
B
C
C
C
C
D
C
C
B
N4
A
B
B
B
B
C
B
103 116
108
C
D
D
115
112 123
119
B
C
125
135 117
129
A
B
Art & Design
102
N4/D N4/D
Drama
D
N4
Design & Manufacture
D
D
Dance (N5)
D
99
Computing Science
D
99
Chemistry
D
104
Care
Business Management (N5)
D
95
Biology
D
Administration & It
C
Accounting (N5)
N4
Overall AfE Score
D
AfE Science Score
D
AfE Maths Score
N4
AfE Reading Score
Business (N4)
CfE Grade Predictions
N4/D N4/D N4/D N4/D N4/D N4/D
.
Measuring Value-Added – Terminology
Subject X
Exam grade
Outcome
120
100
80
-ve VA
+ve VA
60
40
20
04
25
46
BASELINE SCORE
67
8
Raw Residual
Trend Line/Regression Line
SCHOOL A: S2 curriculum (SOSCA) data
Year 8 MidYIS Score 2010/2011 Against AfE Overall Mathematics 2012
170
160
150
140
AfE Score
130
120
110
100
90
80
70
60
50
50
60
70
80
90
100
110
120
130
140
MidYIS Test Score (Standardised on a Nationally Representative Sample)
150
SCHOOL C:
AfE data
SCHOOL C:
AfE data
Standardised Residual Bar Chart
99.7% confidence limit
95% confidence limit
ANY VALUE IN THE INNER SHADED
AREA SHOWS THE EXPECTED RESULTS
i.e. AVERAGE VALUE ADDED
SCHOOL C:
SQ data