Introduction to CEM
Secondary Information Systems
Dr Robert Clark
ALIS Project Manager
The Projects
The Projects
A / AS / Btec / IB etc
Year 13
Year 12
Alis
GCSE
Computer Adaptive Baseline Test
(paper test also available)
GCSE
Year 11
Yellis
Year 10
Computer Adaptive Baseline Test
INSIGHT
Combines curriculum tests with
developed ability
(paper test also available)
Year 9
Year 8
(+ additional)
Year 7
MidYIS
Computer Adaptive Baseline Test
(paper test also available)
Typical Timeline
Typical Timeline
Early Testing &
Predictions
START:
Measure Baseline
Autumn Term
Feedback
Reports
A ugust
END:
September
Collect and Send Results
Value-Added
Feedback
Testing is available from June at end
of previous academic year
i.e. Y6 / Y9 / Y11
The baseline score is measured as
early as possible and the information
sent to the CEM Centre
The institution then receives feedback
from the CEM Centre: baseline test
results and predictions
The public examination results are
sent to the CEM Centre when received
by the institution
The value added data is available
from September via the Institution’s
website at the CEM Centre
Predictions
.
How CEM ‘Predictions’ are made…
Subject
SubjectXX
Outcome
140
A / B120
100
C 80
60
40
20
0
04
25
46
67
8
Baseline
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)
Some Subjects are More Equal than Others….
140
A*
A
Grade
120
B
100
>1 grade
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
80
C
60
D
E
40
5
C
6
B
A7
Average GCSE
A*8
Some Subjects are More Equal than Others….
GCSE (MidYIS or Yellis)
A*
A
GCSE Grades
B
1 grade
C
D
E
F
Test Score
Art & Design
Biology
Chemistry
Economics
English
French
Geography
German
History
Ict
Mathematics
Media Studies
Music
Physical Education
Physics
Religious Studies
Science (Double)
Spanish
Baseline Measurement
Problems with Key Stage Baselines
1.
Not all students have KS baselines
No KS3
Foreign Students
Vocational Students
Adult Learners
2.
KS exams do not always represent ‘Start of Course’ ability
Post-16 : Year(s) out or intermediate years
3.
Prior Value-Added
Can you add value at every Key Stage ?
Under achievement leading to under expectation
One teacher’s output = another teacher’s input
4.
Teaching to the test
Does performance represent ability, effort or exam technique ?
5.
Aptitude & fluency vs Achievement & knowledge
6.
Transfer of Data
Although Key Stage baselines can be a very
good indicator of potential attainment, by
themselves they are not sufficient.
Key Stage baselines are confounded by the
effects of prior treatment.
The Computer Adaptive Test
•
Test performed online – results automatically transmitted to CEM.
•
Minimal installation / setup required - if any.
•
Adaptive – difficulty of questions changes in relation to ability of student.
•
Efficient – no time wasted answering questions that are far too easy or
difficult.
•
Wider range of ability
•
Less stressful on students – more enjoyable experience than paper test.
•
Less demanding invigilation.
•
Cheaper !
In 2010/ 2011 over 200,000 students across yrs 7-13 sat this test
Try it yourself at www.intuproject.org/demos
Baseline Feedback
Reports, Graphs
& Predictions
IPRs
(Individual Pupil Record Sheets)
Look for sections that are
inconsistent
Also available based on MidYIS,
Alis, SOSCA & INSIGHT scores
Intake Profiles
Also available based on MidYIS, Yellis, SOSCA and INSIGHT scores
Intake Profiles (Historical)
Predictions – MidYIS example
Similar spreadsheets available from Yellis, SOSCA, INSIGHT
Predictions - Alis example
Chances Graphs
English Language - band D
English Language - band C
50
40
31
30
24
23
20
10
Percent
Percent
50
10
9
2
1
20
20
0
0
29
30
10
0
35
40
1
2
U
G
7
6
U
G
F
E
D
C
B
A
A*
F
E
Grade
0
50
Percent
21
20
7
10
0
0
5
2
0
0
U
G
F
E
D
C
Grade
B
A
A*
50
40
24
C
English Language - band A
40
30
D
Grade
English Language - band B
Percent
1
0
B
A
A*
35
40
26
30
23
20
10
8
0
0
0
1
U
G
F
E
7
0
D
C
Grade
B
A
A*
Predictions
vs
Targets
What is a ‘Prediction’ ?
 NOT a forecast of the grade the student will get
 An indication of the grade (points score) achieved on average by students
of similar ability in the previous year
Targets ?
•
Minimum Targets – Round CEM prediction down ?
•
Realistic Targets – Use nearest CEM grade ?
•
Challenging Targets – 75th percentile ?
Prior Value Added ?
Arbitrary grade fraction ?
75th Percentile Predictions
•
•
•
•
If all students attain 75th percentile predictions, School VA will be at top 25%
Provided by all Projects in addition to 50th percentile
Alis : Excel spreadsheet - ‘Predictions – Spreadsheet (75th Percentile)’
Pre-16 : Adjust button on predictions spreadsheet
Prior Value Added
•
•
•
•
•
Only where prior VA is positive ?
1 year or 3 ?
Reasonable to use raw residual figures as this is an approximate measure and
raw residuals give grade fractions
Alis : Can be calculated using PARIS software
Pre-16 : Adjust button on predictions spreadsheet
Data used to inform, not replace, professional judgement
75th Percentile
Prior Value Added
75th
Percentile
Step 1
Prior
Value-Added
Value-Added Feedback
Reports & Graphs
.
Measuring
Value-Added – Terminology
Subject X
100
Outcome
Exam grade
120
80
-ve VA
+ve VA
60
40
20
4
5
6 SCORE
BASELINE
Baseline
Trend Line/Regression Line
7
8
Raw Residual
Measuring Value-Added – An Example
National Trend
‘Average’ Student
A*
B
Result
C
D
-ve (- 2 grades)
Alf
Bob
Subject A
+ve (+ 2 grades)
E
Chris
Subject B
F
G
U
Low Ability
Average Ability
High Ability
Baseline Score
The position of the national trend line is of
critical importance
Standardisation of Residuals
•
(Raw) Residuals can be used to examine an individual’s performance
•
Standardised Residuals are used to compare performance of groups
•
Standardised Residuals are independent of year or qualification type
•
For a class, subject, department or whole institution the Average
Standardised Residual is the ‘Value-Added Score’
•
Standardised Residual = Residual / Standard Deviation (National Sample)
•
When using Standardised Residuals then for an individual subject
Standard Error

1
N
where N = number of results in the group
(for combinations of subjects consult the relevant project)
•
95% Confidence Limit = 2.0 x Standard Error
•
99% Confidence Limit = 2.6 x Standard Error
•
99.7% Confidence Limit = 3.0 x Standard Error
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
Value Added Feedback…
Statistical Process Control (SPC) Chart
2000
2001
2002
2003
2004 2005
Year
2006
2007 2008
2009
2010
-4.0
-0.7
Spanish
0.1
Religious Studies
-0.5
Physics
Music
Media Studies
-0.7
Maths
-0.4
Information Technology
-0.4
Home Economics
-0.2
History
German
-0.7
Geography
-0.6
French
-0.1
English Literature
0.1
English Language
-1.0
Drama
Design and Technology
-0.2
Chemistry
-0.2
Biology
Art
Average Standardised Residual
Subject Summary
Standardised Residual Graph
4.0
3.0
2.0
2.0
1.0
0.1
0.4
0.0
-0.2
-0.5
-2.0
-2.2
-3.0
Grade Points Equivalent
The Scatter Plot
Look for Patterns…
General Underachievement / over
achievement ?
Baseline Score
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
PARIS Software
PARIS is …..
•
Software to install and use in school
•
An interactive reporting tool
•
Included free with Alis / Yellis / MidYIS
PARIS analyses …..
•
Potential Performance
•
Intermediate Performance
•
Actual Performance
PARIS provides …..
•
Student level reports
•
Subject Level reports
•
Institution Level Reports
Attitudes
There is more to school / college than exams….
•
•
•
•
•
•
•
Student attitudes
Student Welfare & Safety
Non-academic activities
Support
Social and personal development
Parental Survey
Induction Survey
Self Evaluation (Every Child Matters)
Attitudinal MidYIS
INSIGHT
Attitudinal Yellis
Try it yourself at www.intuproject.org/demos
Full ALIS
Other Issues
Points to mull over…
•
•
•
•
•
•
•
•
•
•
•
•
Independence – no agendas
Transparency of Analysis
Self Evaluation
Straightforward and standardised administration
Prompt Feedback
Full working hours phone / email support
Student focus
Replacement for KS3
Innovative online adaptive testing available – student experience
Longitudinal analysis with appropriate error backgrounds
Non-curriculum embedded baselines available
Attitudinal surveys available – Every Child Matters…
Dr Robert Clark
Alis Project Manager
robert.clark@cem.dur.ac.uk
0191 33 44 193