PRACTICAL EXERCISES FOR NEW ALIS USERS
Exercise 1
Situation
You are the subject teacher and are discussing AS target grades with individual students in the
Autumn term of Year 12. You are about to talk to Belinda who achieved an average GCSE score of
5.9.. This gives a statistical prediction=14.75x5.9-59.57 = 27.5 UCAS points using the regression
formula at AS for this subject (Grade D/E at AS). The computer adaptive test seems to confirm this
prediction.
Chances graph showing % of students achieving each AS grade in previous year based on
24175 students
AS GRADE
a) ‘Most candidates with Belinda’s GCSE background score achieved a D in my subject last year so
Belinda’s target grade should be a D’. What are the weaknesses of this statement?
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------b) What other factors should be taken into consideration apart from chances graph data, when
determining a target grade?
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------1
Exercise 2
Situation
Jane has completed her 6th form studies and a review has been received for her by the college after
A level results. Choose one subject at a time and look carefully at what happened in that subject
both from a baseline of average GCSE grades and from a baseline of the computer adaptive test.
Profile Sheet: Jane (from Average GCSE)
Year: 2007
DOB: 01/06/89
(Average GCSE = 6.00 (Band B))
Review: Final_Result
Review
Points
Review
Grade
Average
Points
Average
Grade
(A1) Health & Social Care
30.00
D
39.64
C
-9.64
-0.66
(A2) Religious Studies
80.00
C
89.84
B/C
-9.84
-0.52
(A2) English Literature
80.00
C
87.29
B/C
-7.29
-0.40
B
91.84
B/C
8.16
0.46
Subject
(A2) Drama And Theatre Studies100.00
Residual Std. Residual
The table above shows Review Grades/Point scores achieved in each subject. Predicted
Grades/Points scores are also given. Review information can be compared with the Alis
predictions to help monitor progress in each subject. If a residual is shown, this is the
difference between the Review Points and the Alis 'Average Grade'. The 'Average Grade'
is the average grade achieved by similar students in examinations taken in 2007. The
Average Grade is based solely upon the average GCSE score. Please note that Alis
predictions do not guarantee future results; it is possible to achieve higher or lower grades
than the Predicted Grade.
Individual Chances Graphs from average GCSE score
The higher a student's prior achievement at GCSE the greater the chance of getting a
higher grade at A level.
2
Profile Sheet: Jane from Computer adaptive test
Year: 2007
DOB: 01/06/89
(Online adaptive test = 0.11 (Band C))
Review: Final_Result
Review
Points
Review
Grade
Average
Points
Average
Grade
(A1) Health & Social Care
30.00
D
25.97
D/E
4.03
0.24
(A2) Religious Studies
80.00
C
79.01
C
0.99
0.04
(A2) English Literature
80.00
C
74.30
C/D
5.70
0.26
B
85.16
B/C
14.84
0.70
Subject
(A2) Drama And Theatre Studies100.00
Residual Std. Residual
Individual chances graphs Computer Adaptive test
a) Can you suggest possible reasons for the differences between the predictions (average grade)
made by the two baselines?
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------b) Did Jane reach her potential in all subjects?
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------c) Jane had been set aspirational targets prior to AS and A level by her teachers as below
Health and Social Care
Drama
Religious Studies
English
C
B
B
B
Where these reasonable target grades?
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Why should these grades not be used for accountability of her teachers?
-----------------------------------------------------------------------------------------------------------------------
3
Exercise 3
Scatter Graphs(A2) Subject Report
Review: Final_Result
Subject
(A2) subject
Year: 2008
Baseline: Average GCSE
No of
Students
No of
Results
Average
baseline
n=22
Points
Pred
n=22
Points
Actual
n=22
Avg Raw
Residual
n=22
Avg STD
Residual Completion
n=22
22
22
6.48±0.16
83.86±3.65
85.45±5.91
1.59±4.87
0.07±0.21
100.00
UCAS Points
Predicted Grade = 23.16 * Average GCSE - 66.31
120
120
100
100
80
80
60
60
40
40
20
20
0
0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
Average GCSE Score
Based on 18233 students:
Correlation Coefficient = 0.63
Raw Residual = Actual Grade - Predicted Grade
Standardised Residual = Raw residual/standard deviation
Year: 2008
(A2) Subject Summary
Review: Final_Result
Baseline Score
Name
Baseline: Average GCSE
DOB
Review
Review Average
Av. GCSESc. Grade
Points
Average
Points
Average
Grade
Raw
Residual
St. Res.
BC
29/07/89
5.15
B
100.00
52.96
D/E
47.04
2.03
ET
19/07/89
6.20
A
120.00
77.28
C
42.72
1.84
GP
16/12/89
7.20
A
120.00
100.44
B
19.56
0.84
JH
03/04/90
6.90
B
100.00
93.49
B/C
6.51
0.28
LG
10/09/89
6.00
D
60.00
72.65
C/D
-12.65
-0.55
LW
16/04/89
5.10
D
60.00
51.81
D/E
8.19
0.35
MJ
02/12/89
7.10
B
100.00
98.13
B
1.87
0.08
OJ
21/01/90
7.00
B
100.00
95.81
B
4.19
0.18
OL
24/09/89
6.60
E
40.00
86.55
B/C
-46.55
-2.01
PJ
27/12/89
6.40
D
60.00
81.91
C
-21.91
-0.95
PA
22/04/90
6.50
B
100.00
84.23
C
15.77
0.68
RT
30/12/89
6.50
C
80.00
84.23
C
-4.23
-0.18
RW
29/06/90
5.80
D
60.00
68.02
C/D
-8.02
-0.35
SH
06/10/89
6.40
C
80.00
81.91
C
-1.91
-0.08
SR
25/01/90
6.50
E
40.00
84.23
C
-44.23
-1.91
SR1
02/06/90
6.30
C
80.00
79.60
C
0.40
0.02
SD
11/07/90
7.20
A
120.00
100.44
B
19.56
0.84
TR
08/10/89
7.90
A
120.00
116.65
A
3.35
0.14
WP
02/05/90
5.60
E
40.00
63.39
D
-23.39
-1.01
WS
31/10/89
7.20
B
100.00
100.44
B
-0.44
-0.02
YE
13/11/89
5.60
C
80.00
63.39
D
16.61
0.72
YJ
06/06/90
7.50
A
120.00
107.39
A/B
12.61
0.54
4
a) Concentrate on the students BC, LW, OL, SD, WS. Satisfy yourself that you can
find each of these students on the scatter graph. TIP…. look at average GCSE
scores and Review points (equivalent to actual grade achieved)
b) Now look at the raw residuals for each of these students.
Can you mark between arrows these residuals on the graph ?
c) Which ability range has the best value added in this subject?
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------d) Why are standardised residuals necessary?
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Exercise 4 Departmental analysis
(based on average GCSE score baseline)
SUBJECT A A LEVEL
1.0
1.0
0.5
0.5
0.0
0.0
Residual
Average Standardised Residual
Final_Result
-0.5
-1.0
-0.5
2003
2004
2005
2006
2007
2008
2009
2010
2011
-1.0
Exam Year
a) What happened in 2006 to subject A? Action was taken by the department. What
happened?
5
SUBJECT B A LEVEL
2.0
2.0
1.0
1.0
0.0
0.0
Residual
Average Standardised Residual
Final_Result
-1.0
-2.0
-1.0
2003
2004
2005
2006
2007
2008
2009
2010
2011
-2.0
Exam Year
b) There may be lots of reasons for changes in performance but here one factor is
known by the school. The subject teacher for subject B goes on long sick leave
for one of the autumn terms. When do you think this happened? To help you the
AS chart is shown below for this same subject. It may or may not be relevant.
SUBJECT B
AS LEVEL
2.0
2.0
1.0
1.0
0.0
0.0
Residual
Average Standardised Residual
Final_Result
-1.0
-2.0
-1.0
2003
2004
2005
2006
2007
Exam Year
6
2008
2009
2010
2011
-2.0
Exercise 5
Choice of Courses
Strengths and weaknesses of this A/AS level student?
a) Which AS/A level subjects might be avoided?
---------------------------------------------------------------------------------------------------------------b) This student chose English, Film Studies, Music Technology and Psychology. Is this a good
choice? Do you foresee any problems?
--------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------
7