Performance Monitoring with feedback: a holistic approach
PERFORMANCE MONITORING
USING
VALUE ADDED DATA
(Post – 16)
Keith Murdoch
Performance Monitoring with feedback: a holistic approach
Overview of Presentation
• Context: The College, the locality and
some political drivers
• An overview of Woodhouse College’s
Performance Monitoring approach
• Performance Monitoring
– College
– Departments
– Students
• Concluding musings
Performance Monitoring with feedback: a holistic approach
Woodhouse College:
1,145 16-19 full time students
• 99% A – level provision
• 58% Female
• 54% BME
• Enrolments from 140+ schools
Performance Monitoring with feedback: a holistic approach
Value Added & the
Common Inspection Framework
Outcomes for Learners
•
Learners’ attainment and progress
Quality of Provision
•
Effectiveness of teaching, training and assessment in
supporting learning and development
•
Effectiveness of the care, guidance and support learners
receive
Leadership and Management
•
Raising expectations and promoting ambition
•
Actively promoting equality and diversity to narrow the
achievement gap
•
Effectiveness of self assessment
Performance Monitoring with feedback: a holistic approach
PERFORMANCE MONITORING
Key Elements: Using value added data
• Monitoring Student Progress / Student
Reviews
• Lesson Observation
• Internal Inspections
• Every Child Matters
• Diversity and Equality
• Self Assessment: Department / College
Performance Monitoring with feedback: a holistic approach
SELF ASSESSMENT
• How well do you know your Department / College?
• How well do you know your data?
• Can you accurately identify your weaknesses?
• Do you have the capacity to make improvements?
• Can you provide evidence to demonstrate improvements?
Performance Monitoring with feedback: a holistic approach
Performance Monitoring
Using ALIS to monitor achievement &
attitudes with feedback to 3 levels
How are we doing and how do we know?
with
ALIS helps us measure:
Students
Progress on Course
Departments
Achievement
Whole College
Achievement & Attitudes
What are we doing about the poor bits?
e.g. Target Setting, Action Plans, Operational Plans, Strategic Plans
Quality Monitoring with feedback: a holistic approach
Aug &
Sept
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Strategic & Operational Planning
Dept. SAR
Achievement
+ Action Plans
Whole
College
SAR
Dept SAR
Phase 1 +
Action Plans
Curriculum Quality Monitoring & Internal Inspections
Data
Capture
GCSE
A/AS
Achievement
Feedback
MAG
Yr 12
Reviews
Yr 13Data
Capture:
Attitudes
Input Feedback
Yr 12 & 13
Reviews
On-course student assessment and monitoring, subject by subject
Yr 12
Reviews
Jul
Performance Monitoring with feedback: a holistic approach
Monitoring the overall
performance of the COLLEGE
Summative monitoring by Senior Leaders
and Governors
Performance Monitoring with feedback: a holistic approach
Performance Monitoring with feedback: a holistic approach
Performance Monitoring with feedback: a holistic approach
Performance Monitoring with feedback: a holistic approach
Performance Monitoring with feedback: a holistic approach
A Level results 2010
No of students on a given MAG achieving A*-U grades
A*
A
B
C
A/B
56
112
40
7
B
67
330
302
139
B/C
3
20
28
26
C
8
47
87
96
C/D
2
4
D
1
4
6
D/E
1
1
E
2
5
1
Grand Total
134
512
469
280
D
2
36
10
60
3
10
1
2
124
% of students on a given MAG achieving A*-U grades
A*
A
B
C
A/B
26%
51%
18%
3%
B
8%
37%
34%
16%
B/C
3%
22%
31%
29%
C
2%
14%
26%
28%
C/D
17%
33%
D
3%
14%
21%
D/E
33%
33%
E
17%
42%
8%
D
1%
4%
11%
18%
25%
34%
33%
17%
E
U
15
2
22
2
6
2
1
19
1
2
2
49
25
E
U
2%
2%
6%
17%
21%
1%
6%
8%
7%
17%
Grand Total
218
891
90
339
12
29
3
12
1594
Grand Total
100%
100%
100%
100%
100%
100%
100%
100%
Performance Monitoring with feedback: a holistic approach
Report to Governors
2010
Subject
No of Retention
%
Student
s
40
100
96
97
41
93
117
98
61
94
124
97
60
100
106
100
36
100
2008-2010
Raw
Resid.
Std.
Resid.
08-10
A-B
%
A-E
%
A-B
%
A-E
%
63
45
64
42
73
36
56
46
69
45
57
51
45
48
65
47
89
42
100
97
100
95
100
98
100
98
100
98
96
95
95
96
100
97
100
99
62
100
2.2
0.1
-0.1
65
100
10.0
0.4
0.5
65
100
4.5
0.2
0.2
64
100
1.7
0.1
0.1
70
100
6.1
0.3
0.3
62
98
1.1
0.0
0.2
50
96
-3.6
-0.1
0.0
76
99
5
0.2
0.4
78
100
11.1
0.6
0.5
Value-Added
Performance Monitoring with feedback: a holistic approach
Monitoring the Performance
of DEPARTMENTS


SELF ASSESSMENT:
 summative monitoring by departments
of their own performance using fair
comparisons of achievement levels.
Analysing student achievement by subject
Performance Monitoring with feedback: a holistic approach
Key Performance Indicators
Success rates
% Grade A* - B at A2 and A - B at AS
Pass rate
Value added (A2, AS): exam year / 3 year
Performance compared to MAG
Retention rate
Learner satisfaction
Attendance
Enrolment number
Gender and Ethnicity monitoring
Lesson Observation grade profile
Internal Inspection grade
Average class size
Performance Monitoring with feedback: a holistic approach
Year Institns in
cohort
GCSE
IT DA
Av. A
Residual
2004
6.4
50.2
7.4
0.36
10%
2005
6.2
48
5.7
-0.4
10%
2006
6.4
49
93.1
0.04
2007
6.5
48
95
-0.04
2008
6.5
48
101
0.31
10%
2009
6.3
94
0.1
25%
2010
6.5
102
0.29
10%
2011
Year
Error
Attitude
to Subject
Attitude to
College
Advising
College
(+/ -)0.2
0.1
(+/ -)0.18
0.09
(+/ -)0.22
0.11
(+/ -)0.2
0.1
(+/ -)0.2
0.1
(+/ -)0.18
0.09
(+/ -)0.2
0.1
(+/ -)0.
3.6
3.7
70%
3.6
3.7
82%
3.4
3.6
3.3
3.9
3.5
3.7
3.3
3.7
3.5
3.7
Analysis of Residuals by GCSE scores
8 - 6.2
8 - 6.9
6.8 - 6.2
6.1 - 5.5
2002
2003
2004
3yr.
2005
3 yr.
2006
0.2
0.3
0.55
0.35
-0.9
-0.02
1.4
56
63
48
0.29
21
0.76
27
54
-0.2
19
-1.3
35
59
1.4
27
1.4
32
2007
2yr.
2008
3yr.
2009
3yr.
2010
3yr.
2011
3yr.
-1
0.20
5.6
2.00
1.5
2.03
4.2
3.77
76
-1.3
0.05
4.6
1.57
2.2
1.83
1.8
2.87
36
-0.7
0.35
6.2
2.30
1.0
2.17
6
4.40
40
80
65
78
32
27
34
48
38
44
5.4 - 4.7
0.3
-0.1
0.9
0.37
-0.7
0.03
-2.4
26
24
32
2.7
0.15
-2
-0.57
2.4
1.03
7.1
2.50
22
38
20
11
42
23
ExtraMural
LSE
Enrol
"A"
Complete Retention
"A"
"A"
1
1
95
102
110
90
94
95
95%
92%
86%
1
115
106
92%
92
90
98%
1
109
104
95%
1
105
104
99%
1
120
117
98%
1
108
105
97%
4.6 - 4
0.5
1.6
0.85
0.98
-0.6
0.62
2.6
8
6
14
-7.8
-2.60
12.6
2.47
-4.4
0.13
7.9
5.37
5
13
11
12
7
3
0
-0.9
-0.6
-0.50
-0.6
-0.70
0
13.7
6.85
8
7.23
31.1
17.60
23.7
20.93
Performance Monitoring with feedback: a holistic approach
Ye ar
GCSE
2010
6.5
T DA
Av. A
Re sidual
102
0.29
E rror
10% (+/-)0.2
0.1
Attitude
Attitude to
to Subje ct Colle ge
3.5
3.7
Advising
Colle ge
Performance Monitoring with feedback: a holistic approach
SELF-ASSSESSMENT: DEPARTMENTS
ALIS DATA ANALYSIS
1.
Summary of Raw Results
•
2.
Analysis by GCSE Score
•
•
3.
•
An explanation of data points which lie outside the control lines
A comment on the moving average
Analysis by Teaching Group
•
7.
Can we identify common features within the high and low achieving groups?
Do the extremes distort the overall picture of the subject performance?
Analysis of Variance
•
•
6.
In terms of actual scores and standardised residuals, how have students from different ethnic minority/gender
groups performed?
Are the differences significant?
Analysis of Extreme Cases
•
•
5.
In terms of residuals how have students in different bands been performing?
Does a preponderance of students in any one band help explain the overall residual?
Analysis by Ethnic Minority and Gender
•
4.
Comparison of raw subject A-level results with the national percentage for the subject
Is there any apparent correlation between set and residual or, over the 3 year period, between member(s) of staff
teaching a set and the residuals achieved, after sets have been analysed for the range of ability?
Students’ Attitudes and learning and Teaching Processes
•
•
Analysis in trends in students’ attitudes to subject
Issues raised by analysis of perceived learning activities.
Performance Monitoring with feedback: a holistic approach
1. Analysis by GCSE Score
–
–
–
In terms of residuals, how have students in
different bands been performing?
Does a preponderance of students in any one
band explain the overall residual?
How does your analysis impact on strategies
for teaching and learning?
Analysis of Residuals by GCSE scores
8 - 6.2
8 - 6.9
2010 4.2 78
3yr. 3.77
1.8
2.87
6.8 - 6.2
34
6.0
4.40
6.1 - 5.5
44
7.1
2.50
5.4 - 4.7
23
7.9
5.37
4.6 - 4
3
24
20.93
1
Performance Monitoring with feedback: a holistic approach
2. Analysis by Gender & Ethnic
Minority
–
–
–
In terms of standardised residuals, how have
students from different e-m/gender groups
performed?
Are the differences significant?
What impact will these differences have on your
teaching and learning strategies?
Ye ar
Ge nde r
No
AvGCSE
AvS
AvR
StdR
Diffe re nce
2010
F
M
54
45
6.8
6.7
95.6
100.4
4.8
12.8
0.2
0.6
0.40
08-10
F
M
178
117
0.35
0.49
0.14
Performance Monitoring with feedback: a holistic approach
Performance Monitoring with feedback: a holistic approach
3. Analysis of Extreme Cases
– An extreme is a student with a raw
residual of + or – 30 (15 for an AS)
– Can we identify common features within
the low and high achieving groups?
– Do the extremes distort the overall
picture of the subject performance?
Performance Monitoring with feedback: a holistic approach
Student Level Residuals
Average
GCSE
Score
Actual
Grade
Predicted
Grade
Raw
Residual
Jaymin
5.7
60
50
10
Davy-Martin
Lee
6.7
120
85
35
Lai
King-Ho
6.1
0
62
-62
Mansigani
Ilaha
6.0
120
60
60
Taylor
Hannah
6.3
100
75
25
Surname
Forename
Chandaria
Performance Monitoring with feedback: a holistic approach
4. Analysis of Variance
– An explanation of data points which
lie between control lines
– A detailed explanation of data points
which lie outside 3SD control line
– A comment on the moving average
Performance Monitoring with feedback: a holistic approach
Performance Monitoring with feedback: a holistic approach
KEY ELEMENT of
SELF ASSESSMENT ANALYSIS
 Identifying strengths and weaknesses
 understanding your weaknesses and
identifying actions for improvement
 impact on strategies for teaching and
learning and supporting students
Performance Monitoring with feedback: a holistic approach
Using ALIS data to identify and
‘unpack’ weaknesses
e.g.
-0.7 Female
-0.1 Male
-0.28 Standardised Residual
AS Physics
-0.64 without Maths
-0.19 with Maths
•Work with Maths Department on creating resources to
support student not taking AS Maths
• Introduce problem-solving consolidation sessions
• Further investigation of girls underachievement – Institute
of Physics, Standards Unit, focus groups etc
Performance Monitoring with feedback: a holistic approach
e.g. … contd.
-0.23 Standardised Residual
3 distinct groups
1. Additionality: 0.1
AS Critical Thinking
2. Full programme : -0.8
3. Applied AS: -0.5
• Restructure external assessment of course – January module
• Increase hours for Applied AS students, formalise
requirements, ‘integrate’ into main programme
• Review appropriateness
Performance Monitoring with feedback: a holistic approach
Monitoring the Performance
of STUDENTS
to raise aspirations and
achievement
Performance Monitoring with feedback: a holistic approach
You know you’ve gone to Woodhouse
when.....
MAG/CAG. They do mean something.
Performance Monitoring with feedback: a holistic approach
PROBLEM
Motivating, and monitoring student progress
is a FORMATIVE process
ALIS is
 RETROSPECTIVE
 SUMMATIVE
 STATISTICAL
HOW CAN WE ‘SQUARE THE CIRCLE?’
Possible because correlations are high and the variation
in the association between AVGCSE and the statistically
‘PREDICTED’ A-Level grade, from one year to the
next, in any given subject, tends to be very SMALL
Performance Monitoring with feedback: a holistic approach
KEY PROCESSES

Departmental Monitoring and Assessment
Practices
which feed
into

Student Reviews


Scheduled monitoring of a student’s progress across their
programme
Mutually supporting academic and pastoral functions
using COMMON DATA
MINIMUM ACCEPTABLE GRADES
Performance Monitoring with feedback: a holistic approach
BENCHMARKING and
TARGET SETTING: Defining Grades
1. MAG: Minimum Acceptable Grade
The statistically predicted grade for each subject (Scale
A/B – E [E] based on the ALIS trend line) which will not
change during the year. Provides an initial benchmark
(with associated health warnings) against which a student’s
progress can be judged.
2. CAG: Current Achievement Grade
The grade a student is currently working at.
Performance Monitoring with feedback: a holistic approach
Defining Grades (contd.)
3. TAG: Target Achievement Grade
The grade (above the MAG) a student is considered
capable of working at and should aim to achieve.
Student Review discussions (where appropriate) would
be focused on negotiating strategies to enable students to
attain this grade.
4. PG: Predicted Grade
The grade that is written on a student’s UCAS form and
subsequently changed or confirmed as part of the return
to Examination Boards
Performance Monitoring with feedback: a holistic approach
CALCULATING THE MINIMUM
ACCEPTABLE GRADE
The student MAG for each subject is based on the
previous year’s ALIS ‘trend-line’ for that subject
Principle = ‘good enough for purpose not statistically flawless’
Example:
Lisa Fry has GCSE grades 2A*, 3A, 4B and 1C
Total Points = 66
(i.e. 2x8 + 3x7 + 4x6 + 1x5)
AVGCSE = 66 / 10
= 6.6
MAP for subject (15.78 x 6.6) – 62.65 = 41.5
MAG = C (at AS)
Performance Monitoring with feedback: a holistic approach
Technical problem!!!
calculation creates number and decimal points
which need to be translated into UCAS grades
College Conversion Table
A-Level
AS Level
MAG
‘Predicted’ Point Score
‘Predicted’ Point Score
A/B
109 – 140
53 – 60
B
>91 - <109
>44 - <53
B/C
89 – 91
43 – 44
C
>71 - <89
>34 - <43
C/D
69 – 71
33 – 34
D
>51 - <69
>24 - <33
D/E
49 – 51
23 – 24
E
<49
<23
• No student can be targeted to attain A* or A grade
• No student can be targeted to ‘fail’, i.e. below grade E
Performance Monitoring with feedback: a holistic approach
Progress in Individual Subjects

College policy on the monitoring of student
progress and achievement
Framework for
 Department Policies
• setting and marking students’ work
• monitoring students’ progress (target setting)
• use of subject tutorial period
 Supporting Teaching and Learning
Performance Monitoring with feedback: a holistic approach
Student Name
1
2
3
4
5
6
MAG: B
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21
100
90
80
70
60
50
40
30
20
10
0
74
72
71
67
63
55
71 72
63
58
65
56
59
48
67 65
55
Performance Monitoring with feedback: a holistic approach
Nayan
90
80
70
60
AS
50
MAG
40
30
20
10
0
1
2
3
4
5
6
7
8
9
10
11
Performance Monitoring with feedback: a holistic approach
Student Review Cycle
Dept
Tutor
Review
Parents
Set Lists
Programme
Interview
2nd Yr Review
MAG calculated
New Parents
2nd Yr meeting
Oct
1st Yr Subject
Assessment
Programme
Interview
1st Yr Review &
Case Conference
Grade Report
Dec
2nd Yr Subject
Assessment
Jan
1st Yr Subject
Assessment
Sept
Grade Report
1st & 2nd Yr
Programme
Interviews
1st & 2nd Yr
Reviews & Case
Conferences
2nd Yr meeting
1st Yr G-Report
& meeting
Feb
Mar
2nd Yr Subject
Assessment
2nd Yr Reports
Jun
1st Yr Subject
Assessment
1st Yr Reports
(July)
Performance Monitoring with feedback: a holistic approach
Start of the Student Review Process
September:
AVGCSE score used to calculate MAG (AS) subject by
subject : included on set lists and programme record card
October:
Autumn Review – with Personal Tutors
Scale:
1 = performing outstandingly
2 = satisfactory
3a/b = study skills &/or conceptual problems
4 = ‘alarm’ signal
5 = very recently joined subject
+ Effort Grade 1 - 4
AS MAG is provided as benchmark
Case Conference
Performance Monitoring with feedback: a holistic approach
Student Review Data
Categories
MAG
Module Result
CAG
Effort
Quality of Work
Meeting Deadlines
Involvement
Attendance
Scales
A/B - E
%
A-F
1-4
1-4
1-4
1-4
% present
Source
MIS
Teacher
Teacher
Teacher
Teacher
Teacher
Teacher
MIS
Performance Monitoring with feedback: a holistic approach
Tutor Programme Review Data
MAG
RES
CAG
EFF
QoW
MD
INV
ATT
310765 Student Name (98%)
Eng-s-ld
AS
WAL
B
57
B
2
2
1
2
97
Hist-s-le
AS
THO
B
60
B
2
2
1
2
100
Law-s-lc
AS
SFR
B
49
B/C
2
2
2
2
96
Psyc-s-la
AS
STE
B
54
B/C
1
2
1
2
100
329368 Student Name (88%)
Geog-s-lb
AS
MCA
C
37
E/F
4
4
4
2
90
Germ-s-ld
AS
PAR
D
28
F
4
4
4
4
88
Pol-s-le
AS
GRI
C
30
F/E
4
4
3
3
83
PSM-s-la
AS
ALN
E
30
E
3
3
3
3
91
313492 Student Name (95%)
Performance Monitoring with feedback: a holistic approach
Some musings……
• measurement gives messages
• evolve your own
• invest in the time
• transparency as a tool not a threat
• know the health warnings
• embed
• never take understanding for granted
• trust the tribe