Identifying students ‘at risk’: early
warning at macro and micro levels
Barry McCluckie & Lorna Love
Introduction
Early warning for Science advisers: Using attendance data
at the macro level

The project & the process used

ROC Analysis- trigger level

Outcomes

Other and future work + stats
Process
Early warning for Science advisers: using attendance data at the macro level

Can we use 1st semester science attendance data to predict which L1 science
students will withdraw or have a poor 1st year? (poor 1 year: GPA <=9 OR <120 credits)
st

Data types gathered: lab, tutorial, workshop, field trip & class test attendance

Data for 19 courses collected by 14 individuals, using Excel/Access.

Data collected in week 6 (for teaching weeks 1 to 5).

Reformatted and made suitable for input into an Access database.

Data is then ‘cleaned’ by comparing expected to actual attendance.

Queries run in database to select ‘at risk’ students in week 7,data made
available to adviser at the end of that week via the Universities ‘Single Sign On’.
ROC curve analysis 1
Receiver Operating Characteristic curve

Real world decisions are made in the presence of uncertainty!

What is a ROC curve?: A graphical representation of the trade off
between the True Positive Rate (TPR) & the False Positive Rate
(FPR) as the discrimination threshold is varied.

Discrimination threshold: The tipping point between a prediction
being classified as ‘Positive’ or ‘Negative’.

As the discrimination threshold is varied the benefit and costs
values will change. This is what we want to know about!
ROC curve analysis 2
Distribution for actually negative
cases (Stayed On/ good performance)
Contingency table:
One possible
discrimination threshold
Distribution for actually positive
cases (Withdraw/poor performance)
ROC curve analysis 3
ROC curve analysis 5
ROC Curve
1
0.9
0.8
-(95%)
0.7
-(90%)
Sensitivity
0.6
-(85%)
0.5
-(80%)
-(75%)
0.4
-(70%)
-(65%)
0.3
-(60%)
-(55%)
-(50%)
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
1-Specificity
0.6
0.7
0.8
0.9
1
ROC curve analysis 4
 Discrimination threshold selected was 75% attendance level in
order to maintain an adviser to student ratio equal to 2.
 1229 first year students (07-08) with 98 (8%) deemed as ‘at risk’.
 Information available to advisers via Universities ‘Single Sign On’.
Example
Matric
0701589
Title FirstName Surname
Email
Telephone Course L2 L3 L4 L5 T2 T3 T4 CT5
MR
JOHN
KING 0701589c@student.gla.ac.uk 01416384321 BIOLOGY1A 1 1 0 0
CHEMISTRY1
0 0
0
GEOGRAPHY1 0
1 1
1
Outcomes
Result for 2007-2008 students

20 students who withdrew correctly identified.
(26% of all withdrawals summer
‘08)

78 ‘at risk’ students continued until the end of the academic
year. (80% of those at risk)

Of these 49 had a negative outcome with 29 having a positive
outcome. (Summer ’08)

21 of the positive students spoke to advisers, while 25 of the
negative students spoke to adviser.

In all 69 ‘at risk’ students withdrew or had a negative
outcome. (70% of those identified as ‘at risk’, summer ‘08)
nd
Other & Future work

All 1st semester attendance data was made available to Advisers of
Studies for their students via the ‘Single Sign On’.

Early Warning System expanded to include the Arts Faculty and
most of LBSS Faculty. [(08-09)1 year courses only]
St

Improve the trigger- regression/survival analysis using data from
Bi-Query and compare ROC curves.

Improve the efficiency of attendance gathering tools. More use of
barcode scanning and databases?; more centralised data
gathering as university moves to new student records system?
STATS
Year
Courses
Advisers Students

07-08
19
63
1229

08-09
85
140
2707

09-10
100+?
200+?
3800+? (80% of L1)
Early warning at a micro level
Early warning: using diagnostic data at the
micro level

Project background
- Study Support Co-ordinator for F.I.M.S. Oct. ’08
- assist struggling students in Mathematics, Computing Science, Statistics

Evolution of fine-grained ‘at risk’ student data

Use of data collected
Dept. of Computing Science
Context

Computing Science
- two Level 1 modules - long, thin courses CS1P and CS1Q
- Level 1 approx.170 students – wide variety of prior experience
- problem with students succeeding in CS1P

Voluntary Peer Assisted Learning (PAL) for CS1P
- early, diagnostic class test
- more than a ‘pass/fail’, categories of strugglers
- ‘fundamentals’
‘hand execution’
‘problem solving’
- PAL options : structured peer revision, supported problem solving sessions
Computing Science Tutors
Diagnostic class test data

Purpose of data
- PAL offered to all CS1P students, targeted specifically at struggling students
- purpose: to inform the PAL co-ordinator and the Level 1 facilitators

‘At risk’ students list
- share this information informally with the tutors
- surprising result ...
Mismatch of concern
Surprising result

Conflict: My ‘at risk’ list ≠ the tutor’s ‘at risk’ list
- some students I was concerned about were thought to be ‘doing ok’
- some students that the tutor was concerned about passed the class test
- problem is that ‘35% in class test’ does not tell the whole picture.
- tutor knows more about the student than their mark
 Examples of categories of progress from the tutor’s perspective
- “not seen student enough to be able to comment”
- “trying hard, making progress, will get there with continued effort”
- “done a lot of programming before but not willing to engage with new concepts”
- “turns up late and unprepared”
The new ‘at risk’ data
Rich collection of data
Recorded tutor’s comments and also added in attendance data.
Now the student data has information from 3 sources
1:
defined via diagnostic class test(s)
2:
defined by tutor (and PAL facilitator)
3:
defined via attendance data
What do we do with this new ‘at risk’ list?
- P.A.L. Co-Ordinator and Facilitators
- Head of Year
- Course director or Lecturer(s)
- Adviser of studies (Science or non-Science)
- Tutor
- Study Support Co-ordinator
Progress ‘chats’ with students
Trial using Study Support Co-Ordinator to intervene

Contacting advisers
- sent email to adviser of each student of concern (roughly 60 from170)
- emailed details of concerns and the additional tailored support offered
- Websurf contains no indication of CS progress until end of year

Texting and meeting students
- asked them to come for an brief, informal chat
- spoke with 15 students, mostly a positive experience
- aim was to direct student to tailored support but found that chat alone useful

New approach
- Study Support Co-ordinator has time
- Study Support Co-ordinator is not
- involved in presentation of lectures or assessment
- Study Support Co-Ordinator is
- approachable, a subject specialist, and a (relatively!) recent graduate of the
Science Faculty
Evaluation
Evaluation
Work in Progress...
Is micro-level early warning worth it?
1. time and effort involved (estimate, average 1 day per week on top of significant ‘set-up’ costs)
2. correctly identify the students that withdrew from the course or failed?
3. identify students not flagged up by the macro warning system?
4. offer correct progression advice based on the diagnosis?
5. information sent to the advisers improve the uptake of PAL?
6. chats with the Study Support Co-Ordinator improve the uptake of PAL, study skills, motivation?
Ultimately, did this increase retention within Level 1 Computing Science?
Next Year
Plans for Next Year

Other modules
- Level 1 and Level 2 Computing Science
- Level 1 and Level 2 Mathematics

Formalise process of data collection, sharing and contacting students
- last year’s was ad-hoc and informal
- timeline with triggers for collecting data from tests, tutors, facilitators and for
contacting students and advisers
- devise useful, informative discrete ‘categories of concern’ from tutors
Other departments
Recommendations

Time and effort
- creating diagnostic, formative assessment is easier for some subjects than others
- once created – easy to manage resulting data
- speaking to tutors is time consuming, categorisation and automation essential
- estimate spend 1 day per week monitoring and tracking approx 60 of 170 students

Use of data
- who deals with the ‘at risk’ students ?
- exit from course or ‘back on track’ ?
- to ‘get back on track’ more than a chat is required – support mechanism
- no point in identifying students if there is no plan of action