Enhancing ODL Students' Success through Risk
Profiling and Prediction: The Case of Unisa
HANLIE LIEBENBERG
Senior Specialist: Institutional Research, Unisa
PROF GEORGE SUBOTZKY
Executive Director: Information & Strategic Analysis, Unisa
DION VAN ZYL
Manager: Information Services, Unisa
Presented at:
NADEOSA Conference, Johannesburg, 30 August 2011
Acknowledgements
• The efforts of numerous DISA staff members in gathering
and preparing information is acknowledged
• In particular, the help and support of Robert Lightbody,
admin Asst/caregiver to Prof Subotzky, was invaluable in
preparing this presentation
Overview
1. Background, Key Challenges & Research
Problem
2. Unisa Student Success Framework
3. Unisa Conceptual & Predictive Models of
Success
4. Unisa Tracking System
5. Data Analysis Challenges
6. Segmented Profiling: Categorising Student
Risk
Background
• Whilst various theoretical models
contribute towards understanding the
various dimensions impacting on student
success, utilising actionable intelligence
to inform effective interventions remains
daunting
• This challenge is particularly formidable
at Unisa, which now has +340 000 mainly
non-traditional, older, part-time,
underprepared students
• They face challenging socio-economic
circumstances, particular work-related
and domestic responsibilities, which
impede on student success
Research Problem
• To address this, Unisa recently developed a
student support & success framework,
comprising 4 elements:
- Conceptual model
- Predictive model
- Student support interventions
- Evaluating impact
• Critical challenge: moving from conceptual
model of student success to profiling,
tracking, assessing and predicting risks to
success
Key Challenges
• Key concerns and critical questions that arose in developing
an integrated Student Support and Success Framework in the
Unisa context
• More particularly, the process of moving from the conceptual
modelling of student success – a necessary first step – to the
detailed student profiling, tracking and predictive modelling
of risks upon which effective interventions are based
• Key challenge: translating and operationalising relevant
constructs of the high-level conceptual model to create a
comprehensive student profile, tracking system and predictive
model which retains sufficient complexity but remains
practicable
The Challenge of Translating Theory into Practice
A theory that could fully explain every aspect of the attrition
process would contain so many constructs that it would
become unwieldy if not unmanageable. Such situations call for
the use of theoretical models which are simplified versions of
reality that strip away the minute details to concentrate on
factors that are assumed or deduced to be important. ... Models
can be judged by their usefulness. A model of the attrition
process should contain sufficient constructs to explain what is
undoubtedly a complex process and yet sufficiently simple to be
understandable and useable. It should be able to explain
collected descriptive data, and it should provide a framework
against which predictions can be hazarded and judgements
made about potential interventions.
Kember (1989: 279-280)
Operationalising the Conceptual Model
This implies:
• Identifying and defining all academic and
non-academic variables needed for construct
measurement, segmentation, profiling and
predictive modelling;
• Utilising suitable data gathering methods that
yield consistent, complete and unbiased data;
and
• Applying appropriate advanced statistical
analysis that can identify complex underlying
multivariate dynamic relationships between
variables and constructs
Elements of the
Unisa Student Success Framework
Extensive
literature review
& conceptual
modeling of all
factors affecting
success in Unisa
context
Comprehensive
profiling,
tracking and
intelligence
gathering
culminating in
predictive model
of student
risks/success
Incrementally
implementing
an institutionwide Student
Support
Framework
Evaluating
impact over time
SHAPING CONDITIONS: (predictable as well as uncertain)
• Social structure, macro & meso shifts: globalisation, political economy, policy; National/local culture & climate
• Personal /biographical micro shifts
STUDENT
IDENTITY & ATTRIBUTES:
• Situated agent: SES, demographics
• Capital: cultural, intellectual, emotional,
attitudinal
• Habitus: perceptions, dispositions,
discourse, expectations
THE STUDENT WALK:
Multiple, mutually constitutive
interactions between student, institution
& networks
• Managing complexity/ uncertainty/
unpredictability/risks/opportunities
• Institutional requirements known &
mastered by student
• Student known by institution through
tracking, profiling & prediction
INSTITUTIONAL
IDENTITY & ATTRIBUTES:
• Situated organisation: history, location,
strategic identity, culture, demographics
• Capital: cultural, intellectual, attitudinal
• Habitus: perceptions, dispositions,
discourse, expectations
TRANSFORMED STUDENT IDENTITY & ATTRIBUTES:
Processes:
• Informed responsibility & ‘choice’
• Ontological/epistemological dev.
• Managing risks/opportunities/
uncertainty: Integration, adaptation,
socialisation & negotiation
F
I
T
F
I
T
Domains:
• Intrapersonal
• Interpersonal
Modalities:
• Attribution
• Locus of
control
• Selfefficacy
F
I
T
F
I
T
FIT
F
I
T
Retention/Progression/Positive experience
Choice,
Admission
Learning
activities
Course
success
Graduation
Employment/
citizenship
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
TRANSFORMED INSTITUTIONAL IDENTITY & ATTRIBUTES:
Processes:
• Informed responsibility & choice
• Managing risks/opportunities:
Transformation, change
management, org. learning,
integration & adaptation
Domains:
• Academic
• Operational
• Social
Modalities:
• Attribution
• Locus of
control
• Selfefficacy
SHAPING CONDITIONS: (predictable as well as he uncertain)
• Social structure, macro & meso shifts: globalisation, internationalisation, political economy, technology, social demand
• HE/ODL trends, policy
• Institutional biography & shifts; Strategy, business model & architecture, culture & climate, politics & power relations
Success
FIT
Key Constructs of the Predictive Model
a) Students' inter-personal attributes:
 Demographics and past socio-economic status, including educational and family
background and exposure to role models;
 Current socio-economic status and life circumstances, measured by the constructs of
time and opportunity to study and stability in life circumstances and support for study;
b) Students' intra-personal attributes:
 Academic readiness and ability;
 Metacognitive skills;
 Psychological attributes and outcomes of other processes;
c) Institutional services, practices & culture:
 The quality of academic and administrative services;
 Institutional culture and practices;
d) Integration, engagement and transformation:
 Students' effective management of their life circumstances and mitigation of risks as well
as meeting learning expectations and utilising opportunities;
 The institution's effective management of academic and support processes and
mitigation of risks.
Student as Situated Agent
•
•
-
Inter-Personal
Background:
Current SES & Life
Demographics
Circumstances:
Past SES
- Time &
Educ. Background
Opportunity
Family Background
- Stability & Support
Role Models
Intra-Personal
Academic
Readiness &
Ability
Success
Student’s Effective Management of:
• Life Circumstances & Risks
• Learning Expectations & Opportunities
Fit: Academic
Choices &
Activities
Fit with
Institutional
Culture &
Practices
Institution’s Effective Management of:
• Academic & Support Processes/Risks
• Student Profile/Risk & Communication
Institutional Services, Practices & Culture
Quality of
Academic
Services
Quality of
Admin
Services
Social:
Institutional
Culture &
Practices
Psychological
Attributes &
Outcomes
Student Walk
Integration, Engagement & Transformation
Utilisation of
Admin/
Support
Services
MetaCognitive
Skills
Formative
Assessment
Course
Success
Graduation
• Satisfaction
• Graduateness
Institution as Situated Agent
Senate
STLSC
School/College TLSC
Student Success Forum
Academic Department
Professional Structures
Lecturer/Supervisor/Online
Mentor/Tutors/Regions
TSDL
DCCAD
Library
DISA
Student Support Coordinator
USGS
Dean of Stud.
Academic
Admin Structures
DSAR
DSAA
SMPPD
Affective
Admin
TRACKING SYSTEM
Profiling, Tracking & Predicting Risk at the level of Student/Module/Qualification/Institution
Student Information
•
•
•
•
•
•
•
•
Applications/Registration
HEMIS
Assessment Performance/Scores
Academic Readiness Self-Assessment
Student Profile Survey
Student Satisfaction Survey
Exit/Tracer Surveys
ICMAs
Operational Processes
•
•
•
•
•
Application/Registration
Study Material
Assessment Management
Finance
HR
Communication/Engagement
•
•
•
•
•
•
•
•
College/School/Department/Regions
E-Tutor/F2F Tutor/Online Mentor
Counsellor
Call Centre
Admin Department
Tutorial Attendance
myUnisa/Library
Student Course Evaluation
Student Profile Design Challenges
• Considerations were given specifically to
question response formats and scaling
• Initial draft survey questionnaire
consisting of over 100 questions based
on key constructs identified in conceptual
and predictive models
• Throughout design process, imperative
to ensure alignment between
questionnaire items, measurements and
constructs
• Final version comprising approximately
50 questions derived
• Methodological and practical issues had
considered in operationalising the
instrument
Data Analysis Challenges
• All questions were designed within
demands of data
• Develop single continuous scale
measure for each construct that is unidimensional, can discriminate across full
spectrum of students and is
valid/reliable
• Four steps in the construction of scale
measures, namely:
1. Item selection
2. Examination of the empirical
relationships of items
3. Combining of items into a scale
measure; and
4. Validating the scale measure
Risk Categories:
Key Element of Segmented Student Profiling
• This involved distilling 3 primary student-related cluster constructs
from the predictive model, namely:
– Academic ability
– Psychological attributes/metacognitive skills and
– Life circumstances
– Effective engagement with the institution (construct left out of the
initial risk categorisation, as this involves complex measurement
through, for example, student engagement surveys)
• A good example of deriving simplified, but meaningful measurable
constructs out of the complexity of the full predictive model
• The challenge was to define risk categories which could be
measured on appropriate scales. Three approaches were explored
Hypotheses
• If sufficient engagement, integration & transformation is
achieved, this will generate:
– Greater utilisation of support services
– Sufficient fit between students' choices, behaviours,
transforming attributes & performance and institutional
communications, practices, expectations and culture
• In turn, this will generate greater success in:
– Formative assessment, course success, graduation, student
satisfaction and required graduate attributes
Risk Model
3 Categorical Risk Measures: High/Medium/Low (27 Permutations)
Academic Ability
Skills/Attributes
H
H
H
H
M
Academic Ability
Able
Able
Able
Challenged
Able
Challenged
Challenged
Challenged
H
Circumstances
Risks
Risk Categories
H
3H
Very Low
H
M
2H 1M
M
H
2H 1M
2 Categorical Risk
Measures: High/Low
(82HPermutations)
H
H
1M
M
M
1H 2M
H
M
1H 2M
Low
Skills/Attributes
Circumstances
5-point Risk Measure
(125 Permutations) Risks
Risk Categories
M
Permutations
Developed
M
Very low risk
H
20 H
Underdeveloped
3-6 L
Conducive
33
Developed
L
7-8
Obstructive
H
M
9-10 L
Conducive
M
L
H
H
L
37
Developed
H
H
25
M Underdeveloped
L
L
M
H
M
H
10 M
Underdeveloped
H
Developed
L
M
Underdeveloped
L
RiskConducive
Score
Risk1H
Categories
H
2M 3L
H
M
11-12
Obstructive
H
13-15 H
Conducive
L
Obstructive
M
2H 1L
Very
Low2L 1H
2H 1L
Low 2L 1H
2H 1L
1H 1M 1L Moderate
Moderate
2L 1H
1H 1M 1L
High
1H 1M 1L1L
2H
High risk
Very
High
1H 1M
1L
1L 2H
1H 1M 1L
1H 1M 1L1L
M
3M
L
1H 2L
Obstructive
Low risk
L
H
L
1H 2L
L
L
H
1H 2L
L
M
M
2M 1L
M
L
M
2M 1L
M
M
L
2M 1L
M
L
L
1M 2L
2H
3H
High
Very High
Very high risk
Reflection on process so far
Challenge 3:
Data analyses
• Use of different multivariate techniques
• “While identifying relevant variables
Step 3
Analyses,
Interpretation
and
Reporting
Step 1
Project
Design
Research
Process
Challenge 1:
Identifying relevant
measures
• Translation of conceptual ideas into
explaining and protecting success is the
meaningful questions/variables for
point of departure, the real challenge, in
profiling, tracking & risk/success
light of the complexities involved, is
determining the combined effects of and
relationships between different predictor
prediction
Step 2
Data Collection
Quantitative
• Defining of measurable constructs &
risk/success categories
variables.” (Subotzky & Prinsloo,
• Scaling considerations
Distance Education 32/2, 2011)
• Definition of risk categories
Challenge 2:
Methodological and operational
considerations
• Tracking system
• Survey design (data gathering method;
timing & frequency; incentives
Questions