Enhancing ODL Students' Success through Risk Profiling and Prediction: The Case of Unisa HANLIE LIEBENBERG Senior Specialist: Institutional Research, Unisa PROF GEORGE SUBOTZKY Executive Director: Information & Strategic Analysis, Unisa DION VAN ZYL Manager: Information Services, Unisa Presented at: NADEOSA Conference, Johannesburg, 30 August 2011 Acknowledgements • The efforts of numerous DISA staff members in gathering and preparing information is acknowledged • In particular, the help and support of Robert Lightbody, admin Asst/caregiver to Prof Subotzky, was invaluable in preparing this presentation Overview 1. Background, Key Challenges & Research Problem 2. Unisa Student Success Framework 3. Unisa Conceptual & Predictive Models of Success 4. Unisa Tracking System 5. Data Analysis Challenges 6. Segmented Profiling: Categorising Student Risk Background • Whilst various theoretical models contribute towards understanding the various dimensions impacting on student success, utilising actionable intelligence to inform effective interventions remains daunting • This challenge is particularly formidable at Unisa, which now has +340 000 mainly non-traditional, older, part-time, underprepared students • They face challenging socio-economic circumstances, particular work-related and domestic responsibilities, which impede on student success Research Problem • To address this, Unisa recently developed a student support & success framework, comprising 4 elements: - Conceptual model - Predictive model - Student support interventions - Evaluating impact • Critical challenge: moving from conceptual model of student success to profiling, tracking, assessing and predicting risks to success Key Challenges • Key concerns and critical questions that arose in developing an integrated Student Support and Success Framework in the Unisa context • More particularly, the process of moving from the conceptual modelling of student success – a necessary first step – to the detailed student profiling, tracking and predictive modelling of risks upon which effective interventions are based • Key challenge: translating and operationalising relevant constructs of the high-level conceptual model to create a comprehensive student profile, tracking system and predictive model which retains sufficient complexity but remains practicable The Challenge of Translating Theory into Practice A theory that could fully explain every aspect of the attrition process would contain so many constructs that it would become unwieldy if not unmanageable. Such situations call for the use of theoretical models which are simplified versions of reality that strip away the minute details to concentrate on factors that are assumed or deduced to be important. ... Models can be judged by their usefulness. A model of the attrition process should contain sufficient constructs to explain what is undoubtedly a complex process and yet sufficiently simple to be understandable and useable. It should be able to explain collected descriptive data, and it should provide a framework against which predictions can be hazarded and judgements made about potential interventions. Kember (1989: 279-280) Operationalising the Conceptual Model This implies: • Identifying and defining all academic and non-academic variables needed for construct measurement, segmentation, profiling and predictive modelling; • Utilising suitable data gathering methods that yield consistent, complete and unbiased data; and • Applying appropriate advanced statistical analysis that can identify complex underlying multivariate dynamic relationships between variables and constructs Elements of the Unisa Student Success Framework Extensive literature review & conceptual modeling of all factors affecting success in Unisa context Comprehensive profiling, tracking and intelligence gathering culminating in predictive model of student risks/success Incrementally implementing an institutionwide Student Support Framework Evaluating impact over time SHAPING CONDITIONS: (predictable as well as uncertain) • Social structure, macro & meso shifts: globalisation, political economy, policy; National/local culture & climate • Personal /biographical micro shifts STUDENT IDENTITY & ATTRIBUTES: • Situated agent: SES, demographics • Capital: cultural, intellectual, emotional, attitudinal • Habitus: perceptions, dispositions, discourse, expectations THE STUDENT WALK: Multiple, mutually constitutive interactions between student, institution & networks • Managing complexity/ uncertainty/ unpredictability/risks/opportunities • Institutional requirements known & mastered by student • Student known by institution through tracking, profiling & prediction INSTITUTIONAL IDENTITY & ATTRIBUTES: • Situated organisation: history, location, strategic identity, culture, demographics • Capital: cultural, intellectual, attitudinal • Habitus: perceptions, dispositions, discourse, expectations TRANSFORMED STUDENT IDENTITY & ATTRIBUTES: Processes: • Informed responsibility & ‘choice’ • Ontological/epistemological dev. • Managing risks/opportunities/ uncertainty: Integration, adaptation, socialisation & negotiation F I T F I T Domains: • Intrapersonal • Interpersonal Modalities: • Attribution • Locus of control • Selfefficacy F I T F I T FIT F I T Retention/Progression/Positive experience Choice, Admission Learning activities Course success Graduation Employment/ citizenship F I T F I T F I T F I T F I T TRANSFORMED INSTITUTIONAL IDENTITY & ATTRIBUTES: Processes: • Informed responsibility & choice • Managing risks/opportunities: Transformation, change management, org. learning, integration & adaptation Domains: • Academic • Operational • Social Modalities: • Attribution • Locus of control • Selfefficacy SHAPING CONDITIONS: (predictable as well as he uncertain) • Social structure, macro & meso shifts: globalisation, internationalisation, political economy, technology, social demand • HE/ODL trends, policy • Institutional biography & shifts; Strategy, business model & architecture, culture & climate, politics & power relations Success FIT Key Constructs of the Predictive Model a) Students' inter-personal attributes: Demographics and past socio-economic status, including educational and family background and exposure to role models; Current socio-economic status and life circumstances, measured by the constructs of time and opportunity to study and stability in life circumstances and support for study; b) Students' intra-personal attributes: Academic readiness and ability; Metacognitive skills; Psychological attributes and outcomes of other processes; c) Institutional services, practices & culture: The quality of academic and administrative services; Institutional culture and practices; d) Integration, engagement and transformation: Students' effective management of their life circumstances and mitigation of risks as well as meeting learning expectations and utilising opportunities; The institution's effective management of academic and support processes and mitigation of risks. Student as Situated Agent • • - Inter-Personal Background: Current SES & Life Demographics Circumstances: Past SES - Time & Educ. Background Opportunity Family Background - Stability & Support Role Models Intra-Personal Academic Readiness & Ability Success Student’s Effective Management of: • Life Circumstances & Risks • Learning Expectations & Opportunities Fit: Academic Choices & Activities Fit with Institutional Culture & Practices Institution’s Effective Management of: • Academic & Support Processes/Risks • Student Profile/Risk & Communication Institutional Services, Practices & Culture Quality of Academic Services Quality of Admin Services Social: Institutional Culture & Practices Psychological Attributes & Outcomes Student Walk Integration, Engagement & Transformation Utilisation of Admin/ Support Services MetaCognitive Skills Formative Assessment Course Success Graduation • Satisfaction • Graduateness Institution as Situated Agent Senate STLSC School/College TLSC Student Success Forum Academic Department Professional Structures Lecturer/Supervisor/Online Mentor/Tutors/Regions TSDL DCCAD Library DISA Student Support Coordinator USGS Dean of Stud. Academic Admin Structures DSAR DSAA SMPPD Affective Admin TRACKING SYSTEM Profiling, Tracking & Predicting Risk at the level of Student/Module/Qualification/Institution Student Information • • • • • • • • Applications/Registration HEMIS Assessment Performance/Scores Academic Readiness Self-Assessment Student Profile Survey Student Satisfaction Survey Exit/Tracer Surveys ICMAs Operational Processes • • • • • Application/Registration Study Material Assessment Management Finance HR Communication/Engagement • • • • • • • • College/School/Department/Regions E-Tutor/F2F Tutor/Online Mentor Counsellor Call Centre Admin Department Tutorial Attendance myUnisa/Library Student Course Evaluation Student Profile Design Challenges • Considerations were given specifically to question response formats and scaling • Initial draft survey questionnaire consisting of over 100 questions based on key constructs identified in conceptual and predictive models • Throughout design process, imperative to ensure alignment between questionnaire items, measurements and constructs • Final version comprising approximately 50 questions derived • Methodological and practical issues had considered in operationalising the instrument Data Analysis Challenges • All questions were designed within demands of data • Develop single continuous scale measure for each construct that is unidimensional, can discriminate across full spectrum of students and is valid/reliable • Four steps in the construction of scale measures, namely: 1. Item selection 2. Examination of the empirical relationships of items 3. Combining of items into a scale measure; and 4. Validating the scale measure Risk Categories: Key Element of Segmented Student Profiling • This involved distilling 3 primary student-related cluster constructs from the predictive model, namely: – Academic ability – Psychological attributes/metacognitive skills and – Life circumstances – Effective engagement with the institution (construct left out of the initial risk categorisation, as this involves complex measurement through, for example, student engagement surveys) • A good example of deriving simplified, but meaningful measurable constructs out of the complexity of the full predictive model • The challenge was to define risk categories which could be measured on appropriate scales. Three approaches were explored Hypotheses • If sufficient engagement, integration & transformation is achieved, this will generate: – Greater utilisation of support services – Sufficient fit between students' choices, behaviours, transforming attributes & performance and institutional communications, practices, expectations and culture • In turn, this will generate greater success in: – Formative assessment, course success, graduation, student satisfaction and required graduate attributes Risk Model 3 Categorical Risk Measures: High/Medium/Low (27 Permutations) Academic Ability Skills/Attributes H H H H M Academic Ability Able Able Able Challenged Able Challenged Challenged Challenged H Circumstances Risks Risk Categories H 3H Very Low H M 2H 1M M H 2H 1M 2 Categorical Risk Measures: High/Low (82HPermutations) H H 1M M M 1H 2M H M 1H 2M Low Skills/Attributes Circumstances 5-point Risk Measure (125 Permutations) Risks Risk Categories M Permutations Developed M Very low risk H 20 H Underdeveloped 3-6 L Conducive 33 Developed L 7-8 Obstructive H M 9-10 L Conducive M L H H L 37 Developed H H 25 M Underdeveloped L L M H M H 10 M Underdeveloped H Developed L M Underdeveloped L RiskConducive Score Risk1H Categories H 2M 3L H M 11-12 Obstructive H 13-15 H Conducive L Obstructive M 2H 1L Very Low2L 1H 2H 1L Low 2L 1H 2H 1L 1H 1M 1L Moderate Moderate 2L 1H 1H 1M 1L High 1H 1M 1L1L 2H High risk Very High 1H 1M 1L 1L 2H 1H 1M 1L 1H 1M 1L1L M 3M L 1H 2L Obstructive Low risk L H L 1H 2L L L H 1H 2L L M M 2M 1L M L M 2M 1L M M L 2M 1L M L L 1M 2L 2H 3H High Very High Very high risk Reflection on process so far Challenge 3: Data analyses • Use of different multivariate techniques • “While identifying relevant variables Step 3 Analyses, Interpretation and Reporting Step 1 Project Design Research Process Challenge 1: Identifying relevant measures • Translation of conceptual ideas into explaining and protecting success is the meaningful questions/variables for point of departure, the real challenge, in profiling, tracking & risk/success light of the complexities involved, is determining the combined effects of and relationships between different predictor prediction Step 2 Data Collection Quantitative • Defining of measurable constructs & risk/success categories variables.” (Subotzky & Prinsloo, • Scaling considerations Distance Education 32/2, 2011) • Definition of risk categories Challenge 2: Methodological and operational considerations • Tracking system • Survey design (data gathering method; timing & frequency; incentives Questions