CSU LEARNING ANALYTICS STRATEGY Version 1.3 Table of Contents 1. Introduction............................................................................................................................................................... 3 2. Glossary ..................................................................................................................................................................... 4 3. Purpose and Vision .................................................................................................................................................... 5 4. Drivers ....................................................................................................................................................................... 5 5. Values ........................................................................................................................................................................ 6 6. Objectives .................................................................................................................................................................. 6 7. Strategies ................................................................................................................................................................... 7 A. Gathering learning analytics data .................................................................................................................... 7 B. Technical architecture ..................................................................................................................................... 8 C. Proactive Analytic Strategies ......................................................................................................................... 10 8. Development and review of policies ....................................................................................................................... 10 9. Governance ............................................................................................................................................................. 12 10. Specific roles and responsiblities .......................................................................................................................... 12 11. Ethical issues and risk mitigation mechanisms ..................................................................................................... 13 12. High level indicators of successful analytics usage at CSU .................................................................................... 13 2|P a g e 1. INTRODUCTION This strategy covers learning analytics, and acknowledges the critical role of the learner, the teacher and course and subject design in the educational process. Note that learning and academic analytics overlap and a clear distinction might not be possible. Learning analytics is situated within the general notion of “analytics” (see Glossary below). Learning analytics can be uni-dimensional but the goal is to develop multi-dimensional models where various factors are integrated and analysed in a systemic way and being an expression of the agency of those involved in the learning and teaching systems at all levels. The Working Party acknowledges that learning is a complex social activity and that technical methods do not fully capture the scope and nuanced nature of learning. It is further critical that the type and nature of interactions are analysed and not merely the amount of interactions. Aspects of learning analytics are dependent on the availability of “big data” and as such are research that uses massive amounts of data. Learning analytics can be provided to the student, teaching staff, student support staff, teaching support staff, and administrators to support adaptive practice and adaptive systems. It can be used to support a wide range of activities from day-to-day teaching and learning activities, to institutional support for at risk students, and the generation of new insights regarding teaching and learning patterns and effectiveness. Below is a description of possible uses of learning analytics which is an adaption of work by George Siemens (2012). 1 Focus of analytics Who Benefits? Items below can be described as “learning analytics” Subject-level: alignment with learning experience design, social networks, conceptual development, language analysis Learners, teaching staff, support staff Aggregate (big data) predictive modeling, patterns of success/failure Learners, teaching staff, support staff Items below can be described as “academic analytics” 1 Institutional: learner profiles, performance of teaching staff, quality of course and subject design, resource allocation Administrators, IR, funders, marketing, learners Regional & National (state/provincial): comparisons between systems Governments, administrators International: ‘world class universities’ National governments (OECD) Siemens, G. (2012). Learning analytics: new insight or new buzzword? ACODE webinar. October 2012 3|P a g e Internationally there is progression to multi-dimensional and systemic learning analytics that e.g. include data external to the LMS and from support services. Focussed research occurs through the international Society for Learning Analytics Research (SoLAR) (http://www.solaresearch.org/) There are a number of types of learning analytics systems (George Siemens, 2012): Dashboards; Recommender Systems; Predictive models; and Alerts/warnings/interventions. Analytics can be provided in the following areas (George Siemens, 2012) that may be applied to students or teaching staff: Analytics around social interactions; Analytics around learning content; Analytics in different spaces; Analytics on interaction with the university system; Analytics on intervention and adaptation; and Assessment of analytics. Please see Section 10 on roles and responsibilities of various players and for more information on developments at CSU. The working party that was established by the ILSC to create this strategy were: Assoc Prof Philip Uys (convenor) (DSL); Nina Clemson (P&A); Simon Thomson (Office of Student Services); Liz Smith (Academic Support); Paul Bristow (DIT); Nadine Mckeown (DSL); Assoc Prof Barney Dalgarno (Sub-Dean L&T, Faculty of Education). The working party was supported by Kate Rose (DSL). Special input was received from Assoc Prof Alan Bain (Faculty of Education). 2. GLOSSARY Academic analytics Analytics AUSSE BICC Business Intelligence Business Intelligence Competency Centre CEQ Community of practice 4|P a g e “The application of business intelligence tools and strategies to guide decision-making practices in educational institutions. The goal ... is to help those charged with strategic planning in a learning environment to measure, collect, decipher, report and share data in an effective manner.” (http://searchcio.techtarget.com/definition/academicanalytics) Note that learning analytics and academic analytics are interrelated and that the course and learning designs determine what a university might wish to analyse through academic analytics. “The use of data, statistical analysis, and explanatory and predictive models to gain insights and act on complex issues.” (EDUCAUSE) Australasian Survey of Student Engagement Business Intelligence Competency Centre A set of methods and techniques that are used by organisations for tactical and strategic decision making. An organisational team that has defined tasks, roles, responsibilities and processes for supporting and promoting the effective use of Business Intelligence across an organisation. Course Experience Questionnaire A group of people who share a craft and/or a profession and/or an interest Data warehousing DWBI ILSC Learning analytics Master data SES UES A consolidation of data from a variety of sources that is designed to support strategic and tactical decision making. Data warehousing and business intelligence Information and Learning Systems Committee Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs. (SoLAR) Note that the learner context referred to above includes relevant computer systems, learning experience design, the role of teaching staff as well as learning and teaching support staff. Persistent, non-transactional data that defines a business entity for which there is, or should be, an agreed upon view across the organisation. CSU Student Experience Survey University Experience Survey 3. PURPOSE AND VISION Purpose: To provide a university-wide framework to guide the use of learning analytics at CSU , and to improve our understanding of models of learning and teaching at CSU and its performance in the context of: o International and national developments and drivers contextualised to CSU; o Institutional values, objectives and strategic priorities; o Relevant institutional policies, governance, roles and responsibilities; and o Ethical issues and institutional risks; Including: o High level objectives and strategies; o High level indicators of successful usage; o Infrastructure and human resource issues; o Risk mitigation strategies; and o Policy and governance issues. This Strategy informs the CSU Educational Technology Framework, the DIT ICT Strategy in this area and the work of the BI Steering Committee. Vision: A high quality student centred learning experience supported by subject, course and institutional analytics that reflect the activities of students, teaching and support staff. 4. DRIVERS EXTERNAL 1. Remaining competitive in an increasingly global market; 2. Increasing public accountability and transparency; 3. Performance in external benchmarking quality indicators such as CEQ, UES, AUSSE and International Student Barometer; 4. CSU’s Compact with the Government highlights retention in the context of increasingly diverse intakes; 5|P a g e 5. TEQSA is increasingly focussing on course delivery and support processes requiring increased accountability; 6. Evidence of successful use of Analytics within Higher Education internationally (e.g. Purdue University) and in other sectors of society (e.g. business); and 7. Increasing expectations by students as customers of a rich and responsive online environment INTERNAL The need for: 1. Improved student retention and progress; 2. More efficient and effective targeting of teaching and support activities and resources; 3. Improved understanding of the relationship between models of learning and teaching and performance indicators at CSU; 4. Evidence-based professional learning of teaching staff; 5. Prioritisation of investment in learning systems and activities; and 6. Improved overall satisfaction of CSU students 5. VALUES 1. Alignment – we seek to improve alignment between models of learning and teaching and performance indicators. 2. Supporting student success – we recognise that the analysis of learning and teaching related behaviours and data can provide valuable insights into the student experience. We use this data for the purpose of supporting student progress and retention and promoting teaching excellence. 3. Student centred – we place students at the centre of the learning experience by accommodating diverse individual characteristics in the learning process; providing them with choice; and allowing them to be active learners who are capable of managing their own learning, also through the use of analytics. 4. Continuous improvement – we recognise the learning and teaching environment and needs of students change constantly. We reassess our practice on a regular basis to ensure we are collecting and analysing relevant and actionable data that presents significant opportunities for improving the student learning and teaching experience. 5. Evidence based decision making – we refine learning and teaching and the support of students in response to differing and changing needs and behaviours and ensure that actions are linked to strong evidence. 6. Respect – we abide by all related confidentiality and privacy regulations and ensure that actions taken as a result of access to student and staff data is done so in an ethical, supportive and respectful manner. 6. OBJECTIVES LEARNING AND TEACHING 1. To provide feedback to students on their learning interactions and progress to improve their likelihood of success, their choice of subjects and their self-management of learning on the basis of models of learning and teaching at CSU. 6|P a g e 2. To provide alerts and reports relating to student activity and progress to teaching and teaching support staff that would enable and inform appropriate intervention strategies on the basis of models of learning and teaching at CSU. 3. To provide feedback to teaching staff on the effectiveness of their learning designs and learning and teaching interactions on the basis of models of learning and teaching at CSU. 4. To provide reports to Course Directors to help inform revisions to the learning designs within subjects and courses as part of curriculum renewal and course reviews using the Smart Tools project as appropriate. 5. To provide alerts and reports to Heads of School that would enable and inform appropriate management interventions and professional development strategies. ORGANISATIONAL 1. To support the student experience through the enhancement of University systems and processes. 2. To provide information to support institutional strategic planning in the learning and teaching area, including indicators to allow the achievement of learning and teaching objectives to be more effectively monitored. 3. To provide information to support the improvement of relevant administrative processes. 4. To allow benchmarking of online learning interactions against other institutions. 5. Support selection, and prioritisation of systems and services through increased awareness of actual usage by students and staff. 6. Provide dynamically, adaptive systems informed by analytics to ensure adaptive learning. 7. STRATEGIES The CSU Learning Analytics Strategy will be underpinned by a whole of institution approach that ensures consistency of approach for all students combined with the flexibility to allow teaching staff and school specific initiatives tailored to the needs of specific student cohorts. Relevant theories, learning and teaching models, analytical methods and technologies will be used to implement the strategies below. A. GATHERING LEARNING ANALYTICS DATA At an institutional level we will collect and use data related to areas like the following: 1. The design of courses and subjects; 2. Student activities in relation to learning design; 3. Historical data such as students Grade Point Average (GPA); previous performance by the student in subjects’ historical performance rates for the subject; 4. Student access of key information (e.g. access to online subject outline within first 3 weeks of session); 5. Student assessment (e.g. failure to submit an assignment, failure of an assignment); 6. Student engagement in compulsory online activities (e.g. introductory forum posting, download essential readings); 7. Student feedback on university wide surveys (e.g. OES, SES, CEQ, UES and AUSSE); 8. Student participation in online orientation and induction activities; 9. Staff uploading of online subject outlines within required timeframe; 10. Staff posting welcome/expectations message in Week 1; 11. Staff turnaround of assignments within appropriate timeframe (currently within 21 days); and 7|P a g e 12. Teaching staff participation in online communication strategies i.e. forum activity; or chat activity; or online meeting activity; 13. Teaching staff access to teaching resources. Note: These factors will be reviewed on a regular basis to ensure continued relevance by relevant parties including Academic support and DSL. Faculties, schools and teaching staff may develop more specific data collection and response strategies according to the individual needs and characteristics of their cohorts for instance using online dashboards. Such strategies will normally be agreed to by the relevant Head of School or Executive Dean, clearly documenting the purpose for data collection, the method and the organisational responsibilities for such data collection and the strategies that will be put in place as a result of this data collection. Such strategies should complement rather than duplicate university wide processes and where possible should integrate existing resources, processes and information. Predictors of student success will be developed based on the cohort, national and international good practice. In addition to predictors of success, further indicators need to be identified that describe deeper student engagement and higher levels of learning such as critical curiosity; meaning making; creativity; resilience; strategic awareness; and the building of learning relationships (Shum, 20122) B. TECHNICAL ARCHITECTURE SOURCE SYSTEMS 1. All University information systems should be considered as potential analytics source systems unless specifically excluded. 2. Learning Management and related systems acquired or built by the University (such as Smart Tools and other course design and management tools) must capture data relevant to learning and academic analytics. 3. Learning Management and related systems acquired or built by the University must provide a reasonable level of ‘operational’ level reporting for student and teaching users. 4. Learning Management and related systems acquired or built by the University must provide direct access to captured learning and academic analytics data for extraction into other systems, such as a data warehouse or other analytic tools. ENTERPRISE ANALYTICS SYSTEMS 1. Learning and teaching data should be processed and consumed as per the pipeline below. That is, learning and teaching data may exist in either transactional data tables, unstructured files or other source. All data sources should be extracted from the source location and transformed into data warehouse structures. In most cases users and tools should consume the learning and teaching data from the data warehouse and associated reporting functionality where needs are not met by the application’s reports. 2. Data warehouse structures should ensure that data is stored at the lowest appropriate level of granularity and that dimensions are ‘conformed’ to other student related data structures i.e. data need to be aligned structurally with other university data. 3. Where application reports exist, as far as practicable data warehouse reports should be consistent with results displayed in source system reports. 2 Shum, S.B. (2012). Our learning analytics are our pedagogy. Ascilite webinar. October 2012. 8|P a g e 4. Learning Management and related systems could be capable of sophisticated learning analytics that could parallel some of the stages in the diagram above. ENTERPRISE REPORTS 1. A suite of standard reports should be developed in consultation with relevant groups, and may be embedded within the Learning Management system, if the system does not provide appropriate reporting functionality. Additional reports and ad hoc requests will be developed on an as needs basis. ACCESS TO ANALYTIC DATA 1. Access to data will be controlled by roles, with appropriate access privileges set depending on the sensitivity of the data and the role of the individual. Policies need to be developed to clarify the approvals required for access to data at various levels. Access to analytics systems should provide audit trails of what data is accessed by whom. 2. Where reports don’t directly require identifying individuals data will be aggregated and anonimised to unlink personal identification with captured data. The core data set will, however, retain the identification data. 3. A community of practice should be responsible for providing contextual information regarding teaching and learning data and reports, such as the meaning of and limitations regarding each data element, the use and inference of analytics and to refer questions to subject matter experts as appropriate. REQUESTING ANALYTICS SERVICES This section relates to the creation of reporting channels as well as access to such channels. 1. 2. 3. 4. 5. 6. Source system reporting tools – eg LMS dashboards, traffic lights. Faculty or School requests authorised by Executive Dean or Head of School for analytics from DSL. Academic Support request from P&A. Committee and executive requests. Self serve access to authorised data using pre built data cubes and visualisation/reporting tools. Custom requests may frequently evolve into standard reporting services. 9|P a g e C. PROACTIVE ANALYTICS STRATEGIES In addition to the provision of a variety of reports for students, subject coordinators, course directors, heads of school and teaching support staff relating to individual or collective student interactions and progress and staff activities, a number of types of alerts will be generated, which will either be delivered via emails or will be presented inside the LMS. Some categories of alert will be generated automatically in all cases, while other categories will be configurable either by the student or the subject coordinator. The following table shows a non-exhaustive list of examples of some of the types of alerts which could be further developed through close consultation with stakeholders especially the faculties. In each case policies need to be developed to make clear whether the display of such alerts is governed by either opt in or opt out decisions. Degree to which configurable Trigger for alert How delivered To whom Issues with the design of courses and subjects Non reading of student forum postings by subject coordinator for period of 3 weeks Unanswered messages on subject forum for period of one week Non access of subject outline by student 2 weeks into the session Non submission of an assignment by the due date Non access by a student to a resource marked as important by the subject coordinator Non completion of a course or subject survey Through the Smart Tools system HOS, Subject coordinator Always generated Message HOS, Subject coordinator Always generated Message Subject coordinator Configurable Message Student Always generated Message Student, International Student Support Officer Configurable Message within LMS Student Configurable Message Student Always generated Traffic indicator within LMS and perhaps also Email depending on severity Student, Subject coordinator Always generated Learning activity profile indicating that student is at risk of failure 8. DEVELOPMENT AND REVIEW OF POLICIES Policies around analytics need to be deeply embedded in the core policies of CSU around all of its learning and teaching functions. Possible relationships between learning analytics and a number of CSU policies are identified below. Critical policy areas include student access to analytics; use of historical data; privacy; access to information; and collection and storage of data Content delivered via a CSU owned or controlled service and accessed via a browser is covered by the CSU Web Policy and is subject to the governance of the Web Management Committee (chaired by the Executive Director, Marketing and Communications). This policy sets out CSU’s management principles on the development, maintenance and use of its website, including CSU Interact. 10 | P a g e o CSU Web Policy Document (2008) Other relevant polices that will need to be reviewed and/or reworded: 1. The conditions and obligations associated with authorised use of CSU’s computing and communication facilities are set out in the CSU Policy for the Use of University Computing and Communication Facilities (2006) o The policy does not explicitly refer to the monitoring and capture of online behaviour for the purpose of L&T analytics. There are several existing sections that could be reworded or strengthened – for example: 12. OBLIGATIONS – Staff (p. 8) 12.2 To assist in fulfilling the above obligations, CSU reserves the right to audit and monitor the use by employees of CSU’s computing and communication network and facilities. 18. CONFIDENTIALITY - Students (p.12) 20.2 CSU reserves the right to audit and to monitor the use of CSU computing and communication facilities. 2. Policies relating to privacy and confidentiality, including the use of records databases will also need to be amended to cover the collection, storage and use of data: o Privacy Management Plan (INF42, 2001) Privacy – A Brief Summary (INF41, 2001) o DIT Privacy Statement This page is also referenced on the ‘Changing your Password’ webpage o It includes text such as ‘What Information is collected’ and ‘Use of Personal Information’ that will need to be reviewed. There is currently no mention of capturing data for analytics. o Division of Human Resources - Staff “Privacy – Your Responsibility” “Charles Sturt University collects and retains considerable personal data regarding its students and staff during the course of their candidature and employment.” o Several Divisions at CSU maintain specific privacy statements on their websites Library Services: Privacy of Information on Online Forms Student Services 3. Use of, and linkage to external services (import and export) o The Policy and Guidelines for the use of External Educational Technologies (under development) to be updated to investigate and/or facilitate the collection of learner analytics across external systems 4. Collection, use and storage of data - Students and teaching staff o Academic Communication With Students Policy o The CSU Student Charter “the expectations students and the University may have of each other” o Changing your Password (staff and students) “By clicking the button below, I am signifying my agreement with the University's Policy for computing and communications and with the conditions stated above” Alternative mechanisms of ‘signifying agreement’ may have to be sought if students are no longer compelled to regularly change their password 11 | P a g e o o Access to Student Records - Overview of rights and responsibilities Access to Student Records “The University's policy that details staff and student access to records, the amendment of records and to assessment items and information” DIT Policy for the Storage and Transmission of Personal/Private Information “This policy covers the storage and transmission of any personal and/or private information within and outside CSU’s network” 5. Stakeholder relationship to other policies and initiatives within the University o Smart Tools, ie to include analytics on the course and subject design process, o Curriculum Renewal and CSU Degree, o BUSS, o STAR. 9. GOVERNANCE This Strategy informs the CSU Educational Technology Framework, the DIT ICT Strategy in this area and the work of the Business Intelligence Steering Committee. 1. Role of the ILSC: monitor implementation of the Learning analytics strategy. 2. Role of BIC Steering Group: to set clear and concise guidelines on the collection and use of student related data. 3. Research Ethics Committee: data for research 4. Academic Senate: managing policies related to learning analytics 5. Curriculum Learning and Teaching Committee: summary reports related to the CLT Plan 6. Learning Analytics implementation working party: a multi-disciplinary team that will coordinate the implementation of the Strategy under the auspices of the ILSC taking into account the needs of diverse stakeholders. Note: Interaction between the ILSC and the Web Management Committee is necessary to ensure that developments pertaining to analytics are properly integrated. 10. SPECIFIC ROLES AND RESPONSIBLITIES 1. Teaching staff: Involvement in the development of the use of learning analytics at CSU. Use the data to improve learning and teaching. 2. Dean of Students, Executive Deans, Heads of School, Course Directors and Sub-Deans and Associate Deans Learning and Teaching: User. Identify data types required. Request and act on data. 3. DSL: Contact point for Faculties, Schools and teaching staff for learning and teaching data. Gathering data, analysis & reporting, as part of DSL’s own sub-data warehouse reporting infrastructure. Resolve data quality issues relating to systems that DSL is the custodian for. Provision of professional learning activities and support to users (teaching staff and managers) of analytics of systems that DSL is the custodian for, and other systems in which learning analytics are provided. Conceptual leadership about learning analytics as it pertains to teaching staff. 4. Planning and Audit: Location of CSU’s Business Intelligence Competency Centre (BICC) which manages CSU’s data warehouse and supports the effective use of BI and analytical data at CSU. This area will be involved in extracting source system data, combining it with other relevant data and processing into reporting structures, and then providing reports, analysis, tools and guidance to consume the final data sets. 12 | P a g e 5. DIT: Supply system usage information from activity logs and other very large information sources in a highly aggregated form. Assess new information systems for ability to supply information to meet analytics needs. Where possible, facilitate fixing data quality issues in source systems. Provision of integrations systems and services to support the transporting and transforming of information between key systems. In partnership with P&A and DSL facilitate the building of reports and dashboards and data warehouse feeds of relevant information. Supply user identity and affiliation information to support access control to analytic information. Ensure the integrity of the data being stored and extracted for analysis. 6. Academic Support: Contact point. Identifying relevant data types and strategies to address results. Production and dissemination of relevant resources identified as a result of analytics and integration with programs such as Student Success Team. Relevant training of students. 7. Student Services: User of analytics and record interactions with students 8. Smart Tools project: Direct access to analytics on course and subject design 9. Student Central: User of analytics and record interactions with students 10. Division of Library Services: User of analytics and record interactions with students 11. ETHICAL ISSUES AND RISK MITIGATION MECHANISMS 1. Privacy and appropriate use of learning analytics are key ethical issues. 2. Policies will need to be developed on who has access to what data about students’ and teaching staff online learning activity and in what circumstances and these will need to be developed to be consistent with some broad ethical principles. 3. Policies and guidelines will also need to be developed specifying how analytics data can be used and for what purposes. Note: These policies would be developed in collaboration with appropriate parties at CSU such as the CSU Ombudsman and Legal Office. 12. HIGH LEVEL INDICATORS OF SUCCESSFUL ANALYTICS USAGE AT CSU 1. Increase in the quality and effectiveness of online teaching as per Student Experience Survey due to adaptive online teaching practice and/or adaptive online systems. 2. Increase in student retention rates through more effective interventions either automated or human. 3. Increase in the quality and effectiveness of online learning as per assessment results. 4. Increase in online engagement due to feedback on learning practices. 5. Increase in the appropriateness of subjects selected by students. 13 | P a g e