Cascade Project: Evaluation Report DRAFT v4 December 2010 Henriette Lundgren Cascade Project Evaluator henriette.lundgren@conted.ox.ac.uk +44 (0) 7511 096672 Marion Manton Project Manager marion.manton@conted.ox.ac.uk +44 (0) 1865 280986 Bridget Lewis Nicola Warren Table of contents Evaluation summary and recommendations ...................................................................... 3 Background, aims and structure of this report .................................................................... 5 1 2 3 4 5 Focus area 1: Online assignment handling ................................................................ 6 1.1 IT support time ................................................................................................... 7 1.2 Administration handling time .............................................................................. 7 1.3 Customer satisfaction ......................................................................................... 8 1.4 Ease of use ........................................................................................................ 9 1.5 Conclusion: new system improvements on their way ........................................10 Focus areas 2 and 3: VLE support and generic content ............................................10 2.1 Overall adoption rate .........................................................................................12 2.2 User friendliness (administrators) ......................................................................13 2.3 Usage rate (students) ........................................................................................14 2.4 Customer satisfaction (students) .......................................................................16 2.5 User acceptance (academics) ...........................................................................17 2.6 IT set-up and support time ................................................................................18 2.7 Conclusion: high acceptance of VLE support for courses ..................................18 Focus area 4: Course design ....................................................................................19 3.1 Technology engagement ...................................................................................20 3.2 Conclusion: technology use in course design increases ....................................21 Focus area 5: Online enrolment and payment...........................................................21 4.1 Administration handling time .............................................................................22 4.2 Adoption rate.....................................................................................................23 4.3 Customer satisfaction ........................................................................................26 4.4 Conclusion: growing number of online enrolments – less handling time ............27 Overall project evaluation..........................................................................................28 5.1 Project management .........................................................................................28 5.2 Stakeholder engagement ..................................................................................29 5.3 Learning as a result of the project .....................................................................29 5.4 Tangible benefits ...............................................................................................29 5.5 Conclusion: achievements against aims and objectives ....................................30 Appendices.......................................................................................................................32 2 Evaluation summary and recommendations The Cascade project evaluation was conducted in three steps. First, an Evaluation Plan was agreed (April 2010); next baseline data was collected and presented in the Baseline Report (June 2010). Finally, a second quantitative and qualitative data set was gathered (November 2010) to close the evaluation loop. For the online assignment handling area, evaluation data was drawn partially from the new as well as from the legacy system. Data from pilot studies were collected as the new and improved online assignment handling system will not be fully rolled out until the academic year 2011-12. Our preliminary time-motion analysis of the new system has shown significant time savings in comparison to paper-based assignment handling, but the handling efficiency still lags behind the legacy online assignment handling system where routine use has decreased administration time. In terms of student adoption and satisfaction, the new system has received positive responses during the pilots. Further efficiency and service improvements are anticipated once the new assignment handling system is fully implemented. VLE support for courses as well as online delivery of generic content have both been well received and adopted by members of the Department. This is especially true in the area of award-bearing courses where the take up of the VLE resources has been good. A significant challenge will be supporting staff and students as numbers increase. However, tools have been put in place to manage this issue, and there is cause to be optimistic about the volume of IT support that will be required. It is anticipated that VLE supported courses, with the integration of generic content, will be an increasingly important tool for administrative staff, tutors, academics and students. In these focus areas, we see full achievements on both aims: efficiency and service. As Departmental staff become more familiar with the VLE and are able to set up their courses more efficiently they will focus more of their attention on other value-adding course activities. Technology use in course design has increased over the last two years. This trend can be in large part directly attributed to the Cascade project work as the project has supported a general trend of technology engagement and confidence within the Department. All new programmes coming through the Department’s approval process in the last nine months now plan at least basic use of the VLE, and because of that we see the innovation aim of this focus area as having been achieved. Online enrolment and payment is now almost as popular as paper-based enrolment. This reduces dramatically the administration handling time and error rate when compared to the manual approach. In the case of a standard online enrolment, the time savings in comparison to paper-based enrolment are immense. Where online enrolment requires an extra feedback loop via email with the course administrator, administration handling time savings still exist but are less significant. In terms of student satisfaction with the online enrolment website, a consistently high level of customer satisfaction can be observed. In this area, improved efficiency and service is clearly identifiable. As a consequence of the evaluation analysis and conclusions, a number of recommendations were established to guide the Department’s efforts beyond the end of the project. Recommendation on Online Assignment Handling System: The Department should drive the full implementation of the new system further, including the proposed roll-out to all legacy system users and new courses in 2011-12 and to all award-bearing courses subsequently. 3 Recommendation on VLE Implementations, including Generic Content: The Department should focus on award-bearing courses where VLEs are used over a longer period of time and therefore maximise the ‘shelf-life’ and return on investment. Recommendation on VLE Improvements: Through the collection of feedback from all key stakeholders – administrative staff, part-time tutors, students and full-time academic staff – course VLEs can be integrated in cycles of continuous improvement. The provision of additional support sessions to all staff when they meet as course teams is recommended to embed the use of the self-service resources and documentation available from the project. Recommendation on Course Design: The Department should continue to offer support to academic staff, to assist them to plan how they will use technology effectively to deliver the curriculum when they are designing new courses. Recommendation on Online Enrolment and Payment: The Department will identify all courses where online enrolment and payment could be successfully implemented i.e. for those courses where the process can be fully automated. By doing so, the Department will reduce the number of errors associated with manual processing, improve transaction times, ensure the enrolment and payment service can be offered 24/7, while at the same time generating significant financial value through the reduction in staffing costs associated with automating the process. Detailed findings in each focus area (Sections 1-4) and for the overall project (Section 5) are presented in this report. Supporting evidence can be found in the Appendices. 4 Background, aims and structure of this report The first step in the evaluation of the Cascade project was to devise an Evaluation Plan – submitted to JISC in March 2010 – and to define the measures of success. In order to construct this Evaluation Plan, all available Cascade documents, such as the original JISC call, the project plan and activity plans for the five focus areas of the project were reviewed and synthesised. Next, several informal meetings were conducted with the project team members. Two days were spent identifying aims and key measures of success for the individual focus areas. Finally, an evaluation plan matrix was agreed, which outlined the various evaluation questions, activities, and suggested data-collection methods. The second step was to take a snapshot of the initial status of the project to establish a baseline. The goal of the resulting Baseline Report – submitted to JISC in June 2010 – was to provide a sound basis on which the success of the project could later be evaluated. The baseline was established following the same structure as the Evaluation Plan; starting with focus area 1 on online assignment handling and concluding with focus area 5 on online enrolment and payment. Where possible, data collection, analysis, and interpretation was separated into three distinct sub-sections. The Baseline Report focused entirely on the description of baseline information and did not include any summative evaluation on the progress of the project. The third and final step in the evaluation process was to summarise our findings in this Evaluation Report. This report draws from various data sources and meetings and an evaluation workshop that took place on 10 November 2010 with the project team. In this report, we aim to compare the baseline data to the data that was collected throughout the project. As well as reporting on the various focus and evaluation areas, we also aimed to link each area back to the core project aims, i.e. to increase efficiency, to innovate and to improve levels of service: Aims Efficiency Innovation Service Focus Areas Online Assignment Handling VLE Support for Courses Online Delivery of Generic Content Course Design Online Payment & Enrolment Evaluation Areas Admin handling time IT set-up and support time User acceptance Adoption rate Ease of use Usage rate and areas Technology engagement Technology confidence Technology best practice Service response time Customer satisfaction Figure 1.1: Pyramid of project aims in focus and evaluation areas (Source: Cascade Evaluation Plan) The Evaluation Report will conclude with a section on overall project evaluation and lessons learned during the project. 5 1 Focus area 1: Online assignment handling Online assignment handling was recognised early on as a high priority for the project, due to the clear benefits it offered the Department both in terms of time and cost savings and in the potential to offer an improved service to students. This was also recognised at the wider University level where there was a parallel pilot of online assignment handling, with which we collaborated to identify areas of common practice. Within the Department, a legacy online assignment submission system (referred to as ‘CASS’) was being used by a few programmes. However this had considerable functionality issues and needed replacing, and as such was not scalable for wider adoption across the full range of the Department's course offerings. Initially, the plan was to have the new online assignment-handling system available to all courses that wanted to use it for the start of the 2010-11 academic year. However, the launch of the new system was delayed to allow more time for extensive testing and piloting. All those staff involved in the piloting of the system have opted to adopt the system. We are now offering the opportunity for a number of courses to trial the system for the remainder of the current academic year, and we anticipate a very significant adoption rate for the start of the 2011-12 academic year. Consequently, an evaluation of this focus area will be done using the following structure: # Evaluation area Main stakeholder(s) Evaluation feasibility check Section in Evaluation Report 1.1 IT support time TALL IT support team Only a small amount of pilot data available; reference to legacy system 1.1 1.2 Administrationhandling time Registry Only small amount of pilot data available; comparison paper vs CASS vs new assignmenthandling system 1.2 1.3 Adoption rate Department Summative data not available – indicative data discussed in focus area conclusion 1.5 1.4 Customer-service response time Students Data not available to evaluate NA 1.5 Customer satisfaction Tutors, Registry and students Only small amount of pilot data available 1.3 1.6 Ease of use Students Only small amount of pilot data available 1.4 Table 1.1: Overview of evaluation data for Focus area 1 As the new online assignment-handling system was not fully implemented at the time of the second data collection, only four of the six evaluation areas have been included in this report. 6 1.1 IT support time In our Baseline Report, we gave an overview of the IT support time it took to support users of the Department’s legacy online assignment-submission system during the period August 2007 to July 2009. The main data we referred to came from an Excel spreadsheet held in the Department that detailed the amount of time spent providing IT support to students, administrative staff and academic staff to support their use of the old online submission system. Ideally, we would have liked to collect the same data for the new online assignment-handling system in order to compare IT support times. However, as the new system was not available as a production service during the evaluation period, no directly comparable data could be collected. We therefore decided to continue reporting IT support data on the legacy system instead: Total IT support time (CASS) Number of programmes supported Baseline 2007/08 3,050min 7 Baseline 2008/09 5,005min 9 New data 2009/10 4,875min 8 Table 1.2: IT support time CASS For more detail on the reported IT support time, please see Appendix 1.1. As Table 1.2 shows, the total IT support time for the legacy system rose from 3,050min in 2007/08 to 5,005min in 2008/09. This increase in support time was mainly due to an increased number of courses being set up and supported. In comparison, the total support time in 2009/10 decreased again to 4,875min as a consequence of the reduction in the number of courses supported. IT support time was also tracked for the new online assignment-handling system during pilot studies. Apart from bug reporting by teams such as the Registry etc. there were no more than two calls to the staff providing IT support for the new system. This meant that the vast majority of academics and students were able to use the system without any issues; a major improvement on the previous system which, despite familiarity with the system still requires significant amounts of IT support, as shown above. While this might be taken to be extremely encouraging, the IT Manager cautions on projecting too optimistically considering the relatively small numbers who have used the system thus far. We do know that we have designed out key problem areas in the previous system. However, the IT Manager still suggests, from experience, that there may be unexpected issues that will only become evident as the service is rolled out to more programmes. 1.2 Administration handling time We also gave an overview of the current administration handling time for paper-based vs. the legacy online assignment-handling system in our Baseline Report. We then referred to two time motion studies that depicted average administration-handling time for each method. Based on data collected during a pilot of the new online assignment system with the British and European Studies course, a third flowchart was drawn up and validated by the Department’s Registry in November 2010. Assignment handling process steps Baseline Paper-based Course assignment set up Preparation for marking 5min 15min Baseline New data Old system New system 5min 10min None None 7 Preparation for moderation 45min Student notification 30min Total end-to-end assignment handling 95min 35min1 10min 50min 45min 10min 65min Table 1.3: Administration handling time for assignment submission Extension approval process Baseline Paper-based Receipt of request Passing request on to decision maker2 Receipt of decision Verification and notification of decision Total end-to-end extension approval 2min 5-30min 2min 12min 21-46min Baseline New data Old system New system 2min 2min 5-30min 5-30min 2min 2min None 2min 9-34min 11-36min Table 1.4: Administration handling time for extension approval For more detail on the reported administration handling time, please see Appendix 1.2. The time motion studies are split into two parts: With the new online assignment-submission system, the administration handling process takes 30 minutes less in assignment submission and marking (Table 1.3) and 10 minutes less for the extension approval process (Table 1.4), in comparison to the paper-based process. These time savings, taken from initial time motion studies in conjunction with pilot testing, initially seem to be lower than the time savings when we compare the old assignment-handling system vs. paper-based process. However, the reason behind the somewhat conservative time savings attributed to the new system is explained, in part, by the familiarity that the Registry staff have with the existing system, coupled with the fact that figures for the new system have been collected when using it in piloting conditions, where things cannot be expected to run as smoothly as in an established service. More specifically, the data is also skewed by the longer moderation time taken as a result of issues with the file compression (zip) and download system encountered during the pilot study. These features have now been resolved in the new system. Overall, there is extremely good reason to believe that when this process is repeated with the final release of new assignment-handling system, time savings will be greater than those observed with the legacy system. Other advantages of the new online vs. paper-based systems are: the reduction in paper handling and photocopying as more courses use the system; better auditing and control; and improvements in assignment transaction storage. Reduction in paper storage is a further advantage both in terms of less physical space being required, and also in terms of less staff time being required to retrieve data from the archive. In summary, additional testing is planned to conclude the formative evaluation of the new online assignment-handling system, and we will then provide a summative statement on the progress of this evaluation area. 1.3 Customer satisfaction Key stakeholders of assignment-handling systems are students who submit their assignments and academics who receive or download the students’ submissions, mark them 1 2 This number was adjusted by Registry from 20min to 35min after the initial baseline data was collected in April 2010. The amount of time taken to pass the request to the decision maker depends on how the request is received and whether additional authorisation needs to be granted. 8 and then notify the students of their grades. We chose two data collection methods in this area to seek feedback from key stakeholders. For the Baseline Report, we first analysed an online survey that involved students, administrative staff and academics from across the Department. The results of this online questionnaire, with 57 respondents from different groups and programmes, depict a positive attitude towards the use of technology in assignment submission and the implementation of a new online assignment-handling system. We analyzed email feedback from users of the legacy system to draw a qualitative baseline. We concluded that, when academic staff experienced problems, they often associated them with the online technology rather than the underlying administrative processes, despite the fact the latter were often the cause. This highlighted the importance of ensuring that technology is well integrated in robust administrative processes in order to maximise user satisfaction. Comparing our baseline information to initial feedback from the new system pilots, the following conclusions can be drawn. Tutors, the Registry and students have all pointed out bugs in the system which have been, or are being, addressed. Modifications to the user instructions have also been made to respond to the issues raised. The most common underlying reason behind issues encountered when using the system was that users had not read the instructions. A few users who had previously used the legacy system took some time to adjust to the new system. There were no negative comments as a result of the change from one system to another and a number of users commented that they felt the new system was an improvement. The following quotes from two students and one Course Director demonstrate levels of customer satisfaction with the new online assignment submission system after the initial pilot: "As far as I can see, everything has worked ok. I got all the responses I was expecting and the processes were simple to follow." (Student 1) "I suppose it's because I've learnt to live with CASS that I found this took some getting used to. I can see though that it's an improvement, and I like the 'control panel' where you can see everything that's gone through at a glance." (Student 2) “Our students found the online system straightforward and easy to use both for essay submission and to request extensions.” (Course Director 1) “System navigation was easy and intuitive. It was simple to download student essays and to provide feedback. In fact, it was much more user-friendly than initially anticipated.” (Course Director 1) These four quotes indicate the positive attitude towards the new online assignment submission system. New technology takes some time to settle; however, we have learned that ease of use and the enhancement of functionality were key to improving customer satisfaction levels for the new online assignment-handling system. 1.4 Ease of use Finally, in our Baseline Report, we looked at the ease of use of the current online assignment-submission system. The legacy system log information was hard to extract and interpret, so we chose to analyze feedback provided by the IT support manager. Four main system issues were reported. These were accessing the system, uploading multiple files, requesting extensions,; and email retrieval by marking tutors. We concluded that the legacy 9 system was missing essential functionality that would enhance the ease of use in online assignment submission. All of the issues outlined above have been addressed in the design of the new online assignment-handling system. Students using the system during our Statistics for Health Researchers pilot study used a dummy assignment to practice uploading files, and amending and deleting them as required. This tested the enhanced functionality and introduced the students to the look and feel of the new system before they were asked to use it to upload real assignments. Of the 17 students on the course, 11 viewed the practice assignment and seven practiced uploading files. This suggests that inclusion of a practice assignment is useful. Also, there were no forum postings relating to problems with assignment submission. For the British and European Studies pilot, we also scanned the log files recorded in the course VLE to see if there was any evidence of students experiencing difficultly uploading assignments and using the system as per the pilot instruction. The analysis showed that students uploaded the assignments without difficulties and also tested the system successfully for requesting extensions. For more details on the pilot study analysis, please see Appendix 1.3. Further testing will be required prior to a summative statement on the progress of this evaluation area being made available. 1.5 Conclusion: new system improvements on their way The Cascade team worked successfully to get the development of the new online assignment-handling system to its current position. However, to evaluate this entire focus area, we could only collect data from a few pilot studies as the full release of the new system will not be until the beginning of the 2011-12 academic year. As the system is not fully implemented, we could only report formative evaluation data in this section. Our initial administration-handling time analysis has shown significant time savings in comparison to paper-based assignment handling, but the handling efficiency still lags behind the legacy online assignment-handling system where familiarity and routine use has decreased administration time. For the new online assignment-handling system, the potential timesaving should be greater in the long term, but the initial pilot data and somewhat conservative time estimates from the Registry do not yet show this efficiency gain. In terms of student adoption and satisfaction with the new system, positive signals have been received during the pilots. Many users thought the new system was an improvement in comparison to the old online assignment-handling system. Also course directors who were involved in the system testing were positive and intend to continue to use the system following their initial pilot use. Recommendation on Online Assignment-Handling: The Department should go ahead with the full implementation of the new online assignment-handling system, including the proposed roll-out to all legacy system users and new courses in 2011-12 and to all awardbearing courses subsequently. 2 Focus areas 2 and 3: VLE support and generic content In recent years, the Department has offered VLE support for courses in three main areas: fully-online courses, blended learning and to support largely face-to-face learning. However, while the first area is well supported and developed, use of a VLE to underpin the other two categories has generally taken place on an ad hoc basis and was only used on a small minority of the programmes that might benefit from this provision. Addressing this in a more systematic fashion was always intended to be a focus of the project. However, identifying which approaches to take was a significant challenge. Of the original 21 sub-themes, 15 were related to this area and much of the process of refining the scope of the project focussed on prioritising work in this area. 10 The pilot studies gave the project team useful insight into how VLEs were being used and how well they were received by both students and staff. The course teams were very positive, despite having to spend time learning how to upload and edit content. They found the Moodle environment very useful for communicating with their students, on both administrative and academic matters, and for streamlining access to information, which resulted in a reduction in the need for photocopying. All course teams that took part in the pilot studies are now implementing VLE support more widely across course offerings in their subject areas, including supporting new audiences such as alumni. Initial student responses to the VLE support provided for their courses were broadly positive, although some students lacked confidence in their skills with technology. This will be partially addressed through the materials provided in the generic content pilot study that was closely interlinked with the VLE developments. One finding from our initial stakeholder engagement activities and the review of new course proposals was the inefficiency inherent in the amount of generic content created and recreated independently by different course teams across the Department. This issue was evident both in administrative areas, such as that covered by our course handbooks, and in training in what might be called 'academic literacies', especially in areas such as basic study skills and library and information skills. It was recognised that, in many cases, this information was delivered at the start of a course when students were not fully cognizant of its importance. It was the case, therefore, that when students needed a specific skill, they often required retraining. This created a demand for reusable online content addressing these areas, accessible by students at their point of need. The Cascade project team worked with library staff and with course teams to identify the topics that would be most valuable to develop. We also sought to identify whether there was existing content within the Department, the University or the wider HE sector, which could fulfil this need but whose existence was not necessarily known to either students or tutors. From our pilots and wider implementation experience this academic year, it is now clear that the materials most frequently included in a VLE by staff have been those that are more administrative in nature. All new courses have included course handbook materials and links to updated library, technical guidance and support information provided through the website; the more “academic” content has seen more varied uptake by course teams. However, the extremely low cost of sustaining these links once set up makes provision of them worthwhile. For this Evaluation Report, the two focus areas of VLE support for courses and online delivery of generic content will be evaluated in one section as outlined in Table 2.1: Main stakeholders Evaluation area # Section in Evaluation Report All Adoption rate of VLE and generic content 2.3 2.1 Administrators User friendliness 2.1 2.2 Students Usage rate 2.4 / 3.3 2.3 Students Customer satisfaction 2.5 / 3.1 2.4 Academics User acceptance 2.6 2.5 TALL support team IT set up and support time 2.2 2.6 Table 2.1: Overview of combined evaluation data for Focus Areas 2 and 3 11 During the evaluation data analysis, we found that these two focus areas were closely related, and that evaluation areas had great overlap. We therefore re-organised evaluation areas around the main stakeholders, i.e. administrators, students, academics and the TALL support team. 2.1 Overall adoption rate The first area on VLE support for courses looks at adoption rates. To establish a baseline, we simply counted the number of courses that were using the VLE in April 2010. For the Evaluation Report, the project team decided to focus on award-bearing courses, which can be split into two categories: PIP award-bearing CPD award-bearing Total Baseline April 2010 8 5 13 New data November 2010 18 8 26 Table 2.2: Number of VLE supported award-bearing courses3 In April 2010, there were eight award-bearing programmes using a VLE in the Department’s Public and International Programmes (PIP) division and five in the Continuing Professional Development (CPD) division. This baseline compares to 18 in PIP and eight in CPD during our second data collection point in November 2010: a doubling of courses using a VLE. In addition to the 26 award-bearing courses using a VLE listed above, two further courses were using WebLearn (the VLE used by the University’s Medical Sciences division) and four courses were delivered fully online and have not been counted in the table above. As such, out of a total of 48 award-bearing course cohorts which could use a VLE in some capacity, there now remain only 17 which do not offer VLE support for their programme. In terms of overall adoption of generic content, links to the following generic content were found in ten analysed award-bearing courses that are supported with a VLE: Link to guidance on using site and technical help (10 courses) Link to course handbook (9 courses) Link to library information websites (8 courses) Link to University Skills Portal (5 courses) Link to University computing information (2 courses) For more detail on the reported generic content links in course VLEs, please see Appendix 2.1. The data show that the majority of course VLEs link to at least four generic study resources, such as the technical help page, the course handbook, the library information website and the University Skills Portal. The analysis also showed that a few courses linked to Departmental or University of Oxford generic content (e.g. a series of 14 transferable skills online training courses available on the University’s Skills Portal site) that many were previously unaware of. We also found that more course teams than we would have predicted linked to external study skills content (e.g. study skills information at the University of Sheffield and The Open University). 3 Where a course programme runs for two or more years and has two year groups of students in parallel, separate course VLEs are set up for each cohort and are counted as separate courses in this table. 12 2.2 User friendliness (administrators) In our Baseline Report, we wrote that the VLE should be perceived as easy to set up as well as reliable and simple to use. For the data collection method, we decided on a qualitative approach and conducted semi-structured interviews with individual course administrators who had agreed to participate in the project. In these interviews, staff were asked about their experiences before and after the implementation of the Cascade project VLE support for courses enhancements. As we had decided to collect this data in a single attempt, qualitative interviews took place only after the pilot implementations in June 2010. In total, three interviews were conducted: Interview Interview 1 Interview 2 Interview 3 Course manager Weekly classes Archaeology Psychodynamic Counselling Date of interview 17 June 2010 16 June 2010 8 June 2010 Table 2.3: Interviews conducted with Course Managers, June 2010 A summary of theses interviews can be found below: Interview 1: “The VLE is a great place to send people to when they make repetitive enquiries about the same thing,” states the weekly classes Course Manager in our interview. As a consequence, the Course Manager thinks that the number of enquiries has been significantly reduced due to the VLE support. Generally the weekly classes are already trying to move many processes into soft copy, so this has just continued that process. However, he still thinks they will be sending out information in letter form for a while as, in the past, the suggestion that all communication will be electronic has resulted in part-time tutors asking for Departmental email accounts. That has disproportionate resource implications and so is not something they want to encourage. The Course Manager thought that the VLE saved time for the class administrator and that the VLE was very easy to use. He did not encounter any technical problems and he reported that other staff around him found the VLE as support for weekly classes equally useful. The interviewee plans to further develop support provided via the VLE in the 2010-11 academic year. Interview 2: The benefits from VLE-supported courses are that communicating with students is made easy and very efficient as the course administrator can contact everyone in one go. “This was especially useful in an emergency like the snow in early 2010” she explained. The VLE is also good for providing information on aspects of the course that were not known at the start of the programme and therefore were not in the handbook, for example, sharing materials such as maps, plans, diagrams, lists etc. before a session. Thus, a benefit is information dissemination and knowing that all students have access to it. However, the VLE is not so much adopted by the archaeology tutors: neither the common room has been used nor the tutor forums as tutors did not engage. Tutors are used to working the old way and find it hard to change. This is something best introduced at the start of a new programme and more likely to work if built in from start, with useful mandatory activities taking place online. Currently, tutors still photocopy articles for students and only once did a tutor ask to put an article on the VLE. This shows that tutors might not have adopted the new VLE support structure yet. In terms of efficiency, the VLE took time at the start but definitely saved time once it got going. The Course Manager is confident that the VLE will save time in the future. Technical problems were found around logging in and remembering passwords. The Course Manager thinks that course tutors could adopt the new system more and that they are still being quite oblivious to its advantages. Several months into the pilot the Director of Studies emailed students rather than using the forums and then continued dialogue via email, despite the VLE as the common communication platform being in place. 13 Interview 3: For the third Course Manager, the News forum and the ability to provide articles online rather than photocopying these for the class were perceived as most useful in the context of VLE support for her courses. The biggest benefit was that the VLE saved time in course administration. The administrator estimated that she has saved in excess of 10 hours of photocopying. This is a conservative estimate, and she would expect it to increase further next year. “For part-time administrators, 20-30 minutes a week copying less in term time can be noticeable” explained the interviewee. Other time savings come from the News forum being a quick and efficient method of communication: there have been fewer requests for resending information than before because someone has lost an email. However, the biggest challenge is explaining the benefit and use of the VLE to tutors. As a matter of fact, tutors barely used the VLE and only saw it as an administrative rather than teaching tool. They also found it very hard to remember passwords. One reason for this “lack of interest” might be that the tutors in psychodynamic counselling are mostly very occasional teachers who are only contracted to teach one or two sessions. The administrator reasons that for these tutors, adoption of total VLE support cannot really be expected. Once they had spent some time with the VLE, however, they seemed to show more appreciation and adoption. In conclusion, the interviewee would definitely recommend the VLE as support for courses to others, and she would be prepared to act as a mentor to others. The Course Manager also stressed the importance of a single sign on (SSO) login for the course VLE in order for students and tutors to have fewer passwords to remember and therefore to increase the adoption of the VLE. Although all three Course Managers refer to different aspects of their course VLE set-up and use, a positive attitude towards the new course platform can be observed. 2.3 Usage rate (students) In the Baseline Report, we had focused on expected usage rate of weekly classes tutors, where we were positively surprised by the high technology usage rate of our 55+ target group surveyed in December 2009. For the Evaluation Report, we therefore focus on students (rather than tutors) to get a better picture on how much they had used both the VLE and the various generic content sources. We analysed the following two data sources: 1. VLE technical log data 2. VLE usage and activity A summary of VLE log data, shown in Table 2.4, gives a list of generic content included in the course VLE and the total number of times each resource was viewed: 14 Count of Full name Information All Archaeology courses offered by the Department Assignments Course Staff General Reading List Group Projects - individual topics How to complete your profile How to use discussion forums Introduction Library guide for Archaeology Markers and Deadlines Online etiquette Schedule of Teaching Sessions 2009-2010 Submission of Assignments and Practical Logbooks Summary of Course requirements for a One-Year Module Tools to support group projects Training Excavation and Practical Weekends Tutorials Useful Telephone Numbers and Contacts Writing Assignments Grand Total Total 25 51 37 22 31 31 3 19 11 26 5 27 12 30 35 37 20 22 7 451 Table 2.4: Summary of total number of page views for the Undergraduate Diploma in Archaeology VLE site, November 2010 The logs for the Undergraduate Diploma in Archaeology programme VLE reveal that none of the features we included as generic content had been ignored. However, as the templates (with their generic content) have only recently been consolidated into their almost-final format, it is difficult to draw conclusions about the items we have included there. For the VLE usage and activity data (see Table 2.5), award-bearing courses that used the VLE templates developed by the Cascade project to establish a VLE to support the delivery of their 2010-11 course were analysed in terms of a) student and staff log-in percentage, b) actions since course launch and c) percentage of days with activity in the VLE. All data was taken on 22 November 2010, and all courses, unless otherwise mentioned, were launched in the second week of October 2010: Course MSc Sustainable Urban Development PG Certificate in Historical Studies Foundation Certificate in History 2009-2011 Foundation Certificate in History 2010-2012 MSc English Local History Undergraduate Diploma in the History of Art Undergraduate Certificate in Archaeology 2010-2012 Undergraduate Certificate in Archaeology 2009 -2011 Undergraduate Diploma in Archaeology 2010-2011 Percentage of students and staff logging in 100% Actions since course launch 5,126 Percentage of days with activity in the VLE 100% 100% 12,954 100% 60% 326 52% 77% 605 75% 100% 60% 2,812 312 75% 54% 80% 1,308 90% 100% 1,370 70% 100% 1,488 92% 15 Course Postgraduate Diploma in Psychodynamic Practice 2009-2011 Postgraduate Certificate in Psychodynamic Counselling Percentage of students and staff logging in 100% Actions since course launch 100% 5,137 5,352 Percentage of days with activity in the VLE Consistent use since then (percentage could not be calculated 78% Table 2.5: Usage rate and actions in existing VLEs, November 2010 The usage data shows that in most cases all students and staff have logged-in to their VLE. It is worth noting that this figure should be treated with caution for many of the courses with lower percentages as for many sessional tutors, the VLE has not yet become relevant as they are only due to teach later in the course calendar. Across the majority of courses, the VLE has been accessed a high percentage of the days it has been available, without any downturn in the rate of access over time, suggesting steady and sustained use. 2.4 Customer satisfaction (students) In this area of evaluation, we looked at the perceived customer satisfaction when using VLEsupported courses. In terms of the data collection method, we decided to analyse data retrieved from various online surveys. For our baseline, we analysed responses from 34 students, of which 14 students were from Archaeology, 16 from Psychodynamic Counselling and 4 from Global Health studies. The data was collected in mid and late 2009. In our baseline analysis we concluded that most students had a positive attitude towards the introduction of a course VLE. Between mid June and mid July 2010 we conducted a new online survey to enquire about students’ customer satisfaction after the pilot studies. A total of 19 responses were collected from three different study programmes: Archaeology (9), Psychodynamic Practice (5) and Psychodynamic Counselling (5). We first asked the students how useful they found the course website (Q2). 46.1% responded that they found the course website useful or extremely useful, 41.2% said it was of some use and 11.8% found it not very useful. Next we asked them: “How easy was the website to use?” (Q4). The response was similar to the previous question: 47.1% found it very easy or extremely easy to use, 52.9% said it was fairly easy to use; no one responded that the course website was not very easy or not very easy at all. Details of the responses given by students to Q2 and Q4 are outlined in Appendix 2.2. In the next two questions, we asked students to rate different website features: which ones they used (Q5) and how valuable they had found these features (Q6). The features that were most used and also found most valuable were: Course information Administrative information relating to the course Reading list and links to library catalogues News forum for course announcements (sent by email) 16 These features echoed many of the areas identified by our administrators in Section 2.2. Towards the end of the questionnaire (Q14), we enquired about the importance of having a course website for future courses: If you were to enrol on a future course how important would it be to you that the course offered online support through a similar course website? 1 Extremely important 2 Very important 3 Of some importance 4 Not very important 5 Not important at all Figure 2.1: Importance of course website for future courses The survey results show that 35.3% of students think that having a VLE site to support their course is very or extremely important. The majority or students surveyed (41.2%) chose the middle answer “of some importance”, which indicates acceptance if not enthusiasm. 23.5% of students responded that having a VLE to support their course was not very important to them. Thus, we could conclude that more than three-quarters of the respondents view the VLE of some or higher importance. 2.5 User acceptance (academics) In terms of user acceptance among academics, we looked at their perception of the VLE templates that help generate an online presence for courses. For the data collection method, we used feedback received after VLE demonstration workshops that we held with academic and administrative staff: “Indeed a very interesting and useful morning! I would be delighted to set up a Moodle for the Diploma in the History of Art asap, a good trial run for the MSt. My feeling is I am happy to base it on the template you have developed, this all looked grand, and it is pliant enough that I may tinker at the edges easily should the mood take me.” (Academic 1, Lecturer in the History of Art) “I know a lot more and have more confidence. I have now established a skill set and can broaden it. I would not have done a podcast in my most recent course if I would have had to learn everything from scratch. The more one uses technology, the broader and more imaginative one’s use is likely to become.” (Academic 2, Lecturer in Local and Social History) “The demo went very well. I gained the impression that the students just took it as a norm that we could provide things like this, which was a shame because Tom and I were so impressed with ourselves (and more so with you and Kiri since you actually did the work!). There was no - wow you really are at the cutting edge of academic technology!!! Some were keen to submit electronically.” (Academic 3, Lecturer in History) 17 “Many thanks for sorting this. The stuff was all there but it will be easier to access it all in one place. We had a meeting with our external today and showed her the first module. She thought that every university in the country would like something similar for both its undergraduate and postgraduate populations.” (Academic 3, Lecturer in History) The first email quote shows the enthusiasm of this academic to use the VLE template for her course. The second email quote clearly indicates the adoption process of new technology skills, which is reflected in the lecturer’s broader use of them. The third and fourth emails also show excitement about the course websites, and email 3 also indicates that students are starting to perceive this feature as a default, while email 4 indicates that our approach, and some of the generic content developed within it, might be of interest for academics beyond the Department and universities beyond Oxford. 2.6 IT set-up and support time For this area, we asked two questions, on VLE set-up and support, to the dedicated VLE support team who provided their feedback in April 2010 and November 2010. In addition, reports on IT support time spent supporting the Department’s course administrators and academics to use VLEs to support their courses were reviewed and analysed. In April 2010, the VLE support team were involved in three main areas: 1. Loading tutors and students on to the system 2. Duplicating sites at the start of the academic year 3. Supporting more general VLE use In November 2010, the VLE support team reported back that generally there has not been much change in the overall IT set-up and support time. Despite more courses using VLE sites (see section 2.1 for more details) the IT support needs and time spent have not gone up significantly from last year, due to the supporting documentation, processes and support developed by the Cascade project. The VLE support team identified that one recurring issue was that staff did not adequately read instructions and support material when first building their VLE from the template. This should improve, as familiarity with these tools improves. In addition, some staff prefer phoning up the support team to ask for help rather than accessing the available support site information. One new issue that had not been anticipated was that as more art courses began to use the VLE, large graphic files were uploaded, which presented problems for the system. Support guidance was necessary, which resulted in extra IT support time being given by the VLE support team. 2.7 Conclusion: high acceptance of VLE support for courses Overall, VLE support for courses and generic content innovations have been well received and taken up by members of the Department. In the area of award-bearing courses, where we have focussed our efforts, the take up of the VLE and resources has been impressive. For individual stakeholders within the Department, the picture is somewhat mixed. Administrative staff expressed a very positive opinion towards VLE-supported courses and the use of generic content, and full-time Departmental academic staff, also reported high acceptance and enthusiasm towards the VLE. However, part-time sessional tutors seem to find the integration of these Departmental innovations into their courses more difficult. A reason for lower adoption among part-time tutors might be their lower degree of involvement and integration into the daily processes of the Department. For students, a steady usage rate 18 and general satisfaction could be observed which leads us to the conclusion that our students perceive VLE supported courses as valuable for their studies. In terms of IT support, a certain level is still required at the moment to set up and maintain course VLEs. The picture, though, remains unchanged from our baseline snapshot and, after the initial start-up support, most courses need very little ongoing support. This is very encouraging, as IT support has remained at a similar level to last year despite the fact that the number of courses that might use it has now doubled. There is also an ongoing expression of interest for using the VLE for the few remaining award-bearing in courses in the Department. Recommendation on VLE implementations, including generic content: The Department should focus on award-bearing courses where VLEs are used over a longer period of time and maximise the ‘shelf-life’ and return on investment. Recommendation on VLE improvements: Through the collection of feedback from all key stakeholders – administrative staff, part-time tutors, students and full-time academics staff – the course VLEs should be integrated in the Department’s cycles of review and improvement. Providing additional support sessions to sessional and contracted staff when they meet as course teams is likely to enhance use of the self-service resources and documentation made available by the project. 3 Focus area 4: Course design Many of the most successful uses of technology in curriculum delivery tend to have been designed into the course from the start rather than added as an afterthought. Thus, while many of the activities of the Cascade project have focussed on existing programmes, we also wanted apply the lessons learned, as appropriate, to as many new programmes being developed by the Department as possible. The project team originally considered supporting this process by running workshops for academics in the Department, which we know, from other projects, can be a powerful learning experience. However, as the project progressed it became clear that academics are more likely to find time to engage with this area when they are actually designing courses. At any one point this was rarely more than one or two individuals, making a workshop less valuable than one-to-one sessions. With this in mind, our approach evolved to the creation of a reference resource, the Course Design Moodle, which academics could access at a time that suited them, and by making individual support sessions available to staff at the relevant point of the specification process of their new course. At the time of preparing the Evaluation Report, there was only a small amount of evaluation data available on the newly created Course Design Moodle and other activities undertaken in this area (see Table 3.1). # Evaluation area Main stakeholders Evaluation feasibility check Section in evaluation report 4.1 Technology engagement Department Only small amount of pilot data available 3.1 4.2 Technology confidence Department Not enough data available to evaluate NA 4.3 Technology best practice Department Not enough data available to evaluate NA Table 3.1: Overview evaluation data for Focus area 4 19 As a consequence, only one out of the three evaluation areas described in the Evaluation Plan will be reviewed here: 3.1 Technology engagement Between October 2007 and October 2008, only two new course proposals were submitted to the Department’s Academic Board for approval and both outlined their intent to integrate some technology elements in course delivery. However the uptake of technology prior to this had been minimal. In October 2008, the course proposal form was amended to include several new questions which required justification of technology use, or not, in several different contexts. Since then, we have tracked the new course proposals forms to identify technology engagement in course design. Table 3.2 lists all received course proposal forms since October 2008 with course title and planned technology engagement. Month first submitted to committees May 20084 November 2008 November 2008 January 2009 February 2009 Course Technology engagement MSc in Sustainable Urban Development MSt in the History of Design VLE support DPhil in Management and Business Studies DPhil in Architectural History MSt in Literature and the Arts April 2009 Postgraduate Certificate in Historical Studies April 2009 Postgraduate Certificate (Research) in Archaeology Postgraduate Certificate in Cognitive Behavioural Therapy Postgraduate Certificate and Diploma in Health Research MSt in Diplomacy DPhil in Mutual and Co-operative Organisation MSc in Surgical Science and Practice Postgraduate Certificate in Cognitive Behavioural Therapy for Psychological Trauma Undergraduate Certificate in Local and Family History April 2009 June 2009 February 2010 February 2010 April 2010 April 2010 April 2010 October 2010 Postgraduate Certificate in Ecological Survey Techniques Using VLE to support course and deliver some content No planned use of VLE No planned use of VLE Using VLE to support course and to deliver two core modules online Using VLE to support course and deliver three core modules online No planned use of VLE VLE support VLE support VLE support No planned use of VLE VLE support VLE support Delivered online with one face-to-face residential session in Oxford Delivered online with one face-to-face residential session in Oxford Table 3.2: New course proposal forms and planned technology engagement 4 Although this course proposal was submitted prior to October 2008, as we have been engaging with this project team throughout the development process up to course launch this term, we have included them here. 20 This analysis shows that – after the proposal form amendment in October 2008 – the majority of academics proposing new programmes now want to use technology for course delivery and, apart from one case, the programmes which have chosen not to use the VLE are Doctoral programmes which have such low student numbers that the VLE would rarely be appropriate for an individual course.5 During the two-year period from October 2008 to October 2010, fifteen proposals were received in total, of which eleven planned to use the VLE to support delivery. Of these, two outlined significant plans for delivering elements of the course online and two propose to deliver the majority of the course online. This trend seems set to continue as TALL has already been contacted by other teams about developing significant elements of online provision for new programmes. 3.2 Conclusion: technology use in course design increases Clearly, engagement with online course elements in the Department has significantly increased since the project began. This trend can largely be directly attributed to the Cascade project. Since April 2010 all new programmes coming through the Department’s approval process intend at least basic use of the VLE. Confidence with technology and technology best practice were areas where data for a summative evaluation were hard to collect. However, with the overall amount of technology use in course design increasing, and from our experience working with staff across the Department, we can conclude that the overall level of confidence with technology among administrative and academic staff has risen. Recommendation on Course Design: The Department should continue offering support to administrative and academic staff in choosing and implementing technology options in course design. 4 Focus area 5: Online enrolment and payment Existing experience had indicated that there were major financial and operational benefits available through the provision of online enrolment and payment services, in particular: Significant savings in processing time by course administration staff; improved data management by creating a direct link between enrolments and payments; improving accuracy as students input their own data, therefore reducing the number of people involved and removing transposition errors; savings in financial administration time due to reduced manual processing of payments and opportunities for batch processing; improved service to our globally distributed students as enrolment is available 24/7. There was a need to increase the number of courses offering online enrolment in order to take advantage of the benefits identified above. Moreover, feedback had been received from users suggesting a number of areas where improvements could be made to the functionality and usability of the service. For example, a key issue raised by users was frustration that returning students needed to enter their details each time they enrol on a course (some students enrol on four or five short courses per year). To improve the user experience, online student login functionality was identified as a priority, providing students with access to manage their data and to help returning students register for courses online without the need to input their details again. 5 At Departmental level we are investigating supporting these with a Graduate School, which would offer broader VLE support to all doctoral students. 21 In this context, all three evaluation areas (4.1 administration handling time; 4.2 adoption rate and 4.3 customer satisfaction), as originally outlined in both the Evaluation Plan and Baseline Report, were reviewed. 4.1 Administration handling time In April 2010, time motion data was collected on paper-based and online enrolments and three different classes of enrolment types were identified: 1. Paper-based enrolment 2. Online enrolment 3. Online enrolment with feedback loop For these three enrolment types, the Department’s manager of Online Weekly Classes conducted a time motion study of how long it took to administer each type of enrolment and provided five samples, each with accurate time measures. A summary of the time motion study data can be found in Table 4.1. Longest enrolment time Shortest enrolment time Average enrolment time Baseline data Paper-based New data Online 13:45min 2:45min 5:45min 00:15min 00:05min 00:10min New data Online with feedback loop 15:00min 3:00min 5:25min Table 4.1: Time motion study results for course administrator handling time for different types of enrolment For more details on this time motion data, see Appendix 4.1a and Appendix 4.1b. For paper-based enrolment the longest sample of end-to-end student enrolment and payment process took 13:45min. In this instance, the system did not hold an existing student record and the administrator needed to create a new one. Also, the specific payment method added to the total time. In contrast, the shortest enrolment time was 2:25min. In this sample, the student record already existed, i.e. the enrolment was from a returning student, and the course payment was made by credit card. The average time it takes to process a paperbased enrolment is 5:45min, correcting for outliers by excluding the longest and the shortest enrolment times from the calculation. Online enrolment is processed automatically; the administrative handling time requires no more than 5-15 seconds to mark a successful online enrolment notification email as read and move it into the respective email archive folder. However, in some instances the online enrolment is not successful due to some missing or conflicting information. We called this type of unsuccessful attempt to enrol “Online enrolment with feedback loop”. An example could be where a student registers for a course with the EU-student fee but submits an address located in the US. In this instance, the administrator needs to undertake the following steps: 1. Read through the student email explaining the online enrolment problem 2. Write back to request further information 3. After further information retrieval, formulate instructions on how to solve the enrolment problem and reply to the student This “Online enrolment with feedback loop” process can result in a number of emails being exchanged between the prospective student and the Online Weekly Classes staff. This 22 process can take up to 15:00min. In best cases, it only takes 3:00min. On average, in our time motion study, this process of clarifying an unsuccessful online enrolment took 5:25min. 4.2 Adoption rate In order to evaluate the adoption rate of online enrolment and payment, we collected records of transaction counts from across the Department. The data source of these records was the Department’s database called InfoSys. To analyse the adoption rate of online enrolment and payment, we looked at three different indicators: 1. Ratio of Departmental courses that offer online enrolment 2. Number of online enrolments by course type 3. Percentage of online vs paper enrolments First, we looked at the number of courses that had the option available to enrol online. Data was collected by running a snapshot report on InfoSys, on the last day of each month, to collect data on the total number of course records for the current academic year onwards (A), the number of ‘live’ courses being marketed on the Department’s website (B), and the number of ‘live’ courses accepting online enrolments (C). The data is presented in Table 4.2. A B C (C/B) (C/A) Type Total course records in InfoSys Live courses accepting enrolments Live courses accepting online enrolments Ratio 1 Ratio 2 Apr-10 1,412 May-10 1,686 Jun-10 1,752 Jul-10 1,779 Aug-10 1,796 Sep-106 820 Oct-10 978 Nov-10 1,065 424 543 650 652 633 453 475 525 297 465 589 598 581 417 430 484 70.04% 21.03% 85.64% 27.58% 90.62% 33.62% 91.72% 33.61% 91.79% 32.35% 92.05% 50.85% 90.53% 43.87% 92.19% 45.45% Table 4.2: Online enrolment enabled courses, April - November 2010 Whereas ‘Ratio 1’ indicates the percentage of ‘live’ course currently being marketed on the Departmental website that offer online enrolment, ‘Ratio 2’ gives the percentage of current and future courses in InfoSys, but not necessarily yet open for enrolment, that are accepting online enrolments. Ratio 2 thus has more course records, which explains why it is much lower than Ratio 1, which only takes live courses into consideration. The table gives end of month snapshot data for the last eight months of the Cascade project. For example, at the end of April 2010, a snapshot of data from InfoSys found 1,412 course records for the current and future academic years, of which 424 courses were tagged as live courses with information on the Departmental website. The remaining 988 courses were either courses that were not marketed online, that had already finished or that had not started taking enrolments yet. Of the 424 live courses, 297 were marked with the flag ‘EnrolOnline’ which means that they have the online enrolment and payment functionality enabled. Therefore 297 out of 424 (70.04% = Ratio 1) of courses marketed on the Departmental website in April 2010 were taking online enrolments, and 297 out of 1412 6 In September 2010 the data collection parameter for course records in InfoSys was changed from ’09-10 courses onwards’ to ’10-11 courses onwards’ to reflect a change of academic year 23 (21.03% = Ratio 2) of all current and future courses currently in InfoSys at the time, but not necessarily yet open for enrolment, were accepting online enrolments. If we compare the April data to the November 2010 data, then we can see a steady increase in the number of courses accepting online enrolment and the number of live courses having the online enrolment and payment functionality enabled. The two ratios likewise indicate a steady upward trend in the percentage of the Department’s courses accepting online enrolments. Secondly, we looked at the number of paper-based and online enrolments, by course type, on a monthly basis. The baseline data was collected for the full academic year (August 2007 to July 2008) before the start of the project and compared to data collected for the two full academic years (August 2008 to July 2010) during which the Cascade project ran. 18,000 16,000 14,000 12,000 10,000 8,000 6,000 4,000 2,000 0 Paper Online 2007-2008 2008-2009 2009-2010 Year Figure 4.1: Total number of online and paper-based enrolments received by type Figure 4.1 shows the total number of paper-based and online enrolments received by the Department over the three year period from August 2007 to July 2010. The graph shows that during this period there has been a steady growth in the number of enrolments being received online, with almost double the number of online enrolments received in 2009-10 compared with 2007-08. In terms of the percentage of overall enrolments, online enrolments have increased from 21% in 2007-08 to 39% in 2009-10. When analysed on a monthly basis, of particular note is the number of online enrolments received during the month of December. Whilst in total number these are not particularly significant compared to other months of the year, as a percentage of overall enrolments, online enrolments in December make up a higher percentage of enrolments than for almost any other month of the year. For example, in December 2009, online enrolments made up 49% of enrolments received (compared to the year average of 39%). As the Department is closed for two weeks each year over the Christmas period, this data indicates that the availability of online enrolments means that students can continue to enrol and pay online even when the Department is closed. In terms of number of transactions, the increase in online enrolments has been particularly significant for the Department’s largest programmes, which provide open access courses for the general public and between them took almost 6,000 online enrolments in 2009-10. 24 500 450 400 350 300 PPWeekly - Online PPWeekly -F2F PPDayWeek - F2F 250 200 150 100 50 Au g0 O 7 ct -0 7 D ec -0 Fe 7 b0 Ap 8 r0 Ju 8 n0 Au 8 g0 O 8 ct -0 8 D ec -0 Fe 8 b0 Ap 9 r0 Ju 9 n0 Au 9 g0 O 9 ct -0 9 D ec -0 Fe 9 b1 Ap 0 r1 Ju 0 n10 0 Month Figure 4.2: Online enrolments for three large course programmes by month Figure 4.2 shows the number of online enrolments, by month, for three different course types offered by the Department’s Public and International Programmes division: 1. Weekly classes delivered online (PPWeekly – Online) 2. Weekly classes delivered face-to-face (PPWeekly – F2F) 3. Day and weekend schools delivered face-to-face (PPDayWeek – F2F) Overall the graphs indicate that enrolment numbers vary considerably by month and that the total number of online enrolments has been increasing steadily. Over the three year period, there has been a 60% increase in the total number of online enrolments received by the three programmes from 3,560 in 2007-08 to 5,683 in 2009-10. Third, we looked at the proportion of enrolments for the Department’s day and weekend schools and weekly classes (both those delivered online and face-to-face) that were taken online compared with the proportion received on paper (see Figure 4.3). 25 8,000 7,000 6,000 5,000 Paper Online 4,000 3,000 2,000 1,000 PPDayWeek 2009-2010 2008-2009 2007-2008 2009-2010 200820-09 2007-2008 0 PPWeekly Figure 4.3: Day and weekend schools and weekly class enrolments by type For the weekly classes programmes (“PPWeekly”) there was both an overall growth in total enrolment numbers as well as the proportion of enrolments received online. The percentage of online vs. total enrolments was already high in 2007-08 with 44% of enrolments taken online. In 2008-09, this percentage of online enrolments grew to 53% of total enrolments, an increase of 9% compared to the previous year. This ratio grew further to 58% in 2009-10. Thus, more than half of these courses are now booked and paid for online. In comparison, the day and weekend schools (“PPDayWeek”) showed an even steeper increase in online enrolments: from 6% in 2007-08 to 32% in 2009-10, which meant an average annual growth of 13 percent in the last two years. Finally, to illustrate total adoption of online enrolment over time, using the latest available data, Table 4.3 provides a summary of the number of online enrolments taken for all courses offered by the Department, and their associated financial value, for the first quarter of the four academic years 2007/08 to 2010/11. 2007-08 2008-09 2009-10 2010-11 No 427 578 835 1,163 August Value £57,549 £92,823 £171,297 £215,218 September No Value 527 £77,947 826 £147,406 1,053 £245,941 2,219 £283,741 No 217 327 566 452 October Value £33,163 £59.831 £190,756 £158,924 No 1,171 1,731 2,454 2,834 Q1 Total Value £168,659 £300,059 £607,994 £657,883 Table 4.3: Q1 Total number of online enrolments and their financial value 2007-08 to 2010-11 Table 4.3 shows clear growth both in terms of the number of online enrolment transactions and their associated financial value. Between 2007-08 and 2010-11, the number of online enrolments transactions increased by 142% from 1,171 in Q1 2007/08 to 2,834 in Q1 2010/11. Growth in financial value was even greater with a 290% increase over the same period. 4.3 Customer satisfaction The last area of baseline data is the attitude of students towards online payment and enrolment expressed as customer satisfaction when using this method of enrolment. Data were collected from an online survey that was made available for students to complete after 26 every online enrolment and payment made via the Department’s website. Key findings from this analysis are summarised in Table 4.4. Number of respondents Found the online enrolment form very clear Concerned or very concerned about online payment Unconcerned or very unconcerned about online payment Overall grade of 8 out of 10 or above for the Department’s online enrolment process Baseline April 2010 1,500 69.9% 9.6% 69.8% New data November 2010 2,377 69.2% 9.0% 71.3% 86.7% 86.3% Figure 4.4: Summary results from online enrolment survey The analysis shows that 69.9% of students surveyed in April found the online enrolment form very clear (69.2% in November). The concerns that students had when enrolling online were around the online payment. 9.6% in April said that they were ‘concerned’ or ‘very concerned’ about using the Department’s online payment process (9.0% in November). This can be contrasted by almost 70% of the students who said in April that they were ‘unconcerned’ or ‘very unconcerned’ about online payment (71.3% in November). 1,300 out of 1,500 respondents (86.7% in April) gave the Department’s online enrolment process an overall grade of 8 out of 10 or above (2,052 out of 2,377 respondents; 86.3% in November). 4.4 Conclusion: growing number of online enrolments – less handling time As outlined in detail in the Final Report7 and the project case studies on business performance8 and supporting prospective students,9 the enhancement and wider adoption of online enrolment and payment has been a major focus of the Cascade project, bringing significant benefits to the Department. As Figure 4.1 indicates, throughout the period of the Cascade project, course enrolments have moved away from paper and towards online enrolment and payment and, as the project’s outputs become embedded, this trend will continue. As a consequence, administrative handling time relating to the enrolment and payment process will be further reduced, saving a significant amount of staff time. This will result in a reduction of staff numbers associated with this non-value adding area of our work, leaving staff to focus on other activities that support student recruitment and retention. While the administrative handling time of paper-based enrolments depends on the payment method – payment by credit card being quicker than payment by invoice – and the pre-existence of a student’s record, the administrative handling time of online enrolments depends on whether additional information is required to complete the enrolment. In the case of a standard online enrolment, the time savings in comparison to paper-based enrolment are immense. Where online enrolment requires an extra feedback loop via email with the support team, administrative handling time savings still exist but are less significant. In terms of student satisfaction with the online enrolment website, a consistently high level of customer satisfaction can be observed, before, and during the Cascade project. By the end of the Cascade project, the Department had already seen measurable benefits in terms of savings in course administration time through the availability of online enrolment 7 8 9 Cascade Project Final Report: http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeFinalReport.doc Case Study 1: Leveraging technology to drive business performance: http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeCaseStudy1.doc Case Study 3: Using technology to support prospective students: http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeCaseStudy3.doc 27 and payment. The greatest impact has been seen in the Department’s Public and International Programmes division, which delivers the Department’s two largest programmes (Day and Weekend Schools and Weekly Classes). In the 2009-10 academic year, 5,683 (87%) of the total of 6,501 online enrolments taken by the Department were for these two programmes. As a result of savings made in course administration time, both programme offices have been able to support an increase in the overall number of course registrations; for example the Weekly Classes programme took 850 additional enrolments in 2009-10 compared to the previous year and reduced their staffing by 0.5 FTE, allowing them to reallocate the staff effort towards marketing and development of further new courses. Recommendation on Online Enrolment and Payment: Through further roll-out of online enrolment and payment functionality, the Department will capitalise on the substantial time savings in administrative handling time, enabling the extra staff time to be used either to support additional student numbers or to provide value-adding services to enhance the quality of course offerings. 5 Overall project evaluation 5.1 Project management Throughout the Cascade project, all reports were submitted to JISC in a timely manner. The project was extended by three months in order to achieve more implementation of results from the development areas that could only go live at the start of a new academic year. All but one of the focus areas – the roll-out of the new online assignment-handling system – were implemented successfully within the project timeline. When the Cascade team reviewed their own project management during the evaluation workshop held on 10 November 2010, the following three areas of improvement were highlighted: 1. Scoping 2. Resource allocation 3. Implementation planning In terms of scoping, the team felt they underestimated the time needed for this important stage of the project, which had a knock on effect on managing the project going forward. If the team was to run the Cascade project again, they would allocate more time and staff resources for requirements gathering and project scoping. In theory, resources in the TALL team were allocated to the Cascade project from the start of the project, but existing commitments restricted these in real terms. Thus the project manager could only put in her full time on the project (and recoup the initial short fall) after Cascade had been running for several months. Also, it took far longer than hoped for to recruit the Research officer. Also, more learning technology and administrative resources rather than technical development resources were available to the project. Since greater technical development than anticipated was required, this was not ideal. If the team were to run the project again, they would plan more realistically against availability of staff time and allow more flexibility to adjust staff allocation to fit the project’s requirements. Some of the implementation timings were not matched with the academic calendar of the Department. As a consequence, a delay of months in the development work of the online assignment-handling system resulted in a delay in full implementation of an entire academic year. The team also discovered that there were many dependencies between the different project focus areas; at the beginning of the project, the team found it difficult to identify and manage these. During the project, the team developed some simple project management documents to assist with overall planning across the five focus areas. These were very 28 helpful; however, the tools could not help where important external dependencies had been missed earlier in the project planning. Were the team to run the Cascade project again, they would identify dependencies (both those within and between focus areas, as well as external considerations) earlier and build them into the implementation plan. 5.2 Stakeholder engagement The project undertook extensive stakeholder consultation and engagement across all divisions of the Department. Against the staff list in December 2010, of 119 staff who had roles that might benefit from the outputs of the Cascade project, 93 (78%) had directly engaged with the project. Through this, the team encountered a real cultural change in attitudes towards technology use. Examples of how this manifested itself included: More academic staff contacting TALL to discuss how they could use technology to deliver their programme, while preparing new course proposals; Colleagues from Registry directed academic staff to consult TALL when they began writing new course proposals; Divisional managers and the Department’s Financial Controller contacted the TALL CoManager (Administration) to discuss online elements for new courses so that they could be fully costed and included in the budget for the new programme. Lastly, it seems clear that the experience of basic VLE support has encouraged individuals in the Department to consider greater use of technology in future courses. 5.3 Learning as a result of the project During the end-of-project evaluation workshop, the project team discussed how much they had learned during the Cascade project, and how much they were still learning as the services developed are embedded in the Department’s work. The project team grouped the key lessons learned under five headings: (1) project scoping; (2) project management, (3) innovation, (4) engagement and (5) embedding. Some examples of this learning are described in earlier sections of this Evaluation Report, for example, the difficulties of convincing stakeholders on new technology adoption (see Section 1.3) or the expectations of students towards technology innovation in the Department (see Section 2.5). A full description of the learning as a result of the project is provided in Section 3.2 of the Final Report.10 In addition, the project team produced five case studies11 to highlight key project activities and to share their learning in these areas with the wider e-Learning community. The service developments, the new system implementations and the cultural changes associated with the project will continue well beyond the official end of the Cascade project. Online enrolment and payment is being adopted by more and more courses, new courses are routinely starting to include technology elements in their course design, course websites will continue to support course delivery and the new online assignment-handling system will be fully rolled-out at the start of the 2011-12 academic year. 5.4 Tangible benefits Across the project, a number of tangible benefits were identified and these are described in Section 3.3 of the project’s Final Report. Quantitative data in several areas of tangible benefit were identified as part of the project’s evaluation activities and, in Table 5.1 below, data collected during the project has been applied to 2009-10 data held by the Department to give 10 11 Cascade Project Final Report: http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeFinalReport.doc Five case studies highlighting the learning as a result of the project are available from the project website at: http://cascade.conted.ox.ac.uk/project-outputs 29 some indicative figures of potential future efficiency gains resulting from the increased use of technology for curriculum delivery. Input variable = Potential annual saving 130 hours = 78 hours = 360 hours = £3,840 = 824 hours Cascade focus area improvement 260 assignments handled by Registry for award-bearing courses 390 extension requests handled by Registry for awardbearing courses 48 award-bearing courses x 48 award-bearing courses x 10,298 paper enrolments received by the Department in 200910 x x x 0.5 hours saved with new online assignment handling submission system and process vs. paper-based 0.2 hours saved with new online extension approval feature and process vs. paperbased 7.5 hours of photocopying time saved using a VLE to provide course resources £80 saved in printing costs by using a VLE to provide course resources 0.08 hours saved in course administration by online enrolment and payment process vs. paper-based Table 5.1: Potential annual savings from Cascade efficiency improvements Based on the evaluation data collected, potential annual efficiency gains in these areas include: 1. Online assignment handling administration time savings (up to 208 hours p.a.). 2. Increased course administration efficiencies through the use of a VLE to provide course resources (up to 360 hours p.a. of time saved photocopying and up to £3,840 reduction in photocopying costs). 3. Administration handling time savings from online enrolments (up to 824 hours p.a.). In total, in these three areas, there is potential to save up to 1,392 productive hours of administrative time, which is equivalent to over 38 weeks. Rather than spending this time on assignment handling and course enrolment, the Department intends to direct these resources towards more value-adding activities, such as development of new courses or the improvement of existing course offerings. 5.5 Conclusion: achievements against aims and objectives The original aims of Cascade were to enable the Department to: Undertake its activities more efficiently so that resources are focused on value-adding activities, such as delivering improvements to the student experience and the creation of tools that support best practice; Develop new, or repurpose existing, activities to support the delivery of its new vision and provide additional revenue streams; Support its ability to deliver academically superb courses to students of the highest calibre through the use of new tools and functionality to augment the services currently offered to students. 30 In the context of the Evaluation Report these aims were simplified into three headers: to increase efficiency, to innovate and to improve levels of service. In Table 5.2, we mapped each focus areas’ achievements against one or more of these three headers. Table 5.2: Matrix linking project outcomes to aims Table 5.2: Matrix linking project outcomes to aims For the online assignment-handling focus area, the project team only partially achieved the desired aims. From the evaluation analysis in Section 1, we see possibilities of efficiency and service improvements, but they have not yet fully materialised due to the delayed roll-out of the new assignment-handling system. In contrast, the combined VLE support for courses and online delivery of generic content focus areas achieved both aims, efficiency and service. Departmental staff are undertaking course set-up activities more efficiently and have more time to focus on value-adding activities. That is also reflected in the enhanced student experience described in Section 2 of this report. More Departmental courses have integrated technology elements in their course design and we therefore see the innovation aim of this focus area as achieved. Some of the achievements outlined in Section 3 were supported by a general trend towards technology adoption that the Cascade team supported by putting in place appropriate tools and support structures. In Section 4 of this report, administrative handling-time savings in the focus area of online enrolment and payment were clearly visible and will continue contributing greatly to the efficiency improvements. Due to the 24/7 availability of the system, and additional course administrator time available to deal with any non-standard enrolments, this area of work has also demonstrated an improvement in levels of service provided to students. 31 Appendices Double click on any table to see in original excel. Appendix 1.1: IT support time spent supporting online assignment submission (IT support team, 2007/08, 2008/09, 2009/10) Course Course Course Course Course Course Course TOTAL 1 2 3 4 5 6 7 Course Course Course Course Course Course Course Course Course TOTAL 1 2 3 4 5 6 7 8 9 Course Course Course Course Course Course Course Course Course TOTAL 1 2 3 4 5 6 7 8 9 Aug-07 120 45 Sep-07 60 30 Oct-07 Nov-07 20 90 30 110 Dec-07 15 55 Jan-08 80 20 15 50 15 30 10 20 110 100 180 50 180 155 170 170 420 35 205 50 380 Aug-08 40 10 5 Sep-08 10 5 30 Oct-08 15 10 20 100 150 Nov-08 25 140 15 110 0 0 175 155 85 465 Dec-08 25 25 115 30 0 15 35 0 0 245 20 40 65 10 115 75 55 5 10 Aug-09 10 50 40 20 0 10 15 75 10 145 Sep-09 30 15 35 0 0 0 50 120 25 130 Oct-09 20 35 10 75 0 0 25 35 10 165 Nov-09 60 195 35 105 0 0 335 120 0 730 Dec-09 80 95 10 60 0 15 45 5 30 305 10 Feb-08 30 Mar-08 110 55 25 90 80 Apr-08 70 90 55 65 15 May 08 95 30 30 240 20 380 110 405 Jan-09 90 70 30 10 0 30 265 170 20 495 Feb-09 30 80 10 60 0 80 45 65 30 305 Mar-09 50 200 110 90 0 10 130 30 5 590 Jan-10 60 70 10 0 0 125 165 140 10 430 Feb-10 Mar-10 60 70 10 40 0 80 150 0 30 410 105 75 5 135 5 50 0 165 90 70 20 450 25 195 Jun-08 10 20 15 10 10 30 10 105 Jul-08 Total mins Total hrs 135 745 12.42 15 480 8.00 25 265 4.42 15 765 12.75 15 305 5.08 30 0.50 10 460 7.67 215 3050 50.83 Apr-09 10 125 10 15 0 135 150 145 10 445 May 09 50 140 30 65 0 30 95 0 35 410 Jun-09 125 70 40 0 0 0 55 110 25 290 Jul-09 Total mins Total hrs 145 615 10.25 20 895 14.92 10 425 7.08 0 480 8.00 0 10 0.17 60 360 6.00 0 965 16.08 95 915 15.25 15 340 5.67 235 3750 62.50 Apr-10 90 35 0 30 0 140 30 40 10 325 May-10 40 30 20 30 0 135 50 25 20 305 Jun-10 20 15 55 0 0 55 55 105 0 200 Jul-10 Total mins Total hrs 105 580 9.67 105 850 14.17 40 270 4.50 0 410 6.83 0 0 0.00 70 795 13.25 5 1015 16.92 30 765 12.75 25 190 3.17 325 3920 65.33 35 10 32 Appendix 1.2: Administration handling time for assignment submission (Registry staff, Apr-10 for legacy system & Nov-10 for new system) Manual assignment handling Time Who During course set-up Assignment set-up Prior to due date Check if there are any extensions that affect the due date At time of submission Preparation for marking (getting script to marker) After marking Preparation for moderation (getting scripts to moderator) Receipt of submissions (issuing receipt) Photocopying scripts Stuffing envelopes Posting Time taken Registry Assistant 5 minutes Covered below Reception / Registry Assistant Comments Who Time taken New online assignement handling system Who Time taken Comments Registry Assistant 10 minutes Registry setting up groups within Moodle and mark sheet Not Registry, Mike? 5 minutes N/A N/A Covered below N/A N/A N/A N/A N/A Seconds (if reception is asked by student for signature) Students write their own receipts at reception and they are not issued if work is posted direct to the Registry N/A N/A N/A N/A N/A Automatic email notification N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Registry Assistant 15 Minutes Record marks on mark sheet Registry Assistant 15 minutes Record marks on mark sheet N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Registry Assistant 10 minutes Files are generally sent to External as a zipped attachment outside of CASS Registry Assistant 20 minutes Had to download comments and assignments in order to zip email to moderator Registry Assistant 10 minutes Registry Assistant 10 minutes Registry Assistant 10 minutes N/A N/A Registry Assistant 15 minutes Central ? Includes time to record marks Depends on number of assignments Registry Assistant 15 minutes Photocopying scripts Registry Assistant 10 minutes Stuffing envelopes Registry Assistant 10 minutes Central ? Receipt of moderated scripts Registry Assistant 10 minutes Returning marked scripts to students Registry Assistant 30 minutes Depends on number of assignments Registry Assistant 10 minutes Notifying students of marks Registry Assistant N/A Done by return of work N/A Done by return of work Receipt of request Registry Assistant 2 minutes Depends how request is received Registry Assistant 2 minutes Passing request on to decision maker Registry Assistant 5 minutes - 30 minutes Depends how request is received and whether it needs additional authorisation. Registry Assistant if more than 2 weeks. Receipt of decision Registry Assistant 2 minutes Depends how decision is received Registry Assistant 2 minutes Verification of decision Registry Assistant / Registry Officer 2 minutes N/A N/A Notifying student of decision Registry Assistant 5 minutes N/A Notifying tutor and other course personnel of decision Registry Assistant 5 minutes N/A Student notification Approximate Subtotal Comments per assignment Receipt of marked scripts Posting After moderation Legacy online assignemnt handling system Task Also a lot of work is returned via Registers 95mins Depends on number of assignments * See separate note in additional comments N/A 50mins Depends on number of assignments, still need to check comments Done by return of Did not have a realistic work test 65mins Registry Assistant 2 minutes Registry Assistant if more than 2 weeks or above permitted number of extensions 5 minutes - 30 minutes if more than 2 weeks Automatic email notification Registry Assistant 2 minutes Record extension on mark sheet N/A N/A N/A N/A N/A N/A Registry Assistant 2 minutes Write note to student N/A N/A N/A N/A N/A 5 minutes - 30 Automatic unless minutes if more more than 2 weeks than 2 weeks. Extension requests Approximate Subtotal 21-46mins 9-34mins 11-36mins 33 Appendix 1.3: Summary of log data for online assignment submission pilot (VLE logs, British and European Studies programme, November 2010) Full name Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 Student 7 Student 8 Student 9 Student 10 Grand Total Number of page accesses for uploading assignment files 4 6 4 8 7 6 4 5 6 4 54 Number of Number of Number of uploads of assignment assignment extension file uploads and extension request views additions support files Grand Total 31 4 2 41 27 4 37 23 2 29 30 5 43 12 4 23 33 3 42 8 2 14 32 1 3 41 16 3 25 10 2 16 222 5 30 311 34 Appendix 2.1: Overview of study programme links within course VLEs (VLE logs, November 2010) Programme Link to guidance on using site and technical help Link to course handbook Llink to library information website(s) MSc Sustainable Urban Development X PG Certificate in Historical Studies X X X Foundation Certificate in History 2009-11 X X X Foundation Certificate in History 2010-12 X X X Link to University computing information Link to University skills portal X Other links Dept website - information for students and University visitors and friends info X X Generic MSc induction and historical studies modules Graduate supervision system Study skills resources Sheffield MSc English Local History X X X X X Virtual training suite internet for history tutorial Programme specification (on dept website) Writing essays and report OU Open learn content University skills portal online courses Diploma in History of Art X X X X Materials from OUCS IT courses Information about Oxford (various links) Undergraduate Certificate in Archaeology 2010-2012, year 1 X X X X Undergraduate Diploma in Archaeology 2010-2011. X X X X Postgraduate Diploma in Psychodynamic Practice 20092011 year 2 X X Postgraduate Certificate in Psychodynamic Counselling X X Total In percentage 10 9 8 5 2 100% 90% 80% 50% 20% 35 Appendix 2.2: Student online survey results Q2; Q4 (Survey Monkey, November 2010) Ho w e a sy wa s the we b site to use ? Ho w use ful d id yo u find the co urse we b site ? Re sp o nse Pe rce nt Answe r Op tio ns Extremely easy Very easy Fairly easy Not very easy Not easy at all If you experienced problems using the site please describe these: Re sp o nse Co unt Answe r Op tio ns 2 6 9 0 0 1 11.8% 35.3% 52.9% 0.0% 0.0% a nswe re d q ue stio n skip p e d q ue stio n Numb e r 1 Extremely useful 2 Very useful 3 Of some use 4 Not very useful 5 Not useful at all 17 2 use Ho w e a sy wa s the we b site Iftoyo u ? e xp e rie nce d p ro b le ms using Re sp o nse D a te the site p le a se d e scrib e the se : Jun 28, 2010 1:47 PMRemembering username & password Couldn't find course marks Extremely easy Very easy Fairly easy Not very easy Not easy at all Re sp o nse Pe rce nt 5.9% 41.2% 41.2% 11.8% 0.0% Re sp o nse Co unt 1 7 7 2 0 a nswe re d q ue stio n skip p e d q ue stio n 17 2 Ho w use ful d id yo u find the co urse we b site ? 1 Extremely useful 2 Very useful 3 Of some use 4 Not very useful 5 Not useful at all 36 Appendix 4.1a: Administration handling time for paper-based enrolments (Manager of online weekly classes, April 2010) Steps Activity 1 Pick up form from fax machine or print from email 2 Open InfoSys 3 Go to student section within Infosys and start query with student's name 4 Evaluate query result If no student record exists, continue with step 5; otherwise go directly to step 7 5 Create new student record 6 Validate new student record and press 'confirm' >7 Register student for course 8 Process student payment 9 Send hard copy of enrolment and payment confirmation to student or send payment (soft copy) request in case of payment by invoice If payment type is by invoice, continue with step 10; otherwise go directly to step 13 10 Receive and file confirmation from finance that invoice was paid to the account 11 Enter payment confirmation in Infosys 12 Send hard copy of enrolment and payment confirmation to student > 13 14 In Infosys, process batch report per payment type Send batch reports to finance via fax or internal mail Sample 1 Sample 2 7 April 15.04 8 April 10.13 Sample 3 Sample 4 8 April 10.22 9 April 11.21 5 secs 5 secs 5 secs 5 secs 10 secs 10 secs 10 secs 10 secs 5 secs 5 secs 5 secs 5 secs No No Yes No 120 secs 240 secs 300 secs 5 secs 5 secs 5 secs 40 secs 40 secs 40 secs 40 secs 20 secs 20 secs 20 secs inv.employer 240 secs write email & attach invoice 240 secs 30 secs 30 secs 30 secs to 13 to 13 to 13 30 secs process with sample 3 30 secs 37 20 secs 20 secs Appendix 4.1b: Administration handling time for online enrolments with feedback loop (Manager of online weekly classes, April 2010) Steps 1 Activity Sample 1 Sample 2 Sample 3 Sample 4 Sample 5 Open email inbox 15 Apr 9.19 16 Apr 9.14 16 Apr 9.36 16 Apr 13.54 19 Apr 10.24 120 secs 60 secs 300 secs 30 secs 60 secs 600 secs 120 secs 120 secs If email states 'online enrolment successfull', continue with step 2; otherwise go directly to step 4 2 Right mouse click on email with "online enrolment successful" and mark as read 3 Move email into respective folder >4 5 Read through student email with online enrolment problem Write back email response with request for more information 180 secs 6 Read student's response to information request 60 secs Repeat steps 5 and 6 where necessary 7 Formulate email with instructions how to solve enrolment problem and send to student Once the problem has been resolved, the online enrolment process is normally concluded with step 1, 2 and 3. Please record start and end date + time of end-to-end process >>> 120 secs 120 secs paper ernolment received 19 Apr 10.30 paper ernolment received 19 Apr 16.10 38