Cascade Project: Evaluation Report
DRAFT v4
December 2010
Henriette Lundgren
Cascade Project Evaluator
henriette.lundgren@conted.ox.ac.uk
+44 (0) 7511 096672
Marion Manton
Project Manager
marion.manton@conted.ox.ac.uk
+44 (0) 1865 280986
Bridget Lewis
Nicola Warren
Table of contents
Evaluation summary and recommendations ...................................................................... 3
Background, aims and structure of this report .................................................................... 5
1
2
3
4
5
Focus area 1: Online assignment handling ................................................................ 6
1.1
IT support time ................................................................................................... 7
1.2
Administration handling time .............................................................................. 7
1.3
Customer satisfaction ......................................................................................... 8
1.4
Ease of use ........................................................................................................ 9
1.5
Conclusion: new system improvements on their way ........................................10
Focus areas 2 and 3: VLE support and generic content ............................................10
2.1
Overall adoption rate .........................................................................................12
2.2
User friendliness (administrators) ......................................................................13
2.3
Usage rate (students) ........................................................................................14
2.4
Customer satisfaction (students) .......................................................................16
2.5
User acceptance (academics) ...........................................................................17
2.6
IT set-up and support time ................................................................................18
2.7
Conclusion: high acceptance of VLE support for courses ..................................18
Focus area 4: Course design ....................................................................................19
3.1
Technology engagement ...................................................................................20
3.2
Conclusion: technology use in course design increases ....................................21
Focus area 5: Online enrolment and payment...........................................................21
4.1
Administration handling time .............................................................................22
4.2
Adoption rate.....................................................................................................23
4.3
Customer satisfaction ........................................................................................26
4.4
Conclusion: growing number of online enrolments – less handling time ............27
Overall project evaluation..........................................................................................28
5.1
Project management .........................................................................................28
5.2
Stakeholder engagement ..................................................................................29
5.3
Learning as a result of the project .....................................................................29
5.4
Tangible benefits ...............................................................................................29
5.5
Conclusion: achievements against aims and objectives ....................................30
Appendices.......................................................................................................................32
2
Evaluation summary and recommendations
The Cascade project evaluation was conducted in three steps. First, an Evaluation Plan was
agreed (April 2010); next baseline data was collected and presented in the Baseline Report
(June 2010). Finally, a second quantitative and qualitative data set was gathered (November
2010) to close the evaluation loop.
For the online assignment handling area, evaluation data was drawn partially from the new
as well as from the legacy system. Data from pilot studies were collected as the new and
improved online assignment handling system will not be fully rolled out until the academic
year 2011-12. Our preliminary time-motion analysis of the new system has shown significant
time savings in comparison to paper-based assignment handling, but the handling efficiency
still lags behind the legacy online assignment handling system where routine use has
decreased administration time. In terms of student adoption and satisfaction, the new system
has received positive responses during the pilots. Further efficiency and service
improvements are anticipated once the new assignment handling system is fully
implemented.
VLE support for courses as well as online delivery of generic content have both been
well received and adopted by members of the Department. This is especially true in the area
of award-bearing courses where the take up of the VLE resources has been good. A
significant challenge will be supporting staff and students as numbers increase. However,
tools have been put in place to manage this issue, and there is cause to be optimistic about
the volume of IT support that will be required. It is anticipated that VLE supported courses,
with the integration of generic content, will be an increasingly important tool for administrative
staff, tutors, academics and students. In these focus areas, we see full achievements on both
aims: efficiency and service. As Departmental staff become more familiar with the VLE and
are able to set up their courses more efficiently they will focus more of their attention on other
value-adding course activities.
Technology use in course design has increased over the last two years. This trend can be
in large part directly attributed to the Cascade project work as the project has supported a
general trend of technology engagement and confidence within the Department. All new
programmes coming through the Department’s approval process in the last nine months now
plan at least basic use of the VLE, and because of that we see the innovation aim of this
focus area as having been achieved.
Online enrolment and payment is now almost as popular as paper-based enrolment. This
reduces dramatically the administration handling time and error rate when compared to the
manual approach. In the case of a standard online enrolment, the time savings in
comparison to paper-based enrolment are immense. Where online enrolment requires an
extra feedback loop via email with the course administrator, administration handling time
savings still exist but are less significant. In terms of student satisfaction with the online
enrolment website, a consistently high level of customer satisfaction can be observed. In this
area, improved efficiency and service is clearly identifiable.
As a consequence of the evaluation analysis and conclusions, a number of
recommendations were established to guide the Department’s efforts beyond the end of the
project.
Recommendation on Online Assignment Handling System: The Department should drive
the full implementation of the new system further, including the proposed roll-out to all legacy
system users and new courses in 2011-12 and to all award-bearing courses subsequently.
3
Recommendation on VLE Implementations, including Generic Content: The Department
should focus on award-bearing courses where VLEs are used over a longer period of time
and therefore maximise the ‘shelf-life’ and return on investment.
Recommendation on VLE Improvements: Through the collection of feedback from all key
stakeholders – administrative staff, part-time tutors, students and full-time academic staff –
course VLEs can be integrated in cycles of continuous improvement. The provision of
additional support sessions to all staff when they meet as course teams is recommended to
embed the use of the self-service resources and documentation available from the project.
Recommendation on Course Design: The Department should continue to offer support to
academic staff, to assist them to plan how they will use technology effectively to deliver the
curriculum when they are designing new courses.
Recommendation on Online Enrolment and Payment: The Department will identify all
courses where online enrolment and payment could be successfully implemented i.e. for
those courses where the process can be fully automated. By doing so, the Department will
reduce the number of errors associated with manual processing, improve transaction times,
ensure the enrolment and payment service can be offered 24/7, while at the same time
generating significant financial value through the reduction in staffing costs associated with
automating the process.
Detailed findings in each focus area (Sections 1-4) and for the overall project (Section 5) are
presented in this report. Supporting evidence can be found in the Appendices.
4
Background, aims and structure of this report
The first step in the evaluation of the Cascade project was to devise an Evaluation Plan –
submitted to JISC in March 2010 – and to define the measures of success. In order to
construct this Evaluation Plan, all available Cascade documents, such as the original JISC
call, the project plan and activity plans for the five focus areas of the project were reviewed
and synthesised. Next, several informal meetings were conducted with the project team
members. Two days were spent identifying aims and key measures of success for the
individual focus areas. Finally, an evaluation plan matrix was agreed, which outlined the
various evaluation questions, activities, and suggested data-collection methods.
The second step was to take a snapshot of the initial status of the project to establish a
baseline. The goal of the resulting Baseline Report – submitted to JISC in June 2010 – was
to provide a sound basis on which the success of the project could later be evaluated. The
baseline was established following the same structure as the Evaluation Plan; starting with
focus area 1 on online assignment handling and concluding with focus area 5 on online
enrolment and payment. Where possible, data collection, analysis, and interpretation was
separated into three distinct sub-sections. The Baseline Report focused entirely on the
description of baseline information and did not include any summative evaluation on the
progress of the project.
The third and final step in the evaluation process was to summarise our findings in this
Evaluation Report. This report draws from various data sources and meetings and an
evaluation workshop that took place on 10 November 2010 with the project team. In this
report, we aim to compare the baseline data to the data that was collected throughout the
project. As well as reporting on the various focus and evaluation areas, we also aimed to link
each area back to the core project aims, i.e. to increase efficiency, to innovate and to
improve levels of service:
Aims
Efficiency
Innovation
Service
Focus Areas
Online
Assignment
Handling
VLE
Support for
Courses
Online
Delivery of
Generic
Content
Course
Design
Online
Payment &
Enrolment
Evaluation Areas
Admin
handling
time
IT set-up and
support time
User
acceptance
Adoption
rate
Ease of use
Usage rate
and areas
Technology
engagement
Technology
confidence
Technology
best
practice
Service
response
time
Customer
satisfaction
Figure 1.1: Pyramid of project aims in focus and evaluation areas
(Source: Cascade Evaluation Plan)
The Evaluation Report will conclude with a section on overall project evaluation and lessons
learned during the project.
5
1 Focus area 1: Online assignment handling
Online assignment handling was recognised early on as a high priority for the project, due to
the clear benefits it offered the Department both in terms of time and cost savings and in the
potential to offer an improved service to students. This was also recognised at the wider
University level where there was a parallel pilot of online assignment handling, with which we
collaborated to identify areas of common practice. Within the Department, a legacy online
assignment submission system (referred to as ‘CASS’) was being used by a few
programmes. However this had considerable functionality issues and needed replacing, and
as such was not scalable for wider adoption across the full range of the Department's course
offerings.
Initially, the plan was to have the new online assignment-handling system available to all
courses that wanted to use it for the start of the 2010-11 academic year. However, the
launch of the new system was delayed to allow more time for extensive testing and piloting.
All those staff involved in the piloting of the system have opted to adopt the system. We are
now offering the opportunity for a number of courses to trial the system for the remainder of
the current academic year, and we anticipate a very significant adoption rate for the start of
the 2011-12 academic year.
Consequently, an evaluation of this focus area will be done using the following structure:
#
Evaluation area
Main
stakeholder(s)
Evaluation feasibility check
Section in
Evaluation
Report
1.1
IT support time
TALL IT
support team
Only a small amount of pilot
data available; reference to
legacy system
1.1
1.2
Administrationhandling time
Registry
Only small amount of pilot data
available; comparison paper vs
CASS vs new assignmenthandling system
1.2
1.3
Adoption rate
Department
Summative data not available –
indicative data discussed in
focus area conclusion
1.5
1.4
Customer-service
response time
Students
Data not available to evaluate
NA
1.5
Customer
satisfaction
Tutors,
Registry and
students
Only small amount of pilot data
available
1.3
1.6
Ease of use
Students
Only small amount of pilot data
available
1.4
Table 1.1: Overview of evaluation data for Focus area 1
As the new online assignment-handling system was not fully implemented at the time of the
second data collection, only four of the six evaluation areas have been included in this report.
6
1.1
IT support time
In our Baseline Report, we gave an overview of the IT support time it took to support users of
the Department’s legacy online assignment-submission system during the period August
2007 to July 2009. The main data we referred to came from an Excel spreadsheet held in the
Department that detailed the amount of time spent providing IT support to students,
administrative staff and academic staff to support their use of the old online submission
system.
Ideally, we would have liked to collect the same data for the new online assignment-handling
system in order to compare IT support times. However, as the new system was not available
as a production service during the evaluation period, no directly comparable data could be
collected. We therefore decided to continue reporting IT support data on the legacy system
instead:
Total IT support time (CASS)
Number of programmes supported
Baseline
2007/08
3,050min
7
Baseline
2008/09
5,005min
9
New data
2009/10
4,875min
8
Table 1.2: IT support time CASS
For more detail on the reported IT support time, please see Appendix 1.1.
As Table 1.2 shows, the total IT support time for the legacy system rose from 3,050min in
2007/08 to 5,005min in 2008/09. This increase in support time was mainly due to an
increased number of courses being set up and supported. In comparison, the total support
time in 2009/10 decreased again to 4,875min as a consequence of the reduction in the
number of courses supported.
IT support time was also tracked for the new online assignment-handling system during pilot
studies. Apart from bug reporting by teams such as the Registry etc. there were no more
than two calls to the staff providing IT support for the new system. This meant that the vast
majority of academics and students were able to use the system without any issues; a major
improvement on the previous system which, despite familiarity with the system still requires
significant amounts of IT support, as shown above. While this might be taken to be extremely
encouraging, the IT Manager cautions on projecting too optimistically considering the
relatively small numbers who have used the system thus far. We do know that we have
designed out key problem areas in the previous system. However, the IT Manager still
suggests, from experience, that there may be unexpected issues that will only become
evident as the service is rolled out to more programmes.
1.2
Administration handling time
We also gave an overview of the current administration handling time for paper-based vs. the
legacy online assignment-handling system in our Baseline Report. We then referred to two
time motion studies that depicted average administration-handling time for each method.
Based on data collected during a pilot of the new online assignment system with the British
and European Studies course, a third flowchart was drawn up and validated by the
Department’s Registry in November 2010.
Assignment handling process steps
Baseline
Paper-based
Course assignment set up
Preparation for marking
5min
15min
Baseline
New data
Old system New
system
5min
10min
None
None
7
Preparation for moderation
45min
Student notification
30min
Total end-to-end assignment handling 95min
35min1
10min
50min
45min
10min
65min
Table 1.3: Administration handling time for assignment submission
Extension approval process
Baseline
Paper-based
Receipt of request
Passing request on to decision maker2
Receipt of decision
Verification and notification of decision
Total end-to-end extension approval
2min
5-30min
2min
12min
21-46min
Baseline
New data
Old system New
system
2min
2min
5-30min
5-30min
2min
2min
None
2min
9-34min
11-36min
Table 1.4: Administration handling time for extension approval
For more detail on the reported administration handling time, please see Appendix 1.2.
The time motion studies are split into two parts: With the new online assignment-submission
system, the administration handling process takes 30 minutes less in assignment submission
and marking (Table 1.3) and 10 minutes less for the extension approval process (Table 1.4),
in comparison to the paper-based process. These time savings, taken from initial time motion
studies in conjunction with pilot testing, initially seem to be lower than the time savings when
we compare the old assignment-handling system vs. paper-based process. However, the
reason behind the somewhat conservative time savings attributed to the new system is
explained, in part, by the familiarity that the Registry staff have with the existing system,
coupled with the fact that figures for the new system have been collected when using it in
piloting conditions, where things cannot be expected to run as smoothly as in an established
service. More specifically, the data is also skewed by the longer moderation time taken as a
result of issues with the file compression (zip) and download system encountered during the
pilot study. These features have now been resolved in the new system. Overall, there is
extremely good reason to believe that when this process is repeated with the final release of
new assignment-handling system, time savings will be greater than those observed with the
legacy system.
Other advantages of the new online vs. paper-based systems are: the reduction in paper
handling and photocopying as more courses use the system; better auditing and control; and
improvements in assignment transaction storage. Reduction in paper storage is a further
advantage both in terms of less physical space being required, and also in terms of less staff
time being required to retrieve data from the archive.
In summary, additional testing is planned to conclude the formative evaluation of the new
online assignment-handling system, and we will then provide a summative statement on the
progress of this evaluation area.
1.3
Customer satisfaction
Key stakeholders of assignment-handling systems are students who submit their
assignments and academics who receive or download the students’ submissions, mark them
1
2
This number was adjusted by Registry from 20min to 35min after the initial baseline data was collected in April 2010.
The amount of time taken to pass the request to the decision maker depends on how the request is received and whether
additional authorisation needs to be granted.
8
and then notify the students of their grades. We chose two data collection methods in this
area to seek feedback from key stakeholders. For the Baseline Report, we first analysed an
online survey that involved students, administrative staff and academics from across the
Department. The results of this online questionnaire, with 57 respondents from different
groups and programmes, depict a positive attitude towards the use of technology in
assignment submission and the implementation of a new online assignment-handling
system.
We analyzed email feedback from users of the legacy system to draw a qualitative baseline.
We concluded that, when academic staff experienced problems, they often associated them
with the online technology rather than the underlying administrative processes, despite the
fact the latter were often the cause. This highlighted the importance of ensuring that
technology is well integrated in robust administrative processes in order to maximise user
satisfaction.
Comparing our baseline information to initial feedback from the new system pilots, the
following conclusions can be drawn. Tutors, the Registry and students have all pointed out
bugs in the system which have been, or are being, addressed. Modifications to the user
instructions have also been made to respond to the issues raised. The most common
underlying reason behind issues encountered when using the system was that users had not
read the instructions. A few users who had previously used the legacy system took some
time to adjust to the new system. There were no negative comments as a result of the
change from one system to another and a number of users commented that they felt the new
system was an improvement.
The following quotes from two students and one Course Director demonstrate levels of
customer satisfaction with the new online assignment submission system after the initial pilot:
"As far as I can see, everything has worked ok. I got all the responses I was expecting
and the processes were simple to follow." (Student 1)
"I suppose it's because I've learnt to live with CASS that I found this took some getting
used to. I can see though that it's an improvement, and I like the 'control panel' where
you can see everything that's gone through at a glance." (Student 2)
“Our students found the online system straightforward and easy to use both for essay
submission and to request extensions.” (Course Director 1)
“System navigation was easy and intuitive. It was simple to download student essays
and to provide feedback. In fact, it was much more user-friendly than initially
anticipated.” (Course Director 1)
These four quotes indicate the positive attitude towards the new online assignment
submission system. New technology takes some time to settle; however, we have learned
that ease of use and the enhancement of functionality were key to improving customer
satisfaction levels for the new online assignment-handling system.
1.4
Ease of use
Finally, in our Baseline Report, we looked at the ease of use of the current online
assignment-submission system. The legacy system log information was hard to extract and
interpret, so we chose to analyze feedback provided by the IT support manager. Four main
system issues were reported. These were accessing the system, uploading multiple files,
requesting extensions,; and email retrieval by marking tutors. We concluded that the legacy
9
system was missing essential functionality that would enhance the ease of use in online
assignment submission.
All of the issues outlined above have been addressed in the design of the new online
assignment-handling system. Students using the system during our Statistics for Health
Researchers pilot study used a dummy assignment to practice uploading files, and amending
and deleting them as required. This tested the enhanced functionality and introduced the
students to the look and feel of the new system before they were asked to use it to upload
real assignments. Of the 17 students on the course, 11 viewed the practice assignment and
seven practiced uploading files. This suggests that inclusion of a practice assignment is
useful. Also, there were no forum postings relating to problems with assignment submission.
For the British and European Studies pilot, we also scanned the log files recorded in the
course VLE to see if there was any evidence of students experiencing difficultly uploading
assignments and using the system as per the pilot instruction. The analysis showed that
students uploaded the assignments without difficulties and also tested the system
successfully for requesting extensions. For more details on the pilot study analysis, please
see Appendix 1.3. Further testing will be required prior to a summative statement on the
progress of this evaluation area being made available.
1.5
Conclusion: new system improvements on their way
The Cascade team worked successfully to get the development of the new online
assignment-handling system to its current position. However, to evaluate this entire focus
area, we could only collect data from a few pilot studies as the full release of the new system
will not be until the beginning of the 2011-12 academic year. As the system is not fully
implemented, we could only report formative evaluation data in this section. Our initial
administration-handling time analysis has shown significant time savings in comparison to
paper-based assignment handling, but the handling efficiency still lags behind the legacy
online assignment-handling system where familiarity and routine use has decreased
administration time. For the new online assignment-handling system, the potential timesaving
should be greater in the long term, but the initial pilot data and somewhat conservative time
estimates from the Registry do not yet show this efficiency gain. In terms of student adoption
and satisfaction with the new system, positive signals have been received during the pilots.
Many users thought the new system was an improvement in comparison to the old online
assignment-handling system. Also course directors who were involved in the system testing
were positive and intend to continue to use the system following their initial pilot use.
Recommendation on Online Assignment-Handling: The Department should go ahead
with the full implementation of the new online assignment-handling system, including the
proposed roll-out to all legacy system users and new courses in 2011-12 and to all awardbearing courses subsequently.
2 Focus areas 2 and 3: VLE support and generic content
In recent years, the Department has offered VLE support for courses in three main areas:
fully-online courses, blended learning and to support largely face-to-face learning. However,
while the first area is well supported and developed, use of a VLE to underpin the other two
categories has generally taken place on an ad hoc basis and was only used on a small
minority of the programmes that might benefit from this provision. Addressing this in a more
systematic fashion was always intended to be a focus of the project. However, identifying
which approaches to take was a significant challenge. Of the original 21 sub-themes, 15
were related to this area and much of the process of refining the scope of the project
focussed on prioritising work in this area.
10
The pilot studies gave the project team useful insight into how VLEs were being used and
how well they were received by both students and staff. The course teams were very
positive, despite having to spend time learning how to upload and edit content. They found
the Moodle environment very useful for communicating with their students, on both
administrative and academic matters, and for streamlining access to information, which
resulted in a reduction in the need for photocopying. All course teams that took part in the
pilot studies are now implementing VLE support more widely across course offerings in their
subject areas, including supporting new audiences such as alumni. Initial student responses
to the VLE support provided for their courses were broadly positive, although some students
lacked confidence in their skills with technology. This will be partially addressed through the
materials provided in the generic content pilot study that was closely interlinked with the VLE
developments.
One finding from our initial stakeholder engagement activities and the review of new course
proposals was the inefficiency inherent in the amount of generic content created and
recreated independently by different course teams across the Department. This issue was
evident both in administrative areas, such as that covered by our course handbooks, and in
training in what might be called 'academic literacies', especially in areas such as basic study
skills and library and information skills. It was recognised that, in many cases, this
information was delivered at the start of a course when students were not fully cognizant of
its importance. It was the case, therefore, that when students needed a specific skill, they
often required retraining. This created a demand for reusable online content addressing
these areas, accessible by students at their point of need. The Cascade project team worked
with library staff and with course teams to identify the topics that would be most valuable to
develop. We also sought to identify whether there was existing content within the
Department, the University or the wider HE sector, which could fulfil this need but whose
existence was not necessarily known to either students or tutors.
From our pilots and wider implementation experience this academic year, it is now clear that
the materials most frequently included in a VLE by staff have been those that are more
administrative in nature. All new courses have included course handbook materials and links
to updated library, technical guidance and support information provided through the website;
the more “academic” content has seen more varied uptake by course teams. However, the
extremely low cost of sustaining these links once set up makes provision of them worthwhile.
For this Evaluation Report, the two focus areas of VLE support for courses and online
delivery of generic content will be evaluated in one section as outlined in Table 2.1:
Main
stakeholders
Evaluation area
#
Section in
Evaluation
Report
All
Adoption rate of VLE and generic
content
2.3
2.1
Administrators
User friendliness
2.1
2.2
Students
Usage rate
2.4 / 3.3
2.3
Students
Customer satisfaction
2.5 / 3.1
2.4
Academics
User acceptance
2.6
2.5
TALL support team
IT set up and support time
2.2
2.6
Table 2.1: Overview of combined evaluation data for Focus Areas 2 and 3
11
During the evaluation data analysis, we found that these two focus areas were closely
related, and that evaluation areas had great overlap. We therefore re-organised evaluation
areas around the main stakeholders, i.e. administrators, students, academics and the TALL
support team.
2.1
Overall adoption rate
The first area on VLE support for courses looks at adoption rates. To establish a baseline,
we simply counted the number of courses that were using the VLE in April 2010. For the
Evaluation Report, the project team decided to focus on award-bearing courses, which can
be split into two categories:
PIP award-bearing
CPD award-bearing
Total
Baseline
April 2010
8
5
13
New data
November 2010
18
8
26
Table 2.2: Number of VLE supported award-bearing courses3
In April 2010, there were eight award-bearing programmes using a VLE in the Department’s
Public and International Programmes (PIP) division and five in the Continuing Professional
Development (CPD) division. This baseline compares to 18 in PIP and eight in CPD during
our second data collection point in November 2010: a doubling of courses using a VLE.
In addition to the 26 award-bearing courses using a VLE listed above, two further courses
were using WebLearn (the VLE used by the University’s Medical Sciences division) and four
courses were delivered fully online and have not been counted in the table above. As such,
out of a total of 48 award-bearing course cohorts which could use a VLE in some capacity,
there now remain only 17 which do not offer VLE support for their programme.
In terms of overall adoption of generic content, links to the following generic content were
found in ten analysed award-bearing courses that are supported with a VLE:





Link to guidance on using site and technical help (10 courses)
Link to course handbook (9 courses)
Link to library information websites (8 courses)
Link to University Skills Portal (5 courses)
Link to University computing information (2 courses)
For more detail on the reported generic content links in course VLEs, please see Appendix
2.1.
The data show that the majority of course VLEs link to at least four generic study resources,
such as the technical help page, the course handbook, the library information website and
the University Skills Portal. The analysis also showed that a few courses linked to
Departmental or University of Oxford generic content (e.g. a series of 14 transferable skills
online training courses available on the University’s Skills Portal site) that many were
previously unaware of. We also found that more course teams than we would have predicted
linked to external study skills content (e.g. study skills information at the University of
Sheffield and The Open University).
3
Where a course programme runs for two or more years and has two year groups of students in parallel, separate course
VLEs are set up for each cohort and are counted as separate courses in this table.
12
2.2
User friendliness (administrators)
In our Baseline Report, we wrote that the VLE should be perceived as easy to set up as well
as reliable and simple to use. For the data collection method, we decided on a qualitative
approach and conducted semi-structured interviews with individual course administrators
who had agreed to participate in the project. In these interviews, staff were asked about their
experiences before and after the implementation of the Cascade project VLE support for
courses enhancements.
As we had decided to collect this data in a single attempt, qualitative interviews took place
only after the pilot implementations in June 2010. In total, three interviews were conducted:
Interview
Interview 1
Interview 2
Interview 3
Course manager
Weekly classes
Archaeology
Psychodynamic Counselling
Date of interview
17 June 2010
16 June 2010
8 June 2010
Table 2.3: Interviews conducted with Course Managers, June 2010
A summary of theses interviews can be found below:
Interview 1: “The VLE is a great place to send people to when they make repetitive
enquiries about the same thing,” states the weekly classes Course Manager in our interview.
As a consequence, the Course Manager thinks that the number of enquiries has been
significantly reduced due to the VLE support. Generally the weekly classes are already trying
to move many processes into soft copy, so this has just continued that process. However, he
still thinks they will be sending out information in letter form for a while as, in the past, the
suggestion that all communication will be electronic has resulted in part-time tutors asking for
Departmental email accounts. That has disproportionate resource implications and so is not
something they want to encourage. The Course Manager thought that the VLE saved time
for the class administrator and that the VLE was very easy to use. He did not encounter any
technical problems and he reported that other staff around him found the VLE as support for
weekly classes equally useful. The interviewee plans to further develop support provided via
the VLE in the 2010-11 academic year.
Interview 2: The benefits from VLE-supported courses are that communicating with students
is made easy and very efficient as the course administrator can contact everyone in one go.
“This was especially useful in an emergency like the snow in early 2010” she explained. The
VLE is also good for providing information on aspects of the course that were not known at
the start of the programme and therefore were not in the handbook, for example, sharing
materials such as maps, plans, diagrams, lists etc. before a session. Thus, a benefit is
information dissemination and knowing that all students have access to it. However, the VLE
is not so much adopted by the archaeology tutors: neither the common room has been used
nor the tutor forums as tutors did not engage. Tutors are used to working the old way and
find it hard to change. This is something best introduced at the start of a new programme and
more likely to work if built in from start, with useful mandatory activities taking place online.
Currently, tutors still photocopy articles for students and only once did a tutor ask to put an
article on the VLE. This shows that tutors might not have adopted the new VLE support
structure yet. In terms of efficiency, the VLE took time at the start but definitely saved time
once it got going. The Course Manager is confident that the VLE will save time in the future.
Technical problems were found around logging in and remembering passwords. The Course
Manager thinks that course tutors could adopt the new system more and that they are still
being quite oblivious to its advantages. Several months into the pilot the Director of Studies
emailed students rather than using the forums and then continued dialogue via email, despite
the VLE as the common communication platform being in place.
13
Interview 3: For the third Course Manager, the News forum and the ability to provide articles
online rather than photocopying these for the class were perceived as most useful in the
context of VLE support for her courses. The biggest benefit was that the VLE saved time in
course administration. The administrator estimated that she has saved in excess of 10 hours
of photocopying. This is a conservative estimate, and she would expect it to increase further
next year. “For part-time administrators, 20-30 minutes a week copying less in term time can
be noticeable” explained the interviewee. Other time savings come from the News forum
being a quick and efficient method of communication: there have been fewer requests for
resending information than before because someone has lost an email. However, the biggest
challenge is explaining the benefit and use of the VLE to tutors. As a matter of fact, tutors
barely used the VLE and only saw it as an administrative rather than teaching tool. They also
found it very hard to remember passwords. One reason for this “lack of interest” might be
that the tutors in psychodynamic counselling are mostly very occasional teachers who are
only contracted to teach one or two sessions. The administrator reasons that for these tutors,
adoption of total VLE support cannot really be expected. Once they had spent some time
with the VLE, however, they seemed to show more appreciation and adoption. In conclusion,
the interviewee would definitely recommend the VLE as support for courses to others, and
she would be prepared to act as a mentor to others. The Course Manager also stressed the
importance of a single sign on (SSO) login for the course VLE in order for students and tutors
to have fewer passwords to remember and therefore to increase the adoption of the VLE.
Although all three Course Managers refer to different aspects of their course VLE set-up and
use, a positive attitude towards the new course platform can be observed.
2.3
Usage rate (students)
In the Baseline Report, we had focused on expected usage rate of weekly classes tutors,
where we were positively surprised by the high technology usage rate of our 55+ target
group surveyed in December 2009. For the Evaluation Report, we therefore focus on
students (rather than tutors) to get a better picture on how much they had used both the VLE
and the various generic content sources. We analysed the following two data sources:
1. VLE technical log data
2. VLE usage and activity
A summary of VLE log data, shown in Table 2.4, gives a list of generic content included in
the course VLE and the total number of times each resource was viewed:
14
Count of Full name
Information
All Archaeology courses offered by the Department
Assignments
Course Staff
General Reading List
Group Projects - individual topics
How to complete your profile
How to use discussion forums
Introduction
Library guide for Archaeology
Markers and Deadlines
Online etiquette
Schedule of Teaching Sessions 2009-2010
Submission of Assignments and Practical Logbooks
Summary of Course requirements for a One-Year Module
Tools to support group projects
Training Excavation and Practical Weekends
Tutorials
Useful Telephone Numbers and Contacts
Writing Assignments
Grand Total
Total
25
51
37
22
31
31
3
19
11
26
5
27
12
30
35
37
20
22
7
451
Table 2.4: Summary of total number of page views for the
Undergraduate Diploma in Archaeology VLE site, November 2010
The logs for the Undergraduate Diploma in Archaeology programme VLE reveal that none of
the features we included as generic content had been ignored. However, as the templates
(with their generic content) have only recently been consolidated into their almost-final
format, it is difficult to draw conclusions about the items we have included there.
For the VLE usage and activity data (see Table 2.5), award-bearing courses that used the
VLE templates developed by the Cascade project to establish a VLE to support the delivery
of their 2010-11 course were analysed in terms of a) student and staff log-in percentage, b)
actions since course launch and c) percentage of days with activity in the VLE. All data was
taken on 22 November 2010, and all courses, unless otherwise mentioned, were launched in
the second week of October 2010:
Course
MSc Sustainable Urban
Development
PG Certificate in Historical
Studies
Foundation Certificate in
History 2009-2011
Foundation Certificate in
History 2010-2012
MSc English Local History
Undergraduate Diploma in
the History of Art
Undergraduate Certificate in
Archaeology 2010-2012
Undergraduate Certificate in
Archaeology 2009 -2011
Undergraduate Diploma in
Archaeology 2010-2011
Percentage of
students and
staff logging in
100%
Actions since
course launch
5,126
Percentage of
days with activity
in the VLE
100%
100%
12,954
100%
60%
326
52%
77%
605
75%
100%
60%
2,812
312
75%
54%
80%
1,308
90%
100%
1,370
70%
100%
1,488
92%
15
Course
Postgraduate Diploma in
Psychodynamic Practice
2009-2011
Postgraduate Certificate in
Psychodynamic Counselling
Percentage of
students and
staff logging in
100%
Actions since
course launch
100%
5,137
5,352
Percentage of
days with activity
in the VLE
Consistent use
since then
(percentage could
not be calculated
78%
Table 2.5: Usage rate and actions in existing VLEs, November 2010
The usage data shows that in most cases all students and staff have logged-in to their VLE.
It is worth noting that this figure should be treated with caution for many of the courses with
lower percentages as for many sessional tutors, the VLE has not yet become relevant as
they are only due to teach later in the course calendar. Across the majority of courses, the
VLE has been accessed a high percentage of the days it has been available, without any
downturn in the rate of access over time, suggesting steady and sustained use.
2.4
Customer satisfaction (students)
In this area of evaluation, we looked at the perceived customer satisfaction when using VLEsupported courses. In terms of the data collection method, we decided to analyse data
retrieved from various online surveys.
For our baseline, we analysed responses from 34 students, of which 14 students were from
Archaeology, 16 from Psychodynamic Counselling and 4 from Global Health studies. The
data was collected in mid and late 2009. In our baseline analysis we concluded that most
students had a positive attitude towards the introduction of a course VLE.
Between mid June and mid July 2010 we conducted a new online survey to enquire about
students’ customer satisfaction after the pilot studies. A total of 19 responses were collected
from three different study programmes: Archaeology (9), Psychodynamic Practice (5) and
Psychodynamic Counselling (5). We first asked the students how useful they found the
course website (Q2). 46.1% responded that they found the course website useful or
extremely useful, 41.2% said it was of some use and 11.8% found it not very useful. Next we
asked them: “How easy was the website to use?” (Q4). The response was similar to the
previous question: 47.1% found it very easy or extremely easy to use, 52.9% said it was
fairly easy to use; no one responded that the course website was not very easy or not very
easy at all.
Details of the responses given by students to Q2 and Q4 are outlined in Appendix 2.2.
In the next two questions, we asked students to rate different website features: which ones
they used (Q5) and how valuable they had found these features (Q6).
The features that were most used and also found most valuable were:




Course information
Administrative information relating to the course
Reading list and links to library catalogues
News forum for course announcements (sent by email)
16
These features echoed many of the areas identified by our administrators in Section 2.2.
Towards the end of the questionnaire (Q14), we enquired about the importance of having a
course website for future courses:
If you were to enrol on a future course how important would it be
to you that the course offered online support through a similar
course website?
1 Extremely important
2 Very important
3 Of some importance
4 Not very important
5 Not important at all
Figure 2.1: Importance of course website for future courses
The survey results show that 35.3% of students think that having a VLE site to support their
course is very or extremely important. The majority or students surveyed (41.2%) chose the
middle answer “of some importance”, which indicates acceptance if not enthusiasm. 23.5%
of students responded that having a VLE to support their course was not very important to
them. Thus, we could conclude that more than three-quarters of the respondents view the
VLE of some or higher importance.
2.5
User acceptance (academics)
In terms of user acceptance among academics, we looked at their perception of the VLE
templates that help generate an online presence for courses. For the data collection method,
we used feedback received after VLE demonstration workshops that we held with academic
and administrative staff:
“Indeed a very interesting and useful morning! I would be delighted to set up a Moodle
for the Diploma in the History of Art asap, a good trial run for the MSt. My feeling is I
am happy to base it on the template you have developed, this all looked grand, and it is
pliant enough that I may tinker at the edges easily should the mood take me.”
(Academic 1, Lecturer in the History of Art)
“I know a lot more and have more confidence. I have now established a skill set and
can broaden it. I would not have done a podcast in my most recent course if I would
have had to learn everything from scratch. The more one uses technology, the broader
and more imaginative one’s use is likely to become.” (Academic 2, Lecturer in Local
and Social History)
“The demo went very well. I gained the impression that the students just took it as a
norm that we could provide things like this, which was a shame because Tom and I
were so impressed with ourselves (and more so with you and Kiri since you actually did
the work!). There was no - wow you really are at the cutting edge of academic
technology!!! Some were keen to submit electronically.” (Academic 3, Lecturer in
History)
17
“Many thanks for sorting this. The stuff was all there but it will be easier to access it all
in one place. We had a meeting with our external today and showed her the first
module. She thought that every university in the country would like something similar
for both its undergraduate and postgraduate populations.” (Academic 3, Lecturer in
History)
The first email quote shows the enthusiasm of this academic to use the VLE template for her
course. The second email quote clearly indicates the adoption process of new technology
skills, which is reflected in the lecturer’s broader use of them. The third and fourth emails
also show excitement about the course websites, and email 3 also indicates that students
are starting to perceive this feature as a default, while email 4 indicates that our approach,
and some of the generic content developed within it, might be of interest for academics
beyond the Department and universities beyond Oxford.
2.6
IT set-up and support time
For this area, we asked two questions, on VLE set-up and support, to the dedicated VLE
support team who provided their feedback in April 2010 and November 2010. In addition,
reports on IT support time spent supporting the Department’s course administrators and
academics to use VLEs to support their courses were reviewed and analysed.
In April 2010, the VLE support team were involved in three main areas:
1. Loading tutors and students on to the system
2. Duplicating sites at the start of the academic year
3. Supporting more general VLE use
In November 2010, the VLE support team reported back that generally there has not been
much change in the overall IT set-up and support time. Despite more courses using VLE
sites (see section 2.1 for more details) the IT support needs and time spent have not gone up
significantly from last year, due to the supporting documentation, processes and support
developed by the Cascade project.
The VLE support team identified that one recurring issue was that staff did not adequately
read instructions and support material when first building their VLE from the template. This
should improve, as familiarity with these tools improves. In addition, some staff prefer
phoning up the support team to ask for help rather than accessing the available support site
information. One new issue that had not been anticipated was that as more art courses
began to use the VLE, large graphic files were uploaded, which presented problems for the
system. Support guidance was necessary, which resulted in extra IT support time being
given by the VLE support team.
2.7
Conclusion: high acceptance of VLE support for courses
Overall, VLE support for courses and generic content innovations have been well received
and taken up by members of the Department. In the area of award-bearing courses, where
we have focussed our efforts, the take up of the VLE and resources has been impressive.
For individual stakeholders within the Department, the picture is somewhat mixed.
Administrative staff expressed a very positive opinion towards VLE-supported courses and
the use of generic content, and full-time Departmental academic staff, also reported high
acceptance and enthusiasm towards the VLE. However, part-time sessional tutors seem to
find the integration of these Departmental innovations into their courses more difficult. A
reason for lower adoption among part-time tutors might be their lower degree of involvement
and integration into the daily processes of the Department. For students, a steady usage rate
18
and general satisfaction could be observed which leads us to the conclusion that our
students perceive VLE supported courses as valuable for their studies. In terms of IT
support, a certain level is still required at the moment to set up and maintain course VLEs.
The picture, though, remains unchanged from our baseline snapshot and, after the initial
start-up support, most courses need very little ongoing support. This is very encouraging, as
IT support has remained at a similar level to last year despite the fact that the number of
courses that might use it has now doubled. There is also an ongoing expression of interest
for using the VLE for the few remaining award-bearing in courses in the Department.
Recommendation on VLE implementations, including generic content: The Department
should focus on award-bearing courses where VLEs are used over a longer period of time
and maximise the ‘shelf-life’ and return on investment.
Recommendation on VLE improvements: Through the collection of feedback from all key
stakeholders – administrative staff, part-time tutors, students and full-time academics staff –
the course VLEs should be integrated in the Department’s cycles of review and
improvement. Providing additional support sessions to sessional and contracted staff when
they meet as course teams is likely to enhance use of the self-service resources and
documentation made available by the project.
3 Focus area 4: Course design
Many of the most successful uses of technology in curriculum delivery tend to have been
designed into the course from the start rather than added as an afterthought. Thus, while
many of the activities of the Cascade project have focussed on existing programmes, we
also wanted apply the lessons learned, as appropriate, to as many new programmes being
developed by the Department as possible.
The project team originally considered supporting this process by running workshops for
academics in the Department, which we know, from other projects, can be a powerful
learning experience. However, as the project progressed it became clear that academics are
more likely to find time to engage with this area when they are actually designing courses. At
any one point this was rarely more than one or two individuals, making a workshop less
valuable than one-to-one sessions. With this in mind, our approach evolved to the creation of
a reference resource, the Course Design Moodle, which academics could access at a time
that suited them, and by making individual support sessions available to staff at the relevant
point of the specification process of their new course.
At the time of preparing the Evaluation Report, there was only a small amount of evaluation
data available on the newly created Course Design Moodle and other activities undertaken in
this area (see Table 3.1).
#
Evaluation area
Main
stakeholders
Evaluation feasibility
check
Section in
evaluation report
4.1
Technology
engagement
Department
Only small amount of
pilot data available
3.1
4.2
Technology
confidence
Department
Not enough data
available to evaluate
NA
4.3
Technology best
practice
Department
Not enough data
available to evaluate
NA
Table 3.1: Overview evaluation data for Focus area 4
19
As a consequence, only one out of the three evaluation areas described in the Evaluation
Plan will be reviewed here:
3.1
Technology engagement
Between October 2007 and October 2008, only two new course proposals were submitted to
the Department’s Academic Board for approval and both outlined their intent to integrate
some technology elements in course delivery. However the uptake of technology prior to this
had been minimal. In October 2008, the course proposal form was amended to include
several new questions which required justification of technology use, or not, in several
different contexts. Since then, we have tracked the new course proposals forms to identify
technology engagement in course design. Table 3.2 lists all received course proposal forms
since October 2008 with course title and planned technology engagement.
Month first
submitted to
committees
May 20084
November 2008
November 2008
January 2009
February 2009
Course
Technology engagement
MSc in Sustainable Urban
Development
MSt in the History of Design
VLE support
DPhil in Management and Business
Studies
DPhil in Architectural History
MSt in Literature and the Arts
April 2009
Postgraduate Certificate in Historical
Studies
April 2009
Postgraduate Certificate (Research) in
Archaeology
Postgraduate Certificate in Cognitive
Behavioural Therapy
Postgraduate Certificate and Diploma
in Health Research
MSt in Diplomacy
DPhil in Mutual and Co-operative
Organisation
MSc in Surgical Science and Practice
Postgraduate Certificate in Cognitive
Behavioural Therapy for Psychological
Trauma
Undergraduate Certificate in Local and
Family History
April 2009
June 2009
February 2010
February 2010
April 2010
April 2010
April 2010
October 2010
Postgraduate Certificate in Ecological
Survey Techniques
Using VLE to support
course and deliver some
content
No planned use of VLE
No planned use of VLE
Using VLE to support
course and to deliver two
core modules online
Using VLE to support
course and deliver three
core modules online
No planned use of VLE
VLE support
VLE support
VLE support
No planned use of VLE
VLE support
VLE support
Delivered online with one
face-to-face residential
session in Oxford
Delivered online with one
face-to-face residential
session in Oxford
Table 3.2: New course proposal forms and planned technology engagement
4
Although this course proposal was submitted prior to October 2008, as we have been engaging with this project team
throughout the development process up to course launch this term, we have included them here.
20
This analysis shows that – after the proposal form amendment in October 2008 – the
majority of academics proposing new programmes now want to use technology for course
delivery and, apart from one case, the programmes which have chosen not to use the VLE
are Doctoral programmes which have such low student numbers that the VLE would rarely
be appropriate for an individual course.5 During the two-year period from October 2008 to
October 2010, fifteen proposals were received in total, of which eleven planned to use the
VLE to support delivery. Of these, two outlined significant plans for delivering elements of the
course online and two propose to deliver the majority of the course online. This trend seems
set to continue as TALL has already been contacted by other teams about developing
significant elements of online provision for new programmes.
3.2
Conclusion: technology use in course design increases
Clearly, engagement with online course elements in the Department has significantly
increased since the project began. This trend can largely be directly attributed to the
Cascade project. Since April 2010 all new programmes coming through the Department’s
approval process intend at least basic use of the VLE.
Confidence with technology and technology best practice were areas where data for a
summative evaluation were hard to collect. However, with the overall amount of technology
use in course design increasing, and from our experience working with staff across the
Department, we can conclude that the overall level of confidence with technology among
administrative and academic staff has risen.
Recommendation on Course Design: The Department should continue offering support to
administrative and academic staff in choosing and implementing technology options in
course design.
4 Focus area 5: Online enrolment and payment
Existing experience had indicated that there were major financial and operational benefits
available through the provision of online enrolment and payment services, in particular:
 Significant savings in processing time by course administration staff;
 improved data management by creating a direct link between enrolments and
payments;
 improving accuracy as students input their own data, therefore reducing the number
of people involved and removing transposition errors;
 savings in financial administration time due to reduced manual processing of
payments and opportunities for batch processing;
 improved service to our globally distributed students as enrolment is available 24/7.
There was a need to increase the number of courses offering online enrolment in order to
take advantage of the benefits identified above. Moreover, feedback had been received from
users suggesting a number of areas where improvements could be made to the functionality
and usability of the service. For example, a key issue raised by users was frustration that
returning students needed to enter their details each time they enrol on a course (some
students enrol on four or five short courses per year). To improve the user experience, online
student login functionality was identified as a priority, providing students with access to
manage their data and to help returning students register for courses online without the need
to input their details again.
5
At Departmental level we are investigating supporting these with a Graduate School, which would offer broader VLE
support to all doctoral students.
21
In this context, all three evaluation areas (4.1 administration handling time; 4.2 adoption rate
and 4.3 customer satisfaction), as originally outlined in both the Evaluation Plan and Baseline
Report, were reviewed.
4.1
Administration handling time
In April 2010, time motion data was collected on paper-based and online enrolments and
three different classes of enrolment types were identified:
1. Paper-based enrolment
2. Online enrolment
3. Online enrolment with feedback loop
For these three enrolment types, the Department’s manager of Online Weekly Classes
conducted a time motion study of how long it took to administer each type of enrolment and
provided five samples, each with accurate time measures. A summary of the time motion
study data can be found in Table 4.1.
Longest enrolment time
Shortest enrolment time
Average enrolment time
Baseline data
Paper-based
New data
Online
13:45min
2:45min
5:45min
00:15min
00:05min
00:10min
New data
Online with
feedback loop
15:00min
3:00min
5:25min
Table 4.1: Time motion study results for course administrator handling time for
different types of enrolment
For more details on this time motion data, see Appendix 4.1a and Appendix 4.1b.
For paper-based enrolment the longest sample of end-to-end student enrolment and
payment process took 13:45min. In this instance, the system did not hold an existing student
record and the administrator needed to create a new one. Also, the specific payment method
added to the total time. In contrast, the shortest enrolment time was 2:25min. In this sample,
the student record already existed, i.e. the enrolment was from a returning student, and the
course payment was made by credit card. The average time it takes to process a paperbased enrolment is 5:45min, correcting for outliers by excluding the longest and the shortest
enrolment times from the calculation.
Online enrolment is processed automatically; the administrative handling time requires no
more than 5-15 seconds to mark a successful online enrolment notification email as read and
move it into the respective email archive folder. However, in some instances the online
enrolment is not successful due to some missing or conflicting information. We called this
type of unsuccessful attempt to enrol “Online enrolment with feedback loop”. An example
could be where a student registers for a course with the EU-student fee but submits an
address located in the US. In this instance, the administrator needs to undertake the
following steps:
1. Read through the student email explaining the online enrolment problem
2. Write back to request further information
3. After further information retrieval, formulate instructions on how to solve the
enrolment problem and reply to the student
This “Online enrolment with feedback loop” process can result in a number of emails being
exchanged between the prospective student and the Online Weekly Classes staff. This
22
process can take up to 15:00min. In best cases, it only takes 3:00min. On average, in our
time motion study, this process of clarifying an unsuccessful online enrolment took 5:25min.
4.2
Adoption rate
In order to evaluate the adoption rate of online enrolment and payment, we collected records
of transaction counts from across the Department. The data source of these records was the
Department’s database called InfoSys.
To analyse the adoption rate of online enrolment and payment, we looked at three different
indicators:
1. Ratio of Departmental courses that offer online enrolment
2. Number of online enrolments by course type
3. Percentage of online vs paper enrolments
First, we looked at the number of courses that had the option available to enrol online. Data
was collected by running a snapshot report on InfoSys, on the last day of each month, to
collect data on the total number of course records for the current academic year onwards (A),
the number of ‘live’ courses being marketed on the Department’s website (B), and the
number of ‘live’ courses accepting online enrolments (C). The data is presented in Table 4.2.
A
B
C
(C/B)
(C/A)
Type
Total course
records in
InfoSys
Live courses
accepting
enrolments
Live courses
accepting
online
enrolments
Ratio 1
Ratio 2
Apr-10
1,412
May-10
1,686
Jun-10
1,752
Jul-10
1,779
Aug-10
1,796
Sep-106
820
Oct-10
978
Nov-10
1,065
424
543
650
652
633
453
475
525
297
465
589
598
581
417
430
484
70.04%
21.03%
85.64%
27.58%
90.62%
33.62%
91.72%
33.61%
91.79%
32.35%
92.05%
50.85%
90.53%
43.87%
92.19%
45.45%
Table 4.2: Online enrolment enabled courses, April - November 2010
Whereas ‘Ratio 1’ indicates the percentage of ‘live’ course currently being marketed on the
Departmental website that offer online enrolment, ‘Ratio 2’ gives the percentage of current
and future courses in InfoSys, but not necessarily yet open for enrolment, that are accepting
online enrolments. Ratio 2 thus has more course records, which explains why it is much
lower than Ratio 1, which only takes live courses into consideration.
The table gives end of month snapshot data for the last eight months of the Cascade project.
For example, at the end of April 2010, a snapshot of data from InfoSys found 1,412 course
records for the current and future academic years, of which 424 courses were tagged as live
courses with information on the Departmental website. The remaining 988 courses were
either courses that were not marketed online, that had already finished or that had not
started taking enrolments yet. Of the 424 live courses, 297 were marked with the flag
‘EnrolOnline’ which means that they have the online enrolment and payment functionality
enabled. Therefore 297 out of 424 (70.04% = Ratio 1) of courses marketed on the
Departmental website in April 2010 were taking online enrolments, and 297 out of 1412
6 In September 2010 the data collection parameter for course records in InfoSys was changed from ’09-10 courses onwards’ to
’10-11 courses onwards’ to reflect a change of academic year
23
(21.03% = Ratio 2) of all current and future courses currently in InfoSys at the time, but not
necessarily yet open for enrolment, were accepting online enrolments. If we compare the
April data to the November 2010 data, then we can see a steady increase in the number of
courses accepting online enrolment and the number of live courses having the online
enrolment and payment functionality enabled. The two ratios likewise indicate a steady
upward trend in the percentage of the Department’s courses accepting online enrolments.
Secondly, we looked at the number of paper-based and online enrolments, by course type,
on a monthly basis. The baseline data was collected for the full academic year (August 2007
to July 2008) before the start of the project and compared to data collected for the two full
academic years (August 2008 to July 2010) during which the Cascade project ran.
18,000
16,000
14,000
12,000
10,000
8,000
6,000
4,000
2,000
0
Paper
Online
2007-2008
2008-2009
2009-2010
Year
Figure 4.1: Total number of online and paper-based enrolments received by type
Figure 4.1 shows the total number of paper-based and online enrolments received by the
Department over the three year period from August 2007 to July 2010. The graph shows that
during this period there has been a steady growth in the number of enrolments being
received online, with almost double the number of online enrolments received in 2009-10
compared with 2007-08. In terms of the percentage of overall enrolments, online enrolments
have increased from 21% in 2007-08 to 39% in 2009-10.
When analysed on a monthly basis, of particular note is the number of online enrolments
received during the month of December. Whilst in total number these are not particularly
significant compared to other months of the year, as a percentage of overall enrolments,
online enrolments in December make up a higher percentage of enrolments than for almost
any other month of the year. For example, in December 2009, online enrolments made up
49% of enrolments received (compared to the year average of 39%). As the Department is
closed for two weeks each year over the Christmas period, this data indicates that the
availability of online enrolments means that students can continue to enrol and pay online
even when the Department is closed.
In terms of number of transactions, the increase in online enrolments has been particularly
significant for the Department’s largest programmes, which provide open access courses for
the general public and between them took almost 6,000 online enrolments in 2009-10.
24
500
450
400
350
300
PPWeekly - Online
PPWeekly -F2F
PPDayWeek - F2F
250
200
150
100
50
Au
g0
O 7
ct
-0
7
D
ec
-0
Fe 7
b0
Ap 8
r0
Ju 8
n0
Au 8
g0
O 8
ct
-0
8
D
ec
-0
Fe 8
b0
Ap 9
r0
Ju 9
n0
Au 9
g0
O 9
ct
-0
9
D
ec
-0
Fe 9
b1
Ap 0
r1
Ju 0
n10
0
Month
Figure 4.2: Online enrolments for three large course programmes by month
Figure 4.2 shows the number of online enrolments, by month, for three different course types
offered by the Department’s Public and International Programmes division:
1. Weekly classes delivered online (PPWeekly – Online)
2. Weekly classes delivered face-to-face (PPWeekly – F2F)
3. Day and weekend schools delivered face-to-face (PPDayWeek – F2F)
Overall the graphs indicate that enrolment numbers vary considerably by month and that the
total number of online enrolments has been increasing steadily. Over the three year period,
there has been a 60% increase in the total number of online enrolments received by the
three programmes from 3,560 in 2007-08 to 5,683 in 2009-10.
Third, we looked at the proportion of enrolments for the Department’s day and weekend
schools and weekly classes (both those delivered online and face-to-face) that were taken
online compared with the proportion received on paper (see Figure 4.3).
25
8,000
7,000
6,000
5,000
Paper
Online
4,000
3,000
2,000
1,000
PPDayWeek
2009-2010
2008-2009
2007-2008
2009-2010
200820-09
2007-2008
0
PPWeekly
Figure 4.3: Day and weekend schools and weekly class enrolments by type
For the weekly classes programmes (“PPWeekly”) there was both an overall growth in total
enrolment numbers as well as the proportion of enrolments received online. The percentage
of online vs. total enrolments was already high in 2007-08 with 44% of enrolments taken
online. In 2008-09, this percentage of online enrolments grew to 53% of total enrolments, an
increase of 9% compared to the previous year. This ratio grew further to 58% in 2009-10.
Thus, more than half of these courses are now booked and paid for online. In comparison,
the day and weekend schools (“PPDayWeek”) showed an even steeper increase in online
enrolments: from 6% in 2007-08 to 32% in 2009-10, which meant an average annual growth
of 13 percent in the last two years.
Finally, to illustrate total adoption of online enrolment over time, using the latest available
data, Table 4.3 provides a summary of the number of online enrolments taken for all courses
offered by the Department, and their associated financial value, for the first quarter of the
four academic years 2007/08 to 2010/11.
2007-08
2008-09
2009-10
2010-11
No
427
578
835
1,163
August
Value
£57,549
£92,823
£171,297
£215,218
September
No
Value
527
£77,947
826
£147,406
1,053
£245,941
2,219
£283,741
No
217
327
566
452
October
Value
£33,163
£59.831
£190,756
£158,924
No
1,171
1,731
2,454
2,834
Q1 Total
Value
£168,659
£300,059
£607,994
£657,883
Table 4.3: Q1 Total number of online enrolments and their financial value 2007-08 to 2010-11
Table 4.3 shows clear growth both in terms of the number of online enrolment transactions
and their associated financial value. Between 2007-08 and 2010-11, the number of online
enrolments transactions increased by 142% from 1,171 in Q1 2007/08 to 2,834 in Q1
2010/11. Growth in financial value was even greater with a 290% increase over the same
period.
4.3
Customer satisfaction
The last area of baseline data is the attitude of students towards online payment and
enrolment expressed as customer satisfaction when using this method of enrolment. Data
were collected from an online survey that was made available for students to complete after
26
every online enrolment and payment made via the Department’s website. Key findings from
this analysis are summarised in Table 4.4.
Number of respondents
Found the online enrolment form very clear
Concerned or very concerned about online payment
Unconcerned or very unconcerned about online
payment
Overall grade of 8 out of 10 or above for the
Department’s online enrolment process
Baseline
April 2010
1,500
69.9%
9.6%
69.8%
New data
November 2010
2,377
69.2%
9.0%
71.3%
86.7%
86.3%
Figure 4.4: Summary results from online enrolment survey
The analysis shows that 69.9% of students surveyed in April found the online enrolment form
very clear (69.2% in November). The concerns that students had when enrolling online were
around the online payment. 9.6% in April said that they were ‘concerned’ or ‘very concerned’
about using the Department’s online payment process (9.0% in November). This can be
contrasted by almost 70% of the students who said in April that they were ‘unconcerned’ or
‘very unconcerned’ about online payment (71.3% in November). 1,300 out of 1,500
respondents (86.7% in April) gave the Department’s online enrolment process an overall
grade of 8 out of 10 or above (2,052 out of 2,377 respondents; 86.3% in November).
4.4
Conclusion: growing number of online enrolments – less handling time
As outlined in detail in the Final Report7 and the project case studies on business
performance8 and supporting prospective students,9 the enhancement and wider adoption of
online enrolment and payment has been a major focus of the Cascade project, bringing
significant benefits to the Department. As Figure 4.1 indicates, throughout the period of the
Cascade project, course enrolments have moved away from paper and towards online
enrolment and payment and, as the project’s outputs become embedded, this trend will
continue. As a consequence, administrative handling time relating to the enrolment and
payment process will be further reduced, saving a significant amount of staff time. This will
result in a reduction of staff numbers associated with this non-value adding area of our work,
leaving staff to focus on other activities that support student recruitment and retention. While
the administrative handling time of paper-based enrolments depends on the payment method
– payment by credit card being quicker than payment by invoice – and the pre-existence of a
student’s record, the administrative handling time of online enrolments depends on whether
additional information is required to complete the enrolment. In the case of a standard online
enrolment, the time savings in comparison to paper-based enrolment are immense. Where
online enrolment requires an extra feedback loop via email with the support team,
administrative handling time savings still exist but are less significant. In terms of student
satisfaction with the online enrolment website, a consistently high level of customer
satisfaction can be observed, before, and during the Cascade project.
By the end of the Cascade project, the Department had already seen measurable benefits in
terms of savings in course administration time through the availability of online enrolment
7
8
9
Cascade Project Final Report:
http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeFinalReport.doc
Case Study 1: Leveraging technology to drive business performance:
http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeCaseStudy1.doc
Case Study 3: Using technology to support prospective students:
http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeCaseStudy3.doc
27
and payment. The greatest impact has been seen in the Department’s Public and
International Programmes division, which delivers the Department’s two largest programmes
(Day and Weekend Schools and Weekly Classes). In the 2009-10 academic year, 5,683
(87%) of the total of 6,501 online enrolments taken by the Department were for these two
programmes. As a result of savings made in course administration time, both programme
offices have been able to support an increase in the overall number of course registrations;
for example the Weekly Classes programme took 850 additional enrolments in 2009-10
compared to the previous year and reduced their staffing by 0.5 FTE, allowing them to
reallocate the staff effort towards marketing and development of further new courses.
Recommendation on Online Enrolment and Payment: Through further roll-out of online
enrolment and payment functionality, the Department will capitalise on the substantial time
savings in administrative handling time, enabling the extra staff time to be used either to
support additional student numbers or to provide value-adding services to enhance the
quality of course offerings.
5 Overall project evaluation
5.1
Project management
Throughout the Cascade project, all reports were submitted to JISC in a timely manner. The
project was extended by three months in order to achieve more implementation of results
from the development areas that could only go live at the start of a new academic year. All
but one of the focus areas – the roll-out of the new online assignment-handling system –
were implemented successfully within the project timeline. When the Cascade team reviewed
their own project management during the evaluation workshop held on 10 November 2010,
the following three areas of improvement were highlighted:
1. Scoping
2. Resource allocation
3. Implementation planning
In terms of scoping, the team felt they underestimated the time needed for this important
stage of the project, which had a knock on effect on managing the project going forward. If
the team was to run the Cascade project again, they would allocate more time and staff
resources for requirements gathering and project scoping.
In theory, resources in the TALL team were allocated to the Cascade project from the start of
the project, but existing commitments restricted these in real terms. Thus the project
manager could only put in her full time on the project (and recoup the initial short fall) after
Cascade had been running for several months. Also, it took far longer than hoped for to
recruit the Research officer. Also, more learning technology and administrative resources
rather than technical development resources were available to the project. Since greater
technical development than anticipated was required, this was not ideal. If the team were to
run the project again, they would plan more realistically against availability of staff time and
allow more flexibility to adjust staff allocation to fit the project’s requirements.
Some of the implementation timings were not matched with the academic calendar of the
Department. As a consequence, a delay of months in the development work of the online
assignment-handling system resulted in a delay in full implementation of an entire academic
year. The team also discovered that there were many dependencies between the different
project focus areas; at the beginning of the project, the team found it difficult to identify and
manage these. During the project, the team developed some simple project management
documents to assist with overall planning across the five focus areas. These were very
28
helpful; however, the tools could not help where important external dependencies had been
missed earlier in the project planning. Were the team to run the Cascade project again, they
would identify dependencies (both those within and between focus areas, as well as external
considerations) earlier and build them into the implementation plan.
5.2
Stakeholder engagement
The project undertook extensive stakeholder consultation and engagement across all
divisions of the Department. Against the staff list in December 2010, of 119 staff who had
roles that might benefit from the outputs of the Cascade project, 93 (78%) had directly
engaged with the project. Through this, the team encountered a real cultural change in
attitudes towards technology use. Examples of how this manifested itself included:



More academic staff contacting TALL to discuss how they could use technology to deliver
their programme, while preparing new course proposals;
Colleagues from Registry directed academic staff to consult TALL when they began
writing new course proposals;
Divisional managers and the Department’s Financial Controller contacted the TALL CoManager (Administration) to discuss online elements for new courses so that they could
be fully costed and included in the budget for the new programme.
Lastly, it seems clear that the experience of basic VLE support has encouraged individuals in
the Department to consider greater use of technology in future courses.
5.3
Learning as a result of the project
During the end-of-project evaluation workshop, the project team discussed how much they
had learned during the Cascade project, and how much they were still learning as the
services developed are embedded in the Department’s work. The project team grouped the
key lessons learned under five headings: (1) project scoping; (2) project management, (3)
innovation, (4) engagement and (5) embedding. Some examples of this learning are
described in earlier sections of this Evaluation Report, for example, the difficulties of
convincing stakeholders on new technology adoption (see Section 1.3) or the expectations of
students towards technology innovation in the Department (see Section 2.5). A full
description of the learning as a result of the project is provided in Section 3.2 of the Final
Report.10 In addition, the project team produced five case studies11 to highlight key project
activities and to share their learning in these areas with the wider e-Learning community.
The service developments, the new system implementations and the cultural changes
associated with the project will continue well beyond the official end of the Cascade project.
Online enrolment and payment is being adopted by more and more courses, new courses
are routinely starting to include technology elements in their course design, course websites
will continue to support course delivery and the new online assignment-handling system will
be fully rolled-out at the start of the 2011-12 academic year.
5.4
Tangible benefits
Across the project, a number of tangible benefits were identified and these are described in
Section 3.3 of the project’s Final Report. Quantitative data in several areas of tangible benefit
were identified as part of the project’s evaluation activities and, in Table 5.1 below, data
collected during the project has been applied to 2009-10 data held by the Department to give
10
11
Cascade Project Final Report:
http://cascade.conted.ox.ac.uk/sites/cascade.conted.ox.ac.uk/files/CascadeFinalReport.doc
Five case studies highlighting the learning as a result of the project are available from the project website at:
http://cascade.conted.ox.ac.uk/project-outputs
29
some indicative figures of potential future efficiency gains resulting from the increased use of
technology for curriculum delivery.
Input variable
=
Potential
annual
saving
130 hours
=
78 hours
=
360 hours
=
£3,840
=
824 hours
Cascade focus area improvement
260
assignments handled by
Registry for award-bearing
courses
390
extension requests handled
by Registry for awardbearing courses
48
award-bearing courses
x
48
award-bearing courses
x
10,298
paper enrolments received
by the Department in 200910
x
x
x
0.5 hours
saved with new online assignment
handling submission system and
process vs. paper-based
0.2 hours
saved with new online extension
approval feature and process vs. paperbased
7.5 hours
of photocopying time saved using a
VLE to provide course resources
£80
saved in printing costs by using a VLE
to provide course resources
0.08 hours
saved in course administration by
online enrolment and payment process
vs. paper-based
Table 5.1: Potential annual savings from Cascade efficiency improvements
Based on the evaluation data collected, potential annual efficiency gains in these areas
include:
1. Online assignment handling administration time savings (up to 208 hours p.a.).
2. Increased course administration efficiencies through the use of a VLE to provide
course resources (up to 360 hours p.a. of time saved photocopying and up to £3,840
reduction in photocopying costs).
3. Administration handling time savings from online enrolments (up to 824 hours p.a.).
In total, in these three areas, there is potential to save up to 1,392 productive hours of
administrative time, which is equivalent to over 38 weeks. Rather than spending this time on
assignment handling and course enrolment, the Department intends to direct these
resources towards more value-adding activities, such as development of new courses or the
improvement of existing course offerings.
5.5
Conclusion: achievements against aims and objectives
The original aims of Cascade were to enable the Department to:



Undertake its activities more efficiently so that resources are focused on value-adding
activities, such as delivering improvements to the student experience and the creation
of tools that support best practice;
Develop new, or repurpose existing, activities to support the delivery of its new vision
and provide additional revenue streams;
Support its ability to deliver academically superb courses to students of the highest
calibre through the use of new tools and functionality to augment the services
currently offered to students.
30
In the context of the Evaluation Report these aims were simplified into three headers: to
increase efficiency, to innovate and to improve levels of service. In Table 5.2, we mapped
each focus areas’ achievements against one or more of these three headers.
Table 5.2: Matrix linking project outcomes to aims
Table 5.2: Matrix linking project outcomes to aims
For the online assignment-handling focus area, the project team only partially achieved the
desired aims. From the evaluation analysis in Section 1, we see possibilities of efficiency and
service improvements, but they have not yet fully materialised due to the delayed roll-out of
the new assignment-handling system. In contrast, the combined VLE support for courses and
online delivery of generic content focus areas achieved both aims, efficiency and service.
Departmental staff are undertaking course set-up activities more efficiently and have more
time to focus on value-adding activities. That is also reflected in the enhanced student
experience described in Section 2 of this report. More Departmental courses have integrated
technology elements in their course design and we therefore see the innovation aim of this
focus area as achieved. Some of the achievements outlined in Section 3 were supported by
a general trend towards technology adoption that the Cascade team supported by putting in
place appropriate tools and support structures. In Section 4 of this report, administrative
handling-time savings in the focus area of online enrolment and payment were clearly visible
and will continue contributing greatly to the efficiency improvements. Due to the 24/7
availability of the system, and additional course administrator time available to deal with any
non-standard enrolments, this area of work has also demonstrated an improvement in levels
of service provided to students.
31
Appendices
Double click on any table to see in original excel.
Appendix 1.1: IT support time spent supporting online assignment submission (IT support team, 2007/08, 2008/09, 2009/10)
Course
Course
Course
Course
Course
Course
Course
TOTAL
1
2
3
4
5
6
7
Course
Course
Course
Course
Course
Course
Course
Course
Course
TOTAL
1
2
3
4
5
6
7
8
9
Course
Course
Course
Course
Course
Course
Course
Course
Course
TOTAL
1
2
3
4
5
6
7
8
9
Aug-07
120
45
Sep-07
60
30
Oct-07
Nov-07
20
90
30
110
Dec-07
15
55
Jan-08
80
20
15
50
15
30
10
20
110
100
180
50
180
155
170
170
420
35
205
50
380
Aug-08
40
10
5
Sep-08
10
5
30
Oct-08
15
10
20
100
150
Nov-08
25
140
15
110
0
0
175
155
85
465
Dec-08
25
25
115
30
0
15
35
0
0
245
20
40
65
10
115
75
55
5
10
Aug-09
10
50
40
20
0
10
15
75
10
145
Sep-09
30
15
35
0
0
0
50
120
25
130
Oct-09
20
35
10
75
0
0
25
35
10
165
Nov-09
60
195
35
105
0
0
335
120
0
730
Dec-09
80
95
10
60
0
15
45
5
30
305
10
Feb-08
30
Mar-08
110
55
25
90
80
Apr-08
70
90
55
65
15
May 08
95
30
30
240
20
380
110
405
Jan-09
90
70
30
10
0
30
265
170
20
495
Feb-09
30
80
10
60
0
80
45
65
30
305
Mar-09
50
200
110
90
0
10
130
30
5
590
Jan-10
60
70
10
0
0
125
165
140
10
430
Feb-10
Mar-10
60
70
10
40
0
80
150
0
30
410
105
75
5
135
5
50
0
165
90
70
20
450
25
195
Jun-08
10
20
15
10
10
30
10
105
Jul-08
Total mins Total hrs
135
745
12.42
15
480
8.00
25
265
4.42
15
765
12.75
15
305
5.08
30
0.50
10
460
7.67
215
3050
50.83
Apr-09
10
125
10
15
0
135
150
145
10
445
May 09
50
140
30
65
0
30
95
0
35
410
Jun-09
125
70
40
0
0
0
55
110
25
290
Jul-09
Total mins Total hrs
145
615
10.25
20
895
14.92
10
425
7.08
0
480
8.00
0
10
0.17
60
360
6.00
0
965
16.08
95
915
15.25
15
340
5.67
235
3750
62.50
Apr-10
90
35
0
30
0
140
30
40
10
325
May-10
40
30
20
30
0
135
50
25
20
305
Jun-10
20
15
55
0
0
55
55
105
0
200
Jul-10
Total mins Total hrs
105
580
9.67
105
850
14.17
40
270
4.50
0
410
6.83
0
0
0.00
70
795
13.25
5
1015
16.92
30
765
12.75
25
190
3.17
325
3920
65.33
35
10
32
Appendix 1.2: Administration handling time for assignment submission (Registry staff, Apr-10 for legacy system & Nov-10 for new system)
Manual assignment handling
Time
Who
During course set-up
Assignment set-up
Prior to due date
Check if there are any extensions that affect the due date
At time of submission
Preparation for
marking
(getting script to
marker)
After marking
Preparation for
moderation
(getting scripts to
moderator)
Receipt of submissions (issuing receipt)
Photocopying scripts
Stuffing envelopes
Posting
Time taken
Registry Assistant
5 minutes
Covered below
Reception / Registry
Assistant
Comments
Who
Time taken
New online assignement handling system
Who
Time taken
Comments
Registry Assistant
10 minutes
Registry setting up
groups within Moodle
and mark sheet
Not Registry, Mike?
5 minutes
N/A
N/A
Covered below
N/A
N/A
N/A
N/A
N/A
Seconds (if
reception is asked
by student for
signature)
Students write their
own receipts at
reception and they
are not issued if
work is posted
direct to the
Registry
N/A
N/A
N/A
N/A
N/A
Automatic email
notification
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Registry Assistant
15 Minutes
Record marks on
mark sheet
Registry Assistant
15 minutes
Record marks on mark
sheet
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Registry Assistant
10 minutes
Files are generally
sent to External as
a zipped
attachment
outside of CASS
Registry Assistant
20 minutes
Had to download
comments and
assignments in order
to zip email to
moderator
Registry Assistant
10 minutes
Registry Assistant
10 minutes
Registry Assistant
10 minutes
N/A
N/A
Registry Assistant
15 minutes
Central
?
Includes time to
record marks
Depends on
number of
assignments
Registry Assistant
15 minutes
Photocopying scripts
Registry Assistant
10 minutes
Stuffing envelopes
Registry Assistant
10 minutes
Central
?
Receipt of moderated scripts
Registry Assistant
10 minutes
Returning marked scripts to students
Registry Assistant
30 minutes
Depends on
number of
assignments
Registry Assistant
10 minutes
Notifying students of marks
Registry Assistant
N/A
Done by return of
work
N/A
Done by return
of work
Receipt of request
Registry Assistant
2 minutes
Depends how
request is received
Registry Assistant
2 minutes
Passing request on to decision maker
Registry Assistant
5 minutes - 30
minutes
Depends how
request is received
and whether it
needs additional
authorisation.
Registry Assistant if
more than 2 weeks.
Receipt of decision
Registry Assistant
2 minutes
Depends how
decision is received
Registry Assistant
2 minutes
Verification of decision
Registry Assistant /
Registry Officer
2 minutes
N/A
N/A
Notifying student of decision
Registry Assistant
5 minutes
N/A
Notifying tutor and other course personnel of decision
Registry Assistant
5 minutes
N/A
Student notification
Approximate Subtotal
Comments
per assignment
Receipt of marked scripts
Posting
After moderation
Legacy online assignemnt handling system
Task
Also a lot of work is
returned via
Registers
95mins
Depends on
number of
assignments
* See separate note in
additional comments
N/A
50mins
Depends on number
of assignments, still
need to check
comments
Done by return of Did not have a realistic
work
test
65mins
Registry Assistant
2 minutes
Registry Assistant if
more than 2 weeks
or above permitted
number of
extensions
5 minutes - 30
minutes if more
than 2 weeks
Automatic email
notification
Registry Assistant
2 minutes
Record extension on
mark sheet
N/A
N/A
N/A
N/A
N/A
N/A
Registry Assistant
2 minutes
Write note to student
N/A
N/A
N/A
N/A
N/A
5 minutes - 30
Automatic unless
minutes if more
more than 2 weeks
than 2 weeks.
Extension requests
Approximate Subtotal
21-46mins
9-34mins
11-36mins
33
Appendix 1.3: Summary of log data for online assignment submission pilot (VLE logs, British and European Studies programme, November 2010)
Full name
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
Student 9
Student 10
Grand Total
Number of page accesses for
uploading assignment files
4
6
4
8
7
6
4
5
6
4
54
Number of
Number of Number of uploads of assignment
assignment extension file uploads and extension request
views
additions
support files
Grand Total
31
4
2
41
27
4
37
23
2
29
30
5
43
12
4
23
33
3
42
8
2
14
32
1
3
41
16
3
25
10
2
16
222
5
30
311
34
Appendix 2.1: Overview of study programme links within course VLEs (VLE logs, November 2010)
Programme
Link to guidance on
using site and
technical help
Link to course
handbook
Llink to library
information website(s)
MSc Sustainable Urban
Development
X
PG Certificate in Historical Studies
X
X
X
Foundation Certificate in History
2009-11
X
X
X
Foundation Certificate in History
2010-12
X
X
X
Link to University
computing
information
Link to University
skills portal
X
Other links
Dept website - information for students and University
visitors and friends info
X
X
Generic MSc induction and historical studies modules
Graduate supervision system
Study skills resources Sheffield
MSc English Local History
X
X
X
X
X
Virtual training suite internet for history tutorial
Programme specification (on dept website)
Writing essays and report OU Open learn content
University skills portal online courses
Diploma in History of Art
X
X
X
X
Materials from OUCS IT courses
Information about Oxford (various links)
Undergraduate Certificate in
Archaeology 2010-2012, year 1
X
X
X
X
Undergraduate Diploma in
Archaeology 2010-2011.
X
X
X
X
Postgraduate Diploma in
Psychodynamic Practice 20092011 year 2
X
X
Postgraduate Certificate in
Psychodynamic Counselling
X
X
Total
In percentage
10
9
8
5
2
100%
90%
80%
50%
20%
35
Appendix 2.2: Student online survey results Q2; Q4 (Survey Monkey, November 2010)
Ho w e a sy wa s the we b site to use ?
Ho w use ful d id yo u find the co urse we b site ?
Re sp o nse
Pe rce nt
Answe r Op tio ns
Extremely easy
Very easy
Fairly easy
Not very easy
Not easy at all
If you experienced problems using the site please describe these:
Re sp o nse Co unt Answe r Op tio ns
2
6
9
0
0
1
11.8%
35.3%
52.9%
0.0%
0.0%
a nswe re d q ue stio n
skip p e d q ue stio n
Numb e r
1 Extremely useful
2 Very useful
3 Of some use
4 Not very useful
5 Not useful at all
17
2
use
Ho w e a sy wa s the we b site Iftoyo
u ?
e xp e rie nce d
p ro b le ms using
Re sp o nse D a te
the site p le a se
d e scrib e the se :
Jun 28, 2010 1:47 PMRemembering username & password Couldn't find course marks
Extremely easy
Very easy
Fairly easy
Not very easy
Not easy at all
Re sp o nse
Pe rce nt
5.9%
41.2%
41.2%
11.8%
0.0%
Re sp o nse Co unt
1
7
7
2
0
a nswe re d q ue stio n
skip p e d q ue stio n
17
2
Ho w use ful d id yo u find the co urse we b site ?
1 Extremely useful
2 Very useful
3 Of some use
4 Not very useful
5 Not useful at all
36
Appendix 4.1a: Administration handling time for paper-based enrolments (Manager of online weekly classes, April 2010)
Steps
Activity
1
Pick up form from fax machine or print from email
2
Open InfoSys
3
Go to student section within Infosys and start query with student's name
4
Evaluate query result
If no student record exists, continue with step 5; otherwise go directly to step 7
5
Create new student record
6
Validate new student record and press 'confirm'
>7
Register student for course
8
Process student payment
9
Send hard copy of enrolment and payment confirmation to student or send payment
(soft copy) request in case of payment by invoice
If payment type is by invoice, continue with step 10; otherwise go directly to step 13
10
Receive and file confirmation from finance that invoice was paid to the account
11
Enter payment confirmation in Infosys
12
Send hard copy of enrolment and payment confirmation to student
> 13
14
In Infosys, process batch report per payment type
Send batch reports to finance via fax or internal mail
Sample 1
Sample 2
7 April 15.04 8 April 10.13
Sample 3
Sample 4
8 April 10.22
9 April 11.21
5 secs
5 secs
5 secs
5 secs
10 secs
10 secs
10 secs
10 secs
5 secs
5 secs
5 secs
5 secs
No
No
Yes
No
120 secs
240 secs
300 secs
5 secs
5 secs
5 secs
40 secs
40 secs
40 secs
40 secs
20 secs
20 secs
20 secs
inv.employer
240 secs
write email &
attach invoice
240 secs
30 secs
30 secs
30 secs
to 13
to 13
to 13
30 secs
process with
sample 3
30 secs
37
20 secs
20 secs
Appendix 4.1b: Administration handling time for online enrolments with feedback loop (Manager of online weekly classes, April 2010)
Steps
1
Activity
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Open email inbox
15 Apr 9.19
16 Apr 9.14
16 Apr 9.36
16 Apr 13.54
19 Apr 10.24
120 secs
60 secs
300 secs
30 secs
60 secs
600 secs
120 secs
120 secs
If email states 'online enrolment successfull', continue with step 2; otherwise go directly to step
4
2
Right mouse click on email with "online enrolment successful" and mark as read
3
Move email into respective folder
>4
5
Read through student email with online enrolment problem
Write back email response with request for more information
180 secs
6
Read student's response to information request
60 secs
Repeat steps 5 and 6 where necessary
7
Formulate email with instructions how to solve enrolment problem and send to student
Once the problem has been resolved, the online enrolment process is normally concluded with
step 1, 2 and 3. Please record start and end date + time of end-to-end process >>>
120 secs
120 secs
paper ernolment
received 19 Apr
10.30
paper ernolment
received 19 Apr
16.10
38