User Experiences with Open Course Ware
E. Sjoer1, T.A. Hennis
1
Delft University of Technology, Faculty of Technology, Policy and Management, Jaffalaan 5, 2628 BX
Delft, The Netherlands (e.sjoer@tudelft.nl)
Abstract
In October 2007, TU Delft (Netherlands) launched its Open Course Ware website (http://ocw.tudelft.nl)
and became a member of the world-wide OpenCourseWare consortium (http://www.ocwconsortium.org).
From this date, educational resources used for the MSc programmes in Civil engineering, Electrical
engineering and Offshore engineering have been made available to the public. In 2005, TU Delft signed the
Berlin Declaration on Open Access to knowledge in the science and humanities. In accordance with the
Declaration free access to course content is being supported by the university. The goal of the
OpenCourseWare initiative is to build and expand knowledge networks, to increase collaboration with
society and to attract (PhD) students and life long learners. Since the TU Delft OpenCourseWare website is
its pilot phase, it needed to be evaluated. Upcoming courses on sustainable development, applied physics
and biomedical engineering should benefit from the evaluation results. In this paper, user statistics on
unique visitors, geographic origination etcetera, will be presented. Furthermore, focus groups were
organized with bachelor students and bachelor-level-teachers. How do potential new students experience
TU Delft OpenCourseWare?
Keywords: educational resources, open course ware, accessibility, evaluation
1.
INTRODUCTION
In October 2007, Delft University of Technology launched its Open Course Ware website. TU Delft
OpenCourseWare is a free and open digital publication of educational materials, organized as courses. The goal of
the OpenCourseWare initiative is build and expand knowledge networks, to increase collaboration with society, as it
is formulated on the website, ‘to eliminate borders and geographic distance as obstacles to the instantaneous
exchange of knowledge and new ideas. […] Educators from around the world may upgrade their classes; students
may enhance their coursework or pursue self study; the general public may glimpse the depth and breadth of what
leading universities are offering and benefit from reading lists and lectures.’[1] The first courses that were open to
the public were the MSc programmes in Water management (Civil Engineering), Micro-electronics (Electrical
Engineering) and Offshore Engineering. Delft OCW builds on the worldwide movement of opening up education.
Examples from other successful projects are MIT OCW and USU OCW, OU OpenLearn Initiative, Carnegie
Mellon’s (CMU) Open Learning Initiative, and National Repository of Online Courses (NROC) [2].
In this paper we briefly address the process before launch; the main focus will be the preliminary results gathered
from user statistics and one of the focus groups consisted of potential new students and their teachers, i.e. bachelor
students and bachelor-level-teachers. The research question can be formulated as follows: What are the preliminary
findings of TU Delft OpenCourseWare in terms of access, and perception of the educational resources by potential
new students and their current teachers? In the conclusion the lessons to be learned from this case study are
discussed so that other (technical) universities might consider Open Course Ware.
2.
STARTING AN OPEN COURSE WARE PROJECT
In June 2007, a project on OpenCourseWare was started at Delft University of Technology, initiated by a group of
people: the rector magnificus, a professor specialized in e-learning, people from the library, and the technical support
department. The budget allocated was K€ 600. The idea was to design an Open Course Ware website, à la MIT, and
pilot six master courses in two successive rounds. Open Course Ware (OCW) or Open Educational Resources (OER)
are defined by the Hewlett foundation, one of the largest supporters of OER as: ‘teaching, learning and research
resources that reside in the public domain or have been released under an intellectual property license that permits
their free use or re-purposing by others. Open educational resources include full courses, course materials, modules,
textbooks, streaming videos, tests, software, and any other tools, materials or techniques used to support access to
knowledge’ [3]. For Delft OCW, work packages were defined, the technical infrastructure was tooled up, the work
processes for delivery of educational material were described, and communication and evaluation strategies were set
up [4]. Further, a work package ‘Strategy’ was put into place to research the sustainability of the project in the long
term, using the framework of David Wiley [5].
In October 2007, the website went ‘live’ with ten courses for three master programmes: Water management (Civil
Engineering), Micro-electronics (Electrical Engineering) and Offshore Engineering. The teachers of these courses
volunteered to put the material of their courses online. They got K€ 50 for each programme to digitalize and update
some of the educational material, original designed for face-to-face delivery. Furthermore, material needed to be
added to the educational database and copyrights need to be arranged. A lot of knowledge was gained on the topic of
copyright. TU Delft decided to take out a creative comments license BY-NC-SA (non-commercial share alike, 3.0,
version July 2007) and a small ‘bureau’ was set up to screen the material submitted by the teachers for copyrights. A
technical infrastructure was also set up and the look and feel of the website was designed using new icons for
educational resources. A screen dump of the website as it is now is shown in figure 1.
FIGURE 1. Screen dump of the website (7 March 2008)
A number of instruments were designed for the evaluation of the first pilot phase [6]. Log facilities were installed to
measure the exposure of and access to the Delft OpenCourseWare website. A short survey was designed to
determine the use of the educational material and the attractiveness of the website. To gain further insight into the
quality of the educational material, focus groups were organized. A focus group consists of a small, homogeneous,
group of people that are led in a guided discussion of a specific issue. Focus groups were carried out with life long
learners, which are defined as working adults of 25 year or older, Bachelor (HBO) students, their teachers and
current Master students of the several domains. In this paper preliminary findings are shown, some statistics results
to give an indication of the amount of access to the site in the first months after launch, and the results of one of the
focus groups: Bachelor students and their teachers.
3.
FINDINGS: ACCESS
Access to the Delft OpenCourseWare website was tracked using Google Analytics. Log data was acquired on the
characteristics of visits and visitors.
3.1
Visits characteristics
The website attracted 12.729 visitors from October 15 2007 – February 22 2008 (19 weeks), 10.472 of these were
unique visitors with different IP-addresses. That is a nearly 100 visitors per day average. 82% of them were first
time visitors. As figure 2 shows, there was a small peak on 28 November 2007 when the number of visitors nearly
doubled. Probably due to an article that mentioned the website [7].
FIGURE 2. Number of visits
Of the visitors that entered the Delft OpenCourseWare website 77% came via references from another site, mostly
from tudelft.nl or ocwconsortium.org and only 9 % came from search engines: nearly all via Google; this percentage
appears to be increasing. Search terms that were most frequently used, were: Open Course Ware and
OpenCourseWare. 63 % of the visitors used Internet Explorer/Windows (84% XP and 8% Vista). Most of the visits
were short.
As the Delft Open Course Ware was given a soft launch, with little publicity there was no visitor’s peak when the
website was launched. Some of the peaks that occurred may be explained by an article mentioning Delft
OpenCourseWare. Note: Open Course Ware is a well known term for open access to digital educational resources,
although some people do have problems with the term ‘course’. A course refers to resources (usually full courses)
that are specifically designed for self-learning, such as courses from the Open University, whereas the Delft Open
Courseware include materials for face-to-face teaching, used in a course, such as lecture notes, syllabi, exams, and
articles, after the example of MIT.
3.2
Visitors characteristics
Who are the visitors to the Delft OpenCourseWare website? We do not have much data on visitor’s characteristics.
Residence can be determined from the log data (see figure 3):
FIGURE 3. Visitors per country
The Netherlands was responsible for half of the traffic, which is not surprising. However, as figure 3 depicts, visitors
came from 140 other countries and resided in 2400 cities. The exposure of educational material geographically could
be significant.
4.
FINDINGS: PERCEPTION
Last year bachelor students of schools of higher education (HBO) are potential new students for the Master
programmes of Delft University of Technology. The OpenCourseWare website provides students and teachers with
the opportunity to get an overview of the educational programme, the level of knowledge required and other relevant,
domain-related issues. The extent to which the website succeeds in doing this was investigated using focus group and
interviews with both bachelor students and their teachers. Three focus groups were organized with students active in
the following domains: Dredging/Offshore engineering, Micro-electronics, and Water management. Their teachers
were also interviewed.
4.1
Outline of focus groups
Before starting an interview, the students were asked to review and use the website for 10 minutes. Although they
had been asked to use and click through the pages on the website in advance, this repetition was designed to refresh
their memories, and to initiate preliminary discussion amongst the participants. The group interviews were divided
into three main parts.
After about 10 minutes the students were given blank notes, and asked to describe in a few catchwords their first
impressions. Although some were a bit hesitant, this proved to be an effective method to break the ice. When the
notes were collected, participants were asked what they deemed to be their most important catchword and were
invited to go into greater depth using this key word. This resulted in an interesting and in-depth discussion of the
student’s impressions of the Delft Open Course Ware website.
The second part of the interview was a more guided discussion on the use and usability of the site and its contents.
Interviewees were asked to describe their expectations and to describe their experiences of using the site i.e. search,
download, etc. The content of the site was discussed, its educational usefulness, and its transparency and
attractiveness. In addition, the students were asked to elaborate on possible improvements for Delft OCW.
Finally, interviewees were posed the hypothetical situation where they were invited to dinner with the president of
the TU Delft, the Delft OCW project manager, and the professor responsible for the domain relevant for their
education. They were then asked to discuss and explain the things they would ask, suggest and mention to the Delft
OCW.
4.2
Results focus groups
The outcomes of the interviews with the teachers were quite different. Some of the interviewees were enthusiastic,
mentioning that they would certainly use (some of) the resources in future education, some were less interested,
although acknowledging the potential of the initiative. Overall, one could say that the interviewed students and
teachers were positive about the project. The main positive aspects are explained below:
Structure. Most interviewees found the structure of the website and its course material obvious and
understandable. It all looked well laid out; it was formatted in PDF documents, with clear descriptions of
the different resources.
Usefulness. The teachers and students were, for the most part, positive about the usefulness of the
contents and the initiative itself. Even though sometimes the resources were hard to understand, they
triggered a desire to know more.
Many suggestions were made and criticisms expressed by teachers and students. The sections below summarize the
most important criticisms and suggestions on the issues of usability and content.
4.2.1
Usability
The questions on usability addressed issues such as ability to find and use resources, navigating the OCW site, visual
attraction and communication aspects. Technical improvements suggested by the actors are passed over.
Interactivity. Most students and teachers would like to see a higher level of interactivity on the site. For
example, a desire for more information about using the site was mentioned, specifically mentioning the added
value and use for specific user groups.
Search & Link. There were some suggestions for searching and linking functionalities. It would be useful if
there were links between resources, linking to other courses or courseware and also an ability to search within
the site’s contents, was stated to be a very useful additional functionality.
Communication. Students mentioned the added value of being able to communicate online with other
people from the same domain, or students doing their course.
4.2.2
Content
Students and teachers were asked whether they understand the resources, if they would use them for teaching and
learning, and if they had any suggestions.
Information. The website should provide information for its users about the entrance level required to
understand the resources and preferably provide revision courses.
Context. Creating a cadre for the resources. Most teachers mentioned that the resources were very specific,
and tell little about the world they represent or in which they are used. Showing projects and more contexts
would enlarge the perspective and understanding of users. It was also suggested that short flash movies about the
content, the professional world of the specific OCW-domain, and pedagogy or ideology would provide added
value. Naturally, the findings were different per domain.
5.
CONCLUSIONS
In October 2007, TU Delft launched an Open Course Ware website. Master courses in Water management, MicroElectronics, and Offshore Engineering were put online, open for everyone who would like to use the content. Since
launch, the website has attracted on average 100 visitors a day. A lot of traffic came from other websites that referred
to the Delft Open Course ware site, such as ocwconsortium.org. The website was visited by people from 140
nationalities. The lessons that can be derived from these statistics were that the (English) website has the potential to
reach a lot of people world-wide, and that publicity is responsible for peaks in the number of visitors, so the way a
website is launched and communicated is very important. A soft launch is recommended if the website is still ‘under
construction’.
One of the goals of the Open Course Ware website was to provide students and teachers with the opportunity to get
an overview of the educational programme in a specific domain and use the resources for teaching and learning. As
was done for life long learners, the perceptions of the TU Delft website were solicited from bachelor students and
their current teachers. Focus groups with bachelor students from schools of higher education were organized and
their teachers were interviewed. The results were very different. In one domain students felt aligned to the content
available both for the knowledge level of the resources as the subject matter whereas students in another domain
considered the provided educational resources ‘difficult’ and ‘academic’. For that reason, it is hard to derive
conclusions per actor. The content and the professional world of each specific OCW-domain could be very different.
Therefore, the content of specific domains should be adapted to the users of these domains.
The bachelor students and their teachers would like to have more information on the educational resources. For
instance: What is the knowledge entrance level for certain materials? If the designers of the Open Course Ware
website want to attract new bachelor students, the website should provide additional links to direct to the information
they require. Students also preferred more communication possibilities, such as an online forum.
Finally, the students recommended making the content more interactive. Slides are hard to understand without
context. The documents and readers were appreciated, but the students would prefer short movies, video clips
etcetera; these will make the site more lively and attractive. It appears that Open Course Ware promotes the updating
of existing material and making it more attractive, with pictures and movies. To fill part of the needs, more videos of
lectures have recently been added to the site.
The Delft OCW repository can be considered as ‘the first layer’. One of our next goals is to research the willingness
of other university teachers to add material to the website. However, if Delft University wants to become a hub in
global knowledge networks and wants to explore the future of learning, a second layer which consists of a depository
where everyone can upload content, and a third layer to facilitate a world-wide community, would be desirable.
According to the advice of Hennis for a sustainable Delft OCW, the project team should immediately start to develop
simultaneously the second and third layer. Moreover, they should involve others because of (i) the potential
contributions (OER, money, engagement) of numerous universities, companies, and individuals worldwide can make
(ii) the required technologies, networks, people, and expertise, and (iii) the present acceptance of the system by those
who helped develop it and are willing to continue in its development. Delft OCW needs to connect with other
organizations, utilize existing tools and technologies, and enable individuals to contribute to the project [8].
References
[1] http://ocw.tudelft.nl/about-opencourseware/ Delft OpenCourseWare website, Retrieved at 7 March 2008.
[2] http://ocw.mit.edu, http://ocw.usu.edu, http://openlearn.open.ac.uk , http://www.cmu.edu/oli,
http://www.courserepository.org
[3] http://www.hewlett.org/Programs/Education/OER/. Retrieved at 20 April 2008.
[4] Delft Open Educational Resources (Delft OpenEr), Projectvoorstel […] (in Dutch), unpublished, 2007.
[5] Hennis, Thieme, 2008, The Future of Delft Open Courseware. How to build a sustainable environment for Open
Educational Resources. Master thesis, Delft University of Technology.
[6] Sjoer, E, Plan van aanpak: Evaluatie Delft OpenER, Eerste fase (in Dutch), unpublished, 2007.
[7] http://education-portal.com/articles/10_Sources_for_Free_Online_Technology_Courses.html 10 sources for free
online technology courses. Retrieved at 7 March 2008.
[8] Hennis, T., W. Veen and E. Sjoer (accepted by Edmedia 2008), The future of Open CourseWare; a case study.