ONLINE
LEARNING LAB
REPORT 2015
Innovating
the way we
teach and learn
Innovating
the way
we teach
and learn
TABLE OF CONTENTS
Executive summary
6
Foreword9
Introduction11
The Year In Numbers
12
Exploring Online Education
17
Design & Delivery: Learning Experiences
Iteratively improving course features
Exploring storytelling, pedagogy and new media
Storytelling & Pedagogy:
Innovation & pedagogy
Flexibility & production
Taking a course whenever you want; the rise of Netflix University
Personal and Glocal: Small Private Online Courses (SPOCS)
Specializations: What’s next?
Activate and Empower: Learners as community builders
Sustainable MOOC development
Crowdfunding open content
17
17
18
18
18
20
20
22
23
23
25
25
Democratization through education
27
Impact on on-campus education
41
MOOCs as a Platform for research
49
Demographics
Areas of employment
Education
Course quality and satisfaction rates
Do our learners return for more?
Performance by age
Where the digital and physical classrooms meet to form the next experience in higher education
Designing the classroom of the future
Lessons learned from Flipping the Classroom
How Leiden University students use our MOOCs
Facilitating research through tooling and analytics
Data management policy
Github tools
Data as a service
Online learning research
Information density and understanding
Effective learning strategies: retrieval practice
Peer reviews: online performance and collaboration 27
29
31
32
37
40
41
41
44
45
49
49
50
50
50
51
52
52
Collaborations53
Next steps
54
Exploring new technologies
Recognition of MOOCs and online learning towards credit
Piloting online proctoring
Designing for deep learning
Adopting an online mindset for our physical spaces
Experiment with alternative monetization structures
Stimulate sustainable development of course content
Exploring the online master space
54
54
55
55
55
55
56
56
6 - Innovating the way we teach and learn
7 - Innovating the way we teach and learn
EXECUTIVE SUMMARY
This report presents the result of the Online Learning Lab providing a quantitative and qualitative
analysis of the Leiden open and online initiatives
from October 2014 to December 2015. So far:
• Leiden Massive Open Online Courses (MOOCs) reached
almost 200K enrollments from 196 countries (EU 37%,
Asia 17%, Africa 5%) or approximately 10.000 monthly
users.
• More than 100K participants enrolled in the course
Terrorism & Counterterrorism: Comparing Theory and
Practice.
• Over 52K participants enrolled in Miracles of Human
Language; an Introduction to Linguistics, making it the
single MOOC run with most enrollments in 2015.
• More than 11.000 exams were submitted and over
12.000 Statements of Accomplishment were rewarded.
• More than one million videos were watched, over 30.000
posts and more than 15.000 comments were made on
the Leiden MOOC social media forums in 2015.
• Over 90% of the learners had a ‘good’ to ‘excellent’
experience in the MOOCs.
• More than 358 Video Lectures were produced for
MOOC content (over 50 hours of educational content)
and 71 video’s for other projects, recorded at 15 locations
on campus and beyond.
• Over 400 volunteers translated subtitles for the
MOOC Miracles of Human Language into 26 languages.
• 8,5% of active MOOC learners is active in the
Facebook groups established around the courses.
• Started with Small Private Online Course (SPOC) design
that combines a virtual classroom with a real classroom.
• For the first time a course successfully crowdfunded an
additional module.
General Findings:
• MOOCs beyond the hype: MOOCs seem to have
outlived the hype and there is still a lot to experiment
and learn from this type of online education.
• MOOCs and SPOCs help to innovate on-campus
education: Your own students are ready to use new
learning formats. Online learning spaces do not make
students less willing to learn if combined with intensive
and rich on-campus learning experiences. For example,
the use of a good mix of storytelling and pedagogy
ensures that such materials are engaging and leverage
their maximum potential in terms of educational
experience and outcomes.
• The rise of 21st century skills: Students increasingly
expect academic education and knowledge gathering
to go hand in hand with (new) skills such as planning,
finance, coding, data visualization and so on.
• Faculty engagement is key: To catalyse the conversation
regarding online learning, the development of faculty
wide Online Learning Strategy Plans is useful.
• Growth of requests for MOOCs and SPOCs: MOOCs
and SPOCs have been included as key strategic points in
relation to on-campus education innovation, and there is
a significant rise in the number of requests to use online
learning facilities of the Online Learning Lab.
• The impact of technological developments: Research,
education and working Technological advances (be it
digital or material innovations) increasingly shape the
physical environment in which students learn, study
and work; the university building of the future will
look different from the ones we have today.
• Recruitment and brand awareness through MOOCs:
The global and massive reaches of MOOCs have recruitment potential. MOOCs and SPOCs are seen by several
Faculties as a way to reach new student groups.
• Networking potential through MOOCs: MOOCs
offer possibilities to build a network of professionals
around the world.
• Importance of flexibility due to pace of technological
change: As technologies become ever more readily
accessible due to prices dropping, a limited budget can
have a big impact in online learning.
• Online Learning Research calls for an interdisciplinary
approach: There is a growing need to cross disciplines
due to increasing complexities in scientific development.
There is a need to create new institutional incentives to
bring students, teachers and researchers from different
backgrounds closer together, and to connect to broader
networks outside the university (inside out and outside in).
Recommendations:
• Explore the opportunity space between 100%
residential and full online MOOCs. Be clear about
what the residential student experience is. Be explicit
and intentional about what students and faculty get
when they engage in class.
• Develop a coherent vision on what an academic
working and learning environment should look like in
the future and use broad experimentation to test
different approaches.
• Dare to question the origin of the curriculum
structure. Don’t keep doing what you do because
you’re comfortable doing it. E.g. experiment with short
cohort settings, mix of tools and delivery modes, and
teacher-mentor coaches. Focus on the residential
experience you provide students with.
• Stimulate Social Learning in course design.
Students learn more (and retain knowledge longer)
from interaction and discussion with each other and the
teacher than simply listening to a lecture, watching a
•
•
•
•
•
•
•
video or reading a text. Social contact in a community,
online as well as face to face, is central to the academic
experience.
Design for change in learning spaces. Develop a
comprehensive approach for classroom and learning
space design and furniture.
Find a way to leverage the potential of MOOCs to attract new students from previously hard-to-reach areas.
Stimulate the use of open online materials.
Understand how students and alumni use our own
and other institutes’ online materials.
Start innovation and pedagogy experimentation.
Explore cutting-edge techniques while harbouring sound
educational principles. E.g. the impact and use of virtual
reality and 360 video as new media tools in education.
Invest in flexible equipment and small studio
environments and high quality staff. Since new technologies are changing the online learning landscape
quickly and new inventions make previous technologies
obsolete, ensure that quickly changing the composition
of your team and the tools on which they rely are possible
throughout the entire year.
Explore new, alternative funding approaches for online
education e.g. crowdfunding institutional partnerships.
Invest in online communities. This includes enlisting
learners to update/upgrade your materials. Learners are
proud of ‘their’ MOOCs. Give them ownership.
“ Even with my english, that isn’t good enough, this
course showed me a new world, a really interesting
world, an invisible world that only a few people can
see how it works. From now on, always that i look a
big corporation on TV, Internet or when i buy some
product or service, I will remember what I learned
with Leiden University.” Student in the MOOC
Rethinking International Tax Law .
8 - Innovating the way we teach and learn
9 - Innovating the way we teach and learn
FOREWORD
Leiden University has always strived to nurture a coherent vision of how new ideas can quickly enter the
university, if they are relevant, and to either find ways
to scale up, or to abandon such ideas and initiatives in
a timely manner. As research confirms “ such space is
necessary […] and [for] the definition and management of dynamic capabilities to renew competences
in line with changing environments”.1
by Vice Rector Magnificus prof. dr. Simone Buitendijk
We are in the midst of a technological revolution,
which is significantly impacting the way we live, learn
and work. Universities need to determine in what way
they want to adapt to these new technologies and how
they wish to use them in teaching and training the
new generation for the job market and the world of
tomorrow. Present day students will encounter a more
globalized and interconnected society than previous
generations did. These developments are shaping a
new type of student, who more and more needs to be
a ‘digital native’, and who will learn differently from
students 10 years ago. Many present day students
need to be prepared for jobs that do not yet exist.
The mission of the research-intensive university is
to drive innovation, help respond to major national
and global problems, and provide the narratives that
make it possible to understand a rapidly changing and
increasingly volatile world.
We started in 2012 on a journey to explore how we
could offer our knowledge to a larger, global audience
and to democratize education through technology,
with our partner Coursera. We remain committed to
this mission to open-up education and TO increase
access globally. At the same time we are welcoming
requests from staff and teachers within our university
to change the way we work and to rethink our campus
education: what does online & open education mean
for (the future of) the university and how can
it enhance the learning experience of students?
The report in front of you looks back at 2015 and
reflects on the various innovation experiments we
conducted. We have tried to formulate a number of
recommendations for ourselves as a research and
teaching organisation, but we hope they will inspire
and be of benefit to a larger community of highereducation innovators.
1 Teece, D. J., Pisano, G., & Shuen, A. (1997). Dynamic capabilities and strategic management. Strategic management journal, 18(7), 509-533, Navarro,
J. R., & Gallardo, F. O. (2003). A model of strategic change: Universities and
dynamic capabilities. Higher education policy, 16(2), 199-212.
10 - Innovating the way we teach and learn
11 - Innovating the way we teach and learn
INTRODUCTION
“We know what we are, but not what we might be”
William Shakespeare
Through the Online Learning Lab (OLL), Leiden
University primarily aims to explore new forms of
education, pedagogy and novel target groups. By providing a truly global audience access to our education,
we hope to make a contribution to the democratisation of education. Through the experience and insight
of these explorations, we aim to leverage innovation
and to strengthen our on-campus education. This
impact on campus education is fortified by research
for which MOOCs provide an excellent platform.
Last but not least, our frontrunner position in online
learning helps establish and reinforce (inter)national
collaborations. In the following chapters we will
evaluate the progress on these four aims.
After launching the first MOOCs and SPOCs in
2013-2014, we introduced several innovations in
the delivery of MOOCs. We enhanced our working
methods, established of a data-driven research team,
and a growing interest in MOOCs and SPOCs from
the part of the faculty which lead to over 25 newly
approved MOOC, SPOC and Flipping the Classroom
projects. These projects are currently in different
stages of development. One of the driving forces
which impacted online learning within the university
were the new Online Learning Strategy plans written
by the Leiden University Faculties, for which we provided advice. From online masters and pre-masters to
active learning in a Flipped Classroom, new flexible
learning spaces, academic skills education
and blended learning for study success are put into
each faculty’s context and are prioritized.
At the same time, the Centre for Innovation’s Online
Learning Lab maintained the role of frontrunner,
always in co-creation with other pioneers, to explore
trends that Faculties might be interested in tomorrow.
This way, we hope to fulfill our role to help Leiden
University prepare for the future of education. To stay
ahead of the curve we decided to become an innovation partner of Coursera which enables us to test
new ideas quicker and push the boundaries of online
learning a bit further.
The future of online and open learning is just
beginning, and by exploring and harnessing the
possibilities presented by rapidly developing information technologies, we aim to be at the forefront of
teaching and learning innovation. It is not yet possible to say what MOOCs could/should look like in
the coming 3 to 5 years and how online learning will
be embedded in our institutions; therefore investment in new initiatives and scalability of working
processes is needed in the coming years.
The report in front of you will provide facts and
figures around the various innovation projects we
ran last year with MOOCs.
Marja Verstelle & Gideon Shimshon, Co-founders
of Leiden University’s Online Learning Lab @ The
Centre for Innovation
12 - Innovating the way we teach and learn
13 - Innovating the way we teach and learn
THE YEAR IN NUMBERS
Table 1: Innovation production figures at a glance
MOOCs delivered
MOOCs Rerun
• Miracles of Human Language; an Introduction to
Linguistics by prof. dr. M. van Oostendorp.
• Federalism & Decentralization; Evaluating Africa’s
track record by dr. J.G. Erk
• Rethinking International Tax Law
by prof. mr. S. Douma.
• The Changing Global order (2 reruns in 2015)
by prof. dr. M. Hosli;
• Configuring the World (2 reruns in 2015)
by prof. dr. R. Griffiths
• Global affairs Capstone Project (2 reruns in 2015)
by prof. dr. R. Griffiths and Prof. Dr. M. Hosli
• Miracles of Human Language (1 rerun in 2015)
by prof. dr. M. van Oostendorp.
MOOCs available On-Demand
MOOCs in development in 2015
• Terrorism & Counterterrorism: Comparing
Theory and Practice by prof. dr. E. Bakker.
• Federalism & Decentralization; by dr. J.G. Erk
• Rethinking International Tax Law;
by prof. mr. S. Douma.
• International Law in Action: a guide to the International Courts &
tribunals in the Hague (Jan 2016) by prof. dr. L. van den Herik et al.
• The Rooseveltian Century (Jan 2016)
by prof. dr. G. Scott-Smith et al.
• Clinical Kidney Transplantation (Jan 2016)
by Dr. M. Reinders et al.
• Be Persuasive: write a Convincing Position Paper or Policy Advice.
Project centred course (Feb 2016) by prof. dr. R. Griffiths.
• Anatomy of the abdomen & Pelvis (April 2016)
By prof. dr. M. De Ruiter et al.
• International Law in Action: Investigating and prosecuting
International Crimes (Spring 2016) by prof. dr. C. Stahn.
• On Being a Scientist (Summer 2016) by prof. dr. B. Haring et al.
• Heritage under Threat (Summer 2016) by dr. S. Mire et al.
• Evolution Today (Fall 2016) by prof. dr. M. Schilthuizen et al.
SPOCs delivered
Flipping the Classes delivered
• Configuring the World; a Global Adventure,
by prof. dr. R. Griffiths.
• Terrorism & Counterterrorism (2 projects) prof. dr. E. Bakker.
• Federalism & Decentralization dr. J.G. Erk
• Rethinking International Tax Law, by prof. mr. S. Douma.
Special audio visual projects
Guides and workshops
• Documentary series of 4 Lectures about the
History of Leiden University (for the 440th
anniversary of Leiden University);
• Production of 30 high quality online video lectures
‘On Minorities’ with the Faculty of Humanities;
• 3-D rotation model ‘Embryonic Intestines’
for Anatomy MOOC;
• A customized video design and trailers
for 7 new MOOCs;
• Special recording Vice-President of the European
Commission Federica Mogherini in collaboration
with Campus The Hague, used in the MOOC
‘The Changing Global Order'.
• Extending studio and faculty services: greenscreen,
new media formats, cameras and equipment
Guides
• How to write & create a knowledge clip?
• Innovation of physical learning spaces
Workshops:
• How to develop a SPOC workshop
(Conference on Innovation in Teaching)
• Screencast recording
(Faculty of Science)
• Communication management Training
(TU Delft and E-merge)
• Development of knowledge clips at LU
(Conference on Innovation in Teaching)
14 - Innovating the way we teach and learn
15 - Innovating the way we teach and learn
Session-based
333.655 Enrollers
Total
164.716 Learners
24.557
15.689
3.263
6.660
1.616
Exam Takers
Certificates
Honours
signature track
Financial aid
423.586
197.389
17.660
8.320
On-demand
89.931 Enrollers
32.673 Learners
1.971 Certificates
1.660 Signature
track
Figure 1: Overview of statistics for on-demand and session-based MOOCs. Data is calculated cumulatively over the years 2013-2015.
Enrollers
Learners
Certificates
Signature track
16 - Innovating the way we teach and learn
17 - Innovating the way we teach and learn
EXPLORING ONLINE EDUCATION
Creating engaging and impactful online education
requires an increasingly interdisciplinary and iterative
approach. By bringing together domain experts,
pedagogy expertise, video/media specialists, coaching
capabilities, and analytics into the course design the
development process becomes highly experimental.
Bringing together all this knowhow to deliver a
course that is efficient in production while at the same
time allows for exploration and innovation required
a process which was flexible yet timeboxed.
Design /
Innovate
Leiden MOOC Learners per 100.000 Internet Users
1
192
- Course design
• Pedagogy
• Community
• Video
- Student
experience
Figure 2: Sign-up rates, per country and completion rates, per region. Based on data of roughly 170.000 learners from 10 session-based MOOCs
Develop
- Video scripts
- Assignments
- Video production
- Course environment
- Tests
Figure 3: Overview delivery process Online Learning Lab
Design and Delivery: Learning Experiences
Based on the work of G. Salmon (University of
Western Australia), who developed an education
development method called Carpe Diem, we adapted
the method as part of the Innovate step in the delivery
process (figure 3). In contrast to previous development methods, Carpe Diem involves organizing an
intensive 2-day workshop with domain experts and
MOOC teams to kick-start the course design process.
This speeds up the completion time for course development in the Design phase and delivers a more
coherent approach to content, assignments, assessment and video. It also fosters trust between the
various team members, which results in a strong
commitment to get the project done together.
Deliver
- Course launch
- Interaction and
feedback
Evaluate
- Course evaluation
- Define improvements
- Provide research data
18 - Innovating the way we teach and learn
Exploring Storytelling, Pedagogy and New Media
How can we translate learning materials into an
online setting? What tools do we use when teaching
online and how can we keep students engaged in
a setting so different than our regular classrooms?
These are some of the questions the Online Learning
Lab seeks to answer for educators who want to develop online learning materials. The following elements
are leading in our approach to video didactics:
• Storytelling and pedagogy ensure that such materials are
engaging and leverage their maximum potential in terms
of educational experience.
• Innovation and pedagogy experimentation with
cutting-edge techniques while harbouring sound
educational principles.
• Flexibility and production ensures that we incorporate
a variety of formats suitable to the course content.
Storytelling and Pedagogy:
Video and interactive online media are a key aspect
of online learning environments. Similar to on-campus education, online teaching materials take
well-established pedagogical models as their point of
departure. However, many of the materials used in
online education rely on new media (as opposed to
physical classroom interaction), and therefore require
additional techniques to achieve the desired level of
engagement. One technique we apply is storytelling.
The essence of great storytelling is well known in
(news) media, (web) documentary production and
filmmaking, and known as one of the oldest forms
of communication1.
1 Bratitsis, T., Barroca, A., Fruhmann, P., & Broer, Y. (2014).
European Educators ’Training Needs For Applying Digital
Storytelling In Teaching Practice, 194–204.
19 - Innovating the way we teach and learn
We apply similar techniques by working with
educators to remodel learning materials and help
them to engage with students by posing intriguing dilemmas and questions. Therefore, the focus becomes
showing instead of simply telling. The first results of
using storytelling are promising and we are looking
to intensify the use of these techniques in 2016.
Innovation and Pedagogy
The convergence of technology in devices means that
these new forms are readily accessible and that the
boundaries between types of media begins to fade2.
For each production, we require the implementation
of a specific innovation which can be tested against
pedagogical insights. Such innovations can focus on
incorporating specific technologies, one example of
which is the 3D animations created for the MOOC
Anatomy of the Abdomen and Pelvis. They can also
focus on creating more engaging video’s, such as our
upcoming experiments using a 360 degree camera.
To us, the process of innovation in online learning
materials is enabled by a collaboration between
experienced documentary makers who look at video
formats and the emotional connection between the
viewer and the subject, as well as pedagogical
expertise geared towards learning outcomes and
formal learning models. Using new media in online
education is relatively novel, but virtual reality (VR)
and augmented reality (AR) are widely recognised
to have a promising future in higher education.3
2 Jenkins, H. (2004). The Cultural Logic of Media
Convergence. International Journal of Cultural Studies, 7(1),
33–43. doi:10.1177/1367877904040603
3 Exploring new media for education is done in collaboration
with Remco van Schadewijk from the Leiden Science Faculty
Video and Pedagogy
To celebrate Leiden University’s 440th birth year
Professor Willem Otterspeer, a historian at Leiden
University developed four knowledge clips.
Each clip in this web series highlights the
communicative power of storytelling in educational
content by taking viewers on a journey through four
centuries of freedom in Leiden.
Using engaging video: which format is right for you?
In a MOOC, instructors translate their scientific knowledge into videos, animations and other new media using
techniques from documentary production, storytelling. The images below show several of these techniques as
they are used in our MOOCs.
Lecture video - ‘Talking Head’
Use of informants
On-site locations
3D animation
20 - Innovating the way we teach and learn
Flexibility and Production
The interaction between instructors and students is
essential for learning outcomes. Therefore, instead
of taking a ‘one size fits all’ approach, each MOOC is
redesigned to fit the instructor’s needs. In the case of
online learning materials, this translates into a flexible
studio setting and a production team that is active
both in The Hague and Leiden. In terms of design
this means that every MOOC has a different look
and feel. Additionally the MOOC-videos are shot on
a wide variety of locations. In 2015 the video team
shot lecture video’s on 15 locations in and around the
University. From the operating room for the course
Clinical Kidney Transplantation to the Linguistics lab
for the course Miracles of Human Language.
With a relatively small team of one director and one
editor, and one flexible staff if needed, the media
team produced 358 Video Lectures and over 50 hours
of educational content. It further produced another
71 videos for additional video projects. The formats
range from discussion videos, interviews, animations
and studio recordings. As the number of requests
to use online learning facilities are rising, there is a
continuous search for optimisation of the production
output. The flexible studio format used at the Online
Learning Lab, allows for this by making sure that
innovative add-ons to the workflow can be added
the year round.
21 - Innovating the way we teach and learn
Taking a Course Whenever You Want:
the Rise of Netflix University
Traditionally, MOOCs were offered in sessions of five
to eight weeks. These sessions would run whenever
the instructor could make the time for them, which
led to a mismatch in supply and demand. In 2015,
Coursera launched its “on-demand platform”,
allowing students to begin a course whenever it suits
them. While this feature is accommodating to the
schedule of learners, it introduces the issue that
without deadlines, one might never actually start
the course.
Coursera further introduced “cohorts”, which
provides learners with deadlines and therefore
reintroduces a part of the session-based system.
As such, this approach attempts to combine the best
of both platforms by allowing flexibility and providing incentives to work on the course materials.
The results of the new system are mixed. Below a
short analysis comparing the ‘session based’ with
the ‘cohort based’ approach.
Terrorism session-based
(session 3 & 4, each 5
weeks)
Terrorism on-demand with
cohorts (from 30-06-2015
to 31-12-2015)
% change
Sign-ups
30.991
33.290
+7.4%
Course completers
(of total sign-ups)
6.5%
3.6%
-44.6%
Payments (of total sign-ups)
2.5%
2.7%
+8%
Hours spent OLL staff
(over entire period)
36
27
-25%
Hours spent academic staff
(over entire period)
30
27
-10%
Engagement Professor
Weekly
Stand-by basis
-
Table 2: Comparison indicators for the on-demand and session-based courses of Terrorism & Counterterrorism.
We compared six months of the continuous ondemand session with the results of two session-based
courses for the MOOC Terrorism and Counterterrorism. These are our main observations:
• Overall, the number of course completers dropped.
However, the number of payments1 went up. As a percentage of total course completers, payments represent
roughly 39% in the session-based platform and 75% in
the on-demand platform.
• As the on-demand platform improved, both retention
and number of certificates increased. The number of
paid certificates increased faster than did the non-paid
certificates.
• The social interaction in the experience also saw a drop
with less students posting on the forum.
1 With ‘payments’, we mean payments that count towards a verified certificate
These results call for a rethinking of our course
designs on the one hand and the need to keep
experimenting with platform features from the
Coursera side and explore alternative monetization
strategies without hindering the learning process.
In 2015, we joined the Innovation Partners group
of Coursera. This enables us to test out and deliver
feedback on additional functionality and to be at the
forefront of MOOC platform developments. So far,
Leiden University opted-in for most didactical tests (6
out of 7), and declined the monetization related tests
(2), as they do not fully align with our vision on open
education. To us, offering learners the possibility to
expand their knowledge is the most important feature
of open education. Our efforts for 2016 will continue
on this path.
22 - Innovating the way we teach and learn
Personal and Glocal:
Small Private Online Courses (SPOCS)
Unlike MOOCs, which cater to massive audiences,
SPOCs are geared towards smaller, private groups
ranging from 20 to 250 and sometimes 500 students.
SPOCs feature intensive moderation by teachers, and
are fully online courses. Earlier SPOC experiment
gave us insight in the development and support
processes required to scale up. In September, we
opened a call for SPOCs, which resulted in a selection
of nine new, very diverse SPOCs to be developed and
evaluated in 2016.
The SPOCs that we are developing are targeted at
several groups:
• Regular students can use them as an online and flexible
course to practice, for example, academic skills.
• Students from several universities can follow SPOCs
together as a means to foster cross-university community
building.
• Professionals can follow SPOCs to deepen and widen
their skill set.
• Pre-master students can follow SPOCs to prepare for
their on-campus education, for example when they
live abroad.
23 - Innovating the way we teach and learn
Learning From Experimental SPOCs
In the summer of 2015, Professor Dr. Richard T.
Griffiths organized a Small Private Online Course
(SPOC) which was based on his successful MOOC
“Configuring the world: a critical political economy
approach”. Students of Dr. Griffiths’ summer school
in China participated in the SPOC together with
pre-university students. While the Chinese students
also received on-campus classes by Dr. Griffiths,
the other students did not. One of the goals of the
course was to develop a lively community between
students of different backgrounds residing in
different geographic areas and taking different
versions of the same course.
The experiment did not have the desired results.
SPOCs need their own approach to community
management that pays more attention to individual learners than one would in a MOOC, just like
material might need an adjustment for this specific
format. Induction and e-tivities embedded into the
course design are important to lower the barrier
to interaction, as is the constant attention of the
moderator, which in a SPOC is the instructor. In
addition thought needs to go into the motivation
of students to go online. The Chinese students not
only felt shy about their English, but had live access
to the professor during the day which did not
motivate them to go online after hours to interact
with the virtual group of pre-university students.
There simply was no need for them to do so.
This experiment was helpful in preparing Online
Learning Lab develop a strategy for community
management and e-moderation for the 9 SPOC
experiments lined up for 2016.
In 2015 Coursera closed down its private course functionality we used in some former SPOC experiments,
and we started a selection process for a new SPOC
platform, which we evaluated along the lines of the
following criteria:
•
•
•
•
•
User-friendliness
Mobile accessibility
Integrated web conferencing
Easy integration of 3rd party tools like TurnitIn
A paywall and easy access for non-Leiden students
Based on these criteria we selected an experimental
platform as a SPOC platform for 2016. This platform
offers us the opportunity to explore online education
for private groups. In the last chapter of this yearly
report we will look ahead at those developments that
we aim to explore in 2016.
Specializations: What’s Next?
In collaboration with Geneva University, we
composed the Specialization Global Affairs consisting
of three MOOCs and a capstone project. While the
average MOOC of four weeks requires a study load
comparable to 1 ECTS, a Specialization compares
more to a semester long 5-10 ECTS course.
After finishing the course, learners have considerable
in-depth knowledge of a topic. We joined the first
launch of Specializations by Coursera, which was also
the first Specializations in the field of social sciences.
The experiences are positive. We continue to look for
topics to develop as a Specialization from scratch,
to offer our learners an in depth learning experience.
Activate and Empower:
Learners as Community Builders
Online education should be available and open to
everybody. One way to achieve this is to lower the
barrier to entry by offering a diverse range in video
captions. We have found that when engaged properly
online learners are happy to give back to the
community of students by performing a variety of
tasks. Our most successful project involving volunteer
translators resulted in the translation of all captions
in the MOOC Miracles of Human Language: an
Introduction to Linguistics to 26 languages. To date,
this effort made it the most widely translated MOOC
on Coursera.
24 - Innovating the way we teach and learn
25 - Innovating the way we teach and learn
“Testing online interaction”
One of Leiden’s popular online courses is given
by Marc van Oostendorp and called ‘Miracles of
Human Language’. In an ideal situation Marc van
Oostendorp teaches students in small groups, making the MOOC a test on how to bring this personal
interaction online: “It seemed like a nice challenge
to investigate how to imitate interaction in online
education. In fact, that is why I started teaching the
MOOC.” Creating an online community including
17.000 participants in an Facebook group was key
to the success of the MOOC. Marc van Oostendorp: “Being together with students online, made
the MOOC experience totally new to me. Knowing
that there are thousands of students at the same
time learning the same content is very stimulating.
That is why for me a MOOC is the best new thing
to study self paced, it is stimulating to learn together. Faculty of Humanities. Instructor Marc van
Oostendorp, course: Miracles of Human Language.
An Introduction to Linguistics.
Russian
Brazilian Portuguese
Chinese
Greek
Italian
Spanish
Ukranian
Arabic
Bahasa Indonesia
Korean
Hungarian
Dutch
Turkish
Vietnamese
Romanian
Portuguese
Bulgarian
Croatian
Japanese
Swedish
Danish
Malay
0
5
10
15
20
25
translators
Figure 4: Number of translators, per language
Sustainable MOOC development
In the course of 2015, Coursera has been experimenting to find additional revenue streams for sustainable
financing. One of the most visible consequences of
this approach resulted in an overhaul of Coursera’s
motto from “the best courses in the world, for free”
to “...accessible for everyone” and later to “Take the
world’s best courses, online”. From Coursera’s perspective, these changes were meant to open opportunities to explore revenue models, while allowing
Coursera to maintain its commitment to 100% financial aid for all learners for whom payment remains a
prohibitive barrier. While individual courses still can
be joined for free, courses that are part of a specializa-
tion feature a popup window with the choice between
1) payment or 2) financial aid. Since Specializations
gathering in prominence on the Coursera website,
many visitors we speak to get the impression that
MOOCs no longer available for free. We regret this
because our aim is that Leiden MOOCs are always
free to take, and that one only pays (or applies for
financial aid) for an -optional- verified certificate. We
are working closely with Coursera to ensure learners
are well-informed of the options available to them.
In part as a result of our feedback, Coursera has
made recent changes to the user interface to make
the financial aid option more prominent. We are not
opposed to experiments with new monetization models as a financially healthy platform provider and the
continuity of the platform are important for Leiden
University. Also, the share the university receives
helps financing MOOC updates and to keep them
running on-demand. Our conviction is that monetization forms like sponsoring, crowdfunding, pay
forward (for others) and pay-what-it-is-worth-to-you
are monetization forms that fit our aims of sharing
knowledge better. Therefore we will experiment with
those optional forms of payment in the coming year.
A first promising experience is described in the next
paragraph. MOOCs organized by Leiden University
will remain free to take.
Crowdfunding Open Content
Even though many of our learners are not interested
in a course certificate, they are willing to donate money. Our crowdfunding experiment in 2015 resulted in
raising enough money for the development of a new
module for a course the learners enjoyed. This shows
that users are willing to donate money to enable their
own learning as well as that of others, and to keep
content open and free.
26 - Innovating the way we teach and learn
27 - Innovating the way we teach and learn
DEMOCRATIZATION THROUGH EDUCATION
40%
Percentage of contributors
Demographics
Figure 6 shows the 10 most featured countries of
origin for our learners. Most of them come from
the United States. Only three non-Western countries
are featured in this list (India, China and the
Russian Federation). When compared to the Coursera
average, Leiden University attracts a lower percentage
of learners from upcoming economies, but attracts a
higher number of European students.
Based on survey data in our session-based MOOCs,
we gather information about the age, occupation, and
course experience of our learners. These statistics are
useful in evaluating our courses, and serve as a basis
to our iterative approach to create better content.
Approximately 5-10% of all learners complete our
surveys. Simultaneously, we use other sources of data
to get a better understanding of user demographics
and engagement.
60%
United States
20%
United Kingdom
Netherlands
India
0%
<€5
€5-€10
€10-€25
€25-€50
€50 - €100
€100 - €500
>€500
Spain
China
Figure 5: Results of our crowdfunding campaign to finance an additional module for the MOOC ‘Miracles of Human Language’
Canada
Germany
Russian Federation
France
0%
5%
10%
15%
Figure 6: Ten most featured countries of origin for our learners. Data is based on roughly 90.000 learners of 7 MOOCs
20%
25%
28 - Innovating the way we teach and learn
29 - Innovating the way we teach and learn
When comparing age groups between regions, we see
that for every continent the largest group falls into the
category of 25-34 years (figure 7). With the exception
of Oceania and, to some extent, the Americas, learners
of age 45 and higher are scarce. Asia, in particular, is
represented by younger learners aged 18-24 and 25-34.
As for gender differences across continents, we
observe that (with the exception of Africa) the overall
gender distribution is roughly equal. With respect
to the Coursera average (roughly 60% Male, 40%
Female), this is a more equal distribution.
AGE GROUPS
EUROPE
40%
35%
30%
25%
20%
15%
10%
5%
30%
25%
AMERICAS
AFRICA
GENDER BALANCE
Roughly 38% of respondents for the course International Tax Law have a background in law, and 20% of
respondents for the same course have a background
in accountancy, banking or finance (figure 9). When
compared to other courses and the Coursera average
(2.4% for the former category and 14.3% for the latter
category), this is a remarkably high rate. Although not
surprising given the subject matter of the course, this
fact, coupled with the observation that the course has
a much lower adoption rate among other employment
areas, indicates that this course is specifically popular
among professionals.
65+
55-64
45-54
35-44
F: 51%
M: 49%
Student (college/university) (15.9%)
25-34
18-24
<18
Teaching and education (11%)
65+
55-64
20%
45-54
15%
35-44
10%
25-34
5%
18-24
40%
35%
30%
25%
20%
15%
10%
5%
Areas of Employment
Our courses are popular with both university students (figure 8, 16% of total) and those employed
in the education sector (11% of total). The former
group is relatively stable over all courses. The second
group is remarkably higher (22% versus average of
8% for other courses and 5.3% Coursera average) in
Miracles of Human Language MOOC. This indicates a high adoption rate among teachers for this
particular course.
Law (7.3%)
F: 53%
M: 47%
Accountancy, banking and finance (6%)
<18
65+
M: 68%
55-64
45-54
35-44
F: 32%
Retired (6%)
Business, consulting and management (5.8%)
25-34
18-24
<18
IT and information services (5.2%)
ASIA
40%
35%
30%
25%
20%
15%
10%
5%
25%
20%
OCEANIA
15%
10%
5%
65+
55-64
45-54
F: 45%
Public sector (5%)
35-44
25-34
M: 55%
18-24
<18
65+
55-64
45-54
Health and social care (2.8%)
F: 50%
35-44
25-34
18-24
<18
Seeking work (3.8%)
Other (12.4%)
M: 50%
Figure 7: Breakdown per continent for age groups and gender. Graph is based on several data sources of roughly 15.000 learners
0%
5%
Figure 8: percentage of participants, per area of employment. Aggregated over all courses
10%
15%
20%
30 - Innovating the way we teach and learn
A similar result can be observed (although less
strong) for the course Wheels of Metals, Urban
Mining for a Circular Economy. This course, which
was primarily aimed at professionals, attracted a
higher-than-average number of learners in areas
such as engineering and manufacturing (14%) and
business, consultancy and management (10%).
31 - Innovating the way we teach and learn
A relatively large group of MOOC participants
(6%) indicate that they are retired. This is interesting
especially because most survey respondents are
between the ages of 25 and 35. When compared to
the Coursera average (4.5%), Leiden courses are
slightly more popular amongst retired learners.
Video lecture by dr. Jan Erk for his course Federalism and decentralization:
Evaluating Africa’s track record.
Education
When we compare the highest level of education
attained for our courses in 2015 with the results for
2014, we see that the figures remain almost identical.
Most respondents (roughly 70% for both years)
indicate that they have enjoyed university education
at the undergraduate or postgraduate level. This is
higher than the Coursera average (roughly 65%).
Conversely, our courses are less popular among
less-well educated learners when compared to
the Coursera average.
39.4%
Postgraduate university
38.6%
31%
Undergraduate university
31.5%
11.5%
College
11.7%
10.3%
Completed high school
Year
10.2%
Some additional training
3.1%
2014
3.3%
2015
2.5%
Some secondary school
2.6%
2%
Prefer not to say
1.9%
0.3%
Primary school
0.2%
0%
10%
20%
Figure 10: Highest level of education among respondents for 2014 and 2015
30%
40%
50%
32 - Innovating the way we teach and learn
33 - Innovating the way we teach and learn
Course Quality and Satisfaction Rates
Most students indicate that they want to learn new
things from our MOOCs (figure 11). This number
is higher among courses that are aimed at a wide
audience (e.g. Federalism & Decentralization and
Miracles of Human Language) rather than professionals (e.g. Wheels of Metals and International Tax law).
For the latter group, improving career prospects is an
important reason to take the course.
“This course gives an opportunity to study for
those people who don’t have a chance to study
in the university. That’s a great thing that you are
doing.” Student in the MOOC Rethinking
International Tax Law.
As in 2014, most learners get what they want from the courses they follow (figure 12). In 2015, Roughly 50% indicate that they got what they wanted out of the course, and an additional 36% indicated that the course exceeded
their expectations. These figures vary little across courses, which reflects learner satisfaction across the board.
“You have pushed me positively into the becoming
the Diplomat I always love to be”. Student in the
MOOC Terrorism and Counterterrorism;
Comparing Theory and Practice.
40.8%
Learn new things
35.4%
Yes, the course exceeded
my expectations
45.9%
35.9%
21.3%
Improve my career prospects
49.6%
Yes, completely
19.5%
13.5%
Some additional training
To some extent
11.9%
Year
9.5%
See what MOOCs are
2015
13.4%
0.8%
6.4%
2014
6.2%
2015
5.7%
Meet new people
2014
14.1%
12.8%
Get a feeling for education
at Leiden University
Unsure
Year
50.2%
No
0.5%
0%
10%
20%
30%
40%
5.6%
0.5%
0.5%
0%
10%
20%
30%
40%
Figure 11: Responses to the question “What would you like to get out of this course?”, aggregated for 2014 and 2015
50%
Figure 12: Responses to the question ‘did you get what you wanted from the course?’, aggregated for 2014 and 2015
50%
60%
34 - Innovating the way we teach and learn
35 - Innovating the way we teach and learn
Learner satisfaction is further reflected in the course ratings (figure 13 for session-based courses, figure 14 for
on-demand courses).
In both cases, learners indicate they are happy with the quality of Leiden MOOCs. By far, most respondents
(>80%) indicate that they find the course ‘excellent’ or ‘very good’.
100%
80%
42.6%
Excellent
42.6%
60%
44.2%
Very good
40.4%
Year
10.2%
Good
2014
13.5%
2015
2.5%
Satisfactory
40%
20%
3%
Poor
0.5%
0%
0.5%
5
0%
10%
20%
30%
40%
4
3
2
1
4
3
2
1
5
4
3
2
50%
Federalism
17 reviews
Figure 13: Respondents’ answers to the question ‘how would you rate the overall quality of this course’, aggregated for 2014 and 2015
5
Figure 14: Ratings for each of our on-demand courses
Tax law
66 reviews
Terrorism &
counterterrorism
555 reviews
1
36 - Innovating the way we teach and learn
37 - Innovating the way we teach and learn
When we look at the engagement of learners with course components, we see that most students (roughly 83%)
watch videos and make quizzes (75%). Conversely, few students indicate that they engage with the forum on a
regular basis.
Do Our Learners Return for More?
Roughly 40% of all our learners has signed up for
more than one MOOC organized by Leiden University (figure 16 ). This means that there exists a large
student body (> 100.000) of learners who return to
follow more MOOCs from Leiden University.
“I am a policy analyst from South Sudan and we
are debating both decentralization and federalism
in South Sudan and this knowledge will be quite
handy for me and i am going to be a better policy
analyst. I would be honored to take another course
with you.” Student in the MOOC Federalism and
Decentralization; evaluating Africa’s Track Record.
100%
never
seldom
some weeks
most weeks
every week
90%
80%
70%
184.418
60%
59.1% | 1 course
16356
4 courses | 5.2%
50%
59.916
40%
19.2% | 2 courses
12.983
5 courses | 4.2%
30%
29.247
20%
9.4% | 3 courses
8.925
>5 courses | 2.9%
10%
Forum discussions
In-video quizzes
Quizzes, exams
and assignments
Readings
Watching video
lectures
Figure 15: Respondents’ answers to questions about their engagement with the course components, aggregated for all courses in 2015.
Figure 16: Figure shows what percentage of learners have respectively signed up for one, two, three, four, five or more Leiden MOOCs over both 2014 and 2015
25.4%
17.9%
24.2%
45.2%
44%
46.3%
38.8%
32.9%
18.3%
29.9%
38.9%
63.6%
Septemb
82.1%
75.8%
54.8%
61.1%
New learners
53.7%
201
3
Se
Ter
Co rorism
u
(ro ntert &
und err
2) orism
Returning learners
40%
30%
Sep
tem
ber
& ism
m or
ris err
rro ert
Te unt 2)
Co und
(ro
50%
61.2%
56%
ism &
Terror terrorism
er
Count
3)
(round
60%
70.1%
4
67.1%
01
74.6%
e2
70%
81.7%
Jun
79.5%
er 2014
80%
Co
wo nfig
rld ur
(ro ing
un the
d1
)
20.5%
The ch
an
Globa ging
l
(round Order
1)
100%
90%
Overall, courses are most clearly linked through time
and content. That is, specialized courses attract fewer
(but different) learners, and learners often sign up
for the next Leiden course.
4
100%
We see that the courses the Changing Global Order
(run 1) and Configuring the World (run 1) most
clearly overlap in users. This is likely due to the fact
that these courses are part of the same specialization.
01
Figure 18 clusters the courses together based on
the overlap in course sign-ups. Here, we see that one
cluster (purple text) is comprised of mainly early
courses (end of 2013 and beginning of 2014), the light
blue cluster concerns courses that were organised in
2015, and the dark blue cluster is comprised of several
unrelated courses.
pte
mb
er
2
When we look at figure 17, we see that a sizeable minority of each course is comprised of learners who have
already followed at least one other Leiden MOOC.
4
39 - Innovating the way we teach and learn
May 2
01
38 - Innovating the way we teach and learn
014
y2
uar
Jan
etals
of M
eels
Wh
36.4%
The Changing
Global Order
(round 2)
Introduction to EU
Law (round 1)
June 2013
20%
Fe
De dera
ce lis
nt m
ra
liz &
ati
on
201
4
20
Ju
ne
April
201
15
20
5
y
ar
2015
March
Figure 17: Courses (x-axis) are ordered chronologically. A ‘returning learner’ is a learner who has signed up for at least one earlier course.
No
vem
ber
u
br
Fe
15
Intern
at
Law (r ional Tax
ound
1)
5
201
he
g t 2)
rin d
gu un
nfi (ro
Co orld
W
e
Jun
Human
les of
1)
Mirac ge (round
Langua
)
)
r
1
n
x
er
er
3)
4)
ld
ld
ls
de
EU )
d1
d2
or )
nd
ma )
l ta ) ord )
eta 1) ord 1) wor 1) und
or ) nd
on 1 un
un
e w nd 2 f hu nd 1 iona nd 1 bal nd 3 (rou
al nd 2 (rou
f Mund obal und the und (ro
ro
h
cti und (ro
b
(
t
o
o
t
u
u
u
u
u
o
o
u
l o
l
a ro gl (ro ism
s o
g o
d ro m
m
ars (ro ng g (ro ring (ro rism
g g (r orism
rin (r racle ge (r ternaw ( ing
ris
tro ( ris
o
ral
Ge
n l
sin
si
s
In Law erro
i
gu
gu
rro
de
err
err
Ri
Ri
M ngua I
Ri
nfi
nfi
T
T
T
Te
o
o
Fe
a
C
C
L
Ter
Co rorism
u
(ro ntert &
und err
4) orism
g
ngin
Cha er
The al ord
b
Glo nd 3)
u
o
(r
10%
October 2014
Figure 18: Result of applying a hierarchical clustering method on learner enrolments. There are three clusters (purple, grey and light blue). Both between and within
the clusters, we see a strong time-based component in learner sign-ups. Additionally, we observe that the first rounds of ‘the changing global order’ and ‘configuring the
world’ are more deeply clustered. This is likely to be the result of the courses belonging to the same specialization track.
40 - Innovating the way we teach and learn
41 - Innovating the way we teach and learn
IMPACT ON
ON-CAMPUS EDUCATION
Performance by Age
Figure 19 shows the final course grade distribution by age group. All the distributions are bimodal (i.e. they have
two peaks), indicating that students either do very poorly or do very well.
0.025
0.020
Nonetheless, the learning experience offered to these
students does not reflect this mindset. Much like the
boundary between television and the internet has
been torn down in the past 20 years, hereby rendering
television sets, mobile phones, tablets and computers
‘portals’ to access the same multimedia content,
we aim to explore the ‘wall’ between online and
on-campus education and figure out how to form
the next learning experience in blended education.
0.015
Density
Today’s university students are digital/networked
natives who are just as comfortable moving around
in the digital world as they are in the physical
world. They are used to quick fact-checking on
their phones using google, sharing images, video
and thoughts via social media, and possess great
aptitude at scanning large amounts of information
to determine its relevance.
0.010
0.005
> 18
18 - 24
> 25
0.000
0%
25%
50%
75%
100%
Figure 19: Density plot of course grades, per age group. Only grades > 0 are included in the graph. Figure is based on the survey responses and performance of roughly
10.000 learners in eight MOOCs.
Under 18 year-old learners (upcoming university
students) are less likely to receive a high grade for
their course. This age group often indicates that the
course difficulty was more difficult than the average
learner. For this group, the bimodal distribution
is least pronounced when compared to the other
groups.
18 to 24-year-old learners (current university
students) are more strictly distributed between poor
performers and good performers.
Above 25-year-old learners (life-long learners) are
the best performers, and also show a less uniform
distribution in passing grades.
At the heart of all this rests the belief that, ultimately,
content and learning experience are what matters.
Contrarily, the medium through which education
is offered should not have an effect on the student’s
experience. To this effect, all learning spaces need to
be geared towards optimizing the learning experience
in order to cater to today’s students.
Where the Digital and Physical Classrooms Meet
to Form the Next Experience in Higher Education
In 2015, the Centre for Innovation Online Learning
Lab has made several concrete steps towards further
congruence of learning spaces at Leiden University.
We set up a Flipped Classroom experiment which
received a very positive reaction and we intensified
the use of our Living Lab, a space modeled after
Stanford’s d-school to better reflect the needs of current-day students and located at the Faculty
of Governance and Global Affairs. Additionally,
we visited other institutions to gain more experience
and to help guide our goals for 2016.
Designing the Classroom of the Future
In 2014, a special commission at MIT published
the Institute-wide Task Force on the Future of MIT
Education (URL: goo.gl/je85Qy). In this report, the
authors forward several recommendations to integrate new technologies with traditional education.
In part, these recommendations focus on an overhaul of the physical learning spaces and on the need
to experiment with pedagogy and various forms of
blended learning in education to remain relevant
and ahead of the curve.
Despite recent advances in online education, our
on-campus educational format still relies on a model
that predates the internet. That is, today’s students are
used to having access to large amounts of information
and a digital freedom of movement that is reflected in
online courses, but not in our on-campus education.
Too many classes are still focused on ‘broadcasting’
knowledge in a linear fashion. For a large part, this
discrepancy arises from learning spaces that are
suited for such a learning experience rather than
allowing students to collaborate, research and shape
their physical space to fit their needs. Future learning
spaces could to be re-imagined in order to meet new
learning needs of students with potentially significant
impact on university wide classroom design. The
blended approach and a new (digital) type of student
are posing new requirements for classroom setups. In
recent years, Harvard moved from high tech classroom to flat floor, everything on wheels, whiteboard
walls with minimal tech (apart from flex beamers/
42 - Innovating the way we teach and learn
projectors and power outlets). Setting up the classroom differently is important for future classroom
setup for teacher innovation. Universities can change
the culture through space changes, providing teachers
with a new environment for learning.
Since its inception in 2012, the Living Lab has been a
collaborative hub in which students and faculty can
organize lectures, seminars, workshops and other
activities. The main goal of the Living Lab is to promote collaboration between students and faculty by
providing a space that is flexible and can serve many
different purposes depending on the needs.
The Living Lab provides students with flexible furniture, a large, open space, beamers, meeting areas and
furniture that can be used as a whiteboard. The effect
we see on the instructor in this new structure asks
Taking students outside of the traditional classroom.
43 - Innovating the way we teach and learn
questions such as: where do I stand? How do I relate
to my students in this new setup? While students ask
what’s my role as a student in this space?
As institutions it is important to realize that students
are not looking for technology for the sake of technology. Rather, they are looking for a learning experience
that fits their needs, and we need to design for this in
our online and physical classrooms and other learning spaces. Key takeaways from the various visits and
working within the Living Lab area are:
Put collaboration central in the classroom (flexible),
Informal spaces are key around the classroom for
deep work and learning and Provide for makerspaces/
testing spaces (also for students of social science, law
and humanities).
44 - Innovating the way we teach and learn
45 - Innovating the way we teach and learn
Lessons Learned from Flipping the Classroom
In 2015, students who were enrolled in the oncampus Masters course Tax Treaties simultaneously
participated in the MOOC Rethinking International Tax Law together with some 14.000 participants
from all around the world. As the on-campus course
mainly consisted of traditional lectures, the MOOC
supplemented this course through participation in
the online MOOC forums, peer reviews, and by
providing additional content.
The data shows that the Leiden University students
outperformed their peers in the other groups (figure
20). Furthermore, Leiden Students were better at
selecting content relevant to them, and received
higher grades for the peer-graded assignments.
Leiden students indicated that they enjoyed participating in the MOOC. They especially enjoyed the
flexibility with which the format presented them, the
supplementary skills they learned, and the additional
35
30
Grade
25
20
information they received from the online lectures
and readings. Additionally, it became clear that the
online format substantially lowered the barrier to
engagement with the content. Simultaneously, the
course instructors of the course were pleased with
the results.
We learned several key lessons from this blended
experiment:
• Students experienced this form of blended learning as
highly valuable. Chiefly, they enjoyed the freedom with
which they could consume course content.
• Even though there was no strict check on the
engagement with the course content, Leiden University
students watched 80% of all videos. Additionally, students
identified lectures important to them and re-watched
these at a higher frequency than other types of students.
• A tight integration between on-campus and online
courses are essential both in content as in set up. In
future blended projects, this will be one of the focus
points.
How Leiden University Students use our MOOCs
For the years 2013 to 2015, we matched our database of MOOC learners with the on-campus
student body database. We wanted to see whether
we could identify students who followed a MOOC
before joining an on-campus course, and wanted to
understand if, and if so, how, our current on-campus
students used our MOOCs.
15
10
5
Regular Coursera
Students
Advanced track
Students
Leiden University
Students
Figure 20: Violin plot of grade distributions for regular Coursera students, advanced track students and Leiden University students that participated in the course ‘International tax law’. The boxplots show the summary statistics of the final exam for each group. The shape around the boxplot
shows the grade distribution for students in each group using a double density plot.
“It is a fun way to learn at your own pace”. Leiden
student in the MOOC “Rethinking International
Tax Law”
“I really like learning through MOOCs. It allows me
greater control over my schedule“ Leiden student in
the MOOC “Rethinking International Tax Law”’
Teaching assistants interviewing Professor of Linguistics N. Chomsky.
46 - Innovating the way we teach and learn
47 - Innovating the way we teach and learn
We identified a total of 1.961 students who participated in one of the session-based MOOCs of Leiden University.
Of these students, we identified 254 (13%) students as having followed a MOOC before attending an on-campus
course. The remaining students either followed a MOOC while already studying at Leiden University or were
alumni.
terrorism & coun. terrorism (round 4)
terrorism & coun. terrorism (round 3)
terrorism & coun. terrorism (round 2)
terrorism & coun. terrorism (round 1)
intro to European law (round 1)
international tax law (round 2)
international tax law (round 1)
miracles of human language (round 1)
miracles of human language (round 1)
rising global order (round 3)
rising global order (round 2)
254
rising global order (round 1)
13% | Potentially recruited students
federalism & decentralization (round 1)
configuring the world (round 3)
configuring the world (round 2)
1.707
87% | Leiden students
Cu
ltu
re
le
an
tr
op
ol
o
gi
Fi
s
Ja
pa
e e caal
ns
n
Pu
re
tu
o
ch
bl
n
di
tw
t
ic
es
(g
ik
Ad
ra
ke
m
du
lin
in
a
gs
ist
so te)
r
In
ci
Ps atio
o
te
yc
lo
n
rn
g
ho
(
at
lo g ra ie
io
gy
du
na
M
lR
id
a
(
gr te)
de
ela
ad
ntio
ua
O
ns
os
te
)
te
an
ns
d
tu
D
d
ip
P
ie
lo
s
m sych
ac
ol
y
og
(
i
gr
e
Cr
a
isi
Cr dua
sa
te
Po
im
nd
lit
in )
Se
ic
ol
al
cu
og
Sc Fis
rit
ie
ie
c
yM
In
a
n
a
te
ce
lr
an
rn
ec
ag
(
at
io
em gra ht
na
du
l R ent
at
e)
ela
(
tio grad
ns
ua
te
(g
)
Li
ra
b
G dua
Re era
l A es
te
ch
)
rt chi
tsg
ele s an ede
ni
d
er
Sc s
dh
ie
ei
nc
d
es
(g
ra
du
a
te
Po
)
lit
i
Be colo
Re stuu gie
ch
rs
In tsge kun
te
de
l
e
rn
er
dh
at
io
e
i
na
d
lS
tu
di
es
configuring the world (round 1)
Less students from
a particular study
enrolled in MOOC
More students from
a particular study
enrolled in MOOC
Figure 21: Results of matching the USIS database with MOOC participants. Our study indicates that MOOCs may help in recruiting new students.
Figure 22: overview of Leiden University student participation in Leiden MOOCs. The x-axis features 15 on-campus programmes which are ordered by the total
amount of enrollments in Leiden MOOCs (shown on the y-axis). Darker color shading indicates higher enrolments in a given MOOC; lighter color shading indicates
less enrolments
48 - Innovating the way we teach and learn
49 - Innovating the way we teach and learn
MOOCS AS A PLATFORM FOR RESEARCH
In 2016, we will conduct further research to
understand the connection between online education
and taking on-campus education for these students.
“Lasting effects”
Professor Stefaan van den Bogaert was the first in
the Netherlands to produce a Massive Open Online
Course: ‘The Law of the European Union’. Starting the
MOOC was a big step into the unknown, but according to Van den Bogaert it left lasting results: “When
starting the online course we had no idea what the
effect would be, we were pioneering. During recording
one cannot see the students, only the red-light of the
camera, but in fact the whole world is watching”. The
MOOC drew some 40.000 participants, following video lectures making quizes and discussing EU-Law on
the forum. Started already two years ago, the results
are still visible today. Van den Bogaert: “After having
given the course online, we noticed a growth in the
number of applications for the Master in EU Law from
both within and outside of the EU. In fact students
approached me to say that they have chosen Leiden
University because of the course. I can say now, that
there is a clear MOOC effect in Leiden.” Faculty of
Law. Instructor Stefaan van den Bogaert, course: The
Law of the European Union. An Introduction (2014).
In the nine months during which our research team
was active in 2015 we focused on developing our
data infrastructure, a research track, and on implementing actionable learning analytics. Together with
researchers at Leiden University and partner institutions, we focused our efforts on the following areas:
• We started building a data infrastructure to manage the
deluge of learner data given to us by Coursera and finding secure ways to pass this data onto other researchers.
We designed a proper infrastructure to store Coursera
data in a fast, flexible and secure way, and further created
tools and documentation to share our expertise.
• We engaged in fundamental research and set up several
research tracks that focus on different aspects of online
learning, such as learning materials, educational interventions, student behavior, and learning outcomes. We
further published four peer-reviewed research papers,
including one open-access publication at a leading conference on learning analytics.
• Learning analytics allows us to impact online learning
materials with practical insights. Among other things,
we published one report in cooperation Dr. Maartje van
den Bogaard (ICLON) in which we evaluated the use
of the MOOC Rethinking International Tax Law in the
on-campus course Tax Treaties, organized by Professor
Sjoerd Douma at the Law faculty.
Facilitating Research Through Tooling and
Analytics
Several tools and services have been created in 2015
that researchers can use to do analysis with. These
consists of a data management policy which defines
how data can be shared and used, a set of github tools
which provide analytics capabilities to researchers.
Video lecture by Dr. Ester van der Voet for her course Wheels of Metal:
Urban Mining for a circular economy.
Data Management Policy
When it comes to managing large datasets, few topics
are as important as user privacy. In order to ensure
the protection of our user’s data, we created a data
management policy in which we outline how we store
and retrieve data, and under which circumstances
and conditions we share this data with third parties
or researchers. In 2016, we further develop this policy
in light of new European privacy laws and new data
formats.
Our tools are open-source
The Online Learning Lab maintains several repositories containing tools to facilitate analytics with
Coursera MOOC data.
These tools are geared towards facilitating researchers
within Leiden University, as well as researchers from
other universities.
Check out our tools here: https://github.com/LU-C4i
50 - Innovating the way we teach and learn
51 - Innovating the way we teach and learn
Online Learning Research
Next to the provision of tools and procedures for
learning analytics, the team also started working on
a research agenda with the University wide Higher
Education Research Group (CROHO), the university’s School of Education (ICLON) and the Centre
for Education and Learning (CEL, a collaboration
between the universities of Leiden, Delft and Rotterdam). MOOCs offer new ways to look at learning.
The approach blends three disciplines; educational
sciences to investigate teaching and learning aspects,
computer science for data analysis and modeling,
and psychology to understand and study individual
processes such as motivation and information
processing.
Github Tools
We realize that the expertise to handle large
datasets is not abounding at our partner institutions,
we therefore have released several tools on the online
sharing platform GitHub. These tools allow researchers
to store, retrieve and manipulate MOOC research data.
In 2016, we will further develop these tools to include
a comprehensive package for the R statistical language.
By evaluating data from 500,000 interaction sessions
with 100 videos from our MOOCs we found that,
indeed, our students need more time finishing a video
when the complexity is either low or high, supporting
our hypotheses.
6
This research line is exemplar for answering fundamental questions within an applied setting. Using
large scale, actual usage data we applied a psycholinguistic model of information processing difficulty and
confirmed classical laboratory research hypotheses
about human attention and interest. Simultaneously,
we pave the way for selecting videos of an optimal level of difficulty for each student’s individual knowledge.
4
- Application and verification of
psycholinguistic findings
- Retrieval practice
2
PSY
- Instructional
design of videos
- MOOC design
- Community
EDU
CS
Actionable learning analytics
- Segment of users on
level of understanding
- Offer different / extra content
Information Density and Understanding
Our most recent research publication addresses a
fundamental issue in didactical videos. Our hypothesis is that students struggle to understand difficult
videos, but similarly struggle to concentrate when the
difficulty is low.
8
Dwelling rate
Data as a Service
For researchers at the University of Leiden, we offer
the opportunity to make appointments to discuss and
receive access to our wealth of learner data. We also
provide assistance in the safe storage of data in
a format familiar to the researcher.
Together, these three areas are important to provide feedback to instructional designers, to improve
actionable learner analytics, and to better understand
students and their learning process.
0
0.5
1.0
1.5
2.0
2.5
rate rate and dwelling rate.
Figure 23: The relationship betweenInformation
video information
The former is a measure of both the complexity of information and the time in
which the information is presented. The latter relates to the amount of a video
that a student sees.
52 - Innovating the way we teach and learn
53 - Innovating the way we teach and learn
COLLABORATIONS
Effective Learning Strategies: Retrieval Practice
To promote meaningful learning, an instructor
should employ educational strategies, which are consistent with human cognitive learning mechanisms.
In the case of online education, this implies that
the range of instructional study objects (e.g. videos,
animations, book chapters, articles and discussion
platforms) should be designed and implemented in
accordance with these principles. The main purpose
of this line of research is to identify the core active
ingredients of instructional study objects and get a
better understanding of how, for whom and in what
context they have an impact on learning. Specifically,
we aim to understand how students process information in an online environment, how we can enhance
the learning process to make it more efficient and
more effective, and how we can Increase the pedagogical quality of MOOCs
These studies will strengthen the body of literature
and provide instructors with evidence-based guidelines for how to design, enhance and optimize open
online courses. This, in turn, will increase the educational benefits of MOOCs for the students.
Over the past year a number of collaborations have
been strengthened or started.
Peer Reviews: Online Performance and Collaboration
Peer assessment is more than merely a scalable assessment procedure for more complex assignments. In
addition to alternatives such as automated essay scoring, peer assessment serves a formative purpose as
well; participants are considered to benefit from both
providing and receiving peer assessments, as well as
from their encounter with others’ work and perspectives. An explorative study by Bart Huisman, a Ph.D
student at the Graduate School of Teaching (ICLON),
investigates this formative aspect of the peer assessment procedure. More specifically, it investigates the
extent to which the ability of peer reviewers influences the subsequent performance of participants.
• The GitHub tools we developed have been shared with
Coursera partners, and formed the basis for successful
further exchange of experiences. It formed a stimulus for
Coursera to launch the Coursera Research Community.
• We participated in a research summit at the University
of Michigan about viable new avenues for research in
MOOCs.
• The Online Learning Lab staff works closely together with PHD-candidates from various organisations
e.g. Centre for Education and Learning from Leiden-Delft-Erasmus.
• University of Utrecht and Edinburgh, we initiated a
meeting for MOOC and online learning researchers of
all LERU partners who aim to connect and collaborate. A
second meeting will be organised in april 2016.
In 2016, our research projects will use MOOCs and
SPOCs as a vehicle for research to answer fundamental questions about teaching and learning in an
applied setting. In doing so, we simultaneously try to
explain student behavior and to stimulate productive
student behavior by answering design questions on
how to promote positive learning outcomes for our
students.
Research:
Collaboration and connections with other researchers
in the field of MOOCs and online learning:
Coursera:
With Coursera there are a number of collaboration
projects which will be built on this coming year:
• In 2015 we decided to become an innovation partner of
Coursera which enables to test new ideas quicker and
know earlier than most how the MOOC space is developing.
• In 2016 the Coursera Partner conference will be hosted
by Leiden University and University of Geneva.
Collaboration with University partners
Looking for ways to collaborate with other partner
universities can be helpful in developing new programmes or influencing policy.
• Discussions regarding the accreditation of MOOCs with
LERU partners and the Dutch ministry of education.
• Further develop specialisations together with University
of Geneva
• Explore the setting up of a Global Master’s programme
with two other universities.
54 - Innovating the way we teach and learn
55 - Innovating the way we teach and learn
NEXT STEPS
In 2016 the Online Learning Lab will take further
steps in the following innovation lines.
Exploring New Technologies
Focus on storytelling, visual technologies and pedagogy. Online learning initiatives represent a new
field of education in which there is still a need to
experiment. First attempts have shown that implementing traditional methods such as lectures and
talking heads in an online environment do not make
use of the full potential of online learning. As such,
the question becomes: how do we move forward to
an evidence based, flexible approach, that allows us to
further test and implement new educational technologies within a rapidly changing technological landscape? The aim is to bring together instructors and
creative professionals to create work as Virtual Reality
(VR) and interactive media that will allow students
to engage with these questions through purposeful
projects that unite storytelling with innovative visual
technologies.
Recognition of MOOCs and Online Learning Towards Credit
How can we ensure that universities recognize each
other’s online courses towards credit? This is both
a question for open courses that currently offer
non-formal certificates and for closed online accredited courses that offer ECTS. Which regulations apply
and, if necessary, how should they be adapted? How
can we guarantee the validity of exams at a distance?
What quality should the instructional design of the
courses have? How far can we go in offering personalized choices while maintaining a coherent curriculum? And how do we scale this processes within our
departments?
In collaboration with the department of Academic Affairs, faculty, LERU and other (inter)national
partners, we will elaborate solutions for recognition
and accreditation of open and closed online courses.
Our vision is that Leiden University students, within
a few years, can benefit from a qualified offer extending beyond the Leiden programs and vice versa, and
can personalize their studies to prepare themselves
optimally for their future careers.
Piloting Online Proctoring
In 2015 we published Effective online education requires valid online assessment: could online proctoring offer the answer? coauthored by Marja Verstelle
(Online Learning Lab) and Marinke Sussenbach
(TU Delft) in the SURF Open and Online Education
Trend Report 2015. The coming year we will organize
an online proctoring pilot to better understand and
develop the monitoring of exams at a distance. Several of the SPOCs we are developing with faculties in
2016 will offer online proctored exams. We will report
on the possibilities and requirements needed to scale
this service.
Designing for Deep Learning
One of the main goals of online learning is to
promote “deep learning”, a term commonly used
to signify learning beyond remembering and reproducing. Deep learning challenges the student to
apply, analyze and evaluate and finally to create new
knowledge. It leads to long-term retention of the
learned as opposed to short-term retention used only
to pass the course. The Online Learning Lab regards
the integration of deep learning in online material
as one of the main challenges for 2016. On the one
hand, our research team looks into ways to stimulate
this in a variety of ways. On the other hand, online
learning can harness deep learning in contact hours,
often referred to as Flipping the Classroom. We will
run 5 Flipped Classroom experiments and evaluate
the impact on deep learning.
Another way to look at this is by asking instructors
what the most important things they want to research
about learning and teaching in their specific field/
subject. What are the concepts students most struggle
with and can we design experiments to figure out
what will fit students best? These can then be modeled
into experiments in an online learning context.
Adopting an Online Mindset for Our Physical
Spaces
In the coming year we focus more on designing
the classrooms and learning spaces. Technological advances (be it digital or material innovations)
increasingly shape the physical environment in
which students learn, study and work; the university
building of the future could look very different from
the ones we have today. All spaces are collaborative
spaces which are adaptable which can be tailored to a
specific learning approach. One vision is that teachers
have a deeper understanding of the use of didactics in
a physical environment. This is where social learning and individual coaching based on data driven
learning analytics come into play. Though tracking
students in the classroom teacher will see a student’s
performance and will be able to define appropriate
action in real time while operating in an interactive
learning environment.
Experiment with Alternative Monetization Structures
In order to secure sustainable funding to keep our
MOOCs running and up to date, we aim to experiment with new forms of user contributions. Only
56 - Innovating the way we teach and learn
57 - Innovating the way we teach and learn
ACKNOWLEDGEMENTS
a small part of learners aims to get a certificate,
while we know many of them would love to support
us in some way to keep making and updating free
MOOCs. The crowdfunding pilot described earlier
in this report proved to be a positive experience. Our
conviction is that monetization forms like sponsoring and crowdfunding are monetization structures
that fit within our vision to maintain open content.
The upcoming MOOC De-mystifying Mindfulness
is a good example of a topic in which these type of
contributions from participants are more opportune
than fees for certificates. In this MOOC we will run
and evaluate a pilot in 2016 with different forms of
contributions and share the outcomes with Coursera
and the MOOC community.
Stimulate sustainable development of course content
Most faculties that are developing MOOCs, SPOCs
and other online materials have the intention of using
the online materials in their on-campus education.
The effort and costs of developing the materials is
high, put pays itself back when the materials can be
used in different settings and over many years. As
such, the Online Learning Lab needs to focus more
and more on modularity: making small chunks of
content which are independent of the specific context
in which they are produced first. For example, this
means leaving out numbering, references to recent
events, and splitting content that will not change soon
from topics that might, although it is not always easy
to foresee (who had expected two years ago that the
Schengen Treaty is under threat today?).
A second approach towards sustainability is the reuse
of materials of others, which we think is as much a
cultural issue as a technical and practical one. Therefore, we start with challenging ourselves to use existing online (materials in our) courses for faculty: teach
as you preach. We aim to develop SPOCs in such a
way that they can be cloned for use in other domains
of Leiden University. And for faculty we aim to develop a simple support addressing three concerns:
• Where can I quickly find high-quality materials?
• How can I adapt them for my class?
• How am I allowed to use it and if needed, how can I clear
copyrights?
Exploring the Online Master Space
Leiden University has at least one blended master
(Advanced Master Air and Space Law) where students do part of the work remotely. Over the past few
years a number of full-fledged online master programmes have been developed which provide more
flexibility for students and a greater reach of existing programmes. In 2016 we will be exploring the
possibilities to develop a number of online (blended)
programmes with various faculties. For example the
Leiden Medical School’s (LUMC) future lab would
like to create a masters programme which will consist
for 75% of online components. In the law (Tax Law)
and Archeology departments there is also interest in
the development of online masters. Online education
can also help develop new and deeper partnerships
with other universities and organisations, for example
through teaming up with a number of universities
and develop a global master programme where students travel to different campuses for short intensive
periods of time and to parts of the degree online.
This report is the collective achievement of a large
number of colleagues who spent massive amounts
of time developing methods, tools and courses.
Without their time, enthusiasm, support and
ideal all this would not have been possible.
Online course teams
Rethinking International Tax Law (MOOC)
(one session-based MOOC and transition to on-demand)
Prof. mr. S. Douma, Mr. drs. Judith Reijnen,
Max van Boxel
volunteers: Pablo Angel, Paula Beneitez, Stewart
Cotteril, Vasileios Dafnomilis, Pedro Luz, Mahesh
Nayak, Jessica (Chu) Shi.
Federalism & Decentralization; Evaluating Africa’s
Track Record (MOOC)
(one session-based MOOC and transition to on-demand)
Dr. J.G. Erk, Miranda Verboon, Gerrit Krol
volunteers: Terresa Lewis, Saskia Kellenbach.
Miracles of Human Language; an Introduction to
Linguistics (MOOC)
(session 1 and 2)
Prof. dr. Marc van Oostendorp, Inge Otto, Marten
Vermeulen
volunteers: Joshua Asmah, Valeria Hernandez Reyes,
Saskia Kellenbach, Terresa Lewis, Kaori Muramatsu,
Jasper Spierenburg, Kate Taylor, and 400 translators.
Terrorism & Counterterrorism: Comparing Theory and
Practice (MOOC)
(session 4 and the transition to on-demand)
Prof. Dr. Edwin Bakker, Jeanine de Roy van Zuijdewijn
Volunteers: Sana Abdeljalil, Ali Ahmed Dyrs,
Joshua Asmah, Stewart Cotteril, Mark Hatz,
Valeria Hernandez Reyes, Saskia Kellenbach,
Kaori Muramatsu, Iris Soliman.
The Changing Global Order (MOOC)
(session 2 and 3)
Prof. dr. Madeleine Hosli, Denise de Buck
Volunteers: Joshua Asmah, Stewart Cotteril,
Terresa Lewis.
Configuring the World; A Critical Political Economy
Approach (MOOC)
(session 2 and 3)
Prof. dr. Richard Griffiths, Einat Shitrit
Volunteers: Danny Damen, Saskia Kellenbach, Leo
Landoll (Sandy), Kaori Muramatsu, Stewart Cotteril.
Configuring the World; A Global Adventure (SPOC)
Prof. dr. Richard Griffiths, Einat Shitrit.
Capstone Global Affairs
(session 2)
Prof. dr. Richard Griffiths, Prof. dr. Madeleine Hosli,
Einat Shitrit.
The course teams of courses that started in 2015 and
launch in 2016
International Law in Action: A guide to the International Courts and Tribunals in the Hague (MOOC)
- January 2016
Prof. mr. Larissa van den Herik, dr. Yannick Radi,
dr. Cecily Rose, Lieneke Louman.
Volunteers: Stewart Cotteril, Tatiana Ramirez,
Jaymie Wink, Louis Léonet.
58 - Innovating the way we teach and learn
59 - Innovating the way we teach and learn
THE ONLINE LEARNING LAB
Centre for Innovation online learning team
The Rooseveltian Century (MOOC) - January 2016
Prof. dr. Giles Scott-Smith, dr. Dario Fazzi
Volunteers: Stewart Cotteril, Saskia Kellenbach.
Clinical Kidney Transplantation (MOOC) - January
2016
Dr. Marlies Reinders, Prof. Frans Claas, prof. Cees
van Kooten, dr. Sebastiaan Heidt, dr. Volkert
Huurman, prof. dr. Hans de Fijter, prof. dr. Eelco de
Koning, dr. Andre Baranski, prof. dr. Ton Rabelink,
dr. Aiko de Vries, dr. Jan Nico Bouwes Bavinck, prof.
dr. Leo Visser, dr. Ingeborg Bajema, Bart Oerlemans,
Erik Pasveer, Manon Dijkerman.
Volunteers: Maarten Engelse, Bram Voorzaat,
J.R. Bank, Greetje Dreyer.
Be Persuasive: write a Convincing Position Paper or
Policy Advice (MOOC) - February 2016
Prof. dr.Richard Griffiths, Miranda Verboon.
Volunteers: Dragana Djordjevic, David Talbot.
Anatomy of the Abdomen and Pelvis; a journey from
basis to clinic. (MOOC) - April 2016
Prof. dr. Marco de Ruiter, Daniel Jansma MSc, MD
Paul Gobee, Dr. Beerend Hierck, Bas Boekestijn MSc.
Volunteers: Teressa Lewis, Denise Arnold,
Yara Willems, Aglaia Hage, Thomas Kluck.
International Law in Action; Investigating and prosecuting International Crimes (MOOC) - Spring 2016
Prof. dr. Carsten Stahn, Lieneke Louman.
Heritage under Threat (MOOC) - Summer 2016
Dr. Sada Mire, drs. Mara de Groot, Krijn Boom, Katrin
Hannen, Prof.dr. Maarten Janssen, Joseph Powderly,
dr. Henrike Florusbosch, Prof. dr. Gerard Persoon,
Prof. dr. Petra Sijpensteijn, drs. Marieke van Haren
Gideon Shimshon
Co-founder Online Learning
Lab & Director of Centre
for Innovation
Marja Verstelle
Co-founder Online Learning
Lab & Learning Innovation
at Centre for Innovation
Evolution Today (MOOC) - Fall 2016
Prof. dr. Menno Schilthuizen, dr. Rutger Vos,
dr. Maurijn van der Zee.
Jasper Ginn
Data analyst & research
Tanja de Bie
Project Coordinator Learning
Innovation & Community
manager
ICTO Program steering committee
Simone Buitendijk, Kurt De Belder, Jan Willem
Brock, Frank Damen, Henk Dekker, Hans van Dommele, Evert Fortuin, Marlou Grobben, Joost Kok, Rick
Lawson, Paul Nieuwenburg, Jos Schaeken, Bernard
Steunenberg, Marja Verstelle, Han de Winde.
Annemieke van den Bijllaardt
Online Learning expert
Thomas Hurkxkens
New Media director
Leontine van Melle
Project Coordinator Learning
Innovation
Einat Shitrit
Project Coordinator Learning
Innovation
Nestor Romero Clemente
Video editor Open &
Online Learning
Frans van der Sluis
Postdoctoral Researcher
On Being a Scientist (SPOC, MOOC) - Summer 2016
Prof. dr. Bas Haring, Prof. dr. Frans van Lunteren,
Prof. ing. Ionica Smeets, Remco Schadewijk, Saskia
Kellenbach.
Coursera Partnership Team
Daphne Koller, Emma Webb, Melanie Lei, Meera
Ramakrishnan, Evan Shore, Isaac Guttman,
Alex Sarlin, and Coursera partner support team.
60 - Innovating the way we teach and learn
61 - Innovating the way we teach and learn
CENTRE FOR INNOVATION - ONLINE LEARNING LAB STRUCTURE
APPENDIX I: METRICS PER COURSE
The Online Learning Lab is founded and managed
by the Centre for Innovation at Leiden University.
The Centre for Innovation helps to prepare Leiden
University for the digital future by identifying and
exploring technology-induced trends in education
and research.
The mission of the Online Learning lab is to experiment with new developments in online and open
education. It further provides an online teaching
platform for teachers to use as starting point for their
own ideas about - and new possibilities of the use of
- online courses. The operating principles regarding
open and online learning are:
• Innovate: develop new concepts around MOOCs and
Online Learning by accelerating new initiatives in online
learning and keep positioning UL at the forefront of
online learning developments.
• Service: Further develop services in online learning,
didactics and production for campus education that can
be used by all faculties in the university.
• Scale: Scale up our current MOOC engagement base
from 150.000 to 1.000.000 joiners by 2017.
• Execute: Delivery of the online learning project proposals
together with university partners.
• Standardize: Develop a professional production team for
online education (MOOCs and SPOCs) that functions
for the entire university.
• Develop: SPOC’s and new online projects with faculty for
education to on campus students.
• Lessons Learned: Based on experience and lessons
learned contribute to strategic discussions of Leiden
University around the new LMS platform and long term
physical infrastructure vision.
The core support team is responsible for learning
and teaching innovation, and project management.
This includes instructional design, video production,
coaching, technical support, development support,
online community management, communication and
participant recruitment, evaluation and data analysis.
The organisation is based on a network model and
deploys agile methods to ensure that innovation and
production quality ar aligned at all times.
A broader network is involved to help with issues
around copyrights clearance (Library UBL) and for
contract and IP issues (LURIS) Also the lab works in
collaboration with various researchers at the Leiden
University centers for research into higher education (ICLON-Leiden University Graduate School of
Teaching and CROHO) and at the Centre for Education and Learning (Leiden University, TU Delft
and Erasmus University). On press and recruitment
approaches the Lab works with the Leiden Strategic
Communication and Marketing Department (SCM).
During the production process MOOC teachers are
supported by a junior teacher or student assistant
who takes care of the course coordination together
with the project management team of Online Learning Lab.
Session-based courses
Course
Session
Enrolments
Signature Track
completed learners
Configuring the World
2
20.225
541
910 (4.5%)
Configuring the World
4**
8.957
428
642
Rethinking Tax Law
1
14.203
538**
767 (5.4%)
Changing Global Order
2
14.899
576
990 (6.6%)
Changing Global Order
3
12.848
569
823 (6.4%)
Federalism and
Decentralization
1
3.788
84
223 (5.9%)
Miracles of Human
language
2
16.058
309
745
(7.1%)
(4.6%)
Miracles of Human
language
1
49.924
435
2536 (5%)
Terrorism and
Counterterrorism
4
11.769
231
703 (5.9%)
Course
Running since
# Total enrolled students
# Paid Certificates
# course completers
Federalism and
Decentralization
September 2015
2.903
62
22
Rethinking international
Tax Law
November 2015
10.468
240
105
Terrorism and
Counterterrorism
March 2015
76.560
1.660
1.844
On-demand courses
** Of which approximately 53 Leiden University students for whom the certificate was gifted by the university.
** There is no third session for this course.
62 - Innovating the way we teach and learn
63 - Innovating the way we teach and learn
Ontwerp: Studio Piraat