Achieving Learning Objectives
Online: Not All Platforms are Equal!
Nick Feamster
Georgia Tech
A Tale of Two MOOCs
• A free Coursera MOOC on Software Defined
Networking to over 50,000 enrolled students
– Scalable delivery, assignments, and forums
• A for-credit Udacity “MOOC” on Computer
Networking for Georgia Tech’s Online MS
program, to 200+ enrolled students
– Not so scalable…
Subtle differences in platform design have significant effects
on a course’s ability to scale.
MOOC #1:
Software Defined Networking
• Extremely hot, new topic
– No existing “real world” courses on the
topic (yet)
– A chance to develop an archetype course
(and material) that others might use in
their own courses
– I wanted to be copied. This seemed like
a good way.
• Boutique topic means that there were
no “set in stone” ways of teaching the
course
– Easier to think outside the box, since
there was no existing material anywhere
http://blog.sflow.com/2012/05/software-defined-networking.html
Two Offerings So Far:
• Summer 2013
(6 weeks)
• Summer 2014
(8 weeks)
Who Takes The Course?
• 50,000+ people
registered each year
– 10,000 people active
each week
– 1,000-3,000
watching lectures
(3,500+ in 2014)
• 500+ turn in
programming
assignments each
year
Other Demographics (Summer 2014)
• 55,000+ Enrolled
• 197 Countries
– 31% from the United States
– 15% from India
• Sex and Age
– 86% Male
– 74% 20-39 years old
• Education and Employment
– 58% Employed full-time
– 26% Full-time students
– 38% have Undergrad degree, 31% have Masters
What the Instructor Sees
How Was the Course Developed?
• Course lesson plan, with learning objectives
• Top down design:
– Figure out modules, continue subdividing until
you get 10-minute “lectures”
• Production
– Lecture Filming and Production
– Assignments
– Syllabus
• Operation
– Forums
Filming Take 1: Studio
• Started in a studio at Georgia Tech
• This quickly proved intractable
– Studio is distracting. Lots of people watching
makes recording nerve-wracking.
– Studio time is limited, contentious, and expensive.
– Post-production quality was not to my liking.
– Faculty members need scheduling autonomy.
• A lot different from a TV interview!
Filming Take 2: Camtasia
• Studio people will claim that quality suffers.
• I believe this is bogus, for several reasons.
– People can take their time recording, take breaks, record
when they are “in the zone”.
– Screen captures from a laptop are easy.
– Recording quality from a good laptop is quite fine.
– Nobody has ever complained about the quality of the
recordings. Content is what matters most.
• Takes a little getting used to at first (looking at the
camera, etc.)
• You have to do your own editing.
Producing Lectures in Camtasia
• General rule: 10-15 minutes of lecture == 2
hours of recording and editing
– This does not include time to prep slides, etc.
• This is way different than a usual lecture!
– Biggest lesson: Silence is quite easy to edit!
– Stumbling is OK: Just say the same thing again
– You can also “practice” while the video is rolling
• Need to find a quiet, well-lit location
• Takes patience, but it starts to be quite fun
General Lecture Production Process
• Determine learning objectives
• Research the topic
• Write slides
– General format: Summary/outline, 7-10 content slides,
wrap-up
• Plan demonstrations
– Plan on-screen landscape
•
•
•
•
Record (one sitting for consistency…big challenge!)
Edit and export
Upload
Note: You can prepare lectures out-of-order!
New Modes of Lecture Production
• On-screen demonstrations
– Videos lend themselves very nicely to onscreen demonstrations
– User can see things in “life size”
– User can pause, rewind, etc. Very helpful
tutorials for working through assignments
• Interviews with luminaries in the field
– Google “On Air” Hangouts
– Can be done anywhere, and people are
willing!
– Process: Invite, script questions, send,
revise. Stay roughly on script
– Many people have since copied this idea
(Nick McKeown, Alex Orso, Mike Hicks, etc.)
Lessons from Lecture Production
• Lots of money is not needed. Students want
content, clearly delivered. Camtasia works.
• A 45–60-minute lecture is about 15 minutes of
well-polished video. Content distills well.
• Lecturer is always “ahead” of the students.
This creates interesting time-shifting effects,
with advantages and disadvantages.
• MOOCs are asynchronous and autonomous
for the student. This is a big reason for their
success. Production should be the same.
How Was the Course Developed?
• Course lesson plan, with learning objectives
• Top down design:
– Figure out modules, continue subdividing until
you get 10-minute “lectures”
• Production
– Lecture Filming and Production
– Assignments
– Syllabus
• Operation
– Forums
Developing Scalable Quizzes:
Stick to the Basics
• Problem: The platform is not very good at handling anything
that’s not multiple choice or simple numerical answer.
– Short answers become massive regular expressions
– …will likely improve as the software gets better
• Stick to multiple choice if possible
• Main differences from a normal MC quiz
– Helps to write up explanations for each option
– Need to figure out if students can re-take the quiz, and if
so, how many times
Developing Assignments is Hard
• Wanted to teach concepts and programming
– Not a “heavy-duty” programming course, but
some concepts are best explained and learned
through short code exercises.
• Code submission possible, but grading must
be done on regular expression evaluation of
output
• Problem: How to guarantee uniform
programming environments?
Uniform Environment: Virtual Machine
• Asked students to:
– Install VirtualBox (or equivalent) virtual machine emulator
– Download 64-bit VM image that had everything prepackaged (no asking students to install software
themselves). Kind of like a lab at home.
• This has generally worked. A few problems
– VM is huge (~1 GB), so have to get it right the first time
– Not everyone has a 64-bit machine
– Certain performance, even when run in an emulator, varies
depending on underlying hardware
How Was the Course Developed?
• Course lesson plan, with learning objectives
• Top-down design:
– Figure out modules, continue subdividing until
you get 10-minute “lectures”
• Production
– Lecture Filming and Production
– Assignments
– Syllabus
• Operation
– Forums
Communicating with 10,000 Students
Can Scale!
• I feared absolute disaster
– Small glitches in
assignments in a classroom
of 50 students create
mayhem.
– I feared that I would be
spending my life answering
minutia in online forums.
• Thankfully, this did not happen!
• Lesson: Giving an assignment
to 10,000 students in a MOOC
is easier than giving it to 50 in a
classroom!
Why Assignments Can Scale
•
•
•
First of all, you do your best to work
out the bugs, mostly out of fear 
– We attempted everything that
we assigned, sometimes on
multiple platforms
More importantly: Self-selection!
– Everyone doing the assignments
wants to be there
– This makes a huge difference
– People not only fix issues
themselves, they rewrite
documentation
Caveat: Course difficulties vs.
platform difficulties are currently
hard to tease apart. Some of this is
“user error”.
A student’s blog post describing part of
the assignment setup in detail…
Some Feedback…
A Tale of Two MOOCs
• A free Coursera MOOC on Software Defined
Networking to over 50,000 enrolled students
• A for-credit Udacity “MOOC” on Computer
Networking for Georgia Tech’s Online MS
program, to 200+ enrolled students
“MOOC” #2: Online MS at GT
• Online version of CS 6250: Computer
Networking
• Two offerings thus far
(Spring and Summer 2014)
– About 120+ students in each offering
– Paying tuition for a “degree”
– Not specialists in networking
• Technical difference: Platform
How Was the Course Developed?
• Top-down planning, with “scripts”
• Several full-time staff members
– Course developer from Udacity
– Video editor from Udacity
• Part-time from others
– Three paid TAs/Graders
– Many other people handling logistics
• Course recorded in studio, ahead of time, with custom
equipment
– Recording done in “marathon stints”
– Not possible to integrate interviews
• No auto-grading, little visibility of content, etc.
• Initially, all material released at once (huge mistake)
Comparing the Approaches
Coursera
(SDN MOOC)
Udacity
(OMS “MOOC”)
Video Delivery
Camtasia-recorded videos,
talking head and slides + onscreen demos
Tablet writing + moving hand
Grading
Simple MC quizzes, autograded programming
assignments.
No auto-grading! TAs graded
assignments “manually”.
Forums
Initially “home brew”. Moved
to Piazza.
Piazza, but the students expect
more direct instructor
engagement.
Pacing
One week at a time,
synchronous (though people
fall behind)
All-at-once release.
Demographics
Professionals and students
taking the course for free, for
enrichment.
Paying students who are not
necessarily interested in the
subject. Entitled.
Cost
300+ hours of my time, 40+
hours of TA time (all “Free”)
Estimated: $150k (staff
salaries, studio time, etc.)
Leave the Pedagogy to the Instructors
• Substance trumps style.
• A MOOC platform should allow instructors to
deliver the content in whatever format they
see fit.
• Mandating a particular “style” emphasizes
style over substance.
GT OMS Degree “MOOCs” Are Neither
Massive nor Open
• One of the important features of MOOCs that allows
scaling is community support.
• Things that dilute this support:
– Students who pay and expect staff contact
– Students who are not interested in the content,
but rather trying to get a degree
– “Go at your own pace” style (students are on their
own, course staff can’t keep up)
• Other platform-specific factors prevent scaling
– Instructor has less autonomy in creation and
management of content
Lesson #1: MOOCs Change the Game
for Instruction
• Everyone is watching you
– In seven years, never got feedback (or
acknowledgment) about on-campus course
– Now, everyone’s eyes are on you
– Huge pressure to do a good job
• MOOCs are a “forcing function” for introducing
new media into the classroom
– I wouldn’t have thought to do “on air” interviews for a
classroom; people would likely decline them anyhow
– I will likely re-use some MOOC content in the “real”
classroom
Lesson #2: Time Investment Pays Off
• Typical Course Creation (45-minute lecture)
– 1-2 hours to create slides
– 1 hour to deliver the lecture
– Number of students: 50-100
– Total: 3 hours per lecture
– … but then you have to do it again! (can’t save an awesome
lecture, students can’t rewatch, etc.)
• MOOC Course Creation (10-minute video)
– 1-2 hours to create slides
– 2-3 hours to produce the lecture
– Total: 3-5 hours per lecture
– Number of students: 500-100,000
– … but, you have the archive. Can keep improving it, use it to
enhance in-classroom discussion, etc.
Lesson #3: Certain Aspects of MOOCs
Scale Surprisingly Well
• Lectures are watchable by thousands
– The scale can be a motivator for guest lectures!
• Programming assignments can be graded
automatically, if designed well
• Self-motivated, self-selected students fix
glitches, help each other out
Conclusions
• Scalable MOOCs require
– Community engagement
– Low barriers to creating content
– Support for a diversity of delivery modes to match
the material
• Not all platforms are equal
– A platform should give the instructor autonomy
and visibility.
• For-credit “MOOCs” face far different scaling
challenges than true MOOCs.
– At the moment: neither massive, nor open