Researching for Better Online
Instructional Methods using AB
Testing
--- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, Neset Demirci and David Pritchard
Teaching is an old practice…
We enter the age of the Internet
Now we can teach like this
Can we find better ways to
teach in an online environment?
In a normal MOOC
Entire MOOC Population
Course Material
A/B Testing
Traditional Teaching Method
Experimental Teaching Method
A/B Testing
Simple Design of an A/B Expriment
Common
Assessment
Treatment
Control
Multiple A/B Experiments in a Course
Unit 1
Experiment 1
Multiple A/B Experiments in a Course
Independent Random Splits
Unit 1
Experiment 1
Unit 2
Experiment 2
Multiple A/B Experiments in a Course
Unit 1
Experiment 1
(part 1)
Unit 2
Experiment 2
Unit 3
Experiment 1
(part 2)
Summary of A/B Experiment Features
•
No limit on the number of groups, nor on the number of
experiments in a course.
•
Each group can receive a different number of materials.
•
Group assignment is random.
•
No student can "opt-out" of any of the experiments.
Examples of A/B Experiments in our MOOC
Our MOOC: 8.MReV
Content: College introductory level mechanics (calculus based)
Length: 14 Weeks (12 required, 2 optional) May 29th ------- Sep 14th
Initially Registered: 10709
Attempted Second Week’s Quiz: ~4100 (estimated)
Our MOOC: 8.MReV
Total Number of AB experiments: 7
Experiment 1: Does length of video make a
difference?
Does length of video make a difference?
Office hour videos: ~10+ minutes
Talk about a challenging homework
problem
Does length of video make a difference?
Short Video
~2 minute video segments
Does length of video make a difference?
Short Video
~2 minute video segments
Does length of video make a difference?
Short Video
~2 minute video segments
Long Video
~10 minutes whole video
Does length of video make a difference?
Short Video
~2 minute video segments
Long Video
~10 minutes whole video
Experiment Procedure
Challenging Homework Problem
(Graded)
Experiment Procedure
Challenging Homework Problem
(Graded)
Long Video
Short Video
Experiment Procedure
Challenging Homework Problem
(Graded)
Long Video
Short Video
Does length of video make a difference?
Research Questions:
• Difference in time spent watching video?
• Difference in number of questions answered?
• Difference in successfully solving the challenging
homework problem?
Increasing Interactivity of Course Material
Interactivity of Course Material
Traditional problem vs. Direct measurement video
Traditional Physics problem
Provides all necessary
information
Direct measurement video (by Peter Bohacek)
Distance measured by on
screen ruler.
Time measured by
counting frames
Player can play frame by frame
Direct Measurement Video
Survey
question
Traditional
Homework
Direct
Measurement
Video
Common
Homework
Problem
Direct Measurement Video
Research Questions:
• Does the extra interactivity result in more
participation, or does the extra work lead to less
participation?
• Can we identify any positive impact on students’
knowledge to justify the cost of creating these
videos?
Interactivity of Course Material
Multiple Choice vs. Drag and Drop Problem
Multiple Choice
Multiple Choice
Multiple Choice
Traditional Multiple Choice question
Extraneous Cognitive Load
Drag and Drop Example
Drag and Drop Example 2
Drag and Drop and Deliberate Practice
Deliberate Practice: (Erickson)
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
• Repetitive and highly focused training
Drag and Drop and Deliberate Practice
Deliberate Practice:
Domain Specific Problem Design
• Focus on practicing one skill at a time
• Providing immediate specific feedback
Problem Design/Auto Grading/
Customized Feedback
• Repetitive and highly focused training
Drag and Drop format
Experimental Setup
Entire MOOC population
Experimental Setup
Experimental Setup
Multiple Choice vs. Deliberate Practice
Research Questions:
• Does increasing interactivity result in more
engagement? (i.e. Are students more
“deliberate” in their practice?)
• Compared to traditional problems, do Drag and
Drop problems serve as better/worse
assessments?
• Do “deliberate practice” result in any detectable
learning gain?
Thank you
Comments and Questions?
Discussion Questions
• Other innovative instructional methods worth putting
online and testing via AB experiment?
Discussion Questions
• How should we design our experiments to suit the (unique)
MOOC population?
• How can we transfer the experimental results obtained
through a MOOC to a "flipped" classroom despite the
difference in population?
Discussion Questions
• What criteria should we use to judge whether one pedagogy
is better than another? Learning? Engagement? More
discussion? Less attrition?
Discussion Questions
• How do we design effective assessments of learning when
students have the freedom to access any external resources?