DEVELOPING A RESEARCH AGENDA
FOR MOOCS IN ENGINEERING
EDUCATION
Don Lewis Millard, Ph.D.
National Science Foundation
Setting the Stage…
 Disruption
 Leveling the Playing Field
 More Bang for the Buck
 Digital Democratization
Disruption Afoot
 MOOCs focused an intense spotlight on higher
education and LEARNING
 MOOCs leverage behavior theory:



Bite-sized video lectures/examples
Computer assignments/reinforcement
Assessment/peer discussion
 MOOCs have only begun to scratch the surface of
the nexus of brain, cognitive science, and education
research
Research-based MOOC Principles *
 MOOC learning is highly dependent on prior knowledge
 MOOC motivation is critical – it determines, directs, and sustains
what students do
 How students organize knowledge in a MOOC - influences how
they learn and apply what they know
 Goal-directed practice, coupled with targeted feedback,
enhances MOOC learning quality (vs. the grade)
 Self-directed (MOOC) learners need to monitor and adjust their
learning approaches
an image, flowcharting)
(flash cards, problem solving, working with a peer, drawing
 Intellectual, social, and emotional climate of the MOOC course
has significant impact on student perception and outcomes
*Adapted from the book: “How Learning Works: Seven Research-Based Principles for Smart Teaching”
Sense and Meaning
 Does this make sense?
Based on experience
 Does it have meaning?
Material relevant to the learner
 Meaning is more significant for longer-term storage
 Students will remember more if provided less at any
given time
(average capacity of working memory is 7 chunks)
Let’s Test This…
7032924620
Was This Your Number?
7032924620
(703) 292-4620
MOOC Dimensions
 Questions: video interspersed with interactive questions
get users thinking, check their understanding, make it
fun, etc…ultimately, help students learn. Questions
should NOT be there to merely evaluate…
 Collaboration: great way to learn (supported by
research) - students need to use collaboration to
enhance learning, not to share answers without
understanding them
 Video Utilization: users retain more if they take notes,
draw diagrams, make notecards, and actively try to
attach meaning & make sense of the material
Acknowledgement: Udacity – from Steve Blank’s “How to Build a Startup”
Need to Focus on Quality
 Content
 Design (focused on student engagement)
 Instructional delivery
 Assessment/Feedback
 Mentoring
 Outcomes
Leveling the Playing Field
Can massive open online courses (MOOCs)
level out the playing field and make higher
education accessible to more students?
One View of the Playing Field
Income
Level
Academic
Achievement
College Degree
Success Rate
Lower Quartile
Upper Quartile
9%
Upper Quartile
Lower Quartile
80%
Acknowledgement: Michael Crow, President, Arizona State University
Leveling the Playing Field
 Unrestricted registration
 Ubiquitous access – ability to personalize
content
 MOOCs can be great organizing structures
for the vast array of on-line content
 MOOCs offer first step toward the potential
to break from historical time restrictions
(various size content, semester/course timelines)
Fertile Ground for Leveling
 Data from MOOCs:
 IP addresses
 Access/Submission times
 Assessment results
 Context-specific recorded data about the video,
lab, discussion, or assessment
 Text-field submissions
 Discussion boards

Logs – User 56937 paused lecture video 13b at 5:48:30 PM
on July 16, 2013, from IP address 194.158.64.0
MOOC Data
 2012 ECE MOOC offering
 154,763 Registrants
 230,000,000 Clicks: Able to zoom out to analyze
1000’s of students or zoom in to follow a single
student’s trajectory
 Instantaneous Data/Feedback
 Enrollment
 Participation
 Curriculum
 Achievement
Approaches to Reform
 Evidence-based approach - believing
teaching and learning will be advanced by
establishing clear rubrics to measure student
outcomes and testing experimental efforts to
increase it
 “Artisan” approach - teachers can improve
over time through improving their intuition
and personal experience (without a need
for data-driven reform)
Direction of Educational Reform
 Tension between quantitative/qualitative methods:




facts vs.Quantitative
knowledge
Facts
skill vs. wisdom
Skill
information
vs. insight
Information
solution vs.
intuition
Solution
Qualitative
Knowledge
Wisdom
Insight
Intuition
 MOOCs are likely to move the needle further and
faster toward quantifiable rubrics and away from
qualitative measures
 Worthy of Consideration: Are engineering educators
measuring the “wrong” things - because the “right” things
are not amenable to measurement?
Increasing the Buck’s Impact
Can MOOCs produce more bang for the
buck: through helping to reduce dropouts
and the time spent in higher education?
Caution…
 Free Ride: very few studies have been
performed to analyze the results on learning
– analysis has been limited to a massive
number of press articles and blogs
 A bit of what has been presented has some
built-in bias (e.g., those who have produced
MOOC offerings)
One Example - SJSU MOOC Pilot Study
 Mentoring had a significant positive impact
Stimulated discussion, provided encouragement
 Fostered an understanding of individual needs
 Advanced critical thinking and problem solving
abilities

 SJSU Pilot Study: SJSU Evaluation Report
Problem with implementation – politically charged,
non-uniform buy-in, rushed…
 A faculty member summed up the pilot with:
“Udacity has brought to the table ways to make the
courses more inquiry-based and added real life
context”

Students in the Driver’s Seat
 MITx Circuits MOOC offering: 155,000  7,157 (4.6%)
approximately 40 years worth of students!
 Students requested website be kept up at the end of the
course
 Students developed a text viewer for mobile devices to
augment MITx platform
 Students stridently pushed for enhanced personalization -
extending homework and exam deadlines
 “Shaky hand drawings” were preferred to polished slides
Promise of “Big Data”
https://newsoffice.mit.edu/2014/mit-and-harvard-release-deidentified-learning-data-open-online-courses
Digital Democratization
Are universities ready to commit themselves
to digital democratization?
Digital Democratization
 MOOCs are forcing universities to increase the
importance of teaching – as part of core mission
 Need to shift mindset from merely presenting
material - to providing focused guidance along
with opportunities to put knowledge into practice
(universities do this best) – building metacognitive
skills
 Key Inflection Point:
When a MOOC-based ENG program of study
leads to a degree from an accredited institution
MOOC Opportunities
 Take advantage of the emergence of the
learning sciences and their application to
educational practice
 MOOCs need to develop ways to incorporate
higher order skills (problem identification,
innovation in design, critical thinking)
 Create new business models - that effectively
combine instructional quality, lower cost, and
increased access (unlimited scalability of MOOCs)
Increasing the Value of a Buck
 MOOC 2.0

Focus on typical students (vs. top performers)

Leverage campus resources/local environments
(e.g., hybrid, flipped classrooms)

Credentials/Certification (whole course?)

Personalized pace - highly motivated students are not
limited by semester timeframe

Increased formative feedback

Integration of social networking/learning – monitored
discussion boards, continual formative feedback

Improve impact beyond CS
 Anything, Anywhere, Anytime, Anyone
Moving Forward
Where do we need to go?
Khan Academy (“Circuits 3” Example)
Embedded Validation…
If R1 = 4, R2 = 2, Rn = 4:
Rtotal = ?
Rtotal =
Rtotal = ([R1║R3] ║ R2) = ([2]║2) = 1
Opportunities
 Improving Undergraduate STEM
Education Program (IUSE)
 Program Description: NSF 14-7513
IUSE Objectives
 Increase student retention in engineering
 Improve students' STEM learning outcomes
 Generate knowledge on how students learn
and on effective practice in undergraduate
engineering classrooms
 Broaden participation in engineering
 Prepare students to participate in
engineering for tomorrow
IUSE Potentials
Projects that build on fundamental
knowledge in undergraduate engineering
education and prior R&D
 Research on design, development, and
wide-spread implementation of effective
engineering learning/teaching knowledge
and practice
 Foundational research on student learning
 Knowledge generating exploratory efforts

Other Funding Opportunities
 Research on Engineering Education (REE),
PD 10-1340
http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503584
 EHR Core Research (ECR), NSF 13-555
http://www.nsf.gov/pubs/2013/nsf13555/nsf13555.htm
 Cyberlearning and Future Learning
Technologies (Cyberlearning), NSF 14-52
http://www.nsf.gov/pubs/2014/nsf14526/nsf14526.htm
Opportunity Abounds
We need more coordinated,
comprehensive research studies
and sharing of data…
Opportunity Abounds
Which is why we’re here!
Thanks
dmillard@nsf.gov