Evidence-Based Learning and
Motivation With MOOCs
Dick Clark
Center for Cognitive Technology
University of Southern California
clark@usc.edu http:\\cogtech.usc.edu
Learning with MOOCs: A Practitioners Workshop
Wednesday August 13th 2014 – Cambridge MA
Center for Cognitive Technology
Topics
1.
Counter-intuitive findings from research on learning from
technology - what are we doing that does not work or makes
learning fail?
1.
What does the best evidence indicate we should do to maximize
learning, motivation and transfer from MOOC courses?
i. Instructional design
ii. Screen design
iii. Motivation strategies
1.
The impact of evidence-based online course design and
suggested MOOC research questions.
Sources of evidence
Why evidence-based MOOC design?
•
•
•
•
•
•
•
•
45% 2 year and 36% 4 year students “no analytical learning gains”1
Those whose learning improved only gained about 4% a year.1
Students remember less than 30% of what they were taught3
Prestige colleges produce no more learning than all others4
STEM students writing and quantitative skills decline2.
Physics students think less like physicists than when they enter3.
Minority and underserved student learning outcomes are worse.
Faculty have discipline expertise BUT most lack pedagogical
expertise YET are overconfident about their teaching skill.3
• MOOC designers either do not know about or are not allowed to use
evidence-based online pedagogy to produce online presentations.
1Aram
& Roska (2011) 2 Bok (2008) 3 Armstrong (2012) 4 Pascarella (2005) 5Bloom (1984)
Counter-Intuitive Research Evidence
• The choice of media for instruction does NOT influence learning.
– Lectures, power point slides and video are media. (Clark 2012).
• Learning is influenced by pedagogy (instructional method).
• Pedagogy provides external support for internal cognitive
processes required to learn.
• Media influence access to, and the cost of, learning.
• Inadequate pedagogy and screen design can cause students to
remember significantly less after they finish a lesson than before
they started (R. E. Clark & Estes, 2008).
• 30% of students like and choose instruction from which they
learn the least when offered a choice between more or less
guidance (Clark & Estes, 2008) .
Counter-Intuitive Research Evidence
• Allowing students to navigate through a course in any way they wish
harms both learning and transfer (Elen & Clark, 2006) .
• For novice learners, most immersive simulations and serious games
are significantly less effective and more expensive than other
ways to teach because of “discovery learning” (Clark, 2007).
• Faculty who are top experts are only able to describe 30% of the
analytical and decision strategies they use to perform complex
tasks (Clark, 2013).
• 50% of students are wrong when asked what and how much they
learned from instruction (Clark & Estes, 2008).
• So what is the problem? Why have we not been more successful?
Cognitive Architecture & Neuroscience
Our cognitive architecture makes learning difficult (perhaps to
protect us?)
• We can only think about 3 +/-1 things at a time
• Stress and distractions take up thinking space
• When we overload our thinking space it “crashes” and we
loose attention and “daydream” (cognitive overload).
• Crashes are pleasurable but learning stops (Clark et al, 2010).
• Helping students avoid cognitive overload requires a careful
focus on what and how we are presenting information.
Why Learning and Motivation are Difficult
We have evolved with two integrated knowledge systems to circumvent
limits on our thinking space:
1. Declarative (what and why) – can only be memorized (30% of
total). Remains conscious unless “forgotten”.
2. Procedural (when and how) – application knowledge – automates
and operates outside of our consciousness (70% of total).
Our procedural (application) system is more effective if courses and lessons
are sequenced as students are expected to apply the knowledge.
•
Or if no sequence, simpler to learn before more difficult topics.
Lessons (modules) should be constructed around a procedure (when and
how to do something).
What MOOC design framework will help us solve these problems?
PSLC Knowledge-Learning-Instruction Framework
“Knowledge is acquired or
modified during unobservable
learning events inside students’
minds. These learning events are
influenced by instructional
events, in which the student’s
learning environment is modified,
and assessment events, in which
student knowledge is inferred
from performance either at the
time of the event or later.”
Koedinger, K.R., Corbett, A.T., & Perfetti, C.
(2012). The Knowledge-learning-instruction
framework: Toward bridging the sciencepractice chasm to enhance robust student
learning. Cognitive Science 36(5), 757-798
External support for the internal
cognitive processes required for
motivation and learning.
From Learning to Instruction
J. R. Anderson’s (Lee & Anderson, 2012) Neuroscience-based ACT-R suggests
that lesson level Instruction requires:
1.
2.
3.
4.
5.
6.
7.
Objective and Reasons (intentional module, goal buffer)
Overview (Declarative module)
Connection to Prior Knowledge (Declarative module)
New Knowledge (Declarative, visual/manual modules)
Demonstrated Procedure (Production system, declarative module)
Practice (Integration of new and old to achieve goal)
Feedback (Intentional module to correct and perfect)
These features mirror David Merrill’s (2006) principles from evidence-based
instructional design systems
Guided Learning - Lesson or Module Level
Evidence-based lessons or modules consist of 8 sections:
1.
2.
3.
4.
5.
6.
7.
Objective (What will be learned in this module)
Reasons (Benefit and Risk)
Overview (Outline and instructional method)
Connection to Prior Knowledge (Analogy, metaphor)
Conceptual Knowledge (concepts related to “how to”)
When and How (Demonstrate decisions and actions)
Practice (Solve engaging problems – part to whole explain)
8. Feedback (what worked, what needs correction)
Evidence for Guided Learning Effectiveness?
•
•
•
•
•
•
•
Learning increased +/- 35%
Learning occurs +/- 20% less time.
Fewer and less important decision errors.
More accurate student explanations.
Better assessment of developing expertise.
Coaching and feedback targets obvious.
Persistence and mental effort increased.
(Clark, 2009)
Multimedia (screen design) Theory - Mayer
10 Principles (Students learn better when …)
%
# of Tests
1. Coherence: Eliminate extraneous visuals and sound 31
14 of 14
1. Signaling: Highlight essential information
17
5 of 6
1. Redundancy: Do not narrate on-screen text
23
5 of 5
1. Visual Contiguity: Text next to part described
38
5 of 5
1. Time Contiguity: Simultaneous graphics & narration
42
5 of 5
Multimedia (screen design) Theory - Mayer
10 Principles (Students learn better when …)
%
# of Tests
6. Pacing: Learner pacing better than system pacing 31
3 of 3
7. Pre-train: Teach conceptual information first
27
5 of 5
8. Modality: Narrate graphics, not text & graphics
32
7 of 17
9. Personalization: Learners imagine prior experience 36
11 of 11
10. Voice: Human voice better than machine voice
3 of 3
25
“What makes me (us)
Effective or Successful?”
Past
experience of
effectiveness
Value
Starting
Self Efficacy
Persisting
Attributions
Mental
Effort
Emotions
MOOC Student Motivation Strategies
Motivation defined as: Starting, Persisting and Mental Effort
Four MOOC motivational strategies:
1.
Increase the personal value of a learning task.
• If students are not persisting.
2. Increase (or Decrease) student task-specific confidence.
• Under or over confidence decrease persisting and effort.
3. Provide more positive emotional climate toward learning task.
• Strong negative emotions decrease motivation.
4.
Reframe students beliefs about the controllability of problems
• When students believe that problems are uncontrollable they
will quit or manufacture an excuse not to start or persist.
Value, Efficacy, Attribution and Emotion vary between cultures.
Guidance based on motivation
Survey to monitor perceived value, confidence and mood
(positivity)
Survey data combined with
starting, completion, time and
mastery data to generate
personalized guidance.
© 2013 Kaplan, Inc.
For example, if a student’s profile
shows low performance, low
confidence, and low time spent,
this triggers guidance messages
aimed at increasing confidence.
© 2013 Kaplan, Inc.
Checklist for MOOC design reviews
1.
2.
3.
4.
5.
6.
7.
Learning Objectives –
a) Alignment ?
b) Measurability?
c) Clarity? and
d) Appropriateness?
Sequencing – courses, lessons?
Knowledge to perform real world
tasks?
Conceptual definitions, examples,
practice and feedback?
Application demonstrations, practice
and corrective feedback?
Students asked to reflect and
integrate?
Clear directions and navigation?
Category
Score
1. Program level (3 items)
2. Course Level (4 items)
3. Lesson level ( 6 items)
4. Media and Navigation Level (7 items)
TOTAL Instructional Design SCORE (max=13)
8. Focus only on Learning Objectives
9. Students control pacing?
10. Required sequence?
11. Text and graphics integrated spatially?
12. Narration rather than lots of print?
13. Conversational style?
14. LMS supports self testing, practice and
immediate corrective feedback?
Research questions for MOOC platforms
Use data mining to suggest when to guide and recommend five
alternative instructional strategies to target individual learners:
① Choosing content and problems based on prior knowledge.
② Providing fully OR partly guided demonstration and practice
③ Immediate OR delayed corrective feedback during practice.
④ Providing OR asking explanations for guidance and feedback.
⑤ Hints when students get stuck OR letting them “figure it out”.
http://www.learnlab.org
http://pact.cs.cmu.edu/pubs/Koedinger,%20Corbett,%20Perfetti%202012-KLI.pdf
Koedinger, K. R. & Aleven, V. (2007) Exploring the assistance dilemma in
experiments with cognitive tutors. Educational Psychology Review. 19. 239-264.
Conclusion
• Our focus should be on student learning NOT on media or technology.
• MOOC design requires three specialties:
1. Faculty contribute content expertise.
2. Instructional designers – Learning, motivation & media expertise.
3. Assessment specialists – measurement expertise.
•
Use guided pedagogy and avoid cognitive overload.
• Diagnose and treat motivation problems online.
• LMS systems must change to accommodate needed pedagogy.
• Research on learning from MOOCs to improve instruction using data.
mining must be based on a thorough review of solid past research.
QUESTIONS? COMMENTS?
Comments?
Questions?
References
Learning Styles:
Plashler, H. McDaniel, M., Rohrer, D, & Bjork, R. (2008) Learning Styles: Concepts and Evidence . Psychological Science in
the Public Interest. 9(3). 105-103.
Student Preferences:
Clark, R. E. (1989) When Teaching Kills Learning: Research on Mathemathantics. In H. Mandl, E. De Corte, N. Bennett, &
H. F. Friedrich (Eds.) Learning and instruction. European Research in an International Context. Volume II. Oxford:
Pergamon.
Clark, R. E. (1982) Antagonism Between Achievement and Enjoyment in ATI Studies Educational Psychologist, 17(2). 92101.
Rapid Pre-testing – Staged Innovation Design:
Kalyuga, S. (2006) Rapid cognitive assessment of learner’s knowledge structures . Learning and Instruction. 16. 1-11.
Wagner, N (1984) Instructional product evaluation using the staged innovation design. Journal of Instructional
Development. 7(2), 24-27.
Guidance, Media, Motivation, Assessment and Personalizing all discussed in:
Bloom, B. S. (1984). The 2-sigma problem: The search for methods of group instruction as effective as one-to-one
tutoring. Educational Researcher, 13, 4–16.
Clark, R. E., Yates, K., Early, S. & Moulton, K. (2010). An Analysis of the Failure of Electronic Media and Discovery-based
learning: Evidence for the performance benefits of Guided Training Methods. In Silber, K. H. & Foshay, R. (Eds.).
Handbook of Training and Improving Workplace Performance, Volume I: Instructional Design and Training Delivery. New
York: John Wiley and Sons. 263-297.
Clark, R. E., Howard, K. and Early, (2006). Motivational challenges experienced in highly complex learning environments.
In Elen, J. and Clark, R. E. (Eds.). Handling complexity in learning environments: Research and Theory. Oxford, G.B.:
Elsevier Science Ltd, 27-43.