RELATE:
REsearch in Learning, Assessing, and
Tutoring Electronically
RELATE.mit.edu
Postdocs:
Phil Dukes
Sofia Morote
Rasil Warnakulasooriya
PI: Dave Pritchard
$: MIT, NSF, DEP
1
Also known as
by EET
and CyberTutor.MIT in publications
an expert system
based on educational expertise, not AI
The most advanced tutorial and assessment system in the world
Made by Effective Educational Technologies, a Pritchard company.
2
Outline
• Objective
• Pedagogy that Works
• Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
3
Perspective
Digital Education Future?!
Broadcast Radio
Passive
Class
Uniform Style
Next Edition
Teacher
Author
High Stakes Tests
Two-way Radio
Interactive
Student
Stylized (e.g. Audio)
Next Day
Coach
Authors/Researchers
Embedded Assessment
4
Why Homework??
Teachers’ Priorities:
1. Lectures
2. Exams
3. Notes and Demonstrations
4. Homework
Students spend most time and learn most from
1. Homework
5
Books, lectures, most WWW education
TWO WAY LEARNING
Students, Teachers, Authors, Researchers
Learn from each other
DATA, EXPERIMENT, ANALYSIS, CONCLUSIONS
6
Teacher
Student
SOCRATIC LEARNING
System
Authors, Researchers
7
Outline
 Objective
• Pedagogy that Works
• Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
8
Pedagogy
Design Philosophy of myCyberTutor
Emulate the interaction between a
human tutor and a student.
The tutor informs the teacher.
The process informs the author.
Results:
an effective interactive learning tool
you can author, deliver and improve content
an expert program embodying your expertise
9
Pedagogy
Student-Centered Instruction
Mastery Learning
The amount learned
should be constant
and time allowed
to vary
Constructivists
Allow students to
construct knowledge
in their own way
Student
Others
Socratic pedagogy
and learning styles
(to be implemented)
10
Pedagogy
Pedagogical Principles
•
•
•
•
•
Actively engage the student
Adapt problem to less skillful students with hints
Prompt feedback addresses wrong answers
Mastery Learning
>90% get solution
Declarative and procedural knowledge are both
important
hints and subproblems
• Solidify and extend the solution
followups
• Free response answers reduce guessing
11
12
13
14
15
16
Results from MIT
Gain on the MIT Final
Exam
December 2000 to May 2001
P-value
0.69
0.69
0.35
0.010
17
Results from MIT
Gain on Force Concept
Inventory
data C. Ogilvie 2000
FCI gain=0.41 for course
P-value
0.854
0.807
0.198
0.087
0.015
18
Outline
 Objective
 Pedagogy that Works
• Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
19
TWO WAY LEARNING
Two Way Learning - Feedback
myCyberTutor interactions:
• Typical student returns to server 10 times during
course of each problem (cf. Web Assign ~4
times per assignment)
• Students achieve the correct answer 90% of the
time (cf. ~60% first time right)
• Students comment on ~3% of all problems
– More if problem has flaws
20
Student
Feedback – Closed Loop Education
Comments
21
Feedback – Closed Loop Education
Wrong
Answers
22
Feedback – Closed Loop Education
Feedback to Author
Improves Problems
• < 90% correct: Need more hints
• Wrong Answers: Respond to common ones
• Comments: Revise wording, remove
confusion, revise program
• Time: Is this problem worthwhile?
23
Feedback Enables Revisions
that Improve Problems
Percent
Answering
Correctly
Percent
Requesting
Solutions
Average
Wrong
Answers
/part
Average
hints/part
Median
Minutes/
part
83.5 %
12.3 %
1.51
0.76
1.5
Spring
2002
91.6 %
6.0%
0.89
0.83
1.5
Fall
2003
93.4%
Spring
2001
Room for even more improvement!!
24
Outline
 Objective
 Pedagogy that Works
 Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
25
Pedagogy
RELATE:
REsearch in Learning, Assessing, and
Tutoring Electronically
RELATE.mit.edu
Postdocs:
Phil Dukes
Sofia Morote
Rasil Warnakulasooriya
26
Objective
Attractions for Researchers
Data with resolution
Capability for Split Class Assignments
Your Data Package
Log of your students’ interactions
Anonymous student # vs name & ID for your
class
Database of your class’ assignments
Plus - General Data Package
Performance Data - your class vs. standard
SML skeleton each problem (subparts, hints, etc.)
Format key
27
Inductive vs. Deductive
Instruction
Inductive: Students learn by doing a problem
from the hints,
from the subproblems
by figuring it out from feedback
Transfer learning to tutorial questions??
Deductive: Students learn from a tutorial
from the learning goal & text
from the hints
from the self-assessment questions
Transfer learning to related problem??
28
Tutorials
• Tutorial problems in Mastering Physics are
carefully planned and sequenced
instruction with SAQ’s
• They are used as instructional material to
impart principles in deductive learning.
Pedagogy
Problems
• Problems require a student to apply an
already familiar concept, formula, or
procedure
• Socratic help is available, including
explanation of the concept, a formula
that is needed, etc.
• Related Problems cover same topic as
adjacent tutorial
30
Deductive:
Related Problem Difficulty Reduced by
working Tutorial First
Harmonic Os.
Newton 3rd Law
Torque
Results from MIT
After working tutorial
p=0.01*
p=0.03*
p=0.06**
31
Inductive:
Tutorial Difficulty Reduced by working Related
Problem First?
Harmonic Os.
Newton 3rd Law
Torque
Results from MIT
32
Results from MIT
Conclusion:
Deductive Works
• Interactive tutorials significantly increase performance on
subsequent related problems (~25% less difficult)
• Students don’t learn inductively from a multi-part example
• We recommend using online tutorials in the old fashioned
way - preparation for subsequent deductive exercises
33
Twice as much learning per unit time spent on the tutorial
compared with time spent on the preparatory problem
Tutorial-first
0.020
Problem-first
0.018
Improvement per unit time
0.016
0.014
0.012
0.010
0.008
0.006
0.004
0.002
0.000
Torque
Newton III
SHM
34
Prior related problem reduces
the hints requested on the related
problem by ~12%
(based on 6 problems)
Unprepared group
Prepared group by solving a related problem
p < 0. 05
0.6
p < 0. 01
0.5
0.4
0.3
0.2
0.1
0.7
fraction of students who request hints
0.7
Prior tutorial reduces
the hints requested on the related
problem by ~19%
(based on 5 problems)
0.6
Unprepared group
Prepared group by solving a tutorial problem
p < 0. 005
0.5
0.4
p < 0. 1
0.3
0.2
0.1
0.0
Finding Torque
0.0
Colliding cars
Newton III
Cross-section for asteroid impact
35
Time to Completion
The real time environment allows us to study
how long it takes students to work problems,
whether good students do problems quicker or
slower, etc.
We have discovered that there are
Three groups of students in time:
36
When Students Finish: Three distinct groups
Quick solvers
< 2.5 minutes
Real-time solvers
2.5 min – 2.2 hours
Interrupted solvers
> 2.2 hours
Colliding cars
0.40
0.35
Rate of completion
0.30
0.25
0.20
0.15
Quick solvers
Interrupted solvers
0.10
0.05
Real-time solvers
0.00
-0.05
7s
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h
4
6
8
Ln(t)
10
17h 2d
12
5d
14
37
0.40
Finding torque
Flywheel kinematics
Parallel-axis theorem
0.4
0.35
Median time ~ 11min
0.30
Rate of completion
Rate of completion
Median time ~ 7min
0.3
0.2
0.1
Colliding cars
Shooting a block up an incline
0.25
0.20
0.15
0.10
0.05
0.00
0.0
7s
2
20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h
4
6
8
2d
10
5d
-0.05
12
7s
20s 55s 2.5m 7m 18m 50m 2.2h
2
14
4
6
2d
10
5d
12
14
Ln(t)
Ln(t)
Cross-section for asteroid impact
Post-collision orbit
The parallel-axis theorem
A person standing on a leaning ladder
Collision at an angle
0.40
6h 17h
8
0.30
Median time ~ 30min
Rate of completion
0.30
0.25
0.20
0.15
Rate of completion
0.25
Median time ~ 18-30min
0.35
0.20
0.15
0.10
0.10
0.05
0.05
0.00
-0.05
0.00
7s
2
20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h
4
6
8
Ln(t)
10
2d
12
7s
5d
14
2
20s 55s 2.5m 7m 18m 50m 2.2h
4
6
8
Ln(t)
6h 17h
10
2d
12
5d
38
14
Real-time solvers make errors and ask for hints
total
no hints, no wrong ans.
no hints, at least one wrong
at least one hint, one wrong
0.25
Rate of completion
0.20
0.15
0.10
0.05
0.00
7s
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h
4
6
8
Ln(t)
10
2d
12
5d
14
39
0.25
total
no hints, no wrong ans.
no hints, at least one wrong
at least one hint, one wrong
Collision at an angle:
Unprepared group
total
no hints, no wrong answers
no hints, at least one wrong
at least one hint, one wrong
Collision at an angle:
Prepared group
0.25
0.20
0.15
Rate of completion
Rate of completion
0.20
0.10
0.05
0.15
0.10
0.05
0.00
0.00
7s
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h
4
6
8
Ln(t)
10
2d
12
5d
7s
14
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h
4
6
8
10
2d
12
5d
14
Ln(t)
Note that:
1. The quick solvers do not make mistakes or ask for hints
2. The real-time solvers make mistakes and ask for hints
3. The interrupted solvers make mistakes and ask for hints
4. Fewer real-time solvers in the prepared group ask for hints
40
Fraction Finished curves with hints & feedback
Collision at an angle
Colliding cars
1.0
0.8
0.8
0.6
0.4
fraction
fraction
0.6
64%
0.4
44%
0.2
0.2
0.0
0.0
7s
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h
4
6
8
Ln(t)
10
17h 2d
12
5d
7s
14
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h
4
6
8
10
17h 2d
12
5d
14
Ln(t)
For 14 problems: fraction of real-time solvers = 65  4%
41
Time to completion curves without hints & feedback
End-of-chapter 10.38
0.8
0.8
0.6
0.6
35%
0.4
End-of-chapter 10.40
1.0
fraction
fraction
1.0
28%
0.4
0.2
0.2
0.0
0.0
7s
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h
4
6
8
10
17h 2d
7s
5d
12
14
4
6
8
10
17h 2d
12
5d
14
Ln(t)
Ln(t)
End-of-chapter 10.46
1.0
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h
0.8
For 3 typical homework problems:
fraction
0.6
fraction of real-time solvers = 29  3%
0.4
0.2
0.0
7s
2
20s 55s 2.5m 6.7m 18m 50m 2.2h 6h
4
6
8
Ln(t)
10
17h 2d
12
5d
14
42
Outline
 Objective
 Pedagogy that Works
 Feedback – Closed Loop Education
 Research from MIT
• Revolution in Assessment
• Closing Thoughts
43
Low-Error
Embedded
Future - Assessment
Assessment
Imagine that a rich ship-owner has hired Socrates to
tutor his children. At the end of the month he desires
to assess the amount they have learned. Would you
advise him to:
a)
Administer a standardized hour-long test to the
children?
b) Ask Socrates how much they have learned?
MyCyberTutor Assessment has ~100 times less
variance due to error than a good final exam!
This Assessment gives ~6 times as reliable an assessment per
unit of student time as a good final exam!
44
Future - Assessment
Embedded Assessment
Final Exam
myCyberTutor vs. Final Exam
SocraticTutor
0.5
Score (Odd Problems)
Score (Odd Problems)
0.5
R2 = 0.986
0.4
0.3
0.2
0.1
R2 = 0.4053
0.4
0.3
0.2
0.1
0
0
0
0.1
0.2
0.3
0.4
Score (Even Problems)
0.5
0
0.1
0.2
0.3
0.4
0.5
Score (Even Problems)
myCyberTutor
•…is 6 times more reliable per unit time
•…has ~100 times less error variance
45
Future - Assessment
Assessment:
Detailed Skill Profile
46
Predicting
Final Exam Score
Future - Assessment
1
0.9
R2 = 0.51
MIT Final Score
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SocraticTutor Prediction
Implication: MyCyberTutor can replace tests
47
Assessment
myCyberTutor
Assessment Implies:
1) More Accurate
2) Fine Grained Assessment on Subtopics
3) Immediate Remediation
-Select Next Problem
4) JITT Guide for Teacher
5) Learning vs. Avoiding Lost Points
6) Predict Test Scores
-Eliminate Tests
7) Incredible Tool for Education Research
8) Replace High Stakes Tests
48
Outline
 Objective
 Pedagogy that Works
 Feedback – Closed Loop Education
 Research from MIT
 Revolution in Assessment
• Closing Thoughts
49
What you can gain
»Write interactive problems
»Educational Research on them
»Educational Research in general
What we can accomplish together
»Partnerships in each - ideally all!!
50
Attractions for Problem Writers
You can write truly interactive material
Improve your problems from student data
Author in just your area of research expertise
Author tutorials, example problems, and
problems involving applets
New education research tools improve your
problems and exercises
51
Attractions for Curriculum
Developers
New research tool with resolution of student’s
capabilities and difficulties
Improve instructional material via Feedback
Experimentally Compare material and
pedagogy
Develop texts, tutorials, traditional exercises,
and more sophisticated problem sequences
52
Credits and Thanks
Effective Educational Technologies, Inc.*
Alex Pritchard - sole programmer for 4 years
Adam Morton - chief programmer
David Kokorowski - content development
Andrea Pritchard - president & treasurer & HR
Postdocs and Undergrads
Gabe Rockefeller - now at U. Arizona
Phil Dukes - now at UTBrownsville
Sofia Morote - now at Dennison College
David Kokorowski - now at Effective Education Tech
Rasil Warnakulasooriya - present
Support:
MIT - esp. Physics Dept. - support with TA’s
NSF
*DEP’s family has controlling financial interest
53
END
54
Results from MIT
Student Opinion
“How does the amount you learn per unit time
with CyberTutor compare with time (including
checking solutions) spent on written
homework?”
Much More
Same
Much Less
MIT Semesters
55
Results from MIT
Student Opinion
“Would you recommend myCyberTutor for
8.01 next year?”
Ratio of Yes to No
MIT Semesters
56
Feedback – Closed Loop Education
57
Help for Authors
Author training and manual
Pedagogy instructional material
Review first problems
Instruction on using feedback data to improve
problems, guidelines for wrong answers
58
Future - Assessment
85% Relilability is Unfair
True Skill
Observed
Score
1/4 of those students who failed had a passing true score
Another 1/4 should have failed, but passed
59
Objective
Attractions for
Teachers
Tutoring available 24/7
Better than written Homework
- not lower quality substitute
Shift grading effort into instruction
Teachers have instant access to detailed
student performance data, facilitating
Just-In-Time Teaching (JiTT)
Problem Library with Metadata on difficulty,
time, student rating, topics involved
60
Pedagogy
Written Homework
Assign
Do homework
Hand in
Grade
Learn from hw result?
• No help for student
when stuck
• Feedback to student
takes 1 week
• Labor to grade
• No feedback to
teacher
• Copying is easy
61
Pedagogy
myCyberTutor is a Web-Based
Homework Tutorial System
Assign
Learn while
doing CyberTutor
homework
Detailed knowledge
base
• Provides immediate
feedback and help to
the student
• Does the grading
• Offers immediate
feedback to the
teacher
• Supplies powerful
insight into the
student’s thinking
62
Pedagogy
Tutorials
• Tutorial problems in myCybertutor
are carefully planned and
sequenced instruction with SAQ’s
• They are used as instructional
material to impart principles in
deductive learning.
Outline
Objective
Pedagogy
• Demo
• Results from MIT
• Feedback – Closed Loop Education
• Future - Assessment
• Closing Thoughts
64
Future - Assessment
Assessment/Testing
Error
Any assessment has testing error
- did you study problem that was on test?
- careless mistakes?
- lucky guess?
How to determine reliability (reproducibility)?
- compare two equivalent tests
- split single test into two equivalent tests
If test is reliable (error-free), split grades will
correlate
65
Future - Assessment
Final Exam 2001-2002
Reliability 0.85
s
error
=0.41
s
observed
66