In-vivo Experimentation
Steve Ritter
Founder and Chief Scientist
Carnegie Learning
©2012 Carnegie Learning, Inc.
An attempt to find meaning in
three acts
• Design: Geometry Contiguity (Vincent
Aleven, Kirsten Butcher)
• Modeling: Adjusting learning curve
parameters (Cen, Koedinger, Junker)
• Personalization: Word problem content
(Candace Walkington)
©2012 Carnegie Learning, Inc.
DESIGN
©2012 Carnegie Learning, Inc.
Geometry angles
©2012 Carnegie Learning, Inc.
Contiguity
Early Version
Research Version
Commercial Version
(Carnegie Mellon)
(Carnegie Learning)
Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and
deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY:
Lawrence Earlbaum.
Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an
Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual
Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.
©2012 Carnegie Learning, Inc.
Early Tutor
©2012 Carnegie Learning, Inc.
Revised (commercial) tutor
©2012 Carnegie Learning, Inc.
Geometry Contiguity
• Design and field experimentation
– Butcher and Aleven (2008)
• Diagram interaction led to better transfer and
retention
• Analysis of impact
– Hausmann and Vuong (2012)
• Unit-level effects mixed
• Advantage for harder skills
©2012 Carnegie Learning, Inc.
Geometry Angles
©2012 Carnegie Learning, Inc.
Lessons
• Change is constant
• Transition from research to production
always requires adaptation
©2012 Carnegie Learning, Inc.
MODELING
©2012 Carnegie Learning, Inc.
Skillometer
©2012 Carnegie Learning, Inc.
Expression Writing
©2012 Carnegie Learning, Inc.
What gets learned?
©2012 Carnegie Learning, Inc.
Bayesian Knowledge Tracing
Cognitive tutor traces these skills differently
©2012 Carnegie Learning, Inc.
Learning Curve Parameter Fitting
•
Field study looking at learning area of geometric figures
–
One group used adjusted learning parameters based on previous year’s data
•
Optimized group took 12% less time to reach same performance
•
•
Significant learning gain in both groups
No difference in learning gain between groups (p = 0.772 )
120
100
80
Optimized
60
Control
40
20
le
irc
C
Po
ly
go
n
Tr
ap
ez
oi
d
Tr
ia
ng
le
Sq
ua
re
Pa
ra
l le
lo
gr
am
0
16©2012 Carnegie Learning, Inc.
Lessons
• Learning efficiency is a great outcome
• Small, systemic changes can have big
impact
• Optimizing skills requires appropriate skill
model
– Koedinger, McLaughlin and Stamper (2012) LFA
©2012 Carnegie Learning, Inc.
PERSONALIZATION
©2012 Carnegie Learning, Inc.
Word problem customization
©2012 Carnegie Learning, Inc.
Personalization field study
• Students who got problems related to their
interests made fewer errors
• Also affected subsequent unit
• Interaction with readability
©2012 Carnegie Learning, Inc.
Lessons
• Content matters
– Challenge for knowledge component
modeling
• Are we personalizing preferences, reading
level or both?
©2012 Carnegie Learning, Inc.
Summary
• It’s not about whether A is better than B
– It’s about why A is better than B
©2012 Carnegie Learning, Inc.