Preparation for NSF Site Visit, May 2007
Simulated Students: Machine Learning by Demonstration for Intelligent Authoring
Report prepared for PSLC Advisory Board Visit, December 2006
PSLC Research Cluster(s): Enabling Technology
Noboru Matsuda
Postdoctoral Fellow, Human-Computer Interaction, CMU
Office: Newell-Simon Hall 2602G, Voice: 412-268-2357, Email: mazda@cs.cmu.edu
William W. Cohen
Associate Research Professor, Center for Automated Learning and Discovery, CMU
Office: Wean Hall 5317, Voice: 412-268-7664, Email: wcohen@cs.cmu.edu
Kenneth R. Koedinger
Professor, Human-Computer Interaction, CMU, PSLC Co-Director
Office: Newell-Simon Hall 3531, Voice: 412-268-7667, Email: koedinger@cmu.edu
Project Plan Abstract
Can machine learning agents help authoring Cognitive Tutors? Can they also help us understand how human students learn? To answer those questions, we have developed SimStudent,
a machine-learning agent that learns cognitive skills from model solutions demonstrated by human problem-solvers. We have then integrated the SimStudent into existing software suits – the
Cognitive Tutor Authoring Tools (CTAT).
CTAT provides an integrated development environment to facilitate building Cognitive
Tutors in two major ways: (1) it provides a tool to build a graphic user interface (GUI) for the
Cognitive Tutor by a drag-&-drop manner, and (2) it helps to build a problem specific cognitive
model that allows the Cognitive Tutor to perform tutoring on a particular problem upon which
the cognitive model is made. The Cognitive Tutor with this type of problem dependence is called
an Example-Tracing Tutor, as opposed to a fully functional Mode-Tracing Tutor. To build a
Model-Tracing Tutor, the author must build a generalized cognitive model represented as a set of
production rules. However, building a generalized cognitive model requires a significant amount
of time and special skills such as cognitive task analysis and AI programming.
We have extended CTAT by adding SimStudent to the authoring environment to help the
author build production rules without programming. The basic idea is that instead of writing production rules by hand, the author simply demonstrates how to perform a subject task with some
example problems. The SimStudent generalizes those demonstrations and generates a set of production rules that not only replicate the problem-solving steps demonstrated but also solve similar problems. If these generalizations are correct – i.e., are correct implementations of the actual
task that is being taught – then they can be added to the cognitive model as correct rules. If the
generalizations are incorrect but “plausible” (or pedagogically beneficial, if you will), then they
can be added to the cognitive model as mal-rules. The author can also perform “incorrect” behavior on purpose so that SimStudent learns mal-rules that represent human students’ common
misconceptions. Both correct and mal-rules would be embedded into a Cognitive Tutor, allowing
the tutor to perform its usual functions, namely, model tracing and scaffolding.
The intellectual merit of the project is twofold: (1) to facilitate cognitive modeling, and
(2) to advance theories of human learning and instruction.
Preparation for NSF Site Visit, May 2007
Achievements
I. Major Findings
Both of the studies listed here used the student-tutor interaction log collected from the classroom studies conducted in LearnLab. DataShop was used to export those data.


Error of commission analysis: In the past, we evaluated the SimStudent’s learning performance (i.e., the “quality” of production rules learned) by counting number of steps that are
correctly model-traced by the production rules learned. This measurement reveals the
number of production rules that are incorrectly generalized hence do not model trace correctly performed problem-solving steps (i.e., the error of omission). However, this evaluation method does not uncover the number of production rules that correctly generates alternative solutions, and more importantly it does not reveal the number of production rules
that generates incorrect steps (i.e., the error of commission). We have developed a communication stream – called the Tutoring Service – between SimStudent and a tutor to assess the correctness of a production-rule application. For each of the problem solving steps
in the test problem, SimStudent sends the possible next steps suggested by production
rules to the Tutoring Service. The Tutoring Service then forward the steps performed by
SimStudent to an appropriate assessor (usually a Cognitive Tutor). An evaluation study in
Algebra I showed that after trained with 15 problems, there were 1 or 2 overly general
rules applied (hence generated a incorrect step) per step in average in the test problems.
Modeling and predicting real-students performance: When human students use Cognitive
Tutors, their performance are logged. Such log data can be seen as the students’ demonstration on how to solve problems as well as how to make errors. We have then fed the individual student-tutor interaction log to SimStudent so that SimStudent can model each individual student’s performance. Once such a model is created, we can use the model to
predict the student’s performance on the novel problems. An evaluation study showed that
when trained on 15 problems, SimStudent accurately predicted the human students’ correct
behavior on the novel problems more than 80% of the time. However, the current implementation of SimStudent does not accurately predict when the human students make errors.
II. Other Studies and System Development

Turning SimStudent into Simulated “Student”: So far, we have asked the author to demonstrate how to solve problems, and for each of the steps demonstrated SimStudent acknowledges the author’s performance with flagged feedback showing if the step is model-traced
with an existing production rule or not. This protocol does not uncover overly general production rules that generate incorrect steps when there is a correct production rule that
model traces a step, which will impede learning (taking longer learning time to eventually
uncover incorrect production rules). One way to improve the SimStudent’s learning performance is to have SimStudent perform the task and let the author provide feedback. With
this learning protocol, the author gives a problem to SimStudent, and SimStudent performs
steps. If there is no production rule that could perform a step, then SimStudent asks a hint,
and the author performs a step. If there is a production rule(s) that can apply, then SimStudent shows the resulted step(s) to the author, and the author gives flagged feedback (cor-
Preparation for NSF Site Visit, May 2007

rect or incorrect). We have been developing this new learning schema – called interactive
learning.
Applying SimStudent to chemistry Stoichiometry domain for bootstrapping: One way to facilitate authoring is to ask the author to solve (both correctly and incorrectly) as many
problems as possible (without interactive with SimStudent) and record those solutions. We
then apply SimStudent to those pre-stored solutions and ask SimStudnet to generalize all
such solutions at once. Because there is no interaction between SimStudent and the author,
this type of learning is quite challenging in some aspects – e.g., there is no focus of attention specified. We have applied SimStudent to chemistry Stoichiometry domain.
Publications






Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007; in press).
Evaluating a simulated student using real students data for training and testing. In Proceedings of the international conference on User Modeling.
Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007; in press).
Predicting students performance with SimStudent that learns cognitive skills from observation. In Proceedings of the international conference on Artificial Intelligence in Education.
Matsuda, N., Cohen, W. W., Sewall, J., & Koedinger, K. R. (2006). Applying machine
learning to cognitive modeling for cognitive tutors (Machine Learning Department Technical Report No. CMU-ML-06-105). Pittsburgh, PA: School of Computer Science, Carnegie Mellon University.
Matsuda, N., Cohen, W. W., Sewall, J., & Koedinger, K. R. (2006). What characterizes a
better demonstration for cognitive modeling by demonstration? (Machine Learning Department Technical Report No. CMU-ML-06-106). Pittsburgh, PA: School of Computer
Science, Carnegie Mellon University.
Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2005). Building Cognitive Tutors with
Programming by Demonstration. In S. Kramer & B. Pfahringer (Eds.), Technical report:
TUM-I0510 (Proceedings of the International Conference on Inductive Logic Programming) (pp. 41-46): Institut fur Informatik, Technische Universitat Munchen.
Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2005). Applying Programming by
Demonstration in an Intelligent Authoring Tool for Cognitive Tutors. In AAAI Workshop
on Human Comprehensible Machine Learning (Technical Report WS-05-04) (pp. 1-8).
Menlo Park, CA: AAAI association.
Plans for June 2007 – December 2007



Complete the implementation of the interactive learning, and conduct evaluation study to
see how it works better than the traditional learning (i.e., learning by demonstration).
Complete the Tutoring Service and run evaluation studies for SimStudent learning with automated evaluators (i.e., the Cognitive Tutors).
Conduct evaluation studies for SimStudent to see how it facilitates authoring Cognitive
Tutors.