QA on Anderson et al. 1995
Intro to CTAT
CPI 494 & 598 Jan 27, 2009
Kurt VanLehn
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
NEXT
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
Task selection
Assessment
Authoring and the software development
Evaluations
Dissemination
Describe the user interface of the Lisp Tutor
• What do students type? What are the <..> things? Steps?
Describe the user interface of the Geometry tutor
• What does student do? Steps?
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
NEXT
Task selection
Assessment
Authoring and the software development
Evaluations
Dissemination
For the Lisp tutor
• What counts as a correct step?
• How did the tutor determine if the student’s
step is correct?
• Would the tutor accept both
– (plus X 5) and (plus 5 X)
– (times X 0.5) and (divide X 2)
– How does it know?
• Can students choose their own variable
names?
For the geometry tutor
• What steps count as correct?
– True but useless propositions?
– Top down vs. bottom up?
• How does the tutor determine if the student’s
step is correct?
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
NEXT
Task selection
Assessment
Authoring and the software development
Evaluations
Dissemination
For the Lisp tutor
• When the student asks for a hint, how does
the tutor determine what step to hint?
• If the tutor allowed students to keep their
incorrect steps on the screen, what step
would the tutor suggest when asked?
• What if the tutor did not give immediate
feedback and were asked for a hint?
For the Geometry tutor
• How does it decide what step to suggest?
• What does ACT theory say about this policy?
• Would it be better to recognize the student’s
plan and help them along by suggesting its
next step?
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
NEXT
Task selection
Assessment
Authoring and the software development
Evaluations
Dissemination
For the Lisp tutor
• Does the “normal” version of the tutor give
immediate feedback, delayed…?
• What other kinds of feedback policies did they
try? What were the results?
• How was hinting usually done?
• Did the tutor ever ask a question when asked
for a hint?
For the Geometry tutor
• Does the “normal” version of the tutor give
immediate feedback, delayed…?
• What other kinds of feedback policies did they
try? What were the results?
• How was hinting usually done?
• Did the tutor ever ask a question when asked
for a hint?
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
NEXT
Assessment
Task selection
Authoring and the software development
Evaluations
Dissemination
Lisp tutor: Assessment
• What is assessed? Motivation? Knowledge?
What kind of …?
• What granularity?
• Give examples of what is assessed?
• What’s the name of the technique?
• When students enter correct step, incorrect
step or hint-request, what happens to the
assessment?
• IF the goal is to get the second element of the
list
• THEN code car and set as a subgoal to pass to
car an argument that is the tail of the list
Geometry tutor: Assessment
• What is assessed?
• Give examples
• How is the assessed stuff different from Lisp’s?
• IF the goal is to prove two triangles are
congruent
• THEN set as a subgoal to prove that
corresponding parts are congruent.
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
NEXT
Assessment
Task selection
Authoring and the software development
Evaluations
Dissemination
Lisp tutor: Task selection
•
•
•
•
•
•
What are the tasks to be selected from?
Are they authored or generated dynamically?
Who does the selection? Tutor or student?
What is the procedure for selection?
What information does the procedure need?
How does this procedure fit with ACT theory?
Geometry tutor: task selection
•
•
•
•
•
•
What are the tasks to be selected from?
Are they authored or generated dynamically?
Who does the selection? Tutor or student?
What is the procedure for selection?
What information does the procedure need?
How does this procedure fit with ACT theory
Initial Framework
• Step loop
–
–
–
–
•
•
•
•
•
User interface
Interpreting student actions
Suggesting good actions
Feedback and hints
Assessment
Task selection
Authoring and the software development
Evaluations
NEXT
Dissemination
Lisp stutor: Evaluations
• List the evaluations done
• For each, is it: Formative? Summative?
Hypothesis testing?
• What are the results of each?
• a fixed set of programming exercises either
with the tutor or in a standard LISP
environment. Students using the tutor
completed the exercises 30% faster and
performed 43% better on a posttest.
• using the tutor completed the exercises 64%
faster and scored 30% higher on posttests
than did students using a standard LISP
environment.
What did this graph show?
What did this graph show?
Geometry tutor: Evaluations
• List the evaluations done
• For each, is it: Formative? Summative?
Hypothesis testing?
• What are the results of each?
• In Peabody HS, one standard deviation
• An ethnography
• Again, but only got positive result with
teacher.