Brett van de Sande
Arizona State University
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
PERC 2010 workshop
Experimental Methods for Studying
Student Metacognition and Affect
during student problem-solving (homework)
1
Our group/team
Kurt.Vanlehn@asu.edu
Also: Kasia Muldner, Winslo Burleson,
Asael Sorenson, Raj Ranganathan
bvds@asu.edu
University of Pittsburgh
Outline
• Theoretical background
– Definitions of Metacognition & Affect
– Why study them?
• Survey of methods, with examples; focus on
Intelligent tutor systems
• Hands on: try solving a problem with Andes
• Possible future studies
3
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Define Metacognition
• Thinking about cognition (memory, reasoning,
learning, etc.) itself.
• It is “thinking about thinking.”
• For learning, this includes:
• Planning: the design of the
learning process, choice of
strategy
• Monitoring: comparing actual
progress to the desired one
• Self assessment: the ability to
correctly evaluate one's own
knowledge level
• Debugging: identifying sources of
failure and overcoming those
4
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Definitions: Affect
• A feeling or emotion, as distinguished from
cognition, thought, or action (conation)
• In learning: boredom, confusion, delight,
flow, frustration, and surprise
• Time scales:
• Long term (weeks/months): “I love/hate physics”
• Short term (seconds): “What am I supposed to do
for step 2?”
• “Affective trajectory” (Michael Baker)
5
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Why study metacognition?
Why study affect?
Why in the context of homework?
• Groups of 3-5, discuss
• Collect results:
• Hw is less structured than class, teacher not direct influence; habits
more important than topics; affect important for time decisions;
motivation <-> affect.
• Experts vs students: experts use m-cog skills, should be part of
assessment. Sense of how student is framing activity: “get an
answer” vs. “understand” mc skill students learn is something to
learn & affects other learning.
• Differentiating between “challenge” & “frustration” determine object
of frustration.
• Identify frustration & intervene (encourage).
6
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Survey of Methods
• Metacognition and Affect dissimilar
• Experimental probes
•
•
•
•
•
Survey
Interview
Physiological (posture sensors)
Behavioral (decision-making)
Neurological (MRI, etc) new, $$$
7
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Surveys
• Example: CLASS survey (Colorado)
4. It is useful for me to do lots and lots of
problems when learning physics.
9. I find that reading the text in detail is a good
way for me to learn physics.
25. I enjoy solving physics problems.
34. I can usually figure out a way to solve
physics problems.
40. If I get stuck on a physics problem, there is
no chance I'll figure it out on my own.
8
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Study: “Are Students’ Responses and Behaviors
Consistent?” (U. Wutchana, N. Emarat and E. Etkina, 2009)
From Summary:
“…the observations indicate that students behave
differently from what they report on the survey.
Thus it might be difficult to access students’
beliefs about physics and learning physics by
using surveys only.”
10
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Surveys
• Good:
• Easy to do
• Probes mental state, “what’s going on inside”
• Bad:
• Longer time scales, limited number of data points
• Hard to scale (what does a “3” mean for an emotion?)
• Susceptible to bias (give answer that is perceived to be
desired by the instructor).
• Meta-metacognition, Meta-Affect: student reflection on …
• “After the fact:” the metacognitive activity or event that
caused the affective state has come and gone.
11
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Interview techniques
• Schoenfeld (1992) advocates study of
metacognition.
• At the middle school level, Lester, Garofalo & Kroll
(1989, June) investigation “designed to study the
role of metacognition (i.e. the knowledge and
control of cognition) in seventh graders’
mathematical problem solving.” Assessment data
and tools employed before, during, and after the
instruction included written tests, clinical
interviews, observations of individual and pair
problem-solving sessions, and videotapes of the
classroom instruction.
12
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Interview techniques
• Good:
• In depth probe of metacognitive processes
• Can adapt to student response
• Bad:
• Time consuming (small sample size)
• Analysis hard (small sample size)
• Affect susceptible to student bias (student
reports what they believe experimenter wants)
• Presupposes a theoretical framework (bias in
interview questions)
13
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Physiological measures
• Probes:
•
•
•
•
Facial expression
Skin resistance
Heart rate
Butt sensor
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
• Example: Art Grasser & AutoTutor
AutoTutor is natural language tutor for qualitative
Newtonian physics. experiments where students are
monitored for facial expressions and butt sensor as
they solve problems with tutor.
Classify affective states: boredom, confusion, delight,
flow, frustration, & surprise
14
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Physiological measures
• Good:
• Hard to fake responses (authentic)
• Real time monitoring
• Bad:
• Intrusive, restricted to lab setting
• Expensive equipment
• Video methods need manual analysis
• Still need to connect to actual Affective state
via survey or interview methods.
15
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Behavioral Measures
• “Decision making in an authentic context”
• Does the student drop the class/change major?
• Do they ask for help or try again without help?
• Read a hint or immediately ask for the next hint?
• Good:
•
•
•
•
Easy to measure (in context of tutor system)
In many cases, measure often, on short time scale
Authenticity possible
Unobtrusive
• Bad:
• Need to connection to student mental state
16
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Example behavioral measure: Gaming studies
• Gaming (Baker):
“Attempting to succeed in an interactive learning
environment by exploiting properties of the system
rather than by learning the material.”
• Examples:
• Guessing & looking at tutor response
• Drilling through hint sequences
• Follows from affective states of boredom &
confusion (Rodrigo et al, 2007)
• Metacognitive strategy
• Student uses gaming to quickly get through previously
mastered, but tedious material.
• Compensate for errors/weaknesses in the tutoring system.
17
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Example: “An Analysis of Gaming Behaviors
in an Intelligent Tutoring System” (K.
Muldner, et al, 2010)
Session as dialog between tutor and student
Use timing of student action after the tutor
responds to student’s previous step:
Tutor Response
Student Action
hint request
student entry
fast
slow fast
slow
bottom-out hint skip hint
copy hint
high-level hint
skip hint
incorrect
correct
guess
no planning
18
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Try the new version of Andes
• Use example problem to explain
instructional strategy
• Features instructors care about
• Try it out!
19
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Andes from the student’s
perspective:
an example from statics
Students at the USNA took an average of 15
minutes (median 13 minutes) to solve this.
20
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Upon request, Andes gives
hints for what to do next
Define variables
Draw free body diagram
(3 vectors and body)
Define coordinates
(3 choices for this problem)
Immediate Red/green
feedback for student actions
Principles-based help for
Incorrect entry
23
Students must provide units
score
Students work with
algebraic expressions
Andes calculates
numerical answer
Instructor’s perspective
• Homework component of a standard two-semester
introductory course (with lecture and textbook)
• Works in any web browser
• Over 500 problems
• 100% coverage of USNA fall course (no thermodynamics)
• 90% of spring course (no calculus or modern physics)
• Not tied to a specific textbook (have used Knight, Giancoli,
& Serway).
• Student testing this summer, classroom beta testing
this fall (October).
• Offered through: Open Learning Initiative (OLI),
WebAssign, & LON-CAPA (thanks to Andy Pawl).
26
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Try it out
• http://andestutor.org/sets
• Start with problem vec1ay and follow the instructions
in the tutor pane.
• If you are stuck or unsure why an entry is incorrect,
you can ask the Andes Tutor for Help. Just click on
the help button at the bottom of the Tutor pane.
• If you have a comment or complaint, type it in the text
box at the bottom of the Tutor pane. These are
logged and will be read by us later.
• When you have completed a problem, click on the
submit button to exit.
27
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Possible future studies
• Probes:
– Log student actions (solution steps, errors, help
requests), as well as tutor interpretation, with time.
– Could have fine-grained logging: keystrokes, mouse
clicks, menu usage.
– Can dialog with students in tutor pane
before/during/after the problem solving process.
• Experimental contexts (each has pros and cons):
– Student in dorm room/home.
– In class (can have working with pairs, microphones)
– Laboratory (can also have video, physiological
sensors).
28
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Future experiments
• Theoretical question
–metacognition and/or affect?
–Time scale of “affective trajectory”
–Verbal and behavioral aspect?
• Experimental design
–Probe & experimental context
–Pre-test/post-test
–Control & experimental conditions
29
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Brett van de Sande
Thank you.
bvds@asu.edu
Kurt.Vanlehn@asu.edu