Cognitive and Metacognitive Educational Systems: Papers from the AAAI Fall Symposium (FS-10-01)
The Role of Prompting and Feedback in Facilitating Students’ Learning
about Science with MetaTutor
Roger Azevedo1, Amy Johnson2, Candice Burkett2, Amber Chauncey2, Ashley Fike2
Mihai Lintean3, Zhiqiang Cai2, & Vasile Rus3
1
McGill University, Department of Educational and Counselling Psychology
University of Memphis, Department of Psychology, Institute for Intelligent Systems
3
University of Memphis, Department of Computer Science, Institute for Intelligent Systems
2
one’s own understanding of the material and the
appropriateness of the current information, and making
adaptations to one’s goals, strategies, and navigational
patterns, based on the results of such monitoring processes
and their resulting judgments (Azevedo, 2005; Azevedo &
Witherspoon, 2009; Opfermann, Azevedo, & Leutner, in
press; Pintrich, 2000; Winne, 2001; Winne & Hadwin,
2008; Zimmerman, 2001; Zimmerman & Schunk, in
press). Although learners should attempt to follow these
guidelines when attempting difficult topics, exploration of
typical learning has demonstrated that few learners, in fact,
engage in effective self-regulated learning. We assume
that, while motivation and affect play a role in determining
learners’ willingness to self-regulate, a lack of selfregulatory skills is the main obstacle to adequate regulation
and therefore deficient learning gains and conceptual
understanding (Azevedo & Jacobson, 2008; Shapiro, 2008;
Schwartz et al., 2009; White, Frederiksen, & Collins,
2009). Therefore, our current research is directed toward
scaffolding learners’ use of self-regulation using artificial
pedagogical agents (PAs) during learning with MetaTutor,
a multi-agent adaptive hypermedia learning environments
that models, scaffolds, and fosters learners’ use of
cognitive and metacognitive SRL processes during
learning about human biology.
Learners attempting to self-regulate often face
limitations in their own metacognitive skills, which, when
compounded with lack of domain knowledge, can result in
cognitive overload in open-ended learning environments
like hypermedia (Azevedo, Johnson, Chauncey, &
Graesser, in press; Leelawong & Biswas, 2008;
McQuiggan, Robinson, & Lester, 2010). One method of
relieving the cognitive burden placed on learners in this
situation is to provide assistance in the form of adaptive
scaffolding. Previous experiments conducted by Azevedo
and colleagues (e.g., Azevedo, Cromley, Winters, Moos, &
Greene, 2005; Azevedo, Moos, Greene, Winters, &
Cromley, 2008) established that adaptive scaffolding
Abstract
An experiment was conducted to test the efficacy of a new
intelligent hypermedia system, MetaTutor, which is intended to
prompt and scaffold the use of self-regulated learning (SRL)
processes during learning about a human body system. Sixtyeight (N=68) undergraduate students learned about the human
circulatory system under one of three conditions: prompt and
feedback (PF), prompt-only (PO), and control (C) condition. The
PF condition received timely prompts from animated pedagogical
agents to engage in planning processes, monitoring processes, and
learning strategies and also received immediate directive
feedback from the agents concerning the deployment of the
processes. The PO condition received the same timely prompts,
but did not receive any feedback following the deployment of the
processes. Finally, the control condition learned without any
assistance from the agents during the learning session. All
participants had two hours to learn using a 41-page hypermedia
environment which included texts describing and static diagrams
depicting various topics concerning the human circulatory
system. Results indicate that the PF condition had significantly
higher learning efficiency scores, when compared to the control
condition. There were no significant differences between the PF
and PO conditions. These results are discussed in the context of
development of a fully-adaptive hypermedia learning system
intended to scaffold self-regulated learning.
Objectives and Theoretical
Framework
When learning about complex science topics such as the
human circulatory system, research indicates that
individuals can gain deep conceptual understanding
through effective use of self-regulated learning (SRL). The
successful use of cognitive and metacognitive SR
processes involves setting meaningful goals for one’s
learning, planning a course of action for attaining these
goals, deploying a diverse set of effective learning
strategies in pursuit of the goals, continuously monitoring
11
provided by a human tutor leads to greater deployment of
sophisticated
planning
processes,
metacognitive
monitoring processes, and learning strategies as well as
larger shifts in mental models of the domain. The purpose
of the current work is to determine if adaptive scaffolding,
provided by pedagogical agents (PAs) within an adaptive,
intelligent hypermedia learning environment, would also
be capable of producing the same or better learning
outcomes and increased use of effective self-regulated
learning processes.
This experiment used a mixed-methodology design that
combined product and process data to examine the effect of
various types of SRL prompting and scaffolding delivered
by pedagogical agents (PAs) in an adaptive intelligent
hypermedia learning environment. Three learning
conditions were used to determine the efficacy of
scaffolding SRL through pedagogical agents: 1) prompting
with feedback condition (PF), 2) prompting only condition
(PO), and 3) no prompting condition (NP). Participants
were randomly assigned to one of the three conditions and
asked to learn about the human circulatory system using a
hypermedia learning environment, MetaTutor, during a
two-day experiment. This experiment included the
collection of concurrent think-aloud protocols, eyetracking data, human-agent dialogue, learning outcome
measures, log-file data, metacognitive judgments during
learning, embedded quizzes, and facial recognition data for
affect classification. Due to the complexity of the data
analyses, we only report on a few of the log-file variables
and the learning outcomes data (presented here as learning
efficiency).
learning environment including 41 pages of text and static
diagrams, organized by a table of contents displayed in the
left pane of the environment (See Figure 1). The version of
MetaTutor used in this experiment comprises material on
the human circulatory system. Along with the table of
contents, the environment includes a timer indicating time
remaining, an SRL palette which learners may use to
instantiate an interaction with the pedagogical agent, and
an overall learning goal (which is the same for all
participants) and sub-goals (which are created by all
participants at the beginning of the learning session.
Additionally, four distinct pedagogical agents (Gavin, Pam,
Mary, and Sam) are displayed in the upper right-hand
corner of the environment, which provide varying degrees
of prompting and feedback throughout the learning session
designed to scaffold students’ SRL skills and content
understanding (see Azevedo, Johnson, Chauncey, &
Burkett, 2010 for details).
Learning Conditions
We designed and tested three versions of the MetaTutor
environment that covered the human circulatory system. In
the Prompt and Feedback (PF) version, participants were
prompted (by PAs) to use specific self-regulatory
processes, and given immediate feedback about their use of
those processes. In the Prompt only (PO) version,
participants received the same prompts as the ones
provided to those in the PF version. However, the agents in
the PO version did not provide feedback. The timing of the
prompts used in both the PF version and the PO version
was adaptive to the individual learner and was determined
using various factors of learner interaction, including time
on page, time on current sub-goal, number of pages visited,
relevancy of the current page for the current sub-goal, etc.
In the No Prompt (NP) version of MetaTutor, participants
did not receive prompts or feedback. All three versions
(PF, PO, NP) provided an SRL palette which allowed
participants to self-select any SRL processes they would
like to use during the learning session.
Method and Data Sources
Participants
Participants were 69 undergraduate students (70% females)
from a large public university in Southeastern United
States. The mean age of the participants was 23 and their
mean GPA was 2.84. All participants were paid $10 per
hour, up to $40 for completion of the 2-day, 4-hour
experiment.
Experimental Procedure
On day one of the experiment, participants completed a
demographics questionnaire and the pretest on the human
circulatory system. Learners were given up to 20 minutes
to complete the pretest.
On day two, the learners engaged in the learning session
and completed the posttest on the human circulatory
system. Before beginning the learning session, the Tobii
T60 (Tobii T60, 2009) eye-tracker was calibrated to each
participant individually. All participants were then
instructed in the think-aloud procedure and shown a short
video demonstrating thinking aloud. Next, each participant
was shown another short video explaining and
demonstrating the various functionalities of MetaTutor and
providing the learners with their overall learning goal (See
Figure 1 for learning goal). This introductory video also
Materials and MetaTutor
The materials used in the experiment consisted of several
computerized elements. The pretest and posttest each
comprised 25 multiple-choice items with four foils for each
question. Items on the pretest and posttest included textbased items (which could be answered by directly referring
one sentence within the content) and inferential items
(which required integrating information from at least two
sentences within the content). Two equivalent forms of the
test were created using a total of 50 items and the forms
used for pretest and posttest were counterbalanced across
participants. The learning environment used by all
participants, MetaTutor, is an adaptive hypermedia
12
demonstrated the use of an electronic note-taking feature
within the environment and instructed the participants to
use the peripheral drawing pad if and when they chose to
draw. Following the introductory videos, the learners were
given two hours to learn about the human circulatory
system using MetaTutor. All participants were provided
the opportunity to take a break during the two hours,
although not all chose to do so. During the learning
session, participant verbalizations and gross looking
behaviors were recorded using an embedded webcam
within the eye-tracker monitor. Immediately after the
learning session, participants were given up to 20 minutes
to complete the posttest. Finally, all participants were paid
and debriefed before leaving the lab.
significantly more pages (M = 38.87, SD = 3.84) than both
the PO condition (M = 33.26, SD = 8.39; p < .05) and the
PF condition (M = 23.56, SD = 10.07; p < .001).
Additionally, the PO condition visited significantly more
pages than the PF condition, p < .001. Subsequent revisits
to the same page were not counted in total.
Amount of Time on Pages and Diagrams
The results indicate that students did not differ
significantly in the amount of time spent on each page (See
Table 1). On average, students spent anywhere from 60
seconds to 90 seconds on each page (p >.05). By contrast,
an ANOVA revealed a significant difference among groups
in the mean time spent viewing individual diagrams within
the environment, F (2,66) = 3.02, p = .052. LSD Post-hoc
analyses revealed that mean diagram view time was greater
for the PF condition (M = 1.05 min, SD = 0.99) compared
to the Control condition (M = 0.54 min, SD = 0.46), p =
.016. The PO condition did not differ significantly from the
remaining two conditions (M = 0.75 min, SD = 0.51).
Number of Sub-Goals Generated during
Learning
An ANOVA indicated a significant difference among the
groups in the number of sub-goals generated during the
learning session, F (2,66) = 8.74, p < .001. LSD post-hoc
analyses revealed that the PO condition (M = 4.13, SD =
1.29) and the Control condition (M = 4.70, SD = 1.72) both
attempted significantly more sub-goals than the PF
condition (M = 3.04, SD = 0.98), p < .01. There was not a
significant difference between the PO condition and the
Control condition.
An ANOVA indicated a significant difference among
the groups in the mean time spent on each individual subgoal during the learning session, F (2,66) = 10.31, p <
.001. LSD post-hoc analyses revealed that the PF condition
(M = 41.39, SD = 18.62) spent significantly longer on each
sub-goal compared to both the PO condition (M = 27.77,
SD = 9.96) and the Control condition (M = 23.30, SD =
12.18), p < .01.
Figure 1. Screenshot of MetaTutor.
Results
In this section we present some of the log-file data and the
learning outcomes data. The results are presented in Table
1.
Learning Time with the Science Content
Learning time was calculated by summing the amount of
time spent viewing the instructional content. Interactions
with the agents, in which the instructional content was not
visible, were not included in learning time. An ANOVA
indicated a significant difference among the groups in
learning time, F (2,66) = 40.71, p < .001. LSD post-hoc
analyses indicated that the Control group had a longer total
learning time (M = 87.94, SD = 12.42) when compared to
both the PO condition (M = 68.31, SD = 11.18) and the PF
condition (M = 56.84, SD = 11.82), p < .001. Additionally,
the PO condition had a significantly longer learning time
compared to the PF condition, p < .01.
Number of Pages Visited. An ANOVA indicated a
significant difference among the groups in the mean
number of pages visited (out of 41 possible1) during the
learning session, F (2,66) = 22.17, p < .001. LSD post-hoc
analyses revealed that the Control group visited
Learning Efficiency
Each participant received one point for each correct answer
selected on the pretest and posttest. From this value, a
learning efficiency score was calculated by dividing the
raw posttest score by the number of minutes the participant
was actually learning (time on task). Time on task was
defined as the sum of all of the time spent viewing domainrelated content (text and/or diagram). During certain
periods of the learning session, the learning content was
hidden from view due to interactions with the agent. To
account for differential learning time, the time each
participant spent viewing the learning content was factored
in to the learning efficiency score (Faw & Waller, 1976;
Simons, 1983).
13
An analysis of variance (ANOVA) on the learning
efficiency scores indicated a significant effect of learning
condition on learners learning efficiency (F [2,66] = 6.64, p
< .01). Post-hoc comparisons revealed that the Prompt and
Feedback (PF) condition significantly outperformed the No
Prompt (NP) condition (d = 0.84). Marginal differences
were demonstrated for each of the remaining two
comparisons (p > .05). See Table 1 for descriptive
statistics.
extend current models of SRL and build more sophisticated
intelligent multi-agent technology-learning environments
designed to detect, trace, model, and foster students’ SRL.
Our study contributes to an emerging field that merges
educational, cognitive, and learning sciences, and
educational technology research by addressing issues
related to learning about complex science topics with
multi-agent environments (Azevedo et al., 2010; Jacobson,
2008; Leelawong & Biswas, 2008; McQuiggan et al.,
2010). Our study also contributes to an emerging body of
evidence which illustrates the critical role of SRL in
students’ learning with hypermedia (e.g., Azevedo, 2009;
Azevedo et al., in press), and extends recent research
regarding the role of intelligent, adaptive scaffolding in
facilitating students’ learning with hypermedia.
Converging temporally-aligned, multi-level data will allow
us to examine the critical role of PAs as external regulatory
agents whose scaffolding methods facilitate students’ selfregulated learning (e.g., Azevedo et al., in press a; Chi et
al., 2001; Graesser et al, 2001). Lastly, both our product
and process data can be applied to inform the design of
intelligent multi-agent hypermedia environments as
Metacognitive tools (Azevedo, 2005, 2009; Azevedo et al.,
in press a) to foster learners’ self-regulated learning of
challenging science topics by providing adaptive
scaffolding (e.g., Jacobson, 2008; Graesser, McNamara, &
VanLehn, 2005; White & Frederiksen, 2005).
Scientific Significance and
Contributions of the Study
Our results show that college students’ learning about a
challenging science topic with hypermedia can be
facilitated if they are provided with adaptive prompting
and feedback scaffolding designed to regulate their
learning. More importantly, we have demonstrated that
PAs are effective in facilitating students’ SRL processes by
providing timely prompting and feedback. Their
effectiveness stems from the system’s ability to determine
optimal times during a learning session (e.g., prompting
learners to activate their prior knowledge at the beginning
of each generated sub-goal; prompting students to assess
whether the current text and diagram are relevant for the
current sub-goal). We have demonstrated the effectiveness
of prompting and feedback by showing that students in this
condition (i.e., PF condition) read less science materials
and navigated through fewer hypermedia pages during the
learning task. They also tended to spend more time on each
page and they also spend more time inspecting each
diagram presented in MetaTutor. Those in the PF condition
also set fewer sub-goals but they spend more time on each
sub-goal. Overall, the data support existing theoretical
frameworks and models of SRL (Pintrich, 2000; Winne &
Hadwin, 2008; Zimmerman, 2006) related to the use of
computers as MetaCognitive tools (Azevedo, 2005, 2009;
Azevedo & Greene, in press; Azevedo et al., in press a,b).
Subsequent analyses of the verbal protocols, metacognitive
judgments, emotions data, and log-file data will allow us to
Acknowledgements
The research presented in this paper has been supported by
funding from the National Science Foundation (Early
Career Grant DRL 0133346, DRL 0633918, DRL
0731828, HCC 0841835) awarded to the first author. The
authors would also like to thank Michael Cox and Rachel
Anderson for the data collection, and Art Graesser and
Danielle McNamara for the design and development of
MetaTutor.
14
References
Chi, M. T.H., Siler, S., Jeong, H., Yamauchi, T., &
Hausmann, R. (2001). Learning from human tutoring.
Cognitive Science, 25, 471-533.
Azevedo, R. (2005). Using hypermedia as a metacognitive
tool for enhancing student learning? The role of selfregulated learning. Educational Psychologist, 40(4), 199209.
Hmelo-Silver, C. E., & Azevedo, R. (2006). Understanding
complex systems: Some core challenges. Journal of the
Learning Sciences, 15(1), 53-61.
Azevedo, R., & Cromley, J.G. (2004). Does training on
self-regulated learning facilitate students’ learning with
hypermedia? Journal of Educational Psychology, 96, 523535.
Jacobson, M. (2008). A design framework for educational
hypermedia systems: Theory, research, and learning
emerging scientific conceptual perspectives. Educational
Technology Research & Development, 56, 5–28.
Azevedo, R., Cromley, J.G., & Seibert, D. (2004). Does
adaptive scaffolding facilitate students' ability to regulate
their learning with hypermedia. Contemporary Educational
Psychology, 29, 344-370.
Jacobson, M.J., & Azevedo, R. (2008). Scaffolding
learning with hypertext and hypermedia: Theoretical,
empirical, and design issues. Educational Technology
Research and Development, 56(1), 1-3.
Azevedo, R., Cromley, J.G., Winters, F.I., Moos, D.C., &
Greene, J.A. (2005). Adaptive human scaffolding
facilitates adolescents’ self-regulated learning with
hypermedia. Instructional Science, 33, 381-412.
Jacobson, M., & Wilensky, U. (2006). Complex systems in
education: Scientific and educational importance and
research challenges for the learning sciences. Journal of
the Learning Sciences, 15(1), 11-34.
Azevedo, R., & Greene, J. A. (in press). Introduction: The
measurement of learners’ self-regulated cognitive and
metacognitive processes while using computer-based
learning environments. Educational Psychologist.
Lajoie, S.P., & Azevedo, R. (2006). Teaching and learning
in technology-rich environments. In P. Alexander & P.
Winne (Eds.), Handbook of educational psychology (2nd
ed.). Mahwah, NJ: Erlbaum.
Azevedo, R., Guthrie, J.T., & Seibert, D. (2004). The role
of self-regulated learning in fostering students’ conceptual
understanding of complex systems with hypermedia.
Journal of Educational Computing Research, 30(1), 87111.
Leelawong, K., & Biswas, G. (2008). Designing learning
by teaching agents: The Betty's Brain System.
International Journal of Artificial Intelligence in
Education, 18, 181-208.
Azevedo, R., Cromley, J.G, Winters, F.I., Moos, D.C.,
&Greene, J.A. (2005). Adaptive human scaffolding
facilitates
adolescents’
self-regulated
learning.
Instructional Science, 33, 381-412.
McQuiggan, S., Robison, J., & Lester, J. (2010). Affective
transitions in narrative-centered learning environments.
Educational Technology & Society, 13, 40-53, 2010.
Azevedo, R., Johnson, A., & Chauncey, A. & Graesser, A.
(in press a). Use of hypermedia to convey and assess selfregulated learning. In B. Zimmerman & D. Schunk (Eds.),
Handbook of self-regulation of learning and performance.
New York: Routledge.
Azevedo, R., Moos, D., Witherspoon, A., & Chauncey, A.
(in press b). Measuring cognitive and metacognitive
regulatory processes used during hypermedia learning:
Issues and challenges. Educational Psychologist
Opfermann, M., Azevedo, R., & Leutner, D. (in press).
Metacognition and hypermedia learning. In N. Seel (Ed.),
Encyclopedia of the learning sciences. Amsterdam, The
Netherlands: Springer.
Pintrich, P.R. (2000). The role of goal orientation in selfregulated learning. In M. Boekaerts, P. Pintrich, & M.
Zeidner (Eds.), Handbook of self-regulation (pp. 451-502).
San Diego, CA: Academic Press.
Azevedo, R., & Witherspoon, A. M. (2009). Self-regulated
use of hypermedia. In D. J. Hacker, J. Dunlosky, & A. C.
Graesser (Eds.), Handbook of metacognition in education.
(pp. 319-339). New York, NY: Routledge.
Puntambekar, S., & Hubscher, R. (2005). Tools for
scaffolding students in a complex learning environment:
What have we gained and what have we missed?
Educational Psychologist, 40(1), 1-12.
Graesser, A., McNamara, D., & VanLehn, K. (2005).
Scaffolding deep comprehension strategies through
Pint&Query, AuthTutor and iSTRAT. Educational
Psychologist, 40(4), 225-234.
White, B., & Frederiksen, J. (2005). A theoretical
framework and approach for fostering metacognitive
development. Educational Psychologist, 40(4), 211-233.
15
Winne, P.H. (2001). Self-regulated learning viewed from
models of information processing. In & B. Zimmerman &
D. Schunk (Eds.), Self-regulated learning and academic
achievement: Theoretical perspectives (pp. 153-189).
Mawah, NJ: Erlbaum.
Zimmerman, B. (2001). Theories of self-regulated learning
and academic achievement: An overview and analysis. In
& B. Zimmerman & D. Schunk (Eds.), Self-regulated
learning and academic achievement: Theoretical
perspectives (pp. 1-37). Mawah, NJ: Erlbaum.
Schwartz, D. L., Chase, C., Chin, D. B., Oppezzo, M.,
Kwong, H., Okita, S. et al. (2009). Interactive
metacognition: Monitoring and Regulating a Teachable
Agent. In D. J. Hacker, J. Dunlosky, & A. C. Graesser
(Eds.), Handbook of metacognition in education. (pp. 340358). New York, NY: Routledge.
Shapiro, A., & Niederhauser, D. (2004). Learning from
hypertext: Research issues and findings. In D. Jonassen
(Ed.).
Handbook
of
Research
for
Education
Communications and Technology (2nd ed). Mahwah, NJ:
Erlbaum.
White, B., Frederiksen, J., & Collins, A. (2009). The
interplay of scientific inquiry and metacognition: More
than a marriage of convenience. In D. J. Hacker, J.
Dunlosky, & A. C. Graesser (Eds.), Handbook of
metacognition in education. (pp. 175-205). New York, NY:
Routledge.
Winne, P., & Hadwin, A. (2008). The weave of motivation
and self-regulated learning. In D. Schunk & B.
Zimmerman (Eds.), Motivation and self-regulated
learning: Theory, research, and applications (pp. 297–
314). Mahwah, NJ: Erlbaum.
Zimmerman, B., & Schunk, D. (2001). Self-regulated
learning and academic achievement (2nd ed.). Mawah, NJ:
Erlbaum.
Zimmerman, B. (2006). Development and adaptation of
expertise: The role of self-regulatory processes and beliefs.
In K. Ericsson, N. Charness, P. Feltovich, & R. Hoffman
(Eds.), The Cambridge handbook of expertise and expert
performance (pp. 705–722). New York: Cambridge
University Press.
16