Cognitive and Metacognitive Educational Systems: Papers from the AAAI Fall Symposium (FS-10-01)
How to Support Meta-Cognitive Skills for Finding and Correcting Errors?
Erica Melis
Andreas Sander
Dimitra Tsovaltzi
German Research Center for Artificial Intelligence (DFKI)
Universität des Saarlandes
German Research Center for Artificial Intelligence (DFKI)
Stuhlsatzenhausweg 3 (Building D3 2)
D-66123 Saarbrücken
Stuhlsatzenhausweg 3 (Building D3 2)
D-66123 Saarbrücken
andreas.sander@dfki.de
D-66123 Saarbrücken
melis@dfki.de
Dimitra.Tsovaltzi@dfki.de
be presented in a variety of ways, with different kinds of
feedback, diverse tutorial strategies and diverse sequencing
of learning material. Adapting such features to the needs of
individual students could contribute to better learning.
We conjecture that erroneous examples will give students
the opportunity to find and reflect upon errors in a way that
will lead to deeper learning, while at the same time not
causing students to feel ashamed or demotivated, as is more
likely when their own errors are exposed.
During the last years we conducted a number of experiments, some of them strictly controlled lab and classroom
studies, other more preliminary less controlled classroom
experiments, and a Wizard-of-Oz experiment.
Abstract
Meta-cognitive skills to be developed in learning for the 21st
century is the detection and correction of errors in solutions. These meta-cognitive skills can help to detect errors
the learner has made her/himself as well as errors others have
made.
Our investigations in learning from errors have the ultimate
goal to adapt the selection and presentation to the learner so
that he/she can better learn from erroneous examples others
have made. In our experiments we found that (1) erroneous
examples with help provision can promote students skill of
find errors, (2) the benefit from erroneous examples depends
on the relation between the student’s level and the example’s
difficulty, i.e. if the student is prepared for the problem, (3)
for many students it is very difficult to correct errors.
Finding Errors in Erroneous Examples
Our experiments (Melis & Kriesell 2009; Tsovaltzi et al.
2009; 2010) tested whether erroneous examples in the domain of fractions can help students learn in the context
of Technology-Enhanced Learning (TEL). This could spare
students the embarrassment and demotivation of confronting
their own errors. We conducted lab studies and classroom
studies with students of different curriculum levels to measure the effects of learning through erroneous examples. We
summarize the results of these studies that compare learning
gains in three experimental conditions: a control condition, a
condition in which students were presented with erroneous
examples without help, and a condition in which students
were provided with help for error detection and correction.
Introduction
Some of the meta-cognitive skills to be developed in learning for the 21st century are the detection and correction of
errors in problem solutions. These meta-cognitive skills can
help to detect errors the learner has made her/himself as well
as errors others have made, e.g., errors hidden in solutions
on the web or in other instructions.
However, confronting students with erroneous solutions
of mathematical problems and asking them to find the error
is rather unusual in most schools (except Japanese schools)
and many teachers are ambivalent about – or even hostile to
– discussing errors in the classroom (Tsamir & Tirosh 2003).
Our ultimate goal is to use adaptive technology to help
students to learn by dealing with errors. That is, help students learn mathematics by presenting erroneous examples
together with exercises in an adaptive fashion within the
TEL system ACTIVE M ATH, a web-based learning environment for mathematics. We hope to improve students’ cognitive competencies in math learning, as well as their metacognitive competencies in error discovery, error correction
and self-monitoring.
Erroneous examples are worked solutions that include one
or more errors that the student is asked to detect and/or correct. Technology opens new possibilities of instruction with
erroneous examples. For example, erroneous examples can
Empirical Results
The erroneous examples’ benefit depends on the relation between the student’s level and the example’s difficulty, i.e.
the student’s readiness statement. and needs, (similar to the
results of investigations dealing with more vs. less interactivity in (vanLehn et al. 2007)).
We hypothesize that the effect of erroneous examples depends on when and how they are introduced to the students
and whether students are supported in finding and correcting
the error.
Design
c 2010, Association for the Advancement of Artificial
Copyright Intelligence (www.aaai.org). All rights reserved.
Erroneous examples in ACTIVE M ATH include instances of
typical errors students make, which address standard prob-
64
lems students face with rule-application, or errors which target common misconceptions and address more fundamental
conceptual understanding infractions. Erroneous examples
consist of two phases: error detection and error correction.
Based on pilot studies, we designed feedback for helping students understand and correct the errors. There are
four types of unsolicited feedback: minimal feedback (flag
feedback), error-awareness and detection (EAD) feedback,
self-explanation feedback (realized as MCQs) and errorcorrection scaffolds.
We conducted lab studies with 6th, 7th and 8th-graders
and school studies with 9th and 10th-graders. The participants came from both, urban and suburban, German schools.
All studies used three experimental conditions. The condition No-Erroneous-Examples (NOEE) was the control and
included standard fraction exercises with minimal feedback
and the correct solution, but no erroneous examples in the intervention. The condition Erroneous-Examples-With-Help
(EEWH) included standard exercises, and erroneous examples with provision of help (EAD, error detection/correction
MCQs, and error-specific help). The condition ErroneousExamples-Without-Help (EEWOH) included standard exercises, and erroneous examples but without additional
help. The design included a pre-questionnaire, a pretest,
a familiarization, an intervention, a posttest and a postquestionnaire. The pre- and posttests were counterbalanced.
when asked to correct the errors in erroneous examples. We
wanted to find reasons for these difficulties.
Experimental Set-Up
The experiment took place at a secondary school in
Heusweiler, Germany. 35 7th grade students (8 male and 27
female students) participated in this experiment and for each
students the experiment took 45 minutes. Before the main
experiment started, each subject received a short introduction about the user interface. During all experiments a total
of 102 instances of erroneous examples were processed and
for 65 instances, students detected the error at first request.
We used a WOz environment developed in our group in
which the student and the Wizard can communicate. The
Wizard (the second author) selected one to four predefined
erroneous examples. The selection criteria included: (1) not
too difficult to find the error, (2) errors are close to errors or
misconceptions the student made in the pretest. Always, the
first question to the subject was whether the computations in
the exercise are correct.
The Wizard’s plan was to first, ask the student why the
result cannot be correct, then the subject should show where
the error actually is and tell what the error is and why it is
an error. Then the student was supposed to correct the error
and simultaneously write down what was in her/his mind. In
cases where subjects had problems with this method during
problem solving, a retrospective protocol after the problem
solving process was used. Both the thinking aloud method
and retrospective protocol did not yet provide much input.
Results
In the studies with different levels of students we had results
that showed differences in how erroneous examples with
error detection and -correction help influence mathematics
learning. We found that, at least, more advanced students
(9th- and 10th-grade) benefit from erroneous examples with
help in terms of cognitive skills in general, as opposed to
erroneous examples without help, or no use of erroneous examples at all. We also found that erroneous examples as a
whole and error detection and -correction help can promote
conceptual understanding for the same students.
Erroneous examples can also influence the metacognitive skill of error detection for less advanced (6th-grade) but
highly competent students.
We did not find evidence for the use of erroneous examples for medium-advanced students (7th- and 8th-grade).
However, this can be a result of the combined high level and
high competence which our participants had in comparison
to the students of the 9th and 10th-grade.
In those experiments we found that most students
do not have the meta-cognitive skill of correcting
errors.
Why is this the case?
Therefore, we
wanted to employ the Wizard-of-Oz (WOz) technique
(cf. http://en.wikipedia.org/wiki/Wizard of Oz experiment)
in order to find out more about reasons of correction weaknesses.
Results
At first, many students claimed to see a correct solution and
did not find the error at first try. This was the case for 47
out of the 102 instances. In 27 out of 65 cases the participants could not correct the error properly at all. That is, in
more than 40% of the cases, in which the students spotted
the errors, they could not correct them .
In the following, we present two typical interaction excerpts which were based on the erroneous example containing a wrong computation. The translation of the task in English is
The rain barrel: Ulli collects rain water in a rain barrel in
his garden and waters the plants in his garden. After several
days of rain the 320 liter barrel was filled. The day after the
rains Ulli asks: yesterday I had 320 liter in the barrel. Then I
used 5 12 liter for my plants. How much is still in the barrel?
and he computes...
1
2
11
= 320 −
2
320 − 11
=
2
309
=
2
... did he compute correctly? Please answer this question
with ”YES” or ”NO”
320 − 5
Correcting Errors in Erroneous Examples
In a WOz experiment we addressed the question, why many
students – even those who can find the errors or can solve
a (fraction) problem on their own – have great difficulties
65
In this interaction the participant first answered, that all
computations in the exercise are correct, but then (after a
short hint) the student found the error. Nevertheless, the student could not correct the mistake (compute the common denominator and adapt the two fractions), because the student
did not know how the procedure works practically.
This is a good example for the dissociation between execution (action) and monitoring (evaluation) or, the distinction between knowing that and knowing how (Ohlsson
1996), which we encountered in our earlier studies (Tsovaltzi et al. 2010). The student knew that the fractions have
to have the same denominator before subtraction of two fractions is possible. However, the student did not know how
to calculate a common denominator for the fractions. This
makes a big difference.
If a student knows what, he/she often only remembers the
(name of the) algorithm that needs to be applied. This does
not mean he/she knows exactly what to do/the single steps.
That is, the student may not even remember the definition
(rather than a vague notion) of a common denominator.
Log-Excerpt 1
Original Log-Excerpt of a participant (behind the
German you find the English translation)
student:Ja En: yes
WOz:Nein, es wurde nicht richtig
net/argumentiert.
En:
no, the
tion/argumentation is incorrect
gerechcalcula-
WOz:Warum kann das errechnete Ergebnis nicht
stimmen? En: why can the result not be correct?
student:Emm weil er die (320-11)/2 nicht soo rechnen
darf. En: emm, because you cannot compute like this
WOz:Genau, warum darf er das nicht? En: correct,
why must he not?
student:Weil er nicht die 320 auf die 11/2 mit drauf
setzten darf. En: because he must not put the 320 on
top of 11/2
Log-Excerpt 2
WOz:Was muss er machen, damit er das auf einen
Bruchstrich schreiben darf? En: what should he do
before writing this in the nominator?
Original Log-Excerpt of another participant
student:Ja En: yes
student:Er muss die Zahl 320 als Bruch umwandeln
entweder 320/1 oder soo un dann den Hauptnenner
rechnen. En: he needs to extend the number 320 to
the fraction 320/1 or so and then compute the common
denominator
WOz:Richtig! Du hast den Fehler gefunden!
Correct, you’ve found the error
WOz:
Nein, es wurde nicht richtig gerechnet/argumentiert. En: no the solution is incorrect
student: lies dir die Aufgabe noch mal genau durch.
En: please read the problem again
En:
Woz:Nein En: no
WOz:Warum kann das errechnete Ergebnis nicht
stimmen? En: why can the result not be correct?
WOz:Korrigiere nun (in deinem Eingabefenster) den
ersten fehlerhaften Schritt und führe die Berechnungen
zu Ende. [...] En: now correct the first incorrect step
and complete the computation
WOz:Hast du Fragen? En: do you have questions?
student:nein En: no
student:keine ahnung En: no idea
WOz:Warum kann das errechnete Ergebnis nicht
stimmen? En: why can the result not be correct?
WOz:Wie würdest du vorgehen, um den fehlerhaften
Schritt zu korrigieren? Beschreibe in Worten. En: what
would you do to correct the wrong step? describe in
your own words.
student:weil ulli die 320 liter auch in einen bruch
umrechnen muss En: because ulli needs to transform
the 320 liter into a fraction
student:Also ich würde die 320/1 mit der 11/2
zusammen auf den Hauptnenner bringen dann das
minus rechnen. En: well, I’d compute the common
denominator of 320/1 and 11/2 and then substract
WOz:Wo ist der Fehler? [...] En: where is the error?
student:Schritt 3;weil ulli die 320 liter auch in einen
bruch umrechnen muss En: step 3; because ulli needs
to transform the 320 liter into a fraction
WOz:Richtig, versuche dies nun rechnerisch umzusetzen! En: correct, now try to compute this
WOz:Richtig! Du hast den Fehler gefunden!
correct! you found the mistake
WOz:Korrigiere nun (in deinem Eingabefenster) den
ersten fehlerhaften Schritt und führe die Berechnungen
zu Ende. [...] Please correct (in the input window) only
the first incorrect step
En:
WOz:Was genau hat er falsch gemacht? En: what
exactly did he make wrong?
student:Schritt 3;er hat die 320 liter nicht in einen
bruch umgerechnet. En: step3; he didn’t transform 320
liter to a fraction
student: wie bringt man es wieder auf den Hauptnenner? En: how is the common denominator computed?
66
examples makes a difference to learning and whether highlighting errors helps students learn from those errors. Their
empirical studies in which no help was provided showed
some learning benefit of erroneous examples, but unlike the
less ambiguous Siegler et al results, the benefit they uncovered was only for learners with strong prior knowledge and
for far transfer learning. Research in other domains, such as
medical education, has shown benefits of erroneous examples in combination with elaborate feedback in the acquisition of problem-solving schemata.
More relevant to the work presented in this paper is Ohlsson’s theory (Ohlsson 1996) argues that errors occur due to
overgeneralization (applying rules to situations where they
do not apply). Error-detection requires declarative knowledge (e.g.knowledge of rules) in the domain. The error
causes a conflict between this knowledge which the learner
believes to be true and what the learner perceives as the
current situation. The way to avoid errors is by specializing knowledge structures to trigger only in the situations
where they apply. Ohlsson distinguished between declarative knowledge (knowing that) and practical knowledge
(knowing how), which he defines as the knowledge required
to choose the appropriate action in a perceive-decide-act cycle.
The contribution of our work are empirical studies that
explicitly tackle this dissociation, in particular, in the domain of fractions. This in turn can inform the design of
erroneous examples and intruction not only in the context
of technology-enhanced learning, which we focus on, but in
general.
WOz:Wie würdest du vorgehen, um den fehlerhaften
Schritt zu korrigieren? En: How would you proceed in
order to correct the incorrect step?
student:Schritt 3;ich würde aus den 320 liter 160/2
machen und dann die 11/2 davon abziehen En: step3;
I’d transform 320 liter to 160/2 and then substract 11/2
WOz: Wie würdest du vorgehen. Erkläre mit deinen
eigenen Worten. Ohne genaue Rechnungen. En: how
would you proceed? explain in your own words.
without exact computations.
WOz:Hast du Fragen? En: do you have questions?
student:Schritt 3; ich würde die 320 liter nehmen und
in 160/2 liter umwandeln, dann die 5 1/2 liter in 11/2
liter um wandeln und dann die 11/2 liter von den 160/2
liter abziehen so wäre das ergebnis: 149/2 liter En:
step 3; I’d take the 320 liter and transform it to 160/2
liter, then transform the 5 1/2 liter into 11/2 liter, and
then substract 11/2 liter from 160/2 liter with the result
149/2 liter.
In this interaction the student found the mistake (subtraction of two fractions with different denominators without
first computing the common denominator) and had an idea
how to correct the error (expansion of 320 to a fraction with
the denominator two), which means he/she was a step ahead
but still could not complete the computation of the common
denominator, i.e., did not fully understand the principle of
expanding fractions to equal denominators..
Would understanding the principle conceptually allow
students to perform the necessary actions automatically?
What exactly it is that prevents them from correcting the error once they recognize it?
Conclusion
Our goal is a cognitive tool that supports learning from error, in particular from erroneous examples and that increases
monitoring alert. As the results show, the tool needs to
adapt to the student’s mastery level for sequencing and metacognitive help provision.
This article describes results on our way to support students in developing the meta-cognitive skills of error detection and error correction in ACTIVE M ATH.
Related Work
Erroneous examples in mathematics have rarely been investigated or used as a learning intervention, either within
a technology-enhanced learning (TEL) system or within
the classroom. Overall, the related previous studies did
not target adaptation by the technology. The empirical results (sometimes inconclusive) evidence that studying errors
can promote student learning (Borasi 1994; Siegler 2002;
Curry 2004; Grosse & Renkl 2004; Monthienvichienchai &
Melis 2005; Grosse & Renkl 2007; Siegler & Chen 2008).
The skill of correcting errors was not addressed before.
A controlled comparison of the study of correct and incorrect examples was done by Siegler and colleagues (Siegler
2002; Siegler & Chen 2008). They investigated whether
self-explaining both correct and incorrect examples is more
beneficial than self-explaining correct examples only. They
found that when students studied and self-explained both
correct and incorrect examples they learned better. They
hypothesized that self-explanation of correct and erroneous
examples strengthened correct strategies and weakened incorrect problem solving strategies, respectively. Grosse and
Renkl studied whether explaining both correct and incorrect
Acknowledgement This work was supported by Deutsche
Forschungsgemeinschaft in the project ME 1136/7-1.
References
Borasi, R. 1994. Capitalizing on errors as ”springboards for
inquiry”: A teaching experiment. Journal for Research in
Mathematics Education 25(2):166–208.
Curry, L. 2004. The effects of self-explanations of correct and incorrect solutions on algebra problem-solving performance. In D. Gentner, K. F., and Regier, T., eds., 26th
Annual Conference of the Cognitive Science Society, 1548.
Mahwah, NJ: Erlbaum.
Grosse, C., and Renkl, A. 2004. Learning from worked examples: What happens if errors are included? In Gerjets, P.;
Elen, J.; Joiner, R.; and Kirschner, P., eds., Instructional design for effective and enjoyable computer-supported learn-
67
ing. Tuebingen: Knowledge Media Research Center. 356–
364.
Grosse, C., and Renkl, A. 2007. Finding and fixing errors in worked examples: Can this foster learning outcomes?
Learning and Instruction 17:612–634.
Melis, E., and Kriesell, D. 2009. Effects of erroneous examples, results of a preliminary school experiment. In V. Dimitrova, R. Mizoguchi, B. d. B., and Graesser, A., eds., AIED
2009, volume 200 of Frontiers in Artificial Intelligence and
Applications, 578–580. IOS press.
Monthienvichienchai, R., and Melis, E. 2005. Implementing
courseware to support learning through real-world erroneous
examples: Student’s perceptions of tertiary courseware and
obstacles to implementing effective delivery through vle. In
Remenyi, D., ed., Proceedings of European Conference on
eLearning (ECEL-2005), 281–289. Academic Conferences
International.
Ohlsson, S. 1996. Learning from performance errors. Psychological Review 103:241–262.
Siegler, R., and Chen, Z. 2008. Differentiation and integration: Guiding principles for analyzing cognitive change.
Developmental Science 11:433–448.
Siegler, R. 2002. Microgenetic studies of self-explanation.
In Granott, N., and Parziale, J., eds., Microdevelopment,
Transition Processes in Development and Learning. Cambridge University Press. 31–58.
Tsamir, P., and Tirosh, D. 2003. In-service mathematics
teachers’ views or errors in the classroom. In International
Symposium : Elementary Mathematics Teaching.
Tsovaltzi, D.; Melis, E.; McLaren, B.; Dietrich, M.;
Goguadze, G.; and Meyer, A.-K. 2009. Erroneous examples: A preliminary investigation into learning benefits. In
Ulrike Cress, Vania Dimitrova, M. S., ed., European Conference for Technology-Enhanced Learning, EC-TEL 2009,
LNCS 5794, 688–693. Springer-Verlag.
Tsovaltzi, D.; Melis, E.; McLaren, B.; Meyer, K.; Dietrich,
M.; and Goguadze, G. 2010. Learning from erroneous examples: When and how do students benefit from them? In ,
ed., EC-TEL 2010, volume - of LNCS, –. Springer-Verlag.
vanLehn, K.; Graesser, A.; Jackson, G.; Jordan, P.; and
C.P.Rose. 2007. When are tutorial dialogues more effective than reading? Cognitive Science 31:3–62.
68