Generating Sensible Feedback
for Gap-Filling Exercises with
Verbal Units in the Chemnitz
Internet Grammar
Christian Eckhardt
ENGLISH LANGUAGE AND LINGUISTICS,
CHEMNITZ UNIVERSITY OF TECHNOLOGY, D-09107 CHEMNITZ1
I. INTRODUCTION
The exercise component of the Chemnitz Internet Grammar (CING) project, as
described by Wilson and Schmied (this volume), will be discussed in more deatail
here. The CALL software market is dominated by old-style drill and practice software.
In most cases textbooks, audio and video material has simply been transfered to CDROMs. "In reviewing these products one realises that, in terms of technology,
enourmous progress has been made; but from an educational point of view, no
progress has been made at all." (Wolff 1998: 69)
One of the central ideas of the CING is to promote learner autonomy and to facilitate
inductive learning. Inductive and autonomous learning both require some guidance of
the learner. Learners need an overview of what they have done so far and where their
problems lie, only then can they take qualified decisions on what to do next. Guidance
can either be immediate feedback given in the exercise component or advice and
information given at any other time. The system's knowledge about the learner should
comprise more than facts such as "s/he has made x% mistakes in the exercise on
present simple and continuous". Ideally it should categorise incorrect answers
according the presumed cause of the deviation and give feedback accordingly.
I want to thank my colleagues Josef Schmied, Angela Hahn and Naomi Hallan for their support
and advice. The Chemnitz Internet Grammar project is located at:
http://www.tu-chemnitz.de/InternetGrammar/
1
Two approaches to the improvement of feedback will be shown: a) error anticipation
for every single item as practiced with authoring systems and b) error analysis
performed by knowledge-based tutoring systems developed in Artificial Intelligence
(AI) research.2 Soria (1997: 44) has termed a compromise he applied to the inflection
of Spanish verbs 'Expert CALL':
In between these two approaches there may be a role for applications based on linguistic
knowledge which is not as complete or as formalised as the afore-mentioned theories [of AI]
propose, but wich is reliable and suffecient for teaching and learning.
In the last section a similar compormise will be proposed for verbal units 3 in gapfilling exercises.
II. THE PROBLEM
An easy way to produce a large number of exercises for CALL is to develop a kind of
authoring system. Such systems allow teachers to create learning units on the
computer without knowledge of any programming language. Anticipated feedback
and hints for expected answers can be provided. Brücher (1994:126) thinks that
authoring systems are very useful because they are easy to use, inexpensive and can be
adapted to the needs of the class or course. The following example from Testmaster
1.0 (cf. Brücher 1994:120) demonstrates how authoring systems work:
(1) 01: In how many ways can you express the names of the former two Superpowers?
02: One has stars and stripes on its flag, the other had a hammer and sickle.
03: (The) [United States of America/United States/States/USA]
04: (The) [USSR/Soviet Union/Union of Socialist Soviet Republics]
05: [America/Russia]#There are more elaborate names than this.
06: (The) CCCP#OK, but you are using English letters as Russian ones.
Line 01 contains the question, line 02 a hint. In lines 03 to 05 the correct answers are
given: the article is optional and is therefore put in brackets. The square brackets
indicate that there is a list of solutions. The different solutions are separated by
slashes. In lines 05 and 06 the answers are provided with comments. The comments
are separated by the "#" symbol. If the learner answers the question (01) with
For an overview of Intelligent Tutoring Systems developed for language learning see Swartz and
Yazdani (1992) and Holland et al. (1995).
3 The term "verbal unit" has been choosen to point out that unlike "verb phrase" is does not
comprise objects and adverbials. "Verbal unit" here denotes only the inflected form of the verb (i.e.
auxiliaries, modals and the main verb).
2
"America" the program will tell him or her "There are more elaborate answers than
this." and accept the answer as correct.
Testmaster is a relatively flexible program because it is able to accept several answers
as correct and comments on other anticipated answers. An advantage of this type of
program is that teachers can adapt the exercises to the needs and the capabilities of
their learners. On the other hand teachers have to spend a lot of time devising the
exercises. The task becomes even more labour-intensive when a teacher wants to
maximise the use of the program by anticipating and commenting as many answers as
possible. Handke (1988:27) demonstrates the problem of the anticipation of answers
with a relatively simple gap-fill exercise:
(2)
John (1) _______ to London yesterday. He (2) ______ there a long time ago.
Now he (3) _______ in America at the University of
The (4) _______ of his studies is to explore expert systems.
Berkeley.
Possible correct solutions are:
(3) gap 1: went, drove, flew, got, walked etc.
gap 2: had (not) been, had stayed, had lived etc.
gap 3: is living, is studying, is staying, is working etc.
gap 4: (primary, main, basic) purpose, aim, idea etc.
Authoring systems become even more problematic when answers that consist of
more than just keywords are required, because the number of possible answers, right
or wrong, increases enormously, so that anticipation becomes impossible. Higgins
(1995: 44-49) lists 555 correct paraphrases of the sentence "Jim hated school because
the teachers were unkind." to illustrate this problem. Taking possible learner errors
into account, the number of possible sentences increases even more.4
Even when the verb to be used in a gap-filling exercise is given, a large number of
different incorrect answers are possible, e.g.:





Students may be unaware of irregular verb forms
They may get person and number wrong.
They may apply the wrong tense.
They may have word order problems. (*He walked has.)
They may have problems with simple and continuous aspect.
Though certainly aware of this fact, Borraccino (1998) has implemented a tutoring system that
allows students to answer in complete sentences. The system then tries to match the answer to
solutions stored in a database. If the answer given by the student cannot be found in the data-base it
will be transfered to a human tutor who then checks the answer "manually", passes it to the student
and adds it to the data base.
4
 They may type nonsense.
Not all answers that do not exactly match the anticipated correct answer are errors in
the sense of being based on a misconception of the grammatical rules concerned.
They may be referred to as mistakes. The most obvious example of a mistake is a mistyped word. Ignorance of irregular verbs may also lead to mistakes that are outside the
focus of the current exercise. The same applies to spelling mistakes (especially edeletion and y-to-i change).
Most programs available today get round this problem by accepting only a single
answer as the correct solution and ruling out all other solutions. This appears to be an
acceptable solution, but the programs usually also restrict their feedback to 'correct' or
'wrong' with no further help given. The "Grammatiktrainer Englisch" by Digital
Publishing (Munich) is an exception. If an error (or deviation) has occured the
program comments on every single word it detects as being wrong, as shown in the
table below:
Error
a) Have you *seed Roger?
b) I visited the Charlisles and *have gone
shopping a bit.
c) She *called me four times since we met
on the weekend.
Feedback
The Participle is wrong, it is an irregular
verb.
This action belongs to the past. Change the
tense.
This is not the present perfect.
Table 1: Feedback provided by the "Grammatiktrainer English"
Unfortunately the program for example cannot deal with the following mistake
properly:
(4)
I visited the Charlisles and *had gone shopping a bit.
Here the progam reports that it could not deal with the word "had" and comments on
"gone" with "This action belongs to the past, change the tense", which is the same
comment as with b) in the table above. Hence the program interprets the answer as
present perfect because it contains a participle, since the second word in the input is
the same as in the present perfect. This shows that the program can only deal with
anticipated answers. In other sections of the "Grammatiktrainer" other kinds of
mistakes have been anticipated. In the "past and past perfect" section the program
recognises "had gone" as a past perfect form. Though the program's feedback is not
always appropriate and does not cover some obvious cases, it is the only program
available on the German market that gives detailed feedback at all.
Another disadvantage is that the number of exercises provided for each topic is very
limited although there are more than 600 exercises on the CD-ROM. This may also be
due to the fact that the exercises are stored in fixed frames, i.e. they are not generated
from a database.
III. ARTIFICIAL INTELLIGENCE (AI) APPROACHES
Intelligent Computer-Assisted Instruction (ICAI) uses methods and tools from
artificial intelligence (AI) for the development of learning systems; these are also
called Intelligent Tutoring Systems (ITS). They are intended to approach the
performance of an idealised human tutor. Leutner (1992: 59) lists four criteria that an
ITS should fulfil:
1. An opportunity for the learner to exchange roles. The program should not only
question the leaner but also allow the learner to put questions. It should be able
to answer questions whose answer has not been explicitely stored.
2. Domain knowledge and teaching knowledge should be separated. This would
allow the program to react to learners' needs flexibly. The program should
follow a general teaching strategy and be able to fill it out with the required
contents in contrast to "non-intelligent" programs where the whole presentation
has been prepared and fixed in advance.
3. It should establish the level of the learner‘s knowledge and categorise their
working habits.
4. Based on the data gained in 3.) the program should be able to adapt to the
knowledge state of the learner and present the contents in the way most suitable
for the learner.
These criteria are met by only few ITS. To fulfil these demands on the system a
minimum of data necessary can be deduced: Self (1990) lists the different kinds of
data he considers necessary for successful adaption of the program to learners' needs:
1. Expert knowledge of the domain to be taught
2. Knowledge of teaching strategies
3. A model of the learner, reflecting his or her learning style, domain knowledge
and behaviour.
The structure of an ITS can best be explained as consisting of four components, as
shown in figure 1. The communication module is responsible for the presentation of
the domain knowledge and the exercises. It also interprets the learner's input and
passes it on to the student module. Most ITSs are text-based, i.e. the learner’s input
has to be typed. There are a few systems that allow spoken language as input and a
few that use spoken language as output (e.g. Spaai 1995). In order to allow input that
is more complex than just keywords or predefined phrases, the communication
module has to parse sentences and make their meaning accessible to the computer (cf.
O'Shea / Self 1983: 92). Therefore an ITS needs a syntactical parser and a semantic
Figure 1: The components of an ITS.
component. The parser should also be able to tolerate minor mistakes. Creating
output is much easier because the computer can make use of predefined messages or
message components.
The student module compares the learner's answer to the solution arrived at by the
expert system (expert module) and draws conclusions about the knowledge and
competence of the learner. It stores its conclusions in the student model. With an
analysis of the learners knowledge at hand the teaching module decides on what to do
next, whether to provide new exercises of the same type or to proceed to the next
topic.
The expert module contains the knowledge about the domain to be taught. It should
be able to solve all questions that the system can ask the user or answer all questions
the user asks the system. In AI research a distinction is made between procedural and
declarative knowledge. Transfered to a language teaching system, declarative kowledge
consists of facts such as words and their syntactical, morphological, phonological and
semantic properties while procedural knowledge is the knowledge of how to combine
these words into more complex words or sentences and to interpret their meaning.
Rule systems can be designed to attempt to model human thought processes or just to
try to arrive at the same output. Since there is no satifactory model of human thought,
algorithms that work independent of any model of human thought processes are
usually more effecient. The best example of such algorithms are chess programs. It
had long been thought that chess was a task that only intelligent humans could
perform well. Scientists had assumed that a computer that could play chess well could
also perform most other human tasks. Thus the construction of an "intelligent" chess
computer had been a primary aim in AI research. Attempts, such as Winograd's
SHRDLU, to model easier tasks humans perform every day, like moving about
objects or finding one's way in unknown territory have shown that this assumption is
not valid. This is basically so because the algorithms underlying most "intelligent"
systems do not model human thought but try to reach human performance in a way
more compatible with computers.
In contrast to these so-called "black-box" models, others, called "glass-box" models,
try to simulate human thought processes. This is not only problematic because human
thought can hardly be observed and there is no sufficient theory but also because
humans have types of data available to them, that computers can hardly access: a
computer cannot look into a learner’s face. Clancey (1983), one of the pioneers of
intelligent tutoring, came to the conclusion that though glass-box models are usually
less efficient, they are more suitable for tutoring, because the learner can be guided
along the same path the computer has used to come to its solution. Burton and
Brown (1983: 82) have suggested using a black-box model to come to the correct
solution an a glass-box model for the tutoring, since "backward reasoning" is much
easier than problem solving from scratch.
An Example
Most ITS are prototypes designed to teach the programming language LISP or the
basics of algebra, only few systems are concerned with the teaching of natural
languages. One of them is the "English Tutor" (ET), which is intended to help Italian
students with English tenses. It will be described here in order to give the reader an
impression of how such a system works.
ET asks the students to do gap-filling exercises and then the expert system solves the
tasks itself. Then ET compares its solution to that of the learner. The expert system,
which will not be dealt with in more detail here, is based on a model of temporal
logic by Matthiessen (1984) that enables it to determine the tense to be used.
If it detects an error it tries to figure out the cause of the error:
(5) ET:
Student:
Exercise-1: "He always (carry) an umbrella when he was in England."
was carrying
Three causes for this error are possible:
1. The student does not know the rules for the use of simple past and past continuous,
2. s/he does not know simple past and therefore uses past continuous or
3. the answer was a mistake, i.e. an unsystematic error.
To find the cause the system tries to verify or falsify alternative theses. Here it tries to
find out if the learner knows the simple past:
(6)
ET:
Student:
ET:
Student:
ET:
Student:
Exercise-2 : "I (share) a room with him when we were students."
shared
Exericise-3 : "I (make) a cake when the light went out."
made
Exercise-4: "Last winter on holiday her mother often (tell) her
that she (spend) too much money."
was telling / was spending
The answer to exericse 3 was wrong. It seems that the student has problems with the
simple past. With exercise 4 the program tests its hypothesis once more. Since the error
has occured again, a mistake (hypothesis 3) can be excluded. Having verifyed one of
its hypotheseses proved, the program gives the following feedback (Fum et al. 1988:
515):
(7)
ET: Answers to exercises 1, 3 and 4 were wrong. Your knowledge about the use of the
simple past and the past continuous seems to be the following:
You utilized the simple present according to the rule "When a clause describes a single
action or a state that takes place in the past, use the simple past."
This rule does not take into account the further condition that: "When a clause
describes a single action or a state that takes place simultaneously to the reference
time of the clause, then the tense to be used is past continuous."
The solution to exercise-1 is:
Tense: simple past
Conjugated verb: carried
because:
the action took place in the past
the action is habitual
...
The program takes the result of this sequence of tasks as a basis for the choice of
subsequent tasks. It will make the student practise the distinction between simple past
and past continuous and try to correct the student's misconception about the use of
these two tenses.
IV. THE SOLUTION APPLIED TO TENSE AND ASPECT IN
THE CING
The tutoring component of the CING will make use of some technologies that have
been developed in AI research. While ITSs are based on the idea of using an expert
system to find the solution to the task. ITSs are expected to be able to guide the
learner to the correct solution in a way that helps him or her to grasp the problem.
With this approach most attention has to be paid to the expert system’s capacity to
solve problems and to the analogy of the expesystem's algorithm to human
reasoning.
With the CING a completely different attempt that was inspired by Burton and
Brown’s idea of backward-reasoning: the CING will do without an expert system to
solve the tasks, instead the solution will be provided with the task:
(8) (a) I (have been living # to live) here since 1991.
(b) He (has never been # to be & never) to England.
(c) Columbus (sought # to seek) for a passage to India.
From these entries all information required for sensible feedback will be derived.
Compared to the information necessary for "Testmaster" shown in (1), a minimum of
information is provided with each task. It will be restricted to the correct solution and
the text to be presented to the learner (the text after the "#" symbol).
In a first step the learner's input will be compared to the correct solution. If both are
not identical, the program will derive the required tense from the correct answer and
the tense used by the learner will be deduced from his or her input. This can be done
with a lexicon that contains no more than modals and auxiliaries in all their
inflectional variants. Additionally "words"5 will be checked for three morphological
affixes ("-s", "-ed" and "-ing"). Words that contain these affixes are assumed to be full
verbs. This assumption is valid because the program does not deal with full sentences
but with verb phrases in a narrow sense, i.e. that part of a verb phrase that is made up
of modals, auxiliaries, adverbs and verbs. Nouns, like thing, king etc. that would cause
problems, are excluded by this definition.
Here the advantage of the proposed method becomes obvious: The analysis of the
learners' answers and the correct solution can be restricted to a minimum, no fullscale understanding of sentences is necessary and the lexicon can be reduced to
functional words. The only problem that remains with this solution are irregular
verbs. Either irregular forms have to be stored with each and every task or they have
Words here are defined as items in a string that are separated by blanks, but not in any linguistic
sense.
5
to be deduced otherwise. It appears to be unacceptable to extend the lexicon with a
list of all irregular verbs, because due to low band-width on the internet the lexicon
has to be kept as small as possible. From a comparison of the infinitive given with the
task and the provided solution, irregular verbs can be detected. From the information
contained in task (8c) it is possible to derive that the inflection of the verb is irregular:
1.
2.
3.
4.
sought  search
sought  search + (e)s
sought  search + (e)d
sought  search + ing
Since the answer has to be correct, the rightmost word in the solution has to be the
verb, hence it has to be an irregular form.
An example
An example will be given for the analysis of errors in the CING. Provided the
student gives the following answer to task (8a)
(9) *has been lived
the program will first compare the answer to the solution. As they are not similar
here, it has to find out what the features of the target phrase are. For that purpose it
uses the lexicon, containing auxiliaries and modals as mentioned above, is used. Here
is a sample of its entries:
be
belongs to
lexeme
-
number
person
-
&
morphologic
al verb form
Infinitive
am
be
1st Sg.
Present
is
be
3rd Sg.
Present
are
be
Present
was
be
were
be
been
be
2nd
Sg.
1st, 2nd or 3rd
Pl.
1st or 3rd Sg.
2nd or 3rd Pl.
2nd
Sg.
3rd Pl.
-
Past
Past
Participle
Table 2: The lexeme "be" in the lexicon
Table 2 contains the possible forms of the lexeme be and their implications for the
characteristics of the verb phrase. Combined with an analysis of the morphological
variation of the main verb, i.e. by checking whether it is an unmarked, a past or a
continuous form, tense, aspect, number and person can be determined. An analysis of
the subject may contribute to the precise identification the number and person of the
given phrase. If no pronoun is found 3rd person is required. The result of the analysis
will be the following:
Words
Lexeme
Category
Person
Number
Tense
have
have
aux
?
?
present
Aspect
?
been
be
aux
?
?
participl
e
?
living
liv
verb
?
?
continuous
Table 3: Analysis of the correct solution
The solution is the auxiliary have in the present tense, followed by the participle of be
and a continuous form of a verb with the root liv. In other words: the solution of task
(9a) is a present perfect continuous of the verb live.6 This way tenses can easily be
defined in the same terms grammar text-books do it.
The learner's answer (9) can be analysed in the same way, as shown in the table below.
Both, the auxiliary have and the participle of be are correct. Only the verb is inflected
wrongly. This error can stem from a mixture of the forms of the present perfect simple
and the present perfect continuous:
Words
Lexeme
Category
Person
Number
Tense
have
have
aux
?
?
present
Aspect
-
been
be
aux
?
?
participle
-
lived
liv?
verb
?
?
past or
participl
e
-
Table 4: Analysis of the incorrect solution.
This example shows that spelling rules (such as e-deletion, e-insertion, y-to-ie change) have to be
taken into account.
6
By using this kind of analysis feedback can be provided on the basis of generalised
definitions that apply for all tasks:
Condition
have
(present)
Feedback
+ be
(participle)
+ verb
You may have confused the form of the
(past or participle) present perfect simple and present perfect
continuous. Either the verb has to be in
the continuous or the participle must not
be here.
Table 5: A generalised definition of feedback.
V. CONCLUSION
The approach described here is allows on the analysis of tense forms in gap-filling
exercises, i.e. natural language processing in a very narrow context. The analysis is
limited to the identification of auxiliaries and the few morphological variants of
English verbs, which is only possible due to the restrictions inherent to gap-filling
exercises. Here much attention is paid to the formal correctness of tense forms
though this is not likely to be a major problem with the users of the CING as it is
aimed at advanced students of English.
It might be argued that the same goal could much easier be achieved by the provision
of dual-choice exercises, where the tense forms would be clear by definition and
appropriate feedback could be given according to the selection of pre-defined
answers. Apart from the assumption that learners may remember their answers better
when they have typed them in, there is another advantage to this approach for the
CING: As the aim of the CING is to observe learner behaviour in an interactive
learning environment (i.e. on the internet), the learners' success will have to be
measured in order to find out about the successfulness of learner strategies. With
exercises performed in traditional lessons it often seems to be the case that learners
perform well in exercises as long as the problem involved is in the focus of the
exercise, while problems remains difficult in other contexts. In an interactive learning
environment it will be possible to mix items from previous exercises with those of the
current exercise in order to check whether students are also successful with items
outside the focus of the current exercise. This is an obvious advantage of gap-filling
over multiple- (or dual-) choice exercises.
References
Brücher, K.H. (1994). Zur Leistungsfähigkeit und Verwendbarkeit von Autorenprogrammen in
CALL. In: J.Fechner (ed.), Neue Wege im computergestützten Fremdsprachunterricht. Langenscheidt, Berlin,
pp. 135-163.
Burton, R.R. and J.S. Brown. (1982). Investigation of Computer Coaching for Informal Learning
Activities. In: D. Sleeman and J.S. Brown (eds), Intelligent Tutoring Systems. Academic Press, London,
pp. 157-183.
Fum, D, P. Giangrandi and C. Tasso. (1988). The ET Project: Artificial Intelligence in Second
Language Teaching. In: Proceedings of the European Conference on Computers in Education, Lousane, pp. 511516.
Handke, J. (1988). Computer-Assisted Language Learning und künstliche Intelligenz. Die neueren
Sprachen 88: 134-147.
Higgins, J. (1995). Computers and English Language Learning. Intellect, Oxford.
Holland, V.M., J.D. Kaplan and M.R. Sams. (1995). Intelligent Language Tutors Theory Shaping Technology.
Lawrence Erlbaum, Mahwah, NJ.
Leutner, D. (1992). Adaptive Lernsysteme. PVU, Weinheim.
O'Shea, T. and J. Self. (1983). Learning and Teaching with Computers. Artificial Intelligence in Education.
Harvester Press, Brighton.
Self, J. (1990). Theoretical Foundations for Intelligent Tutoring Systems. Journal of Artificial Intelligence
in Education 1(4): 3-14
Soria, J. (1997). Expert CALL: Data-Based vs. Knowledge-Based Interaction and Feedback.
ReCALL 9 (2): 43-50.
Spaai, G.W.G. (1995). Feedback in Computer-Assisted Instruction: Complexity and Corrective
Effeciency. In: R.J. Beun, M. Baker and M. Reiner (eds), Dialogue and Instruction. Modelling Interaction in
Intelligent Tutoring Systems. Springer, Berlin, pp. 167-178.
Swartz, M.L. and M. Yazdani. (eds). (1992). Intelligent tutoring systems for foreign language learning: The bridge
to international communication. Springer, New York.
Wolff, D. (1998). Computers and New Technologies in Language Learning. The European English
Messenger 7(2): 69-71.