Variations in L2 spoken and written English:
investigating patterns of grammatical errors across proficiency levels
Mariko Abe, Yukio Tono
Takasaki City University of Economics, Meikai University
mabe@tcue.ac.jp, y.tono@meikai.ac.jp
1. Purpose
The purpose of this research is to investigate the variability of interlanguage by means
of a corpus-based quantitative analysis. Since the previous studies on language
acquisition have focused on a relatively limited amount of data, studies using a large
amount of learner data may be able to make a significant contribution to the field of
L2 acquisition (Biber, Conrad and Reppen 1998: 180). The spoken and written data
of Japanese learners of English was especially created to investigate the difference
between spontaneous production in speaking and written production with less time
pressure. There is a substantial body of research showing that the processing modes
of learners affect their performance in L2. Therefore, it would be essential to show
the acquisition sequence of certain grammatical features on both spoken and written
data. This paper also aims to describe the developmental patterns of grammatical
features in learner English by analysing learners’ use and misuse across different
proficiency levels.
2. Method
2.1. Learner corpora data
Two types of learner data were compared in this study; the NICT JLE Corpus, a
corpus of more than 1,200 Japanese EFL learners’ oral interview transcripts and the
JEFLL (Japanese EFL Learner) Corpus, a corpus of written essays of more than
10,000 Japanese secondary school students.
The spoken data were extracted from the NICT JLE Corpus. The data consists of
1,200 examinees who have taken Standard Speaking Test (SST), and the test has 9
different levels to assess speaking proficiency. The spoken data used in this research
came from 100 examinees whose proficiency level was assessed at SST level 2 to
level 9. Since the size of this sub-corpus is rather small for detailed study (as shown
in the following table), we hope to increase the size in future studies. There are five
different tasks in this speaking test, but only one of them, single picture description
stage, was used in the current study. A picture is chosen from five different pictures,
and the examinees were asked to describe it in two or three minutes. Therefore the
spoken data of this research does not consist of conversation or speech, but of story
retelling production.
SST
LEVEL
File
Token
SST2/3
SST4
SST5
SST6
SST7
SST8/9
total
22
1222
17
1418
16
1755
19
1891
16
2004
10
1370
100
9660
Table 1 Corpus size of spoken data
1
The written data were extracted from the JEFLL Corpus. The data of this learner
corpus consist of five different composition topics written by learners of six different
academic years at several junior and senior high school in Japan. In this paper, the
written composition produced by one particular junior and senior high school was
chosen, and only the composition entitled “My school festival” was selected.
ACADEMIC
YEAR
File
Token
Average
token
TTR
Junior1
(J1)
104
4994
Junior2
(J2)
77
5004
Junior3
(J3)
87
5000
Senior1
(S1)
46
4997
Senior2
(S2)
53
5000
Senior3
(S3)
55
5005
48.02
64.99
57.47
108.63
94.34
91
71.09
10.79
15.91
14.62
18.41
17.02
19.02
-
TOTAL
422
30000
Table 2 The corpus size of the written data
We will also demonstrate the software for processing these two learner corpora later
in the presentation. The Shogakukan Corpus Network (SCN) provides the free service
for JEFLL, while the NICT JLE Corpus has its own specialized retrieval software.
2.2. Data processing
An equal amount of data were sampled and manually error-tagged, and also POStagged by CLAWS7. Major error categories related to (1) parts of speech, and (2)
tense and aspect were identified and their normalised frequencies were compared in
terms of error types and modes of production, speech and writing, together with the
subjects’ proficiency levels in L2.
All of the spoken data were already error-tagged and published as the NICT JLE
Corpus, therefore written data of JEFLL corpus were error-tagged manually by
referring to the same Error Tagging guide (Isahara, Saiga, and Izumi 2002) as used for
the NICT JLE Corpus. The operational error tags such as redundancy, omission, and
misordering were excluded in this research but by adding the correction within the
error tag, we can still retrieve the operational errors from the concordance lines.
There are 6 types of noun-related tagset and 11 types of verb-related tag set provided
in the NICT JLE Corpus, but only the errors shown in the following table are
considered in this paper. This is because some of the errors did not have a high
enough frequency to be worth analysing. Part-of-speech (POS) tags were also
provided by the CLAWS tagger by using C7 tagset of Garside, Leech & McEnery
(1997) in order to calculate the accuracy rate in each category.
In addition to this, vocabulary or phrases that examinees were not able to produce in
English were indicated by a special “<jp>” tag. By analysing this tag, we can
determine the words and phrases that learners avoided or failed to produce in English.
As the data of written composition was all hand-written by the learners, numerous
spelling mistakes were found in written production. The spelling errors were not
considered in this paper, because a tag for misspelling was not included in the tag set
of NICT JLE Corpus. Therefore, it was crucial to make a correction to the output to
POS tagging to avoid skewing the data on accuracy rates.
2
Noun
Verb
others
CATEGORY
Inflection
Agreement
Countability
Case
Lexical errors
Inflection
Agreement
Form (only spoken data)
Tense
Aspect (only written data)
Lexical errors
Japanese words
TAG
<n_inf>
<n_agr>
<n_cnt>
<n_cs>
<n_lxc>
<v_inf>
<v_agr>
<v_fm>
<v_tns>
<v_asp>
<v_lxc>
<jp>
EXAMPLES
*childerens / *peoples / *girls’s
many *book / one *things
*a music / *an information
my *friend house
a *type→a typewriter
*sleeped / *maked
There *are a lady / a cat *sleep in her bed
to *drinks / is *sleep
He *has jogged now.
They *are knowing
She *is black and short hair.
anmitsu / origami / yukata
Table 3 Error tag set of noun and verb
As the size of spoken sub-corpus was different for different levels, the frequency of
erroneously and correctly used nouns and verbs were normalised considering the subcorpus size of each proficiency level and the total frequency of nouns and verbs. The
size of written sub-corpus was almost same, since the composition files were
randomly extracted from each academic year to equalise the corpus size; therefore,
normalisation was only conducted by the total number of nouns and verbs.
3. Results and Discussion
3.1. Error of parts of speech: noun and verb
In this section on the results of the data analysis, we tackle such questions as whether
particular groups of errors are prone to occur at certain developmental stages, and if
so, why that happened. Error categories related to part of speech, especially noun and
verb, were identified and their normalised frequencies were compared in terms of
error types and modes of production, speech and writing, together with the subjects’
proficiency levels in L2.
The following figure in the research of Abe (2004) indicates the proficiency levels of
learners and noun and verb-related errors of spoken data extracted from NICT JLE
Corpus. The learners of SST level 2 and 3 seem to have a strong correlation with
verb-related error categories “v_agr” and “v_tns”, the SST4 to 5 level learners are
connected with various types of error categories, and the SST6 to 9 level learners can
be linked with noun-related error categories such as “n_agr”, “n_cnt”, “n_lxc”, and
“n_num”. The research, which has been conducted using the same spoken data as in
this paper, shows that lower proficiency level learners have firm connection with
verb-related errors and advanced level learners with noun-related errors.
3
行ﾎﾟｲﾝﾄと列ﾎﾟｲﾝﾄ
対称的正規化
1.0
n_agr
sst6
.5
v_agr
v_tns
sst2/3
v_cmp
v_vo v_fm
v_mo
v_fin
v_inf
v_ng
n_dprp
n_gen
v_lxc
0.0
-.5
次
元
sst4
n_inf
sst5
n_cnt
sst7
n_lxc
sst8/9
-1.0
2
-1.5
-1.5
VAR00001
Proficiency level
n_num
VAR00002
-1.0
-.5
0.0
.5
1.0
次元
Figure11 The proficiency levels and errors (Abe, 2004)
The question then is: can this result of previous research be applied to written
production in this paper to describe the language acquisition stages? In the following
table, the distinctions of academic years and SST proficiency levels are not identical,
but they are simply allocated in the same column for the sake of convenience.
wr_noun
wr_verb
sp_noun
sp_verb
J1
SST2/3
5.38
32.07
0.92
12.11
J2
SST4
4.47
10.03
3.96
5.36
J3
SST5
3.59
8.52
2.66
5.12
S1
SST6
4.55
7.50
3.43
1.68
S2
SST7
3.35
5.70
2.37
2.00
S3
SST8/9
3.85
5.22
1.37
1.31
Table 4 Error rate of nouns and verbs in each corpus (%)
The above table shows that the verb-related errors in spoken data are strongly
connected with novice learners, and these errors gradually decrease as the proficiency
levels increase. This tendency of verb-related errors in spoken mode is identical with
writing production. The error rate of verb is strongly linked with the 1st grade of
junior high school (J1) learners, and it gradually decreases as the developmental
stages progress. However, when we focus on the error rate of nouns, it shows a
dissimilar tendency. The error rate of nouns in written data is relatively low
compared with that of verb-related errors, which suggests the possibility that the
acquisition of verbs is much more difficult than that of nouns.
However, the error rate of nouns in written production is almost identical from the
academic year of J1 to S3, and this also implies that the noun-related errors do not
easily vanish during the developmental stages of written English. Concerning the
spoken language, the error rate of nouns shows unpredictable changes, which are
difficult to explain. But one hypothesis can be put forward. It is that the novice
learners of English do not consciously avoid using nouns to prevent themselves from
4
making errors, but they only have a minute stock of noun and repeatedly use them.
This can be gathered from the data where the total number of nouns is similar (by per
thousand words: J1=169.39, J2=157.26, J3=157.26, S1=160.76, S2=163.17,
S3=167.15). It is essential to examine more detailed data to confirm that nounrelated errors have a strong relation with upper level learners, but we may conclude
that the verb-related errors have the same firm connection with novice learners’
written production as in their spoken data.
The next step is to examine which particular grammatical category of noun and verb
has a strong connection with spoken and written modes. The following pie charts
show the accuracy rate and error rate for each mode. Japanese words, preceding noun
of title (e.g. Mr. Ms.), and all of the proper nouns were excluded from the total
number of nouns.
W R accuracy rate (J1-J3)
SP accuracy rate (SST2-SST9)
N _IN F
2%
n_inf n_cs
2%
6%
n_cnt
32%
N _C S
0%
N _LXC
33%
N _A G R
32%
n_agr
29%
n_lxc
31%
N _C N T
33%
W R error rate (J1-J3)
n_cs
9%
n_cnt
3%
SP error rate (SST2-SST9)
n_inf
2%
N _IN F
5%
N _C S
2%
N _LXC
18%
n_lxc
17%
N _A G R
54%
n_agr
69%
N _C N T
21%
Figure 2 Accuracy and error rate of noun in different modes
5
As can be seen from the pie charts, the accuracy rate of “n_cnt” (countability), “n_lxc”
(lexical error), and “n_agr” (agreement) is similar in the two different modes
(countability: WR=32%, SP=33%, lexical error: WR=31%, SP=33%, and agreement:
WR=29%, SP=32%). However, nouns associated with “n_inf” (inflection) and “n_cs”
(case) are more frequently used in written mode than in spoken mode (inflection:
WR=6%, SP=2%, case: WR=2%, SP=0%). One reason for this is the difference in the
tasks between the two different corpora. Since the spoken test task was to describe a
simple picture so that its vocabulary size might be automatically limited to a small range,
there was not so much need to use the plural and possessive form of nouns and the
different case forms. As a result, the use of singular nouns is enormously high in spoken
data, and makes up 81.73% of the total number of nouns.
Another interesting result is that the rate of lexical error in written and spoken corpus is
surprisingly almost identical as the accuracy rate was similar in both sets of data
(WR=17%, SP=18%). Therefore, we can suppose that erroneous and accurate use of
lexical nouns cannot be an indicator of the difference in production modes. In addition,
on the one hand, inflection is used much more in the written mode than in spoken mode
(WR=6%, SP=2%) as mentioned before, but, on the other hand, it is misused more
frequently in spoken data (WR=2%, SP=5%). Similarly, another noun-related
grammatical category “n_cnt” (countability) is misused more regularly in spoken corpus
when we compare it with the written corpus (WR=3%, SP=21%). These two categories,
inflection and countability, will require a mechanical change of form for the speakers of
English, as well as knowledge of vocabulary. Accordingly, we can suppose that speakers
will be more likely to make errors in more time-pressured spoken production, and the
error rate of these grammatical categories are strongly connected with spoken production.
3.2. Morphological error: agreement
When we compared the accuracy and error rate of the two modes, the accuracy rate of
“n_agr” (agreement) was almost identical in both modes (WR=29%, SP=32%), but its
error rate in written data was higher than that of spoken data (WR=69%, SP=54%).
Since there is no agreement rule in Japanese grammar, it can be easily assumed that
agreement must be a troublesome grammar point for Japanese learners of English, as it is
indicated in the high error rate in both modes. However, the error rate of noun agreement
changes slightly in the developmental stage of written production. It is mostly misused in
J1 and its error rate mostly decreases in J3, but for the most part there is not a large
difference. For that reason, we can suppose that noun-related agreement errors cannot be
a strong indicator of the developmental sequence in written production.
Noun agreement errors may increase as modifiers, an ordinal number or a quantifier, are
added and noun phrases become complicated, but as can be seen from the following bar
graph they seem to be especially connected with SST level 6 in spoken production in
some way. Agreement error increases mostly at SST level 6 and continues decreasing by
SST level 8/9. As mentioned previously, noun-related lexical error and accuracy rate in
each mode was almost identical, but in the next bar graph of spoken data it shows an
irregular change in the developmental sequence as in agreement error. These irregular
6
sequences cannot be clearly explained, but they expose the variability in development of
learner English. We cannot observe the same sequence in written mode, so that it might
be caused by the difference of production modes, or by the lack of competency of
learners in written data, because the error rate of agreement does not decrease below the
lowest error rate of J3.
W R error (noun)
SP error (noun)
S3
SST8/9
S2
SST7
S1
SST6
J3
SST5
J2
SST4
J1
SST2/3
0%
10%
20%
30%
n_agr
40%
n_lxc
50%
60%
70%
n_cs
n_cnt
n_inf
80%
90%
100%
0%
10%
20%
30%
N _A G R
40%
N _C N T
50%
60%
N _LXC
N _IN F
70%
80%
90%
100%
N _C S
Figure 3 Noun- related errors in both production modes (%)
In the next stage, the focus will be shifted to the accuracy and error rate of verbs. First of
all, it must be said that the verb-related error tags used for written and for spoken corpora
in this research paper are slightly different, and they must be modified in further studies.
In considering errors of inflection, past and past participle form of be-verbs were
excluded, because an inflection error of be-verbs rarely occurred, even in J1, the first year
of studying English in Japanese language education.
The following pie charts show the accuracy and error rate of verbs in each different
production mode. The first thing to point out is that the accuracy rate of “v_lxc” (lexical
error), “v_tns” (tense), and “v_agr” (agreement) is similar in the two different modes
(lexical error: WR=37%, SP=33%, tense: WR=28%, SP=30%, and agreement: WR=21%,
SP=26%). In spite of this similarity, the error rate of these grammatical categories shows
dissimilarities. In spoken mode, verb agreement is much more misused than in written
production (WR=7%, SP=60%), but tense and lexical errors are more frequently
observed in written corpus data (tense: WR=65%, SP=18%, lexical error: WR=22%,
SP=9%). As a result, it is clear that a verb agreement error has a strong connection with
spoken production, and a tense and lexical error have a firm relation with written
production. Another interesting point is that both accuracy and error rate of verb-related
inflection error is fairly low in speaking mode, but its accuracy rate is relatively high in
7
writing mode. Therefore, in this study, it appear that the verb-related inflection error is
peculiar to written production.
W R accuracy rate (J1-J3)
S P accuracy rate (S S T 2-S S T 9)
v_asp
1%
v_inf
13%
v_fm
10%
V _IN F
1%
V _LX C
33%
v_lxc
37%
V _A G R
26%
v_agr
21%
v_tns
28%
V _T N S
30%
W R error rate (J1-J3)
v_inf
v_agr 3%
7%
S P error rate (S S T 2-S S T 9)
V _IN F
1%
v_asp
3%
V _LX C
9%
V _T N S
18%
v_lxc
22%
v_tns
65%
v_fm
12%
V _A G R
60%
Figure 4 Accuracy and error rate of verb in different modes (%)
As mentioned above, verb agreement error can be regarded as an effective indicator of
novice learners, because it decreases as the proficiency level increases. However, the
error rate does not decrease at SST level 6 and SST level 8/9, and interestingly this shows
the same pattern as in the noun agreement error of spoken production. As can be seen
from the following bar graph, the verb agreement error shows an irregular change at the
8
level of SST6 and SST8/9, while the error in verbal form increases and the verbal lexical
error decreases. Both noun and verb-related agreement errors in spoken data show a
complicated pattern, which is difficult to explain, but it is interesting to discover that
level 6 and level 8/9 can be some kind of turning point for errors especially related with
agreement.
3.4. Errors of tense and aspect
What can we find out when we focus on the accuracy and error rate of verbs in written
corpus data? Firstly, as it can be observed from the pie charts in the previous section,
written production is strongly connected with tense errors. Although the error rate does
not decrease drastically, except from J1 to J2, it keeps to a fairly high level throughout
the academic year. This implies that tense is a problematical grammar point for Japanese
learners of English, and it does not easily disappear in writing, even for the upper level of
learners. Subsequently, we can conclude that it cannot be an index to identify the
proficiency level of written production, but can be an indicator to distinguish the mode.
As mentioned above, tense errors seem to be peculiar to novice learners in written data.
What, then, can we discover from the spoken data? The error frequency starts to increase
at level 4 and level 6, but then decreases at higher proficiency levels. The errors of
simple present and simple past were good indicators of progress in simple tense usage,
however, a picture description task used in this study turned out to be unsuitable for
observing various complicated tense and aspect forms that might be used by learners at
much more advanced levels.
SP error (verb)
W R error (verb)
S3
SST8/9
S2
SST7
S1
SST6
J3
SST5
J2
SST4
J1
SST2/3
0%
20%
40%
v_tns
v_lxc
60%
v_agr
v_inf
80%
100%
0%
20%
40%
V _A G R
v_asp
V _TN S
60%
v_fm
V _LXC
80%
100%
V _IN F
Figure 5: Verb-related errors in both production modes (%)
Regarding aspect errors, learners in academic year of J1 tend to use a progressive form
inappropriately. In J2 level, the error rate increases, but what is more noteworthy here is
9
that same four learners in written corpus have misused aspect in their composition more
than 3 times. This phenomenon might occur because learners have difficulty in using the
same tense and appropriate aspect while they are writing a sequence of English
paragraphs. A significant educational implication that must be concluded from this fact
is that learners need to practice using tense and aspect by writing longer English
compositions.
Another thing that must be pointed out here is that an error rate of aspect suddenly drops
at the J3 level, but once again it is successively misused as the developmental stage
increases. This trend cannot be clearly explained here, but it may be strongly connected
with an introduction of perfect tense in English grammar textbook in the upper academic
year. An error of perfect aspect was especially observed in the upper grade when it is
introduced in English classes at school. Finally, learners seem to have a tendency to use
the progressive and perfect tense when there is no necessity, but they rarely use the
simple tense when they need to use the progressive and perfect tense.
4. Conclusion
Through the detailed examination on error categories, we were able to observe that some
errors have common developmental patterns and a firm correlation with certain
proficiency levels and production mode. The study of learners’ errors showed how the
errors varied according to their stages of language acquisition and the production mode.
And at the same time, there were other categories that can be described not by the pattern
of error rate, but by the accuracy rate. By analysing the errors and proficiency level of
two different production modes, this study has arrived to identify the features of
interlanguage in a more objective way, and also was able to show the value of utilising
learner corpus for SLA research.
It is necessary for future study to enlarge the size of learner corpus and to increase the
variety of tasks used to elicit the data. Also, as a statistical analysis, the multivariate
statistical analysis, called Correspondence Analysis, can be performed on this data to
identify how learner errors change as levels of proficiency increase. Such an analysis
may be able to clarify the correlation of proficiency level and the error frequency of each
production mode. Additionally, larger corpus can represent several features of
interlanguage, but still we need further consideration on error tags. The focus of this
study was agreement and tense/aspect errors, but there is obviously scope for similar
investigations that examine the relationship between proficiency level and other features
of learner language. Also, it would be useful to compare learners’ production with a
native speaker corpus to examine the difference of modes and the points that learners are
avoiding on the process of language acquisition.
References
Abe, M. (2004) A Corpus-based Analysis of Interlanguage: Errors and English
proficiency Level of Japanese Learners of English. Handbook of an International
Symposium on Learner Corpora in Asia (ISLCA), 28-32.
Bardovi-Harlig, K. and Bofman, T. (1989) Attainment of syntactic and morphological
10
accuracy by advanced language learners. Studies in Second Language Acquisition 11,
17-34.
Biber, D., Conrad, S. and Reppen, R. (1998) Corpus Linguistics: Investigating language
Structure and Use (Cambridge: Cambridge University Press).
Chamot, A. (1978) Grammatical problems in learning English as a third language, in E.
Hatch (ed.) Second Language Acquisition (Rowley, Mass.: Newbury House), 175-189.
Chamot, A. (1979) Strategies in the acquisition of English structures by a child bilingual
in Spanish and French, in R. Andersen (ed.) The acquisition and Use of Spanish and
English as First and Second Languages (Washington D.C.: TESOL), 90-106.
Corder, S. P. (1967) The significance of learners’ errors. International Review of Applied
Linguistics 5, 161-169.
Dagneaux, E., Denness, S. and Granger, S. (1998) Computer-aided Error Analysis.
System An International Journal of Educational Technology and Applied Linguistics
26 (2), 163-174.
Ellis, R. (1994) The Study of Second Language Acquisition (Oxford: Oxford University
Press).
Ellis, R. and Barkhuizen, G. (2005) Analysing Learner Language (Oxford: Oxford
University Press).
Garside, R., Leech, G and McEnery, T (eds.) (1997) Corpus Annotation: Linguistic
Information from Computer Text Corpora (Harlow: Longman).
Granger, S. (1999) Use of Tenses by Advanced EFL learners: Evidence form an Errortagged Computer Corpus, in Hasselgard, H and Oksefjell, S (eds.) Out of Corpora:
Studies in Honour of Stig Johansson (Amsterdam: Rodopi), 191-202.
Granger, S. and Rayson, P. (1998) Automatic profiling of learner texts, in Granger, S
(ed.) Learner English on computer (London: Longman), 191-202.
Isahara, H., Saiga, T and Izumi, E. (2002) The TAO Speech Corpus of Japanese Learner
English Error Tagging Manual Ver.1.0.
Izumi, E., Uchimoto, K. and Isahara, H. (2004) A Speaking Corpus of 1200 Japanese
Learners of English (Tokyo: ALC Press Inc.)
Housen, A. (2002) A corpus-based study of the L2-acquisition of the English verb system,
in Granger, S., Hung, J. and Petch-Tyson, S (eds.) Computer Learner Corpora, Second
Language Acquisition and Foreign Language Teaching (Amsterdam, Benjamins), 77116.
James, C. (1998) Errors in Language Learning and Use: Exploring Error Analysis
(Harlow: Longman).
Kathleen Bardovi-harlig. (2000) Tense and Aspect in Second Language Acquisition:
Form, Meaning, and Use (Malden: Blackwell).
Lennon, P. (1991) Error: Some problem of definition, identification and distinction.
Applied Linguistics 12, 180-95.
Tabata, T. (2002) Investigating Stylistic Variation in Dickens through Correspondence
Analysis of Word-Class Distribution, in Saito, T., Nakamura, J and Yamazaki, S (eds.)
English Corpus Linguistics in Japan (Amsterdam: Rodopi), 165-182.
Tono, Y. (1999) A Corpus-based analysis of interlanguage development: Analysing partof-speech tag sequences of EFL learner corpora, in Lewandowska-Tomaszczyk, B. and
Melia, P.J (eds.) PALC’99: Practical Applications in Language Corpora (Frankfurt:
Peterlang), 323-340.
11