PTLC2005 Taniguchi & Tara Relation between direct tests and indirect tests on English intonation for
Japanese learners: nucleus placement:1
Relation between direct tests and indirect tests on
English intonation for Japanese learners:
nucleus placement
Masaki Taniguchi, University of Kochi
Shizuya Tara, University of Kochi
1 Introduction
There have been pros and cons on the use of indirect tests to measure the learner’s
performance in English pronunciation. Lado (1961) did not consider that an indirect test
could substitute for a direct test but considered it may be effective when the latter was
not practical. However, Buck (1989) denied the effectiveness of using the former as a
substitute for the latter, although he admitted its practicality. In Japan, Shirahata (1991)
conducted an experiment on lexical stresses, and affirmed the effectiveness of indirect
tests. Inoi (1995) denied it. Yamauchi (1992) and Komazawa and Ito (1997) conducted
similar experiments on rhythmic stresses, and found a medium degree of correlation
between direct and indirect tests.
This paper examines (1) whether or not there is credibility in the use of indirect tests on
English intonation, especially on nucleus placement, taken by Japanese EFL learners, (2)
whether or not there is any coherent correlation between the two types of tests on
nucleus placement, and (3) whether or not the subject is aware of the correct nucleus
placement (in indirect tests) even when they fail to achieve it (in direct tests).
2 Experiments
2.1 Research Question
The research question was “Can indirect tests test the subject’s performance of nucleus
placement?”
2.2 Types of Tests and Subjects
Two experiments were conducted. In Experiment One, three kinds of tests were
conducted with 111 Japanese university EFL learners: (a) a general English ability test
(written), containing four reading questions, four vocabulary, four grammar, and four
listening comprehension (Shimizu et al, 2004: 82-85), (b) a direct test using a dialogue
written by Masaki Taniguchi with the help of Professor John Wells (Hashimoto &
Taniguchi, 2003: 108) (Dialogue A), and (c) an indirect test using Dialogue A and a
dialogue used in the past Japanese SAT (Dialogue B). Tests (a) and (b) were conducted
a week before Test (c). The direct test preceded the indirect test. The direct test tested
the subjects’ performance, i.e., where they actually placed nuclei, whereas the indirect
test tested their awareness on nucleus placement, i.e., where they thought they should
place nuclei.
In Experiment Two, two tests were conducted with 174 Japanese university EFL
learners (different subjects from those used in Experiment One): (a) an indirect test
using Dialogue A which was used in Experiment One, followed by (b) a direct test using
the same dialogue. The order of the direct and indirect tests was reversed from
Experiment One, in order to examine whether there is any influence of the first test on
the second. In (a), the subjects were asked to mark the words that they thought should
PTLC2005 Taniguchi & Tara Relation between direct tests and indirect tests on English intonation for
Japanese learners: nucleus placement:2
be emphasized, and in (b) they were asked to read all the lines aloud, and their reading
was recorded on tape.
3 Results
The test results were transformed into dichotomous data, using SPSS for Windows 11.5.
The subjects’ performances in the direct tests were analyzed acoustically with Speech
Filing System as well as auditorily. Three major weak points in Japanese learners’
English intonation were discussed in Taniguchi (2001, 2002), and they were used as a
basis for measurement of the subjects’ performance in the present study.
3.1 Results of Experiment One
The results of Experiment One were as shown in Table 1.
(a) (general
(b) (direct test on
(c) (indirect test
English ability
nucleus
on nucleus
test)
placement)
placement)
18
10
10
11.69
0.80
4.28
2.2
1.37
1.34
17
9
8
6
0
1
0.43
0.78
0.23
Table 1. Results of Experiment One
Full mark
Average
SD
Highest
Lowest
Cronbach's α
As is clear from Cronbach’s α coefficients above, the internal consistency reliability was
highest in the direct test, (b), followed by the general English ability test, (a), and it was
lowest in the indirect test, (c). This implies that the reliability of the direct test is
guaranteed, but that of the indirect test is not. Table 2 shows their correlations.
(a)
(b)
(a)
1
(b)
0.088
1
(c)
0.183
0.150
(c)
1
Table 2. Correlations between (a), (b) and (c)
The correlation between the direct and indirect tests was r = 0.150 (p > 0.05), which
indicates that there was no correlation between them. No correlation was found between
either of them and general English ability, either. The concurrent validity of the indirect
test was low. The results, therefore, were that [1] there was little correlation between the
results of the direct and indirect tests, which implies that an indirect test cannot be a
predictor of measuring the learner’s actual ability of nucleus placement, and [2] the
learner’s general English ability cannot be a predictor of her/his ability in nucleus
placement either in direct or indirect tests.
These results show that the learner’s ability in performing nucleus placement must be
tested directly.
PTLC2005 Taniguchi & Tara Relation between direct tests and indirect tests on English intonation for
Japanese learners: nucleus placement:3
3.2 Results of Experiment Two
The results of Experiment Two were as shown in Table 3.
Full mark
Average
SD
Highest
Lowest
Cronbach's α
(a) (indirect test on
(b) (direct test on
nucleus placement)
nucleus placement)
35
35
9.25
16.40
3.56
2.38
19
24
2
11
0.62
0.25
Table 3. Results of Experiment Two
The correlation coefficient between (a) and (b) above was r = 0.109 (p > 0.05), which
indicates that there is no correlation between them. It means that an indirect test cannot
be a predictor of a direct test, as was indicated by the results of Experiment One earlier.
There is one thing, however, different from the results of Experiment One, that is, that on
the average, the subjects scored better in the direct test than in the indirect test, unlike in
Experiment One. In both experiments, the test conducted later produced higher scores,
which may imply that the subjects learnt from the first test to do better in the second.
One reason why their score in the indirect test was lower than in their direct test may be
because they chose more than one word in some intonation groups. They were not
instructed to mark only one word in each intonation group. Neither was the dialogue
divided into intonation groups for them. They were just instructed to “mark the words that
they thought should be emphasized.” For example, in “Shall we meet at three thirty?”,
many subjects chose “three thirty” as a set as the place for emphasis in the indirect test,
but they ended up placing a nucleus only on the first syllable of “thirty”. The same was
true with “five o’clock”, “four o’clock”, “Gold Medals”, and “eighty-nine”.
The results reaffirmed the Japanese learners’ weak points discussed in Taniguchi (2001,
2002): (a) use of monotonous pitch, causing difficulty in perceiving nucleus location, (b)
lack of contrast between old and new information, ending up in highlighting the last word
in each intonation group in actual performance, and (c) inability of using an appropriate
tone on the syllable where they actually seem to aim at doing so.
Use of both direct and indirect tests in this study especially reaffirmed (c) above. In “How
about four thirty?” in response to “Shall we meet at three thirty?”, 158 subjects answered
correctly in the indirect test, but only 42 performed correctly. It means they were aware
of the contrast between old and new information and knew which word was to be
highlighted, but they did not perform as they meant to. They instead tended to use a high
level pitch on the word to be highlighted, and then a moving pitch on the following word,
which was perceived as a nucleus. Table 4 shows some such examples.
indirect test
direct test
How
7
5
about
1
0
four
158
42
thirty
79
136
PTLC2005 Taniguchi & Tara Relation between direct tests and indirect tests on English intonation for
Japanese learners: nucleus placement:4
It
is
a
early
little
1
2
18
40
115
0
6
1
153
38
He’s
an
swimmer
excellent
indirect test
1
1
2
169
direct test
0
0
152
78
Table 4. Some results of indirect and direct tests showing the subjects’ awareness of
correct nucleus placement but failure in actual performance.
(Note: “How about four thirty?” in response to “Shall we meet at three thirty?”; “It is a little
early” in response to “Is three thirty too early?”; and “He’s an excellent swimmer” in
response to “He’s a very good swimmer”)
indirect test
direct test
4 Conclusions
The findings are [1] that there was little correlation between the direct and indirect tests,
which implies that an indirect test cannot be a predictor of measuring the learner’s actual
ability of nucleus placement, [2] that the learner’s general English ability cannot be a
predictor of her/his ability in nucleus placement either in direct or indirect tests, and [3]
that there was a gap between the subjects’ awareness and their actual performance, i.e.,
even when the subjects were aware of the correct nucleus placement, as shown in the
indirect test, if it happened to be on a word other than the last word in the intonation
group, they tended to place a nucleus on the last word instead of the one they aimed at.
This research made us realize the teachers’ responsibility to discriminate auditorily
between correct and incorrect performances of nucleus placement by their students. It is
indispensable for teachers to get phonetic training before teaching students.
5 References
Buck, Gary (1989) Written tests of pronunciation: do they work? ELT Journal, no. 43, pp.
50-56.
Komazawa, Seiji & Ito, Akihiro (1997) A written test of stress in connected speech in the
NCUEE-Test: Its reliability & validity. Chuubu Chiku Eigo Kyoiku Gakkai Kiyo, vol. 27,
pp. 285-292.
Hashimoto, Mitsuhiro & Taniguchi, Masaki (2003) English Sounds and Oral
Communication: Approach to Living English. English Phonetic Society of Japan.
Lado, Robert (1961) Language Testing. London: Longman.
Inoi, Shinichi (1995) The validity of written pronunciation questions: Focus on phoneme
discrimination. Language Testing in Japan. Tokyo: JALT.
Shimizu, Yoshikazu et al (2004) Exploring Cultural Issues: Practice in TOEIC Test
Format. Tokyo: Seibido.
Shirahata, Tomohiko (1991) Validity of paper test problems on stress: Taking examples
from Mombusho’s daigaku nyushi senta shiken. Shizuoka Daigaku Kyoiku Gakubu
Kiyo, no. 23, pp. 161-172.
Taniguchi, Masaki (2001) Japanese EFL learners’ weak points in English intonation.
English Phonetics (English Phonetic Society of Japan (EPSJ)), no. 4, pp. 45-54.
Taniguchi, Masaki (2002) An experiment on Japanese EFL learners’ English intonation:
tonality, tonicity and tone. English Phonetics (EPSJ), no. 5, pp. 91-110.
Yamauchi, Susumu (1992) Validity of measuring oral production ability on pencil-andpaper tests. Kyushu Eigo Kyoikugaku Kenkyu Kiyo, vol. 20, pp. 72-82.