Chi-Fen Emily Chen (Spring 1999)
Analysis of the Test of Spoken English
Outline
I. Introduction:
II.
TSE: oral proficiency language test for nonnative speakers of English, administered
by Educational Testing Service (ETS) through the TOEFL program
Purpose: to measure the ability of nonnative speakers of English to communicate
orally in English.
Use: to select international teaching assistants, and to certify all other individuals’ oral
English proficiency in a professional field
Theoretical construct: communicative language ability
The TSE test methods:
-
Testing environment: a semi-direct audiotaped test, typically administered in language
laboratories with a microphone and a tape-recorder.
-
Test rubric:
(a) TSE consists of 12 questions, each of which requires examinees to perform a
particular language function, such as narrating, persuading, and giving an opinion.
(b) The instruction is given by the interviewer on the test tape and printed in the test
book as well.
(c) The time allotted for each response ranges from 30 to 90 seconds, and some
questions allow the planning time for the response from 30 to 60 seconds. The total
testing time is approximately 20 minutes.
(d) Each TSE answer tape is rated independently by two trained TSE raters and they
assign a score level using descriptors of communicative effectiveness related to
language function, appropriateness of response to audience/situation, coherence and
use of cohesive devices, and linguistic accuracy (see Appendix II). The score scale is
from 20 to 60, divided into 5 levels. The scores of the two raters are averaged and
reported in 5-point increments (i.e. 20, 25, 30, 35, 40, 45, 50, 55, 60).
-
Test input:
(a) The test questions are written in the test book and recorded at normal speed on the
test tape too. Non-language visual stimuli, such as maps, pictures, graphs, and charts,
are used as part of input. The questions specify clearly the audience, the social
context, the role the examinee has to play, and the language function the examinee is
going to perform in each task.
1
Chi-Fen Emily Chen (Spring 1999)
(b) The test questions are of general nature and the language used is context-embedded
everyday English, without any specialized technical vocabulary.
-
Expected response:
(a) All of the responses are “constructed response,” which have to be recorded on the
answer tape. They are relatively short passages of speech (from 30 to 90 seconds).
(b) The responses are expected to demonstrate how the examinee gives appropriate oral
response to different audience and situations while performing various language
functions.
(c) Restrictions: time constraint, restrictions on organizational, propositional, and
illocutionary characteristics, psychological problems from the use of audiotaped
response, and lack of nonverbal communication in the response.
-
Relationship between input and response: It is a nonreciprocal input and response,
which precludes live interaction between the interviewer and the examinee. The
questions are pre-recorded and the examinee is left to respond.
III. The test content of the TSE Practice Questions: (see Appendix I)
IV. Construct validity of the TSE test:
- The test content matches four competencies (functional, sociolinguistic, discourse, and
linguistic competence).
- The test methods affect the examinee’s performance, especially in terms of discourse
competence and strategic competence.
V. Conclusion:
- advantages
- disadvantage
- suggestions
2
Chi-Fen Emily Chen (Spring 1999)
Paper
I. Introduction
Probably the most well-known oral English proficiency test for nonnative speakers of
English is the Test of Spoken English (TSE). It is a member of the Test of English as a
Foreign Language (TOEFL) family of tests developed by Educational Testing Service (ETS).
The TSE test, a standardized instrument, is designed to evaluate the speaking ability of
nonnative speakers of English, at or beyond the post-secondary level, to communicate orally
in English. It does not measure listening, reading, or writing skills in English. The examinee’s
performance on the TSE test is considered to indicate his/her oral language ability in English
to communicate in academic or professional environments. The TSE scores are used by many
North American institutions of higher education to select international teaching assistants, and
to certify all other individuals’ oral English proficiency in a professional field, such as health
professionals (physicians, nurses, pharmacists, and veterinarians). The scores are not regarded
as predictors of academic or professional success, but only as indicators of nonnative
speakers’ oral communicative language ability in English (ETS 1999b).
The TSE test was revised in 1992 in an effort to make it more ‘communicative” in both
its test content and rating scale criteria, focusing on overall communicative language ability
rather than just on linguistic accuracy (i.e. accuracy in terms of pronunciation, grammar, and
vocabulary). The revision project was completed in 1995, which was developed “on the
premise that language is a dynamic vehicle for communication, driven by underlying
competencies, which interact in various ways for effective communication to take place”
(Chen & Pooler 1999). Therefore, the theoretical construct underlying the revised TSE test is
communicative language ability, which, for the purposes of the test, includes strategic
competence and four language competencies: functional competence, sociolinguistic
competence, discourse competence, and linguistic competence (Chen & Pooler 1999). It
corresponds to Bachman’s (1990) model of communicative language ability quite well.
3
Chi-Fen Emily Chen (Spring 1999)
In this paper, I would like to evaluate the TSE test’s construct validity from two aspects:
test methods and test content. Specifically, I will analyze the test methods that TSE uses to
measure oral language ability based on Bachman’s (1990) framework of test method facets.
Then, I will discuss the content of the TSE practice questions (ETS 1999a; see Appendix 1)
and evaluate them to see whether the TSE test content matches its claimed theory,
communicative language ability.
II.
The TSE test methods
According to Bachman’s (1990) framework of test method facets, there are five
categories: (1) testing environment, (2) test rubric, (3) nature of the input the test taker
receives, (4) nature of the expected response to the input, and (5) relationship between input
and response. I will use his model to analyze the test methods that TSE uses and discuss how
the test methods affect the examinee’s oral language performance.
(1) Testing environment:
The TSE test is a semi-direct test delivered via audio-recording equipment and a test
book, and it is typically administered in language laboratories. An interviewer on the test tape
leads the examinee through the test; the examinee responds into a microphone, and responses
are recorded on a separate answer tape. The examinee, therefore, will be tested in a tiny
cubicle and has to interact with a tape-recorder and a microphone, not with a person. This
may cause some affective problems for test takers if they are not familiar with the equipment,
and then may have negative influence on their language performance in the test tasks. The
TSE test designers have taken this factor into consideration. In order to help the examinee
become more accustomed to speaking on tape and to allow for adjustment of the audio
equipment as needed, they provide a “warm-up” task before the test begins: the interviewer
on the test tape asks some simple questions about the examinee and then the examinee can
practice speaking on tape and his response in this part will not be scored.
4
Chi-Fen Emily Chen (Spring 1999)
(2) Test rubric: It includes test organization, instruction, time allocation, and scoring.
The TSE test is composed of twelve questions, each of which requires the examinee to
perform a particular language function such as narrating, persuading, and giving and
supporting an opinion (ETS 1999a). There is no difference in the degree of difficulty or
importance for each question in the TSE test, but only a difference in the functions that the
examinee has to perform.
The instruction given by the interviewer, including the testing procedures, the tasks to be
completed, and the allotted response time and planning time, is printed in the test book and
recorded on the tape too. That is, all the instruction of the TSE test is presented both aurally
and visually in English. The advantage is that the examinee can either listen or read the
instruction of the test and this way can ensure the examinee to understand exactly what he is
going to do in each test task.
The time allotted for each response ranges from 30 to 90 seconds, and some questions
give the planning time for the response from 30 to 60 seconds (The total testing time is
approximately 20 minutes). The examinee actually responds to each question under big
pressure of time constraint. This may also have a negative influence on the examinee’s oral
performance, for he has to always think quickly and make good use of the short period of time
to express himself.
As for scoring, each TSE answer tape is rated independently by two trained TSE raters;
neither knows the scores assigned by the other. Each rater evaluates the response and assigns
a score level using descriptors of communicative effectiveness that are delineated in the TSE
rating scale (ETS 1999b; see Appendix 2). As of July 1995, the TSE score consists of a single
score of communicative language ability, which is reported on the following scale:
60
50
40
30
20
Communication almost always effective: task performed very competently
Communication generally effective: task performed competently
Communication somewhat effective: task performed somewhat competently
Communication generally not effective: task generally performed poorly
No effective communication: no evidence of ability to perform task
5
Chi-Fen Emily Chen (Spring 1999)
The scores of the two raters are averaged and are reported in 5-point increments. If the two
ratings are not shown adequate agreement, the tape is rated by a third independent rater. Final
scores are based on a resolution of the differences among the three scores (ETS 1999b).
With regard to the rating scale criteria of the TSE test, they in fact include four categories:
ability to perform the language function clearly and effectively (functional competence),
appropriateness of the response to the designated audience or situation (sociolinguistic
competence), coherence of the response and use of cohesive devices (discourse competence),
and accuracy in the use of linguistic features (linguistic competence). However, since the
score is given holistically based on the examinee’s overall communicative oral language
performance, no single area of competence, such as linguistic competence, is
over-emphasized. The raters react to the examinee’s speech as a whole: they do not think
about the individual components of the speaking ability or count the number of errors that the
examinee makes. For example, the examinee can get the score of 60 if he communicates
effectively regardless of his minor errors. This scoring method encourages the examinee to
focus on the overall effectiveness of his communication rather than merely on the accuracy of
his language use. It matches the theory of communicative language ability underlying the test.
(3) Test input: It involves the format and nature of language that is used in test input.
The twelve questions in the TSE test are of general nature and designed to inform the
raters about the examinee’s oral communicative language ability in English. All of the
questions, like the instruction, are written in English in the test book and recorded at normal
speed on the test tape as well. Non-language visual stimuli, such as maps, a series of pictures,
graphs, and charts, are used as part of input to elicit the examinee’s speech. The questions
specify clearly the audience, the social context, the role the examinee has to play, and the
language function the examinee is going to perform in each task. This information is very
useful for the examinee to decide how to use his language appropriately in different situations,
which helps to elicit the examinee’s oral performance in terms of pragmatic competence.
6
Chi-Fen Emily Chen (Spring 1999)
Since TSE is a test of general oral language ability, the language used in the questions is
everyday language, without any specialized technical terms or culture-specific vocabulary. In
addition, the language of the test input is context-embedded, for the topics of the questions are
generally familiar or known to all examinees, regardless of native language, type of
educational training, or field of employment (ETS 1999b). Therefore, the examinee’s oral
performance will not be affected due to unfamiliar vocabulary or topics in the input material.
(4) Expected response: It consists of the format, nature of language, and restrictions on
the expected response to the input material in a test.
The examinee is required to record his oral responses in English to the TSE questions on
the answer tape. All of the expected responses are relatively short passages of speech (from 30
to 90 seconds), but not single sentences. Such responses are the productive use of language
responding to the input material, called “constructed response” (Popham 1978, as cited in
Bachman 1990). This type of response requires the examinee to organize his speech
coherently in a discourse level rather than just producing some individual sentences. The
examinee’s discourse competence is expected to be demonstrated in this type of response.
The language required from the examinee is commonly used vocabulary in everyday
communication, but since the TSE test is mainly used to select international teaching
assistants or to certify health professionals, the response may involve the examinee’s
specialized vocabulary in his field. However, the most important part in the expected response
is how the examinee gives appropriate response to different audience and situations in his
speech while performing various language functions. The examinee’s pragmatic competence,
including functional and sociolinguistic competence, is expected to show from his response.
There are some restrictions on the expected response in the TSE test. The most obvious
one is that the response is constrained by time. With only 30 to 90 seconds for the response to
each question, it actually imposes restrictions on organizational, propositional, and
illocutionary characteristics in the examinee’s oral performance. It will be a big challenge for
7
Chi-Fen Emily Chen (Spring 1999)
the examinee to take care of every detail in his response within such a short period of time. In
addition, the response recorded on the tape may restrict the examinee’s ability to perform
normally as he does in real-life situations if he is afraid of speaking into a microphone. Also,
the nonverbal communication such as gestures and facial expressions, which is actually
indispensable part in interpersonal communication, cannot be seen from this type of response.
(5) Relationship between input and response:
The semi-direct audiotaped format of the TSE test results in a nonreciprocal input and
response which precludes live interaction between the interviewer and the examinee. The
questions are pre-recorded and the examinee is left to respond; there is no feedback, and the
language used does not affect subsequent language use. Since communication is a two-way
system, using reciprocal input and response for the oral proficiency test should be more
appropriate to measure an individual’s oral communicative language ability, especially if we
want to find how he uses strategic competence, which can only be measured in the interactive
communication.
III. The test content of the TSE Practice Questions
In the TSE Practice Questions (ETS 1999a; see Appendix 1), questions 1 to 3 require the
examinee to study a map first and then ask him to give directions, state the reasons why he
recommends a place, and why he likes a particular movie. In questions 4 to 7, the examinee is
asked to look at a sequence of pictures, and then he has to tell the story that the pictures show
and give responses to the questions related to the pictures, including giving an opinion,
persuading people to do something for him, and stating advantages and disadvantages. In
questions 8 and 9, he is required to give his opinion on a controversial issue and to define a
term in his field of study. Then in questions 10 and 11, the examinee is asked to describe and
predict information presented in a simple graph. Finally, in question 12, he is asked to present
information from a revised schedule and indicate the revisions, as if he were talking to a
group of people.
8
Chi-Fen Emily Chen (Spring 1999)
IV. Construct validity of the TSE test
We can find that the TSE test methods and the test questions are designed to elicit the
examinee’s oral language performance in carrying out a variety of communicative functions
in different situations. The theoretical construct underlying the TSE test is communicative
language ability (Chen & Pooler 1999). In view of the TSE test content, two most obvious
competencies required from the examinee to perform various communicative functions are
linguistic competence (including the accurate use of vocabulary, pronunciation, and grammar)
and functional competence (including the ability to select appropriate language functions to
effectively address the language tasks). In addition, sociolinguistic competence is required as
well because the examinee needs to be aware of different audience and different social context
in the test tasks and then select appropriate language forms in his response.
As for discourse competence, it also plays a crucial role in the TSE test because the test
is often used for the selection of international teaching assistants and discourse competence is
essential in the instructional language. The examinee is asked to give 30 to 90 second short
passages of speech for each question in TSE. When the examinee does picture narration,
schedule presentation, and opinion or reason stating, the ability to use cohesive devices to link
information and to organize information in a coherent manner in oral discourse is truly
necessary. However, I still want to emphasize the effect of the restricted time for each
response and limited planning time before the response on the examinee’s oral discourse
performance. I doubt whether the examinee’s speech during such a short time can be fairly
measured from the perspective of discourse competence, or I should ask whether such a short
speech performance can actually reflect the examinee’s discourse competence.
The last competence, strategic competence, which Bachman (1990) regards as an entirely
separate element of communicative language ability, is not easy to be measured from this
semi-direct audiotaped format of the TSE test. Due to some problems of the TSE test methods,
including its absent interlocutor, short fixed timing, and pre-set prompts for speaking, the test
9
Chi-Fen Emily Chen (Spring 1999)
tasks provide no opportunity for interaction between the interviewer and the examinee.
Therefore, some important strategies in interpersonal communication such as requests for
clarification, negotiation for meaning, and repairs of communication breakdown will not
occur in the test. These strategies will be impossible to be measured from this type of the oral
proficiency test. Probably that is why no description about strategic competence is used in the
TSE rating scale criteria.
V.
Conclusion
In general, the TSE test is appropriate to be used to measure nonnative speakers’ oral
communicative language ability in English. Each question in the TSE test requires the
examinee to perform a language function in a particular situation. Besides the linguistic and
functional competence he needs, in order to carry out those functions appropriately, the
examinee also has to pay attention to different audience and social contexts provided in the
questions, which involves his sociolinguistic competence as well. In addition, the discourse
competence is definitely required in the test, for the expected response needs to be a short
passage rather than individual sentences. In terms of the TSE test content and the scoring
criteria used to measure the above four competencies holistically, I can say it matches the
theory of communicative language ability (except for strategic competence) behind it.
However, I do question some of the test methods that TSE uses according to the
communicative approach. This semi-direct audiotaped format causes nonreciprocal test
setting which makes live interaction impossible between the interviewer and the examinee,
but, in fact, real-life communication can hardly happen without interaction. I understand there
are some advantages of using an audiotaped test, such as it can test many people at the same
time, it does not need many interviewers to give the test, and the test scoring does not have to
be done in the same place and at the same time as the test itself (Underhill 1987: 35). Even so,
in a possible way, I will still suggest to use a face-to-face interview for oral proficiency test in
order to elicit the examinee’s “real” oral communicative ability. By talking to a real person,
10
Chi-Fen Emily Chen (Spring 1999)
the examinee can have authentic interaction with the interviewer and it will be easy for him to
perform more naturally in oral communication. Another test method I do not feel comfortable
is the allotted time for the response, though I know the use of time allotment is one way to
establish the reliability of the test. I suggest that the time for each response be longer and
more flexible, for example, by giving a time band (from 30 to 60 seconds) instead of a strict
time limit. In this way, the interaction between interlocutors is possible and the examinee’s
strategic competence can be demonstrated from the interactive communication. Therefore, the
examinee’s oral performance can be expected to more closely reflect his oral language ability.
References
Bachman, L. F. (1990). Fundamental considerations in language testing. New York: Oxford
University Press.
Chen, N. N. & Pooler, E. (1999). Testing oral proficiency: TSE and SPEAK. American
Language Review, 3, 2, 20-21.
Educational Testing Service. (1999a). Information bulletin for the Test of Spoken English.
Princeton, NJ: ETS.
Educational Testing Service. (1999b). TSE for educators. Princeton, NJ: ETS.
http://www.toefl.org/edabttse.html
Underhill, N. (1987). Testing spoken language: A handbook of oral testing techniques.
Cambridge, UK: Cambridge University Press.
11
Chi-Fen Emily Chen (Spring 1999)
12
Chi-Fen Emily Chen (Spring 1999)
13
Chi-Fen Emily Chen (Spring 1999)
Appendix 2
TSE RATING SCALE
60
Communication almost always effective: task performed very competently




Functions performed clearly and effectively
Appropriate response to audience/situation
Coherent, with effective use of cohesive devices
Use of linguistic features almost always effective; communication not affected
by minor errors
50 Communication generally effective: task performed competently
 Functions generally performed clearly and effectively
 Generally appropriate response to audience/situation
 Coherent, with some effective use of cohesive devices
 Use of linguistic features generally effective; communication generally not
affected by errors
40 Communication somewhat effective: task performed somewhat competently
 Functions performed somewhat clearly and effectively
 Somewhat appropriate response to audience/situation
 Somewhat coherent, with some use of cohesive devices
 Use of linguistic features somewhat effective; communication sometimes
affected by errors
30 Communication generally not effective: task performed poorly
 Functions generally performed unclearly and ineffectively
 Generally inappropriate response to audience/situation
 Generally incoherent, with little use of cohesive devices
 Use of linguistic features generally poor; communication often impeded by
major errors
20 No effective communication: no evidence of ability to perform task
 No evidence that functions were performed
 No evidence of ability to respond appropriately to audience/situation
 Incoherent, with no use of cohesive devices
 Use of linguistic features poor; communication ineffective due to major errors
14