The analysis and
standardization
Jirada Wudthayagorn, Ph.D.
Skills to be measured
•
•
•
•
•
Listening
Listening-Writing
Speaking
Listening-Speaking
Reading-Speaking
Skills to be measured
• Reading
• Reading-Writing
• Writing
Listening
• 50 questions (4 multiple choices)
• Listen to statements, short
dialogues, short stories, and lectures
• Choose appropriate answers
• A machine scorer
Listening-Writing
•
•
1.
An integrated section
Two tasks
Listen to 3 passages and fill in each blank
space with words or phrases.
2. Listen to 1 passage and summarize what
they have heard based on headings given
in the answer sheet.
• Two human raters
Speaking
• Listen to short questions and respond
to them (a sentence level)
Listening-Speaking
• Listen to a prompt and them respond
to it (a paragraph level).
Reading-Speaking
• Read a text and them speak about it,
for example, summarizing the text or
making suggestions.
• All about speaking is rated by the
examiners (human raters).
Reading
• Read four reading passages (130-150
words).
• For each passage, five questions are
asked, four multiple choices are
offered.
Reading-Writing
• Read the given information and write
an essay using the information
provided.
Writing
• Write an essay based on the given
prompt.
• All about writing is rated by the
examiners (human raters).
Test Administration
• The test was divided into two booklets.
• Booklet 1 covers all activities (listening and
speaking) that require the facilities in a
laboratory (tape player, headphones, and a
microphone) to complete the test tasks
Test Administration
• Booklet 2 covers reading and writing
tasks.
• 20 minutes break between the two.
• Take 2 hours and 30 minutes to
complete the test.
Who took the test?
• 98 examinees.
• Gender: 27% are male, 73% female.
• Education: 63% bachelor’s degree,
24% Master’s degree, 2% high
school, the rest (11%) did not give
the information.
Who took the test?
• Occupations: 27% government
sector, 16% freelances, 15% are
teachers, lecturers, and educators.
Descriptive Statistics
Task
Raw score
Min
Max
Mean
SD
Total
160
12.10
88.90
46.23
17.54
Reading
20
3
19
12.17
3.73
Reading-writing
20
0
9
4.25
2.03
Writing
20
0
9
3.76
2.00
Listening
20
.80
17.20
9.34
3.03
Listening-Writing
20
0
9
7.42
2.39
Speaking
20
0
9.5
4.09
2.49
Reading-speaking
20
0
9.5
4.45
2.64
Listening-speaking
20
0
9.5
4.43
2.77
What does it mean?
• Reading
Able to get most specific and key
information of moderate complex
texts.
Able to differentiate between major
and minor points (ideas).
Have some ability to infer meanings.
Listening
• Able to comprehend some short
utterances.
Speaking
• No functional ability
• Difficult to understand
Listening-Speaking,
Reading-Speaking,
• No functional ability
• Difficult to understand
Reading-Writing, Writing,
Listening-Writing
• The essays were underdeveloped.
• They contained severe and persistent
writing errors.
• They contained inappropriate or
insufficient details, inappropriate
word choices, and many errors in
sentence structures.
Difficulty index and
discrimination power
• Listening (multiple choices)
Difficulty index: mean = 0.46
Discrimination power: mean = 0.37
• Reading (multiple choices)
Difficulty index: mean = 0.61
Discrimination power: mean = 0.46
More….(to enjoy)
Task
Difficulty index
Discrimination
power
Reading-Writing
0.208
0.265
Writing
0.191
0.276
Listening-Writing
0.129
0.249
Speaking
0.204
0.313
ReadingSpeaking
0.212
0.377
ListeningSpeaking
0.204
0.388
Reliability
Reliability
Reliability
coefficient
Standard Error
of Measurement (SEM)
Listening 0.836
Reading 0.763
Listening 3.091
Reading 1.808
Cronbach Alpha
Listening 0.836
Reading 0.763
Listening 3.091
Reading 1.808
Split-Half
Listening 0.830
Reading 0.79
Listening 3.150
Reading 1.702
Kuder-Richardson 20
(KR-20)
Validity
• 63 examinees took TOEIC.
Descriptive Statistics of
the TOEIC test
Task
Min
Max
Mean
SD
Listening
100
495
292.14
91.218
Reading
55
455
241.59
87.560
Total score
200
950
533.73
169.637
Correlation Coefficients
• ELDC*TOEIC (total)
• ELDC*TOEIC (listening)
• ELDC*TOEIC (Reading)
0.850*
0.681*
0.698*
A linear regression
• TOEIC = 126.19 + 7.36 ELDC + e
For example, if you get 100 points of
the ELDC test, we predict that you
are likely to obtain the TOEIC score
of 862.
Summary
• With all the information we have, we
are able to go back and revise the
test.
• We also have a specific test (e.g.,
English for tourism, English for
automobile industry). And they are on
the pilot process.