DEVELOPMENT OF AN ONLINE
READING TEST
Associate Professor Dr. Alisa Vanijdee
Dean, School of Liberal Arts
Sukhothai Thammathirat Open University, Thailand
Alisa.Van@stou.ac.th;avanijdee@hotmail.com
OBJECTIVES
(1) develop an online English reading test;
(2) validate the quality of the test;
(3) study the reading performance-level of the
test-takers;
(4) analyse common weaknesses found in the
test results, and
(5) propose guidelines to improve the reading
performance of test-takers.
INFORMANTS
31 academics and graduate students
RESEARCH INSTRUMENTS
(1) 3 sets of English reading tests;
(2) test evaluation forms, and
(3) a test-specifications crosscheck form.
RESULTS
(1) 3 sets of online reading tests posted on
the website of Sukhothai Thammathirat
Open University;
RESULTS
(2) the tests are qualified as specified:
the tests were designed based on the
reading process and the reading evaluation
process, with the reading performance
level based on CEFR level;
the tests were modified according to the
IOC scores of lower than .50 and the
experts and pilot-test takers suggestions
together with the trial test scores of the
pilot-test takers
the reliability of the 3 sets was analysed
by Kuder Richardson 20 formula, which
yielded .86, .88, and .61 respectively;
the difficulty and discrimination was
accepted at .20 < = p <= .80 — the rest
were modified;
RESULTS
(3) the reading performance level of the 3 sets
of tests:
marks
level
Number of test takers
Version 1
Version 2
Version 3
76-100
C 1 (effective operational proficiency)
3
3
2
51-75
B 2 (independent user: vantage)
12
14
11
26-50
B 1(independent user: threshold)
16
14
18
0-25
Needs improvement
0
0
0
31
31
31
Total number
RESULTS
(4) the test-takers weaknesses included
insufficient understanding of complex structure
and lack of vocabulary;
(5) the guidelines for reading improvement
focus on reading strategies practice at the
cognitive, metacognitive, and socio-affective
level.
DEVELOPMENT PROCESS
Theoretical Framework
-
Reading process framework
(Khalifa & Weir, 2009: 62)
designed based on the mikroskills
from Alderson (2000) and
test usefulness by Bachman & Palmer
(1996)
Test specifications for Reading
• CEFR
• Unaldi’s (2009) comprehensive framework
• Text difficulty
-vocabulary profile of ‘lexitutor’
-Flesch-Kincaid Grade Level
Test specifications for grammar
• selected from Grammar City & Guilds ESOL
Test specifications for Vocabulary
• AWL list
TEST SPECIFICATIONS
• designed based on the mikroskills from
Alderson (2000) and test usefulness by
Bachman & Palmer (1996).
• Unaldi (2009) detailed specification
Unaldi’s detailed specifications
• CONTEXTUAL
•
• Response Method: Gap filling
•
• Text length: up to 500 words
Genre:
everyday materials such as letters and
emails(e.g., enquiries, orders, letters of
confirmation etc.), public information leaflets
brochures, short official documents
straightforward instructions for equipment
expository and informative newspaper/magazine
articles on familiar subjects personal letters
with description of events, feelings and wishes
simple informational sources (e.g., junior
encyclopaedias, leaflets and brochures)
Rhetorical task: narrative, descriptive,
instructive, expository
Pattern of exposition: define, describe,
elaborate, illustrate, compare and
contrast, classify
Explicitness of text structure: explicit
Structural resources
Words/sentence: Ave
Flesch-Kincaid Grade Level: Ave: 8
The complexity of sentence structure:
mostly simple sentences (but some use
of subordinate clauses in PET)
Cohesion: explicit
Lexical resources
95% K1-20 words = K1 84.7 %
K2 8.7% K3 2.3%
AWL words: 2.5%
Nature of information (abstract/concrete):
concrete
Content knowledge: not required: everyday
situations encountered in work, school,
leisure etc., (personal feelings, opinions and
experiences, hobbies and leisure…
familiar topics in expository texts
COGNITIVE
Type of reading:
- understand the main points and/or relevant
points though not necessarily in detail
(description of events, feelings and wishes,
significant and clearly signalled reasoning, and
argumentation)
- identify unfamiliar words from the context
- extrapolate the meaning of occasional
unknown words from the context and deduce
sentence meaning
Text level: word, sentence and
across sentences
SAMPLES OF TEST SPECIFICATIONS ANALYSIS
 See word file
Components of the Test
 Grammar
 Vocabulary AWL words
 Reading
 Test on CD
The level of difficulty of the
Vocabulary
• processed through lexitutor software
(http://www.lextutor.ca/vp/). The analysis
revealed frequencies of vocabulary, types and
levels (K1, K2, K3, AWL); types, tokens,
families, tokens per family, types per family).
• Flesch-Kincaid Grade Level was also taken into
consideration when adapting reading texts to
correspond the specifications in B1 B2 C1.
 Thank you.