Centre for Research in English Language Learning and Assessment
Однажды…
CRELLA
Centre for Research in English Language Learning and Assessment
Однажды…
….. в 1974-oм году …
CRELLA
Centre for Research in English Language Learning and Assessment
Assessing reading – how and why?
An introduction to assessing reading
for language teachers
Dr Lynda Taylor
CRELLA
Designing tasks for
testing L2 reading
CRELLA
We need to think about….
1.
What types of mental (or cognitive)
processing are typically involved in reading?
2.
What types of text are typically involved in
reading activity?
3.
How can different test tasks (question
formats, e.g. MCQ, short answer) help us to
elicit the relevant cognitive processes using
the relevant text-types in our reading tests?
Starting from the right place….
1. the nature of the cognitive processes in
reading
2. the nature of the reading texts
3. the nature of the reading question formats
Starting from the WRONG place?
1. the nature of the reading question
formats
2. the nature of the reading texts
3.
the nature of the cognitive processes
Good reading task design
1.
What types of mental (or cognitive)
processing are typically involved in reading?
2.
What types of text are typically involved in
reading activity?
3.
How can different test tasks (question
formats, e.g. MCQ, short answer) help us to
elicit the relevant cognitive processes using
the relevant text-types in our reading tests?
What actually happens
when we read?
A brief look at cognitive processes
in reading
CRELLA
Creating an intertextual
representation:
Construct an organised
representation across texts
Text structure
knowledge:
Genre
Creating a text level
representation:
Construct an organised
representation of a single text
Remediation where
necessary
Building a mental model
Integrating new information
Enriching the proposition
Rhetorical tasks
General knowledge
of the world
Topic knowledge
Meaning representation
of text(s) so far
Monitor:
goal checking
Goal setter
Selecting appropriate
type of reading:
Inferencing
Establishing
propositional meaning
at clause and sentence levels
Careful reading
LOCAL:
Understanding sentence
Syntactic Parsing
Syntactic knowledge
Lexical access
Lexicon
GLOBAL:
Comprehend main idea(s)
Comprehend overall text
Comprehend overall texts
Lemma:
Meaning
Word class
Expeditious reading
LOCAL:
Scan/search for specifics
Word recognition
GLOBAL:
Skim for gist
Search for main ideas and
and important detail
Visual input
Lexicon
Form:
Orthography
Phonology
Morphology
Visual input
Clearly, the starting point for any reading
activity is the set of marks on a handwritten
or printed page, or on a computer screen – i.e.
a combination of letters, symbols, pictures,
etc.
Word recognition
Word recognition involves matching the form
of a word in a written text with a mental
representation of the orthographic forms of
the language.
Field (2004:234) refers to this as “the
perceptual process of identifying the letters
and words in a text”.
Lexical (vocabulary) access
Accessing the lexical entry containing stored
information about a word’s form and its meaning
from the lexicon. The form includes orthographic
and phonological mental representations of an
item and possibly information on its morphology.
The lemma (the meaning-related part of the
lexical entry) includes information on word class
and the syntactic structures in which the item can
appear and on the range of possible senses for
the word.
Syntactic parsing
Once the meaning of words is accessed, the
reader has to group words into phrases, and
into larger units at the clause and sentence
level to understand the text message.
Establishing propositional meaning
at the clause or sentence level
An abstract representation of a single
unit of meaning: a mental record of the
core meaning of the sentence without
any
of
the
interpretative
and
associative factors which the reader
might bring to bear upon it.
Inferencing (1)
Inferencing is necessary so the reader can
go beyond explicitly stated ideas as the links
between ideas in a passage are often left
implicit.
Inferencing in this sense is a creative
process whereby the brain adds information
which is not stated in a text in order to
impose coherence.
Inferencing (2)
A text cannot include all the information that is
necessary in order to make sense of it. Texts
usually leave out knowledge that readers can be
trusted to add for themselves.
If there was no such thing as inferencing, writing a
text which includes every piece of information
would be extremely cumbersome and time
consuming.
Constructing a mental model
Once the reader has processed the incoming sentence
and elaborated it where necessary and possible through
inferencing, the new information needs to be integrated
into a mental representation of the text so far.
This process entails an ability to identify main ideas, to
relate them to previous ideas, distinguish between major
and minor propositions and to impose a hierarchical
structure on the information in the text.
Creating a discourse-level structure
At a final stage of processing, a discourse-level structure is
created for the text as a whole. The skilled reader is able
to recognise the hierarchical structure of the whole text
and determines which items of information are central to
the meaning of the text. The skilled reader determines how
the different parts of the text fit together and which parts of
the text are important to the writer or to reader purpose.
Establishing a mental representation
across texts
In the real world, the reader sometimes has to
combine and collate
macro-propositional
information from more than one text.
The need to combine rhetorical and contextual
information across texts would seem to place
the greatest demands on processing.
Cognitive processing at A2 to C2
Word recognition
Lexical access
Parsing
Establishing propositional meaning
Inferencing
Building a mental model
Creating a text level structure
Creating an organised
representation of several texts
KET A2
*
*
*
*
(*)
(*)
PET B1
*
*
*
*
*
*
FCE B2
*
*
*
*
*
*
CAE C1
*
*
*
*
*
*
*
CPE C2
*
*
*
*
*
*
*
*
Goal setter
The goal setter in the left hand column is
critical in that the decisions taken on the
purpose for the reading activity will determine
the relative importance of some of the
processes in the central core of the model.
Types of reading
Reading is either
careful or expeditious
and comprehension takes place at the
local and global level.
Local comprehension
Local comprehension refers to the understanding of
propositions at the level of microstructure i.e., the
sentence and the clause.
Basic comprehension questions are used to assess
lexical, syntactic, and semantic abilities and the
ability to understand important information
presented in sentence-level propositions.
The information used in the question and the
information required for the answer are usually in
the same sentence.
Global comprehension
Global comprehension refers to the
understanding of propositions beyond the
level of microstructure, that is, any macropropositions including main ideas, the links
between those macro-propositions and the
way in which the micro-propositions
elaborate upon them.
Careful reading
Careful reading is intended to extract complete
meanings from presented material.
This can take place at a local or a global level, i.e.
within or beyond the sentence right up to the level
of the complete text or texts.
The approach to reading is based on slow, careful,
linear, incremental reading for comprehension.
Expeditious reading
Expeditious reading involves quick, selective
and efficient reading to access desired
information in a text.
Expeditious reading would appear likely to
include skimming, search reading, and
scanning.
Skimming
Skimming is generally defined as reading to
obtain the gist, general impression and/or
superordinate main idea of the whole text.
Search reading
Search reading involves locating information on
predetermined topics. The reader only wants
information necessary to answer set questions
or to provide data for example in completing
written assignments. It differs from skimming in
that the search for information is guided by
predetermined topics so the reader does not
necessarily have to establish a macropropositional structure for the whole of the text.
Scanning
Scanning involves reading selectively, to
achieve very specific reading goals, e.g.
looking
for
specific
words/phrases,
figures/percentages, names, dates of
particular events or specific items in an index
at the local word level.
Types of reading tested at levels A2 to C2
Careful Reading Local
Understanding propositional meaning at
clause and sentence level
Careful Reading Global
Comprehend across sentences
Comprehend overall text
Comprehend overall texts
Expeditious Reading Local
Scanning or search reading
Expeditious Reading Global
Skim for gist
Search reading
KET
A2
PET
B1
FCE
B2
CAE
C1
*
*
*
*
*
(*)
*
*
*
*
*
*
(*)
(*)
CPE
C2
*
*
*
*
Let’s look at a reading test task
• Look at Task 1 which comes from a reading
test.
• Which of the core cognitive reading processes
we have discussed does this task seem to
elicit?
• Which processes does this task apparently
NOT elicit?
Let’s look at a reading test task
Word recognition
Lexical access
Syntactic parsing
Establishing propositional meaning at clause/sentence level
Inferencing
Building a mental model
Creating a text-level representation
Creating an intertextual representation
Let’s look at a reading test task
Word recognition
Yes
Lexical access
Yes
Syntactic parsing
Yes
Establishing propositional meaning at clause/sentence level
Yes
Inferencing
Building a mental model
Creating a text-level representation
Creating an intertextual representation
?
When we design a reading test task…
We need to be explicit about:
• the types of reading that the task requires
• the various cognitive processes we believe the
reading task is eliciting
• how well matched the cognitive processes are to
the level of our students
E.g. Is there a shift from tasks that focus on decoding
to tasks that focus on meaning building from main
ideas, to a text level representation to intertextual
representation
Cognitive validity in the
testing of Reading
The extent to which the tasks we
employ in a reading test elicit
the
cognitive
processing
involved in target reading
contexts beyond the test itself.
What is the nature of
the texts we read?
A brief look at what makes a text
difficult to read
CRELLA
Contextual features of texts and
tasks in reading tests
This relates to the appropriateness of both :
the linguistic and content demands of the text to be
processed (i.e. read and comprehended)
and
the features of the task setting that impact on task
completion (e.g. responding to comprehension questions
or writing a summary)
Controlling contextual features of
reading texts and tasks – 3 questions
• What are the key contextual features (within-text) that we
need to think about when selecting texts for reading test
tasks?
• What degree of complexity should we aim for in these
features across each of the proficiency levels we want to test?
• What are the methods that can help us to understand the
difficulty level of a reading text?
Contextual features in reading
Some illustrative features relating to:
• lexical complexity (decoding)
• structural complexity (syntactic parsing)
• cohesion (the construction of meaning)
Some key lexical parameters
L1
L2
L3
L4
L5
Syllables per word
Type token ratio
Word frequency
Lexical density
Proportion of academic words
L1 : Average syllables per word
• the mean number of syllables per content word
multisyllabic words take longer to read and process than
monosyllabic words [Rayner & Pollatsek, 1989]
“In general, the more syllables per word and the more words per sentence, the higher the
associated grade level of the text”
[White, S. (2011) Understanding Adult Functional Literacy: Connecting Text Features, Task
Demands, and Respondent Skills. Taylor & Francis]
L2 : Type token ratio
• the number of unique words divided by the number of tokens
of the words
Each unique word in a text is a word type. Each instance of a
particular word is a token.
When the type: token ratio is 1, each word occurs only once
in the text; comprehension should be comparatively difficult
because many unique words need to be encoded and
integrated with the discourse context. A low type: token ratio
indicates that words are repeated many times in the text,
which should generally increase the ease and speed of text
processing.
[Templin, M (1957) Certain Language Skills in Children: Their development and
interrelationships. Institute of Child Welfare Monograph Series No. 26. Minneapolis, MN:
University of Minnesota Press]
L3 : Word frequency
• the relative frequency of occurrence of words
Frequency effects have been shown to facilitate decoding:
– frequent words are processed more quickly and understood better
than infrequent ones (Haberlandt & Graesser, 1985; Just & Carpenter,
1980).
– rapid or automatic decoding = strong predictor of L2 reading
performance (Koda, 2005)
– texts which assist such decoding (e.g., by containing a greater
proportion of high frequency words) are easier to process….
The more frequent a word, the more likely it is to be processed with a fair
degree of automaticity, thus increasing reading speed (even among lower
level learners) and freeing working memory for higher level meaning
building. (Crossley, Greenfield and McNamara, 2008)
L4 : Lexical density
• depends on distinguishing between different word types, i.e.
lexical (content) and function words
– lexical: verbs, nouns, adjectives, adverbs
– function: auxiliaries, determiners, pronouns, prepositions,
conjunctions
Accessing the meaning of lexical items requires accessing
the mental lexicon, function words can be dealt with by
pattern matching. Reading focuses mainly on lexical items
and readers tend to skip function words.
L5 : Proportion of academic words
• the incidence of academic words in a text
• proved to be a good predictor of level in a study of FCE, CAE and CPE
reading texts (Weir et al, 2012)
Mean
SD
FCE (B2)
1.61%
1.26%
CAE (C1)
1.63%
1.41%
CPE (C2)
5.82%
2.84%
Some key syntactic parameters
S1
Sentence length
S2
Readability formulae
S3
Higher level constituents
Syntactic complexity
• linear processing of text in careful reading, with the reader
decoding word by word
• assembly of decoded items into larger scale syntactic
structure
(Just & Carpenter, 1987; Rayner & Pollatsek, 1994)
• cognitive demands imposed vary considerably according to
how complex the structure is
(Perfetti, Landi & Oakhill, 2005)
(Crossley ,Greenfield and McNamara, 2008)
S1 : Sentence length in Cambridge ESOL reading papers
Main Suite Level Average number Range
of words per
sentence
KET (A2)
13.2
8 - 17
PET (B1)
FCE (B2)
CAE (C1)
14.9
18.4
18.6
10 - 20
11 - 25
13 - 27
CPE (C2)
19.6
13 - 30
S2 : Readability formulae
• are long-established and widespread in use
• rely heavily on word length and sentence length
• ignore many language and discourse components
that are theoretically expected to influence
reading and comprehension difficulty
…nevertheless…
• texts with longer words and lengthier sentences are more
difficult to read
– longer words tend to be less frequent in the language and
infrequent words take more time to access and interpret
during reading
– longer sentences place more demands on working
memory
– real-time processing means holding information in your
head until you can parse sentences syntactically
– the longer the sentence, the more difficult this may be
S2 : Difficulty/readability estimates in
Cambridge ESOL reading papers
Main Suite
level
Flesch reading
ease score
Flesch-Kincaid Flesch-Kincaid
grade level
range
KET (A2)
78.3
5.5
2 – 7.4
PET (B1)
64.7
7.9
5 – 10.1
FCE (B2)
66.5
8.4
5 – 12.3
CAE (C1)
58.4
9.6
5.7 - 16
CPE (C2)
57.7
9.9
5.6 – 16.1
S3 : Higher level constituents
• the number of main verbs in a sentence is broadly indicative
of the number of clauses - thus of complex syntactic
composition
• the more complex the syntactic composition, the greater the
load on cognitive processing
• the more clauses you have to process in a sentence, the more
propositions you have to hold in working memory and link
together
Cohesion (and coherence)
Cohesion is an objective property of the explicit language and text.
There are explicit features, words, phrases, or sentences that guide the
reader in interpreting the substantive ideas in the text, in connecting ideas
with other ideas, and in connecting ideas to higher level global units (e.g.,
topics and themes). These cohesive devices cue the reader on how to
form a coherent representation.
The coherence relations are constructed in the mind of the reader and
depend on the skills and knowledge that the reader brings to the
situation…coherence is a psychological construct, whereas cohesion is a
textual construct.
[Graesser et al 2004: 193]
Cohesion
• two forms of textual cohesion can be estimated :
• referential cohesion (the extent to which
words in the text co-refer)
• conceptual cohesion (the degree of similarity
between concepts in different parts of a text)
Let’s look at a reading text
• Read the text and, as you do so, think about
aspects of the text’s complexity related to :
– the difficulty of the lexis/vocabulary
– the difficulty of the syntax
– the cohesion of the text
• Don’t worry about answering the questions
over the page. We shall look at those later.
Analysing the text’s complexity
K
E
T
Overall Number Lexis
of words
Approximately
Restricted to common
740-800words
items which normally
occur in the everyday
vocabulary of native
speakers.
Structure
P
E
T
Approximately
1460-1590
words
General vocabulary
Mostly simple sentences but
sufficient for most topics some use of relative and
in everyday life.
other subordinate clauses.
F
C
E
Approximately
2000 words
Good range of
vocabulary. Topics are
addressed in detail and
with precision.
C
A
E
Approximately
3000 words
Broad range of
vocabulary including
idiomatic expressions
and colloquialisms as
well as language
relating to opinion,
persuasion and ideas.
C
P
E
Approximately
3000 words
Mainly simple sentences
A range of sentence
patterns– from the simple to
the complex.
This level is typified by:
many complex sentences
Frequent use of modals
Some use of ellipsis
Complex approaches to
referencing – range of
pronouns and adverbials,
as well as use of
synonymy.
Very wide range of
Most sentences are long
vocabulary including
and complex. No restriction
idiomatic expressions
on the types of structure
and colloquialisms as
employed by the text. Many
well as language
examples of structures
relating to opinion,
typically used for effect in
persuasion and abstract writing – sentences with
ideas.
several subordinate
clauses, for example.
What is the nature of
the reading task?
A brief look at question formats in
reading tests
CRELLA
Response method
•
•
•
•
•
•
Selected response
Multiple choice (MCQ)
True/false
Right/wrong doesn’t say
Multiple matching
Gapped text
•
•
•
•
•
Constructed response
Short answer questions
Information transfer
Random deletion cloze
Selective deletion gapfilling
• Reading into writing
(e.g. summary)
Position of the test questions
• Before the reading text?
or
• After the reading text?
Let’s look at some test questions
• Question 1
• Question 2
• Question 3
• Question 4
Context validity in Reading
Context validity relates to the
appropriateness of both the linguistic
and content demands of the text to be
processed, and the features of the
task setting that impact on task
completion.
When designing reading test tasks we need
to take account of….
1.
What types of mental (or cognitive)
processing are typically involved in reading?
2.
What types of text are typically involved in
reading activity?
3.
How can different test tasks (question
formats, e.g. MCQ, short answer) help us to
elicit the relevant cognitive processes using
the relevant text-types in our reading tests?
Спасибо за внимание!