Toward a Test Theory for the
Interactionalist Era
Robert J. Mislevy
University of Maryland
Samuel J. Messick Memorial Lecture
Presented at the Language Testing Research
Colloquium, Barcelona, Spain, June 9, 2007.
June 9, 2007
LTRC 2007 Messick Address
Slide 1
Chi, Feltovich, & Glaser, 1981
June 9, 2007
LTRC 2007 Messick Address
Slide 2
Key Ideas
Generative patterns from the domain
organized experts’ perceptions,
understanding, and actions
 Experts assembled pieces from patterns
to fit particulars of context and purpose

» F=MA: Conventional v. situated meaning

Critical role of narrative layer
» Integrates principles & equations with context
» Locus for understanding, planning, & action
June 9, 2007
LTRC 2007 Messick Address
Slide 3
Why Am I Telling You This?

Situative / sociocultural / interactionist
perspective is frontier in all domains

Analogues between model-based reasoning
and using language

Test theory is model-based reasoning
» Interactionalist perspective on test theory
» Bottleneck the narrative layer
June 9, 2007
LTRC 2007 Messick Address
Slide 4
Outline
Messick on assessment arguments
 Interactionalist perspective

» Re language, comprehension, cultural
meaning

A narrative space / metaphor for
assessment in this light
» Attention to senses and roles of context

Implications for building and using
measurement models
June 9, 2007
LTRC 2007 Messick Address
Slide 5
The Assessment Argument
What complex of knowledge, skills, or other
attributes should be assessed?
 What behaviors or performances should
reveal those constructs?
 What tasks or situations should elicit those
behaviors?

Messick (1994) “The interplay of evidence and consequences in
the validation of performance assessments”
June 9, 2007
LTRC 2007 Messick Address
Slide 6
Perspectives on the L2 Construct
What is foregrounded?
 Trait
» Focus on underlying abilities of individuals that are
called upon in a wide variety of situations.

Behaviorist
» Focus on context, from external point of view—
success of action in specified classes of situations.

Interactionalist …
June 9, 2007
LTRC 2007 Messick Address
Slide 7
An Interactionalist Perspective
Concern with language used in specific
discursive practices rather than on language
ability independent of context.
Focus on the co-construction of discursive
practices by all participants ...
A set of general interactional resources that
participants draw upon in specific ways in order
to co-construct a discursive practice.
(Young, 2000, p. 5)
June 9, 2007
LTRC 2007 Messick Address
Slide 8
An Interactionalist Perspective
successful interaction presupposes not only
a shared knowledge of the world, …
but also the construction of a shared
internal context … that is built through
the collaborative efforts of the interactional
partners.
Kramsch ( 1986, p. 367)
June 9, 2007
LTRC 2007 Messick Address
Slide 9
Challenges for Assessment


Amending the construct of individual ability
to accommodate [how] language use in a
communicative event reflects dynamic
discourse, which is co-constructed among
participants; and …
reconciling [the notion that language ability is
local] with the need for assessments to yield
scores to generalize across contextual
boundaries.
Chalhoub-Deville (2003, p. 373)
June 9, 2007
LTRC 2007 Messick Address
Slide 10
A Narrative Frame

Themes from, e.g., cognitive psychology,
literacy, neuroscience, anthropology:
» Connectionist metaphor, Associative memory

Situated cognition & information processing
» Construction-Integration (CI) theory of
comprehension (Kintsch and others)

Individual  Sociocultural perspectives
» A cognitive theory of cultural meaning
(Strauss & Quinn, 1997)
June 9, 2007
LTRC 2007 Messick Address
Slide 11
A Cognitive Theory of Cultural Meaning
 “Interactional Resources”

External: knowledge “out there”, patterns
that exist in use by groups, in tools &
processes
» Cultural models: What ‘being sick’ means,
restaurant script, Newton’s laws, complaints
» Linguistic: Grammar, conventions, frames


Interactional: enable the co-construction of
new shared meanings
Internal: patterns in individuals attuned to
shared external patterns
June 9, 2007
LTRC 2007 Messick Address
Slide 12
Inheritance from Schema Theory

Knowledge as organized patterns, at many levels…

Assembled to understand & to create particular
situations in the world

Developed, strengthened, modified by use

Associations of all kinds, including applicability,
affordances, procedures, strategies, affect
» “The user’s knowledge of the language rules is interlocked
with his knowledge of when, where, and with whom to use
them” (Ellis, 1985)
June 9, 2007
LTRC 2007 Messick Address
Slide 13
and all aspects of context…
Inside A
A
Inside B
B
Context
A la Kintsch: “Conventional” meaning, or
propositional content of text / speech…
June 9, 2007
LTRC 2007 Messick Address
Slide 14
Inside A
A
Inside B
B
Context
The C in CI theory, Construction:
•If B hasn’t developed a given pattern in past
Activation of both relevant and irrelevant …
experience, it can’t be activated (although it may
Linguistic models, e.g.
get constructed in the interaction).
•Conventions, Rhetorical frames
•Relevant patterns from LTM may be activated
Cultural models, e.g.,
in some contexts but not others (e.g., physics
•Equilibrium, Human motivation
models; question formation (Tarone)).
June 9, 2007
LTRC 2007 Messick Address
Slide 15
Inside A
A
Inside B
B
Context
The I in CI theory, Integration:
•Resulting synthesis of reinforced activated
cultural / linguistic / situational patterns:
•the Situation model
•Akin to ‘stories’ in Larkin’s physics study
•Situation model is the understanding
June 9, 2007
LTRC 2007 Messick Address
Slide 16
Inside A
A
Inside B
B
Context
Situation model is also the
basis of planning and action.
June 9, 2007
LTRC 2007 Messick Address
Slide 17
Inside A
A
Inside B
B
Context
Context
Context
Context
June 9, 2007
LTRC 2007 Messick Address
Slide 18
Inside A
A
Inside B
B
Context
Context
Context
Context
June 9, 2007
Ideally, participants
activate cultural &
linguistic models that
are compatible in
LTRCrelevant
2007 Messick ways…
Address
Slide 19
Inside A
A
Inside B
B
Context
Context
To lead to co-constructed
meaning / sufficiently (?)
shared understanding
Context
Context
June 9, 2007
Kramsch’s "shared
internal context"
LTRC 2007 Messick Address
Slide 20
Inside A
A
Inside B
B
Context
Context
Preceeding overheads
highlight the importance of a
common narrative space for
“shared
internal context” re nature &
thinking about
assessment:
use of knowledge would help ground
A = Linguist
compatible views of assessment
B = Psychometrician
purpose, design, analysis, and use for
Context
Context
June 9, 2007
the job at hand.
LTRC 2007 Messick Address
Slide 21
Some external aspects of context
are public &
objective,
Can distinguish
external
and e.g.,
internal
aspects of•Setting
context
(e.g., Douglas,
2000)features
Target language
use (TLU)
•Physical
attributes
Assessment
task
Inside A •Directives
A
B features Inside B
(Bachman & Palmer)
Senses and roles of “context”
Context
Context
Context
Context
June 9, 2007
LTRC 2007 Messick Address
Slide 22
Some aspects of context can be interpreted
by an external observer through the lens
of targeted cultural/linguistic models.
e.g., apology situation.
Inside A
A
Inside B is
The question
at issue in assessment
B
In assessment, we can often arrange for
whether the examinee activates targeted
this to be the case.
compatible l/c models,
then constructs and acts accordingly
Noteathe
need to activate
manymodel.
other l/c
through
corresponding
situation
models in order to construct a situation
model, plan, and carry out action.
•Many places to slip, but others to
compensate.
•“Alternative explanations” in
assessment argument.
Context
Context
Context
Context
June 9, 2007
LTRC 2007 Messick Address
Slide 23
Some aspects of context can be interpreted
by an external observer through the lens
of targeted cultural/linguistic models.
e.g., apology situation.
Inside A
A
Inside
B
As assessment
use these
Bdesigners, we
In assessment, we can sometimes arrange
situations that call upon targeted linguistic/
for this to be the case; sometimes watch
cultural models to determine what
for it to happen.
examinee
actions
would signal
This sense
of context
plays arecognition,
key role in
comprehension,
action
through them.hence
•Evaluation
of performance,
•Observable variables that go into a
measurement model.
Context
Context
Context
Context
June 9, 2007
LTRC 2007 Messick Address
Slide 24
What can we say about individuals?
Use of interactional resources in appropriate
contexts in appropriate ways; i.e.,
Attunement to targeted cultural/linguistic patterns:

Recognize markers of externally-viewed patterns?

Construct internal meanings in their light?

Act in ways appropriate to targeted c/l models in the
assessment contexts?

What are the ranges and circumstances of activation?
(variation of performance across contexts)
June 9, 2007
LTRC 2007 Messick Address
Slide 25
Implications for measurement models



Basic form:
Prob X ij  i ,  j
Probability of aspects of performance Xij given
parameters for person i and situation j
(all could be vector-valued)
• Way too simple
• No explicit connection
with
CI comprehension
These are
indeed
properties of the
model, interaction
processes,meaning
etc.
conventional
of the
• Apparent separation
of person
andand
situation
measurement
model
parameters.
characteristics
June 9, 2007
LTRC 2007 Messick Address
Slide 26
An Interactionalist Perspective:
Instantiation in a Context




Xs result from particular persons calling upon
resources in particular contexts (or not, or how)
Mechanically s simply accumulate info across
situations
Our chosing situations and what to observe
drives their situated meaning.
Situated meaning of s are tendencies toward
these actions in these situations that call for
certain interactional resources, via l/c models.
June 9, 2007
LTRC 2007 Messick Address
Slide 27
An Interactionalist Perspective:
Instantiation in a Context



Inference to criterion contexts (TLU) depends
on analysis of what l/c models are called upon
in assessment use argument…
What is similar, what is different, re the
resources task & criterion situations call for?
To what degree does activation and success
in task context correspond to activation and
success in criterion context?
(e.g., Bachman, Chalhoub-Deville, Douglas, Chapelle)
June 9, 2007
LTRC 2007 Messick Address
Slide 28
An Interactionalist Perspective:
Instantiation in a Context
What demands do we minimize via task
design, so needn’t model?
 What resources do we already know
examinees can draw upon, so tasks can
require them but we needn’t model?

» “Hidden” but essential to meaning
» Occupational English Test (McNamara)
» Analogous to ‘focus on forms’ learning
June 9, 2007
LTRC 2007 Messick Address
Slide 29
An Interactionalist Perspective:
Instantiation in a Context
How to model inconsistent performance?



If “unmotivated,” it’s noise; via probability model
Promising direction: Model individual’s degree or
pattern in variation in terms of context features
If “motivated”: Model in terms of s
» Divide & Conquer: Multiple unidimensional tests (OET)
» Exploratory multidimensional: Discover patterns in data.
» Controlled: Structured multidimensional models
(e.g., Embretson, Adams & Wilson, von Davier)
June 9, 2007
LTRC 2007 Messick Address
Slide 30
Structured Multidimensional Models

Still way too simple, but …
» purposeful modeling of motivated variation in
complex tasks when persons differ in targeted ways
» exploit what is known about examinees but not
modeled

Controlled mixes of demand features
» E.g., in OET-like situations, wrt medical knowledge,
complexity of stimulus language, complexity of
language to be produced.

“Throwing the data over the wall” won’t work
June 9, 2007
LTRC 2007 Messick Address
Slide 31
Structured Multidimensional Models
Probabilities modeled in terms of task features:
» Which dimensions are relevant for which observables
from which tasks? (Robinson’s “difficulty” features)
» Task parameters modeled in terms of Robinson’s
“complexity” features.

Hence a priori structure of patterns to interpret
» Can organize s in terms of traits or context features

Coordinated task design and measurement model
» Create tasks within task models
June 9, 2007
LTRC 2007 Messick Address
Slide 32
Conclusion
How much can testing gain from modern
cognitive psychology?
So long as testing is viewed as something that
takes place in a few hours, out of the context of
instruction, and for the purpose of predicting a
vaguely stated criterion,
then the gains to be made are minimal.
Buzz Hunt (1986)
June 9, 2007
LTRC 2007 Messick Address
Slide 33
Conclusion
I have argued that we need to capitalize on
[method effects] by designing tests for
specific populations -- tests that contain
instructions, content , genre, and language
directed toward that population.
The goal is to produce tests … that would
provide information interpretable as evidence
of communicative competence in context.
Douglas (1998)
June 9, 2007
LTRC 2007 Messick Address
Slide 34
Conclusion
Interactionalist view of test theory…



for assembling, analyzing, and interpreting
assessments,
for arguments in interactionalist view of language
Methods and exemplars needed,
but more pressing need is narrative frame …
» To connect view of language proficiency with the
machinery of test theory,
» Toward modeling purposeful variations in a coherent
design space.
June 9, 2007
LTRC 2007 Messick Address
Slide 35