What is the construct in task-based
language assessment?
Robert J. Mislevy
Professor, Measurement, Statistics and Evaluation
Affiliated Professor, Second Language Acquisition
University of Maryland, College Park
Presented in the invited colloquium “Reprising the role of tasks in language assessment”
organized by John Norris and Steven Ross at the Second Language Research Forum
2010, October 14-17, 2010, University of Maryland, College Park, MD.
Supported by a grant from the Spencer Foundation.
Oct 16, 2010
SLRF 2010
Slide 1
What is the construct?

Bachman 2005 LTRC plenary address:
» What is the construct? The dialectic of abilities and
contexts in defining constructs in language assessment.

Challenges from a sociocognitive perspective
(Atkinson, 2002; Chalhoub-Deville, 2003)
» Interplay of extrapersonal and intrapersonal patterns.
» Capabilities as resources to construct and act through
relevant patterns in meaningful situations.

Challenges in task-based language testing
» Many patterns at many levels; interaction; evolving
Oct 16, 2010
SLRF 2010
Slide 2
[I]t seems to me that the critical issue is how we
define the construct to be assessed — as
ability or as task.
Bachman, 2007, p. 71.
Oct 16, 2010
SLRF 2010
Slide 3
[T]he construct of interest in task-based
assessment is performance of the task itself.
Long & Norris, 2000, p. 600.
Oct 16, 2010
SLRF 2010
Slide 4

The final form of a sentence in ordinary conversation
[has] to be understood as an interactional product.
Schegloff, 1995, p.192.

To adapt a social view of performance… is at some
level incompatible with taking the traditional view of
performance as a simple projection or display of
individual competence.
Macnamara & Roever, 2006, p. 46.
Is the construct co-constructed by all of the
participants in the discursive practice ?
Bachman, 2007
Oct 16, 2010
SLRF 2010
Slide 5
The ability components the language user brings to
the situation … interact with situational facets to
change those facets as well as to be changed by
them.
[The construct is] “ability – in language user – in
context.”
Chalhoub-Deville, 2003, p. 372.
Is the construct is strictly local?
Bachman, 2007
Oct 16, 2010
SLRF 2010
Slide 6
My Objective


Propose an consistent sense of “construct” for
assessment, including language testing.
Ground it in …
» a sociocognitive perspective and
» the structure of assessment design and use arguments.

Show how it …
» encompasses most of the senses of construct in
Bachman’s analysis,
» helps answer the problematic questions about constructs.
Oct 16, 2010
SLRF 2010
Slide 7
The Situative Stance
Affordances and abilities … are … inherently relational.
An affordance relates attributes of something in the
environment to an interactive activity by an agent
who has some ability, and an ability relates attributes
of an agent to an interactive activity with something in
the environment that has some affordance. …
It does not go far enough to say that an ability depends
on the context of environmental characteristics, or
that an affordance depends on the context of an
agent's characteristics.
The concepts are codefining...
Greeno, 1994, p. 338.
Oct 16, 2010
SLRF 2010
Slide 8
The assessment [design] argument
(Messick, 1994)

What complex of knowledge, skills, or other
attributes should be assessed?

What behaviors or performances should reveal
those constructs?

What tasks or situations should elicit those
behaviors?
Oct 16, 2010
SLRF 2010
Slide 9
Toulmin’s Argument Structure
Claim
unless Alternative
Warrant
explanation
since
so
Backing
Oct 16, 2010
Data
SLRF 2010
Slide 10
Claim about student
Backing concerning
assessment situation
unless
on account of
Warrant
concerning
assessment
Alternative
explanations
since
so
Data concerning
student
performance
Warrant
concerning
evaluation
since
Data concerning
task situation
Warrant
concerning
task design since
Student acting in
assessment situation
Other information
concerning student vis a vis
assessment situation
Bachman / Macnamara / Chalhoub-Deville
consider: Is the construct … co-constructed
by all of the participants?
Me: The activity and its meaning are coconstructed, but the assessment construct is
the examinee’s capability to act in ways that
Claim
about student
productively contribute the
construction.
Backing concerning
assessment situation
unless
on account of
Features of Warrant
concerning
performance assessment
evaluated in light
of emerging
context.
Macro features
of performance
Features of
context arise over
time as student
acts / interacts.
Alternative
explanations
since
so
Data concerning
student
performance
Warrant
concerning
evaluation
since
Micro features of
performance
Data concerning
task situation
Time
Warrant
concerning
task design since
Especially important in interactive
and
extended
performance
Unfolding
situated
Micro features
of Macro features ofcontexts
performance
Student acting in
assessment situation
Other information
situation asconcerning
it
situation
student
vis a vis
assessment situation
evolves
Concerns features of (possibly
evolving) context as seen from the view
of the assessor – in particular, those
seen as relevant to targets of inference:
•Important because of task qua task?
Claim about student
•Import as opportunities to exhibit
pattern attunement in context?
Backing concerning
assessment situation
•Where do they match / mismatch
unless
Alternative
on account of
explanations
features of use
situations?
Warrant
Evaluation of performance
concerning
since
assessment
concerns clues that suggest
so
attunement to features of
Data concerning
Data concerning
student
task situation
performance
cultural / linguistic models of
Warrant
interest:
Warrant
concerning
concerning
task design since
evaluation since
•Aspects of success in task?
•Aspects of
broader
L/C/S
Other
information
concerning student vis a vis
models? assessment situation
Student acting in
assessment situation
Claim about student
Backing concerning
assessment situation
Dp1
on account of
Ds1
Dp1
p2
OI1
A1
Warrant
concerning
assessment
…
unless
Ds2
since
Dsn
OIn
so
An
Data concerning
student
performance
Warrant
concerning
evaluation
Dpn
p1
OI2
A2
Design Argument
Alternative
explanations
Data concerning
task situation
Warrant
concerning
task design since
Multiple tasks: What do they
have in common / i.e., sampling
from
what domain?
Student acting in
since
Other information
concerning student vis a vis
assessment situation
assessment situation
Use Argument
Claim about student in use
situation
(Bachman)
unless
Warrant concerning
use situation
Alternative
explanations
since
on account of
Backing concerning
use situation
Other information
concerning student vis a
vis use situation
Data concerning
use situation
Claim about student
Backing concerning
assessment situation
unless
on account of
Warrant
concerning
assessment
Alternative
explanations
since
so
Data concerning
student
performance
Warrant
concerning
evaluation
Design Argument
since
Data concerning
task situation
Warrant
concerning
task design since
Student acting in
assessment situation
Other information
concerning student vis a vis
assessment situation
Use Argument
Claim about student in use
situation
unless
Warrant concerning
use situation
Alternative
explanations
since
on account of
Backing concerning
use situation
Data concerning
use situation
Other information
concerning student vis a
vis use situation
Claim about student
Backing concerning
assessment situation
unless
on account of
Warrant
concerning
assessment
since
so
Data concerning
student
performance
Warrant
concerning
evaluation
Design Argument
Actions in assessment
situations and use situations
are always understood
through an interactionalist /
sociocognitive lens.
Data concerning
since
Alternative
explanations
task situation
Warrant
concerning
task design since
Student acting in
assessment situation
Other information
concerning student vis a vis
assessment situation
Use Argument
Claim about student in use
situation
unless
Warrant concerning
use situation
on account of
Backing concerning
use situation
Backing concerning
assessment situation
on account of
Warrant
concerning
assessment
Alternative
explanations
since
The values that the variable(s)
Other information
concerning
student visa
a simplified
can take
induce
vis use situation
view of some aspects of
Claim about student
peoples’ capabilities from
some perspective.
Claim
about student
phrased
The analyst’s
interpretation,
unless
Alternative
explanations
backed
upofbyscore(s)
a compatible
in
terms
on some
since
operationalization,
is the
variable(s)
– conduit
from
so
construct theobservations
assessment to
assessment
Data concerning
Data concerning
seeks
to measure.
student
task situation
use
situations
Data concerning
use situation
performance
Warrant
concerning
evaluation
Design Argument
since
Warrant
concerning
task design since
Student acting in
assessment situation
Other information
concerning student vis a vis
assessment situation
Use Argument
Claim about student in use
situation
unless
Warrant concerning
use situation
Alternative
explanations
since
on account of
Backing concerning
use situation
Data concerning
use situation
Other information
concerning student vis a
vis use situation
This is inherently a statement
Claim about student
about the capabilities or
Bachman Backing
asks:concerning
Construct defined
propensities of the examinee.
assessment
situation
in terms of
abilities
or tasks?
and situated meaning
unlessIts nature
Alternative
on account of
explanations
Me: The assessment construct is
depend on …
Warrant
always about examinees’
concerning
since
•Design choices about the
assessment
capabilities, but can be organized so
features of the situation and
around traits or capabilities toData concerning Data
concerning
student
task
situation
•Features
of performance to
perform in various senses in task
performance
evaluate; and
situations.
Warrant
Warrant
concerning
concerning
about the set of task
task
design •Choice
since
evaluation since
situations; and
Other information
concerning
student
vis a vis
•The relationship
of task
assessment situation
Student acting in
Design Argument
assessment situation
situations to use situations.
Use Argument
Claim about student in use
situation
unless
Warrant concerning
use situation
Alternative
explanations
since
on account of
Backing concerning
use situation
Data concerning
use situation
Other information
concerning student vis a
vis use situation
Claim about student
Backing concerning
assessment situation
Trait-based testing
on account of
unless
Alternative
explanations
•From situative
p.o.v., since
there many situations with similar affordances,
concerning
assessment
amenable to similar capabilitiesso [“invariant”? Me: too strong]
Data concerning
Data concerning
•Trait-based construct presumes
stability
of certain level/kind of pattern
student
task situation
performance
use & capabilities across such
situations.
Warrant
Warrant
•Situation features designed
to
evoke
concerning evidence of traits as conceived.
concerning
task design since
evaluation since
•Correspondence to features
of use situation not critical.
Other information
•Performance features identified as evidence of traits.
concerning student vis a vis
assessment situation
Student acting inmeant to require traits.
•CanDesign
be wide
variety of use situations,
Argument
assessment situation
Warrant
assessment situation
Use Argument
Claim about student in use
situation
unless
Warrant concerning
use situation
Alternative
explanations
since
on account of
Backing concerning
use situation
Data concerning
use situation
Other information
concerning student vis a
vis use situation
Claim about student
Backing concerning
assessment situation
on account of
unless
Alternative
explanations
Task-based testing
I: Focus on competences in performance
Warrant
concerning
•Task featuresassessment
designedsinceto evoke evidence of traits as conceived, in the
so
context of valued real-world use
tasks.
concerning
Data concerning
•Can be language centric,Databut
also pragmatic,
student
task situation sociolinguistic.
performance
•Correspondence to selected features
of use situation important.
Warrant
Warrant
concerning
•Performance features
identified
asdesign
evidence
of traits.
concerning
task
since
evaluation since
•Can be wide variety of use situations, meant to require traits.
Other information
•Construct allows more context dependence of pattern
concerninguse
student&
vis acapabilities.
vis
Design Argument
Student acting in
assessment situation
assessment situation
Use Argument
Claim about student in use
situation
unless
Warrant concerning
use situation
Alternative
explanations
since
on account of
Other information
Data concerning
Bachman, Chalhoub-Deville
consider:
concerning student vis a
use situation
vis use situation
Is the construct … strictly local?
Me: The creation
of performance is
Claim about student
& Norris propose: Construct
strictly local in every instance,Long
but the
task performance?
construct is capability of doingis such
-Backing concerning
assessment situation
Me: Task
performance is of central
stability
/ variability across
unlesssituations
Alternative
on account of
explanations
Task-basedistesting
II: Focus
on aspects
ofbut
performance
interest,
assessment argument
an
empirical
question.
Warrant
concerning
•Task featuresassessment
designedsinceto reflect features
of important
real-world
tasks.
construct
is capability
for targeted
so
of task important.
performance, as
•Correspondence to selected features
of aspects
use situation
Data concerning
Data concerning
observed
in task and inferred
•Performance features identified
as evidence
of capabilities
to act to use
student
task situation
performance
situations.
effectively in use situations.
Backing concerning
use situation
Warrant
concerning
•Assessment construct:
capability
to
perform in corresponding ways in
concerning
task
design
since
evaluation since
corresponding real-world situations.
Warrant
Design Argument
Student acting in
assessment situation
Other information
concerning student vis a vis
assessment situation
Conclusion I
Q: What is the construct?
A: What do you want it to be?
In any given application, the assessment-argument
construct concerns capabilities of the individual (as
opposed to abilities or traits).
It is operationalized by choices wrt assessment design
and intended inferences, which can be grounded in a SC
perspective of capabilities, made explicit in argument
framework, and embodied in the elements and processes
of the assessment machinery.
Oct 16, 2010
SLRF 2010
Slide 22
Conclusion II
Q: Where is the construct?
A: The construct is a frame in the analyst’s cognition: to
recognize, make sense, and reason from patterns and
regularities in peoples’ behaviors in unique situations.
The regularities arise from the way people and situations work
in the real world, through the interplay of extrapersonal and
intrapersonal patterns.
Within this frame, we summarize the ways or extents
examinees act in assessment situations that we (in part) shape
and decide how to characterize in accordance with the frame.
We use this synthesis to reason about use situations.
Oct 16, 2010
SLRF 2010
Slide 23