Sauli Takala
University of Jyväskylä
12th CERCLES Conference
London Sept. 6-8, 2012
•
•
•
•
•
•
Overview
Some current issues in definition of the
construct in language education
Task-based language learning/assessment :
neglect of some relevant sources, critique
of some sources
Some models of TB- assessment
Some arguments for and against a checklist
approach
Assessment using checklists
Concluding remarks
Approaches to definition of construct of
language performance :
• ability-focused (eg,. Bachman 2002, 2006)
• task-focused (eg., Norris 2005)
• interaction-focused.
• See next slide for an overview of approach to
construct definition by Bachman (2007; What is the
construct? The Dialectic of Abilities and Contexts in Defining
Constructs in Language Assessment. In Fox et al. Language Testing
Reconsidered. Ottawa: U of Ottawa Press, 41-71)
Construct
Ability/Trait
Task/Content
Approaches to defining the construct
in language testing/assessment,
Bachman, 2007
Skills & elements;Lado,
1961; Carroll1961
Elements/aspects/levels/
integrated skills
Discrete points,
integrative tasks.
Taxonomy óf tasks.
Direct testing, perform. ass Lang proficiency on tasks,
Clark 72, Jones85, Wesche performance in real life
Tasks mirroring/duplic.
real-life tasks; authentic p.
Pragmatic LT; Oller 1979
Pragm expectancy gramm.
Pragmatic tests
Communicative LT;
Canale & Swain 1980 etc.
Communicative compet.
General language profic.
Meaningful comm. situat.
Authentic tasks
Interaction-ability: Bachman 1990, Bach & Palm96
Communic. lang. ability
Language ability
Test methof facets
Task characteristics
Task.based perf. assessm1
Brindley1994, MacNam96
Ability for language use
Simulation of resl-world
tasks
Task-based perf.assessm2
Norris1998, Brown2002
Ability to accomplish part.
tasks or task types
Performance on particular
tasks ort task types
Minimalist interactionalist Intectional competence/
Kramsch1986, Chapelle98
ability
Collaborative activity,
character. of interaction
Strong interactionalist
He&Young 98, Young00
Discursive practices
Moderate interact.
Chalhoub-Deville03
Interactional competence
Ability-in-individual-in - context
Are there fundamental differences or only a
matter of difference in emphasis?
Are we, in practice, usually interested in
finding out to what extent there is ability in
using language to do some tasks?
Is there an inevitable interaction between
degree of ability and the nature of the
tasks?
Do tasks represent the goals/required
Some useful early sources - but to some extent
neglected
• Linguistic philosophy (esp. later Wittgenstein,
Philosophical Investigations, 1953)
• Sociology and social psychology (Kurt Lewin,
Bales (1950), Festinger (1951), Watzlawick
(1967) – communication and interaction in
small groups
• Linguistic pragmatism – especially early critique
of Austin (1955/1962), and Searle (1969) by
German pragmalinguists (Wunderlich, 1972;
Ehlich & Rehbein, 1972 – praxeograms...)
ROLE/SOCIAL ROLE: also neglected?
In social psychology, a role: a set of connected behaviours, rights
and obligations as conceptualised by actors in a social situation. It is
an expected or free or continuously changing behaviour providing a
given individual social status or social position.
Functionalist and interactionist understandings of society and all
social behaviour, including communicative behaviour (cf. next slide)
• Division of labour - interaction among heterogeneous specialised
positions/roles.
• Roles are occupied by individuals/actors.
• Social roles/behaviours are guided by social norms (expectations,
degree of conventionality).
• Legitimate and constructive roles – anticipation of rewards (vs.
punishments)
• Changed social conditions role change.
External situation
Time,
place,
”noise”
Channels.
media
Code: usage
Persons:
goals,
relationships
Topic/aspect
Processing:
inputoutput
Internal
situaton
Dynamics
of situated
interaction
Fischer et al., Guidelines for task-based university
language testing. Graz: ECML, 2011, p. 21.
Puren, C. ( 2008). Formes pratiques de cominaision entre
perspectives actionnelle et approche communicative: analyse
comparative des trois manuels. Langue Modernes. Cited in Fischer
et al., 2011, p. 22. (note the inclusion of roles!)
Testing/assessment – issues and problems
• Testing/assessment is notoriuosly demanding/
complex
• Testing/assessment should fulfil many criteria –
(a) there are many things that should be got right
• (b) there may be some things that one may not
know well enough (inevitable degree of
ignorance/ even ineptitude?)
• (c) there may be serious consequences.
-> Can testing/assessment be made less demanding,
less complex, and exclude avoidable problems/
errors?
Assessment – So Many Things to Get Right
• Observing aspects of good practice <->
learning to first avoid less good practices
• For instance, in learning to rate, avoiding
Halo (horn) effect
Central tendency effect
Restriction of range effect
Severity or leniency
What about using Checklists?
• Checklists: Everyday use – shopping lists
• Checklists/Inventories – have a long history in eg.
personality inventories (self-assessment;
preferences).
• Checklists – frequent use esp. in health care/
nursing, aviation/pilots, building industry….
• Gawande, A. (2010). The checklist manifesto.
How to get things right. London: Profile Books.
• Western Michigan Checklist Project (www.xxx)
• Scriven, M. (2000/2007). The Logic and
Methodology of Checklists. Western Michigan
University.
Reluctance to use checklists - objections
It somehow feels beneath us to use checklists, an embarrassment. It runs counter to the deeply held beliefs
about the how truly great among us – those we aspire to be
– handle situations of high stakes and complexity. The truly
great are daring. They improvise. They do not have
protocols and checklists. Maybe our idea of heroism need
updating. (Gawande, p. 173)
Checklists – boring -> Professionalism – expectations:
selflessness, skill, trustworthiness, discipline. We are not
built for discipline (novelty, excitement). We have to work at
discipline. (p. 182-3).
What about education?
Checklists are valuable evaluation devices when carefully
developed, validated, and applied.
A sound evaluation checklist
• clarifies the criteria that at least should be considered
when evaluating something in a particular area
• aids the evaluator not to forget important criteria
• enhances the assessment's objectivity, credibility, and
reproducibility.
Moreover, such a checklist is useful in planning, monitoring
and guiding operations, and assessing their outcomes. In
the evaluation vernacular, checklists are useful for both
formative and summative evaluations.
(Scriven, M. The Logic and Methdology of Checklists. Western Michigan University.
2000, 2007)
Checklists are not created equal – more bad
than good checklists?
• Bad checklists: vague, imprecise, too long,
hard to use, impractical
• Good checklists: precise, efficient, to the
point, easy to use, provide reminders (not
exhaustive), practical. Usually 5- 9 points
only.
Types of Checklists
• List (mnemonic) – eg. shopping list; laundry list –
grouping important
• Sequential checklist - order matters: strongly
sequential kind, where the sequencing (of some
or all checkpoints) must be followed in order to
get valid results.
• A weakly sequential checklist: order is of some
importance, but for psychological or efficiency
reasons rather than from logical or physical
necessity
Iterative checklist is sequential, in whole or part, but
requires—or may require—multiple passes in order to reach
a stable reading on each checkpoint.
Diagnostic checklist that is used—for example— by
taxonomists, mechanics, and toxicologists. It typically
supports a classificatory kind of conclusion—one that may
be descriptive or evaluative.
Probably the most important kind of checklist for evaluation
purposes is the criteria of merit checklist ( COMlist).
This is what judges use when rating entries in a skating or
barbeque or farm produce competition (and, of course,
when rating language performance).
Key Requirements for Comlists /Criteria of Merit)
Most of the following are self-explanatory and refer to the
criteria or checkpoints that make up a comlist:
1. The checkpoints should refer to criteria and not mere
indicators.
2. The list should be complete (no significant omissions). The
items should be contiguous, i.e., nonoverlapping (essential if
the list is used for scoring).
3. The criteria should be commensurable.
4. The criteria should be clear (comprehensible, applicable).
5. The list should be concise (to assist its mnemonic function);
i.e., it should contain no superfluous criteria.
6. The criteria should be confirmable (e.g., measurable or
reliably inferrable)
(Scriven, 200, 2007)
Positive developments
• Lack of knowledge /ignorance diminishing:
reasonable guidelines of good practice.
• Transparency is making some headway:
useful research eg., on linking exams to the
CEFR is being carried out and increasingly
reported.
• Training on language testing/assessment is
increasingly available (EALTA, ALTE).
• Thus, we have increasingly better chances
to get things right,