Workshop 1
Using an Assessment Use Argument to develop language tests and
justify their use
Monday 2 June and Tuesday 3 June
Facilitators:
Lyle Bachman, UCLA,
Daniel Dixon, University of Utah
Adrian Palmer, University of Utah
Introduction
Objectives and activities
Schedule
About the instructors
Introduction
Argument-based approaches to validation have been discussed in the educational
measurement literature for over twenty years and in the language assessment literature for
nearly a decade. One of the most highly developed exponents of this general approach is that
of Bachman and Palmer (2010). They provide a detailed description of an “assessment use
argument” (AUA) that can guide the development of language assessments, along with a set
of procedures that test developers can follow in the development of language assessments and
the justification of their use. Their concept of an AUA is beginning to inform language
assessment development efforts around the world, and is also finding its way into the
language assessment.
Objectives and activities
In this workshop, which will be based on Bachman & Palmer, 2010, Language Assessment in
Practice, we will provide a detailed discussion of the process of assessment justification and
the role of an AUA in this. We will also introduce an on-line tool for implementing Bachman
and Palmer’s approach to language test development, and provide participants with hands-on
practice in its use.
Topics to be discussed will include:
1. The basic premises of our approach to the development and use of language
assessments
2. Accountability
3. Assessment justification
4. The structure of an AUA (claims, warrants)
5. Assessment development and use (Initial planning, Design, Operationalization)
6. Collecting backing for an AUA
7. The structure of the on-line system and illustration of its functionality using an
assessment development project
Activities will include:
1. Presentations by workshop instructors
2. Discussions and explanations of illustrative example assessment development projects
3. Hands-on practice using an on-line tool to:
a. Articulate an AUA
b. Develop a Design Statement
c. Develop a Table of Specification (“Blueprint”)
d. Summary discussions of activities.
Expected outcomes:
1. Understanding of Bachman & Palmer’s approach to assessment development and use:
a. The need for accountability
b. The process of assessment justification
c. The structure of an AUA
2. Understanding of the process of assessment development and use
a. Initial planning
b. Design
c. Operationalization
3. Ability to:
a. Articulate an AUA
b. Use an on-line tool to:
i.
Articulate an AUA
ii.
Prepare a Design Statement
iii. Prepare a table of specifications (“blueprint”)
4. For a particular assessment, participants will have completed enough of each of the
following assessment development products to be able to continue to work with these
products on their own:
a. An AUA
b. A design statement
c. A table of specifications
Each participant should bring:
1. A copy of Bachman, L. F. and Palmer, A. S. (2010). Language assessment in
practice: developing language assessments and justifying their use in the real world.
Oxford University Press.
2. His/her own internet-equipped laptop
3. Information about either:
a. a new assessment he/she would like to develop or
b. an existing assessment for which he/she would like to develop an AUA.
This information will provide the basis for the participant’s hands-on activities during
the workshop.
The workshop instructors will provide participants with any additional materials that may be
needed, and access to an on-line website for use during the workshop.
Schedule
Monday, June 3
Session 1:
9:00 a.m. - 10:30 a.m.
10:30 – 10:50 a.m.
Session 2:
10:50 a.m. - 12:15 p.m.
12:15 – 1:30 p.m.
Session 3:
1:30 p.m. - 3:00 p.m.
3:00 – 3:20
Session 4:
3:20 p.m. - 5:00 p.m.
 Basic premises of our approach to the development and use of language
assessments
 Uses of language tests
 Accountability and the process of assessment justification
 Assessment development and use (Pre-Development Planning, Design,
Operationalization)
BREAK
 Pre-Development Planning
 Presentation
 Example AUA for a test development project from B&P (2010)
 Using an on-line tool for language assessment development
 Presentation
 Participants use on-line tool for :Pre-development Planning for
their projects
 Summary discussion
 The structure of an Assessment Use Argument (AUA) (claims and
warrants)
 Articulating an AUA (Claims 1 and 2)
 Presentation
 Example AUA for a test development project from B&P (2010)
 Participants use the web-based tool to articulate Claims 1 and 2 in
the AUAs for their projects.
 Summary discussion
LUNCH BREAK
 Articulating an AUA (Claims 3 and 4)
 Presentation
 Example AUA for a test development project from B&P (2010)
 Participants use the web-based tool to articulate Claims 3 and 4 in
the AUAs for their projects.
 Summary discussion
BREAK
 The Design Statement
 Presentation
 Example Design Statement for a test development project from
B&P (2010)
 Participants use the web-based tool to prepare the Design
Statements for their projects.
 Summary discussion
Tuesday, June 3
Session 5:
 Review of Day 1 topics and activities
9:00 – 10:30 a.m.
 The Table of Specifications (“Blueprint”)
 Presentation
 Example Blueprint for a test development project from B&P
(2010)
 Participants use the web-based tool to prepare the Blueprints for
their projects.
 Summary discussion
10:30 – 10:50 a.m.
BREAK
 The Table of Specifications (Continued)
 Participants use the web-based tool to prepare the Blueprints for
10:50 a.m. - 12.15 p.m.
their projects.
 Developing assessment tasks
 Presentation
 Example assessment tasks from a project from B&P (2010)
Session 6:
12:15 – 1:30 p.m.
 LUNCH BREAK
Session 7:
1:30 p.m. - 3:00 p.m.
 Participants develop assessment tasks for their own projects
 Summary discussion
3:00 – 3:20 p.m.
BREAK
Session 8:
 Collecting backing for the AUA
 Wrap-up: summary, feedback
3:20 – 5:00 p.m.
About the instructors
Lyle F. Bachman is Professor Emeritus of Applied Linguistics at the University of
California, Los Angeles. He is a Past President of the American Association for Applied
Linguistics and of the International Language Testing Association. He has received numerous
awards for his research and service to the profession, including the TESOL/Newbury House
Award for Outstanding Research, the Modern Language Association of America’s Kenneth
Mildenberger Award for outstanding research publication (twice, with A. S. Palmer), the
Sage/International Language Testing Association award for the best book published in
language testing (with A. S. Palmer), the Lifetime Achievement Award from the International
Language Testing Association, and the Distinguished Scholarship and Service Award from
the American Association for Applied Linguistics. He has published numerous articles and
books in language testing and other areas of Applied Linguistics. His current research
interests include validation theory, classroom assessment, and epistemological issues in
Applied Linguistics research.
Adrian Palmer teaches a variety of courses on language assessment, teacher training
and professional development, and language teaching methodology, and he regularly offers
workshops and presentations on these topics. Two of his recent books include Seeking the
Heart of Teaching (University of Michigan Press), 2007, and Language Assessment in
Practice (Oxford University Press), 2010. This latter book recently received the 2012
SAGE/ILTA triennial award for the best book in the area of language testing/assessment.
Daniel Dixon completed his Masters in Applied Linguistics at the University of Utah
in December of 2013. In addition to being a graduate student and ESL instructor at the U of
U, he worked with Dr. Adrian Palmer to develop the online Assessment Use Argument
website. He was awarded a Fulbright Student Award and is currently an English Teaching
Assistant at the Universidade Federal de Mato Grosso in Cuiabá, Brazil for the 2014 academic
year. He also plans to use his ten years of experience working as a television news
photojournalist to produce a documentary during his time in Brazil.