BILC Standardization
Initiatives and
Conference Objectives
Peggy Garza
Associate BILC Secretary
For Testing Programs
Outline
•
Standardization Efforts
- Language Testing Seminars
- Tailored Workshops
- Toolbox for Language Testers
- STANAG 6001 Testing Conferences
•
Conference Topics and Goals
BILC Testing Seminars
• Language Testing Seminar (LTS)
• Advanced Language Testing Seminar (ALTS)
• Language Standards and Assessment Seminar (LSAS)
LANGUAGE TESTING SEMINAR
(LTS)
Language Testing Seminar (LTS)
• Developed and conducted by BILC testing experts
• Background: A BILC initiative aimed at promoting a
common interpretation of STANAG 6001 and
consistency in language proficiency testing
• Specific objective: To give NATO/PfP language
teaching professionals the opportunity to practice
developing test items based on STANAG 6001
Language Testing Seminar (LTS)
• Held since Autumn 2000 at the Marshall Center,
Garmisch, Germany
• Two-week graduate-level seminar
• Offered twice times a year (February and September)
• More than 350 graduates from NATO/PfP/MD nations
ADVANCED LANGUAGE
TESTING SEMINAR
(ALTS)
Advanced Language Testing Seminar
(ALTS)
• Need for continuing
professional development for
test developers identified
•
Stakeholders’ meeting in
September 2008
•
Result: Advanced Language
Testing Seminar (ALTS)
•
Validated in 2009
Advanced Language Testing Seminar
(ALTS)
• Target audience: LTS graduates who continue to
work on STANAG 6001 testing
- STANAG 6001 Edition 4 standardization
sessions
- Focus on Level 3 for practical exercises
- Opportunity to exchange best practices
• Modular approach to seminar
Advanced Language Testing Seminar
(ALTS)
• Pre-seminar assignments
- Approximately 40 hours
• Three-week seminar
- Module 1: Receptive Skills: Listening and Reading (1 week)
- Module 2: Productive Skills: Speaking and Writing (1 week)
- Module 3: Test Analysis (3 days)
- Module 4: Managerial Issues (2 days)
NEW
LANGUAGE STANDARDS
AND ASSESSMENT SEMINAR
(LSAS)
Language Standards and Assessment
Seminar (LSAS)
3-11 July 2013
• For language teachers and program managers
• Objectives
–
–
–
–
STANAG 6001 familiarization
Guidelines for aligning instruction to standards
Test and measurement principles
Classroom assessment and practical exercises
Testing Seminar Schedule in 2014
• Language Testing Seminar (LTS)
‒ 10-21 February
‒ 15-26 September
• Advanced Language Testing Seminar (ALTS)
‒ 2- 20 June
• Language Standards and Assessment Seminar (LSAS)
‒ 23 Apr – 1 May
‒ 3-11 December
STANAG 6001 Toolbox
for Testers
• Recommended by ALTS participants
– Samples of listening and reading items at levels 1-3
– Samples of speaking and writing tests at levels 1-3,
including plus levels
– Frequently Asked Questions (FAQs)
– Self-Assessment Outline for STANAG 6001
Testing Programs
BILC Website
www.natobilc.org
Tailored Workshops for
National Testing Programs
Tailored Workshops
• Ukraine
‒ November 2011, 2012
• Moldova
‒ May 2012
• Russia
‒ June 2013
STANAG 6001
Testing Conferences
• Zagreb, 2009
– Focus on standardization, moderation, and validation
• Sarajevo, 2010
– Dates: 31 August – 2 September
– Focus on testing speaking
• Stockholm, 2011
– Dates: 6-8 September
– Focus on testing writing
• Copenhagen, 2012
– Dates: 3-6 September, 2012
– Focus on testing listening
2013 STANAG 6001 Testing Conference
• Some of your questions:
– Our stakeholders have questions. Is there advice for
how best to communicate with the stakeholders?
– Our stakeholders are asking us to test more and more
people. What should we tell them?
– We have developed items. What’s next?
– How do we know if our tests are good?
– We need to hire new testers. Is there a position
description for testers? How can we recruit for
testers?
– How can we keep professionally current ?
2013 STANAG 6001 Testing Conference
• Conference format
– Presentations
– Guest speakers from NATO and SHAPE
– Panel discussions
o Resolving challenges
– Study Groups meetings and reports
o Making a validity argument
o Washback
o Action plans
o Renorming and standardizing testers
o Piloting items and prompts
Specific Conference Goals
• Review and clarify the entire test construction process
• Identify action plans for test development and
administration
• Provide ideas for selecting the appropriate testing
techniques
• Discuss piloting and pretesting
• Learn more about determining test validity
• Identify practicality issues and discuss lessons learned
• Discuss best practices in renorming and standardizing
testers
• Learn about innovations in testing in the BILC community
– Computer-delivered testing
– Proficiency and specialist testing
– Testing at Level 4
Test Life Cycle Diagram
Communicating with the
Stakeholders
• Identify the stakeholders
• Coordinate with the stakeholders during the needs
analysis and planning stages
– Stakeholders approve personnel and resources
• Close the loop with the stakeholders during the Test
Life Cycle
–
–
–
–
Action plans
Technical reports
Updates
Washback
Test Construction
Stage 1
Needs Analysis
Stage 2
Planning
Stage 3
Design
Stage 4
Test Specifications
Development
Revision
Pre-testing / Analysis
Standard Setting
Evaluation/review
Stage 5
Administration
Stage 6
Monitoring
Evaluation
Development Stage
Part 1
Text selection
Item writing
Item review I (moderating)
Piloting on a small scale
• Retain
Item review II (editing)
• Revise
• Reject
Assembly of trial test
More items
needed?
Development Stage
Part 2
Pre-testing/trialling on a larger scale
Statistical Analyses
• Retain
Item revision
• Revise
• Reject
Standard Setting
Item banking
Assembly of final version of test
2013 STANAG 6001 Testing Conference
• Expectations
–
–
–
–
Active participation and interaction
Frank and open discussions
Provide answers to the difficult questions
Create a forum for future networking and
cooperation
• BILC appreciates your participation and we look
forward to a very productive conference!