The Test of the Future
Using Computers in High Stakes Tests:
How Far Can We Go?
Steven Bakker, dutchTest
Trends in Standards and
Assessment in Europe
• Shift from input-oriented standards to outputoriented standards
–
–
–
–
More autonomy to schools
Development of outcome-based national curricula
Change in role of inspectorates
Growing importance of assessing content and skills
that are not curriculum-specific
Trends in Standards and
Assessment in Europe (2)
• Shift from teaching subject-based facts to
developing general skills
– Less rote-learning of facts, more application of
knowledge in every-day situations
– Less testing of factual knowledge, more assessment
of productive skills
– Introduction performance-based testing and
portfolio assessment
Trends in Standards and
Assessment in Europe (3)
• Policy efforts to increase level of education
(Lissabon!)
– More students applying to general higher education
– More need for effective selection and placement
instruments
– Emphasis on predictive validity of tests
Trends in Education in Eastern
Europe (4)
• Increasing need to regulate university- or
department-based entrance tests
– Slovenia: entrance tests replaced by Matura
– Georgia: entrance tests replaced by Unified
Entrance Exam
– Russia: introduction Unified Entrance Exam
– Ukraine: experimental general admission exam
introduced
Trends in Education in Western
Europe (4)
• Increasing needs to introduce more selective
admission instruments
– Germany: centralized administrative procedure for
student placement replaced by admission by
universities which may include testing
– Netherlands: experiments with special admission
procedures for highly talented students
– UK: debate on lack of selectivity of A-levels and
introduction special university admission tests (e.g.
Law School Admission Test)
These trends lead to
•
•
•
•
•
More large-scale testing
More objective testing
More testing in real-to-life situations
More students to be tested
Higher demands to predictive validity
End hence a growing
pressure on the system
Do we have a view of the future?
What we would like the future to
be…..
…and how far can we go?
Scope of this presentation
• List what already has been achieved on our
way to the future
• Highlight main challenges along the road
What do we want IT for in large
scale testing?
More flexibility, more
administrations per
year
-Better management
of production:
standardization and
more efficient use of
resources
-Broadening the
knowledge and skills
assessed
-Widen range of
assessment methods
Less time between
administration and
reporting of results
Fewer human
markers
AND:
Better or same
reliability, validity,
efficiency,
acceptability
Enhancing quality
of marking
More efficient data
collection
Cleaner data
Introduction CBT in the Netherlands: highstakes achievements test on many levels
Secondary Education: three main streams:
• vmbo
– Leading to vocational education.
– Exams at age 16+
• havo
– Leading to higher professional education (polytechnics)
– Exams at age 17+
• vwo
– Leading to academic education
– Exams at age 18+
Introduction CBT in the
Netherlands: three variants
• Digital exams
– Items on screen
– Student answers on screen
– No paper
• Mixed mode
– Stimuli and items on screen
– Student answers on paper
• ‘Compex’
– For 30% of the items a computer is needed to run subjectspecific applications; the rest is PBT
Introduction CBT in the
Netherlands: computer will be used
on all levels
Implementation plan
2001-2007
•
Pilots on all levels
2006
•
vmbo
–
–
Digital exams for a number of general subjects
Mixed mode for Dance, Drama and Music
2007
•
havo and vwo
–
Compex for a number of subjects (physics, biology, geography, economy)
…. 2012
•
Administration on line, school server or central server??
•
More flexibility in administration
•
vmbo: CBT only; havo/vwo cbt for a number of exams, no mixed mode, compex
integrated in cbt
Digital exam: vmbo French
Click ‘Ecole’ to find out more
about Collège Ste-Madeleine
Digital exam: vmbo French listening
skills
Students are
instructed to click
on the names of
two pupils of the
school, Zeinab
and Bilal. They
then hear Zeinab
and Bilal talking
about themselves
and their school.
The students
answer listening
questions by
clicking options
Which of the statemments
about Zeinab is true? There
are several correct answers
Digital exam: vmbo French: Reading
and writing skills in a chatting
session
The student now
has a chat
session with
Zeinab. He/she
has to answer in
French. The
session is fully
simulated,
including Zeinabs
answers
appearing after a
few seconds!
Hello, this is Zeinab. My teacher gave me
your address. Who are you?
Tell her who you are and what your
nationality is
Digital exam: vmbo French: Reading
and writing skills in a chatting
session
The student is
instructed to click
‘Excursions’ A page is
shown with a report
students from the
French school wrote
about their visit to a
zoo. There are some
reading comprehension questions
The question is about
the entire text
What happened
at which time?
Digital exam: vmbo French:
Listening skills in an interview
Bilal and Zeinab are
interviewing the train
manager. Bilal asks:
what do you do when
you are in Amsterdam?
The student is supposed to play the interview and
to answer listening comprehension questions
Digital exam: vmbo French:
Functional writing in an e-mail
The French students
returned home. As
agreed you write an email message to Bilal.
Start with ‘Dear… ‘
and end with
‘Greetings’. If you
move over an
underlined word, a
translation will pop up.
Press ‘Help’ to see
how to type accents
such as é or à
Include the following issues in your
e-mail:
1. Ask how he is doing and if he
and his friends had a good
journey
2. Tell him that your pictures will
be ready today and that they
are beautiful, especially those
from Amsterdam
Using a computer while taking an exam:
PBT + subject specific application on
computer
Using a computer while taking an
exam: stimuli on screen, answer on
paper
Items following the Yulduz
Usmanova video clip
•In 1998, Usmanova reached the status of mega-star in her own
country.
•Why is this understandable considering the political situation?
Mention two reasons.
•Not only in the music, but also in the Female Factory theatre
show, the Ethno-crossover element is present.
•Name two aspects of Yulduz Usmanova’s dance movements
that are of a non-western (traditional) nature.
•Mention two show elements that are important to the way the
show is staged.
•Name two reasons why Usmanova’s music is not part of the
Western mass culture.
What is in place already?
Institutional: Web based
registration, Institutional
VPN’s
Commercial Test Centers:
e.g., Prometric, Lamark
Automated
marking: OMR, Erater, C-rater
IT-facilitated: OSN
iBT delivery TOEFL test
Institutional systems:
TCS, ECD (ETS), ThPro
(CITO)
Commercial systems:
e.g., Qmark, Examsoft
Instant feedback (scores,
pass/fail), e.g., ECDL
Data on web: NAEP
Many institutional
and commercial
products available
Observations
• IT has already pervaded and changed all
aspects of large scale high stakes testing
• Integration of IT applications for different
aspects is far from trivial
• Bottleneck: test delivery
– Security
– Lack of appropriate infrastructure
– Costs
Some wider thoughts
• Is ‘computerizing’ exams, as we know
them now, the road to the future, or should
we abandon the idea of traditional exams?
• If it is
– How do we solve large scale access and
security problems?
– Will such solutions still be acceptable for all
stakeholders?
– Will they be cost efficient?
In other words: How far can we
go?
Or is it time to press the reset
button?
Thank You!