History, Current Practice
and Perspectives, and
Predictions for the Future of
Computer Based
Assessment in K-12
Education
John Poggio
University of Kansas
(Ancient) History circa…
the 1970s, ’80s & ’90s…

Impersonal, frightens students, computer anxiety,
lack of familiarity, equivalence to P&P at best,

“a dramatically different experience that
guarantees the two modalities are not equivalent”

“CAT will restrict problem solving and only
support solutions in a strict linear manner” …

However, study as early as 1972 through the turn
of the century research found no differences…
Advantages
 Immediacy
of results
Among the most notable features
associated with computerized testing is
the immediacy of results. In most systems
as soon as a student finishes taking the
test, scores are ready for display. Delays
and slowness in reporting results can be a
concern of the past.
Advantages
 Cost
efficiency
Moving tests to computer delivery will
signal a reduction in costs. While upon
first consideration this may not seem
obvious, production of test booklets and
answer sheets are extremely large
expenditure items that can be set aside.
Advantages

Accuracy in data and data collection
Capturing information from students on answer
sheets is among the biggest NEGATIVES of the
testing process. A casual review of information
gathered on answer sheets reveals the
considerable number of errors that students
make. Answers to questions and the information
provided, such as gender, race, age, and even
name are frequently miscoded and mis-marked
by students. Computer-based assessment all but
eliminates this error in the measurement
process.
Advantages
 Student
motivation
A number of researchers and field practitioners
have commented on heightened level of student
motivation when assessments are taken online.
This in turn translates into higher performance.
Whether such motivation will be seen over
time is not clear. However, at this time,
students appear more attentive, comfortable,
and responsive to computer rather than paper
and pencil assessment.
Advantages
 Adaptive
alternative
Moving traditional tests to computer
administration has opened the door to
computer adaptive testing. This approach has
the advantage of reducing traditional tests by
half to as much as 75% of the original length. In
this methodology, while we have considerable
reduction in test length, there is relatively little
loss in test properties as validity and reliability
(…and, assessments may be fairer!)
Advantages
 Reduced
administrator and
instructor effort
As the tests will be administered online, the
traditional activities by administrators and
teachers receiving, unpacking, securing, counting,
sorting, and distributing test booklets and
answer sheets is eliminated. This management
effort that requires considerable commitment
of time and attention is all but eliminated when
we go to computerized testing.
Advantages
 Meeting
the needs of special
populations
Technological advances used with computerized
testing signals the dawn of a new era in testing for
students with special learning needs and other
populations (non native speakers). Largely
unstandardized accommodations of the past can
now be transformed into common and fixed
procedures, for example, increasing text size with
the click of a button, transforming text to speech,
zooming, colorizing objects, English/native
language translations, ASL avatars for the hearing
impaired, etc.
Advantages
 Ready
support for data based
decision making
The speed with which data can come available to
educators allows rapid analysis and evaluation of
performance. Whereas in the past it could take
three months for data to be returned to
administrators and teachers, today’s
computerized tests allow educators to review
results and while the information is fresh and
current make instructional and evaluative
decisions on behalf of students.
Advantages
 Store
information, ready access
to results, and create archives
As students take a test on a computer, the
storage and retrieval of information is efficient
and speedy. In the past reliance on answer
sheets meant data cleanup, scanning and
processing, scoring and check up, and eventual
readying of information for storage and retrieval.
But with computer-based testing, the storage of
information is part of the process and access of
that information is efficient and accurate.
Advantages

Ability to modify tests as necessary
In the past, when an error was discovered in a
test booklet, the educator had little recourse
but to stand by and watch the mistake time
after time during administration. In today's
computerized tests when an error is found it
can be corrected immediately. In this way, test
scores are improved for many students.
Advantages
 Improved
security … some risks
As tests are delivered online, traditional
handling of test materials is eliminated.
Breaches of test security are fewer. There
are threats related to hacking that are not
avoided. However, up to this time there
have been few reports of security issues
because of computerized administration.
Advantages

Increased opportunity for collection
of supplemental information or data
Computerized testing allows for and
supports the collection of additional
information. Students can be quickly and
easily surveyed or additional data can be
gathered. Research and evaluation can
have a place in assessment once more!
Advantages
Equal
(..Stronger?) Performance
Research of the past 25 years is clear and
certain: paper and pencil testing yields
scores equivalent to computer-based testing.
We should continue to study these effects.
But, the real effect to watch is, with student
gaining CBT experience, will CBT scores
become stronger than P&P performance?
Disadvantages:
Implementation Barriers
 Local
resistance to change
Change can come very hard or very slow for
some. This new technology brings with it a
demand for new skills and understandings that
must be mastered. In addition, some will see no
need to change from the way that testing has
been done. Careful, thoughtful steps must be
planned to assist individuals make the transition.
Without assistance and help, change will be very
slow for a few. Individuals must be brought into
the new technology and made to feel
comfortable and be supported.
Disadvantages

Local capacity
A school or school district may not have the
wherewithal to move to online testing.
Equipment may not be current or nonexistent,
or local staff may not have the expertise to
venture into this arena. The inability to move
forward with the innovation will in its own way
limit adoption and understanding. Fear of this
unknown needs your deliberate attention.
Disadvantages
 Risk
– Not changing horses…
These are high stakes tests and failure
may not be tolerated. Careers are on the
line and shifting to a new mode of testing
may not be seen as wise. Again,
awareness and sensitivity are important
considerations; plans should be prepared
with these factors in mind.
Disadvantages
 Minimize
obstacles
or eliminate
Perceived impediments to implementation of
computerized testing will add to frustration
and dissatisfaction. When systems are new to
the teacher and testing is pressing forward
with the threat of accountability, tolerance for
the unexpected will be very low. Make the
systems: work, understandable and
forgiving.
Keys to Future Success

Involve the (state, district and
school) technology staff
Schools must support such personnel,
and they must become involved from the
outset. See to it that they work closely
with educators, and that educators are
comfortable reaching out to them for
assistance and support. Remember:
involve and train all!
Keys to Future Success

Training workshops
Plan training sessions periodically, but
frequently, during the semester - as you
have staff, money and time. Get teachers
working at the same grade and across
grades solving problems together with
technology staff – together! The sessions
can be brief - limited to one or two hours
each.
What of Staffing?

Change in roles and needs...

From P&P staff to...

Programmers and web designers, and

Office staff who are computer literate
and savvy, ...and who have patience and
understanding of others who are lost,
truly lost...
Keys to Future Success

Training materials
Development and utilize student, parent,
and teacher/administrator tutorials,
student training and practice tests, and
implementation guides for teachers and
administrators.
Keys to Success

Communication
Frequent, brief and effective communication
is essential. Share information about the
program and its activities. Constant
communication (weekly newsletters,
announcements, sharing and postings)
...email and the web are your friends!
Build a community of knowledgeable and
involved educators, and share the successes
and failures encountered.
About Cost...

Running “Dual Programs” is very, very
expensive... about twice the cost initially!

Support of the infrastructure to do online
testing must not be shortsighted... It must
be planned for…What you once spent on
paper, the attention of staff, storage and
mailing/ shipping, you will now spend on
servers and programmers!
And now, for some illustrations
of the real thing….
GRE Quantitative…
GRE Vebral…
GRE Writing…
TOEFL CBT Structure
TOEFL CBT Structure
TOEFL CBT Reading
TOEFL CBT Reading
CTB Acuity Reading Comp
CTB Acuity Formative System
CTB Acuity …
ITS Reading…
ITS Short Performance…
AIR Math…
AIR Reading…
AIR Limited CR…
NWEA Math…
NWEA …
NWEA…
Pacific Metrics…
Pacific Metrics Math…
Pacific Metrics Reading…
Pearson Math…
Pearson Reading…
And (of course): Computerized
Assessments and Learning (CAL)…
http://cal-demo.caltesting.org/
CAL Math…
CAL Math…
CAL Math…
CAL Review…
CAL Reading…
CAL Reading…
Computer-Based
Assessment:
Future Considerations
Comparability...
Research and Evaluation...Yes,
absolutely...
 but enough is enough!

Is P&P the “Gold Standard”?...
 It must not be!


And, research advises it has not been an
issue… Advice: study the engine, then
generalize…
Tomorrow’s Innovations...

Research directed at teacher, all student,
educator, parent audiences…

Constructed Response formats: yes;
Adaptive: of course; but... AI/ Automated
Scoring, wow!!

Speaking as the response input… and,

Use the technology to link actively
assessment to instruction
Instruct4Success…
Instruct4Success…
Instruct4Success…
Instruct4Success…