The Testing Industry
Its Evolution and Future Directions
Louis Woodruff
NCTA Conference
San Antonio Texas
August 7, 2009
Texas is as large as all of New England
New York, Pennsylvania, Ohio
and Illinois combined
San Antonio Texas
Texas includes 267,339 square miles.
7.4% of the nation's total area
There are more than 70,000 miles
of highways in Texas
The world’s largest parking lot
is located at DFW Airport
The Airport is larger than Manhattan
Texas is home to Dell and Compaq
Central Texas is often referred to as
the Silicon Valley of the South
The name “Texas” comes from an
Indian word meaning friends or allies
Texas was an independent nation
from 1836 to 1845
Texas is the only State
to enter the nation by treaty
instead of annexation
The last battle of the Civil War
was fought in Texas
It was a Confederate victory
That same year…1865…forty years
before the Wright brother’s historic flight
a Texan flew a coil-spring-powered airplane
The craft soared to tree-top heights
before crashing into a henhouse
killing several chickens and
scaring many children
Rodeo is the official Texas State sport
World's 1st Rodeo was in Pecos Texas
July 4, 1883
Just two years after the 1st Rodeo
Dr Pepper was invented in Waco
There is no period after the Dr in Dr Pepper
75% of the world's Snickers bars
are made in Waco
The
average
American
eats
More
popcorn
is sold
in Dallas, per
capita,
than anywhere
in theannually!!
U.S.
42 quarts
of popcorn
The first word spoken from the moon
on July 20, 1969
was
"Houston"
The armadillo is the official State mammal
In Texas, for $150 you can become
a licensed dead animal hauler
In Texas it's illegal to put graffiti
on someone else's cow
There are 18 million people in Texas
and 16 million cattle
In “Texan” the plural of y’all . . . is all y’all
Texas is a national leader in the alignment
of elementary and secondary
curricula and assessment
Texas Association of Collegiate Testing Personnel
Founded 1988
Administration of educational and psychological testing and
assessment programs at Texas colleges and universities
Promote, implement, and evaluate professional standards
Disseminate information to professional testing personnel
Promote continuing education and foster public understanding
The Testing Industry
How Big is it?
Testing Professional
Not Your College Major
Not Your Original Plan
The Testing Industry
Recently Reported Size
of the Testing Market
• in 1955 - $7,000,000
• in 1997 - $253,000,000
• in 2007 - $700,000,000
The Testing Industry
Recently Reported Size
of the Testing Market
$700,000,000
True
X
False
The Testing Industry
Recently Reported Size
of the Testing Market
Educational Testing Service
$700,000,000 - Reported Annual Cash Flow
$50,000,000 - Contract in California
The Testing Industry
Recently Reported Size
of the Testing Market
$700,000,000 ??
Likely more than triple that size . . .
. . . and growing ! ! !
The Testing Industry
Multi-Billion Dollar Industry
Large Diversified Market
Supporting Technologies
Future Growth Potential
Low Margins
Losses
Limited
Capital
Complex
and
Costly
Development
Diminishing
Economies
of Scale
Expensive
Demanding
Infra
Structure
Low Margins
Losses
Limited
Capital
HOW DID WE
Diminishing
Economies
of Scale
GET INTO
THIS SITUATION ?
Complex
and
Costly
Development
Expensive
Demanding
Infra
Structure
What were the early forces that shaped
the modern standardized testing market?
What role has Academia (the colleges)
played in the evolution of the Industry?
What market forces & business dynamics
are driving the Testing Industry today?
Are there lessons to be learned from the
past that can be applied in the present
to shape a better future for our Industry?
In 1845 Boston schools applied
“uniform” testing for all students
through oral examinations.
In 1874 the superintendent
in Portland Maine, created
a uniform curriculum with “standardized” tests.
He published the scores.
Soon after that
he was removed!
An important lesson
for those seeking change!
A well intentioned action, ahead of its time,
may be throttled and pounded into extinction.
OK…but are there any business lessons
from the past for the testing industry?
There is a lesson from the past
for today’s Testing Industry . . .
to be learned from the
history of the #2 pencil !
Slates and chalk continued to be
used after the pencil was invented
due to high costs for the manual
production of pencils
In 1866 a patent was granted
for a wood-plane machine
enabling mass production . . .
This enabled the unit
cost of the pencil to
be lowered . . .
. . . the “new technology”
(the pencil) was soon
embraced in the
marketplace
Within 4 years over
20 million pencils
were sold
By 1903
the eraser
was added
and the cost
dropped to
a penny
. . . this is the same situation the testing
industry has repeatedly confronted . . .
Having the technology is not enough.
Having what the
market needs is
not enough.
Through mass
production and
robust distribution
channels, manufacturing
has to provide the market
the “right” unit cost !
Market
Need
Low Unit
Price
Distribution
Channels
Funding
Sources
Business
Mandate
Production
Capability
Viable
Product
Enabling
Technology
Before we look at how the testing industry
has handled this business mandate . . .
we should first look at the forces that
gave birth to today’s testing industry?
Today’s testing industry
grew out of Academia in
pursuit of scientific missions!
The College Entrance Examination
Board published subject syllabi for
schools to teach and colleges to test.
Lewis Terman at Stanford created the
Intelligence Quotient (IQ) by dividing
mental age by chronological age.
Edward L Thorndike applied the
scientific method to the construction
of tests to measure academic
performance.
Within two decades multiple
standardized tests were in use.
These tests were developed by
Academics within the Universities.
Those Universities supported the
costs to develop these tests as well
as the cost to bring them to market.
The Stanford Achievement Tests were
created to assess subject knowledge.
The Scholastic Aptitude
Test was introduced.
The California
Achievement Test
was published.
Researchers at University of Iowa
developed the Iowa Test of Basic Skills.
Universities and the College
Board were operating their own
commercial testing business.
However, hand scoring of answer sheets
and manual processing of score reports
kept costs high and limited use.
By the mid-1930’s the fledgling industry
was reaching its capacity for growth.
The testing industry, birthed through Academia,
Measure
was
facing the same challenge faced by the #2 pencil !
Academic
Achievement
The Universities
Scientifically Based
Academic Assessments
Still Needed:
Enabling Technology
Academia’s
Business Challenge
Production Capability
Distribution Channels
It was the business sector
that developed the
Enabling Technology
required to drive things forward
In 1938 IBM created the first electronic
graphite/optical test scoring machine.
$
$
$
$
In 1938 IBM created the first electronic
graphite/optical test scoring machine.
While this financially helped Academia’s
testing initiative, the required business
mandate remained incomplete.
Enhanced Production Capability and
improved Distribution Channels were
still needed to complete “The Model”.
The military funded development
of new assessments to support
recruitment and the war effort.
Following the war, the GI Bill provided
money for returning soldiers to further
their education and attend college.
This placed a burden on the nation’s colleges
who had no means for assessing the
readiness of these non-traditional students.
ACT
wasof
founded
aUniversities
second
The
College
Boardas
asked
its
VP
decentralized
testing
services
A group
influential
non-profit
testing
organization
to
to
develop
a the
testing
operation
could
not handle
the
demands
of
approached
College
Boardto
new
demands.
meet
this
new market
national
demand.
such the
an admissions
challenge.
level
admissions
playing
field.
American Council
on Education
The Chauncey Farm
The Carnegie
Foundation
The testing industry, centered in
these large non-profit institutions,
both flourished and profited !
Market
Need
Low Unit
Price
Funding
Sources
Private andDemand
Public
Increasing
Automated
Scanning
Valid and Reliable Tests
Funding
for Assessment
Scoring
and Reporting
Distribution
Channels
Production
Capability
Viable
Product
Enabling
Technology
Norm-Referenced Test
% of Reusable Test Items
Measures performance relative to a
representative national sample of
examinees called “norming sample”
Test Item Database
Reconstructed Test Forms
Large National Volume
Test Item
Statistics
Test Form
Equating
Test Design
Validity &
Reliability
Test Item
Rotation
Market
Need
Funding
Sources
Low Unit
Price
Distribution
Channels
Viable
Product
Production
Capability
Enabling
Technology
Distribution
Channels
Labor for
Test Delivery
International
Partners
Partnerships for low cost services and facilities
Market
Need
Low Unit
Price
Funding
Sources
National
Norm Referenced
Paper & Pencil Testing
Distribution
Channels
Production
Capability
Viable
Product
Enabling
Technology
$
$
$
$
$
$
$
$
$
What
changed
thethe
testing
industry
So then
why are
vendors
$
$
from
a non-profit
focus
in today’s
Testingservice
Industry
not
to
a for-profit
commercial
focus?
making
substantial
profits?
$
$
$
$
$
$
 Service Issues
 Processing Delays
 Delayed Reporting
 Increasing Costs
 Shipping Problems
 Scoring Errors
How did the industry get into this situation?
Is there a way out of it?
CBT significantly improved
the test product . . . but it
limited production capabilities, required
How did the industry get into this situation?
a new distribution infrastructure and
raised operational and development costs
The delivery advantage of using schools
(inexpensive classrooms for large group testing)
lost its business value for CBT.
Store-front commercial outlets
as a new distribution channel.
CBT altered the business dynamics in the testing
industry by reducing profit margins, expanding
competition, and raising client expectations
beyond what the vendors could profitably deliver
deliver.
Even internet delivery did not resolve things
Cost savings have been minimal
Additional competitors emerged
Test development and overall
operating costs remain higher
than traditional P&P testing
Internet
Delivery
http://www
CBT
Low Unit
Price
Market
Need
Funding
Sources
CBT-IBT
disrupted the
dynamics of the P&P
business model
Distribution
Channels
Production
Capability
Viable
Product
Enabling
Technology
Just eight years ago it appeared
that things were about to change!
The public schools could not support
CBT for such large volumes, leaving
P&P delivery as the most viable option.
The “No-Child Left Behind” legislation
gave the Testing Industry hope for a
return to the profitable P&P model.
An expanded paper & pencil
market could provide the
foundation to support the
CBT side of the business !
The Industry leaders scrambled to
win the State contracts to provide
this federally mandated testing,
expecting significant future profits.




Low cost P&P delivery
Less costly P&P test development
High test volumes
Economies of scale
What turned the NCLB golden goose
into a turkey for the Testing Industry?
Norm-Referenced Tests
Measure performance relative to a
representative national sample of
examinees called “norming sample”
Individual State specs
Lower per test volumes
State owned test items
Criterion-Referenced Tests
Measure performance relative
to mastery of content at certain
pre-defined levels of proficiency
All of this
impacts
all of us!
Short reporting window
Delivery by the client
Low economies of scale
Increased competition
NCLB testing, even with over 33,000,000
tests projected for this year, continues
to be a losing venture for many vendors.
At the same time, CBT & IBT continue
to stress the industry with increasing
client demands, costly operations,
and expanding competition.
Is there a silver bullet that can give the
Testing Industry the profitability needed
to remain stable and effective?
Is there some new technology that will
again catapult the Industry forward
into a new and promising paradigm?
Four Requirements to Promote
this New Test Delivery Paradigm
Competition must drive down the cost of Laptops.
Wireless Technology must make affordable advances
in security and control.
Demand for hand-held full function computers
must reach mass production levels.
Education must integrate personal hand-held
wireless computers into the standard curriculum.
Data Source
Test data download
via the internet
Laptop Server Hub
Host Server
Host and Laptop Servers
stored in movable cabinet
Laptop #3
Laptop #4
Test Delivery Unit #4
Test Delivery
Unit Server
#3
Laptop
Test Delivery
Unit Server
#2 + IDD’s
Laptop
Test Delivery
Unit Server
#1 + IDD’s
Laptop
+ IDD’s
Laptop Server
+ IDD’s
After testing in classrooms
laptop servers returned to
Host for test data upload.
Laptop #1
Laptop #2
Laptop servers & Individual
Display Devices transported
in mobile Test Delivery Units
A Possible Model
Testing Room #4
Test Server #1
Testing Room #3
Test Server
#4
(laptop)
Testing Room #2
(laptop)
Server
TestingTest
Room
#1 #1
Test (laptop)
Server #1
Test Server
#1
(laptop)
(laptop)
A Test Delivery Paradigm Shift
Old Model in a New Dimension
Returning the Test Delivery
Business Advantage
to Partnerships
with Schools
Can we really know how our Industry
will evolve over the next 20 years?
How might technology shape the journey ?
What changes may lie ahead ?
Confirming Examinee Identity
Continuously During Testing
Computerized
Paper
Immediate
On-Site
Answer Sheet
and
Essay Scoring
Computerized
“Test Coat”
for continuous
electronic
monitoring
Quantum Computing Enabled
Photon State Encoding
Current data encryption algorithms rely
on computational schema for protection
and can be deciphered, given sufficient
computing power, knowledge and time.
The Heisenberg Uncertainty Principle
It is impossible to discover both
the momentum and position of a
particle at any given instant in time.
Therefore, data encryption using a
particle in motion as a key could
never be deciphered.
Quantum Computing Enabled
Photon State Encoding
(particle motion as the encoding key)
Data Security
No Longer a Concern !
Projection of a laser generated
3-D image suspended in space
A future test center with
individualized testing
environments supporting
holographic images as
interactive test stimuli.
A Virtual Testing Environment
In the future we may be able to
reach total standardization through
the management of each examinee’s
perceptions during the testing event.
Will future advances in brain mapping
and Psychopharmacology enable
selected modifications of the brain’s
perceptions to achieve absolute virtual
standardization - from the inside out?
Virtual Testing Environment
Transmitting an Ideal, Standardized,
Virtual, Personal Testing Space,
Integrated with the Test
Standardization
as an
Altered State of
Consciousness !
the road ahead
may look lonely
and at times we
may feel unsure
not knowing
which turn to take
We will all need help
to navigate through
the unknowns ahead
ITS ALWAYS BETTER
WITH A FRIEND !
ITS ALWAYS BETTER
WITH A FRIEND !
The Testing Industry
Its Evolution and
Future Directions
Contact:
woodruff@bucks.edu
215-968-8467
Louis Woodruff
NCTA Conference
San Antonio Texas
August 7, 2009