AIRS Certification Program
Computer-Based Testing (CBT)
AIRS Webinar: October 2013
The Original Way
• AIRS Certification has been around for over 15
years
• Eligible candidates take a 100 question exam
and acquire either CIRS, CIRS-A or CRS
Certification if they pass the exam
• The exams have always been proctored with
testing taking place in classroom style settings
The Original Way
• And the exam answers have always been
entered and marked using Scantron forms
Problems with the Original Way
• Scantron forms are not the friendliest pieces of
paper. It is a strain on the eyes and requires a
reasonably intense focus
• Although machine readable, AIRS Certification
staff often hand-scored exams if the score was
within 5 of a passing mark or it looked like a
candidate had made an honest mistake in
alignment
• The exam questions themselves were always at
risk as boxes of question papers were sent
around the US and Canada, then returned
Problems with the Original Way
• Providing timely results to candidates was also
a constant challenge – we had to wait for the
boxes to be returned, retrieve the forms, feed
them through the machine, verify the results –
and during busy times, there would be boxes
going in and out every day, and if one person
was off for a couple of days, then the backlog
would build up, etc. etc. …
• Candidates wanted their results within a few
days. Our target was 30 days and sometimes,
we were not even able to meet that …
First Look at Computer-Based Testing
• CBT has been on the AIRS radar for about 6 years
• Original concerns were:
– Expense to members (costs of up to $100 per person)
– Lack of location choices (most operated through
commercial testing centers – less options, even more
expense)
– Concern of impact on Affiliate conferences
– Sense that not everyone was ready for the switch
– AIRS Certification had other priority (from 2008 to
2012, all AIRS Certifications underwent stringent
psychometric review with new questions and new
exams)
Second Look at Computer-Based
Testing
• More software options with lower pricing and
more flexible options
• More unease about Scantron as suitable for
persons with visual impairments
• Survey of AIRS members in 2012 showed that
40% of members had a training room with
computers/Internet
Second Look at Computer-Based
Testing
• Do you know of a 'training-type room' within your community
(for example, an employment center, a library, community
college) that has a number of Internet-capable computers
that could be easily accessed for a CBT Certification exam?
• Yes, and I am aware that either my own or other agencies
have used this space for various reasons = 39%
• Yes, I am aware of such a place but I have never had cause to
contact them = 33%
• No, but I imagine such a space exists and obtaining it should
not be an obstacle = 15%
• No, this may be a problem = 13%
Second Look at Computer-Based
Testing
• Which of the following best describes your
feelings towards Computer-based testing?
• I fully welcome it = 69%
• I welcome it but have some concerns= 27%
• I am concerned about it but appreciate there are some
benefits= 10%
• I prefer the existing system = 4%
Conclusion: Let’s Make the Move to
Computer-Based Testing
Software Selection
• After testing several CBT softwares, AIRS
chose an application called Questionmark
• What we liked about it:
– Admin end of it was relatively easy to use
– Existing clients included US military, major
technology companies, financial services,
government
– Price … honestly, it was the only one that would
not have driven up the cost of Certification for
AIRS members
Software Selection
• Over past 12 months:
– Loading of 800+ questions each with 4 option
choices
– Testing of software by AIRS Certification
Commission members, volunteer agencies,
Affiliates
– Framing and refining of documentation
– More testing
– Bit more testing
Operating for past few months on
Stealth Mode
But now CBT is the only option for
AIRS Certification exams …
BUT THERE ARE OPTIONS
WITHIN THAT OPTION …
CBT: What stays the same
• This is not a low stakes Certification exam.
There is still a requirement for a proctor
• The application process to be approved as a
candidate to take an AIRS Certification exam
remains the same
• AIRS Affiliates still need to approve tests
taking place in their state/regions
• The ‘atmosphere’ and ‘general expectations’
of an exam environment remain the same
CBT: What is different
• An organization wanting to host an exam must
fulfill more detailed criteria (basically, a set
number of seats with reliable computers and
Internet access with technical backup on hand
for start of exam)
• The proctor’s environment changes and
proctor must enter a password before each
candidate can start their exam
CBT: What is different
• The candidate experience is much improved:
– There is one question per page. It is much easier
to read the question and answer options
– Easy to leave questions unanswered and go back
to them later or to answer them and ‘flag’ them
for later review
– You get the results within 5 business days
Three Exam Options
– An exam hosted by an AIRS member (either at
their own facility or at a community-based one)
– An exam proctored by a college or university
– An exam conducted by an online proctor via a
webcam at home or at work
Hosted by AIRS member
• Host must complete an application form (found
via www.airs.org/certification)
• Requirements include:
–
–
–
–
Facility with computers/laptops and Internet
In a quiet setting
Securing a proctor
Having some technical expertise on hand on the day
of the exam just in case
• No additional cost unless required by the host
site to cover specific costs
Hosted by College/University
• Exam can be taken at certain colleges in the
United States (along with a few in Canada)
that are members of the National College
Testing Association (NCTA at www.nctatesting.org
• Find an institution near you and go through
the required steps
• Additional cost usually of around $25 to $30
per person that is paid directly to the college
Hosted by Webcam-based Proctor
• AIRS working with company called ProctorU
• Allows exam to be taken at home or at work (but
in quiet/private setting) as long as you have
webcam and secure Internet
• Cost is $25 but provides flexibility and no travel
costs
So What Does The Exam
Look Like??
• https://ondemand.questionmark.com/deliver
y/perception.php?customerid=399354
AIRS CBT: First Reports
• The work involved in hosting an exam is a bit more
involved for the AIRS member organization but (as
with all other things), it will seem a lot simpler the
second and third times around
• The feedback from candidate has been very positive.
It makes the exam process so much easier
• However, we are working with technology and
although it has not happened yet, something
somewhere will go wrong – be patient and we will
try our best to fix it
Questions
• How come … ?
• What about … ?
• So what if … ?
And when you are ready, go to
www.airs.org/certification
If stuck or confused, email
clivemjones@gmail.com