Performance
Testing Tools
Survey
Elaine Lopes - Cisco
Julie Timmcke – PTC
Oct 2009
Design & Delivery
Committee


The charter for the Design and Delivery Committee is
to build and share with our valued members a
knowledge base of performance test design,
content development process, and delivery of the
performance tests, thus making the performance
testing adoption easier.
Getting involved
–
–
–
–
Attend the summits
Attend D&D conference calls in between summits
test_design@lists.performancetest.org
Provide critiques on our Request For Comment (RFC)
documents.
Join the Performance Testing Council
Tools Survey
Background Information
•
•
Survey goal: Discover what tools are currently being used for
performance-based testing and identify tools or
functionalities that are needed but not yet available.
Preso goals:
•
•
•
Present survey data as is
Not meant to draw conclusions or make predictions
Numbers:
–
–
–
–
Half (9 out of 18) of the respondents responded on the 1st day
The survey was sent to 715 people
Response rate is 2.5%
40 people started it, 18 completed it – started/completed ratio is
45%
•
•
the length of the survey
technical issue - loop on question 85
Are you a member of the Performance Testing Council?
28.2%
Yes
No
71.8%
Answer Options
Response Percent
Response Count
Yes
28.2%
11
No
71.8%
28
answered question
39
skipped question
0
If you are not a member and would like to receive a high level
summary of the survey results, please provide your contact
information (*your name and email address are required).
Performance Testing Council members will receive detailed
survey results.
Answer Options
Company:
Response
Percent
Response
Count
84.6%
22
answered question
skipped question
26
13
Am. Board of Emergency Medicine
American Board of Anesthesiology
American Society for Quality
Applied Measurement Professionals Inc.
Applied Simulation Technologies
Cisco
FBI
Federation of State Boards of Physical Therapy
FL Nursery, Growers & Landscape Assn
Kaplan IT
LyondellBasell
Microsoft
NAC, NETPDTC, USN
NCIDQ
NKBA
Novell
Red Hat
Sun Microsystems
The Caviart Group
Toolwire
Item Writer/Developer
What is your primary job role?
5.3%
Program Manager
Project Manager
13.2%
31.6%
2.6%
5.3%
2.6%
7.9%
2.6%
Answer Options
Item Writer/Developer
Program Manager
Project Manager
IT Manager
Subject Matter Expert
Psychometrician
Marketing Manager
Quality Assurance/Audit Manager
Software Engineer/Programmer
Director of Testing
Other (please specify)
21.1%
Subject Matter Expert
Psychometrician
Marketing Manager
2.6% 5.3%
Response
Percent
Response
Count
5.3%
13.2%
2.6%
5.3%
2.6%
21.1%
5.3%
2.6%
2.6%
7.9%
31.6%
2
5
1
2
1
8
2
1
1
3
12
answered question
skipped question
IT Manager
38
1
CEO, Chief Scientist
Certification Process Manager
Consultant
Director of Instructional Systems
Design
Executive
Executive Director
Organizational Consultant
Organizational Psychologist
Test Developer
Test Developer and Program
Manager
we don't have a director level role,
but it's closest to what I do
How long have you been in this role?
5.3%
28.9%
Less than 1 year
21.1%
1 year to 3 years
3 years to 7 years
7 years to 10 years
15.8%
Answer Options
Less than 1 year
1 year to 3 years
3 years to 7 years
7 years to 10 years
10 years or more
10 years or more
28.9%
Response
Percent
5.3%
21.1%
28.9%
15.8%
28.9%
answered question
skipped question
Response
Count
2
8
11
6
11
38
1
Considering all exams in your organization's testing
programs, how many test delivery instances are
administered annually (not individual candidates but actual
number of tests taken regardless of outcome)?
3.2%
22.6%
1 to 100
9.7%
101 to 1,200
1,201 to 5,000
12.9%
5,001 to 10,000
10,001 to 25,000
22.6%
22.6%
25,001 to 100,000
100,000 or more
6.5%
Answer Options
Response
Percent
Response
Count
1 to 100
101 to 1,200
1,201 to 5,000
5,001 to 10,000
10,001 to 25,000
25,001 to 100,000
100,000 or more
3.2%
9.7%
12.9%
22.6%
6.5%
22.6%
22.6%
1
3
4
7
2
7
7
answered question
skipped question
31
8
Are performance based items used in any of your
organization’s testing programs?
9.7%
9.7%
Yes
No
Not sure
80.6%
Answer Options
Yes
No
Not sure
Response
Percent
Response
Count
80.6%
9.7%
9.7%
25
3
3
answered question
skipped question
31
8
If yes, please list those certifications that employ any
performance items.
•MySQL DBA 5.1
•Numerous
•Storage virtualization
•These certifications belong to our clients. I don't have
permission to share them. By the way, the survey
assumes that the respondent is a testing program and not
a vendor. Moreover it assumes that the testing program is
a certification one. Obviously, there are programs in other
areas such as education, admissions, licensure, I/O.
•NCCCO, ASNT, CrossFit, NAEC
•Real Estate Licensing - 7 different states; Strength and
Conditioning; Orthotics
•Sun Certified Enterprise Architect, Sun Certified Java
Developer
•Novell Certified Engineer - Enterprise Services, Novell
Certified Linux Professional, Novell Certified Linux
Engineer
•70-620
•HP Certifications for Loadrunner
•FNGLA Certified Landscape Technician, FNGLA Certified
Landscape Maintenance Technician"
•Oral certification examination
•Initial Certification in Anesthesiology
•Health, Professional, IT
•Advancement exam for musicians includes some
multimedia performance-based items (audio and video
files).
•All Microsoft Office 2003 and 2007 certifications,
Microsoft Certified Systems Engineer
•Microsoft Certified Systems Administrator, Microsoft
Certified IT Professional: Windows Server 2008 Server
Administrator, Microsoft Certified IT Professional:
Windows Server 2008 Enterprise Administrator, Microsoft
Certified Technical Specialist: Windows Server 2008
Active Directory, Configuring, Microsoft Certified
Technical Specialist: Microsoft SQL Server 2005,
Microsoft Certified Technical Specialist: Windows Vista –
Configuration"
•Operators
•IP Office Performance Lab
•All of our exam programs use performance based items.
•CCIE lab exams. Associate and Professional exams
also incorporate PBT items via simulations.
•Certified Kitchen Designer, Certified Bath Designer
•NCIDQ Certificate
Considering just your exams employing performance items,
how many test delivery instances are administered
annually (not individual candidates but actual number of
tests taken regardless of outcome)?
7.1%
10.7%
1 to 100
14.3%
101 to 1,200
1,201 to 5,000
17.9%
10.7%
5,001 to 10,000
25.0%
10,001 to 25,000
25,001 to 100,000
100,000 or more
14.3%
Answer Options
Response
Percent
Response
Count
1 to 100
101 to 1,200
1,201 to 5,000
5,001 to 10,000
10,001 to 25,000
25,001 to 100,000
100,000 or more
14.3%
25.0%
14.3%
10.7%
17.9%
10.7%
7.1%
4
7
4
3
5
3
2
answered question
skipped question
28
11
Considering just those testing programs employing performance based
items, what design approaches are you using? Check all that apply.
Answer Options
Live Environment
Monitored Hands-On Exam
Simulating Software using Software
Simulating Hardware Using Software
Simulating Hardware using Hardware AND Software
Simulating Hardware or Environment using Video
Simulating Hardware using 3-D Models
Simulating Process using Software
Emulation
Adaptive Text Case Scenario or Adaptive Case Study
Audio Exam
Board Assessment/Oral Exam
Other (please describe)
Response
Percent
Response
Count
46.4%
46.4%
25.0%
17.9%
10.7%
10.7%
3.6%
14.3%
28.6%
28.6%
21.4%
14.3%
14.3%
13
13
7
5
3
3
1
4
8
8
6
4
4
answered question
skipped question
Many of our exams use multiple choice options merely as a method of gathering a response. The response
had to be derived from one or more performances. I'm confused as to why complex behaviors measured by
text/graphics/video/audio-based multiple choice questions are not included in the list. There are many ways
to measure performance than those listed above.
Proctorless, automated evaluation, determinining end result (not methods or tasks to get there) to score
exam objectives.
Board oral exam using simulated cases
Design scenarios graded by trained interior designers
28
11
50.0%
45.0%
40.0%
35.0%
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
Other (please describe)
Board Assessment/Oral
Exam - Assessment by
panel interview
Audio Exam - Using
audible formats for
testing. Audio can be…
Adaptive Text Case
Scenario or Adaptive
Case Study - Using…
Emulation - An imitation
of a real-life (virtual)
environment, whose…
Simulating Process using
Software - Using
software to simulate a…
Simulating Hardware
using 3-D Models Using 3-D models to…
Simulating Hardware or
Environment using
Video - Using video to…
Simulating Hardware
using Hardware AND
Software - Using…
Simulating Hardware
Using Software - Using
software to simulate…
Simulating Software
using Software - Using
software to simulate…
Monitored Hands-On
Exam - Qualified proctor
performs the scoring…
Live Environment Using the actual
environment to test…
Considering just those testing programs employing performance based items, what design
approaches are you using? Check all that apply.
Considering your exams employing performance-based items, does your
organization use any tools in the Item Development category? Item
Development tools include Content Authoring, Standard Setting, Simulation
Software, Emulation Software and Item Bank Management.
29%
Yes
13%
No
Do not know
58%
Answer Options
Yes
No
Do not know
Response
Percent
Response
Count
58.1%
29.0%
12.9%
18
9
4
answered question
skipped question
31
8
Does your organization use a Content Authoring tool (tool for
developing items and forms)?
16%
Yes
No
84%
Answer Options
Yes
No
Response
Percent
Response
Count
84.2%
15.8%
16
3
answered question
skipped question
19
20
How did your organization obtain this tool?
0%
9%
Purchased ‘as-is’
9%
Purchased and customized to
our own needs
18%
Built in house
Third party uses tool (ie,
work is contracted out and
the contractor uses tool)
Other (please specify)
64%
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (ie, work is contracted out and the
Other (please specify)
Built in house; we are a vendor of the tool.
Response
Percent
Response
Count
9.1%
18.2%
63.6%
0.0%
9.1%
1
2
7
0
1
answered question
skipped question
11
28
If your organization purchased this tool, what is the name of the tool and its
vendor?
Tool Name
Vendor
Exam Studio Atvantus, LLC
no name
self written
WebAssessor Kryterion
Webassessor Kryterion
PerformIT
Kaplan IT
Please rate the criteria used to make the choice above from Not Considered to Most Important.
Answer Options
Cost
Built-in functionality
Ability to customize it to our needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
0
0
1
1
0
0
0
0
4
0
0
0
0
0
0
0
0
1
0
0
1
0
0
0
1
1
2
4
0
1
0
1
3
1
1
0
3
6
1
5
4
2
4
3
2
1
2
5
2
4
3
3
4
0
1
0
0
0
0
1
0
0
0
Response
Count
9
30
To what degree do you think this tool fulfills the needs of your
program?
0%
0%
20%
Not at all
10%
Minimally fills program needs
20%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
50%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
10.0%
20.0%
50.0%
20.0%
0.0%
0
1
2
5
2
0
answered question
skipped question
10
29
What functionality would you add to this tool to
greater meet the needs of your program?
Please state up to the TOP THREE things.
Language
localiztion
integration
with our LMS for reporting, delivery, registration,
security
collecting
Ability
more
item
and storing the proper data from tests
to handle question types beyond 4-part multiple choice.
flexibility in the process and admin
source material documentation
importing
tests/items and exporting results more efficiently
Flexibility
to establish sub-categories of across the board
questions.
more
support for complex item types
Ability
to automatically generate and randomly distribute equivalent
exam forms.
more
reporting, and integration with stats
Do you feel your organization got its money’s worth for this tool?
10%
0%
Yes
No
Not sure (please explain)
90%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
90.0%
0.0%
10.0%
9
0
1
answered question
skipped question
10
29
Does your organization use a Standards Setting tool (tool used to
define cut score on an exam form)?
45%
Yes
No
55%
Answer Options
Yes
No
Response
Percent
Response
Count
45.0%
55.0%
9
11
answered question
skipped question
20
19
Purchased ‘as-is’
How did your organization obtain this tool?
Purchased and customized to
our own needs
0%
Built in house
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
100%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
0.0%
100.0%
0.0%
0.0%
0.0%
0
0
9
0
0
0
answered question
skipped question
9
30
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
1
0
0
1
0
0
4
1
0
0
3
0
2
0
0
1
1
0
0
0
0
1
1
1
1
2
3
1
1
1
1
1
1
0
3
1
2
1
3
1
4
3
1
1
4
4
3
4
2
2
3
1
0
0
0
0
0
0
0
0
0
8
8
8
8
8
8
8
8
8
8
31
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
7
answered question
skipped question
Number
1
2
3
4
5
6
7
7
32
Response Text
Functionality... does it expedite the selection of a standard.
integration
Functionality
Quality
Ability to customize.
Cost
fits within our current processes
To what degree do you think this tool fulfills the needs of your
organization?
0%
Not at all
0%
25%
37%
Minimally fills program needs
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
25%
13%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
25.0%
25.0%
12.5%
37.5%
0.0%
0
2
2
1
3
0
answered question
skipped question
8
31
What functionality would you add to this tool to greater meet the needs of your program?
Please state up to the TOP THREE things.
Answer Options
1.
2.
3.
Response Percent
Response Count
100.0%
100.0%
33.3%
3
3
1
answered question
skipped question
Number
3
36
1.
1 base psychometric equations to prove
2.
form balancing
3.
Easily change number of questions analyzed
2 per exam
Flexibility in final
multiple score
algorithm
Greater flexibility in
pulling data from
external databases
3 more automated
less time intensive
Do you feel your organization got it's money’s worth for this tool?
0%
0%
Yes
No
Not sure (please explain)
100%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
7
0
0
answered question
skipped question
7
32
Does your organization use a Software Simulation tool (software
development kit (SDK) used to construct simulations used on the
exam)?
10%
Yes
No
90%
Answer Options
Yes
No
Response
Percent
Response
Count
10.0%
90.0%
2
18
answered question
skipped question
20
19
How did your organization obtain this tool?
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
0.0%
100.0%
0.0%
0.0%
0.0%
0
0
2
0
0
0
answered question
skipped question
2
37
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
1
0
0
0
1
1
2
0
1
1
1
2
0
0
1
0
1
1
0
1
0
1
0
0
0
0
0
0
0
0
0
2
2
2
2
2
2
2
2
1
2
37
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
2
answered question
skipped question
Number
Response Text
1 integration
2 Ease of use
2
37
To what degree do you think this tool fulfills the needs of your
0%
organization?
0%
0%
Not at all
0%
Minimally fills program needs
50%
50%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
Do not know
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
0.0%
50.0%
50.0%
0.0%
0
0
0
1
1
0
answered question
skipped question
2
37
Do you feel your organization got it's money’s worth for this tool?
0%
0%
Yes
No
Not sure (please explain)
100%
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
2
0
0
answered question
skipped question
2
37
Does your organization use an Emulation Software tool (software
development kit (SDK) used to construct emulations used on the
exam)?
11%
Yes
No
89%
Answer Options
Yes
No
Response
Percent
Response
Count
10.5%
89.5%
2
17
answered question
skipped question
19
20
How did your organization obtain this tool?
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
100.0%
0.0%
0.0%
0.0%
0.0%
0
2
0
0
0
0
answered question
skipped question
2
37
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
100.0%
100.0%
2
2
answered question
skipped question
Number
Tool Name
1 VMware Workstation
2 VMWare
Vendor
VMware Inc.
VMWare
2
37
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
1
0
0
0
0
0
1
2
2
0
2
0
0
2
1
1
0
0
0
0
0
2
0
1
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
2
2
2
2
2
2
2
2
2
37
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
1
answered question
skipped question
Number
Response Text
1 Functionality (It could emulate all of Novell's products)
1
38
To what degree do you think this tool fulfills the needs of your organization?
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
100.0%
0.0%
0.0%
0.0%
0
0
1
0
0
0
answered question
skipped question
1
38
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
2
0
0
answered question
skipped question
2
37
Does your organization use an Item Bank management tool (tool
that manages the item bank/pool and item version)?
42%
Yes
No
58%
Answer Options
Yes
No
Response
Percent
Response
Count
57.9%
11
42.1%
8
answered question
skipped question
19
20
Purchased ‘as-is’
How did your organization obtain this tool?
Purchased and customized to
our own needs
0% 7%
14%
Built in house
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
79%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
7.1%
14.3%
78.6%
0.0%
0.0%
0.0%
1
2
11
0
0
0
answered question
skipped question
14
25
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
100.0%
100.0%
2
2
answered question
skipped question
Number
Tool Name
1 none
2 WebAssessor
Vendor
self
Kryterion
2
37
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
2
1
0
0
0
0
0
1
4
2
0
0
0
0
0
0
0
1
1
0
0
0
0
3
3
3
2
5
2
2
3
3
4
2
2
2
1
3
4
4
3
0
4
1
2
2
6
6
5
6
5
3
5
0
0
0
0
0
0
0
0
0
0
13
12
12
12
12
12
12
12
11
13
26
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
8
answered question
skipped question
Number
1
2
3
4
5
6
7
8
8
31
Response Text
cost
Functionality
Functionality
Quality
Ability to customize.
Flexible workflows and integration with content development and form assembly.
ease of use and technical support
Support from the IT department.
To what degree do you think this tool fulfills the needs of your
organization?
0%
0%
8%
Not at all
15%
Minimally fills program needs
15%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
62%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
15.4%
15.4%
61.5%
7.7%
0.0%
0
2
2
8
1
0
answered question
skipped question
13
26
What functionality would you add to this tool to greater meet the needs of your
program? Please state up to the TOP THREE things.
Answer Options
Response Percent
Response Count
100.0%
75.0%
25.0%
4
3
1
1.
2.
3.
answered question
skipped question
Number
1.
1 integration w/ our LMS
2.
4
35
3.
Mix and match items between Support for adaptive
2 forms
items on forms
Ability to track equivalent
3 difficulty levels in questions
Ability to group
questions in more
detailed categories
(keywords)
Flexibility to adapt to constant Automate exam
4 program changes
assignment
Graphical depiction
of bank health
Do you feel your organization got it's money’s worth for this tool?
8% 0%
Yes
No
Not sure (please explain)
92%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
92.3%
7.7%
0.0%
12
1
0
answered question
skipped question
13
26
Considering your exams employing performance-based items, does
your organization use any tools in the Quality Assurance/Quality
Control category ?
4%
Yes
No
42%
54%
Do not know
Quality Assurance/Quality Control tools include: Content Review/Quality
Assurance, Form Equating and Statistical/Psychometric Analysis.
Answer Options
Yes
No
Do not know
Response
Percent
Response
Count
54.2%
41.7%
4.2%
13
10
1
answered question
skipped question
24
15
Does your organization use a Content Review/Quality Assurance
tool (tool used to assure the quality of the content developed)?
40%
Yes
No
60%
Answer Options
Yes
No
Response
Percent
Response
Count
40.0%
60.0%
6
9
answered question
skipped question
15
24
How did your organization obtain this tool?
Purchased ‘as-is’
0%
0%
Purchased and customized to
our own needs
29%
Built in house
Third party uses tool (work is
contracted out and the
contractor uses tool)
0%
0%
Do not know
71%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
0.0%
71.4%
0.0%
0.0%
28.6%
0
0
5
0
0
2
answered question
skipped question
Number
Other (please specify)
1 we built it as part of Webassessor
2 review by subject matter experts
7
32
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
1
0
0
0
1
0
0
0
1
0
3
1
2
1
1
2
2
2
2
1
1
1
1
1
1
2
0
1
0
2
1
1
2
1
0
1
0
0
0
0
0
0
0
0
0
0
5
5
5
5
5
5
5
5
5
5
34
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
1
answered question
skipped question
Number
Response Text
1 Quality
1
38
To what degree do you think this tool fulfills the needs of your
0%
organization?
0%
0%
Not at all
20%
40%
Minimally fills program needs
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
40%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
20.0%
40.0%
40.0%
0.0%
0.0%
0
1
2
2
0
0
answered question
skipped question
5
34
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
5
0
0
answered question
skipped question
5
34
Does your organization use a Form Equating tool (tool used to make
sure different forms have the same difficulty level and cover the
same objectives/skills)?
33%
Yes
No
67%
Answer Options
Yes
No
Response
Percent
Response
Count
66.7%
33.3%
4
2
answered question
skipped question
6
33
How did your organization obtain this tool?
0%
0%
Purchased and customized to
our own needs
0%
25%
Purchased ‘as-is’
25%
Built in house
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
50%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
25.0%
50.0%
25.0%
0.0%
0.0%
0
1
2
1
0
0
answered question
skipped question
4
35
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
2
2
2
1
1
3
1
1
1
0
0
1
2
0
0
2
1
1
0
1
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
3
3
3
3
3
3
3
3
3
3
36
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
2
answered question
skipped question
Number
Response Text
1 Quality
2 quality
2
37
To what degree do you think this tool fulfills the needs of your
organization?
0%
0%
Not at all
0%
25%
Minimally fills program needs
50%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
25%
Do not know
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
50.0%
25.0%
25.0%
0.0%
0
0
2
1
1
0
answered question
skipped question
4
35
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
4
0
0
answered question
skipped question
4
35
Does your organization use a Statistical/Psychometric Analysis tool
(tool that reports the performance at both form and item levels)?
30%
Yes
No
70%
Answer Options
Yes
No
Response
Percent
Response
Count
70.0%
30.0%
7
3
answered question
skipped question
10
29
Purchased ‘as-is’
How did your organization obtain this tool?
0%
14%
Purchased and customized to
our own needs
0%
28%
Built in house
Third party uses tool (work is
contracted out and the
contractor uses tool)
29%
Do not know
29%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
28.6%
28.6%
28.6%
14.3%
0.0%
0.0%
2
2
2
1
0
0
answered question
skipped question
7
32
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
100.0%
100.0%
2
2
answered question
skipped question
Number
Tool Name
1 Excalibre
2 FACETS
Vendor
Assment Systems
Objective Measurement Inc
2
37
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
0
1
2
0
1
0
0
0
0
0
2
3
0
0
0
0
0
0
0
0
1
1
1
4
1
3
3
2
1
0
5
4
2
0
3
3
3
2
0
0
0
0
4
0
0
1
0
0
0
0
0
0
0
0
0
0
6
6
6
6
5
6
6
6
6
6
33
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
2
answered question
skipped question
Number
Response Text
1 Quality
2 Qualilty
2
37
To what degree do you think this tool fulfills the needs of your
0%
organization?
0%
0%
Not at all
14%
14%
Minimally fills program needs
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
72%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
0.0%
14.3%
71.4%
14.3%
0.0%
0
0
1
5
1
0
answered question
skipped question
7
32
What functionality would you add to this tool to greater meet the needs of
your program? Please state up to the TOP THREE things.
Answer Options
1.
2.
3.
Response
Percent
Response
Count
100.0%
0.0%
0.0%
1
0
0
answered question
skipped question
Number
1.
1 Needs reliability figure that is not Alpha
2.
1
38
3.
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
7
0
0
answered question
skipped question
7
32
Considering your exams employing performance-based items, does
your organization use any tool in the Test Delivery category? Test
Delivery tools include: Item/test system/vendor interoperability,
Test Assignment and Test Delivery.
11%
33%
Yes
No
Do not know
56%
Answer Options
Yes
No
Do not know
Response
Percent
Response
Count
33.3%
55.6%
11.1%
6
10
2
answered question
skipped question
18
21
Does your organization use an Item/Exam System/Vendor
Interoperability tool? (tool that makes sure that forms created
under system or vendor X also works over system or vendor Y)
18%
Yes
No
82%
Answer Options
Yes
No
Response
Percent
Response
Count
18.2%
81.8%
4
18
answered question
skipped question
22
17
Purchased ‘as-is’
How did your organization obtain this tool?
0%
0%
Purchased and customized to
our own needs
0%
Built in house
50%
Third party uses tool (work is
contracted out and the
contractor uses tool)
50%
Do not know
0%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
0.0%
50.0%
0.0%
50.0%
0.0%
0
0
1
0
1
0
answered question
skipped question
2
37
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
1
1
1
1
1
1
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
38
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
1
answered question
skipped question
Number
Response Text
1 integration
1
38
To what degree do you think this tool fulfills the needs of your organization?
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
0.0%
0.0%
100.0%
0.0%
0
0
0
0
1
0
answered question
skipped question
1
38
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
1
0
0
answered question
skipped question
1
38
Does your organization use an Exam Assignment tool (tool that
assigns exam forms and/or PODs to candidates, can be based on predefined business rules)?
42%
Yes
No
58%
Answer Options
Yes
No
Response
Percent
Response
Count
41.7%
58.3%
5
7
answered question
skipped question
12
27
Purchased ‘as-is’
How did your organization obtain this tool?
0%
Purchased and customized to
our own needs
0%
20%
20%
Built in house
0%
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
60%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
20.0%
0.0%
60.0%
20.0%
0.0%
0.0%
1
0
3
1
0
0
answered question
skipped question
5
34
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
0.0%
100.0%
0
1
answered question
skipped question
Number
Tool Name
1
Vendor
Assessment Systems
1
38
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
1
1
0
0
0
0
0
1
1
0
0
1
1
0
0
0
0
2
1
0
0
0
0
0
0
0
0
1
2
1
1
3
3
2
1
0
2
1
1
1
1
2
3
2
1
0
1
2
2
1
0
0
1
0
0
0
0
0
0
0
0
0
0
5
5
5
5
5
5
5
5
4
5
34
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
4
answered question
skipped question
Number
1
2
3
4
Response Text
Functionality
integration
Quality
Support from IT department.
4
35
To what degree do you think this tool fulfills the needs of your
organization?
0%
0%
20%
Not at all
20%
Minimally fills program needs
0%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
60%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
20.0%
0.0%
60.0%
20.0%
0.0%
0
1
0
3
1
0
answered question
skipped question
5
34
What functionality would you add to this tool to greater meet the
needs of your program? Please state up to the TOP THREE things.
Answer Options
1.
2.
3.
Response
Percent
Response
Count
100.0%
50.0%
0.0%
2
1
0
answered question
skipped question
Number
1.
Automatically assign forms and
1 PODs to candidates
2 automate process for all exams
2.
Increased
reporting
capabilities
2
37
3.
Do you feel your organization got it's money’s worth for this tool?
0%
20%
Yes
No
Not sure (please explain)
80%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
80.0%
20.0%
0.0%
4
1
0
answered question
skipped question
5
34
Does your organization use an Exam Delivery tool (tool that
presents the exam form to the candidates)?
46%
Yes
54%
Answer Options
Yes
No
No
Response
Percent
Response
Count
53.8%
46.2%
7
6
answered question
skipped question
13
26
Purchased ‘as-is’
How did your organization obtain this tool?
0%
0%
Purchased and customized to
our own needs
17%
Built in house
0%
Third party uses tool (work is
contracted out and the
contractor uses tool)
50%
Do not know
33%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
16.7%
0.0%
33.3%
50.0%
0.0%
0.0%
1
0
2
3
0
0
answered question
skipped question
6
33
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
75.0%
100.0%
3
4
answered question
skipped question
Number
Tool Name
1 ExamStudio
2 Prometric
3 PerformIT
4
Vendor
Atvantus
Prometric
Kaplan IT
Vue
4
35
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
2
0
0
1
0
0
0
0
1
1
0
2
1
0
2
1
1
0
3
4
3
2
4
2
3
2
2
0
1
0
1
1
1
1
2
0
0
0
0
0
0
0
0
0
0
6
5
5
5
5
5
5
5
5
6
33
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
2
answered question
skipped question
Number
Response Text
1 integration
2 cost and coverage
2
37
To what degree do you think this tool fulfills the needs of your
organization?
Not at all
Minimally fills program needs
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
Do not know
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
20.0%
40.0%
40.0%
0.0%
0
0
1
2
2
0
answered question
skipped question
5
34
What functionality would you add to this tool to greater meet the needs of
your program? Please state up to the TOP THREE things.
Answer Options
1.
2.
3.
Response
Percent
Response
Count
100.0%
100.0%
100.0%
1
1
1
answered question
skipped question
Number
1.
1 reduced cost
2.
more flexilbility
in delivering
exams
1
38
3.
easier/faster
publishing
process
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
5
0
0
answered question
skipped question
5
34
5.3%
Considering your exams employing performance-based items, does your
organization use any tools in the Candidate Administration category?
Candidate Administration tools include: Candidate management,
Candidate registration/scheduling, Candidate validatio
Yes
47.4%
No
Do not know
47.4%
Considering your exams employing performance-based items, does your
organization use any tools in the Candidate Administration category?
Candidate Administration tools include: Candidate management, Candidate
registration/scheduling, Candidate validation/authentication and Candidate
payment/refund processing.
Answer Options
Yes
No
Do not know
Response
Percent
Response
Count
47.4%
47.4%
5.3%
9
9
1
answered question
skipped question
19
20
Does your organization use a Candidate Management tool (tool that
manages candidate information that can be used with EEOC 1978)?
29%
Yes
No
71%
Answer Options
Yes
No
Response
Percent
Response
Count
71.4%
28.6%
10
4
answered question
skipped question
14
25
Purchased ‘as-is’
How did your organization obtain this tool?
10%
Purchased and customized to
our own needs
10%
0%
Built in house
10%
20%
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
50%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
10.0%
20.0%
50.0%
10.0%
0.0%
10.0%
1
2
5
1
0
1
answered question
skipped question
Number
Other (please specify)
1 Custom program written based on Access
10
29
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
100.0%
100.0%
4
4
answered question
skipped question
Number
1
2
3
4
Tool Name
Saba
Integral7
FNGLA Database
I7
Vendor
Saba
Integral 7
WF Dick & Associates
Integral 7
4
35
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
3
1
0
0
0
0
0
2
2
0
0
1
0
0
0
0
0
1
2
1
1
1
1
2
2
1
1
1
0
0
0
1
1
1
0
0
Considerably
Most
Do not Know
Important Important
3
3
3
4
3
2
5
3
1
0
3
3
3
3
3
0
2
2
0
1
1
1
1
1
1
1
1
Response
Count
9
9
9
9
9
9
9
9
8
9
30
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
5
answered question
skipped question
Number
1
2
3
4
5
Response Text
Functionality
built in functionality
Integration of certification with membership database
Quality
Support from the IT department.
5
34
To what degree do you think this tool fulfills the needs of your
organization?
0%
0%
10%
Not at all
0%
Minimally fills program needs
Adequately fills program
needs
30%
Mostly fills program needs
60%
Exceeds program needs
Do not know
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
60.0%
30.0%
10.0%
0.0%
0
0
6
3
1
0
answered question
skipped question
10
29
What functionality would you add to this tool to greater meet the needs of
your program? Please state up to the TOP THREE things.
Answer Options
1.
2.
3.
Response
Percent
Response
Count
100.0%
50.0%
16.7%
6
3
1
answered question
skipped question
Number
1.
2.
Better integration with other development
1 tools
2 ANOVA
3 better document and assignment control
4 More reporting features
5 Capture more candidate details
6 better reporting
assessor
workflow mgt.
tools
test question
management
Improved
reporting
capabilities
6
33
3.
cost. Once a
system gets up
to 150K/year, it
becomes very
compelling to
build your own.
Most big
programs have.
Do you feel your organization got it's money’s worth for this tool?
13%
0%
Yes
No
Not sure (please explain)
87%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
87.5%
0.0%
12.5%
7
0
1
answered question
skipped question
Number
8
31
Not sure (please explain)
1 we were feeling the cost pain and endeavored to build vs. buy, but ran out of time.
Does your organization use a Candidate Registration/Scheduling
tool (tool used to schedule candidates to take an exam, or through
administration and can contain business rules such as retake
policies)?
14.3%
Yes
No
85.7%
Does your organization use a Candidate Registration/Scheduling tool (tool
used to schedule candidates to take an exam, or through administration and
can contain business rules such as retake policies)?
Answer Options
Yes
No
Response
Percent
Response
Count
85.7%
14.3%
12
2
answered question
skipped question
14
25
Purchased ‘as-is’
How did your organization obtain this tool?
0%
9%
Purchased and customized to
our own needs
0%
0%
Built in house
46%
Third party uses tool (work is
contracted out and the
contractor uses tool)
45%
Do not know
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
45.5%
45.5%
0.0%
0.0%
9.1%
0
5
5
0
0
1
answered question
skipped question
Number
Other (please specify)
1 Same tool as previous question
11
28
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
60.0%
80.0%
3
4
answered question
skipped question
Number
Tool Name
1 Saba
2 Prometric
3
4 Quickhire
5
Vendor
Saba
Prometric
WF Dick & Assoc
Integral 7
5
34
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
2
1
0
0
0
0
0
2
3
0
0
0
0
0
0
0
0
1
1
0
1
0
0
1
0
0
0
3
2
3
4
3
3
3
2
2
4
3
3
3
2
2
6
3
1
0
4
3
3
5
4
1
3
2
1
1
1
1
1
1
1
1
1
11
11
11
11
11
11
11
11
10
11
28
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
8
answered question
skipped question
Number
1
2
3
4
5
6
7
8
8
31
Response Text
Functionality
integration
hard to say, we've been in business with Prometric for 15 years.
Again, integration with other tools & databases for association
Quality
customization
Support from IT department.
cost
To what degree do you think this tool fulfills the needs of your
organization?
0%
0%
18%
Not at all
9%
Minimally fills program needs
27%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
46%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
9.1%
27.3%
45.5%
18.2%
0.0%
0
1
3
5
2
0
answered question
skipped question
11
28
What functionality would you add to this tool to greater meet the
needs of your program? Please state up to the TOP THREE things.
Answer Options
1.
2.
3.
Response
Percent
Response
Count
100.0%
50.0%
50.0%
4
2
2
answered question
skipped question
Number
1.
1 Better integration with other
2.
better management of multi-part better channel
2 certification program
security
more visibility into available
3 dates/seats for candidates
4 customization by our staff
better refund
process
4
35
3.
better
integration with
test drivers that
allow for
performance
items
more flexibility
in adding
dates/seats
Do you feel your organization got it's money’s worth for this tool?
10%
0%
Yes
No
Not sure (please explain)
90%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
90.0%
0.0%
10.0%
9
0
1
answered question
skipped question
Number
Not sure (please explain)
I do not use this part of our system on a
regular basis so I am not sure what needs
1 improvement.
10
29
Does your organization use a Candidate Validation/Authentication
tool (tool used on exam security to validate the candidate’s
identity when showing up to take an exam against the candidate
scheduled to take the exam)?
30.8%
Yes
No
69.2%
Does your organization use a Candidate Validation/Authentication tool (tool
used on exam security to validate the candidate’s identity when showing up
to take an exam against the candidate scheduled to take the exam)?
Answer Options
Yes
No
Response
Percent
Response
Count
30.8%
69.2%
4
9
answered question
skipped question
13
26
How did your organization obtain this tool?
Purchased ‘as-is’
0.0%
Purchased and customized to
our own needs
33.3%
Built in house
33.3%
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
0.0%
0.0%
33.3%
Other (please specify)
How did your organization obtain this tool?
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
contractor uses tool)
Do not know
Other (please specify)
Response
Percent
Response
Count
33.3%
33.3%
0.0%
1
1
0
0.0%
0
33.3%
0.0%
1
0
answered question
skipped question
3
36
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
100.0%
0.0%
1
0
answered question
skipped question
Number
Tool Name
1Prometric
Vendor
1
38
Please rate the criteria used to make the choice above from Not Considered to Most Important
Not Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Rating
Average
Response
Count
Cost
0
0
0
1
0
0
1
5.00
2
Built-in functionality
0
0
0
0
1
0
1
5.50
2
0
0
0
0
1
0
1
5.50
2
Integration with other tools
0
0
0
0
0
1
1
6.00
2
Quality
0
0
0
0
1
0
1
5.50
2
Technical support
0
0
0
0
1
0
1
5.50
2
Ease of use
0
0
0
0
0
1
1
6.00
2
Readily available
0
0
0
0
0
2
1
5.67
3
Recommendation from management
0
0
0
0
1
0
1
5.50
2
Answer Options
Ability to customize it to my own needs
answered question
3
skipped question
36
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your
own.
Response
Count
Answer Options
1
answered question
skipped question
Number
Response
Text
1 integration
1
38
To what degree do you think this tool fulfills the needs of your
organization?
0.0% 0.0%
Not at all
Minimally fills program needs
33.3%
33.3%
Adequately fills program needs
Mostly fills program needs
0.0%
33.3%
Exceeds program needs
Do not know
To what degree do you think this tool fulfills the needs of your organization?
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
33.3%
0.0%
33.3%
33.3%
0.0%
0
1
0
1
1
0
answered question
skipped question
3
36
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
3
0
0
answered question
skipped question
3
36
Does your organization use a Payment and Refund Processing tool
(tool to collect candidate payment and issue refunds)?
33.3%
Yes
No
66.7%
How did your organization obtain this tool?
Purchased ‘as-is’
0.0%
Purchased and customized to our
own needs
25.0%
25.0%
0.0%
Built in house
0.0%
Third party uses tool (work is
contracted out and the contractor
uses tool)
Do not know
50.0%
Other (please specify)
How did your organization obtain this tool?
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
contractor uses tool)
Do not know
Other (please specify)
Response
Percent
Response
Count
25.0%
0.0%
50.0%
1
0
2
0.0%
0
25.0%
0.0%
1
0
answered question
skipped question
4
35
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
0.0%
100.0%
0
1
answered question
skipped question
Number
Vendor
1Integral 7
1
38
Please rate the criteria used to make the choice above from Not Considered to Most Important
Not Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Rating
Average
Response
Count
Cost
0
0
0
1
1
1
0
4.00
3
Built-in functionality
0
0
0
1
2
0
0
3.67
3
Ability to customize it to my own needs
0
0
0
1
2
0
0
3.67
3
Integration with other tools
0
0
0
2
0
1
0
3.67
3
Quality
0
0
0
1
1
1
0
4.00
3
Technical support
0
0
0
1
1
1
0
4.00
3
Ease of use
0
0
0
1
2
0
0
3.67
3
Readily available
0
0
0
1
2
0
0
3.67
3
Recommendation from management
1
0
0
0
1
0
0
2.00
2
Answer Options
answered question
3
skipped question
36
Number
Response Text
1 Do not know.
2 integration
3 comes bundled with their data management sysytem
0.0% 0.0%
To what degree do you think this tool fulfills the needs of your
organization?
0.0%
Not at all
Minimally fills program needs
25.0%
Adequately fills program needs
50.0%
Mostly fills program needs
25.0%
Exceeds program needs
Do not know
To what degree do you think this tool fulfills the needs of your organization?
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Response
Percent
Response
Count
0.0%
0.0%
50.0%
25.0%
25.0%
0.0%
0
0
2
1
1
0
answered question
skipped question
4
35
Number
1.
2.
1better functionality for refunds take more forms of payment
Do you feel your organization got it's money’s worth for this tool?
25%
Yes
0%
No
Not sure (please explain)
75%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
75.0%
0.0%
25.0%
3
0
1
answered question
skipped question
Number
Not sure (please explain)
1 I don't deal with payment processes.
4
35
Does your organization use any tools in the Score management
category? Score Management tools include: Score automation and
Score reporting.
7%
Yes
43%
No
Do not know
50%
Answer Options
Yes
No
Do not know
Response
Percent
Response
Count
42.9%
50.0%
7.1%
6
7
1
answered question
skipped question
14
25
Does your organization use an Automatic Scoring tool (tool that
scores the candidate at pre-determined check-points and assigns
them a score)?
50%
Answer Options
Yes
No
Yes
50%
No
Response
Percent
Response
Count
50.0%
50.0%
6
6
answered question
skipped question
12
27
0%obtain this tool?
How did your organization
Purchased ‘as-is’
0%
0%
17%
Purchased and customized to
our own needs
17%
Built in house
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
66%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
16.7%
66.7%
16.7%
0.0%
0.0%
0
1
4
1
0
0
answered question
skipped question
6
33
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
4
2
2
1
2
3
3
2
1
1
2
2
1
2
1
2
1
2
0
1
1
3
1
1
0
1
1
0
0
0
0
0
0
0
0
0
5
5
5
5
5
5
5
5
4
5
34
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
4
answered question
skipped question
Number
1
2
3
4
Response Text
Functionality
integration
Support from IT department.
cost
4
35
To what degree do you think this tool fulfills the needs of your
organization?
0%
17%
Not at all
0%
Minimally fills program needs
33%
Adequately fills program
needs
17%
Mostly fills program needs
Exceeds program needs
33%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
33.3%
33.3%
16.7%
16.7%
0.0%
0
2
2
1
1
0
answered question
skipped question
6
33
What functionality would you add to this tool to greater meet the needs of your
program? Please state up to the TOP THREE things.
Answer Options
Response Percent
Response Count
100.0%
100.0%
100.0%
2
2
2
1.
2.
3.
answered question
skipped question
Number
1.
2.
2
37
3.
1 Automatically score GUI
Increased flexibility to
lower need for
proctors to manually
Faster/more reliable
score
scoring/reinitialization
2 better reporting
easier to add new
tracks/devices
more flexibility in
implementing grading
rules
Do you feel your organization got it's money’s worth for this tool?
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
100.0%
0.0%
0.0%
5
0
0
answered question
skipped question
5
34
Does your organization use a Score Reporting tool (tool that creates
and/or sends the score report to the candidates at the end of the
exam)?
50%
Answer Options
Yes
No
Yes
50%
No
Response
Percent
Response
Count
50.0%
50.0%
6
6
answered question
skipped question
12
27
0%obtain this tool?
How did your organization
0%
Purchased ‘as-is’
0%
Purchased and customized to
our own needs
17%
Built in house
33%
Third party uses tool (work is
contracted out and the
contractor uses tool)
Do not know
50%
Other (please specify)
Answer Options
Purchased ‘as-is’
Purchased and customized to our own needs
Built in house
Third party uses tool (work is contracted out and the
Do not know
Other (please specify)
Response
Percent
Response
Count
0.0%
16.7%
50.0%
33.3%
0.0%
0.0%
0
1
3
2
0
0
answered question
skipped question
6
33
If your organization purchased this tool, what is the name of the tool and its
vendor?
Answer Options
Tool Name
Vendor
Response
Percent
Response
Count
100.0%
100.0%
1
1
answered question
skipped question
Number
Tool Name
1 unknown
Vendor
Prometric
1
38
Please rate the criteria used to make the choice above from Not Considered to Most Important
Answer Options
Cost
Built-in functionality
Ability to customize it to my own needs
Integration with other tools
Quality
Technical support
Ease of use
Readily available
Recommendation from management
answered question
skipped question
Not
Considered
Minimally
Important
Somewhat
Important
Important
Considerably
Important
Most
Important
Do not Know
Response
Count
2
2
1
1
1
1
1
2
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
2
2
1
2
3
3
2
1
0
1
1
1
2
1
1
1
1
0
0
1
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
5
5
5
5
5
5
5
5
4
5
34
What is THE ONE MOST IMPORTANT factor your organization
considered? Choose one from the above list or write your own.
Response
Count
Answer Options
4
answered question
skipped question
Number
1
2
3
4
Response Text
Not applicable
ingtegration
Support from IT department.
integration with other systems
4
35
To what degree do you think0%
this tool fulfills the needs of your
organization?
0%
20%
0%
Not at all
Minimally fills program needs
40%
Adequately fills program
needs
Mostly fills program needs
Exceeds program needs
40%
Answer Options
Not at all
Minimally fills program needs
Adequately fills program needs
Mostly fills program needs
Exceeds program needs
Do not know
Do not know
Response
Percent
Response
Count
0.0%
0.0%
40.0%
40.0%
20.0%
0.0%
0
0
2
2
1
0
answered question
skipped question
5
34
Do you feel your organization got it's money’s worth for this tool?
17%
0%
Yes
No
Not sure (please explain)
83%
Answer Options
Yes
No
Not sure (please explain)
Response
Percent
Response
Count
83.3%
0.0%
16.7%
5
0
1
answered question
skipped question
Number
Not sure (please explain)
1 N/A
6
33
Does your organization use any tools that are not mentioned in this
survey?
Yes
No
Do not know
Answer Options
Yes
No
Do not know
Response
Percent
Response
Count
0.0%
78.6%
21.4%
0
11
3
answered question
skipped question
14
25
Are you interested in Performance Testing Council
Summits?
Yes, I attend summits
Yes, I have not attended in
the past but I plan to attend
future summits
18.8%
31.3%
6.3%
Yes, but I have restrictions
preventing my attendance
No
31.3%
12.5%
Undecided
Answer Options
Yes, I attend summits
Yes, I have not attended in the past but I plan to
attend future summits
Yes, but I have restrictions preventing my attendance
No
Undecided
Response
Percent
Response
Count
31.3%
5
12.5%
2
31.3%
6.3%
18.8%
5
1
3
answered question
skipped question
16
23
If you are interested in attending a Performance Testing
Council Summit, what presentations would you like to see at a
Performance Testing Council Summit?
More case studies of performance testing programs.
Maintaining integrity of exam
Reducing subjectivity in a hands-on proctored exam
Training exam judges to be consistent
not sure
Certification programs that use emulations and process
simulations
Do you know of associates at your company or other organizations
who might be interested in participating in this survey?
13.3%
33.3%
Yes
No
Not certain
53.3%
Do you know of associates at your company or other organizations who
might be interested in participating in this survey?
Answer Options
Yes
No
Not certain
Response
Percent
Response
Count
13.3%
53.3%
33.3%
2
8
5
answered question
skipped question
15
24
What’s next?
(suggestions)



Go deeper on a/some subject(s) uncovered by the survey
Invite vendors to respond
Performance Testing Design Approaches Document
–
–

Add: text/graphics/video/audio-based multiple choice
Add: Board oral exam using simulated cases
Deliver performance exams in a scalable manner may be
challenging
–
–
–
–
Delivery channels
Maintaining integrity of exam (exam security)
Reducing subjectivity in a hands-on proctored exam
Training exam judges to be consistent