Foundations of the
NBCOT Certification Examinations
®
®
INTRODUCTION
Historically, regulation of the health professions in the United States began with a necessity to protect the
public from the under-educated and under-trained professional. Over time, licensure, credentialing, and
certification have continued the tradition of protecting the public but have also increased their scope of
activity to continuously improve the quality of practice in the profession.
Foundations of the NBCOT
Certification Examinations
Our mission...
Certification has become the hallmark credential for professionals in a variety of industries often serving as
a benchmark for hiring and promotion (Microsoft, 2007). Certification is defined as “formal recognition that
an individual has proficiency within, and a comprehension of, a specified body of knowledge” (ASQ, 2005,
p. 1). It can be characterized as the qualification of individuals performing jobs using a predefined set of
standards (DeBaugh, 2005; Mulkey & Naughton, 2005; Roberts, 2005). Certifications denote successful
assessment of a defined body of knowledge, demonstrating proof of work competence (Meisinger, 2004;
Mulkey & Naughton, 2005; Schoneboom, 2005). Certifications have value when certification boards
meet periodically to review and update, as required, area-specific bodies of knowledge, examination
specifications and questions (Roberts, 2005).
®
Serving the public interest by advancing client care and professional practice through evidence-based
certification standards and the validation of knowledge essential for effective practice in occupational
therapy
The primary purpose of awarding the NBCOT credential—OCCUPATIONAL THERAPIST REGISTERED (OTR)
and CERTIFIED OCCUPATIONAL THERAPY ASSISTANT (COTA)—is to serve the public interest by certifying
only those persons who have the necessary knowledge of occupational therapy to practice.
Our vision
NBCOT certification uses a formal process to grant a certification credential to an individual who: 1) meets
academic and practice experience requirements; 2) successfully completes a comprehensive examination
to assess knowledge and skills for practice; and 3) agrees to adhere to the NBCOT Candidate/Certificant
Code of Conduct. Currently, 50 states, Guam, Puerto Rico, and the District of Columbia require NBCOT
initial certification for occupational therapy state regulation (e.g., licensing).
Certified occupational therapy professionals providing effective evidence-based services across all areas of
practice worldwide
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or
by any means (electronic, mechanical, photocopying, recording, or otherwise) without prior written permission of the copyright
owners.
NBCOT has received and maintained accreditation from the American National Standards Institute
(ANSI) and the National Commission for Certifying Agencies (NCCA, 2004). Accreditation ensures that a
certification agency meets a recognized standard for the programs and services it employs, and that the
certification agency is engaged in continuous review and quality improvement.
©2015 National Board for Certification in Occupational Therapy, Inc. “NBCOT®” is a service and trademark of the National
Board for Certification in Occupational Therapy, Inc. “OTR®” is a service and trademark of the National Board for Certification in
Occu-pational Therapy, Inc. “COTA®” is a service and trademark of the National Board for Certification in Occupational Therapy,
Inc. All marks are registered in the United States of America.
National Board for Certification in Occupational Therapy, Inc.
12 South Summit Avenue, Suite 100
Gaithersburg, MD 20877
http://www.nbcot.org
Printed in the United States of America.
Rev 08/20/2015
www.nbcot.org
Page 1
FOUNDATIONS OF THE NBCOT
CERTIFICATION EXAMINATIONS
Using a question and answer format, this monograph presents information about the key psychometric
principles that form the foundations of the NBCOT certification examinations.
QQ:
AA:
QQ:
AA:
QQ:
AA:
Page 2
www.nbcot.org
What guidelines does NBCOT follow when developing its certification
examinations?
The procedures used to prepare the NBCOT certification examinations are consistent with the
technical guidelines recommended by the American Educational Research Association, the
American Psychological Association, and the National Council on Measurement in Education
(AERA, APA, NCME; 1999). Additionally, NBCOT test development and administration
procedures adhere to relevant sections of the Uniform Guidelines on Employee Selection
adopted by the Equal Employment Opportunity Commission, Civil Service
Commission, Department of Justice, and Department of Labor (EEOC, CSC, DOJ,
1978; DOL, 1993).
®
How does NBCOT decide on the content to include in its certification
examinations?
Consistent with certification industry standards, the OTR and COTA examinations are
constructed based on the results of practice analysis studies. The studies identify the
domains, tasks, and knowledge required for occupational therapy practice. The ultimate goal
of a practice analysis study is to ensure that there is a representative linkage of examination
content to practice. The periodic performance of practice analysis studies assists NBCOT
with evaluating the validity of the test specifications that guide content distribution of the
credentialing examinations.
When and how did NBCOT complete its latest practice analysis studies?
NBCOT completed its most recent OTR and COTA practice analysis studies in 2012 (NBCOT,
2013). A large-scale survey design was used to gather practice data from thousands of
entry-level OTR and COTA certificants. Survey respondents were asked to rate the criticality
and frequency of the major domains, tasks, knowledge, and skills for occupational therapy
practice. Results of the study led to the formation of the examination blueprints: the NBCOT
Validated Domain, Task, and Knowledge Statements for the OTR (NBCOT, 2013); and the
NBCOT Validated Domain, Task, and Knowledge Statements for the COTA (NBCOT, 2013).
www.nbcot.org
Page 3
QQ:
AA:
How are the examinations formatted?
Items on the COTA examination consist of classic multiple-choice items. The OTR examination
includes classic multiple-choice items, scenario-based multiple-choice items, and clinical
simulation test (CST) problems.
The multiple-choice item format contains a stem (sometimes also called the premise) and
three or four possible response options. The stem may be a direct question or an incomplete
statement. Of the response options, there must be only one correct or clearly best answer,
referred to as the key. The incorrect response options are called distractors.
QQ:
AA:
Scenario-based multiple-choice items are comprised of a short four- or five-sentence
passage that provides the information needed to answer a series of three to five single
response multiple-choice items. The multiple-choice items are linked to the scenario and are
dependent on the scenario to help formulate a correct response.
• Positive actions: These are actions that represent best practice standards based on
the clinical information presented in the opening scene and previous sections.
Points are awarded for selecting these options.
• Neutral actions: These actions may be completed as part of the OT process but do
not necessarily help in solving the clinical situation or problem presented. Points are
neither awarded nor deducted if a candidate selects these options.
• Negative actions: These actions hinder or distract from solving the problem and may
be harmful (physically, psychologically, and/or financially) to the client and/or
institution. Points are deducted if a candidate selects these options.
Points are neither awarded nor deducted for responses that are not selected.
Page 4
www.nbcot.org
The OTR examination includes two components: a 170 item multiple-choice component and a
three problem CST component. The OTR CST component is presented to all candidates before
the multiple-choice items. There are two short optional tutorials on the OTR examination
(one before each component of the examination). The examination clock starts after the
completion of the CST tutorial. Following delivery of the OTR CST component, candidates can
opt to take a tutorial outlining how to navigate through the multiple-choice component of the
examination. The tutorial time for each tutorial is not deducted from overall testing time.
®
The COTA examination is comprised of 200 multiple-choice items based on blueprint
specifications. Items are presented one at a time to the candidates. Some of the items
may contain a picture or chart. Candidates can mark items for review and change their
item responses as their testing time allows until they submit the examination for scoring.
At the start of the examinations, candidates have the option of taking a tutorial about the
functionality of the test screens. The four-hour test clock starts at the conclusion of the
tutorial.
Candidates progress through a CST problem one section at a time. For each section,
candidates are asked to identify and select all of the actions that represent occupational
therapy best practice standards. In return, they are provided with additional practice-relevant
information for each action selected. Once the action has been checked, the candidate
cannot de-select the response option.
Each individual CST problem is scored based on the options selected. Each option
represents:
The OTR and COTA examinations are computer-delivered. Candidates are allotted a period of
four hours to answer the examination items.
Once a candidate enters the multiple-choice component of the OTR examination, he or she
is not able to reenter the CST component. In the OTR multiple-choice component, candidates
can mark items for review and change their item responses as their testing time allows until
they submit the examination for scoring. The selected response on a marked item will be
submitted for scoring. No credit will be given to a marked item where a response option has
not been selected.
The OTR examination also consists of a separate section of CST problems. Each problem
begins with an opening scene that is followed by four to five sections. Each section builds
upon the previous section by providing additional information and prompts candidates to
select from up to 12 options. The options within a section represent a variety of practicebased actions such as questions to ask, actions to take, or causes of the situation. A typical
CST problem consists of 40 - 60 options across the sections.
In each CST problem section, candidates must select at least one option before proceeding to
the next section. Candidates have the option of scrolling to previous screens within the same
CST problem to view opening scenes, section information, and responses they have selected.
Candidates must complete each CST problem one at a time in the order presented.
How are the examinations delivered?
NBCOT certification examination testing centers are located throughout the United States
and internationally. Candidates can schedule to take the examination any day of the
week during the normal business hours of the testing center the candidate selects.
Scheduling instructions are provided in the candidates’ Authorization to Test Letter.
QQ:
AA:
How are NBCOT examinations scored?
All NBCOT certification examinations are criterion referenced. This means in order to pass the
examination, the candidate must obtain a score that is at least equal to the minimum passing
score. The minimum passing score represents an absolute standard and does not depend
on the performance of other candidates taking the same examination. Scores for the NBCOT
examinations are reported on a scale from 300 to 600. A total scaled score of at least 450 is
required to pass the OTR or COTA certification examination. Candidates with a total scaled
score of less than 450 have not met the minimum passing standard, and therefore, did not
pass the examination. It is important to note that the passing standard is based on candidate
performance across the entire exam. It is not necessary to meet the passing standard in
each individual domain area to pass the OTR or COTA exam. Likewise, for the OTR exam, it
is not necessary to meet the passing standard for both the CST portion and multiple-choice
sections.
www.nbcot.org
Page 5
QQ:
AA:
QQ:
AA:
What is a scaled score?
A scaled score is a mathematical conversion of the number of items that a candidate
correctly answered, which is transformed so that there is a consistent scale used across all
forms of the test. This transformation is similar to converting from pounds to kilograms or
centigrade to Fahrenheit. The weight or temperature has not changed, only the format used
to report the units.
Why use scaled scores?
Scaled scores allow for direct comparisons of scores across the multiple test forms NBCOT
uses. Scaled scoring applies the same passing standard to all test forms, but adjusts for
possible differences in overall test form difficulty. Scaled scores make scoring fair for all
candidates so that no single test form is any easier or more difficult in comparison.
Scaled scores make comparisons over time easier. It will not matter which test form a
candidate took, the scale will remain constant. Using raw scores does not guarantee this
because there may be differences in how easy or hard a test form is.
QQ:
AA:
QQ:
AA:
Why not use the percent correct?
Percent correct and number correct scores are just other ways of reporting raw scores and
therefore do not resolve the issue of comparability of scores across different test forms.
Although each form of the examination tests the same knowledge and skills, the forms
contain a different set of test items. This may mean that one or more questions on one test
form may be a little easier or harder than the questions appearing on another test form.
Simply using a raw score does not account for this difference. Best test practice is to use
scaled scoring to ensure comparable scores and fairness to all candidates.
How is the examination pass point determined?
The NBCOT certification examinations are criterion referenced. This means a candidate’s
performance is compared to a pre-determined minimum standard. This standard is
established by completion of a Standard Setting Study as recommended by the Standards for
Educational and Psychological Testing. The methodology used for the most recent standard
setting studies, the modified Angoff methodology, required panels of subject matter experts
to identify the minimum level of competency required to pass the examinations (Impara &
Plake, 1997).
QQ:
AA:
QQ:
AA:
QQ:
AA:
QQ:
AQ:
Is there more than one version of the OTR or COTA examination?
In certification programs including NBCOT, multiple forms of a test are developed to ensure
the security and integrity of the examinations. Every new form of the certification examination
is built to the same standardized test blueprint. The test blueprint determines what domainlevel content will be tested, no matter which test form the candidate is given. Additionally,
each form contains the same number of items but the items contained in the test differ from
form to form.
How does NBCOT ensure examination versions are comparable?
Although each form of the examination tests the same domain-level content, the forms
contain a different set of test items. This may mean that one or more questions on one test
form may be a little easier or harder than the questions appearing on another test form.
They cannot be expected to be precisely equivalent in level and range of difficulty. NBCOT
takes into account these differences by using appropriate equating technologies. Equating
methods measure the difficulty of each form and adjust the passing score as needed so the
same level of candidate performance is reflected in the passing score regardless of form
difficulty. Equating enables NBCOT to maintain the same passing standard across different
examination forms.
®
How does NBCOT develop questions for the OTR and COTA examinations?
NBCOT annually recruits subject matter experts to develop new questions for the
examinations. Subject matter experts comprise OTR and COTA certificants representative of
the profession in terms of practice experiences, geographic regions, gender, and ethnicity.
After questions or “items” for NBCOT examinations are developed, they undergo several
rigorous subject matter committee reviews designed to: validate that the knowledge and
tasks measured are compatible with the domain-level blueprint specifications; assess the
relative criticality and frequency of each item to occupational therapy practice; and confirm
that each item meets generally accepted fairness guidelines.
How is cultural fairness considered during item development?
NBCOT takes into account the cultural fairness of its examinations during item development.
NBCOT adheres to recognized item writing and test development, and reviews procedures to
ensure readability, neutral language, and the universal accuracy of terms used in its items.
Additional fairness criteria include, but are not limited to:
• Editing items for issues of bias and stereotyping;
Page 6
www.nbcot.org
• Coding items to the approved examination blueprint
specifications;
www.nbcot.org
Page 7
QQ:
AA:
QQ:
AA:
• Referencing items to approved and published resources in occupational
therapy;
• Discriminating power: Point-biserial correlations evaluate how well an item can
discriminate between highly proficient and less proficient candidates.
• Selecting item writers who are OTR and COTA practitioners and educators from
diverse geographical areas, practice experiences, and cultures; and
• Distractor effectiveness:
• Field testing items prior to their use as scored items on the exam.
• All distractors must attract some relatively unknowledgeable candidates in
order to enhance the probability that unknowledgeable candidates will
guess correctly.
How are NBCOT certification examinations constructed?
OTR and COTA certification examinations are constructed using a combination of scored (preequated) items and non-scored (field-test) items. Items for each examination are selected
from the respective banks in proportions that reflect the requirements of the OTR and COTA
blueprints. Items selected as scored items for each examination must have been pre-tested
on a sufficiently large sample of candidates and have acceptable item-level statistics. The
newly constructed exam forms undergo a rigorous review and validation process with a
committee of subject matter experts to ensure the examinations contain content reflective of
current entry-level practice.
Why are field-test (unscored) items used on an exam?
The NBCOT exams include a pre-selected number of field-test items on each test form.
Although these items are not considered when scoring candidates’ exams, performance
data is collected and analyzed. This statistical analysis is an important quality control step
that NBCOT uses to preserve the reliability of the examinations. Candidates are not able to
distinguish between the scored and unscored items.
Once a sufficient number of responses are collected on an item, the item statistics are
reviewed based on pre-determined metrics. Items meeting these metrics are entered into the
bank of items that can be used as scored items on subsequent exams. Item-level statistics
falling below these metrics are used to flag items that need additional review and revision
before undergoing further levels of field-testing.
QQ:
AA:
• For the purposes of the examination, distractors (incorrect responses)
should compete with correct answers as plausible choices.
What do you mean by item-level statistics?
• Test reliability: KR (20) internal consistency reliability measures are used to measure
the overall quality of the test.
QQ:
AA:
QQ:
AA:
• Misfit measures: These measures calculate how well the item fits the chosen IRT
model, essentially assesses whether an item is measuring the same thing as other
items on a test form, or how well it “fits” with other items on the exam.
Are there any other benefits to using IRT methodology?
®
In addition to enhancing the reliability of the exams, the advantages of building preequated test forms using IRT include a significant reduction in the waiting period between
the initial administration of a new form and the release of score reports. Because item
analysis, verification of answer keys, and test equating are completed before new forms are
administered, score reports can be issued with minimal delay.
If the tests are pre-calibrated, why doesn’t NBCOT provide test scores
immediately upon completion of the exam?
Calibration of the test items using IRT statistics is only one of the quality control procedures
NBCOT uses to ensure candidates receive accurate scoring information. Additional
quality control measures regarding test administration processes and procedures take
place after administration of an examination and prior to sending an official score. Based
on the extensive quality control measures in place for examination scoring, candidates can
be assured of accurate score reporting.
NBCOT uses Item Response Theory (IRT) methodology to analyze and calibrate examination
items and to pre-equate test forms. IRT statistics provide test developers with valuable
information on the psychometric properties of each item in the bank. Access to this
information during test construction facilitates the selection of appropriate items for the
new test forms. For each item on the examination, statistics are generated as follows and
evaluated against pre-determined metrics established for the program:
• Item difficulty: b-parameter evaluations describe the degree of challenge or easiness
of the item
Page 8
www.nbcot.org
www.nbcot.org
Page 9
SUMMARY
REFERENCES
This monograph addresses questions commonly asked about the foundational processes and procedures
NBCOT employs for its certification examinations. NBCOT examinations are “high stakes” examinations.
To ensure the defensibility of these examinations, NBCOT uses rigorous psychometric methods and
applies multiple levels of quality controls during every aspect of the examination development process.
Adhering to accredited certification standards, NBCOT serves the public interest by advancing client care
and professional practice through evidence-based certification standards and the validation of knowledge
essential for effective practice in occupational therapy. Through continuous review of standards and
processes, stakeholders can be assured that the NBCOT certification examinations are valid and reliable
measures of occupational therapy.
American Educational Research Association, American Psychological Association, National Council on Measurement in
Education (1999). Standards for Educational and Psychological Testing. Washington, D.C.: AERA.
American Society for Quality. (2005). ASQ Certification. Retrieved February 9, 2007, from http://asq.org/certification/
Angoff, W. H. (1971). Scales, Norms, and Equivalent Scores. In Thorndike, R.L. (Ed.), Educational Measurement, 2nd Edition.
Washington, DC: American Council on Education.
DeBaugh, M. (2003, November/December). The Value of Professional Certification. ABA Bank Marketing, 24(11), 36-39.
Downing, S.M., & Haladyna, T.M. (2006). Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Equal Employment Opportunity Commission (1978). The Office of Personnel Management, U.S. Department of Justice and
U.S. Department of Labor (1979). Questions and Answers Clarifying and Interpreting the Uniform Guidelines on
Employee Selection Procedures. 29 CFR Part 1607 (1988).
®
Equal Employment Opportunity Commission (1978). The Office of Personnel Management, U.S. Department of Justice and
U.S. Department of Labor (1979). Uniform Guidelines on Employee Selection Procedures. 41 CFR Part 603 (1978).
Impara, J.C. & Plake, B.S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353366.
Meisinger, S. (2004, April). Certification is Worth the Effort … and Then Some. HR Magazine, 49(4), 12.
Microsoft (2007). The Value of Certification: Connecting the Dots Between Employers and Employees. Whitepaper retrieved
April 22, 2012, from http://www.microsoft.com
Mulkey, J., & Naughton, J. (2005, January). 10 Myths of Certification, T + D, 59(1), 20-29.
National Organization for Competency Assurance (2005). Standards for the Accreditation of Certification Programs.
Washington, DC: NOCA.
NBCOT (2013). 2012 Practice Analysis of the Certified Occupational Therapy Assistant COTA®: Executive Summary. Retrieved
April 2, 2014, from http://www.nbcot.org
NBCOT (2013). 2012 Practice Analysis of the Occupational Therapist Registered OTR®: Executive Summary. Retrieved April 2,
2014, from http://www.nbcot.org
NBCOT (2013). Validated Domain, Task, Knowledge Statements: Certified Occupational Therapy Assistant COTA®. Retrieved
April 2, 2014, from http://www.nbcot.org
NBCOT (2013). Validated Domain, Task, Knowledge Statements: Occupational Therapist Registered OTR®. Retrieved April 2,
2014, from http://www.nbcot.org
Roberts, C. (2005, August). Building a Better Technician. Fleet Equipment, 31(8), 27.
Schoneboom, K. (2005, October). Professional Certification: What’s In It For Me? Office Pro, 65(7), 25.
U.S. Department of Labor (1993). JTPA: Improving assessment: A technical assistance guide. Washington, DC: Author.
Page 10
www.nbcot.org
www.nbcot.org
Page 11
NOTES
®
National Board for Certification in Occupational Therapy, Inc.
12 South Summit Avenue, Suite 100
Gaithersburg, MD 20877
P: 301.990.7979 F: 301-869.8492
www.nbcot.org