Technical
Specifications
Report with
Cut Scores
2014
American Chiropractic Neurology
Board
Technical Specifications Report with Cut Scores 2014
Table of Contents
Examination Construction/Development
2
Security Overview
2
Written Test Development Overview
3
Written Test Development Process
4
Performance Test Development
6
ADA & ESL Accommodations for Examinations
7
Examination Grading
7
Cut Score Setting
8
Examination Maintenance Policy
9
Equating
10
Complaints/Appeals of Examination
10
Reexamination Policy
10
References
12
1|Page
Technical Specifications Report with Cut Scores 2014
Examination Construction/Development
Security Overview
All examination materials are copyrighted material of the American Chiropractic
Neurology Board, Inc. (hereafter indicated by ACNB). No reproduction or
duplication of the material is permitted, unless authorized by in writing by the
ACNB. When not in use, all physical draft materials utilized by examination
development committee members are kept secured in a double locked enclosed
area, as are the computers and the encrypted, security code accessed electronic
devices. All these items are inventoried regularly.
All committee members sign confidentiality agreements prior to participating in any
portion of the test development or administration process. Secure HighTail®
(previously YouSendIt®) email or secure Dropbox® shared communication is used
for all restricted information and no individual has access to the whole test outside
of the physical office.
Any material considered unusable is shredded prior to disposal. Any electronic
material is deleted and then electronically erased. Historical examination materials
are kept in locked, fireproof file cabinets in a locked room and on an encrypted hard
drive in a safe in a locked room.
All duplication of examination materials has been done in a closed environment
under secure arrangements. Schroeder Measurement Technologies (SMT)
administers the written examination with the test bank maintained in their test
vault. All test items are encrypted and password secured on a CD or DVD and
couriered to SMT and raw data is encrypted and password protected and couriered
from SMT back to the ACNB offices.
All test materials administered by SMT are delivered via Internet to proctored test
sites worldwide by per-arranged scheduling and registration of each candidate.
Passport, driver’s license or government issued identification documents is required
for identity verification before the candidate is admitted into the testing facility. The
written examination is divided into three sections due to time constraints. Each
section of the examination requires the candidate to use a unique password in order
to login to SMT’s website. The proctor is responsible for maintaining that protocol
and troubleshooting any computer or electronic issues.
Performance/Practical exam materials are also couriered to the administration sites
where trained examiners receive them and verify the inventory of each item in the
containers. Flash drives for the video clip portion are password protected and
encrypted. These drives undergo a second level of inventory. Each item is
inventoried prior to being sent by courier back to the ACNB.
2|Page
Technical Specifications Report with Cut Scores 2014
Written Test Development Overview
All candidates are examined using the same form of the comprehensive written
examination. No equivalent forms of the competency evaluation are needed.
The ACNB has established a policy that all examinations will be generated in
English. Written translations of the competency evaluation are not available. No
adaptations of the evaluation material are utilized at this time.
Test items in the examination are keyed to an examination blueprint derived from
the most recent Job Analysis or Job Analysis update and validated by the collective
judgment of the subject matter experts then used by a series of committees involved
in the test construction process.
These committees include objective writers, case study writers, item writers and
distracter writers. These committees validate items using source documents from
the chiropractic neurology field.
All of these activities are conducted under the direction of consultant
psychometricians from the Baylor University Department of Education Psychology
and are subject to the standards acceptable to the Institute for Credentialing
Excellence/National Commission for Competency Assurance (ICE/NCCA) (USA).
There is a final review panel, outside of the examination development committees,
to review the work of the committees and to monitor the validation procedures
used by the committees, the linkages of the test items to the examination blueprint
and associated content specifications.
The procedures used by the panel minimize content error on the part of the
committees and provide assurances that the committees followed generally
accepted principles in item development.
Each examination question undergoes this process and then the items are entered
into an item-banking computerized program, (FileMaker Pro relational database)
allowing the Board to maintain all of the specifications of the test items relating to
its linkage to the examination blueprint, job analysis and content validity.
After a sufficient pool of questions has been developed and banked into the
software, an assessment is made of the number of test items developed for each
content domain to ensure that there is a sufficient pool of items in each
major/minor content area.
Once a sufficient pool of items exists, a form of the examination is generated
according to the content specifications. This form is used in yet another field test by
a selected group of Diplomates who sit the exam as pilot test takers, after these
individuals have signed the appropriate confidentiality forms.
A test review committee and rates the results from the pilot test. Then, this portion
of the exam is included for the performance exam. A consultant psychometrician
3|Page
Technical Specifications Report with Cut Scores 2014
analyzes the pilot scoring data, reviews comments by the pilot group and makes
recommendations for edits to the Test Development committee. The consultant
performs the same analysis on the subsequent exam data.
Written Test Development Process
The Test Blueprint is developed from the Job Analysis and Job Analysis Update
based on the calculations described in the Job Analysis 2014 document. The domain
areas (cognitive domains identified in Job Analysis) and the work activities
(objectives identified in Job Analysis) are created so that the they are represented in
number equal to the percentage of the total written exam or performance exam or
both as enumerated in the Exam Blueprints.
Because the field of Chiropractic Neurology does not have a mandatory residency
requirement, the Board continues to require a more comprehensive written exam
consisting of 450 items. Each year, the blueprints are updated to adjust for changes
in the field reflected by the Job Analysis updates.
The 2012 exam piloted a category called “Red Flag” cases that were defined by the
2010 Job Analysis update and are contained in the individual areas of the neuraxis.
The 2013 exam incorporated these cases in the graded portion of the examination.
(See Written Exam Blueprint 2013).
The 2014 Exam Blueprint reflects the finalized Job Analysis that was published in
January 2014. The 2014 examination is still under construction for administration
in August of 2014.
The process for test development reflects the complexity of the work required to
practice chiropractic neurology. This is a post-doctoral healthcare field in the
specialty of neurology.
The written examination covers the higher order thinking necessary in evaluating
history and physical examination data, diagnosing health issues, ordering special
studies, developing treatment and rehabilitation modalities, identifying the need to
refer to another medical professional and analyzing for appropriate referral
resources and evaluating the effects of treatment. Case presentations comprise 75%
of the written exam with 25% short scenarios or non-case related items. Cases are
submitted by certificants (Diplomates) in good standing from all over the world.
The cases involve patients that have been seen by our doctors. The doctors are
required to submit the case using the Case Template so that we have consistency in
presentation.
All data that is normal is listed as Within Normal Limits (WNL) so as to minimize
extraneous reading on the examination. Each section of the three-section written
examination contains between 15 to 20 cases. The case history is maintained for
each case ever included on the ACNB examinations. Cases are rotated out of the
exam at least every five years.
4|Page
Technical Specifications Report with Cut Scores 2014
Cases for which the post exam analysis rates as poorly representing the area they
are supposed to cover are rotated off after the first year. The head of test
development categorizes all cases that are to be reused from the previous year’s
examination and all cases submitted.
Each case is entered into topical bins. Then, these cases are compared to the newest
Job Analysis and chosen for the examination according to that year's blueprint. Over
50 doctors submit cases each year, which results in a rich case bank. Not all cases
submitted meet the criteria for test usage.
Test development teams of Subject Matter Experts (SME) are chosen and trained in
item writing, especially higher order item writing using Bloom’s Taxonomy
(Anderson, 2001). The team composition includes at least one person with English
as a Second Language (ESL), one person from Australia, one from Canada and two
people from the United States.
Each 5-person team is responsible for no more than 10 cases. The Team Leader
assigns cases from the case bank, one to each member. The first member for that
case edits the case itself, writes the stems for the case using the Work Activities list
in the order that makes sense clinically and writes the correct answer with resource
validation. The case is then passed to the second team member who writes the one
series of distracters. The third team member reviews, edits and writes the second
distracter series and the fourth team member completes the set with the third
distracter.
Because clinical decisions are based upon diagnosis, the distracters must agree with
the incorrect diagnosis from that particular distracter series.
The Team Leader provides the final, comprehensive edit and then submits the case
with the items and distracters to the Test Development Director. The response
possibilities for each question are shuffled using a random number generator.
Once all new cases, their accompanying items and distracters have been submitted
back to the Test Development Director, the SME test compilation and ordering
committee meet face-to-face for three days to review each case, its references, the
items and distracters. This team then orders the cases for each section (150 items)
of the exam so that each section is similar in reading time.
All candidates take all three sections of 150 items each; however, having one section
much more reading intensive is not conducive to optimal performance. Since we are
not testing reading but clinical skills and judgment, it is important to load the
reading requirement for the three sections equally.
When the test is administered, each candidate is given the three-section test in
order of section but the cases with their accompanying items are scrambled within
the section. Each item is specially coded so that data analysis can include Domain
and Work Activity analysis.
5|Page
Technical Specifications Report with Cut Scores 2014
The last stage is a segregated proof reading. At least three different individuals are
responsible for the proof reading and grammar checking. These are usually
individuals with Masters Degrees in English. Again, confidentiality is maintained and
each individual is required to sign a confidentiality agreement. Finally, the exam is
couriered to SMT and their proofreader reviews each item and adds it to our Test
Bank in the SMT vault.
Performance Test Development
Test item development for the Performance examination is derived from the Job
Analysis and its updates. A history-taking portion by the candidates was piloted as
research in 2013 and was deemed unsuccessful.
The reasons for the lack of success were related to the method used to request this
history and the fact that the candidates wrote their process of history taking by
hand in blue books. Their handwriting was not reliably readable, so the scores were
not included. As a result and in consultation with Baylor University Educational
Psychology department, a new method developed more in agreement with the 2014
Job Analysis.
This method is undergoing pilot testing with existing certificants and, if an
appropriate grading rubric can be derived, used in the 2014 Performance
examination. There will be four history forms to the test that will be analyzed for
equality in difficulty and a separate cut score will be calculated for each.
A set of SMEs, different from those for the written exam, developed a patient
examination form derived from a collection thirty of such forms from our
certificants’ offices.
The information from these forms was separated into items. The items agreeing
with the blueprint were chosen and arranged in a systematic examination order.
The candidates must use some systematic process to record the history of the
patient, not necessarily the order on the form but including the items on the form.
Every candidate is provided with the form both in the study guide materials as well
as a clean copy during the examination itself. Since, in practice, doctors use these
forms and make notes on them, we were unable to justify not allowing the
candidates access to the forms.
The video clips portion of the examination consists of eighteen clips with one to two
questions about each clip. The question may ask for the finding (i.e. strength of the
reflex) or for the level of the cause of the abnormality or for the diagnosis suggested
by the finding. The candidates are provided two websites where normal and
abnormal video clips are provided for their study and that cover the entire
neurological examination.
6|Page
Technical Specifications Report with Cut Scores 2014
Every item on the physical exam has the potential for being offered in a video
simulation as part of the exam. This adheres to the blueprints derived from the job
analysis.
There are five forms of the video clips portion of the examination. These forms
cover exactly the same case information but with a different video clip. In order to
assure fair grading, the cut score panel analyzes each clip and the scores are taken
for the exact clips used in each form.
ADA & ESL Accommodations for Examinations
Accommodations for candidates with disabilities are handled on an individual basis
and in accordance with the Americans with Disabilities Act upon special request and
validation of the disability. In accordance with American Psychological Association
Standards for Test Administration, additional time may be provided to candidates
who train in a language other than English, specifically in the performance/oral
examination. Again, this must be requested in advance. Since reading speed is not
one of the constructs of testing, any candidate may request extra time after taking
the practice test and evaluating his or her speed in reading the case material and
items.
The ACNB permits any candidate to have one or two (must be specified) hours in
addition to the normal four hours per part of the exam. There is a sitting fee charged
by SMT that is gauged by the hour, so this must be arranged in advance. The
proctored sites must know in advance the time allotted so that scheduling conflicts
may be avoided.
Examination Grading
The ACNB written examination is electronically administered by Schroeder
Measurement Technologies (SMT) and the raw data sent to the ACNB for analysis
against the scoring key. The ACNB provides full psychometric analysis including
item analysis for inclusion or exclusion from the exam. The data is then compared to
the cut score and the Pass or Fail status for the candidate is determined.
For the Performance Examination, two individual raters who are experienced
Diplomates and have been fully trained in the procedures (see the Associate
Examiner Training Manual) rate each candidate’s performance. If the agreement
between these two raters is less than 95%, a second set of raters regrades the test.
Then, the scores are compared to the cut score and a grade of Pass or Fail is
determined.
The data undergoes further analysis in order to provide the areas of candidate
strengths and weaknesses. The ACNB does not round up on the written exam but
does on the Performance Exam because of the observational nature of that exam.
7|Page
Technical Specifications Report with Cut Scores 2014
The final scores are presented to the Board of Directors for acceptance or
instruction to raise or lower the cut score by one or more standard error of the
measurement.
Cut Score Setting
A cut-score committee was assembled to establish the passing point of the Written
and Performance examinations. The ACNB selected committee members by
screening for their qualifications in the field of chiropractic neurology requiring that
each had to be a certificant in good standing. The facilitator, the data entry
technician and seven Subject Member Experts (SMEs) qualified for the committee
fulfilling the following attributes for the group:

Two Experienced (5 or more years as a Diplomate in good standing),

Two Inexperienced (2 or fewer years as a Diplomate),

One Faculty at a chiropractic college,

One Faculty in one of the training programs leading to eligibility

One with English as a Second Language (ESL).
The facilitator holds a Ph.D. in Experimental Psychology and has extensive
experience in certification exam development in nursing as well as chiropractic
neurology. The data entry technician holds at least Bachelor’s Degree. The ESL
member’s primary language is dictated by determining the most represented first
language of all the registered candidates once the candidate registration is complete.
The geographical representation included North America, Australia, South Korea
and Europe.
The members of the cut score committee selected did not include any Diplomates
(certificants) involved in the item development process.
The cut-score procedure used is a modified Angoff procedure for determining the
various passing points on both written and performance examinations.
Each member of the committee rates each item used in the grading of the
examinations and a running mean score is calculated. When completed, the cut
score for each of the examinations is determined, including the variance, standard
deviation, KR-20 value, and the standard error of measurement.
Each member of the multiple-choice exam and the dichotomous performance exam
Angoff cut-score panel is responsible for determining the passing point as it relates
to the minimally competent candidate.
Prior to their initial rating, committee members are trained on cut-score
methodology and provided a worksheet outlining the process with space on the
worksheet to be used in describing the minimally competent candidate. The
8|Page
Technical Specifications Report with Cut Scores 2014
committee meets via webinars for the three days of the panel’s work. They are
provided the candidate study guide as well as the training materials provided to
item writers and Performance exam raters. The panel takes the examination as a
part of the Angoff process.
The results of the 2013 Angoff panel are listed below. The ACNB does not publish
the actual scores.
Examinations for 2013
Cut Scores for 2013
Written Examination
245 raw score
Performance Examination Form A
448 weighted raw score
Performance Examination Form B
453.9 weighted raw score
Performance Examination Form C
449.7 weighted raw score
Performance Examination Form D
444.9 weighted raw score
Performance Examination Form E
440.7 weighted raw score
Examination Maintenance Policy
All examination materials, including draft materials, electronic forms, reports, and
data analysis files are retained in a secure, double-locked, temperature controlled
environment and inventoried in great detail for legal defensibility purposes.
Validation materials, including the production of job analysis data, content
specifications, examination blueprints, and source documents are retained,
inventoried, and controlled with limited access in a secure, well-ventilated
environment.
Personnel of the Board, identified Board members, and consultants have access to
the assessment instruments and associated data. Documentation logs are kept
outlining who is accessing which materials on specified dates and times.
All assessment instruments have control numbers for quality control purposes. Any
assessment instruments deemed unusable are shredded prior to disposal. Control
numbers on the disposed instruments are invalid for future use. All assessment
instruments are reviewed for accuracy and any discrepancies by at least two
individuals, including the consultant, prior to use.
Electronic forms are backed up routinely for security purposes. Access to electronic
data is limited to only a select group of individuals that are directly involved with
the assessment process. Different levels of access to electronic data are based on
criteria of need. Assessment results are retained in electronic fashion indefinitely
with back-up copies. Printed results are maintained for seven (7) years. Aggregate
data of assessment results produced for documentation are maintained for the life
9|Page
Technical Specifications Report with Cut Scores 2014
of the document.
Equating
The American Chiropractic Neurology Board does not use separate forms for the
written test. Each year twenty percent (20%) of the questions are rotated out and
replaced with new ones for current trends in practice and security purposes.
The American Chiropractic Neurology Board maintains five forms of the
Performance Examination. In all forms, the physical examination remains identical
as this represents current practice as elucidated by the Job Analysis. The history for
clarification and the video-clips viewed by the candidates are different but similar in
content for each form. The equating methodology is via Angoff cut score setting. The
Angoff Panel rates each history form and each video-clip question separately and
the ratings obtained are total to reach the cut score for each particular form. That is,
an Angoff cut score is set for each form of the Practical Examination according to the
difficulty of the items on that form.
Complaints/Appeals of Examination
1. The Board must receive Complaints/Appeals within 3 months of the mailing
date of exam results to be considered.
2. The cost of reviewing failed parts of the examination has been and will
continue to be non-refundable and set annually by the Board.
3. The written examination data was provided from SMT and analyzed by the
ACNB psychometric consultant, Donath Group or the consultants from Baylor
University Educational Psychology department and is not subject to regrading. Other complaints regarding the examination will be adjudicated
according to the general Appeals policy and procedure.
4. Appeal of the Performance examination consists of two Board members who
did not grade that candidate. They review and rescore the examination from
the DVD made from the videotape of the examination. If the two Board
members disagree on the scores, a third will re-grade the examination and
the like scores will be used. This will occur within 30 days after the appeal is
made.
5. See the General Complaints/Appeals Policy in the ACNB Policies and
Procedures Handbook to take the process further than that described above.
Reexamination Policy
Candidates may retake only the part(s) of the examination that were failed. Failed
10 | P a g e
Technical Specifications Report with Cut Scores 2014
parts of the examination must be rewritten within 3 years of the initial failure or the
candidate must rewrite the entire examination, unless special authorization by the
Board of Directors is received. Furthermore, once a candidate has unsuccessfully sat
for the examination 3 times, he/she must return to the neurology education and
retake the 100 hours to requalify for the examination and retake the entire
examination.
The American Chiropractic Neurology Board sets a reexamination fee annually for
the performance examination and the written examination. This fee is the same as
the fee to take each portion of the examination for the first time.
11 | P a g e
Technical Specifications Report with Cut Scores 2014
References
Anderson, Lorin. A Taxonomy for Learning, Teaching, and Assessing: A Revision of
Bloom's Taxonomy of Educational Objectives. New York: Longman, 2001, Print
Joint Committee on Standards for Educational and Psychological Testing of the
American Educational Research Association, American Psychological Association,
and National Council on Measurement in Education. The Standards for Educational
and Psychological Testing. Washington, American Educational Research Association,
1999. Print
12 | P a g e