RN Content Mastery Series 2007
National Standard Setting Study
Executive Summary
The purpose of this study was to develop a set of recommended criterion-referenced cut scores
that nursing schools might choose to adopt as benchmarks for student performance on each of
the nine RN Content Mastery Series (CMS) assessments.
Methods
Criterion-referenced cut scores were set at three levels of proficiency for each of the CMS
assessments. The Angoff method was used to set the lowest level cut score. This method lends
itself especially well to setting a single minimum standard. Using the Angoff method, content
expert judges first developed a definition of what characterizes a student who just meets the
minimum standard (see page 3) and then rated the difficulty of each test item as it related to this
definition. Note that these definitions were based on expertise of the nurse educators who
participated in the cut score study. No empirical study was conducted relating NCLEX-RN®
performance to performance on the Content Mastery Series, nor was any study
conducted demonstrating a statistical relationship between Content Mastery Series
performance and actual job performance. These Proficiency Level definitions were used
by cut score study participants for the purpose of making their empirical ratings of item
difficulty. Cut scores were derived from the mean of these item difficulty ratings.
The Bookmark method was used to set the two higher proficiency levels. Definitions of what
characterizes a student at these two levels were developed in the same manner as the Angoff
method. Instead of making individual item difficulty ratings, content experts using the
Bookmark method made a more holistic decision relating the difficulty of the test to the
proficiency level definitions. The items on the test were presented to the content experts in
difficulty order, with the easiest item first. The task of the content experts was to place a marker
between the two items that were perceived to be the dividing point between students at and
below the proficiency level in question. Cut scores were derived from the median rating of the
content experts.
Selection of the content expert judges. Invitations to participate in the study were sent via e-mail
to nurse educators at each of the RN nursing programs on ATI’s list of clients, and to nurse
educators specifically recommended by members of ATI’s sales staff. A total of 282 invitations
were sent out, from which 108 responses of interest in participation were received. Those
selected to be content expert judges had significant teaching experience in the content area(s) for
which they were to set standards, with most having more than 10 years of teaching experience.
The content experts selected to participate were nurse educators serving in RN programs and
educational institutions of various sizes and types from across the United States.
RN Content Mastery Series 2007 Standard Setting Study
1
ATI owes a great deal of gratitude to these participants. As a result of their hard work and
expertise, nursing programs throughout the country will now have criterion-referenced levels of
proficiency to set defensible benchmarks for student performance on each of the ATI RN
Content Mastery Series assessments.
Use of the ATI Recommended Cut Scores
The CMS is designed to provide assessment data regarding a student’s mastery of concepts
within specific nursing content areas and a formative indication of developing NCLEX®
readiness in these content areas. The CMS is not meant to be a direct predictor of NCLEX®
content area performance or future success in nursing. The fundamental question the CMS
attempts to answer is, “How much proficiency has the student attained in this content area?”
Institutions using the RN CMS for high-stakes purposes (e.g. portion of course grade,
progression) should be extremely cautious, regardless of whether the ATI recommended cut
scores are adopted. If these assessments are used for a portion of a course grade beyond extra
credit, institutions should ensure that the content of the assessments is closely aligned with
program curriculum. ATI strongly discourages the use of the CMS assessments as a sole
criterion for progression or graduation.
Nursing education institutions should recognize that the cut scores set as a result of this study are
recommended rather than required for those using the ATI RN Content Mastery Series
assessments. Institutions are free to use these cut scores or to set their own as they see fit. It
should be recognized that these cut scores were established through a formal study in accordance
with the recommendations of the American Educational Research Association, American
Psychological Association, and National Council on Measurement in Education Standards for
Educational and Psychological Testing. It is recommended that institutions wishing to set their
own cut scores contact the ATI Director of Psychometrics to discuss defensible methods of doing
so.
The percentage of students in each of the four categories shown in Table 1 was derived by
applying the cut scores established in this study to all students who took the RN CMS
assessments during norming, conducted October through December of 2007. Institutions should
not necessarily expect the same percentage classifications to apply to their own students when
using the ATI recommended cut scores. The interpretation of the Level 1, 2, 3, and Below Level
1 proficiency categorizations can best be determined from a review of the attached proficiency
level definitions and sample score report explanation.
RN Content Mastery Series 2007 Standard Setting Study
2
RN Content Mastery Series 2007
Proficiency Level Definitions
A student meeting the criterion for Proficiency Level 1:
-
is expected to just meet NCLEX-RN® standards in this content area.
should demonstrate the minimum level of knowledge in this content area required to
support academic readiness for subsequent curricular content.
should meet the absolute minimum expectations for performance in this content area.
A student meeting the criterion for Proficiency Level 2:
-
is expected to readily meet NCLEX-RN® standards in this content area.
should demonstrate a level of knowledge in this content area that more than adequately
supports academic readiness for subsequent curricular content.
should exceed minimum expectations for performance in this content area.
A student meeting the criterion for Proficiency Level 3:
-
is expected to exceed NCLEX-RN® standards in this content area.
should demonstrate a high level of knowledge in this content area that confidently
supports academic readiness for subsequent curricular content.
should exceed most expectations for performance in this content area.
Note that these definitions were based on expertise of the nurse educators who
participated in the cut score study. No empirical study was conducted relating NCLEXRN® performance to performance on the Content Mastery Series, nor was any study
conducted demonstrating a statistical relationship between Content Mastery Series
performance and actual job performance. These Proficiency Level definitions were used
by cut score study participants for the purpose of making their empirical ratings of item
difficulty.
RN Content Mastery Series 2007 Standard Setting Study
3
RN Content Mastery Series 2007 Standard Setting Study
Table 1. ATI recommended cut scores, percentage of students in each category, and number of students taking the CMS assessments.
% of
% of
% of
% of
Number of
Students
Students
Students
RN CMS 2007 Proctored Level 1 Cut Level 2 Cut Level 3 Cut Students
Students
Meeting
Meeting
Meeting
Assessment
Score
Score
Score
Below
Taking the
Proficiency Proficiency Proficiency
Level 1
Assessment
Level 1
Level 2
Level 3
Fundamentals for
Nursing Practice
10.7%
27.6%
53.4%
8.4%
1938
58.3%
66.7%
80.0%
(60 items)
Adult Medical-Surgical
16.1%
43.6%
32.2%
8.1%
1295
54.4%
64.4%
73.3%
Nursing (90 items)
Nursing Leadership
24.5%
46.3%
27.8%
1.4%
216
61.7%
70.0%
80.0%
(60 items)
Community Health
12.7%
32.4%
50.7%
4.2%
71
56.0%
66.0%
80.0%
Nursing (50 items)
Maternal-Newborn
23.5%
35.2%
38.7%
2.6%
821
63.3%
71.7%
85.0%
Nursing (60 items)
Nursing Care of Children
25.2%
32.7%
39.0%
3.1%
718
56.7%
65.0%
78.3%
(60 items)
Nursing Pharmacology
20.6%
36.9%
34.1%
8.5%
1113
50.0%
63.3%
76.7%
(60 items)
Mental Health Nursing
22.7%
34.3%
40.9%
2.2%
905
61.7%
70.0%
85.0%
(60 items)
Nutrition for Nursing
27.7%
43.1%
24.9%
4.4%
253
46.7%
58.3%
68.3%
(60 items)
4
Sample Score Report Excerpt
Individual Performance Profile
RN Fundamentals 2007 Form A
Individual Name:
XXXXXXXXXXXXXX
Adjusted Individual Total Score:
76.7%
Student Number
1258998797
ATI Proficiency Level:
Institution:
XXXXXXXXXXXXXX ADN
Mean - National:
69.1%
Program Type:
ADN
Mean – Program Type:
69.4%
Test Date:
6/19/2008
Percentile Rank-National:
84
# of Questions:
60
Percentile Rank-Program Type:
83
Level 2
Individual Performance in the Major Content Areas*
Mean
Sub-Scale
#
items
Individual
Score
National
Program
_ _Type_
Percentile Rank
Program
National
Type
1
Individual Score (% correct)
10 20 30 40 50 60 70 80 90
Management of Care .......................... 4............... 63.0% ......... ..................... ..................... ............................................................... ▲
Safety and Infection Control................ 8............... 55.0% .........54.3% ........... 53.1% ........... 60.............. 65 ....................................▲
Health Promotion and Maintenance.... 16............. 75.0% .........71.3% ........... 69.5% ........... 70.............. 75 ................................................ ▲
Psychosocial Integrity ......................... 11............. 75.0% .........71.0% ........... 70.4% ........... 73.............. 75 .................................................▲
Basic Care and Comfort...................... 13............. 80.0% .........73.3% ........... 76.0% ........... 79.............. 73 ................................................... ▲
Pharmacological & Parenteral ............ 5............... 80.0% .........73.3% ........... 76.0% ........... 79.............. 73 ................................................▲
Reduction of Risk Potential ................. 2............... 75.0% ......... ..................... ..................... .................. ..................................................... ▲
Physiological Adaptation ..................... 1............... 75.0% ......... ..................... ..................... .................. ......................................................▲
* Means and percentile ranks for sub-scales with fewer than 5 items are not presented.
RN Content Mastery Series 2007 Standard Setting Study
5
100
Score Explanation and Interpretation
Individual Performance Profile
Below Level 1 – Scores below the Proficiency Level 1
Adjusted Individual Total Score:
The adjusted individual total score is a function of the number
of questions answered correctly divided by the number of
scored questions on the assessment. This percentagecorrect score is then adjusted to account for differences in
the difficulty of the form taken, producing the adjusted
individual total score. For example:
Norm-Referenced Measures:
Number of questions answered correctly
Total number of scored questions on the assessment
standard can be considered below minimum expectations
and may be indicative of significant risk in this content area.
ATI strongly advises these students to develop and complete
an intensive plan for focused review and remediation,
including the use of ATI materials, textbooks, class notes,
reference materials, and assistance from nurse educators.
=
Adjustment
50 = 83.3% ( for form ) = Adjusted Individual
60
difficulty
Total Score
Individual scores can be interpreted through “criterionreferenced” or “norm-referenced” measures. Criterionreferenced measures are best used to determine if an
established standard has been met. Norm-referenced
measures can be useful for comparing performance to other
students.
Criterion-Referenced ATI Proficiency
Levels:
These classifications were developed as the result of a
national standard setting study conducted by ATI,
involving nurse educator content experts from across
the U.S.
Level 3 - Scores meeting the Proficiency Level 3 standard
may be considered to exceed most expectations for
performance in this content area. Scores at this level were
judged by the content expert panel to indicate a student as
®
likely to exceed NCLEX-RN standards in this content area.
ATI advises these students to engage in continuous focused
review to maintain and improve their knowledge of this
content.
Level 2 - Scores meeting the Proficiency Level 2 standard
may be considered to exceed minimum expectations for
performance in this content area. Scores at this level were
judged by the content expert panel to indicate a student as
fairly certain to meet NCLEX-RN® standards in this content
area. ATI advises these students to engage in continuous
focused review in order to improve their knowledge of this
content.
Level 1 - Scores meeting the Proficiency Level 1 standard
may be considered to meet the absolute minimum
expectations for performance in this content area. Scores at
this level were judged by the content expert panel to indicate
a student as likely to just meet NCLEX-RN® standards in this
content area. ATI advises these students to develop and
complete a rigorous plan of focused review in order to
achieve a firmer grasp of this content.
RN Content Mastery Series 2007 Standard Setting Study
Means and percentile ranks can be useful for comparing
performance to other nursing students, both nationally and
within the same RN program type. These means and
percentile ranks are initially set on a volunteer norming
sample. They are reviewed annually, and may be
periodically reset as more students take the assessments.
Mean - National:
The national mean is the average of the individual scores of
all test takers (within a specified sample from the ATI data
pool) for this assessment. The national mean includes all RN
program types.
Mean - Program:
The program mean is the average of the individual scores of
all test takers of your RN program type (within a specified
sample from the ATI data pool) for this assessment.
Percentile Rank - National:
The national percentile rank refers to the proportion of test
takers from all types of RN nursing programs (within a
specified sample from the ATI data pool) whose scores are
the same as or lower than your individual score.
Percentile Rank - Program:
A program percentile rank refers to the proportion of test
takers from your specific type of RN nursing program (within
a specified sample from the ATI data pool) whose scores
were the same as or lower than your individual score.
NA: Data not available
Pretest Items: There are 5 unscored pretest questions
throughout the assessment, and 60 scored questions. The
pretest questions are used for research purposes.
Topics to Review:
Based on the questions missed on this assessment, a listing
of content areas and topics to review is provided. A variety
of learning resources may be used in the review process,
including select components of ATI’s Content Mastery
Series® review modules, online practice assessments, and
®
the ATI-PLAN nursing review DVD series.
6