3/5/2013
Use of Standardized Examinations in
Nursing Education Programs
Finding the Balance in Testing – Improving
Performance and Setting Consequences
Presenter:
Virginia D. Ayars, EdD, MS, RN
Nursing Education Consultant
Texas Board of Nursing
March 2013
Virginia D. Ayars, EdD, MS, RN
Texas Board of Nursing
Austin, TX
Lisette Barton, PhD, RN, FNP-BC
School of Nursing, University of Houston-Victoria,
Sugar Land, TX
Reasons for Consideration
•
•
•
•
Emotional issue for students and family
Reports to Boards of Nursing
Attorneys hired
Legislators contacted
Regulations
• Does your Board of Nursing (BON) have
regulations about the use of standardized
examinations in nursing education programs?
National Data
• Survey completed in February 2013
• Administered through NCSBN web survey tool
• Education consultants from 35 of 60 member
boards participated
▫
The NCSBN membership comprises boards of nursing in:
 Fifty U.S. states
 District of Columbia
 Four U.S. territories-American Samoa, Guam, Northern Mariana Islands, and the Virgin
Islands
 Four states have two boards of nursing, one for registered nurses (RNs) and one for
licensed practical/vocational nurses (LPN/VNs): California, Georgia, Louisiana and West
Virginia
 One state, Nebraska, has both the board of nursing and the board for advanced practice
nurses (APRNs) represented
Guidelines or Best Practices
• Does your BON offer guidelines or best practices
to assist nursing education programs in the
utilization of standardized examinations?
1
3/5/2013
Element of Course Grades
• Does your BON have a position statement about
nursing education programs utilizing
standardized examination scores as an element
in determining course grades?
Benefits
• What do you perceive to be the benefits to
nursing education programs that utilize
standardized examinations?
▫
▫
▫
▫
Validated psychometric properties
Assists in preparing students for NCLEX
Increases faculty accountability
Assists in program evaluation as compares student
performance to national standards
▫ Effective remediation tool
National Data
• Additional Comments:
▫ Recommend that programs utilize standardized
examinations for the intended purpose
▫ Programs need to utilize the exams as a resource to
improve program of study and faculty performance
rather than as high-stakes exit examinations
▫ Ensure that program policies reflect the intended use
of examinations to students
Student Progression or Graduation
• Does your BON have a position statement about
nursing education programs utilizing
standardized examination for student
progression or graduation?
Challenges
• What do you perceive to be the challenges to
nursing education programs that utilize
standardized examinations?
▫ If used as part of grade and to prevent progression in program,
may not be legally defensible
▫ Identification of correct cut-off score
▫ Inconsistent grade weights within courses and programs
▫ Costs to program or student
▫ Programs should rely on sound curriculum and teaching
strategies rather than on the standardized examinations
Texas Board of Nursing (BON)
• Texas BON does not have purview over the use
of standardized examinations
• Texas BON offers Education Guideline: Using
Standardized Examinations
• BON staff interviewed three vendors to
determine the intended purposes in developing
the Guideline
2
3/5/2013
Suggested Purposes
Survey
• An assessment tool of potential or enrolled
students
• An assessment tool of the program’s
effectiveness
• An assistive tool to familiarize students with
computerized testing
• Completed in February 2013
• 82 of 111 Texas BON approved professional
nursing education programs’ deans or directors
responded
• 74 of 99 Texas BON approved vocational nursing
education programs’ directors or coordinators
responded
Use in Nursing Programs
Identification of Exams Utilized
VN
RN
1.2%
6.8%
Yes
No
93.2%
Yes
No
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
98.8%
Purposes
VN
61.8%
44.1%
HESI
• RN Respondents:
▫
▫
▫
▫
NCLEX-RN predictor
Assessment of content mastery
Assessment of faculty/program effectiveness
Admission criteria (HESI A2)
ATI
NLN
Other
(please
specify)
65.4%
RN
38.5%
9.0%
1.3%
HESI
ATI
NLN
Other
(please
specify)
Frequency of Use During Program
VN
• VN Respondents:
▫ Assessment of content mastery
▫ NCLEX-PN predictor
▫ Assessment of faculty/program effectiveness
7.4%
1.5%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
50.0%
45.0%
40.0%
35.0%
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
RN
70.0%
45.5%
43.9%
57.3%
60.0%
33.3%
50.0%
40.0%
16.7%
21.2%
37.3% 41.3%
30.0%
28.0%
26.7%
Each
Beginning
End of
semester of program program
Other
(please
specify)
20.0%
10.0%
Each
course
Each
Beginning End of
semester
of
program
program
Other
(please
specify)
0.0%
Each
course
3
3/5/2013
Benefits of Use
Use of Supplemental Materials
VN
RN
100.0%
100.0%
80.0%
100.0%
41.8%
40.0%
27.3%
20.0%14.5%
20.0%
Other (please specify)
Indication of
instructional
weaknesses/strengths
12.5%
▫ Videos
▫ Practice exams
▫ Tutorials
79.7%
• RN Respondents:
60.0%
60.0%
75.0%
• VN Respondents:
100.0%
80.0%
65.5%
98.6% 87.5%
Supplemental Materials
120.0%
120.0%
86.1%
Data to compare to
benchmarks
86.3%
120.0%
100.0%
80.0%
60.0%
40.0%
20.0%
0.0%
Indication of needs of
individual students for
remediation
66.2%
Other (please specify)
17.7%
Indication of
instructional
weaknesses/strengths
Yes
No
Yes
No
Indication of curriculum
weaknesses/strengths
13.7%
33.8%
79.0% 77.4%
72.6%
Data to compare to
benchmarks
RN
98.4%
Indication of needs of
individual students for
remediation
VN
RN
VN
120.0%
100.0%
80.0%
60.0%
40.0%
20.0%
0.0%
Indication of curriculum
weaknesses/strengths
Incorporate into Course Grade
49.2%
40.0%
28.8%
13.6%
20.0%
8.5%
▫ Practice exams
▫ Review books
▫ Case studies
0.0%
0.0%
1
2
3
4
5
1
6
2
3
4
5
6
Affect Student Ability to Progress or
Graduate
VN
Number of Times Student May Retake
Exam
VN
RN
25.8%
40.3
%
59.7%
RN
15.3%
30.6%
0
52.1%
Yes
No
47.9%
Yes
No
16.7%
45.8%
0
1
9.7%
1
2
19.4%
14.5%
2
3
Other (please
specify)
9.7%
12.5%
3
Other (please
specify)
4
3/5/2013
Faculty Members Take Exams
VN
Exams Reflect Required Content
RN
RN
VN
0.0%
36.7%
38.6%
63.3%
No
Yes
No
Identification of Other Products
• VN Respondents:
▫
▫
▫
▫
Modules to assist ESL students
Medical Terminology modules
Increased information re. missed items
More case studies, videos/scenarios & practice questions
• RN Respondents:
▫
▫
▫
▫
More computer adaptive tests
Faculty training tools
More case studies for simulation lab use
Exit HESI linked to other products (case studies, critical thinking
exams, testing-writing tools, faculty training)
Faculty Member Comments
• VN Respondents:
▫ Highly positive responses
▫ Assist in self-evaluation & course evaluation
▫ Validated curriculum content
• RN Respondents:
▫ Evaluation of faculty performance re. instructional
methodologies
▫ Helps when counseling students and assists in guiding
students to address identified weaknesses
▫ Prepares students and validates performance
Yes
No
Yes
No
61.4%
Yes
1.4%
100.0
%
98.6
%
Student Comments
• VN Respondents:
▫ Hard, but empowering
▫ Anxiety producing
▫ Hate them, but love them after pass NCLEX!
• RN Respondents:
▫
▫
▫
▫
Students appreciate value after taking NCLEX
Exams are accurate predictors of NCLEX success
Exit HESI most helpful to students
Offers practice with NCLEX style questions
Additional Comments
• VN Respondents:
▫ Very comprehensive
▫ Highly beneficial
▫ Concerned about rising costs
• RN Respondents:
▫ Critical that students clearly understand requirements
upon admission to program
▫ Need to keep use in perspective
▫ Strive for balance between not teaching to the test, yet
ensuring students prepared for NCLEX
5
3/5/2013
Recommendations
• Placement across the curriculum with planned
remediation
• Academic progression policy to clarify use and frequency
of standardized examinations and test results
• Review of plan by school’s legal counsel
• Academic progression policy published in accessible and
public format
• Changes in policies implemented when student first
enrolls in program or at beginning of each class or
academic term
• Ask student to sign receipt of academic progression
policy
Cautions
• Not recommended to use as High-Stakes
Testing
• Place at appropriate points consistent with
didactic content and clinical learning
experiences
• Do not pattern curriculum changes solely upon
testing results
• Be aware of any security issues about the
standardized examination test pool
Questions?
• For more information, please contact:
▫ Virginia Ayars
▫ Virginia.ayars@bon.texas.gov
▫ 512-305-7660
• Thank you!
Standardized Predictive Testing:
Practices, Policies, and Outcomes
Lisette Barton, PhD, RN, FNP-BC
School of Nursing, University of Houston-Victoria,
Sugar Land, TX
Rae Langford, EdD, RN
College of Nursing, Texas Woman’s University, Houston, TX
Pamela Willson, PhD, RN, FNP-BC, CNE
Past Director of Research, Elsevier Review & Testing, Houston, TX
Background
• Nurse educators use standardized testing, curricular
standards and guidelines to direct educational
practice.
• Standardized testing predicts student success and
identifies student’s weak content areas to allow for a
focused remediation plan.
• Multiple remediation strategies are often
implemented simultaneously.
• Little is known about the effectiveness of
preparation , remediation, and retesting policies in
nursing educational programs in regards to
standardized outcome testing.
Purpose
The four primary aims of this study were:
• Describe the predictive accuracy of the HESI Exit
Exam for NCLEX-RN success.
• Describe schools remediation policies and practices.
• Determine if students who attend nursing programs
with certain high stakes testing policy components
score better on the HESI Exit Exam than students
who attend nursing programs without certain high
stakes testing policy components.
• Identify which combination of policy components
result in higher HESI Exit Scores.
6
3/5/2013
Sample
A random stratified sample was drawn from
Elsevier’s HESI™ Exit Exam users
1 September 2009 through 31 August
2010
SON Deans/Directors
were queried via password protected email survey
Conceptual Framework
• Classical test theory
▫ Reliability and Validity
▫ Crocker and Algina (1986)
471 schools were invited; 99 schools responded for a
21% response rate
• Vygotsky’s Zone of Proximal Development
▫ Phase I – Testing Policies & Practices
▫ Third Voice
▫ Developmental goal attainment
 Of the 99 schools, 64 reported testing policies and
practices for a 65% response rate
▫ Phase II – Predictive Accuracy of HESI Exit Exam
▫ Eun, Knoetek, & Heining-Boynton (2007)
 Of the 99 schools, 69 reported NCLEX-RN
outcomes for a 70% response rate
Sample
Phase I and
Phase II
Schools
Students
Phase I Instrument
Invited
471
Respon Total
ded
Sample
99
AD
BD
Response
Sample Sample Rate
69
40
29
5122
2869
2253
70%
Data
• HESI Exit Exams Scores
▫ 160-item (10 pilot items) comprehensive exam
▫ Most schools administered the E2 within 6 weeks prior
to completion of the nursing curriculum
• NCLEX-RN Outcomes
▫ First time test takers
▫ Pass/Fail; Unknown; Did not graduate
• HESI E2 Exam Practices/Strategies
Questionnaire
▫ Preparation Components
▫ Remediation Components
▫ Consequence Components
• HESI E2 Exit Exam Practices/Strategies
Questionnaire
• 29-item multiple choice and fill in the blank
electronic survey
▫
▫
▫
▫
▫
▫
▫
Exit Exam program policy
Benchmark use
Strategies used to prepare for the Exit Exam
Duration of time allotted to preparation
Re-testing practices
Remediation required between testing
Consequences of the Exit Exam
Phase I Results:
Testing Policies and Practices (n=64)
• Descriptive Data
▫
▫
▫
▫
▫
▫
Required Preparation Plan
Testing a Component of Course Grade
Mandatory Remediation
Remediation Type
Number of attempts
Consequences
• Correlation Data
▫ Testing Policies and HESI E2 Score
▫ Remediation Strategies and HESI E2 Final Score
7
3/5/2013
Phase I Descriptive Results:
Preparation Plan
• 39 of 64 schools provided students with an exit
exam preparation plan (61%)
• 33 schools required students to use the
preparation plan (85%)
• School’s timeframe
▫ 32 (82%) 6 weeks ahead
▫ 30 (77%) more than a week ahead
Phase I Descriptive Results:
Exit Exam as Course Requirement
• 46 schools (67%) used the E2 as a course
requirement
▫ 11 schools – exam for course completion
▫ 11 schools – pass/fail criteria
▫ 11 schools – counted exam as 5-10% of course
grade
▫ 10 schools – counted exam as 20-30% of course
grade
▫ 3 schools – counted exam as greater than 30% of
course grade
Phase I Descriptive Results:
Benchmark & Re-Testing
N = 64 Schools
Benchmark
Policy
• 54 schools (84.4%) had an exam policy
• 37 (69%) used 850; 10 (19%) used 900
• Range was 700 - 950
Mandatory
Benchmark
• 24 schools (38%) of the 64 schools
achievement of benchmark was
mandatory
Mandatory
Re-Testing
• 37 schools (58%) required re-testing
• 1 time 15 – schools; 2 times – 9
schools; 3 times – 12 schools; 4
times – 1 school
Phase I Descriptive Results:
Preparation Plan (61%)
Types of Preparation
(Multiple components used)
Frequency Percent
(N=39)
Self-guided review (eg., case studies,
study guide, NCLEX type items)
33
85
Faculty guided group review
22
56
Faculty guided individual review
18
46
Formal review (eg., HESI, ATI,
Kaplan)
17
44
Peer/mentor tutoring
13
33
Self-guided formal review course
(online)
5
13
Phase I Descriptive Results
Mandatory Remediation
• 41 of 64 (64%) schools required remediation
after the initial failure to meet benchmark
• 15 (37%) schools required students to pay
additional fees for remediation
• 37 (90%) schools required proof of participation
in remediation
Phase I Descriptive Results:
Consequences
• Of the 54 schools that had a benchmark:
▫ 30 schools (56%)set consequences for NOT meeting
the benchmark
▫ Top 3
1st Course failure (15) 50%
2nd Delay NCLEX candidacy (13)24%
3rd Delay graduation (9)17%
▫ Multiple consequences





Repeat applicable course
Audit applicable course
Delay/Deny graduation
Delay/Deny NCLEX candidacy
Program removal
8
3/5/2013
Phase I Correlation Results:
Testing Policy and Practices
with HESI E2 Score
Phase I: Remediation Policy Strategies:
Correlations to Last HESI E2 Score (N=5038)
Correlation
P value
Policy
Yes
No
Significance
Achieving the HESI E2
benchmark score is
mandatory
n = 1426
Mean = 907.20
SD = 112.38
n = 2908
Mean = 855.16
SD = 124.23
p < .0001**
Formal Review Course
.182
.0001**
.0001**
n = 1469
Mean = 905.47
SD = 112.83
n = 3144
Mean = 853.13
SD = 124.14
p < .0001**
Tailored Self-Guided HESI
Exam Review
.112
Require students to
participate in
preparation plan
Require students to retest
n = 2429
Mean = 888.76
SD = 114.80
n = 2184
Mean = 848.98
SD = 128.47
p < .0001**
Self-Guided Online Review
course
.097
.0001**
Require remediation
after a student fails to
meet the benchmark
n = 2618
Mean = 885.18
SD = 116.99
n = 1995
Mean = 849.91
SD = 127.89
p < .0001**
Self-Guided Case Studies
.068
.0001**
Peer/Mentor Tutoring
.042
.003**
Phase II Results: Predictive Accuracy of the HESI Exit Exam for NCLEX‐RN Outcomes (N = 5038)
HESI Exit
Exam
Category
Student NCLEX
Outcome Passed
Failed
900 and
above
2084
37
Predicted
Accuracy
98.26
850 – 899
879
45
95.13
800 – 849
585
48
92.43
700 – 799
699 or less
780
323
127
130
86.00
71.30
TOTAL
4651
387
92.32
Conclusions
• The HESI E2 Comprehensive Exam is predictive of
NCLEX success.
• Schools of Nursing are designing and implementing
high stakes testing policies to manage the
implementation of the HESI E2 Exam.
• Several components within a high stakes testing
policy are significantly related to individual scores
on the HESI E2 Exam.
• Certain types of remediation policies can be
correlated with HESI E2 individual scores.
• High stakes testing policies need to be crafted to
include certain components that are related to HESI
E2 success.
9