UTA College of Nursing
Planning:
From
Objectives to Outcomes
Carolyn L. Cason, RN, PhD
Susan Scheid, RN, MSN
Smart HospitalTM
University of Texas at Arlington
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Objectives
UTA College of Nursing
• Discuss core components to a healthcare
simulation
• Discuss the importance of outcomes
evaluation and challenges to traditional
assessments
• Discuss the importance of validity, reliability,
and feasibility as it relates to assessment
• Discuss types of assessments and their
application in healthcare education
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Strategy for the Session
UTA College of Nursing
• Describe the process used to develop a near
end-of-program clinical performance exam
• Examine the
– Validity, reliability and standardization challenges
– Evaluation challenges
– Performance outcomes
• Describe the ways in which the outcomes
– Guided exam revision
– Informed curriculum/program effectiveness
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Impetus for Exam
UTA College of Nursing
• Faculty desire for clinical performance
assessment to complement the knowledge
assessment obtained via the HESI Exit exam
• Faculty search for a meaningful and
comprehensive approach to evaluation of
program outcomes
• Faculty search for a way to benchmark/assess
clinical competencies
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
UTA College of Nursing
Junior I
Foundations, Assessment, Health Promotion
Junior II
Medical-Surgical Nursing, Psychiatric-Mental Health Nursing,
Research
Senior I
Pediatric Nursing, OB Nursing, Critical Care Nursing
Senior II
Community Health, Leadership & Management, Nursing
Trends, Gerontology, Capstone
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
OSCE Objectives
UTA College of Nursing
• Provide safe patient care
• Obtain focused objective & subjective
assessment
• Plan, implement and evaluate patient care
based on nursing assessment & diagnosis
• Communicate effectively & professionally with
the patient and healthcare team
• Complete appropriate documentation
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Pilot: July 2010
UTA College of Nursing
• Clinical Skills
– Manual B/P; O2 administration, telephone
order, IV management
– NG insertion, IV start, urinary catheter
insertion, restraints, I/O etc.
• Patient Care Management
– 1st day post-op bowel surgery
– Post-op hemorrhage
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Summer 2010 OSCE
Dallas SUN April 2011
Copyright CLCason
Summer 2010 OSCE
Dallas SUN April 2011
Copyright CLCason
UTA College of Nursing
Lessons Learned
Summer 2010
• More preparation of students and
faculty needed
• Skills check separate from patient
care scenarios
• Faculty involvement needed (Crucial)
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Lessons Learned
Summer 2010
• Inconsistencies
• Faculty decision to use OSCE as
diagnostic assessment
• Incorporate more simulation into
Nursing curriculum
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Fall 2010
OSCE Changes
• Increased faculty involvement
• Increased preparation of students
• Skills Check: separate day in context
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Fall 2010
OSCE Changes
• Patient scenarios individual and
private - confidential
• Use OSCE as diagnostic tool for
Capstone Clinical
• Development of simulation for all
levels of undergraduate nursing
program
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Fall 2010 Outcomes
Skills Validation
Safety:
•
•
•
•
Patient identification failure (15-21%)
No allergy check (26-86%)
Inconsistent hand hygiene (21-28%)
Sterile procedure contamination (2332%)
• Not using alcohol to clean skin or IV
tubing ports (21-57%)
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Fall 2010 Outcomes
Skills Validation
Medication Administration:
• Failure to draw up or give insulin
correctly (23-31%)
• Improper use of gloves (23-42%)
• Recapping used needles (25-39%)
• Inability to hang IVPB correctly (2832%)
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
N 4350 Capstone Skills Validation
Indicates error Red= 60-80% Yellow = <80- 90% Green =>90%
Patient: Collins and Jonas
Student ________________________Date/Start time __________ Clinical Instructor ___
N = 76
Skill
Yes
No
Patient Safety
Percentages
Correct
Incorrect
a.
Introduced self/wearing ID
72
4
94.7%
5.3%
b.
Wash/gel hands on entry
69
7
90.8%
9.2%
c.
Identified patient
64
12
84.2%
15.8%
d.
Assessed allergies
56
20
73.7%
26.3%
1. Vital signs: Set values
2.
3.
a.
Blood Pressure:
64
12
84.2%
15.8%
b.
Pulse:
71
5
93.4%
6.6%
c.
Respirations:
71
5
93.4%
6.6%
Oxygenation:
a.
Applied oxygen cannula correctly
72
4
94.7%
5.3%
b.
Set oxygen flow to 2L/min
71
5
93.4%
6.6%
58
18
76.3%
23.7%
52
24
68.4%
31.6%
c.
Correctly figured and drew up Regular insulin dose:
Reg (4 units) (drew first)
Correctly figured NPH insulin and drew up dose:
(14 units). (18 units total in syringe)
Wiped top of vials with alcohol
73
3
96.1%
3.9%
d.
Selected appropriate site
74
2
97.4%
2.6%
e.
Explained procedure
67
9
88.2%
11.8%
f.
Wore gloves for injection
58
18
76.3%
23.7%
g.
Cleaned skin with alcohol
69
7
90.8%
9.2%
h.
Aspirated
8
68
89.5% no
10.5% yes
i.
Recapped used needle
19
57
75% no
25% yes
70
6
92.1%
7.9%
b.
Adjusted rate of primary IV fluids from 75 mL/hr to
125 mL/hr
Selected Ceftriaxone 1 g/50 mL
68
8
89.5%
10.5%
c.
Primed bag correctly
68
8
89.5%
10.5%
d.
Wiped ports with alcohol
58
18
76.3%
23.7%
e.
Lowered primary bag
48
28
63.2%
36.8%
f.
IVPB at correct rate: 100 mL/hr
68
8
89.5%
10.5%
g.
Unclamped IVPB so it would run
53
23
69.7%
30.3%
h.
Capped empty IVPB
49
27
64.5%
35.5%
Insulin administration:
a.
b.
4.
IV Medications:
a.
5. Foley catheter insertion
a.
Explained procedure
69
7
90.8%
9.2%
b.
Applied sterile gloves without contamination
67
9
88.2%
11.8%
58
18
76.3%
23.7%
c.
Inserted Foley without contamination or verbalized
errors in sterile technique.
Patient Safety
a.
Left bed in low position
75
1
98.7%
1.3%
b.
Siderails up x 2, call light in reach
72
4
94.7%
5.3%
c.
Wash/gel on leaving room
60
16
79%
21%
Evaluator: ___________________________________________________
Fall 2010 Scenarios
UTA College of Nursing
• Focused patient assessment, vital
signs, recognize change in patient
condition, independent nursing
judgment and action
• Capstone faculty – sometimes
present during OSCE scenarios
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Fall 2010
UTA College of Nursing
Scenarios: OSCE Evaluation
• Similar student performance concerns
• Safety issues:
– No patient identification ( 31-40%)
– Inadequate hand hygiene (33- 44%)
– Improper use of gloves (14%)
• No medication administration required
for scenarios except IV Fluid
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Fall 2010 Skills and OSCE
UTA College of Nursing
Deficiencies in student performance
• Related to lack of simulation experience
• Inconsistency of evaluators and operators
- 20 different people involved in check-offs
- Standardized pre-recorded voice
responses of manikins
- Operators – differences in prompting with
responses
- Assistance given by evaluator varied
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Student Feedback
Skills & OSCE Fall 2010
Most students (more than 70%)
agreed or strongly agreed:
• Time was sufficient
• Helped meet learning objectives
• Were able to transfer to clinical setting
• Conducive to learning
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Student Feedback
Skills & OSCE Fall 2010
Some students (30-40%) were
neutral or disagreed:
• Helped them gain confidence in role,
skills, or clinical setting
• Felt it was not realistic (40%)
• Were not satisfied with method of
learning
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
1. Length of
time was
sufficient
2. Helped me
meet learning
objectives
3. Helped me
gain confidence
In RN role
4. Helped me
gain confidence
In skills
5. I am more
confident in
clinical setting
6. Able to
transfer info to
clinical setting
7. Able to
transfer skills
to clinical
setting
8. Session fit
well into course
outcomes
9. Environment
was conducive
to learning
10. Session was
realistic
11. Satisfied
with method of
learning
12
Standardized
patient added
to scenario
Strongly
agree
Agree
Neutral
Disagree
Strongly
disagree
N/A
37
40
9
4
0
0
90
28
43
13
4
2
0
90
22
38
19
8
3
0
90
66.7%
33.3%
17
44
19
8
2
0
90
67.8%
32.2%
15
37
24
11
3
0
90
57.8%
42.2%
25
53
6
4
2
0
90
86.7%
13.3%
20
49
10
4
2
0
85
81.2%
18.8%
16
42
16
4
4
0
82
70.7%
29.3%
20
46
8
7
2
0
83
79.5%
20.5%
19
29
19
11
3
0
81
59.3%
40.7%
22
28
20
8
4
0
82
61.0%
39.0%
19
25
13
0
4
3
68.8%
31.2%
260
N = 90 for # 1-6
474
See totals for # 7-12
176
73
31
3
Total
#
64
S. Agree
Or
Agree
85.6%
78.9%
Neutral
Disagree
or S.
Disagree
14.4%
21.1%
UTA College of Nursing
Student Comments
Four main categories of comments:
• Realism - Some very real, some
unrealistic
• Liked individual thinking and action
• Wanted immediate feedback (debriefing)
• Confidence – mostly yes
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Student Suggestions
Suggestions for Improvement:
• Immediate debriefing
• More simulation throughout program
• Opportunities to practice & more
information
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Summary Fall 2010 OSCE
UTA College of Nursing
• Objectives stayed the same
– Faculty wanted end of program evaluation
– Student performance needed improvement
• Reliability: multiple evaluators and
operators
• Validity: use as diagnostic tool
• Feasibility: labor intensive, costly
• Positive changes
• More work needed
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Spring 2011 Changes
UTA College of Nursing
Faculty Involvement:
• Immediate feedback and debriefing by
clinical instructors at time of scenario
• Emphasis on OSCE as plan for
Capstone Clinical
• Scheduling of students by faculty
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Spring 2011 Changes
UTA College of Nursing
• Reversed order of scenarios for realism
• Revisions for consistency and
standardization
– Identical pre-recorded manikin voice
responses
– More instructions to students, faculty,
operators, and evaluators
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Spring 2011
OSCE Outcomes
• Preliminary results for skills validation
– Similar safety and medication concerns
• Slight improvement of hand hygiene
• Faculty presence and participation
– Less student anxiety
– Favorable student responses
• Overall improved outcomes?
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Planning from Objectives
to Outcomes
Program Outcome Objectives
• Clinical Performance Assessment
• Comprehensive approach to
evaluation of program outcomes
• Determine Clinical Competency
before graduation
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
UTA College of Nursing
Planning from Objectives
to Outcomes
Student Objectives for Skills and
scenarios:
• Provide safe patient care
• Obtain a focused patient assessment
• Plan, implement and evaluate patient
care
• Communicate effectively
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington
Quality Assessment
UTA College of Nursing
•
•
•
•
•
Dallas SUN April 2011
Copyright CLCason
Reliable
Valid
Practical
Replicable
Generalizable
The University
of Texas
at Arlington
Thank You
UTA College of Nursing
Questions?
Dallas SUN April 2011
Copyright CLCason
The University
of Texas
at Arlington