The Quest for Quality:
Monitoring and Measuring Outcomes in
Continuing Nursing Education
Session 3: Monitoring and Measuring
Outcomes for Provider Unit Effectiveness
Pamela S. Dickerson, PhD, RN-BC, FAAN
Director of Continuing Education, Montana Nurses Association
President, PRN Continuing Education
Presented by Ohio Nurses Association
and Montana Nurses Association
1
Disclosures
• Purpose: Participants will effectively develop
learning and provider unit processes that lead to
measurable outcomes demonstrating quality
patient care and/or enhanced professional
development of registered nurses.
• Criteria for Successful Completion: Participate in the
entire webinar. Complete the evaluation and
registration information and return to ONA as
directed.
• Conflict of Interest: There is no conflict of interest for
any planner or presenter for this activity.
2
Objective
• Examine approaches to data collection and
analysis related to the effectiveness of a provider
unit in improving patient care and enhancing
nursing professional development.
3
Reflection
• If you have participated in sessions 1 & 2, please
rate your current ability to do the following:
Strong = 3
Moderate =
2
Weak = 1
Conduct a
needs
assessment
Analyze a
gap
Measure an
activity
outcome
Write a
provider
unit
outcome
Your Score:
4
Reflection Data
• My score is closest to:
•
•
•
•
4
7
9
12
5
Self-Assessment
I know how to
Strong =
3
Moderate = 2
Weak = 1
Develop goals and outcome measures
for my provider unit
Engage stakeholders in provider unit
evaluation
Evaluate provider unit data
Measure provider unit effectiveness in
relation to improved patient care or
enhanced professional development of
registered nurses
Your Score =
6
• My total score was closest to a:
•
•
•
•
4
7
9
12
7
Accreditation Conceptual Framework
ANCC 2013 – used with permission
Structural Capacity
•
•
•
•
Educational Design
Process
Commitment
Accountability
Leadership
Resources
Assessment of Learning Needs
• Planning and Faculty
• Design Principles
• Achievement of Objectives
•
Nursing
Professional
Development
: Continuing
Nursing
Education
Quality Outcomes
Evaluation Process
• Evaluation Participation
• Goals for Improvement
• Value/Benefit to Nursing
Professional Development
•
8
Provider Unit Outcomes Measures
• Outcome: “The impact of structure and process on
the organization…and the value/benefit to nursing
professional development.” (ANCC, 2013).
• Outcome Measurement: “The process of observing,
describing, and quantifying predefined indicators of
outcomes of performance.” (ANA/NNSDO, 2010)
9
Evaluation in Context
Goals for
Provider
Unit
Outcome
Measurement for
Provider Unit
Outcome
Measures
Provider Unit
Structure &
Function
Activity
Evaluatio
n
Learning
Activities
PSD, 2013
10
Why Does Your Provider Unit
Exist?
• Single- focused organization:
o Why is continuing education important enough
to make it your business?
o What does “success” for your organization look
like?
• Multi-focused organization:
o Why is continuing education part of what your
organization does?
o How does continuing education benefit the
organization?
11
Let’s Look at the Organization
• What is its mission?
• What are its goals?
• Who are its stakeholders? (Who cares about the
organization, its success, and its functions?)
12
Your Provider Unit
• What are the goals for your provider unit?
• How do your goals match those of organization? (If
they don’t, it’s time for some serious reassessment.)
13
Think Like A Reporter
14
What Is The Best Time To
Evaluate Your Provider Unit?
•
•
•
•
Quarterly
Annually
When requested by administration
When the provider application is due
15
16
What is the best way to frame
Provider Unit Evaluation?
• How effectively do our structure and processes
support our efforts in achieving our goals?
• Did we accomplish what we planned in our
learning activities as a whole?
• Do we have administrative support and funding for
our next provider application?
• When is our next provider application due?
17
Evaluating Your Provider Unit:
Who
• Structural Capacity
o People
• Right mix of people?
• Right qualifications of people?
• Enough people?
• People doing the right things?
• Do people know how to do what’s needed as
the system evolves?
18
Evaluating Your Provider Unit:
What
• Structural Capacity: Physical and Material
Resources
o Office space
o Office supplies
o Classroom space
o Equipment
• Up-to-date?
• Functional?
19
Evaluating Your Provider Unit - What
• Structural Capacity – Financial and Administrative
Support
o Does primary nurse planner have authority to
ensure adherence to criteria?
o Is there administrative support for the work of the
provider unit?
o Are there financial resources for support of
structure and processes of the provider unit as
well as the educational activities?
20
Evaluating Your Provider Unit - When
• It depends…
o Some factors may need to be evaluated monthly or
quarterly
o It may take time to collect data for some outcomes
measures, ranging from several weeks to a year or
more
o Organization may require an annual report and
annual goals
o Provider renewal time is an excellent opportunity for a
comprehensive review and analysis
21
Evaluating Your Provider Unit - Where
• Within the context of the provider unit itself
• Within the context of the organization
• Each will require unique perspectives, data, and
analyses
22
Evaluating Your Provider
Unit – How
• Look at components of structural capacity
• Look at components of processes – provider unit
functions AND educational design processes
• Measure against your goals, which are supported
by your predetermined outcomes measures
• Engage appropriate stakeholders
23
Stakeholders
• Who are the people who care?
o Learners
o Faculty
o Provider Unit Nurse Planners & Key Personnel
o Organizational Personnel
• Administration
• Finance
• Risk Management
• Others
o Community representatives
o Advisory Committee or Board
o Stockholders or owners, Board of Directors
o Others
24
Evaluate Your Provider Unit: Why
• Prove that you make a difference
o Validate your contribution to quality patient care
o Demonstrate your commitment to professional
development of registered nurses
25
Example: Provider Unit Effectiveness
(how stakeholder input helped
improve a quality outcome measure)
• Outcome Measure: diversity of learning activities.
• Provider Unit Goal: Implement at least 5 web based
learning opportunities for nurses in our hospital
during calendar year 2013.
26
Stakeholder Input
o Learners requested opportunity for web-based
learning activities they could access via the
hospital’s intranet and complete at their
convenience.
o Managers requested that contact hours be
awarded for activities that did not require nurses
to be away from the practice setting for the time
required to attend classes.
27
Data
• In calendar year 2012, we offered 53 continuing
education classes for our nurses – all were “realtime” learning opportunities.
• In calendar year 2013, we offered 3 web-based
independent study activities. Our goal was 5, but
the IT person resigned and has not been replaced.
28
Your Thoughts?
• Does this evidence address the question of how
stakeholder input has helped to improve a quality
outcome measure?
• Yes
• No
29
30
What Could Make It Stronger?
• Did learners feel that their needs were met with
these 3 activities?
• What is the plan to move forward? Need to
advocate for IT resources?
• What feedback has been received from managers
in relation to their original request?
31
Example: Professional Development
• Outcome Measure: leadership development
• Provider Unit Goal: Enhance effectiveness of
charge nurses and managers through a series of
leadership classes and organizational initiatives.
32
Background
• Reason for this goal: Fits with organizational goal of
improving staff satisfaction and retention through better
relationships with front-line leadership.
• Evidence:
o Research demonstrating that new nurses leave 1st
positions in significant numbers, primarily due to
unsatisfactory relationships with front-line leaders.
o In our facility in 2012, 36% of new RNs left their
positions within 18 months of hire.
o In our facility in 2012, 64% of RN staff rated their
satisfaction as moderate to low and indicated
relationship with management a key factor
33
Action Plan
• Survey staff nurses to get their perspectives on
expectations of charge nurses and managers
• Survey charge nurses and managers to get their
perspectives on their expectations of their roles
• Survey nursing administration to get their
perspectives on expectations of charge nurses and
managers
34
Action Plan
• Based on survey results, meet with nursing
administration to develop a plan for role
clarification, position description revisions, and other
organizational initiatives to support more effective
functioning of charge nurses and nurse managers.
Plan coordinated educational initiatives.
• Establish planning committee to develop a series of
leadership classes for front-line managers.
• Timeline for combined interventions – March
through June, 2013
35
Educational Interventions
• Six half-day workshops for front-line managers over
a three month period of time.
• Content from each session built on previous sessions
• Content addressed key areas of concern noted in
surveys: staff team communication,
interprofessional communication, conflict resolution,
time management, workflow management, crisis
intervention, dealing with bullying/intimidation,
feedback & evaluation
36
Outcomes
• In the last 6 months of 2013, only 8% of new nurses
(defined as within 18 months of hire) left their
positions.
• In the staff satisfaction survey conducted in
October of 2013, 72% of RNs ranked their
satisfaction as moderate to high. Anecdotal
comments indicated that the concentrated effort
to better prepare and support front-line managers
was a key factor.
37
Demonstrating the Evidence
% leaving
40
35
30
25
20
% leaving
15
10
5
0
2011
last half 2012
38
More Evidence
100%
90%
80%
70%
60%
50%
2013 by %
40%
2012 by %
30%
20%
10%
0%
Low
Moderate High
39
So What?
• Has the provider unit contributed to the professional
development of the front-line managers?
• Has the provider unit contributed to the professional
development of ALL nurses in the organization?
40
Leadership: Another Example
• A college of nursing provider unit is aware of
national data, supported by the IOM report on the
future of nursing, that there is increased need for
leadership education for registered nurses. A focus
group of CNOs from area healthcare facilities
supports the need at the local level.
41
Leadership Example
• Provider Unit Goal: Increase leadership capability of
alumni and other participants in our CNE activities
• Outcome Measure:
o Professional Development - Leadership
42
Framing Your Plan For Goal
Achievement
• Use of an evaluation model
o Revised Kirkpatrick
o Logic
o Moore
• Have a clear plan in place for what you’re going to
do, what resources will be employed, and what
outcome measurement data you will collect –
begin with the end in mind!
43
Using the Revised Kirkpatrick Model
• What outcomes do we want?
• What behaviors are required of these nurse to
increase their leadership capability?
• Where are the gaps? What are current capabilities
and what are the desired capabilities?
• What KSAs do learners need to fill these gaps?
• What is the best learning strategy to address these
needs?
44
Using the Logic Model
• What do we need? (better prepared leaders)
• To address this need, we must:
• If we do this effectively, the outcome will be:
• When leaders implement this outcome, we will see:
• When this happens, organizations will be better
because:
45
Using The Moore Model
• Does the learner know? (is there a knowledge
deficit?)
• Does the learner know how? (is there a skill deficit?)
• Can the learner show that he/she can use the
knowledge? (is there a performance deficit?)
• Is the learner able to apply the knowledge? (is there
a critical thinking/clinical judgment/application
deficit in practice?)
46
The Outcome for The Provider Unit
• Provider Unit Goal: Increase leadership capability of
alumni and other participants in our CNE activities
• Provider Unit Outcome Measure: professional
development – leadership
• Stakeholders: CON leadership, CNO focus group,
alumni, other learners in related activities
• Where’s the evidence?
o Self-report from participants
o Data from CNOs regarding leadership evidence in their
organizations among nurses who participated in these
activities
o Number of nurses engaged in leadership roles in practice
or professional settings
47
Self-Assessment
I know how to
Strong = 3
Moderate = 2
Weak = 1
Develop goals and outcome
measures for my provider unit
Engage stakeholders in provider
unit evaluation
Evaluate provider unit data
Measure provider unit
effectiveness in relation to
improved patient care or
enhanced professional
development of registered nurses
Your Score:
48
• My total score was closest to a:
•
•
•
•
4
7
9
12
49
50
The Bottom Line
• Evaluation of your provider unit helps you identify
strengths, contributions, and areas of opportunity.
• Providing evidence of accomplishments of your
goals and related outcome measures helps you
validate your worth to the organization and the
nurses you serve.
51
Questions and Discussion
52
For Additional Information
Pam Dickerson, PhD, RN-BC, FAAN
pdickerson@aol.com
pam@mtnurses.org
53
References
•
American Nurses Association & National Nursing Staff
Development Organization (2010). Nursing Professional
Development: Scope and Standards of Practice. Silver Spring,
MD. Nursesbooks.org
•
American Nurses Credentialing Center. (2013). 2013 ANCC
Primary Accreditation Application Manual for Providers and
Approvers. Revision 3.0. Silver Spring, MD. Author.
•
Commission on Accreditation, ANCC. (2012). The Value of
Accreditation for Continuing Nursing Education: Quality
Education Contributing to Quality Outcomes. Retrieved from:
http://www.nursecredentialing.org/Accreditation/ResourcesServi
ces/Accreditation-WhitePaper2012.pdf
•
Dickerson, P. (2012). Rocking the boat: challenges ahead for
continuing education providers. Journal of Continuing Education
in Nursing 43(10), 443-450.
54
References
•
Kirkpatrick, D. (1984). Evaluating Training Programs: The Four Levels.
San Francisco: Barrett-Koehler.
•
Kirkpatrick’s Four Level Evaluation Model. (2012). Retrieved from:
http://www.nwlink.com/~donclark/hrd/isd/kirkpatrick.html
• Moore, D., Green, J., & Gallis, H. (2009). Achieving desired
results and improved outcomes: integrating planning and
assessment throughout learning activities. Journal of
Continuing Education in the Health Professions 29(1), 1-15.
• W.K. Kellogg Foundation. (2004). Logic Model Development
Guide. Battle Creek, MI; Author.
55