Receive your continuing education certificate
1. Log onto the Wifi: “Wifi @ OSU”
– No password required
2. Go to atinursingconference.com/now
– Complete the form
– Enter the conference code: ROBERTS115
If you don’t have the ability to complete this process on a
laptop, tablet, or smart phone, please see an ATI
representative for the form to complete manually.
Using Data to Drive Evidence-Based
Program-Related Decisions
Roberts, 2015
Educator Enrichment Conference | Columbus, OH
Fostering Excellence and Distinction in Nursing
Education
Karin K Roberts, PhD, RN, CNE
Objectives
Program Evaluation
Key Words
• Assessment
• Systematic process
• Continuous
• Evidence-based decisions
• Quality improvement
Program Evaluation
Master Plan of Program Evaluation
• Blueprint for program evaluation
• Terms used to describe plan
– Systematic Evaluation Plan (SEP)
– Program Evaluation Plan (PEP)
– Comprehensive Evaluation Plan (CEP)
Systematic Evaluation Plan - ACEN
• Standard 6.1
– The systematic plan for evaluation of the nursing
education unit emphasizes the ongoing assessment
and evaluation of each of the following:
• Student learning outcomes;
• Program outcomes;
• Role-specific graduate competencies; and
• The ACEN Standards.
– The systematic plan of evaluation contains specific,
measurable expected levels of achievement;
frequency of assessment; appropriate assessment
methods; and a minimum of three years of data for
each component within the plan.
(ACEN, 2013)
Systematic Evaluation Plan - CCNE
• Standard IV.A
– A systematic process is used to determine program
effectiveness
Elaboration:
• Written, ongoing, exists to determine achievement
of program outcomes
• Comprehensive
• Identifies qualitative and quantitative data to be
collected
• Includes timelines for collection, review of
expected and actual outcomes, and analysis
• Periodically reviewed and revised
(CCNE, 2013)
Systematic Evaluation Plan - ACEN
• Standard 6.2
– Evaluation findings are aggregated and trended by
program option, location, and date of completion and are
sufficient to inform program decision-making for the
maintenance and improvement of the student learning
outcomes and the program outcomes.
(ACEN, 2013)
Systematic Evaluation Plan - CCNE
• Standard IV-H
– Data analysis is used to foster ongoing program
improvement
• Actual outcomes compared to expected outcomes
• Discrepancies inform areas of improvement
• Changes to program are deliberate, ongoing, and
analyzed for effectiveness
• Faculty are engaged in program improvement process
(CCNE, 2013)
Key Words
• Systematic process
• Ongoing assessment (trended/aggregate data – 3
years)
• Levels of achievement/Outcomes (actual
compared to expected)
• Program effectiveness
• Maintenance/improvement of SLOs and program
outcomes
Program vs. Student Learning Outcomes
• Outcomes
– Program: Indicators that reflect the extent to which a
program is achieving their mission and goals.
• NCLEX pass rates; Retention rates; Employment rates;
Graduate and Employer Satisfaction rates.
(ACEN, 2013; CCNE, 2013)
– Student Learning Outcomes (SLOs)
• Statements of the characteristics or attributes of
students upon graduation from a program of nursing.
• They are measurable, learner-oriented, and are
attained as a result of program activities.
(ACEN, 2013; CCNE, 2013)
Format of SEP
SEP Matrix Format - ACEN
PROGRAM EVALUATION
ACEN Standard or Program Outcome or Student Learning Outcome
PLAN
Component
Expected
Level of
Achievement
(or program
terminology)
Frequency of
Assessment
IMPLEMENTATION
Assessment Methods
Results of Data
Collection and
Analysis
Including
actual level/s
of achievement
ACEN - http://www.acenursing.net/resources/SampleSEP.pdf
CCNE – does not provide recommended format
Actions for
Program
Development,
Maintenance, or
Revision
Program Outcomes Summary Table - ACEN
PROGRAM OUTCOME SUMMARY
Required Program
Outcomes
Expected Level of
Achievement
(or program’s terminology)
Actual Level of
Achievement
Resulting Actions Taken/To Be Taken with
Timeline
Action(s)
Performance on NCLEX
and/or certifying
examination
Program Completion
Program Satisfaction
---Graduate Satisfaction
---Employer Satisfaction
Job Placement
Other Outcome(a);
identified by program
(optional-not required by
ACEN)
ACEN - http://www.acenursing.net/resources/SampleSEP.pdf
Timeframe
SEP Matrix Format
Data Analysis Feedback Loop
Analyze
Data/
Analyze
New Data
Collect
New Data
Unmet
LOA/
Outcome
Develop
Action Plan
Implement
Action Plan
Roberts, 2015
Data Analysis Feedback Loop
Analyze
Data/
Analyze
New Data
Collect
New Data
Unmet
LOA/
Outcome
Develop
Action Plan
Implement
Action Plan
Roberts, 2015
Compare Actual with Expected LOAs/Outcomes
• Most important part of SEP
– Data is collected, aggregated, trended (>/= to 3 yrs)
Compare Actual with Expected LOAs/Outcomes
• Most important part of SEP
– Data is collected, aggregated, trended (>/= to 3 yrs)
– Analyzed
Compare Actual with Expected LOAs/Outcomes
• Most important part of SEP
– Data is collected, aggregated, trended (>/= to 3 yrs)
– Analyzed (actual vs expected LOAs/Outcomes)
– Evidence-based decisions are made based on data
analysis
Evidence-Based Decisions
Analyze
Data/
Analyze New
Data
• LOA/Outcome met
– Maintain current state of program
Collect
New Data
• LOA/Outcome not met
– Revise a component of the program
Unmet
LOA/
Outcome
Develop
Action Plan
Implement
Action Plan
• LOA/Outcome not met
– Develop a new component of the program
• Use strategy such as “root cause analysis” to analyze
why there is a need for revision or development of a
program component
Root Cause Analysis Strategy
• Structured collaborative process that uses data
analysis to determine underlying/contributing factors
of an adverse event with the goal of implementing
corrective action(s).
(adapted from CMS.gov)
• Problem-solving technique that
attempts to get to the root of a
problem.
Analyze
Data/
Analyze
New Data
Collect
New Data
• Using the” Five Whys” can help
drill down to the root cause
(sixsigma.com, 2015)
Unmet
LOA/
Outcome
Implement
Action Plan
Develop
Action Plan
Root Cause Analysis – 5 “Whys”
UNMET LOA/
OUTCOME
One sentence description of unmet LOA/outcome
Why?
Why?
Why?
Why?
Why?
Root Cause(s)
(adapted from isixsigma.com, 2015)
Root Cause Analysis – 5 “Whys”
UNMET LOA/
OUTCOME
Review of curriculum reveals that the credit hour
requirement of 76 credits is above the state limit and DOE
recommendations (ACEN Standard 4.8).
Why?
Curriculum needs this much time to address content.
Why?
Why?
Why?
Why?
New knowledge requires addition of new content.
No content has been removed from curriculum to make
room for new content.
Content mapping has not been done to accommodate
for the change in content.
It has been 9 years since the curriculum was revised
Root Cause(s)
It has been 9 years since the curriculum was revised
and content was mapped to specific courses.
Subsequently, new content has been layered onto
old content. An increase in credit hours was
necessary to accommodate old and new content.
(adapted from isixsigma.com, 2015)
Root Cause Analysis - Practice
• Using the 5 Whys document provided, conduct a root
cause analysis of your program issue.
• Document the following on your flip chart paper the
issue; summary of possible underlying/ contributing
factors; conclusion(s) regarding root cause of issue.
• Using this information and paper, present the results
of your root cause analysis to the group
Development of Action Plan
• When possible cause of unmet LOA/outcomes has
been determined:
– Develop corrective action plan
– Set timeline for implement action plan and collection
of new data (timeline)
– Analyze actual vs expected LOA/outcome
using new data according to timeline
Analyze
Data/
Analyze
New Data
Collect
New Data
Unmet
LOA/
Outcome
Implement
Action Plan
Develop
Action Plan
Sample Action Plan Template
Unmet SLO/Outcome
Root Cause Analysis
ACEN Standard 4.8- Length
of time and credit hours
for a program are
consistent with…state and
national standards, and
best practices.
It has been 9 years since
the curriculum was
revised and content was
mapped to specific
courses. Subsequently,
new content has been
layered onto old content.
An increase in credit
hours was necessary to
accommodate old and
new content.
Action Plan
Minutes
Committee/Month/
Date
Jan 15, 2016-May 15, 2016
Curriculum Committee
Conduct a literature review to
January 15, 2016
determine appropriate number of Action Plan Developed
credit hours for ADN curriculum
and type of curricula that are
currently being developed.
Aug 15, 2016-May 15, 2017
Revise curriculum, consider
revision to a concept-based
curriculum. Send BON new
curriculum’s required docs and
submit substantive change report
when BON approval is received.
Aug 15, 2017 Implement new
curriculum
May 15, 2018 - Assess success of
new curriculum’s first level of
courses – edit as needed.
May 15, 2019 - Assess success of
new curriculum’s second level of
courses – edit as needed
Development of Action Plan - Practice
• Using the results of the root cause analysis
conducted on your program issue, develop an
action plan and timeline for re-evaluation of
LOAs/outcome(s).
• Document on your flip chart paper a brief
description of the plan and timeline
• Using your paper, present your plan of action
to the group
Using Data to Drive Evidence-Based ProgramRelated Decisions
• Assess LOAs/Outcomes – expected vs. actual
—Analyze data and DOCUMENT (minutes)
—Develop and implement evidence-based
action plans
—Collect new data according to timeline
—Analyze new data
—Determine if LOA/outcomes met
Unmet
LOA/
—Develop new action plan…
Outcome
Analyze
Data/
Analyze
New Data
Collect
New Data
Develop
Action Plan
CONTINUOUS CYCLE
Implement
Action Plan
Roberts, 2015
References
Accreditation Commission for Education in Nursing (2013).
Accreditation manual. Atlanta, GA: ACEN
Billings, D.M. & Halstead, J.A. (2012). Teaching in nursing: A guide
for faculty. St. Louis, MO; Elsevier.
Centers for Medicare and Medicaid Services (2015) Retrieved from
https://www.cms.gov/medicare/provider- enrollment-andcertification/qapi/downloads/guidanceforrca.pdf
Commission on Collegiate Nursing Education (2013) Standards of
accreditation. Washington, DC: CCNE
Escallier, L.A. & Fullerton, J.T. (2012). An innovation in design of
a school of nursing evaluation protocol. Nurse Educator,
37(5), 187-191.
References
iSixsigma (2015) Determine the root cause; The 5 whys.
Retrieved from http://www.isixsigma.com/toolstemplates/cause-effect/determine-root-cause-5-whys/ .
Keating, S. (2010). Curriculum development and evaluation in
nursing. New York, NY: Springer. 11-23.
Roberts. K., (2015). Data-action plan graphic. In Using Data to
Drive Evidence-Based Program-Related Decisions
[PowerPoint slides]. Educator Enrichment Conference,
Columbus, OH.
Stavropoulous, A. & M. Kelesi (2013). Concepts and methods of
evaluation in nursing education – a methodological
challenge. Health Science Journal, 6(1), 11-23.