Inspire
.
Engage.
Lead.
Promoting Educational Research
Capacity through Systematic
Evaluation of Curricular Innovations
Janet Landeen, Lynn Martin, Charlotte
Noesgaard, & Kirsten Culver, McMaster
Donna Carr, Conestoga
Nancy Matthew-Maich, Mohawk
Lead.
Engage.
Inspire
.
Changing Perspectives
Lead.
Engage.
Inspire
.
Overview of Approaches
Lead.
Engage.
Inspire
.
Today’s Session




Setting the Context-Curriculum Renewal
Gathering the Research Team
Identifying the Big Questions
Using Interpretive Description to Organize
Approaches
 Narrowing the Questions, Finding Resources,
Inviting Others to the Dance
 Sharing Progress to Date
Lead.
Engage.
The Context
Inspire
.
Inspire
.
Lead.
Engage.
Designing the Kaleidoscope
Curriculum
Experiences of the McMaster Mohawk
Conestoga BScN Program
Lead.
Engage.
Inspire
.
Honouring the many authors
of the Kaleidoscope
Lead.
Engage.
Inspire
.
Lead.
Engage.
Inspire
.
Kaleidoscope…
Kalos…beautiful
Eidos…shape
Scopeo…to reflect on
PBL/PBL
Thinking Like
a Nurse
Ways of
Knowing
Integrated
Knowing/Acting
/Being
Lead.
Engage.
Inspire
.
Key Areas for Curricular Revision
Retention of Program Philosophy & Goals
Lead.
Engage.
Inspire
.
Lead.
Engage.
Inspire
.
The Person in the Middle
 Encounter the person (individual,
family, group, community)
 Use of narrative to have person come
alive
Inspire
.
Engage.
Lead.
 Turn the Kaleidoscope to a different aspect
Lead.
Engage.
Inspire
.
Clinical Reasoning &
Judgment in Nursing
(Tanner, 2006, p.208)
Inspire
.
Engage.
Lead.
 Turn the Kaleidoscope to a different aspect
Lead.
Engage.
Inspire
.
Pedagogical Approach:
Ways of Knowing in Nursing
 Empiric (scientific, evidence based)
 Ethical (moral component-Codes of Ethics)
 Personal (consistent with therapeutic use of
self)
 Aesthetic (art of nursing)
 Emancipatory (critical reflection & action based
on inequities)
(Carper, 1978; Chinn & Kramer, 2008)
Inspire
.
Engage.
Lead.
 Turn the Kaleidoscope to a different aspect
Lead.
Engage.
Inspire
.
Integrated Approaches to Learning
 Pathophysiology & Evidence Informed Decision
Making
 Scaffolded across multiple courses versus
stand-alone courses
(AACU, no date)
Inspire
.
Engage.
Lead.
So What?
Or
Who Cares?
Lead.
Engage.
Inspire
.
Gathering the Research Team
 Interest in educational reform and research
 Identification of key individuals– Chair of Program Evaluation Committee
– Director of Nursing Education Research Unit
– Leaders within Program
 Individuals looking for career growth
opportunities
 Graduate Students
 Undergraduate Students
Inspire
.
Engage.
Lead.
Identifying the Big Questions
Lead.
Engage.
Inspire
.
Program Evaluation vs.
Educational Research
 Evaluation of all
 In-depth exploration of
aspects of program
specific topics
 Real life/real time
 Comparison group,
curriculum-as-lived
before/after design, or
versus curriculum-assystematic qualitative
planned
exploration of
experience or process
 Used to inform local
decision-making
 Uncover broader
knowledge relevant to
others
Lead.
Engage.
Inspire
.
CASN Accreditation Standard
3.2.5 Ongoing comprehensive evaluation
continually improves program outcomes.
 Descriptor: The program is deliberative and
responsive in making timely improvements
based on data from ongoing rigorous and
ethical evaluations from faculty, learners,
graduates, employers, and others, through the
application of the Key Elements.
 Key Element: The program and curriculum are
monitored and evaluated to ensure currency
and relevance to nursing practice.
Lead.
Engage.
Inspire
.
Program Evaluation at
McMaster Mohawk Conestoga
 Comprehensive Program Evaluation Plan
 Use of Stufflebeam’s CIPP Model (Context,
Input, Process, & Product) to develop
Evaluation Matrix
 Matrix critiqued for congruence with program
philosophy, Stake’s Responsive Evaluation
Model, & Aoki’s suggestion of multiple
perspectives
(Aoki 1991; Stake 2003; Stufflebeam & Shinkfield 2007)
Lead.
Engage.
Inspire
.
Program Evaluation
Situational
Issues
Institutional
Priorities
Student
Expectations
Specific
Questions
Accreditation
Standards
Inspire
.
Program of Research
Faculty Interest
Lead.
Engage.
Funding
Opportunities
Informed by
Program Evaluation
Specific Research
Project
Lead.
Engage.
Inspire
.
Evaluating the Impact of the
Kaleidoscope Curriculum
 Have the educational innovations of the
Kaleidoscope Curriculum made a
difference in the graduates of the BScN
Program?
 Are there unintended consequences of
the curricular changes?
Lead.
Engage.
Inspire
.
Sub-questions
 Do students achieve the same or better
outcomes on standard measures of
performance?
 Do clinical faculty and clinical preceptors notice
a difference in graduating (Level 4) students of
the Kaleidoscope Curriculum? If so, are those
differences consistent with the educational
innovations?
Lead.
Engage.
Inspire
.
Sub-questions cont’d.
 Do graduating students integrate knowledge of
pathophysiology and evidence informed
decision making into their clinical reasoning and
judgement?
 Do stories of the educational experience that
graduates and faculty tell focus on the person
(individual, family, group, community)?
 Do graduating students utilize ways of knowing
in their approaches to nursing practice?
Inspire
.
Engage.
Lead.
Interpretive Description
Lead.
Engage.
Inspire
.
Integrative description as
organizing approach
 When purpose is to gain knowledge that can be
directly applied to enhance nursing education.
 “requires an integrity of purpose deriving from
two sources: (1) an actual practice goal, and (2)
an understanding of what we do and don’t know
on the basis of the available empirical evidence
(from all sources).”
(Thorne, 2008, p. 35)
Lead.
Engage.
Inspire
.
Developing the Approach
Do students achieve the same or better outcomes
on standard measures of performance?
Data Source
Data Analysis Strategy Rationale
Student
performance on
pathophysiology
and EIDM
examinations pre
and post
Kaleidoscope.
Per item analysis on
equivalent or identical
exam questions, and
grouped by content
areas.
Teaching
methodology is
different but overall
content to be learned
has remained the
same. Where
appropriate, direct
comparisons will be
exam on anonymized
examination
responses.
Lead.
Engage.
Progress to Date
Inspire
.
Inspire
.
Engage.
Lead.
Evaluating the Impact of Pedagogical
Innovations: Does the Kaleidoscope
Curriculum Make a Difference in the
Clinical Practice of Final Year
Undergraduate Nursing Students?
Kaleidoscope Research Team
Funded by Centre for Leadership in Learning,
McMaster University $8,000
Lead.
Engage.
Inspire
.
Evaluating Clinical Learning
Outcomes
 Integrative description
 Focus groups or individual interviews with 30
Clinical Faculty who taught students pre and
post Kaleidoscope
 Individuals likely to be sensitive to change in
students but least involved in curriculum
discussions
 Semi-structured interview questions
 Graduate student Research Coordinator
 Poised to begin recruitment
Inspire
.
Engage.
Lead.
Understanding the Experiences of
Students Who are Enrolled in
Joint College/University Collaborative
Educational Programs
Nursing: J. Landeen (Mac), N. Mathew-Maich (Moh), M.
Parzen (Moh), & L. Hagermann (Con)
Bachelor of Technology: L. Bolan (Mac) & D. Bender (Moh)
Medical Radiation Sciences: M. Faquharson (Mac) & L.
Marshall (Moh)
Lead.
Engage.
Inspire
.
“Dual Identity” Project
 McMaster President’s “Forward with Integrity” Fund
$5,000, matched by Dean of Engineering
 Uses interpretive description methodology
 8 undergraduate students (2 per program & site)
paid to develop full proposal, recruit participants,
conduct focus groups, analyze results, & present
findings
 Nursing student researchers receive course credit
prior to assuming paid role
 Interdisciplinary faculty team mentoring & teaching
students
Lead.
Engage.
Inspire
.
Examples of Other Research
Projects
 Evaluation of written clinical evaluation forms 2
years pre and 2 years post Kaleidoscope to
assess degree of uptake of curriculum renewalin proposal writing stage (Noesgaard et al)
 Exploring the Meaning of Consistency in PBL
Programs-study completed (Landeen & Jewiss)
 What Makes an Effective Teacher? Listening to
the Voice of Our Students- in data analysis
(Matthew-Maich et al)
Lead.
Engage.
Inspire
.
Barriers to conducting
educational research
 Work load of faculty
 Cultural & institutional differences between
universities & college partners
 Funding sources for educational research
 Perceptions of value of educational research
versus clinical research
Lead.
Engage.
Inspire
.
Facilitators to Educational
Research
 Resurgence of interest in educational research
 Organizations such as International Society for
Teaching and Learning in Higher Education
(ISSOTL)
 Accreditation standards
 Multiplicity of venues for presenting and
publishing
Enriching the experiences of our
students!
Lead.
Engage.
Inspire
.
References
 Aoki, T. T. (1991). Layered understandings of orientation in
social studies program evaluation. In Pinar. W.F. & Irwin, R.L.
(Ed.), (2005). Curriculum in a new key: The collected works
of Ted. T. Aoki. .(pp.167-185) Mahwah, NJ: Lawrence
Erlbaum..
 Association of American Colleges & Universities. (no date)
Integrative Learning VALUE Rubric. Retrieved from:
http://www.aacu.org/value/rubrics/integrativelearning.cfm
 Carper, B. (1978). Fundamental patterns of knowing in
nursing. Advances in Nursing Science, 1(1), 13-23.
 Chinn, P., & Kramer, M. (2008). Integrated theory and
knowledge development in nursing (7th ed.). St. Louis: Mosby
Elsevier.
Lead.
Engage.
Inspire
.
References cont’d.
 Stake, R. (2003). Responsive evaluation. In T.
Kellaghan & D.L Stufflebeam (Eds). The international
handbook of educational evaluation, . (pp. 63-68).
Dordecht: Klower Academic Publishers.
 Stufflebeam, D.L. & Shinkfield, A.J. (2007). Evaluation
theory, models, and applications. San Francisco:
Jossey-Bass
 Tanner, C.A. (2006). Thinking like a nurse: A researchbased model of clinical judgment in nursing. Journal of
Nursing Education,45(6), 204-211
 Thorne, S.E. 2008. Interpretive description. Walnut
Creek, CA: Left Coast Press
Inspire
.
Lead.
Engage.
Thank you
landeen@mcmaster.ca