Skills and Concepts for
Evaluating
Interprofessional
Practice and Education
Barbara Brandt
Jean A. King
Michelle Gensinger
1
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Barbara Brandt and Jean A. King
INTRODUCTION TO IPECP AND
IMPLICATIONS FOR PROGRAM EVALUATION
2
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
The National Center:
A New Model for Public-Private Partnership
The National Center for Interprofessional Practice and Education
is supported by a Health Resources and Services
Administration $4M, five-year Cooperative Agreement Award
No. UE5HP25067.
In addition, the Josiah Macy Jr. Foundation, the Robert Wood
Johnson Foundation (RWJF), the Gordon and Betty Moore
Foundation, and the John A. Hartford Foundation have
collectively committed up to $8.1 million in grants over five years
to support and guide the center.
3
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
The National Center Vision
Our Goals:
a. Improved quality of experience for people, families, communities and learners
b. Shared responsibility for achieving health outcomes and improving education
c. Reduced cost and added value in health care delivery and education
4
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
National Center Principles
1. We are focused on sustainable national change.
2. We inform action with evidence, leveraging new
approaches to data collection and use.
3. We evaluate at each step and level, building on what
we discover.
4. We engage collaboratively as leaders, as colleagues
and as advisors.
5. We are learners with minds open to new ideas and
approaches.
6. We strive to demonstrate trustworthiness every day.
5
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Schedule for this afternoon’s
IPECP session
Time
Topic
Presenters
1:30-2:20
Introduction to IPECP
and Implications for
Program Evaluation
Barbara Brandt
Jean King
2:30-4:30
Assessing IPECP
Jean King
Michelle Gensinger
6
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Barbara Brandt
INTRODUCTION TO IPECP
7
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Jean A. King
IMPLICATIONS FOR IPECP
PROGRAM EVALUATION
8
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
In search of the “sweet spot”
Evaluation
practice
Evaluation of
IPECP
Interprofessional
education
and
collaborative
practice
(IPECP)
9
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Implications for the overlap
• There is no need to reinvent the wheel
• From the evaluation circle: What existing
evaluation processes will help document
and shape IPECP?
• From the IPECP circle: What is the current
status of program evaluation in IPECP?
How can we adapt or build on what is
already occurring?
10
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Possible evaluation approaches
to consider, for example
1. The CIPP model
2. Utilization-focused
evaluation
3. Developmental
evaluation
11
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Stufflebeam’s CIPP model
•
•
•
•
Context
Input
Process
Product
12
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Stufflebeam’s CIPP model
What’s good?
Any concerns?
• Provides clear focus for
virtually any evaluation
• Typically addresses
management’s
concerns (What about
other concerns that
may be important?)
• Applies to any stage of
a program
• Appreciated by
managers and other
decision makers
• Can result in expensive
evaluations
• Assumes that important
decisions can be
identified in advance
13
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Patton’s utilization-focused evaluation
Evaluation done for
and with specific,
primary intended
users for specific,
intended uses
14
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
The PERSONAL FACTOR in evaluation
"The presence of
an identifiable individual or
group of people
who personally care
about the evaluation
and the findings it generates"
15
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Patton’s utilization-focused evaluation
What’s good?
Any concerns?
• Focuses on people
who will actually use
the evaluation’s
process and/or results
• Requires users willing
to engage with the
evaluator
• Adaptable to virtually
any evaluation situation
and data collection
method
• Can be a challenge if
primary intended user
leaves the organization
• Potential for perception
of bias
16
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Developmental evaluation
• A special form of evaluation
appropriate to situations
that are complex and
dynamic, including social
innovation and systems
change
• A contrast to traditional
formative/summative
evaluation
17
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Developmental evaluation
What’s good?
Any concerns?
• Attends to the
complexity inherent
in many evaluation
settings
• Allows for shaping
and adaptation in
dynamic settings
• Not appropriate to
all evaluation
settings (a niche)
• Requires special
skills on the part of
both evaluator and
developer
18
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
How can you choose among approaches?
The classic evaluator’s answer:
It depends!
19
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
How can you choose among approaches?
The Program Evaluation Standards
• Utilization
• Feasibility
• Propriety
• Accuracy
• [Evaluation
accountability]
20
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Final thoughts
• Building evaluation processes/
systems will require that people
across a variety of settings/silos work
collaboratively
• It will be important to pay attention to
issues of USE
21
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Jean A. King and Michelle L. Gensinger
ASSESSMENT IN INTERPROFESSIONAL
EDUCATION AND COLLABORATIVE
PRACTICE (IPECP)
22
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
© 2013
Regents
theUniversity
University
of Minnesota,
Rights Reserved
Cooperative Agreement Award No. UE5HP25067.
© 2013
Regents of
of the
of Minnesota,
All RightsAll
Reserved.
One of the Center’s strategic priorities:
Evaluation/Informatics/Research
23
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Topics for today’s session
 Identify the challenges of assessment (both
quantitative and qualitative) of IPECP
 Review the National Center's search process and
inclusion criteria for assessment instruments
 Apply an outcome-based framework to group
assessment instruments
 Select from and try out a set of existing instruments
 Explain how people might use IPECP assessment
instruments
24
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Three-step interview
• What is the best assessment
experience you have had, either as a
student who was assessed or as an
instructor assessing someone?
• Questions:
• What can we learn from each others’
assessment experiences?
• What are your specific needs/interests
for assessing IPECP learning?
25
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
The big picture and individual snapshots
IPECP LEARNING ASSESSMENT
26
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
What is assessment?
Educational assessment is
the process of documenting,
usually in measurable terms,
• Knowledge
• Skills
• Attitudes
• Beliefs
27
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Assessment at the National Center
• Learning assessment needs to occur at
three levels (micro-, meso-, and macro-)
• These assessments have different
purposes
• All must rely on measurement instruments
to collect the assessment data—hence the
National Center’s curated collection of
instruments
28
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Editorial: A critical need in the field
 IPECP faculty,
clinicians, researchers,
and evaluators need a
set of solid instruments
that measure different
types of learning
 Good instruments are in
short supply right now
29
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options
•
•
•
•
•
Who
Where
When
What
How
30
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options: WHO
• Individual learners
• Teams in practice (e.g., a class or
workshop participants, clinical teams)
• An institution (university, health system)
• The national learning system
31
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options: WHO and
WHERE (grouped)
• Micro-level
• Individual learners
• Clinical settings
• Meso-level
• An institution (university or health system)
• A group of institutions
• Macro-level
• Policy settings
32
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options: WHEN
33
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options: WHAT
The Interprofessional Education
Collaborative (IPEC) competencies
1. Values and ethics for interprofessional
practice
2. Roles and responsibilities
3. Interprofessional communication
4. Teams and teamwork
34
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
More WHAT: Other needed competencies
• Population health, including social
determinants
• Patient-center decision making
• Evidence-based decision making
• Cost-effective practices
• Quality improvement and safe practice
• Stewardship
• Systems thinking
35
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
The Canadian Interprofessional
Health Collaborative (CIHC) framing
• Outcome Level 1: Attitudes
• Outcome Level 2: Knowledge, skills,
abilities
• Outcome Level 3: Behavior
• Outcome Level 4: Organizational level
• Outcome Level 5: Patient satisfaction
• Outcome Level 6: Provider satisfaction
36
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
A well-known way to frame training outcomes
The 4-level
Kirkpatrick
model
37
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
CIHC Outcomes*
(adapted from Kirkpatrick)
Kirkpatrick Model
Level 1:
Reaction
To what degree participants react
favorably to the training
Level 2:
Learning
To what degree participants
acquire the intended knowledge,
skills, attitudes, confidence and
commitment based on their
participation in a training event
Outcome Level 2:
Knowledge, skills, abilities
Level 3:
Behavior
To what degree participants apply
what they learned during training
when they are back on the job
Outcome Level 3: Behavior
Level 4:
Results
To what degree targeted outcomes
occur as a result of the training
event and subsequent
reinforcement
Outcome Level 4:
Organizational level
Outcome Level 1: Attitudes
Outcome Level 5:
Patient satisfaction
Outcome Level 6:
Provider satisfaction
*Modeled after Barr's (2005) six-level framework based on Kirkpatrick (1967)
38
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options
•
•
•
•
•
Who
Where
When
What
How
39
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
IPECP learning assessment options: HOW
•
•
•
•
Knowledge tests
Interviews
Self-reflection
Surveys
•
•
•
•
Attitude scales
Belief statements
Observations
Ratings by others
(including 360 degree)
PLEASE ADD
HERE. . .
40
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Reflection
• What are the
challenges facing
IPECP assessment?
• Where would you
begin if you wanted
to change practice?
• What counts as a
valid instrument?
41
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
An ever-growing and evolving collection
A CURATED COLLECTION OF
IPECP INSTRUMENTS
42
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
How did we identify the instruments
included in the curated collection?
Step 1- Our initial review criteria:
• EITHER the actual tool was included in the
CIHC listing
• OR it had been used in research leading to
at least one refereed journal article
43
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
How did we identify the instruments
included in the curated collection?
Step 2- More stringent criteria (a proxy for
evidence of use and adaptability)
• Use in two or more distinct projects
• Having been reported in two or more
refereed journal articles
44
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
How did we identify the instruments
included in the curated collection?
Step 3- An Internet search for additional
IPECP instruments using the more stringent
criteria
Step 4- National Center staff contacted each
individual developer to confirm content,
contact information, and availability
45
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
A comment on the presence of the RIPLS
The instrument
everyone loves
to hate
BUT WHY?
46
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Introducing. . .
The National Center website’s
assessment tool collection
www.nexusipe.org
47
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
We seek your input!
Members of the IPECP community can
leave comments about instruments



Experiences with a specific tool
What's good/what's bad about an
instrument– comments welcome
Additional references
60
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Identifying instruments for
specific uses
Consider specific contexts, and use the
curated collection to identify possible
instruments you could use
61
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Submitting your own instruments
o
o
o
These instruments will appear on the
Resource Exchange "landing page" with all
the other Resources
They are distinguished by the field called
"submitted by"
The National Center will review these
instruments, and some will be added to the
curated collection
62
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Wrapping up what we’ve discussed
IN CONCLUSION. . .
63
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Reviewing topics for today’s session
 Identify the challenges of assessment (both
quantitative and qualitative) of IPECP
 Review the National Center's search process and
inclusion criteria for assessment instruments
 Apply an outcome-based framework to group
assessment instruments
 Select from and try out a set of existing instruments
 Explain how people might use IPECP assessment
instruments
64
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
Why is it important to expand and build
this instrument collection?
•
•
•
Assessment data can help you have
important conversations
Assessment can provide data for deciding
what to do/change in your program
If we all work together on developing and
using assessments, we will help the field
65
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.
What
questions
remain?
Thank you for
participating
66
The National Center for Interprofessional Practice and Education is supported by a Health Resources and Services Administration
Cooperative Agreement Award No. UE5HP25067. © 2013 Regents of the University of Minnesota, All Rights Reserved.