Quantifying and Measuring
Professionalism: Demonstrating the
Professionalism Outcomes of
Pharmacy Curricula
Presenters
• Dana Hammer,RPh, PhD
– University of Washington, Pharmacy
• Kristin, K. Janke, PhD
– University of Minnesota, Pharmacy
• Katherine A. Kelley, PhD
– Ohio State University, Pharmacy
• Steven A. Scott, PharmD
– Purdue University, Pharmacy
Session Goals
• Gain an understanding of the unique
challenges of measuring
professionalism
• Explore the unique aspects of
professionalism as it relates to
interdisciplinary health care teams
• Gain and enhanced understanding of
how to use professionalism assessment
data to inform curricular change
Creating a Tool to Measure
Professionalism Behaviors: A MultiInstitutional Pilot
Steven A. Scott, PharmD
Kristin K. Janke, PhD
Katherine A. Kelley, PhD
Best Practices for
Professionalism Assessment
•
•
•
•
•
•
Multiple assessors
Multiple settings
Multiple methods
Formative
Beginning early in the curriculum
Improve existing tools rather than create
new
Lynch, et al, 2004
Activity
When working with colleagues, I behave
in a ways that communicate my respect
for them.
1.Strongly agree
2.Agree
3.Neutral
4.Disagree
5.Strongly disagree
Barriers to Measuring
Professionalism
• Definitions tend to be abstract
• Context specific
• Reluctance to report or address
unprofessional behaviors
• Ceiling effects
Ginsburg et al, 2000
Case Example
• A student you have on rotation dresses
professionally and appropriately
everyday. You have noticed that the
student tends to talk down to and act
rudely toward the office staff members
at your practice site.
– Identify/name the professional behaviors of
this student?
– What professional attitudes would you
attribute to this student?
Professionalism: Internally held
vs Externally Exhibited
• Attitude or behavior?
– I am respectful of others
– I dress professionally on rotation
• Attitude or behavior?
– I am trustworthy
– I protect patient confidentiality
Challenges Associated with
Measuring Professionalism
• Often more difficult to measure professional
behavior during the didactic curriculum than
during the experiential curriculum
–
–
–
–
Fewer opportunities
Situations are often simulated vs. real-life
Expectations may not be consistent
Faculty deal with many more students than in the
practice setting
Challenges Associated with
Measuring Professionalism
• The major focus of assessments and assessors is
to evaluate the student’s knowledge and skills, not
necessarily behaviors.
• Professionalism is more difficult to assess than
knowledge or the demonstration of a skill and
usually plays a minor role in any course grade.
• Most assessors have not been trained to evaluate
professional behaviors. (Bryden, et al, 2010)
Challenges Associated with
Measuring Professionalism
• Assessments of professionalism are often sporadic and
isolated and thus the professional development of
students is difficult to monitor. (Lucey and Souba, 2010)
• Assessment of professionalism is often carried out by a
variety of individuals who have not observed the student
previously.
• Isolated incidences of unprofessional behavior may not be
communicated and thus undesirable behaviors must be
rediscovered as the student matriculates through the
curriculum.
Challenges Associated with
Measuring Professionalism
• Volunteer faculty, other healthcare professionals,
and patients are often uncomfortable making
summative judgments about student
professionalism after limited interactions. (Swick, H.,
2007, Hickson, et al, 2007)
• Other healthcare professionals may focus more on
roles than on behaviors.
• Peers are often hesitant to provide honest
evaluations. (Reed, et al, 2008)
Challenges Associated with
Measuring Professionalism
• Behavioral assessments conducted prior to
graduation are prone to being influenced by
student motivation to achieve a desired grade,
rather than being reflective of the student’s true
value system. (Brown and Ferrill, 2009)
• Although valid assessment tools have been
developed, no standard approach to measuring
professionalism has emerged. (Stern DT, et al, 2006)
Multi-Institutional
Development
• 7 institutions* collaborated to create an
instrument to measure behavioral aspects of
professionalism
• GOAL: use previous literature as a foundation
to create an instrument that minimizes ceiling
effects
• Pilot: Spring 2009 in three schools
*Big 10: Purdue University, University of Illinois Chicago,
University of Iowa, University of Michigan, University of Minnesota,
Ohio State University, University of Wisconsin
Multi-Institutional Work
• 7 institutions* collaborated to pilot the
instrument
• GOAL: cross-validation of instrument
• Spring 2010 with first and third year
students
*Purdue University, University of Illinois Chicago, University of Iowa,
University of Minnesota, Ohio State University, University of Wisconsin,
AND University of Kansas
The Instrument
• Professionalism Assessment Tool
(PAT)
• Foundational instruments:
– Physicianship Evaluation Form
(Papadakis, 2001)
– American Pharmacists Association white
paper on student professionalism (2000)
– Hammer et al 2000
– Chisholm et al 2006
The Instrument
• Domains
– Reliability, responsibility and accountability
– Lifelong learning and adaptability
– Relationships with others
– Upholding principles of integrity and
respect
– Citizenship and professional engagement
Self-assessment Ratings
• Foundations
– Pangaro, 1999
• RIME: Reporter, interpreter, manager,
educator
– Wanted a basic level
• Novice
– Miller, 1990
• Knows, knows how, shows, shows how, does
The Final Ratings
“Know”: I understand these responsibilities, but may perform one or
more inconsistently, at times.
“Know how”: I understand these responsibilities and perform them in
a reliable, consistent and accountable manner.
“Show”: Without prompting or support from instructors, preceptors or
managers, I determine when and how to engage in these
responsibilities.
“Show how and Does”: I am confident in assisting others with these
responsibilities or proposing or creating options to fulfill these
responsibilities.
“Teach how”: I have mastered these responsibilities and desire to
learn more and share my learning with others. I demonstrate
maturity, confidence and an ability to educate others in these areas
through the use of evidence and strong interpersonal skills.
Total Items
Maximum
Value
Possible
M (SD)
8
40
31.3 (7.2)
Lifelong Learning and
Adaptability
7
35
22.3 (6.4)
Relationships with
Others
9
45
31.7 (7.2)
5
25
17.8 (5.0)
4
20
12.6 (4.1)
Factor Names
Upholding Principles of
Integrity and Respect
Reliability,
Responsibility, and
Accountability
Citizenship and
Professional Engagement
Kelley KA., et al. A Multi-Institutional Pilot for Cross-Validation of a Tool to
Measure Professionalism Behaviors. Amer J Pharm Educ, 2011 article 179.
Cross-validation
• Internal consistency was good (all 5
factors coefficient alpha >0.9)
• Both exploratory and confirmatory factor
analysis techniques were used
• Factor analysis demonstrated that all
items were correctly loaded under their
assigned domain
Limitations
• 155 surveys removed due to incomplete
data
• Homogeneity of the survey sample (all 7
pilot schools are public research
intensive)
Future Directions
• Longitudinal data collection to
determine growth over time
• Use of instrument by external
evaluators (preceptors and faculty)
• Examine possibilities in curricular
revision
Lessons Learned
•
•
•
•
•
Measuring professionalism has value
Assessing professionalism is possible
Building on previous literature is helpful
Multi-institutional work can be productive
There is more work to be done!
– Multiple assessors & multiple settings
– Longitudinal: beginning early, using multiple
measures
– Formative (for student and curriculum!)
Challenges, Opportunities, and
Best Practices for Interprofessional
Professionalism
Dana Hammer, RPh, PhD
Closing the Loop: Using
Assessment Data to Improve
Professionalism Curricula
Group Discussion
Discussion Questions
• How do you assess professionalism in
your curriculum? Or how would you like
to assess professionalism?
• How does professionalism assessment
data feedback to improve the
curriculum? Or how would you like to
use professionalism data to improve
your curricula?