The Portfolio Project: experience in the UBC
Anesthesiology Residency program
Clinton Wong MD FRCPC
Clinical Professor
Program Director, Residency Program,
UBC Anesthesiology
Jennifer Joo MD
PGY 3 Resident
UBC Anesthesiology
UBC Faculty of Medicine
Medical Education Rounds
Wed. June 2, 2010
Outline today
• History
• What is presented to the anesthesiology
residents as a group to introduce them to
portfolios
• What I as PD do to review portfolios with
individual residents
• Final words about role of portfolios
• One resident’s perspective
• Questions
• Example of one Portfolio CanMEDS role
Learning objectives
• To learn about the role of portfolios in
documenting educational experiences, especially
non-medical expert CanMEDS roles
• To understand some of the barriers to
implementation of portfolios in a residency
program (“B”)
• To gain knowledge in some strategies to overcome
these barriers (“S”)
• To appreciate the benefits of portfolios from a
resident’s perspective
Anesthesiology Portfolio
Project-History
• Canadian Anesthesiology PD’s needed a way to teach and
assess the CanMEDS roles in their programs
• U of C (Dr. J. Todesco) strategy for the learning and
assessment of the Scholar role was presented in a portfolio
format
• Portfolios traditionally not used in medicine, but more
common in areas such as architecture
• Anesthesiology PD’s expressed interest in learning more
• Dr. J. Frank from RCPCS invited to speak at the Canadian
Anesthesiologists’ Society annual meeting in 2005 on
portfolios (S)
History
• 2005: ACUDA PD’s* discuss Portfolio Project nationally in
anesthesiology at half day workshop led by Dr. Ramona
Kearney (S)
• Association of Canadian University Departments of
Anesthesia Postgraduate Education Committee: Ramona
Kearney, Paul Bragg, Joanne Todesco, Sal Spadafora,
Narenda Vakharia, Rob Brown, Johanne Carrier, Francois
Gerard, Carolyne Goyer, Melanie Jaeger, Mark Levine,
Joanne Madden, Jean-Pierre Morin, Greg Peachey, Jeremy
Pridham, Mateen Raazi, Bruce Ramsey, Linda Wynne, Clinton
Wong
History cont’d
• PDs and educators agreed to try and develop a portfolio
approach to each of the CanMEDS roles
• Preliminary drafts of the portfolios were presented to
ACUDA education committee and further developed Dec/05 (S)
• Generic portfolio documents are now available for all roles –
Dec/06
• Each residency program can modify generic portfolio
document to meet their own program’s specific needs (B)
• All other anesthesiology programs in Canada have begun to
implement Portfolios by 2007 (S)
History of Portfolio project
UBC Anesthesiology Residency
Program
• July 2008: approved by RTC to introduce
Portfolio: Collaborator role to incoming PGY 1
residents, with plan to add 2 more roles each year
from July 2009 onwards. (B & S)
• More senior residents “grandfathered” and not
required to do portfolios (B & S)
• Currently 3 roles approved (Collaborator,
Manager, Health Advocate) for PGY 1-3
residents), with 2 more roles planned for July
2010 (Scholar, Professional)
• Eventually all 7 roles for all 5 PGY years
History of Portfolio project
UBC Anesthesiology Residency
Program
• PD meets with new PGY 1 residents to introduce
concept of Portfolios, and its role & importance
• (see PP presentation later)
• PD reviews each resident’s portfolios in biannual
meeting, discussion of entries, and focus
discussion on reflective component.
• At meeting reinforce/remind importance of
portfolios, esp. reflective component (S)
• Portfolio stays with resident, but notes made in
resident’s file about quality of portfolio for
different CanMEDS roles (B)
• What is presented to residents
during PGY 1 year: introduction to
Portfolios
• An abbreviated presentation, with
acknowledgements to Dr. Kearney for
her PP presentation
Portfolios
for
Anesthesiology
Ramona Kearney MD, MMEd, FRCPC
Dept. of Anesthesiology and Pain Medicine
University of Alberta
Objectives
By the end of this session the residents
will be able to use a portfolio to:
• Plan learning activities during
residency
• Reflect on learning experiences
• Document learning
Learning cycle in residency
assessment
objectives
learning
curriculum
Why assess learning?
To determine the acquisition of
knowledge or skills or attitudes
which are the expected outcomes
required
of a competent anesthesiologist, by
the end of residency
Traditional assessments
• MCQ
• Oral exam
• SAQ
These are used mostly to assess
knowledge and some skills.
Why focus on knowledge
assessment?
The competent anesthesiologist depends on Knowledge for
safe practice until sufficient experience is gained and
he/she becomes an expert
(after about 5 years of consultant practice)
Time alone is insufficient to move from competent consultant
to expert consultant; need reflection in all 7 CanMEDS
roles once out of residency since no one will directly
observe you anymore!
Competencies
Professional
Manager
Communicator
Medical
Expert
Health
Advocate
Scholar
Collaborator
What about other
competencies?
• Traditional assessments have limited value for
determining these non-Medical Expert
competencies
• Defining competence for these roles is difficult
• Different learners have different skill levels and
the goals for learning may be different
• Lifelong learning occurs in these areas
Problem
How then do we assess learning in the
areas other then medical expert?
• Programs need to collect good
information for a FITER and assure
their committees and the resident
that he/she is ready for the Royal
College exams.
• NOT Easy!
Typical Assessments
•
•
•
•
Assessments of daily work
Rotation assessments
Oral examinations twice/year
Written examinations twice/year
(standardized written exams)
Currently the Royal College uses these
assessments in Anesthesiology:
• Oral exam
• Written exam (MCQ + SAQ)
• FITER
Key Tools for assessing the CanMEDS competencies
ITER = In-Training Evaluation Report
OSCE = Objective Structured Clinical Examination
SP = Standardized patient
360°= Multi-source feedback
Medial
Expert
Communicator
Collaborator
Health
Advocate
Manager
Scholar
Professional
Written tests
+++
+
+
+
++
+++
+
Oral exam
+++
+
+
+
+
+
+
Direct
Observation
& ITER
+++
+++
+++
+
++
+++
+++
OSCE / SP
+++
+++
+
+
+
+
+
360° / Peer
evaluation
++
++
+++
++
++
+
++
Portfolio
++
+
+
++
++
+++
++
Simulations
+++
+
+
+
+
+
+
Preferred Assessments
• Direct observation/ITER
• 360° assessment
• Portfolio
Direct Observation and In-Training
Evaluation Reports (ITER’s)
 consists of a set of Likert scales, checklists, or global
assessment scales
 based on observed clinical performance
 includes broad evaluation of content and behaviour areas
 efficient
 evaluator can use multiple sources of information
 provides summative evaluation data
Direct Observation and In-Training
Evaluation Reports (ITER’s)
Disadvantages:
 difficult to assess specific skills/ knowledge content
areas
 prone to halo effect (overall impression influences specific
scores in each area)
 demonstrate leniency bias (evaluators not wanting to
provide negative evaluation)
 range-restriction bias (evaluators using small portion of the
scale for most evaluations)
What is a portfolio?
• “a collection of evidence maintained and
presented for a specific purpose”
• “collection of material, made by a
professional that records and reflects on
key events in that person’s career”
Role of reflection
• Container of evidence
(Cole)
Or
• Container of evidence + reflection
(Rees)
Without reflection, a portfolio becomes a log
(resident log book; RLB)
Portfolios
Learners record their experiences, their selfassessment of the experiences and define and
update learning goals.
 Includes supporting documentation reviewed periodically
by supervisor/program director
 Can sample a large breadth of content areas
 Promotes self-assessment and goal-setting
 Can identify areas of deficiency in individual experience
and program design
 Can assess progress over time
Portfolios
Disadvantages:
• Time consuming for data entry (B)
• Difficult to standardize and assess accuracy and
completeness (B)
• Time consuming for program directors to review
(B)
• Not good for pass/fail decisions or summative
evaluations (B)
Portfolios
Key points
• Unknown how accurate resident records are
(B)
• Relies on learner initiative (B+)
Sample Portfolio:
Collaborator CanMEDS role
• General outline: some didactic sessions, &
reference articles to read (tick boxes)
• Provide specific examples from clinical or
non-clinical settings as a narrative (“in
your own words”)
• Both to be combined with reflection
• (see example Word document)
Meeting with PD-why
portfolios? (motivation) (S)
• Mandatory by RTC for UBC
• Standardized nationally for
anesthesia residents
• To improve in all 7 CanMEDS roles,
esp. non-medical expert
Meeting with PD-why
portfolios? (motivation) (S+)
• Upon completion of residency, assumed clinically
competent, but not yet an expert
• To be an expert takes time (2-5 yrs), plus
reflection.
• Otherwise might practice for 30+ yrs & not be
any different than when first finished residency!
• Program needs to promote reflection during
residency so that it continues after residency
Meeting with PD-why
portfolios? (motivation) (S+)
• When applying for positions,
department assumes competency
(medical expert)
• But what about ability in other nonmedical expert roles? (professional,
communicator, collaborator, manager)
(formative role of portfolios)
Meeting with PD-why
portfolios? (motivation) (S)
• Portfolios can be used as part of an
application, in conjunction with CV and log
book to document learning in all 7
CanMEDs roles
• Residents can use portfolios to
demonstrate learning in non-CanMEDS
roles (much like an architect uses his/her
portfolio)
So why portfolios?
• Learners get an opportunity to reflect on experiences and this
leads to deep learning
• Learners develop insight into their own learning needs and their
development of competence
• Learning is individualized
• Lifelong learning is emphasized and enabled (CanMEDS Scholar
role)
Experiential learning
Kolb
Experience
Application
Reflection
Generalization
Implications
The learning is:
•
•
•
•
self-directed
experiential
reflective
meaningful
Principles of adult learning
Experience in medical
education
• Survey of UK medical schools, 2001 –
40% using portfolios in assessment
• General Medical Council, UK –
proposed a portfolio of activity used
for revalidation
Learners’ issues
• Must understand the purpose of the
portfolio (B)
• Must know what criteria the assessors
will use (B)
• Require a commitment (B+)
• Can find it arduous to use (B)
• Will apply variable effort and consistency
compared with other learners (B+)
• Can’t cram (B)
What do you do with this:
collaborator role
• Read the 2 reference articles
• Complete one narrative of a “difficult”
collaborative experience and have it ready
to present to me on your twice a year
meeting with the program director (first
meeting in the fall after completion of your
first rotation in anesthesia).
• Vital to include your reflective comment
• (end of presentation to PGY 1 residents)
What I do when I meet with individual
residents re: portfolios q6/12? (B)
• “Show me your portfolio”
• Focus on narrative component of experience
(fruitful area of discussion; focus on challenging
experiences)
• Look for reflection:
– what did I learn?
– what aspects did I do well?
– how would I do things differently next time?
• Suggestions for next meeting on portfolio
• Reinforce importance of portfolios (see
“motivation” slides)
Final words: the importance of
reflection on professional role
• “Educators now believe role modeling is
insufficient; it must be combined with
reflection on the action to truly teach
professionalism.”
• “Simply modeling professionalism without
following up with discussion constitutes a
missed opportunity for teaching
professionalism.”
• Stern, DT. et all. The Developing Physician-Becoming a
Professional. NEJM 2006: 355: 1794-9
Final Notes on Role of Portfolios
• RCPSC Accreditation
RCPSC Accreditation Presurvey Questionaire (PSQ): EVALUATION OF
RESIDENT PERFORMANCE (Standard B.6)"There must be mechanisms in
place to ensure the systematic collection and interpretation of evaluation
data on each resident enrolled in the program."
RCPSC Accreditation Presurvey Questionaire (PSQ): EVALUATION OF
RESIDENT PERFORMANCE (Standard B.6)"There must be mechanisms in
place to ensure the systematic collection and interpretation of evaluation
data on each resident enrolled in the program."
Dr. Jennifer Joo,
PGY 3 Anesthesiology
“A resident’s
perspective”
• Questions at end
• Sample Collaborator
Portfolio form at end