Lee Atkinson-McEvoy
Chandler Mayfield
October 24th, 2011


Thinking about implementing in your
program?
Learners (students , residents or fellows) are
using it and are asking for your participation?






Introduction
Portfolio vocabulary
Putting it all together: The Big Picture
Examples of electronic portfolios
Review of portfolios by faculty members
Getting started…

Introduction
 What is the purpose of this workshop?
 Why are we talking about portfolios?
(Why is this culture emerging?)



Orientation to portfolios
Introduction to the process of setting up a
portfolio for learners
Discussion of the role of faculty and mentors
in evaluation of the portfolio

Increasing emphasis on creating lifelong
learners

Portfolios enhance
 Professional development
 Assessment
 Lifelong learning

Growing interest in portfolios from
accreditation and certification boards


Introduction
Portfolio vocabulary

“A portfolio is a purposeful and longitudinal
collection of tangible evidence of learnerselected work that exhibits the learner’s
efforts, progress or achievement.

This portfolio features
 the criteria for selection and judging merit, and
 includes evidence of learner reflection.”
UCSF EPIC Report, Adapted from the Committee on Student Assessment’s portfolio
definition from Reckase, 1995.

Formative or Learning Portfolio: Portfolio
used to aid learning, with accompanying
feedback to improve performance

Summative or Assessment Portfolio:
Portfolio used for evaluating and / or grading
a learner’s performance for the purpose of
determining their readiness for advancing to
the next phase of an educational program
Evidence/
Artifact
Criteria/
Assessment
Learning
Plan
Critical
Reflection
Present
Evidence/
Artifact
Criteria/
Assessment
Learning Plan
Critical
Reflection
Present

Artifacts: Products that indicate individual’s
work and advancement.

Examples:
 Test results
 Poster/abstract/journal article
 Letter from a patient or a patient's family member
 Resume,
 Select narrative assessments from E*Value
 Slides from a lecture
 Blog entry or essay about experience with patient
Evidence/
Artifact
Criteria/
Assessment
Learning Plan
Critical
Reflection
Present
Critical
Reflection

Critical reflection: is analysis of personal
experience to enhance learning and improve
future behavior and outcomes.
 Without it, portfolio would not be a portfolio but a
database.
 By critically reflecting on the artifacts in their
portfolios, learners identify their own strengths,
opportunities for improvement, and development
over time.





Goes beyond “consideration or meditation”
A skill developed over time for learning and
lifelong professional development.
Not just what happened (anecdote)
More than your personal opinion of events
Requires
 Data gathering
 Integration of past, present and future experience
 Analysis and reframing
Adapted from Louise Aronson’s presentation on Critical Reflection

Critical reflection should describe
 how a piece of evidence is related to the
educational competency the learner is addressing
 what was learned about the competency
 what the learner discovered about themselves (as
a learner, clinician or human being) from the
experience
 how this discovery can be used to further
professional development
Evidence/
Artifact
Criteria/
Assessment
Learning Plan
Critical
Reflection
Present

Presentation Portfolio (or view): Display of
evidence chosen by learner with reflection,
organized and communicated to an advisor,
mentor or others for a specific purpose
Evidence/
Artifact
Criteria/
Assessment
Learning Plan
Critical
Reflection
Present

Learning Plan:
 is a learner-specific program or strategy for
learning
 Integration of critical reflections based upon
evidence for all areas into a plan which informs an
efficient process for learning
Evidence/
Artifact
Criteria/
Assessment
Learning Plan
Critical
Reflection
Present




Review of the learning plan by an advisor,
mentor or program/course director.
Should have specific criteria, guidelines by
which to assess where the learner is within
the program.
If part of a summative evaluation some
formal rubric should be used.
This information should be available to the
learner to assist in the critical reflection and
development of a learning plan.



Introduction
Portfolio vocabulary
Putting it all together: The Big Picture




Introduction
Portfolio vocabulary
Putting it all together: The Big Picture
Examples of electronic portfolios

For more information: http://portfolio.ucsf.edu





Introduction
Portfolio vocabulary
Putting it all together: The Big Picture
Examples of electronic portfolios
Review of Portfolios by Advisors/Mentors

Faculty
 Should know what is expected of the learner
 Give feedback in a timely fashion
 Use a rubric if provided to guide feedback

Peer(s)
 Can be used to provide feedback
 Must follow the same guidelines as the faculty

Rubrics: multidimensional scoring guidelines
used to provide consistency in evaluating learner
work
 elucidate scoring criteria so multiple evaluators, using
the same rubric, would arrive at same score or grade
 provide a measurement system for specific tasks and
are tailored to each project
 let learners know what is expected of them
 provide learners the opportunity to do selfassessment to reflect on the learning process
CLINICAL SKILLS COMPETENCY
READ ESSAY FOR EACH STANDARD. ENTER YOUR JUDGMENT IN THE APPROPRIATE COLUMMN AND NOTE EVIDENCE TO SUPPORT YOUR JUDGMENT.
Notes Section: List best evidence and provide a brief
Check One Box
Year 2 Standards
summary to support your decision.
 Can perform individual
components of standard history
and physical and integrate these
components into a
comprehensive evaluation.
Met
□
 Establishes relationships with
patients using a patient-centered
approach.
Met
□
 Differentiates abnormal from
normal physical exam findings.
Met
□
 Reviews past medical records
and uses this information in the
current evaluation of their
patients.
Met
□
Met
With
Concerns
□
Did
Not
Meet
□
Insuff Evid
□
Evidence:
Brief Summary:
Met
With
Concerns
□
Did
Not
Meet
□
Insuff Evid
□
Met
With
Concerns
□
Did
Not
Meet
□
Insuff Evid
□
Met
With
Concerns
□
Did
Not
Meet
□
Insuff Evid
□
Evidence:
Brief Summary:
Evidence:
Brief Summary:
Evidence:
Brief Summary:
Justify your decision
Clinical Skills
Competency
Met
□
Met
With
Concerns
Did
Not
Meet
□
□
Insuff Evid
□
Does not meet
Needs improvement
Meets
Exceeds
Selects evidence that
demonstrates
current progress
relative to the
milestone
Evidence does not
provide information
on the skills/
activities listed in the
milestone
Appropriate evidence
selected, but provides
limited details to be able
to infer progress toward
the milestone; no
information in reflection
that provides context for
the evidence
Evidence
demonstrates the
learner can perform at
the level of the
milestone or has a plan
that makes achieving
the milestone likely
Evidence supports
achievement of the
milestone and has
characteristics of year
two milestone
Provides a
thoughtful reflection
and assessment of
the learner’s
progress toward
competency in the
domain
Describes a situation
that does not address
domain skills;
perhaps focusing on
someone else’s
situation; does not
take the reflection
seriously
Discusses a specific
experience, but does not
indicate critical
reflection; no lessons
learned are drawn from
the evidence/experience
Learner described the
value of the
experience/evidence
and articulated a
fundamental lesson
learned
Learner described the
experience and
addressed multiple
and/or complex
lessons learned.
Describes how
feedback has
informed learning in
the domain.
Does not indicate
what feedback was
obtained to help
develop competency
toward milestone.
Description relies solely
on personal perception
about what and/or how
to improve in this
domain.
Based on own
perceptions and one
other source
formulates a plan that
is consistent with
evidence provided for
this domain.
Indicates more than
one source of
feedback and clearly
describes how that
information has guided
learning.
Describes strategies
for future growth in
the domain
Not clear what the
learner’s strategies
are for future growth
Describes strategies for
future growth but does
not articulate specific
resources or plans
Clearly identifies
relevant next steps for
growth in the domain
including resources,
strategies that are
timely and feasible
Describes specific,
timely, and feasible
plan for future growth
in the domain and
considers personal
and professional S &W

Benchmarks: A reference point that serves as
a basis for judging whether a student has met
the associated educational standard.






Introduction
Portfolio vocabulary
Putting it all together: The Big Picture
Examples of electronic portfolios
Review of portfolios by faculty members
Getting started…




Clear Purpose
Provide criteria for selection/specific
instructions to guide selection of evidence
Require reflection
Set clear expectations
 As to how much evidence (remember lean is
good)
 Clear timeline
 Clearly communicate to other faculty their role
 Make sure that feedback is timely
Portfolio Step*
1.
Collect Evidence
2.
Reflection
3.
Evaluate evidence
4.
Defense
5.
Summative Decision
*
Responsible party
Program/trainee
Trainee
Mentor/Trainee
Trainee to program
Program
Friedman Ben David, M, Davis, MH, Harden, RM, et al. (2001). AMEE Medical Education Guide
No. 24: Portfolios as a method of student assessment. Medical teacher, 23(6), 535-551.


What incentives are there for the learners to
complete this portfolio? (there have to be
teeth in the process for most learners)
What barriers do you anticipate for:
 Learners
 Faculty

Suggestions to overcome barriers…







Introduction
Portfolio vocabulary
Putting it all together: The Big Picture
Examples of electronic portfolios
Review of portfolios by faculty members
Getting started…
Are they worth the effort?
Are learning portfolios worth the effort? No
 Its versatility makes it hard to judge whether a portfolio is
good or bad
 Limited evidence; much obtained from self-report
 Not clear that effort is rewarded
 Benefit may be in a side-effect:
 forces students to write something and teachers to spend some time
with students
Norman, G. BMJ 2008:3337:a514
Are learning portfolios worth the effort? Yes






Portfolio is best solution since it allows the collation
and integration of evidence
Portfolios can guide and coach professional
development
Flexibility is advantage when carefully developed
Mentoring is the single most decisive success
factors
Portfolios must be smart and lean
Careful implementation is crucial
Driessen, E. BMJ 2008:337:a513







Introduction
Portfolio vocabulary
Putting it all together: The Big Picture
Examples of electronic portfolios
Review of portfolios by faculty members
How to get started…
Are they worth the effort? Yes.