Preparing Final Clerkship
Performance Evaluations
A Guide for Clerkship Directors and
Evaluation Teams
2013-4
Table of Contents
• Evaluation Criteria
• Data collection
• Interpreting evaluation data:
– Patient Care (RIME)
– Professionalism and Interpersonal
Communication
• Final grades
• Narrative evaluations
• Appeals
• Resources and Contacts
Evaluation Criteria
Grades and MSPE reporting
• Students may earn a grade of Pass or
Pass with Distinction in each of three
domains:
– Patient Care
– Professionalism and Interpersonal
Communication
– Final Exam
• Grades for each domain are reported
separately in the MSPE
Criteria for Pass
• Patient Care:
– Direct observations of clinical skills complete (2 per
clerkship)
– RIME Interpreter*
• Professionalism/Interpersonal
Communication:
– No significant or consistent concerns
– Patient logs, other assignments complete
• Exam score above clerkship passing
threshold
*See the CBEI website for additional details on the RIME framework.
Criteria for Pass with Distinction
• All Passing criteria must be met AND
• Patient Care:
– RIME Manager
• Professionalism and Interpersonal
Communication (IPC):
– Multisource feedback requested: 1 non-MD staff
member, 1 patient, 1 peer
– Consistent evidence of both Exceptional Professionalism
and IPC with patients AND members of the medical
team
• Exam score above clerkship-specific
threshold for Pass with Distinction
Data Collection
Data Collection
• Ideally
– 100% of potential evaluators
– will complete electronic evaluation forms
in E*Value.
– within a week of an evaluation being
assigned.
Data collection
• In reality
– The minimum acceptable return rate is
50%.
– Final evaluations must be submitted within
6 weeks of the end of the clerkship.
– Some evaluators will not want to use
E*Value.
Recommendations
• Keep an eye on return rates – start
reminding and re-requesting early.
• Clerkships may need to tailor
approaches to data-gathering to suit
the needs of different groups, sites, or
individual evaluators
Recommendations
• Avoid going into an evaluation team review or
submitting final evaluations with a return rate
under 50%. Evaluations based on input from
fewer than 50% of potential evaluators are
unlikely to stand up in an appeal.
• All evaluators who submit information
(residents, faculty, fellows, non-MD staff,
peers) should be listed as Contributing
Evaluators in the final evaluation form.
Data Collection
• For ideas, resources, and practical support in
boosting evaluation return rates, contact
– Jen Deitz (jdeitz@stanford.edu)
– Gretchen Shawver (gshawver@stanford.edu) or
– Elizabeth Stuart (aestuart@stanford.edu)
Interpreting Evaluation Data:
Patient Care (RIME)
Two paths to distinction
Performance
Threshold for Distinction
Time
Students may earn Pass with Distinction by meeting criteria
throughout the clerkship OR improving to meet criteria by the
end of the rotation
Expected Transitions
MANAGER
INTERPRETER
REPORTER
POM
Core
clerkship
Sub-I
Residency
+
Core clerkship students are expected to be in the Interpreter stage.
See the CBEI website for additional details on the RIME
framework.
Interpreting Data from Patient
Care (RIME) forms
• For a student in the Interpreter stage:
– Nearly all evaluators will select
Consistently True for items in the Reporter
section.
– Data will include a mix of Sometimes and
Consistently True for items in the
Interpreter section.
Reporter Items
Interpreter Items
Alert
• If significant number of evaluators throughout the rotation - have selected
Sometimes or Rarely True on Reporting
items – the student has not met expectations
for performance in a core clerkship and
should receive a non-passing grade.
• Clerkship directors should speak directly with
individual evaluators to confirm that a student
is not consistently demonstrating the skills
required in the Reporter stage.
Pass with Distinction
MANAGER
INTERPRETER
REPORTER
POM
Core
clerkship
Sub-I
Residency
+
Pass with Distinction requires consistently strong Reporting and
Interpreting (solid bars) with evidence of being in the transition to the
Manager stage.
Interpreting Data from Patient
Care (RIME) forms
• For a student in the Manager stage:
– Nearly all evaluators will select
Consistently True for items in the Reporter
and Interpreter sections.
– Data will include a mix of Sometimes and
Consistently True for items in the Manager
section.
Manager/Educator Items
Broadly
• Students in the Manager stage are
functioning above the expected level –
at the level of a sub-intern or beyond.
Q&A
Q: Should core clerkship students
really be expected to manage
patient care?
Q & A: “Manager”
A:
The term Manager is not meant to imply
independent implementation of patient
care plans. Students who are Managers
demonstrate a sense of personal
responsibility for knowing as much as
possible about their patients and ensuring
that they receive optimal care. Managers
see themselves – and are seen by others
– as patients’ primary providers and
advocates.
Q & A: M without I?
Q: The evaluators for one of my
students marked Consistently True
for all of the Manager items, but only
Sometimes True in the Interpreter
section. Does the student meet
criteria for being a RIME Manager?
Q &A: M without I?
A:
No. Pass with Distinction requires
consistently strong Reporting and
Interpreting skills in addition to the
skills and attitudes represented by
the Manager items on the evaluation
form.
Q & A: Outliers
Q:
What if one evaluator checked
Rarely or Sometimes True on an
item – when all other evaluators
marked Consistently True?
Q & A: Outliers
A:
When reviewing data for both
Patient Care and
Professionalism/Interpersonal
Communication, Evaluation Teams
should look for trends and themes over time and across evaluators.
Outliers
In sorting through outliers, consider:
•
Role/identity of the evaluator
•
Setting
•
Time the evaluator spent with the
student
•
Timing during the rotation
Interpreting Evaluation Data:
Professionalism and Interpersonal
Communication (IPC)
Fundamentals of Professionalism
and IPC
Interpreting data from the
Professionalism/IPC form
• To Pass, there should be a consistent
trend of meeting expectations, with no
significant or consistent concerns.
Exceptional Professionalism/IPC
Interpreting data from the
Professionalism/IPC form
• For Pass with Distinction, data from multiple
evaluators should show a consistent trend of
exceptional Professionalism AND Interpersonal
Communication with BOTH patients and
members of the medical team:
– “Yes” answers to the yes/no question
– Narrative examples of exceptional
professionalism AND interpersonal
communication
Multisource Feedback
(MSF)
• To meet criteria for Pass with Distinction,
students must request multisource feedback
from
– 1 non-MD staff member
– 1 patient
– 1 peer
• A response by peers, non-MD staff, patients
is not required.
• Clerkships must put systems in place to
record students’ MSF requests.
Q & A: Multisource Feedback
Q:
What if a non-MD staff member reports
concerns about professionalism? Does
that disqualify the student for PWD?
Q:
What if a patient or non-MD staff member
submits a glowing description of a student’s
Interpersonal Communication? Shouldn’t
that information count toward Pass with
Distinction?
Q & A: Multisource Feedback
A:
For the initial phase of CBEI, the content of
multisource feedback is not considered in
determining whether a student has met
criteria for Pass with Distinction
Positive comments from MSF content may
be used as examples in the final
summative evaluation.
NEW: Multisource Feedback
Updated
for 201314
Anonymized multisource
feedback comments should be
included in the formative
narrative section of the final
evaluation.
Final Grades
Final grade options
• As of July 2012 clerkships will enter one of
the following final grades for each student:
–
–
–
–
N - Continuing
Fail
Marginal Pass
Pass
• Pass with Distinction will be recorded
separately for each performance domain
N grade
• N is for failed exams.
• N should not be used routinely for:
– Marginal/non-passing performance in the domains
of Patient Care or Professionalism/Interpersonal
Communication
– Missed time from the clerkship
– Incomplete assignments
• Use Marginal Pass or suspend the evaluation
if a student has not completed all clerkship
requirements*
More on N
N
Use when…
Consequences
• All aspects of performance meet
criteria for passing, but student does
not pass the final exam.
• Should not be used for marginal or
failing performance in any other
domain.
Must retake exam
• Except in case of emergency, failure
to attend the final exam session
without prior permission from the
clerkship director will result in referral
to the Committee on Performance,
Professionalism and Promotion
(CP3) for unprofessional behavior.
After second failed attempt,
student takes oral exam or
suitable alternative, to be
determined by the clerkship
director.
N grades must be corrected
within 12 months of the end of
the clerkship.
See MD Program Handbook for additional details.
Q & A: N grade
Q: I have a student who was
admitted to the hospital with
appendicitis the night before
the shelf exam. Should I wait
to submit his evaluation until
he takes the exam?
Q & A: N grade
A:
No. The evaluation should be
submitted with an N grade within 6
weeks of the end of the clerkship.
Evaluations contain important
information that should be made
available promptly to students.
Evaluation submission should not
be delayed by failed or missed
exams.
Marginal Pass vs. Fail
Use when…
Consequences
MP
• Performance fails to meet criteria in one
domain (other than exam).
• Mild concern about Patient Care or
Professionalism/Interpersonal
Communication
• Reviewed by CP3.
• Remediation plan to be
determined by the clerkship
director.
• 3 Marginal Passes in clerkships
may be considered grounds for
dismissal.
F
• Performance fails to meet criteria in more
than one domain.
• Significant concern about Patient care or
Professionalism/Interpersonal
Communication
• Reviewed by CP3.
• Remediation plan to be
determined by the clerkship
director.
• 2 Failing grades in clerkships
may be considered grounds for
dismissal
See MD Program Handbook for additional details.
Q & A: Evaluation Review Shortcuts?
Q:
Does the full evaluation team need
to review files for all students, or just
those who seem likely to meet
criteria for Pass with Distinction in
one or more domains?
Q & A: Shortcuts?
A:
CBEI introduced not only a new
grade to recognize exceptional
performance, but also a new process
to ensure a fair and balanced review
for each student.
All students should benefit from the
process of a full review.
Q & A: Evaluator Lists
Q:
Does the final evaluation in E*Value
need to list every individual
evaluator by name?
Q & A: Evaluator Lists
A:
Updated
for 201314
All contributors should be listed
individually, by name.
If there is concern about protecting
the identity of an individual evaluator,
the clerkship may opt to list the
names of all evaluator who were
asked to contribute to the final
evaluation.
Narratives
Guidelines for Narratives
Summative vs. Formative
Guidelines for Summative
Narratives
• Summative narratives should be:
– 100-200 words long
– Framed as cohesive paragraphs (not lists
of quotes) reflecting student performance
in each of the key domains:
• Patient Care
• Medical Knowledge
• Professionalism and Interpersonal
Communication
Formative vs. Summative
• Summative evaluation involves a final
judgment and description of a
student's performance during a
clerkship.
• Formative evaluation provides ideas
and recommendations for further
learning and improvement.
Formative vs. Summative:
An Analogy
A cook asks for an evaluation of his soup
Adapted from Stanford School of Medicine Clerkship Evaluation Tutorial, 2006
Summative & Formative:
Analogy
Summative evaluation would
answer the question:
Formative evaluation would
answer the question:
How was the soup?
What might be done to
improve the soup?
Yuck!
Pretty good. Very tasty.
A culinary masterpiece!
Needs a little more salt.
Heat it up a bit more.
Some Tabasco would round
out the flavor.
Formative vs. Summative
• Summative narratives submitted in E*Value
are intended to be cut and pasted verbatim
into the MSPE.
• Formative narratives are not included in the
MSPE. Formative comments in E*Value are
seen by:
– Students
– Advising Deans
– Members of the Committee on Performance,
Professionalism, and Promotion (CP3).
– Members of the Clerkship Evaluations Committee
De-identified Individual Comments
• Comments from individual evaluators
(MD, non-MD, patients, peers) may be
included in the formative section of the
final evaluation as long as the identity
of the individual evaluator cannot be
guessed by the reader.
Q & A: Formative vs. Summative
Q:
I had a student who did the most amazing
physical exams but who had a very hard
time organizing presentations.
Should I put positive comments about his
exams in the summative section and leave
his difficulty with presentations for the
formative section?
from Stanford School of Medicine Clerkship Evaluation Tutorial, 2006
Q & A: Formative vs. Summative
A:
No. The summative narrative is
intended to provide an honest,
accurate, and complete description
of a student’s performance during a
given clerkship. Both "positive" and
"negative” aspects of performance
should be included.
Should I include…?
• “Negative” or less than glowing
comments should be included in the
summative section if they reflect:
– Significant or consistent trends in
performance over time
– Features of performance that did not
change despite mid-rotation feedback
– Characteristics of performance that should
be known to potential residency programs
Q & A: CBEI Wording in Narratives
Q:
Should the summative narrative
mention whether a student’s
performance met criteria for Pass
with Distinction in each domain?
Q & A: CBEI Wording in Narratives
A:
The authors of the MSPE will add
information to each narrative to indicate
whether a student’s performance met
criteria for PWD in each domain.
Clerkship directors do not need to
specifically mention Pass with Distinction
in the narrative.
See the next slide for examples.
MSPE format
Q & A: Concerns
Q:
I have concerns about a student.
He responded to mid-clerkship
feedback and deserves to pass the
rotation – but I worry that he will
have difficulty in future rotations and
beyond. Should I mention the
concerns I had in the formative
section?
Q & A: Concerns
A:
Yes. The Committee on Performance,
Professionalism, and Promotion (CP3)
relies on information in both the formative
and summative narratives in monitoring
student progress. Although formative
comments are not included in the MSPE,
they are helpful to CP3 as a formal
documentation of concerns.
Concerns
• The final evaluation form includes a
confidential question to allow clerkship
directors to report concerns about student
performance.
• These flags are viewed by Advising Deans,
Educators for CARE faculty, and the Director
of Clerkship Education. They serve as tool
for identifying trends in student performance..
• Confidential concerns are not seen by
students and not reflected any where in
the MSPE.
Resources
• For more information on narrative
evaluations, including guidance as to whether
information should be formative or summative
and suggestions for framing “negative” ideas,
contact:
– Jen Deitz (jdeitz@stanford.edu)
– Elizabeth Stuart (aestuart@stanford.edu)
Appeals
Appeals
• Some students will appeal their
final clerkship grades and/or the
content of their narrative
evaluations.
Appeals
• According to the SOM policy on
evaluation appeals, the process begins
with a discussion between the student
and the clerkship director or Advising
Dean.
• Unresolved disagreements are
referred to the Clerkship Evaluations
Committee.
See MD Program Handbook for additional details.
Appeals process
• The CEC will initially contact the appealing
student and the clerkship director to discuss
the evaluation in question.
• The CEC will then assemble a committee to
review the student’s evaluation file to
determine whether:
– the evaluation process was conducted fairly
– the final evaluation was based on sufficient
information
– the final grade and narrative warrant revision.
Appeals process
• For an appeal to be considered, a
written request made to the Division of
Evaluation within eight weeks of the
date that the final evaluation was
submitted in E*Value.
Appeal ≠ Failure
• A request for an appeal does not
constitute failure on the part of the
clerkship evaluation team.
• Appeals provide an opportunity for
learning and calibration. They serve an
important function in achieving the
broader goal of optimizing consistency
in the evaluation across clerkships and
sites.
Resources & Contacts
For more information, see the CBEI website:
http://med.stanford.edu/md/curriculum/CBEI/index.ht
ml
Or contact:
•
Elizabeth Stuart, MD, MSEd, Director of Clerkship Education
aestuart@stanford.edu
•
Jen Deitz, MA, Director of Evaluation jdeitz@stanford.edu
•
Gretchen Shawver, Clerkship Coordinator gshawver@stanford.edu