Faculty Development Session on Evaluations
Graduate Medical Education
Ann Dohn, MA,
Designated Institutional Official (DIO)
Nancy Piro, PhD
Program Manger/Education Specialist
Kim Walker, PhD
Program Manger/Education Specialist
1
Department of Graduate Medical Education (GME)
ACGME Requirements: Seven Domains
I. Institution
II. Program Personnel & Resources
III. Resident Appointments
IV. Educational Program
V. Evaluation
I.
II.
III.
Resident
Faculty
Program
VI. Resident Duty Hours
VII. Experimentation & Innovation
2
Department of Graduate Medical Education (GME)
Faculty Development Session
EVALUATIONS
How do we eliminate unintended bias
from our evaluation process?
3
Department of Graduate Medical Education (GME)
What is cognitive bias…
• Cognitive bias is distortion in the way we
perceive reality / information.
• Response bias is a type of cognitive bias
which can affect the results of an
evaluation if evaluators answer questions in
the way they think they are designed to be
answered, or with a positive or negative
bias toward the fellow being evaluated
4
Department of Graduate Medical Education (GME)
Where does response bias occur?
• Response bias most occurs most often in
the wording of the question.
– Response bias is present when a question
contains a leading phrase or words.
– Response bias can also occur in rating scales.
• Response bias can be in the raters
themselves
– Central Tendency
– Halo Effect
– Similarity Effect
5
Department of Graduate Medical Education (GME)
Examples of Question Bias
• Example 1: "I can always talk to my
Program Director about residency related
problems."
• Problem: Terms such as "always" and
"never" will bias the response in the
opposite direction.
• Result: Data will be skewed.
6
Department of Graduate Medical Education (GME)
Examples of Question Bias
• Example 2: “Career planning resources are
available to me and my program director
supports my professional aspirations."
• Problem: Double-barreled ---resources and
aspirations… Respondents may agree with
one and not the other. Evaluator cannot
make assumptions about which part of the
question respondents were rating.
• Result: Data is useless.
7
Department of Graduate Medical Education (GME)
Examples of Question Bias
• Example 3: "Communication in my program
is good."
• Problem: Question is too broad. If score is
less than 100% positive, researcher/evaluator
still does not know what aspect of
communication needs improvement.
• Result: Data is of little or no usefulness.
8
Department of Graduate Medical Education (GME)
Rating Scale Bias
Competence and knowledge in general
medicine.
Poor Fair Good Very Good Excellent
The data will be artificially skewed in the
positive direction with this scale because
there are far more (4:1) positive than negative
rating options.
9
Department of Graduate Medical Education (GME)
Rater/Evaluator Bias
Response bias can be in the
evaluators themselves
• Central Tendency
• Similarity Effect
• Halo Effect
10
Department of Graduate Medical Education (GME)
Beware the Halo Effect
• The halo effect refers to a cognitive bias whereby the
perception of a particular behavior or trait is
influenced by the perception of the former traits in a
sequence of interpretations.
• Thorndike (1920) was the first to support the halo
effect with empirical research.
– People seem not to think of other individuals in
mixed terms; instead we seem to see each person
as roughly good or roughly bad across all
categories of measurement.
11
Department of Graduate Medical Education (GME)
The Halo Effect and Expectations
• The halo effect is involved in Kelley's
implicit personality theory
– the first traits we recognize in other
people influence our interpretation and
perception of later ones because of our
expectations.
12
Department of Graduate Medical Education (GME)
Halo Effect Extends to Products /
Marketing
The iPod has had positive effects on perceptions of Apple’s other products…
13
Department of Graduate Medical Education (GME)
Could this impact our evaluations here?
Empirical evidence from our HouseStaff….
A question from the most recent GME HouseStaff survey:
“The general feeling in my program is that your ability will
be labeled based on your initial performance.”
14
Department of Graduate Medical Education (GME)
17.9%
10.3%
Overall Peds Fellows
71.8%
24.4%
15.6%
54.7%
Overall SHC-LPCH
Reverse Halo Effect
• A corollary to the halo effect is the
reverse halo effect (devil effect)
– individuals, brands or other things
judged to have a single undesirable
trait are subsequently judged to have
many poor traits, allowing a single
weak point or negative trait to
influence others' perception of the
person, brand or other thing in
general.
15
Department of Graduate Medical Education (GME)
Blind Spots
• In the 1970s, the social psychologist
Richard Nisbett demonstrated that we may
have no awareness of when the halo effect
influences us (Nisbett, R.E. and Wilson, T.D., 1977)
The problem with Blind Spots is that we are blind to them…
16
Department of Graduate Medical Education (GME)
Know the ACGME Core Competencies
and How You are Evaluating Them
17
Department of Graduate Medical Education (GME)
Questions?
18
Department of Graduate Medical Education (GME)