Self- Assessment
Karen V. Mann PhD
‘Fall Together’
Department of Family Medicine
University of Calgary
October 22nd, 2013
1
Objectives:
Participants will:





Explore the concept of self- assessment: how do people know “how
they are doing”?
Discuss an evidence-based model for understanding informed selfassessment
Discuss conditions that can facilitate informed self-assessment and
use of external feedback.
Consider the implications and apply these conditions and
approaches to practical examples.
Identify and develop practical educational strategies to enhance
informed self-assessment and feedback adoption, both for learners
and themselves.
2
How do we know “how we are
doing?”

Consider how you make judgments about
your performance as a physician, or teacher:
What sources of information do you use?

What challenges do you experience in
assessing your own performance ?
3
The self-assessment literature

“in a majority of the relevant studies, physicians do not
appear to accurately self-assess …” (Davis, et al., 2006)

“…the most important educational activity related to
self-assessment … should entail helping people
overcome their personal belief that they can rely on it.”
(Eva & Regehr, 2008)
What’s worse, those with the least
proficiency in a domain are the least
accurate self-assessors
(Kruger, Dunning 1999; Violato, Lockyer 2006)

Background

Self-assessing on one’s own is a little like…
5
Understanding informed self-assessment:
Social psychological theory
“The person and the situation; perspectives of social
psychology” (Ross, Nisbett 1991):

1.
Situational factors are powerful influences of behaviour.
2.
It is actually the person’s perceptions of situational
factors which are influential.”
3.
The social system and individual cognition/ perceptions
are in a dynamic state of flux. Therefore making
inferences about individual behaviour is complex/
difficult.
6
Understanding informed self-assessment:
Constructivist learning theory


Learning takes place in contexts
Learners form or construct much of what they
understand as a function of their


7
Experiences in situations
Social interactions
Understanding informed self-assessment:
Humanist learning theory


Learning is personal growth, self-actualization
 Includes affective and cognitive domains
 Focuses on personal goals and accomplishments
Learning is self-directed (which assumes some
degree of self-assessment)
Honest and accurate external feedback is crucial to
facilitate improvement and to accurate selfassessment
(Larsen, et al., 2008; Boehler, et al., 2007; Rees,Shepherd, 2005)
Background: The terms we use

Self-assessment:
“a global judgement of one’s ability in a particular
domain” (Eva & Regehr, 2010)

Informed Self-Assessment:
“a set of processes through which individuals use
external and internal data to generate an appraisal
of their own performance”. (Sargeant et al, 2011)
Informed self-assessment:
Summary of research

Funding:













Research grants: Medical Council of Canada
Additional funding: ABIM, Dalhousie University
Research team:
Joan Sargeant PhD, Dalhousie University (CA)
Kevin Eva PhD, McMaster University (CA)
Karen Mann PhD, Dalhousie University (CA)
Heather Armson MD, University of Calgary (CA)
Jocelyn Lockyer PhD, University of Calgary (CA)
Elaine Loney, MSc, Qual Research Consultant (CA)
Cees van der Vleuten, PhD, Universiteit Maastricht (NL)
Timothy Dornan MD, PhD, Manchester University (UK)
Eric Holmboe MD, Am Board Internal Medicine (USA)
Benjamin Chesluk PhD, Am Board Internal Medicine (USA)
11
Informed self-assessment: Research
objectives:
To understand 1.
how students, residents and physicians perceive
and use self-assessment in clinical learning and
practice
2.
their views of activities aimed at guiding selfassessment and factors that influence their use
3.
how external feedback might be better tailored to
the needs of health professionals
12
Research design and methods


International qualitative study using focus groups
drawing upon principles of grounded theory
Purposive recruitment from identified programs
using specific approaches to enhance selfassessment
13
Participating programs
3rd yr. medical students
NL, final yr. medical students
Belgium, final yr midwifery students
Undergrad
(students)
UK,
Post grad
(residents)

Physicians

UK, Foundation Program trainees
 US, Internal medicine residents
CA, family doctors, MSF program (PAR)
 CA, family doctors, PBSG learning
 US, internal medicine, chart audit
14
Results: Participants

17 focus groups
 2 for each of 8 programs except 3 for Canadian
MSF program

134 participants
 53 undergraduate/ student learners (3 programs)
 32 postgraduate/ resident learners (2 programs)
 49 practicing doctors/ consultants (3 programs)
15
Results – what we learned:


Self-assessment and “informing” self-assessment are
complex activities
They are:
 multi-faceted
 contextualized
 dynamic
 non-linear
16
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
1. Sources of information:
external (FEEDBACK)
I feel like I’m … kind of working under my own
steam, and thinking, “I hope I’m doing enough,
doing the right thing”. I got a ‘satisfactory’ but
it might be nice for someone to give a bit
more feedback about how we’re actually
doing, because sometimes you feel very
alone. (Medical Student B3)
19
1. Sources of information:
external
Anytime that you have to interact with a
colleague, be it at a consultant level,
personal, group practice level or as a
teacher, that always provides feedback. It is
a general amalgamation of all those things
that gives you an assessment of how you
are doing. (Physician L8)
20
1. Sources of information:
internal
… you are sometimes aware when you are talking
to patients that you are vague or are filling in gaps
that shouldn’t be gaps. I think we all do that to
come across as being coherent, but one should
be able to recognize when one is doing that, and
fill in the gaps... (Physician K3)
21
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
2. Interpretation of information
(REFLECTION)
If at the end of the day, if you haven’t asked
yourself some questions, then that’s a problem.
If you go through the whole day and think you
know all about all the problems that faced you,
then you don’t know. You have to ask yourself
why, because you can’t know everything.
(Physician M5)
23
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
3. Responses to information

. . . somebody gave me a ‘below-standard’
score for something, and at first you’re like,
‘Oh, my life, I can’t believe it! I’m not fit to be
a doctor!!” …then it actually makes you think,
‘yeah, okay, I am weaker in that area, I need to
work on it’. And it does kind of give you a
scare - ‘Okay, in future that’s something that
I’m going to have to do.’ (Postgraduate E6)
25
3. Responses to information
“But for some patients… I thought I’d
been keeping up with things, and I
found (from the audit) that actually,
no… I am missing things.” (Physician
P1)
26
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
4. External and internal
conditions/ influences: Climate
Because there is that level of trust within
the group, I don't mind my peers
knowing that I might not know the
answer to something. I don't feel they
would judge me … (Physician M2)
28
External and internal
conditions/ influences: Internal
“. . . When you are more confident, you can
ask for more detailed feedback.” (Medical
student H9)
29
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
5. Tensions



Overarching tension –”Between people”:

Wishing to learn and improve vs. wishing to
appear knowledgeable and confident among
peers and superiors
“Tensions in learning environment”:

Providing genuine assessment feedback vs
“playing the evaluation game”
“Tensions within self”

Wanting feedback yet fearing disconfirming
information
31
5. Tensions: Within people
“… there is this sense of intimidation that
you don’t want to speak up in case you
might just be deficient in your medical
knowledge.” (Physician M3)
32
5. Tensions: Within learning
environment
“I know you’ve got to play the game, but
that’s how I always feel when I’m doing it. I’m
playing the game. I don’t feel like I’m learning
and evolving and becoming a better person
because I’ve done it.” (Postgraduate E2)
33
Dimensions of Informed Self
Assessment
External & Internal Conditions –e.g.
External - learning climate,
relationships
Internal - emotions, confidence,
perceived credibility of feedback
Sources of
Information External –
feedback,
people,
processes
Internal –
“self”
Interpretation
of Information Reflect,
compare,
calibrate, filter,
assimilate
Tensions Within people
Between people
Within the learning environment
Responses to
InformationIgnore
Reject
Seek
Accept
Summary

Overarching finding:

No one activity was generally effective in informing
self-assessment


Effectiveness was moderated by external
factors, internal perceptions, tensions
The value of a specific activity was in how it was
used, not in the activity itself
Question for discussion
What is your experience in helping learners
to use self-assessment effectively?


What are the challenges? What have you found
helpful?
36
Discussion and activity
Considering our discussion of self- assessment,
today, and the factors that influence it…


What strategies might you use with your
learners?
How would you incorporate it?
37
Helping learners (1)




Providing structured opportunities
e.g., MSF, reflection on portfolios
Modeling self-assessment
Asking reflective questions – helping learners to
reflect on performance
Providing effective feedback
38
Effective Feedback

Information, provided by an agent (e.g., teacher,
peer, book, parent, self, experience) regarding
aspects of one’s performance or understanding.

Its goal: to identify the gap between current and
desired performance and to identify a route to get
there.
(Hattie & Timperley, 2007)
Effective Feedback: What We
Have Learned

Focus feedback on the task not the learner

Provide elaborated feedback to enhance learning

Give feedback in manageable units

Be specific and clear

Make feedback as simple as possible and no
simpler
(Shute, 2008)
40
Effective feedback: two helpful
concepts

Zone of proximal development (Vygotsky)
The distance from where a learner is currently
in their development to the level they can
perform at with assistance.

The importance of scaffolding
Scaffolding provides a framework for the
learner to achieve the task.
41
Helping learners (2)





Assisting with identifying sources of information
Promoting the seeking of feedback
Facilitating the incorporation and use of feedback
Recognizing the emotional content of feedback
Addressing the tensions ( within people, between
people and in the environment)
42
Tips: Educational strategies to
enhance informed self-assessment
Selecting assessment & feedback strategies:
 Feedback (formal, informal) is required to inform selfassessment
 To work, feedback needs to be timely, specific, credible,
authentic, provided by informed & engaged supervisors,
guided by standards
 Use of objective feedback data as possible; e.g., observed,
chart audit, practice review
 Assessment processes need to be credible, authentic,
meaningful (not simply “jumping through another hoop”)
Mann et al , 2011
43
Tips: Educational strategies to
enhance informed self-assessment

Faculty development:
 Increase awareness that feedback (formal,
informal) is required to inform self-assessment
 Provide timely, relevant, specific, feedback
 Engage learners in their own self-assessments
 Practice providing constructive feedback that will
help learners see how they can improve
 Create a safe learning environment where it’s OK
to say “I don’t know” or ask for help
Mann et al , 2011 44
Tips: Educational strategies to
enhance informed self-assessment

Learner/ physician development:






Help them to identify valid sources of external performance
feedback
Help them to develop skills in critically interpreting/ analyzing
the feedback data, and judging their performance
Help them be aware of and manage their emotional response
to corrective feedback
Teach skills for seeking specific feedback from supervisors that
will help them improve
Empower learners to seek feedback
Encourage physicians to seek feedback from colleagues
Mann et al , 2011 45
Summary and next steps

What will you take away?

Thank you!
46
Informed self-assessment:
Publications





Sargeant J, Armson H, Chesluk B, Dornan T, Holmboe E, Eva K, Mann K, Lockyer J,
van der Vleuten C, Loney, E. Processes and dimensions of informed selfassessment: A conceptual model. Acad Med. 2010; 85(7):1212-20.
Lockyer J, Armson H, Chesluk B, Dornan T, Holmboe E; Loney E, Mann K, Sargeant
J. Feedback Data Sources that Inform Physician Self Assessment, Med Teach,
2011; 33(2): Pages e113-e120 (DOI: 10.3109/0142159X.2011.542519).
Sargeant J, Eva KW, Armson H, Chesluk B, Dornan T, Holmboe E, Lockyer J, Loney
E, Mann K,. Features of assessment learners use for informed self-assessments of
clinical performance. Med Ed 2011; 45(6): 636-647. (DOI: 10.1111/j.13652923.2010.03888.x)
Mann K, van der Vleuten C, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E,
Lockyer J, Loney E, Sargeant J. Tensions in informed self-assessment: How the
desire for feedback and reticence to collect/use it create conflict . Acad Med 2011, 86
(9):1120-1127
Eva KW, Holmboe E, Lockyer J, Loney E, Mann K, Sargeant J. Factors influencing
responsiveness to feedback: On the interplay between fear, confidence, and
reasoning processes. Adv Health Sci Ed 2012 Mar;17(1):15-26