Slide 1
Authentic Performance
Assessment
Ellen J. Lehning, Ph.D.
Department of Anesthesiology
Slide 2
Objectives

Define clinical competence and authentic
performance


Design an authentic performance
assessment
Implement an authentic performance
assessment
Slide 3
What is the First Principle of
Education?
Teach to and Test Desired Endpoints:
Educational Outcomes
Slide 4
Desired Outcome:
Pass Written Boards
Pass Oral Boards
Achieve Clinical
Competence
Assess:
Slide 5
What is Clinical Competence?
“Integration of knowledge, skills and
attitudes.”
Rice et al., Competency-based objectives for clinical training.
Can. J. Med. Technol. 57:136, 1995
Slide 6
What is Clinical Competence?
“The habitual and judicious use of
communication, knowledge,
technical skills, clinical reasoning,
emotions, values, and reflection in
daily practice for the benefit of the
individual and community being
served.”
Epstein and Hundert, Designing and assessing professional
competence. JAMA 287:226, 2002
Slide 7
What is Clinical Competence?
Miller’s Pyramid of Competence
Does
Shows how
Knows how
Knows
Miller, The assessment of clinical skills/competencies/performance. Acad.
Med. 65:563-567, 1990.
Slide 8
What is Clinical Competence?
“Competence . . . means being able
to function in context.”
Chambers and Gerrow, Manual for Developing and Formatting
Competency Statements. J. Dental Educ. 58:361, 1994.
Slide 9
How is Clinical Competence
Assessed?
“Competence is concerned with what
people can do . . ., that is, potential.”
“Performance is concerned with what
people do . . ., that is, what is actually done
in the real-life context.”
Competence is inferred from performance!
While, Competence versus performance: which is more important?
J. Adv. Nurs. 20:525, 1994.
Slide 10
What is Authentic Performance
Assessment?
 Performance can be simulated or authentic

Simulated performance assessment uses an
artificial context:
SPs, OSCEs, Long Case - Standardized or
Unstandardized

Authentic performance assessment uses a
real context:
Direct observation of a resident caring for a real
patient with a real problem in a real setting –
Unstandardized!
Slide 11
Why Use Authentic Performance?
“Testing should be as close as possible to
the situation in which one attacks the
problem.”
“Ill-structured problems are not found in
simulated and/or standardized tests.”
“The variation inherent in professional
practice will always elude capture by a
set of rules.”
Wiggins, Assessing Student Performance: Exploring the Purpose
and Limits of Testing, Jossey-Bass, Inc. 1993
Slide 12
How is it Done?
“There are few validated strategies to
assess actual clinical practice.”
- Ronald Epstein
“Assessment at the apex of Miller’s
pyramid, the does, is the international
challenge of the century for all involved
in clinical competence testing.”
- Val Wass
Slide 13
Design Step 1 – Select Competencies

Breakdown ACGME’s six global competencies
into specific competencies

Technical skills, case management skills,
clinical decision-making skills, etc.

Avoid too broad

Avoid too specific

Appropriate for the resident’s training level

Sequenced appropriately over the training
period
Slide 14
Design Step 1 – Select Competencies

Write a competency statement:

Chambers and Gerrow, Manual for
developing and formatting competency
statements. J. Dental Educ. 58:361, 1994.

Verb
Direct Object
Qualifying conditions


Slide 15
Example Competency Statement
Induce anesthesia for an ASA I-II
patient by the end of the three
month period
Slide 16
Example Competency Statements

The Global Communication Skills Competency
Communicate accurately, efficiently and supportively
with the patient, patient’s family and the health care
team

Specific Communication Skill Competency
Build a Relationship (Rapport Building)

Component Skills used to Build a Relationship
Establish initial rapport
Empathy
Reflection
Respect
Support
Partnership
Legitimation
Apology
Slide 17
The Performance Rating Process
Specific performances are judged by expert
rater(s) who synthesize multiple impressions in
comparison to criteria given in a rating task and
filtered through the experience of the rater.
Specific performance(s)
Judgment
Expert
Criteria (Criterion referencing)
Chambers, A primer on competency-based evaluation. J. Dent.
Educ. 61:651, 1997.
Slide 18
Design Step 2 – Select Sampling

Which performances or tasks will be
observed?

How many observations?

When?
Slide 19
Design Step 3 – Select Raters
Raters are qualified individuals who have an
opportunity to observe and draw conclusions about
residents – they judge the presence and the quality
of the competency to be assessed
How many raters
Attendings
Other professional staff
Patients
Self
Slide 20
Design Step 4 – Select Rating Criteria
Rating criteria can either be checklists or
rubrics
 Checklists





Presence or absence of a competency or of
the components of a competency
Do not judge quality
May be appropriate for technical procedures
or beginning level competencies
May not capture sophistication of complex
performances:
Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and
Hanson, M. 1999. OSCE checklists do not capture increasing
levels of expertise. Acad. Med. 74:1129-1134.
Slide 21
Example Checklist
Competency:
Built a relationship?
Specific Skills:
Established initial rapport?
Employed Legitimation?
Provided support?
Demonstrated empathy?
Demonstrated legitimation?
Demonstrated apology?
Showed respect?
Yes
No
















Slide 22
Design Step 4 – Select Rating Criteria

Rubrics





Guidelines, rules, or principles by which
performances are judged
Reflect best thinking as to what constitutes
a good performance
Contain multiple quality levels
All rating levels must be written-down,
defined and described
Best rubrics contain anchor(s) to illustrate
the different points on the quality scale
Arter & McTighe, Scoring Rubrics in the Classroom, Corwin Press,
Inc. 2001
Slide 23
Example Rubric
Excellent – Smooth and efficient. Able to use
knowledge, judgment and skills to adjust
management appropriately to the specific patient
and operative procedure.
Competent – Lacks smoothness and efficiency but is
able to use knowledge, judgment and skills to adjust
management appropriately to the specific patient
and operative procedure.
Beginner – Lacks smoothness and efficiency. Able to
manage the case but exhibits limited use of personal
judgment and responsiveness to the specifics of the
patient and operative procedure. Requires some
limited coaching or attending intervention.
Novice – Can only manage the case with extensive
coaching and attending intervention.
Slide 24
What are the Desirable Characteristics
of Performance Assessment?
Face validity
Feasible
Low Cost
Non-reactive
Generalizable
Valid
Reliable
Slide 25
Psychometric Issues

Write good rubrics

Train the raters
– Only need 1 rater/performance

Sample over a broad array of cases/patients

Our intercase agreements range from 0.2 – 0.9

Wass et al., Generalizability in range of the OSCE

ACGME preliminary data shows adequate
generalizability, validity and reliability
Slide 26
Authentic Performances vs. Authentic
Products
Assessing an:
Authentic Performance
versus the
Product of an Authentic Performance
Authentic Performances
Patient Care
Professionalism
Communication Skills
Authentic Products
Patient Care
Ethics
PBLI
Systems-Based
Slide 27
Why Focus on Authentic Performance
Assessment?


ACGME outcomes mandate
Current assessment is rudimentary

Accountability
Murray et al., The accountability of clinical education: its definition
and assessment. Med. Educ. 34:871, 2000
Bordage et al., Education in ambulatory settings: Developing valid
measures of educational outcomes, and other research priorities.
Acad. Med. 73:743, 1998