Comparison: Traditional vs.
Outcome Project Evaluative
Processes
Craig McClure, MD
Educational Outcomes Service Group
University of Arizona
December 2004
Current Problem
• Increasing public concerns with quality
and safety.
• Variable patterns of care that are not
based on medical science.
• Poor quality of interpersonal “service.”
• Public encounters difficulty in assessing
physician competence (initial and
continuing ) and judging quality.
The ACGME Mission
To improve the quality of health
care in the United States by
ensuring and improving the
quality of graduate medical
educational experiences for
physicians in training.
Problem Plus Mission
• ACGME responded to the challenge by
changing focus to:
– How well do we learn what is being taught
– How well do we practice what we learn?
A new way of thinking
How to change the educational
and accreditation system from…
Structure &
process
Competency
Program Goal
•
OLD: goal was for • NEW: the Program
the Program to
Director must
comply with the
determine if
written RRC
residents achieve
Requirements
the learning
objectives set by
the Program.
Six Domains
•
•
•
•
•
Medical Knowledge
Patient Care
Professionalism
Communication and Interpersonal
Practice Based Learning and
Improvement
• Systems Based Practice
Purpose of Assessment
1. Assess residents' attainment of
competency-based objectives
2. Facilitate continuous improvement
of the educational experience
3. Facilitate continuous improvement
of resident performance
4. Facilitate continuous improvement
of residency program performance
Whatever we measure we tend to
improve.
David C. Leach, M.D.
Executive Director
ACGME
September 12, 2002
Characteristics of good
assessment
• Measures actual performance
• Identifies areas for improvement
• Satisfies reasonable request for
accountability
• Is practical
• Is done over time to discern growth
Types of Evaluation
• Formative
• Improve performance
• Summative
• Note achievement
Both types of evaluation can be used to
evaluate either an individual or a program.
Characteristics of good assessment
•
•
•
•
•
Systematic
Dependable
Comprehensive
Congruent
Practical
Characteristics of good assessment
(continued)
• Makes professional practice more
transparent
• Deconstructs the role of physician
• Clarifies levels of expertise by
distinguishing functional levels
Characteristics of good assessment
(continued)
• Measures actual performance
• Identifies areas for improvement, i.e.,
self, others
• Satisfies reasonable requests for
accountability
Traditional Evaluation
1.Global
2.End of rotation
3.Subjective
1.Anchored to norms seen by attending
(therefore variable)
2.“I like/didn’t like the resident”
4.Focused on rotation goals (not
movement toward competency)
Outcome Based Evaluation
1. Formative, focused on specific
competencies required for a physician
2. Measure the full scope of professional
characteristics from very specific
procedures to skills involving a synthesis
of component abilities
3. Specific evaluative techniques chosen to
match the skill being assessed
Assessment Tools
(The Toolbox)
• 360° Evaluation Instrument
• Chart Stimulated Recall Oral Exam
(CSR)
• Checklist Evaluation of Live or
Recorded Performance
• Objective Structured Clinical Exam
(OSCE)
• Procedure, Operative or Case Logs
The Toolbox (continued)
•
•
•
•
•
•
•
Patient Surveys
Portfolios
Record Review
Simulations and Models
Standardized Oral Exams
Standardized Patients (SP)
Written Exams (MCQ)
Evaluation Method
•
OLD: global
checklist format
• NEW: Type of
evaluation chosen
specifically to
measure the
chosen skill drawn
from the 6 domains
Frequency of Evaluation
• OLD: once per
rotation
• NEW: multiple
intervals
assessing
component
behaviors as well
as the integrated
practice of
medicine.
Timing of Assessment
• OLD: End of
rotation
• NEW: Timing
chosen to facilitate
evaluation of a
specific
competency
Anchors for Evaluation
•
OLD: Most
frequently the
preceptor
evaluated the
resident against
the norm of
previous
residents in that
experience
•
NEW: Criteria
defining
competence are
utilized as the
standard against
which resident
performance is
measured
Target of Evaluation
•
OLD: at best
tended to
address the
resident’s
success at the
goals for the
rotation
•
NEW:Criteria for
evaluation
describe the
qualities of the
competent
physician, so are
more wide
ranging or more
specific
Number of Evaluators
• OLD: typically one
per rotation
• NEW: multiple,
both physician and
non-physician
evaluators
Other Outcome Characteristics
•
•
•
Authentic
More Individualized
Reflection and Self-knowledge
Critical
“Authentic”
• Justification for elements included in
the curriculum is that competence as
a practicing physician requires that
skill, knowledge or attitude
• Evaluation is of the actual skill,
knowledge or attitude used by
practicing physicians
More Individualized
• A principle of a criteria-driven physician
curriculum is that everyone can become
competent with sufficient exposure
• Residents obtain skills at different rates
with requirements for disparate learning
experiences
• An optimal outcome-driven system would
have an intake assessment followed by an
individualized program of study
Reflection and Self-knowledge
Critical
• Criteria for competence are provided
to the learner
• Impetus for improvement arises from
desire to narrow the gap between
criteria and performance
• Accurate self-assessment is
essential to the resident gauging
personal performance
In Summary
Traditional method:
Not systematic
Subjective & Normative based
Global evaluations @ rotation end
Outcomes-based:
Systemic and comprehensive
Based on criteria defining competence
Multiple measures and intervals