Measuring Outcomes for Residency
Graduates
Steven L. Frick, MD
Chairman, Dept. of Orthopaedic Surgery
Director of Medical Education, Nemours Children’s Hospital
Professor and Assistant Dean
University of Central Florida College of Medicine
Orlando, Florida

No financial disclosures.
AAOS Program Committee
POSNA Curriculum Committee
POSNA Residents Review
POSNA Treasurer, Board of Directors
JRGOS Board of Directors
ABOS QWTF
ACGME Milestones Project Workgroup

No competency specifically addressing surgical skills

Macy Foundation Report 2011
The Content and Format of
Graduate Medical Education
Recommendation III-A: The length of GME should be determined by an individual ’s readiness for independent practicedemonstrated by fulfillment of nationally endorsed, specialty-specific standards- rather than tied to a GME program of fixed duration.

“nationally endorsed, specialtyspecific standards ”
Do we have any of those?

Role for CORD
 Optimistic versus Pessimistic
 “an opportunity in every difficulty” versus “ a difficulty in every opportunity”

Evidence Based Medicine
Integrating individual clinical expertise with the best external clinical evidence
Outcome
A final product or end result

A. Flexner - 1910
Medicine can be learned but not taught
Active participation required
Need dedicated educators and students

“get comfortable with uncertainty ”

Role of professional education
 Provide practitioners the intellectual tools to assess information critically, stay abreast of changing knowledge, adapt to continuous change, and reflect on the larger role and responsibilities of the profession in society.
 From Time to Heal by Kenneth Ludmerer

“Is there a core body of knowledge and skills that the finishing resident should possess prior to starting practice or fellowship?
”
- Richard Gross, MD
Need curriculum and competency assessment

Residency fundamentally =
Master-Apprentice

William Halsted:
Residency Training system
Introduced in 1889 at Johns Hopkins based on:
 a fixed period of time for training,
 structured educational content,
 actual experience with patients,
 escalating responsibility for patient care during training, and a period of supervised practice after formal training.
 Remains the cornerstone of surgical training in North America more than a century later

Competency Based Education
 Defined by the outcome of the educational process, not the content
 Develop weighted curriculum to teach and assess (Farmer, Gross, Wadey)
 Assessing competence focuses on what the learner is able to do

How do you assess competency?
"the state or quality of being capable or competent; skill; ability."

Miller’s model of competence
Does
Shows how
Performance or “hands on”
Live Demo;
Multimedia
Knows how
Knows
Miller GE. The assessment of clinical skills/competence/performance.
Academic Medicine (Supplement) 1990; 65: S63-S7.
Read, Listen

OCAP

Dreyfus Model of Skill Acquisition

Mastery Learning Model
-Bloom 1968

Becoming Expert
“The 10,000 Hour Rule”
 About 10 yearsdedicated practice
 Bill Joy- UNIX, Sun
Microsystems;
Mozart; The Beatles;
Bobby Fischer;Bill
Gates

Model of complete clinical care
Opening
Engage
Empathize
Educate
Enlist
Find It
Fix It
Closing

Culturally Competent Care

The ability to understand and work with patients whose beliefs, values, and histories are significantly different from our own.

CCC Education
Medical/
Surgical
Team
Concerns
•Quality of Outcomes
•Patient-Physician
Relationship
•Malpractice Claims
Team
Harmony
& Quality of Work
Life
Error Prevention
Courtesy of A. White, III, MD

Defining / Teaching/ Modeling
Professionalism most important
Drs. Cruess body of work
Hidden curriculum
Social Contract
Individual Awareness

Teach and Model
Professionalism

Surgery- tripartite body of knowledge
Frank Wilson, MD
 Preoperative - evaluation, indications, planning
 Intraoperative - technical execution
 Postoperative - immobilization, weight-bearing, PT
 All 3 necessary for success

Ortho Surgical
Education
 Interns - pre and post operative care, framework of ortho fundamentals, closed management of fxs
 PGY 2/3 - basic decision-making and psychomotor skills
 PGY 4/5 - independent decision-making, subspecialty skills, integrate knowledge

Our Educational philosophy at CMC
 Not training
 Stimulus - Reaction vs
Stimulus Thought - Reaction
 Create one-on-one master-apprentice situations
 Graduated responsibility
 ALWAYS supervised in highest risk activities (OR)
 Have to spend enough time with them to know

How do you assess competency?
I DON ’T KNOW

Charlotte Competency Stages
 Stage I - do not know anything cannot do anything, and know it
 Stage II – know and can do a lot, but do not recognize what you do not know and cannot do DANGER
 Stage III – know and can do a great deal, but realize there is much you do not know

“The beginning of a mountaineer’s career, when energy and enthusiasm outpace experience and judgment, is said to be the most dangerous part.
”
Photo by Guillaume Dargaud

Setting Standards
 Job of Chair and RPD to set standards of excellence
 Graduates of program should meet these standards in all core competencies

Assessing competency
 Complete 5 years of orthopaedic surgery program under watchful eye of PD
 12 months PGY1 / internship, 48 months orthopaedic surgery
 Evaluations and comment by faculty, peer evaluations, portfolio (presentations, courses, outcomes instruments), OITE, operative experience log
 Consensus of PD and faculty

ABOS
I believe this individual is capable of the competent independent practice of orthopaedic surgery.
Steven L. Frick, MD
Residency Program Director

An Expert- Knows
 Knows WHAT to do
 Knows HOW to do it

Knows WHEN to do it (and when not to)
 Knows WHY to do it
 Knows WHEN to ask for help
 Knows WHAT we don ’t know

Is it possible/desirable to define and measure
and then graduate a resident before 5 years?

A Competency-Based Curriculum in Orthopaedic Surgery:
From Idea to Implementation
Markku Nousiainen, MS, MD, MEd, FRCS(C)
Sunnybrook Health Sciences Centre
University of Toronto

Current challenges in residency training
 reduction in work hours
 reduced time spent in OR teaching surgical skills
 reduction in wait times
 improvement in patient safety
} reduced training opportunities for residents

Competency-based education
“Training process that results in proven competency in the acquisition & application of skills & knowledge to medical practice that is not simply dependent on the student ’s length of training & clinical experiences”

“Much of what is counted does not count, and much of what counts cannot be counted.
”

Problems
 Toronto experience- 5 years, now all in for first time
 Still no defined “curriculum”
 More resource intensive than traditional pathway = costs more

Current environment of GME= very dependent on Medicare funding
 Some predict reduction in Medicare GME funding under PPACA 2010

How much of residency education is experiential?
Can we list / define everything you need to learn?
Can we transfer knowledge gained from experience without making residents have the experience?

Duty hours 2003
 First ortho class with 80 hour work weekdouble failure rate on part I ABOS certification exam
 Similar result 2011 exam takers
 Why?

Does this exam measure competency?
 Who do you want – 90%ile or 30%ile?

GME-Decade of Accountability
 To patients by residents, faculty
 Patient safety, Resident safety- RPD
 To residents by faculty, institution
 Societal demands for assurance of competency
 Safe, Effective, Patient centered, Timely,
Efficient, Equitable (IOM)
 Increased requirements by oversight organizations – RPD time
 Professional, ethical behavior demanded

NAS-
Next Accreditation System
 Coming to Ortho July 2013
 No more site visits, PIFs every 5 years
 Annual “Biopsy” of 4 things
– Institutional report
– Annual survey of residents and faculty
– Case logs
– Milestones (q 6 mos reports from
Competency Committee)
 Self report every 10 years

Ortho Milestones- 18 cover PC and MK
All have 5 levels
By graduation resident should be level 4
(competent) in all
For peds- septic arthritis and SC humerus fracture
Surrogates for knowledge in other areas

Milestone- Peds SCH Fracture

NAS www.acgme-nas.org
 Institutional reviews (q 18 mos)
 Milestones reports (q 6 mos)- form a competency committee
 Operative experience database
 Resident annual survey
 Faculty annual survey (new)
 ORTHO JULY 1, 2013

Future of
Orthopaedic Residency Education
 Change is coming
 Need to protect experience, in addition to more rigorous evaluation / oversight
 More evaluations / structured experiences
 Remember importance of graduated independence
 Milestones will be modified as we go
 NAS is on the way- BE AN OPTIMIST!

Thank You