Optimal Assessment for Skill
Improvement in Surgery (OASIS)
A Strategic Model
Howard W. Francis, M.D., MBA
Associate Professor
Dept. Otolaryngology-Head and Neck Surgery
and
The Dr. Mohan Swami Institute of
International Medical Education
Objectives
 Trends in Surgical Education
 Identified Challenges: Needs
 Strategic framework: OASIS
 Innovative Solution: LCMA
Foundations
Apprenticeship in Surgical Training
• Declarative Knowledge
• Procedural Knowledge
• Systems Knowledge
The Miller Triangle (1990)
Impetus for Change
Johns Hopkins
Otolaryngology-Head &
Neck Surgery
Objective Assessment
and
Outcomes-based Surgical
Education
.
Johns Hopkins Otolaryngology Competency Assessment Tools
Otology
Mastoidectomy
Tympanoplasty
Head and Neck
Surgery
Sinus/Plastics
Pediatric/General
Parotidectomy
Endoscopic
Sinus Surgery
I&D Peritonsillar
Abscess
Thyroidectomy
Rhinoplasty
Tracheostomy
At Johns Hopkins, we have already developed and
implemented competency assessment tools for
evaluation of commonly performed otolaryngology
procedures
Neck Node
Excision
Pediatric
Tracheostomy
Rigid
Bronchoscopy
Submandibular
Gland Excision
Pilot-Testing of 2 Assessment Tools
• Assessment tools for
Mastoidectomy and
Endoscopic Sinus
Surgery (ESS) were
pilot-tested for
feasibility, validity,
and reliability over 3
years during annual
dissection courses.
Objective Structured Assessment of Technical Skills
(OSATS)
Mastoidectomy Task-Based Checklist
Laeeq et al, 2009; Francis et al, 2010
Skill Development Curves
Francis et al, 2010
Milestones of Procedural Development
Mastoidectomy
Francis et al, 2010
Citations
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
1: Laeeq K, Waseem R, Weatherly RA, Reh DD, Lin SY, Lane AP, Ishii M, Cummings CW, Bhatti NI. In-training assessment and predictors of
competency in endoscopic sinus surgery. Laryngoscope. 2010 Dec;120(12):2540-5. PubMed PMID: 21082748.
2: Ishman SL, Brown DJ, Boss EF, Skinner ML, Tunkel DE, Stavinoha R, Lin SY. Development and pilot testing of an operative competency assessment
tool for pediatric direct laryngoscopy and rigid bronchoscopy. Laryngoscope. 2010 Nov;120(11):2294-300. PubMed PMID: 20939072.
3: Stewart CM, Masood H, Pandian V, Laeeq K, Akst L, Francis HW, Bhatti NI. Development and pilot testing of an objective structured clinical
examination (OSCE) on hoarseness. Laryngoscope. 2010 Nov;120(11):2177-82. PubMed PMID: 20824785.
4: Laeeq K, Pandian V, Skinner M, Masood H, Stewart CM, Weatherly R, Cummings CW, Bhatti NI. Learning curve for competency in flexible
laryngoscopy. Laryngoscope. 2010 Oct;120(10):1950-3. PubMed PMID: 20824777.
5: Lonze BE, Parsikia A, Feyssa EL, Khanmoradi K, Araya VR, Zaki RF, Segev DL, Ortiz JA. Operative start times and complications after liver
transplantation. Am J Transplant. 2010 Aug;10(8):1842-9. PubMed PMID: 20659090.
6: LaPorte DM, Marker DR, Seyler TM, Mont MA, Frassica FJ. Educational resources for the Orthopedic In-Training Examination. J Surg Educ. 2010
May-Jun;67(3):135-8. PubMed PMID: 20630421.
7: Francis HW, Masood H, Laeeq K, Bhatti NI. Defining milestones toward competency in mastoidectomy using a skills assessment paradigm.
Laryngoscope. 2010 Jul;120(7):1417-21. PubMed PMID: 20578231.
8: Ro K, Cameron JL, Yeh MW. The Hunterian Laboratory of Experimental Medicine. Ann Surg. 2011 May;253(5):1042-8. PubMed PMID: 21451394.
9: Francis HW, Masood H, Chaudhry KN, Laeeq K, Carey JP, Della Santina CC, Limb CJ, Niparko JK, Bhatti NI. Objective assessment of
mastoidectomy skills in the operating room. Otol Neurotol. 2010 Jul;31(5):759-65. PubMed PMID: 20517169.
10: Laeeq K, Infusino S, Lin SY, Reh DD, Ishii M, Kim J, Lane AP, Bhatti NI. Video-based assessment of operative competency in endoscopic sinus
surgery. Am J Rhinol Allergy. 2010 May;24(3):234-7. Epub 2009 Dec 16. PubMed PMID: 20021742.
11: Basafa E, Sheikholeslami M, Mirbagheri A, Farahmand F, Vossoughi GR. Design and implementation of series elastic actuators for a haptic
laparoscopic device. Conf Proc IEEE Eng Med Biol Soc. 2009;2009:6054-7. PubMed PMID: 19963665.
12: Galvagno SM Jr, Segal BS. Critical action procedures testing: a novel method for test-enhanced learning. Med Educ. 2009 Dec;43(12):1182-7.
PubMed PMID: 19930509.
13: Cummings CW. The legacy and obligations of the head and neck surgeon: the2009 Hayes Martin Lecture. Arch Otolaryngol Head Neck Surg. 2009
Nov;135(11):1077-81. PubMed PMID: 19917917.
14: Laeeq K, Bhatti NI, Carey JP, Della Santina CC, Limb CJ, Niparko JK, Minor LB, Francis HW. Pilot testing of an assessment tool for competency in
mastoidectomy. Laryngoscope. 2009 Dec;119(12):2402-10. PubMed PMID: 19885831.
15: Berkow LC, Greenberg RS, Kan KH, Colantuoni E, Mark LJ, Flint PW, Corridore M, Bhatti N, Heitmiller ES. Need for emergency surgical airway
reduced by a comprehensive difficult airway program. Anesth Analg. 2009 Dec;109(6):1860-9. Epub 2009 Aug 27. PubMed PMID: 19713264.
16: Haut ER, Chang DC, Hayanga AJ, Efron DT, Haider AH, Cornwell EE 3rd. Surgeon- and system-based influences on trauma mortality. Arch Surg.
2009 Aug;144(8):759-64. PubMed PMID: 19687381.
17: Lin SY, Laeeq K, Ishii M, Kim J, Lane AP, Reh D, Bhatti NI. Development and pilot-testing of a feasible, reliable, and valid operative competency
assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. 2009 May-Jun;23(3):354-9. PubMed PMID: 19490815.
18: Pierorazio PM, Allaf ME. Minimally invasive surgical training: challenges and solutions. Urol Oncol. 2009 Mar-Apr;27(2):208-13. PubMed PMID:
19285236.
19: Shippey S, Handa VL, Chen TL, Chou B, Bowen CW. Validation of an instrument for evaluation of subcuticular suturing using a plastic tissue model.
J Surg Educ. 2009 Jan-Feb;66(1):31-4. PubMed PMID: 19215895.
20: Reiley CE, Hager GD. Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Med Image Comput Comput Assist Interv.
2009;12(Pt 1):435-42. PubMed PMID: 20426017.
Major Benefits of Skills Assessment
• Formative feedback
– Benefits to resident
– Benefits to faculty
• Refinement of resident training
– Guided/mentored learning and practice
– Early remediation
• Program development
The JH Swami Institute of International
Medical Education (SIIME)
Resource Center for Surgical Skills
Evaluation and Training
SIIME
Dr. David Nichols (Director)
Dr. Howard Francis
Dr. EunMi Park (Ed.D.)
Elizabeth Goodspeed
Christopher Matroniano
JHI
Sanford Wu
Charles Kim
PUGSOM
Dr. Charles Wiener and faculty
Needs Assessment
 Challenges in Five Dimensions
 Based on literature review and expert opinion
 Exploratory & confirmatory interviews with surgical
leadership in Malaysia (June ~ October 2011)
 Ongoing with visiting international scholars at JHM
Challenge-1. Resident
 Ready for
independent practice?
Surgery, 146(4):533, 2009
Concerns:
• Operative skill is learned, not innate
• Current level of operative experience
inadequate
• Teaching and learning in the OR is poorly
understood and assessed
Challenge-2. Faculty
 No formal preparation for
teaching
 Too busy for effective
instruction
 Retention of talented
faculty in academics
• Faculty members require basic training in
educational theory and teaching skills
• Longer term interventions, e.g. seminar
series, longitudinal programs, fellowships
– sustainable change in learning, behavior and
organizational culture
Challenge-3. Healthcare SectorHorizontal
 Variability in standards of patient care
and patient safety within and between
institutions and countries
BMJ, 2012
• Differences found between countries
• Greater differences between units than between
hospitals
Challenge-4. Healthcare SectorVertical
 Limited opportunities to advance
surgical skills locally- e.g. training
abroad needed
 Limited local opportunity for
leadership development and
innovation
Challenge-5. Global
 Needed emphasis
on public health
dimensions of
surgical disease
Challenge-5. Global
• Surgical Disease: 11% of global Disability-Adjusted
Life-Years (DALYs) [Farmer et al, 2008; Debas et al,
2006]
• 2015 millennium development goals (MDGs) attention to surgical capacity through improvements
in infrastructure and training [Kushner et al, 2010;
PLoS Editorw, 2008]
• WHO’s Global Initiative in Emergency and Essential
Surgical Care [WHO, 2010]
OASIS Framework
(Optimal Assessment for Skill Improvement in Surgery)
Facilitates the development of a learning environment
that promotes continuous improvement of operative
skills within a culture of patient safety and service
excellence
Residents: Promotion of life-long learning and skill improvement
Faculty: Development as teachers, education innovators
Professional Boards, MOE, MOH, public: System of accountability in
teaching, learning and outcomes
Outcomes
/ Impact
OASIS Roll-out
Impact
Ensuring
Standardization
1. Program Pilot
2. Resources
Expansion
5. Residents
In Network
6. Faculty
3. Full program
7. Clinics
4. Disciplines
8. Upstream
9. Financial
OASIS Faculty Development
• Participation: Train the trainer model/CME
-OASIS curriculum
-OASIS validated skills assessment tools
-OASIS validated methods
• Implementation: Longitudinal engagement
-Faculty engagement and training
-Integration into clinical work flow
-Quality control
• OASIS certification: OASIS Network
• Research: Best practices
OASIS Faculty Development
Engagement
Participation
/ Actions
• OASIS curriculum
• OASIS tools
• Integrated clinical
work flow
• Remediation
& Coaching
Certified
OASIS
Network
Members
R
E
S
E
A
R
C
H
OASIS
Resources
Surgery, 146(4):533, 2009
Recommendations:
• Make operative competency a requirement for board
certification
• Study and improve teaching in the OR
• Develop pedagogic scheme for teaching operations
• Develop valid assessment tools for resident
performance in the OR
• Increase use of simulations in surgical teaching
March, 2012
• Effective July 2012
• Education outcomes reported semiannually
• Innovation rewarded
New Foundational Endeavor at JHU
RAISE Team
(Research & Advancement in Surgical Education)
• A grass-roots leadership group:
Neurosurgery, Orthopedics, Otolaryngology,
Urology
• Addressing ACGME requirements for Next
Accreditation System
• Developed a new framework for residency
education
LCMA© Framework
Learner Centered Milestone Achievement
(LCMA): A Strategic Model for Measuring,
Interacting, Monitoring, and Reporting
Milestone Achievement Outcomes
EunMi Park, Ed.D.
Howard Francis, MD, MBA
Learner-Centered Teaching
Models
A
B
Learner-centered
Faculty-centered
A Staged Model of Development
Self-Directed
Severe
Learner
Mismatch
Mismatch
Near
match
Match
Involved
Learner
Mismatch
Near
match
Match
Near
match
Interested
Learner
Near
match
Match
Near
match
Mismatch
Match
Near
match
Mismatch
Severe
Mismatch
Authority,
Expert
Salesman,
Motivator
Team
Leader
Delegator
Dependent
Learner
Grow (1991)
A Model of Milestones Achievement
Efficiency /
Refinement
Error
Reduction
FACULTY ----- RESIDENT
Clarification
Identify Discuss Establish Assess
Knows
Shows
(Practice)
Does
LCMA Strategic Principle - 1
1. Milestones guide path to proficiency
 Transparent expectations
 Formative assessment guides progress
 Learning needs and style change along
milestone continuum
LCMA-1: A Model of Milestones
Intentional Assurance
For Patient Safety Culture
Operative Competence
Technical/
motor skills
competence
Cognitive
competence
Other competency domains:
- experience
- opportunity
- clinical judgment
- confidence
- communication skills
- professionalism
LCMA Strategic Principle - 2
2. LCMA is focused on bi-directional interactions

LCMA models progressive learner autonomy
with transition from faculty- to learner-initiated
exchange (modeled after the skillful teacher
and mentor)

Faculty - Learner interactions drive milestone
achievement through shared responsibility
LCMA-2. Interactive Responsibilities
Faculty
1
3
6
8
10
2
4
5
7
9
Resident
LCMA Strategic Principle - 3
3. LCMA focuses on systematic reflective
interactions

I: Inform the expected standard & current data

D: Discuss the learner’s next goal and timeline

E: Establish customized instructional
approaches

A: Assess next milestone achievement status
LCMA-3: Systematic Interactions
e.g. H1: Increased Efficiency with
Structured Intervention?
Thank you!
Howard W. Francis, M.D., MBA
EunMi Park, Ed.D.
The Johns Hopkins
Dr. Mohan Swami Institute for
International Medical Education