Assessing operative
autonomy
Combining theory and software to make
evaluation easy
Jonathan Fryer MD,
Professor of Surgery,
Feinberg School of Medicine, Northwestern University
Disclosures
• I have made no financial gains from this project
• I may in the future
• I intend to continue work on this project regardless
2
What is the most essential goal of surgical
training?
3
Operative Autonomy
• The ability to
independently
perform
operations
safely and
effectively.
4
The Problem
There is growing concern that graduating
surgical residents are not achieving operative
autonomy with essential procedures.
1. Bell RH. Why Johnny cannot operate. Surgery 146, 533–542 (2009).
2. Mattar SG et al. General Surgery Residency Inadequately Prepares Trainees for Fellowship:
Results of a Survey of Fellowship Program Directors. Annals of Surgery September 2013 258,
440–449 (2013).
3. Coleman JJ et al. Early Subspecialization and Perceived Competence in Surgical Training: Are
Residents Ready? Journal of the American College of Surgeons 216, 764–771 (2013).
4. Chen P. Are Today’s New Surgeons Unprepared? Well (2013). at
http://well.blogs.nytimes.com/2013/12/12/are-todays-new-surgeons-unprepared
5
The Problem
• To be able to fix it……
You have to be able to
measure it.
6
The Problem
• We don’t do a very
good job of
assessing residents
in the OR.
7
The Problem
– Currently, summative
assessment of OR
performance is based on:
• # of cases logged by resident
– Role of resident in each case?
• Semi-annual global evaluations
– Memory decay?
8
The Problem
…asking busy surgical
faculty to fill out complex
assessment forms in a
timely manner, doesn’t
work.
9
The Solution
A simple assessment tool that:
• Assesses operative autonomy
• Doesn’t impede surgical workflow
• Facilitates high compliance and
prompt completion
Theoretical Framework
• Inter-related constructs:
– Supervision, Guidance, Autonomy, Performance
• Faculty Supervision (oversight)
≠
• Faculty guidance (physical or verbal help)
•
•
1
Faculty Guidance
=
Resident Autonomy
• Resident Autonomy = ƒ (Resident performance)
•
11
The Solution
– With every case faculty:
• Provide resident supervision.
• Assess and document the
level of operative autonomy
achieved by the resident.
• Progressively reduce the level
of operative guidance they
provide to resident.
The “Zwisch” Scale
• 4 levels of operative
guidance
– Show & Tell
– Active Help
– Passive Help
– Supervision Only
DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing
Residents in the Operating Room. Journal of Surgical Education 70, 24–
30 (2013).
Our method: PASS
(Procedural Autonomy and Supervision System)
Today
Coming soon…
Study Design: Participants and Setting
• Department of general surgery
at a large academic hospital
• All teaching faculty underwent
formal frame-of-reference
training per published protocol1
• All general surgery residents
and trained faculty raters eligible
for inclusion
• IRB-approved
1George
17
et al, J. Surg. Educ. 2013; 70
Results: Feasibility
• A 1 hour rater training session is sufficient to
achieve reliable and accurate ratings1
• 92% response rate using PASS
1George,
B. C. et al. Duration of Faculty Training Needed to Ensure
Reliable OR Performance Ratings. J. Surg. Educ. 70, 703–708 (2013).
Results: PASS Sample (7 mos)
Number of Residents
31
By Year of Residency
19
Year 1
Year 2
Year 3
Year 4
Year 5
9
6
5
5
6
Number of Attendings
27
Number of Procedures
1490
Number of Types of
Procedures
127
Results: PASS Sample
Relative Case Complexity
Easiest 1/3
Middle 1/3
193 (13.0%)
895 (60.1%)
20
Hardest 1/3
402 (27.0%)
Results: Validity: Zwisch Levels by PGY
p=<.001
p=<.001
p=<.001
p=0.21
p-values for sequential
pair-wise Χ2 distributions
23.2%
21
Results: Validity: Zwisch Levels by Complexity
p=<.001
22
p=<.001
p-values for sequential
pair-wise Χ2 distributions
Results: Validity: Zwisch Level by Prior
Experience
p=<.001
23
p-values for sequential
pair-wise Χ2 distributions
Study Design: Data Collection
• Sample 2: Video Sample
– 8 procedures video recorded for additional review
(subset of PASS sample)
– Rated by operating faculty, in-person OR observer,
and video reviewer using Zwisch scale (blinded to other
scores)
– Rated by 2 additional video reviewers using other OR
assessment instruments (modified OPRS and OSCORE)
24
Results: Video Sample
Number of Residents
Number of Attendings
Number of Procedures
Number of Types of Procedures
2 Laparoscopic cholecystectomy
2 Open inguinal hernia repair
2 Parathyroidectomy
1 Total thyroidectomy
1 Laparoscopic ventral hernia repair
25
4 (PGY 2 to 5)
2
8
5
Results: Reliability
• Inter-rater reliability
– Zwisch ratings
– Operating attending, OR observer, and video rater
– ICC = .90, 95% CI = .72 - .98, p < .001.
26
Results: Validity: Zwisch Level correlation with
other OR assessment tools
Item
Operative Performance Rating System (OPRS)
Degree of prompting or direction
Instrument handling
Respect for tissue
Time and motion
Operation flow
Overall performance
Ottawa Surgical Competency OR Eval. (O-SCORE)
Knowledge of procedural steps
Technical performance
Visuospatial skills
Efficiency and flow
Communication
27
ρ
p-value
-.92
.94
.94
.94
.95
.95
.001
.005
.005
<.001
<.001
<.001
.94
.93
.92
.86
.92
<.001
.001
.001
.007
.001
Benefits
•
Faculty and residents constantly reminded of ultimate goal ….
i.e. operative autonomy.
•
Establishes a conceptual framework for teaching and learning in
the OR.
•
Data can be used to:
– Help faculty and residents to set learning goals.
– Help programs monitor operative progress and identify those
who may need additional attention.
– Address regulatory requirements for OR supervision and
operative performance assessment.
– Establish national norms
Limitations
• So far, studied only at a single institution
• Validity analysis based on small convenience sample
• Raters not blinded to resident PGY level
• Comparison with only selected items of OPRS and O-
SCORE
• Unmeasured confounders (time of day, supervising surgeon
experience, etc)
29
Conclusion
• The Zwisch rating scale is a reliable and valid measure
of faculty guidance and resident autonomy
• Deployed on PASS the Zwisch scale can be used to
feasibly record evaluations for the vast majority of
operations performed by residents
30
Vision
• All surgical subspecialties.
• Other procedural specialties.
• Other medical professionals who need to learn to perform
complex clinical tasks.
• Other trades or professions where trainees need to learn
to independently perform complex tasks safely and
effectively.
31
Acknowledgements
Surgical Education Research & Development Team
Jonathan
Fryer
Debra DaRosa
Ezra
Teitelbaum
Shari
Meyerson
Eric Hungness
Mary Schuller
Research supported by:
Excellence in Academic Medicine Program
from the State of Illinois
Augusta Webster Educational Innovation
Grant from the Northwestern University
Center for Education in Medicine
Jay
Zwischenberger
Brian George
Theoretical basis
• Global assessment of performance is simpler,
more accurate, and more reliable than
checklists1
• Faculty guidance is related to resident
performance2
• Faculty can accurately and reliably rate the
amount of guidance provided to residents3
1.
2.
3.
Regehr, G., MacRae, H., Reznick, R. K. & Szalay, D. Comparing the psychometric properties of checklists and
global rating scales for assessing performance on an OSCE-format examination. Acad Med 73, 993–997
(1998).
Chen, X. (Phoenix), Williams, R. G., Sanfey, H. A. & Dunnington, G. L. How do supervising surgeons evaluate
guidance provided in the operating room? The American Journal of Surgery 203, 44–48 (2012).
George, B., Teitelbaum, E., DaRosa, D., Hungness, E., Meyerson, S., Fryer, J., Schuller, M., Zwischenberger, J.
Duration of Faculty Training Needed to Ensure Reliable O.R. Performance Ratings. Journal of Surgical
Education 70(6), 703-708 (2013).
Study
• Over 7 months
• 1490 evaluations
• 27 faculty
• 31 residents
Study Design: Rating Scales
• Zwisch
• Procedural Complexity
• Operative Performance Rating System (OPRS)1
– 6 general items only--excludes items that pertain only
to specific procedures
• Ottawa Surgical Competency Operating Room Evaluation
(O-SCORE)2
– 5 intra-operative items only--excludes items that did not
pertain to intra-operative performance.
1Chen
et al, The American Journal of Surgery 2012; 203
2Gofton et al, Acad. Med. 2012; 87
35
Results: Validity
• Convergent Validity for Guidance/Autonomy and Resident
Performance
– Zwisch level vs. PGY
– Zwisch level vs. Complexity
– Zwisch level vs. Resident Experience
• Construct Validity for Guidance/Autonomy
– Zwisch level vs. OPRS guidance item
• Construct Validity for Resident Performance
– Zwisch level vs. OPRS performance items
– Zwisch level vs. O-SCORE performance items
36
The Team
• Dr. Debra DaRosa
• Dr. Brian George
• Dr. Shari Meyerson
• Dr. Ezra Teitelbaum
• Mary Schuller
• Dr. Nathaniel Soper
• Dr. Joseph Zwischenberger
38
Impact so far
• Over 1000 evaluations collected in 6 months
• Response rate > 90%
• Changes in teaching
• They love to use it!
Next steps
• Dictation of feedback
• Reports
Results: Validity
• Convergent Validity for Guidance/Autonomy and Resident
Performance
– Zwisch level vs. PGY
– Zwisch level vs. Complexity
– Zwisch level vs. Resident Experience
• Construct Validity for Guidance/Autonomy
– Zwisch level vs. OPRS guidance item
• Construct Validity for Resident Performance
– Zwisch level vs. OPRS performance items
– Zwisch level vs. O-SCORE performance items
41
Theoretical Framework
Faculty Guidance ≠ Faculty Supervision
Helping
42
Watching
Next Steps
• I am actively trying to bring this to MGH
• It needs additional development before it can be launched
here
• Multiple other departments have already committed to
supporting this project
43
Questions?
44
Results
80
40
0
Number of procedures
Supervision Levels for PGY5 Residents
1
2
3
Residents
4
5
Results
50% = 60 procedures
Current Status
Milestone Achieved
Cost / time
Development of v1.0 mobile app
Development of v0.9
administrative interface (beta)
Integration with Northwestern
EMR
Development of v2.0 iOS app
$200,000 / 8 months
$75,000 / 3 months
Total
$480,000 + operational
expenses
$45,000 / 2 months
$160,000 / 7 months
(ongoing)
Road Map
Planned Technical Milestones
Target launch date
v2.0 for iOS at Northwestern
v1.0 Administrative interface at
Northwestern
v2.0 for Android at Northwestern
February 2014
April 2014
System integration at MGH
v2.0 iOS at MGH
V2.0 Android at MGH
June 2014
June 2014
July 2014
October 2014
12 month budget
Expense Item
Cost
Design and specification
$30,000
Software Development
$225,000-$300,000
QA testing
$30,000
Server hosting and maintenance
$25,000
User training
$5,000
Administrative
$30,000
Total
$345,000 - $420,000
The “Zwisch” Scale
• 4 levels of guidance
– Show & Tell
– Active Help
– Passive Help
– Supervision Only
DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing
Residents in the Operating Room. Journal of Surgical Education 70, 24–30
(2013).
Theoretical Framework
11
Resident
FacultyAutonomy
Guidance ∝
Faculty
ResidentGuidance
Autonomy
• Stritter FT et al., Handbook for the academic physician. 1986.
• Chen et al., The American Journal of Surgery 2012; 203
• Gofton et al., Acad. Med. 2012; 87
52