Intra-Operative Assessment of Technical Skill on Live
Patients Using Economy of Hand Motion:
Establishing Learning Curves of Surgical Competence
Matthew Roberts, Vanessa Bacal, Mohammed Mahdi, Ethan D. Grober
Mount Sinai & Women’s College Hospital, Division of Urology, Department of Surgery,
University of Toronto
Technical Skill
Acquired
Objectively, reliably assessed
OSATS: Objective Structured
Assessment of Technical Skill

Discriminate novice & experienced surgeons

Improvement with training over time

Skills sets transfer to higher fidelity simulations
Intra-Operative Assessment of Technical Skill
- Real Patients, Real Operating Rooms Subjective, Unvalidated
Intra-Operative Assessment of Technical Skill
- Real Patients, Real Operating Rooms Subjective, Unvalidated
?
Variable patients
Variable operations
Variable instruments & technology
OR team changes
Resource & time pressures
Measurement instruments
Hand-Motion Analysis
 Electro-magnetic
measurement of hand position
& orientation
 # of hand movements
 hand travel distance
 hand speed
Hand-Motion Analysis
 In
the surgical skills lab: Objective, reliable and valid
measure of surgical competence on open, laparoscopic,
endoscopic and microsurgical procedures
 Correlates
well with traditional lab-based measures of
surgical competence – global ratings and checklists scores
by trained experts
 Pilot
work and feasibility studies have been performed in
real operating rooms
Hand-Motion Analysis
Hand-Motion Analysis
Study Objectives
1. Establish the feasibility of performing live
intra-operative hand-motion analysis while
operating on real patients
2. Validate live, intra-operative assessments of
economy of hand motion as an objective
measure of technical skill
3. Use hand-motion analysis to establish
competency-based surgical learning curves
based on standards established by
performance of experienced surgeons
Experienced
Methods
Microsurgery-Vasectomy reversal
2 Surgeons
2 Standardized Surgical
Tasks
Vasectomy
Novice
Hand motion analysis
Video recorded
Methods – Video Analysis
2 blinded, experienced surgeons
 Global rating scores
 Checklists scores
 Final product scores
Methods – Video Analysis
Methods – Video Analysis
Methods – Data Analysis
Experienced
Novice
Hand motion data and blinded expert ratings
of video-based surgical performance were:
 Graphically
 Correlated
compared over time
using Pearson calculations
Total Hand Movements
350
300
250
200
Novice Surgeon
# Movements
150
100
Experienced Surgeon
50
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
700
600
Novice Surgeon
500
400
300
Experienced Surgeon
200
100
0
1
2
3
4
Case #
5
6
17
Hand Travel Distances
35.0
30.0
Hand Travel Distance (m)
25.0
Novice Surgeon
20.0
15.0
10.0
Experienced Surgeon
5.0
0.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
50
45
40
Novice Surgeon
35
30
25
Experienced Surgeon
20
15
10
5
0
1
2
3
Case #
4
5
6
17
Global Rating Scores
25
23
Experienced Surgeon
21
19
17
Global Rating Score
15
13
Novice Surgeon
11
9
7
5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
35
Experienced Surgeon
30
25
Novice Surgeon
20
15
10
5
0
1
2
3
Case #
4
5
6
17
Checklist Scores
22
Experienced Surgeon
20
18
16
Checklist Score
Novice Surgeon
14
12
10
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
12
Experienced Surgeon
10
Novice Surgeon
8
6
4
2
0
1
2
3
Case #
4
5
6
17
Final Product Scores
5
Experienced Surgeon
4
3
Final Product Score
Novice Surgeon
2
1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
5
Experienced Surgeon
4
Novice Surgeon
3
2
1
0
1
2
3
Case #
4
5
6
17
Correlating Hand Motion & Video Performance
# Hand
movements
Hand Travel
Distance
Time
r = -0.83
-0.85
-0.86
Checklist Score
-0.78
-0.81
-0.86
Final Product
Score
-0.82
-0.87
-0.83
# Hand
movements
Hand Travel
Distance
Time
Global Rating
Score
-0.94
-0.95
-0.92
Checklist Score
-0.88
-0.83
-0.78
Final Product
Score
-0.90
-0.85
-0.70
Global Rating
Score
Pearson correlation significant < 0.05 level (2-tailed)
BAD
BETTER
Hand-Motion Analysis:
Feasibility
• Set up < 3mins
• Data obtained in 23/25 cases
• Surgical team easily trained
• HMA software and hardware $3500
Conclusions
1. Establish the feasibility of performing live
intra-operative hand-motion analysis while
operating on real patients
2. Validate live, intra-operative assessments of
economy of hand motion as an objective
measure of technical skill
3. Use hand-motion analysis to establish
competency-based surgical learning curves
based on standards established by
performance of experienced surgeons
Utility of Hand Motion Analysis
350
1) Measure & track intra-operative
surgical performance over time
300
# Movements
250
2) Tool for immediate feedback to
surgical trainees and training
programs
200
150
Novice Surgeon
100
Experienced Surgeon
50
0
1
2
3
Case #
4
5
6
7
8
9
10
11
12
13
3) To establish that different
procedures competency targets.
HMA can be used to identify the ideal
or minimum number of exposures
required for technical competence
14
15
15
16
17
Economy of Hand Motion
- Future Directions Further validate this assessment technology:
 More surgeons - with various levels of skill &
experience
 Different types surgeries - basic/complex,
open/laparoscopic/endoscopic
Intra-Operative Assessment of Technical Skill
- Real Patients, Real Operating Rooms -
Intra-Operative Assessment of Technical Skill on Live
Patients Using Economy of Hand Motion:
Establishing Learning Curves of Surgical Competence
Matthew Roberts, Vanessa Bacal, Mohammen Mahdi, Ethan D. Grober
Mount Sinai & Women’s College Hospital, Division of Urology, Department of Surgery,
University of Toronto