Towards Better Understanding of Cognitive
Workload during Physician-Computer Interaction
Lukasz Mazur, PhD
Division of Healthcare Engineering
Radiation Oncology, UNC Chapel Hill
Agenda
Motivation
•  Mental workload à Performance
Research studies
•  System analysis
•  Computer simulation/optimization techniques
•  Subjective data
•  Stressors
•  NASA-TLX
•  Objective data
•  Eye-data (cross-coverage)
Mental
Workload
Performance
•  To subjectively and objectively quantify physician mental workload during
interactions with EHRs.
•  To redesign systems to help ‘optimize’ physician mental workload during
interactions with EHRs.
•  To develop and use advanced simulation-based training to enhance
physicians’ ability to understand their mental workload.
•  To reduce patient harm.
Motivation
TASK DEMAND MENTAL WORKLOAD (WL)
Attention
resources
SITUATION AWARENESS (SA)
WORKING
MEMORY
Stimuli
Perception
Short-term
sensory
Processing
Interpretation
Comprehension
Projection
Decision Making
Response
Execution
PATTERN MATCHING/ CATEGORIZATION SCHEMA SELECTION LONG TERM
MEMORY
LEARNING/MEMORY STORAGE SCHEMA SCRIPTS RESPONSE FEEDBACK Response
Time-line & Interactions with Computers
Consultation
Planning
UNC
Iterations & Handoffs
IMRT case:
54 steps, 15 hand-offs
(to initiate therapy)
Physics QA
Treatment
Humans cannot effec9vely detect and/or respond to failures Nuclear plant Dams 9ght coupling C AàBàCàD A Level of system coupling Humans can effec9vely detect and respond to mi9gate failures loose coupling D C A à B à C à D A more linear system B D Most universi9es Car manufacturing Chera B, Mazur L, Milowski M, Kim
HJ, Marks L. Improving Patient Safety
in Clinical Oncology: Lessons from
High Reliability Industries. The Journal
of American Medical Association
(JAMA) Oncology. Vol. 1, Issue 2, Page
e1-e7, 2015. doi:10.1001/jamaoncol.
2015.0891
B interac9vely complex Level of system interac9on Failures propagate and interact in a predictable manner Failures propagate and interact in an unpredictable manner Safety Barrier
A Safety Barrier (SB) checks one or multiple patient elements.
Patient element: information specific to a patient. They can represent discrete data elements (patient name,
diagnosis, prescription, treatment beam, etc.) or states (treatment approved, IV in arm, etc.)
1. Registration of Image Sets
2. Delineation of target(s) and organs at risk
info
2. Physician Contours
3. Preliminary prescription parameters
4. Isocenter definition
5. Dose distribution optimization and distribution calculation
info
5. Total and Fractional
Dose
Safety Barrier
6. Set up for image-guidance/motion management
Safety Barrier
Element 2 :
Physician Contours
Element 5 :
Total and Fractional Dose
Adapted from Ford et al 1
Curtsey of Pegah Pooya
Formal Safety Barriers
Element Name
Pre-Visit
Front Desk
Nursing/MD
C/SIM
First Name
Last Name
DOB
Insurance info
Appt. verification
MDs info (RadOnc)
Emergency contact info
Patient Schedule
Tx Site
Path Report
IV Contrast
Patient Consent
MR#
IMRT SBRT Preauth
Tx Side
Vital signs
Lab orders
Pacemaker
Pregnancy
Previous Tx
Concurrent Chemo
Creatinine
Allergies
Diabetic
Hypertension Dentures Imaging Ordered
Imaging Note
1
1 1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1 1 1 1 1
1
1
Dosimetry PrePlanning QA1
Sim
Review
1
1
1
1
1
1
1 1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
Dosimetry Plan
Physician
Review QA2
1 1 1 1
1
1 1 1 1 1 1 1 1 1 1 Physics PreTherapist Chart Therapist QA
treatment
Write-Up QA4
Day QA5
Review QA3
1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Total Number of Elements : 65
Curtsey of Pegah Pooya
Formal Safety Barriers
Number of
Total Cognitive
Elements in SB
Workload
# of SB
Number of
Elements
0
5 Pre-visit
11 22 1 10 Front Desk 7 12 2 17 Nursing/MD 8 21 3 10 CT/SIM 16 38 4 14 Dosimetry Pre-Planning
13 20 5 1 Sim Review
8 20 6 3 Dosimetry Plan Review
33 83 7 1 9 23 8 2 38 98 9 2 Total
65 Physician
Physics Pre-treatment
Review
Therapist Chart Write-Up
Therapist QA Day
22 20 59 52 Safety Barrier (SB)
Curtsey of Pegah Pooya
Results for Element (probability of tripping = 5 %)
Number of Safety Barriers (Swiss Cheese Layers) for a Given Element ()
Reliability of SBs
1 2 3 4 5 6 7 8 9 100% 0 0 0 0 0 0 0 0 0 90% 0.5 % 0.06 % 0 0 0 0 0 0 0 80% 1.2 % 0.2 % 0.04 % 0 0 0 0 0 0 70% 1.6 % 0.5 % 0.2 % 0.06 % 0.02 % 0.01 % 0 0 0 60% 2.1 % 0.8 % 0.4 % 0.2 % 0.06% 0.02 % 0.01 % 0 0 50% 2.6 % 1.3 % 0.7 % 0.3 % 0.2 % 0.08 % 0.07 % 0.02 % 0
40% 3.2 % 1.9 % 1.3 % 0.7 % 0.5 % 0.3 % 0.09 % 0.07 % 0
30% 3.6 % 2.5 % 1.7 % 1.3 % 0.9 % 0.7 % 0.4 % 0.2 % 0.18 %
20% 4.0 % 3.3 % 2.9 % 1.9 % 1.7 % 1.4 % 1.0 % 0.8 % 0.7 %
10% 4.6 % 4.4 % 3.6 % 3.3 % 3.1 % 2.6 % 2.7 % 2.3 % 2.1 %
0% 5.0 % 5.2 % 5.1 % 5.1 % 4.8 % 4.8 % 4.9 % 5.0 % 5.1 %
Probability that an incident 𝑖 reaches a patient
Curtsey of Pegah Pooya
Sensitivity Analysis on Probability of Tripping
Probability of Tripping (All Elements) Reliability of
SBs
Baseline
+ 10% + 20%
(GC data) + 30%
+ 40%
+ 50%
+ 60% + 70% + 80% + 90% + 100%
100% 0.8 % 0.9 % 1.0 % 1.2 % 1.21 % 1.22 % 1.4 % 1.5 % 1.6 % 1.7 % 1.8 % 90% 1.1 % 1.2 % 1.4 % 1.5 % 1.5 % 1.6 % 1.8 % 2% 80% 1.5 % 1.7 % 1.9 % 2.0 % 2.1 % 2.3 % 2.4 % 70% 1.9 % 2.3 % 2.33 % 2.6 % 2.6 % 2.9 % 60% 2.5 % 2.7 % 3.02 % 3.2 % 3.4 % 50% 3.2 % 3.6 % 3.9 % 4.1 % 40% 4.0 % 4.2 % 4.7 % 30% 5.0 % 5.6 % 20% 6.2 % 10% 0% 2.02 % 2.5 % 2.4 % 2.7 % 2.7 % 2.9 % 2.9 % 3.1 % 3.4 % 3.6 % 3.9 % 4.0 % 3.8 % 4.0 % 4.2 % 4.6 % 4.7 % 5.0 % 4.12 % 4.7 % 4.9 % 5.6 % 5.7 % 5.9 % 6.1 % 5.1 % 5.6 % 6.0 % 6.2 % 6.9 % 7.1 % 7.3 % 7.9 % 5.9 % 6.4 % 7.0 % 7.3 % 7.6 % 8.3 % 8.7 % 9.2 % 9.8 % 6.9 % 7.7 % 8.2 % 8.7 % 9.2 % 9.6 % 10.4 % 11.1 % 11.2 % 12.2 % 7.7 % 8.4 % 9.2 % 10.0 % 10.9 % 11.6 % 12.0 % 13.1 % 13.5 % 14.4 % 15.2 % 9.7 % 10.7 % 11.7 % 12.7 % 13.6 % 14.6 % 15.5 % 16.5 % 17.1 % 17.7 % 18.4 % Probability that an incident reaches a patient
Curtsey of Pegah Pooya
You don’t really want to
do that, do you?
UNC
“I would make the
target larger medially”
“Really? Ok,
I guess I can”
If you do, no way I can
cover it and spare cord
UNC
UNC
Peer Review: Transparent critique
leads to reduced rework
Results of Pre-RT
daily peer review
Percent
Rate of
Replans
Chera et al 2012,
Seminars Radiation
Oncology
Distribution of Stressors
•  5 stressors per
case simulation
therapists,
radiation
therapists, and
dosimetrists;
•  3 stressors per
case for
physicists and
radiation
oncologists
Mazur LM, Mosaly PR, Jackson M, Chang SX, Burkhardt KD, Adams RD, Jones EL, Hoyle L, Xu J,
Rockwell J, Marks LB. Quantitative Assessment of Workload and Stressors in Clinical Radiation
Oncology. Int J Radiat Oncol Biol Phys.2012 Aug 1:83(5):e571-6.doi:10.1016/j.ijrobp.2012.01.063.
UNC
Subjective Workload Scores
173 assessments
NASA-TLX:
•  Sim therapists: 30-36
•  Physicists: 51-63
•  Physicians, radiation
therapists, and
dosimetrists: 40-52.
Mazur LM, Mosaly PR, Jackson M, Chang SX, Burkhardt KD, Adams RD, Jones EL, Hoyle L, Xu J,
Rockwell J, Marks LB. Quantitative Assessment of Workload and Stressors in Clinical Radiation
Oncology. Int J Radiat Oncol Biol Phys.2012 Aug 1:83(5):e571-6.doi:10.1016/j.ijrobp.2012.01.063.
UNC
Laboratory Studies
UNC
Impact of Workload on Performance
“Easy” vs. “Difficult”
Case
•  T-test:
•  There was a significantly
lower NASA-TLX scores
for participants who were
willing to approve the
plan (vs. those not
willing) (p=0.004)
•  Easy: palliative opposed lateral 2-field brain.
•  Difficult: curative 4 field post-operative pancreas.
Mazur LM, Mosaly P, Hoyle L, Jones E, Chera B, Marks LB. Relating Physician’s Workload with
Errors during Radiotherapy Planning. Practical Radiation Oncology. Vol. 4, Issue 2, p71–75.
Impact of Workload on Performance
Reduced perfomance:
NASA-TLX approx. 50
Severity Grade of Errors
NASA TLX = 50
Grade 2: Moderate:
Altered the intended
treatment, but not likely
to have a meaningful
clinical impact
2
1
Grade 1: Mild: No direct
clinical consequence
expected
0
Grade 0: No error
0
20
40
60
80
Grade 2: Approving a treatment
prescription without correcting a
purposefully embedded error
between the treatment planning and
treatment verify/delivery system.
Grade 1: Missing prescription
requirements as required by a
standard (site, dose per fraction x #
of fractions, total dose).
100
NASA-TLX
Ordinal logistic regression:
relationship test statistic = 5.37;
Grade 0: No errors.
p-value = 0.02.
Mazur LM, Mosaly P, Hoyle L, Jones E, Chera B, Marks LB. Relating Physician’s Workload with
Errors during Radiotherapy Planning. Practical Radiation Oncology. Vol. 4, Issue 2, p71–75.
Impact of Workload on Performance
Willingness to approve
plan:
•  75% (6/8) during cross
coverage
•  Both unapproved had
NASA TLX > 55
•  100% (8 of 8) during
regular coverage.
•  Cross- vs. Regular: p<0.001 (ANOVA)
Mosaly PR, Mazur LM, Jones EL, Hoyle LM, Zagar T, Chera B, Marks LB. Quantifying the Impact of
Cross Coverage on Physician’s Workload and Performance in Radiation Oncology. Practical Radiation
Oncology. 2013 Oct-Dec;3(4):e179-86.
0.50 0.50
0.45 0.45
0.40 0.40
0.35 0.35
0.30 0.30
TEPR in mm
TEPR (mm) Impact of Cross-coverage on Performance
0.25 0.20 0.15 0.25
0.20
0.15
0.10 0.10
0.05 0.05
0.00 0.00
Cross-­‐coverage Regular-­‐coverage 3
4
5
6
7
8
Letter-Span of Memorization-Recall Experiment
Mosaly PR, Mazur LM, Jones EL, Hoyle LM, Zagar T, Chera B, Marks LB. Quantifying the Impact of Cross Coverage on
Physician’s Workload and Performance in Radiation Oncology. Practical Radiation Oncology. 2013 Oct-Dec;3(4):e179-86.
Summary
Mental workload:
•  Degradation of performance appears to be associated with mental
workload within radiation oncology settings
Limitations:
•  Local-based studies
•  Many confounding variables not addressed
Future:
•  Multi-institutional studies
•  Multi-dimensional studies
•  Advances in measurement (workload and performance)
•  Computer simulation models
•  Simulation-based training
Acknowledgment
People:
•  L. Marks, MD, B. Chera, MD, R. Adams, Ed.D
•  P. Mosaly, PhD, E. Comitz MS
•  J. Ivy, PhD, P. Pooya, PhD student - NCSU
•  Radiation Oncology Department – many involved
Funding source:
•  This project was supported by grant number R18HS023458 from the
Agency for Healthcare Research and Quality. The content is solely
the responsibility of the authors and does not necessarily represent
the official views of the Agency for Healthcare Research and Quality.
•  Center for Innovation, University of North Carolina Health System