Structured Assessment Format for
Evaluating Operative Reports (SAFEOR) in General Surgery
Ashley Vergis1 MD, Lawrence Gillman1 MD, Jason Park1 MD,
Samuel Minor MD FRCSC2, Mark Taylor1 MD FRCSC
1University
of Manitoba, Winnipeg, Canada
2Dalhousie University, Halifax, Nova Scotia, Canada
Medicine is an information intensive and reliant
profession whose practice is “dominated by how
well information is processed or reprocessed,
retrieved and communicated”
Murphy et al. (1992). Good European Health Record. Centre for
Health Informatics and Multi-professional Education University
College London: 1-62.
Background

Operative dictations essential for*:
Caregiver communication
 Quality assurance
 Research
 Medico-legal

 Must be comprehensive and concise!
*Eichholz et al. (2004). Obstetrics and Gynecology 103(2): 342-6.
Background


Multiple previous attempts to judge the
quality of OR notes

Third party billing criteria

Comparison to consensus criteria
Overall quality appears poor
Critical details often omitted
 Superfluous aspects detailed

Background

However…


No validated tools that examine quality of
operative dictations
Consequences
Cannot currently measure the adequacy of an
operative report
 Unclear if operative reports meet their stated
objectives

Objective

Evaluate the ‘Structured Assessment
Format for Evaluating Operative Reports’
(SAFE-OR) in General Surgery
Construct validity
 Inter-rater agreement
 Internal consistency

Methods

Subjects

University of Manitoba general surgery
residents
Novices PGY 1-3
 Experienced PGY 4-6

Methods

Design

View and dictate a videotaped laparoscopic
sigmoid colectomy

Dictation transcribed
Methods

Design

Evaluated by 2 blinded, independent
faculty general surgeons
 ‘SAFE-OR’

Structured Assessment Format


instrument
Adapted from CAGS clinical practice committee
guidelines
Global Quality Rating scale
Methods

Structured Assessment Format

19 Categorical items (yes/no)


E.g., Date of Procedure
7 Ordinal items (5 point anchored Likert
Scale)

E.g., Findings on Exploration
Structured Assessment Form
1. Date of Surgery
2. Date of Dictation
3. Patient Identifiers
4. Name of Surgeon and Assistants
5. Name of Anaesthetist
6. Pre-op Patient Diagnosis
7. Post-op Patient Diagnosis
8. Procedure Performed (includin g side)
9. Copies Sent to Referring Physician/Institution
1 0. Brief History of Present Illness (includin g relevant investigations)
0
No description of preoperative
course or indications
1
2
Preoperative course and
indications were described but
some detail was lacking
3
11. Procedure
a. Type of Anesthesia
b. Skin Preparation and Draping
c. Patient Position
d. Method of Abdominal Access
0
No mention of abdominal
access
e.
2
Method of abdominal access
mentioned but specific details
lacking
NO
NO
NO
NO
NO
NO
NO
NO
NO
4
Complete description of preoperative
course and indications for procedure
YES
YES
YES
NO
NO
NO
3
4
Complete description including location of
incisions/trocar placements/insufflation
pressures (if applicable)
3
4
Completely explains results of exploration
making note of unexpected findings
Findin gs on Exploration of Relevant Body Cavity
0
No mention of findings on
exploration
f.
1
YES
YES
YES
YES
YES
YES
YES
YES
YES
1
2
States findings of exploration
but detail/explanation lacking
Techniq ue of Resection, Including Tissue Mobili zation
0
No description of mobilization
or resection technique
1
2
Mobilization and resection
techniques described but
details regarding operative
steps incomplete
3
4
Complete description of tissue
mobilization and method of
resection including instruments used
(staplers etc.)
N/A
Methods

Global Quality Rating Scale
5 point anchored Likert scale
 9 categories

1. Readability
2. Operative Indications
3. Operative Steps
4. Operative Findings
5. Succinctness
6. Clarity
7. Knowledge
8. Reproducibility
9. Overall Quality
Global Quality Ratings Scale
Please circle the number corresponding to the candidate’ s performa nce in each category.
1. Readability of Dictation
1
Dictation was difficult to
read and disorganized with
incoherent flow
2
3
Dictation was relatively
easy to read but at times
lacked flow
4
5
Dictation was easy to read
and well organized with
coherent flow
3
Preoperative course and
indication were described
but som e detail was lacking
or inaccurate
4
5
Dictation includes com plete
description of preoperative
course and indications for
specific procedure performed
3
Dictation included most
im portant steps but some
detail was missing
4
5
Dictation was comprehensive
and included all important
steps of procedure
2. Description of Operative Indications
1
No description of
preoperative course or
indications was included
2
3. Inclusion of Operative Steps
1
Dictation was incom plete
as most important steps
were missing
2
Analysis

Construct Validity


Novice and experienced groups
compared (Independent samples T test)
Inter-rater Agreement

Intra-class correlation coefficient (ICC)
>0.8 desirable*

Internal Consistency

Cronbach’s alpha
*Wanzel, et al. (2002). Current Problems in Surgery 39(6): 573-659.
Results: Construct Validity
Mean Score
Novice
(n=13)
Experienced
(n=8)
p
Structured
Assessment
(out of 44)
23.3 ± 5.2
34.1 ± 6.0
0.001
Global Quality
(out of 45)
25.6 ± 4.7
35.9 ± 7.0
0.006
Results: Inter Rater Reliability
& Internal Consistency
Overall ICC
(95% CI)
Structured
Assessment
Global Quality
0.98
(0.96 - 0.99)
0.93
(0.83 - 0.97)
Range
Cronbach’s
Alpha
0.73 - 0.90*
0.85
0.65 - 0.91
0.96
* Ordinal items, Kappa 0.73 - 1.00 for categorical items
Conclusion

SAFE-OR
 Satisfies
construct validity
 Excellent inter-rater reliability
 Excellent internal consistency
Project Significance
SAFE-OR is the first tool that
assesses the quality of an
operative report in a valid and
reliable manner
Project Significance

Means to more objectively measure
operative report characteristics

Assess degree to which reports meet stated
functions (quality assurance)

If unsatisfactory, may employ methods to
improve reporting
Project Significance

Potential to develop methods of improving
dictations in surgical education

Implement SAFE-OR for assessment in
training programs


Formative and Summative
Feedback (self and faculty)
Global attributes
 Specific details

Project Significance

Develop, monitor and refine curriculum for
operative dictation

Improve communication
 Implications for patient care
Future Investigation




Analyze characteristics of real world
operative dictations
National survey (Cdn) of resident and
program directors training perceptions
Develop and assess specific templates for
operative dictation
Assess computerized dictation templates
(WebSMR)*
*Edhemovic
et al., Annals of Surgical Oncology, 11(10):941–947.
Acknowledgements

Royal College of Physicians and Surgeons
of Canada Medical Education Research
Grant
1. Date of Surgery
2. Date of Dictation
3. Patient Identifiers
4. Name of Surgeon and Assistants
5. Name of Anaesthetist
6. Pre-op Patient Diagnosis
7. Post-op Patient Diagnosis
8. Procedure Performed (includin g side)
9. Copies Sent to Referring Physician/Institution
1 0. Brief History of Present Illness (includin g relevant investigations)
0
No description of preoperative
course or indications
1
2
Preoperative course and
indications were described but
some detail was lacking
3
11. Procedure
a. Type of Anesthesia
b. Skin Preparation and Drapi ng
c. Patient Position
d. Method of Abdominal Access
0
No mention of abdominal
access
e.
3
4
Complete description including location of
incisions/trocar placements/insufflation
pressures (if applicable)
1
2
States findings of exploration
but detail/explanation lacking
3
4
Completely explains results of exploration
making note of unexpected findings
1
2
Mobilization and resection
techniques described but
details regarding operative
steps incomplete
3
4
Complete description of tissue
mobilization and method of
resection including instruments used
(staplers etc.)
N/A
1
2
Resected structures listed but
clear anatomical boundaries
not well defined
3
4
Exact anatomical description of
structures resected including all
adjacent structures and margins
N/A
4
Comprehensive description of
reconstruction including technique,
equipment used, and integrity of
repair
N/A
1
2
Type of technique mentioned
but steps and results of
reconstruction omitted
3
1
2
Closure described but specific
details omitted
3
Techniq ue of Closure
0
No description of closure
method
j.
k.
l.
NO
NO
NO
Techniq ue of Reconstruction
0
No mention of technique of
reconstruction
i.
YES
YES
YES
Limits of Resection
0
No anatomical description of
structures resected or mention
of surgical margins
h.
4
Complete description of preoperative
course and indications for procedure
Techniq ue of Resection, Including Tissue Mobili zation
0
No description of mobilization
or resection technique
g.
2
Method of abdominal access
mentioned but specific details
lacking
NO
NO
NO
NO
NO
NO
NO
NO
NO
Findin gs on Exploration of Relevant Body Cavity
0
No mention of findings on
exploration
f.
1
YES
YES
YES
YES
YES
YES
YES
YES
YES
Irrigation
Drains and location
Prosthesis (size and type) or other items left in
location
m. Estimated Blood Loss
n. Dressings Applied
o. Sponge and Instrument Count
p. List of Specimens Sent (Micro and Pathology)
4
Comprehensive description including
suture method, size and type of suture
employed if applicable
YES
YES
NO
NO
N/A
N/A
YES
NO
N/A
YES
NO
YES
NO
YES
NO
YES
NO N/A
Please circle the number corresponding to the candidate’s performance in each category.
1. Readability of Dictation
1
Dictation was difficult to
read and disorganized with
incoherent flow
2
3
Dictation was relatively
easy to read but at times
lacked flow
4
5
Dictation was easy to read
and well organized with
coherent flow
3
Preoperative course and
indication were described
but som e detail was lacking
or inaccurate
4
5
Dictation includes com plete
description of preoperative
course and indications for
specific procedure performed
3
Dictation included most
im portant steps but some
detail was missing
4
5
Dictation was comprehensive
and included all important
steps of procedure
3
Operative findings were
described but som e relevant
detail was lacking
4
5
Operative findings were
presented in a relevant and
detailed fashion
2
3
Dictation was relatively
concise and inclusive with
some extraneous detail
included
4
5
Dictation was concise but
inclusive with exclusion of
extraneous detail
2
3
Description of included
steps was relatively clear
and intelligible
4
5
Description of included steps
was clear and complete
2
3
Has a reasonable
understanding of the
procedure and relevant
anatomy
4
5
Demonstrates a full
understanding of the
procedure and relevant
anatomy
4
5
Reader has a complete
understanding of operation
and could recreate procedure
step by step
2. Description of Operative Indications
1
No description of
preoperative course or
indications was included
2
3. Inclusion of Operative Steps
1
Dictation was incom plete
as most important steps
were missing
2
4. Description of operative findings
1
2
Operative findings
described were irrelevant
or omitted com pletely
5. Succinctness of Dictation
1
Dictation was
unnecessarily long with
excessive inclusion of
extraneous detail or
truncated but lacki ng
im portant elements
6. Clarity of Dictation
1
Description of included
steps was vague and
unintelligible
7. Knowledge
1
Clearly lacks an
understanding of the
procedure and relevant
anatomy
8. Reproducibility of Operative Procedure
1
Recreation of operative
events would be
im possible from this
dictation
2
1
Unacceptable dictation
Major deficiencies in
operative dictation skills
2
3
Reader could recreate
events using own
knowledge to fill in gaps
OVERALL PERFORMANCE
3
Acceptable dictation
Some room for
im provem ent
4
5
Excellent dictation
Quality consistent with that
expected of an attending
surgeon
CAGS Clinical Practice Commit tee Components of Operative Notes document for
gastrointestinal (GI) procedures
Date of Surgery
Date of Dictat ion
Pat ient Ident ifiers
Name of Surgeon and Assistant s
Name of Anesthet ist
Name of Others Present
Pre and Post -op Pat ient Diagnosis
Procedur e Performed (including side)
Brief History of Present Illness
Including relevant invest igat ions
Procedur e
1. Pat ient Position
2. Skin P reparat ion and Draping
3. Locat ion of Incision(s)
4. Findings on Explorat ion of Relevant Body Cavity
 Special note of unexpected findings
 If cancer: extent of primary, lymph node and distant metastases
5. Mobilization of Structures to be Operated Upon
6. Technique of Resect ion
7. Limit s of Resection
8. Technique of Reconstruction
9. Irrigat ion
10. Drains and Locat ions
11. Prosthesis
 Size and type
12. Other Items Left and Locat ion
13. Technique of Closure
14. Est imated Blood Loss
15. Dressings Applied
16. Sponge and Inst rument Count
Structured Assessment Format for
Evaluating Operative Reports (SAFEOR) in General Surgery
Ashley Vergis1 MD, Lawrence Gillman1 MD, Jason Park1 MD,
Samuel Minor MD FRCSC2, Mark Taylor1 MD FRCSC
1University
of Manitoba, Winnipeg, Canada
2Dalhousie University, Halifax, Nova Scotia, Canada