Electronic Supplement: Additional tables
Table E1- Case 1 Checklist items. Items with N/A were uniformly not performed,
and thus timing agreement cannot be calculated. Items marked with * were below
our threshold for inclusion in the final rubric. **Note that only one subject was
categorized as novice for case 1, and thus all items were either 100% or 0%.
Item
Identified skin mottling?
Bolused IV fluids?*
Obtained cultures (any type)
Obtained blood cultures?
Obtained urine cultures?
Obtained respiratory cultures?
Administered stress dose steroids?
Identified hypoglycemia?
Administered IV glucose?
Antibiotics given?
Vasopressors initiated?*
Central line placed?
Ultrasonography performed?
Chest X ray obtained?
Source of infection addressed?
Patient intubated?
Intubation preparation discussed?*
Preoxygenation performed?
Sedation performed
Ventilator mode ordered?
Ventilator set for lung protective ventilation?
Pooled reliability:
% Overall
Agreement
80%
100%
80%
100%
60%
80%
100%
80%
60%
100%
80%
80%
80%
80%
100%
80%
80%
80%
60%
100%
100%
83.8%
% agreement on
timing (within 15
seconds
100%
20%
100%
100%
100%
N/A
100%
100%
N/A
60%
25%
100%
N/A
100%
N/A
100%
50%
100%
N/A
N/A
N/A
73.0%
Done by experts
(N=4)
0 (0%)
4 (100%)
4 (100%)
3 (75%)
2 (50%)
4 (100%)
1 (25%)
2 (50%)
2 (50%)
4 (100%)
4 (100%)
4 (100%)
4 (100%)
3 (75%)
0 (0%)
2(50%)
3 (75%)
2 (50%)
1 (25%)
0 (0%)
0 (0%)
58.3%
Done by
novice**
(N=1)
1 (100%)
1 (100%)
1 (100%)
1 (100%)
0 (0%)
1 (100%)
0 (0%)
0 (0%)
0 (0%)
1 (100%)
1 (100%)
1 (100%)
1 (100%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
1 (100%)
0 (0%)
0 (0%)
0 (0%)
47.6%
Table E2- Case 2 Checklist items. Items with N/A were uniformly not performed,
and thus timing agreement cannot be calculated. Items marked with * were below
our threshold for reliability and were not included in the final rubric.
Item
Identified wheezing?
Identified crackles?
Identified sinus tachycardia?*
Identified low urine output
Administered fluid bolus*
Obtained cultures (any type)
Obtained blood cultures?*
Obtained urine cultures?
Obtained respiratory cultures?
Antibiotics administered?*
Ultrasonography performed?
Chest X ray obtained?
Influenza nasal swab obtained?
Respiratory serologies obtained?
Urinary antigen tests obtained?
Intubation preparation discussed?
Preoxygenation performed?
Sedation performed
Patient intubated?
Ventilator mode ordered?
Ventilator set for lung protective ventilation?
Pooled reliability:
% Overall
Agreement
86%
86%
29%
100%
57%
71%
57%
71%
86%
57%
71%
71%
100%
100%
100%
71%
71%
71%
86%
100%
100%
78.2%
% Agreement
in items
which both
reviewers
classified as
"Done"
100%
N/A
0%
N/A
0%
100%
100%
N/A
N/A
N/A
N/A
100%
N/A
N/A
N/A
100%
100%
100%
100%
N/A
N/A
81.8%
Done by
expert
(N= 4)
Done by
novices
(N=4)
3 (75%)
3 (75%)
4 (100%)
0 (0%)
1 (25%)
4 (100%)
4 (100%)
1 (25%)
0 (0%)
3 (75%)
0 (0%)
4 (100%)
0 (0%)
0 (0%)
0 (0%)
4 (100%)
4 (100%)
4 (100%)
4 (100%)
1 (25%)
1 (25%)
53.6%
3 (75%)
2 (50%)
1 (25%)
0 (0%)
2 (50%)
2 (50%)
2 (50%)
0 (0%)
0 (0%)
2 (50%)
0 (0%)
3 (75%)
0 (0%)
0 (0%)
0 (0%)
3 (75%)
3 (75%)
4 (100%)
4 (100%)
0 (0%)
0 (0%)
36.9%
Table E3- Agreement between raters on Case 3 Checklist items. Items with N/A
were uniformly not performed, and thus timing agreement cannot be calculated.
Items marked with * were below our threshold for inclusion in the final rubric.
Item
EKG obtained?*
Cardiology/Cath lab consulted?*
Aspirin given?*
Second antiplatelet given?
Analgesia administered?*
Nitroglycerin given?
Thrombolysis/PCI requested?*
Ultrasonography performed?
Chest X ray obtained?
EKG repeated with rhythm change?
Ventricular tachycardia identified?*
Cardioversion performed?*
Sedation performed?*
Mode of cardioversion specified?
Energy for cardioversion specified?
Pooled reliability:
% Overall
Agreement
100%
75%
50%
100%
25%
75%
50%
100%
100%
100%
50%
100%
100%
75%
100%
80.0%
% Agreement in
items which both
reviewers
classified as
"Done"
50%
50%
100%
N/A
100%
100%
100%
N/A
100%
100%
100%
0%
0%
N/A
N/A
72.2%
Done by
expert
(N=3 )
Done by
novices
(N=2)
3 (100%)
3 (100%)
3 (100%)
0 (0%)
3 (100%)
2 (67%)
3 (100%)
0 (0%)
2 (67%)
1 (33%)
1 (33%)
2 (67%)
2 (67%)
2 (67%)
0 (0%)
60.0%
2 (100%)
1 (50%)
1 (50%)
0 (0%)
2 (100%)
1 (50%)
1 (50%)
0 (0%)
2 (100%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
33.3%