Appendix Bayes factor and the impact of technical errors in laparoscopic colorectal
surgery assessments
Melody Ni
In this appendix, we explain the logic behind using Bayes factor (BF) to assess the impact of
a technical error on the sign off assessments within the National Training Programme of
Laparoscopic Colorectal Surgeries (NTP).
Belief updating and the Bayes’ rule
We conceptualise the sign-off process within NTP as the process of belief updating, whereby
“information is received a piece at a time and integrated into a continuously evolving
impression” ([1], p.144). The impression, in our study, corresponds to the subjective belief an
NTP assessor forms over the safety of the operation. The information corresponds to a
technical error the assessor identified because he or she believed the error was important for
safety. We further hypothesised that 1) this belief was sequentially adjusted as the assessor
identified an increasing number of technical errors and 2) the updated belief was fundamental
to his or her decision over whether to pass the sign-off so that the delegate could embark
upon independent practices outside NTP.
The normative framework to capture this belief updating process is the Bayes’ rule [2].
Within this framework, the impression is the hypothesis (H) that the operation was safe; the
information is the data (D) that a technical error had been identified. A higher probability
corresponds to a stronger belief and vice versa. The Bayes’ rule has the following form:
P(D|H) = P(H)P(H|D)/ P(D),
-(1)
where P(H) is the prior probability or one’s initial belief that the surgery is safe before
observing any error; P(D|H) is the conditional probability of an error being observed in
successful sign offs; P(H|D) is the posterior probability or one’s updated belief that the
surgery is safe following identification of an error. P(D) describes how likely the error will be
observed regardless of whether or not the surgery is safe. As errors are observed one at a time
Bayes’ rule can be applied repeatedly to capture the iterative process of belief updating as
more and more errors were identified.
1
Likelihood ratios and the Bayes’ factor
Since we only considered two possible outcomes of a sign off, pass or fail, a more useful
form of Bayes’ rule is the so-called “odds-likelihood ratio” formulation, used to distinguish
between two mutually exclusive hypotheses, H (successful sign offs for safe surgery
performed) and H1 (otherwise, P(H1) = 1-P(H)), based on observation of error (D):
Posterior odds = Prior odds x Likelihood ratios, - (2)
where posterior odds = P(H|D)/(1- P(H|D)), prior odds = P(H)/(1- P(H)), and likelihood
ratios = P(D|H)/(1- P(D|H)). The biggest benefit of using the likelihood ratio form instead of
(1) is that there is no need to consider the prevalence of error, i.e. P(D).
Bayes’ rule establishes a direct link between one’s subjective beliefs (measured in probability
odds) and evidence. The impact of the evidence is measured by the likelihood ratio that
compares how likely the data is observed in successful (P(D|H)) versus in unsuccessful sign
offs (1- P(D|H)). Bayes’ factor, in its simplest forms, equates the likelihood ratio.
Error and sign off decision
Identification of errors signals lack of safety and decreases the posterior odds of safe surgery.
This in turn reduces the chance that assessor would grant a pass to the sign-off submission.
Following this logic we can measure the impact of an error on sign off decisions by rewriting Equation 2 into:
Posterior odds of safe sign off being successful
= Prior odds of sign off being successful x Bayes Factor of an error, (3)
The more critical an error the smaller its Bayes’ factor will be and vice versa. Further under
the assumption of conditional independence, the Bayes’ framework (Eq. 1 & 2) dictates that
the impact of multiple errors (data) can be measured simply by the multiplication of their
respective likelihood ratios. Suppose n number of errors were observed:
Posterior odds of sign off being successful
= Prior odds of sign off being successful x Bayes Factor of error1 x Bayes Factor of error2…
x Bayes Factor of error1 x Bayes Factor of errorn, (4)
2
Example – how to use Bayes’ factor to estimate sign off success (as if you were an expert
assessor)
1. Assume a prior probability that the sign off would be successful before observation of
any error. We have used 90% in our analysis. This is equivalent to a prior odds of 9 =
(0.9/(1-0.9).
2. From Table 3 (main text) find out the Bayes’ factor of a significant error in the
operative video. For instance the transection error has a Bayes’ factor of 0.20.
3. Apply Equation 3 to get the updated probability of successful sign off.
-
Posterior odds = Prior odds x Bayes’ factor
-
Posterior odds = 9 * 0.2 = 1.8
-
Posterior probability = posterior odds/(1+ posterior odds) = 1.8/2.8 = 64.3%
Therefore a significant breach in transection would reduce the chance of being signed off
from 90% to 64.3%, which is still over the 50% benchmark.
When multiple errors were observed:
Suppose excessive bleeding (BF=0.42) has also been observed.
4. Repeat the above process but use the posterior odds as the new prior odds.
-
Posterior odds = Prior odds x Bayes’ factor = 1.8 * 0.42 = 0.756
Posterior probability given transection error and excessive bleeding is therefore 43% =
(0.756/1.756), in which case the sign off is more likely to be a failure rather than a success
(<50%).
Note this is equivalent to estimate the joint BF for the two errors first and then apply Eq 4.
-
Joint BF of transection error and excessive bleeding = 0.2*0.42=0.084
-
Posterior odds = Prior odds x Bayes’ factor = 9 * 0.084 = 0.756
References:
1.
Anderson NH (1981) Foundations of information integration theory, Academic Press, New
York
2.
Bayes T (1763) An essay towards solving a problem in the doctrine of chances. Phil Trans
53:370-418
3