APPENDIX C. Bayesian Components for Drug-Drug Interaction (DDI) Alert Effectiveness Reporting
To be able to more accurately capture sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for drug-drug interaction (DDI) alerts, the data and its generation should be constructed around the
Bayesian components of alert data. These components of alert data are True Positive (TP), True Negative (TN),
False Positive (FP), and False Negative (FN) illustrated in Table 1 below.
Table 1. Bayesian Components of Alert Data
Relevant to Patient
+ -
Presented to
Clinician
+ TP FP
- FN TN
FN = false negative; FP = false positive; TN = true negative; TP = true positive. True positive is defined as an alert appearing when two medications are ordered that are known to interact and result clinically meaningful harm to the patient. A true negative is defined as no alert appearing when there is no known interaction. A false positive is defined as an alert appearing when there is no clinically relevant interaction between two medications. A false negative is when there is no alert but there is a clinically relevant interaction.
Positive predictive value is the number of true positive divided by the sum of the number of true positives plus the number of false positives. Negative predictive value is the number of true negatives divided by the sum of the number of true negatives plus the number of false negatives.
Table 2 provides examples of what types of data should be included; some are currently available from certain electronic health record (EHR) vendors, while others would be useful if the functionality is developed.
Table 2. Examples of Data to Measure Alert Effectiveness
Bayesian
Data Points Description / Example
Component
True Positives Patient modifiers that increase relevance to patient

Trends in historical lab values

Diagnoses on problem list
Appropriate follow up actions

Lab ordered for monitoring

Alert overridden, a few minutes later the same prescriber alters dose,
1

frequency, or administration times or discontinues one of the interacting medications, or adds another drug

Educate patient: provide ability to schedule or document education from the alert window

Request consultation by another provider: this action should be available from within the alert window and tracked; should be viewable by the next
ADE/errors prevented by alert
Overridden alerts that led to provider that sees the alert


ADEs currently not easily retrievable (subset of accepted alerts)
Prevented errors need to be defined per alert (subset of accepted alerts)

Indication to evaluate content and usability of alert, or possibly need for
ADE
Overridden alerts that led to medication errors clinician education

Evaluate using a random sample of overridden alerts

Leverage existing medication error reporting mechanisms to identify
True
Negatives
False Positives
ADE case reports evaluated those instances where an alert was presented

Highlights opportunities to update clinical knowledge base with DIPS(1)
Patient modifiers that

Alerts were correctly hidden from the clinician because modifiers were decrease relevance to evaluated negligible
Patient modifiers that decrease relevance to

Requires standardized feedback mechanisms

Crowd-source collective clinical knowledge at institutional, national, and negligible, but were NOT evaluated perhaps international levels for inclusion in alert logic

Common reasons for false positives include incorrect medication lists;
False
Negatives
Alerts with no appropriate follow up actions
ADEs where no alert fired ordering and discontinuation sequence.

Appropriate action already taken (e.g., dose already reduced based on clinical experience, prior to alert firing)

Automatic reporting of ADEs is ideal for comparison

Evaluate using a random sample of medication orders that were entered Medication errors where no alert fired
Relevant patient modifiers without any alerts.(2)

Opportunity to increase sensitivity by including in alert logic
2

that were NOT evaluated
ADE = adverse drug event; DIPS = Drug Interaction Probability Score
REFERENCES
1 Horn JR, Hansten PD, Chan LN. Proposal for a new tool to evaluate drug interaction cases. Ann Pharmacother.
2007 Apr; 41 (4):674-80.
2 Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014 2014 May-Jun; 21 (3):487-91.
3