CLINICAL PRACTICE
Accuracy of Very Low Pretest Probability
Estimates for Pulmonary Embolism Using the
Method of Attribute Matching Compared with
the Wells Score
Jeffrey A. Kline, MD, D. Mark Courtney, MD, Martin P. Than, MBBS, Kerstin Hogg, MBChB, MD,
Chadwick D. Miller, MD, Charles L. Johnson, and Howard A Smithline, MD
Abstract
Objectives: Attribute matching matches an explicit clinical profile of a patient to a reference database
to estimate the numeric value for the pretest probability of an acute disease. The authors tested the
accuracy of this method for forecasting a very low probability of venous thromboembolism (VTE) in
symptomatic emergency department (ED) patients.
Methods: The authors performed a secondary analysis of five data sets from 15 hospitals in three countries. All patients had data collected at the time of clinical evaluation for suspected pulmonary embolism
(PE). The criterion standard to exclude VTE required no evidence of PE or deep venous thrombosis
(DVT) within 45 days of enrollment. To estimate pretest probabilities, a computer program selected, from
a large reference database of patients previously evaluated for PE, patients who matched 10 predictor
variables recorded for each current test patient. The authors compared the outcome frequency of having
VTE [VTE(+)] in patients with a pretest probability estimate of <2.5% by attribute matching, compared
with a value of 0 from the Wells score.
Results: The five data sets included 10,734 patients, and 747 (7.0%, 95% confidence interval [CI] = 6.5%
to 7.5%) were VTE(+) within 45 days. The pretest probability estimate for PE was <2.5% in 2,975 of
10,734 (27.7%) patients, and within this subset, the observed frequency of VTE(+) was 48 of 2,975 (1.6%,
95% CI = 1.2% to 2.1%). The lowest possible Wells score (0) was observed in 3,412 (31.7%) patients, and
within this subset, the observed frequency of VTE(+) was 79 of 3,412 (2.3%, 95% CI = 1.8% to 2.9%)
patients.
Conclusions: Attribute matching categorizes over one-quarter of patients tested for PE as having a pretest probability of <2.5%, and the observed rate of VTE within 45 days in this subset was <2.5%.
ACADEMIC EMERGENCY MEDICINE 2010; 17:133–141 ª 2010 by the Society for Academic Emergency
Medicine
Keywords: PE, decision-making, D-dimer, diagnosis, decision rule, venous thromboembolism,
computerized tomography angiography, medical malpractice
From the Department of Emergency Medicine, Carolinas Medical Center (JAK), Charlotte, NC; the Department of Emergency
Medicine, Feinberg School of Medicine, Northwestern University (DMC), Chicago, IL; Christchurch Hospital (MPT), Christchurch, New Zealand; Salford Royal Hospital (KH), Manchester, UK; the Department of Emergency Medicine, Wake Forest University School of Medicine (CDM), Winston-Salem, NC; Pretest Consult LLC (CLJ), Newton, MA; and the Department of
Emergency Medicine, Baystate Medical Center (HAS), Springfield, MA.
Received June 3, 2009; revision received August 11, 2009; accepted August 17, 2009.
Supported by grants from the National Institutes for Health, R41HL074415 and R42HL074415, R42 HL086316, K23HL077404, and
R01 HL074384; a Medical Student Award from the Emergency Medicine Foundation; and a Scholar’s Award from the Translational
Science Institute of Wake Forest University.
JAK and CLJ own stock in CP Diagnostics LLC and Studymaker LLC.
JAK conceived and organized the study, wrote the protocol, obtained funding, collected and analyzed data, and drafted the manuscript. JAK, DMC, CLJ, KH, MPT, CDM, and HAS participated in obtaining funding, data collection and analysis, and drafting and
revising the manuscript. CLJ contributed to the overall study design based upon design and operation of the web-based data collection system, and helped to collect and analyze data and draft the manuscript. All authors have access to all data using the rules of the
data use agreement.
Address for correspondence and reprints: Jeffrey A. Kline, MD; e-mail: jkline@carolinas.org.
ª 2010 by the Society for Academic Emergency Medicine
doi: 10.1111/j.1553-2712.2009.00648.x
ISSN 1069-6563
PII ISSN 1069-6563583
133
134
T
Kline et al.
he effort to improve the effectiveness and efficiency of pulmonary embolism (PE) diagnosis
remains a challenge. Experts continue to suggest that physicians too often fail to diagnose PE and
thus are subject to medicolegal allegations of negligence.1,2 Physicians in the United States are increasingly concerned about medical malpractice secondary
to missed PE.3,4 These influences, together with the
widened availability and acceptance of D-dimer and
computerized tomography pulmonary angiography
(CTPA), may have led to increased testing for PE
among very-low-risk outpatients.5 A low threshold for
investigation for PE has led to high rates of repeated
CTPA imaging in the emergency department (ED) setting.6 Negative consequences of increased CTPA imaging include exposure of healthy patients to radiation
and intravenous contrast material.7–9
In recommending a standard approach to diagnosis
and exclusion of PE, authors often use a structured
method to estimate the pretest probability for PE to
interpret the results of diagnostic testing.10 This procedure must yield a posttest probability that is either high
enough to justify administration of empiric anticoagulation (recommended at a threshold between 25 and 80%)
or low enough to safely exclude PE (suggested at a
threshold between 1 and 4%).10–12
The pretest probability assessment can also be used
to exclude PE prior to any formal testing, assuming that
the value of the pretest probability is accurate, and is
below the test threshold. The test threshold represents
the point of equipoise that balances the risk of missing
the diagnosis with the risk of unnecessary testing;
patients with a pretest probability lower than the test
threshold should not benefit (and may even be harmed)
from diagnostic testing. We estimate the point of equipoise, rounded to the nearest whole percentage, at
2%.13,14 We therefore accept pretest probability values
below 2.5% as potentially exclusionary for PE. Because
2.5% represents the upper limit boundary, the actual
observed rate of PE in a population of patients with a
pretest probability of <2.5% should reside between zero
and 2.5%.
This study measures the diagnostic accuracy of a
computerized method to forecast the probability of PE
using attribute matching.15–17 First, attribute matching
requires the clinician to gather a set of predefined
clinical predictor variables from a patient with suspected PE. The tool employs 10 predictor variables:
pleuritic chest pain, dyspnea, age, heart rate, pulse
oximetry reading, prior history of venous thromboembolism (VTE), hemoptysis, recent surgery, unilateral
leg swelling, and estrogen use.14,18 Attribute matching
works by a selection process whereby a computer
algorithm compares the results of all 10 predictor variables obtained from the patient being evaluated to a
library of research patients previously evaluated for
PE compiled from multiple hospitals. The algorithm
returns from the library only the ‘‘matched’’ patients
who share the same profile of predictor variables as
the patient under consideration and reports the
proportion of patients with disease in this matched
sample.
•
ESTIMATION FOR PE: ATTRIBUTE MATCHING VS. WELLS SCORE
The objective of this study was to test the diagnostic
accuracy of attribute matching for the outcome of PE
using a compilation of five prospectively collected databases of patients evaluated for suspected PE and to
compare its performance to a multivariate logistic equation derived and validated for the purpose of estimating
the pretest probability of PE. All of these patients are
new to the attribute-matching system, meaning that
they are different patients than those in the search
database used to generate pretest probabilities and, as
such, represent validation data. The hypothesis was that
the group of patients with a calculated (by attribute
matching) pretest probability of PE <of 2.5% would
have an actual outcome frequency (i.e., posterior probability) of VTE of <2.5%, and when compared with Wells
logistic regression–based method, attribute matching
would estimate a larger proportion of patients with a
pretest probability of PE of <2.5% and yield a lower
false-negative rate.
METHODS
Study Design
This study was a secondary analysis of five discrete
databases18–22 collected at 15 hospitals in three countries between 2001 to 2006. The study protocol was
approved by the institutional review boards for the
conduct of research on humans of all participating
hospitals.
Study Setting and Population
The methods and patient characteristics of two of these
databases have been published,19,20 with only partial
data from one database (from coauthor MPT) used for
one study.18 The largest of these databases was explicitly collected for this purpose, and the methods of data
collection have also been published.23 All of the databases were collected for the purpose of studying the
diagnostic evaluation of acute PE in symptomatic
patients. Patients were enrolled in the ED based on
clinical suspicion of PE prompting formal diagnostic
testing for PE. The predictor variables used by attribute
matching to assess pretest probability were collected
by clinicians or qualified research personnel while the
patients were present in the ED.
Study Protocol
Criterion Standard for Diagnosis and Exclusion of
VTE. The criterion standard included at least one
objective diagnostic test to exclude PE, including either
a normal quantitative D-dimer and ⁄ or a CTPA without
evidence of PE or a ventilation-perfusion lung scan
(V ⁄ Q) read as normal or read as nondiagnostic,
together with bilateral lower extremity ultrasonography
of the venous system read as negative for deep venous
thrombosis (DVT). Images were interpreted by boardcertified specialists in radiology. The absence of PE
required that the patient and the patient’s medical
record reveal no evidence of diagnosis of PE or DVT
within 45 days after enrollment and no evidence of
death from PE (including autopsy data when available).
Establishment of adequate data for follow-up required
ACAD EMERG MED • February 2010, Vol. 17, No. 2
www.aemj.org
135
telephone or written contact with the patient or
patient’s provider or direct observation of a medical
record sufficient to confirm presence (+) or absence ())
of VTE (e.g., history and physical examination from an
admission note, more than 45 days after enrollment).
Diagnosis of VTE required either CTPA read as positive
for PE, V ⁄ Q scanning read as high probability, or diagnosis of DVT within 45 days. However, the criterion
standard also required written evidence of a clinical
plan to treat with either anticoagulation or vena caval
interruption. For example, a patient with isolated calf
vein thrombosis that was not treated was adjudicated
as VTE()), whereas a patient with an isolated calf vein
that was treated at any point with systemic anticoagulation was adjudicated as VTE(+). At a minimum, all studies used adjudication, requiring agreement of two
independent physicians, for difficult or indeterminate
cases.24
categories as follows: age was trichotomized (<35, 35–50,
and >50 years), heart rate was dichotomized (at 99.5
beats ⁄ min), and SaO2 percentage was dichotomized (at
94.5%). The values of pleuritic chest pain, dyspnea,
prior history of VTE, hemoptysis, recent surgery, unilateral leg swelling, and estrogen use were all dichotomous (yes or no). The tool can therefore produce 3 · 92
or 1,536 unique pretest probability estimates. The program output contains the total number of patients
(Ntotal) returned from the reference database who
matched the 10-attribute predictor profile of each new
patient, and the number of patients who had a criterion
standard diagnosis of VTE+ (NVTE+) within 45 days. The
pretest probability for PE equaled NVTE+ ⁄ Ntotal except
where NVTE+ was zero (i.e., no patients in the database
with a profile had VTE+), in which case, NVTE+ was set
at 0.25. Rationale for this adjustment is that no patient
profile can predict a zero risk of VTE.
Attribute
Matching
Pretest
Probability
Estimates. Pretest probability estimates for this article
were computed post hoc in the concatenated data set.
Data fields for each patient in all databases were
reduced by a study author (JAK) to exclude repeat
patients (patients who had separate presentations on
different days) and to yield the values for the required
predictor variables (attributes), the outcome status of
VTE(+) or VTE()), and a unique identifier for each
patient. Data were exported to produce a flat file for
each of the five databases, each containing one patient
per row with independent predictor and outcome variables in columns. Two of the five databases (Beam et
al.21 and partial data18 from coauthor MPT) were found
to have rows with at least one missing data element (<1
and 14%, respectively); these patients were excluded
from analysis. These five flat-files were then concatenated into one file containing all patients, and this file
was transferred to a second study author (CLJ) who
wrote the code in standard query language supplemented by visual basic script required to execute the
matching process described in the introduction and
demonstrated at http://www.pretestconsult.com/webtrial.15
Comparison to Wells Logistic Regression–based
Model and the PERC Rule. To provide a reference
model, we computed the value obtained from the
numeric scoring system of Wells et al.29 that was
derived from logistic regression. This model converted
the b coefficients for six predictor variables into
numeric values of 1, 1.5, or 3, as shown in Table 1.
Thus, the score can produce values of 0, 1, and then 0.5
steps up to a maximum of 12.5, allowing 25 unique pretest probability estimates. For additional comparison,
we tabulated the number of patients who had the eight
objective components of the PERC rule negative within
the database. The PERC rule is a decision rule,
designed to exclude PE in conjunction with the clinical
gestalt estimate that a patient has a low (<15%) pretest
probability for PE. If used independently of the clinical
gestalt, the eight objective components of the PERC
rule (age < 50 years; heart rate < 100 beats ⁄ min;
SaO2% < 95%; no prior VTE, or recent surgery ⁄ trauma,
estrogen use, hemoptysis, or current unilateral leg
swelling) result in a subpopulation of patients with
approximately a 1.4% outcome rate of PE.14,18 All five
databases planned in advance to collect the criteria for
the Wells score as well as the PERC criteria.
Method of Pretest Probability Assessment Using
Attribute Matching.
Pretest probability estimates
were generated based upon selection of patients with
the same values for the eight predictor variables
encoded in a totally separate reference database containing 12,567 patients who were assimilated from prospective PE diagnostic research studies conducted from
1996 to 2006. These are different patients than are in
the validation set in this report.14,25–28 Eight of the predictor variables were previously selected from over 25
candidate variables using a stepwise logistic regression
technique and are used in the pulmonary embolism
rule-out criteria (PERC) rule.14,18 We subsequently performed classification and regression tree analysis on
the entire reference data set, using identical software,
misclassification cost, and pruning settings as we have
previously described, and based upon these results, the
variables ‘‘pleuritic chest pain’’ and ‘‘dyspnea’’ were
added.15–17 The continuous variables were divided into
Data Analysis
For all diagnostic indexes, the criterion standard for disease positive was VTE(+) within 45 days of enrollment.
•
Table 1
Comparator Predictive Model From Wells et al.29
Predictor Variable
Clinical signs and symptoms of DVT
PE is No. 1 diagnosis, or equally likely
Heart rate > 100 beats ⁄ min?
Immobilization at least 3 days, or
surgery in the previous 4 weeks
Previous, objectively diagnosed PE or DVT?
Hemoptysis?
Malignancy with treatment within
6 mo, or palliative care
Numeric Value
3
3
1.5
1.5
1.5
1
1
DVT = deep vein thrombosis; PE = pulmonary embolism.
136
Kline et al.
The primary statistical analysis was to measure the
outcome frequency for VTE in patients categorized as
having a pretest probability for PE of <2.5% as predicted by attribute matching and to assess standard
diagnostic indexes from 2 · 2 contingency table analysis
with 95% confidence intervals (CIs) from the exact
binomial method. The overall diagnostic accuracy of
the pretest probability from attribute matching and for
the Wells score for all patients in the concatenated
database was tested by constructing a receiver operating characteristic curve and calculating the area under
the curve (AUC) using the Wilcoxon method. A 95% CI
was constructed using the standard error calculated
from the Delong method (StatsDirect, v 2.4.4, Cheshire,
England). We also plotted the percentage of the entire
population that had each pretest probability estimate
provided by attribute matching and each numeric value
from the Wells score (normalized to a scale of 0 to
100%).
RESULTS
After concatenation, the five data sets yielded 10,734
unique patients for whom the pretest probability for PE
was calculated using the attribute-matching system.
Table 2 summarizes the origin, inclusion criteria, diagnostic testing done, and outcome rate of VTE for the
five data sets. The rate of VTE(+) within 45 days of
•
ESTIMATION FOR PE: ATTRIBUTE MATCHING VS. WELLS SCORE
enrollment was 747 ⁄ 10,734 (7.0%, 95% CI = 6.5% to
7.5%). The majority of patients were female (6,914) and
white race (6,199). Most patients had either dyspnea
(6,103) or pleuritic chest pain (5,275) or both symptoms
(2,964). Table 3 provides descriptive data including
means for continuous variables and proportions for
ordinal data for the ten predictor variables used to
assess pretest probability.
Using attribute matching, the pretest probability estimate for PE was <2.5% in 2,975 ⁄ 10,734 patients (27.7%,
95% CI = 26.9% to 28.8%). Table 4 shows that the posttest probabilities (i.e., the observed outcome frequency)
for VTE(+) for patients with a pretest probability of
<2.5% ranged from zero to 5.1% between the five data
sets. The overall observed posterior probability of
VTE(+) for patients with an estimated pretest probability of PE of <2.5% was 48 ⁄ 2,975 or 1.6% (95% CI =
1.2% to 2.1%) with an associated diagnostic sensitivity
of 93.8% (95% CI = 91.6% to 95.2%), specificity of
29.3% (95% CI = 28.4% to 30.2%), and a LR– of 0.22
(95% CI = 0.17 to 0.29).
To compare the overall diagnostic accuracy of the
pretest probability estimates derived from attribute
matching with that of Wells logistic regression–based
score, we plotted the receiver operating characteristic
curves in Figure 1A. The AUCs were identical: the area
for attribute matching was 0.74 (95% CI = 0.72 to 0.75)
and the AUC for Wells score was 0.74 (95% CI = 0.72
Table 2
Description of the Five Studies That Were Used to Generate Pretest Probability Estimates
Study Characteristics
Source
Design
Hogg
et al.20
Single-center,
prospective
observational
Kline
et al.19
Single-center,
prospective,
diagnostic
accuracy
Multicenter,
prospective
observational
Beam
et al.23
Kline
et al.18
and M.P.
Thanà
Kline
et al.24
Total
Testing Done and Outcome
Patient
Origin
Manchester
Royal
Infirmary
ED, UK
Carolinas
Medical Center
ED, Charlotte,
NC
EDs in 12
U.S. cities
Single-center,
prospective
observational
Christchurch
Hospital ED,
NZ
Multicenter,
FDA-qualifying
study of a PE
diagnostic
device
EDs in 4
U.S. cities§
Inclusion
Criterion
CTPA
V⁄Q
Pleuritic
chest pain
425
425
90
168
0
22 (5.1%)
Suspected
PE, any
diagnostic
test
Suspected
PE, any
diagnostic
test
Suspected
PE, any
diagnostic
test
One symptom
and one sign
of PE, CTPA
ordered
174
143
104
4
12
24 (13.8%)
7,937
6,712 4,237
468
987
519 (6.5%)
1,810
1,774
362
185
209
137 (7.6%)
388
388
388
14
90
10,734
9,444
5,183
839
1,227
N
D-dimer
Ultrasound
VTE+*
45 (11.5%)
747 (7.0%)
CTPA = computerized tomographic pulmonary angiogram; DVT = deep venous thrombosis; N = sample size; PE = pulmonary
embolism; V ⁄ Q = ventilation-perfusion lung scan; VTE = venous thromboembolism.
*PE or DVT within 45 days of enrollment.
4,349 were quantitative D-dimer assays.
àPartial set of coauthor M.P. Than data previously published.
§Carolinas Medical Center, Charlotte, NC (JAK); Northwestern University, Chicago, IL (DMC); Baystate Medical Center, Springfield, MA (HAS); and Wake Forest University, Winston-Salem, NC (CDM).
ACAD EMERG MED • February 2010, Vol. 17, No. 2
•
www.aemj.org
Table 3
Clinical Characteristics of the Concatenated Data Set
Feature
Pleuritic chest pain
Dyspnea
Age, yr (mean, ±SD)
Pulse rate (mean, ±SD)
SaO2% (mean, ±SD)
Hemoptysis
Prior VTE
Recent surgery or trauma
Estrogen use
Unilateral leg swelling
Mean or N
SD or %
5,275
6,104
49
90
97
378
1,183
778
1,015
837
49%
57%
±18
±21
±4
3.5%
11.0%
7.2%
9.5%
7.8%
VTE = venous thromboembolism.
to 0.75). Figure 1B plots the percentage of the population as a function of the pretest probability estimates
for each prediction method. Figure 1B suggests that
both methods tend to cluster a large proportion of the
pretest probability estimates toward the zero end of the
range. The lowest possible Wells score (0) was
observed in 3,412 patients (31.7%), and 79 ⁄ 3,412 (2.3%,
95% CI = 1.8% to 2.9%) had a VTE outcome. Additionally, 2,742 patients had the eight objective components
of the negative PERC rule, and 38 ⁄ 2,742 (1.4%) had
VTE(+).
Table 5 provides data to directly compare the proportion of patients categorized as having a very low pretest
probability using attribute matching, Wells score, and
PERC rule negative and the observed outcome (‘‘falsenegative’’) rate for VTE(+) with each forecast. Although
the comparison between proportions showed several
significant differences, in general the overall proportions of patients with a very low pretest probability, as
well as their associated false-negative rates, were very
similar between the three methods.
To estimate the potential clinical impact of using a
pretest probability of <2.5% as a gating point with
respect to pulmonary vascular imaging, we calculated
the proportion of patients with an attribute matching
137
estimate of pretest probability of PE of <2.5% who had
a V ⁄ Q scan or CTPA performed. A total of 6,022 (56%)
of the 10,734 patients in the concatenated data set
underwent either a V ⁄ Q scan or a CTPA scan; 1,140
(19% of those imaged) had a pretest probability of
<2.5%; and 18 ⁄ 1,140 (1.6%) were VTE(+). To determine
the potential value of using attribute matching–derived
pretest probability and a quantitative D-dimer, we
extracted from the largest of the five databases, the
patients who had a pretest probability of <10% and a
quantitative D-dimer result that was below the suggested reference threshold of positive. This process
yielded 2,038 patients, of whom 8 of 2,038 (0.4%, 95%
CI = 0.1% to 0.6%) had VTE(+) within 45 days.21 Using
the Wells score in a similar way yielded 2,288 patients
with a Wells score of <4 and a negative quantitative
D-dimer, of whom, 12 of 2,288 (0.5%, 95% CI = 0.2 to
0.8%) were VTE(+) within 45 days.
DISCUSSION
This study tested the hypothesis that the attributematching method can accurately assess pretest probability of PE in symptomatic patients. PE represents a
significant concern to emergency medicine practitioners because if patients with PE are diagnosed and
treated promptly, their outcomes are generally good,
whereas if the patient is not properly diagnosed,
untreated PE can lead to sudden death, chronic cardiopulmonary dysfunction, and impaired quality of life.
However, overtesting for PE can be costly and expose
patients to harm from false-positive diagnoses, radiation, and intravenous contrast. The primary aim
focused on categorizing patients below 2.5% as a pretest probability threshold. The hypothetical construct
states that the decision to not test for PE will benefit
more patients than it will harm in this very low-risk
subgroup. However, this is only true if the method of
pretest probability assessment is accurate. From a data
set including 10,734 patients who were tested for
PE, the attribute-matching method categorized over
one-quarter as having a pretest probability lower than
Table 4
Outcome Rates of VTE
Source
Hogg et al.20
Kline and Hogg19
Beam et al.21
Kline et al.18 and M.P. Than*
Kline et al.22
Total
Pretest Probability Predicted
From Attribute Matching, %
Observed Outcome
VTE(+), n
Observed Outcome
VTE()), n
VTE(+) Rate, %
‡2.5
<2.5
‡2.5
<2.5
‡2.5
<2.5
‡2.5
<2.5
‡2.5
<2.5
‡2.5
<2.5
17
5
22
2
488
31
127
10
45
0
699
48
192
211
113
37
5259
2159
1191
482
305
38
7060
2927
8.1
2.3
16.3
5.1
8.5
1.4
9.7
2.0
12.9
0.0
9.0
1.6
VTE = venous thromboembolism.
*Partial set of author M.P. Than data previously published.
138
Kline et al.
Figure 1. (A) Receiver operating characteristic curve from the
concatenated data set containing 10,734 patients. The diagnostic test was the numeric value of the pretest probability estimate
for PE derived from attribute matching or the Wells criteria, and
the criterion standard for disease was any treated VTE within
45 days. (B) Frequency distribution of pretest probability estimates from attribute matching or the Wells score (normalized
to range from 0% to 100%). The y-axis represents the percentage of the entire population who had the pretest probability
denoted by the symbols (see insert legend). PE = pulmonary
embolism; VTE = venous thromboembolism.
2.5%. Within this subgroup, the actual incidence of
VTE (either DVT or PE) within 45 days was 1.6%, with
the upper limit of the 95% CI equal to 2.1%. This finding of an outcome rate of VTE less than 2.5% was relatively robust, observed in four of five of the test data
sets, representing patients enrolled from 15 hospitals in
•
ESTIMATION FOR PE: ATTRIBUTE MATCHING VS. WELLS SCORE
three countries with underlying prevalence of VTE
between 5 and 14%.
A survey of emergency physicians in 2005 suggested
that the most common tool used by physicians in diagnostic clinical assessment is that of unstructured pattern recognition.4 Unfortunately this technique is
subject to many biases (such as incorrect mathematical
weighting of clinical variables), which may lead to
flawed decision-making.30 Quantitative assessment of
pretest probability to a numeric value on a scale of 0 to
100% has advantages of precision, transparency, ability
to produce a discrete posttest probability when used
with likelihood ratios, and documentation to support
reimbursement using the Current Procedural Terminology (CPTIII) code 0185T: multivariate analysis of
patient-specific findings with quantifiable computer
probability assessment, including report.31 The attribute-matching method of quantifying the probability of
disease has been described and validated previously for
acute coronary syndrome; however, this is the first
report of its use for PE.15–17
We compared the diagnostic performance of attribute matching to the well-known and validated logistic
regression–based Wells score system. These two methods produced very similar overall diagnostic performance on receiver operating characteristic curve
analysis, and both produced extremely low false-negative rates when combined with a negative D-dimer.
However, when examined for the potential to be used
as a sole exclusionary device for PE, the Wells score of
zero yielded a higher observed false-negative rate
(2.3%, 95% CI = 1.8% to 2.9%) than an attribute matching–derived pretest probability of <2.5% (1.6% 95%
CI = 1.2% to 2.1%). The Wells score requires implicit or
subjective physician judgment for the ‘‘alternative diagnosis’’ question, whereas the attribute-matching system
uses all explicit variables. Previous work by Kabrhel et
al.32 has suggested that the majority of the diagnostic
power in the Wells score is derived from this subjective
component.
To provide some methodologic perspective, attribute
matching can be compared and contrasted to other
methods of prediction. Attribute matching differs from
scoring systems derived from logistic regression or
classification trees, which use predictor variables
expressed by an individual patient under consideration
to guide that patient into a predefined bin that predicts
a probability. This outcome probability is estimated
from knowledge (i.e., the magnitude of importance of
predictor or splitter variables) manifested by the
patients that were used to construct the logit equation
or decision tree. On the other hand, attribute matching
works in reverse fashion. Instead of placing the patient
under consideration into a bin or category, the computer program finds the patients from a reference database who ‘‘look like’’ the patient insofar as they are
identical on the binary predictor variables. Attribute
matching therefore shares methodologic similarity to
the k-nearest neighbor (k-NN) technique.33 The variation of k-NN, referred to as k-means clustering, has
particular similarity to attribute matching, because it
can include parametric and nonparametric predictors.34,35 The objective of k-means clustering is to find a
ACAD EMERG MED • February 2010, Vol. 17, No. 2
•
www.aemj.org
139
Table 5
Comparison of Three Methods to Estimate a Very Low Pretest Probability of PE
Method
Attribute matching <2.5%
Wells score = 0
PERC())
n
% of 10,734
95% CI for
Difference
From 28%
2,975
3,412
2,742
28
32
26
NA
2.9 to 5.3
)1.0 to )3.3
VTE(+), n
% With
VTE(+)
95% CI for
Difference
From 1.6%
48
79
38
1.6
2.3
1.4
NA
0.1 to 1.4
)0.9 to 0.4
PE = pulmonary embolism; PERC = pulmonary embolism rule-out criteria; VTE = venous thromboembolism.
centroid within a cluster of data points plotted in coordinate space, with addresses that could be determined
by the clinical attributes. The nearest neighbor(s), analogous to the match size with attribute matching, are
located relative to the centroid. The number of neighbors depends upon the number of centroids and the
domain size of the neighborhood, both of which must
be chosen by the user. In contrast, attribute matching
uses one predefined structured template, processed
through a singular selection process, to retrieve subjects from an array of whom all have the same values
of their attributes and only one of two class labels—disease present or absent. With attribute matching, the
user cannot control the number of patients matched
(i.e., the size of k), as this is determined by the size and
composition of the database. A feature that distinguishes attribute matching from other methods is that
its search matrix consists of a flat-file of records from
real patients previously evaluated for the specific disease in question, which allows transparency of how the
methods works.17
Our proposed next step will be a randomized trial
using attribute matching as an intervention to reduce
unnecessary diagnostic testing for PE, as we have done
using an attribute-matching method to forecast the
probability of acute coronary syndrome.17 The current
data support a clinical trial design that would provide
the pretest probability for PE derived from attribute
matching to the clinician with one of three recommendations: 1) a pretest probability of <2.5%, together with
appropriately low clinical suspicion, can be used in a
shared decision-making model to recommend no diagnostic test for PE; 2) a pretest probability of ‡2.5% but
<10% can justify ordering a high-sensitivity D-dimer,
which if negative at standard cutoff, should yield
an acceptably low posttest probability for PE; or
3) patients with a pretest probability of >10% should
proceed directly to pulmonary vascular imaging,
whereas patients with a pretest probability of >20%
should be considered for empiric anticoagulation in the
absence of contraindications, followed by immediate
pulmonary vascular imaging.11
LIMITATIONS
Limitations to the significance of the present report
include the absence of data showing device performance in actual practice. Although the computerized
method may have had a slight advantage over the use
of a Wells score of zero to exclude PE, this difference
may not be clinically important, and it did not have an
advantage over the PERC rule. A technical limitation of
attribute matching includes the size and composition of
the database. The attribute-matching method described
here uses 10 variables, one trichotomized and nine
dichotomized. Accordingly, the system can generate
1,536 unique profiles for any new patient to be sent to
the reference database for matching. It remains
unknown how many patients in the reference database
must be returned for any given profile to provide a
reliable estimate. Indeed, despite the large size of the
reference database, several profiles returned zero
patients, suggesting that patients with these patterns
will be rarely encountered, but otherwise providing little inference into the probability of the outcome VTE
with this uncommon presentation. To compensate for
low match sizes that predicted a zero probability, we
arbitrarily added 0.25 to numerator values that were
zero, such that the match size had to exceed 10 to produce a pretest probability below the suggested threshold of 2.5%.
CONCLUSIONS
The novel method of attribute matching categorized
27.7% of 10,734 ED patients as having a pretest probability of pulmonary embolism less than 2.5%. The actual
outcome of any venous thromboembolism within
45 days in this subgroup was 1.6% (95% CI = 1.2% to
2.1%). We conclude that the described method of attribute matching reliably forecasts a very low probability
of pulmonary embolism.
References
1. Ota M, Nakamura M, Yamada N, et al. Prognostic
significance of early diagnosis in acute pulmonary
thromboembolism with circulatory failure. Heart
Vessels. 2002; 17:7–11.
2. Kline JA, Hernandez J, Jones AE, et al. Prospective
study of the clinical features and outcomes of emergency department patients with delayed diagnosis
of pulmonary embolism. Acad Emerg Med. 2007;
14:592–8.
3. Studdert DM, Mello MM, Sage WM, et al. Defensive
medicine among high-risk specialist physicians in a
volatile malpractice environment. JAMA. 2005;
293:2609–17.
4. Runyon MS, Richman PB, Kline JA. Emergency
medicine practitioner knowledge and use of decision
140
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Kline et al.
rules for the evaluation of patients with suspected pulmonary embolism: variations by practice
setting and training level. Acad Emerg Med. 2007;
14:53–7.
Kabrhel C, Matts C, McNamara M, Katz J, Ptak T. A
highly sensitive ELISA D-dimer increases testing
but not diagnosis of pulmonary embolism. Acad
Emerg Med. 2006; 13:519–24.
Kline JA, Courtney DM, Beam DM, King M, Steuerwald M. Incidence and predictors of repeated computed tomographic pulmonary angiography in
emergency department patients. Ann Emerg Med.
2008; 54:41–8.
Einstein AJ, Henzlova MJ, Rajagopalan S. Estimating risk of cancer associated with radiation exposure from 64-slice computed tomography coronary
angiography. JAMA. 2007; 298:317–23.
Brenner DJ, Hall EJ. Computed tomography–an
increasing source of radiation exposure. N Engl J
Med. 2007; 357:2277–84.
Mitchell AM, Kline JA. Contrast nephropathy following computed tomography angiography of the
chest for pulmonary embolism in the emergency
department. J Thromb Haemost. 2007; 5:50–4.
Kline JA, Wells PS. Methodology for a rapid
protocol to rule out pulmonary embolism in the
emergency department. Ann Emerg Med. 2003;
42:266–75.
Hogg KE, Brown MD, Kline JA. Estimating the pretest probability to justify the empiric admininistration of heparin prior to pulmonary vascular
imaging for pulmonary embolism. Thromb Res.
2006; 118:547–53.
Roy PM, Colombet I, Durieux P, Chatellier G, Sors
H, Meyer G. Systematic review and meta-analysis
of strategies for the diagnosis of suspected pulmonary embolism. BMJ. 2005; 331:259. doi:10.1136/
bmj.331.7511.259
Pauker SG, Kassirer JP. The threshold approach to
clinical decision making. N Engl J Med. 1980;
302:1109–17.
Kline JA, Mitchell AM, Kabrhel C, Richman PB,
Courtney DM. Clinical criteria to prevent unnecessary diagnostic testing in emergency department
patients with suspected pulmonary embolism.
J Thromb Haemost. 2004; 2:1247–55.
Kline JA, Johnson CL, Pollack CV, et al. Pretest
probability assessment derived from attribute
matching. BMC Med Informat Dec Mak. 2005; 5:26–
37.
Mitchell AM, Garvey JL, Chandra A, Diercks D, Pollack CV, Kline JA. Prospective study of quantitative
pretest probability assessment for acute coronary
syndromes in patients evaluated in emergency
department chest pain units. Ann Emerg Med.
2006; 47:438–47.
Kline JA, Zeitouni R, Hernandez-Nino J, Jones AE.
Randomized trial of computerized quantitative pretest probability for low-risk chest pain patients:
impact on safety and resource use. Ann Emerg
Med. 2009; 53:727–5.
Kline JA, Courtney DM, Kabrhel C, et al. Prospective multicenter evaluation of the pulmonary embo-
•
ESTIMATION FOR PE: ATTRIBUTE MATCHING VS. WELLS SCORE
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
lism rule-out criteria. J Thromb Haemost. 2008;
6:772–80.
Kline JA, Hogg M. Measurement of expired carbon dioxide, oxygen and volume in conjunction
with pretest probability estimation as a method to
diagnose and exclude pulmonary venous thromboembolism. Clin Physiol Funct Imaging. 2006;
26:212–9.
Hogg KE, Dawson D, Mackway-Jones K. Outpatient
diagnosis of pulmonary embolism: the MIOPED
(Manchester Investigation Of Pulmonary Embolism
Diagnosis) study. Emerg Med J. 2006; 23:123–7.
Beam DM, Courtney DM, Kabrhel C, Moore CL,
Richman PB, Kline JA. Risk of thromboembolism
varies, depending on category of immobility in outpatients. Ann Emerg Med. 2009; 54:147–52.
Kline JA, Courtney DM, Miller CD, Smithline HA,
Lanier R, Hogg MM. Combined use of D-dimer and
the exhaled CO2 ⁄ O2 to exclude pulmonary embolism [abstract]. J Thromb Haemost. 2009; 7(Suppl
2):764.
Kline JA, Johnson CL, Webb WB, Runyon M. Prospective study of clinician-entered research data in
the emergency department using an internet-based
system after the HIPAA privacy rule. BMC Med
Inform Decis Mak. 2004. 2005; 4:17–25.
Kline JA, Mitchell AM, Runyon MS. Electronic
medical record review as a surrogate to telephone
follow-up to establish outcome for diagnostic
research studies in the emergency department.
Acad Emerg Med. 2005; 12:1127–33.
Kline JA, Meek S, Boudrow D, Warner D, Colucciello S. Use of the alveolar dead space fraction
(Vd ⁄ Vt) and plasma D-dimers to exclude acute pulmonary embolism in ambulatory patients. Acad
Emerg Med. 1997; 4:856–63.
Richman PB, Wood J, Kasper DM, et al. Contribution of indirect computed tomography venography
to computed tomography angiography of the chest
for the diagnosis of thromboembolic disease in two
United States emergency departments. J Thromb
Haemost. 2003; 1:652–7.
Kline JA, Israel EG, O’Neil BJ, Plewa MC, Portelli
DC. Diagnostic accuracy of a bedside D-dimer assay
and alveolar dead-space measurement for rapid
exclusion of pulmonary embolism: a multicenter
study. JAMA. 2001; 285:761–8.
Kline JA, Novobilski AJ, Kabrhel C, Richman P,
Courtney D. Derivation and validation of a Bayesian
network to predict pretest probability of venous
thromboembolism. Ann Emerg Med. 2005; 45:282–
90.
Wells PS, Anderson DR, Rodger M, et al. Derivation
of a simple clinical model to categorize patients
probability of pulmonary embolism: increasing the
models utility with the SimpliRED D-dimer. Thromb
Haemost. 2000; 83:416–20.
Kabrhel C, Camargo CA Jr, Goldhaber SZ. Clinical
gestalt and the diagnosis of pulmonary embolism:
does experience matter? Chest. 2005; 127:1627–30.
Thornworth W. CPT Assistant. 2009; 4:7.
Kabrhel C, McAfee AT, Goldhaber SZ. The contribution of the subjective component of the Canadian
ACAD EMERG MED • February 2010, Vol. 17, No. 2
www.aemj.org
141
Pulmonary Embolism Score to the overall score in
emergency department patients. Acad Emerg Med.
2005; 12:915–20.
33. Fix E, Hodges JL. Discriminatory analysis—nonparametric discrimination: consistency properties.
Technical Report Number 4, Project Number 21-49-
004, 261-279. Randolph Field, TX: USAF School of
Aviation Medicine, 1951.
34. Hartigan JA, Wong MA. A K-means clustering
algorithm. Appl Stat. 1979; 28:100–8.
35. Steinley D. K-means clustering: a half-century synthesis. Br J Math Stat Psychol. 2006; 59(Pt 1):1–34.
•