White Paper
Retrospective Study Comparison of
Arcadia Lab’s Galileo and Parascript®
AccuDetect® Galileo CAD systems in
screening environment
Takeaway
■
AccuDetect Galileo performs significantly better than Arcadia
Lab’s Galileo in terms of sensitivity and false positive rates, with
over 60% reduction in false positive rate per image.
■
Hospitals should pay close attention to performance details
when selecting CAD systems, especially false positive rates.
Summary
Computer-Aided Detection (CAD) tools are software programs
that analyze digitized or digital mammography images to find
features that are associated with breast cancer. The following
retrospective study compares the performance of Arcadia
Lab’s Galileo and Parascript AccuDetect Galileo CAD systems.
AccuDetet Galileo is an adaptation of Parascript’s CAD software
called AccuDetect for IMS Giotto Full Field Digital Mammography
(FFDM) system.
What’s in this Paper?
Investigators.............................................................................................................. 2
Background............................................................................................................... 2
Study Design............................................................................................................. 3
Results....................................................................................................................... 4
Conclusions............................................................................................................... 8
References................................................................................................................ 8
About University of Udine....................................................................................... 10
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
Investigators
Dr. Rossano Girometti (Researcher), Prof. Chiara Zuiani (Associate Professor), Dr. Anna
Linda (Staff Radiologist), Dr. Viviana Londero (Staff Radiologist), Dr. Anna Dal Col (Staff
Radiologist), Dr. Silvia De Stefani (Resident in Radiology), Dr. Luisa Battigelli (Resident
in Radiology), Eleonora Di Gaetano (MD, Resident in Radiology), Prof. Massimo
Bazzocchi (Chief of the Department).
All the Investigators work for the Istituto di Radiologia Diagnostica – Azienda
Ospedaliero Universitaria “S. Maria della Misericordia” di Udine, Via Colugna n.5033100 Udine.
University of Udine and their department of radiology conducted the first retrospective
clinical study of Parascript® AccuDetect® Galileo in November and December 2010.
Computer-Aided Detection
This was a nonprofit, no sponsorship study conducted at the Radiology Department of
University of Udine, Italy.
to find features that are
The Udine Hospital has a long experience with digital images processed by Full Field
Digital Mammography. The first mammography equipment was purchased from IMS
Giotto in 2003 (IMS Giotto 3DL) and another FFDM (IMS Giotto SD) was acquired by
the hospital in 2008. IMS DL and IMS SD systems are each set up with Acquisition
Work Station (AWS) and there are also 2 Review Work Stations so each image from both
FFDMs can be viewed on both workstations. For processing (RAW) and for presentation
images are stored in hospital’s picture archiving system. Udine Hospital is doing both
screening and diagnostic breast imaging, therefore occurrence of malignant cancers is
about 10-12 cancer cases per 1,000 women. 20-22 mammogram cases are completed
each day.
associated with breast
Background
cancer.
Computer-Aided Detection (CAD) tools are software programs that analyze digitized or
digital mammography images to find features that are associated with breast cancer.
The purpose of CAD is to “mark” suspected findings on mammographic images, in
order to help Radiologist in the detection of lesions1. CAD is more suited to detect than
to characterize/interpret breast lesions1. In other words, CAD marks potential malignant
lesions that Radiologist could have missed.
(CAD) tools are software
programs that analyze
digitized or digital
mammography images
CAD is an intensive matter for research in the scenario of mammographic screening
programs. Therefore, screening is a cost-effective strategy for earlier detection, and thus
treatment of breast cancer1-2. However, prevalence of disease is lower in the screening
scenario (no symptomatic patients) as compared to the clinical setting (different patient
categories, see below), leading to subtler and less obvious mammographic features,
more difficult to identify1,3. The rate of false-negative cases at screening (missed
cancers) is up to 25%, 27-70% of lesions being visible in retrospect according to
different experiences2. In order to overcome the problem of false-negatives, and to
improve sensitivity in the detection of breast cancer, two methods have been adopted:
(a) conventional double reading strategies, involving two radiologists; (b) a single
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
2
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
reading by radiologist followed by the use of CAD, at lesser human cost1,4-5. Despite the
large use of CAD, especially in the USA screening programs, analysis of these methods
is still incomplete because no randomized controlled trials have been performed
to assess changes in survival6. Since these studies are difficult to achieve, most
experiments use surrogate end-points such as: (a) CAD sensitivity and the proportion
of retrieved cancers (retrospective studies); (b) cancer detection rate and recall rate
due to further investigations in patients with suspect findings (prospective studies)7.
Ideally, a CAD-induced increase in cancer detection should be balanced by a small,
clinically acceptable increase in recall rate due to further investigations on false-positive
cases8.
Overall, available results make CAD a controversial matter. Some authors state that
CAD provides no significant increase in cancer detection rate in clinical practice, at
the expense of excessive recall rates7,9. Most evidence, on the contrary, supports its
use as a second reader in the screening scenario, because of an adequate intrinsic
sensitivity for cancer, especially for microcalcifications, and increased readers’
sensitivity up to 23.7%10. In any case, the reduction of specificity invariably associated
with the CAD use may result in no overall clinical benefit1. CAD is still less sensitive
to masses, especially in dense breasts. Moreover, readers who benefit from CAD are
mainly inexperienced ones1. Better CAD performances have been obtained by using
full-field digital mammography, which provides digital images, compared to screen-film
mammography11.
One emerging issue is9 the occurrence of many false positive marks created by CAD
systems during the screening process. The purpose of this study was to compare
two different CAD systems and determine if false positive rates differ significantly from
system to system.
Study Design
Two different CAD systems were used during the study. Both Computer Aided Detection (CAD) systems were designed for a computerized second read of mammography
images in order to help radiologists confirm that a suspicious area requires further
investigation. AccuDetect® Galileo is an adaptation of Parascript CAD solution called
AccuDetect for IMS Giotto platform.
Galileo is manufactured by Arcadia Lab S.r.L and was specifically developed for IMS
Giotto, a Bologna, Italy based manufacturer of FFDM systems.
The study was performed in collaboration with the Department of Radiology, University
of Udine and focused on standalone results for malignant calcification and mass detection.
The cases for the study were collected from two IMS Giotto Full Field Digital Mammography (FFDM) units and consequently retrieved from Udine Hospital’s picture archiving
system . There were 118 cancer cases (86 masses, 6 calcifications, 26 mixed) and
209 negative (normal) cases. The negative cases had a 12-month follow up radiology
report to mark them as normal. The truth data was obtained by a resident, experienced
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
3
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
radiologist who read each case and accompanying radiology and pathology reports.
The radiologist drew contours of mass lesions and outlined an area of malignant micro
calcifications on the screen of a color monitor. The senior (senior) radiologist then
reviewed the contours and confirmed their validity.
All retrospective normal and cancer cases were sent to Galileo and AccuDetect Galileo
software and the following results were calculated:
a) Percent of mass cancer cases correctly marked by Galileo,
b) Percent of calcification cancer cases correctly marked by Galileo,
c) Percent of mass cancer cases correctly marked by AccuDetect Galileo,
d) Percent of calcification cancer cases correctly marked by AccuDetect Galileo,
e) Percent of mass cancer images correctly marked by Galileo,
f) Percent of calcification cancer images correctly marked by Galileo,
g) Percent of mass cancer images correctly marked by AccuDetect Galileo,
h) Percent of calcification cancer images correctly marked by AccuDetect Galileo.
Figure 1. Parascript AccuDetect® Galileo CAD-system
Results
During the comparison as depicted in Table 1 (next page), AccuDetect Galileo achieved
higher sensitivity per case for all malignant lesions (p<0.012).
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
4
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
Table 1: True Positive (Sensitivity) versus False Positive rate per case and per image for AccuDetect
Galileo and Galileo CAD.
AccuDetect Galileo
achieved higher sensitivity
per case for all malignant
lesions (p<0.012).
Figure 1: FROC curve for AccuDetect Galileo (ADG) and performance data for Galileo at 0.8 False
Positive per case.
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
5
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
AccuDetect Galileo achieved higher sensitivity per image for all malignant lesions
(p<0.0001).
Table 2:Sensitivity and False Positive rates per image.
AccuDetect Galileo
achieved higher sensitivity
per image for all malignant
lesions (p<0.0001).
Figure 2: AccuDetect Galileo (ADG) FROC curve for Masses when compared to Galileo.
AccuDetect Galileo achieved more than 60% lower false positive rate per image for
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
6
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
malignant lesion hypothesis (p<0.001) when the operating point for AccuDetect Galileo
is set to match the sensitivity of Galileo.
Table 3: False positive rate for all cancers (Total FP) and separately for masses and calcifications.
AccuDetect Galileo
achieved more than 60%
lower false positive rate per
image for malignant lesion
hypothesis.
Figure 3: AccuDetect Galileo FROC curve for calcifications when compared to Galileo.
The paired Student’s T-test and Wilcoxon matched-pairs signed-rank test were used to
assess the statistical significance of AccuDetect Galileo results’ improvement for malignant lesions detection.
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
7
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
The paired Student’s T-test used to assess the statistical significance of false-positives
reduction by 60% achieved by AccuDetect Galileo. The results of the tests are summarized in the table below:
Table 4: The paired Student T-test and Wilcoxon test results.
Conclusions
AccuDetect Galileo has
significantly better overall
performance than Galileo
in terms of both sensitivity
and false positive rates.
During the study AccuDetect Galileo was compared against state of the art product –
Galileo CAD, which had shown better performance than widely used iCAD’s Second
Look Digital according to a 2007 study15. The comparison proved that:
AccuDetect Galileo has significantly better overall performance than Galileo in terms of
both sensitivity and false positive rates.
AccuDetect Galileo achieved more than 60% lower false positive rate per image for
malignant lesion hypothesis (p<0.001) when operating point for AccuDetect Galileo is
set to match the sensitivity of Galileo.
The study proved that not all CAD products are alike and in this comparison study, Parascript’s AccuDetect Galileo achieved a significant, 60% reduction in false positive rate
per image and better overall performance in sensitivity. Therefore, hospitals should pay
a close attention to performance details when selecting CAD systems, especially if they
care about a significant false mark reduction on reviewed mammograms.
References
1.Bazzocchi M et al. (2007) CAD systems for mammography: a real opportunity? A
review of the literature. Radiol Med 112:329-353
2.Vyborny CJ et al. (2000) Computer-aided detection and diagnosis of breast cancer.
Radiol Clin North Am 38(4): 725-740
3.Malich A. et al (2006) CAD for mammography: the technique, results, current role and
further developments. Eur Radiol 16:1449-1460
4.Chan HP et al. (1999) Improvement of radiologists’ characterization of mammographic masses by using computer-aided diagnosis: an ROC study. Radiology 212:817-827
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
8
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
5.Thurfjell EL et al. (1994) Benefit of independent double reading in a population-based
mammography screening program. Radiology 191: 241-244
6.Helvie M (2007) Improving mammographic interpretation: double reading and computer aided diagnosis. Radiol Clin North Am 45:801-811
7.Taylor P, Potts HWW (2008) Computer aids and human second reading as interventions in screening mammography: Two systematic reviews to compare effects on cancer detection and recall rate. EJC 44;798-807
8.Gur D et al. (2004) Changes in breast cancer detection and mammography recall
rates after the introduction of a computer-aided detection system. J Natl Cancer Inst
96:185-190
9.Fenton JJ et al. (2007) Influence of computer-aided detection on performance of
screening mammography. N Engl J Med; 356:1399-409
10.Ciatto S et al. (2004) Comparison of two commercial systems for computer-assisted detection (CAD) as an aid to interpreting screening mammograms. Radiol Med
107:480-488
11.Skaane P (2009) Studies comparing screen-film mammography and full-field digital
mammography in breast cancer screening: updated review. Acta Radiol 1:3-14
12.Brancato B et al. (2008) Does computer-aided detection (CAD) contribute to the
performance of digital mammography in a self-referred population? Breast Cancer Res
Treat 111:373-376
13.The JS et al. (2009) Detection of breast cancer with full-field digital mammography
and computeraided detection 192:337-340
14. Balleyguier C et al. (2007) BIRADS classification in mammography. Eur J Radiol
61:192-194
15. University of Udine, Department of Radiology (2007) CAD and mammography:
state of the art and clinical impact. Materials of XXXVI Annual Meeting of the Radiologists of the Alpe - Adria Region. Verona, 11.17.2007
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
9
parascript.com
Retrospective Study Comparison of Arcadia Lab’s Galileo and
Parascript® AccuDetect® Galileo CAD systems in screening environment
About University of Udine
The University of Udine was founded in 1978 as part of the reconstruction plan of Friuli
after the 1976 earthquake. Its aim was to provide the Friulian community with an independent center for advanced training in cultural and scientific studies. The University
currently has 10 faculties in: Agriculture, Economics, Engineering, Law, Foreign Languages, Education, Humanities, Medicine and Surgery, Veterinary Science and Mathematical, Physical and Natural Sciences. The University is actively involved in student and
staff exchange projects with universities within the EU and is currently engaged in close
collaboration with several universities from Eastern Europe and other non-EU countries.
Moreover the University participates in many research projects at national and international level. The present number of students enrolled at the University is approx. 17000.
About Parascript
The Parascript image analysis suite extracts meaningful information from images. Employing patented digital image analysis
and pattern recognition technologies, the Parascript image analysis suite improves decision quality in medical imaging, postal
and payment automation, fraud detection and forms processing operations. Parascript software processes billions of digital
images per year. Fortune 500 companies, postal operators, major government and financial institutions rely on Parascript
products. Organizations include the U.S. Postal Service, Bell + Howell, Fiserv, Elsag, Lockheed Martin, NCR, Siemens and
Burroughs. Parascript is online at http://www.parascript.com.
6273 Monarch Park Place, Longmont, CO 80503 USA
T: 303.381.3100 | toll free: 888.225.0169
F: 303.381.3101 | info@parascript.com
©2012 Parascript, LLC. All rights reserved.
10
parascript.com