Training and Clinical Best Practices
for Using CAD Systems
Ronald M. Summers, M.D., Ph.D.
Senior Investigator
Imaging Biomarkers and Computer-Aided Diagnosis Laboratory
Radiology and Imaging Sciences
NIH Clinical Center
Bethesda, MD
Financial Disclosure
• Patent royalties and CRADA support from
iCAD Medical, Inc.
• License fees from Translational Sciences
Corporation
Disclaimer
• Opinions expressed herein are not
necessarily those of HHS or NIH
1
Group 4 Members
•
•
•
•
•
•
•
•
•
•
Matthew Friedman, Georgetown Univ.
Jesse Lin, FUJIFILM Medical Systems
Ben Shih-Chung Lo, Georgetown Univ.
Julian Marshall, Hologic, Inc.
Pat Milbank
Stephen Vastagh, MITA
Samuel G. Armato III, University of Chicago
Heang-Ping Chan, University of Michigan
Berkman Sahiner, CDRH, FDA
Nicholas Petrick, CDRH, FDA
Learning Objectives
• Understand benefits of PACS integration of CAD.
• Identify potential issues with off-label use of CAD.
• Understand the importance of user training
relating to CAD devices.
• Learn about areas of CAD user training and QA
that could benefit from further research.
What is the Problem We are Trying to
Address?
• Improvement in radiologist performance
with CAD is poor, even when
the CAD performs well in the lab
• Examples from the recent literature:
• Fenton et al.
• Dachman et al.
2
The Problem
• 684 956 women who received more than 1.6
million film-screen mammograms
• “CAD use … is associated with decreased
specificity but not with improvement in the
detection rate … of invasive breast cancer”
• CAD led to increased detection of DCIS
• Were radiologists using CAD as a crutch & not
reviewing the mammograms as diligently?
Fenton et al., JNCI 2011
The Problem
• 100 CTC cases, 19 radiologist readers
• CAD sensitivity in the lab: 91.8%
• Sensitivity of the radiologists
without and with CAD: 46.6%, 52.1%
Dachman et al., Radiology 2010
Overview
• To achieve the highest possible benefit from CAD
systems, best practices are required for clinical
implementation and use of CAD
• Summary of opinions of the
AAPM CAD Subcommittee
3
CAD best practices – Important issues
• Importance of training in the use of CAD devices
• Pitfalls of off-label use
• Research opportunities
Rationale for CAD
• Reduce interobserver variability
•
•
Level the playing field
Less trained can perform closer to experts
• Reduce perceptual error
•
•
Find abnormalities missed by radiologists
Improve reproducibility
User training - Topics
• Importance of understanding the effect of
improper use of CAD on sensitivity and specificity
• Frequency and type of training
• Training the vendors: Feedback on CAD
performance
• Storing CAD marks long term for training/auditing
purposes
4
CAD best practices – User training
• Training is important for reading images without
CAD:
Residencies, fellowships, special certifications
• Inexperienced readers benefit more from CAD
• Training for reading images with CAD
• Little published research
• Educate the physicians on the intended use of the
CAD
CAD User training – It Helps!
• Mammography CAD in the United Kingdom
National Breast Screening Program
• FJ Gilbert et al., Radiology 2006
• CADET II Trial, FJ Gilbert et al., NEJM 2008
CAD User training
• 2 month training of radiologists
• Initial training by vendor
• Practice using 6 sets of 75-100 cases
• Truth provide after each set to improve performance
• Cancer prevalence progressively reduced
from 25% to 5%
• Results
• Single reader with CAD had comparable sensitivity to
double reading without CAD
5
CAD User training – Other Results
• Short-term feedback may not help
• Lung nodules on CXR with CAD,
De Boo
et al., Eur Radiol 2012
• Focused training with CAD may reduce training
requirements
• Polyps on CTC,
Taylor
et al., Br J Radiol 2008
Training Occurs During CAD Use
• Learning curve for radiologist use of CAD
changes over the course of a year
• Breast radiologists initially doubled their recall rate
when using CAD (6.2% to 13.4%), but over a
year, the recall rate decreased to near the level
before CAD implementation (6.75%)
• Dean and Ilvento AJR 2006
User training – Important Questions
• How can we encourage radiologists to spend time
getting trained?
• Does the absence of training impair CAD
performance in the clinic?
• Is there an association between radiologist
performance and attitudes towards CAD before
and after training?
6
User training – What?
• Sensitivity and average false positives per image
• Characteristics of false negatives and false
positives (knowledge of latter benefits efficiency)
User training – What?
• Unique strengths and weaknesses of a particular
CAD (target lesions; susceptibility to artifacts)
• Meaning of the various CAD marks
• Absence of CAD mark should not discourage recall
• Learning curve for use of CAD in actual clinical use
User training – When?
• At initial installation
• Annually or via CME
• At time of CAD updates or modifications
• During residency (currently implemented for
breast imaging rotations under ACR guidelines)
7
User training – How?
• Web-based
• One-on-one
• Case-based
User training – Implementation
• Case-based examples of changes in CAD
behavior after updates or modifications
• Technologist training, especially if radiation dose
or patient positioning affect CAD
Training the CAD Vendor
• Continuous feedback about missed lesions, false
positives
• Recording callbacks that have no CAD marks,
CAD marks that cause additional callbacks,
recalling CAD marks on prior exams when current
exam is being read
8
Storing CAD Marks
• Controversial
• Medicolegal aspects have been emphasized,
patient benefits have not
• CAD has been used in the courtroom and has
helped defendant radiologists
• Brenner
et al., AJR 2006
Storing CAD Marks - Benefits
• Facilitate automatic monitoring of the stability of
the CAD system performance over time
• Help the radiologist learn the characteristics of
dismissed CAD marks on prior exams that turn
out to be true lesions in current exams
• Enable CAD system to use previous readings to
improve its current performance
• Enables CAD prospective performance evaluation
in large populations
Storing CAD Marks - Implementation
• Some CAD operates on raw data or processed
images not shown to the radiologist
• Example: ultrathin CT images for CTC
• Such raw data would also need to be stored along
with CAD marks
• Record CAD metadata in DICOM header or using
Annotation Imaging Markup (AIM)
9
Off-label Use
• Use of device in a manner that is not specifically
stated in the FDA-approved indications for use
• Physicians may use any FDA-approved product
off-label according to their professional judgment
concerning the needs of their patients
• Potential problem: CAD used off-label to improve
productivity rather than sensitivity
• Second reader > Concurrent reader > First reader
Off-label Use
• CAD should be used on ALL cases,
not just selected ones, since radiologist cannot
know in advance which cases will be false
negatives without CAD
Colon Cancer in Americans
•
2nd leading cause of
cancer death
•
131,000 diagnosed
annually
•
•
55,000 annual mortality
6% will develop colon
cancer during their
lifetime (40% die)
Image source: Wikipedia
10
1.4 cm polyp in transverse colon found by CAD
Reading Paradigm
• First read
• Concurrent read
• Second read
First Read
• Radiologist reviews only CAD results,
not entire colon
• Fast interpretation time
• High specificity
• Lower sensitivity
• Presently unlikely to be used clinically
• Sometimes used for mammography CAD
(microcalcifications)
11
Concurrent Read
• CAD marks visible during radiologist’s
primary image interpretation
• Radiologist evaluates CAD marks as they
appear in the image
Concurrent Read
• Reduced interpretation time
• CAD marks may distract radiologist from
other findings in vicinity
(“satisfaction of search” error)
2nd Reader
• Radiologist reviews images and arrives at
preliminary diagnosis
• Then evaluates CAD marks,
revises preliminary diagnosis to arrive at
final diagnosis
• Used for mammography CAD (masses)
12
2nd Reader
• Highest sensitivity (↑ 9 - 25%)
• Lowest specificity (↓ 2 - 14%)
• Longest interpretation times (↑ 2 - 4 min.)
CAD as 2nd Reader
7 mm TA in rectum found by 3 readers with CAD
N. Petrick et al., Radiology 2008
Discouraging Off-label Use
• Hard to see how off-label use of CAD benefits
patients rather than physicians
• Record or track reading behavior
• Record radiologist’s findings prior to displaying
CAD output (auditing; RIS integration of CAD)
• Control the workflow by modifying the display
protocols (requires PACS integration of CAD)
13
PACS Integration of CAD
• Improves usability
• Increases reader sensitivity with minimal impact
on interpretation time
• Lung nodule study,
Bogoni
et al. J Digital Imaging 2012
• Integration hampered by deficiencies in IHE
• Welter
et al. Comput Methods Programs Biomed.
2011
• Le et al. IJCARS 2009
Standardization
• File formats and reported data elements
• APIs for PACS/RIS integration of CAD
• Quality control
• Reporting
• Limited to encourage buy-in from vendor
community
CAD best practices –
Research opportunities
• Assessment of radiologist performance and
training methods
• Monitoring CAD performance changes and
effectiveness over time using large electronic
health records
• Funding opportunities
14
Research opportunities - Training
• Number of cases
• Generalizability: local or global cases
• Re-training requirements
• Overcalls, recall rates, FNs, FPs, efficiency,
subtle lesions, pitfalls, radiation exposure
• Patient preparation and acquisition-specific issues
• Reading paradigms
• Content-based image retrieval
Research opportunities
• Effect of CAD marks (shape, type) on
performance
• Human perception research
• Estimating the likelihood of malignancy of a lesion
• Automated lesion size measurement and
segmentation
Distributed human intelligence for
observer performance assessments
ROC curve - All Detections
1
0.9
0.8
Sensitivity
0.7
0.6
0.5
0.4
0.3
Static
Video
Radiologist Static
Radiologist Video
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(1-Specificity)
Summers et al. MIPS Conference 2011;
McKenna et al., MedIA 2012
15
Future Directions
• CAD keeps improving
• Reimbursements on downward trend
• Physician extenders will increasingly provide care
• First-read paradigm will ultimately prevail
• Satisfaction of search errors (i.e., the Fenton
paper) could become more prevalent
Conclusions
• Use of best practices optimizes benefit of CAD
• Areas of improvement to be had in training users,
discouraging off-label use
• Consensus from members of the CAD community
• Has to be done in a way that is not punitive …
• … but that keeps a sharp focus on benefiting the
patient
To Learn More …
www.cc.nih.gov/drd/summers.html
Acknowledgment:
Viatronix provided V3D
visualization software
16