7/21/2014
FDA QI Initiatives
Quantitative Imaging Initiatives:
Why, Who, What, and How?
FDA QI Initiatives
Nicholas Petrick
CDRH/OSEL/DIAM
U.S. Food and Drug Administration
FDA QI Initiatives
Outline of Talk
•
•
•
•
Motivation
Introduction
FDA data collection
FDA QI studies
–
–
–
–
Estimating minimal detectable change
Volume estimation of low-contrast lesions
Impact of reconstruction slice overlap
QIBA Joint Projects:
• Inter-comparison of lesion sizing techniques
• Inter-comparison of volume estimation software tools
– Material characterization using dual-energy CT
• Summary
7/21/2014
2
FDA QI Initiatives
The FDA Team
•
•
•
•
•
•
Marios Gavrielides
Qin Li
Rongping Zeng
Berkman Sahiner
Kyle Myers
Qi Gong
• Talk today focusing on their research efforts
7/21/2014
3
1
7/21/2014
FDA QI Initiatives
Project Motivation
• Response to therapy is integral part of patient
management
• Imaging is increasingly being used for assessing
tumor response
– Limited by uncertainty in the QI measurement
7/21/2014
4
FDA QI Initiatives
Why FDA?
• QIBs are growing area of interest
• Drugs (CDER)
– Qualifying QIB for use in drug trials
• Devices (CDRH)
– Potential for specific QIB device claims
– Move towards having meaningful error bars on QI
measurements
7/21/2014
5
FDA QI Initiatives
Project Goal
• Goal
– Develop a paradigm for assessing specific QI
tool and biomarker claims
• Help facilitate transformation of radiology into a more
quantitative science
• Methods
– Well-controlled anthropomorphic phantom
studies
• Measure effect sizes
• Identify influential imaging parameters
• Determine smallest measurable change
7/21/2014
7
2
7/21/2014
FDA QI Initiatives
Why Phantom studies?
• Allows the collection of large no. of CT scan &
repeat CT scans with well-defined
characteristics
• Practical
– FDA doesn’t have direct access to clinical cases
7/21/2014
8
FDA QI Initiatives
Phase 1: Lung Nodule Sizing
• Investigate sizing of lung nodules using an
anthropomorphic thorax phantom
• Acquire CT data across a range of
– Acquisition parameters
– Nodule characteristics
7/21/2014
10
FDA QI Initiatives
Anthropomorphic Thorax Phantom
7/21/2014
11
3
7/21/2014
FDA QI Initiatives
Synthetic Nodule Characteristics
Spherical
Spiculated
Elliptical
Mixed density
7/21/2014
Lobulated
12
FDA QI Initiatives
Synthetic Nodules
• Sizes
5-40 mm
• Densities
-630 HU, -10 HU,
+100 HU
• Locations
– Attached
– Unattached
7/21/2014
13
FDA QI Initiatives
Inserting the Nodules
7/21/2014
14
4
7/21/2014
FDA QI Initiatives
CT of phantom with embedded nodules
FDA-050-4746-196.gif
7/21/2014
15
FDA QI Initiatives
CT phantom data collection
• Example Layout: 5 & 8 mm aspherical nodules
8 mm
5 mm
8 mm
5 mm
8 mm
5 mm
5 mm
5 mm
8 mm
8 mm
8 mm
5 mm
+100 HU
-630 HU
7/21/2014
16
FDA QI Initiatives
CT phantom data collection
Nodule Layout
Exposure (120 KVp)
20 mAs
100 mAs
200 mAs
Pitch
Standard
1.2
High
0.9
16x0.75 mm
16x1.5 mm
x10
Collimation
Slice Thickness
Reconstruction Kernel
7/21/2014
0.75
mm
1.5
mm
Detail
3.0
mm
2.0
mm
3.0
mm
Medium
5.0
mm
Recon
17
5
7/21/2014
FDA QI Initiatives
Public Release of FDA Phantom Data
• The Cancer Imaging Archive (TCIA)
– http://cancerimagingarchive.net/
– >1900 series available, 720 additional series in
process, more to come
• Collection: Phantom FDA
7/21/2014
18
FDA QI Initiatives
Phase 2: Hepatic Lesion Sizing
• Joint QIBA/Columbia/FDA project
– Lead by Binsheng Zhao, Colombia University
• Investigate sizing of soft tissue lesion using an
anthropomorphic abdominal phantom
– Acquire CT data across a range of
• Acquisition parameters
• Lesion characteristics
7/21/2014
19
FDA QI Initiatives
Semi-anthropomorphic Liver Phantom
• Two liver inserts
– Arterial phase
• Liver: 80HU;
• Lesions: up to 120HU
– Portal venous phase
• Liver: 110HU;
• Lesions: 45HU - 90HU
8
c 5
m c
m
Arterial Phase
8
c 5
m c
m
Portal Venous
• Nodules
– Spherical
– Ellipsoid
– Lobulated
• Data collection
– Starting in Fall, 2014
7/21/2014
20
6
7/21/2014
FDA QI Initiatives
Phase 3: Cardiac Vessel Calcium Scoring
• Joint FDA/NIH project
• Investigate material characterization using
single- and dual-energy CT
– Joint estimation of cardiac calcium size, density,
texture
– Phantom currently under development
7/21/2014
21
FDA QI Initiatives
FDA QI studies
Sub-project 1
Estimating minimal detectable change
in lung nodules with CT
7/21/2014
22
FDA QI Initiatives
Purpose
• To determine the minimum detectable change in
CT lung nodule volume
– As a function of lesion size
• Clinical question
– How early can true change in nodule volume be
detected with CT
*Gavrielides et al., Academic Radiology, 20: 1364-1370, 2013
7/21/2014
23
7
7/21/2014
FDA QI Initiatives
Study Design
• Synthetic nodules
– 4 shapes
– 4 sizes
• 5, 8, 9, 10 mm
7/21/2014
24
FDA QI Initiatives
Study Design
• Nodules embedded
within phantom
vascular structure of
lung
Styrofoam
buffer
• Nodules don’t directly
touching vasculature
7/21/2014
25
FDA QI Initiatives
Study Design
• Image collection protocols
– Thin slice protocol
• 0.75-3.0 mm reconstructions
– Thick slice protocol
• 2.0-5.0 mm reconstructions
• 10 repeats for each configuration
7/21/2014
26
8
7/21/2014
FDA QI Initiatives
Study Design
• Estimator
– Matched-filter algorithm*
• Minimizing cost between images of target and bank of 3D templates
• Cast determining minimum detectable change as a
detection problem
– Area under the ROC curve (AUC) used as detectability metric
• Higher AUC
Higher Detectability
*Gavrielides et al., IEEE-TMI, 29:1795-1807, 2010
7/21/2014
30
FDA QI Initiatives
Number of measurements
Histograms
300
5mm
8mm
9mm
10mm
200
100
0
0
100
200
300
400
Volume Estimates (mm3)
500
600
• 5→8 mm: ∆= 316%, AUC5,8=1.0
• 8→9 mm: ∆= 42%, AUC8,9=0.995
• 9→10 mm: ∆= 37%, AUC9,10=0.999
7/21/2014
31
FDA QI Initiatives
Number of measurements
Histograms
300
9
𝑝𝐵𝑎𝑠𝑒
5mm
8mm
9mm
10mm
200
C
9
𝑝𝑆ℎ𝑖𝑓𝑡
𝑐
100
0
0
100
200
300
400
500
600
Volume Estimates (mm 3)
7/21/2014
33
9
7/21/2014
FDA QI Initiatives
𝝈
𝝁
CV
as function of shifted volume
0.2
Coefficient of variation
5mm
0.15
8mm
0.1
9mm
10mm
0.05
0
100
200
300
400
Volume Estimates (mm3)
500
600
7/21/2014
34
FDA QI Initiatives
Results: Increase from baseline
70
70
5mm
8mm Thin slice protocol
9mm
50
40
19%
17%
15%
30
20
10
0
5mm
protocol
8mm Thick slice
46%
9mm
60
Min % increase
Min % increase
60
50
40
19%
17%
30
20
10
0.6
0.7
0.8
AUC
0.9
1
0
0.6
0.7
• AUC= 0.95
• AUC= 0.95
– 9 mm: 15%
– 8 mm: 19%
– 5 mm: 17%
– 9 mm: 17%
– 8 mm: 19%
– 5 mm: 46%
0.8
AUC
0.9
7/21/2014
1
35
FDA QI Initiatives
Results
• CT imaging can detect small early changes in
nodule volumes
– Across a range of nodule shapes/size
• Sub-centimeter nodules
– Across a range of acquisition/recon parameters
• Potential lower bound on achievable
performance
– Expect increase in detectable change in clinical scans
7/21/2014
38
10
7/21/2014
FDA QI Initiatives
FDA QI studies
Sub-project 2
Volume estimation of low-contrast
lesions with CT
7/21/2014
39
FDA QI Initiatives
Purpose
• To study the volume estimation performance for
lesions with object-to-background contrast less
than 50HU
– Simulating soft tissue hepatic lesions
• To understand the relationship among
performances obtained from phantom study,
simulation and theoretical analyses
– I’ll concentrate on phantom/simulation results
7/21/2014
*Li et al., Physics in Medicine and Biology (Submitted), 2014.
40
FDA QI Initiatives
Study Design
• Synthetic nodules
– Spherical lesions
Lesion 1
Diameter
Lesion 2
Lesion 3
8 mm
9 mm
Density
104 HU
118 HU
27 HU
Contrast
31 HU
45 HU
-46 HU
7/21/2014
9 mm
41
11
7/21/2014
FDA QI Initiatives
Study Design
• Background
– Gelatin-background ~73HU (~soft tissues) in a
cylindrical container
Lesion 1
Lesion 3
Lesion 1
Lesion 3
Lesion 2
Lesion 2
7/21/2014
Setup
Imaging
42
FDA QI Initiatives
Study Design
1: 8 mm, 31 HU
3: 9 mm, -46 HU
2: 9 mm, 45 HU
• Central slice of Lesions 1-3
– Air pocket found in Lesion 3 due to production flaw.
7/21/2014
43
FDA QI Initiatives
Study Design
• Image collection
protocols
Phantom
20 mAs
– 10 repeats for each
configuration
100 mAs
16×0.75 mm
Acquisition
• Estimator
– Improved matchedfilter algorithm
0.75 mm
Reconstruction
7/21/2014
1.5 mm
Detailed
44
12
7/21/2014
FDA QI Initiatives
Analysis
• Percentage error
• 𝑃𝐸 =
𝑉𝑜𝑙𝐸𝑠𝑡 −𝑉𝑜𝑙𝑇𝑟𝑢𝑒
𝑉𝑜𝑙𝑇𝑟𝑢𝑒
× 100%
• Percent Bias
• 𝑃𝐵 = 𝑃𝐸
• Standard deviation of PE
• 𝑆𝑃𝐸 = stdev(𝑃𝐸)
7/21/2014
45
FDA QI Initiatives
Results: Phantom
0.8 mm slice protocol
(PE ± SPE)
Exposure
1.5 mm slice protocol
(PE ± SPE)
Lesion 1 Lesion 2 Lesion 3 Lesion 1 Lesion 2 Lesion 3
20 mAs
3.6±5.6 -0.5±4.4 2.1±4.8 11.9±5.6 11.1±4.8 4.6±5.7
100 mAs
4.3±2.4
0.3±2.3
5.0±2.3
5.0±2.3
6.8±1.5
1.9±1.7
• Bias:
– Generally higher bias for thicker slices
– Exposure, contrast results, no clear trend
• Variance
– SPE falls with increased exposure with a reduction factor of 1.9-3.0
– SPEs falls as contrast increase
– Slice thickness: no clear trend
7/21/2014
46
phantom scan
Hann 100
1120
1120
10
10
1100
1100
20
20
1080
1080
30
40
30
FDA QI Initiatives
10
20
30
1060
1060
40
40
10
20
30
40
Results: Phantom/Simulation Comparison
Phantom
scan
phantom scan
Hann 75
Simulated
Hann 55
1120
10
10
20
20
30
30
40
40
20
30
1080
1060
40
10
40
4.5
20
30
40
3.5
20
30
Hann 75
1120
10
1
20
0.5
30
0
40
10
1100
1.5
20
1080
2
thin
30
40
• Hann
Good
agreement
55
found among
1120
performances in terms
1100
of SPE
1080
1060
3
4
8mm,thin
8mm,thick
Lesion
2
Lesion
10
201 9mm,thin
30 Lesion
40
thin
20
30
1060
1
1120
• Virtual helical CT
1100
developed to produce
1080
images with properties
1060
similar to real scans
10
40
CRLB
simu
phantom
**
3
2.5
2
10
**
4
Std(PE) (%)
10
1100
Hann 100
1120
10
1100
20
1080
30
1060
40
1120
10
1100
20
1080
30
1060
thick
19mm,thick
Lesion 2
thick
7/21/2014SPEs with error bar (95% CI) for simulation and phantom studies
40
10
20
30
40
48
13
7/21/2014
FDA QI Initiatives
Summary
• FDA anthropomorphic CT phantom data
– Range of nodule, acquisition, reconstruction params
– Useful for understanding source & magnitude of QIB
measurement error
• FDA research expected to result in a viable
assessment strategy
– Support evaluation of technical performance of QIB
claims
– Support QIBA profile development and compliance
procedures
7/21/2014
50
14