AAPM 2004, Pittsburgh, PA
TG18 Display Evaluation
Demonstration Workshop
Director: Ehsan Samei, PhD, DABR
Duke University
samei@duke.edu
Workshop format
1:30
2:10
2:25
Lecture
Moderated review of workstations
Group/individual hands-on rotations
3:20
BREAK
4:00
4:30
4:45
Lecture (repeated)
Moderated review of workstations
Group/individual hands-on rotations
Workshop participants
Ehsan Samei
Hans Roehrig
Duke University Medical Center
University of Arizona, Tucson
Manufacturers:
1. BarcoView, LLC
2. Data Ray Corp.
3. Image Smiths, Inc.
4. Image Systems Corp.
5. National Display Systems
6. Planar Systems, Inc.
7. Totoku Electric Co., Ltd.
Rationale
Electronic displays, a key component of digital
imaging
Image
Processing
Imaging Device
1
1
Rationale
Soft-copy display quality is susceptible to
variations in hardware, calibration, and to
degradations over time
Poor display quality leads to reduced overall
effectiveness of diagnostic imaging,
misdiagnosis (lawsuits), quality mismatch leading
to inconsistent clinical decisions
Monitoring and maintaining display quality is
essential for quality patient care
TG18 Document
In final editing phase
Reviews
– current display metrology standards
– current display technologies
Describes
AAPM TG18
AAPM task group aimed to define standard
and objective methods for
Assessment of display performance for
medical imaging systems
Version 9, Oct. 2002
Version 10, Aug. 2004
IEC Display WG 36
TG18 => IEC
Formed in 2003 to develop an
international standard
– Prerequisites and tools for display assessment
Public draft soliciting comments
– Assessment methods
expected before the end of 2004
Defines
– Acceptance testing and quality control criteria
2
2
Current Display
Evaluation Standards
Basics
Luminance
Diverse – Not standardized
– Visible light emitting from a surface into a unit solid
Do not cover all aspects of display performance
angle per second
Often do not indicate acceptance criteria
Either too subjective or too complicated
– SI unit: lumens/steradian/m2 , cd/m2 or nit
Not yet fully endorsed by the medical community
– Conventional unit: foot-lambert = 3.43 cd/m2
Basics
Illuminance
– Quantity of visible light striking a surface per second
– SI unit: lumens/m2 or lux
5 Initial steps
1.
2.
3.
4.
5.
Display class
The purpose of testing
Availability of test patterns
Availability of required instrumentation
Prerequisite steps
– 1 lux striking a perfect white diffuser surface => 1/
nit
3
3
2. Purpose of Testing
1. Display Class
Primary (i.e., diagnostic)
– Used for primary interpretation
Secondary (i.e., clinical)
– Used for secondary view of medical images
– Acquisition QC displays
1. Acceptance testing (by medical physicist)
–
Includes all key quantitative display characteristics (~1 hr)
2. Annual compliance evaluation (by medical physicist)
–
Redoing of almost all acceptance tests (~30-45 min)
3. Monthly/quarterly evaluation (by QC technician)
–
Mix of quantitative and visual tests (~ 15-20 minutes)
–
Results logged for consistency checks
4. Daily evaluation (by user)
–
Checks with a login test pattern (<1 min)
–
Clear instructions about who to call
3. Test Patterns
4. Instrumentation
Luminance meter
Illuminance meter
Magnifier
Mask
Light source
Other devices
– High-grade CCD camera
4
4
5. Prerequisites
Display placement
Display placement
Absence of strong magnetic fields
Clean face-plate
Ambient lighting setup
Display calibration to GSDF
Ambient light flowchart
Set
L’max
Desired
LR’
Measured
Rd
No specular reflection
Reflection, diffuse
Illuminator
L’min = L’max/LR’
Lamb
Light sources
Telescopic
Luminance-meter
~5 L’min
Display
E
Lamb /Rd
Rd= Lr / I
5
5
Reflection criteria
Testing methods
100
Rd = 0.001 /sr
Maximum allowable illuminance (lux)
Lmin = 1.0 cd/m2
Rd = 0.002 /sr
Rd = 0.004 /sr
Rd = 0.008 /sr
Rd = 0.016 /sr
20
Rd = 0.032 /sr
10
Rd = 0.064 /sr
1
0.001
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
9. Other tests
0.004
0.01
0.1
Specular reflection coefficient (Rs)
Testing methods
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
Geometry
GD =
max(di-dmin)/dmin
GD < 2%
9. Other tests
TG18-QC
6
6
Geometry
Geometry
Image clips:
– Whole pattern visible w/o clips
Clocking distortion:
– All edge line patterns w/o distortion
Local distortion:
– 2 directional measurement per quadrant
– Max deviation, max(di-dmin)/dmin < 2% (5% secondary)
Global distortion:
– Directional measures added together
– Max deviation <2% (5% secondary)
TG18-GD / IEC-GD
Testing methods
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
9. Other tests
Visual assessment
Specular:
– High-contrast white
objects should not
be visible
Diffuse:
– Equivalent visibility
threshold of lowcontrast targets in a
dark field with and
without ambient
lighting
TG18-AD
7
7
Reflection, specular
Light sources
Reflection criteria
The maximum ambient lighting should be such that
– specular reflections of white objects (with 90%
reflection) at low luminance levels is < contrast threshold
E < ( /0.9) x (Ct Lmin / Rs)
Display
Telescopic
Luminance-meter
– contrast reduction due to diffuse reflection at low
luminance levels < 20%
E < 0.25 Lmin / Rd
Rs= Lr / Lo
Testing methods
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
9. Other tests
Lamb
Measured
OR
Lamb = ERd
Display off
8
8
Luminance response
Luminance response
L’max = Lmax+Lamb
> 170 cd/m2 for primary displays
> 100 cd/m2 for secondary displays
~ 5% of desired value
L’min = Lmin+Lamb
> 5Lamb
~ 10% of desired value
LR’ = L’max/L’min
> 250 for primary displays
> 100 for secondary displays
TG18-LN
Luminance response
Luminance response
1000
Luminance, cd/m 2
– Lmin and Lmax mapped to corresponding GSDF JND index
(J)
100
280
cd/m
10
1.5
cd/m2
2
J min = J ( L min )
J max = J ( L max )
J = J max
J min
Jp =
P
J
– JNDs ascertained corresponding to luminance values by
linearly mapping the JND and pixel value (P) spaces
1
0
200
400
JND indice (DICOM 3.14)
600
J i = J min +
P i J max
P
9
9
Luminance response
Luminance response
– Measured and GSDF luminance ascertained as a function
of JND
– Maximum deviation between measured and GSDF JND
per pixel value calculated = incremental contrast per
JND
L di @ J i
– measured and GSDF incremental contrast per JND
ascertained
=
i
d
i
=
2 (L i
(L i + L i
(L
d
i
2 (L di
+ L di
Li
1 )( J i
1
Ji
L di
1
)( J
)
i
1
)
Ji
1
)
1
@ 0 .5 ( J i + J i
)
@ 0 .5 ( J i + J i
1
= Max
)
1
)
Luminance results
JNDmax for L’max
GSDF calibrated
JNDmin for L’min
Uncalibrated
0.100
dL/L for a JND
0.100
JND
< 10% deviation from
GSDF’s for primary
< 20% deviation from
GSDF’s for secondary
d
i
i
0.2 for secondary
Luminance response
L’/L’avg per
(
0.1 for primary
)
TG18-LN
313
cd/m2
dL/L for a JND
L i @ Ji
0.010
0.5
cd/m2
0.001
237
cd/m2
0.010
1
cd/m2
0.001
0
100
200
300
400
JND indice
500
600
700
0
100
200
300
400
500
600
700
JND indice
10
10
versus # of steps
Luminance results
0.4
0.100
n
0.35
Barten model
+ tolerance
< 2.0
< 1.0
dL/L per JND
Jp < 3.0
0.010
Complianace with GSDF contrast
Measured
0.3
0.25
0.2
0.15
n
0.1
0.05
0
0.001
0
200
400
600
1
Visual assessment of
luminance response
10
100
1000
Number of levels (n)
JND index (Dicom 3.14)
Testing methods
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
9. Other tests
TG18-CT pattern: Equivalent perception of lowcontrast objects
TG18-MP pattern: Assessment of bit-depth
resolution via ramps
11
11
Viewing angle problem
Luminance uniformity
TG18-ULN10
Max L/Lavg
@ 10% < 30%
TG18-ULN80
Max L/Lavg
@ 80% < 30%
Angle of regard problem
Lmax and Lmin vs angle
300
6
Phi=0
Phi=0
Phi=45
Phi=45
Phi=90
250
Minimum luminance (cd/m^2)
Maximum luminance (cd/m^2)
200
150
100
50
Viewing distance:
30 cm
Display size:
30x40 cm
Max off-axis viewing angle: 35 degrees
Phi=90
5
Phi=135
Phi=135
4
3
2
1
0
-80
-60
-40
-20
0
20
Viewing angle (degrees)
40
60
80
0
-80
-60
-40
-20
0
20
40
60
80
Viewing angle (degrees)
12
12
Angular response
LR vs angle
300
Phi=0
Phi=45
Phi=90
250
Luminance ratio
Phi=135
200
180
150
145
Viewing distance:
Display size:
Max off-axis viewing angle:
30 cm
30x40 cm
35 degrees
95
100
60
50
0
-80
-60
-40
-20
0
20
40
60
80
Viewing angle (degrees)
LR vs angle
Angular response, hor.
0.100
800
Phi=0
Phi=45
700
Phi=90
Dicom 3.14
Phi=135
0
-20
500
400
LR=250
300
dL/L per JND
Luminance ratio
600
-40
-60
0.010
20
40
60
200
+ tolerance
- tolerance
100
0
-80
-60
-40
-20
0
20
View ing angle (degre es)
40
60
80
0.001
0
200
400
600
800
JND index (Dicom 3.14)
13
13
GSDF calib.
angular tolerance
Lmin
Lmax
LR
(0.3)
(250)
mLCD
3mpx
0.87
715.1
820
±23
±25
Max.
Max.dev.
dev.btw.
btw.measured
measuredand
andGSDF
GSDFinc.
inc.dL/L
dL/L
1.2
1.2
1.0
1.0
Testing methods
Phi=0
Phi=0
Phi=45
Phi=45
Phi=90
Phi=90
Phi=135
Phi=135
0.8
0.8
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
0.6
0.6
0.4
0.4
=0.3
0.2
0.2
0.0
0.0
-80
-80
9. Other tests
-60
-60
-40
-40
-20
-20
0
0
20
20
40
40
60
60
80
80
wing
a ngle (degrees)
(degrees)
Vie
V
iew
ing angle
(assumed perfect on-axis calibration)
TG18-RV50
Resolution evaluation
14
14
TG18-QC
Display MTF
FWHM = 0.20 mm
FWHM = 0.27 mm
MTF(fn)
> 35% for primary
displays
1
0.8
> 25% for secondary
displays
4500
0.6
LSF
MTF
3500
2500
0.4
1500
0.2
500
-500
-0.8
0
-0.4
0
0.4
0.8
Distance (m m )
TG18-CX
0
1
2
3
4
5
6
7
8
Spatial Frequency (cycles/mm)
CX resolution criteria
CX score < 4 for primary displays (RAR < 1.15)
CX score < 6 for secondary displays (RAR < 1.47)
15
15
Testing methods
Noise
TG18-AFC
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
CD = 0.012 mm
CD = 0.027 mm
CD = 0.049 mm
CD = 0.110 mm
9. Other tests
Noise
Testing methods
Noise in radiographic imaging
250,000 quanta/mR-mm2
X 1 mR/exposure
X 0.22 mm2/pixel
X 0.25 detected quanta/incident quanta
= 3,500 detected quanta/pixel
~ 2% relative noise
Assuming display noise < 1/3 quantum noise
~ 0.7%
w/ threshold SNR ~ 3
CD threshold ~ 0.023 mm (Rose model)
w/ 0.018 C/PV (8 bit) and
w/ 0.15 mm/display pixel (5 Mpx)
square target
pixels
1
2
3
4
5
PV
contrast
8
4
3
2
2
CD value
(mm)
0.024
0.024
0.027
0.024
0.030
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
9. Other tests
16
16
Veiling glare
GR =
LB
L
LN
LN
Veiling glare, visual
TG18-GQ
TG18-GQN
Cone
Telescopic
Luminance-meter
Display
TG18-GV
TG18-GVN
TG18-GQB
Glare criteria
Visual test
3rd target (96/4096 PV contrast) visible in both
patterns for primary displays
5rd target (160/4096 PV contrast) visible in both
patterns secondary displays
Testing methods
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
9. Other tests
17
17
Chromaticity
Testing methods
u' = 4x/(-2x +12y+3)
v' = 9y/(-2x+12y+3)
CU =
u’2
+
v’2
1. Geometry
5. Resolution
2. Reflection
6. Noise
3. Luminance
7. Veiling glare
4. Lum. dependencies
8. Chromaticity
CU < 0.01
9. Other tests
TG18-ULN80
Overall display QC
Display artifacts
More visually-based and oriented toward QC
“Clinical” test patterns
– TG18-CH
– TG18-KN
– TG18-MM1
– TG18-MM2
Comprehensive test pattern: TG18-QC
TG18-QC
Line-pair patterns
CX targets
CX scoring reference
16 luminance steps
Low-contrast targets
5 and 95% patches
Continuous Ramps
“Quality Control” targets
Contrast bars
Cross-talk target
18
18
TG18-CH
Clinical patterns
TG18-MM1
1.
2.
3.
4.
5.
6.
7.
8.
9.
1.
2.
3.
4.
5.
6.
Evaluation elements
Degree of difficulty for exam
Overall Contrast
Overall Sharpness
Symmetrical reproduction of the thorax
Medial border of the scapulae
Reproduction of rib cage above the
diaphragm
7. Sharp reproduction of the vascular
patterns, the trachea and proximal
bronchi, the borders of the heart and the
aorta, and the diaphragm
8. Visibility of the retrocardiac lung and the
mediastinum
9. Visibility of the spine through the heart
shadow
10. Visibility of small details in the whole lung
11. Visibility of linear and reticular details
out to the lung periphery
TG18-MM and TG18-KN
Evaluation elements
Degree of difficulty for exam
Overall contrast
Overall exposure/penetration
Overall sharpness (no blur)
Overall positioning - nipple in
profile
Adequate compression
Sharp appearance of Cooper's
ligaments
Structure of the clip and the
presence of the gap at its apex
Appearance and visibility of
subtle microcalcifications
Visibility of structures at the
margins of the breast
19
19
TG18 info
Workshop participants
Ehsan Samei
Hans Roehrig
deckard.duhs.duke.edu/tg18
Duke University Medical Center
University of Arizona, Tucson
Manufacturers:
1. BarcoView, LLC
2. Data Ray Corp.
3. Image Smiths, Inc.
4. Image Systems Corp.
5. National Display Systems
6. Planar Systems, Inc.
7. Totoku Electric Co., Ltd.
20
20