The Joint Agency Commercial Imagery
Evaluation (JACIE) Team
and
Product Characterization Approach
Vicki Zanoni
NASA Earth Science Applications Directorate
John C. Stennis Space Center
ISPRS Commission I/WG2
International Workshop on
Radiometric and Geometric Calibration
December 1-5, 2003
Co-authors
• Lockheed Martin Stennis Operations
– Mary Pagnutti
– Robert Ryan
• South Dakota State University
– Dennis Helder
• U.S. Geological Survey
– Greg Snyder
• Booz-Allen Hamilton, NIMA Commercial Imagery
Program
– William Lehman
– Spencer Roylance
Joint Agency Commercial Imagery
Evaluation (JACIE) Team
• Several U.S. Government agencies have purchased
commercial remote sensing data products to
support research and applications
• The Government’s use of commercial remote
sensing products requires thorough knowledge of
data quality
• The Joint Agency Commercial Imagery Evaluation (JACIE) team
was formed to leverage capabilities for the characterization of
commercial remote sensing products
• Each agency brings unique characterization expertise
– National Aeronautics and Space Administration (NASA) - Systems
Characterization (radiometric, spatial, geometric)
– National Imagery and Mapping Agency (NIMA)- Photogrammetry
and Image Interpretability
– U.S. Geological Survey (USGS) - Cartographic Assessments
JACIE Characterization Team
Characterization
JACIE TEAM
Organization
Spatial/
Quality
Geolocational
Radiometric
NASA
X
X
X
University of
Arizona
X
South Dakota
State University
X
U of Maryland
U of Arizona
S. Dakota State U
NIMA
SCIENCE
USERS
X
X
X
X
X
USGS
EOS
Validation
Teams
Application
X
X
X
University of
Maryland
X
X
Science
Community
Space Imaging
X
X
X
Digital Globe
X
X
X
X
Product Characterization
• For both research and applications, commercial
products must be well-characterized for accuracy and
repeatability.
• Since commercial systems are built and operated with
no government insight or oversight, the JACIE team
provides an independent product characterization of
delivered image and image-derived end products.
• End product characterization differs from the systems
calibration approach that is typically used with
government systems, where detailed system design
information is available.
• The product characterization approach addresses
three primary areas of product performance:
geopositional accuracy, image quality, and radiometric
accuracy.
Geopositional Accuracy Assessment
Method
• Utilize sites containing several “image-identifiable” targets
and compare their known locations with those defined by
the commercial image product.
• Follow Federal Geographic Data Committee (FGDC)
standards on target number and distribution
• Compute accuracy statistics.
– Root Mean Square Errors in X and Y directions (RMSEx and RMSEy)
– Radial (net) RMSE (RMSE r)
– Circular error at a 90% confidence level (CE90)
• commonly used to specify geopositional accuracy
– Circular error at a 95% confidence level (CE95)
Geopositional Accuracy Equations
RMSEx =
 x
img
 xcont 
2
n
RMSEy =
 y
img
 ycont 
2
n
Where
Ximg and Yimg are the image-derived positions of the target
Xcont and Ycont are the actual, measured locations of the target
n is the number of points used in the analysis
RMSEr =
RMSEx 2  RMSEy 2
CE90 = 1.5175 • RMSEr
CE95 = 1.7308 • RMSEr
Above equations assume no systematic bias.
Geopositional Accuracy Targets
Road intersections
Brookings, South Dakota
2.44-meter geodetic target
at NASA Stennis Space Center
Image-identifiable targets are GPS-surveyed to within sub-meter
accuracy, typically within a few centimeters
Spatial Response and Image Quality
• The effective spatial resolution of a system or data
product is driven by several parameters, including
ground sample distance (GSD), point spread
function (PSF), and signal-to-noise ratio (SNR).
• System performance is often specified in terms of
modulation transfer function (MTF) at the Nyquist
frequency.
• The JACIE team employs 2 approaches to
characterize spatial response
– use of edge targets to estimate edge response
• Relative edge response is easier to measure and is
mathematically related to PSF and MTF.
– use of pulse targets to estimate MTF
Edge Response Method
System Point Spread Function
Spatial
Domain
x
Edge Target
*
Edge Response
Slope ~ 1/x
Steepness of edge response effects spatial resolution
Edge Targets
QuickBird image of 20-meter concrete
edge target located at NASA Stennis
Space Center
20-meter X 20-meter
reflectance tarps used in edge
response characterization at
NASA Stennis Space
Center.
Targets should allow for both cross-track and along track measurements.
Size and placement of target are critical to ensuring proper sampling along edges.
Example Edge Responses and Line
Spread Functions
Northing direction
Feb. 17, 2002
Easting direction
Jan. 15, 2002
IKONOS (MTFC-Off)
panchromatic
edge responses, derived
from 20 X 20 meter
tarp edge targets
Numerical differentiation
of the edge response
yields the line spread
function (LSF).
edge response
The full width half
maximum (FWHM) value
of the LSF is often
compared to the
systems reported GSD.
line spread function
Pulse Target Method
• For larger GSD products, it is difficult to find edge
targets large enough to estimate edge response.
• Thus, for the commercial multispectral products, the
pulse target method provides a better estimate of
image quality.
• In the pulse method, long “strip” or pulse targets are
employed.
Pulse Target
Includes material © Space Imaging, L.P.
IKONOS image of 12 meter X 60
meter pulse target tarp used to
characterize multispectral MTF.
The size of the tarp is critical to
the effectiveness of the pulse
method.
Pulse tarp orientation with respect to true
north. Orientation of both pulse and
edge targets must allow for proper pixel
sampling across the edge transitions.
Illustration of Pulse Method
Pulse target input and
output response for
QuickBird blue band,
acquired Aug.25, 2002
The Fourier transform of
the input and output
responses.
The output transform
divided by the input
transform provides the
MTF. The value of MTF at
the Nyquist frequency is
often used to specify
spatial performance.
Image Interpretability
• Imagery is also characterized using the National Imagery
Interpretability Rating Scale (NIIRS) and Essential Elements of
Information (EEIs) , a means of quantifying the ability to identify
certain targets (e.g., railcars, airplanes) within an image
product.
• NIIRS is a 10-level rating scale (0 - 9) that defines the ability to
identify certain features or targets within an image.
– Detect trains or strings of standard rolling stock on railroad tracks
(not individual cars) - NIIRS 3
– Detect individual spikes in railroad ties.- NIIRS 9
• EEIs are certain features and targets (e.g. railcars, aircraft) that
correspond to the various NIIRS levels.
NIIRS Assessment Method
• Several image chips are extracted from images acquired over a
given time period and over multiple locations.
• The image chips are evaluated by a group of NIMA-certified
image analysts.
• The analysts each evaluate the same set of images under the
same conditions. i.e. using the same computer and amount of
image magnification, no additional image processing or
enhancement.
• The analysts assign NIIRS ratings and confidence ratings
associated with identification of EEIs.
• Statistical analyses are performed on analysts’ results to
understand the consistency and reliability of the different
analysts, and to identify any outlier image chips used in the
assessment.
• Good correlation among the analysts provides confidence in the
average NIIRS assigned.
General Imagery Quality Equation (GIQE)
• The GIQE mathematically relates NIIRS to several
parameters as a means of quantifying image quality
NIIRS  10.251  a log 10 GSDGM  b log 10 RERGM  0.656 H GM
0.344G

SNR
where
GSDGM is the geometric mean of the ground sampled distance,
RERGM is the geometric mean of the relative edge response,
HGM is the geometric mean-height overshoot caused by MTFC
(Leachtenauer et al., 1997), and
G is the noise gain associated with MTFC. In the current form of the GIQE,
SNR is estimated for differential radiance levels from Lambertian scenes
with reflectances of 7% and 15% with the noise estimated from photon,
detector, and uniformity noise terms.
If the RER exceeds 0.9, then a equals 3.32 and b equals 1.559; otherwise, a
equals 3.16 and b equals 2.817.
Radiometric Characterization
• Reflectance-based vicarious calibration approach
– Characterize reflectance of large, uniform target at time of
satellite overpass
• Measurements taken of target area and a 99% reflectance
spectralon panel (Jackson BRDF model)
– Characterize atmosphere at time of satellite overpass
– Use radiative transport code to predict at-sensor radiance
– Compare predicted at-sensor radiance to actual radiance
acquired by sensor
• Combine results of multiple independent teams. Each
team has slightly different measurement techniques
and data processing methods.
Radiometric Characterization Sites
Lunar Lake playa, Nevada
QuickBird image of
NASA Stennis Space Center
radiometric characterization site
Grass field in
Brookings, South Dakota
In-situ Data Acquisition for Radiometric
Characterization
Solar radiometer
measures optical
depth
Spectroradiometer measurements
of target albedo
Radiosonde measures
pressure, temperature,
and humidity profiles
Spectroradiometer measurements
of Spectralon 99% reflectance panel
Full sky imager
documents cloud
cover conditions
Multi-filter rotating
shadowband radiometer
measures direct and
diffuse solar irradiation
At Sensor Radiance Prediction Method
Sensor Characteristics
Viewing Geometry
Solar Geometry
Spectral Response
Aerosol Model
MODTRAN
Size and Number
Visibility and Asymmetry
adjusted for best fit
transmittance and diffuse to
global ratio
Predicted
Atmospheric
Transmittance
Solar Radiometer
Measured
Atmospheric
Transmittance
Atmospheric Parameters
H2O, O3, Pressure
MODTRAN
At Sensor
Radiance Prediction
ASD/Spectralon Geometry
Site Geometry
Altitude, Latitude, Longitude, Time
MODTRAN
Verification of input
parameters by comparison
to calibrated ASD radiance
measurements
Surface Characteristics
Predicted
Radiance
ASD Measured
Radiance
ASD/Spectralon
Radiance
ASD Measured Target Surface Albedo
Surrounding Surface Albedo
Target BRDF
Example Radiometric Characterization
Results
Example of JACIE-derived blue band radiometric calibration curve and
coefficient compared to original and revised QuickBird calibration
curves/coefficients. The JACIE-derived curve was the result of several vicarious
calibration activities conducted by JACIE team members during the 2002
acquisition season.
Conclusions
• JACIE product characterizations provide thorough
estimates of data quality for products provided to end
users.
• The team has performed spatial, radiometric, and
geopositional characterization of IKONOS and
QuickBird products
• Several improvements in IKONOS and QuickBird-2 data
quality have resulted
• Characterization results presented at annual JACIE
High Spatial Resolution Commercial Imagery
Workshops
• The JACIE team will soon focus on characterization of
OrbView-3.
Back up
Pre-launch Data Characterizations
• For commercially-developed systems, the government
and science end users do not have direct insight into
pre-launch calibrations
• Some critical parameters can only be accurately
measured in the laboratory
• JACIE reviews pre-launch calibration results that
enable thorough V&V of data specification such as
–
–
–
–
–
–
–
–
–
Spectral filter response curves across sensor FOV
Radiometric calibration coefficients
SNR estimates
Dynamic range/Linearity measurement
Polarization sensitivity measurement
Modulation Transfer Function (MTF) measurement
Geometric calibration coefficients
Band-to-band registration
Bad pixel maps
line spread function
edge response
numerically
differentiate
FWHM
line spread function
Fourier
transform
FWHM