System Level Approach to Satellite
Instrument Calibration
Space Dynamics Laboratory at Utah State
University: Joe Tansock, Alan Thurgood, Gail
Bingham, Nikita Pougatchev, Randy Jost
NIST: Raju Datla
Ball Aerospace & Technologies Corp.: Edward
Knight
ASIC3 Workshop, May 17, 2006
1
Outline
•
•
•
•
•
•
•
Calibration Philosophy
“Specmanship”
Workshop to Improve Calibration
Calibration Planning
Subsystem/Component measurements
Ground Calibration
On-Orbit Calibration
– Internal and external calibration sources
• Satellite Instrument Validation and End-to-end Error
Model
• Summary
ASIC3 Workshop, May 17, 2006
2
Calibration Philosophy
• Calibration
– Provides a thorough understanding of sensor operation and
performance
– Verifies a sensor’s readiness for flight
– Verifies requirements and quantifies radiometric and
goniometric performance
– Provides the needed tools to convert the sensor output to
engineering units that are compatible with measurement
objectives
– Provides traceability to appropriate standards
– Estimates measurement uncertainties
ASIC3 Workshop, May 17, 2006
3
Calibration Philosophy – Cal Domains
• A complete calibration will address five responsivity
domains
– Radiometric responsivity
•
•
•
•
Radiance and irradiance traceable to NIST
Response linearity and uniformity corrections
Nominal/outlying pixel identification
Transfer calibration to internal calibration sources
– Spectral responsivity
• Sensor-level relative spectral response
– Spatial responsivity
• Point response function, effective field of view, optical distortion, and
scatter
– Temporal
• Short, medium, and long-term repeatability, frequency response
– Polarization
• Polarization sensitivity
ASIC3 Workshop, May 17, 2006
4
Calibration Philosophy – Cal Domains
• The goal of calibration is to characterize each domain
independently
– Together, these individually characterized domains comprise
a complete calibration of a radiometric sensor
• Domains cannot always be characterized
independently
– Complicates and increases calibration effort
– Example: Spectral spatial dependence caused by Stierwalt
effect
• Calibration parameters are grouped into two
convenient categories
– Calibration equation
• Converts sensor output (counts, volts, etc.) to engineering units
– Radiometric model
• All parameters not included in calibration equation but required to meet
calibration requirements
ASIC3 Workshop, May 17, 2006
5
Calibration Philosophy – Phases of Cal
• A complete and methodical approach to sensor
calibration should address the following phases:
Calibration planning during sensor design
Ground measurements
Subsystem/component
measurements
Sensor-level engineering tests
and calibration
Sensor-level ground calibration
Integration and test
On-orbit measurements On-orbit calibration
ASIC3 Workshop, May 17, 2006
6
Establishment of Good Specifications
Improves Calibration
• Programs often start with a requirement such as
– “The instrument shall be radiometrically calibrated to a 3%
absolute error, 1.5% band to band error, and a 0.25% intraband pixel to pixel error”
• The designers are then asked for cost, schedule, and
risk to meet this requirement, which could vary
dramatically
– E.g., is “error” a 1-sigma or 3-sigma requirement?
• Furthermore, incomplete, changing, or impossible
specifications are often the cause of cost and
schedule overruns
ASIC3 Workshop, May 17, 2006
7
So, What Makes a Good Specification?
• A good specification clearly communicates what
must be accomplished
– To an audience that is reading (vs. oral communication)
• No other “clues” to help understanding
– To an audience that may not be able to ask questions
easily
• Example: reading the specification at the end of the program after
there’s been personnel turnover
– To an audience that may have a different background,
training, or understanding of the problem than the author
“Good” Requirements Tests
(examine every formal requirement with these tests)
Also see E.
Knight, “Lessons
Learned in
Calibration
Specsmanship,
CALCON 2005
proceedings.
A. Is the requirement complete (domains, interactions, worst cases)?
B. Is the requirement unambiguous (terminology, grammar)?
C. Is the specification free of errors (for example, typos, math mistakes)?
D. Is there at least one identifiable method to implement this requirement?
E. Is there at least one identifiable method to verify this requirement?
ASIC3 Workshop, May 17, 2006
8
Lessons Learned in Specifications
• Lessons
– Cover all domains (spectral, spatial, temporal, radiometric,
polarization)
• Including interactions and “worst case” for requirements
– Scrub for ambiguity
– Use mathematical equations whenever possible to define
requirements
– Have at least one idea for implementation in mind when writing
the specification
• Or upon first round of review/questions
– Have at least one idea for verification in mind as well
• Conclusion
– The chance of an instrument
• Being “poorly calibrated”
• Overrunning cost and schedule targets can be reduced with improved
calibration specsmanship
ASIC3 Workshop, May 17, 2006
9
Workshop to Improve Quality of Calibration
• EO/IR Calibration & Characterization Workshops held in Feb
2005 and March 2006 at SDL/USU
– Envision self governing community based organization with goal of
improving calibration for all participating organizations
• Workshop Objectives
– Explore ways to improve the quality of IR/Visible/UV measurements,
community-wide, based on an ISO 17025 standard, as pioneered by
the RCS community
• Benefits based on experiences of RCS community
– Measurably and quantifiably improve the quality of measurements
made in the community
– Facilitate data comparison between sensors, systems, facilities,
programs and customers
– Increase in customer confidence in measurement results due to
improved: accuracy, uncertainty, repeatability, comparability, consistent
documentation
ASIC3 Workshop, May 17, 2006
10
Workshop to Improve Quality of Calibration
• Universal Agreement
– There is an unmet need that can not be addressed by any one
organization
• Intermediate results will continue to be presented at
annual CALCON (Calibration Conference)
• For more information
– CD available containing the presentations and recommendations
of the 2005 and 2006 workshops.
– http://www.sdl.usu.edu/conferences/eo-ir/
• Based on attendee feedback provided at the 2006
workshop, we have started the planning process for the
next workshop, to be held Spring 2007, at NIST, in
Gaithersburg, MD
ASIC3 Workshop, May 17, 2006
11
Calibration Planning
• Calibration planning
– Start as soon as possible (I.e. requirements definition,
concept design, sensor design, etc.)
– Influence sensor design to allow for efficient and complete
calibration
– Encourages optimum sensor design and calibration
approach to achieve performance requirements
• Planning phase can help shake out problems
– Schedule and cost risk can be minimized by understanding
what is required to perform a successful calibration
– Calibration equipment needs should be identified early to
allow time to build and test any required new equipment
ASIC3 Workshop, May 17, 2006
12
Calibration Planning
•
Identify instrument
requirements that drive
calibration
•
Identify calibration
measurement parameters and
group into:
– Calibration equation
– Radiometric model
•
Flow calibration measurement
parameters to trade study
–
–
–
–
–
•
Schedule
Sensor design feedback
GSE hardware & software
Measurement uncertainty
Risk
Perform trade study to
determine best calibration
approach
Mission Requirements
Instrument Requirements
Calibration Measurement Parameters
• Calibration Equation
• Radiometric Model
Sensor
Design
Cost &
Schedule
Measurement
Uncertainty
Calibration
Planning
GSE Hardware
& Software
Risk
ASIC3 Workshop, May 17, 2006
13
Subsystem/Component Measurements
• Subsystem and/or component level measurements
– Help verify, understand, and predict performance
– Collect Parameters for the Radiometric Model that can't be
measured well at the system level
– Minimize schedule risk during system assembly
• Identifies problems at lowest level of assembly
• Minimizes schedule impact by minimizing disassembly effort to fix a
problem
• System/Sensor level model development and measurements
– Allow for the development of Measurement Equation and
Performance prediction
– Allow for end-to-end measurements
– Account for interactions between subsystems and
components that are difficult to predict
ASIC3 Workshop, May 17, 2006
14
Subsystem/Component Measurements
• Merging component-level measurements to predict
sensor level calibration parameters may bring to light
systematic system-level uncertainties A,B
– Comparison of System-level estimate using component
measurements with end-to-end measurement of SABER
relative spectral responsivity (RSR)
• 9 of 10 channels < 5% difference
• 1 channel 24% difference (reason unknown)
• Helps to resolve and correct for component
degradation and sensor performance after launch
A.) Component Level Prediction versus System Level Measurement of SABER Relative Spectral response,
Scott Hansen, et.al., Conference on Characterization and Radiometric Calibration for Remote Sensing, 1999
B.) System Level Vs. Piece Parts Calibration: NIST Traceability – When Do You Have It and What Does It
Mean? Steven Lorentz, L-1 Standards and Technology, Inc, Joseph Rice, NIST, CALCON, 2003
ASIC3 Workshop, May 17, 2006
15
Engineering Ground Calibration
• Engineering calibration
– Performed before ground calibration
– Perform abbreviated set of all calibration measurements
– Verifies GSE operation, test configurations, and test
procedures
– Checks out the sensor
– Produces preliminary data to evaluate sensor performance
– Feedbacks info to flight unit, calibration equipment,
procedures, etc.
• Engineering calibration data analysis
– Evaluates sensor performance, test procedures, calibration
hardware performance and test procedures
• Based on results of engineering calibration,
appropriate updates can be made to prepare for
ground calibration data collection
ASIC3 Workshop, May 17, 2006
16
Ground Calibration
• Provides complete calibration needed to meet related
requirements
• Is performed under conditions that simulate operational
conditions for intended application/measurement
• Careful in-lab calibration minimizes problems that arise after
launch
– Minimizes risk of not discovering a problem prior to launch
– Promotes mission success during on-orbit operations
• For many sensor applications
– Detailed calibration is most efficiently performed during ground
calibration
– On-orbit calibration will not provide sufficient number of sources at
needed flux levels
– Operational time required for on-orbit calibration is minimized
• Best to perform ground calibration at highest level of assembly
possible
– Sensor-level at a minimum is recommended
ASIC3 Workshop, May 17, 2006
17
Extending Calibration to Operational Environment
• Calibration continues after ground calibration
Sensor Design/Fabrication
Ground Calibration
On-Orbit/Field Operations
Internal Calibration Source Response Trending
On-Orbit/Field Calibration/Verification
• Internal Calibration Source Response Trending
–
–
–
Trend sensor response to quantify relative response
changes over time
Source types
•
Blackbodies, glow bars, diffusers, lamps, etc.
Ensure source is stable and repeatable for sensor
operational life
ASIC3 Workshop, May 17, 2006
18
Internal Calibration Sources
• Challenges
– Ensure calibration source is stable and repeatable for sensor operational
life
– Ideally, calibration source should use same optical path as external
measurements
• Detailed trade to determine best approach is needed for each specific
application
• Considerations => source type, flux level, configuration, power, space, and
weight limitations, etc.
– Sources of variability
•
•
•
•
Temperature stability and/or temperature measurement
Emissivity changes
Thermal variations (external and internal)
Separate drift in observed response between calibration source and sensor
response
• Control and/or monitor electronics
• IR internal calibration source developments are required to achieve
stringent stability requirements of many climate change measurements
ASIC3 Workshop, May 17, 2006
19
On-Orbit Calibration
Verifies Cal and Quantifies Uncertainty
• Track, trend, and update calibration throughout a sensor’s
operational life
– In addition to internal calibration sources make use of external
calibration sources
• External On-orbit sources
– Standard IR stars
• Stars aBoo, aLyra, aTau, aCMa, bGem, bPeg
– Celestial objects
•
•
•
•
Moon
Planets provide bright variable sources
Asteroids, etc.
Sometimes you have to be creative:
– Off-axis scatter characterization using the moon
– Other techniques
• View large area source located on surface of earth (often termed vicarious
calibration)
• Cross-calibration between sensors
• Use of atmospheric lines
• Etc.
ASIC3 Workshop, May 17, 2006
20
Satellite Instrument Validation
• The purpose of validation is to assess actual accuracy and
precision of Satellite Instruments by comparison with validating
measurements
• Apparent differences in results between validating and
measurement system
– Satellite and validation data are not co-located in time and space
– Satellite and validating system have different vertical and horizontal
resolution
– Satellite and validating system have finite accuracy and
repeatability
– Physical measurement differences (I.e. spectral, sensor, platform,
etc.)
• Validation Assessment Model makes comparisons more
accurate by understanding and accounting for theses
differences
– Make results comparable
• Validation Assessment Model can be used as a tool to better
understand the tradeoff between validation approaches
ASIC3 Workshop, May 17, 2006
21
End-to-End Error Model
Overall Concept
True
Profile
xsat
Radiance
ysat
Parameter Error - δb
Noise – ε
Instrument
Performance
Assessment
True
Profile
xval
SDR yˆ
Smoothing
Parameter
Noise
Retrieval
EDR
x̂
δy expected Validation Assessment Model δxexpected
Reconcile differences to make results comparable
ŷ val
xval
yval
x̂ val
Validation System
Radiosondes, Aircraft Measurement Systems, Cross-Calibration, etc.
ASIC3 Workshop, May 17, 2006
22
Summary
• Calibration Philosophy
– What does calibration provide
– Calibration domains
– Phases of calibration
• Planning through operational environment
• Importance and benefit of good specsmanship
– Facilitate clear communication and minimizes risk of failure
• Workshop to improve quality of calibration
– Community wide participation working to improve calibration
• Calibration Planning
– Address all phases of calibration as early as possible
• Specification and design phases
ASIC3 Workshop, May 17, 2006
23
Summary (cont)
• Calibration Measurements
– Subsystem/Component Measurements
• Minimizes schedule risk and facilitates development of instrument
model and measurement equation
– Engineering Calibration and Calibration
• Methodical and careful approach leads to efficient and thorough
calibration
– Extending Calibration to Operational Environment
• Internal calibration sources (I.e. in-flight internal sources)
– Challenges and need for improvement
• External on-orbit sources
– External sources and need for improvement
• Satellite Instrument Validation
– Overall concept and the need for validation assessment
model to account for differences in space, time, resolution,
etc.
ASIC3 Workshop, May 17, 2006
24