Phoenix Project
MET Station Calibration, Characterization, and
Cataloging (CCC) Review Report
Prepared By:
Alex Dudelzak
Phoenix Senior Lidar Advisor
Date
Approved By:
Leslie Tamppari,
Project Scientist
Date
1
MET-Phoenix CCC Review Report
Friday October 21, 2005
York University, Toronto, Canada.
Attended (in person or by phone): Oladige Akinlade, Konstantine Baibakov, Eric Choi, Clive
Cook, Mike Daly, Stephane Desjardins, Tom Duck, Alex Dudelzak, Rejean Fortier, Mike Gross,
John Hahn, Vicky Hipkin, Carlos Lange, Olena Lovina, Pat McCormick, Alain Ouellet, Cote
Owen, Jagruti Pathak, Marianna Shepherd, Peter Smith, James Spinhirne, Leslie Tamppari, Peter
Taylor, Isabelle Tremblay, Wensong Weng, Jim Whiteway, Gregory Wilson, Di Wu.
Review Board: Alex Dudelzak, Pat McCormick, Jim Spinhirne, Cote Owen, Gregory Wilson.
This report follows the structure of the Calibration, Characterisation &
Cataloguing (CCC) Plan discussed at the CCC Workshop in order to identify
items missing in the Plan. In the executive summary the highest priority
recommendations are identified for each MET sensor. In the text that follows
are entered only the comments, discussion points and recommendations made
at the Workshop that effect the Plan. Inputs to the Review by independent
Board Members are attached as Appendices.
******************************************************************************
2
Executive Summary
Separate plans for Pressure, Temperature, Lidar and Tell-tale sensors were presented by members of the Canadian
MET science team and instrument contractor, MDA. Areas of responsibility have been delegated under the CSA
management structure. Success criteria for the review were presented by the Phoenix Project Scientist, Leslie
Tamppari:
A successful review will allow the review board to find that the Meteorology team
o Has an adequate plan for calibration of their instruments
o Appropriate schedule and other resources to conduct the activities presented, with adequate
margin
o Tests are appropriately prioritized
o Has a plan that ensures that the instrument will meet science objectives
o Ensures a good understanding of the output of the instruments
o Ensures a high probability of understanding how the instrument will perform on the surface of
Mars
o Ensures a high probability of interpreting returned datasets to meet the science objectives
(calibration, characterization, cataloging)
o Is ready to proceed with calibrationIn summary, the Review Board finds that:
o The team show good understanding of the output of the instruments. The lidar is is a conventional two
wavelength elastic backscatter system based on Nd:YAG laser. The heritage and application of this type of
lidar for observation of cloud and aerosol goes back over 30 years. Such systems are highly proficient at
the detection and height measurement of scattering layers, which is stated to be the principle goal of the
Phoenix MET lidar instrument. The tell-tale is an immature and inaccurate sensor but the maximum level
wind sensor allowable under mission constraints.
o The lack of a presented budget and schedule raises significant concerns for the implementation of the
proposed plans. Divided responsibilities also appear to be a barrier to implementing full end-to-end testing
for sensors.
o Cataloguing: Archiving of characterisation results though not described in detail in the plans will be
through archiving data in PDS data product formats. This is seen as adequate.
o The description of calibration and characterisation activities included in the plans is adequate though there
is concern about verification at system level. It is strongly recommended specific improvements are
included (see below).We would like to commend the MET team for plans related to Temperature
characterisation which go beyond previous efforts for the Mars surface environment.
In conclusion, the Board finds that the MET team is ready to proceed with calibration with the following highly
recommended additions to MET CCC plans:
1. Development of a more defined work plan and integrated schedule of activities leading up to launch. The
schedule should include clear responsibilities and deliverables for all MET partners.
2. Demonstration of stability of pressure sensor calibration at system level eg. effect of supply voltage
stability.
3. Lidar end-to-end performance tests including:
a. Simple atmospheric test on the EM/EQM unit
b. A plan to validate that lidar performance (alignment, laser stability) is maintained during
shipment, integration, launch, landing and under Mars environmental conditions. We suggest a
simple atmospheric test before shipment from Canada and after integration (as a misalignment
check and to ensure a working lidar is launched). For the Mars environment, we suggest the
science team evaluate whether there are side by side measurements that can be done with the SSI
camera or other checks that can demonstrate lidar performance on the mars surface.
4. There needs to be a clear communication of the plan of measurements by the instrument engineering group
needed for the lidar calibration and to characterize the flight electronics and the stability of the lidar
transceiver through the range of launch and environmental conditions the lidar will be subject to .
5. Modelling of lidar signal and S/N from the Mars atmosphere should be included in the plan [to confirm
science can be achieved]. Modelled values should include measured values of the APD pre-amp noise and
dark current.
6. Calibration of the tell-tale with the same camera type and observational view as on mars. The best
calibration, 'test as you fly', would be to image the mirror and calibrate the accuracy of deflection from
image analysis. Further implementation suggestions are included with the Board's individual reports.
7. Verification that the camera can see the telltale mirror at maximum lander tilt
3
Additional recommendations grouped by sensor:
PRESSURE AND TEMPERATURE
Pressure Plan
 To support end-to end characterisation, MDA should analyse the pressure calibration data
provided y FMI to characterize noise statistics and determine possible problem sources.
Temperature Characterization Plan
 It is recommnended a procedure is added to determine possible solar irradiation effects and
dust on the C-frame / TC.
 It is recommended a baseline temperature calibration without solar input is added.
WIND TELLTAIL
Telltail CCC Plan
 It is recommended that Characterization Plan should invoke making series of images over a
short period of time to assess the telltail oscillations.
 It is recommended that Teflon components are tested for resistance to UV.
4
Appendix 1
Phoenix Project MET CCC Review
Friday, October 21, 2005
York University, Toronto, Canada
Submitted by Board Member, M. Patrick McCormick
November 1, 2005
Overall the review reflected a very capable and experienced team working on MET. They are
well aware of the important issues and have the capabilities/experience to solve the problems of
CCC associated with MET. The plan appears adequate, progress appears reasonable at this point
in the development of MET, and there is adequate time to complete calibration before shipment.
In a number of areas a level of specificity was not present in the review however this was
probably due to a lack of experience with NASA-type reviews, and reflected more of an
“academia” approach to the review. However, when asked for clarification as more depth, it was
forthcoming from the presenter and participants.
Pressure
The pressure measurement technique of FMI using a complete unit of three
barocaps and temperature sensors all integrated together appears to be a superior
technique. There seemed to be a little uncertainty regarding the level of end-toend testing of the units when connected to the data acquisition system. FMI is
calibrating and characterizing these units at temperatures and pressures expected
on Mars.
Temperature The development is more of an in-house effort at York University and reflected
the typical investigator and team involvement with all the details of testing and
calibrating being exercised. Characterization and effects of fluctuating
temperatures and radiation were being investigated at temperatures and pressures
expected on Mars. The use of mean temperature and standard deviation, for
example, was discussed as a means for determining u* and T*. The possibility of
deposition of dust on the thermocouples and its effects were probably going to be
studied at the end of the testing program since it would distribute dust over the
entire testing facility.
Winds
Previously, a hot wire anemometer system was descoped from the mission.
However, the need for measuring winds on the surface of Mars is important to
better understand passing weather systems and water transport issues, and for
aiding in choosing the best time of day for sample delivery. Wind knowledge was
thought to be such an important measurement by the science team that a wind
measurement has been added back to the payload. It is a new technique being
developed and furnished by the Finnish.
The technique is called a ‘telltail’ technique whereby tails hanging from a
structure at the top of the MET boom, in a hangman configuration, are affected by
both wind direction and speed. The angle of the tail for a given wind speed is
calibrated. For the measurements aboard the Phoenix lander the SSI camera looks
at the tails through a mirror under the telltail system oriented at 45º. The pixel
position of the tip of the tail yields both speed and direction in 2-D. Accuracies
5
are expected to be the greater of 1 m/sec or 20% at speeds of 2-5m/s (4-10 knots),
or 40% for speeds of 5-10 m/s (10 to 20 knots). Of course, the determination of
wind direction will be possible at higher speeds. Calibration of the telltail system
is continuing in a wind tunnel under Martian conditions of pressure and
temperature. I suggested the Board recommend that these tests be conducted in a
configuration like that to be used on the lander, using an imager like the SSI, in a
geometry using a mirror to image the telltail in a fashion like that to be used on
the lander. In this manner, they will better know what the expected measurement
errors will be.
Lidar
The lidar development calibration discussions showed a high level of
understanding of what can and can not be calibrated with a lidar system.
First, the timing accuracy can be calibrated using a hard target. Second, the
alignment can be characterized by scanning the transmitter beam through the
receiver field or view. Further characterization is performed with the Field Lidar,
which has the same essential characteristics as the Flight Lidar, e.g. laser
wavelengths, same models of APD and PMT detectors, and analog and photon
counting data acquisition systems. The differences will not adversely affect the
characterization goals. For basic aerosol/cloud lidar, the backscattered signal is
the basic measurement. Throughput and detector responsivity are not necessary
to derive a physical parameter. A non-linear detector, however, will have to be
understood and calibrated. The Field Lidar will be used to determine the physical
limitations of the Flight Lidar, and for functional intercomparison tests of the
Flight Lidar. A number of campaigns are planned for the Field Lidar including
desert dust measurements to possibly simulate the conditions on Mars.
I suggested that the Board recommends two more tests. The first is an
atmospheric test of the Flight Lidar in Canada before shipment for
instrument/spacecraft ATLO. This test, which will depend on weather, should be
scheduled whenever reasonable considering the project schedule. Second, I
suggest the Board recommend strongly that an atmospheric test of the Flight Lidar
takes place during or after ATLO to ensure a working lidar is launched. This test
does not have to be exhaustive or take up much time, but primarily be a firing of
the laser into the atmosphere and a measurement of the return backscattered laser
light.
6
Appendix 2
Phoenix Met Calibration Review: York University, Toronto, Canada
October 21, 2005
Dr. Greg Wilson
Jet Propulsion Laboratory
MS 301-445E
4800 Oak Grove Dr.
Pasadena, CA 91109
818.354.5040
General Comments:
After nearly 7 hours of presentations the review board was asked to comment on the budget and
schedule for MET calibration, characterization and cataloguing. Neither budget of schedule was
ever discussed. While it is accepted that MET is a contribution to the Phoenix mission from the
CAS, the lack of an integrated schedule or budget presentation should be of great concern to the
project.
The MET investigation is highly complex from is programmatic implementation. While each of
the sensors are relatively simple, the process of sensor calibration, system level calibration, and
end-to-end validation and verification is not. Having sensors provided, calibrated, and
characterized by FIM, MDR, Univ. of Denmark, York Univ., Univ. Alberta, and Optech; having
them delivered to a system integrator (MDR), and finally to the spacecraft system contractor
(LMA) is highly complex programmatically. A schedule with clear responsibilities and
deliverables is required.
Another general concern is that of end-to-end calibration. “Test as you fly, fly as you test.” It is
not enough to have each piece calibrated and hope that when you pull it together that it will
work. The V&V process does not work unless you have done some level of system calibration,
and made every possible effort to characterize the performance of the system in a flight like
environment.
Recommendation: CAS needs to work closely with MET team lead to develop an integrated
schedule of activities leading up to launch. From the schedule a budget needs to be formulated
that addresses major milestones. This schedule and budget need to be presented to the project
for approval/acceptance.
Other Comments: Temperature, Pressure, and Teltail
As mentioned before, end-to-end system calibration/characterization is critical for system such as
MET.
Telltail (TT): This is a very immature sensor concept. I could not tell if it would meet the L1
requirement or not.
7
1. TT can and should be tested and characterized at atmospheric pressure, and verified at low
pressure. Use similarity theory to match Reynolds Numbers at higher pressure. This was
effectively used by the MPF Windsock experiment, since the dynamics of such a senor is a
function of Reynold’s Number. This should save time and effort.
2. For this to work, you must show that the Kevlar tale is not temperature sensitive. If it is T
sensitive, I do not know how you will characterize it on Mars.
3. Atmospheric testing would allow for more camera angles, which need to observe both
(simultaneously) vertical and azimuth deflections.
4. Calibration images should view mirror, as well as TT.
5. TT needs to be tested with gallow and at different wind directions relative to the MET mast
and gallow.
6. Lander tilt determination is a concern. If the lander can determine orientation to 1 degree,
what does that related to wind speed or wind direction? What is the error involved? What are
the different sources of error in making this measurement (ie. Camera optics, lander tilt, angular
determination of TT)
7. If lander is at Max tilt, can TT measure full dynamic range of wind speed and direction?
8. Calibration/Characterization, as some point, using the actual viewing geometries as in flight
will be required.
Temperature: End-to end calibration of system is my only concern. It appears best practices
are meet. Very impress with response time calibration.
1. Radiative experiments should be done in C-frame with appropriate coating.
Pressure: Again, end-to-end system performance must be demonstrated.
1. FMI sensor level calibration is a concern. How would small variations in the 12 V reference
effect the reported calibration from FMI?
2. What are you plans for system level verification?
8
Appendix 3
MET-Phoenix CCC Review
Review Report
James Spinhirne
October 21, 2005
NASA Goddard Space Flight Center
General Comments
The overall impression from the presentation was that the instrument groups for the MET
package well understand the science and instrumentation issues for their experiments. What
appears to be primarily lacking at this state of the mission development is a clearly thought
through, comprehensive and organized plan for all the various elements of measurements with
listing and documentation of the work break down for the calibration activities. There also needs
to be a clear statement of the person or groups that are responsible overall, and for each of work
elements. For at least one instrument the overall responsibility for the calibration and
verification was not clear. There were statements that the documentation for the calibration
activities was beyond the planned responsibility of the investigators and needed to be done as a
spare time activity. This indicates a lack of understanding of the importance and difficulty of the
type of planning and documentation needed. These need to be neither involved nor lengthy.
They should be clearly laid out and with the types of thorough but normal recording that are
expected in any good quality experimental work.
Another general issue is a possible problem in the communication and organization between the
science and engineering elements of the instrument development. Of priority is the necessary
testing that is needed to characterize any change in calibration or instrument performance as a
result of the range of environmental conditions that are expected for the mission.
Specific Comment for the Pressure, Temperature and TT Winds sensors
Although the general comment on the need for a more well defined work plan applies, the
overall impression was that instrument science group would have an adequately characterized
and calibrated instrument. Specific issues for improvement were raised in the meeting and
included in the meeting notes, especially for the TT wind indicator, and do not need to be
repeated. One issue though, raised in detail for the lidar discussion below regarding the need for
separate and thorough characterization of the flight electronics interface, is emphasized.
Specific Comments for Lidar
Requirements:
The written and presented plans did not explicitly define the needed calibrations or required
accuracy.
The Phoenix Mission Lidar (PML) is a conventional two wavelength elastic backscatter system
based on Nd:YAG laser. The heritage and application of this type of lidar for observation of
cloud and aerosol goes back over 30 years. Such systems are highly proficient at the detection
and height measurement of scattering layers, which is stated to be the principle goal of the PML
instrument. For the detection and ranging application calibration is needed for ranging accuracy,
but timing accuracy is a very robust in instrumentation and straight forward to measure. A 0.01
9
% timing accuracy should be more than sufficient and easily obtained. Another very important
issue for the basic function of scattering layer detection is the modeling of the expected signal
levels and the signal-to-noise ratio. The science group must have done such modeling, but it is
an omission that such calculations are not included or discussed in the CCC plan. This panel
member carried out independent calculations based on a model of the Mars atmosphere that
verify the basic functionality of the instrument. However, the s/n of the 1064 nm channel is
limited by and critically dependent on the overall dark current and pre-amp noise of the APD
detector. This must be fully defined, rather than assumed, for the modeled signal to be accurate.
The secondary goal inferred for the lidar is to characterize the nature of observed cloud and
aerosol from the strength of the signal. A general issue for such application is framed by the
quote “backscatter lidar sees clouds and aerosol very well, but we don’t know what it’s seeing”.
The aerosol and cloud backscattering cross sections are a complex and multi variable function of
the characteristics of the scattering particles and in addition the retrieval of the backscatter cross
section involves, for single ended lidar, an indeterminate solution for attenuation and multiple
scattering. Typically in applications of backscatter lidar, retrievals are constrained by the
addition of radiometer measurements for total optical depth or calibration and technique to key
measurements to known molecular cross sections. For Mars, from what was presented, there is
apparently no additional measurement to provide a boundary condition for solutions nor will it
be possible to obtain an unambiguously known signal from molecular scattering. However,
given the very limited knowledge of dust and clouds on Mars, the magnitude of the scattering
cross section and color ratio for the lidar signal will very usefully constrain the range of
interpretation of the characteristics, but there is no apparent need for high accuracy. A
calibration of the observed attenuated backscatter cross sections to 20% accuracy should be
sufficient. A calibration measurement to this accuracy can by easily achieved, but the stability of
calibration to this accuracy for the deployment on Mars will be challenging.
Plan Presented:
The written and presented plan, plus answers to questions posed, covered the range of issues
necessary for the instrument calibration and characterization and indicated they were understood
by the instrument group. However the emphasis was wrong and the organization and plan
structure incomplete.
The plan for the lidar calibration and characterization presented is heavily weighted to the
development and use of a separate field lidar, FL, which acquires similar two wavelength
signals. The overall plan is to characterize and calibrate the FL backscatter cross section by
atmospheric measurements made against a 532 nm Raman lidar, and then calibrate the PML
flight unit against the field system through an atmospheric intercomparison. The 20% accuracy
requirement mentioned above should be achievable with careful measurements and analysis,
though the near-range overlap function calibration to such accuracy will be difficult. (A separate
method to characterize the overlap function, such as the horizontal measurement method
employed by some lidar groups, should be used if possible.) The plan also covers an extensive
series of field experiment application of their FL unit to measurement the cross section and color
ratio of terrestrial clouds and aerosol. This is not seen as critical. There is an extensive literature
on the lidar cross sections of clouds and aerosol on earth. For Mars both the clouds and aerosol
are already known to be significantly different than on earth, clouds of CO2 particles for
example.
The most critical issues for the backscatter calibration, including the overlap function, is whether
the calibration will be maintained through the integration, launch, landing and the range of
10
environmental conditions on Mars. As presented, there is no plan for operational recalibration or
stability monitoring on Mars. The only way to ensure the calibration stability is through
engineering analysis, characterization and testing of the instrument. Since as stated the actual
flight electronics will not be mated to lidar transceiver until platform integration, the engineer
measurements will need to include a separated and thorough calibration of the data system
components within the flight electronics.
The critical part of the activities left out of the CCC plan are the needed engineering
measurements and analysis that ensure the stability, especially the boresite alignment, and over
all calibration accuracy of the lidar on Mars. There were statements made in response to
questions about the engineering measurements, that these would take place. However it was
apparent that there had been little communication or planning between the instrument
engineering and science groups. There needs to be a clear work break down structure and plan
for the needed measurements.
Recommendations:






The lidar calibration accuracy requirements need to be clearly stated.
The CCC plan should include modeling of signals and s/n from the Mars atmosphere, or
discussion of and reference to such modeling.
If possible, there should be more than one measurement to measure the system overlap
function.
There needs to be emphasis and a clear plan on the measurements needed from the
instrument engineering group to characterize the flight electronics and the stability of the
lidar transceiver through the range of launch and environmental conditions the lidar will
be subject to. The CCC activities listed in table 6 of the plan should be expanded and
also include the measurements needed from the instrument engineering group.
A full work bread down structure with the responsibilities and reporting clearly defined
for the characterization and calibration of the lidar is needed.
The science group should review if there are any possible techniques whereby the
performance stability and calibration of the instrument on Mars could be characterized,
such as radiometric comparison to the imager.
11
Appendix 4
MET-Phoenix CCC Review, York University, Toronto, Canada
Friday Oct 21, 2005
Reviewer: Owen Cote, Air Force Research Laboratory, Hanscom AFB, MA, USA
1. Summary of Comments on the C-frame Thermocouple Temperature Sensor CCC
The C-frame support contains three thermocouple (active) junctions of Copper-Constantan wires
in a parallel electrical circuit. . Since the thermocouple operates on the Peltier effect, I interpret
the parallel circuit as a form of redundancy for the temperature measurement. If either one or two
of the wires are broken a temperature measurement can still be made. If all three wires are intact
then the Electromotive force (delta E) developed across the sensor is the average of that in each
thermocouple. It was not clear from my understanding of the CCC presentation, whether the
calibration is for each C-frame or for each individual thermocouple or how the C-frame would be
orientated on the Mars landing craft. A C-frame parallel to Mars surface with all three
thermocouples at the same height would be expected to be exposed to smaller temperature
differences from the martian atmosphere than if mounted with the C-frame thermocouple wires
parallel to the Martian surface but each at small height difference above the Mars surface.
Whether this height difference can be a significant effect depends on the magnitude of the
calibration difference for each thermocouple which I don’t believe is not known now.
The three C-frames are to be mounted with separations of .25 meters and .5 meters. The total
height difference between the top and bottom C-frame is 0.75 meters as I understand from the
presentations. A height difference of one meter would be a reasonable separation distance to try
and measure how both the mean temperature difference and the mean square temperature
(turbulence) vary diurnally and seasonally in the Martian atmosphere. If the mean temperature
difference were close to the noise threshold of a single thermocouple, then mounting a number of
them in series, (a thermopile) rather than parallel allows the EMF to be the sum of the individual
EMF’s for each thermocouple rather than the average as in the parallel case. This is a
consideration if the signal to noise ration is low. If the reference junctions were eliminated so
that both thermocouple junctions were measuring the Martian atmosphere you would have an
alternative probe that sacrificed redundancy of the parallel circuit for the larger signal or
sensitivity of the multiple thermocouples in a series circuit.
Regarding the question of dust contamination of the thermocouple sensor, literature on
thermocouples suggest that all contamination issues are basically calibration or re-calibration
issues which in the context of the Mars experiment might be treated as a change in calibration
with aging or exposure of the thermocouple in the Marian atmosphere. I cannot offer a good
calibration strategy for estimating that effect.
2. Comments on radiation of thermocouple wires experiment.
The experimental set-up for alternating flows with two different temperatures past the
thermocouple sensor is excellent. When radiating the wire to determine its effect on temperature
measurement some consideration should be give to two cases: heating the wires but not the
thermocouple junction. This will test the Thompson effect and should not show as large an effect
12
as heating the copper/constantan junction only and not the wires if an appropriate shielding to
accomplish these two tests can be designed. This test might tell you if only the junctions would
need to be shielded on Mars.
3. No substantive comments are offered on the pressure, laser, and telltail CCC
13