ACS Side-1
Anomaly Review Board
Conclusions
STScI TIPS Meeting
21-September-2006
Ken Sembach
Subset of conclusions as summarized in the final ACS Side-1 ARB Report
(some items condensed / omitted for this presentation)
1
Anomaly Review Board Members
Roger Chiei, chairperson
LMTO/Code 442
Randy Stevens, secretary
LM IS&S/Code 441
Steve Arslanian
HTSI/Code 441
Wilber (Dale) Brigham
Code 563
Ken Carpenter
Code 667/441
Ed Cheung
J&T/Code 442
Randy Kimble
Code 667/442
Chuck Harguth
BATC
Olivia Lupie
Code 581/441
Ken Albin
BATC
Mal Niedner
Code 667/440
James Simons
BATC
Mike Prior
Code 441
Andy Hunt
BATC
Art Rankin
J&T/Code 441
Lynette Marbley
Code 303/442
Tim Schoeneweis
BATC
Henning Leidecker
Code 562
Ken Sembach
STScI
Beverly Serrano
RSTX/Code 441
Noman Siddiqi
QSS/Code 562
Hsiao Smith
Code 442
Renee Taylor
Code 303/442
Tom Wheeler
STScI
Consultants
2
Conclusions
•
The following facts are based on on-orbit data at the time of the anomaly:
•
•
•
•
•
•
•
All CEB telemetry collected by the Normal Engineering Data (NED) task went from
valid data to all ones within one sample period.
No indication of anomalous secondary voltages prior to ACS Suspend
No indication of thermal rise prior to and following the anomaly
Anomaly nearly instantaneous; occurred within 100 ms window
Anomaly caused the +28V Bus current to decrease ~ 0.8 to 1.3 Amps
No out-of-family behavior was observed in the science data prior to the Suspend event
The most likely cause of the ACS Side 1 on-orbit anomaly was a loss of the ACS
Side 1 CEB +15V power rail in either an open or shorted condition. This
conclusion is substantiated via
•
•
•
•
engineering evaluations
focused ground tests
historical performance
on-orbit telemetry collected at the time of the anomaly
3
Fault Tree Investigation: LVPS Board #3
• Loss of CEB Power Rail:
• Both WFC and HRC CEBs supplied with power from
common LVPS +5V, +15V and -15V source.
• Loss of the +15V power rail, with or without degradation of
the -15V power rail, results in uncharacteristic performance
of CEB A/D conversion and associated serial data stream.
• Ground testing of Engineering CEB unit resulted in
reproduction of on orbit anomaly
4
Most Likely Location of Fault
• The ARB identified LVPS Board #3 as most likely
location of the cause of the ACS on-orbit anomaly
• LVPS #3 contains all of the active components that produce
and/or interface with the suspected +15V CEB power rail
• However, other possible (less likely) locations exist:
• MEB Backplane: +15V trace to ground fault
• MEB to CEB harnessing: +15V wire short to ground fault
• Short on +15V Input to a CEB prior to power switching
relay
5
Possible Fault Locations
Most likely location
of root cause of
anomaly
LVPS#3
S/N 001
CEB PWR
+15V
Rail
Short or Open
Cannot be rule out the possibility
that the source of the anomaly may
be located in this area
MEB2
Backplane
HRC CEB
Short
2 pins
Cable
Harness
538548
Short or Open
Side
Select
Relays
2 pins
+35V
Rail
Same as
+15V
WFC CEB
-15V
Rail
Same as
+15V
Short
Side
Select
Relays
6 connector pins
per power rail
6
Anomaly Review Board Investigated Possible Failures
Case #
Failure Mode
Anomaly
match
based on
ETU testing
Implicates
Expected Telemetry
+5V
+15V
-15V
+35V
+5V
+15V
-15V
+35V
1
All Nominal
No
Problem intermittent or
+15V open beyond TLM
sample point
2
+15V open
(external)
YES
LVPS #3 Board
+5V
0V
-15V
0V
3
+15V open
(internal)
YES
MFL2815D
+5V
0V
-18V to
-20V
0V
4 or 5
-15V open
(external or
internal)
No
MFL2815D or LVPS #3
+5V
+15V
0V
+35V
6
+15V shorted
(internal or
external)
YES
MFL2815D or LVPS #3 or
MEB Backplane or
Harness to CEBs
+5V
0V
-6V to
-8V
0V
7
-15V shorted
(internal or
external)
No
MFL2815D or LVPS #3 or
MEB Backplane or
Harness to CEBs
+5V
+8V to
+11V
0V
+30V to
+35V
8 or 9
MFL2815D
inhibited or
internally
destroyed
No
(additional
testing in
progress)
MFL2815D
+5V
0V
0V
0V
Fault within MFL2815D
Fault on LVPS #3 Board, not in MFL2815D
Fault mode matches on orbit anomaly signature
7
Conclusions
• A thorough component-level Failure Mode Evaluation and Analysis
(FMEA) was conducted by the ARB on the ACS LVPS3 Board
circuitry related to the +/-15V CEB power rails.
• The FMEA results were compared to a set of voltage rail responses
obtained via CEB Engineering Unit lab tests to determine if the
postulated failure mode could have resulted in the ACS on-orbit
anomaly.
• Based on the FMEA results, fourteen ACS LVPS3 board parts (nine
distinct part types) and associated failure modes have been identified as
possibly resulting in the ACS on-orbit anomaly.
• The most likely cause of the ACS failure resides within LVPS Board #3,
component U2-7, the MFL2815D Interpoint DC-DC converter. The next
most likely cause is a short to ground in the transformer T6 on the same
LVPS board.
8
Conclusions
• ARB quality record review
• Identified an MEB level thermal cycle test failure that matched the
on-orbit failure scenario exactly. A short occurred on the +15V rail
at the +35V T6 step-up transformer.
• The ARB was concerned to find that after the initial replacement of
transformer T6, the anomaly again repeated with the new transformer
during re-testing with no further part removal and replacement. The
ARB has no way to determine quantitatively what potential stress and
possible latent damage may have occurred within the T6 transformer
over the 3 minutes of heating that it was subjected to during the retest.
• The ARB was concerned to find that the flight LVPS3 board was
subjected to uncontrolled heating effects by a heat gun as part of the
anomaly investigation. It is conceivable that this anomaly
investigation technique produced thermal stresses that resulted in a
latent failure on the flight LVPS3 board in question.
9
Conclusions
• ARB quality record review (continued)
• During ground test inspections, an investigation of the LVPS3
board showed that the board had separated between the
Ground/Thermal Plane and the fiberglass board. The ARB cannot
help but wonder if such a separation could have occurred that
ultimately led to a shorting condition or a stressed component
solder joint that resulted in the ACS on-orbit anomaly. Such
separation could have been aggravated by thermal transitions. For
example, ACS is powered down approximately once every month
to perform anneal operations, at which time the LVPS board
temperatures transition approximately 30 degrees with the current
6-hour anneals.
10
ARB Recommendations
• Modify ACS flight software so the NED history buffer contains 16-bit
values exactly as read from the A/D FIFOs. Specifically, do not apply a
bit-mask to the data prior to storage in the buffer. Storing the data in an
unmodified form would allow better insight into the MEB/CEB
interface. Make similar flight software changes to COS and WFC3.
(NICMOS and STIS do not have NED history buffers.)
• New CARD items should be written to prevent invalid “hybrid” power
configurations. These include:
• ACS only: prevent commanding of the “off” CEB
• ACS and STIS: prevent hot switching the CEB internal relays.
• Investigate reducing thermal and power cycling by:
• Leaving the HRC CEB powered on while doing SBC operations.
• Reducing the number of anneals
• Performing anneals in a “hybrid” mode that would leave Side-2 electronics
powered while annealing from Side-1
• Investigations have been referred to the Hybrid LEI Task and the STScI
ACS Team.
11