Feb 8, 2007
GLAST LAT Project
GLAST Large Area Telescope
Gamma-ray Large
Area Space
Telescope
LAT Observatory PER
Stanford Linear Accelerator Center
NCRs and Waivers
1
Feb 8, 2007
GLAST LAT Project
LAT Environmental Test Flow
5/14/06
System Commissioning/
System Test
Shipment
5/08/06
3 days
5/20/06
Offload &
Set-up LAT
CPT
Sine Vibe
5 days
5 days
9 days
PER
5/25/06
Install
Radiator
s
3 days
EMI/EMC
Test
6/14/06
11 days
Acoustic
Test
6/24/06
7 days 7/1/06
7/05/06
7/29/06
Pre TV
7/5/06
T- Bal
T- Cycle
43 days
NCRs and Waivers
9/07/06
CPT
3 days
Remove
Radiator
s
2 days
Weight &
CG
Pack and
Ship
9/12/06 2 days
2 days
PSR
9/15/06
9/18/06
Arrival
SASS
2
Feb 8, 2007
GLAST LAT Project
Post delivery items
• Post delivery CPT successfully completed
• FSW load B0.6.12, B0.6.13, B0.6.14, B0.6.15 and B0.7.0 and
regression test successfully completed
• LAT mechanically integrated to spacecraft
• LAT Inrush measurements completed
• LAT functional test completed
• FSW load B0.8.0, B0.8.1 and B0.8.2 and regression test
successfully completed
• LAT to Spacecraft interface characterization test completed
• FSW load B0.9.0 successfully completed and regression test in
process
NCRs and Waivers
3
Feb 8, 2007
GLAST LAT Project
Unit Run Time
•
These unit run times (in hours) start after the LAT was completely
integrated and cover through the completion of the CPT after TV
– Does not include integration testing
– Does not include unit testing
Config
No.
SIU-P
1
2
3
4
5
6
7
8
9
1441
0
74
0
0
31
0
0
24
1570
LAT UNITS ON
SIU-R GASU-P GASU-R EPU-P0 EPU-P1 EPU-R Towers
0
1742
0
47
54
0
5
5
0
1441
0
0
0
54
31
5
0
0
1853
NCRs and Waivers
1531
0
1742
74
47
0
0
0
5
24
1892
1441
0
74
47
54
0
5
0
24
1645
1441
1742
74
0
0
31
0
5
24
3317
0
1742
0
47
54
31
5
5
0
1884
1441
1742
74
47
54
31
5
5
24
3423
ACD
1441
1742
74
47
54
31
5
5
24
3423
ACD
HV1
1441
0
74
0
0
31
0
5
0
1551
ACD
HV2
0
1742
0
47
54
0
5
0
24
1873
PDU-P
PDU-R
1441
0
74
47
0
0
5
0
0
0
1742
0
0
54
31
0
5
24
1567
1856
4
Feb 8, 2007
GLAST LAT Project
Status of NCRs Promoted to QARs
NCR
Corresponding
QAR
Description
Resolution
880, 881, 902,
948, 949
1196-4894
EPU, SIU Reboots
CND, See Reboot
Presentation
942
1196-4895
+X, -X LAT blanket replacement and
final closeout
Close post TV at final
closeout
946
1196-4876
ACD FREE board 5 power up
CND, No repeat at
Observatory
855, 894
1196-4894
LATC verify and dump errors
CND, No repeat at
Observatory
NCRs and Waivers
5
GLAST LAT Project
Feb 8, 2007
Significant NCRs/QARs at GD
• Issue
– NCR 992 - Tracker tower 10 layer 0 readout via left side
impaired
• Investigation
– Problem tracked to a communication problem in the interface
between a GTRC and associated GTFE 0 which resulted in
readout of stale data
• Resolution
– GTFE can be read out through an alternate path (to the right
instead of to the left)
– Register settings modified to read out this layer from the
right side
• In-orbit impacts
– No impact in current condition
– If right side readout becomes impaired, would lose 1 layer of
576 – not a significant impact
NCRs and Waivers
6
Feb 8, 2007
GLAST LAT Project
Documentation Status
•
•
RFAs from previous reviews
– See Systems Engineering Web site for dPDR, PDR, and CDR
RFA closures
• http://wwwglast.slac.stanford.edu/systemengineering/RFAS/RFAS.ht
m
– PER RFAs all closed
– PSR RFAs all closed
• Well, almost. Still have thermal report (in signoff)
Waivers approved (details follow) NOTE: have 2 to work closure,
one is in SC court
NCRs and Waivers
7
GLAST LAT Project
Feb 8, 2007
LAT Level Verification Status
•
•
Overview
– A total of 458 Level 2B and Level 3 reqts were identified for sell-off
• 452 reqts at the LAT level
• 6 reqts at the Observatory level
Current Status
– 319 approved by NASA <goal is to make this 409 by PER>
– 34 in review cycle <will work with Mark, some will be moved to post build 1>
– 50 TV related, just submitted <will work with Mark>
– 6 compression related, FQT completed and just submitted <will work with Mark>
– 43 GRB related, will close when FSW is completed
– 6 to be closed at Observatory test (e.g. shock, 4 TV cycles)
NCRs and Waivers
8
Feb 8, 2007
GLAST LAT Project
LAT Waivers (1/3)
CCR #
433-0311
433-0356
433-0357
433-0358
433-0360
Title
Description
Status
DC Voltage Tolerance
LAT is required to tolerate 0-40V DC. Due to
MOSFET switches at power feed inputs, LAT can
tolerate minimum 15V, excluding transient events.
Approved
Test Point Short Circuit
Isolation
LAT is required to operate within spec if any test
point is shorted to ground. A shorted external clock
select pin would render the redundant GASU
inoperable.
Approved
DC Voltage Tolerance #2
LAT required to tolerate 0-40V DC. After a voltage
drop analysis, it was found that the TEM MOSFET
switches would receive too low a voltage with the
DAQ feed voltage at 15V. To operate the TEM's
safely, the input voltage needs to be 18.5V
minimum.
Approved
GTFE TID
LAT is required to perform TID testing on all GTFE
ASIC lots. The final two lots were not tested since
previous lots exhibited such large margins.
Approved
Tracker Environmental
Test With Non-Flt or
Missing Cables
Several tracker towers went through environmental
test with a subset of missing or non-flight flex
cables. The replacement flight cables were not
subjected to component-level vibe and will not see
twelve tvac cycles.
Approved
NCRs and Waivers
9
Feb 8, 2007
GLAST LAT Project
LAT Waivers (2/3)
CCR #
Title
Description
Status
24AWG STD Strength Cu
High strength Cu alloy is required for 24AWG wire.
LAT uses standard strength Cu wire. As reported
by the LAT PCB, standard strength 24AWG wire
has been used on previous NASA projects with
GSFC’s approval with no compromise to product
reliability.
Approved
433-0362
J-STD vs NASA STD
LAT circuit card assemblies uses J-STD-001 as the
workmanship standard instead of NASA-STD8739.3.
Approved
433-0367
Tracker Flex Cable and
MCM Coupon Failures
Several flex cables and MCM’s are installed on the
LAT although they have failed coupons.
Approved
Radiator Sine Vibe
The radiators will not be installed for LAT-level sine
vibe test. Instead, the radiators were subjected to
alternative tests, i.e. pull test, tap test, LAT-level
acoustic test.
Approved
EMI Skirt Stay Clear
Center EMI skirt pieces near SC-LAT flexures
exceed the LAT stay-clear by 0.015” max.
Approved
VCHP CECM
The VCHP feed violates the CECM requirement.
The measured value is ~700mVp-p vs the
requirement of 200mVp-p.
Approved
433-0361
433-0368
433-0369
433-0374
NCRs and Waivers
10
Feb 8, 2007
GLAST LAT Project
LAT Waivers (3/3)
CCR #
Title
Description
Status
446-0402
Keyed Test Point
Connectors
Test point connectors are not keyed.
Approved
446-0403
Segregate Signal Types
on Test Connectors
JL-39 contains signals of different signal classes.
Approved
Cable Shield Termination
Floating shields are not properly terminated per
NASA-STD- 8739.4.
Approved
RS103 TKR Noise
Occupancy
Tracker noise occupancy exceeded its limit of 10-4
during RS 103 (vertical polarization) between 95101MHz.
Approved
446-0411
446-0412
NCRs and Waivers
11
Feb 8, 2007
GLAST LAT Project
SC-LAT ICD Waivers
ICN #
-095
-107
-109
Title
Description
Status
LAT Grid Interface Hole
Out-of-Tolerance
Several grid interface hole locations are out of
tolerance. Using the as-built LAT Grid and SC
interface hole locations, the analysis shows the
predicted forces to align the shear pins are
small and a minimum of 0.007” exist between
the bolts and holes in the flexures and mating
should not be an issue.
Approved
Recessed Grid
Bushings
The +Y and –Y LAT grid interface hole
bushings are recessed by 0.022” worst case.
Stress analysis at the SC mount interface
shows the margins of safety for ultimate and
yield bearing strength is 7% which is
acceptable. The margin of safety for pin
bending is >200%.
Approved
SC Overvoltage
Protection for LAT
VCHP Feeds
The SC does not limit the VCHP voltage to
maximum 40V. This is not an issue since the
voltage is limited to 55.3 V and the LAT can
withstand up to 60V.
Approved
Waiver for use of inches instead of meters in
SC model
Approved
-112
Thermal Units
NCRs and Waivers
12
Feb 8, 2007
GLAST LAT Project
SC-LAT ICD Waivers
ICN #
-115
-120
-124
Title
Description
Status
LAT Radiator FOV
Obstruction
Two SC snubber brackets and two separation
switches are located in front of the
LAT radiators.
Pending LAT
Review
LAT Radiator Venting
The LAT radiators partially vent through the
through cutouts near the Solar Array Launch
Lock Struts and the Solar Array Drive
Mechanism. The results of the GSFC analysis
shows this is acceptable.
Approved
LAT Inrush Current
The inrush current when the DAQ feed is
enabled by the SC exceeds the requirement.
LAT analysis shows no damage or degradation
to LAT components.
Pending SC
Review
NCRs and Waivers
13
Feb 8, 2007
GLAST LAT Project
GLAST Large Area Telescope
Gamma-ray Large
Area Space
Telescope
Pre-Environmental Review
LAT Instrument
Performance
J. Eric Grove
Naval Research Lab
LAT Commissioner
NCRs and Waivers
14
Feb 8, 2007
GLAST LAT Project
Introduction
• Continuing to monitor LAT performance via CPT, LPT, and
calibrations from Baseline forward through Observatory
integration
– CPT
• Detector subsystems
• Copper paths, interfaces
– Calibrations
• Detector subsystems
• Performance baseline successfully established at SLAC prior to
shipment
– Successfully verified at NRL and GD
– Awaiting pre-TV calibration…
• Compare with presentation 05 (LAT Test Results) from LAT PSR
NCRs and Waivers
15
Feb 8, 2007
GLAST LAT Project
ACD Performance
• Aliveness
No change from LAT PSR
– PHA
• All channels are alive and calibrated
– Veto
• All channels are alive and can be set to flight thresholds
– Exception: one channel that is not used in flight veto
– CNO
• All channels are alive and can be set to flight thresholds
• Performance notes
– None
– ACD performance is quite stable, operating within spec
NCRs and Waivers
16
Feb 8, 2007
GLAST LAT Project
CAL Performance
• Aliveness
No change from LAT PSR
– Spectroscopy
• All channels are alive and calibrated
– Trigger
• All discriminators are alive and can be set to flight
thresholds
– Data suppression
• All discriminators are alive and can be set to flight
thresholds
• Performance notes
– Front-end noise
• Four channels (out of 6144) out of family at room temp
– No impact to flight performance
– No open NCRs on CAL performance
– CAL performance is stable, operating within spec
NCRs and Waivers
17
Feb 8, 2007
GLAST LAT Project
TKR Performance
•
•
Aliveness
– Data
• Total bad channel count < 0.3%, within spec
– TOT
• All channels are alive and calibrated
– Trigger
• Discriminators in all GTFEs are alive and can be set to flight thresholds
Performance notes
– TKR “noise flares”: old issue, no change since PSR
• Transient increase in noise occupancy
– Noise occupancy and data volume are within spec
– TKR meets science performance requirements. Not an issue.
–
“Bad strip” trending: no change since PSR
• Strips not usable for triggering or tracking
– No significant loss of strips since LAT PSR
– TKR meets science performance requirements. Not an issue.
–
TKR tower 10 layer 0 readout via left side impaired: new issue
• Layer can still be read out from right side
– Loss of redundancy in 1 layer out of 576
– Not an issue for flight performance, no effect on science
• NCR 992 is closed
– See Hascall presentation
NCRs and Waivers
18
Feb 8, 2007
GLAST LAT Project
Summary
• LAT detector status
– Performance baseline successfully established at SLAC
• Baseline CPT and Calibration completed and signed off
– Post-environmental test performance measured at NRL
• Baseline performance confirmed
• Pre-ship CPT and Calibration completed and signed off
– Post-integration performance measured at GDC4
• Baseline functional performance confirmed
• Awaiting pre-TV calibration
– Integrated LAT is ready for Observatory environmental test
NCRs and Waivers
19
Feb 8, 2007
GLAST LAT Project
GLAST Large Area Telescope
Gamma-ray Large
Area Space
Telescope
Observatory IRR/PER
LAT Flight Software
Jana Thayer
Stanford Linear Accelerator Center
NCRs and Waivers
20
GLAST LAT Project
Feb 8, 2007
FSW Configuration Summary
•
•
Currently operating LAT with FSW B0-9-0
– Satisfies 95% of FSW requirements
– Includes resolution to watchdog reboots
– ~50 hours of run-time including regression testing with this build
History of FSW updates since shipment to GD-AIS
– 7/06: B0-6-9
• LAT shipped to GD-AIS with this build
• Installed prior to LAT environmental test
• Fulfills 143/183 FSW requirements
– 9/06: B0-6-12
• Included 6 months of bug fixes, JIRAs accumulated during LAT
environmental testing
• Fulfills 173/183 requirements
– 9/06 – 10/06: B0-6-13, B0-6-14, B0-6-15
• Bug fixes, addition of reboot diagnostics
– 11/06: B0-7-0, B0-7-1
• Event data compression implemented
– 1/07 – 2/07: B0-8-0, B0-8-1, B0-8-2
• RAD750 errata and other reboot related JIRAs addressed
– 2/07: B0-9-0
• Event data compression bug fixes implemented
NCRs and Waivers
21
Feb 8, 2007
GLAST LAT Project
Plan forward
• Build plan for B1-0-0
– Build contents:
• Support for commands to test LAT-GBM interface
• GRB detection algorithm
• Fully address 183 of 183 requirements
– Target build date: 4/23/07
– Target Delta-FQT-B: 4/30/07
– Upload to LAT: 5/1/07 (>1 month prior to Observatory TVAC)
• Support Observatory I&T with critical FSW patches/bug fixes
prior to launch as necessary
• Onboard FSW updates prior to launch are approved by a
program-level CCB
NCRs and Waivers
22
Feb 8, 2007
GLAST LAT Project
Requirement Validation
•
B0-9-0: 173/183 requirements verified at FQT on 4/13/06 and delta-FQT A on
8/14/06
•
Outstanding requirements
– GRB detection algorithm: B1-0-0
• 5.3.10.2.1 GRB Location Accuracy
• 5.3.10.2.2 Modification of GRB criteria
• 5.3.11.3.3 Process Attitude Data
• 5.3.11.6 GRB Alert Message Latency
• 5.3.11.7 LAT GRB Repoint Request Message to SC
– FSW Standards (verified as part of B1-0-0 after GRB detection algorithm is
implemented)
• 5.4.1 System of Units (metric system)
• 5.4.2.x Coordinate Systems (3 requirements)
• 5.4.3 Resource Margin
NCRs and Waivers
23
GLAST LAT Project
Feb 8, 2007
Impact to environmental test of remaining GRB
requirements
• GRB detection algorithm only verifiable on FSW Testbed
• GRB algorithm not required for TVAC or observatory test
– No observatory environmental tests require the presence of
GRB detection algorithm
• Desirable to implement on LAT prior to TVAC:
– GRB detection algorithm for performance baseline
– Infrastructure to test remaining LAT-GBM interface
requirements
NCRs and Waivers
24
Feb 8, 2007
GLAST LAT Project
B1-0-0 - Open JIRAs
• None of the open issues are liens against PER
• Outstanding JIRAs dealing with requirements
– FSW-292 Implement GRB detection algorithm
• JIRAs dealing with bug fixes, significant improvements to operations
– FSW-808 Problem enabling periodic triggers
– FSW-305 Summary/statistics telemetry stream needs to be
created for on-board event processors
– FSW-582 Capture of layer splits in LATC does not consider the FE
mode registers
NCRs and Waivers
25
Feb 8, 2007
GLAST LAT Project
Summary
• FSW fulfilling 173/183 requirements used throughout
Observatory I&T
• Spontaneous reboots addressed by B0-9-0
– Reboot problem had minimal impact on LAT and Observatory
testing
• Clear plan forward to complete FSW
• No LAT FSW liens to observatory environmental test
NCRs and Waivers
26
Feb 8, 2007
GLAST LAT Project
GLAST Large Area Telescope
Gamma-ray Large
Area Space Telescope
LAT Reset Resolution Team (RRT)
March 28 , 2007
Summary Status
Erik Andrews
NCRs and Waivers
27
Feb 8, 2007
GLAST LAT Project
RRT: Background
•
•
•
•
During LAT Instrument Integration and Test, infrequent but unexplained processor resets
were observed.
While these were documented in NCRs, analysis determined they were not preventing
progress on Instrument Integration and Test. Testing continued in parallel with reset
analysis
Subsequent to Instrument delivery and checkout at General Dynamics, the Project created a
team to focus on, analyze and solve these resets.
Goal: Resolve Resets
– Four areas of emphasis:
• Fishbone analysis to focus effort in specific technical domains
• Use/Create Off-line Memory Dump Analysis tools to support investigation
• Develop run-time instrumentation of the FSW to improve insight into processing
• Review dumped data from existing resets
– Set up collaboration website to support task.
• Reboot summary and dump data are maintained on the ISOC / FSW Website:
– http://confluence.slac.stanford.edu/display/ISOC/FSW
•
Operational Plan: Evolved process to handle reboots during observatory test
– Memory dump procedure defined
– FSW on call 24/7 to diagnose reboots
– FRB On-call team identified. Process produced good results as used.
• Phone #s distributed and available for operators
NCRs and Waivers
28
Feb 8, 2007
GLAST LAT Project
Root Cause Analysis
•
The RAD-750 contains a Thermal Assist Unit (TAU) which can be programmed in
an interrupt or a polled mode. LAT decided to implement TAU in an interrupt
mode.
–
•
•
•
(Note: GLAST SC does not implement TAU, and consequently this is a non-issue.)
The RAD-750 provides a Decrementer Register which provides an interrupt back
to the system when the counter expires (reaches 0 and transitions to xffff ffff).
Concurrent use of these two interrupts can cause unpredictable results. This
has manifested itself in corruption of machine registers (cache configurations,
stored PC values), stack pointers, etc. Some of which lead to watchdog
timeouts.
– BAE has reproduced the error in their lab. Seen on LAT Instrument +
Testbed
Quoting from the MPC-750 User Manual (but not listed in any errata document):
– For both the MPC750 and MPC755, no combination of the thermal assist
unit, the decrementer register, and the performance monitor can be used at
any one time. If exceptions for any two of these functional blocks are
enabled together, multiple exceptions caused by any of these three blocks
cause unpredictable results!
NCRs and Waivers
29
Feb 8, 2007
GLAST LAT Project
Summary Of Resets
•
Of the 36 total unexpected reboots on the LAT
– The root cause of 26 have been determined and fixed (as of 0.8.2) – JIRA 863
• Fundamental root cause related to interrupt conflict on RAD750
• Problem Confirmed by BAE. To be documented as (inherited) erratum
very soon
– The remaining 10, while suspected that they’re resolved, root cause remains
unconfirmed
• Many are likely already fixed
– Caused by new BAE erratum but unable to definitively confirm due to lack of
data and inability to reproduce
– FSW has matured since the reboots occurred
• For any not already fixed, we’re in a dramatically improved position to
determine root cause of any future reboots
– Improved diagnostic capabilities in FSW 0.9.0 and beyond
– Improved post-reboot processes in place to ensure all relevant data is
captured
•
Plan forward is to gain confidence in solution with extensive run time.
– Plan is to run and re-run LAT Functional and CPT tests in preparation for
Observatory Testing.
– Anticipate 200 – 250 hours of reset-free powered time on the LAT since ‘fix’
NCRs and Waivers
30
Feb 8, 2007
GLAST LAT Project
Remaining Open Reboots
ID
NCR
Open/Clo
sed
Unit
Type of reboot
LAT Activity
Run
Time (UTC)
Build
During primary to secondary
transition
77013656
1/8/2007
B0-6-15+
intSeSuite:
AcdSuite_AcdLongFunctional.xml
77013266
12/1/2006 13:45
B0-7-0
EPU1
Boottype=4, 0x200 (machine check)
Exception
Boottype=4, CPU Exception, PPC
Vector 0x300 (DSI)
LPA: LAT-20xCNONoPer_0.50hr
77012150
10/26/2006 13:33
B0-6-14
Open
SIU0
Boottype=0, VxWorks kernel panic
LCI: TkrNoiseAndGain_CPT
77011860
10/19/2006 5:16
B0-6-14
13 948-4
Open
EPU2
LCI: calu_collect_ci_calibGen_103
77011727 10/17/2006 16:35
B0-6-13
11 902-3
Open
EPU2
During LatReinit
77010681
9/27/2006
B0-6-12
10 902-2
Open
EPU2
Boottype=0, VxWorks kernel panic
Boottype=4, CPU exception, PPC
Vector 0x300 (DSI)
Boottype=4, CPU exception, PPC
Vector 0x300 (DSI)
During LatPowerOnTurbo
77010653
9/27/2006
B0-6-12
8
948-2
Open
EPU2
Boottype=2, Checkstop, EMC Vector 5 LAT-22x_0.50hr muon run
77009297
8/29/2006 6:09
B0-6-9
5
902-1
Open
EPU0
Boottype=4, CPU exception, PPC
Vector 0x300 (DSI)
Boottype=0,
77004380
5/7/2006 2:15:53
PST
B0-6-8
2
880
Open
SIU0
4/10/2006 15:40
B0-6-6
30 902-4
Open
EPU0
26 949-4
Open
SIU0
18 948-8
Open
14 949-3
Boottype=4, 0x300, exception
During LatReinit, concurrent with
main feed on command
VxWorks reboot
NCRs and Waivers
TkrTotGain_SVC_500hz
31
Feb 8, 2007
GLAST LAT Project
Plan for Observatory Test
• Testing based on LAT CPT
– Core set of tests run across environments
– Includes calibration and other tests that are run only at initial
and final ambient CPT
• Two orbit test
– Demonstrates concurrent SC, GBM and LAT operations for 2
orbits during each execution of the CPT
• CPT and LPT definitions follow
• Day in the life
– Full up operational scenario
NCRs and Waivers
32
Feb 8, 2007
GLAST LAT Project
Observatory Level LAT CPT
•
A LAT CPT performs the following test cases across the 9 redundancy
configurations and across environments
Test Case
Description
L-OBS-00x LAT Power On
L-OBS-00x LAT Power Off
L-OBS-10x
L-OBS-02x
L-OBS-06x
L-OBS-13x
L-OBS-22x
L-OBS-301
L-OBS-401
L-OBS-501
L-OBS-312
L-OBS-412
L-OBS-512
LAT Load Shed
LAT Reinitialization
SIU/EPU Hardware Functional
LAT/Spacecraft Interface Test
LAT Science Operations Demo
LAT ACD CPT
LAT CAL CPT
LAT TKR CPT
LAT ACD LPT
LAT CAL LPT
LAT TKR LPT
Configs
1-9
6-91
1-91
1-4,6
1-2
1,2,4,5
1-6
1
1
1
2
2
2
1. LAT Power Off will be run only post FSW B1.0 delivery.
•
Tests in addition to the CPT are also run as required or at initial/final
ambient test
– For example, L-OBS-04x LAT FSW Upload and L-OBS-90x FSW File
System Verification
NCRs and Waivers
33
Feb 8, 2007
GLAST LAT Project
Observatory Level LAT LPT
• A LAT LPT performs the following test cases in redundancy
configurations 1 & 2
Test Case
L-OBS-00x
L-OBS-10x
L-OBS-13x
L-OBS-22x
L-OBS-31x
L-OBS-41x
L-OBS-51x
NCRs and Waivers
Description
LAT Power On
LAT Load Shed
LAT/Spacecraft Interface Test
LAT Science Operations Demo
LAT ACD LPT
LAT CAL LPT
LAT TKR LPT
Configs
1-2
1-2
1-2
1-2
1-2
1-2
1-2
34
Feb 8, 2007
GLAST LAT Project
Conclusion
•
•
•
LAT subsystem level test program successfully completed
No liens open which preclude entrance to environmental test
LAT is ready for Observatory Environmental test
NCRs and Waivers
35
Feb 8, 2007
GLAST LAT Project
Backup Charts
NCRs and Waivers
36
GLAST LAT Project
Feb 8, 2007
LAT Performance Backup slides
NCRs and Waivers
37
GLAST LAT Project
Feb 8, 2007
TKR Performance: Noise Flares
•
•
•
•
Issue
– 8 (of 612) layers in 17 Trackers have shown infrequent, sporadic flares of increased
noise occupancy. The 8 layers are uncorrelated.
– The flares are correlated across channels in a given ladder, with many or all channels in
the ladder firing at once.
– There is no evidence that the problem was statistically worse in T/V than in atmosphere,
but we cannot rule out a small effect.
Analysis
– Monitor in cosmic-ray data in FM-8 and in 16 towers.
– The affected regions are fully ON and sensitive immediately before and after a flare.
This ruled out intermittent bias connections as a cause.
– Even during flares, all recent runs still satisfy all noise specifications.
– Study in FM-8 versus HV level and humidity:
– Unfortunately, we could not get the problem to recur at all in FM-8, so we did not reach
any conclusion.
– Test at lower bias voltage (80 volts instead of 100 volts) still showed flares
– Data taken during TV indicates no significant change under vacuum
Resolution Plan
– Continue to monitor the effects in 16-tower cosmic-ray data, especially in TV testing.
Impacts on On-orbit performance
– The observed noise is very far from a level that would have any impact at all on
performance. An increase by much more than an order of magnitude, including
spreading to other trays, would have to occur to begin to see impacts. (Overall, the
TKR noise performance is phenomenally good!)
NCRs and Waivers
38
GLAST LAT Project
Feb 8, 2007
TKR Performance: Noise Flares
•
TKR “noise flares”
– Transient increase in noise
occupancy
• Duration is minutes to hours
• Little or no dependence on
– Time (i.e. no increase in rate
of occurrence)
– Temperature
– Bias voltage
– LAT-average noise occupancy
• Mean = 1.3×10-6 over JuneJuly muon runs
– Including flaring episodes
– Mean drops to ~5×10-7
when flaring is excluded
• Worst 90-minute period =
1.5×10-5
Note: LAT occ = Layer occ / 576
NCRs and Waivers
39
Feb 8, 2007
GLAST LAT Project
TKR Performance: Bad Strips
•
Three major categories:
– Hot strips: unusually high occupancy
• Historically anything >10-4 occupancy, but strips well above this
level can still be useful and should not be masked unnecessarily!
• Small numbers, with no trending issues.
– Dead strips: do not respond to internal charge injection
• Either a dead amplifier or a broken SSD strip connected to the
amplifier (usually the latter).
• Very small numbers, with no trending issues.
– Disconnected strips: broken wire bond or trace between…
• (a) ladder and amplifier, mostly due to MCM encapsulation
debonding from silicone contamination,
• or (b) SSDs within a ladder, due to Nusil encapsulation
debonding in thermal cycles.
• The majority of the bad strips are in early towers, and the
delamination definitely propagates somewhat with time.
– Can reattach/detach with temperature change
NCRs and Waivers
40
Feb 8, 2007
GLAST LAT Project
TKR Trending
Disconnected Channel Trend
20
0
-20
•
Bad channel trend
– Trend is essentially flat
– Total number of bad channels after LAT env
test = 3400
Total number of TKR chans ~ 900,000
– Total number of bad channels is within spec
• <0.3% of channels
NCRs and Waivers
ho
t2
34
co
ld
Po 234
st
-T
VA
C
ld
1
co
ns
tra
ho
t1
-60
Test phase
•
Old figure to be updated
-40
Pr line
eTV
AC
10
40
Ba
se
Old figure to be updated
60
r
FS
W
100
80
16
Tw
1000
ha
nd
-o
f
16 f
-T
w
r
FS
Ba W
se
Pr line
eTV
AC
ho
t1
tra
ns
co
ld
1
ho
t2
3
co 4
ld
Po 23
4
st
-T
VA
C
Total number of bad channels
10000
Disc. channel increase from hand-off
Bad Channel Trend
TkrFMA
TkrFMB
TkrFM1
TkrFM2
TkrFM3
TkrFM5
TkrFM4
TkrFM6
TkrFM7
TkrFM9
TkrFM10
TkrFM11
TkrFM12
TkrFM13
TkrFM14
TkrFM15
Test phase
•
Bad channel trend
– Total increment since LAT completion
• +XX disconnected strips
– Small increase in bad count during
environmental test
• <XX% increase
41
Feb 8, 2007
GLAST LAT Project
TKR Bad Strips – Summary
•
•
•
The problem of encapsulation delamination has been well known and discussed for a long
time, including the increase during Tracker TVAC testing, but the project elected to use the
affected MCMs as-is because of
1. the adverse schedule and cost impact of redoing 1/3 of the MCM production
2. and the belief that future degradation would never reach a level at which the science
would be compromised.
Nothing is different today:
– There is some evidence that the problem areas have expanded very slightly during LAT
integration, but
• It is impossible to be sure at any time what channels are really disconnected,
because the wires in delamination regions often make electrical contact even
when the mechanical bond is gone. Many channels of the channels that appeared
to be new disconnects during LAT environmental test were observed to be
disconnected during TKR TVAC testing.
• No disconnected channels have appeared in previously unaffected regions of
MCMs.
We expected that the problem regions would expand during LAT environmental testing at a
level comparable to Subsystem environmental testing.
– Indeed this is what was observed
– Degradation is insignificant with respect to science performance
• LAT environmental test caused bad channel count to change from 0.3% to 0.3%
• Expect Observatory environmental test to cause count to change from 0.3% to
0.4% or less
NCRs and Waivers
42
Feb 8, 2007
GLAST LAT Project
GLAST Large Area Telescope
Gamma-ray Large
Area Space
Telescope
LAT FSW Backup
Stanford Linear Accelerator Center
NCRs and Waivers
43
Feb 8, 2007
GLAST LAT Project
B2-0-0 (post-launch)
• Address FSW changes based on lessons learned in testing
– FSW-562 Make sure that PIG's power sequence is still correct
– FSW-287 Anti-flooding for MSG
– FSW-271 Logical/physical descriptions
– FSW-414 Add internal resources to PIG and eliminate the
LEM_micr argument present in most function prototypes/
– FSW-419 If LSEC cannot encode an event, nothing is placed
into the datagram.
– FSW-280 CAL and ACD bias voltage settings
– FSW-538 There is no way to ignore the AEM when the
LATC_verify operation is performed.
– FSW-791 High and low splits are not separately ignorable
NCRs and Waivers
44
Feb 8, 2007
GLAST LAT Project
Deferred
•
Summary
–
–
–
–
–
–
–
–
–
–
–
–
–
FSW-824 CLONE -Disable memory controller Maximum Bank Active Timeout (would
require change to PBC)
FSW-832 CLONE -Need unique access to all cache lines of LCB I/O buffers during
hardware operation (would require change to PBC)
FSW-875 IVV TIM 1635 - LAT FSW Boot Code (PBC): Duplication of APID definitions in
header & source code files may lead to execution errors
FSW-626 LATC dumps have unexpected GTFE masks on LATC verify error dumps only
FSW-239 vxw_flight RTOS consitutent still has the serial console device enabled
FSW-540 Addition of AEM/EBM memory relocation register control
FSW-697 Set the range for all padded fields to 0-0
FSW-474 Sharpen the definition of the extended counters so that completely accurate
bookkeeping can be done even when there are dropped datagrams
FSW-689 Split LFSFILEID into device, directory, and file name
FSW-724 QSEC does not update the event-time fields in the standard context correctly
FSW-526 NCR 794, problem 6: Add debugging code to LCBD code to trace intermittent
failure
FSW-636 NCR 882: CPU should apply a reset to the LCB after it powers the GASU and
before it checks the LCB for data presence
FSW-753 ACD calibration PHA threshold is not being iterated
NCRs and Waivers
45
Feb 8, 2007
GLAST LAT Project
Unscheduled
•
Unscheduled JIRAs (jbt – JIRA to be updated, most can be scheduled)
– FSW-790 Tracker calibration doesn't work correctly with uneven
splits – schedule it for B2-0-0
– FSW-729 LATC verify error response – schedule it for B2-0-0
– FSW-703 Ensure all registers are set - survey
– FSW-763 EFC IVV code issues - determine whether action is
necessary
– FSW-699 Create report to identify configuration files in use - survey
– FSW-872 Illegal memory reference in LCBD after request list fetch
error – schedule for B2-0-0
– FSW-876 Include LATC ignore file used as part of the run
configuration data – schedule it (B1-0-0)
– FSW-878 CLONE -After integration with the space craft, time tones
do not seem to be properly updated (B1-0-0)
– FSW-799 Decide on desired level of command execution
verification, ability to determine commanded configuration changes
– FSW-838 PPC compiler is treating a char as an unsigned quantity
rather than a signed - survey
NCRs and Waivers
46
Feb 8, 2007
GLAST LAT Project
RRT Backup Charts
NCRs and Waivers
47
Feb 8, 2007
GLAST LAT Project
#2 NCR-880
•
•
•
•
•
•
•
Type: 0, VxWorks reboot
Date/Time: 4/10/2006 3:40:00 PM
Unit: SIU-R (SIU0)
FSW Build: B0-6-6
Activity: TkrTotGain_SVC_500hz (~20s after script start)
Analysis
– Either some application called the reboot() or sysToMonitor() functions (not
likely at all), or the VxWorks kernel issued a panic exception. This is usually
the caused by a "work queue overflow" in the kernel, which can mean an
overflow of timer expirations or interrupts.
– For these reboot types, the kernel should leave a short text string at address
0x0000fd00. It usually contains a very short, not very descriptive message
such as "Kernel panic: work queue overflow". Unfortunately, this was not
looked at after the reboot.
Current status
– Not enough evidence to determine if caused by new RAD750 erratum
– Investigation at dead end
– Procedures/Tools in place to gather additional data should this type of
reboot recur
NCRs and Waivers
48
Feb 8, 2007
GLAST LAT Project
#5 NCR-902-1
•
•
•
•
•
•
•
Type: 4, CPU exception, PPC Vector 0x300 (DSI)
Date/Time: 5/7/2006 9:15:53 PM
Unit: EPU0
FSW Build: B0-6-8
Activity: During LatReinit, concurrent with main feed on command
Analysis
– Exception was generated at the application level, either while the RTOS was initializing,
while the SBC was running, or after the applications had been initialized and running.
– DSI exception occurs when no higher priority exception exists and a data memory
access cannot be performed. DSISR register indicates:
• Exception was caused by the data address being out of bounds for our MMU setup
(CPU DBAT registers).
• Exception occurred on a load access
– DAR register indicates data address which was issued to cause the exception is out of
range for the memory mapping we have implemented
– Address related to error = 0xffffffc3
– SSR0 register indicates the instruction which generated the exception was a "lwz"
instruction near the end of the kernel function "taskUnlock()".
– As with #10, #11, and #30, DSI exception addresses for #5 and #18 are 0x365c6c, but the
memory addresses are small negative values rather than the prepainted stack contents
value of 0xeeeeeeee
Current status
– Not enough evidence to determine if caused by new RAD750 erratum
– Investigation at dead end
– Procedures/Tools in place to gather additional data should this type of reboot recur
NCRs and Waivers
49
Feb 8, 2007
GLAST LAT Project
#8 NCR-948-2
•
•
•
•
•
•
•
Type: 2, Checkstop, EMC Vector 5
Date/Time: 8/29/2006 6:09:00 AM
Unit: EPU2
FSW Build: B0-6-9
Activity: LAT-22x_0.50hr muon run (77009297)
Analysis
– Boot tlm seen about 25 seconds after series of commands to reset LRS
counters. LatReinit after reboot resulted in single bit error tlm, possibly not
related to watchdog timer induced reboot
Current status
– Need to verify that CPU is in fact configured to take the Checkstop option
rather than the Machine Check Exception option.
– Need to make decision on the idea of catching these errors at the software
level. All of these critical errors can be masked off from generating EMC
Vector or Checkstop exceptions. In the cases where the error is masked, it
will be reported instead as a CPU interrupt or exception, resulting in the
execution of a software error handler. The advantages are the abilities to
provide a more detailed report and to be reconfigurable. This must be
weighed against the likelihood that the error is in fact critical, and an
attempt to execute software further would fail.
– Procedures/Tools in place to gather additional data should this type of
reboot recur
NCRs and Waivers
50
Feb 8, 2007
GLAST LAT Project
#10 NCR-902-2
•
•
•
•
•
•
•
Type: 4, CPU exception, PPC Vector 0x300 (DSI)
Date/Time: 9/27/2006 5:52:58 PM
Unit: EPU2
FSW Build: B0-6-12
Activity: During LatPowerOnTurbo (77010653)
Analysis
– EPU2 took the exception during the transition from primary to secondary boot.
Consequently, the LSW trace had not started yet and does not contain any useful
information.
– #10, #11, and #30 all take a DSI exception at address 0x365c6c in VxWorks routine
taskUnlock() while attempting to access memory at address 0xeeeeeeee during startup
– #10, #11, and #30 show a saved link register (lr) value in “application 0” word of the
PBC telemetry of 0x360d70 in VxWorks routine reschedule()
– #10, #11, and #30 differ only in the task control block address for the active task and the
saved stack addresses in the “application 1” and “application 2” words. These are
identical in #10 and #11 but different in #30, probably corresponding to the fact that the
first two crashes occurred while running B0-6-12 while the last one was B0-6-15+. Both
of these builds employ the same V6-11-2 version of VxWorks, and thus have no change
in the addresses of the VxWorks routines.
Current status
– Not enough evidence to determine if caused by new RAD750 erratum
– Potentially eliminated with changes in the startup ordering in B0-8-0
– Investigation at dead end
– Procedures/Tools in place to gather additional data should this type of reboot recur
NCRs and Waivers
51
Feb 8, 2007
GLAST LAT Project
#11 NCR-902-3
•
•
•
•
•
•
•
Type: 4, CPU exception, PPC Vector 0x300 (DSI)
Date/Time: 9/27/2006 11:35:00 PM
Unit: EPU2
Activity: During LatReinit (77010681)
FSW Build: B0-6-12
Analysis
– EPU2 took the exception during the transition from primary to secondary boot. Consequently, the
LSW trace had not started yet and does not contain any useful information.
– Code tried to access invalid address in the RTOS portion of RAM, in the taskUnlock() function within
the V6-11-2 vxw_flight image
– #10, #11, and #30 all take a DSI exception at address 0x365c6c in VxWorks routine taskUnlock() while
attempting to access memory at address 0xeeeeeeee during startup
– #10, #11, and #30 show a saved link register (lr) value in “application 0” word of the PBC telemetry of
0x360d70 in VxWorks routine reschedule()
– #10, #11, and #30 differ only in the task control block address for the active task and the saved stack
addresses in the “application 1” and “application 2” words. These are identical in #10 and #11 but
different in #30, probably corresponding to the fact that the first two crashes occurred while running
B0-6-12 while the last one was B0-6-15+. Both of these builds employ the same V6-11-2 version of
VxWorks, and thus have no change in the addresses of the VxWorks routines.
Current status
– Not enough evidence to determine if caused by new RAD750 erratum
– Potentially eliminated with changes in the startup ordering in B0-8-0
– Investigation at dead end
– Procedures/Tools in place to gather additional data should this type of reboot recur including
• dump about 2K bytes starting about 256 bytes below the stack pointer value (assuming the 2K
bytes would not attempt to read past the end of physical memory)
NCRs and Waivers
52
Feb 8, 2007
GLAST LAT Project
#13 NCR-948-4
•
•
•
•
•
•
Type: 0, VxWorks kernel panic
Date/Time: 10/17/2006 4:35:34 PM
Unit: EPU2
FSW Build: B0-6-14 (or 6-13 per jana?)
Activity: LCI: calu_collect_ci_calibGen_103 (77011727)
Analysis
– Not a lost decrementer since no 520s dropout in
housekeeping
• Current status
– Not enough evidence to determine if caused by new RAD750
erratum
– Investigation at dead end
– Procedures/Tools in place to gather additional data should
this type of reboot recur
NCRs and Waivers
53
Feb 8, 2007
GLAST LAT Project
#14 NCR-949-3
•
•
•
•
•
•
Type: 0, VxWorks kernel panic
Date/Time: 10/19/2006 5:16:08 AM
Unit: SIU-R (SIU0)
FSW Build: B0-6-14
Activity: LCI: TkrNoiseAndGain_CPT (77011860)
Analysis
– SIU was emitting normal SIU statistics housekeeping packets
until about 4 seconds before boot telemetry was observed
• Current status
– Not enough evidence to determine if caused by new RAD750
erratum
– Investigation at dead end
– Procedures/Tools in place to gather additional data should
this type of reboot recur
NCRs and Waivers
54
Feb 8, 2007
GLAST LAT Project
#18 NCR-948-8
•
•
•
•
•
•
•
Type: 4, CPU Exception, PPC Vector 0x300 (DSI)
Date/Time: 10/26/06 13:33
Unit: EPU1
FSW Build: B0-6-14
Activity: LPA: LAT-20xCNONoPer_0.50hr (77012150)
Analysis
– SRR0: 0x00365c6c Instruction that caused problem
– SRR1: 0x0000b030 Assorted bits copied from MSR register
– DAR: 0xffffffff Address the CPU was trying to access
– DSISR: 0x40000000 Basically, memory access error
– PCI Status 2 Reg: 0x02000000
– Mem Status Reg: 0x00000004
– Task ID: 0x07fef864 dLCBDevt
– Application 0: 0x00a3ec08 Link register (calling routine)
– Application 1: 0x07fef580 Stack pointer
– Application 2: 0x07fef4c0 Exception pointer
– Application 3: 0x00000000 (and so on to until application 7)
– Consistent with a pointer walking backwards through memory
– As with #10, #11, and #30, DSI exception addresses for #5 and #18 are 0x365c6c, but the memory
addresses are small negative values rather than the prepainted stack contents value of 0xeeeeeeee
Current status
– Not enough evidence to determine if caused by new RAD750 erratum
– Investigation at dead end
– Procedures/Tools in place to gather additional data should this type of reboot recur
NCRs and Waivers
55
Feb 8, 2007
GLAST LAT Project
#26 NCR-949-4
•
•
•
•
•
•
•
Type: 4, Exception, 0x200 (not DSI)
Date/Time: 2006-12-01 13:45:42
Unit: SIU0
FSW Build: B0-7-0
Activity: intSeSuite: AcdSuite_AcdLongFunctional.xml (77013266)
Analysis
– Received a bad pointer, wrote a word where it shouldn’t have, and
bad things happened
– The register is out on the PCI, which means that the value is byteswapped. The rogue value that’s written has zeroes in most and
least significant bytes, so you can’t tell which direction to read the
number from, left or right.
Current status
– Tracked via FSW-872, on agenda for next FSW CCB
– Not enough evidence to determine if caused by new RAD750
erratum
– Procedures/Tools in place to gather additional data should this type
of reboot recur
NCRs and Waivers
56
Feb 8, 2007
GLAST LAT Project
#30 NCR-902-4
•
•
•
•
•
•
•
Type: 4, Exception, 0x300 (DSI)
Date/Time: 2007-01-08
Unit: EPU0
FSW Build: B0-6-15+
Activity: During primary to secondary transition (77013656)
Analysis
– The task dLCBDevt took the exception while trying to access memory at 0xeeeeeeee
– #10, #11, and #30 all take a DSI exception at address 0x365c6c in VxWorks routine
taskUnlock() while attempting to access memory at address 0xeeeeeeee during startup
– #10, #11, and #30 show a saved link register (lr) value in “application 0” word of the
PBC telemetry of 0x360d70 in VxWorks routine reschedule()
– #10, #11, and #30 differ only in the task control block address for the active task and the
saved stack addresses in the “application 1” and “application 2” words. These are
identical in #10 and #11 but different in #30, probably corresponding to the fact that the
first two crashes occurred while running B0-6-12 while the last one was B0-6-15+. Both
of these builds employ the same V6-11-2 version of VxWorks, and thus have no change
in the addresses of the VxWorks routines.
– Current suspicion is that the problem is related to a race condition when the first
forwarded magic 7 packet arrives at the EPU before startup is completed
Current status
– Analyzing stack dump for the dLCBDevt task
– Potentially eliminated with changes in the startup ordering in B0-8-0
– Procedures/Tools in place to gather additional data should this type of reboot recur
NCRs and Waivers
57
Feb 8, 2007
GLAST LAT Project
Abbreviated Fishbone (1/2)
Cause
Discussion/Rationale
1.
Hardware failure induces reboot
Identical pre-existing, intermittent or environmentally induced
failure of isolated part or board in 5 independent
processors is not a credible cause for reboots.
Recommendation per F. Huegel.
Not Credible
2.
Software induced reboot
Code has been designed to avoid this issue. Desk checking and
static analysis tools affirm design.
Very Unlikely
Code analysis performed to eliminate potential memory overwrite
errors, but see 2.2.1
Very Unlikely
Potential for overwrite documented in FSW-823, FSW-831. Static
code analysis tools utilized by IVV to check for
unprotected memory writes. Issues found in JIRAs 834, 835
and resolved.
Very Unlikely.
2.1.
Operating system flaw
2.1.1.
Priority inversion
2.1.2.OS does not provide memory
protection
2.2.
Status
Application software bug
2.2.1.
2.2.1.1.
Memory overwrite
Generic overwrite
2.2.2.
Interrupt locks
Code has been designed to avoid this issue. Instrumentation
implemented in 0.7.1 and 0.8.0 affirm design.
Very Unlikely
2.2.3.
Task exception
Task exceptions have occurred. Three of the resets trace to
task exceptions. However, could not cause watchdog
timeouts, since the CPU handles the exception first (and
resets). Task exceptions are handled in the exception
vector. System designed to collect information on the task
that attempted to execute illegally. This data is captured
preserved. These resets, when they occur are captured and
provide good insight into what's happened.
Possible
Where race conditions are a potential, system designed to
sequence events thru use of semaphores.
Instrumentation in 0.8.0 affirm implementation.
Unlikely
2.2.4.Race Conditions
NCRs and Waivers
58
Feb 8, 2007
GLAST LAT Project
Abbreviated Fishbone (2/2)
Cause
Discussion/Rationale
3.
Operations/environment
Not a credible cause – environmental testing was
successfully completed with no change to reboot
rate during any environment. Recommendation per
F. Huegel.
4.
LAT FSW interacts with
computer firmware/OS feature
4.1.
Features documented in
vendor errata sheets
Status
Not Credible
General note: examined errata from vendor, including
newly disclosed features.
4.1.1.
Errata 15
LAT was susceptible, documented in FSW-820, FSW-821.
BAE Recommended work-around implemented in Build
0.8.x, so now eliminated.
Not Credible
4.1.2.
Errata 24
LAT was susceptible, documented in FSW-822, FSW-824.
BAE Recommended work-around implemented in Build
0.8.x, so now eliminated.
Not Credible
4.2.
5.
Undocumented and
previously unknown errata
In designing LAT LCB, believe we ran across
undocumented errata on the BAE bridge. Believe
that it is possible here as well. (This turns out
to the be case, confirmed by BAE.)
Possible
EPU/SIU hardware design flaw
5.1.
LCB FPGA error
5.1.1.
LCB incorrectly writes
memory
NCRs and Waivers
Writes to random areas in memory could cause an
exception reboot, but very unlikely to cause a
watchdog timer reboot since corrupted memory
would be more likely to cause computation errors
or exceptions rather than causing a process to
hang.
Very Unlikely
59
GLAST LAT Project
Feb 8, 2007
Reset Analysis-related FSW changes
•
•
Process:
– All RRT recommended changes to FSW are being tracked in the JIRA system
• This means Project-level approval for each/all
JIRAs identified
– BAE Undocumented RAD750 Errata
• Identified conflict between decrementer interrupt and TAU interrupt (#xxx)
– BAE Documented RAD750 board (Bridge) Errata Related
• Erratum #15: Simultaneous Snoop with CPU Read Hang ( #820, #821, #823,
#826 )
• Erratum #24: Memory Controller Max Bank Active Timeout Hang (JIRA #822,
#824*, #832*)
– [* = Clones in SIU/EPU boot code. Deferred.]
–
Desk Checking key sections of the code
• Potential LRA command/response lists processing conflict with Erratum #15 (#
826 )
• Identified & recommended changes to package EDS (#831)
– Augment LSW log entries
• Correcting identified LSW flaws (#812, #813)
• Add Stack Pointer and Watchdog timer values on each context switch (#829)
• Add entry/exit from ISRs (#829)
– SIU task exceptions during power-down (#833)
• LCB getting corrupted data from the GASU when it powers down. Fix is
approved for next build.
– Unprotected/unchecked memory accesses (Identified by IVV tools)
• JIRAs: #834, #835; #837 (in 0.9.0)
NCRs and Waivers
60
– Diagnostic: Cache configuration configurable at boot time.