GLAST LAT Project
Managers F2F 6/15/05
GLAST Large Area Telescope:
Face to Face Managers Meeting
Gamma-ray Large
Area Space
Telescope
LAT Integration & Test
June 15, 2005 2:00 PM
Integration and Test
6 Towers
Elliott Bloom
I&T Manager
Ken Fouts
I&T Engineering Manager
SLAC
Eduardo do Couto e Silva
SVAC Manager
4.1.9 - Integration and Test
1
GLAST LAT Project
Managers F2F 6/15/05
Last Month's Accomplishments (1 of 2)
• Management
– Tracking hardware shortages for LAT integration.
– Procedure Status
– 68 of 70 submitted for sign-off or released
• IFCT
– Completed 4 tower tests.
– Completed I&T receiving tests on 3 TKR’s, 3 CAL’s and 3
TEM’s.
– Completed assembly and installation of the grid cooler (aka
the Chilla from Manila).
– Completed installation of 2 additional towers for a total of 6
towers installed in the grid.
– Preparations complete for 6 tower tests.
4.1.9 - Integration and Test
2
GLAST LAT Project
Managers F2F 6/15/05
Last Month's Accomplishments (2 of 2)
•
Online
– LATTE
• Supported VSC software Python interface implementation
• Exploratory work in progress to demonstrate interaction with
the VSC and its software interface
• 4.9.0 released
–
–
–
–
Initiated by ACD-discovered bug
Bug fixes, Trigger GUI, GPIB support
Updated Hippodraw addressing some memory leaks
Updated trigger GUI to address concerns raised in trigger
meeting
– Bug fix for incorrectly initialized TEM BUSY_LRS_MASK register
(measures TEM deadtime)
• Housekeeping multi-pen strip-chart GUI V2.0 deployed to
cleanroom for real-time monitoring.
• E-logbook Version 3.4.1 released (minor bug fixes with
corresponding tutorial and documentation updates)
4.1.9 - Integration and Test
3
GLAST LAT Project
Managers F2F 6/15/05
I&T Activities
Grid Chiller Unit Installation
Six CAL/TEM’s Installed
4.1.9 - Integration and Test
Grid Cooling
Six Tower’s Integration
4
GLAST LAT Project
Managers F2F 6/15/05
Procedure Status
•
•
•
70 documents total
68 latest versions submitted for sign-off
65 are released
80
70
11
11
11
11
11
11
2
2
2
2
2
2
2
2
12
12
12
12
11
11
2
2
2
2
2
12
12
12
12
12
4
4
4
4
4
4
4
4
7
7
7
7
7
6
6
6
3
4
3
4
3
4
3
4
3
4
15
15
15
15
15
15
15
15
15
15
15
15
15
19
19
19
19
18
18
18
18
18
18
18
18
18
Count of submitted
for signoff
Count of DCN written
Count of DCN and
Doc sent for signoff
Count of Signed by
Author
Count of Signed by
Elliott Bloom
Count of Signed by
QA
Count of Signed by
Others
Count of Released in
LATDocs
12
Count of Internal
Review Complete
12
11
Count of Draft
Complete
12
11
Count of Draft
Started
12
11
Count of Document
Number Assigned
Documents
50
11
Count of Document #
60
11
40
30
Depart.
Online
SVAC
Particle
MGSE
Manage.
IFCT-Mech
IFCT-Elec
20
10
0
4.1.9 - Integration and Test
5
GLAST LAT Project
Managers F2F 6/15/05
Procedure Status – Management
4.1.9 - Integration and Test
6
GLAST LAT Project
Managers F2F 6/15/05
Procedure Status – IFCT Electrical
4.1.9 - Integration and Test
7
GLAST LAT Project
Managers F2F 6/15/05
I&T NCRs by Month Open/Closed Status
18
16
Count of Open/Closed Status
18
Count of Open/Closed Status
NCR Summary
16
14
14
12
12
Current
Status
10
Open/Closed Status
24 Open
38 Closed
Open
Closed
8
Current
Status
6
10
4
Open/Closed Status
As of May 10th
2
Open
Closed
0
8
Sep-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Month
DateNCR
by MonthOpened
6
4
As of June 14th
2
0
Sep-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Jun-05
Date by Month
Month
NCR Opened
4.1.9 - Integration and Test
8
GLAST LAT Project
Managers F2F 6/15/05
Power Outage Impact on I&T
• Emergency Generator allowed work to continue in B33.
• Room 103 circuit breaker/panel fault was resolved by turning
off non-essential power.
• Room 104 was unaffected by the panel fault. TKR 3 was
installed on Thursday, June 19.
• Root cause of the circuit breaker fault was found to be a loose
connection on the main power coming into the panel.
• The building was brought back to 100% with minimal impact to
building users.
4.1.9 - Integration and Test
9
GLAST LAT Project
Managers F2F 6/15/05
Online Progress
– LATTE 5
• Meetings with SE, ISOC and FSW
• Requirements doc in progress
• Took delivery of a Dell blade server
– Used to develop LATTE 5 concepts
– Test mobile rack concepts
– Progress on parallelizing the scripts:
• TKR scripts nearly complete and tested
• CAL CPT and LPT nearly complete and tested.
• INT (e2e, SVAC) scripts complete.
• CalibGen, CalibDac and MuTrg still to go
– ACD script preparation and test support ongoing.
– Risks to schedule:
• Continued ACD problems requiring one or more of the Online
team to make a site visit
• LATTE 5 requirements are converging slowly.
4.1.9 - Integration and Test
10
GLAST LAT Project
Managers F2F 6/15/05
I&T Detail Schedule (1 of 6)
4.1.9 - Integration and Test
11
GLAST LAT Project
Managers F2F 6/15/05
I&T Detail Schedule (2 of 6)
4.1.9 - Integration and Test
12
GLAST LAT Project
Managers F2F 6/15/05
I&T Detail Schedule (3 of 6)
4.1.9 - Integration and Test
13
GLAST LAT Project
Managers F2F 6/15/05
I&T Detail Schedule (4 of 6)
4.1.9 - Integration and Test
14
GLAST LAT Project
Managers F2F 6/15/05
I&T Detail Schedule (5 of 6)
4.1.9 - Integration and Test
15
GLAST LAT Project
Managers F2F 6/15/05
I&T Detail Schedule (6 of 6)
4.1.9 - Integration and Test
16
GLAST LAT Project
Managers F2F 6/15/05
Issues & Concerns
ISSUES
• NCR 497 for connector clips and screws.
• Availability of flight hardware.
– Shortages identified by I&T and being worked with
Subsystems.
• TKR Deliveries
• SIU, EPU Deliveries from ELX
• Flight Cables
CONCERNS
• Continued support from subsystems for NCR closure.
• TPS rework and replacement, availability for future tower
installations.
• Need firm delivery dates for third layer ELX boxes
• LATTE 5.0 ready Oct. 1 for use in LAT Functional testing due to
start 10/14. E2E tests start 11/1.
4.1.9 - Integration and Test
17
GLAST LAT Project
Managers F2F 6/15/05
NCR 497 Summary (1 of 2)
Background:
• An NCR was generated for the orientation of clips and fasteners on
incoming cables from Cicon.
• SLAC specifies 2 Glenair connector types on cables.
• Thread damage was found on fasteners due to a mismatched clip and
fastener configuration.
Findings:
• The Glenair connector drawings are incorrect and confusing.
• Clips will damage fastener threads if installed incorrectly.
• Some clips damage fastener threads when installed correctly.
• Connectors from Glenair delivered to Cicon in wrong configuration.
• Cicon put clips on the way they were received.
Corrective Action:
• Replace clip and fasteners at SLAC with new hardware and inspect
fastener threads per NCR 497 disposition.
4.1.9 - Integration and Test
18
GLAST LAT Project
Managers F2F 6/15/05
NCR 497 Summary
Recess in threads should line
up with square feature in clip
4.1.9 - Integration and Test
19
GLAST LAT Project
Managers F2F 6/15/05
SAS Interfaces
• SVAC is happy with CCB process
– Thanks Richard
• Code release is getting more stable and reliable
– Working with SAS to sharpen validation procedures
– specially calibrations
• Pipeline issues are more infrequent
– We expect to transition into a more robust architecture in a
not so distant future
– Ought to be tailored to the ISOC needs
4.1.9 - Integration and Test
20
GLAST LAT Project
Managers F2F 6/15/05
Data Analysis Issues (up to 4 tower data)
• Hardware failures or problems
– None so far !!! Kudos to the “royal we”
• Anomalies in offline distributions
– Becoming less frequent and mostly due to
– Incorrect configurations (mostly operational procedures)
– Mistakes in the offline data analysis codes
• Understanding how the instrument works
– Time sink because it requires a much broader knowledge of
– Integrated detector system functions and performance
– Details of complex offline software
– How the analysis parameter space changes as we add
more towers
4.1.9 - Integration and Test
21
GLAST LAT Project
Managers F2F 6/15/05
Feedback to Integration and Test
• The Instrument Analysis effort has already provided valuable
feedback to Integration in several ways
– Examples will be provided in the next slides
• Improving Operations
– We provided a cross check to existing procedures
– and suggested improvements
• Understanding Instrument performance
– We generated discussions on a particular topic
– That led to different operational settings
– We generated new tests
– To verify functionality at the integrated level
4.1.9 - Integration and Test
22
GLAST LAT Project
Managers F2F 6/15/05
Feedback to Operations: Procedures
•
SVAC automated reports identified
that GEM was not reporting trigger
information properly for some of the
4 tower tests
•
Action
– stopped data taking to
investigate root cause (Eduardo)
•
Root cause
– incorrect power up of system
– found by Horwitz and
confirmed by Huffer
•
Recommendation to I&T
– Improve procedures
– Introduce online control plots
during data taking
4.1.9 - Integration and Test
23
GLAST LAT Project
Managers F2F 6/15/05
Understanding Performance: TEM diagnostics
•
We are trying to investigate 1%
effects when TEM diagnostics were
enabled (Luis)
– Mike pointed out that we should
be careful
– TEM diag disabled in the
default for on-orbit
operations
– System is too complex and
more measurements are
needed to understand the
timing behavior
4.1.9 - Integration and Test
•
Used trigger group discussions to
– Understand timing diagram with
TEM diagnostics on
– How to quantify effects
•
Thanks for the help
– Su Dong and Martin
– Mike
24
GLAST LAT Project
Managers F2F 6/15/05
Understanding Performance: TKR Settings
•
April 15: Anders looked at the variable
GemCondArrivalTimeTkr in the SVAC
ntuple and …
– he did not understand the spike at 31
•
•
•
4.1.9 - Integration and Test
Bill asked Anders whether one-shots
were used or not
Anders checked with Online (Lester)
and the answer was no
The trigger/ELX group was asked to
chime in and investigate with all
relevant parties whether we turn the
one-shot on or not during data
taking
– Discussion involved on-orbit
operations in the presence of
CNO events
•
May 25: Steve chaired a discussion
in which we decided to change the
way we will configure the TKR for
the 4 tower tests
– One shots were turned on
•
June: Anders is comparing 2 tower
data without one-shots with 4 tower
data with one shots
– stay tuned!
25
GLAST LAT Project
Managers F2F 6/15/05
Understanding Performance: CAL Retriggering
4.1.9 - Integration and Test
•
CAL FLE Retriggering identified
at NRL tests (Sasha)
– Need to study what happens
to FHE
•
Discussion will be brought up
today in the trigger meeting
– Benefits from Eric’s
presence at SLAC
26
GLAST LAT Project
Managers F2F 6/15/05
Understanding the Instrument: Housekeeping
•
Checked temperature
differences from CAL to TKR
– CAL is 1.5 degrees cooler
than top of TKR
– Differences can be
understood because of
convection
– Italy checked that
thermistors work fine
4.1.9 - Integration and Test
27
GLAST LAT Project
Managers F2F 6/15/05
Understanding Performance: Deadtime
•
Warren Implemented an offline
variable to measure time without
being affected by roll over of
counter
•
Measured deadtime in several
configurations
– Trying to understand numbers
right now
Proposed a new test to prescale hits
at the GEM level
– Similar to on-orbit operations
•
– enable more than one
trigger engine
– However, it stresses less the
TD&F
– Therefore keep current
prescaling tests
4.1.9 - Integration and Test
28
GLAST LAT Project
Managers F2F 6/15/05
Do we have enough data yet?
•
•
One tower
Two towers
– End-to-end cosmic ray data
– To exercise trigger and data flow capabilities
– SVAC cosmic ray data
– To perform calibrations
– To do performance studies that require large statistical sample
•
Four towers to Sixteen towers (prior to ACD integration)
– Only planned to collect SVAC data
– May request some tests (most likely between 6 and 8 towers) to
– Configure the instrument to take data with 2 Trigger Engines
(simulate on-orbit operation)
– Study nominal/high rates with TEM diagnostics OFF
– Understand TKR one-shot behavior
– Study TKR cable FIFO errors (caused by induced showers)
4.1.9 - Integration and Test
29
GLAST LAT Project
Managers F2F 6/15/05
Number of Talks in the IA meetings
After a year of effort, here we are….
SVAC
LAT Collaboration
30
25
Workshop 4
Scheduled for
July14-15, 2005
Workshop 2
Workshop 1
20
Workshop 3
15
10
5
0
Jun-72004
Jul-302004
Aug-132004
Sep-32004
Nov-52004
4.1.9 - Integration and Test
Nov-192004
Dec-102004
Jan-212005
Feb-182005
Mar-42005
Mar-182005
Apr-12005
Apr-152005
Apr-292005
30
May-132005
Jun-102005
GLAST LAT Project
Managers F2F 6/15/05
Contributions to Instrument Analysis Effort
SVAC
LAT Collaboration
100%
80%
60%
40%
20%
0%
Jun-72004
Jul-302004
Aug-132004
Sep-32004
Nov-52004
4.1.9 - Integration and Test
Nov-192004
Dec-102004
Jan-212005
Feb-182005
Mar-42005
Mar-182005
Apr-12005
Apr-152005
Apr-292005
31
May-132005
Jun-10
2005
GLAST LAT Project
Managers F2F 6/15/05
Data Analysis Efforts
•
We generated a list of 15 potential topics for
– Immediate analysis with data from 1 to 6 towers
– Presentations during IA workshop July 14-15
•
Working with subsystems (Neil,Dave,Ronaldo,Francesco) to
– Sharpen topics and ideas
– Identify names for presentations
•
Initiated contacts ACD
– Would like to have presentation during IA workshop
– If Dave can support that given the ACD schedule
– Need to discuss details for ACD tests at SLAC
– Test plans
– SAS Offline software
– Pipeline scripts and related infrastructue
4.1.9 - Integration and Test
32