GLAST LAT Project
November 18, 2004
GLAST Large Area Telescope:
Gamma-ray Large
Area Space
Telescope
Integration and Test
Two Tower
Integration Readiness Review
I&T Plan
Ken Fouts / Brian Grist
I&T Two Tower IRR
1
GLAST LAT Project
November 18, 2004
Two Tower Schedule
I&T Two Tower IRR
2
GLAST LAT Project
November 18, 2004
Integration Sequence
• Integration Sequence is defined in LAT-MD-00676-02 LAT Assembly
Sequence
• Two tower starts with the integration of Bay 9 and then Bay 8
• Optical surveys will be completed after each installation to
determine installation accuracy
I&T Two Tower IRR
3
GLAST LAT Project
November 18, 2004
On Bench
Single Bay Integration and Test Flow
TKR Module Post Ship
Test
LAT-PS-03290
TKR Module CPT
LAT-PS-03290
CAL Module Post Ship
Test
TEM/TPS Module
Post Ship Test
LAT-PS-03287
CAL Module CPT
LAT-PS-03287
TEM/TPS Module
CPT
LAT-TD-3415
LAT-PS-03062
In Flight like Single bay
LAT-PS-03058
TKR Module LPT
LAT-PS-03290
Cal Module Install
LAT-PS-03027
CAL Module LPT
LAT-PS-03287
TEM/TPS EICIT/IVT
LAT-PS-03875, 04097, 04098, 04099
CAL Module CPT
LAT-PS-03287
LAT-PS-03062
In Metrology
Bay
TRK Module Install
Cal Module LPT
LAT-PS-03287
TEM-TKR EICIT/IVT
LAT-PS-03875, 04097
Single Bay CPT
LAT-PS-03276
I&T Two Tower IRR
4
GLAST LAT Project
November 18, 2004
CALORIMITER INSTALLATION SEQUENCE
Remove CAL,
TEM/TEMPS
from Metrology
Bay
Install CAL,
TEM/TEMPS
into Single
Bay (TKR
Previously
installed)
Install CAL,
TEM/TEMPS
into Grid (TKR
Previously
installed)
Mate TKR
Connectors
to TEM
Invert Cal
Tower using
single Bay
Test
Mate TKR
Connectors
to TEM
Remove CAL,
TEM/TEMPS
from Single Bay
Invert
Grid
Install Cal
into
Metrology
Bay
Mate Connectors
from
TEM/TEMPS to
CAL
Invert
Single Bay
Demate
TKR Flex
Cables
Final
Test
Receive
CAL
Tower
Legend
Remove EM
TEM/TEMPS
(Demate Cal
Connectors)
Install and
shim Flight
TEM/TEMPS
Turnover Op
Test
Invert
Single Bay
Lift Operation
Manual Op
Connector
Mate/Demate
I&T Two Tower IRR
5
GLAST LAT Project
November 18, 2004
Tracker Installation Sequence
Receive
TKR
Tower
Invert
Single
bay
Demate TKR
Flex Cables
from TEM
Install CAL,
TEM/TEMPS
Mate TKR
Flex
Cables
Install TKR
Tower into
Single Bay
Test
Remove CAL,
TEM, TEM-PS, &
TKR Tower from
Single Bay
Invert
Grid
Invert
Grid
Invert
Single
bay
Invert
Single
bay
Remove TKR
Tower from
Single Bay
Install TKR
Tower into
Grid
Final
Test
Install Cal
Tower into
Single Bay
Mate TKR
Flex cables
to TEM
Install TKR
into alignment
fixture
Legend
Verify TKR
Alignment
Turnover Op
Lift Operation
Manual Op
Connector
Mate/Demate
I&T Two Tower IRR
6
GLAST LAT Project
November 18, 2004
Workaround for Late TEM Delivery
•
Late TEM delivery
– Complete Single Bay test with EM TEM/TPS
– Install TKR in Grid
• Tracker installation into the grid and cable work may take
additional development.
– EM TEM may be installed into the Grid with CAL and replaced
with a flight unit when available.
– Removal of the EM TEM and Installation of the Flight TEM/TPS
will require detailed height measurements in the metrology bay
to make custom shims.
– An optical survey or mechanical measurements will be used to
verify location after installation.
I&T Two Tower IRR
7
GLAST LAT Project
November 18, 2004
I&T Operations
I&T Two Tower IRR
8
GLAST LAT Project
November 18, 2004
I&T Organization
I&T Two Tower IRR
9
GLAST LAT Project
November 18, 2004
I&T Roles and Responsibilities
• I&T Management
– Ensure that all aspects of LAT integration and test planning are in place
and these plans are executed
– Define I&T priorities and resolve resource conflicts
– Provide program scheduling, cost accounting, performance tracking and
reporting for entire subsystem.
– Ensure the implementation of the LAT Integration and Test plan and the
LAT Performance and Operations test plan are achieved.
– Ensure that scientific performance and requirements are met per
applicable requirements.
– Support development of subsystem specifications, verification plans, and
interfaces with neighboring subsystems.
– Control subsystem environmental requirements and performance
metrics.
– Support team meetings and project reviews. Travel to meetings and site
visits for test support.
– Help develop and focus Collaboration support for beam test, data
analysis, calibration, environmental test, spacecraft and observatory.
I&T Two Tower IRR
10
GLAST LAT Project
November 18, 2004
I&T Roles and Responsibilities
• Integration Facilities, Configuration, and Test (IFCT) Department
– Develop I&T plans for the EM units and LAT.
– Develop operations and test procedures.
– Layout, spec, and procure equipment and facilities needed for LAT
integration at SLAC, specifically the LAT integration facility at SLAC
(Building 33).
– Train and certify technicians for LAT flight hardware handling.
– Provide on-going operational support of integration facilities and staff.
– Supervise I&T work in Integration Facility (Building 33).
– Record and maintain records of installation and logs of activities to
provide configuration control of the LAT during assembly and test.
I&T Two Tower IRR
11
GLAST LAT Project
November 18, 2004
I&T Roles and Responsibilities
• Mechanical Ground Support Equipment (MGSE) Department
– Design MGSE to support instrument integration and test.
– Support procurement/fabrication of all MGSE.
– Plan and coordinate the assembly and proof test of LAT MGSE.
– Provide support during integration.
I&T Two Tower IRR
12
GLAST LAT Project
November 18, 2004
I&T Roles and Responsibilities
• Online Software Department
– Develop, test, and support software used to collect data from the EM
units, CU and LAT test stands, saves them to persistent storage, and to
monitor and control the test stand and data acquisition system.
– Responsible for the online software, workstations on which the online
software operates, and network connections.
– The online software includes a test executive that communicates with the
embedded system to control the test stand, collects data from the
hardware under test, a scripting language, Graphical User Interface
(GUI) tools, a database, analysis and monitoring tools, and a mechanism
to archive data.
– Provide a code management and release control system.
– Responsible for the design and implementation of a standard test suite
(scripts) used to qualify the instrument and instrument components.
– The Electronics Subsystem is responsible for substantial elements of the
test stand. Thus, there will be close coordination between the Online
Software department and Electronics Subsystem.
– Online will provide a release control and code management system to
configuration control the Online software, this includes the all test scripts
used by I&T.
I&T Two Tower IRR
13
GLAST LAT Project
November 18, 2004
I&T Roles and Responsibilities
•
Particle Test Department
–
–
–
–
–
–
Develop, plan, and lead in the implementation of particle tests for the
EM units and LAT.
Particle tests will include ground cosmic rays and Van de Graaff
photons for the EM, Calibration Unit (CU), LAT and observatory levels.
These tasks require close coordination with all other departments in
I&T.
Design, construct, and maintain the Van de Graaff accelerator, its
associated diagnostic and support equipment, and scintillator telescope
for cosmic ray data acquisition.
Particle test leads in the design and equipment manufacture for the CU
beam test.
Particle Test is supported by the I&T MGSE department and the SLAC
Experimental Facilities Department (EFD).
I&T Two Tower IRR
14
GLAST LAT Project
November 18, 2004
I&T Roles and Responsibilities
•
Science Verification, Analysis, and Calibration (SVAC) Department
–
–
–
–
The SVAC department is responsible to coordinate and help perform the
data analysis for particle beam and cosmic ray tests.
These tests are designed to ultimately calibrate the LAT, and to validate the
Monte Carlo simulations that will be used to verify the LAT science
performance requirements.
The SVAC department plan builds gradually on the knowledge acquired from
low-level calibration tests on hardware units by the subsystems, in the
process of I&T, and from particle tests using EM units, and the LAT.
The SVAC manager is responsible for developing and leading the
collaboration support for offline data analysis of data taken during the LAT
integration and testing phases.
I&T Two Tower IRR
15
GLAST LAT Project
November 18, 2004
I&T Operating Plan
• The detailed LAT I&T schedule is maintained in MS Project by the I&T Scheduler.
– Schedules will be updated daily via the morning meeting with progress from
previous day.
– Once every other week I&T will hold a planning meeting with the scheduler and
interested or affected subsystems to agree on the plan for the next two weeks.
• Plan 2 shift operation
– Minimum staff will be 1 S&E Associate, 2 Technicians and a Test Conductor or
Test Director on shift (4 people per shift).
• Shifts will run 7:00am-3:30pm and 3:00pm-11:30pm
– Daily morning meeting at 7:00am
– Half hour overlap will facilitate a daily “hand-off” meeting at 3:00pm.
– Swing shift Test Conductor will hold an informal meeting at the end of the shift
and complete the Shift Log.
– Shift hand-off between 2nd shift and the following day will be accomplished via
shift log.
• Baseline plan is 2-8hr shifts/5 days a week.
– Extended shifts and/or Saturdays will be used to maintain schedule as required.
I&T Two Tower IRR
16
GLAST LAT Project
November 18, 2004
Two Shift Operation Flow
• Status of day shift test flow (Shift log review)
• Generate/Assign Swing Tasks from schedule.
• Items for Test Prep
1500 Daily Meeting
Integration Test Director
I&T Staff
• Shift log Review
• Status test flow from previous swing shift
• Assign days tasks from schedule.
0700 Daily Meeting
Integration Test Director
IFCT Staff
Thursday
Friday
Saturday
Sunday
Monday
1500 Meeting
0700 Meeting
0700 Meeting
Swing shift
Day shift
Day shift
1500 Meeting
1500 Meeting
Swing shift
Swing shift
I&T Two Tower IRR
17
GLAST LAT Project
November 18, 2004
I&T Test Roles and Responsibilities
Test Director
Larry Wai
Brian Grist
Brian Horwitz (Backup)
Test Conductors
Brian Horwitz (EE Lead)
John Canfield (S&E Assoc)
Toan Le (Backup)
Electrical Technicians
Toan Le
Phuc Hoang
Mechanical Engineers
Tom Borden (IFCT Engr)
Eliazar Ortiz (Lead)
Mechanical S&E Associates Mark Molini
Leo Manger
Reggie Rogers (MGSE)
Mechanical Technicians
Dave Kiehl
Tom Nieland
I&T Two Tower IRR
18
GLAST LAT Project
November 18, 2004
Problem Resolution in I&T
Problem Observed
Inspection, Testing,
Data Analysis
(HW, SW, Data)
Issue
Closed
False Alarm!
Corrective
Actions
Test Conductor
Error
No
Preliminary NCR
(FLAG)
Test
Procedure/Set Up
Error
Flight HW
Risk?
Test Equipment
Problem
Categorize
by
Test Conductor
and/or
Test Director
Yes
Possible
Flight HW Failure
NCR Issued
Troubleshoot
HW MRB
HW MRB
Root Cause Identified
Corrective Actions
Possible
Flight SW, DB
Failure
NCR Issued
Troubleshoot
SW, DB MRB
SW, DB MRB
Root Cause Identified
Corrective Actions
I&T Two Tower IRR
19
GLAST LAT Project
November 18, 2004
Troubleshooting in I&T
• No troubleshooting will be done without appropriate disposition on an
NCR.
• When an anomaly occurs:
– The test set-up is frozen
– The instrument and test equipment is safed.
– The test set-up and instrument configuration is reviewed and
documented by the Test Conductor.
– Test conductor notifies the TD, Subsystem Engineer and QA.
• QA initiates an NCR with information from TC and convenes MRB.
– At a minimum, MRB consists of the Test Director, Test Conductor,
Subsystem Engineer and QA.
• MRB will provide detailed disposition for troubleshooting.
• The test procedure will be redlined or a separate procedure will be
written (as defined by MRB) to document the troubleshooting activity
defined in the disposition.
I&T Two Tower IRR
20
GLAST LAT Project
November 18, 2004
Procedure Plan
•
•
•
•
•
•
Key Requirements
I&T Documentation Flow
I&T Procedure Development Process
I&T Standard Processes
Status
Configuration Management
I&T Two Tower IRR
21
GLAST LAT Project
November 18, 2004
Key I&T Requirement Documents
Document
LAT-MD-00408-03 LAT Instrument Performance Verification Plan
LAT-MD-02730 LAT Performance and Operations Test Plan
Released
Revision
11/8/04
Draft
LAT-MD-00676-02 LAT Assembly Sequence
3/12/04
LAT-MD-00895 LAT Instrument Survey Plan
Draft
LAT-MD-00446-06 SVAC Plan
10/5/04
LAT-MD-03489-02 Ad Hoc Committee on End-to-End Testing
6/25/04
LAT-DS-02561 LAT Tower and Electronics Assembly
Draft
LAT-DS-02560 LAT Tower Assembly
Draft
LAT-DS-02559-01 CAL-TEM-TPS Module Assembly
11/1/04
LAT-DS-00851-03 TRACKER-LAT IDD
10/11/04
LAT-DS-00233-06 CAL-LAT IDD
5/2/03
Requirements Definition Key to Successful I&T
I&T Two Tower IRR
22
GLAST LAT Project
November 18, 2004
I&T Documentation Flow
Test Performance and
Operation Plan
LAT-MD-02730
Drawings
(Mechanical Assembly,
Cabling, EGSE/MGSE
interface, etc.)
Test Sequence
LAT-MD-00676
Assembly Instruction
Data Sheet (AIDS)
I&T Operating
Processes and
Procedures
QA and
Archive
Task Specific Procedures
(Assembly, Test, Process, Etc.)
Procedure Data Sheet
Electronic Logbooks
(Component installation,
Mate/Demate, Etc.)
FLAG
Non-Conformance
Report (NCR)
I&T Two Tower IRR
23
GLAST LAT Project
November 18, 2004
I&T Procedure Process
• I&T Procedures are developed by the appropriate Electrical or Mechanical
Team members.
• Procedures are reviewed internally by I&T and then the draft is put in LAT
Docs for external review. Subsystems are notified by e-mail of the document
status.
• Whenever possible draft procedures are used to dry run assembly,
integration and testing of EM hardware.
• Document signature cycle is determined by Systems Engineering.
• I&T Procedures (including AIDS) are reviewed and signed off (at a minimum)
by:
– The document author
– Quality Engineer
– Subsystem Engineer (TKR, CAL, ELX, Mech., I&T)
– I&T Manager and/or I&T Engineering Manager.
• Daily “huddles” will be used prior to first use of procedures with flight
hardware to review the steps and “mistakeproof” the process.
• I&T Procedures will be executed by the appropriate Mechanical or Electrical
Technicians and/or Science & Engineering (S&E) Associates.
I&T Two Tower IRR
24
GLAST LAT Project
November 18, 2004
I&T Process Procedures
Document
LAT-PS-03481-02 GLAST LAT Integration Facility Emergency
Response Plan
LAT-PS-03263-01 Fastener Installation and Torqueing Procedure
LAT-PS-04727-01 Connector Mate and Demate
Released
Revision
11/4/04
In Signoff
9/3/04
LAT-MD-04601-01 LAT I&T eLog Book Implementation Plan
In Signoff
LAT-MD-01386-01 LAT Facilities Plan
In Signoff
LAT-PS-03291-01 Flight Hardware Bag and Purge Procedure
In Signoff
LAT-PS-04436-01 AeroGo Bearing Reference Guide
11/4/04
LAT-PS-04511-01 LAT Data Collection Procedure
Draft
LAT-MD-01533-01 LAT EGSE Plan
Draft
LAT-MD01462-02 LAT MGSE Plan
Draft
Document Standard Processes for use in I&T
I&T Two Tower IRR
25
GLAST LAT Project
November 18, 2004
Configuration Management
Figure 1. Controlled Document Release Process Flow Chart
• I&T follows the project CM plan, LATMD-00068.
• The figure on the right shows
document flow through CM
• The figure below from LAT-MD-00068
defines who is required to sign
Document Originator
Document Control Center
Document Approver
Review DCN and
Document for
completeness, work
with author to
correct if necessary
Step 4
Prepare controlled
document draft for
approval
Step 1
Prepare DCN and
reference
documentation
Step 2
Update CIDL
Step 5
Submit DCN and
document draft to
Document Control
Center
Step 3
Send document for
review
Step 6
No
Reviews
Document and
DCN
Step 7
Return Signed
DCN, approval
acknowledgement,
or disapproval
comments
Step 8
Approved?
Incorporate
Comments
Step 9
Yes
Send document for
rereview if
necessary
Step 10
Put released
document in
LATDOC (with
attached DCN)
Step 11
Notice is made to
DCN distribution
list via email
Step 12
I&T Two Tower IRR
26
GLAST LAT Project
November 18, 2004
Procedure Status
•
•
•
61 documents total
34 have been submitted for sign-off
20 have been released
I&T Two Tower IRR
27
GLAST LAT Project
November 18, 2004
Procedure Status (1 of 6) - Management
Green is complete
I&T Two Tower IRR
28
GLAST LAT Project
November 18, 2004
Procedure Status (2 of 6) - MGSE
Green is complete
I&T Two Tower IRR
29
GLAST LAT Project
November 18, 2004
Procedure Status (3 of 6) – IFCT Electrical
Green is complete
I&T Two Tower IRR
30
GLAST LAT Project
November 18, 2004
Procedure Status (4 of 6) – IFCT Mechanical
Green is complete
I&T Two Tower IRR
31
GLAST LAT Project
November 18, 2004
Procedure Status (5 of 6) - Particle
Green is complete
I&T Two Tower IRR
32
GLAST LAT Project
November 18, 2004
Procedure Status (5 of 6) - SVAC
Green is complete
I&T Two Tower IRR
33
GLAST LAT Project
November 18, 2004
Procedure Status – Burndown Chart
I&T Two Tower IRR
34
GLAST LAT Project
November 18, 2004
Backup Charts
I&T Two Tower IRR
35
GLAST LAT Project
November 18, 2004
Integration Facilities, Configuration, and Test (IFCT)
Department (1 of 2)
– IFCT Manager and IFCT Engineer:
• Managers for the department.
• Coordinated activities between IFCT staff.
• Oversees the coordination with other subsystem liaison test groups.
• Building manager and assistant building manager for the Integration Facility (SLAC
building 33).
– Mechanical Floor Engineer:
• Oversee and work on the mechanical and thermal aspects of the instrument assembly,
integration, and test operations.
• Supervise mechanical technicians. Develop flight hardware handling procedures and
training. Play a key role in developing the LAT assembly and integration procedures. Lead
the mechanical assembly, integration and testing of the LAT.
• Responsible engineer to the test and assembly procedures. Responsible for the Assembly
Instruction Data Sheets (AIDS).
• Supervises the mechanical technicians.
• Prepares and coordinates the daily tasks. This person is the test conductor for the
mechanical operations.
– Electrical Floor Engineer:
• Responsible engineer for the electrical test procedures.
• Supervises the electrical technicians.
• Prepares and coordinated the daily tasks.
• This person is the test conductor for the electrical testing.
I&T Two Tower IRR
36
GLAST LAT Project
November 18, 2004
Integration Facilities, Configuration, and Test (IFCT)
Department (2 of 2)
–
–
Mechanical Technicians:
•
Responsible for hands on mechanical assembly of the LAT under the supervision of the
Mechanical Floor Engineer.
•
Follow written procedures and Assembly Instruction Data Sheets (AIDS).
•
Review and implement procedures for integration and test of the LAT.
•
Support fabrication and assembly of integration support equipment. Support calibration of
Instrumentation and the LAT instrument.
•
Conduct test under controlled conditions and report results.
•
Conduct daily integration operations assigned by lead technician/floor engineer. Conduct crane
operation.
•
Identify, maintain and inventory tools and support equipment related to LAT mechanical
integration.
•
The mechanical technicians are responsible for the physical mate demate of flight connectors.
Electrical Technicians:
•
Responsible to execute written electrical test procedures under the direction of the Electrical
Floor Engineer.
•
Follow written procedures and Assembly Instruction Data Sheets (AIDS).
•
Will perform tests using automated test and diagnostic equipment, troubleshooting digital and
analog interfaces using oscilloscopes, logic and protocol analyzers, multi-meters, and signal
generators, and perform interface verifications.
I&T Two Tower IRR
37
GLAST LAT Project
November 18, 2004
I&T Flow For Tower Installation
Receive
CAL
Tower
Receive
TKR
Tower
Install TKR
into alignment
fixture
Measure
TKR and
Align
Cones
Install Cal
Tower into
Metrology
Bay
Inspect,
Metrology
& El Test
Remove CAL,
TEM, TEM-PS, &
TKR Tower from
Single Bay
Install TKR
Tower into
Grid Bay
Shim Flight
TEM/TEMPS
Install TKR
Tower into
Single Bay
Test
Install
CAL, TEM,
& TEM-PS
Test
Install CAL
With Flt
TEM/TEMPS
Prepar
e
Grid
Bay
Final
Test
I&T Two Tower IRR
38