GLAST LAT Project Gamma-ray Large Area Space Telescope GSDR, 19 August 2004 GLAST Large Area Telescope: LAT Instrument Science Operations Center Overview Ground Systems Design Review Bill Craig Stanford Linear Accelerator Center bcraig@slac.stanford.edu 650-926-2983 LAT-PR-TBD Section 1.0 1 GLAST LAT Project GSDR, 19 August 2004 Outline Objectives & Overview, Pipeline – (Craig) 25 minutes Requirements – (Martin) 10 minutes Command, Health and Safety, Ops Scenarios – (Bator) 20 minutes Tools and Architecture – (Culp) 20 minutes Management, Cost, Schedule and Risks – (Craig) 15 minutes LAT-PR-TBD Section 1.0 2 GLAST LAT Project GSDR, 19 August 2004 LAT ISOC’s Role in the GLAST GDS Analysis Software LAT-PR-TBD Section 1.0 3 GLAST LAT Project GSDR, 19 August 2004 Objectives The LAT ISOC is organized to: – safely operate the instrument and – produce the LAT’s science data products Functions: – Maintain and modify FSW and Testbed – Command generation, health and safety monitoring – Performance verification and optimization – Process and archive Level 1 and Level 2 data – Develop and optimize the software pipeline that produces the higher level products These functions are organized as teams that share personnel LAT-PR-TBD Section 1.0 4 GLAST LAT Project GSDR, 19 August 2004 ISOC Functional Organization ISOC Manager Databases Command, H&S FSW Documentation Performance V&O Science Products Science Analysis Software Commanding Thermal Control Calibration Level 1 CRs, gammas, diagnostics Health & Safety Trigger Performance Trending Level 2 Transients, GRBs Calibration Configuration Filter Standard Source Monitoring Pipeline Server Reconstruction & Event Classif. Testbed IRF Generation LAT-PR-TBD Simulation Transient Source Analysis Section 1.0 5 GLAST LAT Project GSDR, 19 August 2004 ISOC REVIEWS and RFA STATUS Reviews LAT CDR ISOC Peer Review ISOC CDR No. RFA Summary Date 12-16 May 2003 2 March 2004 4 Aug 2004 Requestor Actionee Status 1 ISOC Documentation Tree E. Andrews L. Bator Submitted 2 Requirements Traceability Tool E. Andrews J. Martin Plan submitted 3 Add ISOC Architecture Diagram E. Andrews B. Craig Submitted 4 ISOC Requirements and Testing E. Andrews J. Martin ECD 10/1/04 5 SAA Handling Approach E. Andrews L. Bator ECD 8/31/04 6 Launch Critical Support Plan M. Rackley J. Leibee S. Culp ECD 8/27/04 7 Red/Yellow Limit Philosophy J. Leibee L. Bator ECD 8/27/04 8 SAS Verification Approach M. Rackley B. Craig J. Martin Submitted LAT-PR-TBD Section 1.0 6 GLAST LAT Project GSDR, 19 August 2004 LAT ISOC Architecture R/T HK & Cmd Mission Operations Center HSK GLAST Science Support Center Command & Science Planning Level 0 Data Data Processing Commanding, Health & Safety Performance Verification Level 1 & Level 2 Instrument Response Command validation Flight Software & Testbed Instrument Optimization Calibrations Science Analysis Software Data Transfer Reports and Information LAT ISOC Steering Committee LAT-PR-TBD Support Functions Through Deliverables Advisory and/or Revision Role Section 1.0 LAT Science Analysis 7 GLAST LAT Project GSDR, 19 August 2004 Commanding, Health and Safety The Commanding, Health and Safety (CHS) team is responsible for: – generation and validation of commands and command sequences, – passing commands on to the GSSC, – verifying these commands were executed, – receiving Level 0 data from the MOC, – logging and archiving of all commands and Level 0 data, – monitoring that data to ascertain and track the health and safety of the instrument, – continuous knowledge of the configuration of the LAT. LAT-PR-TBD Section 1.0 8 GLAST LAT Project GSDR, 19 August 2004 CHS in the ISOC Architecture R/T HK & Cmd Mission Operations Center HSK GLAST Science Support Center Command & Science Planning Level 0 Data Data Processing Commanding, Health & Safety Performance Verification Level 1 & Level 2 Instrument Response Command validation Flight Software & Testbed Instrument Optimization Calibrations Science Analysis Software Data Transfer Reports and Information LAT ISOC Steering Committee LAT-PR-TBD Support Functions Through Deliverables Advisory and/or Revision Role Section 1.0 LAT Science Analysis 9 GLAST LAT Project GSDR, 19 August 2004 Flight Software The Flight Software (FSW) team is responsible for: – Updating and validation of all flight software files, – Debug or problem fixes to the FSW, – Implementation, and validation on the instrument test bed of authorized upgrades to FSW, – Continuing maintenance of the instrument test bed to ensure it is available to validate code and command sequences as well as to investigate any anomalies seen on orbit LAT-PR-TBD Section 1.0 10 GLAST LAT Project GSDR, 19 August 2004 FSW in the ISOC Architecture R/T HK & Cmd Mission Operations Center HSK GLAST Science Support Center Command & Science Planning Level 0 Data Data Processing Commanding, Health & Safety Performance Verification Level 1 & Level 2 Instrument Response Command validation Flight Software & Testbed Instrument Optimization Calibrations Science Analysis Software Data Transfer Reports and Information LAT ISOC Steering Committee LAT-PR-TBD Support Functions Through Deliverables Advisory and/or Revision Role Section 1.0 LAT Science Analysis 11 GLAST LAT Project GSDR, 19 August 2004 FSW and ISOC Interactions Through FSQ – Coding the software that will operate the LAT – Testing the software – Configuration management – Development of Test Bed Beyond FSQ – Continue code/test/CM cycle as part of the ISOC operations – Maintenance of Test Bed – Review commanding, HK, and performance on a frequent and regular basis LAT-PR-TBD Section 1.0 12 GLAST LAT Project GSDR, 19 August 2004 FSW Testbed Block Diagram SIIS “Solid State Recorder” Discretes (to SIU PIDs) Event Builder (EB) output side. The EB is an element of the GASU. To SSR To SIU To EPU(s) Spacecraft Interface Unit 1553 Rx service LCB Rx service Other Tasks MSG 1 PPS GBM service Interrupt Interrupt 1553 Q Q SC Att./Time Q Q Instr. Phys. Legend Q Q File/Object Telecommand (SC to LAT) Q Q HSK Telemetry (LAT to SC) Q Q Watchdog Master to slave Q Q TCS Slave to master Physics data from LAT Data to SSR Command/Response LAT-PR-TBD Event Processing Unit(s) Other Tasks 1 PPS MSG Interrupt service LCB Rx Q Q Q Q Q Q Q 1553 Tx service Q SC Att./Time Q Q Q File/Object HSK Watchdog LCB Tx service Sensors replaced with Front End Simulators (FES) Slaves Slaves Masters LAT Instrument Q Q Q SC Att./Time Instr. Phys. File/Object HSK Watchdog Q LCB Tx service Q From SIU From EPU(s) Event Assembly Event Builder (EB) input side. The EB is an element of the GASU. Command/Response Unit (CRU). The CRU is an element of the GASU. Section 1.0 13 GLAST LAT Project GSDR, 19 August 2004 FSW Status Team is in place Flight Unit level coding is underway Testbed is well underway: functional, needs to be fully validated SW Integration phase is Q4 – Builds produced as needed to support I&T SW Test Scripts underway, completion in Q4 FSW Qualification Testing begins January 2005 LAT-PR-TBD Section 1.0 14 GLAST LAT Project GSDR, 19 August 2004 Performance Verification and Optimization The Performance Validation and Optimization team (PVO) is responsible for: – instrument calibration from low level through IRFs – continuous monitoring of the LAT science performance, identification of instrument performance trends and resolution of anomalies – generation and initial validation of algorithms that improve on-orbit performance of the LAT – management of test and calibration data collected prelaunch – Configuration and maintenance of the LAT reference geometry and the LAT Monte Carlo Model LAT-PR-TBD Section 1.0 15 GLAST LAT Project GSDR, 19 August 2004 PVO in the ISOC Architecture R/T HK & Cmd Mission Operations Center HSK GLAST Science Support Center Command & Science Planning Level 0 Data Data Processing Commanding, Health & Safety Performance Verification Level 1 & Level 2 Instrument Response Command validation Flight Software & Testbed Instrument Optimization Calibrations Science Analysis Software Data Transfer Reports and Information LAT ISOC Steering Committee LAT-PR-TBD Support Functions Through Deliverables Advisory and/or Revision Role Section 1.0 LAT Science Analysis 16 GLAST LAT Project GSDR, 19 August 2004 PVO Analysis Every data set to be analyzed must have information available on the web for easy access on – Hardware configurations – Register settings used for data taking – Quality reports for digitized data – Quality reports for reconstructed data Report generation – Being implemented as I&T exercises with Engineering Models – Will continue to develop during transition phases between I&T and ISOC The basic infrastructure required for the ISOC PVO team is in place for I&T and forms the basis of our implementation. LAT-PR-TBD Section 1.0 17 GLAST LAT Project GSDR, 19 August 2004 Query List of Runs via the Web Run number Date Particle source Hardware type Hardware orientation LAT-PR-TBD Section 1.0 18 GLAST LAT Project GSDR, 19 August 2004 Configuration Report Register Settings LAT-PR-TBD Section 1.0 19 GLAST LAT Project GSDR, 19 August 2004 TKR calibrations example Tests with data from EM1 prototype Dead strip xml file Output Calibration data Y3 Y2 Y1 LAT-PR-TBD <uniplane tray="4" which="bot" > <stripSpan first= "0" last= "156" /> </uniplane> <uniplane tray="2" which="top" > <stripList strips= " 561 1073 1445 1449 1464 1487 " /> </uniplane> <uniplane tray="2" which="bot”> <stripSpan first= "1151" last= "1535" /> "</uniplane> Section 1.0 20 GLAST LAT Project GSDR, 19 August 2004 Science Products The functions developed by the Science Analysis Software (SAS) subsystem of the LAT are leveraged by the Science Products team to provide deliverables for ISOC The Science Products Team is responsible for – Generation, archiving and distribution of the Level 1 data, – Generation, archiving and distribution of specific Level 2 data needed for reference, – Transient source detection, – Configuration control over all pipeline code and generated data sets. All science products requirements currently satisfied by work already completed by the SAS subsystem. LAT-PR-TBD Section 1.0 21 GLAST LAT Project GSDR, 19 August 2004 SP in the ISOC Architecture R/T HK & Cmd Mission Operations Center HSK GLAST Science Support Center Command & Science Planning Level 0 Data Data Processing Commanding, Health & Safety Performance Verification Level 1 & Level 2 Instrument Response Command validation Flight Software & Testbed Instrument Optimization Calibrations Science Analysis Software Data Transfer Reports and Information LAT ISOC Steering Committee LAT-PR-TBD Support Functions Through Deliverables Advisory and/or Revision Role Section 1.0 LAT Science Analysis 22 GLAST LAT Project GSDR, 19 August 2004 Science Analysis Software The Science Analysis Software Team is responsible for – Development and maintenance of the pipeline machinery – Development and maintenance of the simulation, reconstruction and event classification software – Development and maintenance of the calibration algorithms, including low level and Instrument Response Functions – Development and maintenance of the quicklook transient analyses – Development and maintenance of the high-level diagnostics derived from reconstruction and classification – Development and maintenance of the high-level analysis tools – “Help desk” support of ISOC staff LAT-PR-TBD Section 1.0 23 GLAST LAT Project GSDR, 19 August 2004 SAS in the ISOC Architecture R/T HK & Cmd Mission Operations Center HSK GLAST Science Support Center Command & Science Planning Level 0 Data Data Processing Commanding, Health & Safety Performance Verification Level 1 & Level 2 Instrument Response Command validation Flight Software & Testbed Instrument Optimization Calibrations Science Analysis Software Data Transfer Reports and Information LAT ISOC Steering Committee LAT-PR-TBD Support Functions Through Deliverables Advisory and/or Revision Role Section 1.0 LAT Science Analysis 24 GLAST LAT Project GSDR, 19 August 2004 Overall SAS Test Approach Combination of Engineering Model tests, Data Challenges and LAT Integration Support EM tests – EM1 demonstrated ability to simulate/reconstruct real data from single (non-standard) tower • All within standard code framework/tools • Data analyzed with SAS tools Data Challenges – End to end tests of sky simulation through astro analysis – Exercise pipeline – Involve test engineer to check off requirements as part of the DCs LAT Flight Integration – Combine tools from EM & DC applications – Sim/recon/analysis & pipeline processing and record keeping LAT-PR-TBD Section 1.0 25 GLAST LAT Project GSDR, 19 August 2004 Main Science Tools Package Likelihood Level 1 database access Description First Use Workhorse model fitting for detection & characterization of cosmic gamma-ray sources DC1 Extracts desired event data DC1 Exposure calculation Uses IRFs, pointing, livetime etc. for deriving calibrated source fluxes DC1 Source identification Identifies gamma-ray sources with cataloged counterparts at other wavelengths DC2 GRB analysis Temporal and spectral analyses of burst profiles DC1 Pulsar analysis Phase folding & period searching of gamma-ray pulsars and candidates DC2 High level simulation of observations of the gamma-ray sky with the LAT DC1 Observation simulator DC1 = Data Challenge One, February 2004 LAT-PR-TBD Section 1.0 26 GLAST LAT Project GSDR, 19 August 2004 Science Tools Toolkit Package Description Provider Status PIL, PIL++ IRAF parameter access HEASARC In use cfitsio, CCFits FITS file manipulation HEASARC In use XSPEC, Sherpa For GRB spectral modeling HEA standards Under consideration Root gui etc HEP standard Under consideration python Scripting World standard Under consideration doxygen Code doc tool World standard In use Visual C++/gnu Development envs World standards In use CMT Code mgmt tool HEP standard In use cvsweb Cvs web viewer World standard In use cvs File version mgmt World standard In use LAT-PR-TBD Section 1.0 27 GLAST LAT Project GSDR, 19 August 2004 Example of FRED LAT-PR-TBD Section 1.0 28 GLAST LAT Project GSDR, 19 August 2004 Disk and Archives We expect ~10 GB raw data per day and assume comparable volume of events for MC – Leads to ~40 TB/year for all data types • Not a challenge – keep it all on disk • Have funding approval for up to 200 TB/yr – Use SLAC’s mstore archiving system to keep a copy in the silo • Already practicing with it and will hook it up to Gino – Archive all data we touch; track in dataset catalogue – Not an issue LAT-PR-TBD Section 1.0 29 GLAST LAT Project GSDR, 19 August 2004 Security Network security – application vs network – ssh/vpn among all sites – MOC, SSC and internal ISOC – A possible avenue is to make all applications secure (ie encrypted), using SSL. File and Database security – Controlled membership in disk ACLs – Controlled access to databases – Founded on SLAC security (well maintained, high standard) LAT-PR-TBD Section 1.0 30 GLAST LAT Project GSDR, 19 August 2004 Status of SAS EMs, DCs and Flight Integration will leave us ready for flight EM1 worked with our tools DC1 worked well, showing very good capabilities from sky modeling through astronomical analysis – Plenty of work still to do, but reasonably understood – Will be demonstrated in DC2, 3 and LAT Integration, 16tower cosmic ray tests and the beam test prior to launch LAT Flight Integration prep in full swing now DC2 within a year (being negotiated) LAT-PR-TBD Section 1.0 31 GLAST LAT Project GSDR, 19 August 2004 Summary The majority of code needed for ISOC activities; i.e. the science data pipeline and analysis toolkit, is very mature. The verification and optimization tools are being developed by a strong team in I&T with excellent linkages to ISOC Overall architecture, and a strong staff, are in place and development beginning LAT-PR-TBD Section 1.0 32 GLAST LAT Project Gamma-ray Large Area Space Telescope GSDR, 19 August 2004 GLAST Large Area Telescope: LAT Instrument Science Operations Center Requirements and Traceability Jim Martin SLAC jfm@slac.stanford.edu 650-926-4895 LAT-PR-TBD Section 1.0 33 GLAST LAT Project GSDR, 19 August 2004 Program Plan Rqmts 433-PLAN-0008 Grnd Sys Rqmts Doc 433-RQMT-0006 Ops Data Products ICD 492-MOC-009 Science Data Products ICD Ops Concept Doc. 433-OPS-0001 MOC Test Plan 492-MOC-007 LAT Inst Perf. Spec. Level II(B) LAT-SS-00010 LAT ISOC Spec. Level II(B) LAT-SS-00015 LAT Science Verif & Calib Plan LAT-MD-00446 LAT T&DF Spec LAT-SS-00019 ISOC Level III NASA Level IV LAT Level II NASA Level III LAT MISSION ISOC Document Tree LAT SAS Level III Requirements LAT-SS-00020 LAT Operations Plan LAT-SS-01378 LAT ISOC Verif. & Valid. Plan LAT-SS-02513 Instrument Ops Handbook ISOC Test Plan LAT ISOC Level III Requirements LAT-SS-00021 I&T Calibration Plan ISOC Level IV LAT Ops Facility Spec Tlm & Cmd Handbook ISOC Ground S/W Ops Tools Description Pipeline & Archiving Plan FSW & Testbed Maintenance Plan LAT Config Mgmt Plan LATMD-00068 LAT ISOC Mgmt & Dev Plan LAT-MD-04443 ISOC Config Mgmt Plan ISOC CCB Process ISOC Team Coordination Plan LAT FSW Spec LAT-SS-00399 LAT SAS Lvl 4 Spec LAT-SS-00505 LAT Project Mgmt Plan LAT-MC-00054 ISOC/I&T Transition Document Change Request Procedure ISOC Test Procedures On-orbit Ops Procedues Launch & Early Orbit Operations ISOC Calibration Plan LAT Timeline Generation Note: Documents without numbers have not been written yet Rev. 8/12/04 LAT-PR-TBD Section 1.0 34 GLAST LAT Project GSDR, 19 August 2004 Level II Requirements Level II Science Operations 6 Instrument Operations 17 Analysis Software and Data Processing 20 Interfaces 3 Data Standards and Data Format Standards 7 Availability and Reliability 1 Automation 1 Security 1 TOTAL 56 LAT-PR-TBD Section 1.0 35 GLAST LAT Project GSDR, 19 August 2004 REQUIREMENTS – Level II – LAT-SS-00015 (example) Science Operations # Title Summary Verification 4.2.1 LAT Instrument Calibration Produce, monitor and update calibration constants for the instrument T 4.2.2 Instrument Response Functions (IRFs) Generate IRFs needed for science analysis T 4.2.3 LAT Instrument Performance Assess the LAT instrument performance and compare actual performance against predicted performance. T 4.2.4 LAT Instrument Optimization Optimize the LAT instrument in response to in-flight changes in hardware as described in the ISOC Operations Plan, LAT-SS01378. T 4.2.5 Performance Records Archive data on the performance and calibration of the LAT instrument. D 4.2.6 Retrieval of Archived Records Retrieve archived performance and calibration data at a rate of at least 1 GB in 60 minutes. T LAT-PR-TBD Section 1.0 36 GLAST LAT Project GSDR, 19 August 2004 Driving Requirements 4.3.7 - LAT Instrument Commanding – Generate LAT instrument commands and identify procedures to direct the safe upload of the commands. 4.3.9 - LAT Instrument Flight Software – Maintain, update, and validate the operation and performance of the LAT flight software. 4.3.10 - LAT Instrument Simulator – Maintain an instrument simulator for validating and verifying changes to flight software and command procedures, and also for use in anomaly resolution. 4.4.1 - Level 0 Data Processing – Receive Level 0 data from the MOC post-pass and process the data into Level 1 event data within 24 hours of receipt from the MOC. 4.4.2 - Real-time Housekeeping Data – Autonomously receive real-time housekeeping data in near real-time from the MOC and process the data for diagnostic use. 4.4.6 - Instrument Response Simulation – Develop and maintain software to simulate the detailed response of the LAT to charged particles and gamma rays. 4.4.5 - Event Reconstruction and Classification – Develop algorithms to interpret the subsystem responses, apply calibration constants, and to find and identify incident photons. LAT-PR-TBD Section 1.0 37 GLAST LAT Project GSDR, 19 August 2004 REQUIREMENTS – Level III – LAT-SS-00021 # of Level III 3.1 Configuration and Architecture 3.1.1 Interface 3.1.2 Facilities 3.1.3 Redundancy 3.1.4 Security 3.1.5 Database 3.1.6 Website 3.1.7 Documentation 60 10 11 6 6 5 6 4 3.2 Mission Planning & Scheduling 3.2.1 Flight Dynamics 3.2.2 TDRSS scheduling 3.2.3 Target of Opportunity 3.2.4 Stored Command Load Generation 62 3 2 8 49 LAT-PR-TBD Section 1.0 38 GLAST LAT Project GSDR, 19 August 2004 REQUIREMENTS – Level III – LAT-SS-00021 Continued – 2/3 # of Level III 3.3 3.3.1 3.3.2 3.3.3 3.3.4 3.3.5 3.3.6 LAT-PR-TBD Telemetry, Command & Data Processing Telemetry Processing Data Processing Data Archiving Alert Telemetry Monitoring User Interface Language Commanding 85 29 19 10 1 22 4 Section 1.0 39 GLAST LAT Project GSDR, 19 August 2004 REQUIREMENTS – Level III – LAT-SS-00021 Continued – 3/3 # of Level III 3.4 3.4.1 3.4.2 3.4.3 3.4.4 3.4.5 3.4.6 3.4.7 3.4.8 3.4.9 3.4.10 3.4.11 LAT-PR-TBD Monitoring and Analysis Ground System Monitoring Display Pages Sequential Prints Event Messages Memory Mapping & Maintenance Limit Monitoring Configuration Monitoring Trending & Analysis Anomaly Tracking & Notification Timeline Monitoring Calibration & Performance 176 4 38 15 13 7 14 15 32 29 3 6 Section 1.0 40 GLAST LAT Project GSDR, 19 August 2004 Traceability and Testing Level II and Level III moved to Doors – Leverages LAT’s use of Doors for fabrication phase Have begun mapping – II to III – II to higher levels Will include testing: references to documents describing procedures and results LAT-PR-TBD Section 1.0 41 GLAST LAT Project GSDR, 19 August 2004 Summary Requirements are in the signature cycle – Review and update as needed after GSDR Development approach tailored to Level III Ready to develop LAT-PR-TBD Section 1.0 42 GLAST LAT Project GSDR, 19 August 2004 GLAST Large Area Telescope: Gamma-ray Large Area Space Telescope Ground Systems Design Review ISOC Command, Health and Safety Design Lori Bator SLAC ISOC Lead Engineer lbator@slac.stanford.edu (650) 926-5352 LAT-PR-TBD Section 1.0 43 GLAST LAT Project GSDR, 19 August 2004 Commanding, Health and Safety The Commanding, Health and Safety (CHS) team is responsible for: – generation and validation of commands and command sequences, – passing commands on to the GSSC, – verifying these commands were executed, – receiving Level 0 data from the MOC, – logging and archiving of all commands and Level 0 data, – monitoring that data to ascertain and track the health and safety of the instrument, – continuous knowledge of the configuration of the LAT. LAT-PR-TBD Section 1.0 44 GLAST LAT Project GSDR, 19 August 2004 CHS System ITOS used for command, health and safety functions – HK data limit checking – Telemetry and command definition file validation – Command load verification and validation Other tools – Level 0 receipt and archiving – HK trending – Data transmission – Mission planning and generation of file uploads – Anomaly tracking and notification – Relational database queries for trending and analysis – Configuration management tools LAT-PR-TBD Section 1.0 45 GLAST LAT Project GSDR, 19 August 2004 CHS Team (1/3) CHS Team Lead Instrument Physicist (part time PVO) LAT-PR-TBD Software Engineer Operator #1 Section 1.0 Operator #2 46 GLAST LAT Project GSDR, 19 August 2004 CHS Team (2/3) Weekday, day-time operations – Operator shift coverage • 5 am to 2 pm to cover MOC shift times on East Coast • 9 am to 6 pm to cover ISOC internal coordination needs – On-call support • Operators for real-time commanding or anomaly support • Software engineer for emergency software support • Team Lead and Instrument Physicist for anomalies Instrument Physicist – Monitor HK data for impact on science – Monitor and maintain configuration of instrument – Schedule instrument commanding LAT-PR-TBD Section 1.0 47 GLAST LAT Project GSDR, 19 August 2004 CHS Team (3/3) Software Engineer Responsibilities – Maintain ITOS configuration and other CHS tools – Maintain configuration control of command and telemetry database Operator Responsibilities – Generate LAT commands – Interface with MOC and GSSC – Monitor CHS system processes – Monitor real-time contacts during the shift – Monitor automated report generation – Respond to notification of alerts and anomalies – Serve as backup for Software Engineer to address system problems LAT-PR-TBD Section 1.0 48 GLAST LAT Project GSDR, 19 August 2004 Ops Data Products: ISOC – MOC Level 0 Data ISOC RT HK Telemetry Flight Dynamics Products MOC Science Timeline Integrated Observatory Timeline As-flown Timeline PDB Updates LAT Alerts LAT Instrument Commands LAT Instrument Loads LAT Operations Procedures SC Incident Reports LAT Incident Reports Data Retransmission Request LAT T&C Database Updates LAT-PR-TBD Section 1.0 49 GLAST LAT Project GSDR, 19 August 2004 Ops Data Products: ISOC – GSSC ISOC Long Term Science Schedule GSSC Preliminary Science Timeline LAT Timeline LAT Instrument Commands LAT Instrument Loads Observatory Timeline Package LAT-PR-TBD Section 1.0 50 GLAST LAT Project GSDR, 19 August 2004 GLAST Large Area Telescope: Gamma-ray Large Area Space Telescope Ground System Design Review ISOC Operations Planning & Scenarios Lori Bator SLAC lbator@slac.stanford.edu 650-926-5352 LAT-PR-TBD Section 1.0 51 GLAST LAT Project GSDR, 19 August 2004 LAT Operations Planning and Command Process Mission Operations Center (MOC) GLAST Science Support Center (GSSC) Observatory Timeline Package Long Term Science Schedule As-Flown Timeline Spacecraft Timeline Integrated Observatory Timeline As-Flown Timeline Preliminary and Final Science Timelines LAT Timeline ISOC LAT Timeline ATS Commands LAT Planning Operations Plan LAT Timeline Build & Review Verification & Validation File Uploads Command Procedures Feedback Rev 8/12/04 LAT-PR-TBD Section 1.0 52 GLAST LAT Project GSDR, 19 August 2004 Science Planning Phase 0/1: ISOC maintains LAT science plan – Organization of the scientific activities of the LAT collaboration outside of the ISOC is still being defined • Topic at next collaboration meeting, Sept. 27-29 at SLAC – Committee with collaboration and probably Project Scientist representation to oversee LAT operations planning and serve as the interface between science and operations • Acceptance of updated algorithms for the LAT trigger or event filtering (onboard or ground) • Definitions of conditions when the LAT will autonomously request a repointed observation Phase 2: GSSC generates Long Term Science Schedule – GSSC assists in overall science schedule evaluation and will manage the guest investigator proposal process – Coordinated with LAT collaboration – Input to LAT Planning LAT-PR-TBD Section 1.0 53 GLAST LAT Project GSDR, 19 August 2004 Operations Plan Description of LAT operations for one month Derived from – Evaluated performance of the LAT – As-flown timeline – Science plan [Phase 1] or Long Term Science Schedule [Phase 2] – Calibration needs In general the operations plan will have no impact on the pointing of GLAST – For a scanning sky survey (Phase 1 and likely also most of Phase 2), the ‘science activities’ are the same all the time – Calibration needs occasionally will require special observing modes or scheduling of more than the usual number of TDRS contacts – Most calibration data taking will not require specific orientations for GLAST • Occasional exceptions will be, e.g., pointed observation of ‘standard candle’ celestial sources or possibly scans of the Earth limb and the nadir to characterize the albedo background LAT-PR-TBD Section 1.0 54 GLAST LAT Project GSDR, 19 August 2004 LAT Timeline Contains all LAT commanding information to fulfill a one-week period of the Operations Plan – Commands for Absolute Time Sequence (ATS) – File loads • Table uploads • Configuration changes • FSW loads – Command procedures • Commands requested to be sent in real-time Generated by CHS team about 2 weeks before upload Coordinated with GSSC, MOC and GBM through weekly planning meetings Validated and verified on testbed LAT-PR-TBD Section 1.0 55 GLAST LAT Project GSDR, 19 August 2004 Status Continuing to finalize details of commanding process with GSSC and MOC through – Weekly GOWG meetings – Ops TIMs – Operations Data Products ICD (492-MOC-009) – Operations Agreement LAT operations planning documented in ISOC Operations Plan (LAT-SS-01378) LAT-PR-TBD Section 1.0 56 GLAST LAT Project GSDR, 19 August 2004 LAT Operations Scenarios LAT-PR-TBD Section 1.0 57 GLAST LAT Project GSDR, 19 August 2004 LAT Operations Phases LAT I&T – Pre-FSW (EM) – Post-FSW (Flight unit level) Observatory I&T – NRL – Spectrum Launch and Early Orbit (L&EO) – Phase 0 – LAT power-on and configuration – Initial checkout First year – Phase 1 – Survey mode Second and subsequent years – Phase 2 – Pointed observation – Survey mode LAT-PR-TBD Section 1.0 58 GLAST LAT Project GSDR, 19 August 2004 Pre-launch Testing LAT I&T Phase Tests – Verify and validate development of ISOC tools and functions on testbed and through software simulation Data Challenges – “End to end” test of science analysis software Ground Readiness Tests (GRTs) – Verify and validate system interfaces and data flows Observatory I&T Phase Tests – Verify and validate ISOC capabilities End-to-End (ETE) tests – Verify and validate interface between ground system and observatory Mission Simulations – Verify operation team readiness prior to launch LAT-PR-TBD Section 1.0 59 GLAST LAT Project GSDR, 19 August 2004 Initial Turn-On and Checkout (Phase 0) Turn-on (power-up) procedure will not be executed automatically – Humans required to check environmental conditions prior to significant steps – Must establish correct LAT configuration to ensure communications Functional checkout of DAQ, ACD, CAL and TKR Perform initial calibrations Special requirements – ACD will have control of triggering; therefore no science data taking available for ~ 3 days – Monitor phototube high voltages in ACD during turn-on LAT-PR-TBD Section 1.0 60 GLAST LAT Project GSDR, 19 August 2004 Science Operations (Phases 1 and 2) Data taking – Continuous – A few commands to initiate Calibration – Weekly, biweekly and monthly – A few commands to initiate Load changes to tables and FSW – Infrequent – A few commands and/or file uploads which may be large Load new tables and files – Infrequent – A few commands and/or file uploads which may be large Diagnostics – Infrequent – A few commands and/or file uploads which may be large SAA management – FSW turns down high voltage automatically based on SAA message from S/C LAT-PR-TBD Section 1.0 61 GLAST LAT Project GSDR, 19 August 2004 Monitoring Requirements LAT FSW monitors parameters onboard and will take action (power-off or stop activity) in response to limit violations S/C monitors some temperatures and will power off LAT if needed MOC monitors HK data for limit violations – No critical monitoring or actions required – ISOC will provide limits in LAT T&C database – ISOC will provide and direct use of any contingency procedures ISOC monitors HK and Science data for limit violations and trends – Respond to degradation, noise, changes in performance, and failure by uploading commands or files to change configuration – No critical monitoring except during power-up LAT-PR-TBD Section 1.0 62 GLAST LAT Project GSDR, 19 August 2004 Status ISOC operations concept documented in LAT-SS-01378 LAT Operations Plan Detail on operations will evolve from FSW development and I&T tests – Detail will be captured in Operations Handbook (ECD Oct 2005) and Operations Procedures (ECD L-6 months) • Command sequences • Constraints • Contingency actions LAT-PR-TBD Section 1.0 63 GLAST LAT Project Gamma-ray Large Area Space Telescope GSDR, 19 August 2004 GLAST Large Area Telescope: Ground Systems Design Review Software Architecture Steve Culp SLAC sculp@slac.stanford.edu 650-926-8987 LAT-PR-TBD Section 1.0 64 GLAST LAT Project GSDR, 19 August 2004 Topics Requirements Mapped to Software Software Architecture Network and Hardware Architecture Status LAT-PR-TBD Section 1.0 65 GLAST LAT Project GSDR, 19 August 2004 Requirements Mapped to Software Each requirement was identified as: 1. Procedural (non-software) 42 2. Existing software (commercial or other well-established software -- i.e. implementation Done!) 277 3. Under development by SAS (mostly done, refer to SAS for status) 39 4. To be developed by others (I&T, FSW, ITOS) 21 5. Remaining new development 88 total 467 Some requirements map to multiple categories or SW tools, so total exceeds total number of level IIIs (379) LAT-PR-TBD Section 1.0 66 GLAST LAT Project GSDR, 19 August 2004 1. Procedural (no SW development) Name # Level 3 Reqts % of Reqts N/A 19 4.1% Ops Procedures 12 2.6% SCS 11 2.4% 42 9.0% (e.g. facility reqts) (SLAC Computer Services) Total LAT-PR-TBD Section 1.0 67 GLAST LAT Project GSDR, 19 August 2004 2. Existing software Name Description # Level 3 Reqts % of Reqts Source ITOS satellite C&T package 215 46.0% GSFC ATNS Anomaly Tracking and Notification System 32 6.9% FASAT (commercial) or RXTE SOF Process Mgr (RXTE) FastCopy / DTS secure file transfer 15 3.2% FastCopy (commercial), DTS (HEASARC) CVS file config mgmt tool 5 1.1% Open Source STK Satellite Tool Kit 4 0.9% commercial LATDocs LAT documentation management tool 4 0.9% existing SLAC LAT tool NTP synchronize computers 1 0.2% Open Source email electronic mail 1 0.2% Open Source 277 59.3% Total LAT-PR-TBD Section 1.0 68 GLAST LAT Project GSDR, 19 August 2004 3. Under Development by SAS Name Description # Level 3 Reqts % of Reqts GINO pipeline manager 21 4.5% DATACHKTOOL check quality of science data telemetry 6 1.3% SASTOOL1 generate level 1 science data products 6 1.3% SASTOOL2 generate level 2 science data products 4 0.9% SASAUTO manage automatic generation of level 1 & 2 data products 2 0.4% 39 8.4% Total LAT-PR-TBD Section 1.0 69 GLAST LAT Project GSDR, 19 August 2004 4. To be developed by others Tool Name Description # Level 3 Reqts % of Reqts Source ELOG electronic log book 8 1.7% I&T Online (mostly done) DIAGTOOL diagnostic access 7 1.5% FSW ITOS Enhancements minor changes to ITOS to meet specific reqts 3 0.6% GSFC IRFTOOL generate instrument response function 2 0.4% SVAC DBCAL create database of LAT calibration data 1 0.2% SVAC 21 4.5% Total LAT-PR-TBD Section 1.0 70 GLAST LAT Project GSDR, 19 August 2004 5. Remaining to be developed Tool Name Description # Level 3 Reqts % of Reqts Comments PLOTTOOL plotting 28 6.0% combination of existing tools (e.g. ROOT, HippoDraw, JAS, IDL) & new dev TRENDTOOL trending 16 3.4% combination of IDL, DTAS (used by MOC), and/or TAPS (GSFC) PLANTOOL mission planning 6 1.3% generate timeline and commands for LAT operation CONSTRAINT TOOL check command sequences against constraints 12 2.6% DB database definition and implementation 10 2.1% DBIN ingest ISOC data 7 1.5% WEBTOOL provide web access to data products 9 1.9% 88 18.8% Total LAT-PR-TBD partially done by SAS, I&T, FSW coordinating dev with SAS, I&T, FSW, SCS Section 1.0 71 GLAST LAT Project GSDR, 19 August 2004 Software Architecture LAT-PR-TBD Section 1.0 72 GLAST LAT Project GSDR, 19 August 2004 Primary Data Flows LAT ISOC MOC Level 0 Data, LAT Alerts, Integrated Observ. Timeline, Spacecraft Timeline, As-Flown Timeline, Flight Dynamics Products, SC Incident Reports, PDB updates Level 0 Sci Data Receive & Archive Level 0 Hkpg Data, LAT Alerts Integrated Observ. Timeline, Spacecraft Timeline, As-Flown Timeline, Flight Dynamics Products GSSC Anomaly Tracking & Notification System LAT-PR-TBD Archive & Send GSSC Contingency: LAT Instrument Commands, LAT Instrument Loads, LAT Timeline Archive & Send Tested OK Commands to Verify Alerts, Logs, Process status, File status Level 1&2 Sci Data Products, LAT Timeline CHS monitoring & trending Mission Planning Receive & Archive Prelim. Science Timeline, Long Term Science Schedule, Observ. Timeline Package Science Data Analysis Pipeline Level 1&2 Sci Data Products LAT Test Bed MOC LAT Incident Reports, LAT T&C Database Updates All data into ISOC, and products out Database Section 1.0 Web Access 73 GLAST LAT Project GSDR, 19 August 2004 Automatic Telemetry Processing LAT ISOC Science Data Analysis Pipeline Calib. DB Calibration Trending Level 0 sci & hkpg telemetry, LAT Alerts MOC GINO & FastCopy/DTS GINO & FastCopy/DTS Level 1 processing Receive & Archive Level 0 science telemetry Apply Calibration Reconstruction Archive & Send System-level Monitor Level 2 processing Subsystemlevel monitor GCN Level 1&2 data products Transient Search/ GRB Refinement Level 0 hkpg telemetry, LAT Alerts Realtime Level 0 hkpg telemetry (during contacts), LAT Alerts ITOS Monitor hkpg parameters Text & Stripchart displays (optional) Monitor Trends ITOS Monitor Data Alerts, Logs, Process status, File status FASAT Reports, Log entries, Data Products received and sent Anomaly Tracking & Notification System Database STOL Config monitor scripts Beeper / email message Beeper / email message LAT-PR-TBD Section 1.0 74 GSSC GLAST LAT Project GSDR, 19 August 2004 Telemetry & Command Definition File Verification LAT ISOC Modify LAT T&C definition files LAT T&C def’n files MOC Updated LAT T&C definition files Project Database (master Tlm & Cmd definitions) OK (test passed with new file(s) ITOS Replay Commands & Receive Telemetry CCSDS Commands LAT Testbed telemetry Command log, Error log and optional Text/Stripchart displays Test Suite of STOL Commands LAT-PR-TBD Section 1.0 75 GLAST LAT Project GSDR, 19 August 2004 Command Load Generation & Verification LAT ISOC LAT Timeline (new) Generate Command Load & Perform Constraint Checking Verified OK GSSC STOL commands ITOS Issue Commands & Receive Telemetry CCSDS Commands LAT Test Bed telemetry Command log, Error log and optional Text/Stripchart displays LAT-PR-TBD Section 1.0 76 GLAST LAT Project GSDR, 19 August 2004 ISOC Network and Hardware Architecture SLAC Internet LAT ISOC Web Server … Firewall PVO Workstations FSW Workstations CHS Workstations Linux PC (Hkpg Replay ITOS) (Realtime connection ITOS) Firewall Gateway System (Oracle, GINO, FastCopy/DTS) SCS Storage Farm Abilene Network MOC GSSC Solaris Workstation (VxWorks tools) 1553 LAT Test Bed SIIS LVDS (S/C Sim) Linux PC (Test Bed ITOS) LAT Test Bed Lab LAT-PR-TBD Firewall SCS CPU Farm SAS/SP Workstations Linux PC Anomaly Tracking & Notification System Section 1.0 77 GLAST LAT Project GSDR, 19 August 2004 Status Architecture concept is solid Ready to proceed – Refine and document software design – Implement it – Lots of testing and demos LAT-PR-TBD Section 1.0 78 GLAST LAT Project Gamma-ray Large Area Space Telescope GSDR, 19 August 2004 GLAST Large Area Telescope: LAT Instrument Science Operations Center Management, Cost, Schedule Ground Systems Design Review Bill Craig Stanford Linear Accelerator Center bcraig@slac.stanford.edu 650-926-2983 LAT-PR-TBD Section 1.0 79 GLAST LAT Project GSDR, 19 August 2004 ISOC Management Process The ISOC has established the following meetings in support of ISOC development and coordination: – Weekly ISOC staff meeting to track schedule, RFAs and any coordination issues. – Weekly ISOC-FSW coordination meeting, attended by ISOC staff and the FSW manager, to work issues of joint importance and ensure that jointly held requirements are executed in the most efficient manner. – Bi-weekly meetings between ISOC manager and GLAST project scientist to work overall science planning and mission issues. – Database development meetings. – Regular interaction with I&T including shared personnel. Participate in: • Weekly GOWG meeting with GSFC – Address ICDs, timeline, and operation issues • Weekly FSW, I&T, SAS working group meetings – Coordinate script development and test activities LAT-PR-TBD Section 1.0 80 GLAST LAT Project GSDR, 19 August 2004 Test Philosophy Schedule constructed to test as early as possible – LAT testbed allows frequent and meaningful tests while instrument is being assembled – Data challenges already testing full science data pipeline – I&T optimization and calibration tools validated and tested Frequent demonstrations and simulations precede major software releases. Six end to end tests during Observatory I&T to provide confidence in final major release of software. LAT-PR-TBD Section 1.0 81 GLAST LAT Project GSDR, 19 August 2004 ISOC Software Release Schedule New software architecture in place, centralizing all software releases; eliminating the various separate software release schedule as in the PDR schedule The schedule has been coordinated with GSFC Ground System plans – ISOC Software Release 1 (April 1, 2005) • Support Ground Readiness Test (GRT) 2 and 3 – ISOC Software Release 2 (August 15, 2005) • Support GRT 4 and 5 – ISOC Software Release 3 (December 15, 2005) • Support End-to-End 1, ETE 2, ETE 3, GRT 6, GRT 7, and Mission Sim – ISOC Software Release 4 (July 25, 2006) • Support remaining ETE’s 4, 5, and 6 LAT-PR-TBD Section 1.0 82 GLAST LAT Project GSDR, 19 August 2004 Summary of ISOC Software Capabilities for GRTs & ETEs Date What Release 15-Apr-05 GRT 2 ISOC 1 (1 Apr 05) 15-Jun-05 GRT 3 “ 1-Sep-05 GRT 4 ISOC 2 (15 Aug 05) 15-Nov-05 GRT 5 “ 11-Feb-06 ETE 1 ISOC 3 (15 Dec 05) 15-Feb-06 GRT 6 “ contingencies added 15-Apr-06 ETE 2 “ provide instrument commands and file loads, support memory dumps of instrument 1-May-06 GRT 7 “ clean-up and regression tests 1-May-06 Mission Sims “ full ISOC capabilities to support Mission Sims (from May 2006 through launch) 15-Jun-06 ETE 3 “ support basic observatory operations 25-Aug-06 ETE 4 ISOC 4 (25 Jul 06) 14-Oct-06 ETE 5 “ regression test and contingencies 1-Feb-07 ETE 6 “ final ETE at launch facility LAT-PR-TBD Required ISOC capabilities receive real-time HK data from MOC, provide basic LAT P&S (Planning & Scheduling) level 0 data processing on science data, IOC processes science level 0 data into level 1 products receive level 0 data from MOC provide level 1 and 2 data products to GSSC, provide more complex LAT P&S, provide LAT file uploads to GSSC, provide simulated science data provide Level 1 data products to GSSC, receive Level 0 files from MOC (post-test) instrument turn on Section 1.0 83 GLAST LAT Project GSDR, 19 August 2004 ISOC Development Schedule Phase / Milestone A u g 0 4 S e p 0 4 O c t 0 4 N o v 0 4 D e c 0 4 J a n 0 5 F e b 0 5 M a r 0 5 A p r 0 5 M a y 0 5 J u n 0 5 J u l 0 5 A u g 0 5 S e p 0 5 O c t 0 5 N o v 0 5 D e c 0 5 J a n 0 6 F e b 0 6 M a r 0 6 A p r 0 6 M a y 0 6 J u n 0 6 J u l 0 6 A u g 0 6 S e p 0 6 O c t 0 6 N o v 0 6 D e c 0 6 J a n 0 7 F e b 0 7 ISOC CDR GSDR ITOS setup/configuration ISOC Verif. with Test Bed Diagnostic Tool Dev Mission Planning Dev DB/Web/E-logbook Dev Plotting/Trending Dev ISOC Demos ISOC Simulation Testing ISOC SW Releases 1 2 4 3 Ground Readiness Tests 2 3 4 5 7 6 End-to-End Tests 1 2 3 5 4 Environmental tests at NRL ISOC testing at NRL ISOC takes over LAT operation ISOC used to I/F with LAT (front-door only) Mission Simulations Launch SW Dev Slack Times LAT-PR-TBD Section 1.0 84 6 M a r 0 7 GLAST LAT Project GSDR, 19 August 2004 ISOC Staffing Plan ISOC Staffing by Job Function FTE 60 50 Support 40 Scientist Technician 30 Software 20 Engineering 10 Management 0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 Months since March 04 Plan matches actuals through July 04 LAT-PR-TBD Section 1.0 85 GLAST LAT Project GSDR, 19 August 2004 Procurement plans The large disk and CPU farms needed for pipeline storage and processing are supplied by SCS The handful of ISOC workstations will be procured in phases but will all be in place for the final software release Third party software is largely in place already – Final build/buy decisions on tools by November ’04 No issues foreseen with H/W or purchased S/W LAT-PR-TBD Section 1.0 86 GLAST LAT Project GSDR, 19 August 2004 ISOC Risk Status Number Date Rank Originator Description Mitigation ISOC-0001 5/15/04 1 B. Craig ISOC lacks accepted architecture and plan for software implementation. Trade study between possible front ends to be completed by 6/15/04. Hires into s/w architecture position. Successful CDR retires risk ISOC-0002 5/15/04 3 B. Craig Slow response to PDR RFAs Schedule and track RFA’s weekly. 3 remain as of 8/02 ISOC-0003 5/17/04 2 B. Craig Inadequate staffing plan for ISOC. Draft staffing plan complete. Culp, Lemon, Steele hired, S/W developers needed in Sep/Oct ISOC-0004 5/21/04 4 B. Craig No facility location identified for ISOC Long-term solution identified, short term space to be requested from SLAC management. LAT-PR-TBD Section 1.0 87 GLAST LAT Project GSDR, 19 August 2004 ISOC Risk Status (2) Number Date Rank Originator Description Mitigation ISOC-0005 5/21/04 2 B. Craig No requirements levied on I&T and Flt S/W subsystems Mechanism in place with I&T and FSW Final disposition after joint requirements review. ISOC-0006 5/21/04 1 B. Craig ISOC will be unable to hold schedule due to staffing delays and unscoped work Definition of work plan follows architecture development. Additional support supplied as requested but need to balance new hires vs transfers from other subsystems. LAT-PR-TBD Section 1.0 88 GLAST LAT Project GSDR, 19 August 2004 Issues and Concerns Frontloaded software support needed. – Need to limit hires to account for expected transfers from other subsystems. Database architecture not as well developed as we would like – Need to deliver I&T databases soon while retaining an overall structure that makes sense for the ISOC Requirement completeness, – risk of missing requirements, mitigation in work (DOORS and full requirement review) Many software elements need to interoperate smoothly – Early testing reduces problem, and certainly easier than writing the code anew LAT-PR-TBD Section 1.0 89 GLAST LAT Project GSDR, 19 August 2004 Summary Significant improvements since March peer review Architecture in place – Based largely on existing tools – Tools mapped directly to requirements Planning ISOC verification with Test Bed and 3 demos prior to first Ground Readiness Test Successful CDR on August 4; ready to build, first software release is in April 2005. LAT-PR-TBD Section 1.0 90