CRaTER SOC Operations
LRO MOR Section 10
NASA’s Goddard Space Flight Center
Mr. Michael Golightly
CRaTER Deputy Project Scientist
Boston University
LRO Mission Operations Review (MOR)
Slide - 1
CRaTER SOC Overview
Housekeeping
monitors
MOC
R/T
TCP
Realtime
archive
MOC
DMP
SCP
Raw
archive
PDIST
SCP
Uplink
archive
Pipeline
Processing
MOC
CMD
PDS
PPI
node
Secure
web
server
Archive
mirror
SCP
NASA’s Goddard Space Flight Center
Archive
for PDS
Pipeline
archive
LRO Mission Operations Review (MOR)
Slide - 2
CRaTER Software Component Status
Component
Function
Status
C++ Object library
Object-oriented interface to Level 0, 1, and 2 data
products
100% written
Released
CRaTER Pipeline
Time ordering and gap detection in raw (EDR) files;
Generation of Level 0, 1, and 2 products
100% written
Released
Higher level products
Model-dependent generation of LET spectra
0% written
Data distribution
TCP/IP packet distribution server and clients
100% written
Under review
Engineering display
TCL/Tk graphical interface to display and analyze
secondary instrument science and housekeeping
100% written
Under review
Science display
IDL graphical interface to display and analyze primary
and secondary science data products
90% written
Private web interface
HTTPS/SSL server to provide secure access to
CRaTER data products and processing information
10% written
Public web interface
HTTP server to provide access to CRaTER status
50% written
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 3
CRaTER SOC Functions
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Uplink command generation (SCP to MOC, as needed)
Continuous real-time data collection (TCP from MOC) and display
Daily receipt of recorded data (SCP from MOC)
Daily pipeline processing
Daily quick-look analysis and trending; examine data gaps for existence of
anomalous data loss
Daily report to SOC lead and MOC
Daily distribution to non-secure hosts (CRaTER internal server)
Daily review of system logs for evidence of attempted intrusions
Daily/weekly data back-ups to offline storage
Weekly SOC team meetings
Weekly processing & trending report and security inspection
Monthly archive collection and rearrangement
3-monthly archive generation and transfer to PDS
Yearly security analysis and report
Anomaly response, e.g., replication on engineering unit or flight unit #2
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 4
CRaTER SOC Organization
Principal Investigator
(Spence)
Project
Scientist
(Kasper)
•
•
•
•
Calibration algorithms
Conversion algorithms
Science display
On-orbit commanding
Deputy
Project
Scientist
(Golightly)
•
•
•
•
•
Project Manager1
(Foster)
Project
Engineer3
(Goeke)
SOC
Lead2
(Ford)
Pipeline software
Document preparation
LDWG/PDS Interfaces
Archive generation
IT security and
hardware maintenance
•
•
•
•
1 MIT
GSE control S/W
Engineering display
Document reviews
Code reviews
project management ends Dec 2007
2Golightly
3Support
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
takes over as SOC Lead Jan 2008
reduced to 25% beginning Jan 2008
Slide - 5
CRaTER SOC Staffing
FY08
FY09
O N D J F M A M J J A S O N D J F M A M J J A
Principal
Investigator
Proj. Scientist,
Science Lead
Deputy PS,
SOC Lead
Temporary
Soc Lead
Instrument
Scientist
Spence
Operations
Lead
Science
Software
Bradford
Kasper
Golightly
Ford
Case
Wilson
Student
Student
SOC Operations
Integration/Test
S/W Development/Maintenance
Science Algorithms
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Science Algorithm
Improvements/Evolution
Slide - 6
CRaTER Key Documentation Status
Document
Description/Purpose
Need Date
Current Status
32-01209
SOC Requirements Document — Plans for SOC
component verification
12/01/2006
Rev B: 11/02/2006
Delivered / signed off
32-02080
CRaTER SOC to PDS PPI Node ICD — Defines the
process of data transfer from SOC to PDS
12/01/2006
Rev B 11/21/2006
Delivered / signed off
32-01210
Data Management and Archive Plan — Describes
the generation, format, and archival of CRaTER
pipeline products
12/01/2006
Rev A: 10/25/2006
Delivered / signed off
32-01213
SOC Risk Assessment — Identifies security risks
and their proposed mitigations
04/30/2007
Rev B: 06/01/2007
Delivered / signed off
32-01208
IT Security Plan and Contingency Plan — Describes
the controls to apply to minimize IT risks
06/30/2007
Rev A: 07/01/2007
Delivered / signed off
32-01212
SOC Test Plan — Describes IT testing methodology
and test suites traceable from IT requirements
10/30/2007
Rev 01: 08/31/2007
Draft released
32-01211
Standard Software Interface Specification —
Describes the raw (EDR) and pipeline (CDR)
products and PDS archive organization
05/31/2007
Rev A: 05/31/2007
Rev B: under review
32-01207
Calibration Plan — Describes how detector counts
are converted to energy and lineal energy transfer
10/01/2007
Rev 01: 08/12/2007
Draft released
32-01214
Special Software Interface Specification —
Describes the Level 3 and 4 spectral data products
02/29/2008
Not started
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 7
CRaTER SOC Hardware Facilities
To backup server
BU NETWORK
Logger
console
switch
Crater-Devel
CRaTER LAN
mux
SOC-A
Crater-A
Recorder
(BU)
Printer
SOC-B
Crater-B
RAID
15x500
GB
A CRaTER Team Member’s LAN
2x500 GB
2x500 GB
Printer
UPS
Locked cabinet in key-coded room
“Secure” LAN
External networks
Disk access
“Semi-open” LANs
Power supply
Multiplexed consoles
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 8
CRaTER Engineering Unit
• The SOC will maintain a
CRaTER engineering unit†
and associated 1553 interface
(S/C simulator) to assist in the
diagnosis of post-delivery
problems with the flight unit.
• SOC staff will interact with the
simulator via a UDP interface
to the following set of software
programs
–
–
–
–
–
–
†
Graphical command interface
Output capture to disk
Analog housekeeping display
User status display
Primary science stats display
Output conversion to ASCII
28VDC
CRaTER
Engineering
Unit
1553B-UDP
Emulator
Internet
(UDP)
2x
1553,
1 Hz tick
Example: send commands to the engineering
unit and monitor the resulting housekeeping
channels
[1] Start the graphical command interface:
host# CCmd &
[2] Start the graphical housekeeping interface
host# CHouse &
the CRaTER flight spare can also be used in place of the
engineering unit, e.g., to assist in flight-unit recalibration
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 9
CRaTER Software Maintenance
•
Operating system maintenance
–
–
–
•
Pipeline code development/modifications
–
–
•
Through start of instrument I&T: MIT/Ford, BU/Bradford
After start of instrument I&T: BU/Bradford, BU/Wilson
Configuration Control
–
–
•
SOC-A, SOC-B, Crater-A, Crater-B, and Crater-Devel
Redhat Linux—updates provided under 3-year maintenance contract with Redhat
SOC Operations Lead responsible for applying updates/patches
Code version control maintained under Subversion version control system
(http://subversion.tigris.org/)
Copies of all implemented version will be maintained under version control in case it is
necessary to roll-back to an earlier version
System stability/integrity carefully controlled during all upgrades
–
–
–
Code development/testing done outside SOC on Crater-Devel
User manuals, installation guides, code reference manuals, etc must be updated and released
with new versions/changes to existing software
Changes to operating system, pipeline, or science software implemented in sequence of
distinct steps
•
•
•
•
•
Install changes/updates on Crater-Devel--test for compatibility, stability, & functionality;
Install on the appropriate backup machine (e.g., machine-B if machine-A prime, etc)--test for
compatibility, stability, & functionality;
Switch backup machine to prime, and prime to backup (e.g., machine-B → prime, machine-A → backup)
Install on remaining machine--test for compatibility, stability, & functionality.
NO FLIGHT INSTRUMENT SOFTWARE
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 10
CRaTER Software Testing
•
The Test Plan is directly derived from the following documents:
– 32–01209 CRaTER SOC Requirements Document
– 32–01208 CRaTER IT Security and Contingency Plan
subject to the constraints identified in the following:
–
–
–
–
•
32–01210 CRaTER Data Management and Archive Plan
32–01211 CRaTER Standard Product Software Interface Specification
32–02080 CRaTER Science Team and the PDS PPI Node ICD
431–ICD–000049 External Systems ICD for the LRO Ground System
The Traceability Matrix identifies 40 separate requirements:
– 13 functional — the basic operations expected from the SOC
– 10 design — requirements imposed by external constraints
– 10 performance — ability of the s/w and human interfaces to do the job
– 7 security — factors that protect the system from unintended behavior
which are verified by a set of 10 tests, using various methodologies:
–
–
–
–
5 by explicit test — did the system pass the prescribed procedural test?
3 by demonstration — did the system produce the expected results?
1 by analysis — are the algorithms adequate to the task?
1 by inspection — are the planned resources available?
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 11
CRaTER SOC Development Status
Current Status
• Hardware (5 processors, RAID storage, etc.) is being assembled
• Documentation deliveries are up-to-date
• Pipeline Software (Levels 0, 1, 2) is written and is under review
• Standard product SIS is under review by the PDS PPI node
• Engineering Unit interface is built and tested
• Draft SOC test plan is out; individual test scripts are being written
• Draft calibration plan has been released
• SOC is fully staffed (except for students)
Future Tasks
• Place SOC software under configuration control
• Erect network firewall around secure SOC hardware
• Complete SOC testing, including SOC-MOC and SOC-PDS tests
• Conduct SIS peer review by PDS PPI node
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 12
Backup Charts
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 13
CRaTER SOC Hardware Facilities
• SOC computers SOC-A
–
–
–
–
–
Host data pipeline process
Receive data pushed from MOC
Process data into specified L0, L1, L2 (, L3, L4) products
Push processed data to RAID storage, PDS, and Crater computers
CONNECTIVITY IS ONE WAY— will not be able to access SOC
machines except via computer’s console in controlled access facility
– A & B machines mirror each other for redundancy
• Crater computers Crater-B
– Receive CRaTER data products
– Serves data for public and internal web sites, nominal displays, team
scientists’ applications, etc (link between CRaTER data stores and
anything outside of the SOC firewall)
– No connectivity from Crater computers back to the SOCs
– A & B machines mirror each other for redundancy
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 14
CRaTER SOC Hardware Facilities
To backup server
BU NETWORK
Crater-Devel
switch
CRaTER LAN
Crater-A
SOC-A
Recorder
(BU)
Printer
SOC-B
Crater-B
RAID
15x500
GB
A CRaTER Team Member’s LAN
2x500 GB
2x500 GB
Printer
Secure MOC→SOC Ingest Connection
External networks
SOC→Crater, PDS Push Connection
CRaTER Data Via Restricted Access Network Connections
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Disk Access
Slide - 15
CRaTER SOC Hardware Facilities
• SOC RAID Storage
– RAID storage selected for short-term reliability
• derived both from the SOC requirements and risk assessment plans
• prevent a disk I/O error from crashing SOC-A/B while receiving real-time
data from MOC
– Sized for 4 years of data storage
• Uncompressed Level 0, 1, and 2 products → 789 GB/y
– Serves as backup to CRaTER PDS data submission
NASA’s Goddard Space Flight Center
LRO Mission Operations Review (MOR)
Slide - 16