SM4 Readiness Review
Tuesday, 9 September 2008, 9:00am -12:40pm
Welcome
SMOV Program
Proposal Processing Status
Long Range Planning
Contingency Plans for STIS/ACS
Planning & Scheduling
Hubble Mission Office
Carl Biagetti, 9-9:30am
Denise Taylor, 9:30-9:45am
David Adler, 9:45-10am
Neill Reid, 10 - 10:15am
Merle Reinhart & George
Chapman, 10:15-10:45am
…Break for 15 minutes…
Commanding
Alan Welty, 11-11:15am
Data Processing and Archiving
Faith Abney, 11:15-11:45am
Calibration Pipeline Processing
Warren Hack, 11:45-12:05pm
Operational Readiness
Al Holm, 12:05-12:25pm
Infrastructure Contingency Plans
Doris McClure, 12:25-12:40pm
1
SM4 Readiness Review
Wednesday, 10 September 2008, 9:00-12:30:
SM4 Observatory Support
WFC3 Readiness
COS and STIS Readiness
ACS Readiness
NICMOS Readiness
FGS and OTA Readiness
OPO and ERO Readiness
Chris Long
John MacKenty
Alessandra Aloisi
Linda Smith
Tommy Wiklind
Ed Nelan and Matt Lallo
Mario Livio and Keith Noll
2
SMOV4 Planning Readiness
Carl Biagetti
System Engineering Branch OED
3
SMOV4 Plan Overview …1
SMOV4 Plan
– Provides for the timely commissioning of the
Observatory for science following SM4
• Commissions the newly SIs on channel-by-channel basis
– Introduces GO science on a channel-by-channel basis as
SMOV progresses
• Recommissions the restored SIs on channel-by-channel
basis
• Recommissions serviced Observatory Systems
• Performs Early Release Observations (EROs)
– Satisfies the SMOV4 requirements of 21 March
2007
• Plan presented, reviewed, and approved 12 Oct. 2007
(SMR-4029, CCR 5248)
4
Generic SMOV Process
following each HST Servicing
Spacecraft
Subsystems
Commissioning
Early Release
Observations
New/Revived Instruments
PCS/EPS/TCS…
HST Release
from Shuttle
Focus/Alignment
of SI Optics
SMOV Start
Instrument Preps
Engineering Check-Outs
Outgassing
Cooldown
Instruments
Science Calibrations
GO Science
Ramp-up
As Instruments/Channels
Are commissioned
5
SMOVs 2 – 4
EXTERNAL ORBITS
HISTORICAL COMPARISON
SI/SS
WFPC2
FOC
Spacecraft/FGS
NICMOS
STIS
ACS
WFC3
COS
ERO
TOTALS
SMOV2
SMOV2
SMOV3B SMOV4
PLANNED ACTUAL ACTUAL PLANNED
46
77
31
15
15
69
61
41
81
194
285
77
34
205
267
14
21
186
38
163
164
72
75
70
90
601
780
419
591
Notes:
1. STIS and NICMOS required more orbits than planned because of the NICMOS
thermal short and the opto-coupler resets in both SIs.
2. SMOV4 seems to be the most complex in terms of realtime interactions and
in-line analyses.
6
SMOV4 Plan Overview …2
SMOV4 Plan satisfies the following
drivers
• Perform EROs in time for Jan. AAS release
– i.e., schedule observations in early
December 2008
• Minimize excess FGS3 usage by timely
commissioning of FGS2R2
7
SMOV4 Plan Overview …3
SMOV4 Plan satisfies the following constraints
Long intervals for outgassing and contaminants dissipation
– Bright Earth Avoidance (for 21 days from Release)
• to avoid UV-induced polymerization of exposed optics (WFC3 POM)
– COS NUV = 10 days from Release (per model) to allow internal pressure
below 20 uTORR, before detector activation
– COS FUV = 15 days from Release (per model) to allow internal pressure
below 10 uTORR, before detector activation
– WFC3 = 21 days from Release before cal lamp usage and TEC cooldown
Careful, incremental initial high-voltage ramp-ups
– COS NUV/FUV
– STIS NUV/FUV
– ACS SBC
8
SMOV4 Plan Overview …4
SMOV4 Plan prioritizes SMOV activities to
allow schedule flexibility
Prioritization (High, Medium, Low)
– Based on realistic assumptions about scheduling
rates (external orbits)
– Accommodates ERO schedule and other key
dates
– Accommodates non-SMOV (eg., GO, ERS)
programs before SMOV completion
9
Prioritization of SMOV activities
• High Priority = activities needed to
enable EROs in early Dec.
• Medium Priority = activities needed for
enabling GO science
• Low Priority = all other activities
– Needed to satisfy all other SMOV4
requirements and reference file generation
– Needed for handbook inputs in spring
10
SMOV Prioritization: COS
High Priority Activities (EROs)
– Through NUV and FUV Alignment and Focus
Medium Priority Activities (Science enable)
– All FUV cals beyond alignment/focus and target acq test
– The rest of NUV wavelength scale calibration
Low Priority Activities
– Low 1 – needed for reference files and handbook inputs
(data needed by Feb. 1)
– Low 2 – desirable but not required for handbook inputs
– Low 3 – All others
11
SMOV Prioritization: WFC3
High Priority Activities (EROs)
– Through UVIS & IR Fine Alignment
– In parallel with EROs
• UVIS/IR/FGS alignment, UVIS/IR plate scales
Medium Priority Activities (for “Easy” GO
science)
– Internal flats, flat-field uniformity, photometric zeropoints
Low Priority Activities (for “Hard” GO science)
– UVIS/IR image quality, PSF wings, UVIS PSF
Core Modulation (shutter test), pointing stability, IR
12
grisms
SMOV Prioritization: ACS
High Priority Activities (EROs)
– WFC Optimization Campaign (OC), image quality,
sensitivity
– SBC turn-on, UV monitor
Medium Priority Activities (GO science)
– SBC PSF
– HRC image quality, sensitivity, geometric
distortion, coronagraph commissioning
Low Priority Activities
– None remaining, but assume the option to delay
HRC commissioning in favor of WFC/SBC in early
SMOV
13
SMOV Prioritization: STIS &
NICMOS
NICMOS
– High Priority Activities (EROs)
• Through Aperture Location activity
– Medium Priority Activities (GO science)
• Through Aperture Location activity and SIAF update
• Aperture knowledge without SIAF may be sufficient
STIS
– High = STIS External Focus Check
– Medium = NUV Optical Format Verification, FUV Image
Quality
– Low = CCD Spectroscopic Throughput, Image/Pointing
Stability, NUV Image Quality
14
SMOV4 EXTERNAL ORBITS & ENABLE DATES
- Including EROs Oct. 10, 2008 Launch -- Assumes 55 - 60 prime observing orbits per week
70
ERO
BEA PERIOD (1st 3 weeks)
AVAILABLE SCIENCE ORBITS
FGS
PCS
NICMOS
STIS
ACS
COS
WFC3
60
ORBITS
50
40
30
20
10
0
1
2
3
4
5
6
7
8
9
10
11
12
10/20 10/27 11/3 11/10 11/17 11/24 12/1 12/8 12/15 12/22 12/29 1/5
13
14
1/12 1/19
SMOV WEEK
COS NUV SCI ENABLE
WFC3 UVIS/IR “HARD “SCI ENABLE
ACS SBC SCI
ENABLE
ACS HRC SCI
ENABLE
UV
IR
COS FUV SCI ENABLE, WFC3 IR “EASY” SCI ENABLE
WFC3 UVIS/IR ERO ENABLE, UVIS “EASY” SCIENCE ENABLE,
NIC SCIENCE ENABLE, COS FUV ERO ENABLE
STIS SCI & ERO ENABLE
ACS WFC SCI & ERO ENABLE, NIC ERO ENABLE, FGS2R2 GUIDING
ENABLE
Critical Operations & Operational
Challenges in SMOV
HV Ramp-ups for STIS and COS
Focus & Alignments for COS & WFC3
– Iterative processes for each SI in the same weeks
Realtime Commanding
– GENSLEWs (Use Offsets)
• Used in several proposals for pointing updates before
aperture data (SIAF) is in place
• Needed to meet ERO schedule
– Table uploads, Event flag mgmt, etc.
Fast Data Turnaround (FASTRACK)
– Needed in several places for in-line analyses
16
Critical Operations and Operational
Challenges in SMOV Weeks 1-7
Yellow indicate times of high activity
FTn = Fastrack data delivery requested by n proposals
WEEK
1
2
3
4
S/C &
FGS
BEA, VDT,
AMA ADJ,
FT1
BEA
AMA ADJ,
FT2
BEA
WFC3
OUTGAS
OUTGAS
OUTGAS,
SDB CHK
INIT UVIS
ALIGN, INIT
IR ALIGN,
FT2
COS
OUTGAS,
SDB CHK
OUTGAS
FUV HV,
COS/FGS
ALIGN, NUV
ALIGN, FT2
FUV HV, NUV
ALIGN/FOCU
S, FUV
SENS, FT2
ACS
SDB CHK,
TEMP SET
PNT
WFC OC,
FT1
WFC OC,
FT4
WFC OC,
CCD X-TALK,
FT4
STIS
SDB CHK
FT1
EROs
NICMOS
COOLDOWN
5
6
7
INIT UVIS
ALIGN, INIT
IR ALIGN,
UVIS FINE
ALIGN, FT3
UVIS/IR
FINE ALIGN,
FT2
IR FINE
ALIGN,
EROs, FT1
NUV
ALIGN/FOCU
S, EXT.
SPECTRO.
PERF., FT1
FUV ALIGN
/FOCUS,,
FT2
FUV
STRUCT/TH
ERM STAB,
EROs, FT2
SYSTEM
COOLDOWN
FW TEST,
FOCUS/PAM
ADJ, FT2
VDT
EROs
17
Organization, Management,
Reporting during SMOV
SMOV Team
– Consists of STScI & Project science & engineering
staff
– Lead personnel identified for each critical activity
– Team to meet regularly (“morning meeting”)
• Daily, for 1st couple of weeks, then at a slowly decreasing
frequency
• To assess daily progress, review daily plans
• Determine replans
• Contributions from each SI and subsystem teams
• Daily minutes/reports to be published Project-wide
SMOV TTRB
– To assess replan requests
– Make recommendations to HST MO and Project
18
SMOV4 Contingency Planning
High-level contingency plans are being
worked for each SI/subsystem
– Including SMOV OTA focus contingency
19
SMOV PROPOSAL IMPLEMENTATION TEAM (PIT)
EARNED VALUE (EV) as of 4Sep08
10 EV points/proposal
– 3 pts for prop submit
– 2 pts for 1st PIT mtg
– 2 pts for 2nd PIT mtg
– 3 pts for prop complete
165 SMOV4 PROPOSALS = 1650 total EV points
- 163 PROPOSALS PIT-APPROVED
PIT PROGRESS
- 100 % PLANNED EARNED VALUE for Aug. 31
- 99.0 % ACTUAL EARNED VALUE as of Sep 4
SMOV PIT EARNED VALUE -- NOV2007 - SEP2008
(as of 04SEP08)
1800
POINTS (10 pts/proposal)
1600
1400
1200
PLANNED EV
(FEB 2008)
1000
ACTUAL EV
800
600
400
200
0
Nov-07
Dec-07
Jan-08
Feb-08
Mar-08
Apr-08
May-08
Jun-08
Jul-08
Aug-08
Sep-08
20
Credit goes to dozens and dozens of people
- STScI, GSFC, COS & WFC3 Teams
For a very good SMOV4 Plan
21
Proposal Processing Status
Denise Taylor, Observation
Planning Branch - OED
22
Proposal Processing Status
SMOV Proposals
153 proposals (not including ERO)







35 COS props (164 ext. orb., 111 int. orb.) - 7 props still working
31 ACS props (38 ext. orb., 493 int. orb.) - 1 props still working
42 WFC3 props (163 ext. orb., 370 int. orb.) - All ready
25 STIS props (21 ext. orb., 174 int. orb.) - All ready
12 NIC props (34 ext. orb., 136 int. orb.) - All ready
2 PCS props (24 ext. orb., 26 int. orb.) - All ready
6 FGS props (57 ext. orb.) - All ready
23
Proposal Processing Status
SMOV Proposals (continued)


145 are ready for flight.
8 are being worked:


1 ACS + 1 COS props need to be resubmitted,
reprocessed and re-reviewed
6 COS props need CS reviews
24
Proposal Processing Status
ERO Proposals
9 proposals for ERO
4 are ready for flight (16 orbits):
One WFPC2 ERO
One STIS ERO
One ACS ERO
One COS ERO
4 have been submitted and are being worked (need CS reviews) (20 orbits):
One COS ERO
One WFC3 ERO
One COS ERO
One WFC3 ERO
1 has not been submitted (? orbits)
25
Proposal Processing Status
Early Science Proposals (132 orbits possible for execution before 31 January 2009)
COS/GTO

11520 - QSO Absorbers, Galaxies and Large-scale Structures in the Local Universe
(26/39 orbits)

11534 - Atmosphere of a Transiting Planet (20/20 orbits)
GO





11566 - Imaging Saturn’s Equinoctal Auroras (6/12 orbits)
11612 - Eta Carinae’s Continuing Instability and Recovery - the 2009 Event (9/16 orbits)
11706 - The Parallax of the Planet Host Star XO-3 (1/6 orbits)
11788 - The Architecture of Exoplanetary Systems (11/63 orbits)
11789 - An Astrometric Calibration of Population II Distance Indicators (11/33 orbits)
ERS

11359 - Panchromatic WFC3 Survey of Galaxies at Intermediate z (28/104 orbits)

11360 - Star Formation in Nearby Galaxies (20/110 orbits)
26
Proposal Processing Status
Cycle 17 General Observer Pool
197 GO proposals including those from HST Cycle 17 TAC,
Chandra TAC, and previous HST TACs, for 3627 orbits.
16 SNAP proposals, for 1417 orbits.
24 GTO/COS proposals for 291 orbits.
3 GTO/ACS proposals for 24 orbits.
2 ERS proposals for 214 orbits.
3 GO/PAR proposals for 410 orbits.
27
Proposal Processing Status
Cycle 17 General Observer Pool (continued)




Program Coordinators (PCs) are processing observations in the general pool: ~
6000 orbits.
Instrument teams have begun reviewing these observations and will have most
done before SM4. Remaining will be done according to Long Range Plan
schedule.
Currently 9% unschedulable, mostly due to incompatibility of requested
scheduling constraints (orients, guide stars, timing links, etc.). This is typical for
early ingest; PCs will resolve these issues throughout the coming months.
Final processing steps will wait until after SM4, when the ground system is
reconfigured for 3 gyro mode.
28
Proposal Processing Status
After SM4




SMOV proposals and science proposals will be reworked as needed
during SMOV.
Remaining Cycle 15 and Cycle 16 proposals will be reworked for 3-gyro
scheduling.
Cycle 17 pool will continue processing as usual, to execute
observations according to the Long Range Plan.
Begin processing Cycle 17 calibration proposals (Calibration review on
9/25/08; Phase II deadline TBD).
29
The Long Range Plan
David Adler
Science & Mission Scheduling
Branch - OED
30
Long Range Planning Group
(LRPG) Activity Summary
•
Remainder of Cycle 16:
– 173 orbits before SM4, 441 orbits after SM4 in Cycle 16 “tail.”
•
The LRPG is monitoring remaining Cycle 16 WFPC2 observations.
– all but one 3-orbit ToO follow-up should be completed by SM4.
–
PI of that program will put in a request to change SI if needed.
•
The LRPG has incorporated SMOV, SI commissioning, ERO, ERS, and GTO
constraints into the Cycle 17 planning process.
•
A preliminary Cycle 17 LRP has been built and verified.
•
The LRPG has identified/developed plans for managing SM4 contingencies in
Cycle 17 planning.
31
Preliminary Cycle 17 Plan
Highlights
•
December/January priorities:
–
–
–
–
–
–
High priority SMOV
EROs
ERS/GTOs
Time-critical Cycle 17 science
Medium/low priority SMOV
Cycle 16 GO science
•
Cycle 16 planning information for Oct-Dec programs is being
maintained in case of launch slip.
•
All cycle 16 science planned for Oct-Dec can be replanned for 2009.
•
Under-subscription in early 2009 – LRPG is leaving space for Cycle 17
calibration programs, Target of Opportunity programs, DD, HOPR
repeats, etc.
32
Preliminary Cycle 17 LRP – Oct-Feb
launch
BEA
SM4
SI
commissioning
milestones
ACS
SBC
STIS
COS NUV
ACS WFC/HRC
NIC
WFC3 UVIS “easy”
WFC3 UVIS/IR “hard”
COS FUV
WFC3 IR “easy”
LRPG Transition from SMOV to
Cycle 17 Science
• Preliminary Cycle 17 plan is built
• Statistics:
4200 orbits – Cycle 16 and 17 science in working LRP
+600 orbits - SMOV/ERO (not in plan)
+850 orbits - “unplanned” – 2 orbits a day of calibration, HOPR, ToO, etc
--------------5650 orbits in Cycle 17
-650 orbits - cycle 17 tail – beyond Dec 31, 2009
=5000 orbits from 10/20/08 – 12/31/09 = 11.42 orbits/day = 80 orb/week
• Plan will not be released to the public until a couple of weeks after
deployment when the state of the observatory is known.
34
LRPG Contingency Plans
If there’s a launch slip:
•
Long Range Plan will be rebuilt to reflect the change.
•
For a 1 - 4 week slip, the following can fill the gap:
–
–
–
–
•
Existing Cycle 16 science programs
Cycle 17 NIC science that can schedule in 2-Gyro
Cycle 15/16 Snaps
Cycle 17 NIC Snaps that can schedule in 2-Gyro
5 + week slip:
– Existing science and Snaps, as described above; studies are ongoing to see how
far out this can be extended.
– If more science is needed, LRPG will consult with the Science Policies Group
(SPG) for a course of action.
35
Summary - LRPG Readiness
for SM4
• Transition plan between Cycle 16 – SM4 – SMOV – Cycle 17 is in place.
• Preliminary Cycle 17 LRP has been built and is ready for execution. It is
currently on-hold pending the outcome of SM4.
• The LRP incorporates all constraints/HST orbit resource requirements of
all SMOV (including SI commissioning) and ERO/ERS activities.
• Process/policies are in place for updating the Cycle 17 plan in response
to SM4 contingencies.
• The LRPG is ready for SM4!
36
Contingency plan if STIS
and/or ACS are not available
• The C17 TAC recommended programs based on the
assumption that SM4 would be fully successful, i.e.,
both STIS and ACS would be repaired
• STIS and ACS are scheduled for repair after COS
and WFC3 and their repair is more challenging
• Therefore STIS and ACS carry a higher risk of not
being available in C17 than COS and WFC3
37
Instrument Contingencies
Neill Reid
Science Mission Office
38
C17 allocation by orbit:
• STIS: 8% of total; includes imaging and
spectroscopy
• ACS: 25% of total; includes 1% of SBC
use (currently operational)
• Numbers include parallels but no
snapshots (1357 snapshot targets
awarded in C17; STIS: 140; ACS: 518)
39
C17 proposers were required to address
whether their programs could be switched
to a different instrument if STIS/ACS were
not available
Using this information, the TAC endorsed or
rejected an instrument switch and made a
recommendation to the Director
40
STIS
• Moderate-resolution UV spectroscopy can often be
switched to COS
• Imaging can almost always be done with WFC3 or ACS
• Optical spectroscopy, spatially resolved spectroscopy,
high-resolution spectroscopy, etc., need STIS
• ~40% of all orbits using STIS can be switched
• The Long Range Planning Group has run simulations
showing that when STIS observations are removed,
science with other SIs can be front-loaded in the Cycle
17 long range plan
41
ACS
• Standard broad- and narrow-band imaging can be
switched to WFC3 in most cases
• SBC observations continue to be available
• Optical coronagraphy, polarimetry, high-resolution
imaging, etc., need ACS
• ~85% of all orbits using ACS can be switched
• The Long Range Planning Group has run simulations
showing that when ACS observations are removed,
science with other SIs can be front-loaded in the
Cycle 17 long range plan
42
Impact if STIS/ACS are not
available
• Using the TAC recommendation as a guide-line and after
additional STScI review, PI’s will be notified whether
instruments can be switched
• Process similar to that used when ACS failed and
observations were switched to WFPC2/NICMOS
• PIs may appeal to the TTRB if switch was rejected
• Switch can be done quickly (1 – 2 months)
• Scheduling impact: ≤10% of the total orbits (sum of all
C17 orbits) will be lost because the science
requirements preclude an instrument switch
• Orbits can be replaced, e.g., by snapshots
43
Planning & Scheduling
Merle Reinhart and George
Chapman
System Engineering Branch OED
44
44
Overview
Merle Reinhart
– What is Planning and Scheduling?
– SMOV Database Management
– P&S Procedures and Tools
George Chapman
–
–
–
–
–
–
–
Rendezvous Intercept SMS Process
Ephemeris Management
Deploy SMS Process
SMOV Calendar/SMS Build Process
P&S Team Readiness
Pending Changes
Items Still in Work
45
What is Planning and Scheduling?
Long Range Planning, Short Term Scheduling & PASS
TRANS
Proposals
PRD
SCIOPSDB
Instructions
SPSS
HST/TDRS
Ephemeris
SPIKE
Orbit File
C&C List
MOSS
PASS
CCS
NGSS
FDF
SCS
SMS
GSC
PRD
46
Database
NSSC-1 Flight Software
NSSC-1 FSW updates are required to support SM4
NSSC-1 8.6 supports full complement of SIs during
SM4
– This will be installed Sep. 10 during SMS 252 (SMS 254
is first use)
– Avoids software changes during SM4
NSSC-1 9.0 removes support for COSTAR and
WFPCII
– This will be installed immediately after SM4 during the
H&S SMS
47
Database
PRD/SCIOPSDB
OPSPRD 7.0 has been deployed to SMS 254
– First used after NSSC-1 8.6 FSW is loaded
– PR 60745: TFPF JWOSCPRM for oscilloscope mode parameters
Planned Updates
– PR 60655, 60781: PLCP PIPHTHET, PIHETPHT, PISAFFLT
needed for WFC3
– After SM4 rendezvous, but prior to deployment
– PRD & SCIOPSDB updates for SMOV4 are included in the
OPS/SM4 database merge (described later)
– Other planned updates for use on initial H&S SMS
• PR 60663: PLCP PJSAFHLD for ACS safing recovery including TEC
commanding
• PR 60739: AGCF and SCHF parameters for switch to 3-gyro mode
• PR 60742: Table ODB for NSSC-1 9.0 FSW
– Freeze Waivers will be requested for each update
48
Database
SMOV Preparation Environment
SMOV4 development utilizes the same methodology that was successfully
used for all previous Servicing Missions.
Use a separate database and disk space from Operations, but the same
machines/servers.
– spss_sm4 database for SMOV4 development
– spss database for current SPSS Operations
Uses the operational software and tools, but utilizes different
PRD/SCIOPSDB and SMS Instructions
– Allows simultaneous Flight Operations and SMOV development
– Two environments are isolated
The spss_sm4 database is managed in the same way as the SPSS
Operations database.
This SMOV4 environment is used to:
–
–
–
–
Prepare the SMOV proposals for execution
Create the SMGT products
Create the SMOV4 representative products
Post-SM4 DRM Planning and Scheduling exercise
49
Database
The Merge Process
SMOV environment gets merged into the Operations environment after
rendezvous and berthing occurs.
Unload all the SMOV4 and housekeeping proposals from the spss_sm4
database
– This stores the proposals on disk as an sql load file
– This part of the process nominally occurs right around launch time and takes
about a day
Get a backup of the spss database
– This allows us to easily restore to pre-launch if necessary
Database Administrator moves the contents of specified tables from the
spss_sm4 database into the spss database
– This transfers the PRD/SCIOPSDB information and the Instructions needed
post-SM4
Load the SMOV4 and housekeeping proposals into the spss database
Database Administrator will rebuild the table indices to ensure good
database performance
50
Database
The Merge Process (cont)
Modify any spss database resident
housekeeping visits to 3-Gyro/OBAD
mode
PRD/SCIOPSDB group will audit the
merged spss database for correctness
OPS will audit the proposals for
correctness
End-to-end process generally takes about
2 days.
This Merge process will be tested on
September 15th.
51
Database
PASS Configuration Files
These are files used by the PASS software and configuration controlled by
the P&S Operations group.
Command Loader Basefile changes to disable WFPC2 and COSTAR and
enable WFC3 and COS (or the appropriate combination).
– In the CLBASE Namelist
• Old: NS_SINAME = 'ACS ', 'NIC ', 'STIS', 'CSTR', 'WFII’
• New: NS_SINAME = 'ACS ', 'NIC ', 'STIS', 'COS ', 'WFC3’
• Old: NSREDUNDV = 'A', 'A', 'A', 'B', 'A', 'A’
• New: NSREDUNDV = 'A', 'A', 'A', 'B', 'A', 'B’
• Old: NSSHPSI = '_NSSHPS4’
• New: NSSHPSI = '_NSSHPS5'
52
Database
PASS Configuration Files (cont)
MEGG initialization file changes to swap
in the appropriate new instruments.
– Old:
WFPC standby
COSTAR hold
STIS operate
NICMOS saa_oper
ACS hold
NCS CPL_hold
– New:
WFC3 hold
COS hold
STIS operate
NICMOS saa_oper
ACS hold
NCS CPL_hold
COS_NUV hold
COS_FUV hold
53
Planning and Scheduling
Procedures
The Procedures are written, configured descriptions of a particular
process workflow that describes the tools and actions that should
be performed to accomplish that particular workflow.
Procedures that need modification to support post-SM4
– None of these updated procedures will be put into place until
after the database merge is complete.
• 3.2.01 - Scheduling Unit Selection
– Remove the WFPC2 decon references and add in the WFC3/ACS/STIS
Anneal proposals
• 3.2.02 - Flight Calendar Preparation
– Add the STIS Low Voltage management (STISLs) back in
• 3.2.03 - Flight Calendar Building
– Remove the special checks for non-functional ACS/STIS detectors
– The following procedures were modified and installed
• 3.2.17 - Science Instrument Safe Mode Recovery
– Add WFC3/COS and post_SM4 ACS
– All Visits (excepting WFC3 Soft-Safing) are ready for use
54
Planning and Scheduling
Tools
The tools are configured programs or scripts that are used to
perform certain actions called out in the Procedures used for
building the weekly flight calendars.
Tools which have already been modified and being used for the
initial SMOV calendar builds
– ccl_gsselect
• Selects the guidestars used in the SMS
• Don’t allow FGS2R2 to be used as the dominant guider
– su_gaps.sqr
• Calendar Visit summary report
• Add COS/WFC3
55
Planning and Scheduling
Tools (cont)
Tools requiring modification for SM4
– data_vol_report.py
• Data Volume report for OPUS
• Add COS/WFC3 and remove WFPC2
– gimme_snaps.py
• Builds the lists of what to schedule on a particular week
• Change the default for SI/configs to exclude back to nothing
– Calendar/SMS checklists
• Checks correctness of flight calendars and SMS’s
• Changes yet to be determined
All of these tools will be put into place operationally after the database
merge is complete.
56
Rendezvous
Transitioning HST to Servicing
Mission 4
Objective: prepare HST for rendezvous with the Space Shuttle while
maximizing the science content of the launch week SMS
Methodology: utilize the nominal P&S Intercept capability
– A primary science SMS with “transition” visits
– A pre-planned intercept SMS with Gyro-4 test and rendezvous attitude
maintenance
– Post-launch switch from science to intercept
Testing:
– Previously used for SM3B
– Tested during the System Readiness 4.2 (SR4.2) SMGT
• Intercept loaded and executing but stopped prior to Gyro-4 test and
rendezvous attitude maintenance to reduce test duration
57
Rendezvous
Science SMS
science
rend att science rend att
science
280
287
Intercept SMS
science
load breaks
rend att
G4 Tests
Rend Att Maint
= execution path if launch occurs
58
Rendezvous
Key Transition Components
Transition Visit
– Slews HST to rendezvous attitude (varies with launch date/time)
– Configures SIs to nominal states in preparation for real-time safing
– Provides intercept point and 2 hours of “dead time” to facilitate interception
Science SMS
– Nominal science SMS
– One Transition visit scheduled for each launch window
• Provides for launch delays
– Nominal product delivery to GSFC
Intercept SMS
– Same as science SMS up through the Transition visit
– Gyro-4 test and attitude maintenance replace science after Transition visit
– Products generated and delivered pre-launch (post-science SMS delivery)
• TDRS updates withheld until post-launch
59
Rendezvous
What happens on launch day?
Science SMS is executing
If launch occurs
– Intercept the Science SMS
• P&S updates TDRS schedule to match the Intercept SMS
• FOT uplinks the Intercept loads
– No science after start of launch window +7h
• SIs are safed and NSSC-1 “idled” in real-time at 7h MET via Command
Plan
If launch is delayed
– Science SMS continues uninterrupted
• FOT uplinks the next loads from the Science SMS
– P&S generates and delivers an Intercept SMS for the next
opportunity
60
Ephemerides
Maintain nominal processing and delivery schedule during
servicing mission
– Prevents gaps in data
• reduces manual intervention
• less disruptive to PASS processing
– Ensures recent data is available in an emergency deploy
situation
Support fast turnaround processing and delivery post-deploy
– The process is fully automated
– P&S personnel will be physically at the STScI for these
critical ephems to deal with any unexpected problems
61
Deploy SMS Process
Firing Up an Improved HST
The deploy SMS is a Health & Safety SMS generated using the nominal H&S
generation process
– 3-Gyro Onboard Attitude Determination (OBAD) = “T”mode
– Built using NSSC-1 FSW 9.0
– 36 hour duration
– BEA compliant initial attitude provided by PCS
– Contains no SI commanding
• Allows installation of NSSC-1 FSW 9.0 and COS FSW
– Removes support for COSTAR and WFPC2
– Fixes a COS Lamp startup failure issue
• Allows real-time commanding to configure HST for use
Tested during the System Readiness 4.2 (SR4.2) SMGT
First Science SMS
– NSSC-1 cold-start due to FSW install in previous SMS
– Begin SMOV activities
– Based upon preliminary SMOV template calendar
62
SMOV4 Calendar/SMS Builds
Construct representative SMOV products assuming a nominal
mission and a nominal SMOV
Products will cover the first 4-5 weeks of SMOV
– Determines the scheduling feasibility of the SMOV Plan
– Finds any inconsistencies between the different SMOV Proposals
These products will be widely distributed for review
– Ensures we haven’t missed or misinterpreted something
Will be the starting point for building the templates the Flight
Calendar Builders will use for flight.
Templates will be started by ~T-14 days and handed to the Flight
Calendar Builder by their nominal T-11d build start.
Currently have the first 3 weeks built which includes the first four
post-BEA days.
– As expected, this post-BEA area is oversubscribed in the SMOV
plan
63
P&S Team SM4 Readiness
Operations Training
No difference in SPSS and PASS process and
procedures between pre and post SM4
operations
Therefore, no special training/instruction required to
support post SM4 operations.
All team members participated in training sessions
to become familiar with COS and WFC3
characteristics
Team also familiarized with any changes in check
tools and procedures related to the return to 3Gyro operating mode
Planning and Scheduling utilizing the Post-SM4
DRM proposals
64
P&S Team SM4 Readiness
Operations Support
P&S staff will be on-site at STScI to cover critical
activities as necessary
–
–
–
–
–
Launch to update TDRS schedule
Pre and Post reboost ephem processing
Post-Deploy ephem
HST Attitude from PCS and H&S SMS creation
Follow-on Science SMS creation
Post-deploy scheduling
– Resume normal SMS build and delivery schedule for
SMOV following post-deploy H&S and first Science
SMS
P&S Team ready for SM4, SMOV and Cycle 17
operations
65
Pending Changes
PRD & SCIOPSDB
Things waiting for SM4 that need preapproved freeze waivers
– PRD & SCIOPSDB updates are included in the
OPS/SM4 database merge
– Planned PRD updates
• PR 60655, 60781: PLCP PIPHTHET, PIHETPHT,
PISAFFLT needed for WFC3
• PR 60663: PLCP PJSAFHLD for ACS safing recovery
including TEC commanding
• PR 60739: AGCF and SCHF parameters for switch to
3-gyro mode
• PR 60742: Table ODB for NSSC-1 9.0 FSW
66
Items Still in Work
Database Merge Test
– Sept 15
WFC3 Soft Safing Recovery Procedure and Visits
– Analysis completed by Sept 12
– Visits created and tested by Sept 19
Create Flight Rendezvous/Transition Visits
– Completed by Sept 22
Calendar and SMS Checklist Tool Updates
– Review and determination of needed changes by Sept 12
– Implementation and testing completed by Oct 3
data_vol_report.py and gimme_snaps.py Tool Updates
– Implementation and testing completed by Oct 3
Flight Functional Tests (FT) Support Schedules
– Completed by Oct 3
Finish building Representative SMOV Weeks out to at least 5
– Completed by Oct 3
67
Commanding
Alan Welty
Commanding Branch - OED
68
Commanding
Front-End Science Instrument Support
– Write requirements for PROPINST, APT, TRANS,
and SCIOPSDB related to SI commanding
– Write & maintain the IM “Instructions” – code used
to generate most SMS statements
– PDB tables and groups used in SI commanding
• Define them, with the SI ops working groups
• Create and validate the PDB files
– Certification (see slide 5)
– SMS Review Tool
• Checks for CARD & OLD violations
• Checks to ensure science data quality
69
Regression Testing1
Bit Validation (“bit busting”) of PDB files
–
–
–
–
Create PDB files & bitval test instructions
Run SPSS & PASS to generate the bits
Verify that the bits are correct
Deliver PDB files & bitval test instructions to
Ball/GSFC
• They also generate the bits & deliver them to us
– Compare STScI and Ball/GSFC products
– Do the paperwork
70
Regression Testing2
Bit Validation Status
– ACS-R
• New items Done (3 forms yet to be signed)
• Items unaffected by repair – In progress
– NICMOS
• Done (revalidation not requested - NICMOS is fully operating)
– STIS
• TBD (no changes, so no issues are expected)
– COS
• Done (3 forms yet to be signed)
– WFC3
• GSFC has 2 TBDs and 8 forms to sign
• STScI Done (except for signing 10 forms…)
71
Regression Testing3
Certification
– Create/update certification test proposal (in APT)
•
•
•
•
–
–
–
–
One proposal for each SI
Typically, each visit tests a different ‘major’ capability
Exercise all legal proposal inputs
Negative testing is left to the Test Team
Process proposals with TRANS
Load TRANS output on CMD’s dev database
Generate calendars & SMSs
Verify correctness of SMSs
• Make detailed check of any new commanding
• Explain any differences in pre-existing commanding
– Update SRT if necessary & run it on the SMSs
72
Regression Testing4
Certification Status
– ACS-R
• Last: 05-Sep-08; Up to date
– NICMOS
• Last: 15-Jan-08; Up to date
– STIS
• Last: 30-Jul-04; Up to date
– COS
• Last: 05-Sep-08; Up to date
– WFC3
• Last: 09-Jun-08; Recertification in progress
73
Special Commanding
– ACS-R
• Tested in SFT/SMGT 06-Sep-08
• Optimization Spec Com tested in TV 23-Aug-08
– NICMOS
• N/A
– STIS
• Tested in SFT/SMGT 24-Jun-08
– COS
• Tested in SMGT 08-Sep-07
• Liens SMS run in ESTIF 16-Jan-08
– WFC3
• Tested in SMGT 11-Feb-08
74
Other Work
– Instruction Audit
• SPSS_SM4/IMDB comparison completed 05-Sep-08
– COS & WFC FSW Patchable Constants
• Prep for verification/signoff in progress
– Documentation
• ACS DM-05 update ready for distribution
• COS & WFC3 DM-05 updates in progress (minor)
• COS & WFC3 OP-01 updates TBD
– SMS Review Tool
• Necessary code is complete; some input data updates
are TBD (related to tracking info, not SMS checking)
• To be installed during SM4 (some updates are not
appropriate for pre-SM4 operation of ACS)
75
SM4 Data Management
Software Status
Faith Abney
Data Systems Branch - OED
76
What do we mean by “Data
Processing” ?
• Receipt of science and engineering data
• Reformatting, quality checking,
calibration, to prepare data for archive
• Archiving the data
• Retrieving the data
• Processing & calibration of retrieved data
• Sending data to the user
• User access tools
77
Data Ingest Pipeline
78
Data Distribution
79
Raw Data Receipt and Ingest
•
PACOR interface code has not changed
–
Tested with all ground tests including STScI and PACOR
•
–
•
•
Used daily to support HST operations
No changes to POD ingest processing
–
–
–
–
•
included all SMGT data
All instruments handled identically
No instrument specific code
Tested with all SMGT data for COS and WFC3
Used daily to support HST operations
100% of COS and WFC3 SMGT data have gone through data
receipt and ingest processing
No critical outstanding PRs for this part of the system
80
Data Ingest Pipeline
81
OPUS => CORE
Science Data
Processing
82
82
Core Science Data Processing
• Instrument specific changes in every step
• Biggest changes with:
– World Coordinate System is mostly new code for
each COS and WFC3
– Generic Conversion for COS is almost all new
code
– Generic Conversion for WFC3 reused much of
ACS code
83
Core Science Data Processing
• 100% of COS and WFC3 SMGT data make it
through
– Still have changes to be completed in 2008.4
• 100% of TV went through all relevant stages
– TV data has no schedule information so can’t
complete all pipeline stages
– Haven’t gone back to rerun all the data with latest
software version
• Outstanding SM4 PRs to go into next build
– 59281, 60725, 60640, 60722, 59921, 60240,
60349, 60440, 60630
84
59821
60725
OPUS => CORE
Science Data
Processing
60640
60722
59921,60240, 60349,
60440, 60630
Outstanding PRs to
go into next build
85
85
Data Ingest Pipeline
86
Catalog
• Keywords are extracted from data headers
and are used to populate the database
– Includes searchable parameters such as RA,
DEC, filters, detector, exposure times, etc
– Includes instrument specific tables
• Database tables are searched by archive
users using StarView or the Web Interface
87
Cataloging for COS and WFC3
• Cataloging is working for COS and WFC3
– Cataloged both SMGT and TV data
– TV data gets archived to operational archive using
own set of tables
– SMGT data archived to the test system
• 100% WFC data is cataloged
• 97% of COS data is cataloged
– Shared wavecals don’t work yet
• Don’t expect any in SMOV
– Some ACQs still being worked on
88
Cataloging for COS and WFC3
• WFC3 keyword cataloging is complete
– May add more keywords after SMOV
• COS first cut at keyword cataloging done
– Iterating with the COS team
• To verify catalog contents, an independent
verification tool checks the data headers against the
catalog
– Verifies right values come from the right extensions
– Tool has been used for other instruments, recently updated
for COS and WFC3
• One outstanding PR, 60615
89
Database Tables
Instrument
# Fields
Table Size (KB)
WFPC2
405
837644
STIS
783
1401432
NICMOS
746
2095472
ACS
503
448732
COS
455
30
COS_TV
449
14556
WFC3
357
30
WFC3_TV
386
175576
90
Data Ingest Pipeline
0 PRs
9 PRs
1 PR
1 PR
91
Data Distribution
92
Data Distribution Changes
• Added COS and WFC3 to list of known HST instruments
• Added definitions of “Calibrated” and “Uncalibrated” data
retrieval keywords for COS and WFC3
– Allowed for long extension names from OTFR for COS data (all
other OTFR instruments used 3 char extensions)
• Added more disk space
• Distributed 100% of the COS and WFC3 that was ingested
(SMGT & TV)
– Includes going through OTFR
• No outstanding critical DADS PRs
93
Keyword Extensions
CALIBRATED
UNCALIBRATED
COS
asn, corrtag_*, coretag_a,
coretag_b, flt_a, flt_b, flt,
fitsum, lampflash, spt, trl,
x1d, x1dsum1, x1dsum2,
x1dsum3, x1dsum4,
x1dsum
asn, mdd, pha_a, pha_b,
rawaccum_a, rawaccum_b,
rawaccum, rawacq, rawtag,
rawtag_a, rawtag_b, spt, trl
WFC3
asn, crj, drz, flt, ima, spt, trl
asn, raw, spt, trl
An extension will be returned when a particular keyword is requested if that
file exists for that dataset or within that association.
94
Data Distribution
95
Retrieval Interface
• StarView was rusty railed 4 years ago
– Had support issues with old code and new
databases this spring
– Data Retrieval by StarView was less than
3% of total
– Usage was largely in-house by data
analysts and instrument scientists
96
Retrieval Interface
• StarView functionality migrates to Web
– Retain the name “StarView”
– Retrievals for COS and WFC3 are working on the
MAST science searches
– Search screens for COS and WFC3 instrument
and engineering searches are not yet ready
• Will be delivered as they become ready - not tied to build
• Mostly used by instrument scientists and data analysts
97
Outstanding Critical PRs
• WFC3 - NONE
• COS
–
–
–
–
–
–
–
–
59921 – GC changes
60240 - Exptime when TDF is down
60349, 60630 - COS ACQ files
60440 – SPT Headers
60486 – COS engineering data issue
60615 - Keywords
60722 – WCS
60725 – Grating value
98
Outstanding Critical PRs
• ACS
– 59821 - Oscilloscope mode
– 60640 – Gain value changes
• STIS
– None
NICMOS
– 60429 - SAA clean timing issues
99
Outstanding SM4 Critical PRs
WFC3
0
COS
9
ACS
2
STIS
0
NICMOS
1
TOTAL
12
100
Build Schedule
• DMS 2008.4
– All SM4 critical PRs are in build DMS 2008.4
– Also includes 3 non-critical, database table fixes
– Enhancements for easier reprocessing in
operations
– Installation scheduled for September 30
– Each PR approved by HSTMO prior to installation
• Newly discovered critical problems will be
delivered as quick fixes after 2008.4
• 2009.1 installation planned in early 2009
101
Routine Maintenance after SM4
• Likely items for DMS 2009.1
–
–
–
–
Changes based on SMOV experience
COS PulseHeight Histogram
COS Accumulated Images
Pipeline enhancements
• NICMOS Reprocessing
– 60595,60597,60598, 60610, 60701
102
Calibration Software Status for
SM4
Warren Hack
Science Software Branch - OED
103
Data Ingest Pipeline
Generate metadata from
calibrated datasets
104
Data Distribution
The calibration software also
gets provided to the user
as part of STSDAS.
105
Current Instruments
The pipeline calibration software for the legacy
instruments has been reviewed to see whether any
changes were needed to support SMOV activities.
ACS: The ACS pipeline software will require changes to support the
new commanded values of the gain.
STIS: No changes required for SMOV support.
NICMOS: The NICMOS pipeline have been enhanced with deliveries
to OPUS 2008.3 in order to prepare for reprocessing of pre-SM4
NICMOS data.
WFPC2: All software development has focused on supporting final
reprocessing during SMOV, and the version now in place will be
sufficient.
106
WFC3
CALWF3 Version 0.95 was delivered to OPUS Build 2008.3.
– This version of CALWF3 has been used to successfully process 100% of
the relevant ground-test data from Servicing Mission Ground Test
(SMGT) through OPUS. It has also been used by the WFC3 team to
process all the Thermal-Vac (TV) test data.
• Testing of CALWF3 carried out by OPUS as part of standard regression
testing for operational OPUS builds using SMGT data, and through a
separate OPUS TV pipeline to process WFC3 thermal-vac data as needed by
the WFC3 team.
• OPUS regression testing using SMGT data starts with raw telemetry files
(POD files) processes them through Generic Conversion, CALWF3 and
ingests the calibrated results in the archive for WFC3 team review.
• OPUS regression testing validates the entire calibration pipeline system from
telescope to delivery to the end user, while processing the TV data validates
the algorithms used in CALWF3 for the calibration.
– Updates to the non-linearity and bias-level corrections were identified as result of
the latest TV data, as well as bugs in CR rejection algorithms.
107
WFC3
CALWF3 has been tested using thermal-vac data and SMGT data.
– Thermal-vac data:
• Nearly 60,000 exposures
• Every conceivable detector readout mode, including UVIS Extended Pixel
Edge Response (EPER) mode for CTE, and UVIS and IR special warm
engineering test readout modes which will be used during the SM4 functional
test.
–
SMGT data:
• Regression test suite for OPUS/CALWF3 and for complete end-to-end test of
all WFC3 front-end and back-end systems.
• 30 visits with 99 exposure lines resulting in 172 individual exposures.
• Wide range of possible exposure modes (binned, unbinned, 1-2-4 amp
readout, UVIS and IR subarrays,…)
• 22 associations consisting of pure CR-SPLIT, pure REPEAT-OBS, pure dither
for both UVIS and IR, along with IR REPEAT-OBS plus dither associations.
108
COS
CALCOS Version 2.4 has been delivered for OPUS Build
2008.3.
– This version of CALCOS has been tested using thermal vac (TV)
data prior to delivery to OPUS.
– Regression testing performed by OPUS relies on the SMGT
datasets. These datasets contain blank science arrays, but
complete headers.
– OPUS regression testing has validated that 97% of all the COS
SMGT data can be processed through the entire calibration
pipeline system from telescope to delivery to the end user.
• The only failures coming from the use of shared wavecals.
– The COS Pipeline Verification Team processes the TV data to
validate the algorithms used in CALCOS for the calibration.
109
COS
The COS PVT has reported:
– Functional testing has been completed successfully for CALCOS.
– Verification of the pipeline products based on assessment of the
calibrated data and the algorithms has been performed using TV
data for at least 8 of the 20 modules in CALCOS.
The data used for performing these tests include:
– The TV data from 2006 with more than 2100 exposures, covering
the majority (if not all) observing modes.
– The SMGT data with about 40 associations of both ACCUM and
TIME-TAG data taken by both NUV and FUV channels.
110
OPUS 2008.4
Work continues on CALCOS, with only a couple PRs
which were resolved with the delivery of CALCOS v2.5
for OPUS 2008.4
– CALCOS should become an STSDAS package. (PR 60180)
– Reference file handling for wavelength calibration (PR 60653)
– Use of science exposure SPT for product SPT file (PR 60708)
Recent TV testing of WFC3 has highlighted the need to
make some modifications to CALWF3 to support the
instrument during SMOV. The following OPRs were
resolved with the delivery of CALWF3 V0.99 for OPUS
2008.4.
– Cosmic-ray rejection algorithm updates (PR 60727, 60710,
60564)
– Produce ‘drz’ files (PR 60674)
111
OPUS 2008.4
The repair of ACS will return it to an operational state, and
changes will be delivered for testing under 2008.4 to
support ACS SMOV activities:
– Support the new commanded gain values for ACS (PR 60729)
– Corrections to cosmic-ray rejection (PR 60727, 60710)
112
MultiDrizzle
No issues are expected for pipeline use of Multidrizzle during SMOV.
MultiDrizzle supports all current instruments imaging modes for
ACS,STIS,NICMOS, and WFPC2, as well as WFC3 UVIS and IR images.
Multidrizzle will only be used in the ACS and WFPC2 pipelines during SMOV,
but not for WFC3.
–
–
–
It will not be turned on for WFC3 pipeline use due to the fact that on-orbit
calibrations will need to be performed in order to generate accurate enough
distortion models for WFC3. Verified distortion-correction reference files will be
generated by the WFC3 Instrument team based on calibrations taken in Nov
29 for UVIS and Dec 3 for IR.
SMOV calibration activities will still be able to use any new reference files
generated from on-orbit data using the locally installed version of Multidrizzle.
Pipeline use of MultiDrizzle for WFC3 can be added to OPUS as early as
OPUS 2009.1.
113
Post-SMOV Work
Work continues on CALWF3 based on the results of TV testing of the
IR channel to update the following processing steps:
• the IR reference pixel subtraction (PR 60657)
• the IR non-linearity correction (PR 60659)
A number of issues have been identified for COS and even STIS which should
not affect SMOV support. These issues have been given a low priority and
will be resolved in time for delivery to OPUS 2009.1.
– COS: Trailer file handling may need to be revised based on experience during
SMOV.
– STIS: Zero exposure time members of STIS associations disappear
from the calibrated product (PR 60326)
– STIS: CALSTIS should use spectral order when saving the gross count
rate (PR 60327)
There will additional issues which come up during SMOV, and those will be
prioritized based on the impact to SMOV activities.
114
Routine Maintenance After SM4
Significant testing has been performed to verify that data taken in the most
common modes from each instrument will process successfully through
the entire pipeline; from raw telemetry through calibration to archive
ingest and retrieval from OTFR. However, issues with unexpected
situations will arise during SMOV which will require immediate calibration
software fixes, and a procedure will be in place to address these issues.
Those isolated calibration pipeline problems which prevent the completion of
processing and delivery of data to PIs will be worked with the highest
priority during SMOV by SSB.
The SSB developer will assess whether the problem arises from:
–
–
–
a software bug,
reference file problem (which may or may not require a software change), or
input file formatting issue (which may still require a software change).
For fixes which require calibration software changes:
1.
2.
3.
4.
data causing failure will be provided to SSB
software will be revised to address the issue
functional testing will be performed by SSB along with testing against the
problem data to verify that the problem was adequately resolved while still
successfully processing the functional test data.
new code will be delivered to OPUS using standard mechanism
115
Routine Maintenance After SM4
Any issue identified as a reference file problem will be reported to the
instrument team for resolution, and any relevant software change
will be verified using the procedure outlined here.
Any issue involving the input file format will be reported to OPUS for
discussion and resolution, and any relevant software change will
be verified using the procedure outlined here.
For those issues which do not prevent the processing of data or
impede the deliver of data to the PIs:
OPRs will be filed, and
the work will be prioritized using current methods and criteria.
–
Such issues might include errors in keywords or interpretation of
association tables.
116
Operational Readiness
Albert Holm
Data Processing & Archive Services OED
117
Topics
•
•
•
•
•
•
Rapid data delivery
Pointing adjustments
Activity and staff schedules
Data release policy implementation
Metrics/completeness tracking tools
Anomaly resolution procedure
118
Data Ingest Pipeline
119
Data Distribution
120
Rapid Data Delivery
121
Rapid Data Delivery - 2
Proposals and visits requiring rapid turn
around given fast track service
– routinely identified via the SMOV4
Proposal Implementation Team (PIT)
– 14 ACS, 3 STIS, 2 NICMOS, 7 COS, and 4
WFC3
– SM4 functional test data also will be fasttracked
122
Rapid Data Delivery - 3
Rapid delivery managed with existing DPAS procedure
11.3.3 & Fastrak tool
•
•
PIs asked for destination directories for transfer tests in August
ACS oscilloscope mode data requires special preparations
–
–
–
–
Resource file overrides
Transfer of ACS memory dumps successfully tested
Oscilloscope tables test awaiting processing capability (PR 59821 in DMS 2008.4)
ITSD created a web interface to allow GSFC engineers to access the data
PR 59582, Fastrak hangs when transferring more than 1 GB
of data during session
•
Fix planned for DMS 2008.4
123
Pointing Adjustment Support
Genslews allow small pointing adjustments of
the telescope to correct for inaccuracies in
assumed aperture positions
• Ten SMOV4 proposals requiring Genslews (One ACS
and nine COS) identified via PIT
• ACS coronographic apertures move
• COS aperture locations may need adjustment from
pre-launch measurements
Genslew ops requests managed via existing
DPAS procedure 11.7.9
124
Activity Schedules
Activity schedules for fast track and pointing
adjustments are derived from mission
schedule data delivered by SMSB.
• OpusLib java tool automatically generates a table of
times for pointing adjustments
• Fast track dump times are determined manually from
the RDO files
Staff schedules will be adjusted if needed for
critical events
125
Staff SMOV4 Schedules and Kepler
Kepler Operations Readiness Test duties are
split up among four DPAS personnel:
• Forrest Hamilton, DMC Ops Lead, manages the test setup and
execution, and provides HST trouble shooting
• Deb Kenny and Gustavo Cardona will support the ORTs and
SMOV4 data processing operations
• Tracy Ellis will support SMOV4 and will support the ORTs if
SMOV activities allow
Only ORT-2a is likely to occur during SMOV
• This ORT will simulate a few days during commissioning
• Impact upon SMOV activities should be small since this ORT
will require the processing of only 6 Full Frame Images (FFIs)
126
Data release policy implementation
OPB has set initial proprietary period to
one year for proposals using either new
SIs or serviced modes
• ERO and ERS data will remain proprietary
until their press release
• SMOV and calibration data for COS, WFC3,
and the serviced modes will remain
proprietary until ERO press release
127
Data release policy implementation - 2
Manual process for shortening release
dates will continue
• based on proposal ID, not visit or SI mode
GO data taken with new SIs and with
serviced modes will be embargoed
• authorization to access data from proposal
will be withheld until the PI agrees not to
discuss or present the data publicly until after
ERO press releases
128
Embargo Agreement Text
129
Metrics/completeness tracking tools
Basic tracking tool recognizes COS & WFC3 data
• not thoroughly tested, but should be able to track
whether all expected observations have been
received and processed
• supplies HSTMO metric pages HST Science Data
Availability and HST Processing Status
More detailed metrics tools and reports still in work and
probably will not be ready at start of SMOV4
130
Anomaly resolution procedure
Procedure 11.10.1, Response to Pipeline Processing
Problems, will be followed
1.
2.
3.
4.
5.
Investigate problem using process log files and trailer files
as appropriate
If symptoms are of a known problem with an existing
workaround, execute the workaround.
If the problem is known but no workaround exists,
document the error and the dataset.
Notify DSB of a previously unknown problem if error
occurred in code DSB maintains
Notify SSB and the SI team of a previously unknown
problem which occurs during calibration
131
Infrastructure Contingency
Planning
Doris McClure
IT Services Division
132
Topics Covered
Facilities readiness and failover capabilities
– Backup power
– Building security
Essential computer equipment
–
–
–
–
–
–
Planning and scheduling systems
Data Management Systems
Science instrument support systems
Visiting instrument teams
Infrastructure: central storage, network, email
Off-hours support during SM4 and SMOV
Internet access/data transfer
133
Backup Power Capabilities
All essential network equipment, servers, and storage
are located in computer rooms connected to an UPS
(uninterruptible power supply)
– Provides 30-45 minutes of backup power
In addition, critical servers and core network switches
are attached to power generator
– Provides unlimited backup power
– Covers Planning and Scheduling systems and Data
Management System
Failover to power generator was successfully tested on
8/15
134
Building Security – Muller Bldg.
Normal operating procedures
–
–
–
–
Guard stationed at building entrance 24x7
Building open from 7am to 6pm, all visitors must sign in
Access card for entry to building is required outside these hours
First floor (where all critical computer equipment is located) is
locked down from 6pm to 7am weekdays and on weekends
Enhanced security capabilities
– By launch additional access card readers will be in place providing
capability to:
• Lock down entire building at a moment’s notice
• Lock down selected sections of the building
Plans during SM4
– Access through front door will continue to be from 7am to 6pm
– First floor access will be controlled, i.e., access card required, 24
hours/day, 7 days/week
135
Essential Computer Systems - 1
Planning and Scheduling systems
– Servers protected against power outages by both UPS and
power generator
– Desktop systems protected against power outages by UPS
– Distributed model ensures that failure of any one system
should not affect operations
– If entire site (STScI) becomes unusable, STScI can move
this function to GSFC where offsite backup servers are in
place
• Last successful test was performed 10 July 2008
Data Management Systems
– Servers and storage are protected against power outages by
both UPS and power generator
– No backup/offsite capabilities exist
136
Essential Computer Systems - 2
Science instrument support systems
– Systems are primarily individual desktop/laptop systems
– Protection against power failures
• Systems in 4th floor South offices are protected by UPS and power
generator
• Other offices are not protected
• Alternative locations exist (e.g. Rm 111 “Fishbowl”) that are protected
by UPS and power generator in the event of power failure
Visiting instrument teams
– Space set up in 135 and N409
– Procedure in place to certify visitor systems prior to attaching them
to the STScI network
– No protection against power failure
• Can use Rm 111 in the event of power failure
137
Essential Computer Systems - 3
Network equipment and central storage system
– Critical network equipment (core switches and routers) and the
central storage system are protected against power failures by UPS
and the power generator
– Core switches and routers have failover capability to backup
switch/router
– Central storage system has new areas dedicated to SM4/SMOV
configured in highly available (RAID6) configuration. Two disks can
fail and the storage will continue to function.
Email server
– The email server is also connected to the UPS and generator
– There is no backup server available, however alternatives will be in
place:
• Use of STScI Information Channel (SIC) via web
• Contact list includes alternate email addresses for all essential staff
138
Essential Computer Systems - 4
Off-hours support
– On site support is provided from approximately
7am to 6pm, Monday through Friday
– During mission, extra support will be on site as
required for quick turn around items
– On-call procedures in place for unanticipated
failures of essential computer systems
• ITSD management will be point of contact and on call
24x7 throughout SM4 and SMOV
• Essential IT personnel will be on call and will respond to
emergencies
139
Internet Access/Data Transfer
Internet access is required to:
– Send/receive all data between STScI and GSFC
No backup capability exists at STScI
Plans in event of unanticipated, extended failure
–
–
–
–
Potential use of JHU network for certain data transfers
Data from PACOR would need to be transferred via media (e.g. tape)
Planning and Scheduling operations move to GSFC
Science instrument support staff can go to GSFC
• Laptops are certified and HSTnet ready
• Offices at GSFC will be available through the end of January 2009
Plans to address outages
– Scheduled outages: NISN sends notices of planned outages well in
advance, usually 1-2 weeks. If timing is bad, we will reschedule with
GSFC/NISN.
– Unscheduled outages: STScI will contact GSFC (Simrall) and/or NISN Help
Desk in the event of an unplanned outage and work to resolve as quickly as
possible.
140