NICMOS Suspend Event
STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
2008/069
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• At 2008/069/02:46:03 UTC, NICMOS Suspended with an
NICMOS 632 status buffer message (parameter = 225 octal, 149
decimal)
• Indicates Out of Tolerance move for a commanded move of Filter
Wheel 2 (FW2) to the F110W filter position
– Tolerance is 110 counts
• STB parameter indicates a disparity of 149 counts
– FSW analysis indicates that the final FW2 position was only 13 counts away
from the expected, indicating FW2 did settle at the correct position
• NICMOS was in Observe executing proposal 11168
– The IMF in the Hidden Galactic Starburst W49A
2
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• 10 prior FW 2 moves had been successful since transitioning from
SAA Operate to Operate
– 069/02:12:41 – Transition from SAA Operate to Operate complete
– 069/02:14:28 - 02:43:38 – Blank, F110W, F165M, F207M, F110W, F165M,
F207M, F110W, F165M, F207M
– 069/02:45:50 – F110W (NICMOS 632)
• At the time of the event:
– HST was not in an SAA passage
– HST had entered orbit day at ~069/02:16 (roughly orbit noon)
• HSTAR 11215 has been opened to document this event
3
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• Original occurrence in 1998
– HSTAR 6815
– Occurred during a move from Blank to F110W
– Attributed to known noise in the FW2 electronics
• Most recently in 2007/158
– HSTAR 10851
– Occurred during a move from F171M to F110W
– Also attributed to known noise in the FW2 electronics
• From HSTAR 10851 closure: “Ground testing did demonstrate this susceptibility, however, and it is
attributed to noise in the electronics. Bench testing of these mechanisms before integration into NICMOS
showed torsional oscillation took ~8 seconds to completely dampen out after a motion was commanded.
Review of the code revealed that a settling time of 3 seconds is allocated between moves, but not between
moves and position measurements. Therefore the Filter Wheel is still ringing when the FSW is reading
out the inductosyns. This will occasionally results in an erroneous measurement which is outside the
miss-distance tolerance of 110 counts.”
– SCR N52 - NICMOS Filter Wheel retry failures opened 7/19/07
• Single retry events remain common on all three Filter Wheels
4
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 632
5
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• OR 18212-0 to dump NICMOS memory per ROP NS-09 was
completed at ~069/05:09
– Dump files transferred to Payloads FSW for analysis
• Engineering telemetry extracted and examined for any anomalous
behavior with FW2 or the Mechanism converter (see following)
– Nominal FW2 move from 068 included for comparison
6
Tim e (UTC)
2008.069 02:50:28.991
2008.069 02:49:09.491
2008.069 02:48:37.991
2008.069 02:48:07.491
2008.069 02:47:09.991
2008.069 02:46:38.491
2008.069 02:46:07.991
2008.069 02:45:28.991
2008.069 02:44:39.491
2008.069 02:44:08.491
2008.069 02:43:37.491
2008.069 02:42:39.991
2008.069 02:42:09.491
2008.069 02:41:37.991
2008.069 02:41:07.491
2008.069 02:40:09.991
2008.069 02:39:38.491
2008.069 02:39:07.991
2008.069 02:38:28.991
2008.069 02:37:39.491
2008.069 02:37:08.491
2008.069 02:36:37.491
2008.069 02:35:07.491
Temperature (degC)
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event
Filter Wheel 2 Tem perature
069/ 02:31:00 - 02:51:00
0
-1
-2
-3
-4
NFW2TMP
-5
-6
-7
-8
-9
-10
7
Tim e (UTC)
2008.068 21:46:39.990
2008.068 21:46:07.990
2008.068 21:45:09.490
2008.068 21:44:28.990
2008.068 21:43:37.990
2008.068 21:42:09.490
2008.068 21:41:28.990
2008.068 21:40:37.990
2008.068 21:39:39.490
2008.068 21:39:07.490
2008.068 21:38:08.490
2008.068 21:37:09.990
2008.068 21:36:37.490
2008.068 21:35:38.490
2008.068 21:34:39.990
2008.068 21:34:07.990
2008.068 21:33:09.490
2008.068 21:32:28.990
2008.068 21:31:37.990
2008.068 21:30:39.490
2008.068 21:30:07.490
2008.068 21:29:08.490
2008.068 21:28:09.990
2008.068 21:27:37.490
Temperature (degC)
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event
Filter Wheel 2 Tem perature
068/ 21:27:00 - 21:47:00
0
-1
-2
-3
-4
-5
-6
-7
NFW2TMP
-8
-9
-10
8
Tim e (UTC)
2008.069 02:50:28.991
2008.069 02:49:09.491
2008.069 02:48:37.991
2008.069 02:48:07.491
2008.069 02:47:09.991
2008.069 02:46:38.491
2008.069 02:46:07.991
2008.069 02:45:28.991
2008.069 02:44:39.491
2008.069 02:44:08.491
2008.069 02:43:37.491
2008.069 02:42:39.991
2008.069 02:42:09.491
2008.069 02:41:37.991
2008.069 02:41:07.491
2008.069 02:40:09.991
-8
2008.069 02:39:38.491
-6
2008.069 02:39:07.991
-4
2008.069 02:38:28.991
-2
2008.069 02:37:39.491
0
2008.069 02:37:08.491
2008.069 02:36:37.491
2008.069 02:35:07.491
Voltage (V)
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event
+/- 12 V Resolver Voltages; +15 V, +5 V Mechanism Voltages
069/ 02:31:00 - 02:51:00
16
14
12
10
8
6
4
2
N12V1RES
N12V2RES
N15VMECH
N5VMECH
NM12V1RS
-10
NM12V2RS
-12
-14
9
Tim e (UTC)
2008.068 21:46:39.990
2008.068 21:46:07.990
2008.068 21:45:09.490
2008.068 21:44:28.990
2008.068 21:43:37.990
2008.068 21:42:09.490
2008.068 21:41:28.990
2008.068 21:40:37.990
2008.068 21:39:39.490
2008.068 21:39:07.490
2008.068 21:38:08.490
2008.068 21:37:09.990
2008.068 21:36:37.490
2008.068 21:35:38.490
2008.068 21:34:39.990
2008.068 21:34:07.990
2008.068 21:33:09.490
-8
2008.068 21:32:28.990
-6
2008.068 21:31:37.990
-4
2008.068 21:30:39.490
-2
2008.068 21:30:07.490
0
2008.068 21:29:08.490
2008.068 21:28:09.990
2008.068 21:27:37.490
Voltage (V)
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event
+/- 12 V Resolver Voltages; +15 V, +5 V Mechanism Voltages
068/ 21:27:00 - 21:47:00
16
14
12
10
8
6
4
2
N12V1RES
N12V2RES
N15VMECH
N5VMECH
NM12V1RS
NM12V2RS
-10
-12
-14
10
Tim e (UTC)
2008.069 02:50:28.991
2008.069 02:49:09.491
2008.069 02:48:37.991
2008.069 02:48:07.491
2008.069 02:47:09.991
2008.069 02:46:38.491
2008.069 02:46:07.991
2008.069 02:45:28.991
2008.069 02:44:39.491
2008.069 02:44:08.491
2008.069 02:43:37.491
2008.069 02:42:39.991
2008.069 02:42:09.491
2008.069 02:41:37.991
2008.069 02:41:07.491
2008.069 02:40:09.991
2008.069 02:39:38.491
2008.069 02:39:07.991
2008.069 02:38:28.991
2008.069 02:37:39.491
2008.069 02:37:08.491
2008.069 02:36:37.491
2008.069 02:35:07.491
Current (A)
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event
Mechanism Current
069/ 02:31:00 - 02:51:00
0.000
-0.001
-0.002
NMECHCUR
-0.003
-0.004
-0.005
-0.006
-0.007
-0.008
-0.009
-0.010
11
Tim e (UTC)
2008.068 21:46:39.990
2008.068 21:46:07.990
2008.068 21:45:09.490
2008.068 21:44:28.990
2008.068 21:43:37.990
2008.068 21:42:09.490
2008.068 21:41:28.990
2008.068 21:40:37.990
2008.068 21:39:39.490
2008.068 21:39:07.490
2008.068 21:38:08.490
2008.068 21:37:09.990
2008.068 21:36:37.490
2008.068 21:35:38.490
2008.068 21:34:39.990
2008.068 21:34:07.990
2008.068 21:33:09.490
2008.068 21:32:28.990
2008.068 21:31:37.990
2008.068 21:30:39.490
2008.068 21:30:07.490
2008.068 21:29:08.490
2008.068 21:28:09.990
2008.068 21:27:37.490
Current (A)
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event
Mechanism Current
068/ 21:27:00 - 21:47:00
0
-0.001
-0.002
-0.003
-0.004
-0.005
-0.006
-0.007
-0.008
NMECHCUR
-0.009
-0.01
12
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• Memory dumps have been analyzed by Payloads FSW
– EEPROM shows the expected 5-byte difference for the stored motor
commutations
– No out of limit events recorded in internal limit table
– Internal error log has several instances of previous motor retries
• Error Number 633 = FW2 Retry (internal ESB only)
– 19 events since 2008/001
• Error Number 632 = FW2 Retries Allowed Exceeded (Suspend)
---------------------------------------------------------------------------54 | 2008/069/00:54:13.803 | 633 | 121 | 0x0075
---------------------------------------------------------------------------55 | 2008/069/01:00:33.804 | 633 | 121 | 0x0071
---------------------------------------------------------------------------56 | 2008/069/02:45:56.791 | 633 | 121 | 0x0072
---------------------------------------------------------------------------57 | 2008/069/02:46:02.894 | 632 | 120 | 0x0095
---------------------------------------------------------------------------13
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• Forward Plan
– All NICMOS telemetry indicates there are no hardware or operational
concerns
• Filter Wheel retries are commonplace occurrences
– Recommend recovery to Operate and repositioning of the PAM using real
time commanding
• See MEGG following for command windows
– Next external NICMOS science observation is scheduled for 069/16:15:16
• Proposal 11157 - NICMOS Imaging Survey of Dusty Debris
Around Nearby Stars Across the Stellar Mass Spectrum
14
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• Operations Request 18213-0
– Recover NICMOS from Suspend to Hold mode
• Operate converter relay opened on Side 1
• Suspend flag and Checksum error flag cleared
• NICMOS RTCS processing enabled
– Transition NICMOS from Hold to Boot
• Operate converter relay closed on Side 1
• Suspend AP enabled
• Boot FSW initialized
– Transition NICMOS from Boot to SAA Operate
•
•
•
•
Operate FSW copied to RAM and initialized
Buffer Boxes turned on
Mechanisms initialized
PAM sensor turned on
– Transition NICMOS from SAA Operate to Operate
• Initialize and enable detectors
– Wait at least 1 hour from SAA Operate before moving PAM
– Move PAM to Camera 2 coronagraphic position (PAMC) in preparation for
intercepting the science timeline
15
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
16
NICMOS Recovery to “Operate”
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS Recovery to “Operate” cont’d
PAM Move (At least 1 hr after NICMOS to SAA Operate)
17
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS First Science Observations (Proposal 11157)
18
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS First Science Observations (Proposal 11157)
19
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• On-orbit Validation and Monitoring
– SI SE will monitor transition to SAA Operate in the SMOR
– SI SE will monitor results of first observation and filter wheel move
remotely (actual first observation occurs during LOS)
• System Impacts
– SI
• NICMOS ready for science
– DMS, EPS, I&C, OTA, PCS, Safing, TCS
• No impact
– Critical Science Considerations
• None
20
NICMOS STB 632
“MECH_2_MAX_RETRIES_EXCEEDED”
• Follow-up Activities
21