JWST Program Processing Procedures Editor: Beth Perriello Abstract This document is intended as an introduction to JWST program processing. Throughout the document, examples use 9999 in place of the actual program number. A knowledge of simple unix commands, and an unix editor (like emacs or vi) is assumed in this document. Phase I to Phase II Transition Unlike HST programs, there will not be different IDs used for phase I and phase II. Phase I and phase II do not exist for JWST. Instead, the ID assigned when the Principal Investigator (PI) submits his/her proposal will be the ID used through the entire process. The term proposal will be used for anything that the PI submits for the Telescope Allocation Committee (TAC) to review. Once a proposal is accepted by the TAC, it should be referred to as a program. Accepted programs will be assigned a program coordinator (PC) and possibly a contact scientist (CS). CS assignments are made for all Target of Opportunity programs (ToO), moving target programs and Director’s Discretionary programs (DD), or at the request of the PI or PC. Help Developing Programs 1. Receive notification of program When GO and DD program assignments are complete, each PC will receive an email of the notification letter (which includes the TAC Peer Review comments) for his/her assigned programs. In the case of a GTO, CAL, COM, ENG, ERS or NASA programs, the PC will be given the PI’s name and email address. 2. Bring up the Toolbox The toolbox is available, but all of the buttons do not work yet. Access to most of the required tools can be found in one graphical toolbox. The Toolbox is a family of programs written and maintained by OPB. It combines several tedious functions into one button click and a short question and answer session. To bring up the Toolbox type tools at the unix prompt. When it first comes up, you will need to choose a submenu by clicking on one of the buttons below “working directory”. You can choose any tool within a submenu by clicking on it. When using the Toolbox, you can fill in one or more program numbers (separated by spaces) on the “list” line of the interface or you can search for programs by a specific PI. The working directory listed on the second line is the directory where the Toolbox will look for input files and direct output files. You can change this for the current session, but each time the Toolbox comes up fresh, it will assume that your working directory is the one stored with your preferences. You can change this and other Toolbox preferences by clicking on the Preference button. 2.1 Run Create This step is only necessary if you are working on a new program that was not anticipated (such as a new calibration or DD program). Most GO, GTO and calibration programs will have had a number reserved in the program library (PLIB), so it not necessary to create a program. To add a new program to the PLIB, type the following at the unix prompt: plib create –-­‐help Alternatively, you can use the Create button in the PLIB menu of the Toolbox. Create does the following: -­‐ Ensure that a program with that title and cycle is not already in PLIB -­‐ Uses the plib create command to assign the next sequential program number to the program -­‐ Places the file in the PLIB -­‐ Creates the first records for the new program in the database -­‐ Records the TAC allocation in the database You will get a message in the Toolbox message area telling you that the command ran successfully and what the new ID is. 3. Get Organized It is critical that the PC keep good records because he/she may be the only person who knows what has been done with the program. 3.1 Create a history file The history file is an important place to document what has happened to a program. It is accessible by anyone who may have a question about a program or someone covering for the PC. It is useful if the PC has a less than perfect memory, and the PC can begin to figure out why certain decisions were made. The types of information kept in the history file are: -­‐Changes made to the program. -­‐Notes on implementation work performed The history files are in /data/jwst/implementation/history/ and have the naming 9999.history. To create a new history file use the History File button in the Analysis menu of the Toolbox. If you click on the button and the history file exists already, you will be given an emacs editing session on the file. If you want to create a history file without the toolbox, you will need to be in /data/jwst/implementation/history/. In this directory, type the following history-­‐file 9999 3.2 Shared Email Folders Exchange Public Folders are used to store all observing program related emails. Each JWST observing program has one folder, and each folder is named with the integer ID number of the corresponding JWST observing program. New folders will be created by ITSD in large chunks well in advance of when they are needed. Each folder has its own email address of the format JWST9999@stsci.edu. Messages sent to these addresses will automatically be filed in the appropriate folders. These folders are visible only to Outlook and Outlook Web Application and not other mail clients. Program coordinators may read, drag and drop, and delete to/from the public folders. Instrument and contact scientists have read-­‐
only access to the public folders. It is very important that all email related to a program is placed in the appropriate folder. For more information, please consult the Observing Program Email Folders Procedures. 3.3 Procheck There are many steps in preparing a program for scheduling. This can become confusing when iterating on a single program or working on several different programs at once. A good way to keep track of where you are with a particular program is to use procheck. This is a button found in the Toolbox. When a field is green, that step has been completed. If a field is red, that steps needs to be completed before a visit can be set to scheduling. To bring up procheck via the command line, type the following in your JWST terminal: procheck 9999 & 4. Help PI develop program submission This step can be relatively easy with some programs. The PI may be an experienced JWST observer and want to do something very straightforward. For many programs, PCs will answer questions or seek answers to questions for the PI. 4.1 Working with CSs Only some programs will have a CS assigned. The CS is an instrument scientist who specializes in the primary instrument used in the program. You can send any instrument specific questions to the CS assigned to the program. If no CS is assigned, a pass through help desk (si_csreview) can be used. It is very important throughout the process for a PC and a CS to keep each other in the loop on correspondence with the PI. A PC may make a recommendation that he/she did not realize impacted the science. Conversely, the CS may make a recommendation that adversely affects scheduling. 4.2 Contact the PI Once a program has been approved by the TAC, an introductory email should be sent by the PC. This email will tell the PI when the complete program is due, and it will encourage the PI to use the program number in all correspondence. The email will explain the role of the PC in the program’s development and implementation. There is a tool called hello-­‐pi that does the following: -­‐ Creates a form letter with the program number and due date. -­‐ Allows the PC to edit the letter. -­‐ Sends the letter to the PI. Information on when and how to run this tool is distributed near the date the PIs are notified of their accepted programs. 4.3 Help answer PI’s questions A PC will receive many types of questions from the PI during the program development process. Some PIs will ask questions about how to achieve certain scientific objectives. If a PC does not know the answer, these questions can generally be answered by the CS (or by si_csreview). It is critical that the PC responds to PI inquiries (by email or phone) promptly. If an answer cannot be found within a day, an email should be sent stating that the PC is working on finding the answer. Some questions will be about the particular instrument the PI is using. The primary source of this information is the relevant Instrument Handbook. If the answer is not obvious, the PC should seek guidance from the CS or email si_csreview. Some questions will be about structuring the program or setting up the observations. The primary source for this is the Program Instructions. If the answer cannot be found, another PC should be consulted. Some PIs will know exactly what they want to do but have trouble running the APT software. PCs should be familiar with APT and point the PI to the necessary training movies provided by the developers. 5. Running APT You many need to run APT yourself on a program to answer a PI’s question. To run APT, type: apt-­‐mac & The APT graphical user interface will come up on your screen. Alternatively, you can select the APT application from the Dock or from the Application folder on your Mac. 6. Receive program from PI At or before the deadline, the PI should formally submit the completed program. The PI does this by using the “Submit” button in APT. APT submission does the following: -­‐ Returns a generic acknowledgment to the submitter saying that the PC for the program will be in touch soon -­‐ Looks up who the PC and (CS) for the program is in the database -­‐ Emails the error summary, and PI comments to the PC & (CS) -­‐ Stores the submission in /data/jwst/implementation/apt-­‐submission -­‐ Updates database setting program status to implementation The submission directory (/data/jwst/implementation/apt-­‐
submission/) contains two files for each submission. 9999-­‐Submission-­‐001.apt.email 9999-­‐Submission-­‐001.aptx The .apt.email file is the acknowledgment sent to the PI. The .aptx file is the actual APT file itself. The second submission would be marked -­‐002 and so on. If the PI has made comments or recommendations about the software, be sure to pass these along to the apt group at apt@stsci.edu, or Karla Peterson (peterson@stsci.edu) for consideration. If the PI includes questions, they should be answered promptly. 7. Check program into PLIB The first thing to do with the new program is put it in PLIB. All you have to do is check in the new version. Normally, in order to check a program into PLIB, you must first check it out of PLIB. The checkout/checkin process is intended to help limit the danger of two people working on the same program simultaneously. Only one person may have the program checked out and only that person can check it back in. In the case you just want to check in the new version of the program without having to first check out the old version, you can use the Checkin button in the PLIB menu of the Toolbox. Since you have not checked out the program, you will be prompted to confirm that you really want to substitute a brand new version of the program and whether you want to check in the version from your working directory or from the submission directory. The history file will be brought up in an emacs session so that you can record that you have checked in a new version. The following directions apply to any change you might need to make to the program once it is in PLIB. 7.1 Check the program out of PLIB In your working directory in a terminal window, type: plib checkout 9999 This will put a copy of the APT file (9999.aptx) in your current directory. Note: it will overwrite an existing file named 9999.aptx in that directory. You then can edit the APT file using APT. Alternatively, you can use the Checkout button in the PLIB menu of the Toolbox. It will checkout the program into your working directory and put you into an APT session on the resulting APT file. Regardless of which method you choose, it is necessary to update the history file to reflect what changes were made and why. Label each set of changes in the history file with your name and the date. The Toolbox will pop up the history file in emacs after you checkout (or checkin in the case of no checkout) a program. If you checkout a program and decide that you no longer want to make any changes, you can release the lock on the program by typing: plib unlock 9999 Alternatively, you can use the Unlock button in the PLIB menu of the Toolbox. 7.2 Check the program back in to PLIB Once you have made the desired changes, in your working directory in a terminal window, type: plib checkin –-­‐cleanup 9999 Alternatively, you can use the Checkin button in the PLIB menu of the Toolbox. For help with PLIB commands, type: plib –-­‐help Once the program has been checked into PLIB, APT will run and pop up a box showing a list of changes found in the program and any diagnostics once it has finished. Checkin does the following: -­‐ Locks the program -­‐ Stores a copy of the program in /data/jwst/implementation/diagnostics -­‐ Runs APT on the program -­‐ Stores APT diagnostic files in /data/jwst/implementation/apt-­‐report/diagnostics -­‐ Populates the database -­‐ Pops up a box which summarizes changes and diagnostics -­‐ Unlocks the program 7.3 Other PLIB Tools There are several other PLIB commands. Here are a few examples. The history command shows you each time a particular program has been checked out, checked in or just read. It can be helpful if you have lost track of when a change was made. plib history 9999 To look at a particular revision of the program, you can use the read command to read out a particular revision of the program (Note: it will be stored in the file 9999.aptx in your current directory). plib read –-­‐revision 3 9999 To review the differences between two versions, you can use the diff command. plib diff –-­‐revision1 3 –-­‐revision2 4 9999 If you want the program to be reprocessed but have no changes to make to the program, the command is: plib reprocess 9999 If you want to display the programs that are currently reserved (checked out) by the user, you can run the following command. plib show Note that all of these commands are available from the PLIB menu in the Toolbox. 8. Diagnostic Waiver If a program has any errors, the diagnostic waiver will need to be run before visit update is run. Here is the command for the diagnostic waiver. diagnostic-­‐waiver 9999 This command will pop-­‐up a window that allows the PC to select which errors and warnings to waive. It is possible to waive all of the errors and warnings. The PC will have to provide an explanation for why the errors and/or warnings are being waived. 9. Update Visits Visit update will calculate scheduling windows, science instrument usage, guide stars, MOSS and constraint and restriction overrides. Here is the command to run visit update in a terminal window. visit –update 9999 –start_time=”2018.275:00:00:00” –
end_time=”2020.275:00:00:00” The start time and end time are not necessary. If you leave them off, it will default to the times listed above. However, this is the syntax that you will need to use if you want to run visit update over a time range different from the default. Visit update can be run on a single visit. A visit ID is made up of the program number (5 digits), the observation number (3 digits), and the visit number (3 digits). An example for program 9999, observation 1, visit 1 is 09999001001. Alternatively, visit update can be run from the Processing menu in the Toolbox. 10. Load program into Spike Spike can be used to review the schedulability of visits. 10.1 Bring up Spike GUI Spike can be started by selecting it from the Analysis menu in the toolbox. Another option is to bring it up by typing the following command. emacs –f start-­‐spike-­‐gui & Currently, Spike must be started on a Linux machine such as Taz. 10.2 Put in the override into the Emacs buffer. Note: This step may no longer be needed. It appears that the default now starts on October 01, 2018. However, this needs to be confirmed. In the emacs buffer, hit return, and type the following at the prompt. (override-planning-start 14392.5) ;; Oct-1-2018
Hit return after you type this command.
10.3 Load the program Click on load. Now, a program can be loaded by clicking on Proposal, then Load Proposal. 10.4 Resolve scheduling problems with a program If a visit cannot be scheduled at any time according to Spike, the PC needs to figure out what is causing the problem. The visit task suitability report can be useful in solving problems. 10.5 Create an observer specified constraint (if needed) An observer specified constraint (OSC) provides a way to constrain where an observation can schedule without modifying the program. They can be created and deleted by using the Spike GUI. OSCs can be defined in two different modes: combine and override. In combining mode, an OSC interval is combined with other absolute constraints for the observations. In override mode, the OSC replaces the existing absolute constraints. OSCs should be created with caution. Make sure the OSC is really doing what you want. To create an OSC, load the program into the Spike GUI. Select the Create/Edit OSC menu option. A pop-­‐up window will appear. Select the visit that you want. An OSC identifier and comments are required. In the Time Windows Intervals field, enter the start and end times. These should be in the format of start date to end date. Here is an example: 2020.100:00:00:00 2020.110:00:00:00 Click on Create for this Visit. The OSC will be saved in the database, and the plan windows will be adjusting accordingly the next time the LRP is updated. 11. Verification Verification will confirm that the program does not perform any unnecessary duplication of observations from any other programs. It will verify that the program does not go over its time allocation, and it confirms that the program only uses its approved targets and observing modes. It will list any mandatory TAC comments as well. To run verification on a program, type the following: verification 9999 For additional information about verification, the following command can be used. verification –-­‐help Once the toolbox has been created, you can use the Verification button in the Processing menu of the Toolbox. 12. Test Scheduling Visits can be test scheduled by using the short term scheduling (STS) GUI. To start the GUI, type the following command: sts –gui & Unlike SGS for HST, the STS GUI is more basic, and everything is tied together. To create a STS, type the following in the command line box at the bottom of the GUI. sts –create 2019.165 2019.175 –baseline=NONE The start and end times can be changed to what you need. However, it is important to include the baseline as none because the default is flight which you do not want. To add a visit, select sts –add_visit from the sts menu at the top of the GUI. If you are using a test visit, you should uncheck overlapping plan windows because your visit will not have plan windows. You can uncheck some of the other boxes as well to narrow down the list of visits. Alternatively, you may find it easier to use the command line box to add a visit. Here is an example. sts –add_visit 09999001001 When a visit has been added to the STS, it will appear in the GUI under DSN (Deep Space Network) Contacts. The visit text will be black. To schedule a visit, select sts –schedule under the sts menu at the top of the GUI. You can select the visit you want to schedule, and you can add a start and end time if you want. If the visit is successfully scheduled, the text will turn red in the GUI. To read more about the VSS scheduling commands, please look at the Command Syntax document found in the Documentation section off of the PC homepage. It is unclear how much test scheduling will be needed for JWST. If the PCs do need to do a lot of test scheduling, the GUI will need to be updated and made more robust. 13. Send galley proof to PI Send an email galley proof to the PI for final review. The galley proof email is a request to the PI to do a final review of the program. The galley proof will include any changes made to the program as well as the plan windows for the visit(s). This is done by using the galley proof button in the toolbox. Another option is to type the following command. galley-­‐proof -­‐-­‐program 9999 Galley proofs can be sent by visits. Type the following to send a galley proof for specific visits. galley-­‐proof –-­‐visits 09999001001 14. Set visits to scheduling To mark a visit to schedule, type: flight-­‐ready program observation VISIT Alternatively, you can use the Flight Ready button in the Processing menu of the Toolbox.