Instrument Science Report STIS 1996-20 Implementation of the STIS Exposure Time Calculator on the WWW W. Hack, K. Sahu, E. Kinney, and R. Bohlin October 16, 1996 ABSTRACT A user-friendly WWW interface to calculate exposure times for STIS spectroscopic observations is now available. This report describes the design criteria and implementation of the WWW pages which use the expected STIS sensitivities to estimate exposure times. The Exposure Time Calculator also determines local and global count rates and S/N. The software description includes the interface between the WWW form and the calculation routines and supports debugging of revisions and updates. The support files necessary for maintaining the form and the output products from the calculations are described. The ETC can be accessed via URL http://www.stsci.edu/ftp/instrument_news/ STIS/topstis.html. 1. Introduction Science observations with a completely new HST instrument, such as STIS, will require an observer to learn many new things in order to plan a successful observation. Crucial to this planning is a determination of S/N, which can either be done by hand using material from the STIS Handbook (Baum et al. 1996), by manipulating complex IRAF tasks such as calcphot or simspec within the synphot package, or through a simple WWW interface in a manner similar to that used by FOCSIM. This report describes the WWW form which was created to serve as an easy to use interface to the complex simulation routines necessary for planning STIS spectroscopic observations. 2. Current Simulation Software The calculations used to prepare information for the initial STIS Instrument Handbook relied on IRAF tasks in the synphot package with PGPLOT plotting routines providing the spectral plotting capabilities, and C functions to calculate the S/N of Exposure Time. The synphot task countrate was used to properly normalize the chosen input spectrum or to 1 create a synthetic spectrum, then calculate the count rate that spectrum will produce in the specified STIS observation mode. The full list of input parameters for a typical STIS observation can be seen in the IRAF parameter listing: Table 1. Parameter listing for a typical STIS simulation using the countrate package countrate.instrument = “stis” countrate.detector = “a2d1” countrate.spec_elem = “g140m” countrate.aperture = “1” countrate.cenwave = INDEF countrate.userspec = “bpgsdir$bpgs_1” countrate.synspec = “ “ countrate.synmag = “ “ countrate.reddening = 0. countrate.exptime = 1. countrate.output = “none” countrate.form = “counts” countrate.wavecat = “synphot$data/wavecat.dat” countrate.refwave = INDEF countrate.verbose = yes countrate.flux_tot = 26907932672. countrate.flux_ref = INDEF countrate.refdata = ““ countrate.mode = “a” # EOF This same countrate routine serves as the basis for the countrate calculations performed using the WWW form, although not all of these parameters are directly modified on each simulation. In fact, the parameters which are specified will be the grating, the slit, the spectral shape and flux (whether chosen from a catalog or modeled by synphot itself), and the reddening. The countrate task produces an output table containing the simulated spectrum which is then dumped to an ASCII table using the tdump task. This ASCII table serves as the basis for the rest of the calculations and plotting. Calculation of the S/N for a given observation not only depends on the count rate of the source spectrum as determined using countrate, but also on some knowledge of the noise characteristics for the chosen configuration. Background count rates have been measured from testing of the detectors and from on-orbit observations by other instruments. These measurements were compiled into a set of normalized spectra representing the zodiacal light and bright earth scattered light, as well as a table of dark count rates for each detector. Each set of background sources has a range of values which have been condensed to three sets of options: low, average, and high values. Using these values, the S/N can be calculated using the equations stated in the STIS Instrument Handbook. The new WWW form provides an easier interface for running countrate and for presenting the results. 3. WWW Form Layout Simulating a STIS observation requires a set of input parameters to be selected by the user, some from a defined set of options and others in the form of a specific value. A similar set of options was necessary for running the FOC simulator, FOCSIM. This same basic 2 format was used as the starting point for the STIS WWW form. An introductory paragraph at the top of the form explains in general terms what calculations this form will perform, while links to a help file provide on-line assistance. The form requests the instrumental mode in the first section and then guides the user into providing specific information about the source and exposure. Selection of the particular mode is made at the top of the form, with a heading saying that both a grating and an associated slit must be chosen. Only supported modes which will be available to GOs in Cycle 7 appear in this list, and other modes can easily be added at a later date. The menus preclude the specification of an illegal combination of gratings and slits. Section 2 in the form requires the user decide between calculating a given exposure time to reach a desired S/N or calculating a S/N that will be observed in a chosen exposure time. The user then needs to specify the desired wavelength for the observation. This wavelength is used as the central wavelength setting for which the S/N and count rate will be determined. It will also be used elsewhere in the simulation, especially in determining how much of the spectrum will fall on the detector for the purposes of global count rate determination. A function in the program verifies that this specified wavelength actually falls within the range of the chosen grating/slit combination before running the simulation. In the event that the user inputs a improper wavelength, the program will tell the user what wavelength range the chosen grating/filter combination actually encompasses along with the specified wavelength. Section 3 on the form deals with the target properties. First, its spectrum is selected, where the user can specify either a particular model spectrum, calibration spectrum, have a synthetic spectrum calculated, or provide an arbitrary flux distribution via ftp. Observed spectra from the BPGS catalog and calibration spectra from the CRCALSPEC library are made available. In addition, black body, power-law and flat-spectra can be specified with input boxes for the black-body temperature, or power-law index. Section 4 establishes the normalization and interstellar extinction, and whether the target is a point or diffuse source. The diameter of diffuse sources establishes the filling factor for the specified slit. Associated with the choice of a point source or an extended source is the specification of the target’s flux, with different units for point sources and extended sources. Buttons select whether the target is a point source with a given magnitude, point source with a given flux at a particular wavelength, an extended source with a given magnitude / arcsec2, or an extended source with a flux/arcsec2 at a given wavelength. Finally, the calculation of S/N the background count rates for the observation. Dark count rates for the three detectors depend on the mode selected. Zodiacal light and bright earth scattered light can change from one observation to another. Three choices are provided for both zodiacal light and bright earth light, corresponding to Average, Low and 3 Figure 1: STIS WWW Form showing the 5 major input sections. The title for specifying the user supplied spectrum is the first link that was established to the help file. 4 High expected values, with low earth light being specified as ‘Shadow’. A ‘Submit Simulation’ button runs the simulation. 5 Fig. 2: CCD and MAMA Simulation Results pages for sample cases. STIS Exposure Time Calculator Results S/N and Exposure time at the Specified Wavelength: Exposure time: 1.939 seconds S/N = 10 at 4299.6 Angstroms Remember! S/N is calculated per resolution element and Integrated Counts are calculated per lambda-pixel. CCD Counts/pixel Analysis The brightest pixel would have 49.7 counts at 4336.3 Angstroms. The following plots summarize the results of this calculation: Plot of Integrated counts at each wavelength Plot of S/N at each wavelength Here are the data in tabular form: Table of integrated counts at each wavelength The observational parameters for this calculation were: Grating: g430m Slit: 52x0.1 Specified Wavelength: 4299.9 Angstroms Target was a point source Spectrum: Flat Spectrum Source Flux: v = 10 Reddening of E(B-V) = 0.0 6 STIS Exposure Time Calculator Results S/N and Exposure time at the Specified Wavelength: Exposure time: 0.07006 seconds S/N = 10 at 1400.0 Angstroms Remember! S/N is calculated per resolution element and Integrated Counts are calculated per lambda-pixel. MAMA Count Rate Analysis: This observation would exceed the FUV-MAMA bright object count rate limits! These limits are: 35 counts/sec/pix in any pixel and 200000 counts/sec across the entire detector. This spectrum will have the following count rates: The total counts from the source and background at the specified wavelength: 50.0 Peak local count rate was at 150.9 counts/sec/pix at 1320.0 Angstroms Global count rate: 2.626e+07 counts/sec The following plots summarize the results of this calculation: Plot of Integrated counts at each wavelength Plot of S/N at each wavelength Here are the data in tabular form: Table of integrated count rates at each wavelength The observational parameters for this calculation were: Grating: e140h Slit: 0.2x0.09 Specified Wavelength: 1400.0 Angstroms 7 4. Results Page The design of the output page provides the results, with links to the following: (i) Expected counts from the source and the background in tabular form (ii) A plot of expected counts vs. wavelength, and (ii) A plot of expected S/N vs. wavelength. The output page gives additional information on whether the observation would exceed any of the detector countrate restrictions. An example of this output page appears in Figure 2 for a MAMA simulation and in Figure 3 for a CCD simulation. 5. Software Design Designation of the input parameters and development of the WWW form took care of the initial part of the work necessary for getting a working WWW interface. A program was then custom-designed to act as an intermediary between the form and the simulation software and to serve as an output channel for the results. The interface between the FOCSIM WWW form and the IRAF task FOCSIM served as the basic model for the STIS WWW simulator. The interface between a WWW form and any other program relies on the Common Gateway Interface (CGI) standard and software distributed with the HTML servers. Specific C functions are provided with the HTTP server for use in CGI programs to parse out the parameters passed from the WWW form to the main program. The availability of these parsing routines in C and the ability to perform additional calculations based on the throughput data from CDBS within the interface program makes C the language of choice for this interface. Furthermore, C routines for calculating S/N Exposure Times and BOP limits can be readily developed and maintained as there are those with programming experience with C in the group. WWW Form Interface Routines provided with the HTTPD software have been designed to break down the parameters from any WWW form into a single C structure for manipulation according to the design of the calling program. Input fields or menus on an WWW form are assigned variable names and each user’s choice becomes the value for that field. The choices that a user sees on the form can either be the exact value passed to the CGI program, or a separate value can be assigned. For example, the available slits for each grating are labelled by the width and height in arcseconds, such as 52 x 0.1. However, the value from this selection will be passed on to IRAF, so it is translated by the form as ‘s52x01’ as shown in the HTML: <option selected value="s52x01">52x0.1 8 The value given by the form is then reported verbatim in the output form (as seen in Figure 3). On the other hand, the selection of the spectral shape for the source is passed in the same manner as selected on the form, e.g. A0V for the model spectrum selected in Figure 3. The CGI program must then determine what file corresponds to an A0V spectra, a relation which is provided by reading in the file ‘cat_modelspec.dat’ for the model spectra and ‘cat_stdspec.dat’. A similar catalog is set up for the slits so that the same description for the slit is reported both in the input form and output page. Once all the selections are made on the form, each variable has a value; and all the name/value pairs are strung together by the HTTP software and sent to the CGI program. The functions used to parse these name/value pairs are defined in ‘util.c’, which was copied from the cgi-src directory of the HTTPD distribution for use in our own CGI program. The calls to these functions were then copied directly from the FOCSIM form CGI software as follows: /* Determine the number of parameters from the form */ cl = atoi(getenv("CONTENT_LENGTH")); /* Parse the name-value parameter pairs from the form since they **get passed as 'name=value&name=value&...' from the form. */ for(x=0;cl && (!feof(stdin));x++) { m=x; entries[x].val = fmakeword(stdin,'&',&cl); plustospace(entries[x].val); unescape_url(entries[x].val); entries[x].name = makeword(entries[x].val,'=');} As the comments in the code suggest, this loop parses the name/value pairs from the form and fills an array named ‘entries’, which was defined to have a string ‘.name’ and a string ‘.val’ for each name/value pair. For example, the first selection from the STIS form would be ‘grating = g430m’ for a G430M simulation and they would be parsed as: entries[0].name = ‘grating’ entries[0].val = ‘g430m’ All the selections from the form would then get converted into this one array which would be available as a global variable for use by any other part of the program that needs the information. Interpreting the Form Parameters Performing the desired calculation requires interpreting the information in the entries array as parsed from the form. Several structures were created to serve as holding areas for the information needed in the calculations: obspar, RANGE, and MODEPAR. These variable types (definitions are in the Appendix) provide a place to hold the information from the form, as well as the parameters read in from other sources. This supplementary infor- 9 mation comes from files containing information about the gratings, slits, and the choice of spectra. The files are: cat_modelspec.dat - name of file for each model spectra available cat_stdspec.dat - name of file for each standard spectra available cdfile.dat - column definition for use in converting ASCII tables to SDAS tables gratings.dat - grating wavelength ranges and list of central wavelengths stis_modes.dat - plate scale, read noise,... for each mode stis_slits.dat - height and width for each slit wavecat.dat - filename catalog of wavelength tables for every mode Some functions were then written to determine what grating and slit was selected (get_usr_grating) and to check the desired wavelength against the grating to insure it was in range (read_grating). Checks were then made to determine whether the source object is a point source or an extended source, and whether to calculate the S/N or exposure time. Getting the User Supplied Spectra For the first time in a form at STScI, this form provides the capability for the user to supply his own spectra for use in the calculation. Normally, forms only allow a user to fill in the blanks provided, check the boxes presented, or make a selection from a list. Since the user cannot input a long list of input data directly, our program accesses a particular anonymous directory where the user has placed the spectrum of interest. The first piece of information needed by the program is the name of the file. A field on the form allows the user to supply the name of the file that was uploaded to the FTP directory. The FTP directory selected is the one on ftp.stsci.edu set aside just for outside access: outside-access/in.coming. With the name of the file in place, a script is called by the CGI program that performs the actual transfer: ftp-auto-get. /* They have either a SDAS .tab file, or a FITS (.fit or .fits) file */ /* Create the new file name to be referenced in datadir directory */ /* Rename user file for uniqueness, keeping original extension... */ sprintf(spec,"%s/%s%d.%s",datadir, specname, (int)getpid(), spec ext); /* Now move it from the anonymous ftp directory to datadir directory */ sprintf(ftpcmd,"ftp-auto-get %s %s", entries[nform].val, spec); system(ftpcmd); This is a script using the Expect routine to allow the use of interactive programs using non-interactive scripts. The Expect script is a very simple set of instructions (given in Appendix B) to establish the FTP connection to ftp.stsci.edu, cd to outside-access/in.coming, set the transfer type to binary, then transfer the specified file to the datadir directory as a temporary file. This method relies on having the CGI script, and, therefore, the HTTP server itself, running as a user since this routine relies on spawning a shell to actually run the script. Once set up properly, this method allows the use of not only ASCII files for input, but also any file format, since the transfer uses binary mode, allowing SDAS and FITS tables as well. With the user supplied spectra in place, the calculation can begin. 10 Calling the IRAF Tasks The IRAF routine countrate provides the crucial step of calculating the count rate for the desired source spectrum with the chosen mode at the given wavelength. This involves not only renormalizing the spectra to the chosen magnitude or flux in ergs, but also resampling that spectrum to the wavelength grid for the mode and then multiplying it by the combined throughput of the mode. The latest version of the throughput file is automatically selected using functions provided in ‘cdbsfile.c’, thereby insuring that only the most recent (and presumably most accurate) wavelength range is used in the calculation. The method used for running countrate is to pass a parameter file created by the program based on the form parameters in the entries array, and having the results written out to a directory. In our case, some form values are written as a parameter file and passed to the IRAF tasks in a fashion similar to that used by the FOCSIM CGI program, focpar. The entries array is searched for information on the source spectrum and flux, which is written in the proper format to a temporary file (parfile) using simple fprintf commands. The IRAF task is then started from within the CGI program as if it were being run from the UNIX command line outside of IRAF using the following command: sprintf(command, "/usr/stsci/stsdasx/bin/x_synphot.e countrate @%s > %s", parfile,srepfile); system(command); A separate file listing the IRAF environment variables already exists and is automatically read in when the IRAF routine is started, allowing it to run as if it were within the IRAF environment. The standard reporting done by countrate is also sent to a temporary file (srepfile) where the total count rate can be used for the global count rate check for MAMA observations. The output table of wavelength vs. count rate is then converted to an ASCII table using tdump, after writing out a parameter file in the same manner as used for countrate. However, the wavelength range for tdump is based on the wavelength range for the mode as read (using tselect) from the throughput file in CDBS itself. This provided an absolute wavelength range for both the spectrum and the background count rate tables regardless of whether there is detectable flux there or not. In the end, two ASCII tables exist in the temp directory datadir (set in ‘global.h’), one for the source spectrum and one for the background. The rest of the calculations proceed entirely within the CGI program. S/N and Exposure Time Calculations The primary purpose of this form is to calculate either the S/N that can be reached at a given wavelength in a specified exposure time or the exposure time necessary to reach a given S/N at a specified wavelength. The choice made by the user was placed in the globally defined structure params as params.sn and params.x, where params.x is either the desired S/N or exposure time and sn is a boolean (Yes to calculate S/N, No to calculate 11 exposure time). The following excerpt from the main program illustrates how these parameters are used to calculate the proper values: /* Calculates the final S/N or Exp. Time Here*/ if(params.sn == Yes && params.extended == No) { sn = calc_sn_pt(source, background, q_factor, mode.dark, mode.read, params.nread, params.x); t = params.x; } else if (params.sn == Yes && params.extended == Yes) { sn = calc_sn_ex(source, background, q_factor, mode.dark, mode.read, params.nread, params.x); t = params.x; } else if (params.sn == No && params.extended == No) { time = calc_time_pt(source, background, q_factor, mode.dark, mode.read, params.nread, params.x); /* Determine which array element has the user specified wavelength */ lidx = find_lambda(time.lambda,time.size, atof(entries[11].val)); t = time.counts[lidx]; sn = calc_sn_pt(source, background, q_factor, mode.dark, mode.read, params.nread, t); } else { time = calc_time_ex(source, background, q_factor, mode.dark, mode.read, params.nread, params.x); /* Determine which array element has the user specified wavelength */ lidx = find_lambda(time.lambda,time.size, atof(entries[11].val)); t=time.counts[lidx]; sn=calc_sn_ex(source, background, q_factor, mode.dark, mode.read, params.nread, t); } When a user wants to calculate the exposure time for a given S/N, the exposure time is calculated for each wavelength in the observation, then the exposure time at the given wavelength is pulled out. This is then used to calculate the S/N at that exposure time for all the wavelengths in the observation. This way every computation ends in knowing the S/N at all wavelengths for the exposure. With the S/N and exposure time calculations finished, the integrated counts are then computed based on the count rate spectrum of the source and background for the calculated (or given) exposure time. This array of observed counts per pixel is written to a file to be accessed from the output page and also plotted as a GIF image using PGPLOT functions. Similarly, the array of S/N at each wavelength is plotted using PGPLOT functions, again being written as a GIF image. These images can then be included as on-line images in the output page. The functions for the S/N, exposure time, and integrated count calculations (including calc_cts_pt and calc_cts_ex) are all written by Ellyne Kinney and Kailash Sahu and are defined in ‘stis_snsim.c’. They are all based on the same algorithms that were used to create the results for the STIS Instrument Handbook. The plotting functions are located in the file ‘stis_plotsim.c’. 12 Final Output Once all the calculations are done and arrays written to file and plots created, the program prints its own HTML to create the output results page complete with the embedded plots of the S/N and integrated counts at each wavelength in the observable portion of the spectrum and a link to the ASCII table of integrated counts vs. wavelength. The output routine (output_page in ‘stis_specsim.c’) provides one final check: bright-object limitations for the mode. Both CCD and MAMA modes are treated separately in the specific output that is produced to reflect the differing count or count-rate limits for bright objects. This can be seen in Figure 2 for a MAMA simulation and Figure 3 for a CCD simulation. The primary results (the S/N and exposure time and link to the output table of integrated counts) are listed at the top of the page where the user can quickly see them, while the simulation parameters are given just above the plots. Regardless of whether any bright object messages appear, all these results fit neatly on a single page that can be easily saved or printed for later reference. 6. Development Environment This development of this software was carried out using the HTTPD server running on hector with severely restricted access from other machines. Any HTML browser could view the forms, but only when the form is viewed from an HTTP server with a full http address could the form be run. The WWW pages are now accessed using the URL: http://rue.stsci.edu with the exposure calculator development taking place in the exp_forms directory. The ETC can now be accessed via URL http://www.stsci.edu/ftp/instrument_news/STIS/topstis.html The forms must be developed and debugged on a HTTP server machine. The server hector was used to develop forms for other groups, then those forms were migrated to the public pages on rue with little trouble. This software was migrated to the Ultra-machine ‘rue’ which supports the public access of the form. In order to make this move, a ‘datadir’ directory has been defined on the server for use in storing the temporary parameter files as well as the final ASCII table and GIF plots of the results. In addition, a cgi-bin/stis directory stores all the support files necessary for this program, including a version of ‘zzsetenv.def’ with all the necessary IRAF logicals defined. The source code was then compiled in that directory and referenced from the WWW form by the command : acc -o stis_specsim stis_specsim.c stis_debug.c stis_snsim.c stis_plotsim.c util.c cdbsfile.c -lF77 $PGLIBS chmod 775 stis_specsim 13 In order to compile the debug version of the code, simply use the following command syntax: acc -o stis_spectest stis_specsim.c stis_debug.c stis_snsim.c stis_plotsim.c util.c cdbsfile.c -DDEBUG_TASK -lF77 $PGLIBS The DEBUG_TASK flag is used at compile time to set a switch in the program for using the functions in ‘stis_debug.c’ to create a program that can run without the WWW form as input. It provides the capability for reading in the file ‘entries.dat’. This file needs to be created using a current version of the WWW form that accesses the program /cgibin/post-query to report the name and value pairs for each entry on the form. The resultant output page should then be saved as a ‘Plain Text’ file using the browsers menus as the file ‘entries.dat’. 7. Final Overview This report describes the tasks used to interpret the parameters sent by the WWW form, to determine the instrument mode based on the users choices, and then to calculate the S/N, exposure time and integrated counts for the desired observation. The bulk of the effort has been spent in writing and debugging the CGI program, even using the FOCSIM program as a starting place. Several techniques have been developed to debug the program, resulting in a form that can be compiled to run with or without the WWW form. In the end, a set of C programs were written to perform the following tasks in order: • parse the parameters from the WWW form itself, • write those parameters to an IRAF compatible parameter file, • start the IRAF tasks with the parameter file, • to compute the S/N from the IRAF task output, • write the results as an HTML page with links to output plots and tables. There is an HTML file for the WWW form itself, a set of C routines for running the simulation, and support files necessary for performing the simulation. REFERENCES: Baum S. et al. 1996, STIS Instrument Handbook, version 1.0 (Baltimore: STScI) 14 Appendix A. Structure Definitions The following lines from ‘global.h’ provide the definitions for the structures used throughout the CGI program. typedef struct { char *name; char *val; } wwwpar; wwwpar entries[NUM_FORM_ENTRIES]; typedef struct { char *grating; char *slit; int usrwave; char usrtype[1]; Bool extended; float size; float tcounts; float tback; float wcounts; float wback; int nread; Bool sn; int x; } obspar; obspar params; /* Global structure containing observation parameters */ typedef struct { char *name; float low; float hi; int numi; float *iwave; int numc; float *cwave; } RANGE; typedef struct { float min; float max; } BAND; /* Used to keep track of min and max wavelength for mode */ typedef struct {char *name; float dark; int read; float platescale; float p_light; int d_PSF; Bool MAMA; Bool backup;} MODEPAR; typedef struct {char *name; float width; float height;} SLITPAR; typedef struct {int size; double *lambda; double *counts;} SPECTRUM; 15 Appendix B: ftp-auto-get script The script used by the CGI program to perform the automatic FTP of a user supplied spectrum is provided here for reference. It is based on the Expect package for running interactive programs from scripts. #!/usr/local/bin/expect -f # # Usage: ftp-auto-get filename output # where # filename - name of single file to ‘get’ # output - name with full path for output file # # This script lets a user automatically use ftp to retrieve a file # from ‘ftp.stsci.edu:/outside-access/in.coming’ directory and # copy it into a directory and filename of their choosing. # log_user 0 set timeout -1 #set host [lindex $argv 0] set file [lindex $argv 0] set outfile [lindex $argv 1] while {1} { # spawn ftp $host spawn ftp ftp.stsci.edu expect “Name” send “anonymous\r” expect “Password:” send “nobody@stsci.edu\r” expect “Login failed.” {send “quit\r”;close;wait } \ “ftp> “ break } send “cd outside-access/in.coming\r” expect “ftp> “ send “bin\r” expect “ftp> “ send “get $file $outfile\r” expect “ftp> “ send “quit\r” 16