Computer-assisted proposal submission systems
Drew J. Asson
Space Telescope Science Institute
Baltimore, MD 21218
asson@stsci.edu
ABSTRACT
As institutions and observatories are required to handle more tasks with fewer resources, the need to automate
processing becomes crucial. One of the easiest tasks to partially automate is the front-end process of requesting to
use a service, e.g. a telescope. Automation can include making the proposal forms available, and allowing them to
be submitted, electronically. By providing a standard proposal form, the necessary information contained in the
proposal can be extracted and partially processed by software including tracking and low-level error detection. The
need to have people in the loop is still necessary, but many of the mundane, repetitive tasks can be given to
software, while the mentally difficult tasks can be handled by people. This paper discusses tools available for
automating a proposal submission process. It will also discuss a system which was implemented in two months to
partially automate Phase I submissions for using the Hubble Space Telescope. This system was used for the most
current call for proposals and will continue to be used for upcoming cycles.
Keywords: automation, electronic submission systems, electronic mail, Hubble Space Telescope (HST), software.
1. INTRODUCTION
Observatories and institutions are constantly seeking ways to improve services while reducing costs. Mundane,
repetitive tasks often take up a great deal of time but are essential to the activities of the institution. Handling
proposal submissions is one of these tasks. Proposals are often prepared electronically, printed out on paper, mailed
to the institution, and then partially reentered electronically for review. In order to make the submission system
more efficient, the proposer can submit his/her proposal electronically. Almost all institutions, especially in the
astronomical field, have e-mail and Internet connectivity.
Once a proposal has been received electronically, the institution can decide how much automated processing needs to
be done. By responding electronically to the proposer immediately after they submit, s/he knows their proposal was
properly received. This avoids the need to have a person prepare and send out a letter, either via e-mail or regular
mail, stating that their submission was received and will be processed. An electronic backup copy of the
submission can be saved in case the original submission is lost or changed. If a standardized template with predefined accepted values is used, the contents of the proposal can also be analyzed and a report sent to internal staff
and/or the submitter regarding possible errors with their submissions. This allows the staff to concentrate on the
merits of the proposal instead of whether the form was filled out properly. This also allows the ability to determine
if the proposerÕs laundry list was sent instead of a proposal form.
The next section will discuss some of the free tools that are available to put together an electronic submission
system. The subsequent two sections will discuss actual systems that are being used or are being planned. Finally,
future directions of proposal submission systems will be discussed.
2. TOOLS AVAILABLE
There are many tools available for partial automation and analysis of electronically submitted proposals. The ones
that I will mention are all freeware products that are available on a variety of platforms, even though UNIX was
our delivery platform. Please see Appendix A for information on how to obtain these tools.
For construction of a template that a proposer will fill out in order to propose, LaTeX1 is a useful tool. By defining
various styles, a simple keyword-value system can be set up that is easy for the proposer to fill out, as well as easy
for software to parse. Since LaTeX is widely used for scientific journals, scientists are often familiar with it and
thus avoid learning yet another computer system.
For storage of received proposals, there are a variety of methods and systems available. One can store them as flat
files on disk. Care should be taken to avoid problems if a disk becomes corrupt or damaged. If some method of
revision control is desired, there are a variety of tools available. The source code control system2 (SCCS) comes
with many UNIX systems and provides a way of storing changes to submissions in a minimum amount of space. It
also allows an audit trail of how a proposal has developed over time, especially if proposers are allowed to
continually modify their proposals up until a pre-defined deadline. If more control is desired than SCCS, RCS3
(Revision Control System) and CVS4 (Concurrent Versions System) are also available. These are similar to SCCS,
but contain more features.
For forwarding e-mail, the procmail 5 software system is very useful for sending the submission to the correct
processing software and storage area. procmail allows one to parse mail headers and bodies for keywords (full regular
expressions are allowed) and either forward the mail to another address, store the mail in a file, or invoke a program
and provide the mail message as input. This simple tool is at the heart of the ST ScIÕs Phase I submission system.
This tool alone can be used to automate the most manual of processes any group must go through to handle
submissions. For example, instead of having a proposer call up a dedicated staff person to get the proposal
submission material, s/he can just send e-mail to a procmail front-end system. This system, based on the subject of
the e-mail, can send the proposer the submission forms electronically. Also, at submission time, instead of using a
paper mail system, they can submit electronically and procmail can determine where to send the information.
Procmail is not omniscient, but it can handle a variety of situations where a person would normally be required.
For parsing and extracting information from the submission, two scripting languages are very useful. The first is
perl (Practical Extraction and Reporting Language) created by Larry Wall6. The second is Tcl/Tk developed by John
Ousterhout7. Both languages are a good choice. perl is more low-level and appeals to the UNIX-savvy individual.
Tcl/Tk is good for non-programmer types in that it provides a great deal of power without the learning curve often
found with programming languages. Both languages have powerful string manipulation routines which are useful
for parsing and analyzing the text of a submission. perl has a powerful report writer that can be used to generate
analysis and usage reports for staff. Finally, perl and Tcl have both been ported to a variety of platforms, e.g. UNIX
boxes, VMS machines, PCs and Apple Macintoshes.
3. HST PHASE I SYSTEM
3.1 Original system
In order to use the Hubble Space Telescope (HST), proposers must submit a proposal. A Call for Proposals (CP)
occurs nominally once a year. This period is called Phase I. Anyone can propose to use the telescope and proposals
are evaluated on scientific merit by a review panel of experts. The accepted proposal moves into Phase II, where
proposers are required to create detailed observing programs.
Before the introduction of this new proposal submission system, people wishing to propose to use the HST were
required to fill out both electronic and paper forms. The electronic form was a keyword/value template with
duplicate information being requested in various places. Budget forms had to be typed or photocopied from paper
material sent out by the ST ScI. Some forms were submitted electronically, but all forms had to be submitted via
regular mail as well. Proposers often submitted the electronic portion of the same proposal several times since they
were unsure of whether ST ScI had received them. Almost all of these submissions were checked by people, finding
both technical errors and simple syntactic errors.
3.2 Current system
As part of the Project to Re-Engineer Space Telescope Observing (PRESTO), a new Phase I system 8,9,10 was built.
This system took advantage of the fact that nearly all of HST proposers have access to e-mail. In place of the
variety of forms that used to comprise Phase I, two forms were created. One was for the proposal, the other for
budget requests. The system put in place deals only with the former. Budget forms are still only accepted via
regular mail.
A LaTeX template was created with various LaTeX style files to provide one form that a proposer could fill out and
submit to the ST ScI. The best of both worlds was achieved by using LaTeX to make a simple keyword/value form
that could generate a visually pleasing paper output product as well as a computer-readable product. A proposal
represents an individual or institution, so the proposer desires a product that looks professional. Figure 1 shows a
portion of the Phase I template used. Proposers enter the desired values inside the braces.
\target{}
% Target name (see Target Naming Conventions in the Phase I Proposal
% Instructions). NOTE: In order to allow for the target name to fit
% in the printed table, you cannot have more than seven (7) consecutive
% characters without inserting a hyphen. For example,
% GALCLUS-LOCAL-GROUP-GALAXY-NUMBER1
% will print properly, while
% GALCLUS-LOCALGROUP-GALAXY-NUMBER1
% will not.
\ra
{}
% Right ascension coordinates, ex: {05 23}
\dec
{}
% Declination coordinates, ex: {-37 55}
\magnitude {}
% Visual magnitude of the target
\configuration {}
% WFPC2,FOC/96,FOS/RD,FOS/BL,HRS, or FGS
\mode
{}
% Instrument mode (see Table 4 of the Instructions)
\spectralelements{} % Spectral elements (filters,gratings...) Use spaces to separate items in a list.
\totalorbits {}
% Orbits required for this observation
\flags
{}
% Enter PAR,CVZ,TOO,DUP,DT as applicable, where:
%
PAR <==> parallel,
%
CVZ <==> continuous viewing zone,
%
TOO <==> target of opportunity,
%
DUP <==> duplicate, and
%
DT <==> dark time.
Figure 1. LaTeX Template for Phase I
Figure 2 illustrates the data flow in the system for the Phase I period.
User Site
Incoming Mail to
newprop@stsci.edu
Processing Software
Request
Submission
PostScript
submission
Other
P
R
O
C
M
A
I
L
E-mail templates to requester
Analyze, send mail to internal staff,
send acknowledgement to submitter
Store and Respond to
submitter
Store, forward to staff, respond to
submitter that there is a problem
Logfile
Figure 2. Data Flow
3.2.1 Requesting the forms
At the beginning of the CP period, the ST ScI sends out paper material to previous HST users, astronomical
universities, and anyone that has requested to receive the CP. The CP contains the e-mail address at the ST ScI that
will handle the electronic submissions. An astronomer can send a message to this address with a subject of ÒRequest
TemplatesÓ. They will automatically be sent the Phase I template, any necessary LaTeX style files, and a
README file which explains what all the files they have been sent are for. These files arrive at their site anywhere
from 30 seconds to 5 minutes from receipt of the original request message. This time delay is based purely on how
long the e-mail takes to travel through the Internet, as generation of the response and passing it to the mailer takes
only a few seconds.
An added benefit is that if a proposer wants a pristine copy of the submission materials, s/he can send another e-mail
request. Also, the ST ScI can keep track of all the people who submitted, in order to add them to our mailing lists
for future cycles as well as to keep them informed of any information or problems during the CP period.
3.2.2
Submitting
After the proposer has designed the proposal and filled out the form, they can submit it to the ST ScI. They do this
by sending the completed LaTeX template via e-mail to the same address. The idea of one stop shopping (i.e. one email address) was important in the design of this software. There is no special subject line needed for submitting the
proposal. When the procmail handler receives the submission, it identifies it as a submission and then invokes a
perl program to process it. This program (process-submission) stores the e-mail message in two separate locations
for safety and then passes the e-mail file to another program (parse-phase-1-prop) which actually scans the file for the
relevant information. This is described in the analysis section below.
The e-mail message is saved in its entirety, i.e. it includes all the mail header information from the original
message. The parse-phase-1-prop program only checks the actual proposal template. The information is saved so
that a reply message can be generated to the sender (who might not be the same as the principal investigator listed in
the template). It is also saved as part of an accounting system, i.e. ST ScI will know who submitted and when,
exactly.
Starting in cycle 5, the ST ScI started a trial program of full electronic submission, i.e. a proposer would send all
materials to ST ScI electronically. In this new system, they use LaTeX to generate a professional proposal
document and then print it and mail it to ST ScI. In this trial program, instead of printing out the LaTeX output
product (a PostScript file) , the proposer would e-mail the PostScript file. This file is then printed out at ST
ScI to ensure that it is a valid proposal. The method that the proposer uses to submit this PostScript file is to
send e-mail to the same address used above, but with a subject line of Òpostscript submission <proposerÕs last
name>Ó. The software requires the proposerÕs last name since the PostScript file contains only a graphical
representation of the proposal, and thus it cannot extract a name. This name is then used to correctly store the file.
All e-mail is stored, so if the wrong name is used, nothing is lost.
3.2.3
Analysis
The parse-phase-1-prop program, mentioned in the previous section, performs validation and extraction of
information. The validation checks for missing or illegal values for various keywords in the templates. It also
checks relations between keyword/value pairs, e.g. if the proposal is for archival research then budgetary information
is required to be filled out in the proposal. (Archival research means that a proposer would like to receive funds to
do their research using the HST data archive (HST data that has been released into the public domain.)) Percentages
are calculated which reflect the number of European Space Agency (ESA) members on the proposal team and also
the percentages of the proposal that are devoted to the various science instrument on HST (i.e. cameras,
spectrographs, etc.).
This analysis is written in a report and e-mailed to staff members who are in charge of the Phase I submission
process. They review the analysis and determine what, if any, changes are required to the submission. This allows
the staff to easily detect anomalous submissions with very little effort.
Once the validation phase is done, the program extracts the return e-mail address from the file and sends a message to
the person who submitted the proposal. The content of the message depends on the result of the validation. If
everything was successful, the message thanks the submitter and includes the principal investigatorÕs (PI) name, the
title of the proposal, and the number of orbits requested. This provides feedback to the proposer as to what s/he
actually submitted. If the proposal did not contain a PI name or a title, the submitter is notified of this and asked to
resubmit. If the error(s) are more complicated than that, the proposer is told that the software had a problem
interpreting their submission and that a member of the staff will contact them as soon as possible to discuss the
problem.
Information parsed out of the proposal submission is also archived in a special format for future reference. All
pertinent information is stored in this compact format, and thus staff can use these files to write reports and query
information in the proposals without being required to rerun the parse-phase-1-prop program.
3.3 Preparing the observing program
If a proposal is awarded observing time, the proposer must create a detailed observing plan. However, some
information from the Phase I proposal is required on this new form, mainly the abstract and scientific justifications.
In previous cycles, the proposer had to manually retype this information into the new Phase II proposal. Given that
ST ScI has the proposerÕs Phase I proposals and knows the structure of the Phase II template, software was written
to prepare a partially-filled out Phase II proposal for the proposers. These are generated from either the original
submission (e-mail file) or from the archived proposal data. This template is then mailed to the PI, but not as part
of this system.
3.4 Time scale from idea to operational use
One very important item to note here is that this system was put together with free, off-the-shelf components, in
three weeks by two people. One wrote the storage and analysis software in perl, while the other created the LaTeX
template. The system received its first outside (i.e. from a real proposer) template request on 31 May 1994. When
the submission period ended, a new message was inserted at the procmail level to inform people sending mail to this
account that the account was closed for the season. This involved changing the procmail handler, but did not require
a new code installation or reconfiguration.
One of the reasons that the software was developed so quickly was due to the design process. Before any software
was written, the group tried to understand what the problems were with the old system. This included problems that
the ST ScI encountered, but more importantly, problems that our users (i.e. the proposers) encountered. One of
their main complaints had to do with having to enter duplicate information in the Phase I template. Also, for those
awarded observing time, they disliked the fact that they would have to retype information from their Phase I template
into their Phase II program. Also, the ST ScI wanted to keep the amount of information that we requested in the
template to a minimum. This made the template quicker to fill out as well as easier to process. An example of this
was removing several unnecessary free text questions from the proposal form.
What is important to note is that the group looked at the problems in the system and laid out a plan of action before
implementing a new system. Several solutions were debated as to their pros and cons. When the group was
finished this iterative process, a well-understood and stable system was defined. From this specification, the
software was developed quickly.
Another key feature was that the group was made up of various types of people. There were only two computer
scientists and five astronomers. Four of the astronomers had worked closely with Phase I proposers in previous
CPs. The fifth had overall knowledge of many of the systems in place at the ST ScI. He had helped develop the
new operations concept (PRESTO concept) which put a focus on user support and feedback. This diversity allowed
us to consider options that either group on their own might not have come up with.
3.5 Performance statistics
During Cycle 5 Phase I (early July to 12 August 1994), the system handled a total of 1360 e-mail messages. These
included 350 requests for proposal templates, 863 proposal submissions (non-unique), and 48 PostScript
submissions (i.e. full electronic). Only 2 all-paper proposals were received for Cycle 5.
3.6 User community reaction
The HST user community was pleased with the new Phase I system and saw it as a vast improvement over previous
cycles. Three out of the 125 respondents said that all material should be made available electronically, and just the
minimum amount of paper should be produced. This new software will be used in the upcoming cycles.
Modification to the processing that is done internally will be made before the beginning of the next cycle; however,
as far as the proposers are concerned, the system will be the same.
4. OTHER SYSTEMS
4.1 HST phase II submissions
As described above, HST proposers awarded observing time must develop a detailed observing plan. ST ScI provides
these proposers with a system called RPS211 (Remote Proposal Submission system, version 2). Proposers craft the
program with this software, and then submit them electronically to ST ScI. They may submit the program many
times, but ST ScI staff must keep track of each version they submit. Similar activities as described in ¤3 need to
occur in Phase II as well. Examples are: safe storage of proposals, e-mail response to submitter, and notification of
someone internally that a proposal has arrived and must be dealt with.
The software that was described in ¤3 was used as a starting point for the Phase II submission software. A Submit
button was provided on the RPS2 user interface, so proposers did not have to invoke e-mail themselves. Since
software generated the mail message, the process-submission tool did not have to deal with many erroneous mail
messages. Also, the software placed an id in the mail message that was used at the ST ScI to route the incoming
mail to the appropriate person, instead of one generic person as in Phase I.
4.2 ADASS IV conference abstract submissions
The developer of the LaTeX template used in the HST Phase I submission system was also in charge of handling
abstract submissions for the Fourth Astronomical Data Analysis Software and Systems (ADASS IV) conference held
in Baltimore, MD in 1994. Abstracts for ADASS IV were accepted electronically and the software developed above
seemed perfectly applicable to handle storing abstracts and responding to submitters. Only slight changes in the
response messages were required to use the software in this situation.
4.3 Kitt Peak National Observatory (KPNO) system
The ST ScI system used the KPNO electronic submission system 12,13,14 as a starting point. KPNO uses LaTeX
templates which they provide to their proposers via an e-mail interface. Submissions and template requests are done
via e-mail using procmail. They support full electronic submission by accepting the finished LaTeX template and
any encapsulated PostScript files. The major difference between the two systems is that instead of using one email address and looking at the subject to determine what action to take, KPNO set up multiple e-mail aliases. Each
e-mail address was for one activity, e.g. kpnoprop-request was for getting the proposal forms, kpnoprop-help was for
any questions. A disadvantage of this is that it requires the proposer to know multiple e-mail addresses. A positive
outcome, though, is that it simplifies the system since code to distribute mail based on the subject is not needed.
4.4 European Southern Observatory (ESO) system
ESO uses their Proposal Handling and Reporting System (PHRS)15 to handle proposal submissions. This system
also used LaTeX for their template. Their system used Tcl/Tk instead of the perl system used by the HST systems
described above. This was a useful choice since they built a graphical interface in Tcl/Tk to aid people at ESO in
perusing the proposals that had been submitted. These tools were used by staff handling the submission process.
Referees e-mailed in their ratings of the proposal pools and these ratings were automatically stored in the database
that contained the proposals. Once the proposal pool was selected, a final schedule was produced and made available
via the World Wide Web.
5. CONCLUSIONS
This paper presented two main topics: tools available for automating electronic submissions and several operational
proposal submission systems. The systems were used primarily to aid in handling proposal submissions, but the
tools are adaptable to many activities that occur in institutions and companies daily. Tasks such as tracking
problems, filing leave requests, and searching databases that are only available at certain places can be tackled by
using these pieces of software. One of the responses from the survey mentioned in ¤3.6 was very interesting in
light of this paper. The response was that all proposal information should be made available electronically and that
paper should be used as little as possible. Since computers and e-mail are so much a part of many of our daily lives,
processes might need to be updated to take advantage of this. By using the tools mentioned in this paper, it
becomes possible to focus more on the internals of a process, instead of on the mechanics of the process.
The software described in ¤3 is available. Please send a request to asson@stsci.edu if you are interested.
6. ACKNOWLEDGMENTS
I would like to acknowledge Harry Payne who created the LaTeX template and style files for the system described in
¤3. I would also like to acknowledge the rest of the ST ScI team who worked on the new Phase I process: Mark
Johnston, Piero Madau, Ron Downes, Max Mutchler, and Ray Lucas.
APPENDIX A
All the freeware products that were mentioned in ¤2 are available electronically via anonymous ftp (File Transport
Protocol). When using ftp, log in with a user name of anonymous and a password of your complete e-mail address,
e.g. asson@stsci.edu. Once logged in, change directory (ÔcdÕ) to the directory mentioned below. Then type ÔbinÕ to
tell ftp you will be downloading binary data. Then type Ôget <file name>Õ, e.g. Ôget cvs-1.3.tar.gzÕ. This will
download the file to your local machine. When you are finished using ftp, type ÔquitÕ to exit.
Most often, the files that you download will be compressed tar files. A tar file is one file which contains one or
more files, often in a hierarchical format. All the files listed below will be compressed with either UNIXÕs
ÔcompressÕ command or the Free Software Foundations GNU zip (ÔgzipÕ). If the file name ends in ÔZÕ, it is a
UNIX compress file. If it ends in ÔgzÕ, it is a GNU zip file. gzip will decompress both formats, so you can use
just gzip. It is available from the same place as CVS, RCS, and perl.
System
GNU zip
CVS
RCS
perl
Tcl/Tk
procmail
LaTeX
Site
prep.ai.mit.edu
prep.ai.mit.edu
prep.ai.mit.edu
prep.ai.mit.edu
ftp.cis.ufl.edu
ftp.cs.berkeley.edu
Directory
/pub/gnu
/pub/gnu
/pub/gnu
/pub/gnu
/pub/perl/src/4.0
/ucb/tcl
ftp.informatik.rwth-aachen.de
/pub/packages/procmail
(also at comp.sources.misc archives)
ftp.shsu.edu
/tex-archive
ftp.tex.ac.uk
/tex-archive
ftp.dante.de
/tex-archive
File name
gzip-1.2.4.tar.gz
cvs-1.3.tar.gz
rcs-5.6.0.1.tar.gz
perl4-4.036.tar.gz
perl-4.036.tar.gz
tcl7.3.tar.Z
tk3.6.tar.Z
procmail.tar.gz
(see README there)
7. REFERENCES
1. Lamport, Leslie, LaTeX - A Document Preparation System, Addison-Wesley, Reading, 1986.
2. Gilly, Daniel, UNIX in a Nutshell, ¤17, OÕReilly & Associates, Sebastopol, 1992.
3. Tichy, Walter F., ÒRCS - A System for Version Control,Ó Software -- Practice & Experience 7 (July 1985), 637654.
4. Berliner, Brian, ÒParallelizing Software Development,Ó Proceedings of the Winter 1990 USENIX Conference at
Washington, D.C., The USENIX Association, Berkeley, 1990.
5. There is no paper reference for procmail that I could find. The documentation, history, and acknowledgements
for procmail comes with the source code. Please see Appendix A to find out where to obtain procmail.
6. Schwartz, Randal L., Learning Perl, OÕReilly & Associates, Sebastopol, 1993.
7. Ousterhout, John K., Tcl and the Tk Toolkit, Addison-Wesley, Reading, 1994.
8. Asson, Drew J., ÒPhase I Electronic Submission Software,Ó ST ScI/APSB Technical Report Number 1994-03,
27 December 1994.
9. Payne, H.E., and Asson, D.J., ÒElectronic Submission of HST Phase I Observing Proposals,Ó presented at
Astronomical Data Analysis Software and Systems IV, Baltimore, MD 25-28 September 1994 (to be published in
the Proceedings of the Astronomical Society of the Pacific Conference Series).
10. Mutchler, M., Anderson, K., Asson, D., Downes, R., Lucas, R., Madau, P., and Payne, H., ÒObserving with
HST I: A New Phase I Proposal Process,Ó presented at the 184th Meeting of the American Astronomical Society,
Minneapolis, Minnesota, 29 May-2 June 1994.
11. Bose, Ashim, Miller, Glenn, and Gerb, Andrew, ÒAn Interactive Tool to Aid in Proposal Preparation for the
Hubble Space Telescope,Ó to be presented at AeroSense Ô95, Orlando, FL, 17-21 April 1995.
12. Massey, Phil, and Pilachowski, Caty, ÒA New Procedure to Apply for Telescope Time at KPNO,Ó KPNO,
NOAO Newsletter No. 37, 1 March 1994.
13. Massey, Phil, Barnes, Jeannette, and Patterson, Pat, ÒElectronic Submission of Telescope Proposals,Ó KPNO,
NOAO Newsletter No. 38, 1 June 1994.
14. Massey, Phil, ÒElectronic Proposal Submission Continues,Ó KPNO, NOAO Newsletter No. 41, 1 March 1995.
15. Chavan, A.M., and Albrecht, M.A., ÒAn Information System for Proposal Submission and HandlingÓ, presented
at the ADASS IV Conference, Baltimore, MD, 25-28 September 1994 (to be published in the Astronomical Society
of the PacificÕs Conference Series).