Instrument Science Report ACS-98-04 Performance of the onboard compression algorithm for ACS F.R. Boffi & M. Stiavelli January 28, 1999 ABSTRACT An extensive set of experiments was performed to test the performance of the onboard compression algorithm created for ACS by R. White. We determined the compression factor for realistic simulated images of different astronomical targets as well as of STIS images and ACS darks and flat fields. We present the results of such experiments and our recommendations for the compression settings of primary and parallel observations. We also estimated the time required to compress ACS images on-board, but concluded that these estimates are at present not reliable and therefore suggest an alternative strategy. Finally, for comparison purposes, we also list the compression factors achieved with the UNIX commands gzip and compress. 1. Introduction Compression is one way to deal with many of the problems the 34 Mbyte size of the ACS WFC images raises. A more complete discussion on the subject can be found in the ISR ACS-97-02 by Stiavelli and White. In the present document we present the results of compression tests performed on simulated images ad hoc created (see TIR ACS-98-01 by Boffi and Stiavelli), on real STIS images, and on ACS and STIS instrument flat fields and on ACS darks. The simulated fields represent two clusters of galaxies and two nearby field galaxies. The STIS images used by us are parallel observations of relatively empty fields. These fields may be representative of typical ACS WFC targets and may allow us to accurately test the performance of the compression algorithm on a variety of environments and cases, by varying gain setting, number of cosmic ray hits (i.e. exposure times), and background noise. 1 In addition to the compression factor we also estimate the time required to compress the various images. The compression time depends on the noise level of the image being compressed and on the required compression factor. We timed the compression on a Sparc 4 CPU and on an Intel 483SX/33Mhz CPU and then used some time results from the ACS hardware to convert these measurements into estimates for the ACS on-board compression. 2. The White-Pair algorithm The White-Pair algorithm adopted for compression of ACS WFC images is lossless, i.e. completely reversible and causing no data loss unless there is an overflow of the internal buffer. The buffer is sized depending on the desired compression factor which consequently has to be chosen carefully. It is also worth mentioning that the algorithm compresses data input blocks of 2048 pixels, always produces compressed output data of the specified fixed block size and handles block overflow the same way as the onboard program. On decompression pixels that were dropped due to block overflow are set to zero. 3. Tests of compression Tests were performed on simulated as well as on real images. Simulated images were created for this purpose by Boffi and Stiavelli (TIR ACS-98-01). We focussed mostly on cases where we expect compression to be less effective, since our main purpose is to derive robust minimum compression factors. The different fields (see Introduction) were simulated with a sky background of 500 counts suitable to represent typical long broadband exposures and for different gain settings (1,2,4 and 8). Compression tests were performed on images with a number of cosmic ray hits equivalent to approx. 10% of the entire number of pixels in the WFC 4144x2068 (physical overscan included) frame and on the same images with a third of such cosmic hits (to possibly cover a variety of exposure times). These should represent a one orbit and a typical exposure respectively. The real images, instead, are STIS short exposure parallel observations of reasonably empty fields and were actually summed up together to simulate a longer exposure time. They were all for gain setting equal to one and for a 300 sec. exposure time. The simulated exposure time is 2100 sec. Because of the multiple read-outs this suitable image has somewhat higher read out noise than an equivalent single STIS exposure. However, the resulting noise level per pixel is not higher than that expected for a broad band ACS WFC exposure with medium-bright background. ACS flat fields and two DARKs, and ten STIS flat fields with different gains were also compressed. All results are summarized in Table 1. The first column indicates the image, columns 2 through 5 the various gain settings. For each case the maximum compression 2 factor is listed. In the Table, Ball/FF and Ball/Dark refer to two ACS images that were used for compression experiments at Ball. From the results shown in Table 1 we can see that a compression factor of 2 or more is achieved in the case of simulated images for gain settings of 4 and 8. The images representing the two clusters of galaxies appear to compress more than the two galaxy images. In fact, the former compress up to a factor of 2.3-2.5 while the latter at the most reach a compression factor of roughly 2 for the highest gain setting. The STIS image, sum of seven 300 sec. exposure empty frames, compresses to a factor of 1.6 at gain=1. Flat fields of both ACS and STIS instead do not compress well. In order to try and understand these results and to generalize our understanding of the compression algorithm we estimated the noise levels of all frames. The noise corresponds to a robust estimate of the dispersion of intensity levels in the image (specifically half the percentile width corresponding to a ±1σ for a Gaussian distribution). Compression factors are plotted against noise levels for both sets of ACS simulated images and for the ACS flat fields, in Figure 1. 3 Table 1: Measured compression factors Image gain=1 gain=2 gain=4 gain=8 Clusters/long 1.307 1.702 2.119 2.267 Clusters/1/3 CR 1.329 1.785 2.272 2.448 Galaxies/long 1.071 1.371 1.691 2.060 Galaxies/1/3 CR 1.074 1.407 1.753 2.138 ACS/f.f.#1 0.980 1.025 1.176 1.524 ACS/f.f.#2 0.982 ACS/DARK 1.776 2.168 2.764 3.193 Ball/f.f. 1.03 Ball/DARK 2.147 STIS/ f.f.#1 0.981 STIS/ f.f.#2 0.981 STIS/f.f.33 0.983 STIS/f.f.#4 0.984 STIS/f.f.#5 1.141 STIS/f.f.#6 0.986 STIS/f.f.#7 0.985 STIS/f.f.#8 0.985 STIS/f.f.#9 1.000 STIS/f.f.#10 1.000 STIS/limage 1.618 4 Figure 1: Correlation between compression factor and noise level. In Fig. 1 filled circles represent simulated cluster images, open circles the two galaxy images, and the open squares identify ACS flat fields. The higher the noise the less the compression factor. Such a behavior was already anticipated by Stiavelli and White (1997). It is interesting to notice that each set of data falls within a relatively narrow band in the 2D plane. The flat fields have the lowest compression factors of the three sets. 4. Timing of the compression tests Timing tests were performed on all images discussed in Section 3. The results for a compression factor of 2, i.e. a blocksize of 2048 bytes, are presented in Table 2, which is formatted similarly to Table 1. The first column indicates the image, columns 2 through 5 the various gain settings. For each case we list the time it took to compress each frame on a Sparc4 workstation as measured with the UNIX time command (in units of seconds). A Sparc 4 was used since faster workstations complete the compression too rapidly for an accurate timing estimate. In order to estimate the on-board compression time we plan to establish the relation between our timing measurements and those carried out on the ACS hardware on the two ACS reference images (Ball/FF and Ball/Dark). To verify the reliability of our approach we also timed the compression of STIS images and the two ACS reference images on an Intel 386SX/33Mhz CPU, which has an architecture more similar to the Intel 386/16Mhz used in ACS. In these latter experiments we tested a number of different compression factors since we expect a dependence of the compression time on the 5 commanded block size in addition to the image noise level. In Table 3 we list the results for the two reference images while in Table 4 we list the compression results for the STIS images and various blocksizes. For the reference image timing measurements on the ACS hardware we will focus on the 4 amplifier read-out mode since, as we have checked, the results from other readout schemes can be accurately obtained by rescaling the amount of data that are compressed. Table 2. Compression timings on a Sparc 4 CPU (blocksize=2048). Clusters/long 4.13 s. 3.80 s. 3.66 s. 3.46 s. Clusters/1/3 CR 4.24 s. 3.64 s. 3.67 s. 3.45 s. Galaxies/long 4.27 s. 4.09 s. 3.81 s. 3.40 s. Galaxies/1/3 CR 3.84 s. 3.88 s. 3.70 s. 3.37 s. ACS/f.f.#1 4.34 s. 4.08 s. 4.34 s. 4.93 s. ACS/f.f.#2 4.07 s. ACS/DARK 3.75 s. 3.57 s. 3.26 s. 3.47 s. STIS/f.f.#1 0.55 s. STIS/f.f.#2 0.55 s. STIS/f.f.#3 0.63 s. STIS/f.f.#4 0.58 s. STIS/f.f.#5 0.54 s. STIS/f.f.#6 0.61 s. STIS/f.f.#7 0.52 s. STIS/f.f.#8 0.57 s. STIS/f.f.#9 0.48 s. STIS/f.f.#10 0.55 s. STIS/image 0.51 s. Ball/ f.f. 4.37 s. Ball/DARK 3.7 s. 6 Table 3. Compression timing results for ACS images (in seconds) Name blksize ACS Sparc4 486/33 Ball/FF 1156 65.43 3.84 ± 0.1 18.3 ± 0.2 Ball/FF 2048 95.49 3.94 ± 0.1 18.6 ± 0.4 Ball/FF 4094 164.1 4.36 ± 0.16 21.9 ± 1.4 Ball/Dark 1156 126.8 3.58 ± 0.10 21.6 ± 0.5 Ball/Dark 2048 186.8 3.67 ± 0.10 21.9 ± 0.4 Ball/Dark 4094 186.8 4.06 ± 0.16 22.9 ± 0.2 Table 4. Compression timing results for STIS images (in seconds) Name blksize Sparc4 486/33 STIS/f.f.#1 2048 0.53 ± 0.02 3.13 ± 0.04 STIS/f.f.#1 2560 0.56 ± 0.04 3.13 ± 0.09 STIS/f.f.#1 2730 0.54 ± 0.07 2.97 ± 0.19 STIS/f.f.#1 3724 0.50 ± 0.03 3.40 ± 0.04 STIS/f.f.#5 2048 0.57 ± 0.04 3.44 ± 0.57 STIS/f.f.#5 2560 0.55 ± 0.04 3.52 ± 0.09 STIS/f.f.#5 2730 0.58 ± 0.02 3.29 ± 0.26 STIS/f.f.#5 3724 0.58 ± 0.05 3.76 ± 0.10 STIS/f.f.#7 2048 0.55 ± 0.06 3.03 ± 0.15 STIS/f.f.#7 2560 0.56 ± 0.03 3.14 ± 0.07 STIS/f.f.#7 2730 0.50 ± 0.04 2.78 ± 0.16 STIS/f.f.#7 3724 0.57 ± 0.05 3.16 ± 0.04 STIS/Image 2048 0.55 ± 0.03 3.48 ± 0.11 STIS/Image 2560 0.56 ± 0.04 3.40 ± 0.20 STIS/Image 2730 0.59 ± 0.03 3.25 ± 0.22 STIS/Image 3724 0.60 ± 0.03 3.74 ± 0.23 In Figure 2 we verify how well the Sparc 4 and the 486/33 timings correlate with those measured with the ACS hardware (triangles, squares, and stars indicate blocksizes of 1156, 2048, 4094, respectively). While the Sparc 4 timings do not correlate well with the 7 ACS hardware, we find that the 486/33 timings provide indeed a very good correlation, probably due to the similarity in instruction set between the two CPUs. The best fit relation connecting the 486/33 timing to the ACS is the following: T ( ACS ) = ( 28.0 ± 5.5 ) × T ( 486 ) – ( 447.5 ± 28.3 ) Carrying out timing experiments on the 486/33 is very cumbersome, thus, as a last resort, we attempted to see whether a reasonably good correlation exists between the 486/ 33 timings and the Sparc 4. Since Figure 2 shows a good correlation between ACS and 486/33 timings and no (good) correlation between ACS and Sparc 4, we do not expect 486/33 and Sparc 4 timings to correlate well. We can actually identify a weak correlation when more images are used. In Figure 3 we show this correlation as revealed when 16 STIS images are also considered. The STIS timings have been multiplied by 7.668 to account for the difference in size between ACS images with overscan and STIS images. The relation shows a scatter of about 10 per cent but is statistically significant: T ( 486 ) = ( 9.9 ± 1.9 ) × T ( Sparc4 ) – ( 17.4 ± 2.0 ) By combining these two relations we are able to derive an estimate of the time it would take to compress with ACS any of the images considered in Section 3. Our results on the worst case compression timings are summarized in the following table: Table 5. Extrapolated worst case compression times (in seconds) Image Type Gain=1 Gain=2 Gain=4 Gain=8 Science 250 200 125 100 FF 280 400 360 240 Dark/Bias 105 55 40 30 The overall uncertainty of these results is hard to assess. There is at least a 20 per cent uncertainty due to the two steps involved. However, much larger uncertainties are proba- 8 bly present when we estimate the compression time for images very different from those used to derive the transformation equations. The large degree of uncertainty of our estimates is shown by the fact that an actual GAIN=1 bias compresses on ACS in 186 s while our estimate based on Sparc4 data would give us 105 s. It is our conclusion that a better estimate of ACS compression timings for a scientifically relevant range of images cannot be obtained with these methods. Therefore, we suggest that: • a set of compression experiments be carried out on the ACS hardware or on a bench. These tests should involve either a number of biases and flat field exposures compressed with various blocksizes and/or the compression of simulated images provided by us; • the exact version of the compression code implemented by Ball in the flight software be run on a few representative images in order to better understand the timing discrepancies, and in particular the fact that even the good 486/33 - ACS timing correlation does not appear to be a simple scaling with the clock speed. Figure 2: Correlation between different compression timing experiments. 9 Figure 3: Correlation between Sparc4 and 486 timing experiments 5. Conclusions and recommendations The compression experiments here described allow us to recommend a compression strategy for ACS. We verified that images with a significant amount of noise compress very little. Typical astronomical images will compress more and biases and darks compress best. Our recommended compression factors are provided in Table 5. It is worth noticing that allowing even a small amount of data loss does not improve significantly the compression factors attainable.The timing experiments have provided less clearcut results. However, both the extrapolated compression times from Sparc 4 experiments and the measurements actually carried out at Ball indicate that the compression time should remain below 200 s for all images for which compression is recommended. Only on-orbit experience is likely to allow us to refine these estimates. 10 Table 6: Recommended compression factors WFC observation gain=1 gain=2 gain=4 gain=8 Primary 1.0 1.0 1.0 1.6 Parallel 1.0 1.0 1.5 1.8 Flat Field 1.0 1.0 1.0 1.0 Dark & Bias 1.5 1.9 2.5 3.0 6. References Boffi, F.R., Stiavelli M., 1998, TIR ACS-98-01 Stiavelli, M., White R.L., 1997, ISR ACS-97-02 7. APPENDIX: Tests of compression using gzip and compress In order to provide some term of comparison and to present a complete range of results we also compressed all frames by using standard compression algorithms available in unix, like gzip and compress. We have performed these experiments on all images listed in Table 1. In Table 3 we present the results of these tests: in each cell the compression factors for gzip and compress are respectively given. The integer fits files have the following dimensions: the ACS simulated images and instrument flat fields 17 Mbytes, the STIS images and flat fields 2.2 Mbytes. Gzip and compress compress the images with the two galaxies of some factor comprised in the range 1.2 - 2.4, the higher values being achieved for higher gain settings. The images of two galaxy clusters are compressed of a factor between 1.7 and 4. The two clusters fields appear to compress slightly better than the two galaxies with both compression algorithms, the two galaxies instead are better compressed with gzip than compress. The STIS flat fields are appear to compress slightly less than the ACS. The former compress between 1.2 and 1.5 at the most, the latter between 1.3 and 1.8. Finally the STIS image (sum of seven 300 sec. exposure images) compress of about 1.8 with both algorithms which well agrees with the compression factors obtained for the ACS simulated images. 11 Table 7: Compression factors with gzip and compress Frame gain=1 gain=2 gain=4 gain=8 Clusters/long 1.701; 1.631 1.975; 2.015 2.540; 2.705 3.613; 3.988 Clusters/1/3 CR 1.737; 1.669 2.002; 2.051 2.581; 2.745 3.642; 4.026 Galaxies/long 1.391; 1.186 1.566; 1.430 1.846; 1.786 2.366; 2.383 Galaxies/1/3CR 1.408; 1.198 1.592; 1.456 1.870; 1.815 2.397; 2.419 ACS/f.f.#1 1.296; 1.098 1.409; 1.177 1.542; 1.349 1.759; 1.675 ACS/f.f.#2 1.296; 1.097 ACS/DARK 2.310; 2.398 2.838; 3.042 3.670; 4.062 4.986; 5.887 Ball/f.f. 1.200; 1.031 Ball/DARK 2.451; 2.571 STIS/f.f.#1 1.218; 1.016 STIS/f.f.#2 1.217; 1.013 STIS/f.f.#3 1.228; 1.024 STIS/f.f.#4 1.227; 1.022 STIS/f.f.#5 1.497; 1.328 STIS/f.f.#6 1.262; 1.059 STIS/f.f.#7 1.257; 1.054 STIS/f.f.#8 1.261; 1.057 STIS/f.f.#9 1.418; 1.223 STIS/f.f.#10 1.402; 1.205 STIS/image 1.871; 1.820 Acknowledgements We thank R. White for developing the compression algorithm and for valuable comments. 12