Instrument Science Report ACS-98-04
Performance of the onboard compression
algorithm for ACS
F.R. Boffi & M. Stiavelli
January 28, 1999
ABSTRACT
An extensive set of experiments was performed to test the performance of the onboard
compression algorithm created for ACS by R. White. We determined the compression factor for realistic simulated images of different astronomical targets as well as of STIS
images and ACS darks and flat fields. We present the results of such experiments and our
recommendations for the compression settings of primary and parallel observations. We
also estimated the time required to compress ACS images on-board, but concluded that
these estimates are at present not reliable and therefore suggest an alternative strategy.
Finally, for comparison purposes, we also list the compression factors achieved with the
UNIX commands gzip and compress.
1. Introduction
Compression is one way to deal with many of the problems the 34 Mbyte size of the
ACS WFC images raises. A more complete discussion on the subject can be found in the
ISR ACS-97-02 by Stiavelli and White. In the present document we present the results of
compression tests performed on simulated images ad hoc created (see TIR ACS-98-01 by
Boffi and Stiavelli), on real STIS images, and on ACS and STIS instrument flat fields and
on ACS darks.
The simulated fields represent two clusters of galaxies and two nearby field galaxies.
The STIS images used by us are parallel observations of relatively empty fields. These
fields may be representative of typical ACS WFC targets and may allow us to accurately
test the performance of the compression algorithm on a variety of environments and cases,
by varying gain setting, number of cosmic ray hits (i.e. exposure times), and background
noise.
1
In addition to the compression factor we also estimate the time required to compress
the various images. The compression time depends on the noise level of the image being
compressed and on the required compression factor. We timed the compression on a Sparc
4 CPU and on an Intel 483SX/33Mhz CPU and then used some time results from the ACS
hardware to convert these measurements into estimates for the ACS on-board
compression.
2. The White-Pair algorithm
The White-Pair algorithm adopted for compression of ACS WFC images is lossless,
i.e. completely reversible and causing no data loss unless there is an overflow of the internal buffer. The buffer is sized depending on the desired compression factor which
consequently has to be chosen carefully. It is also worth mentioning that the algorithm
compresses data input blocks of 2048 pixels, always produces compressed output data of
the specified fixed block size and handles block overflow the same way as the onboard
program. On decompression pixels that were dropped due to block overflow are set to
zero.
3. Tests of compression
Tests were performed on simulated as well as on real images. Simulated images were
created for this purpose by Boffi and Stiavelli (TIR ACS-98-01). We focussed mostly on
cases where we expect compression to be less effective, since our main purpose is to
derive robust minimum compression factors. The different fields (see Introduction) were
simulated with a sky background of 500 counts suitable to represent typical long broadband exposures and for different gain settings (1,2,4 and 8). Compression tests were performed on images with a number of cosmic ray hits equivalent to approx. 10% of the
entire number of pixels in the WFC 4144x2068 (physical overscan included) frame and on
the same images with a third of such cosmic hits (to possibly cover a variety of exposure
times). These should represent a one orbit and a typical exposure respectively.
The real images, instead, are STIS short exposure parallel observations of reasonably
empty fields and were actually summed up together to simulate a longer exposure time.
They were all for gain setting equal to one and for a 300 sec. exposure time. The simulated
exposure time is 2100 sec. Because of the multiple read-outs this suitable image has somewhat higher read out noise than an equivalent single STIS exposure. However, the
resulting noise level per pixel is not higher than that expected for a broad band ACS WFC
exposure with medium-bright background.
ACS flat fields and two DARKs, and ten STIS flat fields with different gains were also
compressed. All results are summarized in Table 1. The first column indicates the image,
columns 2 through 5 the various gain settings. For each case the maximum compression
2
factor is listed. In the Table, Ball/FF and Ball/Dark refer to two ACS images that were
used for compression experiments at Ball.
From the results shown in Table 1 we can see that a compression factor of 2 or more is
achieved in the case of simulated images for gain settings of 4 and 8. The images representing the two clusters of galaxies appear to compress more than the two galaxy images.
In fact, the former compress up to a factor of 2.3-2.5 while the latter at the most reach a
compression factor of roughly 2 for the highest gain setting. The STIS image, sum of
seven 300 sec. exposure empty frames, compresses to a factor of 1.6 at gain=1. Flat fields
of both ACS and STIS instead do not compress well. In order to try and understand these
results and to generalize our understanding of the compression algorithm we estimated the
noise levels of all frames. The noise corresponds to a robust estimate of the dispersion of
intensity levels in the image (specifically half the percentile width corresponding to a ±1σ
for a Gaussian distribution). Compression factors are plotted against noise levels for both
sets of ACS simulated images and for the ACS flat fields, in Figure 1.
3
Table 1: Measured compression factors
Image
gain=1
gain=2
gain=4
gain=8
Clusters/long
1.307
1.702
2.119
2.267
Clusters/1/3 CR
1.329
1.785
2.272
2.448
Galaxies/long
1.071
1.371
1.691
2.060
Galaxies/1/3 CR
1.074
1.407
1.753
2.138
ACS/f.f.#1
0.980
1.025
1.176
1.524
ACS/f.f.#2
0.982
ACS/DARK
1.776
2.168
2.764
3.193
Ball/f.f.
1.03
Ball/DARK
2.147
STIS/ f.f.#1
0.981
STIS/ f.f.#2
0.981
STIS/f.f.33
0.983
STIS/f.f.#4
0.984
STIS/f.f.#5
1.141
STIS/f.f.#6
0.986
STIS/f.f.#7
0.985
STIS/f.f.#8
0.985
STIS/f.f.#9
1.000
STIS/f.f.#10
1.000
STIS/limage
1.618
4
Figure 1: Correlation between compression factor and noise level.
In Fig. 1 filled circles represent simulated cluster images, open circles the two galaxy
images, and the open squares identify ACS flat fields. The higher the noise the less the
compression factor. Such a behavior was already anticipated by Stiavelli and White
(1997). It is interesting to notice that each set of data falls within a relatively narrow band
in the 2D plane. The flat fields have the lowest compression factors of the three sets.
4. Timing of the compression tests
Timing tests were performed on all images discussed in Section 3. The results for a
compression factor of 2, i.e. a blocksize of 2048 bytes, are presented in Table 2, which is
formatted similarly to Table 1. The first column indicates the image, columns 2 through 5
the various gain settings. For each case we list the time it took to compress each frame on
a Sparc4 workstation as measured with the UNIX time command (in units of seconds). A
Sparc 4 was used since faster workstations complete the compression too rapidly for an
accurate timing estimate. In order to estimate the on-board compression time we plan to
establish the relation between our timing measurements and those carried out on the ACS
hardware on the two ACS reference images (Ball/FF and Ball/Dark). To verify the reliability of our approach we also timed the compression of STIS images and the two ACS
reference images on an Intel 386SX/33Mhz CPU, which has an architecture more similar
to the Intel 386/16Mhz used in ACS. In these latter experiments we tested a number of different compression factors since we expect a dependence of the compression time on the
5
commanded block size in addition to the image noise level. In Table 3 we list the results
for the two reference images while in Table 4 we list the compression results for the STIS
images and various blocksizes. For the reference image timing measurements on the ACS
hardware we will focus on the 4 amplifier read-out mode since, as we have checked, the
results from other readout schemes can be accurately obtained by rescaling the amount of
data that are compressed.
Table 2. Compression timings on a Sparc 4 CPU (blocksize=2048).
Clusters/long
4.13 s.
3.80 s.
3.66 s.
3.46 s.
Clusters/1/3 CR
4.24 s.
3.64 s.
3.67 s.
3.45 s.
Galaxies/long
4.27 s.
4.09 s.
3.81 s.
3.40 s.
Galaxies/1/3 CR
3.84 s.
3.88 s.
3.70 s.
3.37 s.
ACS/f.f.#1
4.34 s.
4.08 s.
4.34 s.
4.93 s.
ACS/f.f.#2
4.07 s.
ACS/DARK
3.75 s.
3.57 s.
3.26 s.
3.47 s.
STIS/f.f.#1
0.55 s.
STIS/f.f.#2
0.55 s.
STIS/f.f.#3
0.63 s.
STIS/f.f.#4
0.58 s.
STIS/f.f.#5
0.54 s.
STIS/f.f.#6
0.61 s.
STIS/f.f.#7
0.52 s.
STIS/f.f.#8
0.57 s.
STIS/f.f.#9
0.48 s.
STIS/f.f.#10
0.55 s.
STIS/image
0.51 s.
Ball/ f.f.
4.37 s.
Ball/DARK
3.7 s.
6
Table 3. Compression timing results for ACS images (in seconds)
Name
blksize
ACS
Sparc4
486/33
Ball/FF
1156
65.43
3.84 ± 0.1
18.3 ± 0.2
Ball/FF
2048
95.49
3.94 ± 0.1
18.6 ± 0.4
Ball/FF
4094
164.1
4.36 ± 0.16
21.9 ± 1.4
Ball/Dark
1156
126.8
3.58 ± 0.10
21.6 ± 0.5
Ball/Dark
2048
186.8
3.67 ± 0.10
21.9 ± 0.4
Ball/Dark
4094
186.8
4.06 ± 0.16
22.9 ± 0.2
Table 4. Compression timing results for STIS images (in seconds)
Name
blksize
Sparc4
486/33
STIS/f.f.#1
2048
0.53 ± 0.02
3.13 ± 0.04
STIS/f.f.#1
2560
0.56 ± 0.04
3.13 ± 0.09
STIS/f.f.#1
2730
0.54 ± 0.07
2.97 ± 0.19
STIS/f.f.#1
3724
0.50 ± 0.03
3.40 ± 0.04
STIS/f.f.#5
2048
0.57 ± 0.04
3.44 ± 0.57
STIS/f.f.#5
2560
0.55 ± 0.04
3.52 ± 0.09
STIS/f.f.#5
2730
0.58 ± 0.02
3.29 ± 0.26
STIS/f.f.#5
3724
0.58 ± 0.05
3.76 ± 0.10
STIS/f.f.#7
2048
0.55 ± 0.06
3.03 ± 0.15
STIS/f.f.#7
2560
0.56 ± 0.03
3.14 ± 0.07
STIS/f.f.#7
2730
0.50 ± 0.04
2.78 ± 0.16
STIS/f.f.#7
3724
0.57 ± 0.05
3.16 ± 0.04
STIS/Image
2048
0.55 ± 0.03
3.48 ± 0.11
STIS/Image
2560
0.56 ± 0.04
3.40 ± 0.20
STIS/Image
2730
0.59 ± 0.03
3.25 ± 0.22
STIS/Image
3724
0.60 ± 0.03
3.74 ± 0.23
In Figure 2 we verify how well the Sparc 4 and the 486/33 timings correlate with those
measured with the ACS hardware (triangles, squares, and stars indicate blocksizes of
1156, 2048, 4094, respectively). While the Sparc 4 timings do not correlate well with the
7
ACS hardware, we find that the 486/33 timings provide indeed a very good correlation,
probably due to the similarity in instruction set between the two CPUs.
The best fit relation connecting the 486/33 timing to the ACS is the following:
T ( ACS ) = ( 28.0 ± 5.5 ) × T ( 486 ) – ( 447.5 ± 28.3 )
Carrying out timing experiments on the 486/33 is very cumbersome, thus, as a last
resort, we attempted to see whether a reasonably good correlation exists between the 486/
33 timings and the Sparc 4. Since Figure 2 shows a good correlation between ACS and
486/33 timings and no (good) correlation between ACS and Sparc 4, we do not expect
486/33 and Sparc 4 timings to correlate well. We can actually identify a weak correlation
when more images are used. In Figure 3 we show this correlation as revealed when 16
STIS images are also considered. The STIS timings have been multiplied by 7.668 to
account for the difference in size between ACS images with overscan and STIS images.
The relation shows a scatter of about 10 per cent but is statistically significant:
T ( 486 ) = ( 9.9 ± 1.9 ) × T ( Sparc4 ) – ( 17.4 ± 2.0 )
By combining these two relations we are able to derive an estimate of the time it would
take to compress with ACS any of the images considered in Section 3. Our results on the
worst case compression timings are summarized in the following table:
Table 5. Extrapolated worst case compression times (in seconds)
Image Type
Gain=1
Gain=2
Gain=4
Gain=8
Science
250
200
125
100
FF
280
400
360
240
Dark/Bias
105
55
40
30
The overall uncertainty of these results is hard to assess. There is at least a 20 per cent
uncertainty due to the two steps involved. However, much larger uncertainties are proba-
8
bly present when we estimate the compression time for images very different from those
used to derive the transformation equations. The large degree of uncertainty of our estimates is shown by the fact that an actual GAIN=1 bias compresses on ACS in 186 s while
our estimate based on Sparc4 data would give us 105 s. It is our conclusion that a better
estimate of ACS compression timings for a scientifically relevant range of images cannot
be obtained with these methods. Therefore, we suggest that:
•
a set of compression experiments be carried out on the ACS hardware or on a bench.
These tests should involve either a number of biases and flat field exposures compressed with various blocksizes and/or the compression of simulated images provided
by us;
•
the exact version of the compression code implemented by Ball in the flight software
be run on a few representative images in order to better understand the timing discrepancies, and in particular the fact that even the good 486/33 - ACS timing correlation
does not appear to be a simple scaling with the clock speed.
Figure 2: Correlation between different compression timing experiments.
9
Figure 3: Correlation between Sparc4 and 486 timing experiments
5. Conclusions and recommendations
The compression experiments here described allow us to recommend a compression
strategy for ACS. We verified that images with a significant amount of noise compress
very little. Typical astronomical images will compress more and biases and darks compress best. Our recommended compression factors are provided in Table 5. It is worth
noticing that allowing even a small amount of data loss does not improve significantly the
compression factors attainable.The timing experiments have provided less clearcut results.
However, both the extrapolated compression times from Sparc 4 experiments and the measurements actually carried out at Ball indicate that the compression time should remain
below 200 s for all images for which compression is recommended. Only on-orbit experience is likely to allow us to refine these estimates.
10
Table 6: Recommended compression factors
WFC observation
gain=1
gain=2
gain=4
gain=8
Primary
1.0
1.0
1.0
1.6
Parallel
1.0
1.0
1.5
1.8
Flat Field
1.0
1.0
1.0
1.0
Dark & Bias
1.5
1.9
2.5
3.0
6. References
Boffi, F.R., Stiavelli M., 1998, TIR ACS-98-01
Stiavelli, M., White R.L., 1997, ISR ACS-97-02
7. APPENDIX: Tests of compression using gzip and compress
In order to provide some term of comparison and to present a complete range of results
we also compressed all frames by using standard compression algorithms available in
unix, like gzip and compress. We have performed these experiments on all images listed in
Table 1. In Table 3 we present the results of these tests: in each cell the compression factors for gzip and compress are respectively given. The integer fits files have the following
dimensions: the ACS simulated images and instrument flat fields 17 Mbytes, the STIS
images and flat fields 2.2 Mbytes. Gzip and compress compress the images with the two
galaxies of some factor comprised in the range 1.2 - 2.4, the higher values being achieved
for higher gain settings. The images of two galaxy clusters are compressed of a factor
between 1.7 and 4. The two clusters fields appear to compress slightly better than the two
galaxies with both compression algorithms, the two galaxies instead are better compressed
with gzip than compress. The STIS flat fields are appear to compress slightly less than the
ACS. The former compress between 1.2 and 1.5 at the most, the latter between 1.3 and
1.8. Finally the STIS image (sum of seven 300 sec. exposure images) compress of about
1.8 with both algorithms which well agrees with the compression factors obtained for the
ACS simulated images.
11
Table 7: Compression factors with gzip and compress
Frame
gain=1
gain=2
gain=4
gain=8
Clusters/long
1.701; 1.631
1.975; 2.015
2.540; 2.705
3.613; 3.988
Clusters/1/3 CR
1.737; 1.669
2.002; 2.051
2.581; 2.745
3.642; 4.026
Galaxies/long
1.391; 1.186
1.566; 1.430
1.846; 1.786
2.366; 2.383
Galaxies/1/3CR
1.408; 1.198
1.592; 1.456
1.870; 1.815
2.397; 2.419
ACS/f.f.#1
1.296; 1.098
1.409; 1.177
1.542; 1.349
1.759; 1.675
ACS/f.f.#2
1.296; 1.097
ACS/DARK
2.310; 2.398
2.838; 3.042
3.670; 4.062
4.986; 5.887
Ball/f.f.
1.200; 1.031
Ball/DARK
2.451; 2.571
STIS/f.f.#1
1.218; 1.016
STIS/f.f.#2
1.217; 1.013
STIS/f.f.#3
1.228; 1.024
STIS/f.f.#4
1.227; 1.022
STIS/f.f.#5
1.497; 1.328
STIS/f.f.#6
1.262; 1.059
STIS/f.f.#7
1.257; 1.054
STIS/f.f.#8
1.261; 1.057
STIS/f.f.#9
1.418; 1.223
STIS/f.f.#10
1.402; 1.205
STIS/image
1.871; 1.820
Acknowledgements
We thank R. White for developing the compression algorithm and for valuable
comments.
12