1
Compression of 3D Scientific Data for Archival,
Distribution and Fast Classification
First A. Author, Second B. Author, Jr., and Third C. Author, Member, IEEE
Abstract—In this work, we propose a compression algorithm based on Partitioned Vector
Quantization for 3D scientific data, and in particular those generated by remote sensors. One of the
dimensions of the input data is treated as a single vector that is partitioned into subvectors of
(possibly) different lengths, each quantized separately. Quantization indices and residuals are then
entropy coded. Partitioning serves different purposes: vector quantization looses efficiency in high
dimension and partitioning is a way of improving it; it allows a better decorrelation of the data in
the given dimension, improving compression; the concatenation of the quantization indices can be
seen as a projection to a lower dimensional subspace that preserves the original statistical properties
of the data; partition indices can be used for fast (remote) browsing and classification. In fact,
experimental results with NASA AVIRIS hyperspectral images show that the proposed algorithm
achieves state of the art lossless compression on such data, provides near-lossless and lossy
capabilities (that could be used for fast browsing and broadcasting on limited bandwidth
communication channels), is scalable to arbitrary data size, allows fast classification and target
detection in the compressed domain.
Index Terms—About four key words or phrases in alphabetical order, separated by commas. For a
list of suggested keywords, send a blank e-mail to keywords@ieee.org or visit the IEEE web site at
http://www.ieee.org/web/developers/webthes/index.htm.
A
N
I. INTRODUCTION
increasing number of scientific instruments produce high volume of data in the form of
two-dimensional matrices of vectors, also called data cubes. Typical examples are biomedical images,
multi, hyper and ultra spectral images, measurements of volumetric parameters like atmospheric
temperature and pressure by way of microwaves sounders. Compressing these data presents a challenge for
the currently standardized compressors. Beside the three dimensional nature of the measurements, which
exhibit correlation along each dimension, the single measurements have values of 16 bits or more. Current
state of the art compressors do not perform well on data sources with large alphabets. Furthermore, it is
desirable to have a single compressor that can perform well in all of lossless, near lossless, and lossy modes
and can scale to data cubes of arbitrary sizes. Lossless compression is often required for data collection and
archiving due to the cost of the data collection and that it may be destined to analysis and further
2
elaboration. However, when data is downloaded by users, a lossy mode can be employed to speed up the
transmission, since the final users may not need all measurements at their full precision, and in any case,
can request further precision only when needed.
In [1] we propose a compression algorithm based on Partitioned Vector Quantization, where one
dimension of the cube is treated as a single vector, partitioned and quantized. Partitioning the vector into
subvectors of (possibly) different lengths is necessary because the possibly large number of components.
Subvectors are individually quantized with appropriate codebooks. The adaptive partitioning uses a novel
locally optimal algorithm and provides a tool to reduce the size of the source alphabet. Correlation among
the other two dimensions is exploited with methods derived from the image coding domain. Lossless or
near-lossless entropy coding can be applied to the residual error. An additional feature of this compression
method is the possibility to tightly bound error on a pixel by pixel basis. Due to the hierarchical structure of
the data compressed with the proposed algorithm, it is possible to perform classification and target
detection directly in compressed domain, with considerable speed and memory savings. The high speed
classification and target detection algorithm that we describe here can process 90 percent of the compressed
pixels with a simple table lookup, full decompression is only required if exact classification is necessary
and it is limited to the 10 percent remaining pixels. Algorithms have been verified experimentally on the
lossless and near lossless compression of NASA AVIRIS images.
Section 2 briefly reviews the algorithm in order to introduce the necessary notation. Besides good
compression performance, LPVQ presents a number of features that are highly desirable for these kinds of
images. A communication paradigm that shows these advantages is outlined and discussed in Section 3.
The problem of classification and the necessary notation is introduced in Section 4. Section 5 presents
experimental results on the classification error introduced by LPVQ lossy and near lossless compression
modes. Section 6 builds on these experimental observations with an algorithm that speeds up the
classification by accepting or rejecting vectors based only on bounds of the VQ error. These error bounds
are available at design time or can be extracted from the compressed image. In the same section, the
3
asymptotic behavior is derived. Given a compressed image and a classification threshold, the speed up
achieved on the naïve algorithm depends on the acceptance and rejection rates. Experiments on typical
AVIRIS images are performed in the same section in order to assess these rates for a number of
classification thresholds. The acceptance and rejection rates only depend on the bounds on the quantization
error and can be easily derived during the design of the LPVQ. So, these considerations also provide a tool
useful to select the quantization parameters (codebook entries and number of partitions) in order to match a
desired range of classification speeds.
II. THE COMMUNICATION MODEL
An application for which our algorithms were designed is the encoding of information collected by a
remote sensing device having limited computational power (satellite, airplane, biomedical imager) and
losslessly transmitted to a powerful central server (see Figure 1). The central server decompresses and
elaborates the data, performing all necessary adjustments and corrections. Then, re-codes the data in a more
compact representation and distributes the result among a number of users that may subscribe the service at
different quality levels. A common example is weather information collected by geostationary satellites.
The satellites observe of events evolving in a specific geographic area, such as hurricanes, storms, flooding
and volcanic eruptions. The collected data have to be faithfully transmitted to a land based central station
that analyzes the data, applies some form of enhancement, compressed the data further and sends the result
back to the satellite in order to broadcast it to the final users. In this setting, the same satellite is used for
both acquisition and broadcasting. More common is the case of the central server performing the
transmission. While the compression hardware used in the satellite must be simple and require very low
power to function, there is typically little or no limitation to the computational power of the base station.
III. DEFINITIONS
We model a three dimensional data cube as a discrete time, discrete values, bi-dimensional random
source I(x,y) that emits pixels that are D-dimensional vectors I(x,y) . Each vector component Ii (x, y) ,
0  i  D 1, is drawn from a finite alphabet Xi and is distributed according to a space variant probability


4
distribution that may depend on other components. We assume that the alphabet has the canonical form
Xi  0,1, ,Mi.
Our encoder is based on a Partitioned Vector Quantizer (or PVQ), which consists of N independent,
L -levels, di -dimensional exhaustive search vector quantizers Qi  (Ai,Fi,Pi ) , such that
 d  D and:
i
1 i N
d
i i
i
Ai  {c1,c 2 ,...,c L } is a finite indexed subset of R i called codebook. Its elements c ij are the code

vectors.
d
i
i
i
Pi  {S1 ,S2 ,...,SL } is a partition of R i and its equivalence classes (or cells) S ij satisfy:

L
j 1
i
i
S ij  R d i and Sh  Sk   for h  k .
Fi:R  Ai is a function defining the relation between the codebook and the partition as Fi (x)  c ij
di

if and only if x S ij .
The index j of the codeword c ij , result of the quantization of the di -dimensional subvector x , is the
information that is sent to the decoder.
With reference to the previously defined N vector quantizers Qi  (Ai,Fi,Pi ) , a Partitioned Vector
Quantizer can be formally described by a triple Q  A ,P ,F  where:
D

A  A1  A2   AN is a codebook in R ;

D
P  P1  P2   PN is a partition of R ;



D
F :R  A is computed on an input vector x R
D
as the concatenation of the independent

quantization of the N subvectors of x . Similarly, the index vector sent to the decoder is obtained
as a concatenation of the N indices.
Thedesign of a Partitioned VQ aims at the joint determination of the N 1 partition boundaries
b0  0  b1 
 bN  D and at the design of the N independent vector quantizers having dimension
di  bi  bi1 , 1  i  N .

th
In the following, given a source vector I(x,y) , we indicate the i subvector of boundaries bi1 and bi 1

5
with the symbol Ibb ii 11 (for simplicity, the x and y spatial coordinates are omitted when clear from the
context). The mean squared quantization error between the vector I and its quantized representation Iˆ , is
given by
 
I  ˆI
2


i
Ji

where c  c
N
 I
i1
N

i1
i
Ji ,0
, ,c
i
J i ,d i 1
b i 1
b i 1
 ˆIbbii11
Ibb ii 11  c iJi

2

2
 is the codeword of the i
b i 1
  I
N

i 1 h b i 1
th
h
 c Ji i ,h b i 1

2
codebook minimizing the reconstruction error on
Ibb ii 11 , and:

b 1
i
J i  argmin MSE Ib ii 1 ,c l
1l  L

IV. A LOCALLY OPTIMAL PVQ DESIGN
The complexity of building a quantizer that scales to vectors of arbitrary dimension is known to be
computationally prohibitive. Furthermore, the efficiency of a VQ decreases with its dimensionality: the
fraction of the volume of a hypersphere inscribed in a hypercube of the same dimension d goes to zero
when d goes to infinity ([2]-[3]). Partitioning the vectors in consecutive, non-overlapping subvectors is a
common solution to this problem (see the book of Gersho and Gray [4]). While a partitioned VQ leads to a

sub-optimal solution in terms of Mean Squared Error (MSE) because it does not exploit correlation among

subvectors, the resulting design is practical and coding and decoding present a number of advantages in
terms of speed, memory requirements and exploitable parallelism.
In a Partitioned VQ, we divide the input vectors into N subvectors and quantize each of them with an
L -levels exhaustive search VQ. Having N subvectors of the same dimension (except perhaps one, if N
does not divide D) is not a possibility here, since we assume that the components of I(x,y) may be drawn
from different alphabets. In this case their distributions may be significantly different and partitioning the
D components uniformly into N blocks may not be optimal. We wish to determine the size of the N sub
vectors (of possibly different size) adaptively, while minimizing the quantization error, measured for
example in terms of MSE. Once the N codebooks are designed, input vectors are encoded by partitioning
6
them into N subvectors of appropriate length, each of which is quantized independently with the
corresponding VQ. The index of the partitioned vector is given by the concatenation of the indices of the N
subvectors.
Given the number of partitions N and the number of levels per codebook L, it is possible to find the
partition boundaries achieving minimum distortion with a brute-force approach. We first determine, for
every 0  i  j  D , the distortion Dist(i, j) that an L -levels vector quantizer achieves on the input
subvectors of boundaries i and j . Then, with a dynamic program, we traverse the matrix Dist(i, j) and find
the N costs corresponding to the input partition of boundaries b0  0  b1 
 bN  D and whose sum is
minimal. A more sophisticated approach to partitioned vector quantization is discussed in Matsuyama [5].
It uses dynamic programming at each step to decide the current optimal partition, instead of designing one
vector quantizer for each possible partition configuration first and then applying dynamic programming.
Nevertheless, even this approach is computational intensive.
The locally optimal partitioning algorithm that we propose here (LPVQ) provides an efficient alternative
to dynamic programming, while performing comparably in most applications. Our partitioning algorithm is
based on a variation of the Generalized Lloyd Algorithm (or GLA, Linde, Buzo and Gray [6]).
Unconstrained vector quantization
can be seen as the joint optimization of an encoder (the function

d
F:R  A described before) and a decoder (the determination of the codewords for the equivalence
classes of the partition P  {S1 ,S2 ,...,SL }). GLA is an iterative algorithm that, starting from the source
sample vectors chooses a set of codewords (also called centroids) and optimizes in turns encoder and
decoder until improvements on the predefined distortion measure are negligible. In order to define our
PVQ, the boundaries
of the vector partition b0  0  b1 

 bN  D have to be determined as well. Our
design follows the same spirit of the GLA. The key observation is that, once the partition boundaries are
kept fixed, the Mean Square Error (the distortion we use in our application) is minimized independently for
each partition by applying the well-known optimality conditions on the centroids and on the cells.
Similarly, when the centroids and the cells are held fixed, the (locally optimal) partitions boundaries can be

7
determined in a greedy fashion. The GLA step is independently applied to each partition. The equivalence
classes are determined as usual, but as shown in Figure 2, the computation keeps a record of the
contribution to the quantization error of the leftmost and rightmost components of each partition:


2


i   Ib (x, y)  Iˆb (x,y) and i   I b 1 (x,y)  Iˆb 1 (x, y)
x ,y
i 1
i 1
x,y
i
i
2
Except for the leftmost and rightmost partition, two extra components are also computed:

i   I b
x,y
i 1 1



2
(x,y)  Iˆb i 1 1 (x, y) and i   I b i (x,y)  Iˆb i (x, y)
x,y
2
The reconstruction values used in the expressions for i and i are determined by the classification
performed on the components bi1,...,bi . The boundary bi1 between the partitions i 1 and i is changed
according to the criteria
Mi = min( i 1   i , i 1  i 1 , i   i )
if ( Mi = i   i )
bi1  bi 1 1
else if ( Mi = i 1  i 1 )
bi1  bi 1 1
V. ENTROPY CODING
The partitioned vector quantizer is used here as a tool to reduce the dimensionality of the source vectors.
Dependence between the other two dimensions has to be exploited by means of an entropy coder.
Furthermore, when a lossless coding is required, quantization residuals have to be encoded as well, entropy
coded with a conditioning on the subvector indices.
i

i

i
The index vectors J(x,y) and the codewords c j  c j ,b i , ,c j ,b i 1 1 , with 1  i  N and 1  j  L , are
sufficient to a lossy reconstruction of the data. When higher accuracy is needed, the compressed data can be
augmented with the quantization error:


where, for each 1  i  N :

b 1
b 1
b 1

E(x, y)  Eb10 (x, y),E b21 (x, y),...,Eb NN 1 (x, y)
E bbii 11 (x,y)  Ibb ii 11 (x, y)  ˆIbbii11 (x, y)
The unconstrained quantizers work independently from each other and independently on each source
8
vector and an entropy encoder must be used to exploit this residual redundancy. In particular, each VQ
index Ji(x,y) is encoded conditioning its probability with respect to a set of causal indices spatially and
spectrally adjacent. The components of the residual vector E bbii 11 (x,y) are entropy coded with their
probability conditioned on the VQ index Ji(x,y) . In near-lossless applications, a small, controlled error can
be introduced at this stage. Figure 3 depicts a diagram of the LPVQ encoder.
VI. COMPUTATIONAL COMPLEXITY
Vector Quantization is known to be computationally asymmetrical, in the sense that encoding has a
computational complexity which is orders of magnitude higher than decoding. Even more time consuming
is the design of the dictionary which is also not guaranteed to converge into an easily predictable number of
iterations. Numerous methods have been designed to speed up the convergence, for example, by
constraining the search of the closest codeword like in [7], or by seeding the dictionary with vectors having
special properties [8]. Adding structure to the VQ can also simplify encoding at the price of a limited
degradation of the output quality [4]. All these improvements clearly apply to our method as well;
furthermore, practical scenarios arise in which our algorithm can be applied as is without the computational
complexity being an obstacle.
A possible way of mapping the LPVQ algorithm to the communication model described in Section II is
to equip platform used for the remote acquisition with an encoder able to download a dictionary transmitted
from the central server (see Figure 1). This dictionary can be, for the very first scene, bootstrapped as
described in [8], with a number of vectors randomly selected from the image itself, with a dictionary
designed for another image or with any dictionary that believed to be statistically significant. Due to the
nature of our algorithm, the choice of the first dictionary will only impact the compression ratio of the very
first transmission.
Once the central server receives a scene, let’s say at time instant t, a few iterations of LBG may be
necessary to refine the dictionary, adjust the partition boundaries and improve compression. The refined
version of the dictionary is transmitted back to the remote platform and used for the encoding of the scene
9
acquired at time t+1. In Section VIII we report on experiments intended to assess the performance
degradation derived from a mismatched dictionary.
It has been observed in [1] that if the codewords in each codebook are sorted according to their energy,
the N planes constituted by the indices retain important information on the structure of the original image.
These planes alone can be used to browse and select areas of interest and, as we show in section XX, to
extract important information on the nature of the original pixel vectors.
VII. EXPERIMENTAL RESULTS
LPVQ has been tested on a set of five AVIRIS images ([9]). AVIRIS images are obtained by flying a
spectrometer over the target area. They are 614 pixels wide and typically have a height on the order of
2,000 pixels, depending on the duration of the flight. Each pixel represents the light reflected by a 20m x
20m area (high altitude) or 4m x 4m area (low altitude). The spectral response of the reflected light is
decomposed into 224 contiguous bands (or channels), approximately 10nm wide and spanning from visible
to near infrared light (400nm to 2500nm). Each of these vectors provides a spectral signature of the area.
Spectral components are acquired in floating point 12-bit precision and then scaled and packed into signed
16 bit integers. After acquisition, AVIRIS images are processed to correct for various physical effects
(geometrical distortion due to the flight trajectory, time of day, etc.) and stored in scenes of 614 by 512
pixels each (when the image height is not a multiple of 512, the last scene will be smaller). All files for
each of the 5 test images were downloaded from the NASA web site (JPL [9]) and the scenes belonging to
the same flight merged together to form a complete image.
Several experiments have been performed for various numbers of partitions and for different codebook
sizes. The results that we describe here were obtained for N 16 partitions and L  256 codebook levels.
The choice of the number of levels makes also practical the use of off-the-shelf image compression tools
that are fine-tuned for 8 bit data. The LPVQ algorithm trains on each image independently and the
codebooks are sent to the decoder as side information. The size of the codebook is negligible with respect
the size of the compressed data (256 x 224 x 2 bytes = 112 KB) and its cost is included in the reported
10
results.
The partition boundaries for each of the five images are depicted in Figure 4. While similarities exist, the
algorithm converges to different optimal boundaries on different input images. This is evidence that LPVQ
adapts the partitions to input statistics. Starting with a codebook obtained by random image pixels, we have
found that adaptation is fairly quick and boundaries converge to their definitive values in less than one
hundred iterations.
If LPVQ is bootstrapped with the method described in [8], which populates the dictionary with the vector
with highest norm and then adds the vector whose minimum distance from the current dictionary is
maximum, the algorithms shows a much faster convergence. Furthermore, the constrained search for the
closest codeword is generally faster by a factor of 2.
LPVQ performances are analyzed in terms of Compression Ratio (defined as the size of the original file
divided by the size of the compressed one), Signal to Quantization Noise Ratio, Maximum Absolute Error
and Percentage Maximum Absolute Error.
The Signal to Quantization Noise Ratio (SQNR) is defined here as:
The correction factor
1
12
2
10 D 1
  Ii 
SQNR (dB)   log 10  2  1 
Ei
12
D i 0
is introduced to take into account the error introduced by the 12 bit
analog-to-digital converter used by the AVIRIS spectrometer ([9]). This solution also avoids unbounded
values in the case of a band perfectly reconstructed.
th
The Maximum Absolute Error (MAE) is defined in terms of the MAE for the i band
as MAE  max MAE i , where MAEi is:
i
MAE i  max Ii (x,y)  Iˆi (x, y)
x ,y
th
The average Percentage Maximum Absolute Error (PMAE) for the i band having canonical alphabet
Xi  0,1, ,Mi is defined as:
PMAE (%) 
1 D 1 MAE i
 100

D i 0 Mi
11
Table I shows that LPVQ achieves on the five images we have considered an average compression of
3.14:1, 45% better than the 1-D lossless compressor bzip2 when applied on the plane–interleaved images
(worse results are achieved by bzip2 on the original pixel–interleaved image format) and 55% better than
the standard lossless image compressor JPEG-LS. We also compared LPVQ with the method published in
[11], even though it reports experiments on a subset of our data set. The algorithm in [11] uses LBG to
cluster data. An optimized predictor is computed for each cluster and for each band. Prediction error is
computed and entropy coded, along with side information regarding the optimized predictors. The resulting
method outperforms LPVQ, while sharing time complexity similar to the design stage of LPVQ and
lacking any extra feature provided by the latter, like lossy and near-lossless mode, fast browsing and
classification, et cetera. The last column of Table I reports, as a reference, the compression and the SQNR
when only the indices are encoded and the quantization error is fully discarded. As we can see from the
table, on average we achieve 55:1 compression with 25.27dB of SQNR.
More interesting and practical are the results obtained with the near-lossless settings, shown in Table II.
At first, the introduction of a small and constant quantization error across each dimension is considered;
that is, before entropy coding, each residual value x is quantized by dividing x adjusted to the center of the
range by the size of the range; i.e.,
q(x) = (x+)/(2+1). This is the classical approach to the
near-lossless compression of image data and results into a constant MAE across all bands. With this setting,
it is possible to reach an average compression ratio ranging from 4:1 with the introduction of an error
  1 and a MAE  1 to 10:1 with and error of   10 and MAE  10 . While the performance in this
setting seem to be acceptable for most applications and the SQNR is relatively high even at high
compression, the analysis of the contribution to the PMAE of the individual bands shows artifacts that
might be unacceptable. In particular, while the average PMAE measured across the 224 bands of the
AVIRIS cube is low, the percentage error peaks well over 50% on several bands (see Figure 5). Since the
PMAE is relevant in predicting the performance of many classification schemes, we have investigated two
different approaches aimed at overcoming this problem. In both approaches we select a quantization
12
parameter that is different for each band and it is inversely proportional to the alphabet size (or dynamic).
In general, high frequencies, having in AVIRIS images higher dynamic, will be quantized more coarsely
than low frequencies. We want this process to be governed by a global integer parameter .
th
The first method, aiming at a quasi–constant PMAE across all bands, introduces on the i band a
distortion i such that:
1 D 1
 
D i 0 i
th
Since the i band has alphabet Xi  0,1, ,Mi we must have:


D 
1 

i  
 Mi 
 D 1 M
2 
i
 i 0

The alternative approach, aims at a quasi–constant SQNR across the bands. If we allow a maximum
th
absolute error  i on the i band, it is reasonable to assume that the average absolute error on that band
will be
i
. If we indicate with i the average energy of that band and with  the target average maximum
2
absolute error, then the absolute quantization error allowed on each band is obtained by rounding to the
nearest integer the solution of this system of equations:

 2j
 2i
 10log10
i, j 0, ,D  1, i  j
10log 10
2
2

i 2
j 2

D 1
 1



 
 i
 D i 0
As can be seen from Table II, the three methods for near-lossless coding of AVIRIS data are equivalent
in terms of average SQNR at the same compression. However, the quasi-constant PMAE method is indeed
able to stabilize the PMAE across each band (Figure 6). The small variations are due to the lossless
compression of some bands and the rounding used in the equations. The average SQNR is not
compromised as well. Similar results are observed for the quasi-constant SQNR approach. The SQNR is
almost flat (Figure 7), except for those bands that are losslessly encoded and those with small dynamic. The
PMAE is also more stable than the constant MAE method (Figure 8).

13
VIII. LPVQ AND RANDOM DICTIONARIES
In order to evaluate the proposed approach in the framework described in Section II, we performed the
following experiments for each scene of the Moffett Field image:
1) we compute compression ratio and RMSE at each step of the LPVQ algorithm, starting with a
random dictionary;
2) same except that the starting dictionary is the locally optimal one for the previous scene.
As we can see from Figure 9, compression performance is quite stable already after just 50 iterations. The
compression ratio is slightly lower when LPVQ is fed with the dictionary from the previous scene, while
RMSE, on the other hand, is much lower. This suggests that the entropy coder needs to be tuned-up for this
case and that the lossy compression is slightly better when only few iterations are allowed.
A different experimental setting uses six different schemes to generate random dictionaries. One hundred
dictionaries for each scheme are used with PVQ (i.e., the dictionary is not refined with the proposed LBGlike scheme to obtain a locally optimal one) and with both uniform and optimal partitions. Compression
results for each scheme are box-plotted in Figure 10. The results suggest that taking a small sample of the
input data is enough to obtain compression performances reasonably close to the locally optimal solution
produced by LPVQ, and that the non-uniform partitioning is much more effective at high compression.
IX. THE PROBLEM OF CLASSIFICATION
Let each image pixel be a D-dimensional vectors I( x, y)   I 0
disjoint subvectors as I( x, y)   I0
I1
I D 2
I D1  divided into N
I N 2 I N 1  , the n -th partition having dimension dn , for
I1
0  n  N . For simplicity the boundaries are omitted here.
Upon
quantization
J( x, y)   J 0
J1
each
J N 2
vector
is
represented
J N 1  and by E( x, y)   E0
by
E1
an
N -dimensional vector
ED2
of
indices
ED1  , a D-dimensional vector
representing the quantization error. Each component J(x,y) indexes one of the L codebook centroids for
the VQ corresponding to the partition n , 0  n  N . If Iˆk is the quantized value for Ik , LPVQ decomposes

14
each component of an input vector I(x,y) into as Ik  Iˆk  Ek , where 0  k  D .
Given a target vector T   t0
tD2 tD1  , a distortion measure D(I,T) and a classification
t1
threshold , we wish to determine, for every image pixel I(x,y) , whether D(I(x, y),T)   . When this
condition is satisfied, we say that the pixel I(x,y) is a member of the class T .
The Euclidean Distance ED(I,T) 
D 1
 I
k 0
k
 t k  is distortion measures commonly used in classification.
2
The classification threshold , plays an important role since it determines whether the classification is
used to detect a rare pixel instance, eventually present in the image (problem commonly referred to as
target detection) or whether the objective of the classification is the determination of a thematic map,
where almost every pixel belongs to one of several classes.
X. EFFECT OF CODING ERROR ON THE CLASSIFICATION
As we did in the case of the compression we used AVIRIS hyperspectral images to assess the effect of
the coding error on classification. Classification and target detection are algorithms frequently applied to
hyperspectral images; when the images are lossily encoded, it is licit to ask whether and in which measure
the coding error affects the results of these algorithms.

In order to assess the classification errors introduced by LPVQ lossy modes, we have experimented with
the standard images for various errors and threshold parameters. It is important to notice that, while the

lossy results are peculiar of our algorithm, the near lossless apply to any compression scheme that bounds

the maximum absolute error on each vector component. The results for three images, Cuprite, Moffet Field
and Low Altitude, are presented here. Results on the other two images were very similar.
In our experiments, the absolute error introduced by LPVQ in near lossless mode is respectively bounded
to a maximum of 1, 2, 4 and 8 units. The classification uses Euclidean Distance with an average threshold
per band  [0,..,50] ; typically, values in the range  [5,..,35] are of some interest in practical
applications. The figures show the percentage of errors in the classification of 8 targets selected from each
image. The targets are hand picked from the reduced resolution image constituted by the index planes and
15
are meant to represent targets of common interest (buildings, roads, water, vegetation, etc...). The lossless
encoding is used as a reference for the classification and the error percentages reflect the sum of all false
positive (classifications that were not classified in the lossless), false negative (vectors classified in the
lossless and not classified in the lossy) and misclassifications (different clusters in lossless and lossy).
As it is possible to see from Figures 11-13, in near lossless, the classification is fundamentally
unaffected; even for an error of 4 , the percentage of classification errors is, in the worst case, below
0.1%. This is sufficient for most applications. Smaller coding errors, achieve even lower percentages. For
the compression parameters used in Section VII (i.e., 256 code vectors and 16 partitions) the indices and
the code vectors alone (Figure 14, lossy case) achieve a classification error always lower than 3%. While
this error rate may be unsuitable for some applications, as we show in the next section, it is possible to take
advantage of this characteristic to design a faster classifier.
XI. SPEEDING UP THE CLASSIFICATION
Since the LPVQ indices provide most of the information necessary to classify the pixels, we designed a
simple algorithm that, bounding the minimum and maximum quantization error for each component and for
each cluster, can decide the classification of most pixels without inspecting their error vector E(x, y) .
Bounds on the quantization error are available at design time or can be inferred from the encoded image.
The advantages of this algorithm are twofold. Firstly, the number of partitions and the number of centroids
are typically small, so pre-computed values can be arranged into a lookup table. If most vectors can be
classified without the need of the error, this will result into a substantial speed up. Secondly, we can
imagine a user on the field, browsing and trying different classifications on the LPVQ indices instead of the
whole image. Then, when a parameterization of interest is found, the user can query the server for the error
vectors necessary to resolve any classification ambiguity.
Given a threshold , a target vector T and a distortion measure D(I(x, y),T) , we wish to determine,
from the quantization indices of a pixel I(x,y) , two quantities Dmin(I(x, y),T) and Dmax(I(x, y),T) ,
respectively lower and upper bounding the distortion D(I(x, y),T) .
16
Since I(x,y) belongs to the class T only if D(I(x, y),T)   , we can observe that:

If Dmin(I(x, y),T)   , the pixel I(x,y) clearly does not belong to the class T ;

If Dmax(I(x, y),T)   , the pixel I(x,y) trivially belongs to the class T ;

If Dmin(I(x, y),T)    Dmax (I(x,y),T) then the quantization indices and the bounds on the error
are not sufficient to decide the classification. In this case, and only in this case, it is necessary to
access the error vector and compute D(I(x, y),T)   .
Figure 15 shows the pseudocode that implements this scheme. The main assumption is that the quantities
Dmin(I(x, y),T) and Dmax(I(x, y),T) can be computed by summing the individual contributions of each
vector component k , upper and lower bounded by the minimum and maximum of the quantization
residuals min( E k,l ) and max( Ek,l ) .
This is clearly the case of Euclidean Distance, here defined as:
ED(I(x,y),T) 
D 1
 I
k 0
 tk 
2
k
2
Changing the threshold  into  , we can focus on the contribution of min( E k,l ) and max( Ek,l ) to the
individual terms of the sum. For a given component k , 0  k  D , in the partition n , 0  n  N we have:
I k  tk 2  Iˆk  E k  t k  .
2

If Ik has been quantized with the centroid l , 0  l  L :


  
 

  
 
2
2
2


min  Iˆk  min( Ek ,l )  tk , Iˆk  max( E k,l )  tk  Ik  tk 


and
2
2
2


max  Iˆk  min( E k ,l )  tk , Iˆk  max( E k,l )  tk  Ik  tk  .


Summing these bounds over all components of the partition n we obtain Dmin (Cl ,n,T) and Dmax (Cl ,n ,T) .
These quantities are precomputed in the algorithm in Figure 15 by the two nested “for” loops. Since there
are only 2  L N possible values and N and L are typically small ( N 16 and L  256 in Section VII),
Dmin (Cl ,n,T) and Dmax (Cl ,n ,T) can be stored in a lookup table indexed by n and l . The computation of
Dmin(I(x, y),T) and Dmax(I(x, y),T) requires a total of 2  N lookups and 2 (N  1) sums. If one of the two

17
comparisons Dmin(I(x, y),T)   and Dmax(I(x, y),T)   succeeds, then the pixel I(x,y) is classified from
its indices only, with a number of operations proportional to N .
Otherwise, the error vector
E(x, y) must be retrieved,
I(x,y)
exactly reconstructed as
I(x,y)  J(x, y)  E(x, y) and finally D(I(x, y),T)   computed. This requires a sum a subtraction and a
product for each component and D1 sums to accumulate the result, i.e. a total of D subtractions, 2  D1
sums and D products, a number of operations proportional to D (for AVIRIS images, D  224 ).
The speed of the new classification algorithm depends on the probability that the two tests fail and that
we have to use the error vector in order to classify. Let this probability be:
If sub ,  add and mul
PrDmin (I(x, y),T)    Dmax (I(x,y),T)
are respectively the times to perform a subtraction, a sum and a multiplication on a
given computing platform, the total running time of the new classification is:
2 (N 1)  add  D ( sub  mult )  (2 D 1)  add PrDmin (I(x, y),T)    Dmax (I(x,y),T)
Figures 16-18 show an experimental assessment of the quantity PrDmin (I(x, y),T)    Dmax (I(x,y),T)
for three AVIRIS images. They have been classified using 100 randomly selected targets. Average results
are reported for thresholds  [5,..,35] and quantizers having 256, 512 and 1024 levels. It is possible to

see how the percentage of vectors on which the simplified test fails to classify the points reaches a

maximum between 7.5% and 11% for a 1024 levels quantizer and 12% to 18% for a 256 level quantizer.
Practical figures are typically better. Problems like target detection are solved by applying a low threshold,
while classification requires a high one; in these regions of the curve, the percentages may be much lower


than the maxima. These results can be used in two ways. Given an image encoded with LPVQ, we can

improve the speed the naïve classification with the use of a lookup table. Alternatively, we can use this as a
tool to select the number of quantization levels and match a desired average classification speed
XII. CONCLUSIONS
We have presented an extension of the GLA algorithm to the locally optimal design of a partitioned
D
vector quantizer (LPVQ) for the encoding of source vectors drawn from a high dimensional source on R .
18
It breaks down the input space into independent subspaces and for each subspace designs a minimal
distortion vector quantizer. The partition is adaptively determined while building the quantizers in order to
minimize the total distortion.
Experimental results on lossless and near-lossless compression of
hyperspectral imagery have been presented, and different paradigms of near-lossless compression are
compared. Aside from competitive compression and progressive decoding, LPVQ has a natural parallel
implementation and it can also be used to implement search, analysis and classification in the compressed
data stream. High speed implementation of our approach is made possible by the use of small independent
VQ codebooks (of size 256 for the experiments reported), which are included as part of the compressed
image (the total size of all codebooks is negligible as compared to the size of the compressed image and our
experiments include this cost). Decoding is no more than fast table look-up on these small independent
tables. Although encoding requires an initial codebook training, this training may only be necessary
periodically (e.g., for successive images of the same location), and not in real time. The encoding itself
involves independent searches of these codebooks (which could be done in parallel and with specialized
hardware).
REFERENCES
[1] G. Motta, F. Rizzo, and J.A. Storer, “Compression of Hyperspectral Imagery,” in Proceedings Data
Compression Conference, J.A. Storer and M. Cohn, Eds. 2003, pp. 333-342, IEEE Computer Society
Press.
[2] D. Landgrebe, “Hyperspectral Image Data Analysis,” in IEEE Signal Processing Magazine, January
2002, pp. 17–28
[3] M. G. Kendall, A course in the Geometry of n-Dimensions. New York: Hafner, 1961.
[4] A. Gersho and R.M. Gray, Vector Quantization and Signal Compression. Kluwer Academic Press,
1991.
[5] Y. Matsuyama. “Image Compression Via Vector Quantization with Variable Dimension”, TENCON
87: IEEE Region 10 Conference Computers and Communications Technology Toward 2000, Aug.
1987, Seoul, South Korea.
[6] Y. Linde, A. Buzo, and R. Gray, “An algorithm for vector quantizer design”, IEEE Transactions on
Communications 28, 84-95, 1980.
[7] Huang, Bi, Stiles, and Harris, in IEEE Transactions on Image Processing, July 1992.
[8] I. Katsavounidis, C.-C.J. Kuo, and Z. Zhang, “A New Initialization Technique for Generalized Lloyd
Iteration,” in IEEE Signal Processing Letters, vol. 1, no. 10, pp. 144-146, October 1994
[9] JPL [2002]. NASA Web site: http://popo.jpl.nasa.gov/html/aviris.freedata.html
[10] B. Aiazzi, L. Alparone and S. Baronti, “Near–Lossless Compression of 3–D Optical Data”, IEEE
Transactions on Geoscience and Remote Sensing, vol. 39, no. 11, 2001.
19
[11] J. Mielikainen and P. Toivanen, “Clustered DPCM for the Lossless Compression of Hyperspectral
Images”, in IEEE Transactions on Geoscience and Remote Sensing, vol. 41, no. 12, pp. 2943-2946,
December 2003.
[12] F. Rizzo, B. Carpentieri, G. Motta, and J. A. Storer, “High Performance Compression of
Hyperspectral Imagery with Reduced Search Complexity in the Compressed Domain”, in Proceedings
Data Compression Conference, J.A. Storer and M. Cohn, Eds. 2004, IEEE Computer Society Press.
.
[1] G. P. Abousleman [1995]. “Compression of Hyperspectral Imagery Using Hybrid DPCM/DCT and
Entropy-Constrained Trellis Coded Quantization”, Proc. Data Compression Conference, IEEE
Computer Society Press.
[2] B. Aiazzi, L. Alparone, A. Barducci, S. Baronti and I. Pippi [2001]. “Information-Theoretic Assessment
of Sampled Hyperspectral Imagers”, IEEE Transactions on Geoscience and Remote Sensing 39:7.
[3] .
[4] .
[5] JPL [2002]. NASA Web site: http://popo.jpl.nasa.gov/html/aviris.freedata.html
[6] Y. Linde, A. Buzo, and R. Gray [1980]. “An algorithm for vector quantizer design”, IEEE Transactions
on Communications 28, 84-95.
[7] M. Manohar and J. C. Tilton [2000]. “Browse Level Compression of AVIRIS Data Using Vector
Quantization on Massively Parallel Machine,” Proceedings AVIRIS Airborne Geoscience Workshop.
[8] Y. Matsuyama [1987]. “Image Compression Via Vector Quantization with Variable Dimension”,
TENCON 87: IEEE Region 10 Conference Computers and Communications Technology Toward 2000,
Aug. 25-28, Seoul, South Korea.
[9] R. E. Roger, J. F. Arnold, M. C. Cavenor and J. A. Richards, [1991]. “Lossless Compression of
AVIRIS Data: Comparison of Methods and Instrument Constraints”, Proceedings AVIRIS Airborne
Geoscience Workshop.
[10] S. R. Tate [1997], “Band ordering in lossless compression of multispectral images”, IEEE
Transactions on Computers, April, 477-483.
[11] G. Shaw and D. Manolakis [2002], “Signal Processing for Hyperspectral Image Exploitation”, IEEE
Signal Processing Magazine 19:1
[12] S. Subramanian, N. Gat, A. Ratcliff and M. Eismann [1992]. “Real-Time Hyperspectral Data
Compression Using Principal Component Transform”, Proceedings AVIRIS Airborne Geoscience
Workshop.
[13] G. O. Young, “Synthetic structure of industrial plastics (Book style with paper title and editor),”
in Plastics, 2nd ed. vol. 3, J. Peters, Ed. New York: McGraw-Hill, 1964, pp. 15–64.
[1]
[2] NASA, “AVIRIS home page,” http://popo.jpl.nasa.gov, February 2003
[3].
[4].
[5] M. Manohar and J.C. Tilton, “Browse level compression of AVIRIS data using vector quantization on
massively parallel machine,” in Proceedings AVIRIS Airborne Geosciences Workshop, 2000.
SPURGARE LE REFERENCES ED AGGIUNGERE