1
Compression of 3D Scientific Data for Archival,
Distribution and Fast Classification
First A. Author, Second B. Author, Jr., and Third C. Author, Member, IEEE
Abstract—We propose a compression algorithm based on Partitioned Vector Quantization targeted
to 3D scientific data, remote sensing and hyperspectral imagery. Along one of its dimensions, the
input data is treated as a single vector that is partitioned into subvectors of (possibly) different
lengths, each quantized separately. Quantization indices and residual errors are entropy coded.
Partitioning serves different purposes: vector quantization looses efficiency in high dimension and
partitioning is a way of improving it, it allows a better decorrelation of the data in the given
dimension, improves compression. The concatenation of the quantization indices can be seen as a
projection to a lower dimensional subspace that preserves the original statistical properties of the
data; partition indices can be used for fast (remote) browsing and classification. In fact, experimental
results with NASA AVIRIS hyperspectral images show that the proposed algorithm achieves, on
such data, state of the art lossless compression, it provides near-lossless and lossy capabilities (that
could be used for fast browsing and broadcasting on limited bandwidth communication channels), it
is scalable to arbitrary data size, and it allows fast classification and target detection in the
compressed domain.
Index Terms—About four key words or phrases in alphabetical order, separated by commas. For a
list of suggested keywords, send a blank e-mail to keywords@ieee.org or visit the IEEE web site at
http://www.ieee.org/web/developers/webthes/index.htm.
I. INTRODUCTION
A
N
increasing number of scientific instruments produce high volume of data in the form of
two-dimensional matrices of high dimensionality vectors, also called data cubes. Typical examples are
biomedical images, multi, hyper and ultra spectral imagery, measurements of volumetric parameters like
atmospheric temperature and pressure via microwaves sounders. Compressing these data presents a
challenge for the currently standardized general purpose compressors. Beside the three dimensional nature
of the measurements, which exhibit correlation along each dimension, the single measurements have values
of 16 bits or more. Current state of the art compressors do not perform well on data sources with large
alphabets. Furthermore, it is desirable to have a single compressor supporting lossless, near lossless, and
lossy compression modes and can scale to data cubes of arbitrary sizes. Lossless compression is often
2
required for data collection and archiving due to the cost of the acquisition and transmission process and
due to the fact that original data may be needed by unforeseen analyses or further elaboration. However,
when data is downloaded by users, a lossy mode can be employed to speed up the transmission, since the
final users may not need all measurements at their full precision, and in any case, can request further
precision only when needed.
In [1] we propose a compression algorithm based on Partitioned Vector Quantization, where one
dimension of the cube is treated as a single vector, partitioned and quantized. Partitioning the vector into
subvectors of (possibly) different lengths is necessary because of the possibly large number of components.
Subvectors are individually quantized with appropriate codebooks. The adaptive partitioning uses a novel
locally optimal algorithm and provides a tool to reduce the size of the source alphabet. Correlation among
the other two dimensions is exploited with methods derived from the image coding domain. Lossless or
near-lossless entropy coding can be applied to the residual error. An additional feature of this compression
method is the possibility to tightly bound error on a pixel by pixel basis. Due to the hierarchical structure of
the data compressed with the proposed algorithm, it is possible to perform classification and target
detection directly in the compressed domain, with considerable speed up and memory savings. The high
speed classification and target detection algorithm that we describe here can process 90 percent of the
compressed pixels with a simple table lookup, full decompression is only required if exact classification is
necessary and it is limited to the 10 percent remaining pixels. The proposed algorithms have been verified
experimentally on the lossless and near lossless compression of NASA AVIRIS images where, under
conservative assumptions, is able to improve the naïve classification of a factor of five or more.
Section II discusses the communication paradigm under consideration. Section III introduces the
necessary notation and briefly reviews the proposed algorithm. Section IV discusses compression
experiments with LPVQ.
The problem of classification and the necessary notation is introduced in Section V. In this same section
experimental results on the classification error introduced by LPVQ lossy and near lossless compression
3
modes are discussed first. Building on these experimental observations, we describe an algorithm that
speeds up the classification by accepting or rejecting vectors based only on bounds of the VQ error. These
error bounds are available at design time or can be extracted from the compressed image. In the same
section, the asymptotic behavior is derived. Given a compressed image and a classification threshold, the
speed up achieved on the naïve algorithm depends on the acceptance and rejection rates. Experiments are
performed on typical AVIRIS images in order to assess these rates for a number of classification
thresholds. The acceptance and rejection rates only depend on the bounds on the quantization error and can
be easily derived during the design of the LPVQ. So, these considerations also provide a tool useful to
select the quantization parameters (codebook entries and number of partitions) in order to match a desired
range of classification speeds.
II. THE COMMUNICATION MODEL
An application for which our algorithms are designed is the encoding of information collected by a
remote sensing device having limited computational power (satellite, airplane, biomedical imager) and
losslessly transmitted to a powerful central server (see Figure 1). The central server decompresses and
elaborates the data, performing all necessary adjustments and corrections. Then, re-codes the data in a more
compact representation and distributes the result among a number of users that may subscribe the service at
different quality levels. A common example is weather information collected by geostationary satellites.
The satellites observe events evolving in a specific geographic area, such as hurricanes, storms, flooding
and volcanic eruptions. The collected data have to be transmitted to a land based central station that
analyzes the data, applies some form of enhancement, further compresses the data and sends the result back
to the satellite in order to broadcast it to the final users. In this setting, the same satellite is used for both
acquisition and broadcasting. More common is the case of the central server performing the transmission.
While the compression hardware used in the satellite must be simple and has to require very low power to
function, there is typically little or no limitation to the computational power of the base station.

4
III. DEFINITIONS
We model a three dimensional data cube as a discrete time, discrete values, bi-dimensional random
source I(x,y) that emits pixels that are D-dimensional vectors I(x,y) . Each vector component Ii (x, y) ,
0  i  D 1, is drawn from a finite alphabet Xi and is distributed according to a space variant probability
distribution that may depend on other components. We assume that the alphabet has the canonical form
Xi  0,1, ,Mi.
Our encoder is based on a Partitioned Vector Quantizer (or PVQ), which consists of N independent,
L -levels, di -dimensional exhaustive search vector quantizers Qi  (Ai,Fi,Pi ) , such that
 d  D and:
i
1 i N
d
Ai  {c1i,c i2 ,...,c iL } is a finite indexed subset of R i called codebook. Its elements c ij are the code

vectors.
d
Pi  {S1i,S2i ,...,SLi } is a partition of R i and its equivalence classes (or cells) S ij satisfy:

L
j 1
i
i
S ij  R d i and Sh  Sk   for h  k .
Fi:R  Ai is a function defining the relation between the codebook and the partition as Fi (x)  c ij
di

if and only if x S ij .
The index j of the codeword c ij , result of the quantization of the di -dimensional subvector x , is the

information that is sent to the decoder.
With reference to the previously defined N vector quantizers Qi  (Ai,Fi,Pi ) , a Partitioned Vector
Quantizer can be formally described by a triple Q  A ,P ,F  where:
D
A  A1  A2   AN is a codebook in R ;

D
P  P1  P2   PN is a partition of R ;




F :R D  A is computed on an input vector x R D as the concatenation of the independent

quantization of the N subvectors of x . Similarly, the index vector sent to the decoder is obtained
as a concatenation of the N indices.

5
The design of a Partitioned VQ aims at the joint determination of the N 1 partition boundaries
b0  0  b1 
 bN  D and at the design of the N independent vector quantizers having dimension
di  bi  bi1 , 1  i  N .
th
In the following, given a source vector I(x,y) , we indicate the i subvector of boundaries bi1 and bi 1
with the symbol Ibb ii 11 (for simplicity, the x and y spatial coordinates are omitted when clear from the
context). The mean squared quantization error between the vector I and its quantized representation Iˆ , is
given by
I  ˆI
2


i
Ji

where c  c
N
 I
b i 1
b i 1
 ˆIbbii11
 I
b i 1
b i 1
 c iJi
i1
N
i1
i
Ji ,0
, ,c
i
J i ,d i 1

2

2
 is the codeword of the i
b i 1
  I
N

i 1 h b i 1
th
h
 c Ji i ,h b i 1

2
codebook minimizing the reconstruction error on
Ibb ii 11 , and:

b 1
i
J i  argmin MSE Ib ii 1 ,c l
1l  L

A. A Locally Optimal PVQ Design
The complexity of building a quantizer that scales to vectors of arbitrary dimension is known to be
computationally prohibitive. Furthermore, the efficiency of a VQ decreases with its dimensionality: the
fraction of the volume of a hypersphere inscribed in a hypercube of the same dimension d goes to zero
when d goesto infinity
 ([2]-[3]). Partitioning the vectors in consecutive, non-overlapping subvectors is a
common solution to this problem (see the book of Gersho and Gray [4]). While a partitioned VQ leads to a

sub-optimal solution in terms
 of Mean Squared Error (MSE) because it does not exploit correlation among
subvectors, the resulting design is practical and coding and decoding present a number of advantages in
terms of speed, memory requirements and exploitable parallelism.
In a Partitioned VQ, we divide the input vectors into N subvectors and quantize each of them with an
L -levels, exhaustive search VQ. Having N subvectors of the same dimension (except perhaps one, if N
does not divide D) is generally not a possibility since, in this context, we assume that the components of
6
I(x,y) may be drawn from different alphabets. In this case their distributions may be significantly different
and partitioning the D components uniformly into N blocks may not be optimal. We wish to determine the
size of the N sub vectors (of possibly different size) adaptively, while minimizing the quantization error,
measured for example in terms of MSE. Once the N codebooks are designed, input vectors are encoded by
partitioning them into N subvectors of appropriate length, each of which is quantized independently with
the corresponding VQ. The index of the partitioned vector is given by the concatenation of the indices of
the N subvectors.
Given the number of partitions N and the number of levels per codebook L, it is possible to find the
partition boundaries achieving minimum distortion with a brute-force approach: first determine, for every
0  i  j  D , the distortion Dist(i, j) that an L -levels vector quantizer achieves on the input subvectors of
boundaries i and j . Then, with a dynamic programming approach, traverse the matrix Dist(i, j) and find the
N costs corresponding to the input partition of boundaries b0  0  b1 
 bN  D and whose sum is
minimal. (A more sophisticated approach to partitioned vector quantization is discussed in Matsuyama [5].
It uses dynamic programming at each step to decide the current optimal partition, instead of designing one
vector quantizer for each possible partition configuration first and then applying dynamic programming.
Nevertheless, even this approach is computational intensive.)
The locally optimal partitioning algorithm that we propose in this paper (LPVQ) provides an efficient
alternative to dynamic programming, while performing comparably in most applications. Our partitioning
algorithm is based on a variation of the Generalized Lloyd Algorithm (or GLA, see Linde, Buzo and Gray
[6]).
Unconstrained vector quantization can be seen as the joint optimization of an encoder (the function
F:R d  A described before) and a decoder (the determination of the codewords for the equivalence
classes of the partition P  {S1 ,S2 ,...,SL }). GLA is an iterative algorithm that, starting from the source
sample vectors chooses a set of codewords (also called centroids) and optimizes in turns encoder and
decoderuntil improvements on the predefined distortion measure are negligible. In order to define our

7
PVQ, the boundaries of the vector partition b0  0  b1 
 bN  D have to be determined as well. Our
design follows the same spirit of the GLA. The key observation is that, once the partition boundaries are
kept fixed, the Mean Square Error (the distortion we use in our application) is minimized independently for
each partition by applying the well-known optimality conditions on the centroids and on the cells.
Similarly, when the centroids and the cells are held fixed, the (locally optimal) partitions boundaries can be
determined in a greedy fashion. The GLA step is independently applied to each partition. The equivalence
classes are determined as usual, but as shown in Figure 2, the computation keeps a record of the
contribution to the quantization error of the leftmost and rightmost components of each partition:


2


i   Ib (x, y)  Iˆb (x,y) and i   I b 1 (x,y)  Iˆb 1 (x, y)
x ,y
i 1
i 1
x,y
i
i
2
Except for the leftmost and rightmost partition, two extra components are also computed:

i   I b
x,y
i 1 1



2
(x,y)  Iˆb i 1 1 (x, y) and i   I b i (x,y)  Iˆb i (x, y)
x,y
2
The reconstruction values used in the expressions for i and i are determined by the classification
performed on the components bi1,...,bi . The boundary bi1 between the partitions i 1 and i is changed
according to the criteria
Mi = min( i 1   i , i 1  i 1 , i   i )
if ( Mi = i   i )
bi1  bi 1 1
else if ( Mi = i 1  i 1 )
bi1  bi 1 1
IV. DATA COMPRESSION WITH LPVQ
The partitioned vector quantizer is used here as a tool to reduce the dimensionality of the source vectors.
Dependence between the other two dimensions has to be exploited by means of an entropy coder.
Furthermore, when lossless coding is required, quantization residuals have to be entropy coded with a
conditioning on the subvector indices.
i

i
i

The index vectors J(x,y) and the codewords c j  c j ,b i , ,c j ,b i 1 1 , with 1  i  N and 1  j  L , are
sufficient to a lossy reconstruction of the data. When higher accuracy is needed, the compressed data can be

8
augmented with the quantization error:


E(x, y)  Ebb10 1 (x, y),E bb21 1 (x, y),...,Ebb NN 11 (x, y)
where, for each 1  i  N :
E bbii 11 (x,y)  Ibb ii 11 (x, y)  ˆIbbii11 (x, y)
The unconstrained quantizers work independently from each other and independently on each source
vector and an entropy encoder must be used to exploit this residual redundancy. In particular, each VQ
index Ji(x,y) is encoded by conditioning its probability with respect to a set of causal indices spatially and
spectrally adjacent. The components of the residual vector E bbii 11 (x,y) are entropy coded with their
probability conditioned on the VQ index Ji(x,y) . In near-lossless applications, a small, controlled error can
be introduced at this stage. Figure 3 depicts a diagram of the LPVQ encoder.
A. Computational Complexity
Vector Quantization is known to be computationally asymmetrical, in the sense that encoding has a
computational complexity which is orders of magnitude higher than decoding. Even more time consuming
is the design of the dictionary which is also not guaranteed to converge into an easily predictable number of
iterations. Numerous methods have been designed to speed up the convergence, for example, by
constraining the search of the closest codeword like in [7], or by seeding the dictionary with vectors having
special properties [8]. Adding structure to the VQ can also simplify encoding at the price of a limited
degradation of the output quality [4]. All these improvements clearly apply to our method as well;
furthermore, practical scenarios arise in which our algorithm can be applied as is, without the
computational complexity being an obstacle.
A possible way of mapping the LPVQ algorithm to the communication model described in Section II is
to equip the platform used for the remote acquisition with an encoder that is able to download a dictionary
transmitted from the central server (see Figure 1). This dictionary can be, for the very first scene,
bootstrapped as described in [8], with a number of vectors randomly selected from the image itself, with a
dictionary designed for another image, or with any dictionary that is believed to be statistically significant.
9
Due to the nature of our algorithm, the choice of the first dictionary will only impact the compression ratio
of the very first transmission.
Once the central server receives a scene, at time instant t, a few iterations of LBG may be necessary to
refine the dictionary, to adjust the partition boundaries and to improve compression. The refined version of
the dictionary is transmitted back to the remote platform and is used for the encoding of the scene acquired
at time t+1. Later in this section we report on experiments intended to assess the performance degradation
derived from a mismatched dictionary.
It has been observed in [1] that if the codewords in each codebook are sorted according to their energy,
the N planes constituted by the indices retain important information on the structure of the original image.
These planes alone can be used to browse and select areas of interest and, as we show in Section VI, to
extract important information on the nature of the original pixel vectors.
B. Experimental results
LPVQ has been tested on a set of five AVIRIS images ([9]). AVIRIS images are obtained by flying a
spectrometer over the target area. They are 614 pixels wide and typically have a height on the order of
2,000 pixels, depending on the duration of the flight. Each pixel represents the light reflected by a 20 x 20
meters area (high altitude) or 4 x 4 meters area (low altitude). The spectral response of the reflected light is
decomposed into 224 contiguous bands (or channels), approximately 10nm wide, spanning from visible to
near infrared light (400nm to 2500nm). Each of these vectors provides a spectral signature of the area.
Spectral components are acquired in floating point 12-bit precision and then scaled and packed into signed
16 bit integers. After acquisition, AVIRIS images are processed to correct various physical effects
(geometrical distortion due to the flight trajectory, time of day, etc.) and stored in scenes of 614 by 512
pixels each (when the image height is not a multiple of 512, the last scene will be smaller). All files for
each of the 5 test images can be downloaded from the NASA web site (JPL [9]) and the scenes belonging
to the same flight are merged together to form a complete image.
Several experiments have been performed for various numbers of partitions and for different codebook
10
sizes. The results that we describe here were obtained for N 16 partitions and L  256 codebook levels.
The choice of the number of levels makes also practical the use of off-the-shelf image compression tools
that are fine-tuned for 8 bit data. The LPVQ algorithm trains on each image independently and the
codebooks are sent to the decoder as side information. The size of the codebook is negligible with respect
the size of the compressed data (256 x 224 x 2 bytes = 112 KB) and its cost is included in the reported
results.
The partition boundaries for each of the five images are depicted in Figure 4. While similarities exist, the
algorithm converges to different optimal boundaries on different input images. This is evidence that LPVQ
adapts the partitions to input statistics. Starting with a codebook obtained by random image pixels, we have
found that adaptation is fairly quick and boundaries converge to their definitive values in less than one
hundred iterations.
If LPVQ is bootstrapped with the method described in [8], which populates the dictionary with the vector
with highest norm and then adds the vector whose minimum distance from the current dictionary is
maximum, the algorithms shows a much faster convergence. Furthermore, the constrained search for the
closest codeword is generally faster by a factor of 2.

LPVQ performances are analyzed in terms of Compression Ratio (defined as the size of the original file
divided by the size of the compressed one), Signal to Quantization Noise Ratio, Maximum Absolute Error
and Percentage Maximum Absolute Error.
The Signal to Quantization Noise Ratio (SQNR) is defined here as:
2
10 D 1
  Ii 
log
2
10   1 
Ei
12
D
i 0
is introduced to take into account the error introduced by the 12 bit
SQNR (dB) 
The correction factor
1
12
analog-to-digital converter used by the AVIRIS spectrometer ([9]). This solution also avoids unbounded
values in the case of a band perfectly reconstructed.
th
The Maximum Absolute Error (MAE) is defined in terms of the MAE for the i band
as MAE  max MAE i , where MAEi is:
i
11
MAE i  max Ii (x,y)  Iˆi (x, y)
x ,y
th
The average Percentage Maximum Absolute Error (PMAE) for the i band having canonical alphabet
Xi  0,1, ,Mi is defined as:
1 D 1 MAE i
 100
D
Mi
i 0
Table I shows that LPVQ achieves on this five images an average compression of 3.14:1. This is 45%
PMAE (%) 
better than the 1-D lossless compressor bzip2 when applied on the plane–interleaved images (worse results
are achieved by bzip2 on the original pixel–interleaved image format) and 55% better than the standard
lossless image compressor JPEG-LS. We also compared LPVQ with the method published in [11], even
though it reports experiments on a subset of our data set. The algorithm in [11] uses LBG to cluster data.
An optimized predictor is computed for each cluster and for each band. Prediction error is computed and
entropy coded, along with side information regarding the optimized predictors. The resulting method
outperforms LPVQ, while sharing time complexity similar to the design stage of LPVQ and lacking any
extra feature provided by the latter, like lossy and near-lossless mode, fast browsing and classification, et
cetera. The last column of Table I reports, as a reference, the compression and the SQNR when only the

indices are encoded and the quantization error is fully discarded. As we can see from the table, on average
we achieve 55:1 compression with 25.27dB of SQNR.
More interesting and practical are the results obtained with the near-lossless settings, shown in Table II.
At first, the introduction of a small and constant quantization error across each dimension is considered;
that is, before entropy coding, each residual value x is quantized by dividing x adjusted to the center of the
x 
 . This is the classical approach to the near-lossless
2
   1
range by the size of the range; i.e., q( x)  
compression of image data and results into a constant MAE across all bands. With this setting, it is possible
to reach an average compression ratio ranging from 4:1 with the introduction of an error   1 and a
MAE  1 to 10:1 with and error of   10 and MAE  10 . While the performance in this setting seem to
be acceptable for most applications and the SQNR is relatively high even at high compression, the analysis
12
of the contribution to the PMAE of the individual bands shows artifacts that might be unacceptable. In
particular, while the average PMAE measured across the 224 bands of the AVIRIS cube is low, the
percentage error peaks well over 50% on several bands (see Figure 5). Since the PMAE is relevant in
predicting the performance of many classification schemes, we have investigated two different approaches
aimed at overcoming this problem. In both approaches we select a quantization parameter that is different
for each band and inversely proportional to the alphabet size (or dynamic). In general, high frequencies,
having in AVIRIS images higher dynamic, will be quantized more coarsely than low frequencies. We want
this process to be governed by a global integer parameter .
th
The first method, aiming at a quasi–constant PMAE across all bands, introduces on the i band a
distortion i such that:
1 D 1
i  
D
i 0
th
Since the i band has alphabet Xi  0,1, ,Mi we must have:
 D  
1 
 i   D 1  Mi  
 M
2 
 i 0 i

The alternative approach, aims at a quasi–constant SQNR across the bands. If we allow a maximum
th
absolute error  i on the i band, it is reasonable to assume that the average absolute error on that band
will be
i
. If we indicate with i the average energy of that band and with  the target average maximum
2
absolute error, then the absolute quantization error allowed on each band is obtained by rounding to the
nearest integer the solution of this system of equations:

 2j
 2i
10log

10log
i, j 0, ,D  1, i  j

10
2
10
2

i 2
j 2

D 1
 1


 
 i
 D i 0
As can be seen from Table II, the three methods for near-lossless coding of AVIRIS data are equivalent
in terms of average SQNR at the same compression. However, the quasi-constant PMAE method is indeed
13
able to stabilize the PMAE across each band (Figure 6). The small variations are due to the lossless
compression of some bands and the rounding used in the equations. The average SQNR is not
compromised as well. Similar results are observed for the quasi-constant SQNR approach. The SQNR is
almost flat (Figure 7), except for those bands that are losslessly encoded and those with small dynamic. The
PMAE is also more stable than the constant MAE method (Figure 8).
C. LPVQ and Random Dictionaries
In order to evaluate the proposed approach in the framework described in Section II, we performed the
following experiments for each scene of the Moffett Field image:
1) we compute the compression ratio and RMSE at each step of the LPVQ algorithm, starting with a
random dictionary;
2) same except that the starting dictionary is the locally optimal one for the previous scene.
As we can see from Figure 9, compression performance is already quite stable after 50 iterations. The
compression ratio is slightly lower when LPVQ is fed with the dictionary from the previous scene, while
RMSE, on the other hand, is much lower. This suggests that the entropy coder needs to be tuned-up for this
case and that the lossy compression is slightly better when only few iterations are allowed.
A different experimental setting uses six different schemes to generate random dictionaries. One hundred
dictionaries for each scheme are used with PVQ (i.e., the dictionary is not refined with the proposed LBGlike scheme to obtain a locally optimal one), and with both uniform and optimal partitions. Compression
results for each scheme are box-plotted in Figure 10. The results suggest that taking a small sample of the
input data is enough to obtain compression performances reasonably close to the locally optimal solution
produced by LPVQ, and that the non-uniform partitioning is much more effective at high compression.
V. FAST CLASSIFICATION WITH LPVQ
Let each image pixel be a D-dimensional vectors I( x, y)   I 0
disjoint subvectors as I( x, y)   I0
I1
I1
I D 2
I D1  divided into N
I N 2 I N 1  , the n -th partition having dimension dn , for


14
0  n  N . For simplicity the boundaries are omitted here.
Upon
quantization
J( x, y)   J 0
J1
each
J N 2
vector
is
represented
J N 1  and by E( x, y)   E0
by
E1
an
N -dimensional vector of indices
ED2
ED1  , a D-dimensional vector
representing the quantization error. Each component J(x,y) indexes one of the L codebook centroids for
the VQ corresponding to the partition n , 0  n  N . If Iˆk is the quantized value for Ik , LPVQ decomposes
each component of an input vector I(x,y) into as Ik  Iˆk  Ek , where 0  k  D .
Given a target vector T   t0
t1
tD2 tD1  , a distortion measure D(I,T) and a classification
threshold , we wish to determine, for every image pixel I(x,y) , whether D(I(x, y),T)   . When this
condition is satisfied, we say that the pixel I(x,y) is a member of the class T .
The Euclidean Distance ED(I,T) 
D 1
 I
k 0
k
 tk 
2
is the distortion measures commonly used in
classification.
The classification threshold , plays an important role since it determines whether the classification is
used to detect a rare pixel instance, eventually present in the image (problem commonly referred to as
target detection) or whether the objective of the classification is the determination of a thematic map,
where almost every pixel belongs to one of several classes.

A. Effect of Coding Error on the Classification

As we did in the case of compression, we used AVIRIS hyperspectral images to assess the effect of the

coding error on classification. Classification and target detection are algorithms frequently applied to
hyperspectral images; when the images are lossily encoded, it is licit to ask whether and in which measure

the coding error affects the results of these algorithms.
 In order to assess the classification errors introduced by LPVQ lossy modes, we have experimented with
 the standard images for various errors and threshold parameters. It is important to notice that, while the
lossy results are peculiar of our algorithm, the near lossless apply to any compression scheme that bounds
15
the maximum absolute error on each vector component. The results for three images, Cuprite, Moffet Field
and Low Altitude, are presented here. Results on the other two images were very similar.
In our experiments, the absolute error introduced by LPVQ in near lossless mode is respectively bounded
to a maximum of 1, 2, 4 and 8 units. The classification uses Euclidean Distance with an average threshold
per band  [0,..,50] ; typically, values in the range  [5,..,35] are of some interest in practical
applications. The figures show the percentage of errors in the classification of 8 targets selected from each
image. The targets are hand picked from the reduced resolution image constituted by the index planes and
are meant to represent targets of common interest (buildings, roads, water, vegetation, etc...). The lossless
encoding is used as a reference for the classification and the error percentages reflect the sum of all false
positive (classifications that were not classified in the lossless), false negative (vectors classified in the
lossless and not classified in the lossy) and misclassifications (different clusters in lossless and lossy).
As it is possible to see from Figures 11-13, in near lossless, the classification is fundamentally
unaffected; even for an error of 4 , the percentage of classification errors is, in the worst case, below
0.1%. This is sufficient for most applications. Smaller coding errors, achieve even lower percentages. For
the compression parameters used in Section VII (i.e., 256 code vectors and 16 partitions) the indices and
the code vectors alone (Figure 14, lossy case) achieve a classification error always lower than 3%. While
this error rate may be unsuitable for some applications, as we show in the next section, it is possible to take
advantage of this characteristic to design a faster classifier.
B. Speeding up the Classification
Since the LPVQ indices provide most of the information necessary to classify the pixels, we designed a
simple algorithm that, bounding the minimum and maximum quantization error for each component and for
each cluster, can decide the classification of most pixels without inspecting their error vector E(x, y) .
Bounds on the quantization error are available at design time or can be inferred from the encoded image.
The advantages of this algorithm are twofold. First: the number of partitions and the number of centroids
are typically small, so pre-computed values can be arranged into a lookup table. If most vectors can be
16
classified without the need of the error, this will result into a substantial speed up. Second: we can imagine
a user on the field, browsing and trying different classifications on the LPVQ indices instead of the whole
image. Then, when a parameterization of interest is found, the user can query the server for the error
vectors necessary to resolve any classification ambiguity.
Given a threshold , a target vector T and a distortion measure D(I(x, y),T) , we wish to determine,
from the quantization indices of a pixel I(x,y) , two quantities Dmin(I(x, y),T) and Dmax(I(x, y),T) ,
respectively lower and upper bounding the distortion D(I(x, y),T) .
Since I(x,y) belongs to the class T only if D(I(x, y),T)   , we can observe that:

If Dmin(I(x, y),T)   , the pixel I(x,y) clearly does not belong to the class T ;

If Dmax(I(x, y),T)   , the pixel I(x,y) trivially belongs to the class T ;

If Dmin(I(x, y),T)    Dmax (I(x,y),T) then the quantization indices and the bounds on the error
are not sufficient to decide the classification. In this case, and only in this case, it is necessary to
access the error vector and compute D(I(x, y),T)   .
Figure 15 shows the pseudocode that implements this scheme. The main assumption is that the quantities
Dmin(I(x, y),T) and Dmax(I(x, y),T) can be computed by summing the individual contributions of each
vector component k , upper and lower bounded by the minimum and maximum of the quantization
residuals min( E k,l ) and max( Ek,l ) .
This is clearly the case of Euclidean Distance, here defined as:


ED(I(x,y),T) 
D 1
 I
 tk 
2
k
k 0

2
Changing the threshold  into  , we can focus on the contribution of min( E k,l ) and max( Ek,l ) to the
individual terms of the sum. For a given component k , 0  k  D , in the partition n , 0  n  N we have:
I k  tk 2  Iˆk  E k  t k  .
2

If Ik has been quantized with the centroid l , 0  l  L :

17

  
 

  
 
2
2
2


min  Iˆk  min( E k ,l )  tk , Iˆk  max( E k,l )  tk  Ik  tk 


and
2
2
2


max  Iˆk  min( E k ,l )  tk , Iˆk  max( E k,l )  tk  Ik  tk  .


Summing these bounds over all components of the partition n we obtain Dmin (Cl ,n,T) and Dmax (Cl ,n ,T) .
These quantities are precomputed in the algorithm in Figure 15 by the two nested “for” loops. Since there
are only 2  L N possible values and N and L are typically small ( N 16 and L  256 in Section VII),
Dmin (Cl ,n,T) and Dmax (Cl ,n ,T) can be stored in a lookup table indexed by n and l . The computation of
Dmin(I(x, y),T) and Dmax(I(x, y),T) requires a total of 2  N lookups and 2 (N  1) sums. If one of the two
comparisons Dmin(I(x, y),T)   and Dmax(I(x, y),T)   succeeds, then the pixel I(x,y) is classified from
its indices only, with a number of operations proportional to N .
Otherwise, the error vector
E(x, y) must be retrieved,
I(x,y)
exactly reconstructed as
I(x,y)  J(x, y)  E(x, y) and finally D(I(x, y),T)   computed. This requires a sum a subtraction and a
product for each component and D1 sums to accumulate the result, i.e. a total of D subtractions, 2  D1
sums and D products, a number of operations proportional to D (for AVIRIS images, D  224 ).
The speed of the new classification algorithm depends on the probability that the two tests fail and that
we have to use the error vector in order to classify. Let this probability be:

PrDmin (I(x, y),T)    Dmax (I(x,y),T)

If sub ,  add and mul are respectively the times to perform a subtraction, a sum and a multiplication on a
given computing platform, the total running time of the new classification is:




2 (N 1)  add  D ( sub  mult )  (2 D 1)  add PrDmin (I(x, y),T)    Dmax (I(x,y),T)
Figures
 16-18 show an experimental assessment of the quantity PrDmin (I(x, y),T)    Dmax (I(x,y),T)
for three AVIRIS images. They have been classified using 100 randomly selected targets. Average results
are reported for thresholds  [5,..,35] and quantizers having 256, 512 and 1024 levels. It is possible to

see how the percentage of vectors on which the simplified test fails to classify the points reaches a

maximum between 7.5% and 11% for a 1024 levels quantizer and 12% to 18% for a 256 level quantizer.
18
Practical figures are typically better. Problems like target detection are solved by applying a low threshold,
while classification requires a high one; in these regions of the curve, the percentages may be much lower
than the maxima. These results can be used in two ways. Given an image encoded with LPVQ, we can
improve the speed the naïve classification with the use of a lookup table. Alternatively, we can use this as a
tool to select the number of quantization levels and match a desired average classification speed
VI. CONCLUSIONS
We have presented an extension of the GLA algorithm to the locally optimal design of a partitioned
D
vector quantizer (LPVQ) for the encoding of source vectors drawn from a high dimensional source on R .
It breaks down the input space into independent subspaces and for each subspace designs a minimal
distortion vector quantizer. The partition is adaptively determined while building the quantizers in order to
minimize the total distortion.
Experimental results on lossless and near-lossless compression of
hyperspectral imagery have been presented, and different paradigms of near-lossless compression are
compared. Aside from competitive compression and progressive decoding, LPVQ has a natural parallel
implementation and it can also be used to implement search, analysis and classification in the compressed
data stream. High speed implementation of our approach is made possible by the use of small independent
VQ codebooks (of size 256 for the experiments reported), which are included as part of the compressed
image (the total size of all codebooks is negligible as compared to the size of the compressed image and our
experiments include this cost). Decoding is no more than fast table look-up on these small independent
tables. Although encoding requires an initial codebook training, this training may only be necessary
periodically (e.g., for successive images of the same location), and not in real time. The encoding itself
involves independent searches of these codebooks (which could be done in parallel and with specialized

hardware).
19
REFERENCES
[1] G. Motta, F. Rizzo, and J.A. Storer, “Compression of Hyperspectral Imagery,” in Proceedings Data
Compression Conference, J.A. Storer and M. Cohn, Eds. 2003, pp. 333-342, IEEE Computer Society
Press.
[2] D. Landgrebe, “Hyperspectral Image Data Analysis,” in IEEE Signal Processing Magazine, January
2002, pp. 17–28
[3] M. G. Kendall, A course in the Geometry of n-Dimensions. New York: Hafner, 1961.
[4] A. Gersho and R.M. Gray, Vector Quantization and Signal Compression. Kluwer Academic Press,
1991.
[5] Y. Matsuyama. “Image Compression Via Vector Quantization with Variable Dimension”, TENCON
87: IEEE Region 10 Conference Computers and Communications Technology Toward 2000, Aug.
1987, Seoul, South Korea.
[6] Y. Linde, A. Buzo, and R. Gray, “An algorithm for vector quantizer design”, IEEE Transactions on
Communications 28, 84-95, 1980.
[7] Huang, Bi, Stiles, and Harris, in IEEE Transactions on Image Processing, July 1992.
[8] I. Katsavounidis, C.-C.J. Kuo, and Z. Zhang, “A New Initialization Technique for Generalized Lloyd
Iteration,” in IEEE Signal Processing Letters, vol. 1, no. 10, pp. 144-146, October 1994
[9] JPL [2002]. NASA Web site: http://popo.jpl.nasa.gov/html/aviris.freedata.html
[10] B. Aiazzi, L. Alparone and S. Baronti, “Near–Lossless Compression of 3–D Optical Data”, IEEE
Transactions on Geoscience and Remote Sensing, vol. 39, no. 11, 2001.
[11] J. Mielikainen and P. Toivanen, “Clustered DPCM for the Lossless Compression of Hyperspectral
Images”, in IEEE Transactions on Geoscience and Remote Sensing, vol. 41, no. 12, pp. 2943-2946,
December 2003.
[12] F. Rizzo, B. Carpentieri, G. Motta, and J. A. Storer, “High Performance Compression of
Hyperspectral Imagery with Reduced Search Complexity in the Compressed Domain”, in Proceedings
Data Compression Conference, J.A. Storer and M. Cohn, Eds. 2004, IEEE Computer Society Press.
.
[1] G. P. Abousleman [1995]. “Compression of Hyperspectral Imagery Using Hybrid DPCM/DCT and
Entropy-Constrained Trellis Coded Quantization”, Proc. Data Compression Conference, IEEE
Computer Society Press.
[2] B. Aiazzi, L. Alparone, A. Barducci, S. Baronti and I. Pippi [2001]. “Information-Theoretic Assessment
of Sampled Hyperspectral Imagers”, IEEE Transactions on Geoscience and Remote Sensing 39:7.
[3] .
[4] .
[5] JPL [2002]. NASA Web site: http://popo.jpl.nasa.gov/html/aviris.freedata.html
[6] Y. Linde, A. Buzo, and R. Gray [1980]. “An algorithm for vector quantizer design”, IEEE Transactions
on Communications 28, 84-95.
[7] M. Manohar and J. C. Tilton [2000]. “Browse Level Compression of AVIRIS Data Using Vector
Quantization on Massively Parallel Machine,” Proceedings AVIRIS Airborne Geoscience Workshop.
[8] Y. Matsuyama [1987]. “Image Compression Via Vector Quantization with Variable Dimension”,
TENCON 87: IEEE Region 10 Conference Computers and Communications Technology Toward 2000,
Aug. 25-28, Seoul, South Korea.
[9] R. E. Roger, J. F. Arnold, M. C. Cavenor and J. A. Richards, [1991]. “Lossless Compression of
AVIRIS Data: Comparison of Methods and Instrument Constraints”, Proceedings AVIRIS Airborne
Geoscience Workshop.
20
[10] S. R. Tate [1997], “Band ordering in lossless compression of multispectral images”, IEEE
Transactions on Computers, April, 477-483.
[11] G. Shaw and D. Manolakis [2002], “Signal Processing for Hyperspectral Image Exploitation”, IEEE
Signal Processing Magazine 19:1
[12] S. Subramanian, N. Gat, A. Ratcliff and M. Eismann [1992]. “Real-Time Hyperspectral Data
Compression Using Principal Component Transform”, Proceedings AVIRIS Airborne Geoscience
Workshop.
[13] G. O. Young, “Synthetic structure of industrial plastics (Book style with paper title and editor),”
in Plastics, 2nd ed. vol. 3, J. Peters, Ed. New York: McGraw-Hill, 1964, pp. 15–64.
[1]
[2] NASA, “AVIRIS home page,” http://popo.jpl.nasa.gov, February 2003
[3].
[4].
[5] M. Manohar and J.C. Tilton, “Browse level compression of AVIRIS data using vector quantization on
massively parallel machine,” in Proceedings AVIRIS Airborne Geosciences Workshop, 2000.
SPURGARE LE REFERENCES ED AGGIUNGERE