RECONSTRUCTING AND SEGMENTING HYPERSPECTRAL IMAGES FROM
COMPRESSED MEASUREMENTS
Qiang Zhanga Robert Plemmonsb David Kittlec David Bradyc Sudhakar Prasadd
a
Biostatistical Sciences, Wake Forest University, Winston-Salem, NC 27157
b
Computer Science and Mathematics, Wake Forest University, Winston-Salem, NC 27106
c
Electrical and Computer Engineering, Duke University, Durham, NC 27708
d
Physics and Astronomy, University of New Mexico, Albuquerque, NM 87131
ABSTRACT
A joint reconstruction and segmentation model for hyperspectral data obtained from a compressive measurement
system is described. Although hyperspectral imaging (HSI)
technology has incredible potential, its utility is currently limited because of the enormous quantity and complexity of the
data it gathers. Yet, often the scene to be reconstructed from
the HSI data contains far less information, typically consisting of spectrally and spatially homogeneous segments that
can be represented sparsely in an appropriate basis. Such
vast informational redundancy thus implicitly contained in
the HSI data warrants a compressed sensing (CS) strategy
that acquires appropriately coded spectral-spatial data from
which one can reconstruct the original image more efficiently
while still enabling target identification procedures. A codedaperture snapshot spectral imager (CASSI) that collects compressed measurements is considered here, and a joint reconstruction and segmentation model for hyperspectral data obtained from CASSI compressive measurements is described.
Promising test results on simulated and real data are reported.
Index Terms— Hyperspectral images, compressive measurements, reconstruction, segmentation, classification, target
identification.
1. INTRODUCTION
Hyperspectral images are digital images, often taken either
from an airplane or satellite, in which each pixel records not
just the usual three visible bands of light (red at 650nm, green
at 550nm, and blue at 450nm), but on the order of hundreds
of wavelengths so that spectroscopy can be conducted on the
materials in the object or scene. In air to ground remote sensing, the user is then able to identify, for instance, the species
of trees and other vegetation, crop health, mineral and soil
composition, moisture content of soils and vegetation, and
pollution quantities. The technology also has clear military
Research supported by the U.S. Air Force Office of Scientific Research
(AFOSR). Corresponding author: R. Plemmons, plemmons@wfu.edu,
http://www.wfu.edu/˜plemmons
and homeland-security applications, as it enables identification of targets such as buildings and vehicles, even with attempts to camouflage, as well as objects in space. It is also
possible to detect and identify gas plumes such as those arising from leaks even when the gases are invisible to the human
eye. In fact, hyperspectral imaging was used following the
attack on the twin towers and the hurricane Katrina disaster
to identify dangerous gas leaks, providing guidance and protection to rescuers. We are also working with applications to
space situational analysis for the U.S. Air Force [1, 2], where
the targets are space objects including assets, such as satellites, monitored from the ground by hyperspectral imaging
systems. More generally, see [3] and the references therein
for a comprehensive overview of recent hyperspectral data
analysis and target identification trends.
2. JOINT RECONSTRUCTION AND
SEGMENTATION OF SPECTRAL IMAGES TAKEN
BY COMPRESSIVE MEASUREMENTS
In many applications image targets are sparse in the sense
that in some basis they typically occupy a small fraction of
the overall region of interest (the target domain). This sparsity assumption suggests approaching the imaging problem
by using the framework of compressed sensing. At the core
of compressed sensing lies the following problem (here we
focus, as is common in the compressed sensing community,
on the discrete setting). Assume f ∈ Rn is a signal that is
sparse measured by the `0 quasi-norm; i.e., the number of its
nonzero components in f is much less than n, i.e., kf k0 n.
Letting g ∈ Rm be the measurement data vector, then the
compressive sensing linear inverse problem forward model
can be expressed as Hf = g, where H is an m × n system sensing matrix with m n. The goal is to recover f ,
given the data vector g and the sensing matrix H. As m n,
the linear system Hf = g is severely underdetermined and
a unique reconstruction of f is in general impossible. However, due to the sparsity of f one can compute f by solving
the optimization problem
min kf k0 subject to Hf = g.
(1)
Since (1) is NP-hard and thus may be computationally intractable, one often considers instead its convex relaxation,
min kf k1 subject to Hf = g,
(2)
which can be solved by linear and quadratic programming
methods.
Compressive sensing and compressive representation in
imaging are currently very active research areas, with important contributions by, e.g. Candès, Romberg, and Tao [4],
Donoho [5], among others. In practice, it is important to realize that compressive sensing can be done only by a compressive sensor. Our purpose in this section is to outline our work
on reconstructing hyperspectral data obtained by compressive
sensors. such as the coded aperture snapshot spectral imaging
(CASSI) systems being developed by David Brady, et al. [6].
In the conventional acquisition of hyperspectral data,
technologies such as tunable filters or computed tomography
measure every element in a hyperspectral data cube at least
once, and can require large acquisition times, data storage,
and transfer times. Recently proposed compressive imagers
such as CASSI [7, 6, 8] only need take a single snapshot from
which to reconstruct a hyperspectral data cube assumed to
be sparse in some basis. See Figure 1 for a photograph, with
brief system details, of such a CASSI system developed at
Duke University by David Brady and his group.1
The Duke CASSI hyperspectral compressive sensing
cameras have been developed with two modalities, a dual disperser form (DD-CASSI) which can be used to obtain good
spatial resolution possibly at the cost of spectral resolution
[7], and a single disperser version (SD-CASSI) which emphasizes spectral resolution in the reconstructed data [8]. Each
modality requires solving an ill-posed inverse problem for the
numerical reconstruction of the hyperspectral data from the
compressive measurements. We have found that combining
reconstruction with segmentation leads to promising results,
and can often better facilitate target identification from the
estimated data. Technical details on our joint reconstruction/segmentation methods are provided in [10].
Here, for simplicity we concentrate primarily on the DDCASSI system. Although compressed measurements, say
from DD-CASSI, need much less acquisition time and data
storage, they demand strong algorithms for data reconstruction, a problem which is usually highly under-determined.
For example, if the size of a vectorized hyperspectral cube
f is n1 × n2 × n3 , a DD-CASSI system [7] measures only
a 2D vectorized image g with size n1 × n2 , from which we
generally need to reconstruct an estimation of the original
cube for data analysis. Here, the matrix H is the CASSI
system operator having size n1 n2 × n1 n2 n3 . Since there
are fewer rows than columns in H, additional constraints are
generally needed to seek a useful estimation. For the forward
Fig. 1. A CASSI snapshot spectral imaging camera.
model of the CASSI system, g = Hf = SCf , where C is an
aperture coding matrix, and S reflects dispersive shifting and
integration over the wavelengths λ.
In many practical situations the signals to reconstruct are
composed of relatively homogeneous segments or clusters,
e.g. hyperspectral images in remote sensing problems [1],
and in 3D computed tomography images [11]. The basis of
concern for us here is the set of segmentation membership
functions in the two spatial dimensions, which take on values
of either 0 or 1 for hard segmentation or within the interval
[0, 1] for fuzzy segmentation.
Variational segmentation algorithms have become a large
area of research, see e.g. [12, 13], but they are often applied
either directly on the original measurements or after the reconstruction. In contrast, work is beginning on combining
reconstruction and segmentation in a more general linear inverse problem setting. In particular, Li et al. [2, 14] have
coupled the segmentation and deblurring/denoising models to
simultaneously segment and deblur/denoise degraded hyperspectral images. Also, Ramlau and Ring [15] have jointly
reconstructed and segmented Radon transformed tomography
data. However, neither approach tested compressive measurements from real sensors.
In our joint reconstruction/segmentation approach, we
seek a particular form of the compressed sensing solution
that takes advantage of spectrally homogeneous segments in
the two spatial dimensions, thus greatly reducing the number
of unknowns. First, assume the solution f is composed of
a limited number of segments or materials, each of which
essentially has a homogeneous value at each spectral channel. Thus we seek a decomposed solution described in a
continuous form as
f (x, y, λ) =
L
X
ui (x, y)si (λ),
(3)
i=1
1 Other snapshot spectral imagers are also being developed. See, e.g., [9]
for a CASSI-type system developed for medical imaging.
where ui (x, y) is the ith membership function, whose values
can be either 0 or 1 for a hard segmentation or in the interval [0, 1] for a fuzzy segmentation, and satisfies the constraint
PL
th
segment or mai=1 ui = 1. Here, si (λ) represents the i
terial’s spectral signature function. The support of ui lies only
in the two spatial dimensions represented by x and y, and is
independent of the spectral dimension represented by λ. The
spectral signatures, s = {si (λ), i = 1, . . . , L}, vary only
along the spectral dimension. The discrete version of f can
thus be written as,
L
X
fˆ =
ui sTi ,
(4)
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
20
40
60
80
100
120
140
160
180
20
40
60
(a)
80
100
120
140
160
180
(b)
Aluminum
Glue
Solar Cell
1
1
1
0.5
0.5
0.5
Black Rubber Edge
1
0.5
i=1
0
where fˆ ∈ Rn1 n2 ×n3 is the folded hyperspectral cube, ui ∈
Rn1 n2 ×1 is the vectorized membership function and si ∈
Rn3 ×1 . With this decomposed form of f , the savings in the
number of signals to reconstruct is significant, (n1 n2 + n3 )L
unknowns compared to the original n1 n2 n3 , since we can
usually expect n3 L.
We concentrate here on illustrating the numerical aspects
of our joint reconstruction and segmentation model for hyperspectral data obtained from CASSI compressive measurement systems. Specific technical details of our methods are
provided in [10].
0
20
40
0
0
20
0
40
Honeycomb side/Bolt
Copper Stripping
1
1
0.5
0
0.5
0
20
40
0
0
20
0
20
40
0
0
Honeycomb Top
1
1
0.5
0.5
40
0
0
20
40
20
40
Background
0
0
20
40
(c)
Fig. 2. (a) Raw CASSI simulated compressed sensing image.
(b) The reconstructed/segmented CASSI image. False color
identifies the materials in the satellite. (c) The estimated material spectral signatures compared with the original ones.
3. NUMERICAL EXAMPLES
We solve the joint reconstruction/segmentation problem by
integrating a spectral signature solver with a segmentation
solver - our fuzzy segmentation method for hyperspectral images [2, 14]. A sample result from our initial study [10], using
a simulated hyperspectral image of the Hubble Space Telescope satellite developed in our papers [1, 16], is given in Figure 2. Eight materials typical to those associated with satellites are used, Hubble aluminum, Hubble glue, Hubble honeycomb top, Hubble honeycomb side, solar cell, bolts, rubber
edge, and copper stripping. Note the excellent recovery of six
of the seven material spectra; the discrepancy between the estimated and true spectra for the bolts segment is related to its
relatively small abundance in the object.
Next, we consider a real hyperspectral image, of size
320 × 360 × 29 with bands from .453µm to .719µm, taken
from above the campus of University of Southern Mississippi,
Gulfport, Mississippi. The hyperspectral data (as well as LiDAR data) was collected for test purposes by the company
OpTech International, as part of a project led by Professor
Paul Gader at the University of Florida Department of Computer and Information Science and Engineering. The original
hyperspectral dataset has 72 bands ranging from .4µm to
1.0µm, but after matching the calibrated wavelengths of
the CASSI system with those actually measured bands by
OpTech, we chose 29 of the 72 bands and put it through a
CASSI forward model to obtain a simulated CASSI image.
The left image in Figure 3 shows a Google map of the area and
the right image shows the simulated CASSI image obtained
by passing the data though the CASSI forward model. Here,
we ran the reconstruction/segmentation algorithm directly on
the simulated CASSI image with seven known spectra taken
from the original hyperspectral cube: dirt, pavement, grass,
rooftops, sand, trees, water and shadows. The result is shown
in Figure 4. The algorithm clearly separates out the areas of
trees, water/shadows, grass and pavement with a relatively
high resolution. For example, we observe sharp boundaries
between trees and grass, and four thin lines of dirt splitting the
grass area into four parts in the middle slightly to the left. Due
to the limited material information in the chosen bands, the
two road strips at the bottom are recognized as grass, which
can be fixed by including more long wavelength bands. In
terms of target identification, the reconstruction/segmentation
clearly identifies three targets purposely placed on tables just
above the ground near the center of the scene. These consisted of colored cloths placed on tables, and we mark these
targets by the yellow circle in Figure 4.
4. SUMMARY AND FURTHER WORK
We emphasize that compressive sensing can be done only
by a compressive sensor. The process requires new sensing technology, data representations and reconstruction methods, as with coded aperture snapshot spectral (CASSI) systems. We have described here a joint reconstruction and segmentation model for hyperspectral data obtained from compressive imaging systems. Our tests indicate that combin-
[2] F. Li, M. Ng, R. Plemmons, S. Prasad, and Q. Zhang, “Hyperspectral image segmentation, deblurring, and spectral analysis for material identification,” in Proc. SPIE Conf. on Defense, Security and Sensing, Vol. 7701, 2010, Available at
http://www.wfu.edu/˜plemmons/papers.html.
50
50
100
150
100
200
150
250
200
300
[3] A. Castrodad, Z. Xing, J. Greer, E. Bosch, L. Carin, and
G. Sapiro, “Learning discriminative sparse models for source
separation and mapping of hyperspectral imagery,” IMA
Preprint Series No. 2341, University of Minnesota, 2010.
250
350
300
400
100
100
200
300
200
300
400
Fig. 3. The left image shows a Google map of the area while
the right image shows the CASSI raw data.
Water/Shadow
[4] E.J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on information theory, vol. 52, no. 2, pp. 489, 2006.
[5] D.L. Donoho, “Compressed sensing,” IEEE Transactions on
Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.
[6] D.J. Brady, Optical Imaging and Spectroscopy, Wiley Interscience, 2008.
50
Tree
100
Sand
150
Rooftop
200
[7] M.E. Gehm, R. John, D.J. Brady, R.M. Willett, and T.J.
Schulz, “Single-shot compressive spectral imaging with a
dual-disperser architecture,” Optics Express, vol. 15, no. 21,
pp. 14013–14027, 2007.
[8] A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single
disperser design for coded aperture snapshot spectral imaging,”
Applied optics, vol. 47, no. 10, pp. B44–B51, 2008.
Grass
250
Pavement
300
50
100
150
200
250
300
350
Dirt
Fig. 4. The reconstructed/segmentated map estimated from
the CASSI data. The artificial yellow circle in the image identifies location of the three targets placed in the center of the
remote scene at the U. Southern Mississippi site.
ing reconstruction with segmentation in an alternating fashion leads to promising results, and can often be used to perform target identification from compressive data. Although
our tests use only the CASSI model, we feel that the techniques we developed here can be applied to a variety of other
compressive sensors. Technical details on our joint reconstruction/segmentation methods are provided in [10]. A recent study by Kittle, et al. [17] indicates that more than one
frame of measurements may improve the robustness of reconstruction/segmentation against noise, and warrants further
investigation.
5. REFERENCES
[1] Q. Zhang, H. Wang, R. Plemmons, and Pauca P., “Tensor
methods for hyperspectral data analysis: a space object material identification study,” J. Optical Soc. Amer., Series A,
vol. 25, pp. 3001–3012, 2008.
[9] M. Johnson, D. Fink, and G. Bearman, “Snapshot spectral
imagery in ophthalmology,” J. Biomedical Opt., vol. 12, pp.
14330–14344, 2007.
[10] Q. Zhang, R. Plemmons, D. Kittle, D. Brady, and S. Prasad,
“Joint segmentation and reconstruction of hyperspectral data
with compressed measurements,” in preparation, 2011.
[11] R. Maksimovic, S. Stankovic, and D. Milovanovic, “Computed
tomography image analyzer: 3D reconstruction and segmentation applying active contour models–snakes’,” International
journal of medical informatics, vol. 58, pp. 29–37, 2000.
[12] T.F. Chan and L. Vese, “Active contours without edges,” IEEE
Trans. Image Proc., vol. 10, no. 2, pp. 266–277, 2001.
[13] T.F. Chan and J. Shen, Image Processing and Analysis - Variational, PDE, Wavelet, and Stochastic Methods, SIAM Press,
2005.
[14] F. Li, M. Ng, and R. Plemmons, “Coupled segmentation and
denoising/deblurring models for hyperspectral material identification,” Numerical Linear Alg. and Applic., to appear 2011,
available at http://www.wfu.edu/˜plemmons/papers.html.
[15] R. Ramlau and W. Ring, “A Mumford-Shah level-set approach
for the inversion and segmentation of X-ray tomography data,”
J. Comp. Physics, vol. 221, no. 2, pp. 539–557, 2007.
[16] N. Gillis and R. Plemmons, “Dimensionality reduction, classification, and spectral mixture analysis using nonnegative underapproximation,” Optical Engineering, Feb., 2011, in press,
available at http://www.wfu.edu/˜plemmons/papers.html.
[17] D. Kittle, K. Choi, A. Wagadarikar, and D. Brady, “Multiframe image estimation for coded aperture snapshot spectral
imagers,” Applied Optics, vol. 49, pp. 6824–6833, 2010.