Joint Segmentation and Reconstruction of
Hyperspectral Data with Compressed Measurements
Qiang Zhang1,∗ , Robert Plemmons2 , David Kittle3 , David Brady3 ,
and Sudhakar Prasad4
1
Biostatistical Sciences, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA
2
Computer Science and Mathematics, Wake Forest University, Winston-Salem, NC 27106,
USA
3
4
Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA
Physics and Astronomy, University of New Mexico, Albuquerque, NM 87131, USA
∗
Corresponding author: qizhang@wakehealth.edu
This work describes numerical methods for the joint reconstruction and
segmentation of spectral images taken by compressive sensing coded aperture
snapshot spectral imagers (CASSI). In a snapshot, a CASSI captures a twodimensional (2D) array of measurements that is an encoded representation of
both spectral information and 2D spatial information of a scene, resulting in
significant savings in acquisition time and data storage. The reconstruction
process decodes the 2D measurements to render a three-dimensional spatiospectral estimate of the scene, and is therefore an indispensable component
of the spectral imager. In this study, we seek a particular form of the
compressed sensing solution that assumes spectrally homogeneous segments
in the two spatial dimensions, and greatly reduces the number of unknowns,
often turning the under-determined reconstruction problem into one that is
over-determined. Numerical tests are reported on both simulated and real
c 2011 Optical Society of
data representing compressed measurements. ⃝
America
OCIS codes: 100.3010, 110.3010, 110.4234
1
1.
Introduction
Hyperspectral remote sensing technology allows one to capture images using a range of
spectra from ultraviolet to visible to infrared. Multiple images of a scene or object are created
using light from different parts of the spectrum. These hyperspectral images, forming a data
cube, can be used, e.g. for ground or space object identification [1], astrophysics [2], and
biomedical optics [3].
Since a single digital image can typically have a size of 12 megabytes or more, the size of
hyperspectral data cubes could easily move to the gigabyte level. Such high dimensional data
pose challenges in both data acquisition and reconstruction. Technologies such as tunable
filters [4] or computed tomography [5] measure at least as many elements as there are in a
hyperspectral data cube, and require long acquisition time and vast data storage and transfer,
but relatively little effort in reconstruction. Recently proposed compressive imagers such as
a Coded Aperture Snapshot Spectral Imager (CASSI) [6, 7] need only take a single snapshot
from which to reconstruct a hyperspectral data cube if the latter is assumed to be sparse
in some basis. Clearly compressed measurements need much less acquisition time and data
storage, although they demand powerful algorithms for data reconstruction, which is usually
a highly under-determined problem. For example, if the size of a vectorized hyperspectral
cube f is n1 × n2 × n3 , the double disperser CASSI (DD-CASSI, to be discussed later) [6]
measures only a 2D vectorized image g with size n1 × n2 , from which we need to reconstruct
the original cube. Here we consider first a simple least squares approach to estimating f ,
i.e. fˆ = arg minf ∥Hf − g∥22 , where H is the DD-CASSI system matrix having a size n1 n2 ×
n1 n2 n3 . Since there are fewer rows than columns in H, the problem can possess an infinite
number of solutions, and hence additional constraints are needed to obtain a meaningful,
practical solution.
Mathematical analysis of compressive sensing has drawn a great deal of attention after
important results were obtained by Donoho [8, 9] and Candes, Romberg and Tao [10]. The
problem is generally posed as one of finding the sparsest solution to f , with sparsity measured
by the l1 norm or l0 pseudo-norm, i.e.
min ∥f ∥p , subject to Hf = g,
(1)
where p = 0 or 1. Sometimes though the signals themselves are not sparse, an appropriate
or even optimal basis Φ can be found on which the projections of the signals are sparse, i.e.
min ∥Φf ∥p , subject to Hf = g.
f,Φ
(2)
Though it remains important to develop methods to determine the optimal basis for given
2
classes of signals and measurement systems, in many practical situations the signals to be
reconstructed are composed of relatively homogeneous segments or clusters, e.g. hyperspectral images in remote sensing problems [11], and in computed tomography [12]. The optimal
basis here is the set of segment membership functions on the two spatial dimensions, which
take on values of either 0 or 1 for hard segmentation or within the interval [0, 1] for fuzzy
segmentation [13]. Variational segmentation algorithms are a popular area of research, see
e.g. [12, 14–16], but they are often applied either directly to the original measurements or
after the reconstruction. By contrast, work is just beginning on combining reconstruction
and segmentation in a more general linear inverse problem setting. For instance, Li, Ng and
Plemmons [15] have coupled the segmentation and deblurring/denoising models in order
to simultaneously segment and deblur/denoise degraded hyperspectral images. Ramlau and
Ring [17] jointly reconstructed and segmented Radon transformed tomography data. Xing,
Zhou, Castrodad, Guillermo Sapiro, and Carin [18] divided the hyperspectral imagery into
contiguous blocks and used Bayesian dictionary learning method to estimate both the dictionry and the spectra in each block from limited noisy observations. However, neither example
considered compressive measurements directly from a sensor.
We now formalize our approach in the following. Let H ∈ RN ×n1 n2 n3 represent the hyperspectral imager system matrix, f ∈ Rn1 n2 n3 ×1 the vectorized hyperspectral cube, and
g ∈ RN ×1 the vectorized measurement image(s). Consider the following linear least squares
problem for reconstructing compressive measurements,
min ∥Hf − g∥22 .
f
(3)
By compressive measurements, we mean N < n1 n2 n3 . Here we assume the solution f is
composed of a limited number of segments or materials, each of which essentially has a
homogeneous value at each spectral channel. Thus we seek a decomposed solution described
in a continuous form as
L
∑
f (x, y, λ) =
ui (x, y)si (λ),
(4)
i=1
where ui (x, y) is the ith membership function, whose values can be either 0 or 1 for a hard
segmentation or in the interval [0, 1] for a fuzzy segmentation, and satisfies the constraint
∑L
th
segment or
i=1 ui = 1. Here, si (λ) represents the spectral signature function of the i
material. The support of ui lies only on the two spatial dimensions represented by x and y,
and is thus independent of the spectral dimension represented by λ. The spectral signatures,
s = {si (λ), i = 1, . . . , L}, vary only along the spectral dimension. The discrete version of f
3
can thus be written as,
fˆ =
L
∑
ui sTi ,
(5)
i=1
where fˆ ∈ Rn1 n2 ×n3 is the folded hyperspectral cube, ui ∈ Rn1 n2 ×1 is the vectorized membership function and si ∈ Rn3 ×1 . We identify f with fˆ in the remainder of the paper.
With this decomposed form of f , the savings in the number of signals to reconstruct is
significant, L(n1 n2 + n3 ) unknowns compared to the original n1 n2 n3 , since we can usually
expect n3 ≫ L. As shown later, due to this reduction the reconstruction problem to be
solved can be turned from being highly under-determined to being over-determined. Also,
for each i the membership function ui is expected to be sparse in terms of its gradient, since
only near the boundaries is there significant mixing of members in our applications. Thus a
total variation (TV) regularization is particularly suitable here.
This manuscript is organized as follows. In Section 2, we present an alternating least
squares (ALS) approach to separate the original problem (3) into two subproblems and to
solve for ui and si in an alternating fashion. For the first subproblem, namely to solve for
ui given si , when the system matrix H preserves boundaries, e.g. the DD-CASSI system,
we present a generalized segmentation algorithm based on the Chan-Vese model [14] and
the variational model [15, 16] within an inverse problem setting. This enables us to directly
segment a hyperspectral data cube from a single observed image, given known spectral signatures. For more general cases of H, we present and compare three algorithms with increasing
computation complexity to solve the first sub-problem. The second sub-problem, namely to
solve for si , given ui , is typically a highly overdetermined problem, and simple regularized
pseudoinverse methods suffice here, as we shall see. In Section 3, we present both simulated
and real compressed hyperspectral images to be reconstructed with the proposed method.
By increasing the number of measurements N , we also study the relationship between N
and the probability of successful reconstruction from random initial values, and how this
probability depends on the nature of the reconstruction algorithms, thus shedding light on
certain thresholds on the minimum number of measurements required for a finite probability
of success. By comparing the three algorithms, we were pleasantly surprised to find that even
a simple pseudo-inverse approach can achieve quite satisfying results when N is sufficiently
large. We present our main conclusions and comments in Section 4.
2.
Joint Segmentation and Reconstruction
Using the decomposed form of f ≡ fˆ described in (5), we have essentially turned the original linear problem into a non-linear problem. Nevertheless, the reduction in the number of
unknowns enables us to take advantage of compressed measurements to achieve satisfactory
reconstruction results. Our approach to solve this problem is to optimize for ui and si by
4
alternating iterations, given initial values of ui or si . But before we go into details, some
simple results and notation on matrix-vector multiplications are provided.
We denote the concatenation of all membership vectors ui as u ∈ RLn1 n2 ×1 , i.e. u =
(uT1 , uT2 , . . . , uTL )T , and the concatenation of all spectral signatures si as s ∈ RLn3 ×1 , i.e.
s = (sT1 , sT2 , . . . , sTL )T .
Proposition 1. With the decomposed form of f in (5), we have the following equality
Hf = HSu = HU s.
(6)
Here S = S̃ ⊗ I1 , the column vectors of S̃ are the spectral signature vectors si , and I1 ∈
Rn1 n2 ×n1 n2 is the identity matrix. U = Ũ ⊗I2 , Ũ = (ũij )n1 n2 ×L , where each column corresponds
to the vectorized ui , and I2 ∈ Rn3 ×n3 is another identity matrix. ⊗ denotes the Kronecker
product.
Proposition 2. With the decomposed form of f in (5), we have the following equality
Hf =
L
∑
HSi ui =
i=1
L
∑
HUi si .
(7)
i=1
Here Si = si ⊗ I1 , Ui = ũi ⊗ I2 , and ũi is the ith column of Ũ .
The proofs of both propositions are basically bookkeeping by noting the vectorization of
f is done first by the spectral dimension and then by two spatial dimensions, i.e.
f = (f111 , . . . , f11n3 , f121 , . . . , f12n3 , . . . , fn1 n2 1 , . . . , fn1 n2 n3 )T .
(8)
We denote HS by Hs and HU by Hu to represent the system matrices used for solving for
u and s, respectively.
2.A.
Alternating Least Squares (ALS)
The ALS approach turns the original problem (3) into two subproblems, i.e. given s(n) at
step n, we solve,
u(n+1) = arg min ∥Hs(n) u − g∥22 , subject to Eu = 1
u
(9)
and given u(n+1) at step n, we solve
s(n+1) = arg min ∥Hu(n+1) s − g∥22 , subject to s > 0,
s
5
(10)
(n)
(n+1)
where Hs = HS (n) and Hu
= HU (n) by Proposition 1. The two constraints on u
and s respectively are the sum-to-one constraint for the membership functions expressed as
∑
Eu = Li=1 ui = 1, with E = (I1 , I1 , . . . , I1 )n1 n2 ×n1 n2 L , and the nonnegativity constraint for
the spectral signatures.
Since the second sub-problem (10) has only Ln3 unknowns, it is often over-determined, i.e.
N ≫ Ln3 . Hence, simple approaches such as the pseudo-inverse (PI) method with Tikhonov
regularization for noise can be sufficient, and the nonnegativity constraint can be satisfied
with a projection function onto the nonnegative orthant. But because N ≫ Ln3 , the PI
solution could be well within the nonnegative orthant and this would render the projection
step unnecessary.
The first sub-problem (9) has Ln1 n2 unknowns and hence could be under-determined, but
because often times n3 ≫ L, and when Ln1 n2 ≤ N < n1 n2 n3 , we can expect (9) to be exact
or even over-determined. Even better, if we consider a hard segmentation, the number of
nonzeros in u would not exceed n1 n2 , because at each pixel there is exactly only one ui that
is nonzero. In cases when (9) is under-determined, we seek a sparse solution in its null space
with the sparsity defined as the l1 norm of the boundaries of segments, and rewrite (9) as
min
u
L
∑
∥Gui ∥1 , subject to Hs u = g and Eu = 1,
(11)
i=1
where G is the gradient matrix, the discrete version of the more familiar operator ∇, and
thus the problem above can also be regarded as a total variation (TV) minimization problem,
e.g. [19], with the functional
Fu = ∥Hs u −
g∥22
+ αu
L
∑
∥Gui ∥1 ,
(12)
i=1
where αu is the TV regularization parameter.
As a necessary condition for convergence, consider a simpler situation when both (9)
and (10) are exact or over-determined and we only use L2 functionals (informally, square
integrable functionals) for optimizing u and s. We then have the following theorem that
guarantees a nonincreasing sequence of the L2 functional values through the iterations of the
ALS approach.
Theorem 1. When only using the L2 functionals, the alternating least square approach
results in a nonincreasing sequence of functional values, i.e.
∥Hs(n) u(n+1) − g∥22 ≥ ∥Hu(n+1) s(n+1) − g∥22 ≥ ∥Hs(n+1) u(n+2) − g∥22 ,
6
(13)
for n = 0, 1, 2, . . ..
Proof. The proof becomes obvious after noting that by Proposition 1,
Hs(n) u(n+1) = HS (n) u(n+1) = HU (n+1) s(n) = Hu(n+1) s(n) .
Thus by definition of s(n+1) , we have the first inequality,
∥Hs(n) u(n+1) − g∥22 = ∥Hu(n+1) s(n) − g∥22 ≥ ∥Hu(n+1) s(n+1) − g∥22 .
(n+1) (n+1)
And by Hu
ity,
s
(n+1) (n+1)
= Hs
u
(14)
and the definition of u(n+2) , we have the second inequal-
∥Hu(n+1) s(n+1) − g∥22 = ∥Hs(n+1) u(n+1) − g∥22 ≥ ∥Hs(n+1) u(n+2) − g∥22 .
(15)
In the more general case, when (9) is under-determined and when noise is present, we seek
to minimize a combined functional of (10) and (12) with added Tikhonov regularization for
si ,
L
L
∑
∑
F = ∥Hf − g∥22 + αu
∥Gui ∥1 + αs
∥si ∥22 ,
(16)
i=1
i=1
where αs is the Tikhonov regularization parameter. For (16), we have a similar theorem to
guarantee a nonincreasing sequence of F (n) with the ALS approach.
Theorem 2. Using the functional defined in (16), the alternating least square approach
results in a nonincreasing sequence of functional values.
Proof. First we define the solutions of two subproblems as,
u
(n+1)
=
arg min ∥Hs(n) u
u
−
g∥22
+ αu
L
∑
∥Gui ∥1 , subject to Eu(n+1) = 1,
and
s
(n+1)
=
(17)
i=1
arg min ∥Hu(n+1) s
s
−
g∥22
+ αs
L
∑
∥si ∥22 , subject to s > 0.
(18)
i=1
Then the functional value at step n after optimizing for u becomes
Fu(n)
=
∥Hs(n) u(n+1)
−
g∥22
+ αu
L
∑
i=1
7
(n+1)
∥Gui
∥1
+ αs
L
∑
i=1
(n)
∥si ∥22 ,
(19)
and after optimizing for s at step n, it becomes
Fs(n) = ∥Hu(n+1) s(n+1) − g∥22 + αu
L
∑
(n+1)
∥Gui
∥1 + αs
i=1
L
∑
(n+1) 2
∥2 .
∥si
(20)
i=1
By the inequality in (14) and by the definition of s(n+1) , we have
Fu(n) ≥ Fs(n) .
(21)
Similarly, by the inequality in (15) and the definition of u(n+2) , we can see
Fs(n) ≥ Fu(n+1) .
(22)
Next we prove that the nonincreasing sequence leads to a minimizer of (16) in the space
A defined as,
L
∑
A = {(u, s)|ui ∈ BV (Ω), ui ≥ 0,
ui = 1, s ≥ 0},
(23)
i=1
where BV (Ω) is a bounded variation space. We rewrite (16) in its continuous form,
∫
F=
[H(f ) − g] dxdy + αu
2
Ω
L ∫
∑
|∇ui |dxdy + αs
Ω
i=1
L ∫
∑
i=1
s2i (λ)dλ,
(24)
Λ
where H is the continuous version of the system operator H.
Theorem 3. In the space A, there exists a minimizer of the functional defined in (24).
∫
Proof. If we take ui = 1/L and si = 1, then F = Ω [H(1) − g]2 dxdy + αs L|Λ| < +∞. Since
F ≥ 0 in A, we know the infimum of the functional would be finite. Let (u(n) , s(n) ) ⊆ A be a
minimizing sequence of the ALS approach, with u(n) and s(n) defined in (17) and (18). Then
there exists a constant M > 0, such that
F(u(n) , s(n) ) ≤ M.
(25)
Hence each term in F(u(n) , s(n) ) is also bounded, i.e.,
αu
L ∫
∑
i=1
(n)
It is also easy to see that ui
(n)
|∇ui |dxdy ≤ M.
(26)
Ω
is bounded in L1 since ∥ui ∥L1 (Ω) =
8
∫
Ω
(n)
ui dxdy < |Ω|, and by
(n)
the compactness of BV space, up to a subsequence also denoted by {ui } after relabeling,
there exists a function u∗i ∈ BV (Ω) such that
(n)
→ u∗i strongly in L1 (Ω),
(n)
→ u∗i a.e. (x, y) ∈ Ω,
ui
ui
(n)
∇ui
→ ∇u∗i in the sense of measure.
(27)
Also, by the lower semi-continuity of total variation,
∫
∫
(n)
∗
|∇ui |dxdy.
|∇ui |dxdy ≤ lim inf
n→∞
Ω
(28)
Ω
Since u(n) satisfies two constraints, by convergence, so would u∗i . For the convergence of s(n) ,
we have at each iteration
s
(n)
= P+
[(
Hu(n)T Hu(n)
+ αs I
)−1 (
Hu(n)T g
)]
.
(29)
Here we use the discrete form of the objective functional in (18) for to avoid introducing more
(n)
(n)T
notations while maintaining the spirit of the proof. Since the eigenvalues of Hu Hu + αs I
(n)
have a lower bound αs , and both Hu and g are bounded, we know s(n) must have an upper
(n)
bound. Furthermore, because u(n) → u∗i , we know Hu → Hu∗ , where Hu∗ = HU ∗ as defined in
(n)
Proposition 1. Hence we can find an upper bound of Hu for all n and thus an upper bound
of s(n) for all n. By the boundedness of the sequence {s(n) }, we can extract a subsequence
also denoted by {s(n) } and a limit s∗ such that
s(n) → s∗ .
(n)
Since ui
∫
∗
→ u∗i a.e. (x, y) ∈ Ω and s(n) → s∗ , by Fatou’s Lemma we know
[H(f ) − g] dxdy +
Ω
(30)
2
∑∫
i
s∗i (λ)dλ
∫
≤ lim inf
n→∞
[H(f
(n)
) − g] dxdy +
Ω
2
∑∫
(n)
si (λ)dλ, (31)
i
∑ (n)
∑
(n)
where f ∗ = i u∗i (x, y)s∗i (λ) and f (n) = i ui (x, y)si (λ). By (28) and (31), the functional
also satisfies the inequality
F(u∗ , s∗ ) ≤ lim inf F(u(n) , s(n) ),
n→∞
(32)
and we can conclude (u∗ , s∗ ) must be a minimizer.
In the next two sections we focus our attention on solving the first subproblem (12), since
9
the second subproblem is highly over-determined and well-posed, and thus a simple pseudoinverse approach would be sufficient. For the first subproblem, we start from a simpler case,
i.e. when the operator Hs preserves the boundaries. This effectively renders ui independent
from uj , when i ̸= j, and this also makes individual entries within ui independent from each
other. For solving this problem, we generalize two existing segmentation approaches, the
Chan-Vese model [14] and a variational model [15, 16]. We then move to the more general
Hs , i.e. where boundaries are not preserved and where all ui are coupled together by Hs .
2.B.
Boundary Preserving Operator H
Clearly any operations on a hyperspectral data cube only along the spectral dimension will
preserve the boundaries in the two spatial dimensions. One simple example is the summation
operator along the spectral dimension to turn a hyperspectral cube into a 2D image. Another
example, the DD-CASSI system, is similar to the summation operator except that by using
a coded aperture, the system effectively first multiplies the hyperspectral cube with a random aperture code, and then sums along the spectral dimension, as shown in the following
equation.
∑
fijk ci,j−k ,
(33)
gij =
k
where cij is the calibrated 2D aperture code without spectral content, f is the original
hyperspectral cube and g is the observed 2D image. A third example is the correlation
operator, e.g. the moving average method [16], which computes the correlations between
spectral signatures. Here we formalize the definition as following.
Definition 1. We define a system matrix H as boundary preserving if it satisfies
HSi ui = Λi ui ,
(34)
where H ∈ Rn1 n2 ×n1 n2 n3 , Si ∈ Rn1 n2 n3 ×n1 n2 is defined in Proposition 2, and Λi ∈ Rn1 n2 ×n1 n2
is a diagonal matrix.
Notice that the boundary preserving operators have a fixed number of rows, i.e. n1 n2 , for
the apparent reason that the boundary of an object could be exactly the boundary of the
scene. Again using the summation operator as an example, we have H = eT ⊗ I1 , where
e ∈ Rn3 ×1 is a constant vector with all entries 1. It is not hard to verify that HSi is a
diagonal matrix.
Theorem 4. The boundary preserving operator effectively renders each ui independent from
each other, that is to say, we can optimize for each ui separately.
∑
∑
∑
Proof. Because i ui = 1, we have g = i g ⊙ ui = i Diag(g)ui , where ⊙ represents the
element wise product and Diag(g) is a diagonal matrix with elements of g on the diagonal.
10
Hence by Proposition 2 and by the definition of the boundary preserving operator, we have
∥Hs u−g∥22 = ∥
L
∑
L
L
L
∑
∑
∑
HSi ui −
Diag(g)ui ∥22 = ∥
(Λi −Diag(g))ui ∥22 =
∥Λi −Diag(g)∥22 u2i .
i=1
i=1
i=1
i=1
(35)
Clearly ui is decoupled from uj when i ̸= j. Additionally, uij1 is also independent from uij2
for j1 ̸= j2 . Note that the last equality in (35) is due to the hard segmentation assumption,
∑
i.e. the binary ui , and hence the cross terms disappear after expanding ( Li=1 aij uij )2 , where
aij is the j th element on the diagonal of Λi − Diag(g).
Due to the independence of the ui , we can solve for each ui separately through, for example,
by using the popular active contour PDE model, also called the Chan-Vese (C-V) model [14],
which can be described using the following functional,
∫
∫
∫
∫
2
H(ϕ) + α1 (u0 − c1 ) H(ϕ) + α2 (u0 − c2 )2 (1 − H(ϕ)),
F (ϕ, c1 , c2 ) = µ δ(ϕ)|∇ϕ| + ν
Ω
Ω
Ω
Ω
(36)
where ϕ(x, y, t) is the function whose zero level set represents the evolving curve C, and is
chosen to be positive inside C and negative outside C. H(ϕ) is the Heavyside function of ϕ,
which is defined as
{
1 ϕ ≥ 0,
H(ϕ) =
(37)
0 ϕ < 0.
δ(ϕ) is the derivative of H(ϕ), u0 is the observed image, c1 is the mean intensity within C,
and c2 is the mean intensity outside C. Also, µ, ν, α1 , α2 are weighting parameters of the
model. The last two terms on the right are the force terms that either expand or shrink the
initial contour C0 . We refer readers to [14] for further details of the model.
For a hard segmentation, the membership function ui (x, y) is equivalent to the Heavyside
function of ϕ, and g(x, y) is the observed image. The modification only involves slightly
changing two force terms of the original model, i.e.
∫
∫
∫
∫
2
F (ϕ, c1 , c2 ) = µ δ(ϕ)|∇ϕ| + ν
ui + α1 (g̃i − g) ui + α2 (g̃i − g − c2 )2 (1 − ui ), (38)
Ω
Ω
Ω
Ω
where g̃i (x, y) is the image spectrally coded by the ith spectral signature. The discrete form
of g̃i (x, y) is derived by taking the diagonal of HSi .
We can see the only difference from the original C-V model are the force terms. The
modified C-V model in (38) is a generalization of the original model in the sense that if
g̃i (x, y) = c1 , (38) is the same as the original C-V model. It is equivalent to say that we are
segmenting the zero value segment of the image g̃i − g, rather than c1 − g. This becomes clear
if we replace f with its decomposed form and apply the boundary preserving assumption in
11
the following equation,
(
H(f ) = H
∑
)
ui (x, y)si (λ)
=
∑
i
H(si (λ))ui (x, y),
(39)
i
and the difference between H(f ) and g now becomes,
H(f ) − g =
∑
[H(si (λ)) − g(x, y)] ui (x, y).
(40)
i
Here g̃ would simply be H(si (λ)) and its difference from g is the driving force for ui . If
H(si (λ)) = ci , the equation above becomes the regular C-V model.
This modification is crucial since a spectrally homogeneous segment in f could result in
inhomogeneous intensities in the same segment of H(f ) or g. One example would be to
multiply f with a random cube and then sum along the spectral dimension. The variation of
intensities in the original segments would give us wrong segmentation results if we directly
applied the C-V model. An example of such is provided in Figure 1 (a) in Section 3.
The active contour model evolves each individual initial contour according to the given
force terms, and thus it depends on the initial contours and has to be implemented separately
for each segment. The variational model proposed in [15,16] is able to segment all L segments
at the same time and is also formulated for the fuzzy segmentation, i.e. ui ∈ [0, 1]. The model
also accounts for the sum-to-one constraint ui and the nonnegativity constraints on si . Briefly,
the original formulation includes a total variation (TV) regularization term for ui and the
intensity difference between values in the segment and the mean of the area, i.e.
∑∫
∑∫
|∇ui |dxdy + αu
(g − ci )2 u2i dxdy.
(41)
i
Ω
i
Ω
The modification again comes by replacing the force term with the difference between images
g̃ and g.
∑∫
∑∫
|∇ui |dxdy + αu
[g̃i − g]2 u2i dxdy.
(42)
i
Ω
i
Ω
Again, (42) can be seen as a generalization of (41) because when g̃i = ci , we have exactly
the same model. From Theorem 4, we know that after discretization (42) is the same as the
functional Fu in (12). One example is shown by Figure 1 (b) in Section 3, where we were
able to correctly segment and reconstruct a simulated Hubble Satellite Telescope (HST)
hyperspectral cube while only using a single DD-CASSI image.
12
2.C.
General Operator H
In many applications, such as in using the Radon Rransform [20] in computed tomography
or in using the single disperser CASSI (SD-CASSI) system [7], the boundaries cannot be
preserved by the operator. Hence, we cannot always rely on modifying existing segmentation
algorithms. We solve the constrained subproblem (9) with approaches such as pseudo-inverse
methods, methods with total variation minimization, or suboptimal methods for sparse signal
recovery, e.g. the matching pursuit [21] or the orthogonal matching pursuit [22]. In this
section, we will discuss these three options.
As stated before, due to the reduction in the number of unknowns, the subproblem (9) can
be exact or even over-determined when Ln1 n2 ≤ N , while measurements are still compressed
when N < n1 n2 n3 . The pseudo-inverse solution in this case would simply be:
u = PZ2
[(
HsT ∗ Hs
)−1 (
HsT g
)]
,
(43)
where PZ2 is the projection operator onto the space Z2 = {0, 1} for a hard segmentation.
The projection operator onto the space [0, 1] for a fuzzy segmentation is
{
{(
)−1 ( T ) } }
u = min max HsT ∗ Hs
Hs g , 0 , 1 .
(44)
The sum-to-one constraint can be satisfied or closely satisfied by adding a regularization
term to the least square functional, i.e.
α∥Eu − 1∥22 .
(45)
The pseudo-inverse solution often suffers from noise, but this can be effectively reduced
with total variation regularization. Li, Ng and Plemmons [15] proposed the following functional with an auxiliary variable v to jointly segment and deblur/denoise hyperspectral images.
1
αu
FT V = ∥Hs u − g∥22 + ∥v − u∥22 + ∥Gv∥1 ,
(46)
2
2
where v is the auxiliary variable for smoothing u. The following equations, similarly derived
as in [15, 16, 19], can be used to alternatively solve for u and v through the iterations,
p(n) + ϕ∇(div p(n) − αu u(n) )
,
1 + ϕ∇(div p(n) − αu u(n) )
1
= u(n) − div p(n+1) ,
αu
T
= (Hs Hs + αu I)−1 (HsT g + αu v (n+1) ),
p(n+1) =
v (n+1)
u(n+1)
13
(47)
where p(n+1) serves as an intermediate variable. See [15,19] for details such as the satisfaction
of constraints. We call this method the TV regularization method.
The matching pursuit (MP) algorithm finds the “best matching” projections of multidimensional data onto an over-complete dictionary. It iteratively generates for any signal u and
any dictionary Hs a sorted list of indices and scalars which constitute a sub-optimal solution
to the problem of sparse signal representation. The orthogonal matching pursuit algorithm
(OMP) is a modification to MP that maintains full backward orthogonality of the residual
at every step. In each iteration, OMP calculates a new signal approximation u(n) . The approximation error r(n) = u − u(n) is then used in the next iteration to determine which new
element is to be selected. In particular, the selection is based on the inner products between
the current residual r(n) and the column vectors of Hs . The complete algorithm is described
in [23].
Here we modify OMP slightly by introducing a gradient operator, because though the
membership function ui (x, y) is not necessarily sparse, its gradient, being supported over the
segment boundaries, often is. Let û = Gu. We first solve
min ∥û∥1 , subject to Hs G−1 û = g,
û
(48)
through OMP and then set u = G−1 û. We call this approach the gradient orthogonal matching pursuit (GOMP). A similar approach is given in [24] to restore images from subsets of
Fourier transforms. With GOMP, we only consider the hard segmentation, i.e. after solving
for all ui , we search for the maximum ui for each x and y,
{
ui (x, y) =
3.
1 ui (x, y) ≥ uj (x, y), ∀j ̸= i
0 otherwise.
(49)
Numerical Examples
Our experiments used to illustrate the effectiveness of the proposed methods are divided
into two parts, one for the boundary preserving system operator and the other for more
general system operators. Though the methods can also be applied to other compressive
hyperspectral sensing systems, the systems we consider here are the DD-CASSI and SDCASSI.
3.A.
Boundary Preserving Operator
The operator of interest here is the DD-CASSI system, whose forward model has been
described by (33). The details on the optics can be found in [6]. In the three examples
presented in this section, we move progressively from completely simulated data to completely
real data. In the first example, through the forward model (33), we simualte a DD-CASSI
14
image from from a simulated hyperspectral cube of the Hubble Space Telescope (HST)
[25] at size 177 × 193 × 33, then we simulate a DD-CASSI image from a recently acquired
hyperspectral dataset on a urban setting [26] at size 320×360×31, and finally we reconstruct
a hyperspectral cube from a real DD-CASSI image of fluorescent beads [27] at size 600 ×
800 × 59.
We start by testing the generalized C-V model (38) and the generalized variation model
(42) with known spectral signatures in the HST scene. Figures 1 and 2 compare the two
modified models with the original ones and we clearly see the advantages offered by the two
modified models. In Figure 1, an initial square contour is selected in the bottom cylinder of
the satellite, while the modified C-V model progressed to the correct bound and stabilized,
the original C-V model easily broke out of the cylinder boundary and moved to quite arbitrary
places, though it does stabilize in the end. This is due to the highly varying intensities within
each segment in the observed image g. Figure 2 is a similar comparision between the two
total variation (TV) models and clearly the original TV model (41) cannot segment out the
correct areas, while the modified model (42) can.
Next we jointly segmented and reconstructed the original hyperspectral cube from a single
DD-CASSI image. The ALS approach starts from random values of s, i.e. without any prior
knowledge of spectral signatures, and uses the modified variation model (42) to estimate
u(n+1) and the pseudo-inverse method with the Tikhonov regularization to estimate s(n+1) .
The reconstructed tensor is compared with the original tensor, using the l2 norm error,
ϵ=
∥f − f0 ∥22
,
∥f0 ∥22
(50)
where f and f0 are the reconstructed and original hyperspectral cubes, respectively.
Figure 3 shows the first 25 iterations of the estimated membership functions u shown in a
false color map, with one color assigned to each segment. Even without any prior knowledge
on the spectral signatures, we can correctly reconstruct and segment the original HST cube
and the solution u apparently converges after 20 iterations. Figure 4 compares the estimated
spectral signatures with the true ones. Only the fifth spectral signature in the middle differs
from the true one, due to the rather small prevalence of that particular material in the
scene. We also directly ran the general purpose TwIST algorithm [28] without assuming the
decomposed form of solution f , for which the norm error was 0.205 as compared to .012 for
the new approach. The new approach has dramatically reduced the norm error by taking
advantage on the solution form of f .
To test the robustness of the proposed method against noise, we polluted the simulated
DD-CASSI image with white noise having a standard deviation of .3, which effectively results
15
in an SNR of 21 dB, with SNR in dB defined as,
(
SN R = 20 log10
σsignal
σnoise
)
.
(51)
Figure 5 shows the estimated u and s, both of which closely resemble the true ones, though
noisier. The norm error is .05.
In the second example, we considered a real hyperspectral dataset of size 320 × 360 × 29,
with bands from .453µm to .719µm, taken by OpTech (OpTech International, Inc. Kiln,
MS) on the campus of University of Southern Mississippi, in Gulfport, Mississippi. The
data were collected as part of a project led by Professor Paul Gader at the University
of Florida Department of Computer and Information Science and Engineering [26]. The
original dataset had 72 bands ranging from .4µm to 1.0µm, but because the data cube of
DD-CASSI, h(x, y, λ), are only calibrated from .45µm to .72µm, and after matching the
calibrated wavelengths of DD-CASSI with those actually measured bands by OpTech, we
chose 31 of the 72 bands and ran it through a DD-CASSI forward model for a simulated
DD-CASSI image. The left image in Figure 6 shows the Google map of the area and the
right image shows the simulated DD-CASSI image. Here we ran the segmentation algorithm
directly on the simulated DD-CASSI image with seven known spectra taken from the original
hyperspectral cube, shown in Figure 7, and the result is shown in Figure 8. The algorithm
clearly separated out the areas of trees, water/shadow, grass and pavement with a relatively
high resolution. For example, we observe sharp boundaries between trees and grass, and
thin lines of dirt splitting the grass area into four parts in the middle slightly to the left.
Unfortunately due to the chosen bands, the two road strips at the bottom were recognized
as grass, but this can be fixed once we have a system cube covering more long wavelengths.
In terms of target identification, the reconstruction/segmentation clearly identifies three
targets purposely placed on tables just above the ground near the center of the scene. These
consist of colored cloths placed on tables, and we mark these targets by the yellow circle in
Figure 8. This is quite encouraging considering we are only using one snapshot, or 3.45%
of the original data. The only missed target has a similar spectra as grass from .453µm to
.719µm, but if our DD-CASSI system cube is calibrated to longer wavelengths, we will be
able to identify that target as well. The norm error between the reconstructed hyperspectral
data cube and the original cube is .033.
In the third example, we used a real DD-CASSI image of a biomedical scene with fluorescent beads of different colors [27]. The size of the image, 600 × 800, plus the number of
spectral channels, 59, would result in more than 28 million unknowns if we were not using
the decomposed form, and that might render the reconstruction impossible by general purpose algorithms such as TwIST. However, with our decomposition approach we can estimate
16
the spectral signatures of those beads quite close to the estimates given in [27]. Figure 9(a)
shows the original DD-CASSI image and Figure 9(b) shows the reconstructed membership
functions, where we try to match the bead colors with false colors as close as possible, and
they are labeled in the same way as in Figure 9(d). Figure 9(c) and 9(d) are directly taken
from [27] for comparison purposes. Notice that because the signatures of CA and R are quite
close, we cannot quite differentiate between them in Figure 9(b). Also, several long wavelength beads identified in [27], namely C1,S1,S2 and S3, are missing here because pixels of
these beads in the observed DD-CASSI image have weak intensities in the order of 0.01, as
hardly seen in Figure 9(a), while intensities at other beads are between .15 to 1. Hence the
algorithm recognizes them as background. Our reconstruction does show that the boundaries
of beads appear to have different colors from the center parts due to the geometry, while the
reconstruction in [27] treats the whole area of each bead region as uniform in color.
3.B.
General Operator
The system operator of interest here is the SD-CASSI system [7], which can be characterized
using subscript notation as follows:
gij =
n3
∑
ci,j+k fi,j+k,k ,
(52)
k=1
where c is the 2D calibrated aperture code, and the only difference from the DD-CASSI
system is the multiplexing in both the second spatial dimension and the spectral dimension,
and hence the image taken has size n1 × (n2 + n3 − 1). Because of the multiplexing in
the second spatial dimension, this operator does not preserve boundaries. The details of the
optical setup can be found in [7]. Three hyperspectral cubes were used to simulate SD-CASSI
images, the first having only an isolated square object in the image of size 128 × 128 × 33, the
second having two side-by-side rectangular objects in the image to test the method’s ability
to identify sharp boundaries, at size 128 × 128 × 33, and the third being the HST cube at
size 177 × 193 × 33. The forward model (52) was used to generate three SD-CASSI images.
We jointly segmented and reconstructed the original hyperspectral cube from one or more
frames of SD-CASSI images. The ALS approach again starts from random values of s, i.e.
without any prior knowledge, and uses three different algorithms to estimate u(n+1) , the
simple regularized pseudo-inverse (PI) method, the TV regularization model (TV) and the
gradient orthogonal matching pursuit (GOMP) model. The pseudo-inverse method with
Tikhonov regularization was used to estimate s(n+1) .
For the first scene with an isolated object in the scene, we were able to reconstruct exactly
as shown in Figure 10, with only one frame of SD-CASSI image. This tells us that one highly
compressive SD-CASSI image may be sufficient for reconstructing hyperspectral cubes with
17
only isolated objects.
For the second scene with two rectangular objects side by side, using random starting
values of s, we were able to reconstruct exactly from 8 simulated frames of SD-CASSI images
using the simple PI approach, as shown in Figure 11. We also compared the three algorithms
for estimating u with prior known spectral signatures from a single SD-CASSI image. Figure
12 shows the false color images of estimated membership functions by all three algorithms,
where we see an increasing reconstruction quality from top to bottom. Here, GOMP results
on the third row match almost exactly with the true ones, while the TV results are smoother
and closer to truth than those of PI. Clearly, GOMP may be the best candidate among the
three when measurements are most compressed.
Our experience indicates that testing all three algorithms without any prior knowledge
of s, we found a zero probability to reconstruct successfully when the number of frames is
below 4. With the pseudo-inverse approach and with 8 frames of measurements, we were
able to reconstruct exactly to a high probability, and the results are shown in Figure 11. We
further expanded this experiment by running all three algorithms 100 times with different
random starting values, and with different numbers of frames of measurements. A parallel
computation was set up on an eight-core machine and was finished in a week. The probabilities of successful reconstructions are shown in Figure 13(a) and the computation time per
iteration of ALS is shown in Figure 13(b). First we observe that when the number of frames
of observations is below 4, there is no chance to reconstruct successfully for any algorithm.
With 4 frames, the PI approach still could not reconstruct successfully while the TV and
GOMP algorithms have a finite probability, 7% and 8%, respectively. The GOMP has a 40%
success rate at 5 frames compared to 27% for PI and 28% for TV. This informs us that the
GOMP could do reasonably well with more compressive measurements, but given a certain
scene with a fixed number of nonzero projections, there should be a threshold on the minimum number of frames for a successful reconstruction. The TV approach turns out to be
the least successful beyond a sufficiently high number of frames, while we observe an almost
equivalent success rate by PI and GOMP, which is quite encouraging since PI is much faster
as shown in Figure 13(b).
Next, we tried to reconstruct the complex HST scene with a number of frames of SDCASSI images. Here we only tested the PI approach, since the other two would take a
prohibitively long computation time. With random starting values of s, when the number of
frames reach 20, we had satisfactory reconstruction results, as shown in Figure 14, though
in the beginning, the membership function map looks quite messy. Figure 14(b) compares
the estimated six most dominant spectral signatures with the corresponding true ones, and
they all agree quite well.
Finally, the real SD-CASSI images of a color chart image shown in Figure 15(a) were used
18
for reconstructing both the color segments and their spectra. Since the spectral signatures
within each segment are fairly uniform, the scene served as a good candidate for testing
the algorithm. For brevity, we only show the reconstructed colors in the second row as
indicated by the red box in Figure 15(b). The six colors on the second row from left to
right are respectively orange, purplish blue, moderate red, purple, yellow green and orange
yellow, and the spectra of five colors excluding the purple were measured by an Ocean
Optics (Dunedin, Florida) spectrometer. A total of 12 frames of SD-CASSI images were used
for reconstructing a hyperspectral cube in 44 wavelength channels. Figure 15(c) shows the
identified six segments. Clearly we are able to identify the sharp boundaries of the segments.
For the accuracy of reconstructed spectra, we compared them with both references by the
spectrometer in Figure 16(a) and the reconstruction by TwIST in Figure 16(b). Our results
match almost perfectly those obtained by TwIST, and also show good agreements with the
spectrometer references.
4.
Conclusions
By assuming a low-rank representation, we were able to jointly segment and reconstruct
hyperspectral data cubes from compressed measurements using the algorithms proposed in
this study. The assumed form of the solution dramatically reduced the number of unknowns
in the reconstruction, thus allowing better accuracy and faster computation. In our tests, we
obtained promising results after applying the algorithms to a sequence of simulated and real
DD-CASSI and SD-CASSI images.
For boundary preserving measurement operators, such as the DD-CASSI system, we generalized two existing segmentation algorithms, the Chan-Vese model and a total variation
model, to directly segment the compressed measurements along the spatial and spectral dimensions without first reconstructing the data into hyperspectral cubes. In our studies, we
show poor segmentation results after applying the original two models directly on simulated
DD-CASSI images, but very good results after applying the generalized two models. For more
general operators, such as that of SD-CASSI, we proposed three methods, i.e. the pseudoinverse method, the total variation method and the gradient orthogonal matching pursuit
(GOMP) method for estimating the membership functions with given spectral signatures. A
more extensive simulation study shows that several frames of measurements are often needed
for a successful reconstruction, though the theoretical threshold on the minimum number
of frames needs further analysis. With enough observations, the probabilities of successful
reconstruction by different algorithms tend to converge, and even the simple pseudo-inverse
approach can provide successful reconstruction. But when the number of observations is
limited, the GOMP method performs best. For real SD-CASSI data taken on a color chart
scene, we have shown good reconstruction results with the GOMP method.
19
In using a low-rank representation, the algorithms proposed here would better suit situations where distinct spectra lie in some spatially homogeneous areas (segments), e.g.,
cartoon-like scenes. As the size of segments gets smaller and the number of spectra becomes
greater, we would expect a performance reduction given the same number of compressed
measurements. In these situations, we recommend adding measurements as in [29] but when
the scene in study becomes too complex, more general algorithms such as TwIST may be
more reliable. However, we would caution that in these non-sparse situations, the compressive
sensing approach may not be that appealing after all.
5.
Acknowledgement
This work was sponsored in part by the AFOSR under grant number FA9550-08-1-0151.
References
1. K. Jorgensen, J. Africano, K. Hamada, E. Stansbery, P. Sydney, and P. Kervin, “Physical
properties of orbital debris from spectroscopic observations,” Advances in Space Research
34, 1021–1025 (2004).
2. R. Lin, B. Dennis, G. Hurford, D. Smith, A. Zehnder, P. Harvey, D. Curtis, D. Pankow,
P. Turin, M. Bester, and Others, “The Reuven Ramaty high-energy solar spectroscopic
imager (RHESSI),” Solar Physics 210, 3–32 (2002).
3. T. Pham, F. Bevilacqua, T. Spott, J. Dam, B. Tromberg, and S. Andersson-Engels,
“Quantifying the absorption and reduced scattering coefficients of tissuelike turbid media
over a broad spectral range with noncontact Fourier-transform hyperspectral imaging,”
Appl. Optics 39, 6487–6497 (2000).
4. H. Morris, C. Hoyt, and P. Treado, “Imaging spectrometers for fluorescence and Raman
microscopy: acousto-optic and liquid crystal tunable filters,” Applied spectroscopy 48,
857–866 (1994).
5. J. Mooney, V. Vickers, M. An, and A. Brodzik, “High-throughput hyperspectral infrared
camera,” Journal of the Optical Society of America A 14, 2951–2961 (1997).
6. M. Gehm, R. John, D. Brady, R. Willett, and T. Schulz, “Single-shot compressive spectral
imaging with a dual-disperser architecture,” Optics Express 15, 14013–14027 (2007).
7. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded
aperture snapshot spectral imaging,” Applied Optics 47, B44–B51 (2008).
8. D. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory 52, 1289–
1306 (2006).
9. D. Donoho and M. Elad, “Optimally sparse representation in general (nonorthogonal)
dictionaries via l1 minimization,” Proceedings of the National Academy of Sciences of
the United States of America 100, 2197 (2003).
20
10. E. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on information theory 52, 489 (2006).
11. J. Ball and L. Bruce, “Level set segmentation of remotely sensed hyperspectral images,” in “2005 IEEE International Geoscience and Remote Sensing Symposium, 2005.
IGARSS’05. Proceedings,” (2005), pp. 5638–5642.
12. R. Maksimovic, S. Stankovic, and D. Milovanovic, “Computed tomography image analyzer: 3D reconstruction and segmentation applying active contour models–[] snakes’,”
International journal of medical informatics 58, 29–37 (2000).
13. J. Kent and K. Mardia, “Spatial classification using fuzzy membership models,” IEEE
Transactions on Pattern Analysis and Machine Intelligence pp. 659–671 (1988).
14. T. Chan and L. Vese, “Active contours without edges,” IEEE Transactions on image
processing 10, 266–277 (2001).
15. F. Li, M. Ng, and R. Plemmons, “Coupled Segmentation and Denoising/Deblurring
Models for Hyperspectral Material Identification,” Numerical Linear Algebra with Applications (2010).
16. F. Li, M. Ng, R. Plemmons, S. Prasad, and Q. Zhang, “Hyperspectral image segmentation, deblurring, and spectral analysis for material identification,” in “SPIE Symposium
on Defense, Security, and Sensing,” , vol. 7701 (2010), vol. 7701.
17. R. Ramlau and W. Ring, “A Mumford-Shah level-set approach for the inversion and
segmentation of X-ray tomography data,” Journal of Computational Physics 221, 539–
557 (2007).
18. Z. Xing, M. Zhou, A. Castrodad, G. Sapiro, and L. Carin, “Dictionary learning for noisy
and incomplete hyperspectral images,” IEEE Transactions on Geoscience and Remote
Sensing (submitted).
19. A. Chambolle, “An algorithm for total variation minimization and applications,” Journal
of Mathematical Imaging and Vision 20, 89–97 (2004).
20. S. Deans, The Radon transform and some of its applications (Wiley New York, 1983).
21. S. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE
Transactions on signal processing 41, 3397–3415 (1993).
22. J. Tropp, “Greed is good: Algorithmic results for sparse approximation,” IEEE Transactions on Information Theory 50, 2231–2242 (2004).
23. Y. Pati, R. Rezaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive
function approximation withapplications to wavelet decomposition,” in “1993 Conference
Record of The Twenty-Seventh Asilomar Conference on Signals, Systems and Computers,
1993,” (1993), pp. 40–44.
24. R. Maleh, A. Gilbert, and M. Strauss, “Sparse gradient image reconstruction done
21
faster,” in “IEEE International Conference on Image Processing, 2007. ICIP 2007,” ,
vol. 2 (2007), vol. 2.
25. Q. Zhang, H. Wang, R. Plemmons, and V. Pauca, “Tensor methods for hyperspectral
data analysis: a space object material identification study,” Journal of the Optical Society
of America A 25, 3001–3012 (2008).
26. P. Gader, A. Zare, R. Close, and G. Tuell, “Co-registered hyperspectral and LiDAR Long
Beach, Mississippi data collection,” (2010). University of Florida, University of Missouri,
and Optech International.
27. C. Cull, K. Choi, D. Brady, and T. Oliver, “Identification of fluorescent beads using a
coded aperture snapshot spectral imager,” Applied Optics 49, 59–70 (2010).
28. J. Bioucas-Dias and M. Figueiredo, “A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Transactions on Image processing 16, 2992–3004 (2007).
29. D. Kittle, K. Choi, A. Wagadarikar, and D. Brady, “Multiframe image estimation for
coded aperture snapshot spectral imagers,” Applied Optics 49, 6824–6833 (2010).
22
List of Figure Captions
Fig. 1. (a) Segmented bottom cylinder area using the original Chan-Vese model; (b)
Segmented same area using the generalized C-V.
Fig. 2. Segmentation of a DD-CASSI image (a) using the original TV algorithm and (b)
using the modified TV with known si (λ).
Fig. 3. The first 25 iterations of the estimation of membership functions from left to right
and top to bottom.
Fig. 4. The estimated spectral signatures in blue are compared with the true ones in red.
Fig. 5. The joint estimation of membership functions ui in (a) and spectral signatures, si in
(b) in the presence of noise (SNR = 21 dB).
Fig. 6. The left image shows a Google map of the area while the right image shows one band
of the simulated DD-CASSI image.
Fig. 7. The estimated spectra of the segmented areas.
Fig. 8. The segmentation map estimated from the DD-CASSI image.
Fig. 9. (a) The DD-CASSI image of the beads. (b) The estimated membership function
map. (c) The beads spectral signatures from [27]. (d) The pseudo-colored reconstructed
image using TV minimization [27], with the spectral signatures of the marked beads in (c).
Fig. 10. The top left image is a slice of the simulated hyperspectral cube; the top middle is
the simulated SD-CASSI image; the top right is the spectral signature applied to all pixels
in the square to the left; the bottom left is a slice of the reconstructed cube and the bottom
middle is the reconstructed spectral signature.
Fig. 11. The top left image is a slice of the simulated hyperspectral cube; the top middle is
the simulated SD-CASSI image; the top right are two spectral signatures applied to pixels
in two rectangles to the left; the bottom left is a slice of the reconstructed cube and the
bottom middle are the reconstructed spectral signatures.
Fig. 12. Comparison of three reconstruction methods using a single frame observation and
known si (λ). Only the middle 40 rows of the original scene is used here.
Fig. 13. (a) Relationship between the probability of a successful reconstruction and the
number of observed frames for the three reconstruction methods. (b) Computation time of
each method per iteration of ALS.
Fig. 14. (a) The estimated membership function map in 20 iterations. (b) The estimated
spectral signatures in blue, compared to the original in red.
Fig. 15. (a) The imaged color chart object. (b) The original SD-CASSI image with the
reconstructed area in the red rectangle box. (c) The reconstructed segments.
Fig. 16. (a) The reconstructed spectra compared with reference by an Ocean Optics
spectrometer. (b) The reconstructed spectra compared with reconstruction using TwIST.
1500 Iterations
1500 Iterations
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
20
40
60
80
100
(a)
120
140
160
180
20
40
60
80
100
120
140
160
180
(b)
Fig. 1. (a) Segmented bottom cylinder area using the original Chan-Vese
model; (b) Segmented same area using the generalized C-V.
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
20
40
60
80
100
(a)
120
140
160
180
20
40
60
80
100
120
140
160
180
(b)
Fig. 2. Segmentation of a DD-CASSI image (a) using the original TV algorithm
and (b) using the modified TV with known si (λ).
50
50
50
50
100
100
100
100
150
100
150
150
50
100
150
150
50
50
100
100
150
150
50
100
150
150
50
100
150
100
150
50
50
50
50
100
100
100
150
50
100
150
150
50
100
150
100
150
50
50
50
50
100
100
100
150
50
100
150
150
50
100
150
100
150
50
50
50
50
100
100
100
150
100
150
150
50
100
150
150
50
100
150
50
100
150
50
100
150
50
100
150
150
50
100
50
100
150
50
100
150
50
150
50
100
150
150
100
50
150
100
150
50
100
150
100
150
50
150
100
100
100
150
100
150
50
150
50
100
150
Fig. 3. The first 25 iterations of the estimation of membership functions from
left to right and top to bottom.
Glue
Black Rubber Edge
0.8
0.8
0.8
0.8
0.6
0.4
0.6
0.4
0.2
0
400
0
400
600
800
Wavelength (nm)
Honeycomb side/Bolt
Reflectance
1
Reflectance
1
0.2
0.6
0.4
0.2
0
400
600
800
Wavelength (nm)
Copper Stripping
0.6
0.4
0.2
0
400
600
800
Wavelength (nm)
Honeycomb Top
1
0.8
0.8
0.8
0.8
0.4
Reconstructed
Honeycomb side
0.2
0
400
Bolt
600
800
Wavelength (nm)
0.6
0.4
0.2
0
400
Reflectance
1
Reflectance
1
0.6
0.6
0.4
0.2
600
800
Wavelength (nm)
0
400
600
800
Wavelength (nm)
Background
1
Reflectance
Reflectance
Solar Cell
1
Reflectance
Reflectance
Aluminum
1
0.6
0.4
0.2
600
800
Wavelength (nm)
0
400
600
800
Wavelength (nm)
Fig. 4. The estimated spectral signatures in blue are compared with the true
ones in red.
20
40
60
80
100
120
140
160
20
40
60
80
100
120
140
160
180
(a)
Glue
Black Rubber Edge
0.8
0.8
0.8
0.8
0.6
0.4
0.6
0.4
0.2
0
400
0
400
500
600
700
Wavelength (nm)
Honeycomb side/Bolt
Reflectance
1
Reflectance
1
0.2
0.6
0.4
0.2
0
400
500
600
700
Wavelength (nm)
Copper Stripping
0.6
0.4
0.2
0
400
500
600
700
Wavelength (nm)
Honeycomb Top
1
0.8
0.8
0.8
0.8
0.4
0.2
0
400
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0
400
Reflectance
1
Reflectance
1
0.6
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0
400
500
600
700
Wavelength (nm)
Background
1
Reflectance
Reflectance
Solar Cell
1
Reflectance
Reflectance
Aluminum
1
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0
400
500
600
700
Wavelength (nm)
(b)
Fig. 5. The joint estimation of membership functions ui in (a) and spectral
signatures, si in (b) in the presence of noise (SNR = 21 dB).
50
50
100
100
150
200
150
250
200
300
250
350
300
400
50
100
200
300
100
150
200
250
300
350
400
Fig. 6. The left image shows a Google map of the area while the right image
shows one band of the simulated DD-CASSI image.
Pavement
Tree
0.8
0.8
0.8
0.8
0.6
0.4
0
400
0.6
0.4
0.2
600
800
Wavelength (nm)
0
400
Sand
0.6
0.4
0.2
600
800
Wavelength (nm)
0
400
Rooftop
0.8
0.8
0.8
0.4
0.2
0
400
0.4
Reflectance
1
Reflectance
1
0.6
0.2
600
800
Wavelength (nm)
0
400
0.6
0.4
0.2
600
800
Wavelength (nm)
0
400
600
800
Wavelength (nm)
Water/Shadow
1
0.6
Reflectance
1
Reflectance
1
0.2
Reflectance
Grass
1
Reflectance
Reflectance
Dirt
1
0.6
0.4
0.2
600
800
Wavelength (nm)
0
400
600
800
Wavelength (nm)
Fig. 7. The estimated spectra of the segmented areas.
Water/Shadow
50
Tree
100
Sand
150
Rooftop
200
Grass
250
Pavement
300
50
100
150
200
250
300
350
Dirt
Fig. 8. The segmentation map estimated from the DD-CASSI image.
CA3
R3
G2
RO1
100
O1
R1
R4
Y1
BG1
CA4
200
Y3
Y2
100
CA1
CA2
300
200
Y4
Y7
300
YG1
400
R5
400
Y5
R2
G1
500
Y6
500
600
100
200
300
400
500
600
700
800
600
100
200
300
400
500
(a)
600
700
800
(b)
(c)
(d)
Fig. 9. (a) The DD-CASSI image of the beads. (b) The estimated membership
function map. (c) The beads spectral signatures from [27]. (d) The pseudocolored reconstructed image using TV minimization [27], with the spectral
signatures of the marked beads in (c).
1
20
40
40
60
60
80
80
100
100
120
120
20 40 60 80 100 120
0.8
Reflectance
20
0.6
0.4
0.2
50
100
150
0
450
500
550
600
Wavelength (nm)
650
1
20
0.8
Reflectance
40
60
80
100
0.6
0.4
0.2
120
20 40 60 80 100 120
0
450
500
550
600
Wavelength (nm)
650
Fig. 10. The top left image is a slice of the simulated hyperspectral cube; the
top middle is the simulated SD-CASSI image; the top right is the spectral
signature applied to all pixels in the square to the left; the bottom left is a
slice of the reconstructed cube and the bottom middle is the reconstructed
spectral signature.
1
20
40
40
60
60
80
80
100
100
120
120
0.8
Reflectance
20
0.4
0.2
50
20 40 60 80 100120
0.6
100
150
0
450
500
550
600
Wavelength (nm)
650
1
20
Reflectance
0.8
40
60
80
100
0.6
0.4
0.2
120
20 40 60 80 100120
0
450
500
550
600
Wavelength (nm)
650
Fig. 11. The top left image is a slice of the simulated hyperspectral cube; the
top middle is the simulated SD-CASSI image; the top right are two spectral
signatures applied to pixels in two rectangles to the left; the bottom left is a
slice of the reconstructed cube and the bottom middle are the reconstructed
spectral signatures.
Pseudo−Inverse
10
20
30
40
20
40
60
80
100
120
100
120
Total Variation
10
20
30
40
20
40
60
80
Orthogonal Matching Pursuit
10
20
30
40
20
40
60
80
100
120
Fig. 12. Comparison of three reconstruction methods using a single frame
observation and known si (λ). Only the middle 40 rows of the original scene is
used here.
(a)
(b)
Fig. 13. (a) Relationship between the probability of a successful reconstruction
and the number of observed frames for the three reconstruction methods. (b)
Computation time of each method per iteration of ALS.
20
20
20
20
20
40
40
40
40
40
60
60
60
60
60
80
80
80
80
20
40
60 80
20
40
60
80
20
40
60
80
80
20
40
60 80
20
20
20
20
20
40
40
40
40
40
60
60
60
60
80
80
20
40
60 80
80
20
40
60
80
40
60
80
40
60 80
20
20
20
20
40
40
40
40
40
60
60
60
60
60
80
80
80
80
40
60 80
20
40
60
80
20
40
60
80
40
60 80
20
20
20
20
40
40
40
40
40
60
60
60
60
60
80
80
80
80
40
60 80
20
40
60
80
20
40
60
80
80
20
40
60
80
20
40
60
80
20
40
60
80
80
20
20
20
60
80
20
20
20
40
60
80
20
20
80
20
40
60 80
1
0.8
0.8
0.6
0.4
0.4
0.2
500
550
600
Wavelength (nm)
0
450
650
1
1
0.8
0.9
Reflectance
Reflectance
0
450
0.6
0.6
0.4
0.2
450
500
550
600
Wavelength (nm)
650
500
550
600
Wavelength (nm)
0.4
0.2
500
550
600
Wavelength (nm)
650
500
550
600
Wavelength (nm)
650
1.5
0.8
0.7
450
0.6
0
450
650
Reflectance
0.2
0.8
Reflectance
1
Reflectance
Reflectance
(a)
500
550
600
Wavelength (nm)
650
1
0.5
0
450
(b)
Fig. 14. (a) The estimated membership function map in 20 iterations. (b) The
estimated spectral signatures in blue, compared to the original in red.
100
200
300
400
500
600
700
800
900
200
400
(a)
600
800
1000
1200
1400
(b)
10
20
30
40
50
200
400
600
800
1000
1200
1400
(c)
Fig. 15. (a) The imaged color chart object. (b) The original SD-CASSI image
with the reconstructed area in the red rectangle box. (c) The reconstructed
segments.
Purplish Blue
Orange
1
0.4
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0.2
400
800
Yellow Green
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0
400
800
500
600
700
Wavelength (nm)
800
Orange Yellow
1
1
0.8
0.8
Reflectance
Reflectance
0.8
Reflectance
0.6
0.6
0.4
0.2
0
400
1
0.8
Reflectance
Reflectance
0.8
0
400
Moderate Red
1
Ocean Optics
PI
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0
400
800
500
600
700
Wavelength (nm)
800
(a)
Purplish Blue
Orange
Reflectance
0.6
0.4
500
600
700
Wavelength (nm)
0.6
0.2
400
800
Purple
0.6
0.4
0.2
500
600
700
Wavelength (nm)
0
400
800
Yellow Green
1
1
0.8
0.8
0.8
0.6
0.4
0.2
0
400
0.6
0.4
0.2
500
600
700
Wavelength (nm)
800
0
400
500
600
700
Wavelength (nm)
800
Orange Yellow
1
Reflectance
Reflectance
0.8
0.8
0.4
0.2
TwIST
PI
1
Reflectance
Reflectance
0.8
0
400
Moderate Red
1
Reflectance
1
0.6
0.4
0.2
500
600
700
Wavelength (nm)
800
0
400
500
600
700
Wavelength (nm)
800
(b)
Fig. 16. (a) The reconstructed spectra compared with reference by an Ocean
Optics spectrometer. (b) The reconstructed spectra compared with reconstruction using TwIST.