18502
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
A Recurrent Feedback Hyperspectral
Image Super-Resolution Reconstruction Method
by Using Self-Attention-Based Pixel Awareness
Ruyi Feng , Member, IEEE, Zhongyu Guo , and Xiaofeng Wang
Abstract—Hyperspectral images (HSIs) contain abundant spectral information, while the spatial resolution is usually limited. To
obtain high-spatial-resolution HSIs, various HSI super-resolution
(SR) methods are proposed. Currently, deep-learning-based SR
reconstruction methods are studied in depth, which take different
measures to make full use of the spatial and spectral information
of HSIs, and optimize the network with lots of trainings. Although
they have achieved satisfying spatial resolution, the spectral consistency before and after reconstruction is difficult to guarantee. In
this article, we proposed a self-attention-based recurrent feedback
network for hyperspectral SR reconstruction, utilizing pixel-aware
weights and pseudo three-dimensional convolution to enhance the
spatial and spectral consistency during the reconstruction process.
In addition, group reconstruction is used to reduce the redundancy
of information. Spectral consistency regularization is proposed to
ensure the spectral consistency before and after reconstruction. The
effectiveness of the proposed method is tested on one set of natural
images and three hyperspectral remote sensing image datasets.
Index Terms—Feedback embedding, hyperspectral image (HSI),
pixel awareness, recurrent network, super-resolution (SR).
I. INTRODUCTION
YPERSPECTRAL image (HSI) usually contains hundreds or thousands of bands, and each band corresponds
to a specific wavelength range [1]. Compared with multispectral
(MSI) and RGB images, HSI has higher spectral dimensions.
Therefore, HSI is widely used in Earth observation [2], medical
imaging [3], food quality control [4], military intelligence [5],
and other fields. However, due to the technical limitations of
imaging hardware, there is a tradeoff between spectral resolution
and spatial resolution. To guarantee richer and more accurate
spectral information, HSI often results in poor spatial resolution,
which greatly affects its practical application.
Hyperspectral super-resolution (SR) reconstruction, a key
technique for overcoming the spatial-resolution limitations inherent in HSI, aims to enhance low spatial resolution HSI into
H
Received 11 June 2024; revised 25 August 2024; accepted 24 September 2024.
Date of publication 9 October 2024; date of current version 21 October 2024.
This work was supported in part by the National Natural Science Foundation of
China under Grant 41925007 and Grant 42471430, in part by the Knowledge
Innovation Program of Wuhan–Shuguang under Grant 2023010201020336, in
part by Hubei Natural Science Foundation under Grant 2024AFB561, and in part
by the Funds for International Cooperation and Exchange of the National Natural Science Foundation of China under Grant 42311530065. (Corresponding
author: Xiaofeng Wang.)
The authors are with the China University of Geosciences, Wuhan 430074,
China (e-mail: fengry@cug.edu.cn; guozhongyu@cug.edu.cn; wxf@cug.edu.
cn).
Digital Object Identifier 10.1109/JSTARS.2024.3471899
high spatial resolution HSI while preserving the spatial and
spectral information. This approach recovers high-frequency
spatial details and maintains spectral fidelity, thereby significantly improving the utility of hyperspectral data across a wide
range of applications. Recently, hyperspectral SR algorithms
have been researched by more and more scholars, and many
models are studied in depth. Among them, most of the models can be classified in two categories according to whether
using the auxiliary image, single-image-based SR, and multiimage-based SR. Multi-image-based SR usually adopts RGB,
multispectral, panchromatic images as the auxiliary information
[6], [7], [8], [9] in the training process and afford sufficient
reference information during reconstruction process. Although
the multi-image-based SR shows excellent performance, how to
acquire the appropriate auxiliary images is still a big challenge.
While single-image-based does not require the assistance of
auxiliary information. Additionally, single-image-based models typically show lower computational complexity and lower
system requirements, making them suitable for scenarios with
limited image resources. Therefore, single-image-based SR is
more valuable and more practical. In this article, we will focus
on recovering high-resolution HSI from a single low-resolution
(LR) HSI without auxiliary images.
In early study, various interpolation methods are proposed,
i.e., adjacent interpolation [10], biquadratic interpolation [11],
bicubic interpolation [12], etc. But these methods frequently
yield distortions and artifacts attributable to loss detail, particularly at the edges and complex regions of the images. Since
SR reconstruction is a typical ill-posed problem, more and
more studies utilize prior knowledge as regularization terms
and treat SR as a constrained optimization problem, such as
sparse representation based SR approaches [13], [14], [15] and
probabilistic analysis based SR approaches. Simultaneously,
nonlocal similarity [16], low rank constraint [17], [18], [19],
or other spatial or spectral correlations are usually adopted as
prior knowledge. Nevertheless, due to the limitation of these
methods, the quality of reconstructed images is not satisfying.
With the development of deep learning, many scholars tried to
simulate the inverse process of image sampling degradation by
constructing different deep-learning models, and convolutional
neural networks (CNNs) have been introduced and demonstrated
remarkable effectiveness in hyperspectral SR reconstruction.
However, CNN models still exhibit limitations in singleimage SR. HSIs possess abundant of spectral and spatial
© 2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see
https://creativecommons.org/licenses/by-nc-nd/4.0/
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
information. While current CNN methods [20], [21], [22], [23],
[24], [25], [26] usually employ convolutional kernels with fixed
size to extract features, which constrains the model’s receptive
field, limiting its attention to local areas and making it difficult
to capture distant spatial correlations. Meanwhile, CNN models
also face challenges in extracting spectral correlations, resulting in poor consistency in spectral structure before and after
reconstruction.
To reduce the spectral distortion before and after reconstruction, enhance the spatial correlation for local and global image
space, and further improve the quality of hyperspectral SR reconstruction, an improved single image SR reconstruction method
based on deep learning is proposed, named pixel-aware recursive
feedback network (PARF), which operates by grouping the HSI
along the spectral dimension and utilizes a recurrent recursive
network to reconstruct each group. PARF enhances spatial information using self-attention-based pixel awareness and achieving
spectral consistency regularization through enhanced pseudothree-dimensional (3-D) convolutions. Compared to traditional
methods, proposed method approach exhibits superior reconstruction performance. The main innovations are summarized
as follows.
1) The pixel-aware weight based on self-attention is introduced, which can be acquired through the intermediate
results to help reconstruct the missing spatial information. Meanwhile, the self-attention mechanism is used to
expand the receptive field and enhance the local and global
spatial correlations.
2) The optimized pseudo-3-D convolution operation is also
used to ensure spatial and spectral consistency.
The rest of this article is organized as follows. Section II
provides the related works of single-image SR. In Section III,
the proposed model architecture is introduced in detail. Then,
extensive experiments and analyses are presented in Section IV.
Finally, Section V concludes this article.
II. RELATED WORKS
A. Single-Image Super-Resolution Reconstruction
Traditional single-image SR methods need to design a mapping function between LR and high-resolution HSI based on
prior knowledge. While deep-learning-based SR reconstruction
needs a large amount of training data to build models in an endto-end manner and learns the mapping relation autonomously.
Notable models, such as very deep SR [27] and enhanced
deep super-resolution (EDSR) [28], utilizing deep CNNs with
residual learning and dense connections, can transmit information and gradients to learn deeper features more effectively
and significantly enhance the reconstruction quality. Spatial–
spectral prior for super-resolution (SSPSR) [26] uses spectral
self-similarity to select image blocks and improves the effect
of SR reconstruction. These methods learn the complex feature
representation of images through common convolution, residuals, and dense connections, and achieve more satisfying results
in spatial resolution than traditional methods. Recently, generative adversarial networks (GANs) and transfer learning have
been widely employed to enhance the quality of super-resolved
18503
images. GANs, with their ability to generate photorealistic images, have shown promising results in image SR. DGAN [29],
employing multiple generators within a GAN framework, effectively addresses the oversmoothing issue prevalent in traditional
GAN-based image SR, resulting in significantly enhanced perceptual quality and accelerated training convergence. EEGAN
[30] effectively addresses the challenge of recovering highfrequency edge details in noise-contaminated satellite images
by combining an ultradense subnetwork for feature extraction
and an edge-enhancement subnetwork for detailed refinement
within a GAN framework. Meanwhile, transfer learning has been
explored to mitigate the scarcity of training data by leveraging
pretrained models, thereby accelerating the training process and
improving performance. PSRGAN [31] addresses the challenge
of limited infrared image training data by proposing a progressive SR GAN architecture that leverages information from
available visible images through a multistage transfer learning
strategy. However, these models perform poorly in the fidelity
of the spectral structure before and after reconstruction due to
ignoring the spectral relationships.
In order to solve the spectral distortion, a grouped deep
recursive residual network based on grouped embedding recursive module [32] is proposed, which fully considers the spectral correlation information and learns more effective mapping
functions. To more effectively utilize the spectral and spatial
information among consecutive LR bands, an internal fusion
network [33] is proposed, which jointly learns spatial information, spectral information, and spectral difference information
to further explore spatial and spectral information.
For the purpose of better adapting to the high-dimensional
image data of HSI, a 3-D fully CNN [34] is designed to learn
the spatial and spectral correlation simultaneously through a 3-D
convolution. Similar to this, a 3-D GAN is proposed by Li et al.
[35] with band attention mechanism to alleviate the problem
of spectral distortion. Despite these advancements, CNN-based
methods still cannot overcome the drawback of fixed-size convolution, which is unable to obtain remote information associated
with high-dimensional hyperspectral data. The limited receptive
field also hinders the expressive abilities. Besides, traditional
hyperspectral SR reconstruction models require a large HSIs
for training, while the collection of sufficient HSIs poses challenges, and the vast training data burden the model. Particularly,
HSIs exhibit strong correlations between adjacent bands, and it
is crucial to exploit these correlations for hyperspectral highquality SR. Experiments made by Li et al. [32] confirmed this
viewpoint, demonstrating that redundant information between
bands contributes to high-quality SR reconstruction.
To take full advantage of spectral band correlations, inspired
by SSPSR [31] and GDRRN [32], many scholars have proposed
grouping-based model frameworks. Those models split HSI into
distinct groups first, reconstruct spectral data within each group,
respectively, and combine the outcomes of each group finally. It
not only enhances the quality of reconstruction but also improves
the overall computational efficiency. However, these models
only consider spectral correlations within groups, and neglect
the spectral continuity between different spectral groups. To
expand on the concept of grouping, Wang et al. [36] introduced
18504
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
Fig. 1. Error map of the reconstruction results and the ground truths of the
EDSR model in 50 and 100 stages, respectively.
Fig. 2.
the super-resolution via recurrent feedback embedding (RFSR)
network, which employed a recurrent feedback mechanism to
leverage similarities and complementary information within adjacent spectral groups, and enhanced the reconstruction details.
Subsequently, Wang et al. [37] investigated complementary information within adjacent spectral groups to enhance detailed
reconstruction. At the same time, a dual-stage network [38]
was designed to leverage spatial–spectral similarity between
neighboring bands including both coarse and fine stages.
Those innovative approaches enable hyperspectral SR to reconstruct spectral information precisely through feedback from
different groups, fully leverage the continuity between spectral
groups, and enhance the final SR performance. These models
effectively capture the correlation information between distant
spectral bands, but the convolution dependence on fixed-size
kernels restricts models utilizing only a fraction of the available
spatial information.
B. Pixel Aware Weights
Inspired by the idea of loop fusion, the intermediate results
can provide information for subsequent regularization. In pixel
perception based self-supervised SR reconstruction methods
[39], pixel-aware weights and intermediate results are usually
used to optimize and guide the following SR reconstruction
process.
In Fig. 1, it can be observed that the reconstructed background pixels can reach the minimum value in the early stage
of training and then keep constant. But the foreground pixels
behave differently. This suggest that each pixel presents distinct reconstruction difficulties and the background pixels are
easier to reconstruct than those in the foreground. Thus, the
reconstruction result from two different stages can be leveraged
to assess the reconstruction difficulty of individual pixels. By
subtracting the reconstruction results of two different stages,
the difference results reflect the change of pixels in different
regions. Pixels in the foreground area that need to be further
reconstructed exhibit a larger difference, while those in the
background region demonstrate a smaller difference. Therefore,
to measure difference between the reconstruction result of one
stage and the previous one can guide us for the following
reconstruction.
Main structure of pseudo-3-D convolution.
C. Self-Attention Mechanism
Transformer model has been widely used in the field of
deep learning recently. Self-attention, as an important part of
transformer, determines the weight of each position by calculating the similarity between each position and all other positions,
and better capture the information worthy of attention from the
image. The specific process is as follows:
QKT
V
(1)
Attention(Q, K, V) = softmax √
dk
where Q, K, and V represent the input vectors after linear
transformation. dk represents vector dimension. In the proposed
work, self-attention model is adopted to capture spatial attention
information of HSI at each stage after grouping, obtain weights
to guide the reconstruction for the next stage, and improve the
spatial reconstruction effect.
D. Pseudo-3-D Convolution
Traditional 2-D convolution operation has to process MSI
band by band, while 3-D convolution can maintain the original
channel correlation and better extract spatial spectral features
in 3-D space. Generally, to standardize the spatial–spectral
structure of hyperspectral data, 3-D convolution should be used.
Considering the significantly growth in parameters and computational complexity, an efficient pseudo-3-D convolution [40] is
adopted in the proposed model.
The pseudo-3-D convolution is implemented using two convolutional layers, as shown in Fig. 2, in which the first (spectral)
convolution has a filter shape of (1, 1, k) and the second (spatial)
convolution has a filter shape of (k, k, 1) (here, k is set as 3).
By decomposing the 3-D convolution operation into spatial
and spectral parts, pseudo-3-D convolution can explore the
correlation between spatial and spectra with less computational
overhead.
III. PROPOSED METHOD
In this section, we will introduce the overall architecture of
the proposed model first. Then, a detailed explanation of each
part will be made, including a description of the loss function. In
this article, the SR HSI is denoted as H sr ∈ RH×W ×B , where
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
Fig. 3.
18505
Main structure of our proposed PARF includes RF-net module, pixel-aware module, and spatial spectrum consistent module.
H, W, and B represent its height, width, and the number of
band, respectively. x ∈ Rh×w×B represents LR HSI, where h,
w, and B represent its height, width, and band number, respectively (here,h H andw W ). The flowchart of the proposed
method is shown in Fig. 3.
As shown in Fig. 3, the proposed PARF mainly consists of
three parts, i.e., the RF-Net module (in blue box), the pixelaware module (in orange box), and the spatial spectrum consistent part (in grey box). We will give an overview of the
entire work, and then introduce the main network structure, i.e.,
PARF in Section III-A, the acquisition of self-attention-based
pixel-aware weights in Section III-B, the optimized pseudo-3-D
convolutional modules in Section III-C, and loss function in
Section III-D.
For the input LR HSI, x ∈ Rh×w×B , we divide the spectral
lr
lr
segments into G groups first, e.g.,x = [elr
1 , . . . , eg , . . . , eG ].
Each of these groups contains c spectral bands, and they will be
fed into the network for reconstruction. The grouping operation
can better utilize the correlation between adjacent spectral bands.
Meanwhile, the feature dimension can also be reduced in the
network.
To fully maintain the continuity between groups, PARF extracts continuous information from the hidden state si , which is
an abstract feature containing the spectral information of previous groups. With the flow of information between groups, the
spatial–spectral information of the previous group is transmitted
to the current group as the form of hidden state, which constrains
the spectrum of the current group and helps reconstruction. The
key process is as follows:
sr
lr b
(2)
êsr
g = RFNet sg−1 , êg−1 , eg , eg , wg−1
b
where elr
g represents the input LR-group, eg is the HSI
sr
after bicubic interpolation, and êg is the output after
reconstruction.êsr
g−1 is the result of the previous group of SR
reconstruction. With the previous set of reconstruction results,
a hidden state sg−1 is also added to capture the long-distance
dependence of the continuity of the spectral sequence. Inspired
by the network of Xie et al. [41], we add pixel-aware weights
wg−1 to obtain the reconstruction difficulty of pixels, and passes
it backward to help the reconstruct of spatial resolution. The
orange box in Fig. 3 is the way to obtain the pixel-aware weights,
where Ei represents the result of the reconstruction of the ith
group, and Ei = êsr
i .E0 is initialized with 0.
sr
eb1 is upsampled by bicubic from elr
1 , and s0 , ê0 , and w0 are all
initialized with 0. With Formula 2, we obtain the reconstruction
result êsr
1 , and update s0 to s1 as the hidden state of second group.
In pixel-aware module, we obtain new weights w1 through E0
sr
and current results êsr
1 . Then, we send the s1 , ê1 , and w1 into
the second group and use the same method as the first group
for reconstruction. Repeat previous operation until the end of
last group of reconstruction. In this way, the high precision
reconstruction can be guaranteed and enhanced through the
information transfer.
Finally, we combine all the reconstructed results group
by group and obtain the reconstructed result, i.e., Ĥ sr =
sr
sr
[êsr
1 , . . . , êg , . . . , êG ]. To ensure the spectral consistency, we
add a deeper 3-D convolution operation to perform the spatial–
spectral regularization to generate the final result. The expression is as follows:
H sr = PP 3D+ Ĥ sr
(3)
18506
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
upsampled image through the residual structure
h
b
ˆh
êsr
g = fg + eg + Up fg
Fig. 4.
Main model structure of recurrent feedback network.
where PP 3D+ (·) represents the pseudo-3-D module, which will
be introduced in detail in Section III-D. H sr represents the final
reconstruction result with high spatial–spectral resolution.
A. Recurrent Feedback Network Based on Pixel-Aware
Optimization
PARF is dedicated to obtaining information from feedback information and complementing the information between groups.
h×w×c
is the gth
Its specific structure is shown in Fig. 4. elr
g ∈R
group input LR image. The reconstruction results êsr
g−1 of the
,
and
the
hidden
state sg−1
previous group, the current group elr
g
on the channel are connected, and then obtain an abstract feature
representation by a 3×3 convolution. The convolution process
is as follows:
lr
.
(4)
fgl = C3×3 sg−1 , êsr
g−1 , eg
Among them, [·] represents the Catch operation on the channel, and C3×3 represents the 3×3 convolution operation.
Considering the correlation between spectral data, RCAB
modules [42] based channel attention mechanism is utilized,
which is expressed as
fˆgh = RRCABD · · · RRCABd · · · RRCAB1 (fgl ) · ·· · ·· . (5)
The RCAB module consists of two convolutional layers
and a channel attention module. The channel attention module
contains a global pooling operation, two convolution operations,
and a softmax function. The correlation information fˆgh ,
obtained by RCAB module, will be input for two branches.
One is the 3×3 convolution operation, which is used to obtain
long-distance dependence sg , acted as the feedback information
of the next group.
Subsequently, pixel perception is added based on the original
RF-Net, functioned as a weight wg−1 , which is the weight to
measure the difficulty of pixels reconstruction, obtained by
calculating the difference between the previous two groups.
Then, the generated weight is applied to the current group, and
formulated as follows to get fgh , which is a feature containing
the spatial weight information:
(6)
fgh = wg−1 Up fˆgh
where U p(fˆgh ) represents the intermediate result after upsampling. Subsequently, we combine the obtained fgh and the
(7)
where fgh is the intermediate result obtained by formula (6), and
ebg is the unsampled image with bicubic interpolation. Up(·) is
denoted as function performed by PixelShuffle operator [43].
This module can fuse the current group’s information with the
preceding group, and transmit to the next group, which enhances
the intergroup information correlation, and greatly improves the
spectral consistency.
B. Self-Attention-Based Pixel-Aware Weights
At present, transformer is widely used in various fields owing
to the excellent performance of self-attention. Self-attention
mechanism can capture more information from image itself. As
mentioned before, the loss of background pixels can reach the
minimum in the early training stage, and then remain unchanged,
while the foreground pixels and details continue convergence.
Here, self-attention assists to search for more foreground regions, making the model fit data better. Then, we combine the
previous reconstruction group and the current one to obtain the
pixel-aware weights for the following reconstruction process,
which can be defined as
(8)
wg = Sigmoid SA (Eg−1 ) − SA êsr
g
where êsr
g represents the reconstruction results of group g,
Eg−1 represents the reconstruction results of the previous group,
and SA(·)represents the self-attention mechanism. This module
cannot only obtain larger receptive field feature representations through self-attention, but also distinguish foreground and
background regions through the difference between two groups
and transmit targeted learning to the next group, which further
enhances the model’s performance especially in spatial domain.
C. Pseudo-3-D+
Two layers of pseudo-3-D convolution has been used in
RF-Net to construct a pseudo-3-D residual block. Although
the spatial–spectral consistency can be maintained to a certain
extent, the extraction and expression of features are limited. To
address this issue, we increased the number of layers of 3-D
convolution and added RELU and regularization in the middle,
making the module more nonlinear, and better abstract features.
Specifically, the final result is obtained by stacking multiple
pseudo-3-D modules
mid1×1×3 = RELU BN CSpe1×1×3 Ĥ sr
PP 3D+1 Ĥ sr = RELU (BN (CSpa3×3×1 (mid1×1×3 )))
H sr = PP 3D+K · · ·PP 3D+k · · ·PP 3D+1 (Ĥ sr ) · ·· · ··
(9)
where mid1×1×3 represent the middle result. CSpa3×3×1 and
CSpe1×1×3 represent spatial convolution and spectral convolution, respectively. Ĥ sr and H sr represent the intermediate result
and the final reconstruction result, respectively. By stacking k
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
layers of pseudo-3-D convolutions, the reassembled reconstruction results can be further regularized, and ensure the spectral
structure consistency. It has been verified after several experiments that a better result can be achieved when k is taken as 3.
D. Loss Function
Most current SR reconstruction methods use pixel-level loss
to measure the distance between the reconstruction result SR
and the real HR image. Due to the oversmoothing result of
the L2 loss, we choose L1 as the context loss to measure the
reconstruction accuracy of the network, which is defined as
18507
spectral information is inherently encoded within the gradient
structure of the hyperspectral data. Preserving the gradient information ensures that the reconstructed image maintains the same
underlying spectral variations as the ground truth. In summary,
the final target loss is defined as
Ltotal (Θ) = Lhr + Llr + λ1 Lspe + λ2 Lgra
(15)
where λ is a tradeoff parameter for different losses. Specifically,
λ1 controls the weight of the spectral term and the gradient term
is constrained by λ2, which controls the weight of the spatial
and spectral structure.
N
Lhr =
1
Hnhr − Hnsr 1 .
N n=1
(10)
A. Datasets and Experimental Setup
Among them, Hnhr and Hnsr are the nth of real HSI and reconstructed HSI, respectively. N is the number of images in a batch.
Considering the inevitable information loss in the process of HR
HSI downsampling to obtain LR HSI, we added the dual loss.
For the reconstructed high-resolution image SR, the LR SR Lsr
n
is obtained through the same downsampling method, and adopt
the L1 as spatial loss, which is specifically expressed as follows:
N
Llr =
1
Llr − Lsr
n 1
N n=1 n
(11)
sr
where Llr
n and Ln are, respectively, the nth of LR HSI and
results after SR downsampling.
Apart from the context loss and dual loss, spectral measurement is also utilized in this model to constrain the spectral
structure, which is expressed as
N
Hnhr · Hnsr
1
1
arccos
.
(12)
Lspe =
N n=1 π
Hnhr 2 Hnsr 2
In addition, as gradient information can enhance the spatial
relationship of SR images [44], [45], in this article, we also
design a gradient map of the HSI by computing the difference
between adjacent pixels, we calculate the gradient maps in the
height (h), width (w), spectral directions (s) dim, and then add
the three results through the L2 norm
M (H) = ∇H2 s.t. ∇H = (∇h H, ∇w H, ∇s H).
(13)
Among them, M ( · )represents the operation of extracting
the gradient map. Considering gradient maps can better reflect
the spatial–spectral dependence of HSIs; to mitigate the spatial
spectral distortion, a gradient constraint is utilized to provide additional supervision and set gradient loss for image reconstruction. The gradient loss is defined as the gradient map difference
between the super-resolved image and the corresponding HR
image
N
Lgra (Θ) =
1
M (Hnhr ) − M (Hnsr ) 1 .
N n=1
IV. EXPERIMENTS AND RESULTS
(14)
By minimizing this loss, the network is encouraged to preserve the gradient structure of the original image, thereby reducing spectral distortions. The rationale behind this is that the
The proposed method is tested on four extensive datasets,
including a natural hyperspectral dataset CAVE [46] and three
hyperspectral remote sensing datasets, i.e., Pavia Centre, Chikusei [47], and Washington DC datasets. Five methods, i.e., bicubic
interpolation, EDSR [28], SSPSR [26], RFSR [36], and GELIN
[37], are selected as comparison algorithms for analysis. Due to
the lack of spectral grouping GAN-based methods for SR of HSI.
Therefore, we have chosen to compare the proposed method
with other state-of-the-art grouping-based methods in order to
make a more direct and relevant comparison within this specific
research area. As the baseline model, RFSR is a recurrent feedback network for HSI SR, focusing on exploiting spectral correlations through group-based processing and spatial–spectral
regularization. Bicubic interpolation as a classical image SR
method is chosen to compare. EDSR is a deep residual network
optimized for image SR by removing redundant modules and expanding network size. SSPSR employs group convolutions and
a spatial–spectral prior network to effectively extract features
from hyperspectral data. GELIN is a grouping-based HSI SR
method that employs spatial–spectral embedding learning and
neighboring group integration to enhance feature extraction and
reconstruction.
Meanwhile, five reference indicators, including average
PSNR, spectral angle mapper (SAM), structural similarity index
index, relative global error (ERGAS), and root mean square error
(RMSE), are adopted for quantitative evaluation.
B. Experimental Results on CAVE
The CAVE dataset contains 32 HR HSIs, and the size of each
HSI is 512×512 with 31 bands. We randomly select 23 HSIs as
training set, 2 images as validation set, and 6 HSIs as testing set.
For each image in the dataset, a 64×64 region with a step size
of 32 pixels is extracted to increase the number of images used
for training. We use zero value to fill the spectral band to 32
bands and divide them into 8 groups during training procedure.
The LR HSI is generated by bicubic downsampling of HR HSI.
In the test phase, we divide the image into the same size as the
training set and send it to the model for testing. The average
value of each index is taken as the final data of the image.
In the experiment, the scale factor is set as 4 and 8, and
the results are shown in Table I. The bold represents the
18508
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
TABLE I
QUANTITATIVE EVALUATION RESULTS OF DIFFERENT METHODS ON THE CAVE DATASET
Fig. 5.
Error maps of three test HSIs in the CAVE dataset.
best result, and the underline represents the second-best result. Since bicubic only reconstructs values by interpolation,
the results of spectral reconstruction are far inferior to other
methods, which can be observed in Table I. As the MPNSR,
deep-learning-based methods can reach more than 42 dB, but
bicubic is only 40.2358 dB. Furthermore, due to the lack of
spectral information, the SAM of bicubic reaches the lowest
at 5.484. Differing from bicubic interpolation reconstruction,
EDSR focuses on the use of spatial information, enhancing
spatial relationship effectively, as MPSNR indicate that it has
1.8 dB higher than the bicubic. Similarly, it has better results
on ERGAS and RMSE. However, due to lack of spectral consideration, spectral distortion is unavoidable, the SAM index
also shows that the method leads to obvious spectral distortion.
Through grouping strategy, SSPSR and RFSR not only consider
spatial information, but also impose spectral similarity, and there
is a significant improvement in SAM, which is about 0.3 higher
than EDSR. GELIN excels in both spatial and spectral performance by associating spatial and spectral information from more
groups. While compared with the proposed method, GELIN
shows slightly inferior to the proposed method for both 4 and 8
scale in all indicators. Specifically, in terms of spatial resolution
reconstruction, the proposed method surpasses the second-best
approach by approximately 0.74 dB, which is attributed to the
pixel perception weight and provides more reliable spatial and
spectral information for reconstruction. And in terms of spectral
fidelity, the application of optimized pseudo-3-D convolution
operation provides more spectral information to the model,
and the proposed method outperforms the second-best method
by 0.172.
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
Fig. 6.
18509
Error maps of two test HSIs in the CAVE dataset.
TABLE II
QUANTITATIVE EVALUATION RESULTS OF DIFFERENT METHODS ON THE PAVIA CENTRE DATASET
To better compare the different SR reconstruction methods,
we calculate the error between the reconstruction results and
ground truths from both spatial and spectral aspects. Fig. 5 shows
three examples of the difference from CAVE dataset on the 30th
channel at the scale factor 4, and the PSNR and SAM values
of the reconstructed image are labeled at the bottom for comparison. Fig. 6 is the reconstruction result at the scale factor 8.
From Fig. 6, it can be noticed the proposed method inherits
the advantages of the RFSR method in restoring the spatial
information of HSIs, and performs better in the reconstruction
of object edges, especially for the marked region compared with
the EDSR and SSPSR.
To further compare the RFSR and proposed methods, we also
compare the reconstructed images of the two methods on the
same image and use the error map. As shown in Fig. 7, the
proposed method is significantly better in edge and object texture
reconstruction, owing to the training of self-attention-based
pixel-aware weights.
Fig. 7.
Error maps of balloons and flowers between RFSR and PARF.
In addition, the proposed model has also achieved the best
results in spectral restoration, and the SAM index reached a
satisfying level. Fig. 8 shows the spectral error curves of the
reconstructed image and the ground truths on each band. It
18510
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
divide the image into multiple 64×64 sizes by taking 32 pixels as
the step size, and obtain the training set of downsampling 4-fold
and 8-fold by means of bicubic interpolation. During training,
to divide the whole spectral bands into 13 groups, 2 zero-value
bands are constructed and put in the final group. In the test phase,
we segment the image into the same size as training set and send
it to the model for testing. The average value of each index is
taken as the final data of the image.
Table II shows the quantitative results of the five methods on
the evaluation indicators. From the results, it can be observed
the proposed method is obviously better than other methods on
both scales, which shows that the proposed method is robustness
in both natural images and remote sensing images.
Fig. 9 also shows the overall error map of different test images
at the scale factor 4. It can be noticed that the proposed model
has smaller errors than other models in loss of roofs and river
banks. This is mainly attributed to pixel-aware weight based
on self-attention that can help reconstruct the missing spatial
information through intermediate results.
Fig. 10 displays the spectral error. Compared to CAVE dataset,
the Pavia dataset exhibits a noticeable increase in the number of
bands, which makes spectral consistency more challenge. It can
be observed that the grouping-based methods (SSPSR, RFSR,
GELIN, and proposed method) demonstrate a significant advantage in spectral consistency over bicubic interpolation and EDSR
by effectively capturing the correlations between distant spectral
bands. While incorporating multiple layers of pseudo-3-D modules for spectral regularization, the proposed method obtains
smaller average differences and illustrates greater stability in
this regard.
In Fig. 11, we select the 60th, 31st, and 12th band to synthesize
true color image, and visualize the reconstruction results with
a scale factor of 4. From the figure, it can be observed EDSR
and SSPSR are partially blurred, and the images have significantly spectral distortion. Although the baseline model RFSR
has a relatively satisfying performance, it is not smooth in the
reconstruction of edge information. The results obtained by the
proposed method is better visually.
Fig. 8. Mean spectral difference curve in the CAVE dataset at the scale factor 4.
(a) Balloons.mat. (b) Flowers.mat. (c) Egyptian_statue.mat.
can be found the loss of the proposed method on each band
can mostly reach the minimum, mainly because the feedback
information is added to the spectral information. Besides, the
optimized pseudo-3-D convolution also provides the model with
more nonlinearity by stacking multiple 3-D convolution, which
further improves the results of spectral regularization.
C. Experimental Results on Pavia
The Pavia Centre dataset is a scene captured by the ROSIS
sensor as it flew over Pavia, northern Italy, and contains 102 spectral bands. The Pavia Centre dataset is a 1096×1096 HSI. Since
some areas in the original image contain limited information, we
discard this part and finally obtain a 1096×715 image. First, we
D. Experimental Results on Chikusei
The Chikusei dataset contains HSI of Ibaraki, Japan, obtained
by the Hyperspec–VNIR–CIRIS. The size is 2517 × 2335, and
128 bands in the spectral range of 363–1018 nm. We select the
part of 2048 × 2048 to evaluate the proposed method. Similar
to the above datasets, it will first be divided into 16 images with
512 × 512, and then be divided into 64 × 64 as the HR image,
and perform 4-fold and 8-fold downsampling through bicubic
interpolation to get the LR HSI, which contains 128 bands. Then,
grouping is taken to divide these data into 16 spectral groups,
and each group contains 8 spectral bands.
For Pavia data, due to the high spatial resolution and abundant
spectral information, EDSR exhibits inferior performance in
spatial and spectral reconstruction, and also performs poorly on
Chikusei dataset. Therefore, for the Chikusei dataset, we solely
conduct comparative experiments with the bicubic, SSPSR,
RFSR, and GELIN models. Table III presents the quantitative
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
Fig. 9.
18511
Error maps of three test HSIs in the Pavia Centre dataset.
TABLE III
QUANTITATIVE EVALUATION RESULTS OF DIFFERENT METHODS ON THE
CHIKUSEI DATASET
Fig. 10. Mean spectral difference curve of Pavia Centre dataset at scale
factor 4/8. (a) Spectral error under the scale factor of 4. (b) Spectral error under
the scale factor of 8.
results for the four methods under 4× and 8× reconstruction.
Proposed method outperforms the second-best GELIN method
by approximately 0.35 dB in MPSNR. As the increase of spectra, proposed method performs better in keeping the spectral
structure, while bicubic is significantly worse than the other
methods due to the fact that it only reconstructs values by interpolation. The other grouping-based methods typically achieve
SAM values around 3.3 to 3.5, proposed method achieves 2.85,
highlighting the advantages of pseudo-3-D+ in preserving spectral structure. Despite encountering challenges in 8× reconstruction ratio due to information loss during downsampling,
proposed method still achieves the best results across all indices
compared to the baseline methods.
Fig. 12 shows the error map of three test images. The darker
the color, the smaller the error. It can be clearly seen that errors
of the proposed method are the smallest for large farmland and
the junction of river water and river bank owing to the excellent
feature extraction ability of the self-attention mechanism for
18512
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
Fig. 11. Reconstructed images of two test HSIs in the Pavia Centre dataset
with spectral bands 60-31-12 as R-G-B with the scale factor 4.
large patch areas. In Fig. 13, to better visualize the results,
the 70th, 100th, and 36th bands are selected to synthesize true
color image with scale factor of 4 and 8. It can be noticed
other methods have a certain blur in the visual sense, while the
proposed method is much clearer.
Fig. 12.
factor.
Mean spectral difference curve in the Chikusei dataset at the scale
E. Experimental Results on Washington DC
Washington DC dataset is an aerial HSI acquired by HYDICE
sensor, primarily focusing on Washington DC shopping center.
The data comprise 191 bands ranging from 0.4 to 2.4 μm,
covering both visible and near-infrared spectral regions. The
size of image is 1208×307, capturing features such as roofs,
streets, paved roads, grasslands, trees, water bodies, and shadows. Following the same methodology as before, we partitioned
the dataset into 64×64 patches first, and then performed bicubic
downsampling by factors of 4 and 8 to construct the training set.
Differing from the previous approach, we selected images size
of 256×256 for testing due to the limitations of original image
size. In training process, we also use zero value to fill the spectral
band to 192 bands and divide it into 24 spectral groups, and each
group contains 8 spectral bands. The experimental results are
presented in Table IV in detail.
In Table IV, as the number of bands reaches 191, the proposed
model still achieves 0.55 dB higher than the second-best method
in MPSNR indicator. Concerning spectral metrics, RFSR faces
challenges in capturing long-distance correlations between
groups due to its restricted information transfer, which is
limited to the preceding group and the subsequent one, and
results in difficulties in maintaining spectral structure. While
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
18513
TABLE IV
QUANTITATIVE EVALUATION RESULTS OF DIFFERENT METHODS ON THE
WASHINGTON DC DATASET
Fig. 13. Reconstructed images in the Chikusei dataset with spectral bands
70-100-36 as R-G-B with the scale factor 4 and 8.
GELIN is adept at preserving spectral structure as it can
continually associate information between the current group
and its preceding and succeeding groups (three groups in total).
The proposed method also achieves outstanding results by
collecting spatial and spectral information from the first two
groups during the transmission process.
Fig. 14. Mean spectral difference curve in the Washington DC dataset.
(a) Spectral error under the scale factor of 4. (b) Spectral error under the scale
factor of 8.
Fig. 14 illustrates the spectral errors of different methods on
Washington DC. As previously analyzed, the spectral error of
RFSR usually shows a relatively poorer performance when the
number of bands is large, due to the limited information transfer
between groups. In contrast, the proposed method can maintain the as much as spectral feature benefiting from enhanced
information transmission ability.
In Fig. 15, the 60th, 27th, and 17th bands are selected as
RGB bands and synthesized true color image with scale factor
18514
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
TABLE V
QUANTITATIVE PERFORMANCE OF DIFFERENT LOSS FUNCTIONS
TABLE VI
QUANTITATIVE PERFORMANCE OF DIFFERENT MODULE
Fig. 15. Reconstructed images in the Washington DC dataset with spectral
bands 60-27-17 as R-G-B.
of 4 and 8. From Fig. 15, it can be observed the proposed method
exhibits clearer and sharper textures due to the better capturing of
spatial structure by pixel-aware weight based on self-attention.
F. Ablation Study
Within the proposed method, the pixel-aware weights,
pseudo-3-D+ and the losses will influence the performance.
To testify how they affect the performance of the proposed
method, the following ablation studies were conducted on the
Chikusei dataset. In ablation experiments, RFSR is used as the
baseline model, and L1 loss and spectral loss are employed as
constraints. Subsequent experiments are conducted by, respectively, incorporating the dual loss and gradient loss to validate
the effectiveness of adding losses. The results are presented in
Table V.
The results in Table V show that the addition of dual loss and
gradient loss can slightly improve the accuracy, but the effect
is minimal. Only by combining the two losses, the accuracy
of reconstruction will have a certain gain. Dual loss reduces
the size of solution space of model, so that the model can be
solved better. Gradient loss uses the gradient information of the
spectrum to enhance the model’s constraints on the spectrum
and avoid spectral distortion as much as possible.
Further ablation experiments were conducted on the proposed self-attention-based pixel-aware weights and enhanced
pseudo-3-D+ convolutions, Table VI shows the results. Without
pixel-aware weights and pseudo-3-D+ module, the quantitative
evaluation indexes reach the lowest level. After integrating the
self-attention-based pixel-wise weights, the spatial performance
has a significant enhancement, improved by 0.28 dB. Similarly,
the introduction of pseudo-3-D+ regularization also optimize
the spectral structure by applying regularization operations from
both spatial and spectral perspectives. According to the results
in Table VI, the proposed method could reconstruct high-quality
HR estimation when using the two modules, indicating the
effectiveness of the proposed modules.
V. CONCLUSION
In this article, to further improve the spatial resolution of
HSI SR reconstruction and maintain spectral consistency, selfattention-based pixel-aware weights is proposed to add to the
recurrent feedback SR reconstruction model. In addition, the
grouping operation is also used for HSIs for spectral consistency,
and the correlation between the spectra is fully utilized by
intergroup information transmission and enhanced pseudo-3-D
FENG et al.: RECURRENT FEEDBACK HYPERSPECTRAL IMAGE SUPER-RESOLUTION RECONSTRUCTION METHOD
convolution. Meanwhile, the pixel perception weight is continuously updated through the reconstruction results of the previous
group to help the reconstruction of the subsequent spectral
groups. Besides, the structure of pseudo-3-D convolution is
optimized, and the model can better regularize the spectrum.
On the loss function, dual loss as well as gradient loss is used to
enhance the SR reconstruction. Experiments with four datasets
have achieved excellent results in both qualitative and quantitative aspects. While the proposed method achieves promising
results, there is still room for improvement. The computational
cost associated with self-attention and the dataset-specific nature
of grouping strategies are limitations. Future article can focus on
developing more efficient architectures and exploring adaptive
grouping strategies. Moreover, the potential applications of our
method in various domains, such as remote sensing and medical
imaging, warrant further investigation.
REFERENCES
[1] W. Zhao and S. Du, “Spectral–spatial feature extraction for hyperspectral image classification: A dimension reduction and deep learning approach,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 8, pp. 4544–4554,
Aug. 2016, doi: 10.1109/TGRS.2016.2543748.
[2] J. Hu, T. Li, M. Zhao, and J. Ning, “Hyperspectral image superresolution
via deep structure and texture interfusion,” IEEE J. Sel. Topics Appl. Earth
Observ. Remote Sens., vol. 14, pp. 8665–8678, 2021, doi: 10.1109/JSTARS.2021.3107311.
[3] S. Puustinen et al., “Towards clinical hyperspectral imaging (HSI)
standards: Initial design for a microneurosurgical HSI database,” in
Proc. IEEE 35th Int. Symp. Comput. Med. Syst., 2022, pp. 394–399,
doi: 10.1109/CBMS55023.2022.00077.
[4] C. Cevoli, L. Di Cecilia, L. Ferrari, A. Fabbri, and G. Molari, “Potential of
in-field vis/NIR hyperspectral imaging to monitor quality parameters of
alfalfa,” in Proc. IEEE Int. Workshop Metrol. Agriculture Forestry, 2021,
pp. 341–345, doi: 10.1109/MetroAgriFor52389.2021.9628816.
[5] D. Lorts, “Real-time embedded hyperspectral image compression for
tactical military platforms,” in Proc. Appl. Imagery Pattern Recognit.
Workshop, 2002, p. 140, doi: 10.1109/AIPR.2002.1182267.
[6] R. Dian, S. Li, and L. Fang, “Learning a low tensor-train rank representation for hyperspectral image super-resolution,” IEEE Trans. Neural Netw. Learn. Syst., vol. 30, no. 9, pp. 2672–2683, Sep. 2019,
doi: 10.1109/TNNLS.2018.2885616.
[7] W. Dong et al., “Hyperspectral image super-resolution via non-negative
structured sparse representation,” IEEE Trans. Image Process., vol. 25,
no. 5, pp. 2337–2352, May 2016, doi: 10.1109/TIP.2016.2542360.
[8] M. R. Vicinanza, R. Restaino, G. Vivone, M. Dalla Mura, and J. Chanussot,
“A pansharpening method based on the sparse representation of injected
details,” IEEE Geosci. Remote Sens. Lett., vol. 12, no. 1, pp. 180–184,
Jan. 2015, doi: 10.1109/LGRS.2014.2331291.
[9] R. Fei, J. Zhang, J. Liu, F. Du, P. Chang, and J. Hu, “Convolutional sparse representation of injected details for pansharpening,” IEEE
Geosci. Remote Sens. Lett., vol. 16, no. 10, pp. 1595–1599, Oct. 2019,
doi: 10.1109/LGRS.2019.2904526.
[10] H. Chang, D.Y. Yeung, and Y. Xiong, “Super-resolution through neighbor
embedding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2004,
p. I, doi: 10.1109/cvpr.2004.1315043.
[11] D. Zhou, Y. Liu, X. Li, and C. Zhang, “Single-image super-resolution based
on local biquadratic spline with edge constraints and adaptive optimization
in transform domain,” Vis. Comput., vol. 38, no. 1, pp. 119–134, Jan. 2022,
doi: 10.1007/s00371-020-02007-z.
[12] J. Liu, Z. Gan, and X. Zhu, “Directional bicubic interpolation—A new
method of image super-resolution,” in Proc. 3rd Int. Conf. Multimedia
Technol., 2013, pp. 470–477, doi: 10.2991/icmt-13.2013.57.
[13] N. Akhtar, F. Shafait, and A. Mian, “Bayesian sparse representation for hyperspectral image super resolution,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2015, pp. 3631–3640,
doi: 10.1109/CVPR.2015.7298986.
[14] Y. Xu, Z. Wu, J. Chanussot, and Z. Wei, “Nonlocal patch tensor
sparse representation for hyperspectral image super-resolution,” IEEE
Trans. Image Process., vol. 28, no. 6, pp. 3034–3047, Jun. 2019,
doi: 10.1109/TIP.2019.2893530.
18515
[15] X.-H. Han, B. Shi, and Y. Zheng, “Self-similarity constrained
sparse representation for hyperspectral image super-resolution,” IEEE
Trans. Image Process., vol. 27, no. 11, pp. 5625–5637, Nov. 2018,
doi: 10.1109/TIP.2018.2855418.
[16] R. Dian, L. Fang, and S. Li, “Hyperspectral image super-resolution via nonlocal sparse tensor factorization,” in Proc. IEEE/CVF Conf. Comput. Vis.
Pattern Recognit., 2017, pp. 3862–3871, doi: 10.1109/CVPR.2017.411.
[17] R. Dian and S. Li, “Hyperspectral image super-resolution via
subspace based low tensor multi-rank regularization,” IEEE Trans.
Image Process., vol. 28, no. 10, pp. 5135–5146, Oct. 2019,
doi: 10.1109/TIP.2019.2916734.
[18] J. Xue, Y.-Q. Zhao, Y. Bu, W. Liao, J. C.-W. Chan, and W. Philips, “Spatialspectral structured sparse low-rank representation for hyperspectral image
super-resolution,” IEEE Trans. Image Process., vol. 30, pp. 3084–3097,
2021, doi: 10.1109/TIP.2021.3058590.
[19] S. H. Mahajan and V. K. Harpale, “Adaptive and non-adaptive image
interpolation techniques,” in Proc. Int. Conf. Comput. Commun. Control
Automat., 2015, pp. 772–775, doi: 10.1109/ICCUBEA.2015.154.
[20] Y. Li, J. Hu, X. Zhao, W. Xie, and J. Li, “Hyperspectral image superresolution using deep convolutional neural network,” Neurocomputing,
vol. 266, pp. 29–41, 2017, doi: 10.1016/j.neucom.2017.05.024.
[21] Y. Li, L. Zhang, C. Dingl, W. Wei, and Y. Zhang, “Single hyperspectral image super-resolution with grouped deep recursive residual network,” in Proc. IEEE 4th Int. Conf. Multimedia Big Data, 2018, pp. 1–4,
doi: 10.1109/BigMM.2018.8499097.
[22] S. Mei, X. Yuan, J. Ji, Y. Zhang, S. Wan, and Q. Du, “Hyperspectral image
spatial super-resolution via 3D full convolutional neural network,” Remote
Sens., vol. 9, no. 11, 2017, Art. no. 1139, doi: 10.3390/rs9111139.
[23] J. Yang, Y. Zhao, C. Chan, and L. Xiao, “A multi-scale wavelet 3D-CNN
for hyperspectral image super-resolution,” Remote Sens., vol. 11, no. 13,
2019, Art. no. 1557, doi: 10.3390/rs11131557.
[24] Q. Li, Q. Wang, and X. Li, “Mixed 2D/3D convolutional network for
hyperspectral image super-resolution,” Remote Sens., vol. 12, no. 10, 2020,
Art. no. 1660, doi: 10.3390/rs12101660.
[25] Q. Wang, Q. Li, and X. Li, “Spatial-spectral residual network for hyperspectral image super-resolution,” 2020, arXiv:2001.04609.
[26] J. Jiang, H. Sun, X. Liu, and J. Ma, “Learning spatial-spectral prior for
super-resolution of hyperspectral imagery,” IEEE Trans. Comput. Imag.,
vol. 6, pp. 1082–1096, 2020, doi: 10.1109/TCI.2020.2996075.
[27] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using
very deep convolutional networks,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., 2016, pp. 1646–1654, doi: 10.1109/CVPR.2016.182.
[28] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep
residual networks for single image super-resolution,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit. Workshops, 2017, pp. 1132–1140,
doi: 10.1109/CVPRW.2017.151.
[29] M. Zareapoor, M. E. Celebi, and J. Yang, “Diverse adversarial network
for image super-resolution,” Signal Process.-Image Commun., vol. 74,
pp. 191–200, 2019, doi: 10.1016/j.image.2019.02.008.
[30] K. Jiang, Z. Wang, P. Yi, G. Wang, T. Lu, and J. Jiang, “Edgeenhanced GAN for remote sensing image superresolution,” IEEE Trans.
Geosci. Remote Sens., vol. 57, no. 8, pp. 5799–5812, Aug. 2019,
doi: 10.1109/TGRS.2019.2902431.
[31] Y. Huang, Z. Jiang, R. Lan, S. Zhang, and K. Pi, “Infrared image superresolution via transfer learning and PSRGAN,” IEEE Signal Process. Lett.,
vol. 28, pp. 982–986, 2021, doi: 10.1109/LSP.2021.3077801.
[32] Y. Li, L. Zhang, C. Dingl, W. Wei, and Y. Zhang, “Single hyperspectral image super-resolution with grouped deep recursive residual network,” in Proc. IEEE 4th Int. Conf. Multimedia Big Data, 2018, pp. 1–4,
doi: 10.1109/BigMM.2018.8499097.
[33] J. Hu, X. Jia, Y. Li, G. He, and M. Zhao, “Hyperspectral image superresolution via intrafusion network,” IEEE Trans. Geosci. Remote Sens., vol. 58,
no. 10, pp. 7459–7471, Oct. 2020, doi: 10.1109/TGRS.2020.2982940.
[34] S. Mei, X. Yuan, J. Ji, Y. Zhang, S. Wan, and Q. Du, “Hyperspectral image
spatial super-resolution via 3D full convolutional neural network,” Remote
Sens., vol. 9, no. 11, 2017, Art. no. 1139, doi: 10.3390/rs9111139.
[35] J. Li et al., “Hyperspectral image super-resolution by band attention
through adversarial learning,” IEEE Trans. Geosci. Remote Sens., vol. 58,
no. 6, pp. 4304–4318, Jun. 2020, doi: 10.1109/TGRS.2019.2962713.
[36] X. Wang, J. Ma, and J. Jiang, “Hyperspectral image super-resolution via
recurrent feedback embedding and spatial–spectral consistency regularization,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no. 5503113,
doi: 10.1109/TGRS.2021.3064450.
[37] X. Wang, Q. Hu, J. Jiang, and J. Ma, “A group-based embedding learning and integration network for hyperspectral image super-resolution,”
IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no. 5541416,
doi: 10.1109/TGRS.2022.3217406.
18516
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 17, 2024
[38] Q. Li, Y. Yuan, X. Jia, and Q. Wang, “Dual-stage approach toward hyperspectral image super-resolution,” IEEE Trans. Image Process., vol. 31,
pp. 7252–7263, 2022, doi: 10.1109/TIP.2022.3221287.
[39] W. Wei, J. Nie, L. Zhang, and Y. Zhang, “Unsupervised recurrent
hyperspectral imagery super-resolution using pixel-aware refinement,”
IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no. 5500315,
doi: 10.1109/TGRS.2020.3039534.
[40] Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with
pseudo-3D residual networks,” in Proc. IEEE Int. Conf. Comput. Vis.,
2017, pp. 5534–5542, doi: 10.1109/ICCV.2017.590.
[41] S. Xie, S. Chen, H. Jonathan, Z. Tu, and K. P. Murphy, “Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video
classification,” in Proc. Eur. Conf. Comput. Vis., 2017, pp. 318–335,
doi: 10.1007/978-3-030-01267-0_19.
[42] C. Dong, C. C. Loy, K. He, and X. Tang, “Image superresolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295–307, Feb. 2016,
doi: 10.1109/TPAMI.2015.2439281.
[43] W. Shi et al., “Real-time single image and video super-resolution
using an efficient sub-pixel convolutional neural network,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1874–1883,
doi: 10.1109/CVPR.2016.207.
[44] Q. Yan, Y. Xu, X. Yang, and T. Q. Nguyen, “Single image superresolution
based on gradient profile sharpness,” IEEE Trans. Image Process., vol. 24,
no. 10, pp. 3187–3202, Oct. 2015, doi: 10.1109/TIP.2015.2414877.
[45] J. Sun, J. Sun, Z. Xu, and H.-Y. Shum, “Gradient profile prior and
its applications in image super-resolution and enhancement,” IEEE
Trans. Image Process., vol. 20, no. 6, pp. 1529–1542, Jun. 2011,
doi: 10.1109/TIP.2010.2095871.
[46] F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar, “Generalized assorted
pixel camera: Postcapture control of resolution, dynamic range, and
spectrum,” IEEE Trans. Image Process., vol. 19, no. 9, pp. 2241–2253,
Sep. 2010, doi: 10.1109/TIP.2010.2046811.
[47] N. Yokoya and A. Iwasaki, “Airborne hyperspectral data over chikusei,”
Space Appl. Lab., Univ. Tokyo, Tokyo, Japan, Tech. Rep. SAL-2016-0527, 2016.
Ruyi Feng (Member, IEEE) received the B.S. degree
in geographic information system from the Hunan
Normal University, Changsha, China, in 2011, and the
M.S. degree in surveying and mapping engineering
and the Ph.D. degree in photogrammetry and remote
sensing from the Wuhan University, Wuhan, China,
in 2013 and 2016, respectively.
She has been with the School of Computer Science,
China University of Geosciences, Wuhan, since 2016,
and is currently an Associate Professor. Her research
interests include sparse representation, deep learning,
hyperspectral image analysis, high-resolution remote sensing understanding,
and intelligent interpretation of remote sensing imagery.
Zhongyu Guo received the B.E. degree from the
Hubei University of Education, Wuhan, China, in
2021, and the M.E. degree in computer technology
from the China University of Geosciences, Wuhan,
in 2024.
His research interests include object detection,
super-resolution reconstruction, change detection,
and deep learning.
Xiaofeng Wang received the doctoral degree in structural geology from the China University of Geosciences, Wuhan, China, in 2015.
He has been with the China University of Geosciences, since 2013, and engaged in teaching and
research in the field of geosciences information engineering. His research interests include deep learning
for remote sensing, semiautomated lineament detection, and geoscience information engineering.