Abstract
Hyperspectral remote sensing is well-known for its extraordinary spectral distinguishability to discriminate different materials. However, the cost of hyperspectral image (HSI) acquisition is much higher compared to traditional RGB imaging. In addition, spatial and temporal resolutions are sacrificed to obtain very high spectral resolution owing to the limitations of sensor technologies. Therefore, in this paper, HSIs are reconstructed using easily acquired RGB images and a convolutional neural network (CNN). As a result, high spatial and temporal resolution RGB images can be inherited to HSIs. Specifically, a two-stage CNN, referred to as the spectral super-resolution network (SSR-Net), is designed to learn the transformation model between RGB images and HSIs from training data, including a band prediction network (BP-Net) to estimate hyperspectral bands from RGB images and a refinement network (RF-Net) to further reduce spectral distortion in the band prediction step. As a result, the learned joint features in the proposed SSR-Net can directly predict HSIs from their corresponding scenes in RGB images without prior knowledge. Experimental results obtained on several benchmark datasets demonstrate that the proposed SSR-Net outperforms several state-of-the-art methods by ensuring higher quality in HSI reconstruction, and significantly improves the performance of traditional RGB images in classification.
Similar content being viewed by others
References
Li J, Marpu P R, Plaza A, et al. Generalized composite kernel framework for hyperspectral image classification. IEEE Trans Geosci Remote Sens, 2013, 51: 4816–4829
Li W, Du Q, Zhang B. Combined sparse and collaborative representation for hyperspectral target detection. Pattern Recogn, 2015, 48: 3904–3916
Huang X, Zhang L. An SVM ensemble approach combining spectral, structural, and semantic features for the classification of high-resolution remotely sensed imagery. IEEE Trans Geosci Remote Sens, 2013, 51: 257–272
Ma M, Mei S, Wan S, et al. Video summarization via block sparse dictionary selection. Neurocomputing, 2020, 378: 197–209
Zhang Z J, Pang Y W. CGNet: cross-guidance network for semantic segmentation. Sci China Inf Sci, 2020, 63: 120104
Ma S, Pang Y W, Pan J, et al. Preserving details in semantics-aware context for scene parsing. Sci China Inf Sci, 2020, 63: 120106
**e J, Pang Y W, Cholakkal H, et al. PSC-Net: learning part spatial co-occurrence for occluded pedestrian detection. Sci China Inf Sci, 2021, 64: 120103
Cao J, Pang Y, Li X. Learning multilayer channel features for pedestrian detection. IEEE Trans Image Process, 2017, 26: 3210–3220
Alparone L, Wald L, Chanussot J, et al. Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Trans Geosci Remote Sens, 2007, 45: 3012–3021
Bendoumi M A, He M Y, Mei S H. Hyperspectral image resolution enhancement using high-resolution multispectral image based on spectral unmixing. IEEE Trans Geosci Remote Sens, 2014, 52: 6574–6583
Zhang Y. Spatial resolution enhancement of hyperspectral image based on the combination of spectral mixing model and observation model. In: Proceedings of SPIE, 2014. 9244: 201–204
Li X, Ling F, Foody G M, et al. Generating a series of fine spatial and temporal resolution land cover maps by fusing coarse spatial resolution remotely sensed images and fine spatial resolution land cover maps. Remote Sens Environ, 2017, 196: 293–311
Fu Y, Zheng Y, Huang H, et al. Hyperspectral image super-resolution with a Mosaic RGB image. IEEE Trans Image Process, 2018, 27: 5539–5552
Zhang L, Wei W, Zhang Y, et al. Cluster sparsity field: an internal hyperspectral imagery prior for reconstruction. Int J Comput Vis, 2018, 126: 797–821
Liebel L, Körner M. Single-image super resolution for multispectral remote sensing data using convolutional neural networks. Int Arch Photogramm Remote Sens Spatial Inf Sci, 2016, XLI-B3: 883–890
Li Y, Hu J, Zhao X, et al. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing, 2017, 266: 29–41
Hu J, Li Y, **e W. Hyperspectral image super-resolution by spectral difference learning and spatial error correction. IEEE Geosci Remote Sens Lett, 2017, 14: 1825–1829
Mei S, Yuan X, Ji J, et al. Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens, 2017, 9: 1139
Mei S, Jiang R, Li X, et al. Spatial and spectral joint super-resolution using convolutional neural network. IEEE Trans Geosci Remote Sens, 2020. doi: https://doi.org/10.1109/TGRS.2020.2964288
Nguyen R M H, Prasad D K, Brown M S. Training-based spectral reconstruction from a single RGB image. In: Proceedings of European Conference on Computer Vision, 2014. 186–201
Arad B, Ben-Shahar O. Sparse recovery of hyperspectral signal from natural RGB images. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2016. 19–34
Yi C, Zhao Y Q, Chan J C W. Spectral super-resolution for multispectral image based on spectral improvement strategy and spatial preservation strategy. IEEE Trans Geosci Remote Sens, 2019, 57: 9010–9024
Jia Y, Zheng Y, Gu L, et al. From RGB to spectrum for natural scenes via manifold-based map**. In: Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV), 2017. 4715–4723
Arad B, Ben-Shahar O, Timofte R, et al. NTIRE 2018 challenge on spectral reconstruction from RGB images. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Los Alamitos, 2018
Can Y B, Timofte R. An efficient CNN for spectral reconstruction from RGB images. 2018. Ar**v: 1804.04647
Han X, Yu J, Xue J, et al. Spectral super-resolution for RGB images using class-based BP neural networks. In: Proceedings of 2018 Digital Image Computing: Techniques and Applications (DICTA), 2018. 1–7
Koundinya S, Sharma H, Sharma M, et al. 2D-3D CNN based architectures for spectral reconstruction from RGB images. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018
Zhang L, Zhang L, Du B. Deep learning for remote sensing data: a technical tutorial on the state of the art. IEEE Geosci Remote Sens Mag, 2016, 4: 22–40
Mei S, Ji J, Hou J, et al. Learning sensor-specific spatial-spectral features of hyperspectral images via convolutional neural networks. IEEE Trans Geosci Remote Sens, 2017, 55: 4520–4533
Yuan Q, Zhang Q, Li J, et al. Hyperspectral image denoising employing a spatial-spectral deep residual convolutional neural network. IEEE Trans Geosci Remote Sens, 2019, 57: 1205–1218
Zhang M, Li W, Du Q. Diverse region-based CNN for hyperspectral image classification. IEEE Trans Image Process, 2018, 27: 2623–2634
Yasuma F, Mitsunaga T, Iso D, et al. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans Image Process, 2010, 19: 2241–2253
Choi I, Jeon D S, Nam G, et al. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans Graph, 2017, 36: 1–13
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. Ar**v: 1409.1556
Chen L, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation. 2017. Ar**v: 1706.05587
Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. In: Proceedings of International Conference on Learning Representations, 2016
Zhao H, Gallo O, Frosio I, et al. Loss functions for image restoration with neural networks. IEEE Trans Comput Imag, 2017, 3: 47–57
Saxe A M, Mcclelland J L, Ganguli S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In: Proceedings of International Conference on Learning Representations, 2014. 1–22
Kingma D P, Ba J. Adam: a method for stochastic optimization. 2014. Ar**v:1412.6980
Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. 2015. Ar**v: 1502.03167
Camps-Valls G, Gomez-Chova L, Munoz-Mari J, et al. Composite kernels for hyperspectral image classification. IEEE Geosci Remote Sens Lett, 2006, 3: 93–97
Acknowledgements
This work was partially supported by National Natural Science Foundation of China (Grant No. 61671383). Seed Foundation of Innovation and Creation for Graduate Students in Northwestern Polytechnical University (Grant No. CX2020020), and Hong Kong RGC (Grant No. 9042820 (CityU 11219019)).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mei, S., Geng, Y., Hou, J. et al. Learning hyperspectral images from RGB images via a coarse-to-fine CNN. Sci. China Inf. Sci. 65, 152102 (2022). https://doi.org/10.1007/s11432-020-3102-9
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11432-020-3102-9