We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.


SpringerLink
Log in

Multi-FAN: multi-spectral mosaic super-resolution via multi-scale feature aggregation network

  • Short Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper introduces a novel method to super-resolve multi-spectral images captured by modern real-time single-shot mosaic image sensors, also known as multi-spectral cameras. Our contribution is twofold. Firstly, we super-resolve multi-spectral images from mosaic images rather than image cubes, which helps to take into account the spatial offset of each wavelength. Secondly, we introduce an external multi-scale feature aggregation network (Multi-FAN) which concatenates the feature maps with different levels of semantic information throughout a super-resolution (SR) network. A cascade of convolutional layers then implicitly selects the most valuable feature maps to generate a mosaic image. This mosaic image is then merged with the mosaic image generated by the SR network to produce a quantitatively superior image. We apply our Multi-FAN to RCAN (residual channel attention network), which is the state-of-the-art SR algorithm. We show that Multi-FAN improves both quantitative results (PSNR and SSIM) and inference time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and, lightweight super-resolution with cascading residual network. ar**v preprint ar**v:1803.08664 (2018)

  2. Bell, J.F., Wellington, D., Hardgrove, C., Godber, A., Rice, M.S., Johnson, J.R., Fraeman, A.: Multispectral imaging of mars from the mars science laboratory mastcam instruments: Spectral properties and mineralogic implications along the gale crater traverse. In: AAS/Division for Planetary Sciences Meeting Abstracts, vol. 48 (2016)

  3. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

  4. Chang, C.I.: An information-theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysis. IEEE Trans. Inf. Theory 46(5), 1927–1932 (2000)

    Article  Google Scholar 

  5. Doering, D., Vizzotto, M., Bredemeier, C., da Costa, C., Henriques, R., Pignaton, E., Pereira, C.: Mde-based development of a multispectral camera for precision agriculture. IFAC-PapersOnLine 49(30), 24–29 (2016)

    Article  Google Scholar 

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI (2016)

  7. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: ECCV (2016)

  8. Fu, Y., Zheng, Y., Huang, H., Sato, I., Sato, Y.: Hyperspectral image super-resolution with a mosaic RGB image. IEEE Trans. Image Process. 27(11), 5539–5552 (2018)

    Article  MathSciNet  Google Scholar 

  9. Fujimoto, A., Ogawa, T., Yamamoto, K., Matsui, Y., Yamasaki, T., Aizawa, K.: Manga109 dataset and creation of metadata. In: Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding, p. 2. ACM (2016)

  10. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014)

  11. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. ar**v preprint ar**v:1406.2661 (2014)

  12. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)

  13. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

  14. Jaiswal, S.P., Fang, L., Jakhetiya, V., Pang, J., Mueller, K., Au, O.C.: Adaptive multispectral demosaicking based on frequency-domain analysis of spectral correlation. IEEE Trans. Image Process. 26(2), 953–968 (2016)

    Article  MathSciNet  Google Scholar 

  15. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)

  16. Kim, J.H., Choi, J.H., Cheon, M., Lee, J.S.: Ram: Residual attention module for single image super-resolution. ar**v preprint ar**v:1811.12043 (2018)

  17. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. ar**v preprint ar**v:1412.6980 (2014)

  18. Lahoud, F., Zhou, R., Süsstrunk, S.: Multi-modal spectral image super-resolution. In: European Conference on Computer Vision, pp. 35–50. Springer (2018)

  19. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network

  20. Li, Y., Hu, J., Zhao, X., **e, W., Li, J.: Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266, 29–41 (2017)

    Article  Google Scholar 

  21. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPRW (2017)

  22. Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Iccv Vancouver: (2001)

  23. Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems, pp. 2204–2212 (2014)

  24. Monno, Y., Kikuchi, S., Tanaka, M., Okutomi, M.: A practical one-shot multispectral imaging system using a single image sensor. IEEE Trans. Image Process. 24(10), 3048–3059 (2015)

    Article  MathSciNet  Google Scholar 

  25. Najafi, M., Namin, S.T., Petersson, L.: Classification of natural scene multi spectral images using a new enhanced crf. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3704–3711. IEEE (2013)

  26. Shi, Z., Chen, C., ** of snapshot mosaic image sensors via a texture sensitive residual network

  27. Shoeiby, M., Robles-Kelly, A., Timofte, R., Zhou, R., Lahoud, F., Susstrunk, S., **ong, Z., Shi, Z., Chen, C., Liu, D., Zha, Z.J., Wu, F., Wei, K., Zhang, T., Wang, L., Fu, Y., Zhong, Z., Nagasubramanian, K., Singh, A.K., Singh, A., Sarkar, S., Baskar, G.: PIRM2018 challenge on spectral image super-resolution: Methods and results. In: European Conference on Computer Vision Workshops (ECCVW) (2018)

  28. Shoeiby, M., Robles-Kelly, A., Wei, R., Timofte, R.: Pirm2018 challenge on spectral image super-resolution: Dataset and study (2019)

  29. Takumi, K., Watanabe, K., Ha, Q., Tejero-De-Pablos, A., Ushiku, Y., Harada, T.: Multispectral object detection for autonomous vehicles. In: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 35–43. ACM (2017)

  30. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: ICCV (2017)

  31. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  32. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR (2018)

  33. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. ar**v preprint ar**v:1807.02758 (2018)

  34. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: European Conference on Computer Vision, pp. 294–310. Springer (2018)

  35. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR (2018)

  36. Zhou, R., Achanta, R., Süsstrunk, S.: Deep residual network for joint demosaicing and super-resolution. In: Color and Imaging Conference, vol. 2018, pp. 75–80. Society for Imaging Science and Technology (2018)

Download references

Author information

Authors and Affiliations

  1. CSIRO-DATA61, Canberra, Australia

    Mehrdad Sheoiby, Saeed Anwar & Lars Petersson

  2. CSIRO-DATA61/ANU, Canberra, Australia

    Sadegh Aliakbarian

Authors
  1. Mehrdad Sheoiby
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sadegh Aliakbarian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saeed Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lars Petersson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehrdad Sheoiby.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sheoiby, M., Aliakbarian, S., Anwar, S. et al. Multi-FAN: multi-spectral mosaic super-resolution via multi-scale feature aggregation network. Machine Vision and Applications 32, 47 (2021). https://doi.org/10.1007/s00138-021-01174-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-021-01174-w

Keywords

Search

Navigation