SpringerLink
Log in

Attentive frequency learning network for super-resolution

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Benefiting from the strong capability of capturing long-range dependencies, a series of self-attention based single image super-resolution (SISR) methods have achieved promising performance. However, the existing self-attention mechanisms generally suffer great computational costs both in training and inference. In this study, we propose an innovative attentive frequency learning network (AFLN) for single image super-resolution. Our AFLN can greatly reduce computational costs of self-attention mechanism yet well capture long-range dependencies in SISR tasks. Specifically, our AFLN mainly consists of a series of extensive attentive frequency learning blocks (AFLB). In each AFLB, we firstly integrate the hierarchical features by residual dense connections and decompose the original features into low- and high-frequency domains with a half size of original features via discrete wavelet transform (DWT). Then, we adopt self-attention to explore long-range dependency relations in low- and high-frequency feature domains, respectively. In this way, we can model the self-attention in the quarter size of original input image, greatly reducing computational costs. In addition, the separating attention from low- and high-frequency domain can effectively maintain detailed information. Finally, we adopt the inverse discrete wavelet transform (IDWT) to reconstruct these attentive features. Extensive experiments on publicly available datasets demonstrate the efficiency and effectiveness of our AFLN against the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Hayit G (2009) Super-resolution in medical imaging. Comput J 52(1):43–63

    Google Scholar 

  2. Pejman R, Tonis U, Sergio E, Gholamreza A (2016) Convolutional neural network super resolution for face recognition in surveillance monitoring. In: International conference on articulated motion and deformable objects, pp 175–184. Springer

  3. Chen C, Chen Q, Jia X, Koltun V (2018) Learning to see in the dark. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3291–3300

  4. He K, Sun J, Tang X (2010) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353

    Google Scholar 

  5. Zhu K, Jiang X, Fang Z, Gao Y, Fujita H, Hwang Jenq-Neng (2021) Photometric transfer for direct visual odometry. Knowl Based Syst 213:106671

    Article  Google Scholar 

  6. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  7. Li X, Wang W, **aolin H u, Yang J (2019) Selective kernel networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 510–519

  8. Yi W, **aoyan J, Zhijun F, Yongbin G, Hamido F (2021) Multi-modal 3d object detection by 2d-guided precision anchor proposal and multi-layer fusion. Appl Soft Comput 108:107– 405

    Google Scholar 

  9. Dong C, Loy C C, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  10. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  11. Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: Proceedings of the IEEE international conference on computer vision, pp 4799–4807

  12. Zhang Y, Tian Y, Kong Y, Zhong B, Yun F (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  13. Zhu X, Li Z, Zhang **ao-Yu, Li C, Liu Y, Xue Z (2019) Residual invertible spatio-temporal network for video super-resolution. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 5981–5988

  14. Zeng K, Ding S, Jia W (2019) Single image super-resolution using a polymorphic parallel cnn. Appl Intell 49(1):292–300

    Article  Google Scholar 

  15. Yuantao C, Linwu L, Volachith P, Ke G, Runlong X, **gbo X, Qian Z, Kai Y (2021) Image super-resolution reconstruction based on feature map attention mechanism. Appl Intell 1–14, https://scholar.google.com.hk/scholar?q=Image+superresolution+reconstruction+based+on+feature+map+attention+mechanism&hl=zh-CN&as_sdt=0&as_vis=1&oi=scholart

  16. Lim B, Son S, Kim H, Nah S, Lee K M (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144

  17. Zhang Y, Li K, Li K, Wang L, Zhong B, Yun F (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the european conference on computer vision, pp 286–301

  18. Liu D, Wen B, Fan Y, Loy C C, Huang TS (2018) Non-local recurrent network for image restoration. In: Advances in neural information processing systems, pp 1673–1682

  19. Zhang Y, Li K, Li K, Zhong B, Fu Y (2019) Residual non-local attention networks for image restoration. In: International conference on learning representations

  20. Dai T, Cai J, Zhang Y, ** Y, Shuzhen W, Kaihao Z, **aochun C, Haifeng S (2020) Single image super-resolution via a holistic attention network. In: European conference on computer vision, pp 191–207. Springer

  21. Huang H, He R, Sun Z, Tieniu T (2017) Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In: Proceedings of the IEEE international conference on computer vision, pp 1689–1697

  22. Liu P, Zhang H, Zhang K, Lin L, Zuo W (2018) Multi-level wavelet-cnn for image restoration. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 773–782

  23. Xue S, Qiu W, Liu F, ** X (2020) Wavelet-based residual attention network for image super-resolution. Neurocomputing 382:116–126

    Article  Google Scholar 

  24. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  25. Jun F, Liu J, Tian H, Li Y, Bao Y, Fang Z, Hanqing L (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3146–3154

  26. Yoo J, Youngjung U, Chun S, Kang B, Ha J-W (2019) Photorealistic style transfer via wavelet transforms. In: Proceedings of the IEEE international conference on computer vision, pp 9036– 9045

  27. Li Y, Liu M-Y, Li X, Yang M-H, Kautz J (2018) A closed-form solution to photorealistic image stylization. In: Proceedings of the European conference on computer vision (ECCV), pp 453– 468

  28. Guo T, Mousavi HS, Tiep Huu V, Monga V (2017) Deep wavelet prediction for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 104–113

  29. Timofte R, Agustsson E, Gool LV, Yang M-H, Lei Z (2017) Ntire 2017 challenge on single image super-resolution Methods and results. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 114–125

  30. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding

  31. Roman Z, Michael E, Matan P (2010) On single image scale-up using sparse-representations. In: International conference on curves and surfaces, pp 711–730. Springer

  32. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5197–5206

  33. David M, Charless F, Doron T, Jitendra M (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE international conference on computer vision. iCCV 2001, vol 2, pp 416–423 IEEE

  34. Yusuke M, Kota I, Yuji A, Azuma F, Toru O, Toshihiko Y, Kiyoharu A (2017) Sketch-based manga retrieval using manga109 dataset. Multimed Tools Appl 76(20):21811–21838

    Article  Google Scholar 

  35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc 13(4):600–612

    Article  Google Scholar 

  36. Chao D, Chen C L, **aoou T (2016) Accelerating the super-resolution convolutional neural network.. In: European conference on computer vision, pp 391–407. Springer

  37. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 624–632

  38. Tai Y, Yang J, Liu X, Memnet CX (2017) A persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vision, pp 4539–4547

  39. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1664–1673

  40. Li Z, Yang J, Liu Z, Yang X, Jeon G, Wei W (2019) Feedback network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3867–3876

  41. Liu J, Zhang W, Tang Y, Tang J, Gangshan W (2020) Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2359–2368

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 61877002), Bei**g Natural Science Foundation and Fengtai Rail Transit Frontier Research Joint Fund (No. L191009) , Scientific Research Program of Bei**g Municipal Education Commission (No. KZ202110011017) and Postgraduate Research Capacity Improvement Program from Bei**g Technology and Business University in 2021.

Author information

Authors and Affiliations

  1. School of Computer, Bei**g Technology and Business University, Bei**g, China

    Fenghai Li, Qiang Cai, Haisheng Li, Jian Cao & Shanshan Li

  2. National Engineering Laboratory for Agri-product Quality Traceability, Bei**g, China

    Fenghai Li, Qiang Cai, Haisheng Li, Jian Cao & Shanshan Li

  3. Bei**g Key Laboratory of Big Data Technology for Food Safety, Bei**g Technology and Business University, Bei**g, China

    Fenghai Li, Qiang Cai, Haisheng Li, Jian Cao & Shanshan Li

  4. Canvard College, Bei**g Technology and Business University, Bei**g, China

    Yifan Chen

Authors
  1. Fenghai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiang Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haisheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yifan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jian Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shanshan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Qiang Cai or Haisheng Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, F., Cai, Q., Li, H. et al. Attentive frequency learning network for super-resolution. Appl Intell 52, 5185–5196 (2022). https://doi.org/10.1007/s10489-021-02703-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02703-w

Keywords

Search

Navigation