SpringerLink
Log in

Weighted sparse gradient reconstruction model with a robust fidelity for edge-aware image smoothing

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Smoothing out image details while preserving the salient edges is of significance to the field of computational photography. In this paper, we propose a novel optimization model for edge-aware image smoothing, which consists of a regularization term and a fidelity term. The regularization term is based on the idea of weighted sparse gradient reconstruction, which ensures edge-awareness. The fidelity term is based on an \(L_1\) loss, which is robust to outliers. Our model is sophisticated and thus can be non-trivial to solve. In this paper, we propose an iterative solution based on the augmented Lagrange multiplies, where the computational cost in each iteration is dominated by a least square problem that can be efficiently solved in the Fourier domain. We have conducted extensive experiments to evaluate the proposed filter. Both quantitative and qualitative results indicate that our filter is advantageous to the state-of-the-art filters on a variety of image processing and vision tasks. Furthermore, the proposed filter is efficient, it takes approximately 2 s to process images with 1 megapixel on a modern CPU.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability statement

All the data sets explored in this paper are publicly available.

References

  1. Liu, W., Zhang, P., Chen, X., Shen, C., Huang, X., Yang, J.: Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans. Circ. Syst. Video Technol. 30(1), 23–35 (2020)

    Article  Google Scholar 

  2. Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L\({}_{\text{0 }}\) gradient minimization. ACM Trans. Graph. 30(6), 174 (2011)

    Article  Google Scholar 

  3. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27(3), 67 (2008)

    Article  Google Scholar 

  4. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002)

    Article  Google Scholar 

  5. Mun, H., Yoon, G., Song, J., Yoon, S.M.: Texture preserving photo style transfer network. IEEE Trans. Multimed. 24, 3823–3834 (2022)

    Article  Google Scholar 

  6. Lu, C., Xu, L., Jia, J.: Combining sketch and tone for pencil drawing production. International Symposium on Non-Photorealistic Animation and Rendering pp. 65–73 (2012)

  7. Li, Q., Chen, G., Zhan, K., Zhang, X., Saruta, K., Terata, Y.: Multifocus image fusion using structure-preserving filter. J. Electron. Imaging 28(2), 023005 (2019)

    Article  ADS  Google Scholar 

  8. Zhu, Y., Lu, Y., Gao, Q., Sun, D.: Infrared and visible image fusion based on convolutional sparse representation and guided filtering. J. Electron. Imaging 30(4) (2021)

  9. Yang, J., Zhang, J., Li, M., Wang, M.: DBRS2: dense boundary regression for semantic segmentation. J. Electron. Imaging 27(05), 053033 (2018)

    Article  ADS  Google Scholar 

  10. Guo, M., Huang, B., Zhang, J., Wang, F., Zhang, Y., Fang, Z.: Dfbdehazenet: an end-to-end dense feedback network for single image dehazing. J. Electron. Imaging 30(3) (2021)

  11. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. International Conference on Computer Vision pp. 839–846 (1998)

  12. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  PubMed  Google Scholar 

  13. Yin, H., Gong, Y., Qiu, G.: Side window filtering. IEEE Conference on Computer Vision and Pattern Recognition pp. 8758–8766 (2019)

  14. Liu, W., Zhang, P., Chen, X., Shen, C., Huang, X., Yang, J.: Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans. Circ. Syst. Video Technol. 30(1), 23–35 (2020)

    Article  Google Scholar 

  15. Feng, Y., Deng, S., Yan, X., Yang, X., Wei, M., Liu, L.: Easy2hard: Learning to solve the intractables from a synthetic dataset for structure-preserving image smoothing. IEEE Trans. Neural Netw. Learn. Syst. 33(12), 7223–7236 (2022)

    Article  PubMed  Google Scholar 

  16. Xu, L., Ren, J.S.J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. Int. Conf. Mach. Learn. 37, 1669–1678 (2015)

    Google Scholar 

  17. Liu, W., Zhang, P., Lei, Y., Huang, X., Yang, J., Ng, M.: A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6631–6648 (2022)

    Article  Google Scholar 

  18. Yang, Y., Hui, H., Zeng, L., Zhao, Y., Zhan, Y., Yan, T.: Edge-preserving image filtering based on soft clustering. IEEE Trans. Circ. Syst. Video Technol. 32(7), 4150–4162 (2022)

    Article  Google Scholar 

  19. Yang, Y., Zheng, H., Zeng, L., Shen, X., Zhan, Y.: L1-regularized reconstruction model for edge-preserving filtering. IEEE Trans. Multimed. (2022)

  20. Yang, Y., Tang, L., Zeng, L., Wang, X., Zhan, Y.: L0 image smoothing via iterating truncated L1 gradient regularization. J. Electron. Imaging 31(5), 053016 (2022)

    Article  ADS  Google Scholar 

  21. Yang, Y., **ong, Y., Cao, Y., Zeng, L., Zhao, Y., Zhan, Y.: Fast bilateral filter with spatial subsampling. Multimed. Syst. 29(1), 435–446 (2023)

    Article  Google Scholar 

  22. Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009)

    Article  Google Scholar 

  23. Young, S.I., Girod, B., Taubman, D.: Gaussian lifting for fast bilateral and nonlocal means filtering. IEEE Trans. Image Process. 29, 6082–6095 (2020)

    Article  ADS  MathSciNet  Google Scholar 

  24. Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 29, 500–508 (2020)

    Article  ADS  MathSciNet  Google Scholar 

  25. Kou, F., Chen, W., Wen, C., Li, Z.: Gradient domain guided image filtering. IEEE Trans. Image Process. 24(11), 4528–4539 (2015)

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  26. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)

    Article  ADS  MathSciNet  Google Scholar 

  27. Badri, H., Yahia, H., Aboutajdine, D.: Fast edge-aware processing via first order proximal approximation. IEEE Trans. Vis. Comput. Graph. 21(6), 743–755 (2015)

    Article  PubMed  Google Scholar 

  28. Liu, W., Zhang, P., Huang, X., Yang, J., Shen, C., Reid, I.: Real-time image smoothing via iterative least squares. ACM Trans. Graph. 39(3), 28:1-28:24 (2020)

    Article  Google Scholar 

  29. Xu, L., Ren, J.S.J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. Int. Conf. Mach. Learn. 37, 1669–1678 (2015)

    Google Scholar 

  30. Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. CoRR. arxiv:abs/1709.00643 (2017)

  31. Zhu, F., Liang, Z., Jia, X., Zhang, L., Yu, Y.: A benchmark for edge-preserving image smoothing. IEEE Trans. Image Process. 28(7), 3556–3570 (2019)

    Article  ADS  MathSciNet  Google Scholar 

  32. Li, Y., Huang, J., Ahuja, N., Yang, M.: Joint image filtering with deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1909–1923 (2019)

    Article  PubMed  Google Scholar 

  33. Kim, B., Ponce, J., Ham, B.: Deformable kernel networks for joint image filtering. Int. J. Comput. Vis. 129(2), 579–600 (2021)

    Article  Google Scholar 

  34. Wu, H., Zheng, S., Zhang, J., Huang, K.: Fast end-to-end trainable guided filter. CoRR. arxiv.abs/1803.05619 (2018)

  35. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  PubMed  Google Scholar 

  36. Deng, G., Galetto, F., Al-nasrawi, M., Waheed, W.: A guided edge-aware smoothing-sharpening filter based on patch interpolation model and generalized gamma distribution. IEEE Open J. Signal Process. 2, 119–135 (2021)

    Article  Google Scholar 

  37. Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 29, 500–508 (2020)

    Article  ADS  MathSciNet  Google Scholar 

  38. Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. European Conference on Computer Vision pp. 815–830 (2014)

  39. Xu, J., Liu, Z., Hou, Y., Zhen, X., Shao, L., Cheng, M.: Pixel-level non-local image smoothing with objective evaluation. IEEE Trans. Multimed. 23, 4065–4078 (2021)

    Article  Google Scholar 

  40. Ham, B., Cho, M., Ponce, J.: Robust guided image filtering using nonconvex potentials. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 192–207 (2018)

    Article  PubMed  Google Scholar 

  41. Huang, J., Wang, H., Wang, X., Ruzhansky, M.: Semi-sparsity for smoothing filters. IEEE Trans. Image Process. 32, 1627–1639 (2023)

    Article  ADS  Google Scholar 

  42. Xu, L., Yan, Q., **a, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. 31(6), 139:1-139:10 (2012)

    Article  CAS  Google Scholar 

  43. Bi, S., Han, X., Yu, Y.: An L\({}_{{1}}\) image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans. Graph. 34(4), 78:1-78:12 (2015)

    Article  Google Scholar 

  44. Liu, W., Chen, X., Shen, C., Liu, Z., Yang, J.: Semi-global weighted least squares in image filtering. CoRR. arxiv.abs/1705.01674 (2017)

  45. Yeganeh, H., Wang, Z.: Objective quality assessment of tone-mapped images. IEEE Trans. Image Process. 22(2), 657–667 (2013)

    Article  ADS  MathSciNet  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 61402205, and in part by the Postgraduate Research and Practice Innovation Program of Jiangsu Province under Grant No. SJCX21_1693, in part by the Jiangsu University under Grant No. 13JDG085, and in part by the Jiangnan University under Grant No. 20ST0206.

Funding

This work is supported by National Natural Science Foundation of China, Grant No. 61402205 and 62072150.

Author information

Authors and Affiliations

  1. Department of Computer Science, Jiangsu University, Xuefu, Zhenjiang, 212013, Jiangsu, China

    Lanling Zeng, Yucheng Chen & Yang Yang

Authors
  1. Lanling Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yucheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LZ and YC wrote the original manuscript text; YC conducted the experiments; LZ and YY revised the manuscript; YY supervised the research; All authors reviewed the manuscript.

Corresponding author

Correspondence to Yang Yang.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Ethical approval

Not applicable.

Additional information

Communicated by B. Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, L., Chen, Y. & Yang, Y. Weighted sparse gradient reconstruction model with a robust fidelity for edge-aware image smoothing. Multimedia Systems 30, 59 (2024). https://doi.org/10.1007/s00530-023-01209-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-023-01209-4

Keywords

Search

Navigation