SpringerLink
Log in

Two-stage single image reflection removal with reflection-aware guidance

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Removing undesired reflection from an image captured through a glass surface is a very challenging problem with many practical applications. For improving reflection removal, cascaded deep models have been usually adopted to estimate the transmission in a progressive manner. However, most existing methods are still limited in exploiting the result in prior stage for guiding transmission estimation. In this paper, we present a novel two-stage network with reflection-aware guidance (RAGNet) for single image reflection removal (SIRR). To be specific, the reflection layer is firstly estimated due to that it generally is much simpler and is relatively easier to estimate. Reflection-aware guidance (RAG) module is then elaborated for better exploiting the estimated reflection in predicting transmission layer. By incorporating feature maps from the estimated reflection and observation, RAG can be used (i) to mitigate the effect of reflection from the observation, and (ii) to generate mask in soft partial convolution for mitigating the effect of deviating from linear combination hypothesis. A dedicated mask loss is further presented for reconciling the contributions of encoder and decoder features. Experiments on five commonly used datasets demonstrate the quantitative and qualitative superiority of our RAGNet in comparison to the state-of-the-art SIRR methods. The source code and pre-trained model are available at https://github.com/liyucs/RAGNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability statement

The datasets used during the training are available in http://host.robots.ox.ac.uk/pascal/VOC/ and https://github.com/ceciliavision/perceptual-reflection-removal. The datasets used during the testing are available in https://github.com/fqnchina/CEILNet and https://sir2data.github.io/.

References

  1. Pal SK, Pramanik A, Maiti J, Mitra P (2021) Deep learning in multi-object detection and tracking: state of the art. Appl Intell 51(9):6400–6429

    Article  Google Scholar 

  2. Yang J, Ge H, Yang J, Tong Y, Su S (2022) Online multi-object tracking using multi-function integration and tracking simulation training. Appl Intell 52(2):1268–1288

    Article  Google Scholar 

  3. Punnappurath A, Brown MS (2019) Reflection removal using a dual-pixel sensor. In: IEEE conference on computer vision and pattern recognition, pp 1556–1565

  4. Fan Q, Yang J, Hua G, Chen B, Wipf D (2017) A generic deep architecture for single image reflection removal and image smoothing. In: IEEE international conference on computer vision, pp 3238–3247

  5. Wei K, Yang J, Fu Y, Wipf D, Huang H (2019) Single image reflection removal exploiting misaligned training data and network enhancements. In: IEEE conference on computer vision and pattern recognition, pp 8178–8187

  6. Asif M, Chen L, Song H, Yang J, Frangi AF (2021) An automatic framework for endoscopic image restoration and enhancement. Appl Intell 51(4):1959–1971

    Article  Google Scholar 

  7. Chang Y, Jung C (2018) Single image reflection removal using convolutional neural networks. IEEE Trans Image Process 28(4):1954–1966

    Article  MathSciNet  Google Scholar 

  8. Zhang X, Ng R, Chen Q (2018) Single image reflection separation with perceptual losses. In: IEEE conference on computer vision and pattern recognition, pp 4786–4794

  9. Yang J, Gong D, Liu L, Shi Q (2018) Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In: European conference on computer vision, pp 654–669

  10. Heydecker D, Maierhofer G, Aviles-Rivero AI, Fan Q, Chen D, Schönlieb C-B, Süsstrunk S (2019) Mirror, mirror, on the wall, who’s got the clearest image of them all?—a tailored approach to single image reflection removal. IEEE Trans Image Process 28(12):6185–6197

    Article  MathSciNet  MATH  Google Scholar 

  11. Kim S, Huo Y, Yoon S-E (2020) Single image reflection removal with physically-based training images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5164–5173

  12. Li C, Yang Y, He K, Lin S, Hopcroft JE (2020) Single image reflection removal through cascaded refinement. In: IEEE conference on computer vision and pattern Recognition, pp 3565–3574

  13. Lei C, Huang X, Zhang M, Yan Q, Sun W, Chen Q (2020) Polarized reflection removal with perfect alignment in the wild. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1750–1758

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  15. Peng Y-T, Cheng K-H, Fang I-S, Peng W-Y, Wu J-S (2022) Single image reflection removal based on knowledge-distilling content disentanglement. IEEE Signal Process Lett 29:568–572

    Article  Google Scholar 

  16. Chang Y-C, Lu C-N, Cheng C-C, Chiu W-C (2021) Single image reflection removal with edge guidance, reflection classifier, and recurrent decomposition. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 2033–2042

  17. Agrawal A, Raskar R, Nayar S.K, Li Y (2005) Removing photography artifacts using gradient projection and flash-exposure sampling. In: ACM SIGGRAPH, pp 828–835

  18. Fu Y, Lam A, Sato I, Okabe T, Sato Y (2013) Separating reflective and fluorescent components using high frequency illumination in the spectral domain. In: IEEE international conference on computer vision, pp 457–464

  19. Schechner Y.Y, Kiryati N, Basri R (2000) Separation of transparent layers using focus, pp 25–39

  20. Sarel B, Irani M (2004) Separating transparent layers through layer information exchange. In: European conference on computer vision, pp 328–341

  21. Wieschollek P, Gallo O, Gu J, Kautz J (2018) Separating reflection and transmission images in the wild. In: European conference on computer vision, pp 89–104

  22. Xue T, Rubinstein M, Liu C, Freeman WT (2015) A computational approach for obstruction-free photography. ACM Trans Graph 34(4):1–11

    Article  Google Scholar 

  23. Szeliski R, Avidan S, Anandan P (2000) Layer extraction from multiple images containing reflections and transparency. In: IEEE conference on computer vision and pattern recognition, pp 246–253

  24. Li Y, Brown MS (2013) Exploiting reflection change for automatic reflection removal. In: IEEE international conference on computer vision, pp 2432–2439

  25. Guo X, Cao X, Ma Y (2014) Robust separation of reflection from multiple images. In: IEEE conference on computer vision and pattern recognition, pp 2187–2194

  26. Gai K, Shi Z, Zhang C (2012) Blind separation of superimposed moving images using image statistics. IEEE Trans Pattern Anal Mach Intell 34(1):19–32

    Article  Google Scholar 

  27. Sinha SN, Kopf J, Goesele M, Scharstein D, Szeliski R (2012) Image-based rendering for scenes with reflections. ACM Trans Graph 31(4):1–10

    Article  Google Scholar 

  28. Yang J, Li H, Dai Y, Tan RT (2016) Robust optical flow estimation of double-layer images under transparency or reflection. In: IEEE conference on computer vision and pattern recognition, pp 1410–1419

  29. Sun C, Liu S, Yang T, Zeng B, Wang Z, Liu G (2016) Automatic reflection removal using gradient intensity and motion cues. In: ACM international conference on multimedia, pp 466– 470

  30. Han B-J, Sim J-Y (2017) Reflection removal using low-rank matrix completion. In: IEEE conference on computer vision and pattern recognition, pp 5438–5446

  31. Cheong JY, Simon C, Kim C-S, Park IK (2017) Reflection removal under fast forward camera motion. IEEE Trans Image Process 26(12):6061–6073

    Article  MathSciNet  Google Scholar 

  32. Simon C, Kyu Park I (2015) Reflection removal for in-vehicle black box videos. In: IEEE conference on computer vision and pattern recognition, pp 4231–4239

  33. Li T, Chan Y-H, Lun DP (2020) Improved multiple-image-based reflection removal algorithm using deep neural networks. IEEE Trans Image Process 30:68–79

    Article  Google Scholar 

  34. Han B-J, Sim J-Y (2018) Glass reflection removal using co-saliency-based image alignment and low-rank matrix completion in gradient domain. IEEE Trans Image Process 27(10):4873– 4888

    Article  MathSciNet  MATH  Google Scholar 

  35. Levin A, Weiss Y (2007) User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans Pattern Anal Mach Intell 29(9):1647–1654

    Article  Google Scholar 

  36. Wan R, Shi B, Duan L-Y, Tan A-H, Gao W, Kot AC (2018) Region-aware reflection removal with unified content and gradient priors. IEEE Trans Image Process 27(6):2927–2941

    Article  MathSciNet  MATH  Google Scholar 

  37. Arvanitopoulos N, Achanta R, Susstrunk S (2017) Single image reflection suppression. In: IEEE conference on computer vision and pattern recognition, pp 4498–4506

  38. Sandhan T, Young Choi J (2017) Anti-glare: tightly constrained optimization for eyeglass reflection removal. In: IEEE conference on computer vision and pattern recognition, pp 1241–1250

  39. Li Y, Brown MS (2014) Single image layer separation using relative smoothness. In: IEEE conference on computer vision and pattern recognition, pp 2752–2759

  40. Wan R, Shi B, Tan A-H, Kot AC (2016) Depth of field guided reflection removal. In: IEEE international conference on image processing, pp 21–25

  41. Shih Y, Krishnan D, Durand F, Freeman W.T (2015) Reflection removal using ghosting cues. In: IEEE conference on computer vision and pattern recognition, pp 3193–3201

  42. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. ar**v:1409.1556

  43. Dong Z, Xu K, Yang Y, Bao H, Xu W, Lau RW (2021) Location-aware single image reflection removal. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5017–5026

  44. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  45. Chen Y, Zhang H, Liu L, Chen X, Zhang Q, Yang K, **a R, **e J (2021) Research on image inpainting algorithm of improved gan based on two-discriminations networks. Appl Intell 51 (6):3460–3474

    Article  Google Scholar 

  46. Zhang J, Liu Y, Guo C, Zhan J (2022) Optimized segmentation with image inpainting for semantic map** in dynamic scenes. Appl Intell, pp 1–16

  47. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, pp 694–711

  48. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  49. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  50. Wan R, Shi B, Duan L-Y, Tan A-H, Kot AC (2017) Benchmarking single-image reflection removal algorithms. In: IEEE International Conference on Computer Vision, pp 3922–3930

  51. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. ar**v:1412.6980

  52. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L (2019) Pytorch: an imperative style, high-performance deep learning library. In: Advances in neural information processing systems, pp 8026–8037

Download references

Acknowledgements

This work was partially supported by National Natural Science Foundation of China (Nos. 62172127 and U22B2035), Natural Science Foundation of Heilongjiang Province (No. YQ2022F004).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150001, China

    Yu Li, Ming Liu, Yaling Yi, Qince Li, Dongwei Ren & Wangmeng Zuo

Authors
  1. Yu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yaling Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qince Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dongwei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wangmeng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dongwei Ren.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Liu, M., Yi, Y. et al. Two-stage single image reflection removal with reflection-aware guidance. Appl Intell 53, 19433–19448 (2023). https://doi.org/10.1007/s10489-022-04391-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04391-6

Keywords

Search

Navigation