SpringerLink
Log in

Single-image shadow removal using detail extraction and illumination estimation

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Deep learning-based shadow removal methods are frequently hard to obtain a detail-rich and boundary-smoothing shadow removal result. In this work, we propose an illumination-sensitive filter and a multi-task generative adversarial networks architecture to tackle these problems. Firstly, we detect the shadow for the input shadow image and use the illumination-sensitive filter to extract the texture information for generating a coarse image with fewer texture details. Secondly, we conduct illumination estimation for this coarse shadow image to remove the shadow indirectly. Next, we restore the shadow boundary realistically inspired by the idea of image in painting. Finally, we recover the texture details for obtaining the final shadow removal result. Besides, we filter two large benchmark datasets, i.e., SRD and ISTD, to create a Low Error Synthesized Dataset (LESD). The extensive experiments demonstrate that our method can achieve superior performance to state of the arts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Arbel, E., Hel-Or, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33(6), 1202–1216 (2010)

    Article  Google Scholar 

  2. Raj, N. B., N, V.: Single image haze removal using a generative adversarial network. Ar**v Preprint ar**v:1810.09479 (2018)

  3. Cun, X., Pun, C.M., Shi, C.: Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan. In: AAAI, pp. 10680–10687 (2020)

  4. Ding, B., Long, C., Zhang, L., **ao, C.: Argan: attentive recurrent generative adversarial network for shadow detection and removal. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10212–10221 (2019)

  5. Fu, X., Zeng, D., Huang, Y., Zhang, X.P., Ding, X.: A weighted variational model for simultaneous reflectance and illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2782–2790 (2016)

  6. Fu, X., Zeng, D., Huang, Y., Zhang, X.P., Ding, X.: A weighted variational model for simultaneous reflectance and illumination estimation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2782–2790 (2016)

  7. Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  8. Gong, H., Cosker, D.: Interactive shadow removal and ground truth for variable scene categories. In: BMVC, pp. 1–11 (2014)

  9. Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: CVPR 2011, pp. 2033–2040. IEEE (2011)

  10. Guo, X., Li, Y., Ling, H.: Lime: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2016)

    Article  MathSciNet  Google Scholar 

  11. Guo, X., Li, Y., Ling, H.: Lime: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)

    Article  MathSciNet  Google Scholar 

  12. Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 42(11), 2795–2808 (2020)

    Article  Google Scholar 

  13. Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-shadowgan: Learning to remove shadows from unpaired data. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2472–2481 (2019)

  14. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer (2016)

  15. Khan, S.H., Bennamoun, M., Sohel, F., Togneri, R.: Automatic shadow detection and removal from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 38(3), 431–446 (2015)

    Article  Google Scholar 

  16. Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+d net: Training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 680–696 (2018)

  17. Liu, D., Long, C., Zhang, H., Yu, H., Dong, X., **ao, C.: Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  18. Liu, D., Long, C., Zhang, H., Yu, H., Dong, X., **ao, C.: Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8139–8148 (2020)

  19. Liu, F., Gleicher, M.: Texture-consistent shadow removal. In: European Conference on Computer Vision, pp. 437–450. Springer (2008)

  20. Liu, Z., **ang, Q., Tang, J., Wang, Y., Zhao, P.: Robust salient object detection for RGB images. Vis. Comput. 36(9), 1823–1835 (2020)

  21. Mohajerani, S., Saeedi, P.: Cpnet: A context preserver convolutional neural network for detecting shadows in single RGB images. In: 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–5 (2018)

  22. Mohan, A., Tumblin, J., Choudhury, P.: Editing soft shadows in a digital photograph. IEEE Comput. Graph. Appl. 27(2), 23–31 (2007)

    Article  Google Scholar 

  23. Nguyen, V., Yago Vicente, T.F., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4510–4518 (2017)

  24. PNVR, K., Zhou, H., Jacobs, D.: Sharingan: Combining synthetic and real data for unsupervised geometry estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  25. Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.: Deshadownet: A multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4067–4075 (2017)

  26. Sidorov, O.: Conditional gans for multi-illuminant color constancy: Revolution or yet another approach? In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1748–1758 (2019)

  27. Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. ACM Trans. Graph. 28(5), 1–9 (2009)

    Article  Google Scholar 

  28. Tian, J., Qi, X., Qu, L., Tang, Y.: New spectrum ratio properties and features for shadow detection. Pattern Recogn. 51, 85–96 (2016)

    Article  Google Scholar 

  29. Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: European Conference on Computer Vision, pp. 816–832. Springer (2016)

  30. Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1788–1797 (2018)

  31. Wang, R., Zhang, Q., Fu, C.W., Shen, X., Zheng, W.S., Jia, J.: Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6849–6857 (2019)

  32. Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1880–1889 (2020)

  33. Yago Vicente, T.F., Hoai, M., Samaras, D.: Leave-one-out kernel optimization for shadow detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3388–3396 (2015)

  34. Yagyu, S., Sakiyama, A., Tanaka, Y.: Edge preserving multiscale image decomposition with customized domain transform filters. In: 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 458–462. IEEE (2015)

  35. Yang, X., Zhu, S., **a, S., Zhou, D.: A new TLD target tracking method based on improved correlation filter and adaptive scale. Vis. Comput. 36(9), 1783–1795 (2020)

  36. Zhang, L., Long, C., Zhang, X., **ao, C.: Ris-gan: explore residual and illumination with generative adversarial networks for shadow removal. Ar**v preprint ar**v:1911.09178 (2019)

  37. Zhang, L., Long, C., Zhang, X., **ao, C.: Ris-gan: explore residual and illumination with generative adversarial networks for shadow removal. AAAI 2020: Thirty-Fourth AAAI Conf. Artif. Intell. 34(7), 12829–12836 (2020)

  38. Zhang, S., Liang, R., Wang, M.: Shadowgan: shadow synthesis for virtual objects with conditional adversarial networks. Comput. Vis. Media 5(1), 105–115 (2019)

    Article  Google Scholar 

  39. Zhang, X.C., Barron, J.T., Tsai, Y.T., Pandey, R., Zhang, X., Ng, R., Jacobs, D.E.: Portrait shadow manipulation. ACM Trans. Graph. 39(4), 78 (2020)

    Google Scholar 

  40. Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.: Distraction-aware shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5167–5176 (2019)

  41. Zhu, L., Deng, Z., Hu, X., Fu, C.W., Xu, X., Qin, J., Heng, P.A.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 121–136 (2018)

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of **njiang Autonomous Region in China (NO. 2020D01A48) and the National Social Science Foundation Western Project (NO. 20XGL029).

Author information

Author notes

  1. Wen Wu and **antao Wu contributed equally to this work.

Authors and Affiliations

  1. School of Information Engineering, **njiang Institute of Technology, Aksu, 843100, China

    Wen Wu

  2. School of Information Science and Engineering, **njiang University, Urumqi, 830046, China

    **antao Wu

  3. School of Electrical and Electronic Engineering, Wenzhou University, Wenzhou, 325035, China

    Yi Wan

Authors
  1. Wen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. **antao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, W., Wu, X. & Wan, Y. Single-image shadow removal using detail extraction and illumination estimation. Vis Comput 38, 1677–1687 (2022). https://doi.org/10.1007/s00371-021-02096-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02096-4

Keywords

Search

Navigation