SpringerLink
Log in

Annotate less but perform better: weakly supervised shadow detection via label augmentation

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Shadow detection is essential for scene understanding and image restoration. Existing paradigms for producing shadow detection training data usually rely on densely labeling each image pixel, which will lead to a bottleneck when scaling up the number of images. To tackle this problem, by labeling shadow images with only a few strokes, this paper designs a learning framework for Weakly supervised Shadow Detection, namely WSD. Firstly, it creates two shadow detection datasets with scribble annotations, namely Scr-SBU and Scr-ISTD. Secondly, it proposes an uncertainty-guided label augmentation scheme based on graph convolutional networks, which can propagate the sparse scribble annotations to more reliable regions, and then avoid the model converging to an undesired local minima as intra-class discontinuity. Finally, it introduces a multi-task learning framework to jointly learn for shadow detection and edge detection, which encourages generated shadow maps to be comprehensive and well aligned with shadow boundaries. Experimental results on benchmark datasets demonstrate that our framework even outperforms existing semi-supervised and fully supervised shadow detectors requiring only 2% pixels to be labeled.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)

    Article  PubMed  Google Scholar 

  2. Al-Amaren, A., Ahmad, M.O., Swamy, M.: A low-complexity residual deep neural network for image edge detection. Appl. Intell. 53(9), 11282–11299 (2022)

    Article  Google Scholar 

  3. Al-Huda, Z., Peng, B., Algburi, R.N.A., Alfasly, S., Li, T.: Weakly supervised pavement crack semantic segmentation based on multi-scale object localization and incremental annotation refinement. Appl. Intell. 53(11), 14527–14546 (2023)

    Article  Google Scholar 

  4. Chen, X.D., Wu, W., Yang, W., Qin, H., Wu, X., Mao, X.: Make segment anything model perfect on shadow detection. IEEE Trans. Geosci. Remote Sens. 61, 1–13 (2023)

    Google Scholar 

  5. Chen, Z., Zhu, L., Wan, L., Wang, S., Feng, W., Heng, P.A.: A multi-task mean teacher for semi-supervised shadow detection. In: Proceedings of Computer Vision and Pattern Recognition, 5611–5620. IEEE (2020)

  6. Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al.: Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). ar**v preprint ar**v:1902.03368 (2019)

  7. Cucchiara, R., Grana, C., Piccardi, M., Prati, A., Sirotti, S.: Improving shadow suppression in moving object detection with hsv color information. In: Proceedings of Intelligent Transportation Systems, 334–339. IEEE (2001)

  8. Ecins, A., Fermüller, C., Aloimonos, Y.: Shadow free segmentation in still images using local density measure. In: Proceedings of International Conference on Computational Photography, 1–8. IEEE (2014)

  9. Ge, Y., Zhou, Q., Wang, X., Shen, C., Wang, Z., Li, H.: Point-teaching: weakly semi-supervised object detection with point annotations. In: Proceedings of the AAAI Conference on Artificial Intelligence, 37, 667–675 (2023)

  10. Guan, Y.P.: Wavelet multi-scale transform based foreground segmentation and shadow elimination. Open Signal Process. J. 1, 1–6 (2008)

    Article  ADS  Google Scholar 

  11. Gulshan, V., Rother, C., Criminisi, A., Blake, A., Zisserman, A.: Geodesic star convexity for interactive image segmentation. In: Proceedings of Computer Vision and Pattern Recognition, 3129–3136. IEEE (2010)

  12. Guo, M.H., Lu, C.Z., Hou, Q., Liu, Z., Cheng, M.M., Hu, S.M.: Segnext: Rethinking convolutional attention design for semantic segmentation. In: Proceedings of Advances in Neural Information Processing Systems. MIT Press (2022)

  13. Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: Proceedings of Computer Vision and Pattern Recognition, 2033–2040. IEEE (2011)

  14. He, R., Dong, Q., Lin, J., Lau, R.W.: Weakly-supervised camouflaged object detection with scribble annotations. In: Proceedings of the AAAI Conference on Artificial Intelligence, 37, 781–789 (2023)

  15. Hu, X., Wang, T., Fu, C.W., Jiang, Y., Wang, Q., Heng, P.A.: Revisiting shadow detection: a new benchmark dataset for complex world. IEEE Trans. Image Process. 30, 1925–1934 (2021)

    Article  PubMed  ADS  Google Scholar 

  16. Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of Computer Vision and Pattern Recognition, 7454–7462. IEEE (2018)

  17. Huang, X., Hua, G., Tumblin, J., Williams, L.: What characterizes a shadow boundary under the sun and sky? In: Proceedings of International Conference on Computer Vision, 898–905. IEEE (2011)

  18. Joshi, I., Kothari, R., Utkarsh, A., Kurmi, V.K., Dantcheva, A., Roy, S.D., Kalra, P.K.: Explainable fingerprint roi segmentation using monte carlo dropout. In: Proceedings of Winter Conference on Applications of Computer Vision, 60–69 (2021)

  19. Junejo, I.N., Foroosh, H.: Estimating geo-temporal location of stationary cameras using shadow trajectories. In: Proceedings of European Conference on Computer Vision, 318–331. Springer (2008)

  20. Kang, S., Kim, J., Jang, I.S., Lee, B.D.: C2shadowgan: cycle-in-cycle generative adversarial network for shadow removal using unpaired data. Appl. Intell. 53(12), 15067–15079 (2023)

    Article  Google Scholar 

  21. Karsch, K., Hedau, V., Forsyth, D., Hoiem, D.: Rendering synthetic objects into legacy photographs. ACM Trans. Graph. 30(6), 1–12 (2011)

    Article  Google Scholar 

  22. Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: Proceedings of Advances in Neural Information Processing Systems. MIT Press (2017)

  23. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: Proceedings of International Conference on Learning Representation, 1–14 (2017)

  24. Kittler, J.: On the accuracy of the Sobel edge detector. Image Vis. Comput. 1(1), 37–42 (1983)

    Article  Google Scholar 

  25. Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. In: Proceedings of Advances in Neural Information Processing Systems. MIT Press (2011)

  26. Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Detecting ground shadows in outdoor consumer photographs. In: Proceedings of European Conference on Computer Vision, 322–335. Springer (2010)

  27. Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. 98(2), 123–145 (2012)

    Article  MathSciNet  Google Scholar 

  28. Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+d net: Training a shadow detector with adversarial shadow attenuation. In: Proceedings of European Conference on Computer Vision, 662–678. Springer (2018)

  29. Liu, D., Cui, Y., Yan, L., Mousas, C., Yang, B., Chen, Y.: Densernet: Weakly supervised visual localization using multi-scale feature aggregation. In: Proceedings of the AAAI Conference on Artificial Intelligence, 35, 6101–6109 (2021)

  30. Liu, Y., Cheng, M.M., Hu, X., Wang, K., Bai, X.: Richer convolutional features for edge detection. In: Proceedings of Computer Vision and Pattern Recognition, 3000–3009. IEEE (2017)

  31. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of International Conference on Computer Vision, 10012–10022. IEEE (2021)

  32. Luo, D., Liu, G., Bavirisetti, D.P., Cao, Y.: Infrared and visible image fusion based on VPDE model and VGG network. Appl. Intell. 53(21), 24739–24764 (2023)

    Article  Google Scholar 

  33. Meng, Q., Sinclair, M., Zimmer, V., Hou, B., Rajchl, M., Toussaint, N., Oktay, O., Schlemper, J., Gomez, A., Housden, J., et al.: Weakly supervised estimation of shadow confidence maps in fetal ultrasound imaging. IEEE Trans. Med. Imaging 38(12), 2755–2767 (2019)

    Article  PubMed  PubMed Central  Google Scholar 

  34. Mikic, I., Cosman, P.C., Kogut, G.T., Trivedi, M.M.: Moving shadow and object detection in traffic scenes. In: Proceedings of International Conference on Pattern Recognition, P1, 321–324. IEEE (2000)

  35. Nguyen, V., Yago Vicente, T.F., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: Proceedings of International Conference on Computer Vision, 4510–4518. IEEE (2017)

  36. Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: Estimating surface normals from shadows under unknown reflectance and lighting conditions. In: Proceedings of International Conference on Computer Vision, 1693–1700. IEEE (2009)

  37. Panagopoulos, A., Samaras, D., Paragios, N.: Robust shadow and illumination estimation using a mixture model. In: Proceedings of Computer Vision and Pattern Recognition, 651–658. IEEE (2009)

  38. Pei, J., Tang, H., Wang, W., Cheng, T., Chen, C.: Salient instance segmentation with region and box-level annotations. Neurocomputing 507, 332–344 (2022)

    Article  Google Scholar 

  39. Pu, M., Huang, Y., Liu, Y., Guan, Q., Ling, H.: Edter: Edge detection with transformer. In: Proceedings of Computer Vision and Pattern Recognition, 1402–1412. IEEE (2022)

  40. Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)

    Article  Google Scholar 

  41. Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: Proceedings of International Conference on Machine Learning, 6105–6114. PMLR (2019)

  42. Tang, M., Djelouah, A., Perazzi, F., Boykov, Y., Schroers, C.: Normalized cut loss for weakly-supervised cnn segmentation. In: Proceedings of Computer Vision and Pattern Recognition, 1818–1827. IEEE (2018)

  43. Unal, O., Dai, D., Van Gool, L.: Scribble-supervised lidar semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2697–2707 (2022)

  44. Vicente, T.F.Y.: Large-scale weakly-supervised shadow detection. Ph.D. thesis, State University of New York at Stony Brook (2018)

  45. Vicente, T.F.Y., Hoai, M., Samaras, D.: Leave-one-out kernel optimization for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 40(3), 682–695 (2017)

    Article  PubMed  Google Scholar 

  46. Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: Proceedings of European Conference on Computer Vision, 816–832. Springer (2016)

  47. Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of Computer Vision and Pattern Recognition, 1788–1797. IEEE (2018)

  48. Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., Xu, W.: Occlusion aware unsupervised learning of optical flow. In: Proceedings of Computer Vision and Pattern Recognition, 4884–4893. IEEE (2018)

  49. Wu, L., Cao, X., Foroosh, H.: Camera calibration and geo-location estimation from two shadow trajectories. Comput. Vis. Image Underst. 114(8), 915–927 (2010)

    Article  Google Scholar 

  50. Wu, W., Chen, X.D., Yang, W., Yong, J.H.: Exploring better target for shadow detection. Knowl.-Based Syst. 273, 110614 (2023)

    Article  Google Scholar 

  51. Wu, W., Wu, X., Wan, Y.: Single-image shadow removal using detail extraction and illumination estimation. Vis. Comput. 38(5), 1677–1687 (2022)

    Article  MathSciNet  Google Scholar 

  52. Wu, W., Yang, W., Ma, W., Chen, X.D.: How many annotations do we need for generalizing new-coming shadow images? IEEE Transactions on Circuits and Systems for Video Technology 1–12 (2023)

  53. Wu, W., Zhang, S., Tian, M., Tan, D., Wu, X., Wan, Y.: Learning to detect soft shadow from limited data. Vis. Comput. 38(5), 1665–1675 (2022)

    Article  Google Scholar 

  54. Wu, W., Zhang, S., Zhou, K., Yang, J., Wu, X., Wan, Y.: Shadow removal via dual module network and low error shadow dataset. Comput. Graph. 95, 156–163 (2021)

    Article  Google Scholar 

  55. **e, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and Efficient Design for Semantic Segmentation with Transformers. MIT Press, Cambridge (2021)

    Google Scholar 

  56. **e, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of Computer Vision and Pattern Recognition, 1492–1500. IEEE (2017)

  57. Xu, B., Liang, H., Gong, W., Liang, R., Chen, P.: A visual representation-guided framework with global affinity for weakly supervised salient object detection. IEEE Trans. Circ. Syst. Video Technol. 1–12 (2023)

  58. Yang, W., Wu, W., Chen, X.D., Tao, X., Mao, X.: How to use extra training data for better edge detection? Appl. Intell. 1–15 (2023)

  59. Ye, S., Chen, D., Han, S., Liao, J.: Learning with noisy labels for robust point cloud segmentation. In: Proceedings of International Conference on Computer Vision, 6443–6452 (2021)

  60. Zhang, B., **ao, J., Jiao, J., Wei, Y., Zhao, Y.: Affinity attention graph neural network for weakly supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 44(11), 8082–8096 (2022)

    Article  PubMed  Google Scholar 

  61. Zhang, J., Yu, X., Li, A., Song, P., Liu, B., Dai, Y.: Weakly-supervised salient object detection via scribble annotations. In: Proceedings of Computer Vision and Pattern Recognition, 12546–12555 (2020)

  62. Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.: Distraction-aware shadow detection. In: Proceedings of Computer Vision and Pattern Recognition, 5167–5176. IEEE (2019)

  63. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., **ang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of Computer Vision and Pattern Recognition, 6881–6890. IEEE (2021)

  64. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of Computer Vision and Pattern Recognition, 2921–2929. IEEE (2016)

  65. Zhu, J., Samuel, K.G., Masood, S.Z., Tappen, M.F.: Learning to recognize shadows in monochromatic natural images. In: Proceedings of Computer Vision and Pattern Recognition, 223–230. IEEE (2010)

  66. Zhu, L., Deng, Z., Hu, X., Fu, C.W., Xu, X., Qin, J., Heng, P.A.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of European Conference on Computer Vision, 121–136. Springer (2018)

  67. Zhu, L., Xu, K., Ke, Z., Lau, R.W.: Mitigating intensity bias in shadow detection via feature decomposition and reweighting. In: Proceedings of International Conference on Computer Vision, 4702–4711. IEEE (2021)

  68. Zhu, Y., Fu, X., Cao, C., Wang, X., Sun, Q., Zha, Z.J.: Single image shadow detection via complementary mechanism. In: Proceedings of the ACM International Conference on Multimedia, 6717–6726 (2022)

Download references

Funding

The work was supported by the National Natural Science Foundation of China (61972120), the General Research Project of Zhejiang Provincial Department of Education (Y202044861), the Zhejiang Provincial Natural Science Foundation (Regional Innovation Joint Funds) (LQZSZ24E050001).

Author information

Authors and Affiliations

  1. School of Computer Science, Hangzhou Dianzi University, Hangzhou, China

    Hongyu Chen, **ao-Diao Chen, Wen Wu & Wenya Yang

  2. **nchuang Haihe Laboratory, Tian**, China

    **ao-Diao Chen

  3. Department of Computer Science and Engineering, University of Yamanashi, Kofu, Japan

    **aoyang Mao

Authors
  1. Hongyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. **ao-Diao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenya Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. **aoyang Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

WW was involved in conceptualization, writing—original draft, writing—review & editing. HC helped in validation, methodology, project administration, funding acquisition. X-DC contributed to visualization, formal analysis, data curation. WY and XM helped in supervision.

Corresponding authors

Correspondence to **ao-Diao Chen, Wen Wu or Wenya Yang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethics approval

The work follows appropriate ethical standards in conducting research and writing the manuscript. This work presents computational models trained with publicly available data, for which no ethical approval was required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Chen, XD., Wu, W. et al. Annotate less but perform better: weakly supervised shadow detection via label augmentation. Vis Comput (2024). https://doi.org/10.1007/s00371-024-03278-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-024-03278-6

Keywords

Search

Navigation