SpringerLink
Log in

Foreground Detection and Segmentation in RGB-D Images

  • Chapter
  • First Online:
RGB-D Image Analysis and Processing

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

  • 1776 Accesses

Abstract

Depth information available in RGB-D images facilitate many computer vision tasks. As a newly emerging and significant topic in the computer vision community, foreground detection and segmentation for RGB-D images have gained a lot of research interest in the past years. In this chapter, an overview of some foreground-based tasks in RGB-D images is provided, including saliency detection, co-saliency detection, foreground segmentation, and co-segmentation. We aim at providing comprehensive literature of the introduction, summaries, and challenges in these areas. We expect this review to be beneficial to the researchers in this field and hopefully, encourage more future works in this direction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (Canada)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (Canada)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://hzfu.github.io/proj_rgbdseg.html.

  2. 2.

    https://rmcong.github.io/proj_RGBD_cosal.html.

References

  1. Almomani R, Dong M (2013) Segtrack: a novel tracking system with improved object segmentation. In: ICIP, pp 3939–3943

    Google Scholar 

  2. Arbelaez P, Maire M, Fowlkes C, Malik J (2009) From contours to regions: an empirical evaluation. In: CVPR. IEEE, pp 2294–2301

    Google Scholar 

  3. Armeni I, Sax A, Zamir AR, Savarese S (2017) Joint 2D-3D-semantic data for indoor scene understanding. ar**v:1702.01105

  4. Cao X, Wang F, Zhang B, Fu H, Li C (2016) Unsupervised pixel-level video foreground object segmentation via shortest path algorithm. Neurocomputing 172:235–243

    Article  Google Scholar 

  5. Cao X, Zhang C, Fu H, Guo X, Tian Q (2016) Saliency-aware nonparametric foreground annotation based on weakly labeled data. IEEE Trans Neural Netw Learn Syst 27(6):1253–1265

    Article  MathSciNet  Google Scholar 

  6. Chang K, Liu T, Lai S (2011) From co-saliency to co-segmentation: an efficient and fully unsupervised energy minimization model. In: CVPR, pp 2129–2136

    Google Scholar 

  7. Chen H, Li Y (2018) Progressively complementarity-aware fusion network for RGB-D salient object detection. In: CVPR, pp 3051–3060

    Google Scholar 

  8. Chen H, Li Y (2019) Three-stream attention-aware network for RGB-D salient object detection. IEEE Trans Image Process PP(99):1–12

    Google Scholar 

  9. Chen H, Li Y, Su D (2019) Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection. Pattern Recognit 86:376–385

    Article  Google Scholar 

  10. Cheng MM, Mitra NJ, Huang X, Torr PH, Hu SM (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582

    Article  Google Scholar 

  11. Cheng Y, Cai R, Li Z, Zhao X, Huang K (2017) Locality-sensitive deconvolution networks with gated fusion for RGBD indoor semantic segmentation. In: CVPR, pp 1475–1483

    Google Scholar 

  12. Cinbis RG, Verbeek J, Schmid C (2013) Segmentation driven object detection with fisher vectors. In: ICCV, pp 2968–2975

    Google Scholar 

  13. Cong R, Lei J, Fu H, Cheng MM, Lin W, Huang Q (2018) Review of visual saliency detection with comprehensive information. IEEE Trans Circuits Syst Video Technol PP(99):1–19

    Google Scholar 

  14. Cong R, Lei J, Fu H, Huang Q, Cao X, Hou C (2018) Co-saliency detection for RGBD images based on multi-constraint feature matching and cross label propagation. IEEE Trans Image Process 27(2):568–579

    Article  MathSciNet  Google Scholar 

  15. Cong R, Lei J, Fu H, Huang Q, Cao X, Ling N (2019) HSCS: hierarchical sparsity based co-saliency detection for RGBD images. IEEE Trans Multimed 21(7):1660–1671

    Article  Google Scholar 

  16. Cong R, Lei J, Fu H, Lin W, Huang Q, Cao X, Hou C (2019) An iterative co-saliency framework for RGBD images. IEEE Trans Cybern 49(1):233–246

    Article  Google Scholar 

  17. Cong R, Lei J, Zhang C, Huang Q, Cao X, Hou C (2016) Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Process Lett 23(6):819–823

    Article  Google Scholar 

  18. Desingh K, Krishna KM, Rajan D, Jawahar C (2013) Depth really matters: improving visual salient region detection with depth. In: BMVC

    Google Scholar 

  19. Fan X, Liu Z, Sun G (2014) Salient region detection for stereoscopic images. In: ICDSP, pp 454–458

    Google Scholar 

  20. Feng D, Barnes N, You S, McCarthy C (2016) Local background enclosure for RGB-D salient object detection. In: CVPR, pp 2343–2350

    Google Scholar 

  21. Fu H, Xu D, Lin S (2017) Object-based multiple foreground segmentation in RGBD video. IEEE Trans Image Process 26(3):1418–1427

    Article  MathSciNet  Google Scholar 

  22. Fu H, Xu D, Lin S, Liu J (2015) Object-based RGBD image co-segmentation with mutex constraint. In: CVPR, pp 4428–4436

    Google Scholar 

  23. Fu H, Xu D, Zhang B, Lin S, Ward R (2015) Object-based multiple foreground video co-segmentation via multi-state selection graph. IEEE Trans Image Process 24(11):3415–3424

    Article  MathSciNet  Google Scholar 

  24. Gu K, Wang S, Yang H, Lin W, Zhai G, Yang X, Zhang W (2016) Saliency-guided quality assessment of screen content images. IEEE Trans Multimed 18(6):1098–1110

    Article  Google Scholar 

  25. Guo C, Li C, Guo J, Cong R, Fu H, Han P (2019) Hierarchical features driven residual learning for depth map super-resolution. IEEE Trans Image Process 28(5):2545–2557

    Article  MathSciNet  Google Scholar 

  26. Guo J, Ren T, Bei J (2016) Salient object detection for RGB-D image via saliency evolution. In: ICME, pp 1–6

    Google Scholar 

  27. Guo X, Liu D, Jou B, Zhu M, Cai A, Chang SF (2013) Robust object co-detection. In: CVPR, pp 3206–3213

    Google Scholar 

  28. Gupta S, Girshick R, Arbeláez P, Malik J (2014) Learning rich features from RGB-D images for object detection and segmentation. In: ECCV, pp 345–360

    Google Scholar 

  29. Han J, Chen H, Liu N, Yan C, Li X (2018) CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE Trans Cybern 48(11):3171–3183

    Article  Google Scholar 

  30. Han S, Vasconcelos N (2006) Image compression using object-based regions of interest. In: ICIP, pp 3097–3100

    Google Scholar 

  31. He Y, Chiu W, Keuper M, Fritz M (2017) STD2P: RGBD semantic segmentation using spatio-temporal data-driven pooling. In: CVPR, pp 7158–7167

    Google Scholar 

  32. Hickson S, Birchfield S, Essa I, Christensen H (2014) Efficient hierarchical graph-based segmentation of RGBD videos. In: CVPR, pp 344–351

    Google Scholar 

  33. Hou Q, Cheng MM, Hu X, Borji A, Tu Z, Torr P (2017) Deeply supervised salient object detection with short connections. In: CVPR, pp 5300–5309

    Google Scholar 

  34. Huang Q, Wang W, Neumann U (2018) Recurrent slice networks for 3D segmentation on point clouds. In: CVPR, pp 2626–2635

    Google Scholar 

  35. Jacob H, Padua F, Lacerda A, Pereira A (2017) Video summarization approach based on the emulation of bottom-up mechanisms of visual attention. J Intell Inf Syst 49(2):193–211

    Article  Google Scholar 

  36. Joulin A, Bach F, Ponce J (2010) Discriminative clustering for image co-segmentation. In: CVPR, pp 1943–1950

    Google Scholar 

  37. Ju R, Ge L, Geng W, Ren T, Wu G (2014) Depth saliency based on anisotropic center-surround difference. In: ICIP, pp 1115–1119

    Google Scholar 

  38. Kim G, **ng E, Fei-Fei L, Kanade T (2011) Distributed cosegmentation via submodular optimization on anisotropic diffusion. In: ICCV, pp 169–176

    Google Scholar 

  39. Kong S, Fowlkes C (2018) Recurrent scene parsing with perspective understanding in the loop. In: CVPR, pp 956–965

    Google Scholar 

  40. Lei J, Duan J, Wu F, Ling N, Hou C (2018) Fast mode decision based on grayscale similarity and inter-view correlation for depth map coding in 3D-HEVC. IEEE Trans Circuits Syst Video Technol 28(3):706–718

    Article  Google Scholar 

  41. Lei J, Wu M, Zhang C, Wu F, Ling N, Hou C (2017) Depth-preserving stereo image retargeting based on pixel fusion. IEEE Trans Multimed 19(7):1442–1453

    Article  Google Scholar 

  42. Lerma C, Kosecká J (2015) Semantic parsing for priming object detection in indoors RGB-D scenes. Int J Robot Res 34:582–597

    Article  Google Scholar 

  43. Li C, Guo J, Cong R, Pang Y, Wang B (2016) Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Trans Image Process 25(12):5664–5677

    Article  MathSciNet  Google Scholar 

  44. Li Z, Gan Y, Liang X, Yu Y, Cheng H, Lin L (2016) LSTM-CF: unifying context modeling and fusion with LSTMs for RGB-D scene labeling. In: ECCV, pp 541–557

    Google Scholar 

  45. Lin D, Chen G, Cohen-Or D, Heng P, Huang H (2017) Cascaded feature network for semantic segmentation of RGB-D images. In: ICCV, pp 1320–1328

    Google Scholar 

  46. Mishra A, Shrivastava A, Aloimonos Y (2012) Segmenting “simple” objects using RGB-D. In: ICRA, pp 4406–4413

    Google Scholar 

  47. Ni M, Lei J, Cong R, Zheng K, Peng B, Fan X (2017) Color-guided depth map super resolution using convolutional neural network. IEEE Access 2:26666–26672

    Article  Google Scholar 

  48. Niu Y, Geng Y, Li X, Liu F (2012) Leveraging stereopsis for saliency analysis. In: CVPR, pp 454–461

    Google Scholar 

  49. Pei D, Liu H, Liu Y, Sun F (2013) Unsupervised multimodal feature learning for semantic image segmentation. In: IJCNN, pp 1–6

    Google Scholar 

  50. Peng H, Li B, **ong W, Hu W, Ji R (2014) RGBD salient object detection: a benchmark and algorithms. In: ECCV, pp 92–109

    Google Scholar 

  51. Qi C, Su H, Mo K, Guibas L (2017) PointNet: deep learning on point sets for 3D classification and segmentation. In: CVPR, pp 77–85

    Google Scholar 

  52. Qi C, Yi L, Su H, Guibas L (2017) PointNet++: deep hierarchical feature learning on point sets in a metric space. In: NIPS, pp 5099--5108

    Google Scholar 

  53. Qi X, Liao R, Jia J, Fidler S, Urtasun R (2017) 3D graph neural networks for RGBD semantic segmentation. In: ICCV, pp 5209–5218

    Google Scholar 

  54. Qu L, He S, Zhang J, Tian J, Tang Y, Yang Q (2017) RGBD salient object detection via deep fusion. IEEE Trans Image Process 26(5):2274–2285

    Article  MathSciNet  Google Scholar 

  55. Ren X, Bo L, Fox D (2012) RGB-(D) scene labeling: features and algorithms. In: CVPR, pp 2759–2766

    Google Scholar 

  56. Rother C, Minka T, Blake A, Kolmogorov V (2006) Cosegmentation of image pairs by histogram matching-incorporating a global constraint into MRFs. In: CVPR, pp 993–1000

    Google Scholar 

  57. Sahin C, Kim TK (2019) Recovering 6D object pose: a review and multi-modal analysis. In: ECCV workshops, pp 15–31

    Google Scholar 

  58. Sahin C, Kouskouiras R, Kim TK (2016) Iterative hough forest with histogram of control points. In: IROS, pp 4113–4118

    Google Scholar 

  59. Silberman N, Fergus R (2011) Indoor scene segmentation using a structured light sensor. In: ICCV workshops, pp 601–608

    Google Scholar 

  60. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. ar**v:1409.1556

  61. Socher R, Huval B, Bath B, Manning C, Ng AY (2012) Convolutional-recursive deep learning for 3D object classification. In: NIPS, pp 665–673

    Google Scholar 

  62. Song H, Liu Z, Du H, Sun G, Le Meur O, Ren T (2017) Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning. IEEE Trans Image Process 26(9):4204–4216

    Article  MathSciNet  Google Scholar 

  63. Song H, Liu Z, **e Y, Wu L, Huang M (2016) RGBD co-saliency detection via bagging-based clustering. IEEE Signal Process Lett 23(12):1722–1726

    Article  Google Scholar 

  64. Song S, **ao J (2016) Deep sliding shapes for amodal 3D object detection in RGB-D images. In: CVPR, pp 808–816

    Google Scholar 

  65. Song S, Yu F, Zeng A, Chang A, Savva M, Funkhouser T (2017) Semantic scene completion from a single depth image. In: CVPR, pp 190–198

    Google Scholar 

  66. Sun J, Liu X, Wan W, Li J, Zhao D, Zhang H (2015) Database saliency for fast image retrieval. IEEE Trans Multimed 17(3):359–369

    Article  Google Scholar 

  67. Sun L, Zhao C, Stolkin R (2017) Weakly-supervised DCNN for RGB-D object recognition in real-world applications which lack large-scale annotated training data. ar**v:1703.06370

  68. Toshev A, Shi J, Daniilidis K (2007) Image matching via saliency region correspondences. In: CVPR, pp 1–8

    Google Scholar 

  69. Uijlings JRR, van de Sande KEA, Gevers T, Smeulders AWM (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171

    Article  Google Scholar 

  70. Vicente S, Rother C, Kolmogorov V (2011) Object cosegmentation. In: CVPR, pp 2217–2224

    Google Scholar 

  71. Wang A, Lu J, Wang G, Cai J, Cham T (2014) Multi-modal unsupervised feature learning for RGB-D scene labeling. In: ECCV, pp 453–467

    Google Scholar 

  72. Wang J, Wang Z, Tao D, See S, Wang G (2016) Learning common and specific features for RGB-D semantic segmentation with deconvolutional networks. In: ECCV, pp 664–679

    Google Scholar 

  73. Wang W, Lai Q, Fu H, Shen J, Ling H (2019) Salient object detection in the deep learning era: an in-depth survey. ar**v:1904.09146

  74. Wang W, Neumann U (2018) Depth-aware CNN for RGB-D segmentation. In: ECCV, pp 144–161

    Chapter  Google Scholar 

  75. Wang W, Shen J, Yu Y, Ma KL (2017) Stereoscopic thumbnail creation via efficient stereo saliency detection. IEEE Trans Vis Comput Graph 23(8):2014–2027

    Article  Google Scholar 

  76. Wang W, Yu R, Huang Q, Neumann U (2018) SGPN: similarity group proposal network for 3D point cloud instance segmentation. In: CVPR, pp 2569–2578

    Google Scholar 

  77. Wang X, Gao L, Song J, Shen H (2017) Beyond frame-level CNN: saliency-aware 3-D CNN with LSTM for video action recognition. IEEE Signal Process Lett 24(4):510–514

    Article  Google Scholar 

  78. **e Q, Remil O, Guo Y, Wang M, Wei M, Wang J (2018) Object detection and tracking under occlusion for object-level RGB-D video segmentation. IEEE Trans Multimed 20(3):580–592

    Article  Google Scholar 

  79. Yi L, Kim VG, Ceylan D, Shen IC, Yan M, Su H, Lu C, Huang Q, Sheffer A, Guibas L (2016) A scalable active framework for region annotation in 3D shape collections. ACM Trans Graph 210:1–12

    Article  Google Scholar 

  80. Zhang D, Fu H, Han J, Borji A, Li X (2018) A review of co-saliency detection algorithms: fundamentals, applications, and challenges. ACM Trans Intell Syst Technol 9(4):1–31

    Article  Google Scholar 

  81. Zhang Y, Li L, Cong R, Guo X, Xu H, Zhang J (2018) Co-saliency detection via hierarchical consistency measure. In: ICME, pp 1–6

    Google Scholar 

  82. Zhu C, Li G (2018) A multilayer backpropagation saliency detection algorithm and its applications. Multimed Tools Appl 77:25181–25197

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Bei**g Key Laboratory of Advanced Information Science and Network Technology, Institute of Information Science, Bei**g Jiaotong University, Bei**g, 100044, China

    Runmin Cong

  2. Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, China

    Hao Chen

  3. Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore, Singapore

    Hongyuan Zhu

  4. Inception Institute of Artificial Intelligence, Abu Dhabi, United Arab Emirates

    Huazhu Fu

Authors
  1. Runmin Cong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongyuan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huazhu Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huazhu Fu .

Editor information

Editors and Affiliations

  1. School of Computer Science and Informatics, Cardiff University, Cardiff, UK

    Paul L. Rosin

  2. School of Computer Science and Informatics, Cardiff University, Cardiff, UK

    Yu-Kun Lai

  3. IEEE, University of East Anglia, Norwich, UK

    Ling Shao

  4. Department of Computer Science, Edge Hill University, Ormskirk, UK

    Yonghuai Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cong, R., Chen, H., Zhu, H., Fu, H. (2019). Foreground Detection and Segmentation in RGB-D Images. In: Rosin, P., Lai, YK., Shao, L., Liu, Y. (eds) RGB-D Image Analysis and Processing. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-030-28603-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-28603-3_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-28602-6

  • Online ISBN: 978-3-030-28603-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation