SpringerLink
Log in

Probabilistic-Based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

The high-dimensional nature of the 4-D light field (LF) poses great challenges in achieving efficient and effective feature embedding, that severely impacts the performance of downstream tasks. To tackle this crucial issue, in contrast to existing methods with empirically-designed architectures, we propose a probabilistic-based feature embedding (PFE), which learns a feature embedding architecture by assembling various low-dimensional convolution patterns in a probability space for fully capturing spatial-angular information. Building upon the proposed PFE, we then leverage the intrinsic linear imaging model of the coded aperture camera to construct a cycle-consistent 4-D LF reconstruction network from coded measurements. Moreover, we incorporate PFE into an iterative optimization framework for 4-D LF denoising. Our extensive experiments demonstrate the significant superiority of our methods on both real-world and synthetic 4-D LF images, both quantitatively and qualitatively, when compared with state-of-the-art methods. The source code will be publicly available at https://github.com/lyuxianqiang/LFCA-CR-NET.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data Availability

As indicated in the manuscript, the used datasets are deposited in publicly available repositories.

References

  • Alain, M., & Smolic, A. (2017). Light field denoising by sparse 5d transform domain collaborative filtering. In International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6. IEEE.

  • Ashok, A., & Neifeld, M. A. (2010). Compressive light field imaging. In Three-Dimensional Imaging, Visualization, and Display 2010 and Display Technologies and Applications for Defense, Security, and Avionics IV, volume 7690, pp. 221–232.

  • Beck, A., & Teboulle, M. (2009). Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Transactions on Image Processing, 18(11), 2419–2434.

    Article  MathSciNet  Google Scholar 

  • Chen, J., Hou, J., & Chau, L.-P. (2018). Light field denoising via anisotropic parallax analysis in a cnn framework. IEEE Signal Processing Letters, 25(9), 1403–1407.

    Article  Google Scholar 

  • Chen, J., Hou, J., Ni, Y., & Chau, L.-P. (2018). Accurate light field depth estimation with superpixel regularization over partially occluded regions. IEEE Transactions on Image Processing, 27(10), 4889–4900.

    Article  MathSciNet  Google Scholar 

  • Chen, Y., Zhang, S., Chang, S., & Lin, Y. (2022). Light field reconstruction using efficient pseudo 4d epipolar-aware structure. IEEE Transactions on Computational Imaging, 8, 397–410.

    Article  Google Scholar 

  • Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.

    Article  MathSciNet  Google Scholar 

  • Damianou, A., & Lawrence, N. D. (2013). Deep Gaussian processes. In Artificial Intelligence and Statistics, pp. 207–215. PMLR.

  • Dansereau, D. G., Bongiorno, D. L., Pizarro, O., & Williams, S. B. (2013). Light field image denoising using a linear 4d frequency-hyperfan all-in-focus filter. In Computational Imaging XI, volume 8657, pp. 176–189. SPIE.

  • Derin Babacan, S., Ansorge, R., Luessi, M., Matarán, P. R., Molina, R., & Katsaggelos, A. K. (2012). Compressive light field sensing. IEEE Transactions on Image Processing, 21(12), 4746–4757.

    Article  MathSciNet  Google Scholar 

  • Gal, Y. & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050–1059. PMLR.

  • Gaochang, W., Liu, Y., Dai, Q., & Chai, T. (2019). Learning sheared epi structure for light field reconstruction. IEEE Transactions on Image Processing, 28(7), 3261–3273.

    Article  MathSciNet  Google Scholar 

  • Guo, M., Hou, J., **, J., Chen, J., & Chau, L.-P. (2020). Deep spatial-angular regularization for compressive light field reconstruction over coded apertures. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 278–294.

  • Guo, M., Hou, J., **, J., Chen, J., & Chau, L.-P. (2022). Deep spatial-angular regularization for light field imaging, denoising, and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6094–6110.

    Article  Google Scholar 

  • Gupta, M, Jauhari, A., Kulkarni, K., Jayasuriya, S., Molnar, A., & Turaga, P. (2017). Compressive light field reconstructions using deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 11–20.

  • Heber, S., Yu, W., & Pock, T. (2017). Neural epi-volume networks for shape from light field. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2252–2260.

  • Hog, M., Sabater, N., & Guillemot, C. (2016). Light field segmentation using a ray-based graph structure. In European Conference on Computer Vision (ECCV), pp. 35–50.

  • Honauer, K., Johannsen, O., Kondermann, D., & Goldluecke, B. (2016). A dataset and evaluation methodology for depth estimation on 4d light fields. In Asian Conference on Computer Vision (ACCV), pp. 19–34.

  • Hou, J., Chen, J., & Chau, L.-P. (2018). Light field image compression based on bi-level view compensation with rate-distortion optimization. IEEE Transactions on Circuits and Systems for Video Technology, 29(2), 517–530.

    Article  Google Scholar 

  • Inagaki, Y., Kobayashi, Y., Takahashi, K., Fujii, T., & Nagahara, H. (2018). Learning to capture light fields through a coded aperture camera. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 418–434.

  • Jang, E., Gu, S., & Poole, B. (2017). Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations (ICLR).

  • **, J., Hou, J., Chen, J., & Kwong, S. (2020a). Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2260–2269.

  • **, J., Hou, J., Yuan, H., & Kwong, S. (2020). Learning light field angular super-resolution via a geometry-aware network. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11141–11148.

  • **g, D., Zhang, S., Cong, R., & Lin, Y. (2021). Occlusion-aware bi-directional guided network for light field salient object detection. In Proceedings of the ACM International Conference on Multimedia, pp. 1692–1701.

  • **, J., Guo, M., Hou, J., Liu, H., & **ong, H. (2023). Light field reconstruction via deep adaptive fusion of hybrid lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 12050–12067.

    Google Scholar 

  • **, J., & Hou, J. (2022). Occlusion-aware unsupervised learning of depth from 4-d light fields. IEEE Transactions on Image Processing, 31, 2216–2228.

    Article  Google Scholar 

  • **, J., Hou, J., Chen, J., Zeng, H., Kwong, S., & **gyi, Yu. (2022). Deep coarse-to-fine dense light field reconstruction with flexible sampling and geometry-aware fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4), 1819–1836.

    Article  Google Scholar 

  • Kalantari, N. K., Wang, T.-C., & Ramamoorthi, R. (2016). Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, 35(6), 1–10.

    Article  Google Scholar 

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. ar**v:1412.6980.

  • Lamba, M., Rachavarapu, K. K., & Mitra, K. (2020). Harnessing multi-view perspective of light fields for low-light imaging. IEEE Transactions on Image Processing, 30, 1501–1513.

    Article  MathSciNet  Google Scholar 

  • Li, Z., Baker, H., & Bajcsy, R. (2013). Joint image denoising using light-field data. In IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1–6. IEEE.

  • Li, N., Ye, J., Ji, Y., Ling, H., & Yu, J. (2014). Saliency detection on light field. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2806–2813.

  • Liang, C.-K., Lin, T.-H., Wong, B.-Y., Liu, C., & Chen, H. H. (2008). Programmable aperture photography: Multiplexed light field acquisition. In ACM SIGGRAPH 2008 Papers, pp. 1–10.

  • Liang, Z., Wang, Y., Wang, L., Yang, J., Zhou, S., & Guo, Y. (2023). Learning non-local spatial-angular correlation for light field image super-resolution. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Liang, Z., Wang, Y., Wang, L., Yang, J., & Zhou, S. (2022). Light field image super-resolution with transformers. IEEE Signal Processing Letters, 29, 563–567.

    Article  Google Scholar 

  • Liu, H., Simonyan, K., & Yang, Y. (2019). Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR).

  • Lytro. (2016). http://lightfield.stanford.edu/acq.html.

  • Lyu, X., Zhu, Z., Guo, M., **, J., Hou, J., & Zeng, H. (2021). Learning spatial-angular fusion for compressive light field imaging in a cycle-consistent framework. In Proceedings of the ACM International Conference on Multimedia, pp. 4613–4621.

  • Maggioni, M., Boracchi, G., Foi, A., & Egiazarian, K. (2012). Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms. IEEE Transactions on Image Processing, 21(9), 3952–3966.

    Article  MathSciNet  Google Scholar 

  • Marwah, K., Wetzstein, G., Bando, Y., & Raskar, R. (2013). Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, 32(4), 1–12.

    Article  Google Scholar 

  • Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005). Light field photography with a hand-held plenoptic camera. PhD thesis, Stanford University.

  • Ng, R., et al. (2006). Digital light field photography (Vol. 7). Stanford University.

    Google Scholar 

  • Park, K., Lee, K. M., et al. (2017). Robust light field depth estimation using occlusion-noise aware data costs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10), 2484–2497.

    Google Scholar 

  • Premaratne, S. U., Liyanage, N., Edussooriya, C. U. S., & Wijenayake, C. (2020). Real-time light field denoising using a novel linear 4-d hyperfan filter. IEEE Transactions on Circuits and Systems I: Regular Papers, 67(8), 2693–2706.

    Article  Google Scholar 

  • RayTrix. 3d light field camera technology. https://raytrix.de/.

  • Ren, C., He, X., Wang, C., & Zhao, Z. (2021). Adaptive consistency prior based deep network for image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8596–8606.

  • Sepas-Moghaddam, A., Correia, P. L., & Pereira, F. (2016). Light field denoising: Exploiting the redundancy of an epipolar sequence representation. In 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), pp. 1–4. IEEE.

  • Shi, J., Jiang, X., & Guillemot, C. (2019). A framework for learning depth from a flexible subset of dense and sparse light field views. IEEE Transactions on Image Processing, 28(12), 5867–5880.

    Article  MathSciNet  Google Scholar 

  • Shin, C., Jeon, H.-G., Yoon, Y., Kweon, I. S., & Kim, S. J. (2018). Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4748–4757.

  • Smith, L. N., & Topin, N. (2019). Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006, pp. 369–386.

  • Sunder Raj, A., Lowney, M., & Wetzstein, G. Stanford lytro light field archive. http://lightfields.stanford.edu/LF2016.html.

  • Tassano, M., Delon, J., & Veit, T. (2020). Fastdvdnet: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1354–1363.

  • Tomita, K., Tsutake, C., Takahashi, K., & Fujii, T. (2022). Denoising multi-view images by soft thresholding: A short-time dft approach. Signal Processing: Image Communication, 105, 116710.

    Google Scholar 

  • Vadathya, A. K., Girish, S., & Mitra, K. (2020). A unified learning-based framework for light field reconstruction from coded projections. IEEE Transactions on Computational Imaging, 6, 304–316.

    Article  Google Scholar 

  • Vaibhav, V., & Andrew, A. Light field gantry. https://raytrix.de/.

  • Van Duong, V., Huu, T. N., Yim, J., & Jeon, B. (2023). Light field image super-resolution network via joint spatial-angular and epipolar information. IEEE Transactions on Computational Imaging, 9, 350–366.

    Article  Google Scholar 

  • Wang, T.-C., Efros, A. A, & Ramamoorthi, R. (2015). Occlusion-aware depth estimation using light-field cameras. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3487–3495.

  • Wang, Y., Liu, F., Wang, Z., Hou, G., Sun, Z., & Tan, T. (2018). End-to-end view synthesis for light field imaging with pseudo 4dcnn. In European Conference on Computer Vision (ECCV), pp. 333–348.

  • Wang, T., Piao, Y., Li, X., Zhang, L., & Lu, H. (2019). Deep learning for light field saliency detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8838–8848.

  • Wang, Y., Wang, L., Liang, Z., & Yang, J., et al. (2023b). Ntire 2023 challenge on light field image super-resolution: Dataset, methods and results. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1320–1335.

  • Wang, Y., Wang, L., Yang, J., An, W., Yu, J., & Guo, Y. (2020). Spatial-angular interaction for light field image super-resolution. In European Conference on Computer Vision (ECCV), pp. 290–308.

  • Wang, S., Zhou, T., Yao, L., & Di, H. (2022). Detail-preserving transformer for light field image super-resolution. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2522–2530.

  • Wang, T.-C., Zhu, J.-Y., Hiroaki, E., Chandraker, M., Efros, A. A., & Ramamoorthi, R. (2016). A 4d light-field dataset and cnn architectures for material recognition. In European Conference on Computer Vision (ECCV), pp. 121–138.

  • Wang, X., Lin, Y., & Zhang, S. (2023). Multi-stream progressive restoration for low-light light field enhancement and denoising. IEEE Transactions on Computational Imaging, 9, 70–82.

    Article  Google Scholar 

  • Wang, Y., Liu, Y., Heidrich, W., & Dai, Q. (2017). The light field attachment: Turning a dslr into a light field camera using a low budget camera ring. IEEE Transactions on Visualization and Computer Graphics, 23(10), 2357–2364.

    Article  Google Scholar 

  • Wang, N., Shiming, X. I. A. N. G., Pan, C., et al. (2021). You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9), 2891–2904.

    Article  Google Scholar 

  • Wang, Y., Wang, L., Gaochang, W., Yang, J., An, W., **gyi, Yu., & Guo, Y. (2023). Disentangling light fields for super-resolution and disparity estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 425–443.

    Article  Google Scholar 

  • Wang, Y., Yang, J., Wang, L., Ying, X., Tianhao, W., An, W., & Guo, Y. (2021). Light field image super-resolution using deformable convolution. IEEE Transactions on Image Processing, 30, 1057–1071.

    Article  Google Scholar 

  • Wanner, S., & Goldluecke, B. (2013). Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3), 606–619.

    Article  Google Scholar 

  • Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., & Levoy, M. (2005). High performance imaging using large camera arrays. ACM Transactions on Graphics, 24(3), 765–776.

  • Wu, G., Zhao, M., Wang, L., Dai, Q., Chai, T., Liu, Y. (2017). Light field reconstruction using deep convolutional network on epi. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6319–6327.

  • Yang, J., Wang, L., Ren, L., Cao, Y., & Cao, Y. (2023). Light field angular super-resolution based on structure and scene information. Applied Intelligence, 53(4), 4767–4783.

    Article  Google Scholar 

  • Yeung, H. W. F., Hou, J., Chen, J., Chung, Y. Y., & Chen, X. (2018). Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues. In European Conference on Computer Vision (ECCV), pp. 137–152.

  • Yeung, H. W. F., Hou, J., Chen, X., Chen, J., Chen, Z., & Chung, Y. Y. (2019). Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Transactions on Image Processing, 28(5), 2319–2330.

    Article  MathSciNet  Google Scholar 

  • Yoon, Y., Jeon, H.-G., Yoo, D., Lee, J.-Y., & Kweon, I. S. (2015). Learning a deep convolutional network for light-field image super-resolution. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 24–32.

  • Zhang, S., Lin, Y., & Sheng, H. (2019). Residual networks for light field image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11046–11055.

  • Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7), 3142–3155.

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Key R &D Program of China No. 2022YFE0200300, in part by the Hong Kong Research Grants Council under Grants 11218121 and 21211518, and in part by the Hong Kong Innovation and Technology Fund under Grant MHP/117/21.

Author information

Authors and Affiliations

  1. Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong

    **anqiang Lyu & Junhui Hou

  2. City University of Hong Kong Shenzhen Research Institute, Shenzhen, China

    **anqiang Lyu & Junhui Hou

Authors
  1. **anqiang Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junhui Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junhui Hou.

Ethics declarations

Conflict of interest

The authors declare that they do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Communicated by Chongyi Li.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lyu, X., Hou, J. Probabilistic-Based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising. Int J Comput Vis 132, 2255–2275 (2024). https://doi.org/10.1007/s11263-023-01974-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01974-9

Keywords

Search

Navigation