SpringerLink
Log in

EvAC3D: From Event-Based Apparent Contours to 3D Models via Continuous Visual Hulls

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13667))

Included in the following conference series:

Abstract

3D reconstruction from multiple views is a successful computer vision field with multiple deployments in applications. State of the art is based on traditional RGB frames that enable optimization of photo-consistency cross views. In this paper, we study the problem of 3D reconstruction from event-cameras, motivated by the advantages of event-based cameras in terms of low power and latency as well as by the biological evidence that eyes in nature capture the same data and still perceive well 3D shape. The foundation of our hypothesis that 3D-reconstruction is feasible using events lies in the information contained in the occluding contours and in the continuous scene acquisition with events. We propose Apparent Contour Events (ACE), a novel event-based representation that defines the geometry of the apparent contour of an object. We represent ACE by a spatially and temporally continuous implicit function defined in the event x-y-t space. Furthermore, we design a novel continuous Voxel Carving algorithm enabled by the high temporal resolution of the Apparent Contour Events. To evaluate the performance of the method, we collect MOEC-3D, a 3D event dataset of a set of common real-world objects. We demonstrate EvAC3D’s ability to reconstruct high-fidelity mesh surfaces from real event sequences while allowing the refinement of the 3D reconstruction for each individual event. The code, data and supplementary material for this work can be accessed through the project page: https://www.cis.upenn.edu/~ziyunw/evac3d/.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (Canada)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Barrow, H.G., Tenenbaum, J.M.: Interpreting line drawings as three-dimensional surfaces. Artif. Intell. 17(1–3), 75–116 (1981)

    Article  MATH  Google Scholar 

  2. Baudron, A., Wang, Z.W., Cossairt, O., Katsaggelos, A.K.: E3D: event-based 3D shape reconstruction. ar**v preprint ar**v:2012.05214 (2020)

  3. Baumgart, B.G.: Geometric modeling for computer vision. Stanford University (1974)

    Google Scholar 

  4. Carneiro, J., Ieng, S.H., Posch, C., Benosman, R.: Event-based 3D reconstruction from neuromorphic retinas. Neural Netw. 45, 27–38 (2013)

    Article  Google Scholar 

  5. Chaney, K., Zhu, A.Z., Daniilidis, K.: Learning event-based height from plane and parallax. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3690–3696 (2019). https://doi.org/10.1109/IROS40897.2019.8968223

  6. Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_38

    Chapter  Google Scholar 

  7. Cipolla, R., Blake, A.: Surface shape from the deformation of apparent contours. Int. J. Comput. Vision 9(2), 83–112 (1992)

    Article  Google Scholar 

  8. Gehrig, D., Gehrig, M., Hidalgo-Carrió, J., Scaramuzza, D.: Video to events: recycling video datasets for event cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    Google Scholar 

  9. Giblin, P.: Reconstruction of surfaces from profiles. In: Proceedings of 1st International Conference on Computer Vision, London, 1987 (1987)

    Google Scholar 

  10. Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: A papier-mâché approach to learning 3D surface generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 216–224 (2018)

    Google Scholar 

  11. Ha, D., Dai, A., Le, Q.V.: Hypernetworks. ar** framework based on octrees. Autonomous Robots (2013). https://doi.org/10.1007/s10514-012-9321-0. https://octomap.github.io

  12. Kim, H., Leutenegger, S., Davison, A.J.: Real-time 3D reconstruction and 6-DoF tracking with an event camera. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 349–364. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_21

    Chapter  Google Scholar 

  13. Laurentini, A.: The visual hull: a new tool for contour-based image understanding. In: Proceedings of the 7th Scandinavian Conference on Image Analysis, vol. 993, p. 1002 (1991)

    Google Scholar 

  14. Lin, C.H., et al.: Photometric mesh optimization for video-aligned 3D object reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 969–978 (2019)

    Google Scholar 

  15. Marr, D.: Analysis of occluding contour. Proc. Roy. Soc. London. Ser. B. Biol. Sci. 197(1129), 441–475 (1977)

    Google Scholar 

  16. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460–4470 (2019)

    Google Scholar 

  17. Mitchell, E., Engin, S., Isler, V., Lee, D.D.: Higher-order function networks for learning composable 3D object representations. ar**v preprint ar**v:1907.10388 (2019)

  18. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174 (2019)

    Google Scholar 

  19. Rebecq, H., Gallego, G., Mueggler, E., Scaramuzza, D.: EMVS: event-based multi-view stereo-3D reconstruction with an event camera in real-time. Int. J. Comput. Vision 126(12), 1394–1414 (2018)

    Article  Google Scholar 

  20. Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: High speed and high dynamic range video with an event camera. IEEE Trans. Pattern Anal. Mach. Intell. 43(6), 1964–1980 (2019)

    Article  Google Scholar 

  21. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  22. Szeliski, R.: Rapid octree construction from image sequences. CVGIP: Image Understanding 58(1), 23–32 (1993)

    Google Scholar 

  23. Wang, Z., Isler, V., Lee, D.D.: Surface HOF: surface reconstruction from a single image using higher order function networks. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2666–2670. IEEE (2020)

    Google Scholar 

  24. Wang, Z., Mitchell, E.A., Isler, V., Lee, D.D.: Geodesic-HOF: 3D reconstruction without cutting corners. ar**v preprint ar**v:2006.07981 (2020)

  25. Wong, K.Y., Cipolla, R.: Structure and motion from silhouettes. In: Proceedings Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 2, pp. 217–222. IEEE (2001)

    Google Scholar 

  26. Zhou, Q.Y., Park, J., Koltun, V.: Open3D: a modern library for 3D data processing. ar**v:1801.09847 (2018)

  27. Zhou, Y., Gallego, G., Rebecq, H., Kneip, L., Li, H., Scaramuzza, D.: Semi-dense 3D reconstruction with a stereo event camera. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 242–258. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_15

    Chapter  Google Scholar 

  28. Zhu, A.Z., Chen, Y., Daniilidis, K.: Realtime time synchronized event-based stereo. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 438–452. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_27

    Chapter  Google Scholar 

  29. Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: Unsupervised event-based learning of optical flow, depth, and egomotion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 989–997 (2019)

    Google Scholar 

Download references

Acknowledgement

We thank the support from the following grants: NSF TRIPODS 1934960, NSF CPS 2038873, ARL DCIST CRA W911NF-17-2-0181, ARO MURI W911NF-20-1-0080, ONR N00014-17-1-2093, DARPA-SRC C-BRIC, and IARPA ME4AI. We also thank William Sturgeon from the Fisher Fine Arts Materials Library for providing the Artec Spider scanner and assistance.

Author information

Authors and Affiliations

  1. University of Pennsylvania, Philadelphia, PA, 19104, USA

    Ziyun Wang, Kenneth Chaney & Kostas Daniilidis

Authors
  1. Ziyun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kenneth Chaney
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kostas Daniilidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ziyun Wang or Kenneth Chaney .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 10126 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Chaney, K., Daniilidis, K. (2022). EvAC3D: From Event-Based Apparent Contours to 3D Models via Continuous Visual Hulls. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20071-7_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20070-0

  • Online ISBN: 978-3-031-20071-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation