SpringerLink
Log in

Unsupervised Event-to-Image Reconstruction Based on Domain Adaptation

  • Conference paper
  • First Online:
Digital Multimedia Communications (IFTC 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2067))

Abstract

Event camera outputs a stream of asynchronous events, which suffering from a lot of noise, sparse texture, and lacking of static background information. Existing event-to-image reconstruction (E2IR) methods mostly adopt supervised learning approaches. However, it’s hard to collect ground-truth images for real events. To tackle this challenge, we present a novel unsupervised E2IR method based on domain adaption (DA) in this paper. First, we design a long-short time event memory integration method to convert the unstructured events into structured tensor. The structured tensor could effectively alleviate the problem of missing background, but is still noisy. The E2IR method requires further learning about how to improve the quality of the structured tensor, similar to what is learned in conventional noisy image enhancement (IE) process. Thus we subsequently propose a novel DA-based strategy by transferring the knowledge from IE to E2IR. To better transfer the knowledge, we additionally design a multi-scale adversarial training method at the image-level features to bridge the domain gap. Experimental results show that the performance of our proposed Multi-scale Domain Adaptation based E2IR (MDAE2I) method is comparable to the state-of-the-art.

Supported by the Fundamental Research Funds for the Central Universities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 95.50
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 64.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bardow, P., Davison, A.J., Leutenegger, S.: Simultaneous optical flow and intensity estimation from an event camera. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 884–892 (2016)

    Google Scholar 

  2. Brandli, C., Berner, R., Yang, M., Liu, S.C., Delbruck, T.: A 240 \(\times \) 180 130 db 3 \(\upmu \)s latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circuits 49(10), 2333–2341 (2014)

    Article  Google Scholar 

  3. Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. ar**v preprint ar**v:2204.04676 (2022)

  4. Chen, S., Guo, M.: Live demonstration: Celex-v: A 1m pixel multi-mode event-based sensor. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1682–1683 (2019)

    Google Scholar 

  5. Gallego, G., et al.: Event-based vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 154–180 (2022)

    Article  PubMed  Google Scholar 

  6. Gehrig, D., Gehrig, M., Hidalgo-Carrió, J., Scaramuzza, D.: Video to events: recycling video datasets for event cameras. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3583–3592 (2020)

    Google Scholar 

  7. Guo, M., Ding, R., Chen, S.: Live demonstration: a dynamic vision sensor with direct logarithmic output and full-frame picture-on-demand. In: 2016 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 456–456 (2016)

    Google Scholar 

  8. Huang, T., Li, S., Jia, X., Lu, H., Liu, J.: Neighbor2neighbor: self-supervised denoising from single noisy images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14781–14790, June 2021

    Google Scholar 

  9. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    Google Scholar 

  10. Kim, H., Handa, A., Benosman, R., Ieng, S.H., Davison, A.: Simultaneous mosaicing and tracking with an event camera. In: Proceedings of the British Machine Vision Conference (2014)

    Google Scholar 

  11. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105–114 (2017)

    Google Scholar 

  12. Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M., Yang, M.H.: Diverse image-to-image translation via disentangled representations. In: Proceedings of the European Conference on Computer Vision (ECCV), September 2018

    Google Scholar 

  13. Lichtsteiner, P., Posch, C., Delbruck, T.: A 128 \(\times \) 128 120 db 15 \(\upmu \)s latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits 43(2), 566–576 (2008)

    Article  Google Scholar 

  14. Ma, J., et al.: Blind image quality assessment with active inference. IEEE Trans. Image Process. 30, 3650–3663 (2021). https://doi.org/10.1109/TIP.2021.3064195

    Article  PubMed  Google Scholar 

  15. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013)

    Google Scholar 

  16. Moeys, D.P., et al.: A sensitive dynamic and active pixel vision sensor for color or neural imaging applications. IEEE Trans. Biomed. Circuits Syst. 12(1), 123–136 (2018)

    Article  PubMed  Google Scholar 

  17. Moorthy, A.K., Bovik, A.C.: A two-step framework for constructing blind image quality indices. IEEE Signal Process. Lett. 17(5), 513–516 (2010)

    Article  Google Scholar 

  18. Mueggler, E., Rebecq, H., Gallego, G., Delbruck, T., Scaramuzza, D.: The event-camera dataset and simulator: event-based data for pose estimation, visual odometry, and slam. Int. J. Robot. Res. 36(2), 142–149 (2017)

    Article  Google Scholar 

  19. Paredes-Vallés, F., de Croon, G.C.H.E.: Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3445–3454 (2021)

    Google Scholar 

  20. Posch, C., Matolin, D., Wohlgenannt, R.: A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds. IEEE J. Solid-State Circuits 46(1), 259–275 (2011)

    Article  Google Scholar 

  21. Rebecq, H., Gehrig, D., Scaramuzza, D.: Esim: an open event camera simulator. In: Proceedings of the 2nd Conference on Robot Learning, vol. 87, pp. 969–982, 29–31 October 2018

    Google Scholar 

  22. Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: Events-to-video: bringing modern computer vision to event cameras. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3852–3861 (2019)

    Google Scholar 

  23. Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: High speed and high dynamic range video with an event camera. IEEE Trans. Pattern Anal. Mach. Intell. 43(6), 1964–1980 (2021)

    Article  PubMed  Google Scholar 

  24. Reinbacher, C., Graber, G., Pock, T.: Real-time intensity-image reconstruction for event cameras using manifold regularisation. Int. J. Comput. Vis. 126(12) (2018)

    Google Scholar 

  25. Scheerlinck, C., Rebecq, H., Gehrig, D., Barnes, N., Mahony, R.E., Scaramuzza, D.: Fast image reconstruction with an event camera. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 156–163 (2020)

    Google Scholar 

  26. Stoffregen, T., et al.: Reducing the sim-to-real gap for event cameras. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12372, pp. 534–549. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58583-9_32

    Chapter  Google Scholar 

  27. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103 (2008)

    Google Scholar 

  28. Wang, L., Kim, T.K., Yoon, K.J.: Eventsr: from asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8312–8322 (2020)

    Google Scholar 

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  PubMed  Google Scholar 

  30. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018). https://doi.org/10.1109/CVPR.2018.00068

Download references

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, **dian University, **’an, China

    Jupo Ma, Zhenqiang Zhao & Wen Yang

  2. Pazhou Lab, Huangpu, 510555, China

    Jupo Ma & Wen Yang

Authors
  1. Jupo Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenqiang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen Yang .

Editor information

Editors and Affiliations

  1. Shanghai Jiao Tong Univeristy, Shanghai, China

    Guangtao Zhai

  2. Shanghai Jiao Tong University, Shanghai, China

    Jun Zhou

  3. Communication University of China, Bei**g, China

    Long Ye

  4. Shanghai Jiao Tong University, Shanghai, China

    Hua Yang

  5. Shanghai University, Shanghai, China

    ** An

  6. Shanghai Jiao Tong University, Shanghai, China

    **aokang Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ma, J., Zhao, Z., Yang, W. (2024). Unsupervised Event-to-Image Reconstruction Based on Domain Adaptation. In: Zhai, G., Zhou, J., Ye, L., Yang, H., An, P., Yang, X. (eds) Digital Multimedia Communications. IFTC 2023. Communications in Computer and Information Science, vol 2067. Springer, Singapore. https://doi.org/10.1007/978-981-97-3626-3_15

Download citation

Publish with us

Policies and ethics

Search

Navigation