SpringerLink
Log in

Ultra-High-Speed Charge-Domain Temporally Compressive CMOS Image Sensors

  • Chapter
  • First Online:
Coded Optical Imaging

  • 606 Accesses

Abstract

Multi-tap charge modulators can implement temporal compressive sensing in the charge domain, which allows efficient sampling of spatiotemporal information. This processing requires no signal processing circuit on the image sensor. The operation speed is limited only by the speed of the charge modulator, not by the readout frame rate. This chapter explains the concept and benefits of and signal processing for compressive imaging in ultra-high-speed single-event filming and light detection and ranging (LiDAR). Based on our prototype ultra-high-speed CMOS image sensor that works at a burst frame rate of 303 Mfps, single-event filming of laser-induced plasma in laser processing and the decomposition of multiple reflections, which are one kind of multi-path interference in LiDAR, are demonstrated.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 279.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kinko Tsuji. The Micro-World Observed by Ultra High-Speed Cameras. Springer, 2018.

    Book  Google Scholar 

  2. Fabio Remondino Stoppa and David. TOF Range-Imaging Cameras. Springer, Heidelberg, 2013.

    Google Scholar 

  3. Claudio Bruschini, Harald Homulle, Ivan Michel Antolovic, Samuel Burri, and Edoardo Charbon. Single-photon avalanche diode imagers in biophotonics: review and outlook. Light: Science & Applications, Vol. 8, No. 1, p. 87, dec 2019.

    Google Scholar 

  4. S. Cova, A. Longoni, and A. Andreoni. Towards picosecond resolution with single-photon avalanche diodes. Review of Scientific Instruments, Vol. 52, No. 3, pp. 408–412, mar 1981.

    Google Scholar 

  5. R.J. McIntyre. Recent developments in silicon avalanche photodiodes. Measurement, Vol. 3, No. 4, pp. 146–152, oct 1985.

    Google Scholar 

  6. Futa Mochizuki, Keiichiro Kagawa, Shin-ichiro Okihara, Min-Woong Seo, Bo Zhang, Taishi Takasawa, Keita Yasutomi, and Shoji Kawahito. Single-shot 200Mfps 5x3-aperture compressive CMOS imager. In 2015 IEEE International Solid-State Circuits Conference - (ISSCC) Digest of Technical Papers, pp. 1–3. IEEE, feb 2015.

    Google Scholar 

  7. Keiichiro Kagawa, Tomoya Kokado, Yuto Sato, Futa Mochizuki, and Hajime Nagahara. Multi-tap macro-pixel based compressive ultra-high-speed CMOS image sensor. In International Image Sensor Workshop, pp. 2–5, 2019.

    Google Scholar 

  8. Keiichiro Kagawa, Masaya Horio, Anh Ngoc Pham, Thoriq Ibrahim, Shin-ichiro Okihara, Tatsuki Furuhashi, Taishi Takasawa, Keita Yasutomi, Shoji Kawahito, and Hajime Nagahara. A Dual-Mode 303-Megaframes-per-Second Charge-Domain Time-Compressive Computational CMOS Image Sensor. Sensors, Vol. 22, No. 5, p. 1953, mar 2022.

    Google Scholar 

  9. Futa Mochizuki, Keiichiro Kagawa, Ryota Miyagi, Min Woong Seo, Bo Zhang, Taishi Takasawa, Keita Yasutomi, and Shoji Kawahito. Separation of multi-path components in sweep-less time-of-flight depth imaging with a temporally-compressive multi-aperture image sensor. ITE Transactions on Media Technology and Applications, Vol. 6, No. 3, pp. 202–211, 2018.

    Google Scholar 

  10. Masaya Horio, Yu Feng, Tomoya Kokado, Taishi Takasawa, Keita Yasutomi, Shoji Kawahito, Takashi Komuro, Hajime Nagahara, and Keiichiro Kagawa. Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor. Sensors, Vol. 22, No. 7, p. 2442, mar 2022.

    Google Scholar 

  11. D.L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, Vol. 52, No. 4, pp. 1289–1306, apr 2006.

    Google Scholar 

  12. Richard Baraniuk. Compressive Sensing [Lecture Notes]. IEEE Signal Processing Magazine, Vol. 24, No. 4, pp. 118–121, jul 2007.

    Google Scholar 

  13. E.J. Candes and M.B. Wakin. An Introduction To Compressive Sampling. IEEE Signal Processing Magazine, Vol. 25, No. 2, pp. 21–30, mar 2008.

    Google Scholar 

  14. Rudolf Schwarte, Zhan** Xu, Horst-Guenther Heinol, Joachim Olk, Ruediger Klein, Bernd Buxbaum, Helmut Fischer, and Juergen Schulte. New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device (PMD). Proceedings of SPIE, Vol. 3100, pp. 245–253, sep 1997.

    Google Scholar 

  15. Cyrus S. Bamji, Swati Mehta, Barry Thompson, Tamer Elkhatib, Stefan Wurster, Onur Akkaya, Andrew Payne, John Godbaz, Mike Fenton, Vijay Rajasekaran, Larry Prather, Satya Nagaraja, Vishali Mogallapu, Dane Snow, Rich McCauley, Mustansir Mukadam, Iskender Agi, Shaun McCarthy, Zhan** Xu, Travis Perry, William Qian, Vei Han Chan, Prabhu Adepu, Gazi Ali, Muneeb Ahmed, Aditya Mukherjee, Sheethal Nayak, Dave Gampell, Sunil Acharya, Lou Kordus, and Pat O’Connor. 1Mpixel 65nm BSI 320MHz demodulated TOF Image sensor with 3\(\upmu \)m global shutter pixels and analog binning. In Digest of Technical Papers - IEEE International Solid-State Circuits Conference, Vol. 61, pp. 94–96, 2018.

    Google Scholar 

  16. Min-Sun Keel, Daeyun Kim, Yeomyung Kim, Myunghan Bae, Myoungoh Ki, Bumsik Chung, Sooho Son, Hoyong Lee, Heeyoung Jo, Seung-Chul Shin, Sunjoo Hong, Jaeil An, Yonghun Kwon, Sungyoung Seo, Sunghyuck Cho, Youngchan Kim, Young-Gu **, Youngsun Oh, Yitae Kim, JungChak Ahn, Kyoungmin Koh, and Yongin Park. A 4-tap 3.5 \(\upmu \)m 1.2 Mpixel Indirect Time-of-Flight CMOS Image Sensor with Peak Current Mitigation and Multi-User Interference Cancellation. In 2021 IEEE International Solid- State Circuits Conference (ISSCC), pp. 106–108. IEEE, feb 2021.

    Google Scholar 

  17. David Stoppa, Nicola Massari, Lucio Pancheri, Mattia Malfatti, Matteo Perenzoni, and Lorenzo Gonzo. An 80x60 range image sensor based on 10\(\upmu \)m 50MHz lock-in pixels in 0.18\(\upmu \)m CMOS. In 2010 IEEE International Solid-State Circuits Conference - (ISSCC), Vol. 3100, pp. 406–407. IEEE, feb 2010.

    Google Scholar 

  18. Donguk Kim, Seunghyun Lee, Dahwan Park, Canxing Piao, Jihoon Park, Yeonsoo Ahn, Kihwan Cho, Jungsoon Shin, Seung Min Song, Seong ** Kim, Jung Hoon Chun, and Jaehyuk Choi. IndirectTime-of-Flight CMOS Image Sensor with On-Chip Background Light Cancelling and Pseudo-Four-Tap/Two-Tap Hybrid Imaging for Motion Artifact Suppression. IEEE Journal of Solid-State Circuits, Vol. 55, No. 11, pp. 2849–2865, 2020.

    Google Scholar 

  19. Kunihiro Hatakeyama, Yu Okubo, Tomohiro Nakagome, Masahiro Makino, Hiroshi Takashima, Takahiro Akutsu, Takehide Sawamoto, Masanori Nagase, Tatsuo Noguchi, and Shoji Kawahito. A Hybrid Indirect ToF Image Sensor for Long-Range 3D Depth Measurement under High Ambient Light Conditions. In 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits), pp. 46–47. IEEE, jun 2022.

    Google Scholar 

  20. Daniel; Van Nieuwenhove, Ward van der Tempel, and Maarten Kuijk. Novel standard detector using majority current for guiding photo-generated electrons towards detecting junctions. Symposium IEEE/LEOS Benelux Chapter, pp. 229–232, 2005.

    Google Scholar 

  21. Y. Ebiko, H. Yamagishi, K. Tatani, H. Iwamoto, Y. Moriyama, Y. Hagiwara, S. Maeda, T. Murase, T. Suwa, H. Arai, Y. Isogai, S. Hida, S. Kameda, T. Terada, K. Koiso, F. T. Brady, S. Han, A. Basavalingappa, T. Michiel, and T. Ueno. Low power consumption and high resolution 1280X960 Gate Assisted Photonic Demodulator pixel for indirect Time of flight. In Technical Digest - International Electron Devices Meeting, IEDM, Vol. 2020-Decem, pp. 33.1.1–33.1.4, 2020.

    Google Scholar 

  22. Shoji Kawahito, G Baek, Z Li, S M Han, and M W Seo. CMOS Lock-in Pixel Image Sensors with Lateral Electric Field Control for Time-Resolved Imaging. In International Image Sensor Workshop, Vol. 2, pp. 3–5, 2013.

    Google Scholar 

  23. Keita Yasutomi, Yushi Okura, Keiichiro Kagawa, and Shoji Kawahito. A Sub-100\(\upmu \)m-Range-Resolution Time-of-Flight Range Image Sensor With Three-Tap Lock-In Pixels, Non-Overlap** Gate Clock, and Reference Plane Sampling. IEEE Journal of Solid-State Circuits, Vol. 54, No. 8, pp. 2291–2303, 2019.

    Google Scholar 

  24. Ming-Woong Seo, Keiichiro Kagawa, Keita Yasutomi, Yoshimasa Kawata, Nobukazu Teranishi, Zhuo Li, Izhal Abdul Halin, and Shoji Kawahito. A 10 ps Time-Resolution CMOS Image Sensor With Two-Tap True-CDS Lock-In Pixels for Fluorescence Lifetime Imaging. IEEE Journal of Solid-State Circuits, Vol. 51, No. 1, pp. 141–154, jan 2016.

    Google Scholar 

  25. Min Woong Seo, Yuya Shirakawa, Yoshimasa Kawata, Keiichiro Kagawa, Keita Yasutomi, and Shoji Kawahito. A time-resolved four-tap lock-in pixel CMOS image sensor for real-time fluorescence lifetime imaging microscopy. IEEE Journal of Solid-State Circuits, Vol. 53, No. 8, pp. 2319–2330, 2018.

    Google Scholar 

  26. Yuya Shirakawa, Keita Yasutomi, Keiichiro Kagawa, Satoshi Aoyama, and Shoji Kawahito. An 8-Tap CMOS Lock-In Pixel Image Sensor for Short-Pulse Time-of-Flight Measurements. MDPI Sensors, Vol. 20, No. 4, p. 1040, feb 2020.

    Google Scholar 

  27. Cristiano Niclass, Mineki Soga, Hiroyuki Matsubara, Masaru Ogawa, and Manabu Kagami. Cristiano Niclass, Mineki Soga, Hiroyuki Matsubara, Masaru Ogawa, Manabu Kagami. IEEE International Solid-State Circuits Conference, pp. 488–489, 2013.

    Google Scholar 

  28. Toshiki Arai, Jun Yonai, Tetsuya Hayashida, Hiroshi Ohtake, Harry Van Kuijk, and Takeharu Goji Etoh. A 252- V/, 16.7-Million-frames-per-second 312-kpixel back-side-illuminated ultrahigh-speed charge-coupled device. IEEE Transactions on Electron Devices, Vol. 60, No. 10, pp. 3450–3458, 2013.

    Google Scholar 

  29. Takeharu Etoh, Tomoo Okinaka, Yasuhide Takano, Kohsei Takehara, Hitoshi Nakano, Kazuhiro Shimonomura, Taeko Ando, Nguyen Ngo, Yoshinari Kamakura, Vu Dao, Anh Nguyen, Edoardo Charbon, Chao Zhang, Piet De Moor, Paul Goetschalckx, and Luc Haspeslagh. Light-In-Flight Imaging by a Silicon Image Sensor: Toward the Theoretical Highest Frame Rate. MDPI Sensors, Vol. 19, No. 10, p. 2247, may 2019.

    Google Scholar 

  30. Manabu Suzuki, Yuki Sugama, Rihito Kuroda, and Shigetoshi Sugawa. Over 100 Million Frames per Second 368 Frames Global Shutter Burst CMOS Image Sensor with Pixel-wise Trench Capacitor Memory Array. MDPI Sensors, Vol. 20, No. 4, p. 1086, feb 2020.

    Google Scholar 

  31. Yasuhisa Tochigi, Katsuhiko Hanzawa, Yuri Kato, Rihito Kuroda, Hideki Mutoh, Ryuta Hirose, Hideki Tominaga, Kenji Takubo, Yasushi Kondo, and Shigetoshi Sugawa. A global-shutter CMOS image sensor with readout speed of 1-Tpixel/s burst and 780-mpixel/s continuous. IEEE Journal of Solid-State Circuits, Vol. 48, No. 1, pp. 329–338, 2013.

    Article  Google Scholar 

  32. Yusuke Oike. Evolution of Image Sensor Architectures With Stacked Device Technologies. IEEE Transactions on Electron Devices, Vol. 69, No. 6, pp. 2757–2765, jun 2022.

    Google Scholar 

  33. Ayush Bhandari, Micha Feigin, Shahram Izadi, Christoph Rhemann, Mirko Schmidt, and Ramesh Raskar. Resolving multipath interference in kinect: An inverse problem approach. In Proceedings of IEEE Sensors, Vol. 2014-Decem, pp. 614–617, 2014.

    Google Scholar 

  34. Refael Whyte, Lee Streeter, Michael J. Cree, and Adrian A. Dorrington. Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods. Optical Engineering, Vol. 54, No. 11, p. 113109, 2015.

    Google Scholar 

  35. Kazuya Kitano, Takanori Okamoto, Kenichiro Tanaka, Takahito Aoto, Hiroyuki Kubo, Takuya Funatomi, and Yasuhiro Mukaigawa. Recovering temporal psf using tof camera with delayed light emission. IPSJ Transactions on Computer Vision and Applications, Vol. 9, pp. 2–7, 2017.

    Article  Google Scholar 

  36. Robert K. Henderson, Nick Johnston, Sam W. Hutchings, Istvan Gyongy, Tarek Al Abbas, Neale Dutton, Max Tyler, Susan Chan, and Jonathan Leach. A 256 \(\times \) 256 40nm/90nm CMOS 3D-Stacked 120dB Dynamic-Range Reconfigurable Time-Resolved SPAD Imager. In 2019 IEEE International Solid- State Circuits Conference - (ISSCC), pp. 106–108. IEEE, feb 2019.

    Google Scholar 

  37. Oichi Kumagai, Junichi Ohmachi, Masao Matsumura, Shinichiro Yagi, Kenichi Tayu, Keitaro Amagawa, Tomohiro Matsukawa, Osamu Ozawa, Daisuke Hirono, Yasuhiro Shinozuka, Ryutaro Homma, Kumiko Mahara, Toshio Ohyama, Yousuke Morita, Shohei Shimada, Takahisa Ueno, Akira Matsumoto, Yusuke Otake, Toshifumi Wakano, and Takashi Izawa. A 189x600 Back-Illuminated Stacked SPAD Direct Time-of-Flight Depth Sensor for Automotive LiDAR Systems. In Digest of Technical Papers - IEEE International Solid-State Circuits Conference, Vol. 64, pp. 110–112, 2021.

    Google Scholar 

  38. Yasunobu Hitomi, **wei Gu, Mohit Gupta, Tomoo Mitsunaga, and Shree K. Nayar. Video from a single coded exposure photograph using a learned over-complete dictionary. Proceedings of the IEEE International Conference on Computer Vision, pp. 287–294, 2011.

    Google Scholar 

  39. Hajime Nagahara, Toshiki Sonoda, Kenta Endo, Yukinobu Sugiyama, and Rin Ichiro Taniguchi. High-speed imaging using CMOS image sensor with quasi pixel-wise exposure. 2016 IEEE International Conference on Computational Photography, ICCP 2016 - Proceedings, pp. 81–91, 2016.

    Google Scholar 

  40. Yi Luo, Jacky Jiang, Mengye Cai, and Shahriar Mirabbasi. CMOS computational camera with a two-tap coded exposure image sensor for single-shot spatial-temporal compressive sensing. Optics Express, Vol. 27, No. 22, p. 31475, 2019.

    Google Scholar 

  41. Michitaka Yoshida, Akihiko Torii, Masatoshi Okutomi, Kenta Endo, Yukinobu Sugiyama, Rin Ichiro Taniguchi, and Hajime Nagahara. Joint optimization for compressive video sensing and reconstruction under hardware constraints. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 11214 LNCS, pp. 649–663, 2018.

    Google Scholar 

  42. Tadashi Okawara, Michitaka Yoshida, Hajime Nagahara, and Yasushi Yagi. Action Recognition from a Single Coded Image. In 2020 IEEE International Conference on Computational Photography (ICCP), pp. 1–11. IEEE, apr 2020.

    Google Scholar 

  43. Felix Heide, Matthias Hullin, James Gregson, and Wolfgang Heidrich. Light-in-flight: Transient imaging using photonic mixer devices. ACM SIGGRAPH 2013 Emerging Technologies, SIGGRAPH 2013, 2013.

    Google Scholar 

  44. Achuta Kadambi, Refael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. Coded time of flight cameras: Sparse deconvolution to address multipath interference and recover time profiles. ACM Transactions on Graphics, Vol. 32, No. 6, 2013.

    Google Scholar 

  45. Chengbo Li, Wotao Yin, Hong Jiang, and Yin Zhang. An efficient augmented Lagrangian method with applications to total variation minimization. Computational Optimization and Applications, Vol. 56, No. 3, pp. 507–530, dec 2013.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Institute of Electronics, Shizuoka University, Shizuoka, Japan

    Keiichiro Kagawa

  2. Institute of Datability Science, Osaka University, Osaka, Japan

    Hajime Nagahara

Authors
  1. Keiichiro Kagawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hajime Nagahara
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Keiichiro Kagawa .

Editor information

Editors and Affiliations

  1. Laboratory of Applied Computational Imaging, Institut National de la Recherche Scientifique Université du Québec, Varennes, QC, Canada

    **yang Liang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Kagawa, K., Nagahara, H. (2024). Ultra-High-Speed Charge-Domain Temporally Compressive CMOS Image Sensors. In: Liang, J. (eds) Coded Optical Imaging. Springer, Cham. https://doi.org/10.1007/978-3-031-39062-3_28

Download citation

Publish with us

Policies and ethics

Search

Navigation