Hardware Implementation for Spiking Neural Networks on Edge Devices

  • Chapter
  • First Online:
Predictive Analytics in Cloud, Fog, and Edge Computing

Abstract

In the modern data-intensive Internet of Things (IoT) applications, intelligence is enabled by collecting data at the edge devices for processing. The general lack of data processing power in these devices (due mainly to resource constraints) means that the data collected needs to be transmitted to the cloud for processing. The key issues arising from the use of the data communication channel include large energy consumption, communication delays and bandwidth issues, and potential security risks. An attractive approach to avoid these issues is to imbue the edge devices with more powerful data processing capabilities. The main design challenges to improve the data processing capabilities of edge devices are associated with the tight energy and hardware resource constraints in these edge devices. Moreover, the neural networks that are widely deployed for IoT applications demand a large amount of memory and energy. In this chapter, we review the techniques that aim to improve the data processing capabilities of edge devices. Generally, the research community has proposed techniques aimed at improving the algorithms as well as the underlying hardware. We will first discuss the artificial neural network (ANN) and the spiking neural network (SNN) to motivate our focus on hardware design techniques for emulating SNN on the edge devices. Thereafter, this chapter will present a survey of the hardware architecture and optimization techniques proposed to emulate the state-of-the-art SNN on the edge devices.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 139.09
Price includes VAT (France)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 179.34
Price includes VAT (France)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info
Hardcover Book
EUR 179.34
Price includes VAT (France)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Davidson S, Furber SB (2021) Comparison of artificial and spiking neural networks on digital hardware. Front Neurosci 15:345. https://doi.org/10.3389/fnins.2021.651141

    Article  Google Scholar 

  2. Véstias MP (2019) A survey of convolutional neural networks on edge with reconfigurable computing. Algorithms 12(8):154. https://doi.org/10.3390/a12080154

    Article  Google Scholar 

  3. Roy A, Venkataramani S, Gala N, Sen S, Veezhinathan K, Raghunathan A (2017) A programmable event-driven architecture for evaluating spiking neural networks. In: ISLPED, IEEE, Piscataway, pp 1–6. https://doi.org/10.1109/ISLPED.2017.8009176

    Google Scholar 

  4. Lee C, Sarwar SS, Panda P, Srinivasan G, Roy K (2020) Enabling spike-based backpropagation for training deep neural network architectures. Front Neurosci. https://doi.org/10.3389/fnins.2020.00119

  5. Guo W, Fouda ME, Eltawil AM, Salama KN (2021) Neural coding in spiking neural networks: a comparative study for robust neuromorphic systems. Front Neurosci 15:212. https://doi.org/10.3389/fnins.2021.638474

    Article  Google Scholar 

  6. Dayan P, Abbott LF et al. (2001) Theoretical neuroscience, vol 806. MIT Press, Cambridge, MA. https://doi.org/10.1086/421681

  7. Stuijt J, Sifalakis M, Yousefzadeh A, Corradi F (2021) μBrain: An event-driven and fully synthesizable architecture for spiking neural networks. Front Neurosci 15:538. https://doi.org/10.3389/fnins.2021.664208

  8. Morcos B, Stewart TC, Eliasmith C, Kapre N (2018) Implementing NEF neural networks on embedded FPGAs. In: FPT, IEEE, Piscataway, pp 22–29. https://doi.org/10.1109/FPT.2018.00015

  9. Yin S, Venkataramanaiah SK, Chen GK, Krishnamurthy R, Cao Y, Chakrabarti C, Seo JS (2017) Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. In: BioCAS, IEEE, Piscataway, pp 1–5. https://doi.org/10.1109/BIOCAS.2017.8325230

    Google Scholar 

  10. Buhler FN, Brown P, Li J, Chen T, Zhang Z, Flynn MP (2017) A 3.43 TOPS/W 48.9 pj/pixel 50.1 nj/classification 512 analog neuron sparse coding neural network with on-chip learning and classification in 40 nm CMOS. In: IEEE Symp. VLSI Circuits, IEEE, pp C30–C31. https://doi.org/10.23919/VLSIC.2017.8008536

  11. Hecht-Nielsen R (1992) Theory of the backpropagation neural network. In: Neural networks for perception. Elsevier, Amsterdam, pp 65–93. https://doi.org/10.1016/B978-0-12-741252-8.50010-8

    Chapter  Google Scholar 

  12. Diehl PU, Neil D, Binas J, Cook M, Liu SC, Pfeiffer M (2015) Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In: IJCNN, IEEE, Piscataway, pp 1–8. https://doi.org/10.1109/IJCNN.2015.7280696

  13. Chen R, Ma H, **e S, Guo P, Li P, Wang D (2018) Fast and efficient deep sparse multi-strength spiking neural networks with dynamic pruning. In: IJCNN, IEEE, Piscataway, pp 1–8. https://doi.org/10.1109/IJCNN.2018.8489339

  14. Hebb DO (2005) The organization of behavior: a neuropsychological theory. Psychology Press. https://doi.org/10.4324/9781410612403

  15. Song S, Miller KD, Abbott LF (2000) Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nat Neurosci 3(9):919–926. https://doi.org/10.1038/78829

    Article  Google Scholar 

  16. Kheradpisheh SR, Masquelier T (2020) Temporal backpropagation for spiking neural networks with one spike per neuron. Int J Neural Syst 30(06):2050027. https://doi.org/10.1142/S0129065720500276

    Article  Google Scholar 

  17. Kim J, Kwon D, Woo SY, Kang WM, Lee S, Oh S, Kim CH, Bae JH, Park BG, Lee JH (2021) Hardware-based spiking neural network architecture using simplified backpropagation algorithm and homeostasis functionality. Neurocomputing 428:153–165. https://doi.org/10.1016/j.neucom.2020.11.016

    Article  Google Scholar 

  18. Thiele JC, Bichler O, Dupret A (2018) Event-based, timescale invariant unsupervised online deep learning with STDP. Front Comput Neurosci 12:46. https://doi.org/10.3389/fncom.2018.00046

    Article  Google Scholar 

  19. Schuman CD, Mitchell PJ, Patton RM, Potok TE, Plank JS (2020) Evolutionary optimization for neuromorphic systems. In: NICE Workshop, pp 1–9. https://doi.org/10.1145/3381755.3381758

  20. Schuman CD, Young SR, Maldonado BP, Kaul BC (2021) Real-time evolution and deployment of neuromorphic computing at the edge. In: IGSC. IEEE, Piscataway, pp 1–8. https://doi.org/10.1109/IGSC54211.2021.9651607

  21. Schuman C, Patton R, Kulkarni S, Parsa M, Stahl C, Haas NQ, Mitchell JP, Snyder S, Nagle A, Shanafield A et al. (2022) Evolutionary vs imitation learning for neuromorphic control at the edge. Neuromorph Comput Eng 2(1):014002. https://doi.org/10.1088/2634-4386/ac45e7

    Article  Google Scholar 

  22. Fra V, Forno E, Pignari R, Stewart T, Macii E, Urgese G (2022) Human activity recognition: suitability of a neuromorphic approach for on-edge IoT applications. Neuromorph Comput Eng. https://doi.org/10.1088/2634-4386/ac4c38

  23. Frenkel C, Lefebvre M, Legat JD, Bol D (2018) A 0.086-mm2 12.7-pJ/SOP 64k-synapse 256-neuron online-learning digital spiking neuromorphic processor in 28-nm CMOS. IEEE Trans Biomed Circuits Syst 13(1):145–158. https://doi.org/10.1109/TBCAS.2018.2880425

    Google Scholar 

  24. Park J, Lee J, Jeon D (2019) A 65-nm neuromorphic image classification processor with energy-efficient training through direct spike-only feedback. IEEE J Solid-State Circuits 55(1):108–119. https://doi.org/10.1109/JSSC.2019.2942367

    Article  Google Scholar 

  25. Wang D, Chundi PK, Kim SJ, Yang M, Cerqueira JP, Kang J, Jung S, Kim S, Seok M (2020) Always-on, sub-300-nw, event-driven spiking neural network based on spike-driven clock-generation and clock-and power-gating for an ultra-low-power intelligent device. In: A-SSCC. IEEE, Piscataway, pp 1–4

    Google Scholar 

  26. Izhikevich EM (2003) Simple model of spiking neurons. IEEE Trans Neural Netw 14(6):1569–1572. https://doi.org/10.1109/TNN.2003.820440

    Article  MathSciNet  Google Scholar 

  27. Brader JM, Senn W, Fusi S (2007) Learning real-world stimuli in a neural network with spike-driven synaptic dynamics. Neural Comput 19:288–2912. https://doi.org/10.1162/neco.2007.19.11.2881

    Article  MathSciNet  MATH  Google Scholar 

  28. Guerguiev J, Lillicrap TP, Richards BA (2017) Towards deep learning with segregated dendrites. ELife 6:e22901. https://doi.org/10.7554/eLife.22901.001

    Article  Google Scholar 

  29. Stimberg M, Brette R, Goodman DF (2019) Brian 2, an intuitive and efficient neural simulator. Elife 8:e47314. https://doi.org/10.7554/eLife.47314

    Article  Google Scholar 

  30. Knight JC, Nowotny T (2021) Larger GPU-accelerated brain simulations with procedural connectivity. Nat Comput Sci 1(2):136–142. https://doi.org/10.1038/s43588-020-00022-7

    Article  Google Scholar 

  31. Akopyan F, Sawada J, Cassidy A, Alvarez-Icaza R, Arthur J, Merolla P, Imam N, Nakamura Y, Datta P, Nam GJ et al. (2015) TrueNorth: Design and tool flow of a 65 mW 1 million neuron programmable neurosynaptic chip. IEEE TCAD 34(10):1537–1557. https://doi.org/10.1109/TCAD.2015.2474396

    Google Scholar 

  32. Rast A, ** X, Khan M, Furber S (2008) The deferred event model for hardware-oriented spiking neural networks. In: ICONIP. Springer, Berlin, pp 1057–1064. https://doi.org/10.1007/978-3-642-03040-6_128

    Google Scholar 

  33. Cheung K, Schultz SR, Luk W (2016) NeuroFlow: a general purpose spiking neural network simulation platform using customizable processors. Front Neurosci 9:516. https://doi.org/10.3389/fnins.2015.00516

    Article  Google Scholar 

  34. Zheng N, Mazumder P (2018) A low-power hardware architecture for on-line supervised learning in multi-layer spiking neural networks. In: ISCAS. IEEE, Piscataway, pp 1–5. https://doi.org/10.1109/ISCAS.2018.8351516

  35. Krithivasan S, Sen S, Venkataramani S, Raghunathan A (2019) Dynamic spike bundling for energy-efficient spiking neural networks. In: ISLPED. IEEE, Piscataway, pp 1–6. https://doi.org/10.1109/ISLPED.2019.8824897

    Google Scholar 

  36. Fang H, Shrestha A, Zhao Z, Li Y, Qiu Q (2019) An event-driven neuromorphic system with biologically plausible temporal dynamics. In: ICCAD. IEEE, Piscataway, pp 1–8. https://doi.org/10.1109/ICCAD45719.2019.8942083

  37. Stewart K, Orchard G, Shrestha SB, Neftci E (2020) Online few-shot gesture learning on a neuromorphic processor. IEEE J Emerg Sel Top Circuits Syst 10(4):512–521. https://doi.org/10.1109/JETCAS.2020.3032058

    Article  Google Scholar 

  38. Nguyen TNN, Veeravalli B, Fong X (2022) An FPGA-based co-processor for spiking neural networks with on-chip stdp-based learning. In: ISCAS. IEEE, Piscataway

    Google Scholar 

  39. Boahen KA (1998) Communicating neuronal ensembles between neuromorphic chips. In: Neuromorph. Syst. Eng. Springer, Berlin, pp 229–259. https://doi.org/10.1007/978-0-585-28001-1_11

    Chapter  Google Scholar 

  40. James MD (2020) Address-event representation for spiking neural networks. https://jamesmccaffrey.wordpress.com/2020/01/03/address-event-representation-for-spiking-neural-networks/. Accessed 7 Apr 2022

  41. Sebastian A, Le Gallo M, Khaddam-Aljameh R, Eleftheriou E (2020) Memory devices and applications for in-memory computing. Nat Nanotechnol 15(7):529–544. https://doi.org/10.1037/s41565-020-0655-z

    Article  Google Scholar 

  42. Davies M, Srinivasa N, Lin TH, Chinya G, Cao Y, Choday SH, Dimou G, Joshi P, Imam N, Jain S, et al. (2018) Loihi: A neuromorphic manycore processor with on-chip learning. Micro 38(1):82–99. https://doi.org/10.1109/MM.2018.112130359

    Google Scholar 

  43. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. ar**v preprint ar**v:170404861. https://doi.org/10.48550/ar**v.1704.04861

  44. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: CVPR, pp 6848–6856. https://doi.org/10.1109/CVPR.2018.00716

  45. Boroumand A, Ghose S, Akin B, Narayanaswami R, Oliveira GF, Ma X, Shiu E, Mutlu O (2021) Google neural network models for edge devices: analyzing and mitigating machine learning inference bottlenecks. In: PACT. IEEE, Piscataway, pp 159–172. https://doi.org/10.1109/PACT52795.2021.00019

  46. Miriyala VPK, Ishii M (2020) Ultra-low power on-chip learning of speech commands with phase-change memories. ar**v preprint ar**v:201011741. https://doi.org/10.48550/ar**v.2010.11741

  47. She X, Long Y, Mukhopadhyay S (2019) Improving robustness of reram-based spiking neural network accelerator with stochastic spike-timing-dependent-plasticity. In: IJCNN. IEEE, Piscataway, pp 1–8. https://doi.org/10.1109/IJCNN.2019.8851825

  48. Nguyen VT, Trinh QK, Zhang R, Nakashima Y (2021) STT-BSNN: an in-memory deep binary spiking neural network based on STT-MRAM. IEEE Access 9:151373–151385. https://doi.org/10.1109/ACCESS.2021.3125685

    Article  Google Scholar 

  49. Das D, Cen Y, Wang J, Fong X (2022) Bilayer-Skyrmion based design of neuron and synapse for spiking neural network. ar**v preprint ar**v:220302171. https://doi.org/10.48550/ar**v.2203.02171

  50. Hill AJ, Donaldson JW, Rothganger FH, Vineyard CM, Follett DR, Follett PL, Smith MR, Verzi SJ, Severa W, Wang F, et al. (2017) A spike-timing neuromorphic architecture. In: ICRC. IEEE, Piscataway, pp 1–8. https://doi.org/10.1109/ICRC.2017.8123631

  51. Mitchell JP, Schuman CD, Patton RM, Potok TE (2020) Caspian: a neuromorphic development platform. In: NICE Workshop, pp 1–6. https://doi.org/10.1145/3381755.3381764

  52. Eliasmith C, Anderson CH (2003) Neural engineering: computation, representation, and dynamics in neurobiological systems. MIT Press, Cambridge, MA. https://doi.org/10.1086/425829

  53. Bekolay T, Kolbeck C, Eliasmith C (2013) Simultaneous unsupervised and supervised learning of cognitive functions in biologically plausible spiking neural networks. In: CogSci, vol 35. https://cogsci.mindmodeling.org/2013/papers/0058/paper0058.pdf

  54. Shrestha SB, Orchard G (2018) Slayer: spike layer error reassignment in time. In: NIPS 31. https://doi.org/ar**v:1810.08646

  55. Plank JS, Schuman CD, Bruer G, Dean ME, Rose GS (2018) The TENNLab exploratory neuromorphic computing framework. IEEE Lett Comput Soc 1(2):17–20. https://doi.org/10.1109/LOCS.2018.2885976

    Article  Google Scholar 

  56. Mitchell JP, Schuman C (2021) Low power hardware-in-the-loop neuromorphic training accelerator. In: ICONS, pp 1–4. https://doi.org/10.1145/3477145.3477150

  57. Bekolay T, Bergstra J, Hunsberger E, DeWolf T, Stewart TC, Rasmussen D, Choo X, Voelker A, Eliasmith C (2014) Nengo: a python tool for building large-scale functional brain models. Front Neuroinform 7:48. https://doi.org/10.3389/fninf.2013.00048

    Article  Google Scholar 

  58. Blackstock M, Lea R (2014) Toward a distributed data flow platform for the web of things (distributed NODE-RED). In: WoT, pp 34–39. https://doi.org/10.1145/2684432.2684439

  59. Rothganger F, Warrender CE, Trumbo D, Aimone JB (2014) N2a: a computational tool for modeling from neurons to algorithms. Front Neural Circuits 8:1. https://doi.org/10.3389/fncir.2014.00001

    Article  Google Scholar 

  60. Kim S, Jeong J, Kim J, Yun YS, Kim B, Jung J (2020) Develo** IoT applications using spiking neural networks framework. In: RACS, pp 196–200. https://doi.org/10.1145/3400286.3418271

  61. DeWolf T, Jaworski P, Eliasmith C (2020) Nengo and low-power ai hardware for robust, embedded neurorobotics. Front Neurorobot. https://doi.org/10.3389/fnbot.2020.568359

  62. Morcos B (2019) NengoFPGA: an FPGA backend for the nengo neural simulator. Master’s Thesis, University of Waterloo

    Google Scholar 

  63. Rathi N, Srinivasan G, Panda P, Roy K (2020) Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In: ICLR. https://doi.org/2005.01807

  64. Sen S, Venkataramani S, Raghunathan A (2017) Approximate computing for spiking neural networks. In: DATE. IEEE, Piscataway, pp 193–198. https://doi.org/10.23919/DATE.2017.7926981

  65. Shi Y, Nguyen L, Oh S, Liu X, Kuzum D (2019) A soft-pruning method applied during training of spiking neural networks for in-memory computing applications. Front Neurosci 13:405. https://doi.org/10.3389/fnins.2019.00405

    Article  Google Scholar 

  66. Nguyen TNN, Veeravalli B, Fong X (2021) Connection pruning for deep spiking neural networks with on-chip learning. In: ICONS, pp 1–8. https://doi.org/10.1145/3477145.3477157

  67. Deng L, Wu Y, Hu Y, Liang L, Li G, Hu X, Ding Y, Li P, **e Y (2021) Comprehensive SNN compression using ADMM optimization and activity regularization. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3109064

  68. Tang H, Kim H, Kim H, Park J (2019) Spike counts based low complexity SNN architecture with binary synapse. IEEE Trans Biomed Circuits Syst 13(6):1664–1677. https://doi.org/10.1109/TBCAS.2019.2945406

    Article  Google Scholar 

  69. Koo M, Srinivasan G, Shim Y, Roy K (2020) SBSNN: Stochastic-bits enabled binary spiking neural network with on-chip learning for energy efficient neuromorphic computing at the edge. IEEE Trans Circuits Syst I Regul Pap 67(8):2546–2555. https://doi.org/10.1109/TCSI.2020.2979826

    Article  Google Scholar 

  70. Chuang PY, Tan PY, Wu CW, Lu JM (2020) A 90 nm 103.14 TOPS/W binary-weight spiking neural network CMOS ASIC for real-time object classification. In: DAC, IEEE, Piscataway, pp 1–6. https://doi.org/10.1109/DAC18072.2020.9218714

  71. Wang Y, Xu Y, Yan R, Tang H (2020) Deep spiking neural networks with binary weights for object recognition. IEEE Trans Cogn and Develop Syst 13(3):514–523. https://doi.org/10.1109/TCDS.2020.2971655

    Article  Google Scholar 

  72. Kheradpisheh SR, Mirsadeghi M, Masquelier T (2021) BS4NN: binarized spiking neural networks with temporal coding and learning. Neural Process Lett. https://doi.org/10.1007/s11063-021-10680-x

  73. Lu S, Sengupta A (2020) Exploring the connection between binary and spiking neural networks. Front Neurosci 14:535. https://doi.org/10.3389/fnins.2020.00535

    Article  Google Scholar 

  74. Chen Y, Yu Z, Fang W, Huang T, Tian Y (2021) Pruning of deep spiking neural networks through gradient rewiring. In: Zhou ZH (ed) International joint conferences on artificial intelligence organization, IJCAI, pp 1713–1721. https://doi.org/10.24963/ijcai.2021/236

  75. Furuya K, Ohkubo J (2021) Semi-supervised learning combining backpropagation and STDP: STDP enhances learning by backpropagation with a small amount of labeled data in a spiking neural network. J Phys Soc Jpn 90(7):074802. https://doi.org/10.7566/JPSJ.90.074802

    Article  Google Scholar 

  76. Krizhevsky A, Hinton G et al. (2009) Learning multiple layers of features from tiny images. Master’s Thesis, Department of Computer Science, University of Toronto

    Google Scholar 

  77. Deng L (2012) The MNIST database of handwritten digit images for machine learning research. IEEE Signal Process Mag 29(6):141–142. https://doi.org/10.1109/MSP.2012.2211477

    Article  Google Scholar 

  78. Deng L, Wu Y, Hu X, Liang L, Ding Y, Li G, Zhao G, Li P, **e Y (2020) Rethinking the performance comparison between SNNs and ANNs. Neural Netw 121:294–307. https://doi.org/10.1016/j.neunet.2019.09.005

    Article  Google Scholar 

  79. He W, Wu Y, Deng L, Li G, Wang H, Tian Y, Ding W, Wang W, **e Y (2020) Comparing SNNs and RNNs on neuromorphic vision datasets: similarities and differences. Neural Netw 132:108–120. https://doi.org/10.1016/j.neunet.2020.08.001

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xuanyao Fong .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Nguyen, T.N.N., Veeravalli, B., Fong, X. (2023). Hardware Implementation for Spiking Neural Networks on Edge Devices. In: Thakkar, H.K., Dehury, C.K., Sahoo, P.K., Veeravalli, B. (eds) Predictive Analytics in Cloud, Fog, and Edge Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-18034-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18034-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18033-0

  • Online ISBN: 978-3-031-18034-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Navigation