SpringerLink
Log in

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13672))

Included in the following conference series:

Abstract

Spiking Neural Networks (SNNs) have recently emerged as a new generation of low-power deep neural networks, which is suitable to be implemented on low-power mobile/edge devices. As such devices have limited memory storage, neural pruning on SNNs has been widely explored in recent years. Most existing SNN pruning works focus on shallow SNNs (2–6 layers), however, deeper SNNs (\(\ge \)16 layers) are proposed by state-of-the-art SNN works, which is difficult to be compatible with the current SNN pruning work. To scale up a pruning technique towards deep SNNs, we investigate Lottery Ticket Hypothesis (LTH) which states that dense networks contain smaller subnetworks (i.e., winning tickets) that achieve comparable performance to the dense networks. Our studies on LTH reveal that the winning tickets consistently exist in deep SNNs across various datasets and architectures, providing up to \(97\%\) sparsity without huge performance degradation. However, the iterative searching process of LTH brings a huge training computational cost when combined with the multiple timesteps of SNNs. To alleviate such heavy searching cost, we propose Early-Time (ET) ticket where we find the important weight connectivity from a smaller number of timesteps. The proposed ET ticket can be seamlessly combined with a common pruning techniques for finding winning tickets, such as Iterative Magnitude Pruning (IMP) and Early-Bird (EB) tickets. Our experiment results show that the proposed ET ticket reduces search time by up to \(38\%\) compared to IMP or EB methods. Code is available at Github.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 93.08
Price includes VAT (France)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 116.04
Price includes VAT (France)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/Yanqi-Chen/Gradient-Rewiring

References

  1. Akopyan, F., et al.: TrueNorth: design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 34(10), 1537–1557 (2015)

    Article  Google Scholar 

  2. Bai, Y., Wang, H., Tao, Z., Li, K., Fu, Y.: Dual lottery ticket hypothesis. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=fOsN52jn25l

  3. Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., Maass, W.: Long short-term memory and learning-to-learn in networks of spiking neurons. In: Advances in Neural Information Processing Systems 31 (2018)

    Google Scholar 

  4. Brix, C., Bahar, P., Ney, H.: Successfully applying the stabilized lottery ticket hypothesis to the transformer architecture. ar**v preprint ar**v:2005.03454 (2020)

  5. Burkholz, R., Laha, N., Mukherjee, R., Gotovos, A.: On the existence of universal lottery tickets. ar**v preprint ar**v:2111.11146 (2021)

  6. Chen, G.K., Kumar, R., Sumbul, H.E., Knag, P.C., Krishnamurthy, R.K.: A 4096-neuron 1M-synapse 3.8-pJ/SOP spiking neural network with on-chip STDP learning and sparse weights in 10-nm finfet CMOS. IEEE J. Solid-State Circuits 54(4), 992–1002 (2018)

    Google Scholar 

  7. Chen, T., et al.: The lottery ticket hypothesis for pre-trained BERT networks. In: Advances in Neural Information Processing Systems, vol. 33, pp. 15834–15846 (2020)

    Google Scholar 

  8. Chen, T., et al.: Sparsity winning twice: better robust generalization from more efficient training. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=SYuJXrXq8tw

  9. Chen, Y., Yu, Z., Fang, W., Huang, T., Tian, Y.: Pruning of deep spiking neural networks through gradient rewiring. ar**v preprint ar**v:2105.04916 (2021)

  10. Chen, Y.H., Emer, J., Sze, V.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH Comput. Archit. News 44(3), 367–379 (2016)

    Article  Google Scholar 

  11. Christensen, D.V., et al.: 2022 roadmap on neuromorphic computing and engineering. Neuromorphic Computing and Engineering (2022)

    Google Scholar 

  12. Comsa, I.M., Fischbacher, T., Potempa, K., Gesmundo, A., Versari, L., Alakuijala, J.: Temporal coding in spiking neural networks with alpha synaptic function. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8529–8533. IEEE (2020)

    Google Scholar 

  13. Davies, M., et al.: Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38(1), 82–99 (2018)

    Article  Google Scholar 

  14. Deng, L., et al.: Comprehensive SNN compression using ADMM optimization and activity regularization. IEEE Trans. Neural Networks and Learn. Syst. (2021)

    Google Scholar 

  15. Deng, S., Li, Y., Zhang, S., Gu, S.: Temporal efficient training of spiking neural network via gradient re-weighting. In: International Conference on Learning Representations (2022., https://openreview.net/forum?id=_XNtisL32jv

  16. Desai, S., Zhan, H., Aly, A.: Evaluating lottery tickets under distributional shifts. ar**v preprint ar**v:1910.12708 (2019)

  17. Ding, J., Yu, Z., Tian, Y., Huang, T.: Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. ar**v preprint ar**v:2105.11654 (2021)

  18. Ding, S., Chen, T., Wang, Z.: Audio lottery: speech recognition made ultra-lightweight, noise-robust, and transferable. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=9Nk6AJkVYB

  19. Fang, W., et al.: Spikingjelly (2020). https://github.com/fangwei123456/spikingjelly

  20. Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., Tian, Y.: Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems 34 (2021)

    Google Scholar 

  21. Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. ar**v preprint ar**v:1803.03635 (2018)

  22. Frankle, J., Dziugaite, G.K., Roy, D.M., Carbin, M.: Stabilizing the lottery ticket hypothesis. ar**v preprint ar**v:1903.01611 (2019)

  23. Furber, S.B., Galluppi, F., Temple, S., Plana, L.A.: The spinnaker project. Proc. IEEE 102(5), 652–665 (2014)

    Article  Google Scholar 

  24. Girish, S., Maiya, S.R., Gupta, K., Chen, H., Davis, L.S., Shrivastava, A.: The lottery ticket hypothesis for object recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 762–771 (2021)

    Google Scholar 

  25. Guo, W., Fouda, M.E., Yantir, H.E., Eltawil, A.M., Salama, K.N.: Unsupervised adaptive weight pruning for energy-efficient neuromorphic systems. Frontiers in Neuroscience p. 1189 (2020)

    Google Scholar 

  26. Han, S., et al.: Dsd: Dense-sparse-dense training for deep neural networks. ar**v preprint ar**v:1607.04381 (2016)

  27. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. Advances in neural information processing systems 28 (2015)

    Google Scholar 

  28. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  29. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. ar**v preprint ar**v:1502.03167 (2015)

  30. Izhikevich, E.M.: Simple model of spiking neurons. IEEE Trans. Neural Networks 14(6), 1569–1572 (2003)

    Article  MathSciNet  Google Scholar 

  31. Kalibhat, N.M., Balaji, Y., Feizi, S.: Winning lottery tickets in deep generative models. ar**v preprint ar**v:2010.02350 (2020)

  32. Kim, Y., Li, Y., Park, H., Venkatesha, Y., Panda, P.: Neural architecture search for spiking neural networks. ar**v preprint ar**v:2201.10355 (2022)

  33. Kim, Y., Panda, P.: Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, p. 1638 (2020)

    Google Scholar 

  34. Kim, Y., Panda, P.: Visual explanations from spiking neural networks using interspike intervals. Sci. Rep. 11, 19037 (2021). https://doi.org/10.1038/s41598-021-98448-0 (2021)

  35. Kim, Y., Venkatesha, Y., Panda, P.: Privatesnn: privacy-preserving spiking neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 1192–1200 (2022)

    Google Scholar 

  36. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

  37. Kundu, S., Pedram, M., Beerel, P.A.: Hire-snn: harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5209–5218 (2021)

    Google Scholar 

  38. Ledinauskas, E., Ruseckas, J., Juršėnas, A., Buračas, G.: Training deep spiking neural networks. ar**v preprint ar**v:2006.04436 (2020)

  39. Lee, C., Sarwar, S.S., Panda, P., Srinivasan, G., Roy, K.: Enabling spike-based backpropagation for training deep neural network architectures. Front. Neurosci. 14 (2020)

    Google Scholar 

  40. Lee, J.H., Delbruck, T., Pfeiffer, M.: Training deep spiking neural networks using backpropagation. Front. Neurosci. 10, 508 (2016)

    Article  Google Scholar 

  41. Lee, N., Ajanthan, T., Torr, P.H.: Snip: Single-shot network pruning based on connection sensitivity. ar**v preprint ar**v:1810.02340 (2018)

  42. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. ar**v preprint ar**v:1608.08710 (2016)

  43. Li, Y., Deng, S., Dong, X., Gong, R., Gu, S.: A free lunch from ann: towards efficient, accurate spiking neural networks calibration. ar**v preprint ar**v:2106.06984 (2021)

  44. Li, Y., Deng, S., Dong, X., Gu, S.: Converting artificial neural networks to spiking neural networks via parameter calibration. ar**v preprint ar**v:2205.10121 (2022)

  45. Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., Gu, S.: Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems 34 (2021)

    Google Scholar 

  46. Liu, S., et al.: Deep ensembling with no overhead for either training or testing: the all-round blessings of dynamic sparsity. ar**v preprint ar**v:2106.14568 (2021)

  47. Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. ar**v preprint ar**v:1810.05270 (2018)

  48. Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. ar**v preprint ar**v:1608.03983 (2016)

  49. Martinelli, F., Dellaferrera, G., Mainar, P., Cernak, M.: Spiking neural networks trained with backpropagation for low power neuromorphic implementation of voice activity detection. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8544–8548. IEEE (2020)

    Google Scholar 

  50. Mehta, R.: Sparse transfer learning via winning lottery tickets. ar**v preprint ar**v:1905.07785 (2019)

  51. Morcos, A., Yu, H., Paganini, M., Tian, Y.: One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In: Advances in Neural Information Processing Systems 32 (2019)

    Google Scholar 

  52. Mostafa, H.: Supervised learning based on temporal coding in spiking neural networks. IEEE Trans. Neural Networks Learn. Syst. 29(7), 3227–3235 (2017)

    Google Scholar 

  53. Movva, R., Zhao, J.Y.: Dissecting lottery ticket transformers: structural and behavioral study of sparse neural machine translation. ar**v preprint ar**v:2009.13270 (2020)

  54. Neftci, E.O., Mostafa, H., Zenke, F.: Surrogate gradient learning in spiking neural networks. IEEE Signal Process. Mag. 36, 61–63 (2019)

    Article  Google Scholar 

  55. Neftci, E.O., Pedroni, B.U., Joshi, S., Al-Shedivat, M., Cauwenberghs, G.: Stochastic synapses enable efficient brain-inspired learning machines. Front. Neurosci. 10, 241 (2016)

    Article  Google Scholar 

  56. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011)

    Google Scholar 

  57. Orchard, G., et al.: Efficient neuromorphic signal processing with loihi 2. In: 2021 IEEE Workshop on Signal Processing Systems (SiPS), pp. 254–259. IEEE (2021)

    Google Scholar 

  58. Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S.W., Dally, W.J.: Scnn: an accelerator for compressed-sparse convolutional neural networks. ACM SIGARCH Computer Architecture News 45(2), 27–40 (2017)

    Article  Google Scholar 

  59. Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS-W (2017)

    Google Scholar 

  60. Rathi, N., Panda, P., Roy, K.: Stdp-based pruning of connections and weight quantization in spiking neural networks for energy-efficient recognition. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 38(4), 668–677 (2018)

    Article  Google Scholar 

  61. Rathi, N., Roy, K.: Diet-SNN: a low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Trans. Neural Networks Learn Syst. (2021)

    Google Scholar 

  62. Rathi, N., Srinivasan, G., Panda, P., Roy, K.: Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. ar**v preprint ar**v:2005.01807 (2020)

  63. Roy, K., Jaiswal, A., Panda, P.: Towards spike-based machine intelligence with neuromorphic computing. Nature 575(7784), 607–617 (2019)

    Article  Google Scholar 

  64. Schuman, C.D., Kulkarni, S.R., Parsa, M., Mitchell, J.P., Kay, B., et al.: Opportunities for neuromorphic computing algorithms and applications. Nature Comput. Sci. 2(1), 10–19 (2022)

    Article  Google Scholar 

  65. Shi, Y., Nguyen, L., Oh, S., Liu, X., Kuzum, D.: A soft-pruning method applied during training of spiking neural networks for in-memory computing applications. Front. Neurosci. 13, 405 (2019)

    Article  Google Scholar 

  66. Shrestha, S.B., Orchard, G.: Slayer: Spike layer error reassignment in time. ar**v preprint ar**v:1810.08646 (2018)

  67. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ICLR (2015)

    Google Scholar 

  68. Venkatesha, Y., Kim, Y., Tassiulas, L., Panda, P.: Federated learning with spiking neural networks. ar**v preprint ar**v:2106.06579 (2021)

  69. Vischer, M.A., Lange, R.T., Sprekeler, H.: On lottery tickets and minimal task representations in deep reinforcement learning. ar**v preprint ar**v:2105.01648 (2021)

  70. Wang, C., Zhang, G., Grosse, R.: Picking winning tickets before training by preserving gradient flow. ar**v preprint ar**v:2002.07376 (2020)

  71. Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems 29 (2016)

    Google Scholar 

  72. Wu, H., et al.: Training spiking neural networks with accumulated spiking flow. ijo 1(1) (2021)

    Google Scholar 

  73. Wu, J., Xu, C., Zhou, D., Li, H., Tan, K.C.: Progressive tandem learning for pattern recognition with deep spiking neural networks. ar**v preprint ar**v:2007.01204 (2020)

  74. Wu, Y., Deng, L., Li, G., Zhu, J., Shi, L.: Spatio-temporal backpropagation for training high-performance spiking neural networks. Front. Neurosci. 12, 331 (2018)

    Article  Google Scholar 

  75. Wu, Y., Deng, L., Li, G., Zhu, J., **e, Y., Shi, L.: Direct training for spiking neural networks: Faster, larger, better. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1311–1318 (2019)

    Google Scholar 

  76. Wu, Y., Zhao, R., Zhu, J., Chen, F., Xu, M., Li, G., Song, S., Deng, L., Wang, G., Zheng, H., et al.: Brain-inspired global-local learning incorporated with neuromorphic computing. Nat. Commun. 13(1), 1–14 (2022)

    Google Scholar 

  77. **ao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ar**v preprint ar**v:1708.07747 (2017)

  78. Yao, M., Gao, H., Zhao, G., Wang, D., Lin, Y., Yang, Z., Li, G.: Temporal-wise attention spiking neural networks for event streams classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10221–10230 (2021)

    Google Scholar 

  79. Yin, R., Moitra, A., Bhattacharjee, A., Kim, Y., Panda, P.: Sata: Sparsity-aware training accelerator for spiking neural networks. ar**v preprint ar**v:2204.05422 (2022)

  80. You, H., et al.: Drawing early-bird tickets: Towards more efficient training of deep networks. ar**v preprint ar**v:1909.11957 (2019)

  81. Yu, H., Edunov, S., Tian, Y., Morcos, A.S.: Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. ar**v preprint ar**v:1906.02768 (2019)

  82. Zhang, Z., Chen, X., Chen, T., Wang, Z.: Efficient lottery ticket finding: less data is more. In: International Conference on Machine Learning, pp. 12380–12390. PMLR (2021)

    Google Scholar 

  83. Zheng, H., Wu, Y., Deng, L., Hu, Y., Li, G.: Going deeper with directly-trained larger spiking neural networks. ar**v preprint ar**v:2011.05280 (2020)

  84. Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: Zeros, signs, and the supermask. In: Advances in Neural Information Processing Systems 32 (2019)

    Google Scholar 

Download references

Acknowledgment

We would like to thank Anna Hambitzer for her helpful comments. This work was supported in part by C-BRIC, a JUMP center sponsored by DARPA and SRC, Google Research Scholar Award, the National Science Foundation (Grant#1947826), TII (Abu Dhabi) and the DARPA AI Exploration (AIE) program.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Yale University, New Haven, CT, USA

    Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, Ruokai Yin & Priyadarshini Panda

Authors
  1. Youngeun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuhang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hyoungseob Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yeshwanth Venkatesha
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruokai Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Priyadarshini Panda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Youngeun Kim .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 286 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kim, Y., Li, Y., Park, H., Venkatesha, Y., Yin, R., Panda, P. (2022). Exploring Lottery Ticket Hypothesis in Spiking Neural Networks. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13672. Springer, Cham. https://doi.org/10.1007/978-3-031-19775-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19775-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19774-1

  • Online ISBN: 978-3-031-19775-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation