SpringerLink
Log in

Multi-level brain-guided fusion to reinforce spiking neural network in image classification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Spiking Neural Networks (SNNs) can be a suitable alternative to deep artificial neural networks due to efficient computation and low-power event-driven information analysis. Nevertheless, SNN cannot compete with Artificial Neural Network (ANN) in terms of performance due to the challenges in the training process. In this work, an attempt has been made to improve the performance of SNN in the image classification task without changing the training method. To this end, SNN is used to encode image in a brain-guided model. Batch normalization through time layers are used in SNN architecture to train from scratch with low latency. In this work, EEG-evoked signal information, provided by an LSTM-based network, is used to construct a bio-visual path. A pooling-based spatial coding is proposed to down-sample the visual feature maps. Manifold transferring is applied to project each visual feature into the corresponding implicit EEG-based feature. The image-based reduced feature and its EEG-based feature, detected by manifold transferring, are fused to prepare the input for an SVM classifier. In the inference phase, without the need for EEG acquisition, the output of the manifold transfer block, provided in the training process, is combined with reduced visual features to classify the input image. The experimental results show that the multi-level fusion strategy in the brain-guided network improves the classification accuracy of SNN in 3-class and 5-class image classification problems by up to 4% and 15%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Data will be made available on request.

Notes

  1. Highest Common Factor.

References

  1. Zheng X, Chen W, You Y, Jiang Y, Li M, Zhang T (2020) Ensemble deep learning for automated visual classification using EEG signals. Pattern Recogn 102:107147

  2. Kaneshiro B, Guimaraes MP, Kim H-S, Norcia AM, Suppes P (2015) A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. PLoS ONE 10(8):e0135697

    Article  Google Scholar 

  3. Fong RC, Scheirer WJ, Cox DD (2018) Using human brain activity to guide machine learning. Sci Rep 8(1):1–10

    Article  Google Scholar 

  4. Spampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M (2017) Deep learning human mind for automated visual classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp 4503–4511

  5. Fares A, Zhong S, Jiang J (2019) EEG-based image classification via a region-level stacked bi-directional deep learning framework. BMC Med Inform Decis Mak 19(6):268

    Article  Google Scholar 

  6. Thara DK, PremaSudha BG, **ong F (2019) Auto-detection of epileptic seizure events using deep neural network with different feature scaling techniques. Pattern Recognit Lett 128:544–550

    Article  Google Scholar 

  7. Maitin AM, Romero Muñoz JP, García-Tejedor ÁJ (2022) Survey of machine learning techniques in the analysis of EEG signals for Parkinson’s disease: a systematic review. Appl Sci 12(14):6967

    Article  Google Scholar 

  8. Taheri S, Ezoji M, Sakhaei SM (2020) Convolutional neural network based features for motor imagery EEG signals classification in brain–computer interface system. SN Appl Sci 2(4):1–12

  9. Gu X et al (2021) EEG-based Brain-Computer Interfaces (BCIs): a survey of recent studies on signal sensing technologies and computational intelligence approaches and their applications. IEEE/ACM Trans Comput Biol Bioinf 18(5):1645–1666

    Article  Google Scholar 

  10. Song T, Zheng W, Song P, Cui Z (2018) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541

    Article  Google Scholar 

  11. Jebelli H, Khalili MM, Lee S (2019) Mobile EEG-based workers’ stress recognition by applying deep neural network. In: Mutis I, Hartmann T, (eds) Advances in informatics and computing in civil and construction engineering. Springer, Cham, pp 173–180

  12. Zhong S, Fares A, Jiang J (2019) An attentional-LSTM for improved classification of brain activities evoked by images. In: Proceedings of the 27th ACM international conference on multimedia (MM'19), Nice, France, pp 1295–1303

  13. Neftci EO, Mostafa H, Zenke F (2019) Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Process Mag 36(6):51–63

    Article  Google Scholar 

  14. Rathi N, Srinivasan G, Panda P, Roy K (2020) Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. ar**v Prepr. ar**v2005.01807

  15. Wu Y, Deng L, Li G, Zhu J, Shi L (2018) Spatio-temporal backpropagation for training high-performance spiking neural networks. Front Neurosci 12:331

    Article  Google Scholar 

  16. Mozafari M, Kheradpisheh SR, Masquelier T, Nowzari-Dalini A, Ganjtabesh M (2018) First-spike-based visual categorization using reward-modulated STDP. IEEE Trans Neural Netw Learn Syst 29(12):6178–6190

    Article  Google Scholar 

  17. Mozafari M, Ganjtabesh M, Nowzari-Dalini A, Thorpe SJ, Masquelier T (2019) Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks. Pattern Recognit 94:87–95

    Article  Google Scholar 

  18. Sengupta A, Ye Y, Wang R, Liu C, Roy K (2019) Going deeper in spiking neural networks: VGG and residual architectures. Front Neurosci 13:95

  19. Fang W, Yu Z, Chen Y, Masquelier T, Huang T, Tian Y (2021) Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, pp 2641–2651

  20. Kim Y, Panda P (2021) Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Front Neurosci 15:773954

    Article  Google Scholar 

  21. Kundu S, Datta G, Pedram M, Beerel PA (2021) Spike-thrift: towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp 3953–3962

  22. Jiang J, Fares A, Zhong S-H (2019) A context-supported deep learning framework for multimodal brain imaging classification. IEEE Trans Hum-Mach Syst 49(6):611–622

    Article  Google Scholar 

  23. Palazzo S, Spampinato C, Kavasidis I, Giordano D, Schmidt J, Shah M (2020) Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Trans Pattern Anal Mach Intell 43(11):3833–3849

    Article  Google Scholar 

  24. Das Chakladar D, Kumar P, Roy PP, Dogra DP, Scheme E, Chang V (2021) A multimodal-Siamese Neural Network (mSNN) for person verification using signatures and EEG. Inf Fusion 71:17–27

    Article  Google Scholar 

  25. Kavasidis I, Palazzo S, Spampinato C, Giordano D, Shah M (2017) Brain2image: converting brain signals into images. In: Proceedings of the 25th ACM international conference on multimedia (MM'17), New York, NY, United States, pp 1809–1817

  26. Cudlenco N, Popescu N, Leordeanu M (2020) Reading into the mind’s eye: boosting automatic visual recognition with EEG signals. Neurocomputing 386:281–292

    Article  Google Scholar 

  27. Imani Z, Ezoji M, Masquelier T (2023) Brain-guided manifold transferring to improve the performance of spiking neural networks in image classification. J Comput Neurosci 51(4):475–490

  28. Lobo JL, Del Ser J, Bifet A, Kasabov N (2020) Spiking neural networks and online learning: an overview and perspectives. Neural Netw 121:88–100

    Article  Google Scholar 

  29. Wu Y, Deng L, Li G, Zhu J, **e Y, Shi L (2019) Direct training for spiking neural networks: faster, larger, better. Proc AAAI Conf Artif Intell 33:1311–1318

    Google Scholar 

  30. Rathi N, Roy K (2023) DIET-SNN: a low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Trans Neural Networks and Learn Syst 34(6):3174–3182

  31. Aoudi W, Barbar AM (2016) Support vector machines: a distance-based approach to multi-class classification. In: 2016 IEEE International Multidisciplinary Conference on Engineering Technology (IMCET), Beirut, Lebanon, pp 75–80

  32. Russakovsky O et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  33. Palazzo S, Spampinato C, Schmidt J, Kavasidis I, Giordano D, Shah M (2020) Correct block-design experiments mitigate temporal correlation bias in EEG classification,” ar**v Prepr. ar**v2012.03849

  34. Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(10):281–305

  35. Kim Y, Panda P (2021) Visual explanations from spiking neural networks using inter-spike intervals. Sci Rep 11(1):19037

    Article  Google Scholar 

  36. Saffar M, Kalhor A (2023) Evaluation of Dataflow through layers of convolutional neural networks in classification problems. Expert Syst Appl 224:119944

    Article  Google Scholar 

  37. Karimi A, Kalhor A, Ahmadian M (2023) A forward-backward learning strategy for CNNs via separation index maximizing at the first convolutional layer. In: 34th British Machine Vision Conference (BMVC), Aberdeen, UK, pp 20–24

Download references

Acknowledgements

The authors acknowledge the funding support of Babol Noshirvani University of Technology through Grant program No. BNUT/389079/ 1402-2.

Author information

Authors and Affiliations

  1. Faculty of Electrical and Computer Engineering, Babol Noshirvani University of Technology, Babol, Iran

    Zahra Imani & Mehdi Ezoji

Authors
  1. Zahra Imani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehdi Ezoji
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Zahra Imani: Conceptualization, Methodology, Software, Formal analysis, Validation, Visualization, Writing—original draft. Mehdi Ezoji: Project administration, Supervision, Conceptualization, Methodology, Investigation, Validation, Writing—review & editing.

Corresponding author

Correspondence to Mehdi Ezoji.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Imani, Z., Ezoji, M. Multi-level brain-guided fusion to reinforce spiking neural network in image classification. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19665-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19665-z

Keywords

Search

Navigation