Abstract
Spiking neural networks (SNNs), as the third generation of neural networks, are based on biological models of human brain neurons. In this work, a shallow SNN plays the role of an explicit image decoder in the image classification. An LSTM-based EEG encoder is used to construct the EEG-based feature space, which is a discriminative space in viewpoint of classification accuracy by SVM. Then, the visual feature vectors extracted from SNN is mapped to the EEG-based discriminative features space by manifold transferring based on mutual k-Nearest Neighbors (Mk-NN MT). This proposed “Brain-guided system” improves the separability of the SNN-based visual feature space. In the test phase, the spike patterns extracted by SNN from the input image is mapped to LSTM-based EEG feature space, and then classified without need for the EEG signals. The SNN-based image encoder is trained by the conversion method and the results are evaluated and compared with other training methods on the challenging small ImageNet-EEG dataset. Experimental results show that the proposed transferring the manifold of the SNN-based feature space to LSTM-based EEG feature space leads to 14.25% improvement at most in the accuracy of image classification. Thus, embedding SNN in the brain-guided system which is trained on a small set, improves its performance in image classification.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
The custom code for data analysis is available upon request from the corresponding author.
References
Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., & Maass, W. (2018). Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural Information Processing Systems, 31.
Cudlenco, N., Popescu, N., & Leordeanu, M. (2020). Reading into the mind’s eye: Boosting automatic visual recognition with EEG signals. Neurocomputing, 386, 281–292.
Deng, L., et al. (2020). Rethinking the performance comparison between SNNs and ANNs. Neural Networks, 121, 294–307.
Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., & Tian, Y. (2021). Deep Residual Learning in Spiking Neural Networks. Advances in Neural Information Processing Systems, 34, 21056–21069.
Fares, A., Zhong, S., & Jiang, J. (2019). EEG-based image classification via a region-level stacked bi-directional deep learning framework. BMC Medical Informatics and Decision Making, 19(6), 268.
Fu, Q., & Dong, H. (2021). An ensemble unsupervised spiking neural network for objective recognition. Neurocomputing, 419, 47–58.
Garg, I., Chowdhury, S. S., & Roy, K. (2021). DCT-SNN: Using DCT To Distribute Spatial Information Over Time for Low-Latency Spiking Neural Networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4671–4680.
Ghojogh, B., Karray, F., & Crowley, M. (2019). Fisher and kernel Fisher discriminant analysis: Tutorial. ar**v Prepr. ar**v1906.09436
Hajizadeh, R., Aghagolzadeh, A., & Ezoji, M. (2022). Mutual neighborhood and modified majority voting based KNN classifier for multi-categories classification. Pattern Analysis and Applications, 25(4), 773–793.
Hojjaty Saeedy, R. (2021). Biologically Inspired Computer Vision/ Applications of Computational Models of Primate Visual Systems in Computer Vision and Image Processing. University of New Hampshire.
Jebelli, H., Khalili, M. M., & Lee, S. (2019). Mobile EEG-based workers’ stress recognition by applying deep neural network. In Advances in Informatics and Computing in Civil and Construction Engineering, Springer, 173–180.
Jiang, J., Fares, A., & Zhong, S.-H. (2019). A Context-Supported Deep Learning Framework for Multimodal Brain Imaging Classification. IEEE Transactions on Human-Machine Systems, 49(6), 611–622.
Jiang, Y., Krishnan, D., Mobahi, H., & Bengio, S. (2018). Predicting the generalization gap in deep networks with margin distributions. ar**v paper. ar**v:1810.00113
Ju, R., Hu, C., & Li, Q. (2017). Early diagnosis of Alzheimer’s disease based on resting-state brain networks and deep learning. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 16(1), 244–257.
Kaneshiro, B., Guimaraes, M. P., Kim, H.-S., Norcia, A. M., & Suppes, P. (2015). A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. PLoS ONE, 10(8), e0135697.
Kavasidis, I., Palazzo, S., Spampinato, C., Giordano, D., & Shah, M. (2017). Brain2image: Converting brain signals into images. In Proceedings of the 25th ACM international conference on Multimedia, 1809–1817.
Kundu, S., Pedram, M., & Beerel, P. A. (2021). Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 5209–5218.
Lee, C., Panda, P., Srinivasan, G., & Roy, K. (2018). Training deep spiking convolutional neural networks with stdp-based unsupervised pre-training followed by supervised fine-tuning. Frontiers in Neuroscience, 12, 435.
Lee, E., Choi, J.-S., Kim, M., Suk, H.-I., & Initiative, A. D. N. (2019). Toward an interpretable Alzheimer’s disease diagnostic model with regional abnormality representation via deep learning. NeuroImage, 202, 116113.
Lotfi Rezaabad, A., & Vishwanath, S. (2020). Long short-term memory spiking networks and their applications. In International Conference on Neuromorphic Systems, 1–9.
Masquelier, T. (2017). Spike-based computing and learning in brains, machines, and visual systems in particular (HDR Report). Ph. D. Dissertation. https://doi.org/10.13140/RG.2.2.30232.49922
Neftci, E. O., Mostafa, H., & Zenke, F. (2019a). Surrogate gradient learning in spiking neural networks. IEEE Signal Processing Magazine, 36, 61–63.
Neftci, E. O., Mostafa, H., & Zenke, F. (2019b). Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6), 51–63.
Oh, S. L., et al., (2018). A deep learning approach for Parkinson’s disease diagnosis from EEG signals. Neural Computing and Applications, 1–7.
Palazzo, S., Spampinato, C., Kavasidis, I., Giordano, D., Schmidt, J., & Shah, M. (2020). Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11), 3833–3849.
Panda, P., Aketi, S. A., & Roy, K. (2020). Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14.
Rathi, N., Srinivasan, G., Panda, P., & Roy, K. (2020). Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. ar**v Prepr. ar**v2005.01807
Rueckauer, B., Lungu, I.-A., Hu, Y., Pfeiffer, M., & Liu, S.-C. (2017). Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience, 11, 682.
Russakovsky, O., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.
Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019). Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience, 13.
Shrestha, S. B., & Orchard, G. (2018). Slayer: Spike layer error reassignment in time. Advances in Neural Information Processing Systems, 31, 1412–1421.
Song, T., Zheng, W., Song, P., & Cui, Z. (2018). EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, 11(3), 532–541.
Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., & Shah, M. (2017). Deep learning human mind for automated visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6809–6817.
Srinivasan, G., Lee, C., Sengupta, A., Panda, P., Sarwar, S. S., & Roy, K. (2020). Training Deep Spiking Neural Networks for Energy-Efficient Neuromorphic Computing. In ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8549–8553.
Taheri, S., Ezoji, M., & Sakhaei, S. M. (2020). Convolutional neural network based features for motor imagery EEG signals classification in brain–computer interface system. SN Applied Science, 2(4), 1–12.
Taherkhani, A., Belatreche, A., Li, Y., Cosma, G., Maguire, L. P., & McGinnity, T. M. (2020). A review of learning in biologically plausible spiking neural networks. Neural Networks, 122, 253–272.
Wu, J., et al. (2021). Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 7824–7840.
Wu, Y., Deng, L., Li, G., Zhu, J., & Shi, L. (2018). Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12, 331.
Yoo, Y., et al. (2018). Deep learning of joint myelin and T1w MRI features in normal-appearing brain tissue to distinguish between multiple sclerosis patients and healthy controls. NeuroImage Clinical, 17, 169–178.
Zhang, M., et al. (2021). Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE Transactions on Neural Networks Learning Systems, 33(5), 1947–1958.
Zheng, X., Chen, W., You, Y., Jiang, Y., Li, M., & Zhang, T. (2020). Ensemble deep learning for automated visual classification using EEG signals. Pattern Recognition, 102, 107147.
Zhong, S., Fares, A., & Jiang, J. (2019). An Attentional-LSTM for Improved Classification of Brain Activities Evoked by Images. In Proceedings of the 27th ACM International Conference on Multimedia, 1295–1303.
Author information
Authors and Affiliations
Contributions
Zahra Imani: Conceptualization, Methodology, Software, Formal analysis, Validation, Visualization, Writing - original draft. Mehdi Ezoji: Supervision, Conceptualization, Methodology, Investigation, Validation, Writing - review & editing. Timothee Masquelier: Supervision, Conceptualization, Methodology, Investigation, Validation, Writing - review & editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Action Editor: Pulin Gong
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Imani, Z., Ezoji, M. & Masquelier, T. Brain-guided manifold transferring to improve the performance of spiking neural networks in image classification. J Comput Neurosci 51, 475–490 (2023). https://doi.org/10.1007/s10827-023-00861-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10827-023-00861-z