Abstract
With the continuing development of brain–computer interface technology, the analysis and interpretation of brain signals are becoming increasingly important. In the field of brain–computer interfaces, motor imagery (MI) is an important paradigm for generating specific brain signals through thought alone, rather than actual movement, for computer decoding. Functional near-infrared spectroscopy (fNIRS) imaging technology has been increasingly used in brain–computer interfaces due to its advantages of non-invasiveness, low resource requirements, low cost, and high spatial resolution. Scientists have done a lot of work in channel selection, feature selection, and then applying traditional machine learning methods for classification, but the results achieved so far are still insufficient to meet the conditions for realizing fNIRS brain–computer interfaces. To achieve a higher level of classification of fNIRS signals, we propose a method that fuses CNN and attention mechanisms to analyze the near-infrared signals of motor imagery and mental arithmetic data, which is fed into a neural network by deriving signals of changes in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) concentrations through the modified Beer–Lambert law, and then applied to the fNIRS dataset of 29 healthy subjects to validate the proposed method. In the fNIRS-based BCI, the average classification accuracy of the MI signal from HbR and HbO reaches 85.92% and 86.21%, respectively, and the average classification accuracy of the MA signal reaches 89.66% and 88.79%, respectively. The advantage of our approach is that it is lightweight and improves the classification accuracy of current BCI fNIRS signals.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahmadi A et al (2021) Computer aided networks for ADHD subtypes. Biomed Signal Process Control 63:102227
Ali MU et al (2023) OptEF-BCI: an optimization-based hybrid EEG and fNIRS–brain computer interface. Bioengineering 10(5):608
Alotaibi FM (2023) An AI-inspired spatio-temporal neural network for EEG-based emotional status. Sensors 23(1):498
Altan A, Karasu S (2020) Recognition of COVID-19 disease from X-ray images by hybrid model consisting of 2D curvelet transform, chaotic salp swarm algorithm and deep learning technique. Chaos Solitons Fractals 140:110071
Ang KK et al (2015) A randomized controlled trial of EEG-based motor imagery brain–computer interface robotic rehabilitation for stroke. Clin EEG Neurosci 46(4):310–320
Aydin EA (2020) Subject-specific feature selection for near infrared spectroscopy based brain–computer interfaces. Comput Methods Programs Biomed 195:105535
Chaudhary U, Birbaumer N, Ramos-Murguialday A (2016) Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol 12(9):513–525
Chiarelli AM et al (2018) Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification. J Neural Eng 15(3):036028
Cordonnier J-B, Loukas A, Jaggi M (2019) On the relationship between self-attention and convolutional layers. ar**v preprint ar**v:1911.03584
Ergün E, Aydemir Ö (2020) A hybrid BCI using singular value decomposition values of the fast Walsh Hadamard transform coefficients. IEEE Trans Cogn Dev Syst 15(2):454–463
Ergün E, Aydemir Ö (2018) Decoding of binary mental arithmetic based near-infrared spectroscopy signals. In: 2018 3rd international conference on computer science and engineering (UBMK). IEEE
Gao Z et al (2021) Complex networks and deep learning for EEG signal analysis. Cogn Neurodyn 15:369–388
Gao Y, Jia B, Houston M, Zhang Y (2023) Hybrid EEG-fNIRS Brain Computer Interface Based on Common Spatial Pattern by Using EEG-Informed General Linear Model. IEEE Trans Instrum Meas 72:1–10. https://doi.org/10.1109/TIM.2023.3276509
Goshvarpour A, Goshvarpour A (2023) Matching pursuit-based analysis of fNIRS in combination with cascade PCA and reliefF for mental task recognition. Expert Syst Appl 213:119283
Hong K-S, JawadKhan M, Hong MJ (2018) Feature extraction and classification methods for hybrid fNIRS-EEG brain–computer interfaces. Front Hum Neurosci 12:246
Jiang X et al (2023) Characterizing functional brain networks via spatio-temporal attention 4D convolutional neural networks (STA-4DCNNs). Neural Netw 158:99–110
Jiang X et al (2019) Independent decision path fusion for bimodal asynchronous brain–computer interface to discriminate multiclass mental states. IEEE Access 7:165303–165317
** J, Wang Z, Xu R, Liu C, Wang X, Cichocki A (2023) Robust Similarity Measurement Based on a Novel Time Filter for SSVEPs Detection. IEEE Trans Neural Netw Learn Syst 34(8):4096–4105. https://doi.org/10.1109/TNNLS.2021.3118468
Kim H et al (2022) Task-related hemodynamic changes induced by high-definition Transcranial direct current stimulation in chronic stroke patients: an uncontrolled pilot fNIRS study. Brain Sci 12(4):453
Kroese DP, Rubinstein RY (2012) Monte Carlo methods. Wiley Interdiscip Rev Comput Stat 4(1):48–58
Kunjan S et al (2021) The necessity of leave one subject out (LOSO) cross validation for EEG disease diagnosis. In: Brain informatics: 14th international conference, BI 2021, Virtual Event, September 17–19, 2021, Proceedings 14. Springer International Publishing
Kwak Y, Song W-J, Kim S-E (2022) FGANet: fNIRS-guided attention network for hybrid EEG-fNIRS brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng 30:329–339
Laima S et al (2023) DeepTRNet: time-resolved reconstruction of flow around a circular cylinder via spatiotemporal deep neural networks. Phys Fluids 35(1):015118
Lawhern VJ et al (2018) EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng 15(5):056013
Leoni J et al (2021) Automatic stimuli classification from ERP data for augmented communication via brain–computer interfaces. Expert Syst Appl 184:115572
Michel P, Levy O, Neubig G. Are sixteen heads really better than one?. Advances in neural information processing systems. 2019;32. https://doi.org/10.48550/ar**v.1905.10650
Midha S et al (2021) Measuring mental workload variations in office work tasks using fNIRS. Int J Hum Comput Stud 147:102580
Özçelik YB, Altan A (2023) Overcoming nonlinear dynamics in diabetic retinopathy classification: a robust AI-based model with chaotic swarm intelligence optimization and recurrent long short-term memory. Fractal Fract 7(8):598
Pinti P et al (2020) The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Ann N Y Acad Sci 1464(1):5–29
Rabbani MHR, Islam SMR (2023) Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks. Cogn Neurodyn. https://doi.org/10.1007/s11571-023-09986-4
Sharma R, Kim M, Gupta A (2022) Motor imagery classification in brain–machine interface with machine learning algorithms: classical approach to multi-layer perceptron model. Biomed Signal Process Control 71:103101
Shi X, Li B, Wang W, Qin Y, Wang H, Wang X (2023) Classification algorithm for EEG-based motor imagery using hybrid neural network with spatio-temporal convolution and multi-head attention mechanism. Neurosci. 527. https://doi.org/10.1016/j.neuroscience.2023.07.020
Shin J et al (2016) Open access dataset for EEG+ NIRS single-trial classification. IEEE Trans Neural Syst Rehabil Eng 25(10):1735–1745
Shoeibi A, Ghassemi N, Khodatars M et al (2023) Automatic diagnosis of schizophrenia and attention deficit hyperactivity disorder in rs-fMRI modality using convolutional autoencoder model and interval type-2 fuzzy regression. Cogn Neurodyn 17:1501–1523. https://doi.org/10.1007/s11571-022-09897-w
Sun Z et al (2020) A novel multimodal approach for hybrid brain–computer interface. IEEE Access 8:89909–89918
Susan Philip B, Prasad G, Hemanth DJ (2023) A systematic review on artifact removal and classification techniques for enhanced MEG-based BCI systems. Brain-Computer Interfaces 10(2–4):99–113. https://doi.org/10.1080/2326263X.2023.2233368
Taheri SM, Hesamian G (2013) A generalization of the Wilcoxon signed-rank test and its applications. Stat Pap 54:457–470
Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11):2579–2605
Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30, 5998–6008.
Wang W, Li B, Wang H (2022) A novel end-to-end network based on a bidirectional GRU and a self-attention mechanism for denoising of electroencephalography signals. Neuroscience 505:10–20
Wen D et al (2021) Combining brain–computer interface and virtual reality for rehabilitation in neurological diseases: a narrative review. Ann Phys Rehabil Med 64(1):101404
Zang B et al (2021) A deep learning method for single-trial EEG classification in RSVP task based on spatiotemporal features of ERPs. J Neural Eng 18(4):0460c8
Zhang Y et al (2022) Combining robust level extraction and unsupervised adaptive classification for high-accuracy fNIRS-BCI: an evidence on single-trial differentiation between mentally arithmetic-and singing-tasks. Front Neurosci 16:938518
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors do not have any possible conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shi, X., Li, B., Wang, W. et al. Classification algorithm for motor imagery fusing CNN and attentional mechanisms based on functional near-infrared spectroscopy brain image. Cogn Neurodyn (2024). https://doi.org/10.1007/s11571-024-10116-x
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11571-024-10116-x