Abstract
The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG–fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO2, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN–LSTM, CNN–GRU, LSTM–GRU, and CNN–LSTM–GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN–LSTM–GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN–LSTM. The experimental result shows that decision-fused EEG–HbO2–HbR and EEG–fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN–LSTM–GRU in Dataset 01 and CNN–LSTM in Dataset 02 yield the highest performance.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig11_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig12_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig13_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig14_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig15_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig16_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig17_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11571-023-09986-4/MediaObjects/11571_2023_9986_Fig18_HTML.png)
Similar content being viewed by others
References
Al-Shargie F, Tang TB, Kiguchi M (2017) Stress assessment based on decision fusion of EEG and fNIRS signals. IEEE Access 5:19889
Andreessen LM, Gerjets P, Meurers D, Zander TO (2020) Toward neuroadaptive support technologies for improving digital reading: A passive BCI-based assessment of mental workload imposed by text difficulty and presentation speed during reading. User Model User Adap Inter 31:75–104
Appriou A, Cichocki A and Lotte F (2018) Towards robust neuroadaptive HCI: exploring modern machine learning methods to estimate mental workload from EEG signals. In: ACM CHI conference on human factors in computing systems. https://doi.org/10.1145/3170427.3188617
Bagheri M, Power SD (2022) Simultaneous classification of both mental workload and stress level suitable for an online passive brain-computer interface. Sensors 22(2):535
Cao J, Garro EM, Zhao Y (2022) EEG/fNIRS based workload classification using functional brain connectivity and machine learning. Sensors. https://doi.org/10.3390/s22197623
Chiarelli AM, Croce P, Merla A, Zappasodi F (2018) Deep learning for hybrid EEG–fNIRS brain-computer interface: application to motor imagery classification. J Neural Eng. https://doi.org/10.1088/1741-2552/aaaf82
Cho J-H, Jeong J-H, Lee S-W (2021) NeuroGrasp: real-time EEG classification of high-level motor imagery tasks using a dual-stage deep learning framework. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2021.3122969
Cicalese PA, Li R, Ahmadi MB, Wang C, Francis JT, Selvaraj S, Schulzc PE, Zhang Y (2020) An EEG–fNIRS hybridization technique in the fourclass classification of alzheimer’s disease. J Neurosci Methods. https://doi.org/10.1016/j.jneumeth.2020.108618
Dashtestani H, Miguel HO, Condy EE et al (2022) Structured sparse multiset canonical correlation analysis of simultaneous fNIRS and EEG provides new insights into the human action-observation network. Sci Rep. https://doi.org/10.1038/s41598-022-10942-1
Deligani RJ, Borgheai SB, McLinden J, Shahriari Y (2021) Multimodal fusion of EEG–fNIRS: a mutual information-based hybrid classification framework. Biomed Opt Express 12(3):1635–1650
Fatimah B, Pramanick D and Shivashankaran P (2020) Automatic detection of mental arithmetic task and its difficulty level using EEG signals. In: 2020 11th International conference on computing, communication and networking technologies (ICCCNT). pp 1–6. https://doi.org/10.1109/ICCCNT49239.2020.9225647.
Ghaffar MSBA et al (2021) Improving classification performance of four class FNIRS-BCI using Mel frequency cepstral coefficients (MFCC). Infrared Phys Technol 112:103589
Gupta A, Siddhad G, Pandey V, Roy PP, Kim BG (2021) Subject-specific cognitive workload classification using EEG-based functional connectivity and deep learning. Sensors. https://doi.org/10.3390/s21206710
Hollender N, Hofmann C, Deneke M, Schmitz B (2010) Integrating cognitive load theory and concepts of human-computer interaction. Comput Hum Behav 26(6):1278–1288
Hosni SMI, Borgheai SB, McLinden J et al (2022) a graph-based nonlinear dynamic characterization of motor imagery toward an enhanced hybrid BCI. Neuroinformatics 20:1169–1189. https://doi.org/10.1007/s12021-022-09595-2
Khademi Z, Ebrahimi F, Kordy HM (2022) A transfer learning-based CNN and LSTM hybrid deep learning model to classify motor imagery EEG signals. Comput Biol Med. https://doi.org/10.1016/j.compbiomed.2022.105288
Khan MJ, Hong KS (2017) Hybrid EEG–fNIRS-based eight-command decoding for BCI: application to quadcopter control. Front Neurorobot. https://doi.org/10.3389/fnbot.2017.00006
Kocsis L, Herman P, Eke A (2006) The modified Beer–Lambert law revisited. Phys Med Biol 51(5):N91
Li MA, Wang YF, Jia SM, Sun YJ, Yang JF (2019) Decoding of motor imagery EEG based on brain source estimation. Neurocomputing 339(1):182–193
Li Y, Zhang X, Ming D (2023) Early-stage fusion of EEG and fNIRS improves classification of motor imagery. Front Neurosci. https://doi.org/10.3389/fnins.2022.1062889
Ma T, Chen W, Li X, **a Y, Zhu X, He S (2021) fNIRS signal classification based on deep learning in rock-paper-scissors imagery task. Appl Sci. https://doi.org/10.3390/app11114922
Ma T et al (2021) CNN-based classification of fNIRS signals in motor imagery BCI system. J Neural Eng 18(5):056019
Mughal NE, Khan MJ, Khalil K, Javed K, Sajid H, Naseer N, Ghafoor U, Hong K-S (2022) EEG–fNIRS-based hybrid image construction and classification using CNN–LSTM. Front Neurorobot. https://doi.org/10.3389/fnbot.2022.873239
Ortega P and Faisal A (2021a) HemCNN: deep learning enables decoding of fNIRS cortical signals in hand grip motor tasks. In: 2021a 10th International IEEE/EMBS conference on neural engineering (NER). pp 718–721. https://doi.org/10.1109/NER49283.2021.9441323.
Ortega P, Faisal AA (2021b) Deep learning multimodal fNIRS and EEG signals for bimanual grip force decoding. J Neural Eng. https://doi.org/10.1088/1741-2552/ac1ab3
Peng H et al (2018) Single-trial classification of fNIRS signals in four directions motor imagery tasks measured from prefrontal cortex. IEEE Trans Nanobiosci 17(3):181–190. https://doi.org/10.1109/TNB.2018.2839736
Peng H, Li C, Chao J, Wang T, Zhao C, Huo X, Hu B (2021) A novel automatic classification detection for epileptic seizure based on dictionary learning and sparse representation. Neurocomputing 424:179–192
Qiu L, Zhong Y, **e Q, He Z, Wang X, Chen Y, Zhan CA, Pan J (2022) Multi-Modal Integration of EEG–fNIRSfor Characterization of Brain Activity Evoked by Preferred Music. Front Neurorobot. https://doi.org/10.3389/fnbot.2022.823435
Rabbani MHR and Islam SMR (2021) Integration of Decision Fusion and Feature Fusion on EEG and fNIRS Signal. CMPBUP-D-21-01026, Available at SSRN: https://ssrn.com/abstract=3993174
Saadati M, Nelson J and Ayaz H (2020) Multimodal fNIRS–EEG classification using deep learning algorithms for brain-computer interfaces purposes. In: Advances in neuroergonomics and cognitive engineering, AHFE 2019, Advances in intelligent systems and computing, vol 953. Springer. https://doi.org/10.1007/978-3-030-20473-0_21
Sakhavi S, Guan C, Yan S (2018) Learning temporal information for brain–computer interface using convolutional neural networks. IEEE Trans Neural Netw Learn Syst 29(11):5619–5629. https://doi.org/10.1109/TNNLS.2018.2789927
Shin J et al (2017) Open access dataset for EEG + NIRS single-trial classification. IEEE Trans Neural Syst Rehabil Eng 25(10):1735
Shin J, Kwon J, Im CH (2018a) A ternary hybrid EEG-NIRS brain-computer interface for the classication of brain activation patterns during mental arithmetic, motor imagery, and idle state. Front Neuroinform. https://doi.org/10.3389/fninf.2018.00005
Shin J, et al. (2018b) Open access repository for hybrid EEG–NIRS data. In: 2018b 6th International conference on brain–computer interface (BCI). pp 1–4. https://doi.org/10.1109/IWW-BCI.2018.8311523.
Shin J, Lühmann A, Kim D, Mehnert J, Hwang H, Müller K (2018c) Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Sci Data. https://doi.org/10.1038/sdata.2018.3
Singh K, Malhotra J (2022) Smart neurocare approach for detection of epileptic seizures using deep learning based temporal analysis of EEG patterns. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-12512-z
Wickramaratne SD and Mahmud MS (2021) A deep learning based ternary task classification system using Gramian angular summation field in fNIRS neuroimaging data. In: 2020 IEEE international conference on e-health networking, application and services (HEALTHCOM). pp 1–4. https://doi.org/10.1109/HEALTHCOM49281.2021.9398993.
**ao X, Fang Y (2021) Motor imagery EEG signal recognition using deep convolution neural network. Front Neurosci. https://doi.org/10.3389/fnins.2021.655599
**e Y, Oniga S (2023) Classification of motor imagery EEG signals based on data augmentation and convolutional neural networks. Sensors. https://doi.org/10.3390/s23041932
Yuksel BF, Oleson KB, Harrison L, Peck EM, Afergan D, Chang R and Jacob RJK (2016) Learn piano with BACh: an adaptive learning interface that adjusts task difficulty based on brain state. In: Proceedings of the 2016 CHI conference on human factors in computing systems—CHI. https://doi.org/10.1145/2858036.2858388.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rabbani, M.H.R., Islam, S.M.R. Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks. Cogn Neurodyn (2023). https://doi.org/10.1007/s11571-023-09986-4
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11571-023-09986-4