SpringerLink
Log in

A Transfer Learning-Based CNN Deep Learning Model for Unfavorable Driving State Recognition

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

The detection of unfavorable driving states (UDS) of drivers based on electroencephalogram (EEG) measures has received continuous attention from extensive scholars on account of directly reflecting brain neural activity with high temporal resolution and low risk of being deceived. However, the existing EEG-based driver UDS detection methods involve limited exploration of the functional connectivity patterns and interaction relationships within the brain network. Therefore, there is still room for improvement in the accuracy of detection. In this project, we propose three pretrained convolutional neural network (CNN)-based automatic detection frameworks for UDS of drivers with 30-channel EEG signals. The frameworks are investigated by adjusting the learning rate and choosing the optimization solver, etc. Two different conditions of driving experiments are performed, collecting EEG signals from sixteen subjects. The acquired 1-dimensional 30-channel EEG signals are converted into 2-dimensional matrices by the Granger causality (GC) method to form the functional connectivity graphs of the brain (FCGB). Then, the FCGB are fed into pretrained deep learning models that employed transfer learning strategy for feature extraction and judgment of different EEG signal types. Furthermore, we adopt two visualization interpretability techniques, named, activation visualization and gradient-weighted class activation map** (Grad-CAM) for better visualizing and understanding the predictions of the pretrained models after fine-tuning. The experimental outcomes show that Resnet 18 model yields the highest average recognition accuracy of 90% using the rmsprop optimizer with a learning rate of 1e − 3. The overall outcomes suggest that cooperating of biologically inspired functional connectivity graphs of the brain and pretrained transfer learning algorithms is a prospective approach in reducing the rate of major traffic accidents caused by driver unfavorable driving states.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Hao W, Daniel J. Driver injury severity related to inclement weather at highway-rail grade crossings in the United States. Traffic Inj Prev. 2016;17:31–8.

    Article  Google Scholar 

  2. Chen J, Wang H, Hua C. Assessment of driver drowsiness using electroencephalogram signals based on multiple functional brain networks. Int J Psychophysiol. 2018;133:120–30.

    Article  Google Scholar 

  3. Chen J, Wang H, Wang Q, Hua C. Exploring the fatigue affecting electroencephalography based functional brain networks during real driving in young males. Neuropsychologia. 2019;129:200–11.

    Article  Google Scholar 

  4. Yan X, Li X, Liu Y, Zhao J. Effects of foggy conditions on drivers’ speed control behaviors at different risk levels. Saf Sci. 2014;68:275–87.

    Article  Google Scholar 

  5. Min J,  **ong  C, Zhang  Y, Cai M Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model. Biomed Signal Process Control. 2021;69.

  6. Ma Y, Zhang S, Qi D, Luo Z, Li R, T. Potte T, Zhang Y. Driving drowsiness detection with EEG Using a modified hierarchical extreme learning machine algorithm with particle swarm optimization: a pilot study, Electronics. 2020;9.

  7. Lin Z, Qiu T, Liu P, Zhang L, Zhang S, Mu Z. Fatigue driving recognition based on deep learning and graph neural network. Biomed Signal Process Control. 68;2021.

  8. Hu J. Comparison of different features and classifiers for driver fatigue detection based on a single EEG channel. Comput Math Methods Med. 2017.

  9. Zou S, Qiu T, Huang P, Bai X, Liu C. Constructing multi-scale entropy based on the empirical mode decomposition (EMD) and its application in recognizing driving fatigue. J Neurosci Methods. 2020;341.

  10. Hu J. Automated detection of driver fatigue based on AdaBoost classifier with EEG signals. Front Comput Neurosci. 2017;11.

  11. Yildirim O, Talo M, Ay B, Baloglu UB, Aydin G, Acharya UR. Automated detection of diabetic subject using pre-trained 2D-CNN models with frequency spectrum images extracted from heart rate signals. Comput Biol Med. 113 (2019).

  12. Abbasi AA, Hussain L, Awan IA, Abbasi I, Majid A, Nadeem MSA, Chaudhary QA. Detecting prostate cancer using deep learning convolution neural network with transfer learning approach. Cogn Neurodyn. 2020;14:523–33.

    Article  Google Scholar 

  13. Cheng G, Yang CY, Yao XW, Guo L, Han JW. When deep learning meets metric learning: remote sensing image scene classification via learning discriminative CNNs. IEEE Trans Geosci Remote Sens. 2018;56:2811–21.

    Article  Google Scholar 

  14. Zhu YX, Sun WM, Cao XY, Wang CY, Wu DY, Yang Y, Ye N. TA-CNN: Two-way attention models in deep convolutional neural network for plant recognition. Neurocomputing. 2019;365:191–200.

    Article  Google Scholar 

  15. Zhang M, Li B, Liu Y, Tang R, Lang Y, Huang Q, He J. Different modes of low-frequency focused ultrasound-mediated attenuation of epilepsy based on the topological theory. Micromachines. 2021;12.

  16. Luo C, Li F, Li P, Yi C, Li C, Tao Q, Zhang X, Si Y, Yao D, Yin G, Song P, Wang H, Xu P. A survey of brain network analysis by electroencephalographic signals. Cogn Neurodyn. 2022;16:17–41.

    Article  Google Scholar 

  17. Boersma M, Smit DJA, de Bie HMA, Van Baal GCM, Boomsma DI, de Geus EJC, Delemarre-van de Waal HA, Stam CJ. Network analysis of resting state EEG in the develo** young brain: structure comes with maturation. Hum Brain Mapp. 2011;32:413–25.

    Article  Google Scholar 

  18. **n J, Zhou K, Wang Z, Wang Z, Chen J, Wang X, Chen Q. Hybrid high-order brain functional networks for schizophrenia-aided diagnosis. Cogn Comput. 2022;14:1303–15.

    Article  Google Scholar 

  19. Chang W, Wang H, Yan G, Liu C. An EEG based familiar and unfamiliar person identification and classification system using feature extraction and directed functional brain network. Expert Syst Appl. 2020;158.

  20. Hu F, Wang H, Wang Q, Feng N, Chen J, Zhang T. Acrophobia quantified by EEG based on CNN incorporating granger causality. Int J Neural Syst. 2021;31.

  21. Li T, Li G, Xue T, Zhang J. Analyzing brain connectivity in the mutual regulation of emotion-movement using bidirectional Granger causality. Front Neurosci. 2020;14.

  22. Zhang Q, Yang Q, Zhang X, Bao Q, Su J, Liu X. Waste image classification based on transfer learning and convolutional neural network. Waste Manage. 2021;135:150–7.

    Article  Google Scholar 

  23. **e W, Wei S, Zheng Z, Jiang Y, Yang D. Recognition of defective carrots based on deep learning and transfer learning. Food Bioprocess Technol. 2021;14:1361–74.

    Article  Google Scholar 

  24. Hossain SI, de Herve JdG, Hassan MS, Martineau D, Petrosyan E, Corbin V, Beytout J, Lebert I, Durand J, Carravieri I, Brun-Jacob A, Frey-Klett P, Baux E, Cazorla C, Eldin C, Hansmann Y, Patrat-Delon S, Prazuck T, Raffetin A, Tattevin P, Vourc'h G, Lesens O,  Nguifo EM. Exploring convolutional neural networks with transfer learning for diagnosing Lyme disease from skin lesion images. Comput Methods Programs Biomed. 2022;215.

  25. Togacar M, Ergen B. Classification of cloud images by using super resolution, semantic segmentation approaches and binary sailfish optimization method with deep learning model. Comput Electron Agric. 2022;193.

  26. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int J Comput Vision. 2020;128:336–59.

    Article  Google Scholar 

  27. Shahbakhti M, Beiramvand M, Rejer I, Augustyniak P, Broniec-Wojcik A, Wierzchon M, Marozas V. Simultaneous eye blink characterization and elimination from low-channel prefrontal EEG signals enhances driver drowsiness detection. IEEE J Biomed Health Inform. 2022;26:1001–12.

    Article  Google Scholar 

  28. MK. Wali, M. Murugappan, B. Ahmmad, Wavelet Packet transform based driver distraction level classification using EEG. Math Probl Eng. 2013.

  29. Fu R, Wang S, Wang S. Real-time alarm monitoring system for detecting driver fatigue in wireless areas. Promet-Traffic & Transportation. 2017;29:165–74.

    Article  Google Scholar 

  30. Wang L, Johnson D, Lin Y. Using EEG to detect driving fatigue based on common spatial pattern and support vector machine. Turk J Electr Eng Comput Sci. 2021;29:1429–44.

    Article  Google Scholar 

  31. Zhang W, Wang F, Wu S, Xu Z, ** J, Jiang Y. Partial directed coherence based graph convolutional neural networks for driving fatigue detection. Rev Sci Instrum. 91;2020.

  32. Shahbakhti M, Beiramvand M, Nasiri E, Chen W, Sole-Casals J, Wierzchon M, Broniec-Wojcik A, Augustyniak P, Marozas Ieee V. The importance of gender specification for detection of driver fatigue using a single EEG channel, 14th Biomed Eng Int Conf. (BMEiCON)Songkhla Lipe, Thailand, 2022.

  33. Ko LW, Komarov O, Lai WK, Liang WG,  Jung TP. Eyeblink recognition improves fatigue prediction from single-channel forehead EEG in a realistic sustained attention task. J Neural Eng. 2020;17.

  34. Shahbakhti M, Beiramvand M, Nasiri E, Far SM, Chen W, Sole-Casals J, Wierzchon M, Broniec-Wojcik A, Augustyniak P, Marozas V. Fusion of EEG and eye blink analysis for detection of driver fatigue. IEEE Trans Neural Syst Rehabil Eng. 2023;31:2037–46.

    Article  Google Scholar 

Download references

Funding

This study was funded by the National Natural Science Foundation of China (grant no. 62101355), China Postdoctoral Science Foundation (grant no. 2021M692228), Natural Science Foundation of Liaoning Province (grant no. 2022-BS-176), and Basic Research Project of Education Department of Liaoning Province (grant no. LJKMZ20220457).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shenyang University of Technology, Shenyang, Liaoning, 110870, China

    Jichi Chen

  2. School of Mechanical Engineering and Automation, Northeastern University, Shenyang, Liaoning, 110819, China

    Hong Wang

  3. School of Chemical Equipment, Shenyang University of Technology, Liaoyang, Liaoning, 111000, China

    Enqiu He

Authors
  1. Jichi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Enqiu He
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Jichi Chen; methodology: Jichi Chen; software: Jichi Chen and Enqiu He; visualization: Jichi Chen and Enqiu He; resources: Jichi Chen and Hong Wang; writing—original draft: Jichi Chen; writing—review and editing: Jichi Chen, Hong Wang, and Enqiu He; funding acquisition: Jichi Chen; supervision: Hong Wang.

Corresponding authors

Correspondence to Jichi Chen or Enqiu He.

Ethics declarations

Ethics Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, J., Wang, H. & He, E. A Transfer Learning-Based CNN Deep Learning Model for Unfavorable Driving State Recognition. Cogn Comput 16, 121–130 (2024). https://doi.org/10.1007/s12559-023-10196-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-023-10196-7

Keywords

Search

Navigation