SpringerLink
Log in

Surface Electromyography Signal Recognition Based on Deep Learning for Human-Robot Interaction and Collaboration

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

The interaction between humans and collaborative robots in performing given tasks has aroused the interest of researchers and industry for the development of gesture recognition systems. Surface electromyography (sEMG) devices are recommended to capture human hand gestures. However, this kind of technology raises significant challenges. sEMG signals are difficult to acquire and isolate reliably. The creation of a gesture representative model is hard due to the non-explicit nature of sEMG signals. Several solutions have been proposed for the recognition of sEMG-based hand gestures, but none of them are entirely satisfactory. This study contributes to take a step forward in finding the solution to this problem. A sEMG capturing prototype device was used to collect human hand gestures and a two-step algorithm is proposed to recognize five valid gestures, invalid gestures and non-gestures. The former algorithm step (segmentation) is used for sEMG signal isolation to separate signals containing gestures from signals containing non-gestures. The latter step of the algorithm (recognition) is based on a deep learning method, a convolutional neural network (CNN) that identifies which gesture is in the sEMG signals. The performances of the prototype device and recognition architecture were compared successfully with the off-the-shelf sEMG device Myo. Results indicated that the segmentation process played an important role in the success of the gesture recognition system, excluding sEMG signals containing non-gestures. The proposed system was applied successfully in the control loop of a collaborative robotic application, in which the gesture recognition system achieved an online class recognition rate (CR) of 98%, outperforming similar studies in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code Availability

The code that support the findings of this study are available from the corresponding author, Nuno Mendes, upon reasonable request.

References

  1. Neto, P., Simão, M, Mendes, N., Safeea, M.: Gesture-based human-robot interaction for human assistance in manufacturing. The International Journal of Advanced Manufacturing Technology 101 (1), 119–135 (2019). https://doi.org/10.1007/s00170-018-2788-x

    Article  Google Scholar 

  2. Mendes, N., Ferrer, J., Vitorino, J., Safeea, M., Neto, P.: Human behavior and hand gesture classification for smart human-robot interaction. Procedia Manufacturing 11, 91–98 (2017). https://doi.org/10.1016/j.promfg.2017.07.156

    Article  Google Scholar 

  3. Simao, M., Mendes, N., Gibaru, O., Neto, P.: A Review on Electromyography Decoding and Pattern Recognition for Human-Machine Interaction. IEEE Access 7, 39564–39582 (2019). https://doi.org/10.1109/ACCESS.2019.2906584

    Article  Google Scholar 

  4. Du, Y., **, W., Wei, W., Hu, Y., Geng, W.: Surface emg-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors 17(3). https://doi.org/10.3390/s17030458 (2017)

  5. Allard, U.C., Fall, C.L., Drouin, A., Campeau-Lecours, A., Gosselin, C., Glette, K., Laviolette, F., Gosselin, B.: Deep learning for electromyographic hand gesture signal classification by leveraging transfer learning. IEEE Trans Neural Syst Rehabil Eng 27(4), 760–771 (2019). https://doi.org/10.1109/TNSRE.2019.2896269

    Article  Google Scholar 

  6. Jochumsen, M., Waris, A., Kamavuako, E.N.: The effect of arm position on classification of hand gestures with intramuscular emg. Biomedical Signal Processing and Control 43, 1–8 (2018). https://doi.org/10.1016/j.bspc.2018.02.013

    Article  Google Scholar 

  7. Pimentel, M.A.F., Clifton, D.A., Clifton, L., Tarassenko, L.: A review of novelty detection. Signal Process. 99, 215–249 (2014). https://doi.org/10.1016/j.sigpro.2013.12.026

    Article  Google Scholar 

  8. Jiang, X., Merhi, L.K., **ao, Z.G., Menon, C.: Exploration of Force Myography and surface Electromyography in hand gesture classification. Medical Engineering and Physics 41, 63–73 (2017). https://doi.org/10.1016/j.medengphy.2017.01.015

    Article  Google Scholar 

  9. Liu, Y., Huang, H.: Towards a high-stability emg recognition system for prosthesis control: A one-class classification based non-target emg pattern filtering scheme. In: 2009 IEEE International Conference on Systems, Man and Cybernetics, pp 4752–4757 (2009)

  10. Li, Z., Wang, B., Yang, C., **e, Q., Su, C.: Boosting-based emg patterns classification scheme for robustness enhancement. IEEE Journal of Biomedical and Health Informatics 17(3), 545–552 (2013). https://doi.org/10.1109/JBHI.2013.2256920

    Article  Google Scholar 

  11. Liu, J., Zhang, D., Sheng, X., Zhu, X.: Quantification and solutions of arm movements effect on semg pattern recognition. Biomedical Signal Processing and Control 13, 189–197 (2014)

    Article  Google Scholar 

  12. Wahid, M.F., Tafreshi, R., Al-Sowaidi, M., Langari, R.: Subject-independent hand gesture recognition using normalization and machine learning algorithms. Journal of Computational Science 27, 69–76 (2018). https://doi.org/10.1016/J.JOCS.2018.04.019

    Article  Google Scholar 

  13. Sayin, F.S., Ozen, S., Baspinar, U.: Hand gesture recognition by using sEMG signals for human machine interaction applications. In: Signal Processing - Algorithms, Architectures, Arrangements, and Applications Conference Proceedings, SPA, vol. 2018, pp 27–30. IEEE (2018)

  14. Wang, W., Li, R., Diekel, Z.M., Chen, Y., Zhang, Z., Jia, Y.: Controlling object hand-over in human-robot collaboration via natural wearable sensing. IEEE Transactions on Human-Machine Systems 49(1), 59–71 (2019). https://doi.org/10.1109/THMS.2018.2883176

    Article  Google Scholar 

  15. Zhang, Z., Yang, K., Qian, J., Zhang, L.: Real-Time Surface EMG Pattern Recognition for Hand Gestures Based on an Artificial Neural Network. Sensors 19(14), 3170 (2019). https://doi.org/10.3390/s19143170

    Article  Google Scholar 

  16. Rescio, G., Leone, A., Siciliano, P.: Supervised machine learning scheme for electromyography-based pre-fall detection system. Expert Syst. Appl. 100, 95–105 (2018). https://doi.org/10.1016/j.eswa.2018.01.047

    Article  Google Scholar 

  17. Gu, Y., Yang, D., Huang, Q., Yang, W., Liu, H.: Robust EMG pattern recognition in the presence of confounding factors: features, classifiers and adaptive learning. Expert Syst. Appl. 96, 208–217 (2018). https://doi.org/10.1016/j.eswa.2017.11.049

    Article  Google Scholar 

  18. Kaczmarek, P., Mańkowski, T., Tomczyński, J.: putEMG-A Surface Electromyography Hand Gesture Recognition Dataset. Sensors 19(16), 3548 (2019). https://doi.org/10.3390/s19163548

    Article  Google Scholar 

  19. Feng, N., Shi, Q., Wang, H., Gong, J., Liu, C., Lu, Z.: A soft robotic hand: design, analysis, sEMG control, and experiment. Int. J. Adv. Manuf. Technol. 97(1-4), 319–333 (2018). https://doi.org/10.1007/s00170-018-1949-2

    Article  Google Scholar 

  20. Geng, W., Du, Y., **, W., Wei, W., Hu, Y., Li, J.: Gesture recognition by instantaneous surface EMG images. Sci. Rep. 6(November), 6–13 (2016). https://doi.org/10.1038/srep36571

    Google Scholar 

  21. Tam, S., Boukadoum, M., Campeau-Lecours, A., Gosselin, B.: A fully embedded adaptive real-time hand gesture classifier leveraging hd-semg and deep learning. IEEE Transactions on Biomedical Circuits and Systems 14(2), 232–243 (2020). https://doi.org/10.1109/TBCAS.2019.2955641

    Article  Google Scholar 

  22. Olsson, A.E., Sager, P., Andersson, E., Björkman, A., Malešević, N., Antfolk, C.: Extraction of Multi-Labelled Movement Information from the Raw HD-sEMG Image with Time-Domain Depth. Sci. Rep. 9(1), 7244 (2019). https://doi.org/10.1038/s41598-019-43676-8

    Article  Google Scholar 

  23. Bao, T., Zaidi, S.A.R., **e, S., Yang, P., Zhang, Z.: A CNN-LSTM Hybrid Framework for Wrist Kinematics Estimation Using Surface Electromyography. ar**v, 1–9 (2019)

  24. Cao, X., Iwase, M., Inoue, J., Maeda, E.: Gesture recognition based on ConVLSTm-attention implementation of small data SEMG signals. In: UbiComp/ISWC 2019- - Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, pp 21–24. ACM Press, New York (2019)

  25. Wei, W., Dai, Q., Wong, Y., Hu, Y., Kankanhalli, M., Geng, W.: Surface-Electromyography-Based Gesture Recognition by Multi-View Deep Learning. IEEE Trans. Biomed. Eng. 66(10), 2964–2973 (2019). https://doi.org/10.1109/tbme.2019.2899222

    Article  Google Scholar 

  26. Simão, M.A., Neto, P., Gibaru, O.: Unsupervised gesture segmentation of a real-time data stream in matlab. In: IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society, pp 809–814 (2016)

  27. Simao, M.A., Neto, P., Gibaru, O.: Unsupervised gesture segmentation by motion detection of a real-time data stream. IEEE Transactions on Industrial Informatics 13(2), 473–481 (2017). https://doi.org/10.1109/TII.2016.2613683

    Article  Google Scholar 

  28. Benalcazar, M.E., Motoche, C., Zea, J.A., Jaramillo, A.G., Anchundia, C.E., Zambrano, P., Segura, M., Benalcazar Palacios, F., Perez, M.: Real-time hand gesture recognition using the myo armband and muscle activity detection. In: 2017 IEEE Second Ecuador Technical Chapters Meeting (ETCM), pp 1–6 (2017)

  29. Bisi, S., De Luca, L., Shrestha, B., Yang, Z., Gandhi, V.: Development of an emg-controlled mobile robot. Robotics 7(3). https://doi.org/10.3390/robotics7030036 (2018)

  30. Moin, A., Zhou, A., Rahimi, A., Benatti, S., Menon, A., Tamakloe, S., Ting, J., Yamamoto, N., Khan, Y., Burghardt, F., Benini, L., Arias, A.C., Rabaey, J.M.: An emg gesture recognition system with flexible high-density sensors and brain-inspired high-dimensional classifier. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp 1–5 (2018)

  31. Ma, J., Thakor, N.V., Matsuno, F.: Hand and wrist movement control of myoelectric prosthesis based on synergy. IEEE Transactions on Human-Machine Systems 45(1), 74–83 (2015). https://doi.org/10.1109/THMS.2014.2358634

    Article  Google Scholar 

  32. Ribeiro, J., Mota, F., Cavalcante, T., Nogueira, I., Gondim, V., Albuquerque, V., Alexandria, A.: Analysis of man-machine interfaces in upper-limb prosthesis: A review. Robotics 8(1). https://doi.org/10.3390/robotics8010016 (2019)

  33. Dick, F.S., Bert, U.K., Bernd, G.L., Johannes, P.V.D.: High-density surface emg: Techniques and applications at a motor unit level. Biocybernetics and Biomedical Engineering 32(3), 3–27 (2012). https://doi.org/10.1016/S0208-5216(12)70039-6

    Article  Google Scholar 

  34. Phinyomark, A., Khushaba, R.N., Scheme, E.: Feature extraction and selection for myoelectric control based on wearable emg sensors. Sensors 18(5). https://doi.org/10.3390/s18051615 (2018)

  35. Phinyomark, A., Scheme, E.: A feature extraction issue for myoelectric control based on wearable emg sensors. In: 2018 IEEE Sensors Applications Symposium (SAS), pp 1–6 (2018)

  36. Côté-Allard, U., Fall, C.L., Campeau-Lecours, A., Gosselin, C., Laviolette, F., Gosselin, B.: Transfer learning for semg hand gestures recognition using convolutional neural networks. In: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp 1663–1668 (2017)

  37. Farrell, T.R., Weir, R.F.F.: A comparison of the effects of electrode implantation and targeting on pattern classification accuracy for prosthesis control. IEEE Trans. Biomed. Eng. 55(9), 2198–2211 (2008). https://doi.org/10.1109/TBME.2008.923917

    Article  Google Scholar 

  38. Phinyomark, A., Limsakul, C., Phukpattaranont, P.: A novel feature extraction for robust EMG pattern recognition. Journal of Computing 1(1), 71–79 (2009)

    Google Scholar 

  39. Reining, C., Niemann, F., Moya Rueda, F., Fink, G.A., Ten Hompel, M.: Human Activity Recognition for Production and Logistics-A Systematic Literature Review. Information 10(8), 245 (2019). https://doi.org/10.3390/info10080245

    Article  Google Scholar 

  40. Mendes, N., Simao, M., Neto, P.: Segmentation of electromyography signals for pattern recognition. In: IECON Proceedings (Industrial Electronics Conference), vol. 2019, pp 732–737. IEEE, Lisbon (2019)

  41. Kim, J., Kwak, Y.H., Kim, W., Pak, J.J., Kim, K.: Futuristic input device based on gesture recognition. In: ISERD International Conference, pp 54–56, Madrid (2017)

  42. Neto, P., Pereira, D., Pires, J.N., Moreira, A.P.: Real-time and continuous hand gesture spotting: An approach based on artificial neural networks. In: 2013 IEEE International Conference on Robotics and Automation, pp 178–183 (2013)

  43. Mendes, N., Neto, P.: Hand gesture dataset based on sEMG data captured from the Technaid human-robot interaction system. Zenodo, https://doi.org/10.5281/zenodo.1325173 (2018)

  44. Safeea, M., Neto, P.: Kuka sunrise toolbox: Interfacing collaborative robots with matlab. IEEE Robotics Automation Magazine 26(1), 91–96 (2019). https://doi.org/10.1109/MRA.2018.2877776

    Article  Google Scholar 

  45. Safeea, M., Bearee, R., Neto, P.: End-effector precise hand-guiding for collaborative robots. In: Ollero, A., Sanfeliu, A., Montano, L., Lau, N., Cardeira, C. (eds.) ROBOT 2017: Third Iberian Robotics Conference, pp 595–605. Springer International Publishing, Cham (2018)

  46. Mendes, N., Safeea, M., Neto, P.: Flexible programming and orchestration of collaborative robotic manufacturing systems. In: 2018 IEEE 16th International Conference on Industrial Informatics (INDIN), pp 913–918 (2018)

  47. Matrone, G.C., Cipriani, C., Carrozza, M.C., Magenes, G.: Real-time myoelectric control of a multi-fingered hand prosthesis using principal components analysis. Journal of NeuroEngineering and Rehabilitation 9(40), 1–13 (2012). https://doi.org/10.1186/1743-0003-9-40

    Google Scholar 

  48. Cipriani, C., Antfolk, C., Controzzi, M., Lundborg, G., Rosen, B., Carrozza, M.C., Sebelius, F.: Online myoelectric control of a dexterous hand prosthesis by transradial amputees. IEEE Trans. Neural Syst. Rehabil. Eng. 19(3), 260–270 (2011). https://doi.org/10.1109/TNSRE.2011.2108667

    Article  Google Scholar 

  49. Došen, S., Cipriani, C., Kostić, M., Controzzi, M., Carrozza, M.C., Popovič, D.B.: Cognitive vision system for control of dexterous prosthetic hands: Experimental evaluation. Journal of NeuroEngineering and Rehabilitation 7(42), 1–14 (2010). https://doi.org/10.1186/1743-0003-7-42

    Google Scholar 

  50. Jiang, N., Vujaklija, I., Rehbaum, H., Graimann, B., Farina, D.: Is accurate map** of emg signals on kinematics needed for precise online myoelectric control?. IEEE Trans. Neural Syst. Rehabil. Eng. 22(3), 549–558 (2014). https://doi.org/10.1109/TNSRE.2013.2287383

    Article  Google Scholar 

  51. Ortiz-Catalan, M., Rouhani, F., Branemark, R., Hakansson, B.: Offline accuracy: A potentially misleading metric in myoelectric pattern recognition for prosthetic control. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 1140–1143 (2015)

  52. Vujaklija, I., Roche, A.D., Hasenoehrl, T., Sturma, A., Amsuess, S., Farina, D., Aszmann, O.C.: Translating research on myoelectric control into clinics–are the performance assessment methods adequate? Frontiers in Neurorobotics 11. https://doi.org/10.3389/fnbot.2017.00007 (2017)

Download references

Acknowledgements

Authors acknowledge Fundação para a Ciência e a Tecnologia (FCT - MCTES) for its financial support via the project UIDB/EMS/00667/2020 (UNIDEMI).

Funding

The research leading to these results received funding from Fundação para a Ciência e a Tecnologia (FCT - MCTES) under Grant Agreement No UIDB/EMS/00667/2020 (UNIDEMI).

Author information

Authors and Affiliations

  1. Research and Development Unit for Mechanical and Industrial Engineering, NOVA University of Lisbon, Campus da Caparica, 2829-516, Lisbon, Portugal

    Nuno Mendes

Authors
  1. Nuno Mendes
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Nuno Mendes developed the algorithms, created the datasets, designed the system architecture, designed and performed the experiments, analyzed the data, wrote and review the manuscript.

Corresponding author

Correspondence to Nuno Mendes.

Ethics declarations

Ethics approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study was approved by the Ethics Council of the NOVA University of Lisbon (No. HT.STF.HHG.3541.4361824 4).

Consent for Publication

The author affirms that human research participants provided informed consent for publication of the images in Fig. 6 and Fig. 8. All of the participants have consented to the submission of the results of this study to the journal.

Conflict of Interests

The authors declare they have no conflict of interests.

Additional information

Availability of data and material

All datasets from this study are available in the Zenodo repository through the link http://doi.org/10.5281/zenodo.1325173.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mendes, N. Surface Electromyography Signal Recognition Based on Deep Learning for Human-Robot Interaction and Collaboration. J Intell Robot Syst 105, 42 (2022). https://doi.org/10.1007/s10846-022-01666-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-022-01666-5

Keywords

Search

Navigation