SpringerLink
Log in

An Automated Recognition System of Sign Languages Using Deep Learning Approach

  • Chapter
  • First Online:
Cyber Security in Intelligent Computing and Communications

Part of the book series: Studies in Computational Intelligence ((SCI,volume 1007))

  • 757 Accesses

Abstract

Speech-impaired persons frequently use hand-based gestures and movements to communicate. Regrettably, many people cannot understand the semantics of these signs. So, the communication between the hearing people and the deaf community is very challenging. Since the deaf community is generally less skilled while writing a spoken language, so the alternative of written communication is another challenge. Also, the face to face communication is very slow between the hearing and deaf people. Therefore, to compensate for this type of problem, we proposed an automated real-time Sign Language recognition system. The major objective of the given paper is to contribute and additionally advance to the field of automated Sign Language recognition systems. The main focus of our work is to recognize the signs or gestures. The proposed model was successful in surpassing state of the art testing. We can achieve the highest generalized testing accuracy of 98.56% on the validation data and 99.91% on the test data. We aspire to add Natural Language Processing so the model can make words and sentences out of the letters it recognizes, which will be more practical to use.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Murray, J. (2018). World Federation of the deaf. Rome, Italy. http://wfdeaf.org/ourwork/%20Accessed%202020-01-30

  2. K. Grobel, M. Assan, Isolated sign language recognition using hidden markov models, in Systems, Man, and Cybernetics, 1997. Computational Cybernetics and Simulation. IEEE International Conference on 1997, vol. 1 (IEEE, 1997), pp. 162–167

    Google Scholar 

  3. C.-L. Huang, W.-Y. Huang, Sign language recognition using model-based tracking and a 3D hop field neural network. Mach. Vis. Appl. 10(5–6), 292–307 (1998)

    Article  Google Scholar 

  4. I. N. Sandjaja, N. Marcos, Sign language number recognition, in Proceedings of 2009 Fifth International Joint Conference on INC, IMS and IDC (2009), pp. 1503–1508

    Google Scholar 

  5. N. Pugeault, R. Bowden, Spelling it out: real-time ASL fingerspelling recognition, in IEEE Workshop on Consumer Depth Cameras for Computer Vision (2011)

    Google Scholar 

  6. N.H. Dardas, N.D. Georganas, Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. Instrument. Measur. 60, 3592–3607 (2011)

    Article  Google Scholar 

  7. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake, Real-time human pose recognition in parts from single depth image, Commun. ACM (CACM) (2011)

    Google Scholar 

  8. S. Qin, X. Zhu, H. Yu, S. Ge, Y. Yang, Y. Jiang, Real-time markerless hand gesture recognition with depth camera, in Advances in Multimedia Information Processing (2012), pp. 186–197

    Google Scholar 

  9. D. Deora, N. Bajaj, Indian sign language recognition, in IEEE Xplore, Conference 19–21 Dec 2012. https://doi.org/10.1109/ET2ECN.2012.6470093

  10. H.S. Yeo, B.G. Lee, H. Lim, Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware. Multimedia Tools Appl. (2013)

    Google Scholar 

  11. Z. Ren, J. Yuan, J. Meng, Z. Zhang, Robust part-based hand gesture recognition using Kinect sensor. IEEE Trans. Multimedia 15(5), (2013)

    Google Scholar 

  12. F. Dominio, M. Donadeo, P. Zanuttigh, Combining multiple depth-based descriptors for hand gesture recognition. Pattern Recogn. Lett. 101–111 (2014)

    Google Scholar 

  13. C. Dong, American sign language alphabet recognition using Microsoft Kinect, Thesis (2015)

    Google Scholar 

  14. J.R. Balbin, D.A. Padilla, F.S. Caluyo, J.C. Fausto, C.C. Hortinela, C.O. Manlises, C.K.S. Bernardino, E.G. Finones, L.T. Ventura, Sign language word translator using neural networks for the aurally impaired as a tool for communication, in Proceedings of the 2016 6th IEEE International Conference on Control System, Computing and Engineering (ICCSCE) (2016), pp. 425–442

    Google Scholar 

  15. C. Ong, I. Lim, J. Lu, C. Ng, T. Ong, Sign-language recognition through gesture & movement analysis (SIGMA). Mechatron. Mach. Vis. Pract. 3, 232–245 (2018)

    Google Scholar 

  16. L.K.S. Tolentino, R.O. Serfa Juan, A.C. Thio-ac, M.A.B. Pamahoy, J.R.R. Forteza, X.J.O. Garcia, Static sign language recognition using deep learning. Int. J. Mach. Learn. Comput. 9(6) (2019)

    Google Scholar 

  17. R. Rastgoo, K. Kiani, S. Escalera, Sign language recognition: a deep survey. Published by Elsevier Ltd (July 2020). https://doi.org/10.1016/j.eswa.2020.113794

    Article  Google Scholar 

  18. A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Deep learning for computer vision: a brief review. Hindawi Comput. Intell. Neurosci. 1–13 (2018). https://doi.org/10.1155/2018/7068349

  19. J. Wu, Convolutional neural networks. LAMDA Group, National Key Lab for Novel Software Technology Nan**g University, China (2019). https://cs.nju.edu.cn/wujx/teaching/15%7B%5C_%7DCNN.pdf

  20. T. Wang, Recurrent neural network. Machine Learning Group, University of Toronto, for CSC2541, Sport Analytics (2016). https://www.cs.toronto.edu/%7B~%7Dtingwuwang/rnn%7B%5C_%7Dtutorial.pdf

  21. G. Hinton, Deep Belief Nets (NIPS, Vancouver, B.C., Canada, 2007)

    Google Scholar 

  22. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Nets (NIPS, Monteral, Canada, 2014)

    Google Scholar 

  23. R. Grosse, CSC321 Lecture 20: Autoencoders (Toronto University, 2017). http://www.cs.toronto.edu/%7B~%7Drgrosse/courses/csc321%7B%5C_%7D2017/slides/lec20.pdf

  24. C. Doersch, Tutorial on Variational Autoencoders (2016). ar**v:1606.05908

  25. A. Khan, A. Sohail, U. Zahoora, A.S. Qureshi, A survey of the recent architectures of deep convolutional neural networks. Artif. Intell. Rev. (2020). https://doi.org/10.1007/s10462-020-09825-6

  26. J. Bouvrie, 1 Introduction Notes on Convolutional Neural Networks (2006). https://doi.org/10.1016/j.protcy.2014.09.007

  27. C. Szegedy, W. Liu, Y. Jia, et al., Going deeper with convolutions, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2015), pp. 1–9

    Google Scholar 

  28. J. Pang, K. Chen, J. Shi, et al., Libra R-CNN: towards balanced learning for object detection (2020)

    Google Scholar 

  29. T.Y. Lin, P. Dollár, R. Girshick, et al., Feature pyramid networks for object detection, in Proceedings—30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR, 2017)

    Google Scholar 

  30. Z. Cai, N. Vasconcelos, Cascade R-CNN: high quality object detection and instance segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (2019). https://doi.org/10.1109/tpami.2019.2956516

    Article  Google Scholar 

  31. X. Chen, G. Wanga, H. Guoa, C. Zhanga, Pose guided structured region ensemble network for cascaded hand pose estimation. Neurocomputing (2018). https://doi.org/10.1016/j.neucom.2018.06.097

    Article  Google Scholar 

  32. E. Dibra, T. Wolf, C. Oztireli, M. Gross, How to refine 3D hand pose estimation from unlabelled depth data? in International Conference on 3D Vision (3DV) (Qingdao, China, 2017)

    Google Scholar 

  33. B. Doosti, Hand Pose Estimation: A Survey (2019). ar**v: 1903.01013

    Google Scholar 

  34. E. Escobedo-Cardenas, G. Camara-Chavez, Multi-modal hand gesture recognition combining temporal and pose information based on cnn descriptors and histogram of cumulative magnitudes. J. Vis. Commun. Image Represent. (2020)

    Google Scholar 

  35. F. Gomez-Donoso, S. Orts-Escolano, M. Cazorla, Accurate and efficient 3D hand pose regression for robot hand tele-operation using a monocular RGB camera. Expert Syst. Appl. 136, 327–337 (2019)

    Google Scholar 

  36. L. Zheng, B. Liang, A. Jiang, Recent advances of deep learning for sign language recognition, in 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA) (Sydney, NSW, Australia, 2017)

    Google Scholar 

  37. H. Guo, G. Wang, X. Chen, Towards Good Practices for Deep 3D Hand Pose Estimation (2017). ar**v:1707.07248

  38. J. Supancic, G. Rogez, Y. Yang, J. Shotton, D. Ramana, Depth-based hand pose estimation: methods, data, and challenges. Int. J. Comput. Vis. 1180–1198 (2018)

    Google Scholar 

  39. K.Y. Huang, C.H. Wu, Q.B. Hong, et al., Speech emotion recognition using deep neural network considering verbal and nonverbal speech sounds, in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing—Proceedings (2019)

    Google Scholar 

  40. Kaggle Dataset. https://www.kaggle.com/datamunge/sign-language-mnist

  41. N. Neverova, C. Wolf, G. Taylor, F. Nebout, Hand segmentation with structured convolutional learning, in Asian Conference on Computer Vision (ACCV) 2014: Computer Vision, Singapore (2014), pp 687–702

    Google Scholar 

  42. A. Toshev, C. Szegedy, DeepPose: Human Pose Estimation via Deep Neural Network (2014). ar**v:1312.4659

  43. B. Kang, S. Tripathi, T. Nguyen, Real-time sign language finger-spelling recognition using convolutional neural networks from depth map, in 3rd IAPR Asian Conference on Pattern Recognition (ACPR) (Kuala Lumpur, Malaysia, 2015)

    Google Scholar 

  44. M. Han, J. Chen, L. Li, Y. Chang, Visual hand gesture recognition with convolution neural network, in 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), China (2016)

    Google Scholar 

  45. J. Duan, S. Zhou, J. Wany, X. Guo, S. Li, Multi-modality fusion based on consensus-voting and 3D convolution for isolated gesture recognition (2016). ar** Real-time Hand Tracking Interfaces using Convolutional Neural Networks (GitHub Repository, 2017). https://github.com/victordibia/handtracking/tree/master/docs/handtrack.pdf

  46. A. Dadashzadeh, A. Tavakoli Targhi, M. Tahmasbi, HGR-Net: A Two-stage Convolutional Neural Network for Hand Gesture Segmentation and Recognition (2018). ar**v:1806.05653

  47. G. Anantha Rao, K. Syamala, P.V.V. Kishore, A.S.C.S. Sastry, Deep Convolutional Neural Networks for Sign Language Recognition (SPACES, IEEE Xplore, 2018). https://doi.org/10.1109/SPACES.2018.8316344

  48. O. Kopuklu, A. Gunduz, N. Kose, G. Rigoll, Real-time hand gesture detection and classification using convolutional neural networks, in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). https://doi.org/10.1109/fg.2019.8756576

  49. P.M. Ferreira, D. Pernes, A. Rebelo, J.S. Cardoso, DeSIRe: deep signer-invariant representations for sign language recognition. IEEE Trans. Syst. Man Cybern. Syst. 1–16 (2019). https://doi.org/10.1109/tsmc.2019.2957347

  50. A. Elboushaki, R. Hannane, K. Afdel, L. Koutti, MultiD-CNN: a multidimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences. Expert Syst. Appl. 139 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, G.L. Bajaj Institute of Technology & Management, Greater Noida, U.P., India

    Ganesh Prasad Pal

  2. Department of Computer Science & Engineering, Accurate Institute of Management & Technology, Greater Noida, U.P., India

    Archana Das

  3. Department of Mechanical Engineering, IIMT College of Engineering, Greater Noida, U.P., India

    Saswat Kumar Das & Mayank Raj

Authors
  1. Ganesh Prasad Pal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Archana Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saswat Kumar Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mayank Raj
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. GL Bajaj Institute of Technology and Management, Greater Noida, Uttar Pradesh, India

    Rajeev Agrawal

  2. Kennesaw State University, Kennesaw, GA, USA

    **g He

  3. Malaviya National Institute of Technology (MNIT), Jaipur, India

    Emmanuel Shubhakar Pilli

  4. Department of Master of Computer Applications, GL Bajaj Institute of Technology and Management, Greater Noida, Uttar Pradesh, India

    Sanjeev Kumar

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Pal, G.P., Das, A., Das, S.K., Raj, M. (2022). An Automated Recognition System of Sign Languages Using Deep Learning Approach. In: Agrawal, R., He, J., Shubhakar Pilli, E., Kumar, S. (eds) Cyber Security in Intelligent Computing and Communications. Studies in Computational Intelligence, vol 1007. Springer, Singapore. https://doi.org/10.1007/978-981-16-8012-0_23

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-8012-0_23

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-8011-3

  • Online ISBN: 978-981-16-8012-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation