Development of a Sign Language Translator Based on Gestures-To-Words Using IoT

Abstract

Sign language is one of the world’s languages, but instead of using speech or voice, it communicates by hand signals and other body parts. Speech and hearing-impaired people need to be able to communicate in sign language. A speech impaired person is someone who is unable to talk due to a reluctance or inability to speak. A person who has been hearing-impaired since birth or loss may develop later. Malaysian Sign Language (MSL) will be used by Malaysians speech and hearing-impaired population to communicate, but it will be an obstacle as they need to converse with normal people. Currently, sign language translators are mostly focused on general sign language and American Sign Language (ASL). This project aimed to develop a sign language translator system based on MSL with NodeMCU ESP8266, flex sensors and MPU6050 accelerometer gyroscope sensor module that would enable speech and hearing-impaired people to communicate with regular people who are unfamiliar with sign language. It used a data glove approach method that can convert a hand gesture into words. In this context, the flex sensors defined the sensor values based on finger bending, meanwhile, the accelerometer gyro sensor module defined the hand position. This project implements IoT technology to display the meaning of sign language gestures on a smartphone. A NodeMCU ESP8266 module was also used as a source IoT platform by integrating with the Blynk application in the smartphone to display the output of the system. The Arduino IDE and PLX-DAQ software are used to build and evaluate a graph of resistance value against flex sensor deflection to integrate hardware and software. The prototype was found to work smoothly for translation during the hardware and software testing.