Deep Learning-Based Hand Gesture Recognition System and Design of a Human–Machine Interface

Abstract

Hand gesture recognition plays an important role in develo** effective human–machine interfaces (HMIs) that enable direct communication between humans and machines. But in real-time scenarios, it is difficult to identify the correct hand gesture to control an application while moving the hands. To address this issue, in this work, a low-cost hand gesture recognition system based human-computer interface (HCI) is presented in real-time scenarios. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) feature extraction and gesture classification using five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human–machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) smoothing of virtual mouse pointer using of Kalman filter. In our work, five pre-trained CNN models (VGG16, VGG19, ResNet50, ResNet101, and Inception-V1) and ViT have been employed to classify hand gesture images. Two multi-class datasets (one public and one custom) have been used to validate the models. Considering the model’s performances, it is observed that Inception-V1 has significantly shown a better classification performance compared to the other four CNN models and ViT in terms of accuracy, precision, recall, and F-score values. We have also expanded this system to control some multimedia applications (such as VLC player, audio player, playing 2D Super-Mario-Bros game, etc.) with different customized gesture commands in real-time scenarios. The average speed of this system has reached 25 fps (frames per second), which meets the requirements for the real-time scenario. Performance of the proposed gesture control system obtained the average response time in milisecond for each control which makes it suitable for real-time. This model (prototype) will benefit physically disabled people interacting with desktops.

Appendices

A Appendix

1.1 A.1 VGG16

VGG16 architecture is an one type of variant of VGGNet [27] model, consisting of 13 convolutional layers with kernel sizes of $(3 \times 3)$ and Relu activation function. Each convolution layer is pursued by max-pooling layer with filter size $(2 \times 2)$. Finally the FC layer is added with softmax activation function to produce the final output class label. In this architecture, the depth of network is increased by adding more convolution and max-pooling layers. This network is trained on large-scale ImageNet [36] dataset, ImageNet dataset consists of millions of images and having more than 20,000 class labels developed for large scale visual recognition challenge. VGG16 has reported test accuracy of 92.7% in the ILSVRC-2012 challenge.

1.2 A.2 VGG19

VGG19 is also a variant of VGGNet network [27], it comprises 16 convolutional layers and three dense layers with kernel /filter size of $(3 \times 3)$ and then the max-pooling layer is used with filter size $(2 \times 2)$. This architecture is pursued by the final FC layer with softmax function to deliver the predicted class label. This model has achieved second rank in ILSVRC-2014 challenge after being trained on ImageNet [36] dataset. This model has an input size of $(224 \times 224)$.

1.3 A.3 Inception-V1

Inception-V1 or GoogleNet [29] is a powerful CNN architecture with having 22 layers built on the inception module. In this module, the architecture is limited by three independent filters such as $(1 \times 1)$, $(3 \times 3)$ and $(5 \times 5)$. Here in this architecture, $(1 \times 1)$ filter is used before $(3 \times 3)$, $(5 \times 5)$ convolutional filters for dimension reduction purpose. This module also includes one max-pooling layer with pool size $(3 \times 3)$. In this module the outputs by using convolutional layers such as $(1 \times 1)$, $(3 \times 3)$ and $(5 \times 5)$ are concatenated and form the inputs for next layer. The last part follows the FC layer with a softmax function to produce the final predicted output. This input of this model is $(224 \times 224)$. This architecture is trained with ImageNet [36] dataset and has reported top-5 error of 6.67% in ILSVRC-2014 challenge.

1.4 A.4 ResNet50

Residual neural network [28] was developed by Microsoft research, This model consists of 50 layers, where 50 stands total number of deep layers, containing 48 convolutional layers, one max-pooling. Finally global average pool layer is connected to the top of the final residual block, which is pursued by the dense layer with softmax activation to generate the final output class. This network has input size of $(224 \times 224)$. The backbone of this architecture is based on residual block. In the case of residual block, the output of one layer is added to a deeper layer in the block, which is also called skip connections or shortcuts. This architecture also reduces the vanishing and exploding gradient problems during training. ResNet50 architecture was trained on the ImageNet dataset [36] and has achieved a good results in ILSVRC-2014 challenge with an error of 3.57%.

1.5 A.5 ResNet101

ResNet101 model consists of 101 deep layers. Like ResNet50, this architecture is also based on the residual building block. In our experiment, we have loaded the pre-trained version of this architecture, trained on ImageNet dataset [36] that comprises millions of images. This model’s default input image size is $(224 \times 224)$.

1.6 A.6 Vision Transformer

A standard transformer architecture consists of two components (1) a set of encoder and (2) a decoder. But in the case of ViT, it doesn’t require the decoder part as it contains only encoder part. In Vision Transformer, firstly image is split into fixed-sized patches, and each patch is implemented for the patch-embedding phase. In the case of patch embedding, each patch is flattened to produce a one-dimensional vector. After the patch-embedding phase, positional embedding is added with the patches to retain the positional information about the image patches in the sequence. Next, they are moved to the transformer encoder. The transformer encoder [37] comprises two components: (1) a Multi-head self-attention block (MHSA) and (2) MLP (multiple-layer perceptron). Hence MHSA block splits the inputs into several number of heads so that each head can learn different levels of self-attention. Then, the outputs of multiple attention heads are concatenated and delivered to the MLP. Next, the classification task is performed by the MLP layer.

B Appendix

1.1 B.1 Statistical Hypothesis Testing

We have also performed a statistical analysis in order to check statistical significance of our model. In Experiment-1 4.4, we have conducted one sample t-test with the help of IBM SPSS statistical analysis tool. In case of null hypothesis, we have to assume that our model is not statistically significant.

To obtain the value of t, the following formula is used:

t=$\frac{(\overline{X}-\mu )}{\frac{SD}{\sqrt{k}} }$

Table 13 One sample statistics

Table 14 One-sample T-test result

Where $\overline{X}$ is the mean of samples. $\mu$ the test value. SD sample standard deviation. k size of samples.

To get the value of $\overline{X}$, firstly, we have used the ten-fold-cross-validation strategy using Dataset-1, and have calculated the fold-wise accuracy followed by computing the average (considered as sample mean) of these accuracy values with the Inception-V1 (best-selected model for Experiment-1 4.4) model.

Table 13 shows that the sample mean ($\overline{X}$), sample size (k), test value ($\mu$), and the standard deviation (SD) are 99.83, 10, 99, and 0.2907, respectively, and the entire statistical analysis of one-sample t-test has been demonstrated in Table 14.

The results in Table 14 exhibit that p-value < 0.001. Here p-value is used for hypothesis testing to determine whether there is evidence to reject the null hypothesis.

If p < $\alpha$ where, $\alpha$ (confidence level) = 0.05, then null hypothesis is rejected.

In Table 14, it is observed that p-value is very less than $\alpha$, so the null hypothesis is rejected, and we can say that there is a statistically significant difference in the mean of the accuracy values.