Abstract
Gesture recognition has been widely used for human-robot interaction. At present, a problem in gesture recognition is that the researchers did not use the learned knowledge in existing domains to discover and recognize gestures in new domains. For each new domain, it is required to collect and annotate a large amount of data, and the training of the algorithm does not benefit from prior knowledge, leading to redundant calculation workload and excessive time investment. To address this problem, the paper proposes a method that could transfer gesture data in different domains. We use a red-green-blue (RGB) Camera to collect images of the gestures, and use Leap Motion to collect the coordinates of 21 joint points of the human hand. Then, we extract a set of novel feature descriptors from two different distributions of data for the study of transfer learning. This paper compares the effects of three classification algorithms, i.e., support vector machine (SVM), broad learning system (BLS) and deep learning (DL). We also compare learning performances with and without using the joint distribution adaptation (JDA) algorithm. The experimental results show that the proposed method could effectively solve the transfer problem between RGB Camera and Leap Motion. In addition, we found that when using DL to classify the data, excessive training on the source domain may reduce the accuracy of recognition in the target domain.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
S. S. Rautaray, A. Agrawal. Vision based hand gesture recognition for human computer interaction: A survey. Artificial Intelligence Review, vol. 43, no. 1, pp. 1–54, 2015. DOI: https://doi.org/10.1007/s10462-012-9356-9.
J. P. Wachs, M. Kölsch, H. Stern, Y. Edan. Vision-based hand-gesture applications. Communications of the ACM, vol. 54, no. 2, pp. 60–71, 2011. DOI: https://doi.org/10.1145/1897816.1897838.
F. Weichert, D. Bachmann, B. Rudak, D. Fisseler. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors, vol. 13, no. 5, pp. 6380–6393, 2013. DOI: https://doi.org/10.3390/s130506380.
C. L. P. Chen, Z. L. Liu. Broad learning system: An effective and efficient incremental learning system without the need for deep architecture. IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 1, pp. 10–24, 2018. DOI: https://doi.org/10.1109/TNNLS.2017.2716952.
L. Yang, S. J. Song, C. L. P. Chen. Transductive transfer learning based on broad learning system. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan, pp. 912–917, 2018. DOI: https://doi.org/10.1109/SMC.2018.00162.
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, V. Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, vol. 17, no. 1, pp. 2096–2030, 2016. DOI: https://doi.org/10.5555/2946645.2946704.
D. M. Roy, L. P. Kaelbling. Efficient Bayesian task-level transfer learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI, Hyderabad, India, pp. 2599–2604, 2007. DOI: https://doi.org/10.5555/16252751625694.
S. J. Pan, Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, 2010. DOI: https://doi.org/10.1109/TKDE.2009.191.
P. Rashidi, D. J. Cook. Activity knowledge transfer in smart environments. Pervasive and Mobile Computing, vol. 7, no. 3, pp. 331–343, 2011. DOI: https://doi.org/10.1016/j.pmcj.2011.02.007.
X. Zhang, Q. Yang. Transfer hierarchical attention network for generative dialog system. International Journal of Automation and Computing, vol. 16, no. 6, pp. 720–736, 2019. DOI: https://doi.org/10.1007/s11633-019-1200-0.
S. Ruder, M. E. Peters, S. Swayamdipta, T. Wolf. Transfer learning in natural language processing. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, Association for Computational Linguistics, Minneapolis, Minnesota, pp. 15–18, 2019. DOI: https://doi.org/10.18653/v1/N19-5004.
Z. Chen, T. Y. Qian. Transfer capsule network for aspect level sentiment classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, pp. 547–556, 2019. DOI: https://doi.org/10.18653/v1/P19-1052.
G. Domeniconi, G. Moro, A. Pagliarani, R. Pasolini. Markov chain based method for in-domain and cross-domain sentiment classification. In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, IEEE, Lisbon, Portugal, pp. 127–137, 2015.
P. H. C. Guerra, A. Veloso, W. Meira, V. Almeida. From bias to opinion: A transfer-learning approach to real-time sentiment analysis. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, San Diego, USA, pp. 150–158, 2011. DOI: https://doi.org/10.1145/2020408.2020438.
X. Yin, X. Yu, K. Sohn, X. M. Liu, M. Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 5697–5706, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00585.
I. D. Apostolopoulos, T. A. Mpesiana. Covid-19: Automatic detection from x-ray images utilizing transfer learning with convolutional neural networks. Physical and Engineering Sciences in Medicine, vol. 43, no. 2, pp. 635–640, 2020. DOI: https://doi.org/10.1007/s13246-020-00865-4.
K. Aukkapinyo, S. Sawangwong, P. Pooyoi, W. Kusakunniran. Localization and classification of rice-grain images using region proposals-baeed convolutional neural network. International Journal of Automation and Computing, vol. 17, no. 2, pp. 233–246, 2020. DOI: https://doi.org/10.1007/s11633-019-1207-6.
Z. W. He, L. Zhang, F. Y. Liu. Discostyle: Multi-level logistic ranking for personalized image style preference inference. International Journal of Automation and Computing, vol. 17, no. 5, pp. 637–651, 2020. DOI: https://doi.org/10.1007/s11633-020-1244-1.
B. Kulis, K. Saenko, T. Darrell. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 1785–1792, 2011. DOI: https://doi.org/10.1109/CVPR.2011.5995702.
M. Raghu, C. Y. Zhang, J. Kleinberg, S. Bengio. Transfusion: Understanding transfer learning for medical imaging. In Proceedings of Advances in Neural Information Processing Systems, Vancouver, Canada, pp. 3342–3352, 2019.
M. Oquab, L. Bottou, I. Laptev, J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, pp. 1717–1724, 2014. DOI: https://doi.org/10.1109/CV-PR.2014.222.
J. E. Liu, M. Shah, B. Kuipers, S. Savarese. Cross-view action recognition via view knowledge transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 3209–3216, 2011. DOI: https://doi.org/10.1109/CVPR.2011.5995729.
V. W. Zheng, S. J. Pan, Q. Yang, J. J. Pan. Transferring multi-device localization models using latent multi-task learning. In Proceedings of the 23rd National Conference on Artificial Intelligence, Chicago, USA, pp. 1427–1432, 2008. DOI: https://doi.org/10.5555/1620270.1620296.
D. H. Hu, Q. Yang. Transfer learning for activity recognition via sensor map**. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Spain, pp. 1962–1967, 2011. DOI: https://doi.org/10.5555/2283696.2283729.
S. J. Pan, I. W. Tsang, J. T. Kwok, Q. Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, vol. 22, no. 2, pp. 199–210, 2011. DOI: https://doi.org/10.1109/TNN.2010.2091281.
M. S. Long, J. M. Wang, G. G. Ding, J. G. Sun, P. S. Yu. Transfer joint matching for unsupervised domain adaptation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, pp. 1410–1417, 2014. DOI: https://doi.org/10.1109/CVPR.2014.183.
L. X. Duan, I. W. Tsang, D. Xu. Domain transfer multiple kernel learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 3, pp. 465–479, 2012. DOI: https://doi.org/10.1109/TPAMI.2011.114.
M. Kurz, G. Hölzl, A. Ferscha, A. Calatroni, D. Roggen, G. Tröster. Real-time transfer and evaluation of activity recognition capabilities in an opportunistic system. Machine Learning, vol. 1, no. 7, pp. 73–78, 2011.
D. Roggen, K. Förster, A. Calatroni, G. Tröster. The adARC pattern analysis architecture for adaptive human activity recognition systems. Journal of Ambient Intelligence and Humanized Computing, vol. 4, no. 2, pp. 169–186, 2013. DOI: https://doi.org/10.1007/s12652-011-0064-0.
G. Marin, F. Dominio, P. Zanuttigh. Hand gesture recognition with jointly calibrated Leap Motion and depth sensor. Multimedia Tools and Applications, vol. 75, no. 22, pp. 14991–15015, 2016. DOI: https://doi.org/10.1007/s11042-015-2451-6.
M. S. Long, J. M. Wang, G. G. Ding, J. G. Sun, P. S. Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 2200–2207, 2013. DOI: https://doi.org/10.1109/ICCV.2013.274.
Q. Sun, R. Chattopadhyay, S. Panchanathan, J. P. Ye. A two-stage weighting framework for multi-source domain adaptation. In Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, pp. 505–513, 2011. DOI: https://doi.org/10.5555/2986459.2986516.
Y. H. Jang, H. Lee, S. J. Hwang, J. Shin. Learning what and where to transfer. [Online], Available: https://arxiv.org/abs/1905.05901, 2019.
Acknowledgements
This work was supported by National Nature Science Foundation of China (NSFC) (Nos. U20A20200, 61811 530281, and 61861136009), Guangdong Regional Joint Foundation (No. 2019B1515120076), Fundamental Research for the Central Universities, and in part by the Foshan Science and Technology Innovation Team Special Project (No. 2018IT100322).
Author information
Authors and Affiliations
Corresponding author
Additional information
Recommended by Associate Editor Hui Yu
Bi-**ao Wu received the B. Eng. degree in electrical engineering from Soochow University, China in 2019. She is currently a master student in control engineering at South China University of Technology, China.
Her research interests include human-robot interaction, gesture recognition and transfer learning.
Chen-Guang Yang received the B. Eng. degree in measurement and control from Northwestern Polytechnical University, China in 2005, the Ph. D. degree in control engineering from National University of Singapore, Singapore in 2010, and postdoctoral training with the Imperial College London, UK. He received Best Paper Awards from IEEE Transactions on Robotics, and over 10 international conferences.
His research interests include robotics and automation.
Jun-Pei Zhong received the B. Eng degree in control science and computer science from South China University of Technology, China in 2006, the M. Phil degree in electrical engineering from Hong Kong Polytechnic University, China in 2010, and the Ph. D. degree in computer science from University of Hamburg, Germany in 2015. He has been awarded the Marie-Curie fellowship for his doctoral study from 2010 to 2013. From 2014 to 2016, he has participated in different European Union and Japanese funded projects at University of Hertfordshire, UK, Plymouth University, UK and Waseda University, Japan.
His research interests include machine learning, computational intelligence and cognitive robotics.
Rights and permissions
Open Access
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Wu, BX., Yang, CG. & Zhong, JP. Research on Transfer Learning of Vision-based Gesture Recognition. Int. J. Autom. Comput. 18, 422–431 (2021). https://doi.org/10.1007/s11633-020-1273-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-020-1273-9