Abstract
Falls have become the second leading cause of accidental death of the elderly. The serious consequences of falls in the elders can be reduced effectively if they can be detected early. This paper proposes a fall detection method based on keypoint attention module and temporal feature extraction. Firstly, the object detection model (YOLOv3) and the pose estimation model (Multi-stage Pose Estimation Network) are used to obtain the body keypoints. Then, we design a sliding window to preprocess the keypoints. The sliding window divides the keypoints in 30 consecutive frames into a group so that the subsequent network can extract the dynamic features from the keypoints. After that, an adaptive keypoint attention module is designed to strengthen the fall-related keypoints. We improve the long-short-term memory network, and utilize it on the strengthened features to extract the dynamic temporal features. Finally, the fully connected layers are used to recognize falls and normal poses. Experimental results show that the proposed approach achieves an accuracy of 99.73% and 99.62% when tested with UR Fall Detection Dataset and Le2i Fall Detection Dataset.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability statement
Data generated during the current study will be made available at reasonable request.
References
Alarifi A, Alwadain A (2021) Killer heuristic optimized convolution neural network-based fall detection with wearable iot sensor devices. Measurement 167:1–10
Ali SF, Khan R, Mahmood A et al (2018) Using temporal covariance of motion and geometric features via boosting for human fall detection. Sensors 18(6):1–19
Beddiar DR, Oussalah M, Nini B (2022) Fall detection using body geometry and human pose estimation in video sequences. J Vis Commun Image Represent 82:1–13
Cao Z, Simon T, Wei SE et al (2017) Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7291–7299
Charfi I, Miteran J, Dubois J et al (2012) Definition and performance evaluation of a robust svm based fall detection solution. In: 2012 eighth international conference on signal image technology and internet based systems, pp 218–224
Chen Y, Wang Z, Peng Y et al (2018) Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7103–7112
Cheng Y, Wang B, Yang B et al (2021) Monocular 3d multi-person pose estimation by integrating top-down and bottom-up networks. CoRR abs/2104.01797. https://arxiv.org/abs/2104.01797
De A, Saha A, Kumar P et al (2022) Fall detection method based on spatio-temporal feature fusion using combined two-channel classification. Multim Tools Appl. https://doi.org/10.1007/s11042-022-11914-3
Dentamaro V, Impedovo D, Pirlo G (2021) Fall detection by human pose estimation and kinematic theory. In: 2020 25th international conference on pattern recognition (ICPR), pp 2328–2335. https://doi.org/10.1109/ICPR48806.2021.9413331
Dhillon A, Verma GK (2020) Convolutional neural network: a review of models, methodologies and applications to object detection. Prog Artif Intell 9(2):85–112
Fan Y, Wen G, Li D et al (2018) Early event detection based on dynamic images of surveillance videos. J Vis Commun Image Represent 51:70–75
Fei K, Wang C, Zhang J et al (2022) Flow-pose net: an effective two-stream network for fall detection. Vis Comput. https://doi.org/10.1007/s00371-022-02416-2
Galvão YM, Ferreira J, Albuquerque VA et al (2021) A multimodal approach using deep learning for fall detection. Expert Syst Appl 168:1–9
Inturi AR, Manikandan VM, Garrapally V (2022) A novel vision-based fall detection scheme using keypoints of human skeleton with long short-term memory network. Arab J Sci Eng. https://doi.org/10.1007/s13369-022-06684-x
Islam MM, Tayan O, Islam MR et al (2020) Deep learning based systems developed for fall detection: a review. IEEE Access 8:166 (117–166, 137)
Javed MH, Yu Z, Li T et al (2022) Hybrid two-stream dynamic cnn for view adaptive human action recognition using ensemble learning. Int J Mach Learn Cybern 13(4):1157–1166
Kolotouros N, Pavlakos G, Daniilidis K (2019) Convolutional mesh regression for single-image human shape reconstruction. CoRR abs/1905.03244. https://arxiv.org/abs/1905.03244
Kreiss S, Bertoni L, Alahi A (2019) Pifpaf: Composite fields for human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11977–11986
Kwolek B, Kepski M (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Methods Prog Biomed 117(3):489–501
Li W, Wang Z, Yin B et al (2019) Rethinking on multi-stage networks for human pose estimation. ar**v:1901.00148
Mamchur N, Shakhovska N et al (2022) Person fall detection system based on video stream analysis. Procedia Comput Sci 198:676–681
Qi T, Bayramli B, Ali U et al (2019) Spatial shortcut network for human pose estimation. ar**v:1904.03141
Ramirez H, Velastin SA, Meza I et al (2021) Fall detection and activity recognition using human skeleton features. IEEE Access 9:33532–33542
Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. ar**v:1804.02767
Ren L, Peng Y (2019) Research of fall detection and fall prevention technologies: a systematic review. IEEE Access 7:77702–77722
Şengül G, Karakaya M, Misra S et al (2022) Deep learning based fall detection using smartwatches for healthcare applications. Biomed Signal Process Control 71:1–13
Soni PK, Choudhary A (2022) Grassmann manifold based framework for automated fall detection from a camera. Image Vis Comput 122:1–9
Wang BH, Yu J, Wang K et al (2020) Fall detection based on dual-channel feature integration. IEEE Access 8:103443–103453
Wu X, Zheng Y, Chu CH et al (2022) Applying deep learning technology for automatic fall detection using mobile sensors. Biomed Signal Process Control 72:1–9
**ao Y, Yin H, Duan T et al (2021) An intelligent prediction model for ucg state based on dual-source lstm. Int J Mach Learn Cybern 12(11):3169–3178
Yadav SK, Luthra A, Tiwari K et al (2022) Arfdnet: an efficient activity recognition & fall detection system using latent feature pooling. Knowl Based Syst 239:1–11
Zhang J, Tu Z, Yang J et al (2022) Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 13232–13242
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61973185, in part by the Development Plan of Young Innovation Team in Colleges and Universities of Shandong Province under Grant 2019KJN011 , in part the Natural Science Foundation of Shandong Province under Grant ZR2020MF097 and the Colleges and Universities Twenty Terms Foundation of **an City (2021GXRC100).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no confict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, J., Gao, M., Li, B. et al. KAMTFENet: a fall detection algorithm based on keypoint attention module and temporal feature extraction. Int. J. Mach. Learn. & Cyber. 14, 1831–1844 (2023). https://doi.org/10.1007/s13042-022-01730-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-022-01730-4