Abstract
In this paper, a fully immersive serious game system that combines two Natural User Interfaces (NUIs) and a Head Mounted Display (HMD) to provide an interactive Virtual Environment (VE) for patient rehabilitation is proposed. Patients’ data are acquired in real-time by the NUIs, while by the HMD the VE is shown to them, thus allowing the interaction. A Long Short-Term Memory Recurrent Neural Network (LSTM-RNN), previously trained by healthy subjects (i.e., baseline), processes patients’ movements in real-time during the rehabilitation exercises to provide the degree of their performance. By comparing the functionalities of the proposed system with the ongoing state-of-the-art, it is worth noting that the reported fully immersive serious game system provides a concrete contribute to the current literature in terms of completeness and versatility. The results obtained by three rehabilitation exercises, chosen as reference case studies, performed on real patients affected by Parkinson’s disease have shown the effectiveness of the presented approach. Finally, the analysis of the feedbacks received by the therapists and patients who have used the system have highlighted remarkable results in terms of motivation, acceptance, and usability.
Similar content being viewed by others
References
Ai B, Zhou Y, Yu Y, Du S (2017) Human pose estimation using deep structure guided learning. In: Winter conference on applications of computer vision (WACV), pp 1224–1231
Alimanova M, Borambayeva S, Kozhamzharova D, Kurmangaiyeva N, Ospanova D, Tyulepberdinova G, Gaziz G, Kassenkhan A (2017) Gamification of hand rehabilitation process using virtual reality tools: using leap motion for hand rehabilitation. In: 1th International conference on robotic computing (IRC), pp 336–339
Angra S, Ahuja S (2017) Machine learning and its applications: a review. In: International conference on big data analytics and computational intelligence (ICBDAC), pp 57–60
Asadi-Aghbolaghi M, Clapés A, Bellantonio M, Escalante HJ, Ponce-López V, Baró X, Guyon I, Kasaei S, Escalera S (2017) A survey on deep learning based approaches for action and gesture recognition in image sequences. In: 12th International conference on automatic face gesture recognition (FG), pp 476–483
Avola D, Spezialetti M, Placidi G (2013) Design of an efficient framework for fast prototy** of customized human –computer interfaces and virtual environments for rehabilitation. Comput Methods Programs Biomed 110(3):490–502
Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. Trans Neur Netw 5(2):157–166
Brau E, Jiang H (2016) 3d human pose estimation via deep learning from 2d annotations. In: 4th International conference on 3D vision (3DV), pp 582–591
Brooke J (2013) Sus: a retrospective. J Usab Stud 8(2):29–40
Byeon W, Breuel TM, Raue F, Liwicki M (2015) Scene labeling with lstm recurrent neural networks. In: Conference on computer vision and pattern recognition (CVPR), pp 3547–3555
Cho K, van Merrienboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: encoder-decoder approaches. In: 8th Workshop on syntax, semantics and structure in statistical translation (SSST), pp 103–111
Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: Deep learning and representation learning workshop (NIPS), pp 1–9
Crosbie JH, Lennon S, McNeill MDJ, McDonough SM (2006) Virtual reality in the rehabilitation of the upper limb after stroke: The user’s perspective. Cyberpsychol Behav 9(2):137–141
Desai PR, Desai PN, Ajmera KD, Mehta K (2014) A review paper on oculus rift-a virtual reality headset. Int J Eng Trends Technol 13(4):175–179
Dorsey ER, Constantinescu R, Thompson JP, Biglan KM, Holloway RG, Kieburtz K, Marshall FJ, Ravina BM, Schifitto G, Siderowf A, Tanner CM (2007) Projected number of people with parkinson disease in the most populous nations, 2005 through 2030. Neurology 68(5):384–386
Feng X, Liu C, Guo Q, Bai Y, Ren Y, Ren B, Bai J, Chen L (2013) Research progress in rehabilitation treatment of stroke patients: a bibliometric analysis. Neural Regen Res 8(15):1423–1430
García-Martínez S, Orihuela-Espina F, Sucar LE, Moran AL, Hernández-Franco J (2015) A design framework for arcade-type games for the upper-limb rehabilitation. In: International conference on virtual rehabilitation (ICVR), pp 235–242
Gargantini A, Terzi F, Zambelli M, Bonfanti S (2015) A low-cost virtual reality game for amblyopia rehabilitation. In: 3rd Workshop on ICTs for improving patients rehabilitation research techniques (REHAB), pp 81–84
Gobron SC, Zannini N, Wenk N, Schmitt C, Charrotton Y, Fauquex A, Lauria M, Degache F, Frischknecht R (2015) Serious games for rehabilitation using head-mounted display and haptic devices. In: International conference on augmented and virtual reality (AVR), pp 199–219
Golomb MR, McDonald BC, Warden SJ, Yonkman J, Saykin AJ, Shirley B, Huber M, Rabin B, AbdelBaky M, Nwosu ME, Barkat-Masih M, Burdea GC (2010) In-home virtual reality videogame telerehabilitation in adolescents with hemiplegic cerebral palsy. Arch Phys Med Rehabil 91(1):1–9
Guna J, Jakus G, Pogačnik M, Tomažič S, Sodnik J (2014) An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors 14(2):3702–3720
Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334
Hansard M, Lee S, Choi O, Horaud R (2013) Time of flight cameras: principles, methods, and application. In: SpringerBriefs in computer science. 1st edn. Springer-Verlag, London, p 95
Harrington MCR (2011) Empirical evidence of priming, transfer, reinforcement, and learning in the real and virtual trillium trails. IEEE Trans Learn Technol 4(2):175–186
Hermans M, Schrauwen B (2013) Training and analysing deep recurrent neural networks. In: Burges C J C, Bottou L, Welling M, Ghahramani Z, Weinberger K Q (eds) Advances in neural information processing systems, pp 190-198
Hirsch M, Farley B (2009) Exercise and neuroplasticity in persons living with parkinson’s disease. Eur J Phys Rehabil Med 45(2):215–229
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Holzinger A, Scherer R, Seeber M, Wagner J, Müller-Putz G (2012) Computational sensemaking on examples of knowledge discovery from neuroscience data: towards enhancing stroke rehabilitation. In: 3th International conference on information technology in bio- and medical informatics (ITBAM), pp 166–168
Horaud R, Hansard M, Evangelidis G, Ménier C (2016) An overview of depth cameras and range scanners based on time-of-flight technologies. Mach Vis Appl 27(7):1005–1020
Huang Z, Wan C, Probst T, Van Gool L (2017) Deep learning on lie groups for skeleton-based action recognition. In: Conference on computer vision and pattern recognition (CVPR), pp 6099–6108
Ij**a EP, Chalavadi KM (2017) Human action recognition in rgb-d videos using motion sequence information and deep learning. Pattern Recognit 72(Supplement C):504–516
Jorissen P, Wijnants M, Lamotte M (2005) Dynamic interactions in physically realistic collaborative virtual environments. IEEE Trans Vis Comput Graphics 11(6):649–660
Kato N, Tanaka T, Sugihara S, Shimizu K, Kudo N (2016) Trial operation of a cloud service-based three-dimensional virtual reality tele-rehabilitation system for stroke patients. In: 11th International conference on computer science education (ICCSE), pp 285–290
Keus SH, Munneke M, Nijkrake MJ, Kwakkel G, Bloem BR (2009) Physical therapy in parkinson’s disease: evolution and future challenges. Mov Disord 24(1):1–14
Knight A, Carey S, Dubey R (2016) An interim analysis of the use of virtual reality to enhance upper limb prosthetic training and rehabilitation. In: 9th ACM International conference on pervasive technologies related to assistive environments, pp 1–4
Kwakkel G, de Goede C, van Wegen E (2007) Impact of physical therapy for parkinson’s disease: a critical review of the literature. Parkinson Related Disord 13(Supplement 3):S478–S487
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444
Li S, Zhang W, Chan AB (2017) Maximum-margin structured learning with deep networks for 3d human pose estimation. Int J Comput Vis 122(1):149–168
Luis MAVS, Atienza RO, Luis AMS (2016) Immersive virtual reality as a supplement in the rehabilitation program of post-stroke patients. In: 10th International conference on next generation mobile applications, security and technologies (NGMAST), pp 47–52
Miljkovic D, Aleksovski D, Podpečan V, Lavrač N, Malle B, Holzinger A (2016) Machine learning and data mining methods for managing parkinson’s disease. In: Machine learning for health informatics: state-of-the-art and future challenges, pp 209–220
Munroe C, Meng Y, Yanco H, Begum M (2016) Augmented reality eyeglasses for promoting home-based rehabilitation for children with cerebral palsy. In: 11th ACM/IEEE International conference on human robot interaction, pp 565–565
Nielsen J, Molich R (1990) Heuristic evaluation of user interfaces. In: SIGCHI Conference on human factors in computing systems, pp 249–256
Oak JW, Bae JH (2014) Development of smart multiplatform game app using unity3d engine for cpr education. Int J Multimed Ubiquit Eng 9(7):263–268
Paterson RE (2015) Basics of human binocular vision. In: Human factors of stereoscopic 3D displays, pp 9–21
Pei W, Xu G, Li M, Ding H, Zhang S, Luo A (2016) A motion rehabilitation self-training and evaluation system using kinect. In: 13th International conference on ubiquitous robots and ambient intelligence (URAI), pp 353–357
Pellecchia MT, Grasso A, Biancardi LG, Squillante M, Bonavita V, Barone P (2004) Physical therapy in parkinson’s disease: an open long-term rehabilitation trial. J Neurol 251(5):595–598
Placidi G, Avola D, Iacoviello D, Cinque L (2013) Overall design and implementation of the virtual glove. Comput Biol Med 43(11):1927–1940
Placidi G, Avola D, Ferrari M, Iacoviello D, Petracca A, Quaresima V, Spezialetti M (2014) A low-cost real time virtual system for postural stability assessment at home. Comput Methods Programs Biomed 117(2):322–333
Rawat S, Vats S, Kumar P (2016) Evaluating and exploring the myo armband. In: International conference system modeling advancement in research trends (SMART), pp 115–120
Rego P, Moreira PM, Reis LP (2010) Serious games for rehabilitation: a survey and a classification towards a taxonomy. In: 5th Iberian conference on information systems and technologies (CISTI), pp 1–6
Saini S, Rambli DRA, Sulaiman S, Zakaria MN, Shukri SRM (2012) A low-cost game framework for a home-based stroke rehabilitation system. In: International conference on computer information science (ICCIS), pp 55–60
Sak H, Senior AW, Beaufays F (2014) Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Interspeech, pp 338–342
Sargano AB, Wang X, Angelov P, Habib Z (2017) Human action recognition using transfer learning with deep representations. In: International joint conference on neural networks (IJCNN), pp 463–469
Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61(Supplement C):85–117
Sen SL, **ang YB, Ming ESL, **ang KK, Fai YC, Khan QI (2015) Enhancing effectiveness of virtual reality rehabilitation system: durian runtuh. In: 10th Asian control conference (ASCC), pp 1–6
Shiratuddin MF, Hajnal A, Farkas A, Wong KW, Legradi G (2012) A proposed framework for an interactive visuotactile 3d virtual environment system for visuomotor rehabilitation of stroke patients. In: International conference on computer information science (ICCIS), pp 1052–1057
Singh D, Merdivan E, Psychoula I, Kropf J, Hanke S, Geist M, Holzinger A (2017) Human activity recognition using recurrent neural networks. In: International cross-domain conference on machine learning and knowledge extraction, pp 267–274
Sosa GD, Sánchez J, Franco H (2015) Improved front-view tracking of human skeleton from kinect data for rehabilitation support in multiple sclerosis. In: 20th Symposium on signal processing, images and computer vision (STSIVA), pp 1–7
Wasenmüller O, Stricker D (2016) Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In: Asian Conference on computer vision (ACCV), pp 34–45
Weiss PL, Rand D, Katz N, Kizony R (2004) Video capture virtual reality as a flexible and effective rehabilitation tool. J NeuroEng Rehabil 1(1):1–12
Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE Multimed 19(2):4–10
Zhang XY, **e GS, Liu CL, Bengio Y (2017) End-to-end online writer identification with recurrent neural network. IEEE Trans Human-Mach Syst 47 (2):285–292
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Avola, D., Cinque, L., Foresti, G.L. et al. VRheab: a fully immersive motor rehabilitation system based on recurrent neural network. Multimed Tools Appl 77, 24955–24982 (2018). https://doi.org/10.1007/s11042-018-5730-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-5730-1