SpringerLink
Log in

An empirical practice of design and evaluation of freehand interaction gestures in virtual reality

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Nowadays, virtual reality (VR) technologies are increasingly used in various domains. Thus, there is an urgent need to sufficiently explore a natural approach to human–computer interaction that can be customized to various situations across different user groups. In this study, we explored hand gesture types that are more effective and natural for manipulating three-dimensional (3D) objects for application in freehand interaction within a VR environment. More specifically, gestures for 3D manipulation including directional navigation, rotation, scaling, and teleportation were tracked and recognized through a Leap Motion sensor equipped in front of a head-mounted VR device and on top of 3D glasses, respectively. To systematically validate the efficiency, usability, and reliability of the hand gestures, we designed a series of representative tasks in a specifically designed VR application and then recruited 40 participants to complete all required interaction tasks using the newly designed gestures and baseline hand controller. The results showed that the gestural interaction method was sufficiently effective for accomplishing most of the interactive tasks, indicating that this method could be a viable alternative to the conventional hand controller interaction method. Moreover, users preferred the new gestural interaction method more than the conventional one in terms of user experience and immersion. Based on these results, guidelines and strategies were discussed for develo** freehand interaction techniques in general VR applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Achenbach J, Waltemate T, Latoschik ME, Botsch M (2017) Fast generation of realistic virtual humans. In: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST), pp 1–10. https://doi.org/10.1145/3139131.3139154

  2. Feng A, Suma E, Shapiro A (2017) Just-in-time, viable, 3d avatars from scans. In: ACM SIGGRAPH 2017 talks, Article no. 19. ACM, Los Angeles, pp 1–2. https://doi.org/10.1145/3084363.3085045

  3. Vosinakis S, Koutsabasis P (2018) Valuation of visual feedback techniques for virtual gras** with bare hands using Leap Motion and Oculus Rift. Virtual Real 22(1):47–62. https://doi.org/10.1007/s10055-017-0313-4

    Article  Google Scholar 

  4. Li Z, Jiang Y, Zhu Y et al (2022) Modeling the Noticeability of User-Avatar Movement Inconsistency for Sense of Body Ownership Intervention. Proc ACM Interact Mob Wearable Ubiquitous Technol: 12071. https://doi.org/10.1145/3534590

  5. Wu H, Zhang S, Liu J, Qiu J, Zhang X (2019) The gesture disagreement problem in free-hand gesture Interaction. Int J Human-Computer Interact 35(12):1102–1114. https://doi.org/10.1080/10447318.2018.1510607

    Article  Google Scholar 

  6. Sreenath S, Daniels DI, Ganesh ASD, Kuruganti YS, Chittawadigi RG (2021) Monocular tracking of human hand on a smart phone camera using mediapipe and its application in robotics. 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC), Bangalore, India pp 1–6. https://doi.org/10.1109/R10-HTC53172.2021.9641542

  7. Reipschlger P, Flemisch T, Dachselt R (2020) Personal augmented reality for information visualization on large interactive displays. IEEE Trans Vis Comput Graph 27(2):1182–1192

    Article  Google Scholar 

  8. Guerra-Segura E, Travieso CM, Alonso JB (2017) Study of the variability of the leap motion’s measures for its use to characterize air strokes. Measurement 105:87–97. https://doi.org/10.1016/j.measurement.2017.04.016

    Article  Google Scholar 

  9. Fh A, Va A, Kai LB, Ch A, Bp A (2021) Mma B estimating depth information of vascular models: a comparative user study between a virtual reality and a desktop application. Comput Graph 98:210–217. https://doi.org/10.1016/j.cag.2021.05.014

    Article  Google Scholar 

  10. Li B, Zhang C, Han C, Bai B (2019) Gesture recognition based on Kinect v2 and leap motion data fusion. Int J Pattern Recognit Artif Intell 33(5):1955005. https://doi.org/10.1142/S021800141955005X

    Article  Google Scholar 

  11. Ogdon DC (2019) Hololens and vive pro: virtual reality headsets. J Med Libr Assoc JMLA 107(1):118–121. https://doi.org/10.5195/JMLA.2019.602

    Article  Google Scholar 

  12. Facebook. Oculus Quest 2. Available online: https://www.oculus.com/quest-2/. Accessed 16 Mar 2021

  13. Tian F, Lyu F, Zhang XL, Ren X, Wang H (2017) An empirical study on the interaction capability of arm stretching. Int J Hum Comput Interact 33(7–9):565–575. https://doi.org/10.1080/10447318.2016.1265782

    Article  Google Scholar 

  14. Wu H, Luo W, Pan N, Nan S, Deng Y, Fu S et al (2019) Understanding freehand gestures: a study of freehand gestural interaction for immersive VR shop** applications. Human-centric Comput Inform Sci 9(1):1–26. https://doi.org/10.1186/s13673-019-0204-7

    Article  Google Scholar 

  15. Masurovsky A, Chojecki P, Runde D, Lafci M, Przewozny D, Gaebler M (2020) Controller-free hand tracking for grab-and-place tasks in immersive virtual reality: design elements and their empirical study. Multimodal Technol Interact 4:91. https://doi.org/10.3390/mti4040091

    Article  Google Scholar 

  16. Lee M, Kwahk J, Han SH, Lee H (2020) Relative pointing interface: a gesture interaction method based on the ability to divide space. Int J Ind Ergon 75:102878. https://doi.org/10.1016/j.ergon.2019.102878

    Article  Google Scholar 

  17. Venkatakrishnan R, Venkatakrishnan R, Chung CH, Wang YS, Babu S (2022) Investigating a combination of input modalities, canvas geometries, and inking triggers on on-air handwriting in virtual reality. ACM Trans Appl Percept 19(4):1–19. https://doi.org/10.1145/3560817

    Article  Google Scholar 

  18. Allgaier M, Amini A, Neyazi B, Sandalcioglu IE, Preim B, Saalfeld S (2021) Vr-based training of craniotomy for intracranial Aneurysm surgery. Int J Comput Assist Radiol Surg 17(3):449–456. https://doi.org/10.1007/s11548-021-02538-3

    Article  Google Scholar 

  19. Adkins A, Lin L, Normoyle A, Canales R, Ye Y, Jrg S (2021) Evaluating gras** visualizations and control modes in a vr game. ACM Trans Appl Percept 18(4):19. https://doi.org/10.1145/3486582

    Article  Google Scholar 

  20. Argelaguet F, Andujar C (2013) A survey of 3D object selection techniques for virtual environments. Comput Graph 37(3):121–136

    Article  Google Scholar 

  21. Schfer A, Reis G, Stricker D (2022) Comparing controller with the hand gestures pinch and grab for picking up and placing virtual objects. 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops. https://doi.org/10.1109/VRW55335.2022.00220

  22. Bowman DA, Hodges LF (1997) An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In: Proceedings of the 1997 symposium on Interactive 3D graphics 35

  23. Su GE, Sunar MS, Ismail AW (2020) Device-based manipulation technique with separated control structures for 3d object translation and rotation in handheld mobile ar. Int J Hum Comput Stud 141:102433. https://doi.org/10.1016/j.ijhcs.2020.102433

    Article  Google Scholar 

  24. Mendes D, Relvas F, Ferreira A, Jorge JA P (2016) The benefits of DOF separation in mid-air 3D object manipulation. VRST ‘16-22nd ACM Conference on Virtual Reality Software and Technology. ACM

  25. Gerber D, Bechmann D. The spin menu: a menu system for virtual environments. IEEE Conference on Virtual Reality. In Proceedings of the IEEE Conference on Virtual Reality, Bonn, Germany, 12–16 March 2005: 271–272. https://doi.org/10.1109/VR.2005.1492790

  26. **a Z, Oyekoya O Hao Tang (2022) Effective gesture-based user interfaces on mobile mixed reality. SUI ‘22 In: Proceedings of the 2022 ACM Symposium on Spatial User Interaction, pp 1–2. https://doi.org/10.1145/3565970.3568189

  27. Al Zayer M, MacNeilage P, Folmer E (2020) Virtual locomotion a survey. IEEE Trans Vis Comput Graph 26(6):2315–2334. https://doi.org/10.1109/TVCG.2018.2887379

    Article  Google Scholar 

  28. Paulo SF, Medeiros D, Lopes D, Jorge J (2022) Controlling camera movement in VR colonography. Virtual Real 26:1079–1088. https://doi.org/10.1007/s10055-021-00620-4

    Article  Google Scholar 

  29. Zhang F, Chu S, Pan R, Ji N, **, L (2017) Double hand-gesture interaction for walk-through in VR environment. In 2017 IEEE/ACIS 16th international conference on computer and information science (icis), pp 539–544. Wuhan, China.

  30. Caggianese G, Capece N, Erra U, Gallo L, Rinaldi M (2020) Freehand-steering locomotion techniques for immersive virtual environments: a comparative evaluation. Int J Hum Comput Interact 36:1734–1755. https://doi.org/10.1080/10447318.2020.1785151

    Article  Google Scholar 

  31. Poupyrev I, Ichikawa T, Weghorst S, Billinghurst M (1998) Egocentric object manipulation in virtual environments: empirical evaluation of interaction techniques. Comput Graphics Forum 17(3):41–52. https://doi.org/10.1111/1467-8659.00252

    Article  Google Scholar 

  32. Habgood M, Moore D, Wilson D, Alapont S. Rapid, continuous movement between nodes as an accessible virtual reality locomotion technique. In: Proceedings of the 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Tuebingen/Reutlingen, Germany, 18–22 March 2018: pp 371–378

  33. Medeiros D, Sousa A, Raposo A, Jorge JM (2019) Magic carpet: Interaction fidelity for flying in VR. IEEE Trans Vis Comput Graph 26:2793–2804. https://doi.org/10.1109/TVCG.2019.2905200

    Article  Google Scholar 

  34. Argelaguet F, Hoyet L, Trico M, Lécuyer A (2016) The role of interaction in virtual embodiment: effects of the virtual hand representation. In: IEEE Virtual Reality, Mar 2016, Greenville, United States. IEEE, pp 3–10. https://doi.org/10.1109/VR.2016.7504682

  35. Tran TQ, Shin H, Stuerzlinger W, Han J (2017) Effects of virtual arm representations on interaction in virtual environments. In: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology pp 1–9

  36. Venkatakrishnan R, Raveendranath B, Pagano CC, Robb AC, Lin WC, Babu SV (2023) How virtual hand representations affect the perceptions of dynamic affordances in virtual reality. IEEE Trans Vis Comput Graph 29(5):2258–2268

    Article  Google Scholar 

  37. Lougiakis C, Katifori A, Roussou M, Ioannidis IP (2020) Effects of virtual hand representation on interaction and embodiment in HMD-based virtual environments using controllers. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Atlanta, pp 510–518. https://doi.org/10.1109/VR46266.2020.00072

  38. Ismail AW, Billinghurst M, Sunar MS (2015) Vision-based technique and issues for multimodal interaction in augmented reality[C] ACM. In: 8th International Symposium on Visual Information Communication and Interaction (VINCI). ACM, Tokyo, pp 75–82. https://doi.org/10.1145/2801040.2801058

  39. Du MH, Cui H, Wang Y, Duh HBL (2023) Learning from deep stereoscopic attention for simulator sickness prediction. IEEE Trans Vis Comput Graphics 29(2):1415–1423. https://doi.org/10.1109/TVCG.2021.3115901

    Article  Google Scholar 

  40. https://developer.oculus.com/resources/hands-design-intro/

  41. Ankit C, Jagdish RL, Karen D, Sonia R (2011) Intelligent approaches to interact with machines using hand gesture recognition in natural way: a survey. Int J Comput Sci Eng Survey 2(1):122–133

  42. Lou ZH, Yin JB (2018) Analysis of the gesture design principles in human-computer interaction. Softw Guide 17(4):19–24

    Google Scholar 

  43. Ni T, Bowman DA, North C, Mcmahan RP (2011) Design and evaluation of freehand menu selection interfaces using tilt and pinch gestures. Int J Hum Comput Stud 69(9):551–562

    Article  Google Scholar 

  44. Ramadoss P, Rapetti L, Tirupachuri Y, Grieco R, Milani G, Valli E et al (2022) Whole-body human kinematics estimation using dynamical inverse kinematics and contact-aided lie group kalman filter. https://doi.org/10.48550/ar**v.2205.07835

  45. Frøkjær E, Hertzum M, Kasper (2000) Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? The SIGCHI conference. ACM, pp 345–352. https://doi.org/10.1145/332040.332455

  46. ISO I. 9241--11 (1998) Ergonomic requirements for work with visual display terminals (VDTs) - Part 11: Guidance on usability. CEN, Brussels

  47. Borg GA (1982) Psychophysical bases of perceived exertion. Med Sci Sports Exerc 14(5):377–381. https://doi.org/10.1249/00005768-198205000-00012

    Article  Google Scholar 

  48. Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Advances in psychology 52(6):139–183. https://doi.org/10.1016/S0166-4115(08)62386-9

  49. Brooke JSUS (2013) A retrospective. J Usability Stud 8(2):29–40. https://doi.org/10.1080/01440368008530710

    Article  Google Scholar 

  50. Wei LuZ, Tong ChuJ (2016) Dynamic hand gesture recognition with leap motion controller. IEEE Signal Process Lett 23(9):1188–1192. https://doi.org/10.1109/LSP.2016.2590470

    Article  Google Scholar 

Download references

Acknowledgements

We thank all reviewers for their insightful comments on this work. This research was supported by the National Natural Science Foundation of China under the grant number of “61902097”; the PRC Industry-University Collaborative Education Program under the grant number of “CES/Kingfar202209RYJG15”; the “Natural Science Foundation of Zhejiang Province” under the grant number of “LQ19F020010”; and the Zhejiang Provincial Philosophy and Social Science Planning Project under the grant number of “19NDQN301YB”.

Author information

Authors and Affiliations

  1. School of Media and Design, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China

    Ying Wu, Yigang Wang, **aolong Lou & Mingwei Zhang

  2. Jiaxing Nanhu University, Jiaxing, 314001, Zhejiang, China

    Ying Wu

  3. State Key Laboratory of Virtual Reality System and Technology, Beihang University, Bei**g, China

    **aolong Lou

Authors
  1. Ying Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yigang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. **aolong Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mingwei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yigang Wang.

Ethics declarations

Conflict of interest

The author has declared that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Below is the link to the electronic supplementary material.

(MP4 15.4 MB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Wang, Y., Lou, X. et al. An empirical practice of design and evaluation of freehand interaction gestures in virtual reality. Multimed Tools Appl 83, 52481–52507 (2024). https://doi.org/10.1007/s11042-023-17640-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17640-8

Keywords

Search

Navigation