SpringerLink
Log in

Understanding virtual drilling perception using sound, and kinesthetic cues obtained with a mouse and keyboard

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

A Correction to this article was published on 28 August 2023

This article has been updated

Abstract

This work examined whether a virtual drilling task can be appropriately simulated using combinations of sound, and kinesthetic stimuli obtained with movement of a basic 2D mouse in the absence of a haptic device. An experiment was conducted where participants were asked to drill a virtual block of wood under different sound conditions using a standard computer mouse, or keyboard. Although preliminary and greater work remains, the results of these experiments indicate that kinesthetic cues can be used to provide a suitable perceptual drilling experience without a haptic device. This preliminary work has provided a greater understanding of virtual drilling perception using traditional computer interfaces in the absence of haptic devices. This is particularly important when considering remote learning where trainees may be able to practice various psychomotor-based skills at home with commonly available computer hardware and devices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Change history

References

  1. Halamek LP, Kaegi DM, Gaba DM, Sowb YA, Smith BC, Smith BE, Howard SK (2000) Time for a new paradigm in pediatric medical education: Teaching neonatal resuscitation in a simulated delivery room environment. Pediatrics 106:E45

    Google Scholar 

  2. Yu P, Pan J, Qin H, Hao A, Wang H (2020) Real-time suturing simulation for virtual reality medical training. Comput Anim Virtual Worlds 31:e1940

    Google Scholar 

  3. Willaert WIM, Aggarwal R, Van Herzeele I, Cheshire NJ, Vermassen FE (2012) Recent advancements in medical simulation: patient-specific virtual reality simulation. World J Surg 36:1703–1712

    Google Scholar 

  4. Bloom BS (1994) Reflections on the development and use of the taxonomy. In: Rehage KJ, Anderson LW, Sosniak LA (eds) Bloom’s taxonomy: a forty-year retrospective. Yearbook of the national society for the study of education. University of Chicago Press, Chicago

    Google Scholar 

  5. Casa-Ortiz A, Echeverria J, Santos OC (2023) Intelligent systems for psychomotor learning: a systematic review and two case studies. In: du Boulay B, Mitrovic A, Yacef K (eds) Handbook of artificial intelligence in education. Elgar, Cheltenham

    Google Scholar 

  6. Hayward V, Astley OR, Cruz-Hernandez M, Grant D, Robles-De-La-Torre G (2004) Haptic interfaces and devices. Sens Rev 24:16–29

    Google Scholar 

  7. Giri GS, Maddahi Y, Zareinia K (2021) An application-based review of haptics technology. Robotics 10:29

    Google Scholar 

  8. Oakley I, McGee MR, Brewster S, Gray P (2000) Putting the feel in 'look and feel'. In Proc. ACM SIGCHI conference on Human Factors in Computing Systems, 415–422

  9. Pacchierotti C, Prattichizzo D, Kuchenbecker KJ (2016) Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery. IEEE Trans Biomed Eng 63:278–287

    Google Scholar 

  10. Zhou M, Tse S, Derevianko A, Jones DB, Schwaitzberg SD, Cao CG (2011) Effect of haptic feedback in laparoscopic surgery skill acquisition. Surg Endosc 26:1128–1134

    Google Scholar 

  11. Burbach S, Steiger A, Gießer C (2022) Suitability testing of the LucidGloves prototype 4 for extended reality medical teaching. Curr Dir Biomed Eng 8:419–422

    Google Scholar 

  12. Coles TR, Meglan D, John NW (2011) The role of haptics in medical training simulators: a survey of the state of the art. IEEE Trans Haptics 4:51–66

    Google Scholar 

  13. Galdieri R, Camardella C, Carrozzino M, Frisoli A (2022) Designing multi-purpose devices to enhance users’ perception of haptics. J Multimodal User Interfaces 16:335–342

    Google Scholar 

  14. Hamza-Lup FG, Goldbach IR (2021) Multimodal, visuo-haptic games for abstract theory instruction: grabbing charged particles. J Multimodal User Interfaces 15:1–10

    Google Scholar 

  15. Våpenstad C, Hofstad EF, Lang T, Mårvik R, Chmarra MK (2013) Perceiving haptic feedback in virtual reality simulators. Surg Endosc 27:2391–2397

    Google Scholar 

  16. Chen X, Sun P, Liao D (2018) A patient-specific haptic drilling simulator based on virtual reality for dental implant surgery. Int J CARS 13:1861–1870

    Google Scholar 

  17. Karbasizadeh N, Zarei M, Aflakian A, Tale Masouleh MA, Kalhor A (2018) Experimental dynamic identification and model feed-forward control of Novint Falcon haptic device. Mechatronics 51:19–30

    Google Scholar 

  18. Kapralos B, Collins K, Uribe-Quevedo A (2017) The senses and virtual environments. Senses Soc 12:69–75

    Google Scholar 

  19. Shams L, Kim R (2010) Crossmodal influences on visual perception. Phys Life Rev 7:269–284

    Google Scholar 

  20. Obrenovic Z, Starcevic D (2004) Modeling multimodal human-computer interaction. Computer 37:65–72

    Google Scholar 

  21. Azofeifa JD, Noguez J, Ruiz S, Molina-Espinosa JM, Magana AJ, Benes B (2022) Systematic review of multimodal human–computer interaction. Informatics 9:13

    Google Scholar 

  22. Crowely J (2018) Put that there: 20 years of research on multimodal interaction. In Proc. 20th ACM Int. Conf. Multimodal Interaction, Boulder, CO, USA

  23. Jaimes A, Sebe N (2006) Multimodal human–computer interaction: A survey. Comput Vis Image Underst 108:116–134

    Google Scholar 

  24. Schölkopf L, Lorenz M, Stamer M, Albrecht L, Klimant P, Hammer N, Tümler J (2021) Haptic feedback is more important than VR experience for the user experience assessment of in-car human machine interfaces. In: Proc. 31st CIRP Design Conference 2021 (CIRP Design 2021), pp.601–606

  25. Kapralos B, Moussa F, Collins K, Dubrowski A (2017) Levels of fidelity and multimodal interactions. In: Wouters P, van Oostendorp H (eds) Techniques to improve the effectiveness of serious games, Springer advances in game-based learning book series. Springer, Cham, pp 79–101

    Google Scholar 

  26. Kinoshita H, Nagahata M, Takano N, Takemoto S, Matsunaga S, Abe S, Yoshinari M, Kawada E (2016) Development of a drilling simulator for dental implant surgery. J Dent Educ 80:83–90

    Google Scholar 

  27. Wang Q, Qin J, Wang W, Shan J, Zhang J, Liu X, Heng P (2015) Haptic rendering of drilling process in orthopedic surgical simulation based on the volumetric object. In: Proc. 2015 IEEE Int. Conference on Digital Signal Processing, pp. 1098–1101

  28. Corrêa CG, Tokunaga DM, Ranzini E, Nunes FLS, Tori R (2016) Haptic interaction objective evaluation in needle insertion task simulation. In: Proc. 31st Annual ACM Symposium on Applied Computing, pp. 149–154

  29. Cucinotta D, Vanelli M (2020) WHO declares COVID-19 a pandemic. Acta Biomed 91:157–160

    Google Scholar 

  30. Lee J, Gozen BA, Ozdoganlar OB (2012) Modeling and experimentation of bone drilling forces. J Biomech 45:1076–1083

    Google Scholar 

  31. Wu G, Yu D, Wang Y, Zhang Y, Wang CCL (2009) Voxel-based interactive haptic simulation of dental drilling. In: Proc. 29th Computers and Information in Engineering Conference, pp. 39–48

  32. Parsian A, Magnevall M, Beno T, Eynian M (2017) Sound analysis in drilling, frequency and time domains. Procedia CIRP 58:411–415

    Google Scholar 

  33. Nguyen M, Melaisi M, Cowan B, Uribe Quevedo AJ, Kapralos B (2017) Low-end haptic devices for knee bone drilling in a serious game. World J Sci Technol Sustain Dev 14:241–253

    Google Scholar 

  34. van Dongen KW, van der Wal WA, Borel Rinkes IHM, Schijven MP, Broeders IAMJ (2008) Virtual reality training for endoscopic surgery: voluntary or obligatory? Surg Endosc 22:664–667

    Google Scholar 

  35. Grant BL, Yielder PC, Patrick TA, Kapralos B, Williams-Bell M, Murphy BA (2019) Audiohaptic feedback enhances motor performance in a low-fidelity simulated drilling task. Brain Sci 10:21

  36. Bartlett JD, Lawrence JE, Stewart ME, Nakano N, Khanduja V (2018) Does virtual reality simulation have a role in training trauma and orthopaedic surgeons? Bone Jt J 100:559–565

    Google Scholar 

  37. **a P, Lopes AM, Restivo MT (2013) Virtual reality and haptics for dental surgery: a personal review. Vis Comput 29:433–447

  38. Steinberg AD, Bashook PG, Drummond J, Ashrafi S, Zefran M (2007) Assessment of faculty perception of content validity of PerioSim, a haptic-3D virtual reality dental training simulator. J Dent Educ 71:1574–1582

    Google Scholar 

  39. Kaluschke M, Weller R, Hammer N, Pelliccia L, Lorenz M, Zachmann G (2020) Realistic haptic feedback for material removal in medical simulations. In: Proc. 2020 IEEE Haptics Symposium, pp. 920–926

  40. Khwanngern K, Tiangtae N, Natwichai J, Kattiyanet A Kaveeta V, Sitthikham S, Kammabut K (2020) Jaw surgery simulation in virtual reality for medical training. In: Proc. Int. Conference on Network-Based Information Systems, pp. 475–483

  41. Ha-Van Q, Schwendinger H, Kim Y, Harders M (2020) Design and characterization of an actuated drill mockup for orthopedic surgical training. IEEE Trans Haptics 13:655–667

    Google Scholar 

  42. Wen Necsulescu KD, Sasiadek J (2005) Haptic force control based on impedance/admittance control. In: IFAC Proceedings, vol. 38, pp. 427–432

  43. Skarbez R, Brooks FP Jr, Whitton MC (2018) A survey of presence and related concepts. ACM Comput Surv 50:1–39

    Google Scholar 

  44. Storms SL, Zyda MJ (2000) Interactions in perceived quality of auditory-visual displays. Presence Teleoperators Virtual Environ 9:557–580

  45. Mastoropoulou G, Debattista K, Chalmers A, Troscianco T (2005) The influence of sound effects on the perceived smoothness of rendered animations. In: Proc. of the 2nd Symposium on Applied Perception in Graphics and Visualization, pp. 9–15

  46. Hulusic V, Debattista K, Aggarwal V, Chalmers A (2011) Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction. Visual Comput 27:57–66

    Google Scholar 

  47. Bonneel N, Suied C, Viaud-Delmon I, Drettakis G (2010) Bimodal perception of audio-visual material properties for virtual environments. ACM Trans Appl Percept 7:1–16

    Google Scholar 

  48. Larsson P, Västjäll D, Kleiner M (2003) On the quality of experience: A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. In: Proc. 1st International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems

  49. Machemehl M, Michaels J, Tümler J (2020) Investigating the benefits of haptic feedback during in-car interactions in virtual reality. In: Proc. Int. Conference on Human-Computer Interaction: HCI in Mobility, Transport, and Automotive Systems. Automated Driving and In-Vehicle Experience Design, pp 404–416

  50. Melaisi M, Nguyen M, Quevedo A, Kapralos B (2017) The effect of sound on haptic fidelity perception. In: Proc. IEEE Global Engineering Education Conference, pp. 714–717

  51. Moreno D, Melaisi M, MacDonald S, Uribe-Quevedo A, Kapralos B, Martin MV (2017) A brain-computer interface to examine the effects of sound on a haptic-based virtual drilling task. In: Proc. 8th IEEE Int. Conference on Information, Intelligence, Systems & Applications (IISA), pp. 1–4

  52. Melaisi M, Rojas D, Kapralos B, Uribe-Quevedo A, Collins K (2018) Multimodal interaction of contextual and non-contextual sound and haptics in virtual simulations. Informatics 5:1–13

    Google Scholar 

  53. Rakkolainen I, Freeman E, Sand A, Raisamo R, Brewster S (2021) A survey of mid-air ultrasound haptics and its applications. IEEE Trans Haptics 14:2–19

    Google Scholar 

  54. Lécuyer A, Coquillart S, Kheddar A, Richard P, Coiffet P (2000) Pseudo-haptic feedback: Can isometric input devices simulate force feedback? In: Proc. IEEE Virtual Reality Conference, pp. 83–90

  55. Aldridge R, Carr KT, England R, Meech JF, Solomonides T (1996) Getting a grasp on virtual reality, In: Proc. Int. Conference on Computer Human Interfaces, pp. 229–230

  56. Lécuyer A, Burkhardt J-M, Etienne L (2004) Feeling bumps and holes without a haptic interface. In: Proc. Conference on Human Factors in Computing Systems, pp. 239–246

  57. Dominjon L, Lecuyer A, Burkhardt J.-M, Richard P, Richir S (2005) Influence of control/display ratio on the perception of mass of manipulated objects in virtual environments. In: Proc. IEEE Virtual Reality Conference, pp. 19–25

  58. Collins K, Kapralos B (2019) Pseudo-haptics: a review of leveraging cross-modal perception in virtual environments. Senses Soc 14:313–329

    Google Scholar 

  59. Faul F, Erdfelder E, Buchner A, Lang A-G (2009) Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods 41:1149–1160

    Google Scholar 

  60. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Mahwah, p 1988

    Google Scholar 

  61. Al-Balas M, Al-Balas HL, Jaber HM, Obeidat K, Al-Balas H, Aborajooh EA, Al-Taher R, Al-Balas B (2020) Distance learning in clinical medical education amid COVID-19 pandemic in Jordan: Current situation, challenges, and perspectives. BMC Med Educ 20:1–7

    Google Scholar 

  62. Maclntyre B, Smith TF (2018) Thoughts on the future of WebXR and the immersive web, In: Proc. 2018 IEEE Int. Symposium on Mixed and Augmented Reality Adjunct, pp. 338–342

  63. Schrepp M, Hinderks A, Thomaschewski J (2014) Applying the user experience questionnaire (UEQ) in different evaluation scenarios. In: Marcus A (ed) Design, user experience, and usability. Theories, methods, and tools for designing the user experience. Springer, Cham, pp 383–392

    Google Scholar 

  64. Brockmyer JH, Fox CM, Curtiss KA, McBroom E, Burkhart KM, Pidruzny JN (2009) The development of the game engagement questionnaire: A measure of engagement in video game-playing. J Exp Soc Psychol 45:624–634

    Google Scholar 

Download references

Acknowledgements

This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) in the form of individual Discovery Grants to B. Kapralos and A. Quevedo, a Research Excellence Chair from Ontario Tech University to B. Kapralos, and by the Research Center for Biomedical Engineering and Research Institute of Electronics, Shizuoka University, Japan.

Author information

Authors and Affiliations

  1. Faculty of Business and Information Technology and maxSIMhealth Group, Ontario Tech University, 2000 Simcoe Street North, Oshawa, ON, L1H 7K4, Canada

    Guoxuan Ning, Brianna Grant, Bill Kapralos, Alvaro Quevedo & Adam Dubrowski

  2. Faculty of Sciences, Ontario Tech University, Oshawa, ON, Canada

    Guoxuan Ning

  3. Faculty of Health Sciences, Ontario Tech University, Oshawa, ON, Canada

    Brianna Grant & Adam Dubrowski

  4. School of Information Technology, Carleton University, Ottawa, ON, Canada

    KC Collins

  5. Research Institute of Electronics, Shizuoka University, Hamamatsu, Japan

    Kamen Kanev

Authors
  1. Guoxuan Ning
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brianna Grant
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bill Kapralos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alvaro Quevedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. KC Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kamen Kanev
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Adam Dubrowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bill Kapralos.

Ethics declarations

Conflict of interest

Each of the seven authors declare that they have no conflict of interest and no financial interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised to correct the fifth author first name and minor typographical errors.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ning, G., Grant, B., Kapralos, B. et al. Understanding virtual drilling perception using sound, and kinesthetic cues obtained with a mouse and keyboard. J Multimodal User Interfaces 17, 151–163 (2023). https://doi.org/10.1007/s12193-023-00407-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-023-00407-8

Keywords

Search

Navigation