Abstract
Immersive entertainment is an important driver of mixed reality platforms. As technology becomes better and cheaper, this will spark the next major shift in digital entertainment. We identify nine research topics on immersive entertainment across theoretical-scientific, creative, systems, and social dimensions. For each topic, we discuss possible research directions that can lead to advances in the field. This chapter is an extended version of a paper originally published in the proceedings of the XX Brazilian Symposium on Computer Games and Digital Entertainment. Each of the topics is discussed in more depth and a separate challenge for input devices and body tracking was added.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aeschbach, L.F., Perrig, S.A., Weder, L., Opwis, K., Brühlmann, F.: Transparency in measurement reporting: a systematic literature review of CHI PLAY. Proc. ACM Hum.-Comput. Interact. 5(CHI PLAY), 1–21 (2021)
Ahmad, A., Migniot, C., Dipanda, A.: Tracking hands in interaction with objects: a review. In: 2017 13th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pp. 360–369. IEEE (2017)
Alabdulkarim, A., Li, S., Peng, X.: Automatic story generation: challenges and attempts. ar**v preprint ar**v:2102.12634 (2021)
Alcaniz Raya, M., Marín-Morales, J., Minissi, M.E., Teruel Garcia, G., Abad, L., Chicchi Giglioli, I.A.: Machine learning and virtual reality on body movements’ behaviors to classify children with autism spectrum disorder. J. Clin. Med. 9(5), 1260 (2020)
Alkomah, F., Ma, X.: A literature review of textual hate speech detection methods and datasets. Information 13(6), 273 (2022)
Ashktorab, Z., Vitak, J.: Designing cyberbullying mitigation and prevention solutions through participatory design with teenagers. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 3895–3905 (2016)
Attal, B., Huang, J.B., Zollhoefer, M., Kopf, J., Kim, C.: Learning neural light fields with ray-space embedding networks. ar**v preprint ar**v:2112.01523 (2021)
Azuma, R.: Fundamentals of wearable computers and augmented reality (2015)
Bernal, G., Hidalgo, N., Russomanno, C., Maes, P.: Galea: a physiological sensing system for behavioral research in virtual environments. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 66–76. IEEE (2022)
Blaga, A.D., Frutos-Pascual, M., Creed, C., Williams, I.: Freehand gras**: an analysis of gras** for docking tasks in virtual reality. In: 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 749–758. IEEE (2021)
Blaga, A.D., Frutos-Pascual, M., Creed, C., Williams, I.: A grasp on reality: understanding gras** patterns for object interaction in real and virtual environments. In: 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 391–396. IEEE (2021)
Bowman, D., Koller, D., Hodges, L.: Travel in immersive virtual environments: an evaluation of viewpoint motion control techniques. In: Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality, pp. 45–52 (1997). https://doi.org/10.1109/VRAIS.1997.583043
Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: An introduction to 3-D user interface design. Presence 10(1), 96–108 (2001)
Breyer, F., et al.: Narrative-led interaction techniques. In: Rouse, R., Koenitz, H., Haahr, M. (eds.) ICIDS 2018. LNCS, vol. 11318, pp. 217–229. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04028-4_22
Brockmyer, J.H., Fox, C.M., Curtiss, K.A., McBroom, E., Burkhart, K.M., Pidruzny, J.N.: The development of the game engagement questionnaire: a measure of engagement in video game-playing. J. Exp. Soc. Psychol. 45(4), 624–634 (2009)
Burns, A., Tulip, J.: Detecting flow in games using facial expressions. In: 2017 IEEE Conference on Computational Intelligence and Games (CIG), pp. 45–52. IEEE (2017)
Cavazza, M., Aylett, R., Dautenhahn, K., Fencott, C., Charles, F.: Interactive storytelling in virtual environments: building the “holodeck”. In: Proceedings of VSMM, pp. 678–687. Citeseer (2000)
Chang, E., Kim, H.T., Yoo, B.: Virtual reality sickness: a review of causes and measurements. Int. J. Hum.-Comput. Interact. 36(17), 1658–1682 (2020)
Choi, I., Ofek, E., Benko, H., Sinclair, M., Holz, C.: Claw: a multifunctional handheld haptic controller for gras**, touching, and triggering in virtual reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2018)
Cukor, J., Spitalnick, J., Difede, J., Rizzo, A., Rothbaum, B.O.: Emerging treatments for PTSD. Clin. Psychol. Rev. 29(8), 715–26 (2009). https://doi.org/10.1016/j.cpr.2009.09.001
Dadvar, M., De Jong, F.: Cyberbullying detection: a step toward a safer internet yard. In: Proceedings of the 21st International Conference on World Wide Web, pp. 121–126 (2012)
De Guzman, J.A., Thilakarathna, K., Seneviratne, A.: Security and privacy approaches in mixed reality: a literature survey. ACM Comput. Surv. 52(6), 1–37 (2019)
Dekker, A., Champion, E.: Please biofeed the zombies: enhancing the gameplay and display of a horror game using biofeedback. In: DiGRA 2007-Proceedings of the 2007 DiGRA International Conference: Situated Play, pp. 550–558 (2007)
Denisova, A., Nordin, A.I., Cairns, P.: The convergence of player experience questionnaires. In: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, pp. 33–37. ACM (2016)
Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., Bhamidipati, N.: Hate speech detection with comment embeddings. In: Proceedings of the 24th International Conference on World Wide Web, pp. 29–30 (2015)
Egliston, B., Carter, M.: Oculus imaginaries: the promises and perils of Facebook’s virtual reality. New Media & Society, p. 1461444820960411 (2020)
Figueiredo, L.S., Livshits, B., Molnar, D., Veanes, M.: Prepose: privacy, security, and reliability for gesture-based programming. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 122–137. IEEE (2016)
Freeman, G., Zamanifard, S., Maloney, D., Acena, D.: Disturbing the peace: experiencing and mitigating emerging harassment in social virtual reality. Proc. ACM Hum.-Comput. Interact. 6(CSCW1), 1–30 (2022)
Freeman, J., Avons, S.E., Meddis, R., Pearson, D.E., IJsselsteijn, W.: Using behavioral realism to estimate presence: a study of the utility of postural responses to motion stimuli. Presence: Teleoperators Virtual Environ. 9(2), 149–164 (2000)
Gafni, G., Thies, J., Zollhofer, M., Nießner, M.: Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8649–8658 (2021)
Grassini, S., Laumann, K.: Questionnaire measures and physiological correlates of presence: a systematic review. Front. Psychol. 11, 349 (2020)
Halbig, A., Latoschik, M.E.: A systematic review of physiological measurements, factors, methods, and applications in virtual reality. Front. Virtual Real. 2, 694567 (2021)
Hameed, A., Perkis, A.: Spatial storytelling: finding interdisciplinary immersion. In: Rouse, R., Koenitz, H., Haahr, M. (eds.) ICIDS 2018. LNCS, vol. 11318, pp. 323–332. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04028-4_35
Hampali, S., Sarkar, S.D., Rad, M., Lepetit, V.: Handsformer: keypoint transformer for monocular 3D pose estimation of hands and object in interaction. ar**v preprint ar**v:2104.14639 (2021)
Houzangbe, S., Christmann, O., Gorisse, G., Richir, S.: Fear as a biofeedback game mechanic in virtual reality: effects on engagement and perceived usability. In: Proceedings of the 13th International Conference on the Foundations of Digital Games, pp. 1–6 (2018)
IJsselsteijn, W.A., De Kort, Y.A., Poels, K.: The game experience questionnaire. Tech. rep., Technische Universiteit Eindhoven (2013)
Ishaque, S., Rueda, A., Nguyen, B., Khan, N., Krishnan, S.: Physiological signal analysis and classification of stress from virtual reality video game. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 867–870. IEEE (2020)
Jackson, S.A., Eklund, R.C.: Assessing flow in physical activity: the flow state scale-2 and dispositional flow scale-2. J. Sport Exerc. Psychol. 24(2), 133–150 (2002)
Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)
Lages, W.S.: Opportunities and challenges in immersive entertainment. Anais Estendidos do XX Simpósio Brasileiro de Jogos e Entretenimento Digital, pp. 943–946 (2021)
Lang, B.: In ‘horizon’ Facebook can invisibly observe users in real-time to spot rule violations. https://www.roadtovr.com/facebook-horizon-privacy-monitoring-moderation/
Lattas, A., et al.: Avatarme: realistically renderable 3D facial reconstruction “in-the-wild”. In: CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE (2020)
Laugwitz, B., Held, T., Schrepp, M.: Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (ed.) USAB 2008. LNCS, vol. 5298, pp. 63–76. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89350-9_6
LaViola, Jr, J.J., Kruijff, E., McMahan, R.P., Bowman, D., Poupyrev, I.P.: 3D user interfaces: theory and practice, 2nd edn. Addison-Wesley Professional (2017)
Liu, L., Habermann, M., Rudnev, V., Sarkar, K., Gu, J., Theobalt, C.: Neural actor: neural free-view synthesis of human actors with pose control. ACM Trans. Graph. 40(6), 1–16 (2021)
Lombard, M., Ditton, T.: At the heart of it all: the concept of presence. J. Comput.-Mediat. Commun. 3(2), JCMC321 (1997)
Maloney, D., Freeman, G., Wohn, D.Y.: “Talking without a voice’’ understanding non-verbal communication in social virtual reality. Proc. ACM Hum.-Comput. Interact. 4(CSCW2), 1–25 (2020)
Masurovsky, A., Chojecki, P., Runde, D., Lafci, M., Przewozny, D., Gaebler, M.: Controller-free hand tracking for grab-and-place tasks in immersive virtual reality: design elements and their empirical study. Multimodal Technol. Interact. 4(4), 91 (2020)
Matamoros-Fernández, A., Farkas, J.: Racism, hate speech, and social media: a systematic review and critique. Telev. New Media 22(2), 205–224 (2021)
Meehan, M., Insko, B., Whitton, M., Brooks, F.P., Jr.: Physiological measures of presence in stressful virtual environments. ACM Trans. Graph. 21(3), 645–652 (2002)
Mehrabi, A., Siekkinen, M., Kämäräinen, T., yl̈ J̈ski, A.: Multi-tier cloudvr: leveraging edge computing in remote rendered virtual reality. ACM Trans. Multimed. Comput. Commun. Appl. 17(2), 1–24 (2021)
Meta: Notice of monitoring and recording to improve safety in horizon worlds. https://store.facebook.com/legal/quest/monitoring-recording-safety-horizon/
Miller, M.R., Herrera, F., Jun, H., Landay, J.A., Bailenson, J.N.: Personal identifiability of user tracking data during observation of 360-degree VR video. Sci. Rep. 10(1), 1–10 (2020)
Murray, J.H.: Hamlet on the Holodeck, Updated Edition: The Future of Narrative in Cyberspace. MIT Press, Cambridge (2017)
Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L.: Biofeedback game design: using direct and indirect physiological control to enhance game interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 103–112 (2011)
NewZoo: The global games market reaches \$99.6 billion in 2016, mobile generating 37%. https://newzoo.com/insights/articles/global-games-market-reaches-99-6-billion-2016-mobile-generating-37. Accessed 01 Sep 2021
Oh Kruzic, C., Kruzic, D., Herrera, F., Bailenson, J.: Facial expressions contribute more than body movements to conversational outcomes in avatar-mediated virtual environments. Sci. Rep. 10(1), 1–23 (2020)
Oriti, D., Manuri, F., Pace, F.D., Sanna, A.: Harmonize: a shared environment for extended immersive entertainment. Virtual Real. 2021, 1–14 (2021). https://doi.org/10.1007/s10055-021-00585-4
Park, J., Park, H., Yoon, S.E., Woo, W.: Physically-inspired deep light estimation from a homogeneous-material object for mixed reality lighting. IEEE Trans. Visual Comput. Graphics 26(5), 2002–2011 (2020)
Pater, J.A., Kim, M.K., Mynatt, E.D., Fiesler, C.: Characterizations of online harassment: comparing policies across social media platforms. In: Proceedings of the 19th International Conference on Supporting Group Work, pp. 369–374 (2016)
Patten, T., Park, K., Leitner, M., Wolfram, K., Vincze, M.: Object learning for 6D pose estimation and gras** from RGB-D videos of in-hand manipulation. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4831–4838. IEEE (2021)
Phan, M.H., Keebler, J.R., Chaparro, B.S.: The development and validation of the game user experience satisfaction scale (GUESS). Hum. Factors 58(8), 1217–1247 (2016)
Razzaque, S., Kohn, Z., Whitton, M.C.: Redirected walking. In: ACM (ed.) Proceedings of Eurographics, pp. 289–294. Citeseer (2001)
Sanghrajka, R., Young, R.M., Salisbury, B., Lang, E.W.: ShowRunner: a tool for storyline execution/visualization in 3D game environments. In: Cardona-Rivera, R.E., Sullivan, A., Young, R.M. (eds.) ICIDS 2019. LNCS, vol. 11869, pp. 323–327. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33894-7_32
Schrepp, M., Hinderks, A., Thomaschewski, J.: Design and evaluation of a short version of the user experience questionnaire (UEQ-S). Int. J. Interact. Multim. Artifi. Intell. 4(6), 103–108 (2017)
Segovia, K.Y., Bailenson, J.N.: Virtually true: children’s acquisition of false memories in virtual reality. Media Psychol. 12(4), 371–393 (2009)
Sekhavat, Y.A., Sisi, M.J., Roohi, S.: Affective interaction: using emotions as a user interface in games. Multimed. Tools Appl. 80(4), 5225–5253 (2021)
Shin, J.E., Kim, H., Parker, C., Kim, H.I., Oh, S., Woo, W.: Is any room really ok? The effect of room size and furniture on presence, narrative engagement, and usability during a space-adaptive augmented reality game. In: 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 135–144. IEEE (2019)
Siler, C., Ware, S.: A good story is one in a million: solution density in narrative generation problems. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 16, pp. 123–129 (2020)
Skarbez, R., Brooks, F.P., Jr., Whitton, M.C.: A survey of presence and related concepts. ACM Comput. Surv. 50(6), 1–39 (2017)
Slater, M.: A note on presence terminology. Presence Connect 3(3), 1–5 (2003)
Slater, M.: How colorful was your day? Why questionnaires cannot assess presence in virtual environments. Presence 13(4), 484–493 (2004)
Slater, M., et al.: The ethics of realism in virtual and augmented reality. Front. Virtual Real. 1, 1 (2020)
Souza, V., Maciel, A., Nedel, L., Kopper, R.: Measuring presence in virtual environments: a survey. ACM Comput. Surv. 54(8), 1–37 (2021)
Stauffert, J.P., Niebling, F., Latoschik, M.E.: Latency and cybersickness: impact, causes, and measures: a review. Front. Virtual Real. 1, 582204 (2020)
Steed, A., Takala, T.M., Archer, D., Lages, W., Lindeman, R.W.: Directions for 3D user interface research from consumer VR games. IEEE Trans. Visual Comput. Graphics 27(11), 4171–4182 (2021). https://doi.org/10.1109/TVCG.2021.3106431
Svensson, T.: Using VR to simulate interactable AR storytelling. In: Cardona-Rivera, R.E., Sullivan, A., Young, R.M. (eds.) ICIDS 2019. LNCS, vol. 11869, pp. 328–332. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33894-7_33
Szczurowski, K., Smith, M.: Measuring presence: hypothetical quantitative framework. In: 2017 23rd International Conference on Virtual System & Multimedia (VSMM), pp. 1–8. IEEE (2017)
Szolin, K., Kuss, D., Nuyens, F., Griffiths, M.: Gaming disorder: a systematic review exploring the user-avatar relationship in videogames. Comput. Hum. Behav. 128, 107124 (2022)
Tewari, A., et al.: Advances in neural rendering. In: Computer Graphics Forum, vol. 41, pp. 703–735. Wiley Online Library (2022)
Thomas, K., et al.: Sok: hate, harassment, and the changing landscape of online abuse. In: 2021 IEEE Symposium on Security and Privacy (SP), pp. 247–267. IEEE (2021)
Tseng, W.J., et al.: The dark side of perceptual manipulations in virtual reality. In: CHI Conference on Human Factors in Computing Systems, pp. 1–15 (2022)
Usoh, M., Catena, E., Arman, S., Slater, M.: Using presence questionnaires in reality. Presence: Teleoperators Virtual Environ. 9(5), 497–503 (2000)
Voigt-Antons, J.N., Kojic, T., Ali, D., Möller, S.: Influence of hand tracking as a way of interaction in virtual reality on user experience. In: 2020 Twelfth International Conference on Quality of Multimedia Experience, pp. 1–4. IEEE (2020)
Vorderer, P., et al.: MEC spatial presence questionnaire. Retrieved Sept 18, 2015 (2004)
Wee, C., Yap, K.M., Lim, W.N.: Haptic interfaces for virtual reality: challenges and research directions. IEEE Access 9, 112145–112162 (2021)
Weech, S., Kenny, S., Lenizky, M., Barnett-Cowan, M.: Narrative and gaming experience interact to affect presence and cybersickness in virtual reality. Int. J. Hum. Comput. Stud. 138, 102398 (2020)
Wienrich, C., Latoschik, M.E.: Extended artificial intelligence: new prospects of human-AI interaction research. ar**v preprint ar**v:2103.15004 (2021)
Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence questionnaire. Presence: Teleoperators Virtual Environ. 7(3), 225–240 (1998)
Wolf, D., Rogers, K., Kunder, C., Rukzio, E.: JumpVR: jump-based locomotion augmentation for virtual reality. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2020)
Yee, N., Bailenson, J.N.: The Proteus effect: the effect of transformed self-representation on behavior. Hum. Commun. Res. 33(3), 271–290 (2007). https://doi.org/10.1111/j.1468-2958.2007.00299.x
Yu, R., et al.: Bookshelf and bird: enabling real walking in large VR spaces through cell-based redirection. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI), pp. 116–119. IEEE (2017)
Zhang, L., Oney, S.: Flowmatic: an immersive authoring tool for creating interactive scenes in virtual reality. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 342–353 (2020)
Zhao, X., Pang, Y., Yang, J., Zhang, L., Lu, H.: Multi-source fusion and automatic predictor selection for zero-shot video object segmentation. ar**v preprint ar**v:2108.05076 (2021)
Zhu, K., Chen, T., Han, F., Wu, Y.S.: Haptwist: creating interactive haptic proxies in virtual reality using low-cost twistable artefacts. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, p. 693. ACM (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lages, W.S. (2023). Nine Challenges for Immersive Entertainment. In: Santos, R.P.d., Hounsell, M.d.S. (eds) Grand Research Challenges in Games and Entertainment Computing in Brazil - GranDGamesBR 2020–2030. GranDGamesBR GranDGamesBR 2020 2021. Communications in Computer and Information Science, vol 1702. Springer, Cham. https://doi.org/10.1007/978-3-031-27639-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-27639-2_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27638-5
Online ISBN: 978-3-031-27639-2
eBook Packages: Computer ScienceComputer Science (R0)