Abstract
Chatbots are increasing their relevance in the global market. Nonetheless, companies are still struggling to develop chatbots that provide their clients with an optimal experience and, so far, few insights have been obtained to improve their related User Experience (UX). This study investigates whether chatbots that consider users’ gender identity result in an improved UX, and whether sensitivity towards this social matter moderates this relationship. Therefore, a one-factor within-subjects experiment was conducted, involving participants interacting with two versions of a buying-assistant chatbot. In one condition, the chatbot provided a more personalised interaction by presuming the user’s gender identity by their sex assigned at birth and conversing with them using a gender-specific language (e.g., ‘women’s clothing’, ‘men’s clothing’). The second condition, instead, used a gender-neutral approach, using gender-neutral language (e.g., ‘clothing’). We hypothesised that the chatbot presuming a cisgender identity of the user would result in a higher UX than the gender-neutral chatbot, and that this effect would be more substantial for users who score low on gender sensitivity. UX was measured by evaluating the chatbot’s Usability, Empathy and Supportiveness. Results indicate that the chatbot presuming a cisgender identity was considered significantly more usable and supportive, but less empathetic. A moderation effect of gender sensitivity on the evaluation of the chatbots was not found.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Følstad, A., Brandtzæg, P.B.: Chatbots and the new world of HCI. Interactions. 24, 38–42 (2017). https://doi.org/10.1145/3085558
Van Pinxteren, M.M.E., Pluymaekers, M., Lemmink, J.G.A.M.: Human-like communication in conversational agents: a literature review and research agenda. JOSM. 31, 203–225 (2020). https://doi.org/10.1108/JOSM-06-2019-0175
Grudin, J., Jacques, R.: Chatbots, Humbots, and the Quest for Artificial General Intelligence. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–11. ACM, Glasgow, Scotland, UK (2019)
Juniper Research: Chatbots to facilitate $142 billion of retail spend by 2024, driven by omnichannel strategies. https://www.juniperresearch.com/press/chatbots-to-facilitate-$142-billion-of-retail
Gnewuch, U., Morana, S., Maedche, A.: Towards designing cooperative and social conversational agents for customer service. Presented at the 38th International Conference on Information Systems, Seoul (2017)
Ait Baha, T., El Hajji, M., Es-Saady, Y., Fadili, H.: The power of personalization: a systematic review of personality-adaptive chatbots. Sn Comput. Sci. 4, 661 (2023). https://doi.org/10.1007/s42979-023-02092-6
Daniel, F., Matera, M., Zaccaria, V., Dell’Orto, A.: Toward truly personal chatbots: on the development of custom conversational assistants. In: Proceedings of the 1st International Workshop on Software Engineering for Cognitive Services. pp. 31–36. ACM, Gothenburg Sweden (2018)
Følstad, A., Skjuve, M.: Chatbots for customer service: user experience and motivation. In: Proceedings of the 1st International Conference on Conversational User Interfaces. pp. 1–9. ACM, Dublin Ireland (2019)
The Deloitte Consumer Review: Made-to-order: The rise of mass personalisation. Deloitte (2015)
Chakrabarti, C., Luger, G.F.: Artificial conversations for customer service chatter bots: Architecture, algorithms, and evaluation metrics. Expert Syst. Appl. 42, 6878–6897 (2015). https://doi.org/10.1016/j.eswa.2015.04.067
Ling, E.C., Tussyadiah, I., Tuomi, A., Stienmetz, J., Ioannou, A.: Factors influencing users’ adoption and use of conversational agents: A systematic review. Psychol. Mark. 38, 1031–1051 (2021). https://doi.org/10.1002/mar.21491
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., Weston, J.: Personalizing Dialogue Agents: I have a dog, do you have pets too? In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2204–2213. Association for Computational Linguistics, Melbourne, Australia (2018)
Hu, T., et al.: Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media User Experience - Interactions (2018). http://arxiv.org/abs/1803.02952
Stokoe, E.H., Smithson, J.: Making gender relevant: conversation analysis and gender categories in interaction. Discourse Soc. 12, 217–244 (2001). https://doi.org/10.1177/0957926501012002005
Kitzinger, C.: Doing gender: a conversation analytic perspective. Gend. Soc. 23, 94–98 (2009). https://doi.org/10.1177/0891243208326730
Thüring, M., Mahlke, S.: Usability, aesthetics and emotions in human–technology interaction. Int. J. Psychol. 42, 253–264 (2007). https://doi.org/10.1080/00207590701396674
Bevan, N., Kirakowski, J., Maissel, J.: What is Usability? In: Human Aspects in Computing: Proceedings of the 4th International Conference on HCI, Stuttgart, Germany (1991)
Broadbent, E., et al.: Robots with display screens: a robot with a more humanlike face display is perceived to have more mind and a better personality. PLoS ONE 8, e72589 (2013). https://doi.org/10.1371/journal.pone.0072589
Nowak, K.L., Rauh, C.: The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction. J Comp Mediated Comm. 11, 153–178 (2005). https://doi.org/10.1111/j.1083-6101.2006.tb00308.x
Araujo, T.: Living up to the chatbot hype: the influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Comput. Hum. Behav. 85, 183–189 (2018). https://doi.org/10.1016/j.chb.2018.03.051
Reeves, B., Nass, C.: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Center for the Study of Language and Information (1996)
Bainbridge, W.A., Hart, J., Kim, E.S., Scassellati, B.: The effect of presence on human-robot interaction. In: RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication, pp. 701–706. IEEE, Munich, Germany (2008)
Lee, S., Choi, J.: Enhancing user experience with conversational agent for movie recommendation: effects of self-disclosure and reciprocity. Int. J. Hum. Comput. Stud. 103, 95–105 (2017). https://doi.org/10.1016/j.ijhcs.2017.02.005
Schuetzler, R., Grimes, M., Giboney, J., Buckman, J.: Facilitating natural conversational agent interactions: lessons from a deception experiment. In: Human Computer Interaction, Auckland (2014)
Mahlke, S.: User Experience of Interaction with Technical Systems (2008)
Kaptein, M., Markopoulos, P., De Ruyter, B., Aarts, E.: Two acts of social intelligence: the effects of mimicry and social praise on the evaluation of an artificial agent. AI & Soc. 26, 261–273 (2011). https://doi.org/10.1007/s00146-010-0304-4
Laban, G., Araujo, T.: The effect of personalization techniques in users’ perceptions of conversational recommender systems. In: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pp. 1–3. ACM, Virtual Event Scotland UK (2020)
Laban, G., Araujo, T.: Don’t Take it Personally: resistance to individually targeted recommendations from conversational recommender agents. In: Proceedings of the 10th International Conference on Human-Agent Interaction, pp. 57–66. ACM, Christchurch New Zealand (2022)
Trivedi, J.: Examining the customer experience of using banking chatbots and its impact on brand love: the moderating role of perceived risk. J. Internet Commerce 18, 91–111 (2019). https://doi.org/10.1080/15332861.2019.1567188
Przegalinska, A., Ciechanowski, L., Stroz, A., Gloor, P., Mazurek, G.: In bot we trust: a new methodology of chatbot performance measures. Bus. Horiz. 62, 785–797 (2019). https://doi.org/10.1016/j.bushor.2019.08.005
Hassenzahl, M., Tractinsky, N.: User experience - a research agenda. Behav. Inform. Technol. 25, 91–97 (2006). https://doi.org/10.1080/01449290500330331
Plocher, T.A., Garg, C., Chestnut, J.: Connecting culture, user characteristics and user interface design. In: On Human-Computer Interaction: Ergonomics and User Interfaces, pp. 803–807. Lawrence Erlbaum, Munich, Germany (1999)
Wood, W., Eagly, A.H.: Gender identity. In: Handbook of Individual Differences in Social Behavior, pp. 109–125. The Guilford Press (2009)
Jenkins, K.: Amelioration and inclusion: gender identity and the concept of woman. Ethics 126, 394–421 (2016). https://doi.org/10.1086/683535
Jenkins, K.: Toward an Account of Gender Identity. Ergo Open Access J. Philos. 5 (2018). https://doi.org/10.3998/ergo.12405314.0005.027
Yen, C., Chiang, M.-C.: Trust me, if you can: a study on the factors that influence consumers’ purchase intention triggered by chatbots based on brain image evidence and self-reported assessments. Behav. Inform. Technol. 40, 1177–1194 (2021). https://doi.org/10.1080/0144929X.2020.1743362
Tsai, W.-H.S., Liu, Y., Chuan, C.-H.: How chatbots’ social presence communication enhances consumer engagement: the mediating role of parasocial interaction and dialogue. JRIM. 15, 460–482 (2021). https://doi.org/10.1108/JRIM-12-2019-0200
Müller, L., Mattke, J., Maier, C., Weitzel, T., Graser, H.: Chatbot acceptance: a latent profile analysis on individuals’ trust in conversational agents. In: Proceedings of the 2019 on Computers and People Research Conference, pp. 35–42. ACM, Nashville TN USA (2019)
Bartneck, C., Kulić, D., Croft, E., Zoghbi, S.: Measurement instruments for the anthropomorphism, animacy, likeability, perceived Intelligence, and Perceived Safety of Robots. Int. J. Soc. Robot. 1, 71–81 (2009). https://doi.org/10.1007/s12369-008-0001-3
Ren, R., Castro, J.W., Acuña, S.T., De Lara, J.: Evaluation techniques for chatbot usability: a systematic map** study. Int. J. Soft. Eng. Knowl. Eng. 29, 1673–1702 (2019). https://doi.org/10.1142/S0218194019400163
Laranjo, L., et al.: Conversational agents in healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25, 1248–1258 (2018). https://doi.org/10.1093/jamia/ocy072
Kocaballi, A.B., et al.: The personalization of conversational agents in health care: systematic review. J. Med. Internet Res. 21, e15360 (2019). https://doi.org/10.2196/15360
United Nations: Guidelines for gender-inclusive language in English. https://www.un.org/en/gender-inclusive-language/guidelines.shtml
Kocaballi, A.B., Laranjo, L., Coiera, E.: Understanding and measuring user experience in conversational interfaces. Interact. Comput. 31, 192–207 (2019). https://doi.org/10.1093/iwc/iwz015
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., Mctear, M.: Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? In: Proceedings of the 31st European Conference on Cognitive Ergonomics, pp. 207–214. ACM, BELFAST United Kingdom (2019)
Rubin, R.B., Martin, M.M.: Development of a measure of interpersonal communication competence. Commun. Res. Rep. 11, 33–44 (1994). https://doi.org/10.1080/08824099409359938
Habarth, J.M.: Development of the heteronormative attitudes and beliefs scale. Psychol. Sexuality 6, 166–188 (2015). https://doi.org/10.1080/19419899.2013.876444
R Core Team R: A language and environment for statistical computing (2022). https://www.R-project.org/
Figner, B., et al.: Standard Operating Procedures For Using Mixed-Effects Models (2020). http://decision-lab.org/wp-content/uploads/2020/07/SOP_Mixed_Models_D2P2_v1_0_0.pdf
Lüdecke, D., Ben-Shachar, M., Patil, I., Waggoner, P., Makowski, D.: Performance: an R package for assessment, comparison and testing of statistical models. JOSS 6, 3139 (2021). https://doi.org/10.21105/joss.03139
Dixon, M., Freeman, K., Toman, N.: Stop Trying to Delight Your Customers (2010). https://hbr.org/2010/07/stop-trying-to-delight-your-customers
Haugeland, I.K.F., Følstad, A., Taylor, C., Bjørkli, C.A.: Understanding the user experience of customer service chatbots: an experimental study of chatbot interaction design. Int. J. Hum. Comput. Stud. 161, 102788 (2022)
Brandtzæg, P.B., Skjuve, M., Kristoffer Dysthe, K.K., Følstad, A.: When the social becomes non-human: young people’s perception of social support in chatbots. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–13. ACM, Yokohama Japan (2021)
Litman, D.J., Pan, S.: Designing and evaluating an adaptive spoken dialogue system. User Model. User-Adap. Inter.Adap. Inter. 12, 111–137 (2002). https://doi.org/10.1023/A:1015036910358
Go, E., Sundar, S.S.: Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions. Comput. Hum. Behav.. Hum. Behav. 97, 304–316 (2019). https://doi.org/10.1016/j.chb.2019.01.020
Følstad, A., Nordheim, C.B., Bjørkli, C.A.: What makes users trust a chatbot for customer service? an exploratory interview study. In: Tiropanis, T., Vakali, A., Sartori, L., Burnap, P. (eds.): INSCI 2015. LNCS, vol. 9089. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18609-2
Liu, X., Wang, Q.: Study on Application of a quantitative evaluation approach for software architecture adaptability. In: Fifth International Conference on Quality Software (QSIC 2005), pp. 265–272. IEEE, Melbourne, Australia (2005)
Singmann, H., Bolker, B., Westfall, J., Aust, F., Ben-Shachar, M.S.: afex: Analysis of Factorial Experiments (2022). https://CRAN.R-project.org/package=afex
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cocchi, A., Bosse, T., van Pinxteren, M. (2024). Should Conversational Agents Care About Our Gender Identity?. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2023. Lecture Notes in Computer Science, vol 14524. Springer, Cham. https://doi.org/10.1007/978-3-031-54975-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-54975-5_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54974-8
Online ISBN: 978-3-031-54975-5
eBook Packages: Computer ScienceComputer Science (R0)