SpringerLink
Log in

A Conversational Agent that Reacts to Vocal Signals

  • Conference paper
  • First Online:
Intelligent Technologies for Interactive Entertainment (INTETAIN 2016 2016)

  • 1037 Accesses

Abstract

Conversational agents are increasingly being used for training of social skills. One of their most important benefits is their ability to provide natural interaction with humans. This work proposes to extend conversational agents’ benefits for social skills training by analysing the emotion conveyed by the user’s speech. For that, we developed a new system that captures emotions from human voice and, combined with the context of a particular situation, uses this to influence the internal state of the agent and change its behaviour. An example of the system’s use is shown and its limitations and advantages are discussed, together with the internal workflow of the system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 42.79
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 53.49
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Obviously, vocal signals are not the only aspect of behaviour that is relevant for communication training. Other aspects include facial expression, gestures, and posture, among others. However, these aspects are beyond the scope of this paper.

References

  1. Cassell, J., Sullivan, J., Prevost, S., Churchill, E.: Embodied Conversational Agents. MIT Press, Cambridge (2000)

    Google Scholar 

  2. Bosse, T., Provoost, S.: Towards aggression de-escalation training with virtual agents: a computational model. In: Zaphiris, P., Ioannou, A. (eds.) LCT 2014. LNCS, vol. 8524, pp. 375–387. Springer, Heidelberg (2014). doi:10.1007/978-3-319-07485-6_37

    Google Scholar 

  3. Bruijnes, M., Linssen, J.M., op den Akker, H.J.A., Theune, M., Wapperom, S., Broekema, C., Heylen, D.K.J.: Social behaviour in police interviews: relating data to theories. In: D’Errico, F., Poggi, I., Vinciarelli, A., Vincze, L. (eds.) Conflict and Multimodal Communication. Computational Social Sciences, pp. 317–347. Springer, Heidelberg (2015)

    Google Scholar 

  4. Hays, M., Campbell, J., Trimmer, M., Poore, J., Webb, A., Stark, C., King, T.: Can role-play with virtual humans teach interpersonal skills? In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) (2012)

    Google Scholar 

  5. Jeuring, J., Grosfeld, F., Heeren, B., Hulsbergen, M., IJntema, R., Jonker, V., Mastenbroek, N., Smagt, M., Wijmans, F., Wolters, M., Zeijts, H.: Communicate! — a serious game for communication skills —. In: Conole, G., Klobučar, T., Rensing, C., Konert, J., Lavoué, É. (eds.) EC-TEL 2015. LNCS, vol. 9307, pp. 513–517. Springer, Heidelberg (2015). doi:10.1007/978-3-319-24258-3_49

    Google Scholar 

  6. Kim, J., Hill, R.W., Durlach, P., Lane, H.C., Forbell, E., Core, C., Marsella, S., Pynadath, D., Hart, J.: BiLAT: a game-based environment for practicing negotiation in a cultural context. Int. J. Artif. Intell. Educ. 19(3), 289–308 (2009)

    Google Scholar 

  7. Vaassen, F., Wauters, J.: deLearyous: Training interpersonal communication skills using unconstrained text input. In: Proceedings of ECGBL, pp. 505–513 (2012)

    Google Scholar 

  8. Juslin, P.N., Scherer, K.R.: Vocal expression of affect. In: Harrigan, J.A., et al. (eds.) The New Handbook of Methods in Nonverbal Behavior Research. Oxford Press, Oxford (2005)

    Google Scholar 

  9. Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)

    Article  Google Scholar 

  10. Russel, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)

    Article  Google Scholar 

  11. Yik, M., Russel, J., Steiger, J.: A 12-point circumplex structure of core affect. Emotion 11(4), 705–731 (2011)

    Article  Google Scholar 

  12. Scherer, K.R., Shorr, A., Johnstone, T.: Appraisal Processes in Emotion: Theory, Methods, Research. Oxford University Press, Canary (2001)

    Google Scholar 

  13. ElAyadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recogn. 44, 572–587 (2011)

    Article  MATH  Google Scholar 

  14. Truong, K.P., van Leeuwen, D.A., de Jong, F.M.G.: Speech-based recognition of self-reported and observed emotion in a dimensional space. Speech Commun. 54, 1049–1063 (2012)

    Article  Google Scholar 

  15. Lefter, I., Rothkrantz, L.J.M., Burghouts, G.: Aggression detection in speech using sensor and semantic information. Text, Speech and Dialogue 7499, 665–672 (2012)

    Google Scholar 

  16. Rodriguez, H., Beck, D., Lind, D., Lok, B.: Audio analysis of human/virtual-human interaction. In: Prendinger, H., Lester, J., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 154–161. Springer, Heidelberg (2008). doi:10.1007/978-3-540-85483-8_16

    Chapter  Google Scholar 

  17. Bevacqua, E., Pammi, S., Hyniewska, S.J., Schröder, M., Pelachaud, C.: Multimodal backchannels for embodied conversational agents. In: Allbeck, J., Badler, N., Bickmore, T., Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS (LNAI), vol. 6356, pp. 194–200. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15892-6_21

    Chapter  Google Scholar 

  18. Acosta, J.C., Ward, N.G.: Achieving rapport with turn-by-turn, user-responsive emotional coloring. Speech Commun. 53, 1137–1148 (2011)

    Article  Google Scholar 

  19. Cavazza, M., Pizzi, D., Charles, F., Vogt, T., Andre, E.: Emotional input for character-based interactive storytelling. In: Proceedings of the 8th International Conference on Autonomous Agents and multi-agent systems, AAMAS 2009, pp. 313–320 (2009)

    Google Scholar 

  20. DeVault, D., et al.: SimSensei kiosk: a virtual human interviewer for healthcare decision support. In: Proceedings of the 13th International Conference on Autonomous Agents and Multi-agent Systems, AAMAS 2014, pp. 1061–1068 (2014)

    Google Scholar 

  21. Ben Youssef, A., Chollet, M., Jones, H., Sabouret, N., Pelachaud, C., Ochs, M.: Towards a socially adaptive virtual agent. In: Brinkman, W.-P., Broekens, J., Heylen, D. (eds.) IVA 2015. LNCS (LNAI), vol. 9238, pp. 3–16. Springer, Heidelberg (2015). doi:10.1007/978-3-319-21996-7_1

    Chapter  Google Scholar 

  22. Eyben, F., Wöllmer, M., Schuller, B.: openEAR - introducing the munich open-source emotion and affect recognition toolkit. In Proceedings of the 4th International HUMAINE Association Conference on Affective Computing and Intelligent Interaction 2009 (ACII 2009). IEEE, Amsterdam (2009)

    Google Scholar 

  23. Nwe, T., Foo, S., De Silva, L.: Speech emotion recognition using hidden Markov models. Speech Commun. 41, 603–623 (2003)

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the Brazilian scholarship program Science without Borders - CNPq {scholarship reference: 233883/2014-2}.

Author information

Authors and Affiliations

  1. Department of Computer Science, VU University Amsterdam, De Boelelaan 1081, 1081 HV, Amsterdam, The Netherlands

    Daniel Formolo & Tibor Bosse

Authors
  1. Daniel Formolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tibor Bosse
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Formolo .

Editor information

Editors and Affiliations

  1. Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

    Ronald Poppe

  2. Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

    John-Jules Meyer

  3. Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

    Remco Veltkamp

  4. Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

    Mehdi Dastani

Rights and permissions

Reprints and permissions

Copyright information

© 2017 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Cite this paper

Formolo, D., Bosse, T. (2017). A Conversational Agent that Reacts to Vocal Signals. In: Poppe, R., Meyer, JJ., Veltkamp, R., Dastani, M. (eds) Intelligent Technologies for Interactive Entertainment. INTETAIN 2016 2016. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 178. Springer, Cham. https://doi.org/10.1007/978-3-319-49616-0_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-49616-0_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-49615-3

  • Online ISBN: 978-3-319-49616-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation