Keywords

1 Introduction

In most developed countries, especially in Europe, the population of the age group over 65 years is increasing [3]. As getting older, most people will need to stay living a healthy life and to maintain mental and physical autonomy. Thus, a coaching approach can be a solution for retaining their wellbeing. The speedy development of the ICT technology can play a key role in supports of the requirements of the ageing population. Indeed, technology can maximize the successfulness of coaching older individuals at larger scale reducing the economic burden of well-being intervention and at longer term, of their care, However, technology stays advanced for many older adults.

The project NESTORE works to beat the restrictions of current solutions. In fact, most healthcare technological systems tackle two or less wellbeing domains [7]. NESTORE is a virtual coach that addresses healthy older individuals and cover five key dimensions of wellbeing: physical, nutritional, cognitive, social and emotional. Its vision is based on pathways proposed by experts and the experience is personalized on user preferences and needs. SOC and HAPA models [9] known as Behavioral theories are embedded in the coaching algorithms. Data is collected through environmental sensors, wearable devices, beacons, games, sleep monitoring, smart scales and the virtual coach itself, through natural conversation. The whole system is personalized by an intelligent and innovative Decision Support System (DSS) [2] that works on short-term and long-term analysis through analyzing static and dynamic profile. As for the intervention, the virtual coach itself is the only way to deliver data of the whole NESTORE intelligence cloud and devices to the user.

NESTORE e-coach plays three essential roles: a coach, a friend, a companion. We designed NESTORE as a coach that provides coaching activities and plans in each domain of wellbeing in order to give personalized intervention techniques such as short- and long-term goals, feedbacks based on user’s score. We designed NESTORE e-coach as a friend giving it the ability of listening to the user upon his or her request and of understanding the user natural language and the powerful intelligence of the system that is able to learn and adapt to users’ specific behaviors and context. We also design this e-coach as a companion by being omnipresent via its tangibility or its online availability. Most importantly, we define this sense of companionship by having an empathic relationship with the user using emotions detection algorithm [12]. As a result, our e-coach’s final goal is to transform elderly to co-producers of their own health life.

The e-coach is composed of three different interfaces where each interface plays the three roles mentioned before. The three interfaces come in different forms and capabilities: a mobile application, a chatbot integrated in the mobile application, which is a text based conversational agent and a tangible coach, which is an embodied conversational agent that acts as a vocal assistant. These three interfaces can interact with user exclusively, sequentially, concurrently and synergistically. The e-coach interfaces work as a team to act and give the user the sense of coaching, friendship and companionship visioning a long-term human-computer interaction with seniors. In this article, we introduce the NESTORE system with a scenario of an elderly using the e-coach and the role played by the different interfaces. Next, we present the three essential roles of our e-coach by being a friend, coach and companion and how our e-coaches and its three interfaces acts to achieve these roles. Then, we present each interface of the e-coach and the multimodality aspect toward the elderly concluding the article by the status of the project and some future work.

2 Previous Work

We conducted a systematic review to study the previous work for a virtual coach for elderly’s wellbeing [7]. We explored the different definitions of the previous e-coaches especially their role as a friend or companion. Most of the papers found mention the importance for e-coaches to be warm, cuddly and useful. However, there were no studies that explore how each role can affect the design of this virtual coach [7]. We also explored the different embodiments and intervention delivery modalities of the virtual coach: a study [13] has built a system that schedules some physical activities for the user: it is a digital human-like coach in computer program. The embodiment here is an animated virtual coach that also spoke up the scheduled messages that appear on the screen for the user using text-to-speech converter. The coach is almost always visually present on the screen for the user. Another study [8] explains a virtual coach called Bandit in a robotic form that mimics human physical activities in order to help elderly to do certain activities. The robot’s lip movements are synchronized with the robot’s speech so that the lips open at the start until the end of a spoken utterance. Results showed that most elderly were very motivated for such a virtual coach because it is enjoyable, useful and helpful. However, no studies were found that show the multi-modality aspect that we are going to explore in Sect. 6. In fact, most studies explore one type of interface such as [4] and [5]. Finally, most studies were tackling only one domain, which emphasizes the need to explore a multi-domain intervention with multi-modal interaction.

3 E-coach Accompanying Older Adults for a Better Lifestyle: A Scenario

Laura is 65 and has recently retired. She is Italian and lives currently in Milano. She loves cooking and dancing and now that she is free, she would like to spend more time exploring new recipes and re-living her youth dancing in front of the mirror. Sandro, her husband, is also retired and lives with her at home with their dog. Sandro and Laura love gardening together and this is what is hel** them to stay fit. To get a healthier lifestyle, Laura got NESTORE e-coach. This virtual coach comes in different forms of interfaces such as a mobile application, a chatbot integrated inside the mobile app and a tangible coach with vocal and tangible interaction capabilities. The chatbot on-boards Laura, introduces her to the system and starts to get to know more about her habits and preferences. Thus, her e-coach chooses “cooking”, “dancing” and “gardening” as the main three focuses. The e-coach starts setting quantitative goals for Laura and with time while observing Laura’s progress with the e-coach proposed tasks, the e-coach starts sending other goals that optimize Laura’s health. Such propositions and recommendations are sent by the chatbot, the mobile application or the tangible coach, according to the user availability or preferences. If the user is not at home, then the e-coach understands her unavailability of having a vocal interaction with the tangible coach and will send its messages to the mobile chatbot instead. Moreover, depending on what the user chooses or uses more often among chatbot or tangible or mobile application, the e-coach will start sending these recommendations to the most used interface over time. In the beginning, Laura is put to a two- week monitoring (motivational) phase to assess her wellbeing path and then she is shifted to the intervention (volitional) phase where Laura conducts a behavioral change and becomes a co-producer on her own health. This behavioral change model called HAPA is proposed by our experts [9]. Daily and weekly progress and proposition of activities are given by the mobile application. The DSS [2] will select coaching plans based on personalized indications, advices on healthy lifestyle and elderly decline prevention. Hence, the e-coach starts monitoring and tracking the progress of Laura, outlines her behavior changes and manages her loss of motivation using different type of intervention techniques such as feedback, recommendations, praise etc. Since all services in the system are proposed by experts in each domain of wellbeing then “dancing” is treated under the umbrella of the physical domain sub-pathways service which is “aerobic fitness” [6]. The e-coach proposes to Laura to “walk her dog” or “gardening” with her husband under the umbrella of combining two domains: the physical and social domain. The chatbot, as its job in the nutritional domain, asks Laura to send a photo of the current meal to analyze her calories and nutrients [14]. On the other hand, the chatbot sends Laura different recipes based on her heathy eating progress and food preferences. In addition, the e-coach, through the mobile application, suggests taking dancing lessons or joining dancing groups nearby Milano. The e-coach, knowing that she and her husband are both fans of gardening, will suggest for them to go to an exhibition on “Italian gardens” via the mobile application interface. Laura can also connect with her friends or family on the platform in order to join them in some activities with them. Not only can the e-coach connect to Laura, but the latter can also ask questions and get an overview of her performance at the end of each day via the tangible coach. These different modalities are activated independently according to the CASE model [10]. The e-coach is also capable of delivering the intervention by combining the multiple interfaces in a synergistic manner [10]. For instance, the tangible coach explains vocally the recipe of a specific food, accompanying Laura step by step during her performance. The tangible coach also sends the full description of the recipe via the chatbot for a visual interaction. Figure 1 shows the user journey through the NESTORE e-coach.

Fig. 1.
figure 1

User journey using NESTORE e-coach

Full size image

4 Older Adults E-coach Vision and Principles

NESTORE e-coach is based on co-design principles held in 4 different countries: Italy, England, Netherlands and Spain. NESTORE methodology is called after the Greek Nestor, known for his great wisdom, the advisor of Troy. This e-coach guides people to better lives. It plays three essential roles with elderly:

  1. (1)

    a coach: based on experts in each field of wellbeing to help the user achieve his goals by giving coaching tips, tracking and monitoring the user.

  2. (2)

    a companion: based on being omnipresent and empathic with the user. It gives the older adults a sense of companionship so that they will not feel alone.

  3. (3)

    a friend: based on being loyal to the user. It gives also many advices to build a certain trust between the e-coach and the senior.

These three roles allows the e-coach to mediate into the senior’s life and recommend the user to change one or more behavior at once.

4.1 NESTORE as a Coach

NESTORE e-coach mentors the senior in five domains: physical, nutritional, cognitive, social and emotional. Domain experts provided specific recommendations that are orchestrated by the coach following a behavioral changing model called HAPA. The e-coach refers to a coaching timeline that is composed of a three-layer system of pathways, coaching activity plans and coaching events. As a coach, the latter helps the user to choose his or her general pathway. In fact, after the 2 weeks motivational phase, the e-coach analyses the user’s behaviors and recommend him or her the wellbeing pathway to follow. Next, the e-coach helps the user to select his or her favorite coaching plan activity. The e-coach then helps the user to schedule the proposed coaching activities into his or her calendar matching the user’s time availability. Finally, the e-coach guides the users towards their objective by using different intervention techniques such as praise messages, reminders, setting short term to eventually reach long term goals, personalized recommendations, and constructive feedback [7]. The e-coach also give a scoring system defined for each structured activity by our experts and a daily/weekly/monthly progress summary. This kind of interactions are spread among the different interfaces of this e-coach depending on user’s preference and availability. For example, the chatbot and the mobile app show a detailed graphical representation of senior’s wellbeing progress as in Fig. 2, whereas the tangible coach spells out vocally the status of the progress. Also, seniors can ask the tangible to send a summary of their score to the chatbot. NESTORE e-coach has an in depth understanding of the individual’s health status including current and previous diseases, health issues and any sort of medication taken through soft and hard analysis. It uses third party devices coming along with the NESTORE system for hard data collection such as sleep monitoring, step counters, a smart scale and environmental sensors spread in the user’s house. The user will also have some beacons on him during his journey in order to detect the amount of social interaction. The chatbot send some questionnaires provided by experts in the cognitive and social domains for soft data collection. The mobile application suggests some cognitive games such as “numerical updating tasks”, “digit span backward” and Math problem-solving exercise in order to monitor how well the senior is doing toward his or her wellbeing [6].

Fig. 2.
figure 2

Mobile application for showing data, scheduling activities and playing games

Full size image

The chatbot also have food recognition capabilities: it asks the user to send a photo of the current meal he or she is eating. Using machine learning algorithms, the e-coach detects the food in the photo, calculates the calories and the nutrients provided by the meal [14]. To sum up, NESTORE as a coach pushes the individual to convert a habit or follow a healthier routine via a well-studied psychological intervention.

4.2 NESTORE as a Friend

NESTORE e-coach “knows” and “understands” the user which are the two criteria that makes our e-coach a friend. It collects and gathers user’s personal data such as the user’s health issues, preferences, lifestyle, past experiences, and so many other aspects. Data are collected in two forms: static and dynamic. Static data are basically the user’s general information such as users’ demographic and environmental characteristics, preferences and baseline data. Whereas, dynamic data are collected via sensors, applications, and contextual APIs. By collecting all these data, the NESTORE e-coach gets to know really well the senior user. On the user side, once the mobile application is downloaded and the tangible coach is installed, the chatbot starts introducing the whole system and switch from formality to informality after a while as a friendship sign. The e-coach will also be able to execute specific tasks for the senior such as weather cast, news updates, general informing and healthy coaching personalized updates that will help seniors to form eventually a sort of a friendship and a bond to the coach. The amount of the use of the e-coach and the type of questions asked by the user will give the e-coach the possibility to become more open to informality. In particular, NESTORE e-coach implements novel algorithms for detecting and monitoring of important indicators related to user status and behavior. The personalization of the system plays a big role in building a friendship between the e-coach and the senior. The e-coach gives personalized feedback and advice. This latter makes the user feel eventually appreciated, loved and known as a close friend. According to [16], casual friendships are very helpful in relieving stress and depression, but having a close friend takes it to a whole new level of levering. This is what makes e-coach very important in ones’ life. This e-coach is the literal definition of “I’m always here for you”; whenever the user calls out for it; it directly replies, which makes it really functional under all circumstances.

In addition to giving advice; e-coach is also a secret keeper. Confidentiality is a one of e-coach’s core essence, which strengthen the relationship between the user and e-coach. GDPR is taken into consideration [15] since data collection is consensual and will never be leaked. At longer term, the e-coach will be the best projection of a person you’ve known your whole life.

4.3 NESTORE as a Companion

Adults encounter all sort of companionship throughout their daily basic day at work, streets, friends and even family. Paper [18] shows that the following does not apply to elderly. The author elaborated that seniors, once retired from jobs and with their children moving away or their friends and/or spouses passing away, do not have always opportunities for social engagement. This might lead to seniors becoming housebound, especially if they lose the ability to drive or become ill. Companionship is a main part of “senior care” and not solely via assistance in daily duties but also via a beneficial conversational interaction. Whenever human companionship is not available for the senior, the e-coach might, at least partially, compensate this lack. In fact, NESTORE e-coach builds an empathic relationship with the user by (1) being omnipresent with the user at any time, (2) understanding the user sentiments and acting upon it and (3) interacting with the user through tangible and vocal interaction. To begin with, the e-coach is always available for the user. Seniors can always have access by opening their mobile application and start chatting with the e-coach. The physical device part of the coach also plays a role of being an omnipresent and a portable battery power e-coach as shown in Fig. 3. Second, the chatbot and the tangible coach can detect the user’s feelings based on senior’s text input or the senior’s voice and tonality. In fact, particular sentences spelled or written by the user are sent to the Emotion Well-Being Engine that analyzes semantically the emotional valence of the phrase. The emotional wellbeing Engine is based on sematic reference model [12] which make understands the user’s current affect state. Once the chatbot or the tangible coach detects the user’s emotion, they let the user know that they understand his or her feelings and start acting upon it by motivating and supporting the user. The tangible coach, in particular, has a lightning system as shown in Fig. 3 in order to interact with the user. The tangible coach integrates for each emotion lights patterns with different colors for each emotion detected such as “happiness”, “sadness”, “surprised”, “disgust”, “anger” and “fear” [12]. These lights follow the Plutchick’s diagram colors of feelings [17]. To sum it up, the tangible coach plays a huge role in kee** company by the ability of letting the user to “touch” the coach and “speak” using vocal voice. The user has the opportunity to have a physical device in his or her house, always present and ready to start a conversation with the user.

Fig. 3.
figure 3

Tangible coach and the lightning system, listening (a), thinking (b)

Full size image

5 E-coach Architecture and Interfaces for Multi-modality

The use of digital devices to support elderly’s wellbeing has been an open field [7]. Most elderly people, nowadays, own at least one electronic device such as smartphone, smart wearable, tablet, PC or laptop computers and even vocal assistants. In particular, conversational agents have become the center of research and industrial use. According to chatbots.org [19], a conversational agent is a software program which understand and respond the user based on statements sent in ordinary natural language. A conversational agent can be a text-based messaging application such as woebot [20] or a vocal conversational agent such as ALEXA, GOOGLE HOME. They can be used for different purposes such as customer service or information acquisition which make them more useful and popular. Conversational agents can also be used in healthcare.

One of the main goals of the NESTORE virtual coach is to build a ubiquitous interaction with the conversational agents. Commands change with activities across the daily routine and takes place in different time and location. A conversational agent embedded in a mobile application is very useful because it can travel with the user but having a chatbot based conversational agent might decrease the user experience due to its limited modality capability. In fact, a conversational agent which is able to communicate via multiple kind of interfaces preserving the awareness of the user’s preferences and context, can enhance the final user experience and provide a more effective interaction [21]. However, the transition from one device to the other should be seamless and opportune to the particular activity performed at that very moment. In order to do that, one should know exactly the capability of each interface. For instance, two of the main advantages of the speech based conversational agent are optimizing time and enabling multi-tasking. Such interface can be used for particularly daily tasks and should be integrated in daily activities that involves having both hands occupied, for example, while cooking. Text-based Chatbots have other principles of interaction. A text-based chatbot is able to perform the demanded tasks effectively and precisely using buttons on user’s interface and more importantly visually.

In order to give the best user experience, NESTORE e-coach is constituted of two different types of conversational agents: a vocal tangible coach, a text-based chatbot and the mobile application.

The mobile application is an android app. It contains the senior profile. The latter can also change his or her personal data. The senior can find in the mobile app personalized activities proposed and recommended by the system and can also check his or her progress in each domain of wellbeing. Finally, the user can schedule his or her next activities using the calendar in the mobile app. Such interface shows also feedback on different wellbeing charts.

The chatbot, in NESTORE case, is an intelligent conversational software agent activated by natural language text input. This chatbot provides written and generated conversational output in response using intelligent algorithms and a database. The chatbot, like the mobile application, can also schedule activities for the user, knowing the user’s available time, and can propose for the user also activities depending on his health progress through a written scenario of conversations between the chatbot and the user. Users can also chat with the chatbot about their day and their feelings. The chatbot sends also personalized recommendations, feedback, praise messages and notifications on a daily, weekly and monthly analysis. The chatbot can give an overview of the score upon user’s request.

The tangible coach is based on vocal and tangible interaction. In the vocal interaction, the tangible coach will listen to the senior and then generate a response to the user depending on his or her question. Through the lighting system of the tangible coach, the senior will know when the tangible is listening, thinking or speaking. The senior can ask any general questions about the system, the weather, global news, general cultural information and the tangible device is always ready to answer. Like the mobile application and the chatbot, the senior can also ask about his or her scheduled and proposed activities and his or her weekly and monthly scores. The answers are all given vocally in the chosen language. Furthermore, the tangible notifies the user every night asking about their sentiments throughout their day to monitor their emotional wellbeing. Finally, like the chatbot, the senior can have a one to one conversation with the tangible coach. The tangible interaction gives the ability to the user to physically control the data collection process. In fact, the tangible coach has two main states, which are the sleep state and the wake state. The user needs to put the tangible coach in the wake state in order to speak with. The two states are differentiated based on the position of the tangible coach. Figure 4 shows the two different states of the tangible coach. As a result, the tangibility and the possibility to caress and to feel this tangible coach can create eventually an emotional bonding with the user.

Fig. 4.
figure 4

Tangible coach in sleep state (a) and wake state (b)

Full size image

Conversational agents of the e-coach (chatbot and tangible coach) are based on conversations and scenarios managed by the Conversational Coach Engine, which is developed in Node.js. In particular, user’s free-text inputs are treated by 4 instances of the RASA Core/NLU Server, each trained in the different languages supported by the chatbot (English, Spanish, Italian and Dutch).

It is easier usually to use just one type of interface and device to get a task done, however this means that the user must actively engage in the task in order to get the job done. For example, if the user focuses only on the tangible interaction, this may lead to the loss of having a visual understanding of the information received.

A multi-modal multi-interface capability is proposed. Interfaces are used sequentially (that is, device A then device B) and are chosen based on their suitability and availability. Different interfaces are used as well simultaneously for task completion (that is, A and B). For instance, since the tangible coach lacks a screen, combining it with the chatbot can expand the tangible coach’s capability, by sending links, images or videos.

The multimodal multi-interface interaction is based on a multimodal bridge shown in the Fig. 5. The Multimodal Coach Bridge decides when each interface interacts with the user, when to trigger recommendations about invoking multiple interfaces, and whether one or more than one interface should interact independently, sequentially or simultaneously.

Fig. 5.
figure 5

NESTORE e-coach architecture and the different interfaces of the e-coach (Tangible interface, chatbot, mobile application, serious game, social platform)

Full size image

6 Conclusion

NESTORE project is funded by the European union under the H2020 program and is intended to be finished by year 2021. The NESTORE project is based on a virtual coach that helps the user to maintain and improve his or her wellbeing. This is achieved by designing a multi-modal interaction and giving the e-coach different roles for interacting with the elderly. We presented a user journey with the NESTORE e-coach. We also explained the three roles of the NESTORE e-coach in improving and maintaining senior’s wellbeing. NESTORE e-coach is actually a coach, a friend and a companion to older adults. It helps the user to train in different domains in wellbeing by a set of intervention defined by our experts. It also understands the user and act as support system for the user throughout his journey for maintaining or improving the wellbeing. It finally builds an empathic relationship through natural conversation. We also presented the interfaces of this virtual coach and mentioned each roles and capability towards user’s wellbeing. Finally, we showed the architecture of our system and multimodal coach bridge for dispatching data for the different interfaces. The current version of the system is currently being tested with 60 users in 3 sites across Europe: Italy, Spain and the Netherlands. At the current stage of the project, NESTORE provides coaching in four domains (physical, nutritional, social and cognitive). Because of the complexity of the emotional domain, the system can recognize and monitor user’s emotions but does not provide coaching support in this domain. As for the multi-modality, the different interfaces currently can be used exclusively, but the synergistic interaction scenarios are under development.

Thanks to the multiple roles and the improved user experience provided by multimodal interfaces, the NESTORE e-coach is a promising friend and companion for improving older adults’ wellbeing.