BeaCON - A Research Framework Towards an Optimal Navigation

Abstract

In an optimally integrated HMS (Human Machine System), human must understand the machine as well as the machine must understand the human user. Same principle applies for car NS (Navigation System) which is a human-in-the-loop system. An ideally integrated NS knows how, when and what navigation information must be provided for the user and create minimal interruption for the primary task. To do the same, NS must hold the behavioral models of the user for providing the guidance information in an effective way. A research framework which uses these principles, is needed to create such models as well as for conducting further analysis for the research problem of “Giving the driver adequate navigation information with minimal interruption”. Until now no such research framework exists and because of that further analysis of the mentioned research problem cannot be conducted. In this paper we present the research framework BeaCON: Behavior-and Context-Based Optimal Navigation that enables detailed analysis of this research problem.

1 Introduction

A car navigation system is used to provide navigation information which guides the user to reach a destination. Identified research gaps in the area of optimum integration of HMS where human and machine learns together which is applicable for modern day NS is listed in [1]. NS shows a user the current location on the map and gives both audio and visual information for efficient travel from one location to another such as the path to be taken which is calculated based on graph theory as well as the dynamic information i.e. Traffic [3]. The main research problem of “Giving the driver adequate navigation information with minimal interruption” can be divided into following sub problems

• 1. Given a set of route and map information, what is an optimal guidance information for a user?

• 2. How to find optimal guidance information for a user, provided set of inputs to create the same is given?

• 3. How to find when, how and what guidance information must be given to the user?

These research questions must be addressed for achieving the optimal human NS integration which is needed to avoid driver distraction [4] created by NS as well as to reduce the cognitive load associated with navigation task. This paper introduces BeaCON (Behaviour-and Context-Based Optimal Navigation) - a research framework for enabling research in the mentioned problems. The driving context and the driver behavioral aspects must be considered to decide when, how and what guidance information must be given to the user [2]. Natural guidance as well as mobility graph addresses a very minimal aspect of this problem but with tailored solutions [5]. As an initial step to understand the research problems better, [2] conducts a survey to identify the scenarios where optimal integration between the human user and NS is not achieved. The critical survey questions addressed in [2] are regarding.

• 1. Driver distraction by NS

• 2. The extent to which NS understands the user intentions

• 3. Optimal integration of NS with the user

Based on the responses from users, behavioral models are created for these critical survey questions [2], which is also used in BeaCON. Since these models are created only once and will not change, the same can be called static behavioral models. An example of static behavior model created by using the data collected as a part of [2] is given in Fig. 1. Also BeaCON holds dynamic behavioral models created using WEKA machine learning suite [6] for different navigation specific user contexts, created per user. Static and dynamic models are created based on decision tree algorithm C4.5 in order to achieve high interpretability. The navigation specific user contexts created while conducting experiments using BeaCON is abstracted in a concept called Entity Of Interest (EOI) [2]. EOI can be visualized as the context which influence the cognitive load. EOI is currently created for junctions, roundabout and manoeuvres. Whenever a new behavior is identified for the user, the dynamic behavioral models will be recreated or modified using the machine learning algorithms. These models along with other components of BeaCON enables analysis of the research problems mentioned above.

Fig. 1.

An example of a static behavioral model (created for the extent by which NS understanding the user intentions)

2 Related Work

For designing intelligent navigation systems, a deeper understanding of their effects on human navigation behavior is necessary [7]. The dynamic behavioral models in BeaCON incorporate the effects of navigation system on human driver along with other characteristics. Classification of driver’s cognitive state to improve the in-vehicle information by using the drivers cognitive load and driving situation is conducted in [8]. But the classification approach given by [8] is not connecting the driving situation to the real-world map entities (Maneuvers, Roundabout etc.) as well as to the environmental factors (fog, rain etc.), by making the same also as a part of the behavioral models, which is addressed by BeaCON. Also [8] uses support vector machine (SVM) based machine learning models and for driver behavioral models while BeaCON uses decision tree-based driver behavioral models and conditional interconnection between them for enhancing the interpretability of the same [9]. A framework for automated driving testable scenarios including failure mitigation strategies is given in [10]. The list of behavioral competency scenarios mentioned in [10] for automated driving benchmarking is applicable for human driver-based scenarios also. But [10] focuses only on automated driving system as a whole and the optimization steps involved in human based and automated agent-based driving scenarios are different.

3 BeaCON and Its Components

The research framework BeaCON: Behaviour-and Context-Based Optimal Navigation enables to conduct experiments using human-in-the-loop systems, create behavioral data and based on that find user cognitive load points, optimize the machine learning behavioral models, which are the models used by the system to understand the user. The components of BeaCON which also shows the interconnection between different components is given in Fig. 2. Many steps related to BeaCON i.e. start, stop, create cognitive load report etc., are currently automated as well as near real time performance is achieved while conducting experiments. BeaCON consists of 5 main components which are

Fig. 2.

BeaCON and its components

• 1 Driving Simulator (DS)

• 2 Navigation System (NS)

• 3 Cognitive Models for Navigation (CMN)

• 4 Simulator for DS

• 5 Human User (HU), The description of these components is given below

3.1 Driving Simulator (DS)

CARLA open source simulator for autonomous driving research [11] is used in BeaCON with extensive adaptations. Since CARLA simulator is primarily designed for testing autonomous driving agents, extensive enhancements for existing interfaces are done to support the same for human driver interface using Logitech G920 driving hardware. CARLA provides extensive configuration support for creating different environments for experiments. For example, it is possible to control weather, number of pedestrians, state of different objects etc. in the simulator. Using these facilities, BeaCON supports conducting experiments for different cognitive load environments. A DS screen for conducting experiments in environments which induce heavy cognitive load is given in Fig. 3.

Fig. 3.

Sample GUI for the DS from CARLA

3.2 Navigation System (NS)

A custom-made NS is created and integrated to BeaCON. Currently the image from city 1 from CARLA is used for the NS. Navigation system tracks the car position in the route as well as highlights the path to be taken by the human user while conducting experiments. In the NS different images are used to represent different entities like junction, roundabout or a maneuver. Different symbols are used to indicate the start and end points of the test path in the map. GUI for NS is shown in Fig. 4.

Fig. 4.

GUI of NS process

The path taken in the map while conducting the experiment is configurable. Once a specific path from the map is configured, the same will be highlighted in the map. Different entities are represented using different images and the semantic of each entity must be informed to the candidate before conducting experiments. Different entities and their semantics are given in Table 1. Another functionality of NS is to give guidance information for the user based on the current behavior models.

Table 1. Images used for different NS entities

3.3 Cognitive Models for Navigation (CMN)

CMN holds the static and dynamic behavioral models and using the same the cognitive load values of the user for different driving context (EOI) are generated. The models in CMN are created using C4.5 algorithm. The static behavior models are created based on a statistical survey from 77 candidates from different countries [2]. Dynamic behavior models are created for different entities per user. A basic model for different entities is supplied to the system and reconfiguration of the same will be conducted at later point. When a new driver behavior is observed the dynamic behavioral models are updated accordingly. Static and dynamic behavior together create the complete behavior model for the user. It is possible to see the result of cognitive load calculation for all the questionnaires which was a part of the static analysis. An example GUI for one of the static analysis questions for NS creating distraction while driving is given in Fig. 5.

Fig. 5.

An example of interface to static behavior models of BeaCON

Similarly, it is possible to view the results for other static behavior models based on different input. Different dynamic contexts which occur during the navigation which is represented by EOI is represented by different dynamic models. For example, a very basic dynamic model created for roundabout is given in Fig. 6. User behavioral aspects are applied on the basic dynamic models to create user specific dynamic models for different EOIs.

Fig. 6.

Basic dynamic model created for Roundabout

A generic dynamic factor attribute is currently used as a placeholder for different traffic flow related factors. It is also possible to see the configuration of different EOI for a user in the system. For example, for a selected user the current value of cognitive load for a junction for a context can be identified by a GUI given in Fig. 7. Similarly, it is possible to identify the cognitive load values for different contexts for different users. It is also possible to make the system learn for a different cognitive load value for an EOI for a user manually by using the similar GUI.

Fig. 7.

Interface for dynamic model for junction for a user

3.4 Simulator for DS

This is used to study, develop as well as to configure different CMN models without the use of DS, which enables fast research of the CMN component. It is possible to simulate the driving between two configured points in the map as well as to verify the CMN behavior during the process since CMN is the component which learns and holds the behavior models for different users for different contexts. The simulator for DS is currently integrated as a functionality within the NS.

3.5 Human User (HU)

Dynamic models are created per user in CMN, so it is possible to conduct independent incremental experiments per user basis. Dynamic models for each of the EOIs are created and stored per user basis and the user identification is based on configuration values.

4 BeaCON Framework Setup

The components DS, NS, CMN are created as separate process and use inter process communication mechanism. The components in BeaCON are loosely coupled. Start script starts all the processes as well as initializes the driving simulator hardware setup. Preloaded configuration decides the path to be highlighted in the map for conducting human-in-the-loop experiments. Similarly, automation scripts are used for other functionalities related to BeaCON for ease of usability. Separate start scripts are provided for different driving environments as well as based on the hardware used to run BeaCON. For example, it is possible to run BeaCON on a less powerful hardware at the cost of rendering quality of DS. The generated report from the experiments contains x and y position, throttle, acceleration, brake, POI information, as well as the cognitive load values corresponding to each position. BeaCON set up used is shown in Fig. 8. It is also possible to configure the responsiveness of steering, brake and accelerator using the Logitech gaming software application as well as by controlling different parameters in the script used for interfacing the driving simulator hardware. For integrating other cities supported by CARLA necessary changes are needed for the map configuration which is used for highlighting the test route.

Fig. 8.

BeaCON framework setup

5 Conducting Human in Loop Experiments

Presently three different environments are supported for conducting experiments which are low cognitive load (50 Pedestrians, clear weather), medium Cognitive Load (50 Pedestrians, Soft rain noon) and high Cognitive Load (300 Pedestrians, Hard rain sunset). The algorithm for conducting human in loop experiments are given below, the steps for conducting experiments using BeaCON is given in Fig. 9.

Fig. 9.

Conducting experiments in BeaCON

//Input: Candidate choose and drive between two points on the map using a selected configuration //Output: Performance evaluation of the driver behavior, CMN models are re-created based on new behavior observed Step 1: Two points in the map and configuration are selected for conducting experiment Step 2: Candidate drives between the selected points, collect the behavioral data Step 3: Once user reached the destination, stop collecting behavioral data Step 4: Give the behavioral data input to the custom bench marking tool Step 5: Custom bench marking tool creates necessary logs for driving behavior Step 6: Measure cognitive load at different contexts of driving, create report Step 7: Replay the driving behavior to re-verify the findings from report with the user Step 8: Cognitive load values at different points are used to recreate the dynamic behavioral models for EOI

Individual measurements from the test (steering, brake, throttle values etc.) are getting logged in separate files so that the same can be easily used in different data analysis tasks. Using a replay mechanism, it is possible to show different scenarios to the candidate which occurred during the test and to understand why the candidate has a certain behavior at certain EOIs. Replay mechanism helps to confirm the assumptions regarding the root cause of cognitive load with the user. Once the root cause is identified the same information can be used for configuring the dynamic models in the CMN, so that optimal guidance information for the subsequent tests might be generated. The following parameters of the guidance information in the navigation system varies based on the corresponding cognitive load

• Distance to the entity at which the guidance information must be triggered beforehand.

• The symbols used for indicating the entity in the NS display, different symbols for low, medium and high cognitive load contexts

It is also possible to configure other parameters of the NS in scenarios with different cognitive load values. The methodology used for cognitive load measurement, which is the step 6 given above in the algorithm for conducting human in loop experiments is explained in Sect. 6. The result of different experiments by following the steps mentioned in Fig. 9 is given in Sect. 7.

6 Cognitive Load Measurements

Steering entropy method [12] with custom enhancements are used for measuring the cognitive load. The principle of steering entropy method is that when user can assess the actions needed in an effective way, the steering angle motion is in a smoother and predictive way. When driver is distracted vehicle deviates from the planned position and user must apply more corrective measures. These corrective measures decrease steering angle predictability. One of the main advantages of this method is that driver is not interrupted by the process of collecting the steering entropy data. The data is collected every 150 ms which is a justifiable best human response time. The error is calculated based on the predicted and observed values as shown in Fig. 10.

Fig. 10.

Steering entropy method visualized. X axis: Increment, Y axis: Steering Angle

Where configurable time interval (cti) is 150 ms by default. The entities corresponding to the EOIs are configured in the framework, so the report generated contain the cognitive load for the corresponding EOIs. A configurable radius around the EOIs is used for recognizing the entities which created the cognitive load. A custom-made location map** tool is used to map the cognitive load points to the location on the map. Cognitive load can be calculated as f(e(n)) + f(u(n)) where u(n) is a user specific value for the increment n for a user in a context and e(n) is the steering entropy value calculated for the increment n. The user specific values mainly use the static models. It is also possible to enhance u(n) with behavior models deducted from previous experiments.

7 Experimental Results

Multiple experiments are conducted with candidates of different driving experience and different age group. BeaCON enables to calculate the cognitive load for different EOIs, different instantaneous speed as well as for different accelerations. Patterns as well as root cause deductions are possible from the analysis data. One of the cognitive load measurement results from an experiment conduced with a candidate with very less driving experience is given in Fig. 11, Fig. 12, Fig. 13. The spikes indicate the locations where the candidate experiences more cognitive load than normal.

Fig. 11.

Cognitive Load at Different Locations for a candidate with very less driving experience for low cognitive load inducing route conditions without route familiarization. Y axis: Cognitive Load Points, X axis: Point of Interest (POI)

Fig. 12.

Cognitive Load at Different Locations for a candidate with very less driving experience for medium cognitive load inducing route conditions, once the candidate is familiarized with the route. X axis: Cognitive Load Points, Y axis: Point of Interest (POI)

Fig. 13.

Cognitive Load at Different Locations for a candidate with very less driving experience for high cognitive load inducing route conditions, once the candidate is familiarized with the route. Y axis: Cognitive Load Points, X axis: Point of Interest (POI)

Along with instantaneous cognitive load, it is also possible to calculate the cumulated value of the cognitive load value introduced by a specific EOI. This is done by aggregating the high cognitive load values observed, where measurement starting from a configured point before the user reaches the EOI. It can be observed that the more the candidate is familiarized with the route the cognitive load is reduced independent of harsh driving conditions as well as independent of less driving experience.

For example, the variation of cognitive load for different instantaneous speed and different accelerations for a candidate with very less driving experience is given in Fig. 14 and Fig. 15 respectively. The high cognitive load is experienced at the points with high instantaneous speed when the user drives through different EOIs as shown in Fig. 14.

Fig. 14.

Cognitive Load at Different Instantaneous Forward Speed for a candidate with very less driving experience for high cognitive load inducing route conditions without route familiarization. X axis: Cognitive Load Points, Y axis: Instantaneous Forward Speed in m/s

Fig. 15.

Cognitive Load at Different Acceleration for a candidate with very less driving experience for high cognitive load inducing route conditions without route familiarization. X axis: Cognitive Load Points, Y axis: Acceleration in m/s^2

For experienced driver, the cognitive load is less without the route familiarization itself for the route which induces low cognitive load. The test conducted with a highly experienced driver is given in Fig. 16.

Fig. 16.

Cognitive Load at Different Locations for a candidate with high driving experience for low cognitive load inducing route conditions without route familiarization. Y axis: Cognitive Load Points, X axis: Point of Interest (POI)

But for the route which induces high cognitive load, test conducted with highly experienced driver shows that the high cognitive load points are occurring as shown in Fig. 17.

Fig. 17.

Cognitive Load at Different Locations for a candidate with high driving experience for high cognitive load inducing route conditions without route familiarization. Y axis: Cognitive Load Points, X axis: Point of Interest (POI)

Analysis of the results deduct some of the information regarding cognitive load of the candidates. They are given below

• The familiarization of the route reduces the cognitive load significantly independent of the weather conditions for candidate with very less driving experience

• High speed contributes to high cognitive load points during the test

• A sudden deceleration is followed by a high cognitive load peak

• The timely guidance information from NS reduces the cognitive load related to maneuvers especially with harsh weather conditions

• Years of driving experience lead to less cognitive load values for low inducing cognitive load environments

• High cognitive load instances are less during low speed driving

Similarly, it is possible to conduct other similar experiments and measure the cognitive load values with different combinations in terms of behavioral aspects and technology aspects of the contexts.

8 Future Work

A fusion with other cognitive load measurement techniques can be done and is in progress for better analysis of the root cause of cognitive load. For example, steering entropy together with eye tracking mechanism can be used to understand the reason for cognitive load in more granular way. The contribution of cognitive load at each stage of human cognition can be analyzed for identification of the root cause.

The root cause identification can also make use of interpretable algorithms like LIME [13]. This enables more automation of the root cause identification task which is currently more manual in nature with the help of replay mechanism. Once the context which is creating the cognitive load is identified from all the collected data, the semantics must be identified for finding the root cause. A model which include human cognitive constraint is needed for automatic identification of this semantics which is shown in Fig. 18. Integration of this model, which is in the road map for BeaCON enable identification of the root cause of cognitive load with less manual work.

Fig. 18.

Detailed analysis for cognitive load root cause with reduced manual work

Mixed reality environment [14] can be integrated to the research framework for avoiding the creations of stubs as well as to get more close realistic driving environments with a 3D view. Also, mixed reality environment will give more provision for identification of the root cause, for example identification of where the user is looking at a moment of time as well as where the user must be looking ideally based on the context. Brake, accelerator and the steering can be from the real environment and the other entities of the driving simulator can be presented via virtual environment. The NS screen can be projected as a head up display in the virtual environment.

The behavior of CMN can adapt the rules suggested by [15] for human AI interaction. Especially CMN must update and adapt cautiously as well as to what is the reconfiguration occurred, why the same occurred, and inform the same to the user. Also integrating enhanced interpretability should not compromise the human centered nature of the system.

9 Conclusion

The research problem “Giving the driver adequate navigation information with minimal interruption” and its relevance is presented. Introduced BeaCON for analyzing this research problem as well as the importance of this is justified. Comparison with the state of the art is done as well as value addition by BeaCON is presented. The experimental results confirm that the cognitive load of the user can be measured quantitatively as well as the reason for cognitive load can be analyzed with the same. The roadmap of BeaCON for deducting more granular root cause information which can be used for effective reconfiguration of the behavioral models using machine learning algorithms are presented. BeaCON enables state of the art research for providing optimal guidance information by integrating with best suited machine learning algorithms as well as with modern driving simulators.

The design of BeaCON is aligned with the research in the area of connected vehicles and infrastructure conducted by many OEMs [16] for future vehicles. This design enables to incorporate new features to BeaCON based on connected vehicles and infrastructure which lead to more optimal navigation solutions, which can be easily accepted by the automotive industry. Also, the result produced by BeaCON can be used to enhance the connected vehicles and infrastructure.