Keywords

1 Introduction

When drivers use automated driving systems (especially, the level 3 in the SAE definition [1]), they might be engaged in non-driving related activities, such as reading texts, matching a movie, gaming, and listening to the music [2]. However, “slee**” will not be allowed, because the drivers should be in the loop (physical control of the vehicle and monitoring the driving situation) [3] when the operational design domain of the automated driving systems exceeds its functional limitation. Monitoring of a driver’s drowsiness and predicting of the influences of the lower arousal level on a transition from the automated to manual driving are one of the fundamental functions to develop driver monitoring systems installed in future automated driving systems. This paper describes an investigation of evaluation indices for assessing the driver drowsiness both in virtual and real driving environments. We measured driver conditions while using an automated driving system and driver behaviors after the transition to manual driving, and then analyzed correlations between the driver states and the transition behaviors.

Several indices have been proposed to assess the driver’s arousal level in the general experimental conditions as well as in the automotive human factor domains [4]. In summary, the following changes of the evaluation indices, focusing on eye-related metrics, were suggested when a human felt sleepy:

  • Number of blinks increases or decreases [5, 6].

  • Blinking duration prolongs and blinking duration ratio increases [7, 8].

  • Blinking speed decreases [9].

  • PERCLOS (Percent of Eyelid Closure) increases [10].

  • Pupil diameter decreases [11].

  • Pupil low frequency fluctuation occurs [11].

  • Velocity and amplitude of saccadic eye movements decrease [12].

  • Slow eye movement occurs [13].

The driver’s blinking frequency increased and the driver’s blinking duration ratio increased when he/she uses an adaptive cruise control system and a lane-kee** system simultaneously in comparison with a manual driving condition [8, 14]. PERCLOS also increased in the highly automated driving condition than that in manual driving condition, which were suggested based on a driving simulator experiment [15]. When the authors compared the eye-related measurement indices between in a driving seat and in a passenger seat (assuming a human condition under an automated driving mode), the blinking duration was longer in the passenger seat than that in the driving seat [16].

Previous studies were conducted either in a simulator or in a real road condition, and little has investigated the evaluation metrics for assessing a driver drowsiness through virtual and real road traffic environments. In this study, we investigated indices of the driver drowsiness assessment that can be adapted to non-wearable detection under real road traffic environments. Non-wearable detection technique is an important to contribute to practical implementation of the driver monitoring systems.

First, we conducted a driving simulator experiment to select the metrics for assessing the driver’s alert level within a variety of eye-related measurement indices. In the driving simulator experiment, we used wearable eye-mark device to detect the driver’s eye and eyelid movements. A prototype of a driver monitoring system that can use non-wearable device (camera) has been developed based on the findings obtained from the simulator experiment. Second, a test course experiment was conducted to confirm a possibility of the driver state assessment through the driver monitoring system in a real sunlight environment. Finally, we carried out a field operational test to confirm whether the evaluation indices obtained from the driving simulator and test course experiments were valid for assessing a driver drowsiness for longer travel duration in real road traffic environments. We have also investigated an influence of the driver lower alertness on a transition behavior after a driver received information on RtI (Request to Intervene).

2 Driving Simulator Experiment

2.1 Methods

Driving simulator experiments were performed to evaluate the driver’s state, in terms of recognition capabilities, while he/she is using an automated driving system. Scenario assumed a situation where the vehicle was running on a four-lane highway (Fig. 1).

Fig. 1.
figure 1

Driving course in driving simulator experiment. Operational design domain of automated systems is “on highway”, and transition from automated to manual driving occurs before the exit of the highway.

Full size image

It was running alone on the left lane in the automated mode. After driving about 20 min, the vehicle approached a junction (a simulated highway exit), where the participant was instructed to suspend automatically the automated mode (RtI, visual icon and verbal message, occurred) and switch to manual driving before changing lanes to enter the junction. The RtI was presented 10 s before reaching the highway exit. This scenario simulates the situation where a switch to manual driving is required, while running in level 3 automatic driving, to exit from a highway. Assuming the possibility of gradual change in the driver’s arousal level before reaching the junction, the arousal level at the time of RtI presentation was evaluated from the facial expression and the subjective evaluation using KSS (Karolinska Sleepiness Scale) [17].

The participant was instructed to exit from the highway when he/she comes to an intersection junction. The following instruction was given to the participant: there is no need to monitor the automated systems and the surrounding driving conditions, and the automated driving system is not capable of handling procedures of exit from the highway. Because the automated drive mode had been turned off before the intersection came into view (the mode change took place before the “Exit Sign” was given), the participant was instructed to drive manually by himself to exit from the highway after he/she received the RtI. The contents of RtI, including presentation timing, interface design, and automatic shutdown of the automated systems just after the RtI were also instructed to the participants before the experimental trial.

The driving simulator consisted a real vehicle cabin, a 6 degrees-of-freedom electric motion system, and a 300-degree field of view screen. 44 drivers (26 females and 18 males, average age: 39.1 years old from 21 to 73 years old) participated in the driving simulator experiment.

Eye movements and Eyelid movements were measured, and we calculated the following indices: number of saccade, amplitude of saccade, velocity of saccade, pupil diameter, number of blinks, blinking duration, PERCLOS (Percent of Eyelid Closure). Eyelid motions were recorded using a wearable eye camera (eye-mark recorder EMR-9, nac Image Technology Inc.) to determine the percentage length of time of eyelid closure in one minute, where eye closure was defined as the state pupil diameter becomes unmeasurable. Eye movement data recorded by the same eye-mark recorder in two minutes before RtI were used to calculate the indices for saccadic eye movements.

2.2 Results

Driver States Indices in Automated Driving.

We divided the participants into two groups based on the results of the KSS: low arousal group (29 participants, average KSS: 7.93, SD: 0.92) and high arousal group (15 participants, average KSS: 4.47, SD: 1.81). Figure 2 presents the comparison results of the evaluation indices for assessing the participants’ drowsiness before the RtI. A significant difference was found in PERCLOS. PERCLOS was higher in the low arousal group than that in the high arousal group. The difference between the two groups in the pupil diameter suggests significant tendency. The pupil diameter decreased when the participants felt subjectively a drowsiness.

Fig. 2.
figure 2

Results of evaluation indices for assessing driver’s drowsiness while using the automated driving system. Each graph shows average and standard deviation (error bar) of the assessment metrics.

Full size image

Relationship Between Driver States Before RtI and Driving Performance After RtI.

Figure 3 presents the results of a relationship between PERCLOS and the steering operation response time. PERCLOS was calculated before RtI and the steering response was the driving performance index after RtI. The time to onset steering operation was significantly correlated with PERCLOS (correlation coefficient: 0.33, p < 0.05). The steering response time was later when the participant’s arousal level was lower quantitatively. The pupil diameter was also significantly correlated with the time to initiate the steering operation (correlation coefficient: −0.40, p < 0.01).

Fig. 3.
figure 3

Results of a relationship between PERCLOS before RtI and time to initiate steering operation after RtI. The data were divided into three categories based on PERCLOS (from 0 to 30%, from 30 to 65%, and higher than 65%). The error bar presents standard errors of PERCLOS and the steering response time respectively.

Full size image

The results obtained from the driving simulator experiment indicate that PERCLOS and pupil diameter were sensitive to the driver’s drowsiness, and these two indices were significantly correlated with the driving performance after the transition from the automated to manual driving. Pupil diameter is influenced by the brightness in surrounding environments. In a driving simulator, the brightness does not change, and thus the pupil diameter could be applied to the driver drowsiness assessment. The brightness is dramatically changed under real road traffic environments due to a time period of a day, weather conditions, shadows by several kinds of structures.

3 Test Course Experiment

A prototype of the driver monitoring systems has been developed to detect the driver states in the automated driving mode. The prototype consists of a camera that detects the position and direction of driver’s face, driver’s eye movements, and driver’s eyelid movements. The aim of the test course experiment is to estimate whether the prototype of the driver monitoring system can measure the driver drowsiness based on the driver’s eyelid movements.

3.1 Methods

Drivers were categorized into two groups, which are different from the viewpoint of driving conditions. In one group, the participants were instructed to continue monitoring your surroundings and the state of the automatic driving system to guard yourself against accidents. The driving time for one trial was 15 min. An automatically driven vehicle runs following a leading vehicle along a closed test course. There were no other vehicles on the test course. RtI presented at the end of the trial, and the participant subsequently steered the car manually. 20 drivers (9 females and 11 males, average age was 42.5 years old) were assigned to this group.

In another group, the following instructions were given to the participants: “While you are driving, you are not required to monitor the driving situations, and you do not need to pay attention to the situations around you and the state of the automatic driving system. While in automatic driving mode, please keep sitting in your seat quietly without doing anything.” The driving time for one trial was 25 min. The other experimental protocol was the same as the former group. 31 drivers (15 females and 16 males, average age was 47.7 years old) participated in the test course experiment with this scenario.

3.2 Results

Figure 4 presents the results of the driver state metrics measured via the driver monitoring system in the test course experiment. Road environments of a test course have little variety, and driving on a test course could be boring. The instruction of “without the need of monitoring” and the longer driving time could lead to the decrease of the driver’s arousal level while using an automated driving system. The number of blinks, the blinking duration, and PERCLOS suggest significant differences in the comparison between the monitoring and no monitoring conditions.

Fig. 4.
figure 4

Results of evaluation indices for assessing driver’s drowsiness in the test course experiment. (a), (b), (c) were calculated from the eyelid movement data for 1 min per one participant, and (d) was from the eye movement data for 2 min per one participant. Each graph shows average and standard deviation (error bar) of the assessment metrics.

Full size image

PERCLOS increased in the no monitoring group, suggesting a similar tendency to the driving simulator experiment. The number of blinks was lower and the blinking duration was higher in the no monitoring group. Although there were not statistically significant differences, the changes of the two indices when the driver might feel a drowsiness were the same as the driving simulator experiment.

4 Public Road Experiment

4.1 Methods

Instrumented vehicles for the field operational test of the driving monitoring systems were a Tesla Model S and a Benz E-class both equipped with Level 2 automated driving systems. 42 drivers (18 females and 24 males, average age: 33.5 years old, average driving experience: 14.5 years, non-participation in the simulator and test course experiments) drove each of the instrumented vehicles. The driving route was on Tomei and Shin-Tomei expressways in Japan. Time length of driving was about 3.5 h per one participant.

An on-site questionnaire was carried out in 15 min time interval by an operator who rode in the instrumented vehicle: the operator asked the participants about their arousal level using the KSS. The driver states were detected by the driver monitoring system, and the vehicle states were measured via a CAN-based recording system during the measurement trial.

Pseudo-RtI (visual icon and auditory message) presented 10 times per one driving route. The automated driving mode terminated after the driver received the pseudo-RtI, and he/she drove manually for about 5 min. Then the driver turned on the automated driving and continued the automatic driving mode. We recorded the driver’s hand movements installed in the instrumented vehicle (see Fig. 5) and calculated a time for gri** the steering wheel again after he/she received the pseudo-RtI (the driver grasped the lower part of the steering wheel while the automated system was active).

Fig. 5.
figure 5

Driving scene and recording image in public road experiment. The CAN-based recording system measured driving speed, accelerations, accelerator and brake pedals applications, steering operation, headway distance to a leading vehicle, position of the instrumented vehicle via GPS.

Full size image

4.2 Results

Figure 6 and Fig. 7 present the results of the relations between KSS and PERCLOS and between KSS and duration of blinking in the public road experiment, respectively.

Fig. 6.
figure 6

Results of PERCLOS based on KSS in the public road experiment. Average and standard deviation (error bar) of PERCLOS were calculated per each KSS score, using all of the data from the on-site questionnaires of 42 participants.

Full size image
Fig. 7.
figure 7

Results of duration of blinking based on KSS in the public road experiment. Average and standard deviation (error bar) of the duration of blinking were calculated per each KSS score, using all of the data from the on-site questionnaires of 42 participants.

Full size image

PERCLOS increased as the KSS became higher. Especially, the averages of KSS were more than 5% when the values of KSS exceeded 7 (sleepy). The duration of blinking increased when the participants subjectively felt sleepy, indicating a similar tendency to the relation between the KSS and PERCLOS. The durations of blinking were higher in the values of KSS more than 4, compared to those in the values of KSS less than 3. The results imply that the duration of blinking increases at an earlier drowsy level, whereas PERCLOS increases when the driver’s drowsiness reaches sleepy and very sleepy levels.

The average of KSS exceeded 5% when the KSS exceeded 7 (sleepy). The driver states while driving on the highways were categorized into two groups: high arousal (alert) and low arousal, based on PERCLOS measured before the pseudo-RtI presentation. Figure 8 presents the comparison result of the steering response time after the participant received the pseudo-RtI. The steering response time was longer in the low arousal group than that in the alert group, suggesting that the driver’s response was delayed when he/she became drowsy.

Fig. 8.
figure 8

Results of a relationship between PERCLOS before the pseudo-RtI and time for gri** the steering wheel again after the pseudo-RtI. We categorized the participants’ drowsiness into two groups based on PERCLOS, and compared the steering response time between in the high arousal (alert) group and in the low arousal group.

Full size image

On the other hand, there were large standard deviations in the time for gri** the steering wheel again after the pseudo-RtI. We describe a scatter plot of the relationship between PERCLOS and the steering response time (Fig. 9). In the case of the data above PERCLOS 5%, the correlation coefficient between PERCLOS and the steering response time was 0.34, similar to the correlation coefficient obtained from the driving simulator experiment.

Fig. 9.
figure 9

Scatter plot of the relationship between PERCLOS before the pseudo-RtI and steering response time after the pseudo-RtI. Large variation was found in the steering response time below PERCLOS 5%. The duration of blinking might contribute to estimating longer steering response time in the data below PERCLOS 5%.

Full size image

When we focused on the steering response time, which were measured in PERCLOS below 5%, we found out large variation of the response time from about 0 to 6 s. Then, we divided the response time data into two groups, based on the duration of blinking in which the threshold value was 180 ms from Fig. 7. The major part of longer steering response time more than 3 s was found in longer duration of blinking (more than 180 ms). We analyzed the images of the driver’s face recorded in the longer steering response. The video images implied that the participants were fighting off drowsiness.

5 Discussion

The results obtained from the driving simulator, test course, and public road experiments indicate that PERCLOS is highly related to driver drowsiness in the automated driving systems, and the camera-based driver monitoring system could detect such eyelid movements in real road traffic environments. PERCLOS is well used to estimate a human awake level and to predict consequential behavioral delays or errors in several kinds of experimental settings [18, 19]. Our findings suggest that PERCLOS can be applied to evaluating the driver’s drowsiness while an automated driving system is active and to predicting the delay of the driver’s response after a transition from the automated to manual driving.

One limitation of measuring PERCLOS concerns a depth of the estimated driver’s drowsiness level. PERCLOS could not detect a light slee** condition due to the necessity for detecting fully eye closures. The results of our public road experiment implied such limitation: Longer time to respond to RtI presentation was found in the lower PERCLOS region (when PERCLOS was less than 5%). The duration of blinking could be applied to an estimation of the lower drowsiness level: The duration of blinking increased at lower KSS values compared to PERCLOS, and the longer steering response time could be categorized based on the average duration of blinking corresponding to the KSS value of 4. Our findings imply that PERCLOS, duration of blinking, and their combination might contribute to evaluating the driver arousal level more precisely and to predicting the delay of driver’s steering response after a transition from the automated to manual driving.

Camera-based monitoring systems have limitations for detecting small ranges of velocity and amplitude of saccadic eye movements. We used the number of saccade in addition to the velocity and amplitude, suggesting no differences in the saccadic eye movements between when a driver awakes and when a driver feels sleepy. Further research should be necessary to investigate the possibility for applying the saccade-based driver assessment techniques.