1 Introduction

As it stands today, autonomous vehicles require human supervision and assistance to safely roam the roads, be it in a confined area or on the highway (Habibovic and Chen 2021). With the help of stable network communication and a remote operator interface, humans can monitor and command the vehicle from virtually anywhere. The benefits of remote operation supersedes having a safety driver on-board the vehicle in the driver’s seat from a cost perspective. However, simultaneously, many new challenges emerge from replacing the safety driver to a remote location and broadening the task of managing one vehicle to potentially managing multiple vehicles.

Remote control refers to operating a system at a distance which SAE Recommended Practice J3016 (SAE International 2018) defines as: “A driver who is not seated in a position to manually exercise in-vehicle braking, accelerating, steering, and transmission gear selection input devices (if any) but is able to operate the vehicle.”

2 A systemic view of remote control

In 2020, a pre-study on human factors in the remote operation of automated heavy vehicles showed that the success of the remote operation will be affected by many interdependent factors (Habibovic et al. 2020). For example, the situational awareness (Hosseini and Lienkamp 2016), telepresence (Bout et al. 2017), and workload (Chucholowski 2016; Neumeier et al. 2019) are factors that need to be addressed. Moreover, from an organizational standpoint, human operators in remote driving might need other skills and training compared to drivers of manually operated trucks. It is also essential to consider when transitions will occur between different control modes in the remote operation task (Michon 1985). The cognitive workload can be expected to vary when switching between tasks and control modes (Squire and Parasuraman 2010) if the operator environment is not adequately designed from a human factors standpoint. Other factors to consider is the operational design domain (ODD) in which the given vehicle and remote system will operate, and the degree of automation that the vehicle is capable of operating in. A less capable vehicle automation system will require the remote operator to be more engaged, while a more capable system will put the remote operator in a supervisory role. Figure 1 illustrates and exemplifies the human-technology view of remote operation from a systems perspective. The study presented in this paper mainly focus on task, HMI and operator aspects, while acknowledging that other perspectives have to be considered for a successful implementation in a real-life setting.

Fig. 1
figure 1

A systemic view of remote operation

Full size image

3 Remote control applications

Remote operation can be done on operational, tactical, or strategic control modes (Michon 1985), which are often intertwined, and carried out in combination with automated driving functions. For a feasible business case for remote operation in general, we envision that one operator will handle several vehicles. It presupposes vehicle automation capability that can operate autonomously most of the time, but not always and not in all conditions. In strategic and tactical control modes, it is likely that the remote operator will monitor and plan for several vehicles at a time and intervene only when something does not go according to plan. In the project, the remote operation was envisioned to be utilized for the following applications:

  • Remote assessment: Remote assessment enables the remote operator to investigate issues. In remote assessment, the information flow is one-way, i.e., the vehicle sends error messages and system state information to the human operator, but the operator does not directly control the vehicle. The supervisory task is always relevant and can be seen as a base case for remote operations.

  • Remote assistance: Remote assistance enables the remote operator to help the vehicle understand and give input to solve a situation. For example, by diagnosing an event, providing new or updated way points, or accepting a suggested route around an obstacle. However, the vehicle drives itself. For a single vehicle, the remote assistance task is sometimes relevant.

  • Remote driving: Remote driving enables the remote operator to “drive” or evacuate the vehicle in an emergency (e.g., at roadworks or when the vehicle is stuck in a complex or high-risk situation). For a single vehicle, this is rarely relevant but presumably very critical when it is needed.

In the present study, we explored the remote operation task in the mentioned applications: remote assessment, remote assistance, and remote driving by designing a remote operator interface and evaluating it in a simulator. The purpose of the study was to explore requirements for remote operator work and human–machine interaction in a hub-to-hub transportation setting.

4 Methodology

In the exploratory simulator study, 15 test participants (ten male, five female) were recruited within Scania. Eight participants worked with R&D of automated vehicle functions, three worked with R&D of remote operation of heavy vehicles, one worked as a control room operator in a testing facility, and one worked as a fleet manager of manual trucks. Experience from working as a truck driver was not part of the selection criteria. Six participants had previous experiences working in a control center or working with the development of control centres. Four participants stated no experience working with autonomous trucks. Their age spanned from 25 to 55 years (mean = 38.4 y, SD = 9.03 y).

4.1 Simulator study setup and procedure

A control room simulator was designed for human operators to assess, assist, and actively drive automated vehicles remotely. The control room simulator was divided into two workstations: one with a mouse, keyboard, and speakers for the assessment and assistance tasks and one with a steering wheel and pedals for the driving task. The two stations were set up on a regular office desk allowing the test participants to adjust the height according to their preferences. For seating, a regular office chair was used with wheels allowing the test participant to move between the stations (Fig. 2).

Fig. 2
figure 2

Workstation for assessment and assisting operations (left) and remote driving (right)

Full size image

The information provided on the screens to the participants included the following features. The position of each feature is marked with a number in Fig. 3.

  • Vehicle list (1): List of vehicles in operation, the name of each vehicle, its current activity destination, speed, estimated time to arrival, and deviations from each vehicle’s schedule.

  • Vehicle Information (2): Specific information about vehicles selected by the participant, such as planned mission, speed and direction. This information also included vehicle camera views and CCTV cameras from the hubs, a safe stop function, and an offboard software parking brake. It was also possible to see the departure time after loading.

  • Schematic view (3): The schematic view is a visualization of the vehicles’ positions in time in relation to each other (temporal distance relation). The view also showed the vehicles’ names and if the vehicles were driving between the hubs or were operating in the hubs.

  • Geographical map (4): The geographical map shows the vehicles’ geographical positions, directions, and activities. It was possible to zoom in and out and pan the view.

  • Alerts (5): The alerts were displayed in three places on the screen with a sound: (I) incoming alerts on top of the screen with a red border to draw the participants’ attention. These alerts disappeared after a few seconds not to distract the operator while being occupied with other tasks, (II) incoming alerts are displayed in the vehicle list next to the vehicle it concerned, and (III) a list of all the incoming alerts (old and new) in the right top corner of the screen. Red dots indicated new alerts (Fig. 3). These incoming alerts did not disappear until they were acknowledged or cleared by the participants.

Fig. 3
figure 3

User interface for remote assessment. Numbered areas in the map corresponds to the bullet list above

Full size image

4.2 Experiment procedure

The test leader introduced the test participants to the study (background, aim, and purpose) and also described their role as remote operators and the tasks they were asked to accomplish. The test leader also explained the control room simulator’s different functions and how to use the two workstations. The participants were also given time to practice with the two workstations to perform given tasks in the three control modes (assessment, assistance, and remote driving). The test leader assessed when the participant had reached the level of training and task knowledge needed to start with the test.

The participants were given three main tasks—to assess ten automated trucks driving between two hubs, maintain an even traffic flow, and act accordingly on messages and alerts from the system. The overall assessment was mainly done by monitoring the HMI showed in Fig. 3. The flow monitoring task was accomplished using the schematic view. The messages and alerts guided the operator to take a specific action and solve the occurred problem. Vehicles had different speeds in different parts of the map (highway and hub). The operators were asked to act on events that could occur between the two hubs. The ten trucks were monitored simultaneously, and the operator acted on events when a specific truck needed assistance. The events are described below Fig. 4.

Fig. 4
figure 4

A map showing the road between the two simulated hubs, Rosersberg and Cargo City (blue line). Text indicates where the scenarios occurred along the road

Full size image

Five events could occur along the route. The events were introduced by a second test leader sitting at another desk behind the participant and the primary test leader. The events consisted of different types of obstacles and problems.

  • Road works event: A road work which forced the vehicles to slow down when passing. The delays for the specific vehicles were shown in the vehicle list, but no other notice was given that a road work occurred. Action: no specific actions required by the operator.

  • Water puddle event: The water puddle made it impossible for the vehicle sensors to confirm that the route was safe to drive. The system requested the remote operator to drive through the puddle and then give back control to the vehicle (Fig. 5). Action: the operator takes control and drives the vehicle through the water puddle.

Fig. 5
figure 5

The water puddle event. The vehicle requests the remote operator to drive the vehicle. The red lines indicate where the vehicle cannot drive autonomously and needs driving support

Full size image

  • Bathtub event: A bathtub blocked parts of the drive lanes in both directions. The vehicle cannot pass unless it crosses the road's side lines. The system requests assistance from the operator. Action: the operator temporarily gives permission to the vehicle to cross the side-lines to pass the obstacle (Fig. 6).

Fig. 6
figure 6

The bathtub event. User interface for remote assistance. The turquoise line indicates the suggested route around the obstacle on the road. The vehicle has requested permission from the remote operator to cross the side lane marking to pass the bathtub blocking the way. The remote operator can accept or decline this suggestion

Full size image

  • Loading dock event: The vehicle stopped at a parking pocket and sent an alert to the operator requesting help to drive to the specific loading bay. Action: the operator takes control and drives the vehicle to the designated area for the vehicle.

  • Sensor degradation event: The automation system is not working due to sensor malfunction. The vehicle reduces the speed to 30 km/h. The system alerts the operator to initiate a safe stop manoeuvre. Action: the operator initiates a safe stop manoeuvre, and the vehicle drives to a safe stop position.

The test session took around 1.5 h to complete. During the test, the participants were also asked to think aloud, i.e., to orally communicate to the test leaders about their uncertainties, questions and thoughts, and to describe what they were thinking and how they were acting on the different messages and alerts. Operator workload was assessed after each of the five events using the NASA-TLX method for subjective rating (Hart and Staveland 1988). The participants rated mental demand, temporal demand, performance, effort, and frustration (physical demand was excluded). In addition, the overall monitoring task was assessed after finishing the experiment. After the sessions, the test participants were asked about their experiences, perceived challenges and problems, missing information, and other areas of improvement regarding the human–machine interfaces. Finally, the participants were asked if they had additional comments and thoughts to share about the system, the HMI and their experiences as a remote operator.

5 Results

First, the participants’ feedback from the think-aloud protocol and post-interviews on the HMI are presented. This is followed by an analysis of the assessment, assistance, and driving tasks and subjective workload assessments.

5.1 Evaluation of HMI components

The usefulness of the vehicle list received mixed feedback, where some information was seen as redundant. In the current study, (only) ten vehicles were monitored, and sorting, filtering, and prioritization were not necessary to see all vehicles. In a scenario where the number of vehicles is scaled up, functions to filter out to see, e.g., a list of all vehicles with deviations or in a defined area, will probably be of higher importance. Concerning individual vehicle information, test participants sought more assurance when they needed to make changes to the vehicle’s operation. More off-board support in terms of recommendations and what the changes mean in terms of impact on operations (e.g., loss of deliveries) were requested. In the simulated environment, the operators (naturally) optimized operations from the available information on system performance. In general, most of the test persons appreciated the task-based schematic view, which was given higher priority than the map. There were, however, questions regarding how to interpret the visualization. The test persons liked the simplicity of the visualization and how it supported kee** the vehicles at an even time distance. However, manual calculations on adjusting time distances could have been automated. Several participants mentioned that they did not use the geographical map to any great extent. The schematic view was perceived as more helpful to do the task of kee** an even flow. Apparently, the map feature was used more as a general overview and did not contribute much to the actual job in the simulated environment. For the alerts, several operators expressed the need to prioritise alarms to know when to drop a task to act upon another incoming alarm. In the hub-to-hub simulation, sensitive bottlenecks in the system could be, e.g., entrances and exits to the hub and blocking of loading docks, where several vehicles are quickly affected by a failing vehicle. Hence, alarm prioritization must be done not only based on vehicle parameters but also considering the working context and overall system impact of a failure. Several operators missed incoming alarms indicating the need for salient alarms where the user needs to confirm incoming alerts.

5.2 Evaluation of remote operator tasks—assessment, assistance, and driving

5.2.1 Remote assessment

The remote assessment task was characterised by kee** track of the ten vehicles, adjusting and maintaining an even flow of vehicles between the hubs, and responding to alarms by transitioning to remote assistance or remote driving. The HMI allowed for the delay of vehicles at each hub, respectively, as a tool to even out the flow. The participants expressed the expectation and need to be able to communicate with roadside assistance to remove obstacles and solve problems along the route. Several operators express that the job would probably become boring in the long run. According to several participants, ten vehicles could be monitored without problems from the assessment perspective if there are no other issues in parallel. Participants expressed worries about handling the situation in case of problems with several vehicles at once, or cascading events, pointing out the need to consider the organization and teamwork in the job design (Habibovic et al. 2020). The experiment illustrated how the role and tasks of the operators and factors, such as vehicle capability, deviations in the environment, task design, available information, and organizational support, are highly interdependent.

5.2.2 Remote assistance

The current study shows examples of how, to some extent, the tasks of assessing and assisting automated vehicles overlap. In our experiment, the remote assessment task was mainly a cognitive task with a high-level assistance control component to even out vehicle time distances. Also, it is mainly the control system that drives the vehicles, and the operator waits for alarms or messages to act on. In the assistance mode, the operator transitions to a more detailed decision-making situation with a single vehicle where understanding the situation and the surrounding context becomes critical. Several participants expected to handle problems (both momentarily and removing the cause) either by solving them themselves or handing over the issue to another instance, as well as warning other road users of dangerous traffic situations. This means that when designing an interface for a remote operator, the need to communicate with others safely and efficiently must be considered either by automatic messages through the system or by direct communication (e.g., roadside assistance, service units, terminal workers, and infrastructure). The teamwork aspect was not part of the test setup in this experiment but invites for future research on team coordination in remote operation.

5.2.3 Remote driving

The remote driving task was performed as the system prompted participants to take over the vehicle to solve the operational problems in the water puddle and loading dock events. The simplistic simulator setup for driving remotely in the experiment had some noticeable flaws. The limited field of view made it difficult to get an overview. The operator did not use the available camera views to any greater extent since they were placed in the assistance/assessment station and not readily available in the driving station. The operators reflected on this difficulty and suggested better fields-of-view and other sensor capabilities, e.g., remote operator driver assistance systems, to improve situational awareness. Despite the shortcomings of the simplified simulator setup, the results point out findings of importance to future remote driving implementations. The scarcity of information due to limited field-of-view also impacted the feeling of driving safely. Questions of liability also arose—“Who is responsible if I hit something, and the information was not sufficiently available?” One participant experienced that the only way to solve the situation is to trust the system, even though the participant would have preferred more information to feel comfortable. This could also occur outside the simulated environment and impose needs, e.g., communication and reassurance of situations. Another question posed was how liability would be handled during remote operations. Can the remote operator’s driving license be withdrawn if a “wrong” decision is made, even if the appropriate information is scarce? A challenge is that this scarcity of information may only become evident in hindsight, e.g., when accident analysis has established the facts.

Further, before the experiment we hypothesized that task-switching between assessment, assistance and driving would be effortful or at least time-consuming as operators needed to shift focus back and forth. However, the participants perceived these shifts as a natural part of the work and did not bring forward these switches as anything out of the ordinary. Test leader observations also concluded that it was easier than expected. Apparently, it was the tasks and problem-solving that imposed the operator workload rather than the task-switching.

5.3 Workload assessments

In general, the bath tub and water puddle scenarios received low perceived workload ratings using the NASA-TLX workload assessment. The sensor degradation scenario also received relatively low ratings but with a higher inter-quartile variation. The road works and loading dock scenarios received slightly higher medians. A Friedman test was performed but it did not show any statistically significant differences. Interviews revealed that the loading dock event was experienced as stressful mainly due to the limited field of view that was insufficient for the operator to maintain awareness of the surroundings, navigate to the correct loading dock, and drive in a safe way without hitting obstacles. There were no indications that shifting between the monitoring/assistance and driving workstations had an impact on perceived workload Fig. 7.

Fig. 7
figure 7

Boxplots showing combined NASA-TLX scores (min, Q1, median, Q3, and max) for all scenarios introduced during the simulation. Water puddle and loading dock were driving events. The bath tub and sensor degradation scenarios required assistance without driving. Road works caused a delay but without need for operator interference. Monitoring was performed continuously between events

Full size image

The field of view is an obvious improvement for future use of the simulator environment and points out the importance of seeing and sensing the surrounding environment. The road works scenario was a hidden problem while it did not yield any alarms and was more difficult to diagnose. Without omnipotent sensing systems, these types of “silent” events can be expected to cause challenges and a higher workload for future remote operators. The water puddle, bath tub, and sensor degradation scenarios represent events that has a more clear guidance from the control system, e.g., where the control system gives suggestions on how to act, or presents available choices that the automation implements upon approval from the remote operator. It is noteworthy that the distribution in the results is high, indicating large differences in perceived workload between participants. The difference between individuals, and suitable psychological profile for remote operators, is a subject for further investigation. The interview results indicate that individual differences depend both on, e.g., high perceived workload from a need to act without sufficient basis for the decision as well as the low workload of just accepting what the control system suggests (without much concern for the system’s ability to make correct judgements, i.e., possible over-reliance). This finding stresses the importance of looking deeper into the variation of the subjective NASA-TLX ratings and pair them with qualitative analysis of interview material.

6 Discussion and conclusion

The present study explored the requirements for remote operator work in supervisory control of ten heavy vehicles in hub-to-hub transportation by means of a simulator study. Despite a very short introduction and learning period before the experiment (15 min.), the participants could perform the assigned tasks indicating that the simulator had a basic level of usability. However, there are also several points for improvement. Participants asked for prognostic tools when making vehicle adjustments to be able to understand the fleet-level effect of vehicle-specific changes. This need resonates with the concept of situation awareness and the ability to project future status (Endsley et al. 2003), which should be supported through design. The schematic view of time distances was appreciated since it facilitated the task of delaying vehicles to achieve an even flow. It points out the relevance of task-based displays (see e.g., Jamieson et al., (2007) for an example from process control) that are tightly connected to the goals or KPIs of the operator work. In a real system, this could be more complex tasks such as kee** delivery times by re-planning and optimization tasks that are facilitated by task-based visualizations. Interestingly, the geographical map was not perceived as a key feature by the participants since the physical distance on the map did not provide sufficient information about the flow of vehicles when speeds varied between the hubs (a relatively short distance on the map could still be a long time due to low speed). The expressed need for support in task prioritization in relation to alerts from the vehicles also highlights the need for system analysis, systems modelling and risk analyses for remote operation. If contextual factors are not considered, appropriate decision support will be very difficult to provide. Here, scenario-based methodologies could provide useful input (Kettwich et al. 2022). Although only working for 1.5 h, some participants mentioned being bored from working as a remote operator. Boredom highlights the irony of automation (Bainbridge 1983)—when everything works, very little needs to be done, and staying vigilant is challenging for humans in general. Boredom is a known caveat in monitoring highly automated systems and seems valid also in this context. Since an operator cannot be expected to be vigilant over longer periods, appropriate alarm systems (Thunberg and Osvalder 2007) must also be designed and implemented for the remote vehicle operation type of systems. Designing the remote operator job in context, the worker should be expected to be part-time out-of-the-loop. Therefore, we argue that design effort should be focused on how operators can work themselves back into the loop at the appropriate abstraction level (from fleet to single vehicle and vice versa) as needed. In comparison to driving automation, remote operation of vehicles will also benefit from further advancing the understanding of the out-of-the-loop concept for this particular domain (Merat et al. 2019). The results from the simulator study show the importance of taking a systems perspective in the development and implementation of remote operation control centres. Aspects such as the operational context, control modes, vehicle capabilities, operator tasks, HMI, and organization of work have to be considered in parallel to ensure operational safety and performance (Habibovic et al. 2020). The study also showed how classical automation-related challenges such as situation awareness, boredom, vigilance, out-of-the-loop performance problems, and the need for appropriate alarm systems can be expected also in the domain of remote operation of heavy vehicles.

7 Future remarks

The present study emulated a certain remote operation solution with a specific set of circumstances where, e.g., the remote operation task spanned from remote driving to remote assessment, there was a set amount of vehicles to supervise, the task of the vehicles were to transport goods between two specific locations, the remote operators had experience in the type of task that remote operation entails, and there was only one remote operator in the remote center. The results of this study will thus be limited to these circumstances and although some of the results might be generalizable to other similar remote operation circumstances, it should be noted that differences in each factor will affect interrelated factors. A socio-technical system perspective on the topic of remote operation can aid in map** out these factors, and the relationship between them, to guide design of a remote operation eco-system.