Keywords

1 Introduction

Teleoperation, i.e. the manipulation of remotely located machines, is commonly seen in disaster relief, space exploration, and other dangerous or inaccessible environments. As an example, space robots often have to be tele-operated in the process of shuttle docking at space station and various maintenance and repair tasks in the space.

Resulted from the nature of space, there are several challenges in space robotic teleoperation. Time delay caused by long distance is one of the major problems, which may directly affect operators’ perception and interpretation of the current situation. It is critical for operators to maintain awareness of robotic arms’ position and configuration under time delay condition.

Time delay from action input to visual feedback display is an acknowledged shortcoming of current virtual environment and teleoperation technology [1]. There have been some studies focused on Single-Operator-Single-Robot teleoperation with time delay, most of which showed that time delay significantly reduces operator’s performance in error rate, completion time, efficiency and other measures of performance. Thompson showed that the linear relationship of time delay and completion time occurred when the constraint difficulty increased [2]. Hill found, in his comparison of seven performance measures in a time-delayed manipulation task, that time delay decreased the performance in terms of error rate and efficiency [3]. In his study, the efficiency was measured by fraction of time moving (MRATIO) and mean time per move (MBAR). In addition, it’s commonly accepted that operators tend to adopt a move-and-wait strategy in case of time delay [3, 4].

Compared with single operator teleoperation, teamwork in teleoperation makes it possible to undertake complex tasks. However, at the same time, teamwork makes teleoperation more challenging because operators have to work together harmoniously to overcome difficulties. Time delay may significantly influence team performance and safety by increasing collisions between robotic arms, leading to great damage [5]. There are few literatures on Multi-Operator-Multi-Robot teleoperation with time delay yet.

The control loop of Two-Operator-Tow-Robot teleoperation system can be shown as Fig. 1. Time delay occurs in every stage of information transition process, especially during the transition process between masters and slaves. Moreover, we can find that the result of a single operation commanded by a operator affects not only on the environment but also his teammate. Higher attention demand for the situation of teammate and his robotic arm is an important feature of teamwork. Chong worked on Multi-Operator-Multi-Robot teleoperation problem, tried to conduct visual and audition aids to decrease the influence of time delay, and found that unavoidable time delay is the most severe problem affecting team performance and safety in terms of error rate and completion time [5–8].

Fig. 1.
figure 1

The control loop of Two-Operator-Two-Robot teleoperation system

Full size image

In most of previous studies on teleoperation with time delay, team performance was mostly measured in terms of error rate and completion time. However, there is few discussion about other aspects of team performance, sauch as operators’ situation awareness and workload. According to NASA Generic Robotics Training (GRT) performance assessment metric [9], scan pattern and reach/joint limits and singularities are also important measures to evaluate operators’ skill and performance.

In this study on Two-Operator-Two-Robot teleoperation, team performance was measured with operational performance metrics, eye movement metrics and subjective rating. The goal is to compare teleoperation team performance with and without time delay.

2 Method

2.1 Participants

Sixteen pairs of right-handed male engineering undergraduate students in Tsinghua University participated in the experiment (mean age = 19.8, SD = 1.2) and were randomly assigned to teams. None of them had previous experience in operating a robotic arm, using joysticks or teleoperation.

2.2 The Virtual Robot Experimentation Platform

In this study, the experiment was conducted on the virtual robot experiment platform (V-REP) developed by COPPELIA ROBOTICS. The arm was simulated based on SIASUN SR6C industrial robotic arm, which had six degrees of freedom: shoulder pitch, shoulder yaw, elbow pitch, elbow yaw, wrist pitch and wrist roll.

As is shown in Fig. 2, the participants controlled the robotic arm with a posture joystick (right side) and a translation joystick (left side). Their eye movements were tracked by SMI iView RED.

Fig. 2.
figure 2

A participant in experiment

Full size image

Cameras were located on the wall of the room, above the table and on the end effector of the arms. The positions and angles of the cameras besides the wall and above the table were fixed while the cameras on the end effectors of the arms moved with the arms. While operating the arms, the operators could display simultaneously as many as three camera views and a digital panel window. The visual display is illustrated in Fig. 3 (take the view of the operator using the orange arm as an example). The view of the camera on the end of a robotic arm is located on upper left and the direction of the coordinate system changed with the movement of the arm. The upper right view and the lower left view are shot by the cameras on the wall and above the table respectively.

Fig. 3.
figure 3

Visual display of orange arm operator

Full size image

2.3 Tasks

Each participant was responsible for operating a robotic arm and cooperating with his teammate to complete the object moving task. As shown in Fig. 4, each participant was required to move an object to the target area correspondingly. Only when both objects were located in their target area, the task was completed. The task fail if it was not completed in 15 min. In addition, collisions and joint limit reach should be avoid as possible. Since the moving paths were designed to be crossed each other and verbal communication was mot allowed, each participant had to pay attention to the position and configuration of the robotic arm operated by his teammate while making decision on movement direction and velocity.

Fig. 4.
figure 4

Illustration of the task

Full size image

For each participant, this task involves four steps: (1) adjusting the posture of arm to aline the gripper to the object, (2) gras** the object, (3) moving the object and aligning it to the target area, (4) releasing the object. If the object was not put into the target area, it can be picked up and re-moved until the task success or the time was out.

2.4 Experiment Design

The independent variable in this study was time delay (two different levels, 0 s vs. 3.73 s). Specifically, although time delay occured in every stage of information transition process (as shown in Fig. 1), here we only studied the transition delay from masters to slaves.

Three types of measures were recorded to evaluate team teleoperation performance, including operational performance metrics, eye movement metrics and operators’ subjective evaluation, as listed in Table 1. Only data in successful trials was used to measure operational performance.

Table 1. Team teleoperation performance measures
Full size table

Each team was required to perform two replicates with each time delay level. To counteract learning and fatigue effects, the order was counterbalanced.

2.5 Procedure

Before the experiment session, the participants attended a 2-hour training session to learn about the operation of a robotic arm and the experiment task. The training consisted of 15 min Power Point introduction about the configure of the arm, the way to operate joysticks and the task, 65 min individual operation practice, 10 min rest and 30 min team operation practice. The task of team operation practice was similar to the formal experiment task and verbal discussions were allowed during the practice. All participants were able to complete the training session.

In the experiment session, the participants filled out the SART-10D and TWA questionnaires after each trial. Before each trial, the eye tracker was calibrated.

3 Results

3.1 Operational Performance

During the experiment, the position and status of slaves (robotic arms) and masters (the joysticks) were recorded with 2.5 Hz. According to these data, the operational performance were compared between different time delay levels from three aspects, i.e. accident rate (including collisions and joint limit reaches), completion time and controlling efficiency (including number of moves, MRATIO and MBAR). Note that when discussing accident rate and completion time, only data in successful trials were used. The data were analyzed with paired t-test and Wilcoxon signed rank signed test.

  • Accident rate

    The average collision number with 0 s delay (M = 3.46, SD = 2.99) was greater than average collision number with 3.73 s delay (M = 5.75, SD = 7.4). Similarly, when time delay increased from 0 s to 3.73 s, average number of joint limit reaches increased from 0.38 to 1.58, with standard deviation 0.51 and 3.41, respectively. However, statistical analysis show no significant difference on the number of collisions and joint limit reaches between different levels of time delay due to the large variance.

With time delay, the standard deviation of the number of collisions (F = 0.16, p = 0.01) and joint limit reaches (F = 0.02, p < 0.001) increases significantly, which implies that instability of the participants’ performance increases, meaning individual ability may play an important role.

  • Completion time

    Similar to the conclusion in Single-Operator-Single-Robot studies, completion time increased significantly when time delay presented (F = 6.471, p = 0.0217; see Fig. 5). Average completion time was lifted from 484.48 s to 610.70 s, meaning a 126.1 % increase.

    Fig. 5.
    figure 5

    Completion times (error bar represent standard deviation)

    Full size image

  • Control efficiency

    In this study, we evaluated the control efficiency through the number of master moves, MRATIO and mean time per move (MBAR).

There was no significant effect of time delay on the number of master moves and MBAR. The results showed that even in the scenario without time delay, the number of master moves was high, which suggested that in team teleoperation, the participants may tend to use move-and-wait strategy whenever there was time delay or not.

Hill found MRATIO, or the fraction of time moving, is the most sensitive variable to time delay [3]. The result of this study showed a similar result, MRATIO decreased significantly when the time delay increase from 0 s to 3.73 s (t = 2.49, p = 0.02; see Fig. 6), implying significant decrease of control efficiency.

Fig. 6.
figure 6

MRATIO (error bar represent standard deviation)

Full size image

3.2 Eye Movement

Eye tracking data was recorded from one of the participants in each team. As for saccade amplitude, it had a 6.8 % increase from 10.62° to 11.34°, though without significance. Similarly, the blinking frequency also showed no significant difference. These results might suggest that, in the task of this experiment, information acquisition is difficult, but the critical task is not visual searching.

3.3 Subjective Evaluation

Self-reported situation awareness according to SART questionnaire decreased slightly (0 s: M = 47.03, SD = 0.3; 3.73 s: M = 45.56, SD = 0.03) while team workload evaluated by team workload assessment technique (also was reported by participants themselves) showed no difference.

4 Discussion and Conclusions

In this study, we compared performance with 0 s and 3.73 s time delay to discuss the influence of time delay on team teleoperation performance.

According to the experiment results, time delay significantly influenced completion time and MRATIO. These results indicated that time delay significant decreased team performance because of less control efficiency and slower operating speed. Additionally, the significant increase of the variance of the number of collisions and joint limit reaches also suggested that inter-individual difference may play an important role and the operation performance became less stable. However, in this case, time delay did not significantly influence the eye movement or subjective evaluation. However, it would be difficult to say that the influence of time delay to Multi-Operator-Multi-Robot operation is less than its influence to Single-Operator-Single-Robot operation. Differently from the results in Single-Operator-Single-Robot studies, in team teleoperation, there is no significant transition from continuous to the interrupted move-and-wait strategy, according to the analysis of the number of master moves. In team teleoperation, the task is more challenging and operators have to pay attention to the situation of not only their own robot but also the robots of team members. Thus, operators may tend to use move-and-wait strategy whenever there is time delay or not.