Keywords

1 Introduction

For delicate operations of robotic arms under unstructured working environment, such as the Space Station Remote Manipulation System (SSRMS) [1], teleoperation surgeries, computer automation system itself is not able to fully meet the task demand. In these cases, manual teleoperation mode in which human operators obtain information through Human Computer Interface (HCI) still plays an indispensable role [4]. In order to evaluate operators’ performance effectively, objective metrics of robotic arm teleoperation are essential. Valid objective metrics help not only improve operator’s ability, but improve the system design accordingly as well, in this way, the success probability, the operation efficiency, and the safety risk, will achieve better states.

When teleoperating a robotic arm, operators mainly depend the visual information which is captured by the cameras to obtain the perception of the working environment [6]. There are 2 basic kinds of cameras: the cameras which provide local views, and the fixed cameras which provide overall views. An operator should do the cognitive processing and spatial imagination based on these visual information [7]. In addition to that, an operator must pay attention to the running state information, the value information, warning information of the robotic arm. Hence, teleoperation of a robotic arm, is a complicated task requires highly developed operation skills of human. And the evaluation of operators’ operation becomes an important issue of system design and training [8].

In the past 20 years, the training of SSRMS mainly depended on the subjective evaluation of NASA trainers who observed operator’s operation and gave their scores [5]. And a Genetic Robotic Training (GRT) handbook was created by NASA, Table 1 showed partial content of it. There were a few objective metrics in GRT such as whether a collision occurred, the times of reaching joint limitation, but still GRT was mostly consist of subjective metrics, which was not suitable for standardized assessment as well as self-study.

Table 1. NASA GRT standards (excerpts).
Full size table

Besides NASA, several researches explored the utility of objective metrics of robotic arm teleoperation were carried out. Fry et al. [10] picked and verified smooth operation input and multi-axes input as metrics in their study. Lamb and Owen [11] picked completion time, operation error times, and deviation as objective metrics in their study using SSRMS simulation environment. Akagi et al. [12] picked completion time, correct control times, moving distance and other 3 metric in a 3 Dimension of Freedom (DOF) robotic arm experiment. Yet these studies didn’t treat the metrics themselves as primary research goal, and their experiment tasks were mainly focused on fly-to mission, which didn’t consider the delicate moving operation of robotic arm’s end-effector.

In faced with teleoperation task of robotic arms, this paper summarized and created several objective operational performance metrics. And on a simulated robotic arm operation environment an experiment which asked operators to control the end-effector dock a target was carried out, the metrics were collected accordingly; by analyzing the differences of these metrics under different experimental conditions, this paper on one hand seek key display factors influencing this task, and on the other hand testified the validity of the objective metrics.

2 The Objective Metrics of a Robotic Arm Teleoperation Task

Based on the above-mentioned studies and the ISO standard: Ergonomics of human-system interaction – Usability methods supporting human-centered design [16], considering the feathers of robotic arm teleoperation task with hand-controllers, this paper raises 11 items of objective performance metrics, among which four items (8–11) are originally proposed, see Table 2.

Table 2. Objective performance metrics of robotic arm operation with hand controllers.
Full size table

Metric 8 (incorrect operation times) reflects operators’ cognitive ability of mental rotation when watching the video information provided by virtual cameras; less incorrect operation times means better awareness of the location and orientation relationship between the target and the robot arm. Metric 9 (translation efficiency of end-effector) reflects the operating efficiency on the three translation DOF, higher value of Metric 9 means longer moving distances in certain time which suggests a smoother operating of translation hand-controller. Lower values of Metric 10 and Metric 11 (operating efficiency of translation hand-controller and orientation hand-controller) mean less operating of the hand-controllers of certain deviation which means the efficiency is higher. These metrics will be tested and analyzed in the experiment.

3 Method

3.1 Experimental Environment

The experimental scene was simulated on V-Rep software platform. The robotic arm was built based on the **nsong SR6C industrial robot which had 6 joints so as to ensure the end-effector was able to move inside the workspace with 6 DOF (3 translation DOF and 3 orientation DOF). In the experiment, subjects must observe the visual information provided by virtual cameras on HCI and operated two hand-controllers (Laishida PNX-2103) with both hands to control the movement of the robotic arm. The terminate frame was set as the control frame.

3.2 Subject

16 subjects were selected to do this experiment. They were all male, right-handed, and their average age was 24.4 (s = ±1.5). The subjects had no experience of teleoperating a robotic arm and thus need a standard process of operation training.

3.3 Task

The operation task was designed as followed: in the simulated scene, a cross-shaped facility for docking was connected to the end-effector, and its direction was marked by a colored block, see Fig. 1. There were 2 docking target cubes whose surfaces were set with cross-shaped grooves, each cube had 3 grooves therefore a total number of 6 target grooves were in the working space. In each simulation trail, one groove would be selected randomly to be the docking target, and a colored block would be marked on it. An operator needed to control the end-effector till the cross-shaped facility moved into the groove with the 2 colored marks in the same direction. The success criteria were: the relative translation deviation was less than 0.004 m and the relative orientation deviation was less than 4° in each DOF. If the task could not be done in 240 s, it was considered a failure. During the operation, if any joint of the robotic arm reached the limitation position, or the robotic arm had a collision with other objects (especially the target cubes), a warning window would appear to remind the subject to move away the robotic arm to a suitable state.

Fig. 1.
figure 1

The simulated experimental scene.

Full size image

The manual operation of the task could be vaguely divided into 2 phases: firstly the subject watched the video image (mostly the overall cameras), controlled the robotic arm till the target surface appeared in the local camera’s view; secondly the subject mainly depended the local camera’s view, as well as the value information (when provided), adjusted the position and orientation of the end-effector, and completed the task.

The speed of end-effector was constant, and the difficulty of each trail was in the same level.

3.4 Design

The experiment was interclassed with 2 factors and 2 levels. The 2 factors were whether the value information was provided, and different number of virtual cameras (2 or 4), the layout plans were shown in Fig. 2.

Fig. 2.
figure 2

Four layout plans of HCI. a: no numerical information + 2 cameras (staring interests areas: end-effector camera + big global camera); b: no numerical information + 4 cameras(staring interests areas: end-effector camera + big global camera + far global camera); c: numerical information + 2 cameras (staring interests areas: end-effector camera + numerical information); d: numerical information + 4 cameras (staring interests areas: end-effector camera + big global camera + numerical information)

Full size image

The value information included: relative deviation value of 6 DOF, joint angle value of 6 joints (and their limitation range), running time of task and others. In the 2 cameras condition, one was the local view cameras which moved with the end-effector, and the other was the overall view camera which was fixed and able to see all 6 target surfaces. In the 4 cameras condition, besides the mentioned 2 cameras, 2 fixed cameras which respectively observed the 2 cubes from another position were added, since in one trial there was only one target cube, in this condition, only 3 cameras were valid.

Based on the attention allocation theory of Xu et al. [13], an operator should not pay attention to more than 4 items of information in the same time. On the condition of numerical value and 4 cameras, since only 3 cameras functioning, it reached this limit.

16 subjects operated 2 trails on each of the 4 conditions. In order to balance the study effect, the sequence of experiment was in Latin square design. The original performance data was recorded by the software system, and the data was calculated into the metrics in Table 2.

4 Results

The analysis of variance (ANOVA) towards 11 performance metrics showed that with value information all metrics except translation efficiency of end-effector were better than without value information, and 5 metrics (whether succeed, collision times, operation time, efficiency of translation hand-controller and efficiency of orientation hand-controller) were significantly better (p < 0.050). A larger number of metrics on 4 cameras condition were better than those on 2 cameras condition, yet only the metric of reaching limitation times was statistically significant better (F = 4.299, p = 0.040). The interaction analysis of the 2 factors showed that interaction effect significantly influenced the metric of translation efficiency of end-effector (F = 4.690, p = 0.032). The 7 metrics that showed significant differences were shown in Table 3.

Table 3. Comparison of performance metrics (who have significant difference) under 4 different experimental conditions.
Full size table

The success rate of all trails was 73.17%. Comparing the other 10 metrics between the success trails and fail trails, the significance analysis showed that all the metrics in success trails except collision times, multi-axe operation ratio, and translation efficiency of end-effector were significantly better than those in fail trails (p < 0.003). ANOVA within the success trails showed that whether to provide value information had significant influences (p < 0.026) on the metrics of collision times, angle accuracy and efficiency of orientation hand-controller, seen Fig. 3; while different camera layout plans merely significantly influenced (F = 5.417, p = 0.022) the metric of collision times. In the fail trails, the 2 factors had significant influence on none of the metrics.

Fig. 3.
figure 3

The metrics (times of collision, angle accuracy, efficiency of orientation HC) comparison under different experimental conditions.

Full size image

Whether succeed, collision times, reaching limitation times and operation time were 4 basic and important metrics, and using the efficiency metrics to do the correlation analysis with the 4 basic metrics would help to find which efficiency metrics benefited the basic metrics, and the most 3 related metrics to the 4 basic metrics are listed in Table 4. The results showed whether succeed and operation time were significantly correlated to all the other metrics. However, for the 2 safety metrics (collision times and reaching limitation times), the majority of efficiency metrics were not able to predict them, only 2 metrics (translation efficiency of end-effector and efficiency of orientation hand-controller) were significantly correlated to reaching limitation times.

Table 4. The 3 most correlated metrics to vital performance metrics.
Full size table

5 Discussion

5.1 The Influence of Experimental Factors

For the trails providing value information, subjects usually began to focus on the value information (mostly relative deviation value) and operated the robotic arm as to make the relative deviation value decline till meet the success criteria, hence the trails with value information had a significant higher success rate (F = 4.545, p = 0.035). In the task, collision only occurred when the end-effector was close enough to the target, and value information helped subjects to judge if there existed the risk of collision, hence the collision times were significantly decreased (F = 5.214, p = 0.024). Value information also helped subjects decide whether they were operating the hand-controllers correctly, if the relative deviation values were decreasing, it generally (not always) means the current moving direction of the hand-controllers were correct, hence the trails with value information had significant better metrics of efficiency of hand-controllers (F = 4.043, p = 0.047; F = 7.015, p = 0.009).

As introduced in Sect. 3.3, different camera layout plans mainly affected the task in the first phase. In the 4 cameras (3 valid cameras) layout trails, subjects could observe whether a joint was likely to reach the limitation more directly (than observe value information), so the metric of reaching limitation times was significantly lower (F = 4.299, p = 0.040). Besides this, despite that 4 cameras plan showed superiority than 2 cameras plan in most metrics, no statistically significant difference appeared. Additional cameras didn’t help the subjects to do the task much, the reason might be that the local camera which moved with the end-effector played a most important role in this operation, operators almost need to rely on it all the time, whereas the importance of overall cameras was weakened respectively, hence the overall cameras’ influence on performance declined accordingly.

On the interaction of the 2 experimental factors, there was a significant change (F = 4.690, p = 0.032) in the metric of translation efficiency of end-effector. It indicated that in the first phase of the operation, subjects’ comprehension of value information interacted with their observation of camera views, and it might cause the overload of information which made it difficult for the subjects to allocate their attention [13], hence the performance suffered a significant decline.

5.2 Analysis of the Performance Metrics

According to the task analysis, the experimental factors analysis, and the results of the metrics, the performance metrics will be discussed in the following:

  1. (1)

    Whether succeed and completion time. Whether an operation is successful or not is a core metric in this experiment as well as in any other robotic arm teleoperation task. In our task, completion time and docking accuracy (distance and angle) make no sense unless the operation succeeds, therefore correlations between these 3 metrics and whether succeed are not taken into consideration. Reaching limitation times has a significant influence on whether succeed (see Table 3), it is because when a joint reaches its limitation, a subject need to do complicated decisions and operations to get rid of the unsafe state, which will usually cost much time and make the task failed. The operation efficiency metrics put forward in this article, especially incorrect operation times and efficiency of hand-controllers, can straightly reflect the effectiveness of a subject’s operation, hence they are able to predict the metric of whether succeed and completion time well.

  2. (2)

    Safety metrics. Collisions are unacceptable accidents in most actual robotic arm teleoperation tasks, in this experiment, the efficiency metrics don’t have strong correlations with the metric of collision times, which means collision times should be an independent metric to investigate. Another important safety metric is reaching limitation times [9], more times a joint reaches its limitation position means more easily the joint will be damaged. To avoid that, it demands subjects keep their situation awareness (SA) [15] of the gesture of the robotic arm, which reflects onto the efficiency metrics, especially translation efficiency of end-effector and efficiency of hand-controllers (seen Table 3).

  3. (3)

    Incorrect operation times and multi-axe operation ratio. Similar metrics have been used in GRT and other researches [9, 11, 12], [15], yet in different tasks their detailed definitions differ. In our experiment, incorrect operation times is able to predict whether succeed, completion time and reaching limitation times, this metric reflects the subjects’ proficiency of operating the hand-controllers and comprehension of situational information. The metric of multi-axe operation ratio doesn’t show significant predictability in this task.

  4. (4)

    Translation efficiency of end-effector. This metric primarily functions in the first phase of the operation, and it is affected under the interaction of the 2 factors. This metric indicates the fluency of an operator’s control, so that reflects the consistency of his decision-making, hence this metric is a significant prediction of the 2 safety metrics and the metric of completion time.

  5. (5)

    Efficiency of hand-controllers. These 2 metrics (which were defined in this article) indicate operators’ ability of situation perceiving, and hand-controllers operation during the whole task. Less operation of hand-controllers (under the same task difficulty) indicates a more effective and adequate operation. These 2 metrics are significantly correlated with the metric of whether success, completion time and reaching joint limitation times (seen Table 4), which means trainings towards the improvement of the metrics of efficiency of hand-controllers would benefit the performance in the robotic arm teleoperation task displayed equations are centered and set on a separate line.

6 Conclusion and Future Work

Facing with the background of robotic arms teleoperation task (especially the space station robotic arm), this article abstracted and analyzed 11 items of objective performance metrics for evaluating an operation. These metrics were collected in a simulated experiment whose factors were whether providing numerical information and different camera layout plans (2 or 4). Under this experimental task and condition, primary conclusions are as follows:

  1. (1)

    With numerical information, operators perform better on the metrics of whether succeed, collision times, as well as many operation efficiency metrics. Under 4 camera layout plans, reaching joint limitation times decrease. Hence in this task 4 camera layout with numerical information is the better HCI design plan.

  2. (2)

    The metrics of reaching joint limitation times, incorrect operation times, and efficiency of hand-controllers are able to predict the metric of whether succeed. The metric of collision times is relatively independent with the other metrics; therefore, it should be examined individually.

  3. (3)

    The metrics of translation efficiency of end-effector and efficiency of hand-controllers which are put forward in this article, are able to predict the crucial metrics including whether succeed, reaching joint limitation times and completion time. So, they can be used to evaluate operation efficiency in this task and give instructions in operation training.

In future studies, the objective metrics in this article can be used to build a quantitative model for comprehensively evaluate the space station robotic arm teleoperation. Furthermore, operators’ situation awareness and the relationship between SA and performance metrics should be analyzed.