Keywords

1 Introduction

Visuo-haptic displays that are used to provide visual and tactile sensations to users and maintain these two sensations consistent, both spatially and temporally, can promote natural and efficient user interaction in augmented reality (AR) applications. To facilitate effective visuo-haptic AR experiences, it is essential to ensure the spatial and temporal consistency of the visual and haptic sensations. Conventional systems allowed achieving the consistency by using optical combiners such as a half-mirror  [10] or video see-through systems including a head-mounted display  [2] to overlay visual information onto a haptic device. However, these systems require inserting visual display devices between a user and the haptic device, which constrains the field-of-view (FOV) and interaction space, and potentially deteriorates the user experiences. They also prevent multi-user interactions. A possible solution to this problem is to integrate a tactile panel into a flat panel display  [1]. This enables a spatiotemporally consistent visuo-haptic display to facilitate multi-user interactions without covering their FOVs. However, such displays are limited to flat surfaces at the moment.

Another promising approach of visuo-haptic AR display to overcome the above-mentioned limitations is to combine a projection-based AR system for displaying visual information and a haptic display attached on a user’s finger which is controlled by the luminance of each projected pixel  [6, 9]. This approach can be used to maintain the temporal and spatial consistency between the visual and haptic sensations while not being limited to flat surfaces owing to the projection map** technology. However, potentially, the displayed image quality may be significantly degraded, as the luminance of the original image needs to be spatially modulated depending on the desired haptic information.

In this paper, we propose a visuo-haptic display based on the projection-based AR approach to provide both visual image and haptic control information. The proposed system controls a haptic display attached to a user’s finger using temporal brightness information imperceptibly embedded in projected images using pixel-level visible light communication (PVLC)  [4]. The embedded information varies with each pixel. We embed the temporal brightness pattern in a short period of each projector frame so that the modulation does not significantly affect the perceived luminance of the original projection image. Owing to the short and simple temporal pattern, the haptic feedback is presented with an unnoticeably short latency. We can design a visuo-haptic display with various surface shapes as the projection map** technique can overlay images onto a non-planar physical surface. Multiple users can experience the system in which no visual display device needs to be inserted between the users and the surface.

We develop a prototype system comprising a high-speed projector that embeds spatially-varying haptic information into visual images based on VLC principle and a haptic display device that changes vibrations according to the obtained information. Through a system evaluation, we confirm if the proposed system can consistently represent visuo-haptic sensations. We can also use the system as a novel experimental platform to clarify the spatio-temporal perceptual characteristics of visual-haptic sensations. A user study is conducted to investigate whether the delay time and misalignment of visual-haptic asynchrony are within an acceptable range for user experiences.

2 Methods and Implementation

The proposed visuo-haptic AR display can represent haptic sensations corresponding to projected images when users touch and move the haptic display device on a projection surface. The system keeps the consistency of time and position between the visual and haptic sensations at a pixel level. Figure 1 shows the concept of the proposed system. The system comprises a projection system that can embed imperceptible information in each pixel of images and a haptic display device that can control a vibration.

Fig. 1.
figure 1

Concept of the proposed system

Full size image

2.1 Projection System

We utilize PVLC  [4] for embedding haptic information into projected images using a DLP projector. When a projector projects an original image and its complement alternately at a high frequency, human eyes see only a uniform gray image owing to the perception characteristics of vision. Although human eyes cannot distinguish this imperceptible flicker, a photosensor can detect it and use it as signal information.

We employed a high-speed DLP projector development kit (DLP LightCrafter 4500, Texas Instruments) to project images using PVLC. We can control the projection of binary images using the specified software of the development kit. Each video frame consists of two segments. The first segment consists of 36 binary images that correspond to each bit of synchronization information and data for controlling haptic displays and takes 8.5 ms for projection. The second segment displays a full-color image for humans, which also compensates for the luminance nonuniformity caused in the first segment and takes 12 ms for projection. Thus, the time for projection in a frame is 20.5 ms, which means the frame-rate is 49 Hz. We embedded the 26 bits data on x and y coordinates (x = 10 bits, y = 11 bits) and the index number of the vibration information (5 bits) corresponding to a projected texture image using PVLC.

2.2 Haptic Display Device Controlled by PVLC

We developed a wearable haptic display controlled by PVLC. It comprises a receiver circuit with a photodiode (S2506-02, Hamamatsu Photonics), a controller circuit, a vibration actuator, and a Li-Po battery. The controller circuit has a microcontroller (Nucleo STM32F303K8, STMicroelectronics) and an audio module (DFR0534, DFRobot) for playing the audio corresponding to the vibration. The microcontroller is used to acquire the position and spatial haptic information by decoding the received signals to determine a type of vibration, and send it to the audio module that drives the vibration actuator. We use the linear resonant actuator (HAPTIC™ Reactor, ALPS ALPINE) as a vibration actuator. This actuator responds fast and has good frequency characteristics over the usable frequency band for haptic sensation. Therefore, the proposed haptic display device (hereinafter referred to as “Device HR”) can present various haptic sensations.

Figure 2 provides an overview of the proposed system and the appearance of its user interface. We employed the data obtained from the LMT haptic texture database  [8] as a source of projected images and the spatial haptic information. This database provides image files with textures and audio files with corresponding vibrations. We stored the vibration information in the audio module of Device HR in advance, and the device presents haptic feedback by playing the received index number of vibration information.

Fig. 2.
figure 2

Overview of the proposed system—the system comprises a projection system with a screen and a haptic display device controlled by the obtained light information

Full size image

2.3 Latency Evaluation

The latency of the proposed system is defined as the duration from the time when the haptic display device is placed inside an area to the time when the device performs vibration. This latency (\(T_{late}\)) can be calculated as follows:

$$\begin{aligned} T_{late} = T_{wait} + T_{recv} + T_{vib} \end{aligned}$$
(1)

where \(T_{wait}\) is the waiting time for synchronization, \(T_{recv}\) is the time to receive the data, and \(T_{vib}\) is the time to perform vibration using the actuator. According to the estimation of the previous work  [3] and settings of the proposed system, we calculate \(T_{recv}\) equal to 8.5 ms and \(T_{wait}\) equal to 10.2 ms.

We measure \(T_{vib}\) by calculating the time from the moment when the microcontroller sends a control signal to the actuator of the two devices to the moment when the actuator is enabled. We project the sample image with the control information embedded for turning on the actuator in the left half of a projected image and that for turning off in the right half. We attach an acceleration sensor (KXR94-2050, Kionix) to the devices to detect vibration. We place the device at each on and off areas on the screen and conduct the measurement a 100 times using the microcontroller at each boundary of the area. As a result, the averaged values of \(T_{vib}\) are 66.5 ms for the Device HR.

Table 1 shows the values of \(T_{wait}\), \(T_{recv}\), \(T_{vib}\), and \(T_{late}\) corresponding to each device. \(T_{vib}\) of the Device HR is a sum of latency values of the audio module and that of the actuator (HAPTIC™ Reactor) itself. We measure the latency of the audio module 100 times and the average value was 48.8 ms. Therefore, we can estimate that the latency of the HAPTIC™ Reactor is about 66.5 − 48.8 = 17.7 ms.

Table 1. Delay time between providing the haptic information and visual information when using the proposed system—the evaluation was performed for Device HR
Full size table

3 User Study

We conducted a user study to investigate the human perception characteristics of the threshold time of perception of the visual-haptic asynchrony and the misalignment tolerance of the visual-haptic registration accuracy of the proposed system. According to the previous studies, this threshold time was approximately 100 ms  [7], and this misalignment tolerance was approximately 2 mm  [5]. However, we could not simply apply these values to the proposed system. The visuo-haptic displays of the previous studies covered the user’s view by visual displays from the user’s fingers to which the haptic feedback was provided, while our system allows the user to see the finger directly. In the present user study, we performed the two experiments using the proposed system. The first experiment was focused on the evaluation of the threshold time of perception of a visual-haptic asynchrony, and the second was aimed to estimate the misalignment tolerance of the visual-haptic registration accuracy.

3.1 Setup for User Study

Figure 3 represents the experimental situation of the user study. We employed a tabletop display in which the proposed projection system was built-in. The screen size of the tabletop display was 0.77 m \(\times \) 0.48 m, and its height was 0.92 m. We embedded the data of vibration control and the delay time into a projected image in each of the green and red areas separated by numbers. The haptic device turned on the vibration in the green area after the set delay time and turned off in the red area. We set the width of the moving area equal to 182 pixels in the projected image, and the resolution of the projected image was 1.22 pixels/mm in this setup; therefore, the actual width of the moving area was approximately 150 mm.

We implemented an alternative haptic display device for this study, which can present vibrations faster than Device HR by function limitation; this means it focuses on on/off of a constant vibration without the audio module. We used another actuator (LD14-002, Nidec Copal) in the device (hereinafter denoted as “Device LD”), and revealed the latency of Device LD is 34.6 ms by the same latency evaluation. Given that we could set the waiting time for sending a control signal to an actuator using the embedded data, the system was able to provide haptic feedback in the specific delay time (\(\ge \)34.6 ms).

Fig. 3.
figure 3

Experimental situations considered in the user study, (a) appearance of the experiment related to the threshold time of perception of visual-haptic asynchrony, (b) appearance of the experiment related to the visual-haptic registration accuracy (Color figure online)

Full size image

3.2 Participants and Experimental Methods

Ten participants (seven males and three females, aged from 21 to 24, all right-handed) volunteered to participate in the present user study. They were equipped with a haptic device on the index finger of their dominant hand. In the first experiment, we prepared 12 variations of the delay time from 50 to 160 ms at intervals of 10 ms (these delay times included the device-dependent delay time). The participants were instructed to move their index fingers from a red to a green area and answer whether they noticed the delay corresponding to the haptic sensation with respect to the moment when they visually identified the finger entering the green area. In the second experiment, we prepared 12 variations of misalignments from 0 (0 mm) to 26 px (21.32 mm). The participants were instructed to move their index fingers between the white boundary lines within each of the red and green areas as many times as they wanted. Then, they answered whether the timing of haptic feedback matched with crossing the red and green boundaries of the projected image. To influence the moving speed of the user’s fingers, we displayed a reference movie in which a user was moving his/her finger at the speed of 150 mm/s during the experiment. The participants performed each procedure 12 times as the projected image had 12 areas in both the experiments. We defined the 10 different patterns of interconnections between providing haptic sensations with a delay time or a misalignment and the areas to cancel the order effects. The participants experienced these patterns in each experiment, thereby repeated each procedure 120 times in total.

3.3 Results and Discussion

Fig. 4.
figure 4

Percentages of positive answers in the experiment for a threshold time of a visual-haptic asynchrony (left) and that for a visual-haptic regisration accuracy (right).

Full size image

Figure 4 represents the averaged percentage of positive answers obtained in the first experiment. Herein, error bars represent the standard error of the means, and the curve is fitted using a sigmoid function defined as below:

$$\begin{aligned} y = \frac{1}{1 + \exp (-k(x - x_0))}\times 100 \end{aligned}$$
(2)

In Figure 4 (left), we obtained the following values of parameters: \(k = 0.04\) and \(x_0 = 103\), as a result of the fitting and used these values in calculations. We identified the threshold time of perception of the visual-haptic asynchrony (\(T_{th}\)); It was set as the time at which users perceive the delay with a 50% possibility. The results indicated \(T_{th} \approx \) 100 ms, as presented in the fitted sigmoid curve. Similar results of \(T_{th}\) were reported in the previous research  [7], that supported the result of this experiment. As the latency of Device HR is 85.2 ms, the proposed haptic displays meet the requirement of having the delay time such that users cannot perceive visual-haptic asynchrony with their eyes.

In Fig. 4 (right), we obtained the values of parameters: \(k = 0.22\) and \(x_0 = 9.7\), as a result of the fitting and used these values as parameters of the sigmoid function. We identified the misalignment tolerance of the visual-haptic registration accuracy (denoted as \(L_{th}\)) equal to the length perceived as a misalignment by half of the users. The results indicated \(L_{th} \approx \) 10 mm, as presented in the fitted sigmoid curve. The result indicates that the proposed system can provide haptic feedback to users without a perception of misalignment if it is kept within 10 mm, which can be considered as design criteria for visuo-haptic displays. Additionally, \(L_{th}\) (10 mm) is larger than the value (2 mm) reported in the previous research  [5]. We can also conclude that the proposed system can extend the misalignment tolerance of the visual-haptic registration accuracy as the users can visually observe their finger with the haptic display in the system.

4 Conclusion

In the present paper, we proposed a novel visuo-haptic AR display that allowed eliminating visual-haptic asynchrony of the time and position perceived by the users. We implemented the projection system that could embed information into each pixel of images and the haptic display device that could control vibrations based on the obtained information. We conducted the user studies and revealed that the threshold of visual-haptic asynchrony obtained using the proposed visual-haptic display was about 100 ms for the time delay and about 10 mm for the position. From this result, we can conclude that the proposed display device can represent visual and haptic sensations in a synchronized manner as the system can represent them with the pixel-precise alignment at the delay of 85 ms. As future work, we will conduct similar user studies with various directions of the hand moving and with more participants to investigate the systematic thresholds. Furthermore, we will design a model to determine the vibration intensity and frequency of the haptic display based on haptic information corresponding to the texture image and the user’s movements on a display.