Abstract
The most important aspect of virtual reality (VR) is the degree by which a user can feel and experience virtual space as though it is reality. Until recently, the experience of VR had to be satisfied with operations using a separate controller along with the visual and auditory elements. However, for a far more realistic VR environment, users should be able to experience the delicacy of tactile materials. This study proposes tactile technology, which is inexpensive and easy to use. To achieve this, we analyzed the unique patterns of materials through image filtering and designed a computing model to deliver realistic vibrations to the user. In addition, we developed and tested a haptic glove so that the texture of the material can be sensed in a VR environment.
1. Introduction
Owing to the development and wide distribution of computer hardware technology and virtual reality (VR) head-mounted display (HMD) devices, the VR content market is growing. VR has a significant amount of applications in fields such as advertising, shopping, travel, medical care, simulation, and education, as well as games and media [1–3]. To expand the VR market further, it is necessary to minimize the inconveniences of making the use of VR content to feel more realistic. One of the essential aspects in this context is the degree of realism in the touch of materials that the user can experience in a VR environment. Until recently, most of the VR technology was limited to audible and visual experiences. However, with the development of various controllers, people can now interact with the VR contents using the same gestures they use in everyday life. In addition, equipment such as a VR glove can be used to experience the sense of touch using vibrations while interacting with the VR contents. However, there is a limit to universal use due to the costliness of a VR glove and inconvenience in wearing them owing to the size of the gloves. As shown in Table 1, many of the currently released VR gloves in the market are expensive and inconvenient to wear because of separate devices such as finger artificial tendons. In contrast, the haptic glove proposed in this study, which allows you to experience the sense of touch through the texture vibration of an object, is less expensive and light in weight. Manus VR’s “Manus VR Glove” and Sensoryx’s “VRfree” provide hand-tracking technology to make users feel more immersed in the VR environment. However, as it does not provide any other function except for a simple haptic (on/off) switch, the effect of increasing immersion through touch is almost insignificant. Plexus’s Plexus VR Glove tends to be heavy because it uses a VR controller on the glove, not a direct hand-tracking technique. Dexmo VR Glove from Dexta Robotics provides contact resistance through hand-tracking technology and artificial tendons; however, it is heavy and expensive.
The following is a summary of our contributions. First, in this study, we have designed a computing model that combines the pattern of a material with haptic technology using an image-filtering algorithm. Second, we have implemented a haptic-sensing model with which users can feel a deeper sense of immersion in a virtual reality experience. Third, we have developed an inexpensive and convenient VR haptic glove to sense the surface characteristics of the material during a VR experience. In virtual figure model crafting (VFMC) [4] using a leap motion controller, we have measured the degree to which users felt the texture of clay through haptic when they shaped the virtual figure model using clay.
2. Related Works
There are visual methods for users to feel a deeper sense of immersion in a VR experience. However, natural manipulation methods for tracking hand movements such as leap motion do not seem to be actively utilized. Leap motion helps users use gestures in various set-ups, such as computers or virtual reality, through infrared sensors installed in the mouse [5–7]. VR gloves have become one of the most important devices in VR interactions, and various studies are under way to develop the next best VR glove. In 2017, there was a new product announcement almost every month [8]. VR gloves are being developed to create the ability to control and interact within the VR by capturing the movements and gestures of the hands and fingers in real time while wearing gloves on behalf of the controller. VR gloves will enable users to experience more realistic virtual reality, such as patients’ rehabilitation [9], virtual surgery and experimentation, implementation of work sites, and playing virtual reality musical instruments [10].
In addition to glove-based manipulation and interaction, it also provides virtual touch using vibration and heat. Choi et al. conducted a study of VR gloves that simulated the touch or capture of objects by zooming the resistance to the user when coming in contact with artificial tendons on the back of the hand [11]. Ma and Ben-Tzvi also provided a minimal action force and touch force conducted in a study for producing a haptic glove [12].
In recent years, some studies have shown that VR gloves are being produced that provide users with force feedback, such as the Dexta Robotics VR Glove as shown in Table 1 [13]. Research has shown that users feel more immersed in VR if they can experience touches in the forms of haptic interaction or heat. However, research by Kim et al. have shown that the important thing when experiencing VR with a VR glove is not to become immediately immersed in experiencing and feeling through the VR glove [14].
In addition, Perret and Vander Poorten mentioned in their study [8] that different people’s hands are of different sizes and shapes, and one hand of a person may not be the same as or symmetrical to the other. Hence, the most sensitive hand in the body should be lighter, smaller, and faster for the user to adapt the VR glove.
| ||||||||||||||||||||||||||||||||||||||||||||||||||
Algorithm 1: Material texture analysis. | ||||||||||||||||||||||||||||||||||||||||||||||||||
A fair amount of research is being conducted on the methods of analyzing materials. A study was conducted to measure the physical nature of a material to determine how it relates to the psychological feeling of contact, to measure roughness using an illumination sensor, and to quantify the material using friction and strength [15]. Kerzel et al. used optical force sensors to analyze the spectrum of texture data and classify the material [16]. Research is being conducted to make the analyzed materials experience the same force as the three dimensional (3D) surface and texture that comes in contact in the VR. Benko et al. used the matrix of the drive pin to render virtual objects, including detailed surfaces [17]. Hoang et al. conducted a study that increased immersion by providing the touch captured by the computer through passive deformed alloy gloves [18].
Additionally, recent progressive research studies regarding 3D control sensors and glove-based interface technology with effective results are mentioned in [4, 19, 20].
3. Design of Vibration for Haptic Glove
3.1. Material Texture Analysis
Virtual figure model crafting (VFMC) is a VR application where users can experience clay art in virtual reality through realistic and natural motion recognition using leap motion [21]. In VFMC, the user touches the clay and transmits the vibration of the touch when pressing the User Interface (UI) during the experience. The types of clay used in VFMC include candle clay, cork clay, foam clay, and normal clay. In this study, we used cork clay and foam clay, which have characteristics of the surface of the material and have considerable changes.
In order to make the user feel the material of the clay through vibrations, we first proceeded to analyze the patterns of the clay material. The surface of the clay was first photographed for the same. To make any changes clearer in the photograph, a height map was extracted and fast Fourier transform (FFT) was conducted.
FFT is an algorithm that can convert images into frequency components and analyze and process them at high speeds, as shown in Algorithm 1. It is applied to various fields such as for analyzing patterns and searching for images or [22] for sentence plagiarism [23].
If FFT is conducted on an image, a two-dimensional frequency space can be obtained. As the two-dimensional two-wave space obtained at this time is a shift image that converts the origin to the center, the image was divided into four equal parts using the ImageQuatered function and only the images in one of the quadrants were used. The images thus obtained showed the characteristics of the frequency component. To find the characteristics of the frequency component, the FindPeakVertices function was used. We selected a peak whose is or more using (images of three quadrants) obtained from the ImageQuatered function defined above. The repetition frequency can be computed by calculating the distance between coordinates and coordinates from the origin. If the position of the selected pixel is close to the origin then that pixel corresponds to low frequency. Conversely, a pixel at a position far from the origin corresponds to a higher frequency. In addition, the brightness of the FFT image can be used to determine the selection of the pixel. For instance, pixels with high brightness were assigned a relatively large vibration strength.
Table 2 shows the surface of the foam clay and cork clay, the picture for FFT extraction, and the results from FFT.
Figure 1 shows the original photograph of the cork clay texture (Figure 1(a)) and the FFT result in four equal parts (Figure 1(b)).
(a)
(b)
In the case of cork clay, white color was sprayed after photographing, so that it does not affect the extraction of the height map. In the image in Figure 1(b), only the pixels with brightness values higher than 230 have a specific pattern, excluding the minor peaks. The distances from the upper right and the origin can be calculated using the and coordinates of the image.
The image in Figure 1(b) shows only three pixels with a brightness of 230 or more, along with their distance from the origin and brightness values. Based on the FFT results, we further conducted material vibration modeling to express the material in the form of vibrations, as shown in Table 3. In this study, five vibration motors were used to realize the different vibrations on the five fingers. The rated voltage of each vibration motor was 1.5 V. The diameter of the vibration motor was 8 mm, and its thickness was 2.7 mm; hence, it was quite small and easy to use. The strength of the vibration motors can be expressed in revolutions per minute (RPM). RPM can be controlled by pulse width modulation (PWM). For convenience, we denote PWM in Hertz (Hz). We also converted Hz to RPM for better understanding, as shown in Table 3.
3.2. Vibration Glove
In the previous stage, we mapped rough materials with vibrations using their experience in VFMC and constructed a haptic glove so that users can feel these vibrations. The operation process of the haptic glove is shown in Figure 2. When a user touches an object during an experience in the VFMC, it firstly checks whether that object is a clay. If it is clay, then the type of clay is identified and a signal is sent. The signal for cork clay is A, and for foam clay the signal is B. These signals are transmitted through the port connected with Arduino.
If it is a normal clay or a button, it transmits a C signal. If the users remove their hands off the object they touched, it sends a D signal. The haptic glove connected to the computer through serial communication receives the signal and confirms what type of signal is being sent. In the case of A or B signals, vibration occurs at the interval and intensity of the signals stored in the array; in the case of C signals, a simple vibration is provided to the user through one vibration. In addition, when the D signal is stopped, the vibration is generated according to the behavior within the VFMC by stopping the vibration.
The haptic glove was based on the Arduino Uno board, and five vibration motors were used to transmit vibrations to all five fingers that were touching the clay in the VFMC. In addition, the HC-06 Bluetooth module was used to communicate with the computer for wire-free use, by not having to connect with the computer. The circuit diagram used in the haptic glove is shown in Figure 3(a). To increase durability, the white glove (as seen in Figure 3(b)) was placed inside the gray glove (Figure 3(c)).
(a)
(b)
(c)
In addition to the Arduino Uno board and Bluetooth modules, 9 V batteries and five vibration motors were used. The 9 V battery has a capacity of 550 mAh. The consumption of each vibrator is 70 mA. Assuming that the consumption rate is 0.7 and we continue to use five vibration motors, we can expect 1.1 hours. In practice, however, it is expected that the vibrating motor will be more usable than this because it will not continue to operate.
The price of the parts used is about $30, which is cheaper than the price of any VR glove currently sold in the market. Furthermore, the gloves produced in this study can be widely distributed to users. In contrast to other VR gloves, not many types of parts are used in our design, and since the method of conveying the touch is not resistance based but a method of analyzing the texture and mapping by vibrations, it is inexpensive and lightweight at only 120 g as it does not have a separate device attached at the wrist, as shown in Figure 4.
4. Experimental Results
4.1. Haptic Vibration Analysis
The period of vibration was provided differently based on the pixel distance from the origin. The strength of vibration was set differently based on the brightness of the FFT image. In order to express the strength of vibration on the five fingers, sections were defined and mapped, as shown in the aforementioned Table 3. In addition, vibrations of the thumb, index, middle, ring, and little finger were applied in the order of strength of vibration. If the strength of vibration was less than 5 Hz, the above order was applied, and the remaining empty vibration motor was applied with the same strength as the last vibration. The intensity of vibration felt by the user when the clay is touched in the VR is denoted by the curve, as shown in Figure 5, produced by applying the computing model to the information of the two materials being analyzed.
(a)
(b)
The graph in Figure 5(a) depicts the vibration curve of foam clay, which shows a considerable difference in intensity and change in vibration among the five fingers when vibration strengths of the ring and little fingers are set to the same strength as the difference between them is minor. The graph in Figure 5(b) depicts the vibration curve for cork clay, which shows a significant difference in intensity and change in vibration among all five fingers. In addition, Figure 6 shows that the surface of the clay used in the material analysis for foam clay is constant compared to the cork clay and the height change is not significant.
(a)
(b)
4.2. User Study
User evaluation was conducted to see how the haptic glove constructed through the proposed research affects the sense of immersion of the users in VR content experience, as shown in Figure 7.
The user evaluation was conducted with 20 people aged between 23 and 27 who were interested in VR and game production. The questionnaire consisted of five questions addressing the comfort and portability of the gloves, and thoughts about receiving haptic feedback while playing.
The steps followed for the user evaluation were as follows: First, the operation method was demonstrated to participants before the questionnaire was provided. Second, the participants clicked the button on the HUD UI next to their left hands after wearing the VR glove and they proceeded to create the foam clay and the cork clay. Third, they touched the clay to feel the texture haptic and finally they answered the questionnaire.
Table 4 shows the questionnaire that participants had to answer after experiencing VFMC. The questions are explained as follows: Question 1 and 2 asked whether the experience of the VR environment is more realistic with the haptic glove than without, and if the gloves are convenient to use during a VR experience. Question 3 addressed the difference in the textures of the clays as felt by the users.
In questions 4 and 5, we asked the participants about the difference in vibration intensities on their fingers and the realism in the haptic feedback during the VR experience. The results of the questionnaire showed a high score of 4 or more as shown in Figure 8. Question 1 was critically evaluated to determine whether the VR experience was more realistic with the use of the gloves.
5. Conclusion
In this study, we constructed and implemented VR haptic gloves through material analysis using a filtering algorithm to provide a realistic sense of touch to users in VR environments. Vibration mapping through material analysis is possible by implementing characteristics of different materials. However, there might be limitations such as fine differences that could not be detected or distinguished by some users because their fingers seemed to be sensitive to each other. In addition, there is scope of improvement in the glove construction; for instance, instead of using the Arduino Uno board, we can make use of smaller devices such as Arduino Pro Mini and smaller batteries such as coin batteries instead of 9 V batteries, making it even more lightweight and convenient to wear. Furthermore, we plan to vary the glove size in a future study. We observed through the user evaluation that the differences in hand size and the sensitivity of vibration to each finger are different for each person.
Our future research will be focused on providing users with touch experiences such as heat and force in addition to the material haptic. For that purpose, we plan to use only gloves instead of the leap motion used in this study to create economical, lightweight, and wearable gloves that can interact with users’ hands as well as the VR environments.
Data Availability
The data used to support the findings of this study have not been approved because of fund agency policy.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research was supported by the National Research Foundation (NRF) (No. 2018R1D1A1B07042566 and No. 2015R1D1A1A01057725).