Abstract
This study focuses on design of user-adaptive tactile keyboard on mobile device. We are particularly interested in its feasibility of user-adaptive keyboard in mobile environment. Study 1 investigates how tactile feedback intensity of the virtual keyboard in mobile devices affects typing speed and user preference. We report how different levels of feedback intensity affect user preferences in terms of typing speed and accuracy in different user groups with different typing performance. Study 2 investigates different tactile feedback modes (i.e., whether feedback intensity is linearly increased, linearly decreased, or constant from the centroid of the key, and whether tactile feedback is delivered when a key is pressed, released, or both pressed and released). We finally design and implement user-adaptive tactile keyboards on mobile device to explore the design space of our keyboards. We close by discussing the benefits of our design along with its future work.
1. Introduction
Touchscreen-based mobile devices are pervasive, and a number of smartphone users prefer typing on a virtual keyboard in mobile phone as they spend more time on it [1]. Mobile-based text entry system using virtual keyboards has many benefits. Its portability allows people not to carry on their own keyboards, and it does not require significant learning effort from users since it sometimes adapts the training-free QWERTY layout. Due to this, it is even possible to perform eyes-free typing on mobile device [2]. Such benefits of text entry with mobile device can bring another opportunity in other computing areas such as virtual reality (VR).
A good example that effectively takes these benefits is VR. Typing in VR is becoming more important as it gradually replaces conventional communication media. In this sense, typing is required in a broad range of areas in VR such as education, media consumption, gaming, and training. However, typing in VR is still difficult due to many technical challenges such as poor hand and finger tracking, lack of tactile feedback, limitation of vertical field of view, and so on. Since user’s typing fingers are invisible in HMD-based VR, there is a need for alternative ways to achieve typing in VR. One good approach would be text entry on virtual keyboard using mobile device in VR. Since text entry on mobile device can provide portability, convenient, and even eyes-free typing, it can be a good match with VR as today’s VR focuses more on the interactive aspects of VR. In fact, a number of studies [3–5] showed the great benefits and promising results for using mobile device as text input.
In this study, we focus more on how to improve the typing performance and final user experience to the level of conventional mechanical keyboard typing scenario. Assuming that one of the important missing cues in current virtual keyboard typing is the absence of proper tactile feedback, we investigate the effect of various factors in the key click vibration on the final user experience. Two tactile cues during typing are essential for typing experience: (1) mechanical click feeling to provide a user confirmation of pressing and (2) the feeling of valley among keys to help a user navigating the fingers to locate a specific key. In virtual keyboard, the former is partly provided by playing vibrotactile feedback when clicking, but there are huge variations in individual preference on this virtual click feedback due to low fidelity of the feedback. In addition, the latter cue, while this significantly affects the typing accuracy, is not usually provided in virtual keyboard typing scenario.
To overcome these shortcomings, we proposed two new techniques to improve the quality of user’s typing experience. First, in order to cope with various demands of the users about the virtual key click, the relationship between the user’s expertise on typing and user’s preference on the strength of the virtual click feedback is experimentally investigated. This idea is based on our initial hypothesis that expert users do not rely on the feedback since they already have confidence on their typing skill, while novice users like to have feedback to strengthen their internal confidence. Using this relationship, we designed an adaptive key click strength control scheme. This scheme predicts user’s typing expertise in real time based on his or her typing speed and adaptively controls the strength of the feedback to provide optimum experience.
Our second approach is an initial investigation on providing cue equivalent to the valley among keys using vibrotactile feedback. To achieve this goal, vibrotactile feedback is designed in a way that its strength is systematically altered when a user clicks the edge of the key so that he or she can notice the edge of keys. Two different algorithms are provided, and the effect of these is investigated through user experiment.
We finally integrate all the proposed techniques in one virtual tactile keyboard and test the significance of the approaches by observing user’s workload using a TLX questionnaire.
The main contributions of the paper are as follows:(1)New algorithm to adaptively control the amplitude of key click vibration through user preference estimation based on his/her typing expertise(2)New key click vibration rendering technique to deliver virtual keys’ edge information(3)Experimental validations on the new above algorithms
2. Related Works
There exists a broad range of research works focusing on user experience improvement in virtual keyboard using sensory modality. Auditory feedback [6–8] can be a good solution to improve the user experience on virtual keyboards, but it has clear limitation in noisy environment. Visual feedback is the most commonly used modality for user experience and usability of virtual keyboard [9–27]. Sears [24] investigated the palm-style QWERTY keyboard, and they changed their keyboard size and location to investigate the user performance. Even though the size does not affect the user’s typing performance, they revealed that there is a difference in user performance between numeric type keyboard and QWERTY keyboard. Nakagawa and Uwano proved the relationship between location of keyboard and user’s performance such as typing error rate and typing speed. This clearly showed that the location of keyboard is also important factor for the user experience [21]. Mackenzie and Zhang [18] designed the new soft keyboard and their keyboard design improved the user’s text entry speed. Kim et al. [19] developed one key keyboard that can be worn on the wrist. Their keyboard could increase the input speed of users and accelerate their text entry learning ability.
Tactile feedback is another sensory modality that is available on virtual keyboards [8, 28–36]. Brewster et al. [28] found that typing with tactile feedback improves typing performance on mobile device. Users were able to enter more texts, make fewer errors, and correct more errors with tactile feedback. They demonstrated that tactile feedback is important role in touchscreen devices. Rabin and Gordon [29] studied the role of tactile feedback. They analysed kinematics of the right index finger with and without tactile feedback with special gloves and sensors. Their results suggested that tactile cues can provide information about the start location of the finger in which it is necessary to perform finger movement more accurately. Hoffmann et al. [30] developed a new tactile device for text entry. They used tactile feedback as a detector to prevent errors during the typing. Basic concept is that if a user types an incorrect key, the resistive force of the key becomes stronger so that the key makes the user to press the incorrect key harder. Kaaresoja et al. [32] used tactile feedback to mobile touchscreen and they demonstrated that tactile feedback is helpful in both usability and user experience. Nishino et al. [33] used tactile feedback as a communication modality. In their study, they used various types of vibration pattern such as strength, length, and effect. As a result, they found out the guidelines for building a practical system for tactile communication. Lylykangas et al. [37] focused on tactile feedback output delay and duration time. In their study, they found out the optimal duration and delay of tactile feedback when button is pressed. More recently, there exist a number of studies that are focused on perception [38] and performance [39–41] using tactile feedback in mobile and flat keyboard environments.
3. Study 1: Effects of Tactile Feedback Intensity
Study 1 investigates how tactile feedback intensity of the virtual keyboard in the mobile device affects typing speed and how strong user prefers based on their typing speed. We classified four levels of different feedback intensity and conducted a user study to observe the typing speed along with a follow-up questionnaire.
3.1. Apparatus
We built an Android-based typing program application with Mackenzie and Soukoreff phrase set [42]. In this program, a phrase is displayed on the top of the screen and the virtual keyboard is located under the phrase display area. The typing application stores typing speed, total elapsed time, and experimental condition. The typing application runs on Samsung Galaxy Note 3 due to its large touchscreen for typing phrases.
The vibration in the experiment is generated by the phone’s internal linear resonant actuator (LRA) with 200 Hz resonant frequency and 4.043 m/s2 maximum acceleration.
3.2. Participants
A total of eighteen university students participated in this study (mean age: 28.34, SD = 3.93). All participants were paid for their participation. They reported no disabilities. They also reported that they had prior experience of typing in English and familiar with virtual keyboards.
3.3. Experimental Design
We have four levels of tactile feedback intensity: None, Low, Mid, and High. Each has different intensity of tactile feedback signal of 0, 3.430, 3.798, and 4.043 m/s2, respectively. The signal frequency was set to 200 Hz for all four levels of signal. We chose four levels of signal intensity because of JND of tactile feedback since participants may not be able to distinguish among the levels if the number of level exceeds four. Number of level less than four may not give us enough data to analyse the experiment.
In each level, twenty phrases were randomly assigned. The order of intensity level was selected by Latin square to reduce any ordering effect [43].
3.4. Procedure
A number of preliminary typing trials were conducted prior to the main experiment in order to allow participants to be accustomed to the typing application. In the main experiment, we asked participants to take a seat and naturally hold the touchscreen phone with their two hands (Figure 1). We then asked participants to type as fast and as accurately as they can based on the given phrase using their two thumbs [44]. We kept silence in the room during the experiment to avoid any noise effect. For each session, we recorded typing speed in words per minute (WPM), total elapsed time, and intensity level.
(a)
(b)
After completing each condition, we asked participants to rate the application for the following three questions on a 7-point Likert-type scale from 1 (strongly disagree) to 7 (strongly agree): Typing speed—this tactile feedback is helpful for increasing typing speed; Typing accuracy—this tactile feedback is helpful for reducing typing errors; Preference—I prefer this feedback. We also had a debrief session to ask more questions about their typing experiences with different intensity levels after the main experiment.
3.5. Data Analysis
We measured several performance metrics for this experiment. We measured typing speed in words per minute (WPM), keystroke per character (KSPC) for measuring typing efficiency, and minimum string distance (MSD) [44] for typing accuracy.
3.6. Results: Feedback Intensity versus Typing Speed
Figure 2 shows typing speed with different intensity levels. It is clearly observed that the WPM increases when the intensity level increases. Although one-way repeated-measure ANOVA shows that there is only a weak difference in typing speed (F = 2.1, ), the trend is well observed.
3.7. Results: Feedback Intensity versus User Type
We further divided the participants into three groups based on their typing speeds—that is, Beginner, Intermediate, and Expert. Participants having average WPM value lower than 25 were categorized in Beginner group, and participants who scored average WPM higher than 30 were classified into Experts. Rest of the participants were classified as Intermediates. These thresholds are decided based on [44] and our observation on the distribution of the WPM. As a result, Beginner group and Expert group had 5 participants each, and Intermediate group had 8 participants.
As can be seen in Figures 3–5, people in each group prefer different intensity levels. Beginner group reported that they prefer Mid level of feedback intensity, whereas Intermediate group reported that they prefer both Mid and High levels of feedback intensity. Expert group reported that they prefer Low level of feedback intensity.
(a)
(b)
(a)
(b)
(a)
(b)
For different expertise groups and different measurements, one-way repeated-measure ANOVA tests were conducted. For all cases, no significant effect was observed. However, weak evidences (0.05 < value < 0.1) of the effect of feedback intensity on user preferences were captured through a post hoc test (Bonferroni test) in some conditions as shown in the figures as blue lines.
3.8. Discussion
In this study, we first found that increasing feedback intensity is likely to lead to higher typing speed. Although the results were only weakly supported (0.05 < value < 0.1), we clearly observed a linear trend of performance improvement with increased feedback intensity. We then observed the preference of different user group. Interestingly, people in Beginner and Intermediate groups preferred comparably higher intensity levels (Mid for Beginner and Mid/High for Intermediate) than those in Expert group (Low for Expert). It seems that High level was too strong for people in Beginner. However, they clearly preferred relatively higher intensity as they think that it helped them in increasing the typing speed and accuracy.
For people in Expert group, low tactile feedback intensity showed promising results as compared to other expertise groups. It is probably due to the fact that people in Expert group are already good at typing on virtual keyboards so that they neither need any strong key click feedback nor no feedback. This can be weak evidence that our initial hypothesis is on the relationship between typing expertise and the preference on feedback strength.
4. Study 2: Effects of Vibrotactile Edge
The goal of this study is to investigate the effect of vibrotactile feedback that encodes information about the keyboard edge. The idea of the encoding is that the distance between the finger and the intended keyboard is linearly mapped to the strength of the click vibration. Two different mapping functions were used: linearly increasing as distance increases, and the other way around. Additionally, we also investigated the effect of the moment when the encoded feedback is provided, at the moment of key pressing or at the moment of key releasing. In this experiment, we experimentally find the best suitable combination among various combinations of the two mapping functions and two moments of feedback by observing the users’ preference on their typing speed, error, and users’ confidence.
4.1. Apparatus
We built another Android-based typing program apparatus with customized virtual keyboard (Figure 6). In this keyboard, we provide different intensity of tactile feedback signal based on the location of user’s key press on the key. The intensity of signal is determined by the experimental conditions (see the later subsection for further explanation). This keyboard further supports temporal feedback conditions in which tactile feedback is delivered based on key’s touch state. For example, tactile feedback is delivered when key is pressed, released, or both pressed and released. The typing application stores user’s typing speed in WPM, number of key pressed, and the position of finger touched on the key.
(a)
(b)
4.2. Participants
A total of ten university students who did not participate in Study 1 participated in this study (mean = 25.34, SD = 1.92). Participants were paid for their participation. They reported that they are healthy and have no disabilities. They also reported that they had prior experience of typing in English and familiar with virtual keyboards.
4.3. Experimental Conditions
Table 1 shows the combinations of spatial-temporal feedback conditions—we call this tactile feedback mode.
As shown in Figures 7 and 8, and Table 1, we have three spatial tactile feedback conditions for this experiment: Linear Feedback, Reversed Feedback, and Constant Feedback. In Linear Feedback, tactile feedback intensity is linearly increased in regard to the distance of the touched location from the centroid of the key. Stronger tactile feedback is delivered when touched point is relatively farther from the centroid of the key. In Reversed Feedback, tactile feedback intensity is linearly decreased in regard to the touched point from the centroid of the key. In Constant Feedback, the constant intensity of tactile feedback is delivered no matter which area is touched within a key. We also provide no feedback when the centroid of key is touched, providing confidence to the users that they touched the centroid of the key—we call this Dead Zone.
(a)
(b)
(a)
(b)
We also have three temporal tactile feedback conditions: Pressed, Released, and Pressed-Released. In Pressed, tactile feedback is delivered when key is being pressed (touched). In Released, tactile feedback is delivered when key is being released. In Pressed-Released, tactile feedback is delivered when key is being pressed and being released.
4.4. Data Analysis
We measured several performance metrics for this experiment. We measured typing speed in words per minute (WPM), keystroke per character (KSPC) for measuring typing efficiency, and minimum string distance (MSD) [44] for typing accuracy. We also measured key distance ratio for typing accuracy. Given that P(x, y) is user’s touch point on the key, we divided the touched key into four regions using two diagonal lines to obtain a distance ratio from the centroid to user’s touch point P(x, y) (Figure 9).
If P(x, y) is in Surface 1 or 3, we calculate the ratio by the following equation:
And if P(x, y) is in Surface 2 or 4, then:where height and width are the sizes of actual virtual key.
4.5. Procedure
We conducted a typing test experiment to measure the typing performance on a virtual keyboard apparatus as shown in Figure 6. Similar to Study 1, we asked participants to type as fast and accurately as possible using their two thumbs while holding the touchscreen phone.
Combinations of three feedback intensity conditions (i.e., Linear, Reversed, and Constant) and three feedback delivery time conditions (i.e., Pressed, Released, and Pressed-Released) were used for this experiment. No feedback was also used for the comparison. Each session was composed of 10 lines of phrases and the session is repeated three times, yielding thirty tasks in total. Latin square was used to reduce the ordering effect.
After completing each session, we asked participants to rate their typing experiences based on the following questions on a 7-point Likert-type scale from 1 (strongly disagree) to 7 (strongly agree): Typing speed—this tactile feedback is helpful for increasing typing speed; Typing accuracy—this tactile feedback is helpful for reducing typing errors; Preference—I prefer this feedback; and Confidence—this tactile feedback gives me the confidence of key click. We added this Confidence question to find out whether degradation of confidence is caused by spatial or temporal feedback.
4.6. Results: Typing Speed and KSPC
Figure 10(a) shows typing speed in WPM. As observed in this figure, the feedback condition of the “Attached” and “Constant” feedback reached the highest. Figure 10(b) shows typing efficiency in KSPC. We observed that the lowest KSPC (meaning highest efficiency) was observed in the “Attached” and “Linear” condition and second lowest KSPC was observed in “Attached” and “Constant” condition. A two-way repeated-measure ANOVA confirmed that feedback condition was not a significant factor for typing speed (F = 0.276, ) or typing efficiency (F = 0.45, ).
(a)
(b)
4.7. Results: Different Ratio
We also measured the calculated ratio used in the data analysis section. This value can be an indicator of how close user’s key press is to the centroid of a key. Figure 11 shows the ratio for each experimental condition. As illustrated in this figure, the lowest ratio was observed in AC (Attached and Constant) condition, meaning highest typing accuracy.
4.8. Results: User Preference
Figure 12 shows the results of user preference on Accuracy, Speed, Comfort, and Confidence for temporal feedback conditions (Attached, Detached, Both, or None), respectively. It is clearly showed that participants preferred tactile feedback that is provided when key is attached (pressed) in terms of accuracy, but preferred feedback that is provided when key is detached (released) in terms of confidence.
Figure 13 shows the results of user preferences for spatial feedback conditions (No, Linear, Reversed, or Constant), respectively. It is clearly shown that participants preferred Reversed Feedback condition to provide key click confidence. However, participants preferred Constant Feedback condition in terms of typing speed.
A two-way repeated-measure ANOVA showed that there was a significant effect of both temporal conditions and spatial conditions for Confidence measurement (F(2, 90) = 3.326, = 0.05 for temporal condition and F(2, 90) = 4.122, = 0.02 for spatial conditions), but not for other measurements. Post hoc Tukey tests on both spatial and temporal feedback conditions showed the significantly different pairs (see red connecting lines in Figures 12 and 13 for value less than 0.05 and blue connecting lines for value less than 0.1).
4.9. Discussion
In this study, we confirmed that the spatial and temporal modifications on the vibration feedback did not give us a physical performance enhancement as shown in Figure 10. In addition, spatial modification of the feedback did not show significant effect on the user’s preference compared to the Constant Feedback case, while the feedback itself was clearly advantageous on giving a user confidence. Temporal modification on the feedback also did not have clear effect on user’s preference. This result indicates that the proposed temporal and spatial modification techniques are not very effective on physical performance and on user preference.
We presume that these results are due to the fact that participants are naïve to the information embedded on the feedback, so they did not successfully utilize the feedback. This can lead to the need of more intensive experiment that involves prolonged usage of the interface.
In next section, in order to partially avoid this problem, we examined the user’s workload when using a virtual keyboard with new feedback techniques. This time, participants were aware of the meaning of difference in the feedback.
5. User Evaluation: Work Load Analysis
Based on the findings from our first study, we learned that there exist different preferences of tactile feedback for different groups of users. We first discovered that people with higher typing performance (i.e., people who type fast with mobile keyboard) prefer comparably reduced intensity of tactile feedback, whereas beginner and intermediate-level users prefer comparably higher intensity of tactile feedback. From the second study, although there was no statistical meaning, we also had evidences that people prefer Attached with Constant tactile feedback to increase typing speed and provide confidence for their key click confirmation. Based on these findings, we propose a user-adaptive tactile keyboard on mobile device and compare it with the existing keyboard to explore the feasibility of our work (Figure 14).
5.1. Development of User-Adaptive Tactile Keyboards
We developed two different versions of user-adaptive tactile keyboards for this study. First one is feedback mode change-based keyboard. Basically, it measures the user’s typing speed in real time and adaptively changes its feedback mode based on user’s typing performance—we call this keyboard Feedback Mode Change Keyboard. For example, if typing speed is slow, the mode becomes DR (Detached-Reversed—meaning feedback is delivered when key is detached, and tactile feedback intensity is linearly decreased in regard to the touched point from the centroid of the key (Table 1)). If typing speed is increased up to intermediate level, the mode becomes BL (Attached/Detached-Linear—meaning feedback is delivered when key is attached and also detached, and tactile feedback intensity is linearly increased in regard to the touched point from the centroid of the key). If typing speed is further increased up to expert level, the mode is changed to AC (Attached-Constant—meaning feedback is delivered when key is attached, and tactile feedback intensity is constantly provided).
We also developed another keyboard that changes its tactile feedback intensity based on user’s typing speed but does not consider the feedback mode—we call this keyboard Feedback Intensity Change Keyboard. In this keyboard, we set a number of intensity levels and only allowed the level to change one level at a time. This is due to the fact that people felt uncomfortable when the tactile feedback intensity changed dramatically during our pilot study.
For the baseline of this experiment, we also used an ordinary virtual keyboard in which tactile feedback intensity is fixed and constant at all times. We set the size of all keys to the same size and added tactile feedback for space bar and delete key. We stored user’s typing speed and all the characters of the keys that they typed.
5.2. Participants
A total of ten university students who did not participate in Study 1 or Study 2 participated in this study. Participants were paid for their participation. They reported that they are healthy and have no disabilities. They also reported that they had prior experience of typing in English and familiar with virtual keyboards.
5.3. Procedure
Similar to studies 1 and 2, we asked participants to take a seat and naturally hold the touchscreen phone with their two hands for thumb typing (Figure 15). However, instead of asking them to type as fast and as accurately as they can, we asked them to type as comfortable as they can just like they perform the typing task during ordinary days. Since the goal of this study is to find the feasibility of our user-adaptive tactile keyboard in daily life setting, we focus more on comfort use of user-adaptive tactile keyboard than typing performance. For this reason, we provided 10 lines of multiple sentences to simulate real-world scenarios of daily text entry on mobile phone. Three sessions with three different keyboard types (Feedback Mode Change Keyboard, Feedback Intensity Change Keyboard, and baseline) were provided for each user.
(a)
(b)
After each session, we asked participants to evaluate the workload by providing a questionnaire based on NASA-TLX [45]. A total of six questions were asked: mental demand, physical demand, temporal demand, overall performance, frustration level, and effort (Figure 16).
In order to calculate the final workload of the conditions, we used analytic hierarchy process (AHP) to objectify the subjective response from the participants by assigning a weight to subjective response from NASA-TLX [45].
5.4. Results and Discussion: NASA-TLX
Based on the weight from AHP, we calculated the workload with each keyboard condition. The workload for Feedback Mode Change Keyboard and Feedback Intensity Change Keyboard is 31.44 and 38.72, respectively. The workload for baseline was 36.98. Compared to the baseline, the Feedback Mode Change Keyboard reduced the workload by 17%. This is notable since the feedback modes did not have statistical effect in study 2. From this, we can speculate that the feedback modes have positive effect to reduce user’s mental load, although the users do not have preference.
6. General Discussion
This work focuses on feasibility of user-adaptive tactile keyboard on mobile touchscreen. We noticed that there exists a number of tactile feedback that mobile device can provide. We also noticed that not every user likes the same and simple tactile feedback. We hypothesized that there exists a relationship between feedback intensity and users, and we further hypothesized that these users can be grouped by a factor—such as typing speed. We also believed that we can build a user-adaptive tactile keyboard for better usability and performance, and this can be extended to virtual and augmented reality.
We first observed how mobile users behave based on tactile feedback intensity and what intensity level that different user group prefers. We then studied if different tactile feedback mode affects the user preference based on user’s typing speed. Interestingly, we discovered that users preferred tactile feedback that is provided when key is attached (pressed) in terms of accuracy, but preferred feedback that is provided when key is detached (released) in terms of confidence. We also discovered that people in Beginner and Intermediate groups preferred comparably higher levels of feedback intensity (Mid for Beginner and Mid/High for Intermediate) than those in Expert group (Low for Expert).
Based on our findings, we built two different versions of user-adaptive tactile keyboard on mobile phone. We conducted a user study to investigate the feasibility of the keyboards by analysing the workload. As results, Feedback Mode Change Keyboard reduced the workload by 17 percent. We believe that this achievement will shed light on the development of user-adaptive tactile keyboard on mobile platform. Our future work will extend the present study by considering the use of adaptive keyboard in VR setting as typing is one of most challengeable tasks in VR and user-adaptive keyboard can be a good solution to address this issue.
7. Conclusions
This work investigates the effects of user-adaptive tactile keyboard on mobile touchscreen. We performed two studies to investigate the relationship between tactile feedback intensity and user preference. We then implemented user-adaptive tactile keyboards on mobile platform to verify their feasibility. We performed a user study to evaluate the workload of our proposed keyboard and showed the improvement in workload.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the NRF of Korea through the Basic Research Program (NRF-2017R1D1A1B03031272) and by the MSIP through IITP (no. 2017-0-00179) (HD Haptic Technology for Hyper Reality Contents).