Abstract
This paper presents a novel wearable device for gesture capturing based on inertial and magnetic measurement units that are made up of micromachined gyroscopes, accelerometers, and magnetometers. The low-cost inertial and magnetic measurement unit is compact and small enough to wear and there are altogether thirty-six units integrated in the device. The device is composed of two symmetric parts, and either the right part or the left one contains eighteen units covering all the segments of the arm, palm, and fingers. The offline calibration and online calibration are proposed to improve the accuracy of sensors. Multiple quaternion-based extended Kalman filters are designed to estimate the absolute orientations, and kinematic models of the arm-hand are considered to determine the relative orientations. Furthermore, position algorithm is deduced to compute the positions of corresponding joint. Finally, several experiments are implemented to verify the effectiveness of the proposed wearable device.
1. Introduction
The gesture is a natural and efficient way for communication and plays an important role in human-human interaction to express and transmit information. Therefore, the technology of gesture recognition has become a hot research topic. Through gesture recognition, results are used to communicate with computers, assess the kinematics of the hand, control electronic equipment, and so on. It has been wildly used in several application areas, such as rehabilitation, sports, and animation industry [1, 2]. In order to identify human gestures, we need to obtain the positions, speeds, directions, and other information of the gesture by a certain gesture capture technology.
At present, there are two main kinds of motion capture technology, namely, vision based and contact based devices [3, 4]. Vision based devices capture the video streams for analysis to determine the hand motion. On the other hand, contact based devices depend on physical interaction with the user. Based on vision gesture method, users generally do not need to wear collection equipment and can move more freely. But it is more susceptible to background, like illumination, occlusion, and other environmental factors. Furthermore, the camera frame rate and placement requirements are higher [5]. In comparison, contact based devices are easy to implement. Examples of contact based devices are mobile touch screens, EMG-based devices, and data gloves. EMG-based device is portable, but EMG signal is easily influenced by acquisition placement, individual differences, physical condition, and other factors. Moreover, fine finger and hand motion is still difficult to determine [6]. Data gloves use multiple sensor such as optical fiber sensor, pressure resistance sensor, pressure sensor, magnetometer, accelerometer, and gyroscope, to perceive the movement [7]. This kind of method can well reflect the spatial motion trajectory, attitude, and time sequence information of the hand by capturing the position, direction, and angle of the finger, which are restricted to the environmental conditions. But, at present, the product price is very expensive, so the development of low-cost data gloves has become the goal of our research. Different types of sensory gloves have been developed overtime, both commercial and prototype ones. The commercial products [8] usually use expensive motion-sensing fibers and resistive-bend sensors and are too costly for the consumer market [9]. Consequently, the prototype data gloves are developed to lower the cost of such equipment [10]. The flex sensors or bending sensors are integrated into the data gloves. However, the above sensors just measure the relative orientation of articulated segments by mounting the sensor across the joint of interest. This requires an accurate alignment of sensors with a particular joint. Moreover, recalibration is necessary to mitigate estimation errors due to the sensor displacements. General disadvantages of the data gloves are the lack of user customization for individual subject’ hands and obstruction of tactile sensing from the palmar surface of the hand. Often this inherently goes with the lack of mounting space for embedding the sensors in the cloth. To overcome these shortcomings, the inertial and magnetic sensors are induced.
In recent years, the MEMS technology has developed tremendously. The microinertial sensors have so many advantages like low-cost, small size, low-power consumption, large dynamic range, and so on. It has gradually become one of the most popular sensors for human motion capturing [11]. Meanwhile, the magnetic sensors are commonly used together with inertial sensors for accurate and drift-free orientation estimation [12]. Inertial and magnetic measurement unit (IMMU) has been proved to be an accurate approach to estimating the orientations of body segments without the external cameras [13]. It is nonobtrusive, comparably cost effective, and easy to setup and use. It also demonstrates higher correlation and lower error compared with a research-used visual motion capture system when the same motions are recorded [14]. Furthermore, the wearable inertial and magnetic sensors are becoming increasingly popular for the gesture motion capturing.
Gesture motion capture device based on microinertial sensors can generally be divided into three types, which are hand-hold type [15], wrist-wearable type [16], and glove type [17]. Compared with the hand-hold and wrist-wearable types, the glove type is available for having more numbers and types of inertial sensors and provides more accurate results of gestures. Therefore, the forms of gloves are most commonly used. The KHU-l data glove [17] consists of six three-axis accelerometers, but it can only capture several kinds of gestures. In [18], the data glove is developed based on sixteen microinertia sensors, which can capture the movements of each finger and palm, but the information of the heading angle is missing. In [19], the inertial and magnetic measurement unit is used, but it only uses four inertial and magnetic measurement units, which is unable to obtain the information of each finger joint. Power Glove [20] is developed, which includes six nine-axis microinertial sensors and ten six-axis microinertial sensors. It covers each joint of the palm and fingers, and motion characteristics can be better evaluated. However, it does not make full use of nine microinertial sensors, and in some state, the heading angle solution is instable so that it may lead to the estimation errors of the joint angle. The research shows that the current gesture capture device does not take into account the motion of the arm, and, at the same time, the movement of the hand cannot be fully captured. Therefore, we use the inertial and magnetic measurement units to develop a new gesture capture device, which can fully capture the motion information of the fingers, hands, and arms.
On the other hand, the IMMU-based device should be focused on the following two main aspects: calibration and fusion algorithm. Calibration is important for improving the performance of IMMU. Generally, there are two phases that included inertial sensors calibration [21] and magnetometers calibration in field [22]. The inertial sensors are of low cost and low precision, the deterministic errors such as bias and scale factor exist inevitably, and the calibration methods should be designed to improve the accuracy. Moreover, the magnetometers in the field would be affected by the iron-based materials, which generate their own magnetic field. Therefore, the measurements of magnetometers are the combination of earth’s magnetic field and the extra magnetic field caused by the environmental effects. Hence, magnetometers calibration should be implemented for lessening the disturbance and improving the azimuth angle. Fusion algorithm is another key procedure to estimate orientations by combining the signals of gyroscopes, accelerometers, and magnetometers. The Kalman filter is a useful tool for sensor fusion. The extended Kalman filter is [23] a general method for estimating orientations and has been applied in the products of AHRS [24]. Nevertheless, the EKF is not easy to choose the appropriate parameters and hard for computation. Then the complementary filter that combines two independent noisy measurements of the same signal is proposed [25]. Complementary filters are the high-pass signals provided by gyroscopes and the data from low-pass accelerometers and magnetometers which provide relatively accurate measurements at low frequencies fused to estimate the true orientation. Reference [26] proposed a nonlinear invariant observer, with respect to the symmetries of the system equations. In addition, the deterministic algorithm is another class for estimating the orientations.
The novel wearable device proposed is developed for gesture capturing based on the IMMUs. The paper is organized as follows. Section 2 presents the designs of the wearable device. Section 3 describes the calibration of the sensors and the algorithms of attitude estimation, and the position estimation is deduced. Section 4 reports the results of the calibration, the orientations, and gesture capturing experiments, which verify the effectiveness of the proposed device. Section 5 gives the conclusions of the paper.
2. Wearable Device Design
2.1. Inertial and Magnetic Measurement Unit Design
One of the major designs of the wearable device is the development of the low-cost inertial and magnetic measurement units. Commercial IMMU commonly contains processing units and transceiver modules except the MEMS inertial and magnetic sensors. This increases the weight and packaging size; hence, it is not suitable to use these IMMUs to put on the right positions of the fingers or arms. Moreover, it is difficult to add more IMMUs with small distance in order to benefit from redundant measurements or gain more accurate measurements of fingers because of the structure and size of the unit. Here, the MPU9250 [27] that deploys system in package technology and combines 9-axis inertial and magnetic sensors in a very small package is used. Hence, the low-cost, low-power, and light-weight IMMU can be designed and developed. It also enables powering of multiple IMMUs by microcontrol unit (MCU), which reduces the total weight of the system. Moreover, small IMMU can be fastened to the glove, which makes it more convenient and easier to use. The MPU9250 sensor is mounted on a solid PCB with dimensions of 10 × 15 × 2.6 mm and a weight of about 6 g.
Connectivity is another important issue in the design. Different types of connections between sensor units and access points on the body have been described in [28]. Wireless networking approaches for the connections are convenient, but the complexity of wireless networks is increased and the system has to trade off energy consumption for data rates. To avoid this problem, a wired approach is used in [29]. All sensor units are directly connected to a central controlling unit by cables, which lead to a very complex wiring. In the present work, a cascaded wiring approach [30] is used and developed by exploiting the master SPI bus of each IMMU. This approach simplifies wiring without any need for extra components. Since measurements reading from a string of IMMUs, the MCU need not switch to all the IMMUs to fetch the data, which leads to lower power consumption. Meanwhile, the textile cables are used to connect the IMMU to each other and to the MCU for increasing the flexibility. Here the STM32F4 microcontroller is used to develop the MCU.
2.2. Device Design
After determining the above designs, the wearable design of device can be determined. There are thirty-six IMMUs in the device, and a pair of device is separately put on the right hand and left hand. Each side has eighteen IMMUs, which covers all the segments of the arm, palm, and fingers. Each string deploys three IMMUs, and six strings are used. Five of them are used to capture the motions of the five fingers, and the other one is used to capture the motions of the palm, upper arm, and forearm. The battery and MCU are attached to the wrist. The wearable device is shown in Figure 1.
The proposed wearable device is designed based on the low-cost IMMUs, which can capture more information of the motion than traditional sensors. The traditional sensors used in the data glove such as fiber or hall-effect sensors are frail. Nevertheless, the board of inertial and magnetic sensor is an independent unit. It is more compact, more durable, and more robust. Commercial data gloves are too costly for the consumer market, but the proposed data glove in the paper is low-cost (US$ 200). Moreover, the proposed wearable device can not only capture the motion of hand but also capture the motion of arm, and the estimated results of motion are outputting real time.
3. Methods
In this section, the gesture capture algorithms are presented. First, models of the sensors are described, and the calibration method is presented to improve the measurements of the IMMUs. Then the absolute orientations filter based on quaternion-based extended Kalman filter is deduced, and the relative orientations algorithm integrated kinematics of arm-hand are proposed. Finally, the position estimation algorithm is deduced.
3.1. Models of the Sensors
Before analyzing the models of inertial and magnetic sensors, two coordinate frames that are the navigation coordinate frame and the body frame need to be set up. The orientations of a rigid body in the space are determined when the axis orientation of a coordinate frame attached to the body frame with respect to the navigation frame is specified. According to the frames, the sensors’ models are separately built as follows.
(1) Rate Gyros. Because the MEMS rate gyros do not have enough sensitivity to measure the earth angular velocity, the model can get rid of the earth angular vector. And the output signal of a rate gyro is influenced by noise and bias; that is,where is measured by the rate gyros, is the true value, is the gyro’ bias, and is the noise that supposed to be Gaussian with zero-means.
(2) Accelerometers. The measurements of accelerometers in the body frame can be written aswhere is 3 × 3 matrixes measured by the accelerometers, denotes the Orientation Cosine Matrix representing the rotation from the navigation frame to the body frame, and are the gravity vector and the inertial acceleration of the body, respectively, expressed in the navigation frame, and m/s2 denotes the gravitational constant; is 3 × 3 matrixes that scale the accelerometers outputs; is the vector of accelerometers’ bias; is the vector of Gaussian with zero-means.
Commonly, the absolute acceleration of the rigid body in the navigation frame is supposed to be weak or the rigid body is static. Then, the model of accelerometers can be simplified as
(3) Magnetometers. The ideal magnetic vector expressed in the navigation frame is modeled by the unit vector . The measurements in the body frame are given bywhere is a 3 × 3 matrix that scales the magnetometers outputs, denotes the disturbance vector including magnetometers’ bias and magnetic effects, and is the noises that supposed to be Gaussian with zero-means.
According to the above presentations, we can know that the models of accelerometers and magnetometers include errors, which would lead to errors in estimating orientations. Hence, the calibration should be implemented to improve the accuracy of the sensors.
3.2. Calibration
In order to improve the accuracy of the IMMU, calibration is a necessary procedure. The typical calibration method is to assign the inertial sensors to a known angular velocity and linear acceleration. This method normally needs some specific equipment, such as turntable. Here, a novel calibration method is proposed that do not need the specific equipment and is easy to implement. The calibration procedure has two steps. First, the accelerometers and magnetometers are calibrated by the offline procedure. Then the gyro is calibrated by the online procedure.
The unified mathematical models of the calibration of the accelerometers and magnetometers can be used as follows:where triaxial sensor model is written in the vector form and , are constant offset that shift the outputs of sensors.
The calibration is the process that determines coefficients , , , to improve the measurements of sensors. When the unit has omnidirectional rotation, the magnitudes of the true magnetic field and gravity field remain constant, and the loci of the true magnetic field measured and are spherical. Meanwhile, the measured and are ellipsoids, and they can be expressed as follows [31]:where , , , , and and are assumed noise.
Equations (6) are the expressions of the ellipsoid. In other words, the measurements are constrained to lie on an ellipsoid. Thus, the calibration of the magnetometers and accelerometers is to seek ellipsoid-fitting methods to solve the coefficients of , , , . And the least squares algorithm is commonly used to determine the parameters. The device is rotated adequately to ensure that each unit gets enough measurements to determine the calibration parameters by least squares algorithm. Hence, the accelerometers and magnetometers of all units of the device are entirely calibrated.
After calibration, the magnetometers are calibrated to lessen the environmental magnetic effect, and the biases of the sensor outputs are compensated. Accordingly, the previous equation can be rewritten as follows:
Since the scale and bias errors from the accelerometers are calibrated, the models of the accelerometers rewritten as follows:
The offline calibration is implemented to determine the bias and scale of the accelerometers and magnetometers. And the online calibration is implemented to remove the gyro bias. We keep the data glove stationary for a while before use. So the readings are zeros. The bias can then be computed by the mean value of the measurements. The model is expressed as follows:where are the measurements of gyros, are the true angular velocity, and are the noise of gyros.
The calibration parameters of the sensors are compensated into the measurements so that the accuracy can be improved for further work.
3.3. Orientation Filters
Based on these kinds of sensors, two independent ways can determine the attitude and heading. One is determined by open-loop gyros. The angular rate of the rigid body is measured using gyros with respect to its body axis frame. The angles are estimated by the open-loop integration process, which has high dynamic characteristic. However, the gyro errors would cause wandering attitude angles and the gradual instability of the integration drifting. The other way is determined from open-loop accelerometers and magnetometers. The orientations can be correctly obtained from accelerometers and magnetometers in the ideal environment. However the disturbance and noises would lead to large errors and make results lack reliability. The two ways are both quite difficult to achieve acceptable performance. Sensor fusion is a great choice to attain the stable and accurate orientations. Then, the fusion algorithm of quaternion-based extended Kalman filter will be deduced.
The frames are shown in Figure 1. In the figure, the global frame is and local reference frame is, respectively, located in each IMMU. The global reference frame -axis is defined along the axial axis (from the head to the feet) of the subject, -axis along the sagittal (from the left shoulder to the right shoulder) axis, and -axis along the coronal axis (from the back to the chest). The local frame -axis is defined along the axial axis (normal to the surface of IMMU along the downward) of the subject, -axis along the sagittal (from the left side to the right side of the IMMU) axis, and -axis along the coronal axis (from the back to the forward of IMMU). Meanwhile, two assumptions about data glove in use are made: the body keeps static and only arm and hand are in motion; the local static magnetic field is homogeneous throughout the whole arm.
3.3.1. Absolute Orientation Filter
Combining the measured angular velocity, acceleration, and magnetic field values of one single IMMU, it can stably determine the orientations with respect to a global coordinate system. The global coordinate and each coordinate of IMMUs are shown in Figure 1. The transformation between the representations of a 3 × 1 column-vector between and is expressed aswhere quaternion , is the antisymmetric matrix given by
The attitude matrix is related to the quaternion bywhere is the identity matrix.
The state vector is composed of the rotation quaternion. The state transition vector equation iswhere the gyro measurement noise vector is assumed small enough that a first-order approximation of the noisy transition matrix is possible.
Then the process noise covariance matrix will have the following expression:
The measurement model is constructed by stacking the accelerometer and magnetometer measurement vectors:
The covariance matrix of the measurement model iswhere the accelerometer and magnetometer measurement noise and are uncorrelated zero-mean white noise process and the covariance matrixes of which are and , respectively.
Because of the nonlinear nature of (15), the EKF approach requires that a first-order Taylor-Maclaurin expansion is carried out around the current state estimation by computing the Jacobian matrix:
Then, the orientations are estimated by the following EKF equations.
Compute the a priori state estimate:
Compute the a priori error covariance matrix:
Compute the Kalman gain:
Compute the a posteriori state estimate:
Compute the a posteriori error covariance matrix:
According to the above algorithm, the absolute orientations of each IMMU can be estimated. Then, the kinematics of the human arm and hand is considered.
3.3.2. Relative Orientation Filter
The kinematic frames of the arm, hand, and the forefinger are presented. There are six joints, and the coordinate frames are built including the global coordinate (), upper-arm coordinate (), forearm (), palm coordinate (), proximal coordinate (), medial coordinate (), and distal coordinate (). And the lengths between the joints are , , , , , and . The frames are shown in Figure 2. Then the relative orientations between two consecutive bodies can be determined by the following: where is the quaternion of the relative orientations, is the quaternion of the absolute orientations of the first coordinate, and is the quaternion of the absolute orientations of the second coordinate.
Meanwhile, the kinematic models of the arm, hand, and fingers are considered to determine the orientations of each segment. Human arm-hand motion can be supposed to be the articulated motion of rigid body parts. These segments are upper arm (between the shoulder and elbow joints), forearm (between the elbow and wrist joints), hand (between the wrist joint and proximal joint), proximal finger (between the proximal joint and medial joint), medial finger (between the medial joint and distal joint), and distal finger (from the distal joint). Every joint has its own local frame. Shoulder can be modeled as a ball joint with three DOFs and a fixed point representing the center of the shoulder. Movements are represented as the vector between the upper arm and body. Elbow is the rotating hinge joint with two DOFs. Wrist is modeled as a rotating hinge joint with two DOFs that is calculated between the vector representing the hand and forearm. Proximal joint is modeled as rotating hinge joint with two DOFs. Medial joint and distal joint are modeled as rotating joint with one DOF. Thus, the kinematic of this model consists of ten DOFs: three in the shoulder joint, two in the elbow joint, two in the wrist joint, two in the proximal joint, one in the medial joint, and one in the distal joint. Hence, the responding constraints are used to determine the orientations of each segment.
3.4. Positions Estimation
We assume the body keeps static and the motion of arm and hand is formed by rotation of the joints. Hence, the position of the fingertip , expressed in the hand coordinate frame (see Figure 2) can be derived using forward kinematics:where the transformation between two consecutive bodies is expressed by , , , , , and .
The total transformation is given by the product of each consecutive contribution:where is the orientation of the distal phalanx with respect to the body and is the position of the distal frame expressed in the global frame.
3.5. Summary
The gestures capture method is proposed in the section. The two procedures of the calibration are firstly implemented to improve the measurements of the sensors. The inertial sensors are calibrated by the offline calibration, and the gyros are calibrated by the online calibration. Then the orientations of the IMMUs are estimated by the QEKF. And the kinematics of arm-hand is considered and the constraints are integrated to determine the relative orientations. Finally, the kinematic parameters are used to estimate the positions. A complete diagram of the method is depicted in Figure 3.
4. Experiments and Results
The calibration is implemented to improve the sensors accuracy, and then the comparison experiments are testified to investigate the stability and accuracy of the orientations estimation. The real-time gesture motion capture experiments prove the validness of the proposed device.
4.1. Calibration Results
The calibration results of the triaxis accelerometers and triaxis magnetometers are presented in the section. The calibration samples of IMMUs are collected by rotating in various orientations. Then, the proposed method was used to determine the correctional calibration parameters as , , , and . The outputs of the accelerometers and magnetometers are shown in Figure 4. The blue sphere in Figure 4 is the raw date of sensors and the red sphere is the calibrated data of sensors. The calibration parameters are listed in Table 1.
(a) The calibration results of accelerometers
(b) The calibration results of magnetometers
4.2. Evaluation Experiments
After calibrating all the sensors of the device, the evaluation experiments are designed to assess the accuracy of the orientation estimation. One of the strings of the device is attached to the robotic arm. Three IMMUs are set the same directions, shown in Figure 5. And the robotic arm is rotated from the first state to the second state. Then the estimated orientations of IMMUs compare with the true orientations of robotic arm. The results of the orientations of three IMMUs are shown in Figure 6. And the results of the root mean square error (RMSE) of the orientations are listed in Table 2.
(a) The first state
(b) The second state
As shown in Figure 6, during 5~8 s, the robotic arm is dynamic. And, in static case, it can be known that the results have smaller variance and higher precision than the results of the dynamic case. Moreover, the yaw angle has the lager variance compared with the roll and pitch. It should be caused by the motors of the robotic arm, because the magnetometers are disturbed. When the robotic arm is in static, the device can compute the accuracy orientations. The comparison results proved the accuracy of the orientations of the IMMUs of the device. Then the real-time gesture motion capture experiments are implemented.
4.3. Gesture Capture Experiments
The IMMUs’ data is sampled, collected and computed by the MCU, and subsequently transmitted via Bluetooth to the external devices. The MCU processes the raw data, estimates the orientations of the each unit, encapsulates them into a packet, and then sends the packet to the PC by Bluetooth. The baud rate for transmitting data is 115200 bps. The frequency is 50 HZ. By using this design, the motion capture can be demonstrated by the virtual model on the PC immediately. The interface is written by C#. The wearable system is shown as Figure 7.
To further verify the effectiveness of proposed device, upper arm is swung up and down. The results of the orientations of the gestures are shown in Figure 8. And the positons of the fingertip of the right forefinger are shown in Figure 9. As shown in the figures, the wearable device can determine the realistic movements of the arm and hand. Meanwhile, the accuracy of the results is assessed by the statistics. The root mean square error (RMSE) of the orientations is listed in Table 3. The RMSE of the positons are listed in Table 4. Furthermore, the wearable device is also tested by ten healthy participants; the real-time motion gestures capture experiments are implemented to prove the validity of the proposed system.
4.4. Discussion
In this paper, a novel wearable device for gestures capturing based on magnetic and inertial measurement units is proposed. As a practically useful application, the proposed device satisfies the requirements including the accuracy, computational efficiency, and robustness. First, the calibration method is designed to improve the accuracy of the measurements of the sensors. In particular, the magnetic effects of the field are attenuated in advance. Then the orientations of gesture are divided into absolute orientations and relative orientations. The absolute orientations are determined by the QEKFs, and the relative orientations are estimated by integrating the kinematics of the arm-hand. The positions are then easily computed. The proposed method is simple and fast. The algorithm is operated in the embedded system at 50 Hz. The results of the experiments proved the advantages of the proposed device.
5. Conclusion
This paper presents the design, implementation, and experimental results of a wearable device for gestures capturing using inertial and magnetic sensor modules containing orthogonally mounted triads of accelerometers, angular rate sensors, and magnetometers. Different from commercial motion data gloves which usually use high-cost motion-sensing fibers to acquire hand motion data, we adopted the low-cost inertial and magnetic sensor to reduce cost. Meanwhile, the performance of the low-cost, low-power, and light-weight IMMU is superior to some commercial IMMU. Furthermore, the novel device is proposed based on thirty-six IMMUs, which cover the whole segments of the two arms and hands. We designed the online and offline calibration methods to improve the accuracy of the units. We deduced the 3D arm and hand motion estimation algorithms that integrated the proposed kinematic models of the arm, hand, and fingers and the attitude of gesture and positions of fingertips can be determined. For real-time performance and convenience, the interface with virtual model is designed. Performance evaluations verified that the proposed data glove can accurately capture the motion of gestures. The system is developed in the way that all electronic components can be integrated and easy to wear. This makes it more convenient and appealing for the user.
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
This work has been supported by the National Science Foundation for Young Scientists of China (Grant no. 61503212) and National Natural Science Foundation of China (Grant nos. U1613212, 61327809, 61210013, and 91420302).