Abstract
Recent advances in mobile virtual reality (VR) devices have paved the way for various VR applications in education. This paper presents a novel authoring framework for mobile VR contents and a play-based learning model for marine biology education. For interactive and immersive mobile VR contents, we develop a multilayer 360° VR representation with image-based interactions such as mesh deformation and water simulation, which enable users to realistically interact with 360° panoramic contents without consuming excessive computational resources. On the basis of this representation, we design and implement play-based learning scenarios to increase the interactivity and immersion of users. Then, we verify the effectiveness of our educational scenarios using a user study in terms of user-created VR contents, interactivity, and immersion. The results show that more experienced elements in VR contents improve the immersion of users and make them more actively involved.
1. Introduction
Following the development of consumer virtual reality (VR) devices using smartphones and tablets, the demand for mobile VR contents in various fields such as games, movies, and education has increased. Recently, user-created VR contents have been rapidly increasing with the advancement in low-cost image or video-capturing equipment for mobile VR. However, VR contents that use images or videos suffer from limitations with the lack of interactivity by only providing passive information; thus, the contents cannot draw active involvement of users. In education, we know that VR encourages learners to become more active through immersive experience [1]. Learning with real-time interactions provides learners instant results and helps in the decision making to reach their goals. To achieve a higher level of immersion and maximize educational effects, more engaging interactions between the learners and mobile VR contents are essential.
We aim to satisfy the interactive and immersive experience using the following criteria in terms of education.(i)Environment. We target mobile VR platforms. Compared with the desktop-based VR, the cost of mobile VR hardware such as smartphones and mobile headsets is affordable, making it suitable for production and distribution of educational contents for a large number of users.(ii)Engagement. We allow users to navigate in the environment and perform appropriate interactions according to the educational purpose within the VR environment, providing natural adaption to the virtual learning environment.(iii)Immersion. We provide a personalized learning environment to maximize the educational effect by increasing the immersion and participation in virtual space compared with traditional teaching methods.
This paper presents a novel authoring framework for mobile VR contents using 360° photographs or videos. To create engaging interactions in the surrounding scenes, we introduce a multilayer representation where each layer can have different interaction types or properties such as deformable objects, moving objects, and water simulation. On the basis of the proposed representation, we develop play-based learning scenarios on a mobile VR platform for education in marine biology and conduct a user study on the interactivity and immersion of the scenarios. The overall pipeline of our play-based learning is shown in Figure 1.
The main contributions of this paper are summarized as follows:(i)For interactive 360° VR contents, we introduced a multilayer representation using image-based interactions that are effective on mobile devices with limited performance.(ii)We designed a learning model to increase the interactivity and immersion of users, which includes viewer-directed immersive forms, layers for interactivity, achievements, and engaging contents.(iii)We verified our learning framework where we employed a user study in terms of user-created VR contents, interactivity, and immersion.
The remainder of this paper is structured as follows: Section 2 provides an overview of related works, and Section 3 describes our multilayer representation using different interactions and immersive experience in mobile VR. We present our active learning model in Section 4 and evaluate the developed learning scenarios through a user study in Section 5. Finally, we discuss our conclusions and future work directions in Section 6.
2. Related Work
The 360° VR contents for mobile platforms are usually created by including panoramic photographs or videos. This image-based content generation can reduce the time and cost of building a virtual space compared with the same process using three-dimensional (3D) graphics in traditional VR such as games. In addition, real-time manipulation is possible on mobile devices because it consumes a small amount of memory. The main advantage of using image-based techniques is that anyone can use their own images; thus, learner-driven personalized education is possible with the use of user-created VR contents.
2.1. Mobile VR in Education
In recent years, mobile VR education has been shown to provide more interactive learning with fully immersive contents than traditional education approaches [1, 2] A study on motivational active learning methods of several subjects that make learners quickly lose interest, such as Science, Technology, Engineering, and Mathematics [3], measured the level of immersion of learners using the Game Engagement Questionnaire (GEQ) to evaluate the ability of these learning methods. To create VR contents using synthetic 3D graphics, special skills, and extensive efforts are required. Furthermore, delivering user personalized experience and recreating virtual spaces according to the change in the learning context are difficult. The study in [4] provided a conceptual framework for user-generated VR learning across three contexts such as paramedicine, journalism, and new media production. In that study, an active learning environment using student-created 360° videos was proposed where the students could share the videos on social networks to collaborate together. Although 360° contents are more realistic than synthetic 3D VR, limitations on the interactions with the contents have been identified to exist.
2.2. Interactivity in 360° Contents
The 360° contents, which are called 360° photographs or videos, are captured by several wide-angle cameras, which are stitched and mapped in a spherical or hemispherical space to observe the surrounding scenes based on the camera [2]. Most research works for 360° content creation focus on accurate stitching for spherical mapping [5, 6], image-based lighting effects [7–9], resolution improvement, and distortion correction of images from wide-angle cameras [10]. Some researchers investigated methods to improve the interactivity in image-based VR contents by adding URLs to 360° images [11], changing scenes using button clicks [12, 13], and adding special effects to reduce the differences between composite boundaries [14]. The study in [8] recently presented an immersive system using powerful PC-tethered VR headsets that provide interactive mixed reality experiences by compositing 3D virtual objects into a live 360° video. However, there are few studies on interactive 360° contents for mobile VR headsets, such as the Samsung Gear VR and Google Daydream View, where all of the processing is done on mobile phones.
2.3. Immersive Experience in Mobile VR
Immersion comes from the feeling that a user actually exists in a virtual space reproduced by a computer [15, 16]. Unlike the existing computer-based environment, the mobile VR environment can create more immersive feelings owing to the development of VR devices such as head-mounted displays and tracked controllers [17]. The immersion in VR is not only created by elements such as visual, auditory, and haptic information [15] but also by content resolutions, stereo cues, behavioral fidelity of what is being simulated, and system latency [18]. Mobile VR is especially affected by factors such as resolutions and latency because of low-power constraints. Therefore, to increase the immersive experience in mobile VR, visual quality, sound quality, and intuitive interactions can be considered without consuming excessive computational resources [19]. To improve the immersion in mobile VR in terms of user interaction, a few methods to synchronize the virtual and physical spaces have been presented. The methods enable users to navigate the virtual world by mapping the user movement in the physical space to the virtual space [20–22]. In addition, interactive devices that use hand-gesture recognitions have been developed [23, 24]. However, only a few studies have been conducted on the optimization of visual quality or 3D sounds that consider mobile VR platforms.
2.4. Measurement of Immersive Experience
The measurement of immersion in virtual environments has been studied for a long time, mainly in the field of games [25]. GEQ [26] has been widely used as a method for measuring user engagement in games. Recently, a more accurate method to measure immersion in a virtual environment has been proposed in which 10 rating scales and 87 questions are used to show the reliability of the questionnaire [27]. We verify the effectiveness of our framework using the questions corresponding to the flow, immersion, presence, emotion, and judgment among the presented 10 scales.
3. Interactivity and Immersion of 360° VR Contents
In this section, we present methods to improve the interactivity and immersion of mobile VR contents using 360° photographs and videos for education in marine biology.
3.1. Interactive 360° VR Contents with Multiple Layers
Watching 360° contents using a VR headset replaces the user real environment with a simulated one and makes the user feel like he or she is actually there. Thus, 360° panoramic contents create a more immersive experience than regular photographs or videos. However, the viewers can usually look around by moving their head without engaging any interactions because the panoramic contents are generally mapped on a spherical surface as a single layer.
To create more engaging contents, we propose interactive 360° VR contents with multiple layers, as shown in Figure 2. The original input image can be manually segmented by its context such as sky, water, and land by using image tools, and the segmented regions are assigned to different layers. Then, images, animated images, and videos can be added to the original image as separate layers. Images and animated images are usually used for foreground objects, while videos are used for backgrounds. Different interaction properties can be assigned to these multiple layers based on image semantics or user intention. Finally, all the layers are mapped onto each individual sphere. The presented method uses a multilayer representation not only for image composition but also for different types of interactions that enable users to more realistically interact with the 360° panoramic contents.
3.2. Creating Deformable Objects
In our education scenarios for marine biology, interactive image deformation is used to create animation for some objects such as seaweeds and starfish based on transformation of the vertices of a 2D mesh on a particular layer. The resulting visual quality depends on the geometric resolution of the mesh, but mobile devices with limited computational power reduce the maximum achievable quality. Moreover, computationally expensive deformation techniques cause increased latency when the user interacts with the objects within a mobile VR environment. Therefore, we adopt a force-based approach to deform a mesh on a spherical surface, as shown in Figure 3.
The force is attenuated using the inverse proportion of the square of the distance from the center of the force, for example, user picking point . Attenuated force is defined by dividing original force by one plus the square of the distance, i.e., , to ensure that the force is at full strength when the distance is zero.
The force can be converted into acceleration via , where is the mass. To simplify, we set mass = 1 for each vertex. Therefore, is ignored as . Then, the change in velocity is defined by Equation (2), where is the time difference:
Vertex velocity is computed using Equation (3), where the direction is derived by normalizing the vector that points from the center of the force to the vertex position. The vertices are moved to new positions via .
3.3. Realistic Simulation of Water Surface
In our virtual marine contents, realistic simulation of the water surface is an essential method to increase immersion. Real-time performance of the simulation is also required to work with various user interactions in the mobile-based VR environment. To satisfy these requirements, we adopt a physics-based approach to simulate water surface waves by deforming a 2D ocean mesh and implementing it on a graphics processing unit (GPU) hardware. The dynamic behavior of the water surface, such as the ripple effects, is governed by the following 2D wave equation:
Equation (4) defines height displacement on the mesh surface depending on wave speed . refers to time, and and represent the coordinates of the 2D mesh. To numerically solve this second-order partial differential equation, we discretize it by replacing the partial derivatives with the central differences using a regular spatial grid of spacing () and a constant time step (), yielding the following equation:
New height displacement can be explicitly integrated from old displacements and .
Although this explicit method is fast and easy to implement as a GPU shader, it is only conditionally stable. If the time step becomes very large, the system can become unstable. To increase the stability of the simulation, an artificial attenuation can be added to the new displacement, i.e.,
In our experiments, we set attenuation coefficient to 0.985 to generate the ripple effects that naturally disappear. Figure 4 shows the results of water simulation in terms of mouse movement and the attenuation coefficients.
(a)
(b)
3.4. Immersive Experience in Mobile VR Environments
The immersion experience in a mobile VR environment is mainly influenced by the viewing devices and content types, as shown in Figure 5. Because the VR headsets block out all sights of the outside world, watching 360° images or videos using them makes the users more immersed than watching regular photographs or videos. The wide viewing angle of the headset, such as more than 110°, and the higher quality of the visuals, such as more than 4K resolution, as well as the faster and more stable frame rates would result in deeper immersive experience.
Furthermore, wearing headphones makes the users even more immersed by also blocking outside sound. Realistic 3D audio effects on a mobile platform will increase their perception of immersion and reduce the side effects, including headaches and motion sickness. Therefore, the users can focus on the learning objectives in a virtual environment and become active learners.
4. Active Learning Design: Marine Biology Case Study
The learning style of the new generation tends to be more self-directed, engaged, and flexible. Therefore, interactive engagement and active involvement are key elements in digital learning environments for the new generation. The simple forms of mobile VR contents are 360° photographs or videos. When we use the content for educational purpose, lack of interactivity makes the learners become passive in the virtual world. Thus, improvement in the interactivity and immersion in mobile VR contents is crucial to engage learners in the contents and keep them focused and motivated.
For active participant learning, we present play-based learning for education in marine biology. The learning scenarios are designed and implemented as mobile VR environment and allow the users to interact with the immersive mobile VR contents. A low-cost approach to user-created VR contents makes the mobile VR environment potentially learner-centered education to enable learners to create active participant VR contents. The main features of our learning scenarios include viewer-directed immersive form, adding layers for interactivity, achieving play-based learning, and engaging contents for active involvement and immersion.
4.1. Viewer-Directed Immersive Form
The viewer can decide to look at any part of the 360° scenes. The scene can be typically captured from what is around the camera (an outward view) or around an object at different angles (an inward view), as shown in Figure 6. Whereas consumer 360° cameras are mainly focused on outward views, a few apps enable users to capture 360° scenes by having the camera circle around an object. Some cameras simply capture this scene as a hemisphere (a 180° × 360° view), and others can show it as a sphere (a 360° × 360° view). Immersive 360° contents are a valuable tool to create focused learning experience. Figure 7 shows that the user can navigate a 360° marine scene using a VR headset and can control what he or she wishes to view.
(a)
(b)
(c)
4.2. Adding Layers for Interactivity
Layers are a powerful method of creating interactivity and branched stories. For personalized learning, the user can integrate marine plants and animals on additional layers together with animation effects, as shown in Figure 8, which are designed to more engage the learners to the content.
4.3. Achievements in Play-Based Learning
Achievements can be a great means of increasing the learner engagement within the learning content and encourage learners to experiment with features that are not commonly used. In addition, achievements can be a fun way for learners to engage in a light-hearted competition. Figure 9 shows that the adapted picture from [28] illustrates the three zones of marine life according to the depth of the sea, and Figure 10 shows the user interface of our virtual fishing-game scenario. A high score is given to the player when he or she catches the more difficult fish in the deep sea. Table 1 shows the different scoring depending on the marine life zones.
(a)
(b)
(c)
4.4. Engaging Contents for Active Involvement and Immersion
By adding the deformable objects discussed in Section 3.2 on a particular layer or moving objects such as lobsters as shown in Figure 11(a), we can create more interesting contents. During a fishing game, when the player casts a fishing rod, the movement of surface water can be simulated by the method presented in Section 3.3, producing more realistic effects as shown in Figure 11(b). To maximize the immersion, engagement, and learning outcome, we also created marine scenes with head-tracking 3D audio effects and video layers as shown in Figures 11(c) and 11(d).
(a)
(b)
(c)
(d)
5. Experiments
We developed the proposed learning framework on a desktop computer using Unity3D engine and then exported the educational scenarios to a mobile VR platform using Samsung Galaxy S7, Gear VR, and a dedicated controller as an interaction tool.
When we created an interactive 360° VR content, the number of vertices in the 3D spherical model for the foreground image layer was very important for realistic mesh deformation. Figure 12 shows that the results using a high-resolution 3D sphere (right) provided smoother interactions than those using a low-resolution 3D sphere (left). Because the realistic interaction methods adopted in this study relatively needed a small amount of computation for mesh processing and used only a simple 3D spherical model compared with the full 3D VR contents, our learning contents were maintained over 60-frame-per-second performance without latency even if we used a 3D spherical model with relatively high resolution.
(a)
(b)
(c)
To verify the effectiveness of the proposed interactive and immersive learning using 360° VR contents, we experimented on three scenarios in marine biology (see scenarios 1–3 in the supplementary materials (available here)), which were created based on the active learning design described in Section 4. In scenario 1, users can explore an underwater environment and interact with diverse marine life on the added deformable image layers. In scenario 2, the users can discover marine life while playing virtual fishing on a boat. In scenario 3, they can learn about the types of fish according to the sea depth while playing a waterside fishing game. When they catch a fish, the detailed information of the fish, such as scientific name and species name, is shown on the billboard, and different scores are given based on the sea depth. Each scenario consisted of three different levels of scenes: (1) simple user-created scenes with multiple layers from different image sources, (2) interactive scenes with animations such as realistic water simulation and deformable objects, and (3) immersive scenes with video layers and 3D audio effects. The participants answered questionnaires on two key VR elements, namely, interactivity and immersion, after finishing their experience in each scenario. Then, we analyzed the response results and evaluated whether the immersion of educational contents increased with the addition of more interactive elements.
For the user study, we used Samsung Galaxy S7, Gear VR, and a dedicated controller. Prior to the user study, the participants were asked about their previous experience in computer usage and VR devices. For a smooth user-study process, the participants used mobile VR devices and controllers for approximately 5 min to adapt to the interaction style. Then, experiments on interactivity and immersion were conducted by randomly choosing two of the three scenarios to keep their concentration on the experiments by limiting the maximum duration to 20 min. Each scenario required approximately 6 min to perform from Levels 1 to 3, and the average time for the experiments took a total of 17 min, including the warm-up time.
In each scenario, Level 1 showed the most basic 360° VR images, whereas Level 2 showed the scene with realistic interactive animation such as moving waves of water and object animation using mesh deformation. In Level 3, the scenes could have a video layer as a moving object or background, such as sky and water, and 3D audio effects could be added to enhance the immersive feelings. Table 2 lists the added factors to improve the user-created form, interactivity, and immersion in each learning scenario.
The participants were encouraged to experience the scenes using one or more of the techniques included in each experience category, whichever was selected among the given three scenarios. To compare the interactivity and immersion according to the level of experience, the participants provided points from 1 to 10 (10 is the best) for each experience category after finishing all level experience in each scenario. Then, two different questionnaires in Table 3 were given to measure the interactivity (nine questions) and immersion (17 questions). Twenty-six people who were 14 to 30 years old participated in this user study. We included both, subjects who were familiar with IT and had previous experience in the use of VR and those who were unfamiliar with IT and had no experience of VR, in the user study.
In our experiment, the participants chose Scenario 1 for 20 people, Scenario 2 for 13 people, and Scenario 3 for 19 people. After experiencing the scenarios, they provided score for the interactivity and immersion in each scenario. Figure 13 shows the results of the average scores and standard deviations according to the level of experience in each scenario. In the interactivity case, the number of immersive elements was shown to increase from Levels 1 to 3. The interactivity also increased in all three scenarios. In the immersion case, Scenario 3 showed the highest scores at Level 1. The reason for this result is that the 3D audio effects were added to Scenario 3 to increase the immersion, and it was not reflected in Scenario 1.
(a)
(b)
Next, we conducted a survey on the interactivity and immersion for each scenario. The questionnaire for the interactivity consisted of nine questions: four for usability, four for interactivity, and one for immersion. The questionnaire for immersion consisted of 17 questions according to the measurement of user experience in the immersive virtual environment [27]. Our questionnaire consisted of two questions for flow, five for immersion, one for presence, four for emotion, three for judgment, and two for descriptive evaluation of our framework.
We analyzed the response of each participant by calculating the scores. The answers to the positive questions are marked with “P” and the negative questions are marked with “N” in Table 3; we calculated scores according to the 5-point Likert scale (strongly agree = 5, agree = 4, neutral = 3, disagree = 2, and strongly disagree = 1). We also set the scores in the opposite order for the case of negative questions (marked with “N” in Table 3). Thus, we could obtain consistent values from negative and positive responses. The total participant response for each question was calculated as an average score. Then, the average score for each scale was calculated according to the number of questions in the scale. Figure 14 shows the evaluation scores and standard deviations of the interactivity and immersion for each scenario according to the different scale, which are marked in Table 3. The reason ‘immersion’ appears in both interactivity and immersion is that interaction methods or scene elements can affect immersion [27].
(a)
(b)
When we analyzed the overall answers, no negative values (less than 3) in all scenarios were obtained that appeared to be a negative response, but standard deviations existed among the scenarios depending on the different scale properties. In the questionnaire on interactivity, Scenario 1 showed the highest values in the interactivity and usability scales. We analyzed that Scenario 1 contained a small number of immersive experience elements; thus, it provided a simple interaction style such as touching marine lives underwater. From the viewpoint of usability and interactivity, we found that more mixed immersive experience was included, which showed the lower scores in Scenarios 2 and 3, compared to Scenario 1. In the questionnaire about immersion, each scenario was shown to have different characteristics depending on the scale properties. Scenario 2 showed higher judgment and flow, and Scenario 3 exhibited higher scores in the overall scales, except judgment. It shows that as complex immersive experience elements were added, the addition positively influenced the overall scales.
In addition, in the descriptive evaluation, the participants responded that it was good to create realistic user-oriented content and generate natural and realistic animation using mesh deformation. The participants also suggested that the rendering quality in some scenes should be improved and optimized.
6. Conclusions
We have developed an interactive and immersive 360° content-authoring framework for marine biology education. The developed framework is suitable for many students to study using low-cost mobile VR devices because the developed contents can be installed in smartphones or standalone VR equipment. For active participations, we presented user-created forms using additional layers, realistic image-based interactions utilizing elements in the given images, and more immersive contents using animated objects, videos, and 3D audio effects. We conducted a user study to verify the interactivity and immersion of our education scenarios using different levels of experience. As a result, we found that more experience elements improve the immersion of users and make them become more active.
Because the proposed method is processed based on images, creating contents using various low-cost 360° image-capturing devices is easy; thus, it can be utilized not only for educational contents but also for various image-based applications such as marketing and journalism. In the future, the image-based interactions can be improved by applying adaptive mesh deformation to enable more realistic animation depending on the object properties and by automatically segmenting some elements in the images based on the contexts. We will also investigate methods to optimize the presented framework on mobile devices to use more layers and various interactions in an effective manner.
Data Availability
The data used in our paper are 360° images. The 360° images were purchased by a third party license and are applied to “Data Owned by a Third Party” of the Sharing Option. The 360° images data used to support the findings of this study were supplied by Shutterstock under license and so cannot be made freely available. Requests for access to these data should be made to Shutterstock, https://www.shutterstock.com.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-2016-0-00312) supervised by the IITP (Institute for Information & communications Technology Promotion).
Supplementary Materials
A video clip including three play-based learning scenarios for marine biology education. (Supplementary Materials)