Abstract
In the field of technology education, virtual reality (VR) training has received significant attention in terms of its efficacy in use. Given its many advantages, there is a specific need to emphasize concrete measures for the implementation of VR training in the field of tech education. VR training based on mobile environments has been touted as a means of not only enhancing presence, flow, and learning authenticity, but also of minimizing spatial and temporal constraints. The present study has developed an evaluation tool for VR training contents, including those based on mobile environments. After categorizing VR training contents in the field of tech education into structure comprehension type, procedure learning type, and equipment experiment type contents, we constructed items for each evaluation area. The considered areas included learning, media, and content quality. By conducting Delphi surveys with a panel of experts, we confirmed that the derived evaluation items differed in number across different types of content. Under the learning area, satisfaction was found to be adequate for all content types. Items such as flow, interactivity, and learning effects were found to be adequate for procedure learning and equipment experiment type contents. The media area indicated marked variability in item adequacy depending on the content type. Usability was found to be adequate only for procedure learning type content. For equipment experiment type content, items such as presence, usability, and manipulability were all found to be adequate. All evaluation items under the content design area were found to be adequate across all content types. Thus, regardless of the type of content, it is necessary to fulfil the basic elements within the content design area in order to establish the efficacy of VR training as educational content in the field of tech education.
1. Introduction
Virtual reality (VR) training has recently been touted as an alternative means of utilizing cutting-edge technology for supplementing/strengthening industrial training. Because of its ability to produce lifelike virtual environments mimicking relevant situations in the fields of manufacturing, defense, health care, and disaster relief, which enables trainees to effectively conduct flow-inducing training/learning within a safer and less expensive setting than actual training, the VR training industry has received attention as a cutting-edge ICT(Information and Communication Technology) convergent industry. Some of the advantages of virtual reality training using cutting-edge ICT technology are cost cutting, enhanced spatial/temporal accessibility, and increase in safety owing to the use of a virtual environment. Furthermore, from the perspective of trainees, VR training has advantages such as participation in actual hands-on training with high on-site relevance and passing down of the experience and know-how of skilled workers. In view of this, VR training has been utilized in diverse industrial fields including medicine, defense, safety, manufacturing/production/construction, education/culture, and sports/leisure. However, the use of VR training in the field of technology education is still in its early stages. In consideration of the multifaceted advantages presented by VR training, there is an urgent need to explore specific methods for increasing the use of VR training in the field of tech education and to improve its effectiveness.
In addition to such technological advances, there is a need to consider methods for designing and utilizing evaluation areas for VR training contents. The evaluation areas and methods of VR training vary depending on the type of industry. For instance, whereas the key evaluation areas in health care and sports/leisure, respectively, are improvement in patient condition and improvement in entertainment or athletic functionality, the key evaluation area in tech industries is improvement in task performance. Therefore, there is a need to consider the unique properties of the tech industry in developing evaluation areas and methods for improving the efficacy of VR training.
In this study, we identify the features that distinguish training via the VR medium from training based on conventional media. By extracting factors that influence the efficacy of training and reflecting them in the development of question items for evaluating the efficacy of VR training, we aim to present an evaluation method for improving the efficacy of VR training contents.
We aim to present evaluation methods for improving the efficacy of VR training contents by developing measures for evaluating their efficacy. Specifically, the purpose of this study is to identify the features of VR training that distinguish it from training based on conventional media and to develop evaluation question items to measure the efficacy of VR training contents by extracting/reflecting upon the factors that influence the efficacy of training.
2. Theoretical Background
2.1. The Mobile Environment and VR Training
VR refers to a new digital environment constituted by digital objects and spaces within a virtual continuum, unlike the real world. The media technology system of VR space is a technological system that allows users the lifelike experience of seeing and interacting with environments that they would seldom experience in everyday life, without having to directly engage with these environments [1].
VR can provide users with a high degree of flow and lifelike presence. Therefore, introducing VR in education and training allows users to achieve higher flow with the training topic, while improving learning efficacy by enabling individualized interactions. It is very effective for providing training for areas that cannot be easily administered via text and 2D materials, for areas with insufficient training materials, or for those involving high-risk/high-cost equipment. Within this context, there has been a rise in the development and adoption of VR training contents for vocational training in the field of technology engineering via experiential/hands-on learning-based training. Due to the nature of technology training, using equipment relevant for actual on-site situations is crucial. However, because such equipment is frequently very expensive and hazardous to handle and operate manually in some cases, they can present numerous risk factors, all of which must be considered during training. VR training has the advantage of allowing trainees to learn repeatedly and safely using actual equipment within lifelike work settings, which enables the trainees to better operate equipment in real-life situations. Furthermore, due to technological changes, the consequent acceleration of the obsolescence cycles of equipment, and the constraints involved in continuously dedicating resources to the provision of new equipment, substituting these with VR training systems can be very effective. In particular, there is a growing demand for experiential/hands-on VR training contents that can replace high-risk/high-cost industrial equipment training [2–4].
The introduction of mobile environments in VR training is also being increasingly emphasized. In particular, because smart learning via mobile environments in learning environments based on 3D-augmented reality technology resolves the spatial/temporal limitations of conventional e-learning while simultaneously providing presence and flow to users, it can open up new possibilities for training services. Other types of training using smart learning include mobile web-based video learning, app-based learning, location-based learning, collaborative learning via social networks, and simulation-type learning via VR and other technologies. In particular, simulation-type learning is seen to improve learners’ sense of flow and presence through its emphasis on affinity to on-site learning. VR training via mobile environments has received significant attention due to its ability to enhance authenticity, presence, and flow in learning, as well as its advantages in circumventing spatial and temporal constraints [5, 6].
2.2. The Importance of VR Training Contents in Technology Training
In training situations, VR is a learning tool that can help strengthen immersion and increase the efficacy of education via interaction with objects. It is not only flexible enough to visualize abstract objects beyond realistic constraints, but it also enables learners to gain and share diverse and unique experiences. In particular, in the field of technology training, introducing VR can provide learners with an environment in which they can freely go beyond the constraints of what can be realistically experienced [2].
Some of the expected benefits of introducing VR in technology training are cost saving through the replacement of high-end or high-cost training equipment with VR analogues, enabling online learning via websites, realizing the movements of the internal components of equipment and enabling assembly/disassembly training (differentiated from the training effects of conventional training methods), and enabling trainees to practice equipment maintenance and experience hazardous situations within a safe virtual environment.
2.3. Types of VR Training Contents
In tandem with the growing diversity in the concepts and technological typologies of VR, VR has found applications in a very broad range of fields. In the past, due to its relatively high cost and technologically demanding nature, VR was mainly used in institutes and research labs conducting architectural simulations or military/aerospace experiments and training. More recently, however, due to the diversification of media platforms and the widespread adoption of hardware, the VR market has seen continued growth and a concurrent diversification in the fields that have adopted it. In particular, active discussion is currently being conducted regarding the educational use of simulated 3D VR spaces based on computer and Internet networks that enable interactions between multitudes of users. This draws on the strengths of VR, which enable users to experience high presence and flow.
The development and utilization of VR training contents for experiential/hands-on-based vocational training in the field of technology engineering has seen particularly active growth. Advances in industrial technology have led to the demand for more complex and newer technologies in industrial sites, thus intensifying the importance of tech training.
In response, there has been an increase in the number of studies that review and propose VR training contents suited to the field of technology training. Here, we consider some of the most prominent examples.
First, in a study that aimed to design learning modules in the field of technology training by applying competency-unit elements based on the NCS (National Competency Standards), the types of contents were categorized according to the scope of utilization and manipulation, as well as the characteristics of the VR realization, into the component type, equipment training type, experience type, and simulator type [5]. In another study, previous VR training contents that were developed as specialized coursework for technology training based on the NCS were categorized into three types—the component type, scenario type, and equipment training type [2]. While such content types are meaningful in that they categorize the goals or central functions of VR training contents according to their characteristics, their nomenclature, such as component or scenario type, are not intuitive. Another shortcoming is that their attention is limited to equipment rather than the entirety of the areas pertaining to technology training, such as tools, equipment, and facilities.
Meanwhile, other studies that focused on the types of AR (augmented reality) training contents have categorized the types into tangible AR, mobile AR, and collaborative AR, depending on the realizable characteristics from a technical standpoint. Also, in the present study, we divide AR into general forms of AR and mobile AR, further subdividing the former into the observation/manipulation type and experiment activity type, and the latter into the on-site learning guidance type and the on-site problem solution type [7].
In this study, we present a typology of the VR training contents that are currently in use in vocational training programs (E-KoreaTech, Korea University of Technology and Education) and outlined the characteristics of each type. Through a review of the previous literature [2, 4, 7, 8], we categorized the types of VR training contents for tech education based on their characteristics, meaning how content was utilized for each learning objective. The characteristics of each of the three types of contents are listed below.
2.3.1. Structure Comprehension Type
Through the rendering of precision equipment or facilities, structure comprehension type contents specialize in allowing users to comprehend the internal and external structure of large-scale, microscale, or high-risk equipment. In the case of equipment that is difficult to disassemble, 3D modelling is used to train learners in the role and function of each structure. In this type of content, the most crucial factor is the precision and fidelity with which complex structures are rendered in 3D. Although it is usually impossible to completely disassemble complex equipment, using this type of content enables learners to fully disassemble equipment, thereby facilitating their understanding of equipment structure.
2.3.2. Procedure Learning Type
In the field of tech education, some tasks need to be performed using equipment and in certain sequential steps. Procedure learning type contents enable users to learn the task procedures in sequence with regards to equipment or facilities. By providing content that enables users to learn the steps from equipment startup to shutdown in sequential order, procedure learning type content is useful for the training of equipment functions in each step of the tasks, in addition to providing a safe means of learning how to respond to malfunctions or accident situations. Due to the risks involved, it is often difficult to reenact malfunctions or accident situations in real-life settings. Because of this, trainees usually learn about how to respond to such situations through theoretical learning alone. The use of procedural learning type contents for the training of tasks that require procedural action can be effective in reenacting risky or urgent situations and instructing learners on how to respond to them.
2.3.3. Equipment Experiment Type
Equipment experiment type contents consist of training on how to operate equipment. The greatest strength of such content is that it provides an experimental environment wherein users can operate virtual equipment that may result in various outcomes. Even in the case of high-cost or complex equipment, which often present difficulties in conducting real-life operational training, users of the equipment experiment type VR training contents are able to learn how to properly operate such equipment. Thus, in this type of content, the crucial factor is the realization of the simulative functions so that the virtual equipment will react appropriately to the user’s actions in operating them.
2.4. The Evaluation Areas of VR Training Contents
Most studies on the evaluation of VR-based education and training tend to use levels of achievement and satisfaction as indices for measuring learning efficacy. However, considering the characteristics of VR training in the field of tech education, there is a need to supplement measures of knowledge acquisition with behavioral indices such as mastery (i.e., performative proficiency) and applicability to real-life situations. Additionally, satisfaction with education or training refers to a positive affective state that is experienced by the learners as a result of participating in the overall training activity. It is an important index because it is a representative measure of training efficacy while also being a key influencing factor of overall learning outcomes. As subfactors of satisfaction, satisfaction with training in addition to the intention to recommend it to others should be included.
Meanwhile, media characteristics, presence, and flow were found to have significant impacts on learning efficacy [9–13]. In consideration of the unique characteristics of VR-based education and training, we shall include media characteristics, presence, and flow as key evaluative indices. In constructing evaluation items for improving the efficacy of VR training contents in the field of tech education, there is a need to explore the existing literature regarding the contents and evaluation items with respect to the use of online education or VR in general education/training in various fields. Simultaneously, it will also be necessary to emphasize the unique characteristics of VR training in the field of tech education.
To set up the evaluation items regarding VR contents for tech training, it is necessary to consider whether the training has contributed to change or improvement in learner satisfaction or the achievement of learning objectives (Learning Area), whether the characteristics of VR were adequately realized to contribute to better training outcomes (Media Area), and whether adequate consideration was given to the training’s unique characteristics content-wise (Content Quality Area).
Thus, in this study, we devised the following three evaluation areas for VR training contents, along with subitems for each area. In order to set up the evaluation items for each area, we searched both domestic and international sources for thesis and academic articles, using VR training and technology education as key words. We extracted evaluation areas that were discussed by previous studies into evaluation item pools. First, for the Learning Area, we reviewed the list of previous studies that evaluated the learning effects of VR-based education [14–16] to establish five evaluation items—Activeness, Learning Satisfaction, Flow, Interactivity, and Learning Effects. For the Media Area, we reviewed previous studies regarding the evaluation of VR media [9, 12, 17–20]. We also broadly reviewed evaluation indices, such as the Igroup Presence Questionnaires (IPQ) [21], Flow State Scale [22], Suitability Evaluation Questionnaire (SEQ) [23], Virtual Experience Test [24], and the User Satisfaction Evaluation Questionnaire (USEQ) [25]. Based on this review, we drew up six evaluation items for the Media Area—Sensory Immersion, Usability, (Media) Satisfaction, Manipulability, Navigability, and Presence. For the final area, the Contents Area, we established three evaluation items—Instructional Design, Teaching and Learning Strategies, and Learning Contents—based on the “Guideline for E-Learning Quality Management” [26].
The concepts regarding these subitems are summarized in Table 1 below.
3. Results and Discussion
3.1. Construction of Evaluation Areas and Items
The evaluation areas of the VR training contents were derived based on the analysis of existing literature.
First, the learning area was constructed to include items for the evaluation of the “training” aspect of the contents. Second, the media area included items relevant to the characteristics of VR as a medium, in order to verify the aspect of “virtuality.” Finally, the content quality area included quality management items for e-learning contents [26] that were deemed relevant for this study’s purposes, in order to assess the “content” characteristics.
The composition of the evaluation areas and items that were compiled by the authors of this study via literature analysis is presented in Table 2. Specifically, the learning area consists of 5 items, namely, activeness, satisfaction, flow, interactivity, and learning effects. The media area consists of 6 items, namely, sensory immersion, usability, (media) satisfaction, manipulability, navigability, and presence. The content quality area consists of 3 items, namely, instructional design, teaching/learning strategies, and learning contents.
3.2. Evaluation of the Evaluation Items via the Delphi Method
In order to assess the validity of the evaluation items for the VR training contents, analysis was conducted with a 20-person panel of experts using the Delphi method. In accordance with the content validity ratio (CVR) [34], defined here as the ratio of positive responses—4 (adequate) and 5 (very adequate)—on a 5-point scale, values of 0.42 or higher were considered as adequate.
3.2.1. Composition of Expert Panel and Survey Design
The composition of the expert panel reflected the characteristics of the VR training contents. In order to broadly consider matters such as the training goals, the unique properties of virtual training, and the characteristics of the content, the panel consisted of twenty experts from various fields, including five experts in technology engineering, four practitioners in the field of technology, two instructors of content operation, five experts in educational technology, and four experts in educational contents.
The process undertaken in the Delphi study was as follows: The panel of experts, who were selected based on their previous consent, were asked to assess the adequacy of the evaluation areas and items for the VR training contents. The panel also provided suggestions regarding possible modifications to be made to the areas and items. The experts on the panel were presented with the types of VR training contents derived in this study (along with their characteristics and examples), in addition to explanations of the concepts of the evaluation items for each area so as to minimize discrepancies in perception between panel members. The Delphi study was conducted over two rounds, during which experts were asked to suggest modifications for the evaluation areas and items based on their assessment of the adequacy of the question items with which they were presented.
3.2.2. Results of the 1st Round Delphi Study
Results of the 1st round of the Delphi study conducted for assessing the adequacy of the evaluation items indicated different levels of adequacy across content types. The detailed analysis results are presented in Table 3. First, in the case of structure comprehension type contents, the only items that were found to be adequate were satisfaction (CVR = 0.47) under the learning area and usability (CVR = 0.58) under the media area. On the other hand, under the content quality area, all items including instructional design, teaching/learning strategies, and learning contents were found to be adequate for content evaluation. Procedure learning type contents were found to have more adequate evaluation items than the learning area. Thus, all items under the learning area excluding activeness (CVR = 0.26) and sensory immersion (CVR = 0.26), navigability (CVR = −0.05), and presence (CVR = 0.16) under the media items were found to be adequate for evaluation. Finally, in the case of equipment experiment type contents, all evaluation areas and items were found to be adequate for evaluation.
While collecting the views of experts during the 1st round Delphi study for assessing the adequacy of the evaluation items, many experts expressed the need to modify the items of the media area. To prevent overlap between items and improve conciseness, items that shared similarities under the media area, such as sensory immersion and presence, usability and satisfaction, and navigability and manipulability needed to be merged or modified. Furthermore, given the aim of this study, namely, the evaluation of VR training contents, the wording of items under the content quality area were modified from “quality” to “design” and from “instructional” to “content.” The list of evaluation areas and items after the modifications suggested by the experts that were incorporated is presented in Table 4, and the newly constructed evaluation areas and concepts for each item are summarized in Table 5.
3.2.3. Results of the 2nd Round Delphi Study
Because some of the evaluation areas and items were modified based on the results of the 1st round Delphi study, in the 2nd round Delphi study, the evaluation items were not discarded based on the content validity values obtained from the 1st round study. However, the content validity values from the 1st round study were presented during the 2nd round Delphi study to inform the responses. The results of the 2nd round Delphi study are presented in Table 6.
Results of the 2nd round Delphi study indicate similar results as those of the 1st round study in terms of structure comprehension type contents. The satisfaction item (CVR = 0.47) under the learning area and the three items under content design, namely, content design (CVR = 0.89), teaching/learning strategies (CVR = 0.68), and learning contents (CVR = 0.68), were found to be adequate for content evaluation. In the case of the media area, all the items that had been found to be adequate in the 1st round Delphi study, including usability, were found to be inadequate. This may be attributable to the fact that because structure comprehension type contents simply aim to convey the structure, role, or function of equipment/facilities, media characteristics such as presence, usability, or manipulability were seen to be relatively less important.
Procedure learning type contents were also found to have similar results to those of the 1st round study. The four items under the learning area, namely, satisfaction, flow, interactivity, and learning effects, except for activeness (CVR = 0) were found to be adequate, while only the usability item (CVR = 0.68) was found to be adequate under the media area. Furthermore, all three items under the content design area were found to be adequate. Thus, given that procedure learning type contents aim to foster proficiency in the operation of equipment/facilities, items such as flow, interactivity, learning effects, and usability were found to be more emphasized than in structure comprehension type contents.
Finally, in the case of equipment experiment type contents, all evaluation areas and items were found to be adequate. This result is consistent with that of the 1st round Delphi study. Given that equipment experiment type contents aim for mastery rather than comprehension or learning, this result may be interpreted as indicative of the relative importance of the diverse characteristics of VR contents. Thus, based on the results of the Delphi studies conducted over two rounds, we have confirmed differences in the perceived adequacy of evaluation areas and items depending on the type of VR training contents.
4. Construction of Finalized Evaluation Areas and Items by Content Type
Based on the obtained results, we constructed a finalized version of the evaluation areas and items in compliance with the literature analysis and the two rounds of Delphi studies. These are summarized in Table 7. Studying these more closely reveals a marked change in the number of evaluation items that were found to be adequate. First, while only four evaluation items were found to be adequate in the case of structure comprehension type contents, all eleven of the evaluation items for the equipment experiment type contents were found to be adequate.
Reflecting in detail upon the results based on the evaluation items, the satisfaction item under the learning area was found to be adequate for evaluating all types of VR training contents. This indicates that in the case of the learning area, satisfaction is the most basic and representative evaluation item for measuring learners’ response to VR training contents. Furthermore, items such as flow, interactivity, and learning effects were found to be adequate for the evaluation of procedure learning type and equipment experiment type contents. On the contrary, the activeness item was found to be adequate only for evaluating equipment experiment type contents, where responses to the operation of contents were emphasized.
The media area demonstrated the most marked variation in adequacy depending on the type of contents. Usability, which was found to be adequate for the evaluation of the most types of contents, included procedure learning type contents and equipment experiment type contents. This highlights the need for media characteristics of VR training contents to crucially consider the ease of use on the part of the learner who will be actually using the contents. On the other hand, items such as presence and manipulability were found to be adequate only for the evaluation of equipment experiment type contents. This suggests that in cases where the aim of VR training extends beyond simple comprehension or experience and includes mastery, there is a need to sufficiently consider factors such as the content’s presence or manipulability.
Finally, the three evaluation items under the content design area were found to be adequate for all types of content. This implies that there is a need to consider the basic characteristics of the contents in developing/implementing them, regardless of the type of content.
5. Conclusions
This study was conducted with the aim of developing items for evaluating VR training contents and exploring means for improving their efficacy, by extracting the factors that influence the educational efficacy of mobile-based VR training contents. The adequacy of the evaluation areas and items that were extracted through the analysis of previous literature was assessed through Delphi surveys conducted with experts in each field, and the results thereof were used to derive the evaluation items.
In view of this, some points need to be taken into consideration when utilizing this evaluation tool. First, it should be considered that this tool was constructed such that the items would be selected depending on the type of content to be evaluated. It will be adequately utilized as a tool for assessing the contents’ characteristics as a learning tool and for providing feedback. Thus, when utilizing this evaluation tool, it would be appropriate to set it up by selecting evaluation items that suit the type of content.
Second, as a feedback tool for the development and implementation of contents, the evaluation tool presented here could aid in guiding improvements to the contents depending on the evaluation results across each area. By examining the degree of content acceptance across the areas of learning, media, and content design, it would be possible to suggest improvements for each evaluation area.
Third, the results obtained from this evaluation tool may provide guidelines for the design and development of content. For instance, in the case of structure comprehension type content, motivational factors relevant to the experience of learning effects should be prioritized within the learning area, while in the case of equipment experiment type content, consideration must be given to all evaluation items. This is because the virtual experiments may include factors such as structure comprehension as well as procedure learning.
In this study, we assessed the adequacy of evaluation areas and items depending on the type of content. Further studies should extend beyond the assessment of adequacy by also considering the weights assigned to the evaluation areas and items for each type of content. This would allow for the establishment of more concrete standards and evaluation factors for content development.
Data Availability
The statistical data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by Virtual Training Project Grant of the E-KoreaTech in Korea University of Technology and Education (VT-2017-0003). The authors are grateful to a panel of experts who participated for investing time and energy in this project.