Abstract
Traditional social skills training (SST) programs are delivered in person and suffer from significant time, financial, and opportunity barriers that limit their reach and potential benefits for youth. This paper describes the design and preliminary evaluation of Hall of Heroes, a digital game that presents SST through an engaging superhero-themed virtual story world. Participants were randomly assigned to complete the digital game (n = 15) or to a waitlist control condition (n = 14) and were compared on parent-report measures of social emotional functioning. Youth who completed Hall of Heroes significantly improved in their abilities to relate to others (both peers and family members) as well as to accept affection and express emotions with others, compared to youth who did not complete the SST intervention. Further, youth in the treatment condition showed a significantly greater decline in feelings of anxiety, depression, and hopelessness than did youth in the control condition. Both parents and youth reported high levels of engagement in and acceptability of the Hall of Heroes. This study adds to the research literature, supporting the potential of a game-based SST platform for effectively helping youth develop prosocial social problem-solving skills.
1. Introduction
1.1. Social Relationships in Early Adolescence
The early adolescent period is a turbulent and stressful one for youth who must adapt and adjust to a myriad of concurrent life changes. At the same time as significant physical and cognitive growth is occurring, youth move from the smaller, more protective elementary school setting, with smaller classrooms and strong teacher support, to the larger, more impersonal middle school environment. In late elementary and middle school, academic demands increase considerably with more difficult assignments, higher expectations for independence, and more stringent grading [1, 2]. Concomitant with escalating academic demands, the social environment shifts from smaller, more protective settings to much larger, varied, and competitive social groups [3–6].
The turbulence of these academic, social, and developmental shifts, combined with a decline in adult supervision, leads to a period of heightened vulnerability. For many youth, the early adolescent period is associated with loss of self-esteem, increased anxiety and depression, and drops in academic performance [3, 7]. And, adolescents with lower social skills (e.g., poor emotion regulation, difficulty with communication) and poorer peer relations are at greater risk of academic failure, behavioral problems, and emotional difficulties [8–11]. In contrast, positive social support and prosocial behaviors serve as protective factors, helping youth adjust more quickly to changing demands and fostering greater school engagement and academic success [2, 3, 8].
1.2. Social Skills Training
Social skills training (SST) interventions have repeatedly been shown to significantly improve social skills, behavior, and interpersonal relationships [12–15]. Effective SST can serve a protective function, as youth learn and practice those social skills that increase prosocial and/or inhibit maladaptive behaviors [16, 17]. For example, cooperation fosters companionship—a key function of peer relations—while impulsive and domineering behaviors undermine the development and maintenance of positive social relationships. As Peterson et al. [7] explain, “Improvements in social skills and problem-solving abilities allow the preadolescent to build a supportive network and engage in appropriate social activities that may ultimately contribute to reduced school drop-out rates as well as other indices of improved adaptive functioning” (pp. 289). Effective SST programs that improve social problem solving and social skills are essential for preparing youth for the transition to middle school [8, 10].
Unfortunately, SST is traditionally delivered in person by trained clinicians or school staff which presents significant logistical barriers, including a lack of trained providers, scheduling difficulties, and significant time commitments, which too often prevent youth from participating in and benefiting from SST [14, 15]. Poor implementation fidelity by providers (e.g., nonadherence to the in-person treatment protocol) has also been shown to undermine outcomes for youth [7, 18]. And, lastly, SST is typically delivered in a group format with one provider to many youths. Research suggests tailoring interventions to the individual may be necessary to achieve intended gains in social and academic outcomes [19, 20].
1.3. Games for Impact
To help overcome the limitations of in-person SST, a growing number of intervention developers are employing emerging technologies [21]. Game-based platforms offer a cost-effective means to deliver program content in an accessible and engaging way, eliminate travel and provider staffing and training requirements, and standardize program delivery on a broad scale [19] (Merry et al., 2012). When digital games are dynamic and individualized, the user benefits from increased practice opportunities as well as a more tailored and personalized learning experience [19, 22–24]. Further, the engaging and interactive nature of games, and the relative anonymity of digital learning environments, means youth are often motivated to more fully and openly participate in game-based SST compared to traditional in-person SST [25, 26].
There is growing support for the effectiveness of digital games for social, emotional, and behavioral development [21, 27–30]. Digital platforms have been shown to increase comprehension and recall of presented material over conventional instructional methods and produce greater generalization of acquired skills to real-life behavior (Clark et al., 2016) [31, 32]. Further, there is increasing evidence that digital environments are especially effective at targeting higher-order thinking skills of analysis, synthesis, and evaluation, which form the basis of social problem solving [28, 33].
1.4. The Development of Hall of Heroes
Drawing on the foundational principals of SST and digital game-based learning, 3C Institute developed Hall of Heroes, an online single-player point-and-click adventure game wherein the youth navigate a virtual school-like world, engaging with other characters to solve tailored social problem-solving tasks. As illustrated in Figures 1 and 2, the full Hall of Heroes software includes a personalized avatar builder, a gameplay training scene, six baseline assessment scenes, seven tutorial scenes, a final skill mastery application scene, six cinematic cut scenes, and seven knowledge monitoring quizzes. Each of the first six scenes serves as an introduction to and baseline measure of one of Hall of Heroes’ six core social skills: impulse control, communication, cooperation, social initiation, empathy, and emotion regulation. Each of the seven tutorial scenes provides dynamic instructional content focusing on two or three particular social problem-solving strategies for these core skills (see Table 1 for scene overviews and learning objectives). The skill concepts covered in these scenes are reinforced in the knowledge assessments where players are asked a series of application questions relevant to the skill nuances explored in the preceding scenes and provided with dynamic feedback based on their answers.
(a) Personalized avatar builder
(b) Scene selector
(c) Knowledge monitoring assessment
(d) Cinematic cut scenes
Hall of Heroes was developed by a multidisciplinary team comprising clinical and developmental psychologists, SST intervention developers, computer programmers, artists and animators, and experts in computer games for learning. The development process followed best practices in educational game design, with thoughtful consideration for the unique social and developmental challenges of early adolescence [23, 24, 28, 32, 34]. These considerations led us to employ multiple gameplay styles, varying the settings and backgrounds for each scene, and to incorporate a compelling storyline through both playable scenes and cinematic cut scenes, resulting in a motivating and engaging game environment where all types of players can find something to attract them [26, 35–37]. Figure 3 presents a sample of the range of gameplay experiences in Hall of Heroes, which include simulations, drill and practice mini-games, opportunities for open exploration, dialog driven content, and action-based challenges. By incorporating direct instruction, varied practice exercises, dynamic feedback, and self-directed elements, the game is able to effectively target multiple modes of learning [23, 26, 34, 37].
(a) Simulation: players act as the director during a video shoot
(b) Dialog menu: players discuss where to sit during lunch
(c) Skills application: players demonstrate receptive communication in a notes-taking exercise
(d) Mini-game: players practice deep breathing techniques
(e) Practice exercise: players practice using a combination lock to open their locker
(f) Practice exercise: players choose where to stand when joining in
Situating the game in a school-like setting allows for the inclusion of tasks that directly address the routine aspects of middle school, such as changing classes and using a locker, which repeatedly come up as areas of pretransition concern and anxiety for youth [7, 38]. In addition, the school setting allowed us to simulate challenging social situations unique to a school environment, such as choosing a table for lunch and understanding when and how to engage a teacher for help in a bullying situation. Incorporating these common transition experiences into a superhero world serves to balance fantasy and realism, maintaining engagement and allowing for deeper learning and skills transfer [32].
To further develop this rich narrative world, Hall of Heroes is populated with a large and diverse cast of nonplayable characters (NPCs) which serve narrative and pedagogical roles that mirror traditional instructional roles: peer, teacher, mentor, learning companion, etc. Experts in the field of educational game design agree that the inclusion of interactive embodied social pedagogical agents enhances student motivation and engagement and improves learning outcomes including retention and skills transfer (e.g., [39–41]). NPCs as embodied pedagogical agents support the social cognitive aspects of learning, serving as an interactive bridge to help players understand complex concepts [39, 42]. Rather than simply presenting information, the NPCs in Hall of Heroes engage and motivate players through direct experiences with the game world to actively participate in the construction of social problem-solving knowledge, for example, by guiding players to look for clues in school security footage in order to check their assumptions about a situation. NPCs also serve to model and guide players through difficult cognitive processes, for example, by facilitating positive self-talk through conversation and branching dialog menus.
In the current study, we conducted an initial investigation of the efficacy and acceptability of the Hall of Heroes game for SST using a randomized-control design. We expected youth who interacted with Hall of Heroes would demonstrate significantly greater improvements in social, emotional, and behavioral outcomes compared to youth in the waitlist control condition who did not complete the game.
2. Methods
2.1. Participants
Participants were recruited via postings on national parenting, game research, and school listservs and social media sites. Interested parents completed an online survey to determine eligibility. To be eligible to participate, youth must have (1) been attending the 5th or 6th grades; (2) received instruction in a regular education classroom for at least 40% of the school day; (3) been English language proficient; and (4) had access to a computer with Internet use at home.
Eligible participants (N = 32) were randomly assigned to either the treatment (TX) or waitlist control (WLC). Nine percent of participants were attrited over the course of the study (i.e., failed to complete post-data collection tasks), resulting in a final sample of 29 youths. Chi-squared tests revealed no significant differences between participants who completed the study and those who did not based on age, race, gender, or study condition. The final sample of youth ranged in age from nine to thirteen years (M = 11.35, SD = 0.88), with 76% male and 31% representing a racial or ethnic minority group. There were no significant demographic differences across the two conditions.
2.2. Measures
Positive Behavioral and Emotional Skills. Youth behavioral and emotional strengths were measured using The Behavioral and Emotional Rating Scale, Second Edition (BERS-2 [43]), a 58-item strength-based assessment that measures youth positive emotions, behaviors, and interpersonal skills. Parents reported how many statements, such as “acknowledges painful feelings,” “considers consequences of own behavior,” and “completes homework regularly,” were like their child over the past 30 days on a 4-point scale ranging from 0 (not at all) to 3 (very much). The BERS-2 composite Strength Index was calculated by summing across all items and converting this raw score to a standard score (M=100, SD=15). Further, the composite score was broken down into its five subscales: interpersonal strength (strengths in how the child relates to others), affective strength (strengths in how the accepts affection and expresses emotions), family involvement (strengths in the child’s relationships with family members), intrapersonal strength (child’s internal emotional strengths, outlook on one’s own competencies and accomplishments), and school functioning (strengths in completing school-related tasks). Raw scores for each subscale were computed by summing scores for all relevant items and converting them to standard scores (M=10, SD=3). Higher scores reflect greater behavioral and emotional strength in that area. The BERS-2 Strength Index demonstrated good reliability (Cronbach’s α = .97) and prior factor analyses have established the measure’s content, criterion, and construct validity [43].
Psychosocial Distress. Youth psychosocial distress was measured using the 64-item parent-report Youth Outcomes Questionnaire, Second Edition (Y-OQ [44]). Parents were asked to think about their child over the past 7 days and report how true statements were for their child on a 5-point scale from 0 (never or almost never) to 4 (almost always or always). The Y-OQ Total Distress score was computed by summing across all items and respective items were combined for six subscales: intrapersonal distress (18 items covering anxiety, depression, and hopelessness), somatic (8 items including headaches, dizziness, and stomachaches), interpersonal relations (10 items covering arguing, defiance, and communication problems), social problems (8 delinquent or aggressive behavior items), behavioral dysfunction (11 items covering organization, concentration, handling frustration, and ADHD-related symptoms), and critical items (9 items consisting of symptoms often found in youth receiving inpatient services, such as paranoid ideation, hallucinations, mania, and suicidal feelings). Higher scores on each scale indicate higher levels of psychosocial distress. The Y-OQ showed high reliability (Cronbach’s α = .97) and its publishers detail several studies establishing content, criterion, and construct validity with both clinical and nonclinical (community) samples [44].
Game Evaluation. Following completion, youth and parents were asked to provide feedback on the Hall of Heroes program via evaluation surveys. Youth were asked to rate how much they agreed with statements regarding their engagement and liking of Hall of Heroes. Parents rated the extent to which they agreed with statements such as “I would recommend Hall of Heroes to other parents of or grade youth” and “Hall of Heroes was easy for my child to understand.” Youth and parent evaluations were made using a 5-point scale with 1 = strongly disagree and 5 = strongly agree. Open-ended responses were also collected, to collect more detailed feedback on youth and parent impressions of the SEL intervention.
2.3. Procedures
Precautions were taken to ensure study ethics and protection of human subjects. The study protocol was approved by 3C Institute’s institutional review board. Parent consent and youth assent were obtained from all participants prior to participation. All parts of the study were completed online through a secure project website.
All parents in the TX and WLC groups completed the BERS-2 and Y-OQ at pretest. The week following pretest data collection, youth in the TX group began interacting with Hall of Heroes. The game was released in four segments, with one segment released per week for four weeks; youth had one week to complete each segment. Within one week following the TX group’s four-week interaction with the program, parents in both the TX and WLC groups completed the same set of outcome measures again, as posttest measures, and youth and parents in the TX group completed product evaluation surveys.
3. Results and Discussion
Preliminary analyses were conducted using a Multivariate Analysis of Variance (MANOVA) with condition as the between-subjects factor to determine whether scores on the BERS-2 and the Y-OQ differed at baseline (preintervention) across the two conditions. These analyses revealed no significant difference between groups at baseline on any subscale of the BERS-2 or the Y-OQ. However, given the small sample size, we elected to use an Analysis of Covariance (ANCOVA) approach in order to control for baseline scores when analyzing intervention effects [45, 46].
3.1. Positive Behavioral and Emotional Skills
As can be seen in Table 2, results of the ANCOVA revealed significant improvements in positive behavioral and emotional skills for youth in the treatment condition, as measured by the BERS-2. As expected, parents reported that youth who completed Hall of Heroes showed significantly greater growth in their total Strength Index (a measure of youth overall behavioral and emotional strengths) compared to WLC youth who did not complete the intervention. Further, when this index was broken down into its composite subscales, youth in the treatment condition showed particular gains in three areas: interpersonal strength, affective strength, and family involvement. Thus, youth who completed Hall of Heroes significantly improved in their abilities to relate to others (both peers and family members) as well as to accept affection and express emotions with others, whereas youth who did not complete Hall of Heroes showed essentially no change in these areas. These results were expected given that Hall of Heroes directly targets social skills that are critical for positive interpersonal relationships, including communication, cooperation, and social initiation as well as emotional awareness and emotion regulation skills which would directly impact the youth’s affective abilities.
No significant difference by treatment condition was found for the BERS-2 intrapersonal strength subscale. It may be that the duration of the study was too brief (4 weeks) to impact youth’s internal emotional strengths and opinions of their own competencies and accomplishments, which may take more time and real-world experience to alter than possible with this study design. No significant difference was found for the BERS-2 school functioning subscale either. However, Hall of Heroes does not directly address academics or school-related tasks, so lack of a treatment effect for this subscale may be expected.
While intrapersonal and school academics were not directly targeted by the Hall of Heroes intervention, improvements for these subscales may emerge with greater exposure to the intervention and/or more time for these more distal outcomes to be impacted by social skill improvements for the youth. As the youth practice and implement the social skills taught in Hall of Heroes, effects on self-efficacy and academic competence may emerge. Future research with a longer intervention period could directly test this hypothesis.
3.2. Psychosocial Distress
As can be seen in Table 3, ANCOVA results demonstrated TX youth experienced significantly lower Overall Psychosocial Distress on the Y-OQ at postintervention compared to WLC youth, after controlling for preintervention scores. This result suggests that Hall of Heroes’ bolstering of social emotional skills may be key to helping youth cope with stressful situations. Results further indicated that this overall significant effect was largely due to significant improvement in intrapersonal distress for youth in the TX condition. Thus, youth who completed Hall of Heroes showed a significantly greater decline in feelings of anxiety, depression, and hopelessness than did youth who did not participate in the intervention.
While not statistically significant, two other Y-OQ subscales showed trends suggesting youth who completed Hall of Heroes appeared to experience fewer symptoms of somatic distress (e.g., headache, stomach, and dizziness) and fewer critical psychological symptoms (e.g., thoughts of suicide, paranoia, and mania) compared to youth who did not complete the intervention. We know, from the literature, youth who experience social problems often develop somatic complaints, either as a function of the anxiety associated with interpersonal problems or as a coping mechanism to avoid socially stressful situations, such as school [9]. The trend towards fewer somatic symptoms may be an indirect effect from the overall lessening of psychosocial distress in the area of anxiety for youth in the treatment condition. Similarly, lower psychosocial distress in the areas of depression and hopelessness would be expected to decrease the level of serious psychological (critical) symptoms experienced by a youth.
No statistically significant between-group differences were found on the Y-OQ for the interpersonal relationships or social problems subscales. This finding is somewhat surprising given the significant findings on similar interpersonal strength subscale on the BERS-2, reported above. However, for the Y-OQ, interpersonal relations focuses on arguing, defiance, and negative communication problems and social problems focuses on delinquent and aggressive behavior problems. While greater (but nonsignificant) declines in these problem areas were seen for youth in the TX condition compared to those in the WLC condition, these areas represent antisocial behavior problems that are not the primary focus of Hall of Heroes. Hall of Heroes focuses on teaching prosocial skills and behaviors and positive social problem solving. We know that antisocial behavior problems are more resistant to change [13] and that positive impact on antisocial behavior problems may take longer than the 4-week intervention period of this study. It may be that declines in these antisocial behavior problems would emerge with greater exposure to Hall of Heroes as well as greater real-world experience using prosocial skills that are antithetical to antisocial behaviors.
With regard to the behavioral dysfunction of the Y-OQ, this subscale focuses on ADHD-related symptoms, such as disorganization, poor concentration, and trouble handling frustration, which are not directly addressed by the Hall of Heroes intervention. Thus, it is not too surprising that there was no significant group difference for this particular subscale. However, while Hall of Heroes is not designed as a treatment tool for any specific disorder (e.g., Attention Deficit Hyperactivity Disorder, Conduct Disorder), it is possible that improvements in these types of behavioral problem areas may arise over time as youth practice new social emotional skills in the real-world. Future longitudinal research would be needed to test whether group differences in antisocial and ADHD-related behavior problems emerge over time for youth.
3.3. Hall of Heroes Evaluation
Because Hall of Heroes was designed to be used independently by youth and because greater engagement in intervention is associated with greater skill gains (Clark et al., 2016) [47], it was important to examine how youth evaluated the game. In general, as can be seen in Table 4, both youth and parents rated Hall of Heroes favorably across all areas. Parents supported Hall of Heroes as engaging, and youth reported they found the game fun and enjoyed both the characters and story. Also of note were youths’ high marks for interest in playing more games like Hall of Heroes and parents’ belief that Hall of Heroes is an effective tool for teaching social skills with youth.
Follow-up requests for open-ended feedback helped to put these ratings into context. When the youth were asked what they learned in Hall of Heroes, nearly all noted at least one of the program goals, including “cooperation with others,” “respect for others,” “how to deal with feelings,” and “communication and working together,” to name a few. Representative comments from parents were like “my child learned about working together for a common goal,” “my child had fun enough to keep her wanting to play, yet helped her with some new ideas about how to interact with others,” and “there were more than few scenarios that my child encountered that forced him to reevaluate and change directions,” among other comments. Further, parents reported they were likely to recommend Hall of Heroes to other parents of adolescents.
4. Conclusion
The results of this study provide support for the usability and acceptability of Hall of Heroes as well as initial evidence of its potential as a game-based social skills training (SST) intervention for adolescents. High ratings from both youth and parents, and the positive open-ended feedback, support the idea that the digital game Hall of Heroes is an acceptable and engaging way to implement SST. Also encouraging was the fact that we found all participants were able to complete the intervention in the allotted time, and there were no reports of technical or user errors with the software. These findings indicate youth are able to complete Hall of Heroes independently and are sufficiently engaged in the program to maintain interest and motivation throughout SST.
Findings also provided initial support for the efficacy of participation in Hall of Heroes for increased interpersonal and emotional strength and decreased psychosocial distress, key ingredients to preparing youth for the transition to middle school. Despite the small sample size and brief intervention period, youth who completed the game-based SST showed improvements in multiple areas. In fact, those areas found to change significantly as a function of intervention (e.g., lower intrapersonal distress, greater interpersonal and affective strengths) showed moderate to high effect sizes, a treatment benefit commiserate with more traditional in-person SST programs (see [13]).
This work supports the growing body of evidence showing that digital games are potentially effective for delivering a wide array of psychotherapeutic interventions, including SST. As evidence mounts confirming the importance of social skills as a protective force in early adolescence, particularly in helping youth cope with the stressful transition to middle school, the need for effective intervention methods that can overcome the barriers to traditional SST programs becomes increasingly urgent. Hall of Heroes and programs like it which deliver dynamic, personalized, and engaging social skills instruction offer a low-cost, accessible, and easy to scale alternative to reach youth with effective SST on a broader scale than ever before.
4.1. Limitations and Future Directions
This initial study of the Hall of Heroes game-based SST intervention adds to the growing literature indicating digital games can effectively address a wide array of social, emotional, and behavioral concerns (see review in [21]). This pilot test represents the first step in evaluating the efficacy of Hall of Heroes specifically and as such a number of limitations should be considered. First, our sample size was quite small which limits statistical power and our ability to investigate potentially informative subgroup differences (e.g., grade or gender differences) and also can potentially limit generalizability of findings. It is important to note that while the sample was recruited from the broader community with no requirement that youth have a preexisting social skills problem, it appears that parents who signed up were at least concerned about their child’s social skills or social functioning. Examination of baseline levels of each outcome measure showed elevated distress in all areas of the Y-OQ and below average strengths on the BERS-2 for both TX and WLC youth (no group differences were present at baseline). Given what appears to be a self-selection bias by parents for participation in this intervention study, results should be considered generalizable to adolescents in need of social skills intervention, rather than youth more generally. Future studies employing a larger sample size are needed to confirm treatment effects and investigate potential differential treatment response by subgroup, including youth with varying degrees of baseline social difficulties.
Second, the current study relied on parent report for outcome measures. Given our design, it was not possible to include independent measures where the reporter was not aware of the intervention condition of youth. Because parents were not blind to treatment condition, biases may have influenced their reported behavioral changes. However, it is important to note that the pattern of results was not equivalent across all subscales. If biases were solely responsible for parental ratings, we would expect all subscales to be equally impacted. However, those subscales that were most closely related to the content taught through Hall of Heroes (prosocial skills, positive social problem solving) were those that were found to be significantly improved by participation in the intervention. Outcome areas which are not directly targeted by the intervention (e.g., school-related performance, antisocial behavior problems, and attentional problems) or which may require a longer time period than 4 weeks to see change were not found to be significant in this study. Future investigations with multiple informants and informants who are blind to treatment condition, such as teachers, as well as a longer longitudinal design, would be important for clarifying the observed changes by parents in the current study.
Third, the brief intervention period meant data collection and intervention delivery occurred during the spring semester for all participants. Further research is needed to examine whether time of year impacts treatment effects. Studies which compare timing relative to the transition to middle school (e.g., late grade, early/late grade) and include measurement beyond immediate postintervention would be useful for determining the optimal conditions for delivering the intervention.
Data Availability
Individuals interested in the data generated through this study or gaining access to the Hall of Heroes game should contact the first author.
Conflicts of Interest
The authors acknowledge possible conflicts of interest related to this study. While the authors do not directly benefit financially from the results of this study, both authors are affiliated with the company that sells Hall of Heroes (https://www.Centervention.com) and as such may indirectly benefit in the future from the results of the study. They have administered this study in accordance with ethical research guidelines and do not believe this potential conflict has impacted the reported results.
Acknowledgments
This research was supported by the US Department of Education, Institute of Education Sciences, through contract ED-IES-13-C-0041. The authors would like to thank the parents and youth who participated in this research as well as the Hall of Heroes development team for their contributions to this program. In addition, the authors would like to thank Dr. Ashley Craig and Emily Brown for their contributions to earlier drafts of this paper as well as their contributions to the development and testing of Hall of Heroes.