Abstract
Undoubtedly the biggest success amongst the recent games console releases has been the launch of the Nintendo Wii. This is arguably due to its most innovative attribute—the wireless controller or “Wiimote.” The Wiimote can be used as a versatile game controller, able to detect motion and rotation in three dimensions which allows for very innovative game play. Prior to the Wii, and with much less furor, Nokia launched its 5500 model phone which contains 3D motion sensors. Using the Sensor API library available for the Symbian OS, this sensor data can be used by developers to create interesting new control schemes for mobile games. Whilst 3D motion can be utilized for ondevice games, in this paper we present a novel system that connects these phones to large public game screens via Bluetooth where it becomes a game controller for a multiplayer game. We illustrate the potential of this system through a multiplayer driving game using the Microsoft XNA framework and present preliminary feedback on the user experience from a public trial which highlights that these controls can be both intuitive and fun.
1. Introduction
Traditionally, the console game industry has divided potential players into two main categories: hardcore and casual gamers [1]. Of these categories it is the hardcore gamer that the industry has targeted itself towards as they exhibit features it wishes to exploit, such as [1] their tendency to purchase and play many games, their ability to enjoy longer play sessions, their ability to tolerate high levels of functionality in the user interface, their decision to play games as a lifestyle preference or priority. The result of the focus on this market has led to a console user demographic dominated by young white males and arguably the genres of first person shooter (FPS), sports and driving games [1].
However, there have been successful attempts to broaden the console user demographic, most notably by Nintendo through its DS console and innovative titles such as Nintendogs, WarioWare, and BrainAge, the latter of which created huge sales amongst previously nonconsole gamers.
The success of the DS led Nintendo to develop the Wii which has achieved phenomenal sales since its launch [2] by capturing the imagination of users not traditionally considered part of the current console gaming market [2]. From the perspective of both the game developer and game player, the most innovative feature of the Wii is the “Wiimote.‘’ The Wiimote itself was one of the primary design aspects of the Wii, and in an interview by Kenji Hall for Business Week in November 2006, Shigeru Miyamoto describes part of Nintendo’s rationale:
“The classic controller was something we had become fond of and gamers had become comfortable with. It had many important elements. But it also had come to dictate a lot of what went into games—the way graphics were made, the way battles were fought in role-playing games, the arc of in-game stories. They were all being made to fit one standard. Creativity was being stifled, and the range of games was narrowing.”
Their solution was the Wiimote, a wireless controller which is able to sense both rotational orientation and translational acceleration along three-dimensional axes. It achieves this through the use of inbuilt accelerometers, together with a light sensor. This light sensor is used in conjunction with an array of light-emitting diodes centrally positioned above or below the console’s display [3] which allows for six degrees of freedom. The Wiimote can be augmented with additional features, one of which is the “Nunchuk,” which features an accelerometer and a traditional analog joystick with two trigger buttons [3]. The overall result is a game interface capable of supporting a huge array of input possibilities that will enable new and exciting video game experiences.
Whilst mobile phones are in their relative infancy as gaming devices [4] compared to 7th generation consoles such as the Wii, their ubiquity and increasingly rich feature sets means they are equally capable of addressing the issue of widening the game player demographic [5]. Furthermore, as ubiquitous consumer devices they are ideal for interacting with urban computing environments and in particular large public displays [6]. Further, with the ever expanding feature set on handsets, mobile game developers can increasingly turn to innovative forms of user input such as RFID/NFC, cameras, microphone, and so forth [7].
Whilst some of the aforementioned input mechanisms have been the subject of a variety of research projects [7], the use of 2D accelerometers on mobile devices has been the subject of few studies [8] and the potential for using 3D accelerometers on phones appears completely unexplored [7]. This is principally due to the fact that the first 3D accelerometers have only just been integrated into mobile phones and these phones have yet to become widespread. The Nokia 5500 was one of the first such equipped phones having been targeted at sports users, utilizing the built-in motion sensors as a pedometer and speed/distance tracker for various exercising purposes. Other phones with motion sensors include Samsung’s SCH-S310 (claimed to be the world’s first 3D motion sensing mobile), NTT’s N702, and more recently the Nokia N95.
Although there is an obvious potential for innovation in game play on mobile phones themselves, in this paper we are concerned with the potential use of a phone as a controller for games (or indeed a variety of applications) running on large public displays. In the following sections, we present the generic design and implementation of such an interactive system together with issues related to the implementation on this particular phone. Furthermore, we highlight the potential for the general use of accelerometers on phones in a variety of control interfaces. To illustrate the opportunities of this interface, we highlight its operation through the development of a novel multiplayer car racing game using an analogue control mechanism. This extends upon our previous work [9] based on the old arcade classic Tron Light Cycles in which a digital control scheme was employed. We then present the user experience from participants playing the game at a public event in Budapest, before drawing our overall conclusions.
2. Phone Controller System
Whilst there are obviously possibilities for creating innovative interfaces using 3D motion as the input mechanism for mobile games, in this project we explore using the phone as the games controller for multiplayer games shown on external displays and, in particular, large public screens. Using a large screen offers a number of benefits: it frees the games developer from constraints of the limited graphics capabilities of the mobile screen [10], enables a greater amount of movement to the participating players, provides a rich social atmosphere [6] and affords an opportunity for rich social interaction [11] in a variety of urban landscapes. However, we do acknowledge the practical deployment of such displays is often fraught with difficulty [12].
The system developed is part of a generic framework, which we have termed Poppet,1 for utilizing on-board phone sensors such as cameras, accelerometers, radio frequency identification/near field communications (RFID)/(NFC), which can be linked to games running on large public displays via Bluetooth as shown in Figure 1.
Although Poppet is capable of addressing a range of devices, in the next section, we specifically describe the design challenges involved in producing a mobile client for the Nokia 5500 together with its on-board accelerometers. However, the game server design is sufficiently generic to allow interaction with a wide range of phones using Bluetooth as the means of communication.
2.1. Mobile Client Design
Accessing the 3D motion sensors requires the use of the Symbian Sensor API which is similar in function to J2ME’s Mobile Sensor API (JSR-256). Both of these APIs provide the potential to access a wide range of sensors such as accelerometers, thermometers, barometers, and humidity monitors, in fact any type of sensor designed to be incorporated in a mobile phone, or those accessible via Bluetooth [13]. Sensors need only be supported by the API library to be usable. The Symbian Sensor API is available from Nokia and requires the use of the Symbian S60 3rd Edition SDK. Whilst there is support for the Symbian Sensor API on several mobile devices, there are currently no mobile phones that include JSR-256 [13].
The general Poppet framework uses J2ME, as this is currently the most widely deployed mobile platform. Although the sensor data is not directly accessible from J2ME, because of the lack of JSR-256 as previously highlighted, the problem can be overcome by using a socket connection on the mobile phone to allow access to native services. The general solution for accessing native services from J2ME on Symbian S60 phones is by opening a low level socket connection in a Symbian C++ application then connecting to the defined port on the loop-back address from the J2ME application [14]. Thus the Symbian C++ application can retrieve sensor/other data from the phone and then forward this to any J2ME applications that may be listening. This solution has been applied in our simple mobile client to allow J2ME applications to access the 3D sensor data.
The connection between the game client and server is based on Bluetooth, which creates a reasonably high bandwidth (data rates can vary between 1 Mbps and a few Kbps depending upon the type of transfer mode initiated) between the devices. In order to allow a device to become discoverable by others, it is necessary to advertise at least one Bluetooth service. In the case of Poppet, the mobile client implementation uses the official Java Bluetooth API (JSR-82) to alleviate porting issues.
The Nokia 5500 utilizes a 6g accelerometer, that is, it can detect acceleration forces with a magnitude of up to six times that of earth’s gravity. The accelerometer outputs three 12-bit signed data values at a frequency of around 37 Hz. These outputs correspond to the three phone axes (x, y, z) as shown in Figure 2.
In terms of illustrating the opportunities for creating novel interaction methods using phones with inbuilt accelerometers, it is worth considering the following three Figures (3, 4, 5) which show the recorded output from the sensors on the Nokia 5500 in three different scenarios. Note that output has been expressed as acceleration forces in g for clarity.
Figure 3 shows the three accelerometer outputs when the phone was statically placed upon a desk (representing a horizontal plane). It can be seen that outputs x and y are approximately zero (although some sensor noise is evident) and output z is showing a positive 1g force which is the effect of gravity on the device. Note, although sensor noise could be reduced by the application of a digital filter on the output values, in this case we felt showing the raw data was more informative. Overall, this figure illustrates that gravity provides a means of deducing the orientation of the phone which can then be utilized to provide a “tilt-based’’ controller.
In Figure 4, we see the accelerometer outputs resulting from several rapid lateral movements of the phone on a desk. At the start of the graph, we see outputs in the same state as in Figure 4 and then very large acceleration forces on axis y produced by the movement. Note that the output rapidly alternates between positive, then negative data values indicating the transitions from acceleration to deceleration and vice versa.
There are some smaller residual artifacts from this movement, seen in both x and z , which result from testing by hand rather than using a fixed jig (as it is difficult to isolate a movement completely under such conditions). Figure 4 depicts the potential for utilizing “hand gestures” for the control of events within games [15] and without the problems associated with using the camera to obtain the movement [15]. However, gesture recognition requires careful study as the variation in how the user holds the phone could produce anomalous outputs and there is no method of obtaining the phone’s physical position within actual space as provided by the “sensor bar’’ for the Wii. This is because both rotation and translational acceleration affect the accelerometer’s output (essentially they can produce the same internal forces on the accelerometer).
Figure 5 visualizes the output after the phone has been dropped. The trace shows that initially the phone is held upright with the screen held at the top with gravity acting on x. When the phone is dropped, the effect of gravity is overcome and all three accelerometers output values approximating zero, before the phone hits the floor with a jolt. The aim of this test was to show that the phone could be used to measure activity of a player within a game, such as jumping or running, whereby the phone would simply be worn rather than held and directly controlled.
2.2. Game Server Design
Using Bluetooth on a PC requires the implementation of a Bluetooth stack. Furthermore, different models of USB Bluetooth dongle have different stacks that may or may not be compatible with Microsoft Windows’ Bluetooth stack. To alleviate some of these compatibility issues, the game server was implemented using the Franson Bluetooth SDK for C#. This SDK supports the two most common Bluetooth stacks, the Windows stack and the WIDCOMM stack, which should ensure that the Poppet game server can be used with most Bluetooth devices. The basic communication architecture is shown in Figure 6 which also illustrates the on-phone socket communication used to transfer accelerometer data between the native application and a J2ME application.
The game server performs two main tasks: it locates/connects the Bluetooth phone controllers and it also processes the data to be used within the game. The first requires device discovery, which looks for the previously advertised service number on each phone controller in range of the PC. Once a phone controller is found, the PC tries to establish a connection with it. If connecting to the mobile is successful, both sides have all the necessary information to send and receive data between each other. The communication is based on streams at both ends.
The remaining task of the game server is to process the received data. As data triplets are arriving at the PC approximately 37 times per second which is fine in this case although, there might not be enough time to process every triplet and maintain the operation of the game in real-time. Therefore, if a real-time game is created, the developer has three options: they can drop packets, interpolate missing accelerometer samples, or assume that all samples can be processed before the next change in game state.
3. Game Implementation
Even though the project presents some interesting technical challenges, it is principally aimed at developing a theoretical understanding of the user’s experience in utilizing this type of tangible interface in conjunction with a game played on a public display. This understanding will be achieved by collecting empirical data from game prototypes played by various groups. Our first prototype was the game we called Mobi-Tron [9] which operated with a simple digital steering control and was tested amongst staff and students at Lancaster University.
When questioned on how to improve the experience, a number of players expressed a wish that the degree of tilt of the phone should correlate to the speed of the light cycle, as the current game only allows for changes in direction but not speed [9]. Other comments suggested incorporating sounds which the current version lacks and one person suggested that rather than simply turning at right angles, the full 360 degrees of movement should be used [9]. This has been subsequently incorporated in our new game called TiltRacer which is presented in this paper and shown in Figure 7.
TiltRacer utilizes the Poppet framework previously described and the actual driving game was developed in C# using Microsoft’s XNA game framework. The initial version of the game was a simple figure of eight shaped track shown in the previous figure, although this was expanded to a more complex track, shown in Figure 8, for the user trial as our initial in-house feedback considered it too easy to master. The game itself is a simple first to complete a selected number of tracks and can be played by up to 4 players simultaneously although in the trials we limited this to two.
4. User Experience
In this section, we present the results of testing the system amongst participants of the Forum Nokia Technical Days and European Mobile Innovation Competition2 held between the 6th and 8th June 2007 in Budapest, Hungary. The game was displayed on a large screen at the evening reception for the event is shown in Figure 8 and participants were invited to try the game out. To ascertain aspects of the user experience as a whole we decided to adopt and ethnographic approach [16] because of the nature of the event which combined video and still image recording by three researchers of the 30–35 individuals who tried the game.
The general response to the interface was that it was fun, as highlighted by some of the players’ facial reactions in Figure 9 and that it was also intuitive, as participants quickly gained an understanding of how their movements affected game avatars with little or no instruction from the research team or preceding players. Further, as is evidenced by Figure 9, the players were concentrating solely on the screen and not looking at the interface which aided their immersion in the experience. Whilst this observation is interesting in relation to research concerning the display of public information on a large screen against private information on a small screen it should be noted that this largely influenced by the nature of the game-play and something like a card game would provide more interesting analysis of this area.
The overall response to the game is probably best summed up by Harri Lehmuskallio, a user experience specialist for Idean who was part of the judging panel,
“The application expands the interaction from mobile to the physical world. The game concept is simple, but it demonstrates that casual games can be played this way in public spaces. The fact, that TiltRacer was created, opens new possibilities for the industry as it shows that innovating is fun and doable."
5. Conclusions
The Wii console has captured the imagination of users who previously have not participated in console-based gaming. Whilst the particular game genres chosen are a factor in user acceptance, it is undoubtedly its innovative controller functionality that has proved its most attractive feature.
Whilst we are certainly not suggesting that mobile phones could rival the Wii, the inclusion of 3D accelerometers opens up the possibility for the same type of innovation for games running on mobile phones. Further, they also provide users with an innovative game interface that they already carry around as part of their everyday day lives. This could allow them to interact in novel ways with games forming part of the environment. In particular, the increasing presence in our cities of large public displays is making this hybridization of virtual and real space available to the mass market and thus brings new opportunities for games.
Although this game can be considered a fairly simple example in terms of graphics and complexity, and there is certainly a requirement for further study using other types of game, it does highlight that a novel interaction mechanism coupled with a fun group activity can provide an enjoyable social experience, with high levels of user interaction.
Notes
1In folk-magic or witchcraft, a “Poppet’’ is a doll made to represent a person, for casting spells on that person. The intention is that whatever are the performed actions, the doll is transferred to the subject. These dolls are often incorrectly referred to as Voodoo dolls.
2The game represented the UK entry for the innovation completion, having previously won through the UK heat, and was the ultimate winner of the event.
Acknowledgment
The authors thank Nokia for the provision of software and hardware to the Mobile Radicals research group which were used in the implementation of this project.