An Introduction to Augmented Reality in Gaming Jonathan Mearns B00105243 Abstract Augmented reality gaming is a form of gaming that enables the player to use the environment that they are in as part of the game that they are playing. The technology for creating such a game is quite widespread but is rarely used on a wide scale. It is also only played by a niche market. Augmented reality gaming could become more popular in the future when the technology is perfected and is less jittery, but until that time, augmented reality gaming will probably not become mainstream as it’s still in it’s infancy. Introduction This paper surveys the field of Augmented Reality, in which 3-D virtual objects are integrated into a 3-D real environment in real time. This paper provides a starting point for anyone interested in what Augmented Reality is and the applications it can be used for. Primarily gaming. It also aims to point out the benefits to augmented reality gaming as well as any detriments. History of Augmented Reality Augmented Reality took it's first steps toward becoming reality around 1960. A cinematographer named Morton Heilig creates the Sensorama, a machine which is similar to a modern television except that the viewer's head is encompassed by a screen. The viewer sits in a chair which can tilt in accordance with what is being shown on the screen while simultaneously being subjected to stereo sound, ambient noise where appropriate and even smells when the situation merits it. The machine itself never went into mass production due to the inventor being unable to procure financial backing, but it wasn't long before others started to catch on to the idea and put their own spin on it. In 1966, Ivan Sutherland invents the first head mounted display that encompasses the wearer's vision which allowed a person to feel totally immersed in what they could see In 1999, Hirokazu Kato develops the ARToolKit which is a computer vision tracking library that allows for the creation of augmented reality applications [Kato99b]. This toolkit is widely in use at the time of this paper’s creation and several indie applications have been created and examples can be found throughout the internet. One such use of this toolkit is the basketball application where the basket is projected onto a marker held up in physical space and the player moves his hand to throw the virtual ball into the basket. While all of the above methods of providing an alternate reality game are effective, it wasn't until the year 2000 that the first outdoor augmented reality game was made. This game combined the virtual aspect with the real world and as such made the player feel like they were more like living the game compare to just sitting there and playing it. This game was ARQuake. ARQuake makes use of a headset, a standard laptop, a GPS system and a custom made controller. The player would wear the headset while carrying the laptop in a backpack and carrying the gun controller in their hands. This would allow them to move about in an indoor or outdoor area using the environment as they're playing field. Any "enemies" encountered would seem to be moving about in actual real places. The whole process was not without its flaws as the GPS could only be accurate to so many feet and would not always give the players actual correct position. While more accurate instruments could be used now, it was still a remarkable achievement combining the real and the virtual to make a game. In the year 2009, Pranav Mistry develops the Sixth Sense wearable gestural interface[Mistry09]. This method of augmented reality uses a spatial display to project information onto objects rather than viewing them through a lens or a monitor. This technology is still being perfected at the time this paper was produced so it isn’t mainstream yet, although the technology is intended to be released to the mainstream population when the technology is commercialized. Later in 2009 the application Wikitude was released to the general public for use on their smart phones ( i.e. Apple’s iPhone, Android and Symbian platform phones ). This application layered real time data over what the camera on their phone displayed. It would calculate the user’s position using Global Positioning System (GPS), a compass and an accelerometer to determine the user’s location on the planet and what direction they were facing. It would then use this data to inform the user of any points of interest that were in the direction they were facing and give them information such as the history of the landmark, contact details and longitude and latitude of the landmark. Inputs for Augmented Reality Augmented Reality games require a way of allowing the player to participate, so different methods are employed to allow them to see their reality with a virtual augmentation in it. One such method is the head mounted display. Head Mounted Displays These offer some of the most immersive atmospheric gaming available. The user wears a helmet with a six degrees of freedom sensor, which takes into account the position of the user and which direction they are looking in, and overlays the virtual data in the viewer’s vision by way of a half silvered mirror to create the illusion of a virtually augmented reality. This was the type of display used for the first augmented reality game ARQuake. See figure 1 below. [Figure 1] Handheld displays These are the most common forms of Augmented Reality for gaming. While not limited to mobile phones, these are the most common platforms for AR games due to their portability and their back mounted cameras. These are used like looking through a magnifying glass only instead of magnifying the image of what it’s pointed at, it will often overlay the image with the virtual aspect of the game such as coins to collect or seeing a pitfall in front of you when there is nothing there. See figure 2 below. [Figure 2] Spatial Displays The third type of display is less common but allows the user to use their hands as an input device. For example, while wearing the spatial displays digital projector, the user can make certain motions in front of the display’s camera and the projector will display relevant data. Uses for this are not really used for gaming but for practical uses such as, when the user draws a circle on their wrist with their finger, the projector will display a clock on their wrist with the current time. See figure 3 below. [Figure 3] Most commercial Augmented Reality games make use of the ARToolKit. This open source library is widely used by hobbyist and companies alike. Requirements for Augmented Reality Gaming For the ARQuake game, we require: • A mobile computer which can be carried by the user, with enough processing power to generate the 3D graphics in real time. • A suitable head mounted display which the user looks through to see the computer generated graphics. • Tracking devices which can measure the position and orientation of the user’s head (and possibly the hands and arms as well) so the computer can accurately generate graphics to overlay the physical world. • An input device to allow the user to control the game, shooting at monsters, changing weapons, reloading levels, etc. • A software application which interfaces with the hardware devices, and provides augmented reality 3D overlay using optical or video methods. [Piekarski] All of the have been deemed necessary for an augmented reality game, according to Piearski to function for a head mounted display. If the display were handheld, the input and display would be one and the same. Possible Problems All of the above displays require substantial processing power and access to large amounts of RAM to work effectively. This is perhaps one of the downsides to Augmented Reality gaming. To effectively play a game at a constant frame rate, a lot of factors need to be taken into account. Will the game be able to display if the user is looking at the ceiling? Will it display if the user is looking into the sky? Can the display keep up with the player if the user is moving at speed i.e. in a car? Will the display work properly if the user is running and jostling the display about? All physical factors must be taken into account to prevent hang-ups and crashing of the game system. This seems an impossibility as an infinite amount of possibilities cannot be anticipated and compensated for before they happen. As such, Augmented Reality gaming will always be much more prone to instability. This is regarded as a cardinal sin amongst gamers. Why make a game that can crash at any given time? How will my data be saved? These types of problems are something that gamers are worried about. If they are going to spend a lot of their time playing a game and progressing in it, what would happen if all their progress was lost simply because they looked directly up while wearing a head mounted display? Another problem faced in Augmented Reality gaming is the registration problem. The objects in the real and virtual worlds must be properly aligned, otherwise the illusion that the two worlds co- exist will be compromised. An example in gaming would be a pick up that has to be acquired to progress. If the player is to stand over it to acquire it, then this might be a problem if the location of the object is improperly calibrated and is instead inside a wall or other physically inaccessible location. A way around this would be to use RTK (Real Time Kinematics) instead of using a standard GPS system. A standard GPS system is only accurate to within a few metres but an RTK system is accurate to within a few centimetres. This may seem like a positive thing but the RTK system requires a ground based station to be put into place along with using standard sattellite positioning. These can be an expensive purchase for the casual gamer although a technically minded gamer could make their own if they were so inclined. Full instructions and soirce code is available to download via the following site. http://gpspp.sakura.ne.jp/rtklib/rtklib.htm "Registration problems also exist in Virtual Environments, but they are not nearly as serious because they are harder to detect than in Augmented Reality. Since the user only sees virtual objects in VE applications, registration errors result in visual-kinesthetic and visual-proprioceptive conflicts. Such conflicts between different human senses may be a source of motion sickness"[Pausch92]. This means that the person is actually touching/ interacting with a certain object but may perceive it to be something different. This could lead to confusion or as Pausch states, motion sickness. Other studies also support this theory such as a study of motion sickness by CM Oman."However, the identification of other variants such as spectacle sickness and flight simulator sickness in which the physical motion of the head and body is normal or even absent has led to a succession of "sensory conflict" theories that offer a more comprehensive etiologic perspective."[Oman90]. Developing a multiplayer game for an Augmented Reality system is also a significant challenge. The precision of the instruments used to calculate position and direction must be incredibly accurate to allow an accurate representation in game. One such problem would be a cooperative mission in a FPS environment. If the player’s positions are not clearly identifiable then several problems could occur in game. One such problem is friendly fire. Even if each player is not pointing a weapon at the other, friendly fire could still occur if the calibration of the equipment used to read in their positions is off, or if the equipment is too slow to register a change in one or both of the player’s positions. “In developing a multi-user augmented reality video conferencing system, precise registration of the virtual images with the real world is one of the greatest challenges.”[Kato99a]. Possible Benefits Augmented Reality might apply to all senses, not just sight. So far, researchers have focused on blending real and virtual images and graphics. However, AR could be extended to include sound. The user would wear headphones equipped with microphones on the outside. The headphones would add synthetic, directional 3–D sound, while the external microphones would detect incoming sounds from the environment. This would give the system a chance to mask or cover up selected real sounds from the environment by generating a masking signal that exactly cancelled the incoming real sound [Durlach95]. This form of Augmenting Reality would be infinitely useful in gaming. If the headphones were integrated into a HMD ( Head Mounted Display ), the user could not only see enemies in front of them in the case of a FPS, but would also be able to hear them in the direction they are in the virtual space. Augmented Reality can also be used for games based learning. Consistently, attempts at getting students to learn through gaming have fallen flat for one simple reason. The games are not fun. If the games are not fun the players lose interest quickly and thus they learn very little. If the learning game was less of a game and more of an experience then things could certainly change. AR games go beyond purely providing information; they give students experiences such as conducting a virtual investigation. Games are organized around problem solving activities, activities where players must research and discern the value of information, reason from evidence, and construct new representations of their understandings. Using simulation technologies, AR games may also go beyond project-based learning by entering students’ plans and creations in simulated worlds, allowing them to learn through the consequences of their work [Squire06]. A primary benefit of games-based approaches is that they ask students to try on roles other than being students; games can allow learning to occur through the lens of a particular identity (such as being an environmental engineer, journalist or historian) [Gee03] [Shaffer04] [Squire06]. Henry Jenkins (2001) uses the term memorable moments to describe the logic by which games operate. Drawing on the work of Seldes (1957) Jenkins argues that aesthetically, games are less about telling formal stories, and more about setting up interactions that result in memorable moments for the player. A challenge for educators is how to create such memorable moments that are not only fun, but academically meaningful.[Squire06]. Benefits to the Gaming Society as a Whole Augmented Reality gaming can benefit the gaming society in many ways. One of which is the most obvious and yet most prominent. Physical interaction with other real live human beings. While some people may be content to sit at home and play for dangerously long sessions over the internet with other people, augmented reality gaming could bring an incentive to these people to take their gaming to a new level of realism and even a beneficial one. Many of this era's youths are becoming more and more obese. This could be attributed to a lot of factors, not just gaming. However, for those that would rather game online than get some physical exercise, this may be the answer they were seeking. Studies have been conducted as to whether the WiiFit is useful as a game that benefits a user's health. One particular study conducted by the University of Queensland concluded that "Activity fostered by Wii Fit™ showed an immediate effect on balance and strength that needs confirmation by statistically powered studies."[Nitz09]. The WiiFit isn't an augmented reality game but it is still a game that urges the player to exert themselves physically to acheive better results in game. The improvement to their health and wellbeing is a side effect, but it is a beneficial one. This sort of gaming could be applied to augmented reality gaming such as olympic type sports performed with virtual objects, i.e. a virtual javelin throw performed in augmented reality. This type of Augmented Reality gaming could also have other benefits financially if this was to be used on a wide scale. The above mention of olympic events could be carried out without the use of all the expensive equipment that goes with it. Augmented Reality gaming could also be used as a training tool for jobs such as becoming a pilot. Rather than having to spend thousands of pounds on things like a training plane / helicopter and the fuel to fly it, a real time simulator could be built and the costs and risk to the trainee would be lowered. This all depends on which type of training device is used some simulators cost may outweigh the costs standard training but the risk to an inexperienced pilot is still reduced. A fully realistic simulator could be used but would cost in exess of $15 million but a semi realistic one could be used for a substantially lower cost of around $2 million.[Rosenkopf98]. Conclusions In conclusion, augmented reality gaming is a positive use of technology as players gain real world experience through teaching games and simulators, gain additional health benefits through physically active games and open their minds to new ways of thinking and learning. Although the technology is not yet perfected, as augmented reality can often be inaccurate with regards to the player's positioning relative to the environment, improvements are constantly being made and this might be viable for a mainstream application in the near future such as an augmented reality console. References Durlach 95 Durlach, Nathaniel I. and Anne S. Mavor (editors). Virtual Reality: Scientific and Technological Challenges. (Report of the Committee on Virtual Reality Research and Development to the National Research Council) National Academy Press (1995). ISBN 0-309-05135-5. Gee 03 Gee, J. P, What Video Games Have to Teach Us About Learning and Literacy. New York: Palgrave Kato 99a Kato, Hirokazu and Mark Billinghurst, Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System, http://www.hitl.washington.edu/projects/artoolkit/Papers/IWAR99.kato.pdf (1999). Kato 99b Kato, Hirokazu, ARToolKit Home Page, http://www.hitl.washington.edu/artoolkit/, (1999). Mistry09 Mistry,Pranav, Sixth Sense - a Wearable Gestural Interface, http://www.pranavmistry.com/projects/sixthsense/, (2009). Nitz 09 Nitz, J.C., S. Kuys, R. Isles, S. Fu, Is the Wii Fit™ a new-generation tool for improving balance, health and well-being? A pilot study, http://informahealthcare.com/doi/abs/10.3109/13697130903395193, (2009). Oman 90 Oman, CM, Motion sickness: a synthesis and evaluation of the sensory conflict theory, http://www.ncbi.nlm.nih.gov/pubmed/2178753 (1990). Pausch 92 Pausch, Randy, Thomas Crea, and Matthew Conway. A Literature Survey for Virtual Environments: Military Flight Simulator Visual Systems and Simulator Sickness. Presence: Teleoperators and Virtual Environments 1, 3 (Summer 1992), 344-363. Piekarski Piekarski, W and Thomas, Bruce H , ARQuake - Modifications and Hardware for Outdoor Augmented Reality Gaming, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.79.6851&rep=rep1&typ e=pdf, date of print unknown. Rosenkopf 98 Rosenkopf, Lori and Tushman, Michael L, The Coevolution of Community Networksand Technology: Lessons from the Flight Simulation Industry, http://icc.oxfordjournals.org/cgi/content/abstract/7/2/311, (1998). Shaffer 04 Shaffer, D. W, Pedagogical praxis: The professions as models for post-industrial education. Teachers College Record, 106(7). Squire 06 Squire, Kurt D, Mingfong Jan, James Matthews, Mark Wagler, John Martin, Ben Devane, Chris Holden, WHEREVER YOU GO, THERE YOU ARE: PLACE- BASED AUGMENTED REALITY GAMES FOR LEARNING, http://22.214.171.124:9090/progress?pages&id=2356384517&fileName=MTYtc3 F1aXJlLXNoZWx0b24tYm9vay5wZGY=&url=aHR0cDovL3dlYnNpdGUuZWR1Y2 F0aW9uLndpc2MuZWR1L2tkc3F1aXJlL3RlbnVyZS1maWxlcy8xNi1zcXVpcmUtc 2hlbHRvbi1ib29rLnBkZg==&foo=12 (2006).