Toward the indoor use of location-based augmented reality by xumiaomaio


									                                 Chapter 1. Toward the indoor use of location-based augmented reality

Chapter 1

Toward the indoor use of
location-based augmented reality

(short summary of the motivation and research questions, followed by a short version of the con-
clusions, to be written when the rest of the content is near final)

1.1         Introduction
The meaning of the term mobile phone has broadened over the past few years. While a mobile
phone today is still capable of making telephone calls from virtually anywhere, its capability to
do things other than telephony has greatly increased.
    The technologies introduced to the user of the mobile phone include among others bigger
screens, increased processing power, mobile internet connectivity and a range of sensors. With
the rise of these new technologies and capabilities, applications for mobile phones can provide the
user with richer content and gather more information about the surroundings of the user.
    A survey of the technologies available to the user of the mobile phone and the programmer of
mobile applications is provided in section 1.5.

    An example of an application that uses mod-
ern mobile phone hardware and capabilities in the
Layar augmented reality browser1 .     Layar searches
the internet for information on objects of inter-
est nearby the user.       Furthermore, Layar over-
lays the locations of interesting objects on the im-
age of the phone’s camera.        If the user would
turn around, the image changes with the user,
showing the interesting objects in the new di-
rection.    Figure 1.1 shows an example of La-
yar, overlaying houses for sale in the city of
Amsterdam on top of the phone’s camera im-
                                                           Figure 1.1: An example of the aug-
    The Layar augmented reality browser is a prime ex- mented reality browser Layar.
ample of mobile use of augmented reality; it augments
information from a remote source onto the reality that is displayed on the mobile phone’s screen.
Many more such applications exist, all following the basic principle of augmenting information on
the slice of reality visible to the user.
    The principle of augmented reality on a mobile phone has little limits; the information that
is overlayed on the camera image is limited by the information available and the imagination of
the developer. In practice, however, the technology used for determining the position of the user
and the direction the camera is pointed is not very well suited for use indoors. Possible use cases
for augmented reality indoors would be finding one’s way inside a large building or displaying
additional information about art objects in a museum.
  1 Layar   augmented reality browser,

Toward the indoor use of location-based augmented reality                                          1
Simulations & Games

1.2                     Motivation
The most logical way for a mobile device to determine its location is the Global Positioning
System. The use of the Global Positioning System—or GPS—relies on signals from multiple
satellites orbiting the planet. As the phone receives precisely timed signals from these satellites, it
can determine its position by triangulating the distances from the different satellites. This works
great outdoors, but is not feasible within a large building; the reinforced concrete in the building’s
structure will disrupt the signals coming from outside. This disruption makes GPS unusable inside
a building.
    The direction a device is held can be measured with the use of a compass. As the compass
relies on the magnetic field of the earth, however, this can become unreliable inside of a building.
Reinforced concrete and even nearby metal objects are known to disrupt the earth’s magnetic field
locally, causing a compass to occasionally measure the wrong value. This creates a necessity for
the aid of some other means of measuring direction inside a building.

   The problems with both GPS and compasses makes the use of augmented reality on a mobile
phone inside of a large building a challenge. As the uses for augmented reality are virtually
limitless, enabling the use of it indoors would create opportunities for the technology in a new

1.3                     Research questions
Focussing on using a mobile phone, the research question for enabling the use of augmented reality
inside of a building, given the usability of GPS and a compass becomes as follows:

                      How can augmented reality be used inside a building using standard mobile phone hard-

   As becomes clear from the previous section, there are two specific measurements that fail inside
a building: position and direction. In order to enable the use of augmented reality inside, then,
ways to measure these two aspects inside of a building are needed. An answer to the research
question stated above thus involves two aspects:

Positioning How can a mobile phone determine its position inside of a building without the use
      of GPS?
Direction How can a mobile phone determine its field of view inside of a building without the use
      of a compass?

1.4                     Method
In order to provide an way of determining the position and direction of a mobile phone inside of a
building, a survey of the components in a mobile phone has been compiled. From this list, suitable
candidates for measuring position and direction have been selected. The expected accuracy of these
measurements will form the basis of an assessment whether the use of augmented reality inside of
a building is feasible.

1.5                     Available hardware components
To be able to make a selection on candidate components for position and direction measurements,
a list of available components for mobile phones is needed. The term “mobile phone” encompasses
a great variety of devices, ranging from simple cellular phones with little more than a microphone
and a speaker to devices packed with communication and measurement components. For the scope
of this research, however, the type of mobile phone capable of augmenting real time information
on a live image from an embedded camera is considered.
    A pair of well-documented mobile phones will be the continued example of currently available
mobile hardware in this section. The selection of these devices is based on their wide use and
known capability for augmented reality applications like Layar and Wikitude2 . The two devices
selected for this section are the Apple iPhone 4, called “iPhone” from here on, and the HTC
Magic, called “Magic” from here on.
           2 Wikitude,

2                                                                    Study Tour Pixel 2010 - University of Twente
                              Chapter 1. Toward the indoor use of location-based augmented reality

    The iPhone and Magic differ in manufacturer and hardware design. Based on information from
their respective manufacturers, a list of available hardware components or compatible technologies
is presented Table 1.1. [Apple Inc., 2010b; HTC Corporation, 2009]

                                            Apple iPhone 4       HTC Magic
                      Released                July 2010          April 2009
                      Operating system          iOS 4            Android 1.6
                      Accelerometer              !                  !
                      GPS                        !                  !
                      Bluetooth                  !                  !
                      Camera                 5 Megapixels       3 Megapixels
                      Compass                    !                  !
                      Gyroscope                  !                  %
                      Memory                  Undisclosed      192 Megabytes
                      Screen size           3.5” (960×640)     3.2” (320×240)
                      Processor speed         Undisclosed          528MHz
                      WiFi                         !                  !
       Table 1.1: List of hardware components for the Apple iPhone 4 and the HTC Magic

    Both devices are capable of running mobile augmented reality applications as stated before, so
there are many similarities between the two. The iPhone, however, being released over a year later,
has more powerful hardware. While the speed of its central processing unit is not provided by
Apple, it is rumoured to run at 1GHz; about twice as fast as the processor of the Magic. The same
goes for the amount of memory available on the iPhone, which is rumoured to be 512 Megabytes,
significantly more than that of the Magic. A second noteworthy difference is the availability of a
gyroscope in the iPhone, providing an additional means of measuring motion. The difference in
operating systems is of little concern; many popular augmented reality applications run on both
iOS and Android and both have a programming interface to access any sensor available in the
selected devices. [Apple Inc., 2010a; Google Inc., 2010]

1.6      Usability of hardware components
As is discussed in section 1.2, the use of GPS and the compass in the iPhone and Magic is not
feasible inside of a building. In stead, the position of the device and the direction it is pointed
in have to be measured using other components in the list presented in section 1.5. This section
outlines the use of some of these components for the tasks of measuring the required parameters.

1.6.1    Triangulation of wireless signals
The mobile phones introduced in section 1.5 both have wireless communication capabilities other
than cellular telephony (which is not listed in the table). Wireless signals all operate on the same
basic technique, broadcasting information as radio waves at a predetermined radio frequency. One
of the well-known properties of this technique is that the strength of the signal that is broadcasted
becomes weaker when the distance between the sender and reveiver becomes greater.
    By using the Received Signal Strength Indicator—or RSSI—of a wireless receiver, the fraction
of the power that is measured at the receiving end gives an indication of how far the receiver is
away from the sender. When multiple such senders are present, the distances from each of these
senders and the locations of these senders can be combined into an indication of the receiver’s
current location. [Harney, 2009]
    The following sections provide an insight into the usage of wireless technologies available to the
mobile phones discussed in section 1.5 for location estimation and their feasibility for the scope of
this research.

WiFi has been around for quite a few years as a means to create wireless local area networks in
buildings. Its presence in buildings makes it an iteresting candidate to use for triangulating the
position of a WiFi-capable device.
   A number of teams around the world have researched the feasibility and accuracy of measuring
WiFi signals to determine the position of a device. Researchers from the university of Pittsburgh

Toward the indoor use of location-based augmented reality                                           3
Simulations & Games

concluded that WiFi signals can be triangulated into a position and show the accuracy of such a
measurement can be considered accurate to about three meters. However, the results also show
that the impact of walls on the measurements increase the position error of the measurement,
drifting up to five meters when two walls are between the WiFi sender and receiver [Prasithsan-
garee et al., 2002]. Similar researches in 2004 back these findings, but add the note that even
the position of the person holding the WiFi receiver can influence the readings, increasing error
margins, reducing the accuracy of the measured location [Kaemarungsi and Krishnamurthy, 2004;
Bill et al., 2004].

    As becomes apparent in all the results provided above, location measurement with WiFi signals
has a typical accuracy of three meters in the most ideal situation. Taking the increased position
error caused by walls and the user into account, it would seem that using WiFi for a position
measurement suitable for augmented reality application inside a building is not feasible.

Some research has gone into using bluetooth as a means to measure the position of a device
between a number of bluetooth base stations. Experiments by Silke Fedlmann and others in 2003
lead to the conclusion that measurements in signal strength of bluetooth technology is indeed a
viable way of triangulating the position of a device in an office space. In this case, measurements
were accurate to about two meters [Feldmann et al., 2003]. A second publication in 2006 by
Hiroyuki Ochi and others implements a similar approach, lining a room with bluetooth reference
points also achieves an accuracy of about two meters [Ochi et al., 2006]. Another research in 2009
by Thomas King and others leads to approximately the same result; the accuracy achieved in
measuring position with bluetooth is about two and a half meters [King et al., 2009].
    All approaches use a fixed infrastructure of base stations—in the case of bluetooth often called
“dongles”. The two German teams use a bluetooth capable mobile device with software to mea-
sure the signal strength, while the Japanese team prove the principle with a number of fixed
devices. King’s research methodology is the more interesting for the scope of this research, as it
experiments with the impact of a few variations in setup. The most notable variation is the use of
extra bluetooth dongles, halving the position error from about five meters with the use of a single
dongle to two and a half meters when eight dongles are used while at the same time reducing the
standard deviation of the position error to just over a meter.

    Concluding from the results provided above, it seems that bluetooth would indeed be a viable
way to determine the position of a mobile device inside of a building. The average position error of
two meters is however high compared to the something like width of a corridor inside a building.
Whether this would be a problem when used in a real implementation depends on the use case;
a user requiring information on what is at the end of the corridor would experience little trouble
with a location error of two meters, but a user looking at the wall of the same corridor might.
    As an advantage of bluetooth over the use of WiFi for positioning, a typical bluetooth dongle is
relatively cheap, costing around ten euros at the time of writing. As the accuracy of the measured
position increases with the amount of available reference points, installing many dongles in a
building would enable an indoor positioning system for bluetooth capable mobile phones.

Location knowledge
The use of both WiFi and bluetooth signal strength to measure location relies on knowledge of the
location of the reference points provided at the location where an augmented reality application is
to be used. For real world usability, an augmented reality application needs to be able to obtain
this information either from the infrastructure itself, by having the base stations broadcast their
locations, or from some central service provided by the location where the application is to be
used. The actual implementation of this information exchange is however beyond the scope of this

1.6.2                 Motion tracking with accelerometers or gyroscopes
The direction in which a mobile phone is held can be measured using the phone’s compass and
accelerometer; the compass being responsible for the values concerning the horizontal plane, the
accelerometer being capable of measuring the direction of gravity and thus the pitch of a device.
An accelerometer can however also measure change in direction in the horizontal plane. Combining

4                                                      Study Tour Pixel 2010 - University of Twente
                             Chapter 1. Toward the indoor use of location-based augmented reality

accelerometer measurements of motion with the known direction of a reference point, a new
direction—relative to the reference point—can be determined.
    A gyroscope—a sensor that remains stationary when the mobile phone is rotated in any
direction—can be used to continuously read the difference between the current direction and
a reference point.
    Research by Peter Lang and other in 2002 showed the feasibility of using the combination of
accelerometers and gyroscopes to track motion. As both accelerometers and gyroscopes suffer
from a certain drift over time, some other system is necessary to periodically reset the reference
point for the accelerometer and gyroscope [Lang et al., 2002].

1.6.3    Specific visual recognition
As a camera is required to augment information on reality represented by the image of the camera,
the availability of a camera on a mobile phone can also be used to determine the current location
or direction of the phone.
   Visual markers, placed at locations of interest or at regular intervals, could provide a mobile
phone with both its current location if the marker is recognized at being at a certain location
and its direction as the user needs to be in front of the marker to accurately capture it with the
   Research into augmented reality games in 2007 demonstrates the use of visual markers to de-
termine the phone’s orientation [Rohs, 2007]. In 2002, an Austrian team implemented a tracking
system using visual markers on a mobile device. This solution uses visual markers near prominent
objects like doors and a knowledge base that maps these markers to locations in order to determine
the device’s location and direction inside of a building [Wagner and Schmalstieg, 2003].

    The use of recognizing visual markers at certain locations is very useful as a reference point
for other sensors. By aquiring a certain value for the current location and direction of the mobile
phone, scanning a visual marker resets all drift on sensors that are used to determine the location
and direction as the user of the application moves through a building.

1.6.4    Combining technologies
Because the principle of augmented reality requires both a mobile phone’s location and direction,
the techniques described above to determine both inside of a building need to be combined in
order to complete the picture.
   Looking at the two described alternatives to the Global Positioning System as the source for
a mobile phone’s location, it seems that the use of bluetooth signal strength triangulation is the
most feasible.
   The use of accelerometers and gyroscopes when available for tracking the motion of the mobile
phone will provide an augmented reality application with the direction needed to determine the
correct field of view. Occasionally resetting the reference point of the accelerometer and gyro-
scopes will counteract the drift these sensors experience over time.

    The mobile augmented reality application Junaio has recently introduced the implementation
of indoor augmented reality using the combination of visual markers and the sensors of a mobile
phone that work inside of building. [Metaio Inc., 2010]

1.7     Conclusions
Looking at the proposed ways of measuring location and direction using available hardware on a
mobile phone, an answer to the question stated in section 1.3 can be provided.

   Determining the position of a mobile phone inside a building is possible using the triangulation
of wireless signal strengths. Based on the research available, bluetooth would be the technology
best suited for this job, as it provides a slightly better accuracy and cheaper implementation than
WiFi. Visual markers also form a feasible way of determining the current location of the user.
   Both approaches require knowledge of the location of either the base stations in range of the
wireless receiver for using triangulation techniques or the visual markers.

   Measuring direction in cases where the compass of a mobile phone can not be trusted can
be accomplished using accelerometers, gyroscopes and visual markers. Where accelerometers and

Toward the indoor use of location-based augmented reality                                        5
Simulations & Games

gyroscopes provide a live value for the direction, visual markers may be used to reset the drift on
the sensors, occasionally providing a new reference point.

   As both issues for using augmented reality on a mobile device indoors have been addressed, it
can only be concluded that the use of augmented reality is possible inside of a building. It should
be noted, however, that the accuracy of the user’s position inside of a building is in some cases
required to be higher than in the outside world. In a three meter wide corridor, for example, a
location error of three meters could put the user in an office adjacent to the corridor in stead on in
the corridor itself. Augmenting information about the office in stead of the corridor would confuse
the user at this point. Because of this, it is hard to say whether augmented reality would provide
the same user experience indoors than it would outdoors.

1.8                   Future work
Although the research results provided in section 1.6 make it apparent that augmented reality
applications could be used inside of a building, these statements have to be checked in the real
world shed light on the provided user experience. Implementing the proposed methods for deter-
mining a mobile phones location and direction will show the real world feasibility of indoor mobile
augmented reality. A real world implementation could also uncover additional issues that need to
be addressed before the user experience of indoor mobile augmented reality matches that of the
use outdoors.
   Furthermore, a real life implementation of the methods proposed in this paper could be com-
pared to the implementation of Junaio’s indoor methods, which Metaio claims to be absolutely
accurate [Metaio Inc., 2010].

1.9                   References
Apple Inc. iOS Reference Library.,

Apple Inc.         iPhone 4 - Size,        weight,          battery    life,   and    other   specs., 2010b.
Bill, R. et al. Indoor and Outdoor Positioning in Mobile Environmentsa Review and some Inves-
  tigations on WLAN-Positioning. Geographic Information Sciences, 10(2):91–98, 2004.
Feldmann, S. et al. An Indoor Bluetooth-Based Positioning System: Concept, Implementation and
  Experimental Evaluation. In International Conference on Wireless Networks, pages 109–113.
Google Inc. Android Developers Reference.,

Harney, M. Wireless Triangulation Using RSSI Signals. EE Times Education & Training, 2009.
HTC    Corporation.        HTC     -  Products    -    HTC     Magic              -   Specification., 2009.
Kaemarungsi, K. and Krishnamurthy, P. Properties of indoor received signal strength for WLAN
 location fingerprinting. In The 1st IEEE International Conference on Mobile and Ubiquitous
 Systems: Networking and Services. 2004.
King, T. et al. BluePos: Positioning with bluetooth. In WISP 2009 - 6th IEEE International
  Symposium on Intelligent Signal Processing - Proceedings, pages 55–60. 2009.
Lang, P. et al. Inertial tracking for mobile augmented reality. In IEEE Instrumentation and
  Measurement Technology Conference, volume 1, pages 1583–1588. 2002.
Metaio Inc. Junaio Home Page., 2010.
Ochi, H., Tagashira, S. and Fujita, S. A localization scheme for sensor networks based on wireless
  communication with anchor groups. IEICE Transactions on Informations and Systems E Series
  D, 89(5):1614–1621, 2006.

6                                                      Study Tour Pixel 2010 - University of Twente
                             Chapter 1. Toward the indoor use of location-based augmented reality

Prasithsangaree, P., Krishnamurthy, P. and Chrysanthis, P.K. On indoor position location with
  wireless LANs. In The 13th IEEE International Symposium on Personal, Indoor, and Mobile
  Radio Communications (PIMRC 2002), Lisbon, Portugal. 2002.
Rohs, M. Marker-based embodied interaction for handheld augmented reality games. Journal of
  Virtual Reality and Broadcasting, 4(5):0009–6, 2007.
Wagner, D. and Schmalstieg, D. First steps towards handheld augmented reality. 2003.

Toward the indoor use of location-based augmented reality                                      7

To top