Augmented-Realy by jizhen1947



     Ping Gai
     HFE 760
 Augmented- Reality Definition
 Augmented Reality vs. Virtual Reality
 Visual Display Systems for AR
 Video Keying and Image Registration
 System Design Issues
 Augmented Reality Application
    Augmented Reality Definition
 Augmented Reality is a growing area in
  virtual reality area.
 An Augmented Reality system generates a
  composite view for the user. It’s a
  combination of the real scene viewed by the
  user and a virtual scene generated by the
  computer that augments the scene generated
  by the computer that augmented the scene
  with additional information.
    Augmented Reality Definition
   Typically, the real-world visual scene in an AR
    display is captured by video or directly viewed.
   Most current AR displays are designed using see-
    through HMDs which allow the observer to view
    the real world directly with the naked eye.
   If video is used to capture the real world, one may
    use either an opaque HMD or screen-based system
    to view the scene.
                     AR vs. VR
   Virtual Reality: a computer generated, interactive,
    three-dimensional environment in which a person
    is immersed.(Aukstakanis and Blatner, 1992)
    – Virtual Environment is a computer generated three
      dimensional scene which requires high performance
      computer graphics to provide an adequate level of
    – The virtual world is interactive. A user requires real-
      time response from the system to be able to interact
      with it in an effective manner.
    – The user is immersed in this virtual environment.
                AR vs . VR
   VR: the user is completely immersed in an
    artificial world and becomes divorced from
    the real environment. The generated world
    consists entirely of computer graphics.
               AR vs. AR
 VR strives for a totally immersive
  environment. The visual, and in some
  systems aural and sense are under control of
  the system.
 In contrast, an AR system is augmenting the
  real world sense of presence in that world.
  The virtual images are merged with the real
  view to create the augmented display.
                 AR vs. VR
   For some applications , it may be desirable
    to use as much as possible real world in the
    scene rather creating a new scene using
    computer imagery. For example, in medical
    applications, the physician must view the
    patient to perform surgery, in telerobotics
    the operator must view the remote scene in
    order to perform tasks.
               AR vs. VR
 A main motivation for the use of AR relates
  to the computational resources necessary to
  generate and update computer-generated
  scene. In VR, The more complex the scene,
  the more computational resource needed to
  render the scene.
 AR can maintain the high-level of detail and
  realistic shading that one finds in the real
               AR vs. VR
 NO simulator sickness. Vertigo, dizziness
  introduced by sensory mismatch within
  display environment can be a problem when
  one uses an HMD to view a virtual world.
 If the task is to show an annotation to the
  real world.
    Visual Display System for AR
 Hardware for display visual images
 A position and orientation sensing system
 Hardware for combining the computer
  graphics and video images into one signal
 The associated system software
    Visual Display System for AR
   There are two main ways in which the real world
    and the computer generated imagery may be
    combined to form an augmented scene.
    – Direct viewing of the real world with overlaid computer
      generated imagery as an enhancement.In this case, the
      the real world and the CG images are combined
    – Combining the camera-captured video of the real world
      with CG imagery viewed using either an opaque HMD,
      or a screen-based display system.
    Visual Display System for AR
   Two basic types of AR system
    – Opaque HMD or screen-based AR.
       These systems can be used to view local or remote video views
         of real world scenes, combined with overlaid CG.The viewing
         of a remote scene is an integral component of telepresence
    – Transparent HMD AR.
       This system allows the observer to view the real world directly
         using half-silvered mirrors with CG electronically composited
         into the image. An advantage id that the real-world can be
         directly viewed and manipulated.
Visual Display System for AR
                 Video Keying
 Relevant when an opaque HMD with video
  input is used to create an AR scene. Video
  and synthetic image are mixed using a video
  keyer to form an integrated scene.
 Video Keying is a process that is widely
  used in television, film production and CG.
    (weather report)
               Video Keying
   When using video keying to design AR
    scenes, one signal contains the foreground
    image and the other one contains the
    background image. The keyer combines the
    two signal to produce a combined video
    which is then sent to the display device.
                Video Keying
   Keying can be done using composite or
    component video signals.
    – A composite video signal contains information
      about color, luminance, and synchronization,
      thus combining three piece of information into
      one signal.
    – With component video, luminance
      synchronization are combined, but chroma
      information is delivered separately.
                Video Keying
   Chroma keying involves specifying a desired
    foreground key color. Foreground areas containing
    the keying color are then electronically replaced
    with the background image. This results in the
    background image being replaced with the fore
    ground image in areas where the background
    image contains chroma color.
   Blue is typically used for chroma keying
    (Chromakey blue) rarely shows up in human skin
             Video Keying
 If a video image of the real world is chosen
  as the foreground image, parts of the scene
  that should show the computer-generated
  world are rendered blue.
 In contrast, if video of the real world is
  chosen as the background image, the
  computer generated environment will be
  located in the foreground.
Video Keying
                 Video Keying
   A luminance keyer works in a similar manner to a
    chroma keyer, however, a luminance keyer
    combines the background image wherever the
    luminance values are below a certain threshold.
   Luminance and chroma keyers both accomplish
    the same function but usa of a chroma keyer can
    result in a sharper key and has greater flexibility,
    whereas a luminance keyer is typically lower
    resolution and had less flexibility.
   Figure is a schema of the z-key method. The z-
    key method requires images with both depth
    information (depth map) as inputs. The z-key
    switch compares depth information of two images
    for each pixel, and connects output to the image
    which is the nearer one to the camera. The result
    of this is that real and virtual objects can occlude
    each other correctly. This kind of merging is
    impossible by the chroma-key method, even if it is
    accompanied with some other positioning devices
    such as magnetic or acoustic sensor, since these
    devices provide only a gross measurement of
              Image Registration
   It’s required that the computer generated images
    accurately register with the surroundings in the
    real world. In certain applications, image
    registration is crucial.
   In terms of developing scenes for AR displays, the
    problem of image registration, or positioning of
    the synthetic objects within the scene in relation to
    real objects, is both a difficult and important
    technical problem to solve.
            Image Registration
   With applications that require close
    registration, accurate depth information has
    to be retrieved from the real world in order
    to carry out the calibration of the real and
    synthetic environments. Without an
    accurate knowledge of the geometry of the
    real world and computer-generated scene,
    exact registration is not possible.
           System Design Issues
   Frame rate, update rate, system delays, and the
    range and sensitivity of the tracking sensors.
   Frame rate is a hardware-controlled variable
    determining the number of images presented to the
    eye per second. AR displays which show stereo
    images alternatively to the left and right eye
    typically use a scan rate doubler to transmit 120
    frames per second so that each eye has an effective
    frame rate of 60 Hz.
            System Design Issues
   Update rate of the display is the rate at which new
    images are presented to the viewer.
   With a low update rate, if the user using an AR
    display moves his head, the real and computer-
    generated images will no longer be registered until
    the next update. Small errors in registration are
    easily detectable by the visual system.
   What limits the update rate is the relationship
    between the complexity of the scene and the
    computational power of the computer system used
    to generate the scene. This relationship is esp.
    important for computationally intensive
    applications such as medical imaging.
             System Design Issues
   The lag in image generation and tracking is
    noticeable in all HMDs but is dramatically
    accentuated with see-through HMDs. This is an
    crucial problem if exact image registration is
   There are two types of system delays which will
    affect performance in AR: computational and
    sensor delays.
    – As the complexity of the CG image increases, the computational
      delay is a major factor determining the update of a display.
    – In addition, sensor delay, the time requires updating the display, is
      an important variable in determining performance in augmented
    – Many HMD-based systems have combined latencies over 100ms,
      which become very noticeable.
           System Design Issues
   Sensor sensitivity
    – The head-tracking requirements for AR displays.
    – A tracker must be accurate to a small fraction of a
      degree in orientation and a few millimeters in position.
    – Errors in head orientation(pitch, roll, yaw) affect image
      registration more so than error in position(x, y, z),
      leading to the more stringent requirements for head-
      orientation tracking.
    – Positional tracking errors of no more than 1 to 2 mm
      are maximum for AR system.
           System Design Issues
   In addition to visual factors, cognitive factors
    should be considered in the design as well.
   Users of systems form mental models of the
    system they interact with and the mental model
    they form influence their performance.
   With AR displays the designer must take into
    account two mental models of the environment,
    the mental model of the synthetic imagery and of
    the real image.
   The challenge will be to integrate the two stimuli
    in such a way that a single mental model will be
    formed of the augmented scene.
        System Design Issues

               Integrated Mental Model

  Mental Model of               Mental Model of
  synthetic                     real envoronment

Virtual world stimuli          Real world stimuli
Auditory, haptic,              Auditory, haptic,
visual                         visual
 Augmented-Reality Application
 Medical
 Entertainment
 Military Training
 Engineering Design
 Robotics and Telerobotics
 Manufacture, Maintenance and Repair
 Consumer Design
    Augmented-Reality Application
   Medical
     Most of the medical applications deal with image guided surgery. Pre-
      operative imaging studies, such as CT or MRI scans, of the patient
      provide the surgeon with the necessary view of the internal
      anatomy. From these images the surgery is planned. Visualization
      of the path through the anatomy to the affected area where, for
      example, a tumor must be removed is done by first creating a 3D
      model from the multiple views and slices in the preoperative study.
      AR can be applied so that the surgical team can see the CT or MRI
      data correctly registered on the patient in the operation theater
      while the procedure is progressing. Being able to correctly register
      the images at the point will enhance the performance of the
      surgical team and eliminate the need for the painful and
      cumbersome stereotactic frames currently used for registration.
Augmented-Reality Application
Augmented-Reality Application
   Entertainment
    –   Weather report
    –   Virtual studio
    –   Movie special effect
    –   Advertisement
 Augmented-Reality Application
   Military Training
The military has been using display in cockpits that
  present information to the pilot on the windshield
  of the cockpit or the visor of their flight helmet.
  This is a form fo AR display.
Augmented-Reality Application
   Engineering Design           The scenario for this application consists of an office
                                 manager who is working with an interior designer on the
    Distributed Coollaberation   layout of a room. The office manager intends to order
                                 furniture for the room. On a computer monitor the pair see a
                                 picture of the room from the viewpoint of the camera. By
    Product visualizatoin        interacting with various manufacturers over a network, they
                                 select furniture by querying databases using a graphical
                                 paradigm. The system provides descriptions and pictures of
                                 furniture that is available from the various manufactures who
                                 have made models available in their databases. Pieces or
                                 groups of furniture that meet certain requirements such as
                                 colour, manufacturer, or price may be requested. The users
                                 choose pieces from this "electronic catalogue" and 3D
                                 renderings of this furniture appear on the monitor along with
                                 the view of the room. The furniture is positioned using a 3D
                                 mouse. Furniture can be deleted, added, and rearranged
                                 until the users are satisfied with the result; they view these
                                 pieces on the monitor as they would appear in the actual
                                 room. As they move the camera they can see the furnished

                                 room from different points of view.
Augmented-Reality Application
   Robotics and Telerobotics
    Augmented-Reality Application
   Manufacturing,
    Maintenance, and
   One application area that is currently being explored involves
    mechanical maintenance and repair. In this scenario a
    mechanic is assisted by an AR system while examining and
    repairing a complex engine. The system may present a variety
    of information to the mechanic. Annotations may identify the
    name of parts, describe their function, or present other
    important information like maintenance or manufacturing
    records. AR may lead the mechanic through a specific task by
    highlighting parts that must be sequentially removed and
    showing the path of extraction. The system may also provide
    safety information. Parts that are hot or electrified can be
    highlighted to constantly remind the mechanic of the danger of
    touching them. The mechanic may also be assisted by a remote
    expert who can control what information is displayed on the

    mechanic's AR system.
Augmented-Reality Application
   Consumer Design
    House Design
    Fashion, beauty industry
 Virtual Environments and Advanced
  Interface Design, edited by Woodrow
  Barfield, Thomas A.Furness III
   Augmented Reality Sites - North America
   MIT
   Image Guided Surgery home page
   Intelligent Room project
   J P Mellor's home page
   Media Lab Wearable Computer page
   CMU
   Z-Key project
   Magic Eye project
   Columbia University
   Virtual Worlds research
   Architectural Anatomy
   University of North Carolina - Chapel Hill
   Ultrasound Visualization Research
   Hybrid Tracking Research
   Latency in Augmented Reality
   Ronald Azuma's Augmented Reality page
   Telepresence Research Group
   Rich Holloway's Home Page
   USC Computer Graphics and Immersive Technologies Laboratory
   University of Washington Human Interface Technology Lab (HITL)
   Colorado School of Mines
   Hazardous waste management Bozidar Stojadinovi, Virtual Reality Lab, University of Michigan
   Augmented Reality work at the University of Toronto
   Argonne National Labs - Michael E. Jebb's Augmented Reality page
   NIST description of the Boeing project
   Colorado State Univ. - Michael L. Croswell's Augmented Reality page
   Ross Whitaker's Augmented Reality page
   Mihran Tuceryan's Augmented Reality page
   The WorldBoard Project
   Vision-based Augmented Reality for Guiding Assembly Rajeev Sharma, Jose Molineros, University of Illinois at Urbana-Champaign
   Alexander Chislenko's Intelligent Information Filters and Enhanced Reality Page
   Microvision's Virtual Retinal Display

To top