VLunch-Nov-2008

Document Sample
VLunch-Nov-2008 Powered By Docstoc
					Multi-Camera Real Time
 3D Reconstruction of
 Urban Environments

         The problem
 Overview of current techniques
        Research goals
              The Problem
• Given a vehicle with multiple onboard
  cameras, we want to construct a 3D map
  of the environment it travels through.
• We also want to:
  ◦ avoid active scanning technologies, for
    example laser range finding,
  ◦ avoid dependency on GPS, and
  ◦ perform processing onboard in real time.
              Project Goals
• Develop a multi-camera system capable of
  ◦ tracking its own location,
  ◦ mapping its surroundings, and
  ◦ reconstructing a 3D model.
• Research lighting-invariant reconstruction.
• Develop techniques for
  ◦ estimating a lighting-neutral surface texture.
  ◦ relighting a previously captured model to
    match different lighting conditions.
             3D Model Reconstruction:
                   Overview
                 See [Pollefeys08], [Cornelis08] for examples.
 Image
 stream




Ego-motion                                                 3D
  tracking                                            reconstruction



Trajectory                                               3D model
          3D Model Reconstruction:
                Overview
             GPS readings    INS readings
 Image
 stream




Ego-motion                                       3D
                        Fusion
  tracking                                  reconstruction



                      Trajectory              3D model
Above: 4 vehicle mounted cameras.
Upper Right: A resulting image set (for one frame of
video)
Lower Right: An example of a trajectory is shown in
green, with feature points in blue. The 3D
reconstruction is underlaid.
Below: Views of a 3D reconstructed model.




                                                       Source: [Mordohai07] and
                                                       http://www.cs.unc.edu/Research/
                                                       urbanscape/
             3D Model Reconstruction:
                   Overview
                   GPS readings    INS readings
   Image
   stream




                                                       3D
Vehicle tracking              Fusion
                                                  reconstruction



                            Trajectory              3D model
               Example:
        Pollefeys et al. system
• Video input
 http://www.youtube.com/watch?v=KSAJkN6QH8Q


• Reconstruction 1
 http://www.youtube.com/watch?v=3RF26nWzxhc


• Reconstruction 2
 http://www.youtube.com/watch?v=UdYX9UZDjzY


• See [Pollefeys08]
          Ego-motion Tracking:
              The Problem
• Estimation of the vehicle’s position and
  orientation
• Some visual-only methods have been tried
  – These tend to accumulate inaccuracies
  – These often will not recognise a location visited twice
• Other researchers fuse GPS and INS data with
  the output from the vision algorithm
  + The model is georegistered
  – System is dependent upon GPS to avoid drift
            Ego-motion Tracking:
             Current approaches
• SLAM (Simultaneous Localisation And Mapping)
   ◦ Active area of research, especially in UK
• Visual odometry
   ◦ Designed to solve vehicle tracking problem
   ◦ Used in Pollefeys et al. system
• Both systems have a similar data flow
         Ego-motion Tracking:
           Typical Structure
Image
stream


             Extract 2D
           feature tracks

                            Extrapolate 3D
                              position of
                             features and
                                camera

                                             Perform global
                                              optimisation
          3D Model Reconstruction:
                Overview
                   GPS readings    INS readings
   Image
   stream




                                                       3D
Vehicle tracking              Fusion
                                                  reconstruction



                            Trajectory              3D model
          3D Reconstruction
• Using the trajectory information, we can
  perform stereo reconstruction
• Plane-sweep algorithm is widely used for
  this
  ◦ Number of planes can be kept small to
    improve run time
  ◦ Urban scenes often contain approximately
    planar objects
  ◦ Effective for Lambertian (non-glossy) surfaces
    (due to photo-consistency constraint)
         3D Reconstruction:
   The Plane-Sweeping Algorithm
• Photo-consistency constraint
  ◦ Assumes that surfaces reflect the same light in
    all directions – Lambertian surfaces
• A series of planes are swept to find
  matching regions
• The model is formed as the set of regions
  found
  ◦ Planar regions approximating the real scene
The Plane-Sweeping Algorithm:
        Basic Approach
   Camera 1                           Camera 2



                                                 Scene object,
                                                 e.g. building
              No match

                      Match found at surface
Sweeping
through
planes

              Scene object,
              e.g. building
The Plane-Sweeping Algorithm:
        Basic Approach
  Camera 1                          Camera 2



                                               Scene object,
                                               e.g. building




                        Reconstructed
                        model


             Scene object,
             e.g. building
  The Plane-Sweeping Algorithm:
   Multiple Sweeping Directions
       Camera 1                   Camera 2



                                             Scene object,
                                             e.g. building




Sweeping
through
planes
                  Scene object,
                  e.g. building
    3D Reconstruction:
Multiple Sweeping Directions
Camera 1                          Camera 2



                                             Scene object,
                                             e.g. building




                      Reconstructed
                      model


           Scene object,
           e.g. building
                   Research Systems:
                   General Structure
                   GPS readings    INS readings
   Image
   stream




                                                       3D
Vehicle tracking              Fusion
                                                  reconstruction



                            Trajectory              3D model
                      Model
• 3D mesh of textured polygons
  ◦ Can be reprojected to provide visually
    accurate views near to the vehicle trajectory
  ◦ Disadvantages in current systems:
     ▪ It has holes, especially on reflective objects
     ▪ It contains artefacts around moving objects such as
       cars, people and trees
     ▪ There is no semantic knowledge about what
       objects in the model represent – no cartographic
       information is present
                         Model: Example
Source: [Akbarzadeh06]
                Assessing Model Quality
Source: [Pollefeys08]




Ground truth model                          Reconstructed model




Accuracy                                    Completeness
      Above figures: blue represents error < 15cm; red represents error > 60cm
      3D Model Reconstruction:
            Summary
• Real-time 3D reconstruction of a
  moderately sized urban scene can be
  demonstrated
  ◦ System split into a pipeline
  ◦ Multiple processors and graphics cards
• Such systems need further improvements
  such that they are accurate, flexible,
  reliable and useful
             Research Goal:
           Lighting Invariance
• Recognising a previously visited location
  may be difficult if lighting conditions have
  changed significantly
• Techniques exist for removing shadows
  and other lighting effects
  ◦ Could compensate for weather changes
• Night-time scenes present a greater
  challenge
Source:
[Troccoli08]
Source:
[Troccoli08]
Source:
[MRL02]




          :   ::




          :
                               References
[Akbarzadeh06] A. Akbarzadeh, J.-M. Frahm, P. Mordohai, B. Clipp, C. Engels, D. Gallup, P.
    Merrell, M. Phelps, S. N. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewénius, R. Yang, G.
    Welch, H. Towles, D. Nistér, M. Pollefeys, Towards Urban 3D Reconstruction from
    Video. 3DPVT 2006: Third International Symposium on 3D Data Processing,
    Visualization and Transmission; available online at
    http://www.vis.uky.edu/~dnister/Publications/2006/Urban/
    Akbarzadeh_UrbanReconstruction06.pdf

[Cornelis08] N. Cornelis, B. Leibe, K. Cornelis, L. Gool, 3D Urban Scene Modeling Integrating
    Recognition and Reconstruction. International Journal of Computer Vision Vol 78; No
    2-3; July 2008; available online at http://www.vision.ee.ethz.ch/~bleibe/papers/
    cornelis-3durbanscene-ijcv07final.pdf

[MRL02] Y.Y. Chuang, A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D.H. Salesin, Image
   analogies: Image Colorization. http://mrl.nyu.edu/projects/image-
   analogies/colorize.html accessed Nov 2008

See also: A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin. Image Analogies.
    SIGGRAPH 2001 Conference Proceedings.
                           References
[Mordohai07] P. Mordohai, J.-M. Frahm, A. Akbarzadeh, B. Clipp, C. Engels, D.
   Gallup, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H.
   Stewénius, H. Towles, G. Welch, R. Yang, D. Nistér, M. Pollefeys, Real-Time
   Video-Based Reconstruction of Urban Environments. Proceedings of the 2nd
   ISPRS International Workshop 3D-ARCH 2007: 3D Virtual Reconstruction
   and Visualization of Complex Architectures; available online at
   http://www.cs.unc.edu/~mordohai/public/
   UNC-UKY_UrbanReconstuction07.pdf

[Pollefeys08] M. Pollefeys, D. Nistér, J. M. Frahm, A. Akbarzadeh, P. Mordohai, B.
    Clipp, C. Engels, D. Gallup, S. J. Kim, P. Merrell, Detailed Real-Time Urban 3D
    Reconstruction from Video. International Journal of Computer Vision Vol 78;
    No 2-3; July 2008; available online at
    http://vision.ai.uiuc.edu/~qyang6/publications/detailed_urban3D_IJCV08.pdf

[Troccoli08] A. Troccoli, P. Allen, Building Illumination Coherent 3D Models of
    Large-Scale Outdoor Scenes. International Journal of Computer Vision Vol
    78; No 2-3; July 2008; available online at
    http://www1.cs.columbia.edu/~allen/PAPERS/ijcv08.pdf.

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:32
posted:4/11/2011
language:English
pages:30