Multi-Camera Real Time 3D Reconstruction of Urban Environments The problem Overview of current techniques Research goals The Problem • Given a vehicle with multiple onboard cameras, we want to construct a 3D map of the environment it travels through. • We also want to: ◦ avoid active scanning technologies, for example laser range finding, ◦ avoid dependency on GPS, and ◦ perform processing onboard in real time. Project Goals • Develop a multi-camera system capable of ◦ tracking its own location, ◦ mapping its surroundings, and ◦ reconstructing a 3D model. • Research lighting-invariant reconstruction. • Develop techniques for ◦ estimating a lighting-neutral surface texture. ◦ relighting a previously captured model to match different lighting conditions. 3D Model Reconstruction: Overview See [Pollefeys08], [Cornelis08] for examples. Image stream Ego-motion 3D tracking reconstruction Trajectory 3D model 3D Model Reconstruction: Overview GPS readings INS readings Image stream Ego-motion 3D Fusion tracking reconstruction Trajectory 3D model Above: 4 vehicle mounted cameras. Upper Right: A resulting image set (for one frame of video) Lower Right: An example of a trajectory is shown in green, with feature points in blue. The 3D reconstruction is underlaid. Below: Views of a 3D reconstructed model. Source: [Mordohai07] and http://www.cs.unc.edu/Research/ urbanscape/ 3D Model Reconstruction: Overview GPS readings INS readings Image stream 3D Vehicle tracking Fusion reconstruction Trajectory 3D model Example: Pollefeys et al. system • Video input http://www.youtube.com/watch?v=KSAJkN6QH8Q • Reconstruction 1 http://www.youtube.com/watch?v=3RF26nWzxhc • Reconstruction 2 http://www.youtube.com/watch?v=UdYX9UZDjzY • See [Pollefeys08] Ego-motion Tracking: The Problem • Estimation of the vehicle’s position and orientation • Some visual-only methods have been tried – These tend to accumulate inaccuracies – These often will not recognise a location visited twice • Other researchers fuse GPS and INS data with the output from the vision algorithm + The model is georegistered – System is dependent upon GPS to avoid drift Ego-motion Tracking: Current approaches • SLAM (Simultaneous Localisation And Mapping) ◦ Active area of research, especially in UK • Visual odometry ◦ Designed to solve vehicle tracking problem ◦ Used in Pollefeys et al. system • Both systems have a similar data flow Ego-motion Tracking: Typical Structure Image stream Extract 2D feature tracks Extrapolate 3D position of features and camera Perform global optimisation 3D Model Reconstruction: Overview GPS readings INS readings Image stream 3D Vehicle tracking Fusion reconstruction Trajectory 3D model 3D Reconstruction • Using the trajectory information, we can perform stereo reconstruction • Plane-sweep algorithm is widely used for this ◦ Number of planes can be kept small to improve run time ◦ Urban scenes often contain approximately planar objects ◦ Effective for Lambertian (non-glossy) surfaces (due to photo-consistency constraint) 3D Reconstruction: The Plane-Sweeping Algorithm • Photo-consistency constraint ◦ Assumes that surfaces reflect the same light in all directions – Lambertian surfaces • A series of planes are swept to find matching regions • The model is formed as the set of regions found ◦ Planar regions approximating the real scene The Plane-Sweeping Algorithm: Basic Approach Camera 1 Camera 2 Scene object, e.g. building No match Match found at surface Sweeping through planes Scene object, e.g. building The Plane-Sweeping Algorithm: Basic Approach Camera 1 Camera 2 Scene object, e.g. building Reconstructed model Scene object, e.g. building The Plane-Sweeping Algorithm: Multiple Sweeping Directions Camera 1 Camera 2 Scene object, e.g. building Sweeping through planes Scene object, e.g. building 3D Reconstruction: Multiple Sweeping Directions Camera 1 Camera 2 Scene object, e.g. building Reconstructed model Scene object, e.g. building Research Systems: General Structure GPS readings INS readings Image stream 3D Vehicle tracking Fusion reconstruction Trajectory 3D model Model • 3D mesh of textured polygons ◦ Can be reprojected to provide visually accurate views near to the vehicle trajectory ◦ Disadvantages in current systems: ▪ It has holes, especially on reflective objects ▪ It contains artefacts around moving objects such as cars, people and trees ▪ There is no semantic knowledge about what objects in the model represent – no cartographic information is present Model: Example Source: [Akbarzadeh06] Assessing Model Quality Source: [Pollefeys08] Ground truth model Reconstructed model Accuracy Completeness Above figures: blue represents error < 15cm; red represents error > 60cm 3D Model Reconstruction: Summary • Real-time 3D reconstruction of a moderately sized urban scene can be demonstrated ◦ System split into a pipeline ◦ Multiple processors and graphics cards • Such systems need further improvements such that they are accurate, flexible, reliable and useful Research Goal: Lighting Invariance • Recognising a previously visited location may be difficult if lighting conditions have changed significantly • Techniques exist for removing shadows and other lighting effects ◦ Could compensate for weather changes • Night-time scenes present a greater challenge Source: [Troccoli08] Source: [Troccoli08] Source: [MRL02] : :: : References [Akbarzadeh06] A. Akbarzadeh, J.-M. Frahm, P. Mordohai, B. Clipp, C. Engels, D. Gallup, P. Merrell, M. Phelps, S. N. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewénius, R. Yang, G. Welch, H. Towles, D. Nistér, M. Pollefeys, Towards Urban 3D Reconstruction from Video. 3DPVT 2006: Third International Symposium on 3D Data Processing, Visualization and Transmission; available online at http://www.vis.uky.edu/~dnister/Publications/2006/Urban/ Akbarzadeh_UrbanReconstruction06.pdf [Cornelis08] N. Cornelis, B. Leibe, K. Cornelis, L. Gool, 3D Urban Scene Modeling Integrating Recognition and Reconstruction. International Journal of Computer Vision Vol 78; No 2-3; July 2008; available online at http://www.vision.ee.ethz.ch/~bleibe/papers/ cornelis-3durbanscene-ijcv07final.pdf [MRL02] Y.Y. Chuang, A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D.H. Salesin, Image analogies: Image Colorization. http://mrl.nyu.edu/projects/image- analogies/colorize.html accessed Nov 2008 See also: A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin. Image Analogies. SIGGRAPH 2001 Conference Proceedings. References [Mordohai07] P. Mordohai, J.-M. Frahm, A. Akbarzadeh, B. Clipp, C. Engels, D. Gallup, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewénius, H. Towles, G. Welch, R. Yang, D. Nistér, M. Pollefeys, Real-Time Video-Based Reconstruction of Urban Environments. Proceedings of the 2nd ISPRS International Workshop 3D-ARCH 2007: 3D Virtual Reconstruction and Visualization of Complex Architectures; available online at http://www.cs.unc.edu/~mordohai/public/ UNC-UKY_UrbanReconstuction07.pdf [Pollefeys08] M. Pollefeys, D. Nistér, J. M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S. J. Kim, P. Merrell, Detailed Real-Time Urban 3D Reconstruction from Video. International Journal of Computer Vision Vol 78; No 2-3; July 2008; available online at http://vision.ai.uiuc.edu/~qyang6/publications/detailed_urban3D_IJCV08.pdf [Troccoli08] A. Troccoli, P. Allen, Building Illumination Coherent 3D Models of Large-Scale Outdoor Scenes. International Journal of Computer Vision Vol 78; No 2-3; July 2008; available online at http://www1.cs.columbia.edu/~allen/PAPERS/ijcv08.pdf.