Learning Center
Plans & pricing Sign in
Sign Out



									Next Generation 4D Distributed Modeling and Visualization

 Automated 3D Model
Construction for Urban

                Christian Frueh
                John Flynn
                Avideh Zakhor
           University of California, Berkeley

                     June 13, 2002
Presentation Overview

  Introduction
  Ground based modeling
   - Mesh processing
  Airborne modeling
   - Aerial photos
   - Airborne laser scans
  3D Model Fusion
  Rendering
  Conclusion and Future Work

 Goal: Generate 3D model of a city for virtual
 walk/drive/fly-thrus and simulations
       • Fast
       • Automated
       • Photorealistic

For Fly-Thru:               For Walk/Drive-Thru:
• 3D Model of terrain       • 3D model of street
  and buildings tops          scenery & building
  & sides                     façades
• coarse resolution         • highly detailed

3D Model of terrain            3D model of building
and building tops              façades

Airborne Modeling              Ground Based Modeling
•Laser scans/images            • Laser scans & images
    from plane                 from acquisition vehicle


             Complete 3D City Model
Airborne Modeling

 Goal:   Acquisition of terrain shape and
         top-view building geometry

 Available Data:
 • Aerial Photos
 • Airborne laser scans

 Texture:    from aerial photos
 Geometry:          2 approaches:
             I) stereo matching of photos
             II) airborne laser scans
Airborne Modeling
Approach I : Stereo Matching               (last year)
Stereo photo pairs from city/urban areas, ~ 60% overlap


   Manual:                        Automated:
 • Segmentation             • Camera parameter
                            • Matching,
                            • Distortion reduction,
                            • Model generation
Stereo Matching

 Stereo pair from downtown Berkeley and the estimated
 disparity after removing perspective distortions
Stereo Matching Results

  Downtown Oakland
Airborne Modeling

 Approach II: Airborne Laser Scans

 Scanning city from plane

• Resolution 1 scan point/m2
• Berkeley: 40 Million scan points

                                     point cloud
Airborne Laser Scans
• Re-sampling point cloud
• Sorting into grid
• Filling holes

     Map-like height field
usable for:
• Monte Carlo Localization
• Mesh Generation
Textured Mesh Generation

1. Connecting grid vertices to mesh
2. Applying Q-slim simplification

3. Texture mapping:
   • Semi-automatic
   • Manual selection of few correspondence
     points: 10 mins/entire Berkeley
   • Automated camera pose estimation
   • Automated computation of texture for
Airborne Model

     East Berkeley campus with campanile
Airborne Model

 Downtown Berkeley
Ground Based Modeling

   Goal:      Acquisition of highly detailed
              3D building façade models


                                           2D laser   v


Acquisition vehicle          Scanning setup                    x

Truck with rack:             • vertical 2D laser scanner
• 2 fast 2D laser scanners     for geometry capture
• digital camera             • horizontal scanner for pose
Scan Matching & Initial Path Computation

   Horizontal laser scans:
    • Continuously captured during vehicle motion
    • Overlap
      Relative position estimation by scan-to-scan matching

t = t0          Translation (u,v)     Adding relative
                Rotation              steps (ui, vi, i)
                                                 (ui, vi, i)
                                                  (ui-1, vi-1, i-1)

t = t1
                     (u, v)                (u2, v2, 2)
                                          (u1, v1, 1)

             Scan                         path (xi,yi,i)
                                 3 DOF pose (x, y, yaw)
6 DOF Pose Estimation From Images

 Scan matching cannot estimate vertical
  - Small bumps and rolls
  - Slopes in hill areas
 Full 6 DOF pose of the vehicle is
  important; affects:
  - Future processing of the 3D and intensity data
  - Texture mapping of the resulting 3D models
 Extend initial 3 DOF pose by deriving
  missing 3 DOF (z, pitch, roll) from images
6 DOF Pose Estimation From Images
  Central idea: photo-consistency

  • Each 3D scan point can be projected into
    images using initial 3 DOF pose
  • If pose estimate is correct, point should
    appear the same in all images
  • Use discrepancies in projected position
    of 3D points within multiple images to
    solve for the full pose
6 DOF Pose Estimation – Algorithm

 3DOF of laser as initial estimate
 Project scan points into both images
 If not consistent, use image correlation
  to find correct projection
 Ransac used for robustness
6 DOF Pose Estimation – Results

                with 3 DOF

                with 6 DOF
6 DOF Pose Estimation – Results

  Model      Number of   Completely     Less
 Number       Seams       Removed     noticable   No Change   Worse
    1           48           42           0           4         2
    2           15           8            0           7         0
    3           29           25           2           2         0
    4           24           18           1           5         0
    5            33          24           0          9          0
    6            18          15           0          3          0
    7            34          28           1          3          2
  Total         201         160           4          33         4
Percentage                   80           2          16         2
 Monte Carlo Localization (1)
Previously: Global 3 DOF pose correction using aerial
a) path before MCL correction b) path after MCL correction

    After correction, points fit to edges of aerial image
Monte Carlo Localization (2)
             Extend MCL to work with airborne
             laser data and 6 DOF pose

No perspective shifts of
building tops, no shadow lines
  •   Fewer particles necessary,
      increased computation speed
  •   Significantly higher accuracy
      near high buildings and tree

Use terrain shape to estimate
z coordinate of truck
  •   Correct additional DOF for
      vehicle pose (z, pitch, roll)
  •   Modeling not restricted to
      flat areas
Monte Carlo Localization (3)
Track global 3D position of vehicle to correct relative
6 DOF motion estimates

Resulting corrected path overlaid with airborne laser
height field
Path Segmentation
                          24 mins, 6769 meters

                         vertical scans: 107,082
                           scan points: ~ 15 million

                       Too large to process as
                       one block!

  • Segment path into quasi-
    linear pieces
  • Cut path at curves and
    empty areas
  • Remove redundant
Path Segmentation

Resulting path segments overlaid with
edges of airborne laser height map
Simple Mesh Generation
Simple Mesh Generation


   Point cloud                           Mesh
• Partially captured
  foreground objects
• erroneous scan points
  due to glass reflection

    Side views look “noisy”

    Remove foreground: extract facades
Façade Extraction and Processing (1)
    1. Transform path segment into depth image

     2. Histogram analysis over vertical scans
                              split   main
depth value sn,υ for
                              depth   depth                        split        main
a scan point Pn,υ
                                                                   depth        depth


                                                       local minimum

scanner                                       10


                                                   1      6   11   16      21     26    31    36
                  ground points                                                              depth

Façade Extraction and Processing (2)
3. Separate depth image into 2 layers:

                            =trees, cars etc.

                            =building facades
Façade Extraction and Processing (3)
4. Process background layer:
  • Detect and remove invalid
  scan points
 • Fill areas occluded by foreground
 objects by extending geometry
 from boundaries
     •Horizontal, vertical, planar
     interpolation, RANSAC
 • Apply segmentation
 • Remove isolated segments
 • Fill remaining holes in large
  • Final result: “clean” background
Façade Extraction – Examples (1)

 without processing    with processing
Façade Extraction – Examples (2)

                   without processing

                     with processing
Façade Extraction – Examples (3)


Facade Processing
Foreground Removal
Mesh Generation

Downtown Berkeley
Automatic Texture Mapping (1)
Camera calibrated and synchronized with laser scanners
    Transformation matrix between camera image and
    laser scan vertices can be computed

1. Project geometry into images
2. Mark occluding foreground objects in image
 3. For each background triangle:
   Search pictures in which triangle is not occluded,
   and texture with corresponding picture area
Automatic Texture Mapping (2)

Efficient representation: texture atlas
Copy texture of all triangles into “mosaic” image

   Typical texture reduction: factor 8..12
Automatic Texture Mapping (3)

Large foreground objects: Some of the filled-in
triangles are not visible in any image!
    “texture holes” in the atlas

Texture synthesis: preliminary

•   Mark holes
    corresponding to non-
    textured triangles in
    the atlas
• Search the image for areas matching the hole

• Fill the hole by copying missing pixels from
  these image
Automatic Texture Mapping (4)


Automatic Texture Mapping (5)
Ground Based Modeling - Results

   Façade models of downtown Berkeley
Ground Based Modeling - Results

   Façade models of downtown Berkeley
Model Fusion

        Fusion of ground based and airborne
        model to one single model

    Façade model

                              Airborne model
 Model Fusion:
  1. Registration of models
  2. Combining the registered meshes
Registration of Models
Models are already registered with each
via Monte-Carlo-Localization !

     Which model to use where?
Preparing Ground Based Models
Intersect path segments with each other Remove
degenerated, redundant triangles in overlapping


Preparing Airborne Model

 Ground based model has 5-10 times
 higher resolution

     Remove facades in airborne model
      where ground based geometry is
     Add ground based façades
     Fill remaining gaps with a “blend
      mesh” to hide model transitions
Preparing Airborne Model

   Initial airborne model
Preparing Airborne Model

    Remove facades where ground
    based geometry is available
Combining Models

   Add ground based façade models
Combining Models

   Fill remaining gaps with a “blend
   mesh” to hide model transitions
Model Fusion - Results
Ground based models:
• Up to 270,000 triangles, 20
  MB texture per path
• 4.28 million triangles, 348
  MB texture for 4 downtown
• Difficult to render interactively!

    Subdivide model and create
     multiple level-of-details (LOD)
    Generate scene graph,
     decide which LOD to render when
Multiple LODs for façade meshes

  Highest LOD

•Qslim mesh

  Lower LOD
• Geometry: 10%
• Texture: 25%
of original mesh
Façade Model Subdivision for Rendering

Subdivide 2 highest LODs of façade meshes along
cut planes

                    Sub-      Submesh   LOD 0
                    scene               LOD 1
         Path                           LOD 0
         segment                        LOD 1
Global             LOD 2                LOD 0
scene                                   LOD 1
Interactive Rendering

  Downtown blocks: Interactive rendering with
  web-based browser!
Future Work

 Resolution enhancement and post-
  processing of LIDAR data
 Devise new data acquisition system and
  algorithms to capture
  - both sides of street simultaneously
  - texture for upper parts of tall buildings
 Include foreground objects in model
 Add temporal component to dynamically
  update models
 Compact representation
 Interactive rendering

To top