Docstoc

PowerPoint Presentation - Google Code

Document Sample
PowerPoint Presentation - Google Code Powered By Docstoc
					New Image Rectification
Schemes for 3D Vision Based
on Sequential Virtual Rotation


Jin Zhou
June 16th, 2009
Dissertation Defense
Outline

   Introduction
   Rectification based on Virtual Sequential Rotation
   Image Rectification for Stereoscopic Visualization
   Camera Calibration
   Stereoscopic View Synthesis from Monocular
    Endoscopic Sequences
   Rapid 3D Modeling from Single Images
   Robot Vision
   Conclusions and Future Work
The Geometry of 3D to 2D




       Images are 2D projections of the 3D world
3D Vision – The Problem
   How do we extract 3D information from 2D images?




                                                             ?
                                         3D of the objects




                                                             ?
             3D of the cameras
                                 ?         ?            ?
3D Vision – Applications

 Augmented Reality
 Scene Modeling
 Virtual Touring
 3D Imaging
 Robots
3D Vision – A Human Perspective

 Size
 Linear Perspective
 Object Connections
 Stereo
 Motion
 Shading
 Texture
3D Vision – Computational Approaches
   Different approaches use different cues
   Different problems requires different
    approaches.
   Structure from Motion (SfM)
     Rely   on point correspondences
   Single View Based Modeling (SVBM)
     Rely   on knowledge of the scene
   Camera calibration
     All approaches requires the images are calibarated
      first (either manual or automatic)
3D Vision – Practical Challenges

 Camera information is unavailable
 Point correspondences is not reliable and
  time-consuming
 Image resolution is limited
 Degeneracy
3D Vision – Limitations of Current Approaches

 Distortion
 Degeneracy
     Dueto high degree of freedom of geometric
      models
   Lack of geometric meaning
     Most approaches are purely based on
      algebraic derivations or imaginery objects.
   Not accurate or not convinient (SVBM).
3D Vision – Our Contributions
   Novel image rectification schemes are proposed
    based on sequential virtual rotation
   Novel approaches are proposed for the following
    problems
     Image Rectification for Stereoscopic visualization
     Camera Calibration
     Stereoscopic View Synthesis from Monocular
      Endoscopic sequences
     Rapid Cones and Cylinders Modeling
     Monocular Vision Guided Mobile Robot Navigation
3D Vision – Results of Our Approaches

 No affine/projective distortion
 Can handle degeneracy
 Intuitive geometric meanings
     Lead   to insights of particular problems
   Accurate and fast
Publications
What is Image Rectification?




  Image rectification is a process to transform the original
  images to new images which have desired properties.
General Image Transformations




                               H?

                            x'  H x

Image transformation can be defined by a 3x3 matrix H, which is
called Homography.
Image Rectification based on Virtual
Rotation
   Homography of camera rotation/zooming
                P  KR[ I | C ], P '  K ' R '[ I | C ]

    x '  P ' X  K ' R '[ I | C ] X  K ' R '( KR) 1 KR[ I | C ] X  ( K ' R ')( KR) 1 x

                               H   ( K ' R ')( KR)1
   If we normalize the coordinates
                       x  K 1x
                       ˆ                   x'  K '1 x'
                                           ˆ

   Assume R = I
                                 ˆ
                                 H  R '              Camera orientation is
                                                       determined at the same time
Advantages of the New Rectification
Schemes
   Intuitive geometric meaning
   Robust
     Rotation parameters can be computed by various
      basic image features, such as points, lines and
      circles.
   Can be used for camera calibration.
   Can be used for 3D information extraction.
   Lead to non-distorted results
     Reason:   Rotation do not introduce affine/projective
      distortion
Rotations are Decomposed as Euler Angles


                      1    0        0 
            Rx ( )  0 cos( )  sin( ) 
                                          
                      0 sin( ) cos( ) 
                                          

                       cos( ) 0 sin( ) 
            Ry ( )   0
                                 1   0    
                        sin( ) 0 cos( ) 
                                           


                      cos( )  sin( ) 0 
            Rz ( )   sin( ) cos( ) 0 
                                          
                       0
                                  0     1
Rotation Parameters Can Be Estimated by
Basic Image Features

   Each rotation has only one degree of freedom and thus
    only needs one constraint.
   Example: transforming a point on to y axis




                           Rz p  (0, d , c)T
                              ˆ                                          / 2
                                                                   
                                                            [ ]        / 2
                          a cos( )  b sin( )  0                
                                                                         otherwize
                                                      Normalize
                              arc tan(a, b)                       [arc tan(a, b)]
             Ambiguity!
Image Rectification for Stereoscopic
Visualization




          The Principle of Stereoscopic (3D) Visualization
Motivation
   Stereo content is scarce
   Stereo cameras/camcorders are expensive
   Common users seldom use stereo
    cameras/camcorders
   We want to generate stereo content from
    images/videos taken by common cameras
The Problem




   Given two arbitrary images, rectify them so that the
   results look like a stereo pair.
Our Approach – Rectification based on
Virtual Rotation




    We can “rotate” camera to standard stereo setup.

              H i  KR( Ki Ri )1
Calibrated Case
Constraints of the stereo camera pair:
1. The two cameras have the same intrinsic parameters (K) and
   orientation (R)
2. The camera’s optical axis is perpendicular to the baseline (C1 – C2)
    i.e. the camera’s x axis has the same direction with the baseline
               UnKnown                     R  (r , s, t )T

                                           r  (C1  C2 ) / || C1  C2 ||
                              1
        H i  KR( Ki Ri )                  t  (r  p) / || r  p ||
                                           s  r t
                K1   Known                                       Any vector
Uncalibrated Case

    For the uncalibrated case, all K, R and C are unknown.
     We can only start from the fundamental matrix and point
     correspondences.
    Estimate H2 (homography for the second image)
                         H i  KR( Ki Ri )1
                                         ?
             f    0    px         px  w / 2
                                                       R2  I
    K  K2   0
                  f   p y  with  p y  h / 2
                            
             0                   
                  0    1          f  ( w  h) / 2
Determine R based on Sequential Virtual
Rotation
                                                  
Constraints:          Re2  (1, 0, 0)T
                       ˆ                   e2  K 2 1e2
                                           ˆ



First rotate around z axis so that the point is transformed to x axis (i.e. y = 0)

         Rz (a, b, c)T  (d , 0, c)T           z  [arctan(b, a)]

Rotate around y axis so that the point is transformed to infinity.

     Ry (d ,0, c)T  (e,0,0)T  (1,0,0)T         y  [arctan(c, d )]


                             R  Ry Rz
Estimate H1
 H1  KR( K1 R1 )1  H 2 K2 R2 ( K1 R1 )1  H 2 M
 M  ( I  e2 vT ) M 0        F  [e2 ] M
 H1  H 2 M
  H 2 ( I  e2 vT ) M 0
  H 2 ( M 0  e2 vT M 0 )
  H 2 M 0  H 2 e2 vT H 2 1 H 2 M 0                 H 2 e2  KRK 2 1e2  K [r , s, t ]T r
  ( I  H 2 e2 v1T ) H 2 M 0 (v1T =vT H 2 1 )        K (1, 0, 0)T  (k , 0, 0)T



                 H1  ( I  H 2 e2 vT ) H 2 M 0  H A H 2 M 0
                       a b c 
                 H A   0 1 0
                             
                       0 0 1
                             
Estimate H1

   Determine a, b and c
   Property of standard stereo setup:
     For  two points with the same depth, their projection
       on different images should have the same distance
       (Points with the same depth should have the same
       disparity).
   Approach
     Group points by similar disparities
     Then compute a, b by minimizing

                                                                            
                                                                                                                                                2
                                                                                            ( a ( xi  xi )  b( yi  y j )  ( xi ' x j '))
                                                                                                  ˆ ˆ            ˆ ˆ            ˆ ˆ
                                                                          2
                    (|| H1 x i  H1 x j ||  | H 2 x i ' H 2 x j ' ||)
     p i , j  Ap                                                             p i , j Ap


                                         ax0  by0  c  x0 '
                                          ˆ     ˆ        ˆ
Results
                                     Shear distortion




  Original Pair   Hartley’s Method     Our Method
Results
                                    Shrink horizontally




 Original Pair   Hartley’s Method    Our Method
Camera Auto-Calibration from the
Fundamental Matrix
Traditional Approaches:

Kruppa Equations                
                        [e2 ] w2 [e2 ]  Fw1 F T

                       dual image of the absolute conic       wi  K i K iT

Huang-Faugeras constraints
                   1
                     trace 2 ( EE T )  trace( EE T ) 2  0
                   2
                   E  K 2 FK1
                         T




Cons:
Complex and hard to understand!
Derivation for degenerate cases are purely algebraic.
Our Approach
   We transform the original pair to a standard stereo pair
   through sequential virtual rotation and zooming

     7 DOFs
                        
                  F  K 2 T R2 z R2 y F S Rx R1 y R1z K11
                             T    T


7 Parameters       ( f1 , 1z , 1 y ,  x , f 2 ,  2 z ,  2 y )

                                                                       Rz (1z )e1  (d1 , 0, c1 )T




                                   0 0 0 
        F
                             F S  0 0 1
                                         
                                   0 1 0 
                                         
Decomposition Illustration
                     
 Fz  R2 z FR1z1  K2 T R2 y F S Rx R1y K11
                          T




      c2     0  0  a b            a  c1 0 0 
Fz    0      1 0  c d             c  0 1 0 
                                              
      0
             0 d 2   a b
                                   a   0 0 d1 
                                                 


                                             f1d 2
      a   f1 f 2 sin( x ),     c                 cos( x ),
                                           cos( 2 y )
              f 2 d1                           d1d 2
     b               cos( x ), d                          sin( x )
            cos(1 y )                  cos(1 y ) cos( 2 y )
Degeneracy Analysis
Degeneracy Analysis
Results of Monte Carlo Simulation
Stereoscopic View Synthesis From
Monocular Endoscopic Videos




   3D imaging helps to enable faster and safer surgical operations

  Two view image rectification can not be applied to the new problem

   Challenges: 1. Image quality is poor 2. Degeneracy
The Framework

                We proved:
                1. Affine 3D reconstruction is
                   sufficient.
                2. Linear interpolation in
                   normalized disparity field is
                   equal to linear interpolation
                   in 3D space.
 Strategy for Solving Degeneracy
We assume the initial two frames have same orientation (i.e. they are rectified)
The assumption makes the DOF of the fundamental matrix from 8 to 2!



                                                               No Assumption

                                                               Degeneracy!



                                                               Assume the two
                                                               frame are
                                                               rectified
Interpolation
                a) Shows the disparities
                based on the SfM results


                b) We do Delaunay
                triangulation and interpolate
                each triangle


                c) We pick a set of grid points
                from b) and do bilinear
                interpolation

                d) We fill holes using Laplacian
                interpolation and do smoothing.
Results of Synthetic Data

                            Disparity image after
                            triangulation


                            Final disparity image



                            Ground truth




                            Stereo images
Results of Real Data
Rapid Cylinders and Cones Modeling from A
Single Image
Overview of Our Approach
   Goal: Rapid + Accurate
   Camera Calibration (Orientation Estimation)
     Vertical lines
     Vanishing line   of horizontal plane
     A Cone
   Modeling from Image
     Cones  (two points / four points)
     Cylinders (two points / four points)
The Coordinate Systems
 Observation: Most objects are standing on the ground




      (X,Y,Z,O) -- World Coordinate System
      Y is perpendicular to the ground

      (x,y,z,O) -- Camera Coordinate System
      Camera center is at the origin
  Orientation Estimation from Vertical Lines




                                  R
       (a, b, c)T                                         (0,1,0)T

First rotate around z axis so that the point is transformed to y axis (i.e. x = 0)

Rotate around x axis so that the point is transformed to infinity.
Orientation Estimation from a Cone



                RzRxRz


R   Edges are symmetric to y axis   Rx   Cross section is a circle




                Rx (π/2)
Illustration
Metric Rectification of the Ground Plane
                                                  1 0 0 
                              H g  Rx ( / 2)  0 0 1 
                                                        
                                                  0 1 0
                                                        




          Original           Rectified




         Original            Rectified
Modeling Cones
Cone Parameters:
        Μ0  (c  (0, yc ,1)T , r , yv )
             Μ  ( R, d , M0 )

2D Control Points for Cones Modeling:




Standard view      Cones on Ground         General Cones
Cones Modeling from the Standard View

   Rectify the standard view to the ground plane
    view by Rx(π/2)
     The cross section is rectified to a circle
     The edge line is tangent to the circle
     The center and radius can be determined for any
      point on the edge line
Modeling Cones Standing on the Ground




              Ry




Four points   Standard   Ground view   3D Mesh
              view
Modeling General Cones



                 R




   Five points       Standard   3D Mesh
                     View
Modeling Cylinders
Cylinder Parameters:

         L0  (c  (0, yb ,1)T , r , yt )
              L  ( R, d , L0 )
2D Control Points for Cones Modeling:




    Standard View       Cylinders on Ground   General Cylinders
Modeling Cylinders Standing on the Ground




             Ry




two points   Standard   Ground view   3D Mesh
             view
Modeling General Cylinders



                R




  Four points       Standard   3D Mesh
                    View
Screenshots of Real Data Experiments
Exploiting Vertical Lines for Monocular
Vision based Mobile Robot Navigation

In man-made environments, vertical lines are omni-present:
buildings, boxes, bookshelves, cubicle walls, door frames

Many vision based systems assume the image plane is
perpenticular to the ground plane

We proposed methods to rectify an image plane with general
pose to be vertical based on vertical lines.
Rectified Images for Ground Plane
Detection
 The normalized homography of the ground plane has a special
 form in the rectified images:


              ˆ
              H  R ( I  CnT / d )

      n0  (0,1, 0)T          C  ( x0 , 0, z0 )T


                 cos( ) x0 / d     sin( ) 
             H  0
             ˆ
                             1         0          4 DOFs
                  sin( ) z0 / d
                                    cos( ) 
                                             
Results

          Special Form

          4 DOFs



          General Form

          8 DOFs
Ground Rectification for Mosaic based
SLAM
  From vertical image plane, it’s easy to get the ground plane image

  After rectification, the relationship between any two views
  directly indicates their relative locations and orientations.
Results
Vision Based Control

                               Occupied                   Given an rectified image
    Nearest left                          Nearest right
                                                          Find the object to track
                   7   8   9
                                                          Identify the obstacle or ground plane
      5
            6                  10
                                    11                    Output a turn angle.
…                                            …
                                                          Adjust the camera to make object
                                                          always stays at the center of the image.



                                                                                      Movie 1
                                                                Vanishing line
                                                                                      Movie 2
                                                                                      Images
                                                                                      Obstacles



                                    Turn angle                                   Rectified Image
                                                                       Occupied
Other Potential Applications

   Surveillance/Activity Recognition/ Path Planning

                                    Different object’s size




                                     Same object’s size
                                     Ground is rectified
                                     Right angles are
                                     recovered
                                     Speed is reflected on
                                     the image
Conclusions and Future Works
   Novel image rectification schemes are proposed in the
    context of exploring several practical 3D vision problems
   For each of the problem, we designed novel algorithms
    and nice results are achieved. Moreover, we gain new
    insights to the problems.
   In the future, we should combine different approaches
    and exploiting more visual cues for 3D vision problems.
Questions?


Thanks!

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:8/15/2012
language:Latin
pages:66