Docstoc

class12

Document Sample
class12 Powered By Docstoc
					Structure from motion
       Class 12




       Read Chapter 5
            Assignment 2
Chris       MS regions
Nathan      …
Brian       M&S LoG features
Li          SIFT features
Chad        MS regions
Seon Joo    SIFT features
Jason       SIFT features
Sudipta     T&VG elliptic features
Sriram      …
Christine   …
                    Assignment 3

•   Collect potential matches from all algorithms for all pairs
•   Matlab ASCII format, exchange data


                                   naming convention: firstname_ij.dat
                                                  chris_56.dat

•   Implement RANSAC that uses combined match dataset
                        [F,inliers]=FRANSAC([chris_56; brian_56; …])
•   Compute consistent set of matches and epipolar geometry
•   Report thresholds used, match sets used, number of consistent
    matches obtained, epipolar geometry, show matches and
    epipolar geometry (plot some epipolar lines).

Due next Tuesday, Nov. 2
                   Papers

• Each should present a paper during 20-25
  minutes followed by discussion. Partially outside
  of class schedule to make up for missed classes.
  (When?)

• List of proposed papers will come on-line by
  Thursday, feel free to propose your own
  (suggestion: something related to your project).
• Make choice by Thursday, assignments will be
  made in class.
• Everybody should have read papers that are
  being discussed.
                3D photography course schedule
                     Introduction
 Aug 24, 26          (no course)                 (no course)
 Aug.31,Sep.2        (no course)                 (no course)
 Sep. 7, 9           (no course)                 (no course)
 Sep. 14, 16     Projective Geometry     Camera Model and Calibration
                                               (assignment 1)
 Feb. 21, 23    Camera Calib. and SVM         Feature matching
                                               (assignment 2)
 Feb. 28, 30       Feature tracking           Epipolar geometry
                                               (assignment 3)
 Oct. 5, 7          Computing F             Triangulation and MVG

 Oct. 12, 14       (university day)               (fall break)
Oct. 19, 21             Stereo                  Active ranging
Oct. 26, 28     Structure from motion           Self-calibration
 Nov. 2, 4      Shape-from-silhouettes          Space carving
 Nov. 9, 11          3D modeling            Appearance Modeling         Nov.12 papers
                                                                        (2-3pm SN115)

 Nov. 16, 18          (VMV’04)                     (VMV’04)
 Nov. 23, 25     papers & discussion            (Thanksgiving)
 Nov.30,Dec.2    papers & discussion        papers and discussion       Dec.3 papers
                                                                        (2-3pm SN115)

 Dec. 7?        Project presentations
            Ideas for a project?
Chris          Wide-area display reconstruction
Nathan         ?
Brian          ?
Li             Visual-hulls with occlusions
Chad           Laser scanner for 3D environments
Seon Joo       Collaborative 3D tracking
Jason          SfM for long sequences
               Combining exact silhouettes and
Sudipta
               photoconsistency
Sriram         Panoramic cameras self-calibration
Christine      desktop lamp scanner
            Today’s class

• Structure from motion

  • factorization
  • sequential

  • bundle adjustment
               Factorization

• Factorise observations in structure of the scene
  and motion/calibration of the camera

• Use all points in all images at the same time

 Affine factorisation
 Projective factorisation
Affine camera
The affine projection equations are
                   X j                          X j 
     xij   Pi   
                  x

     y    P y  Yj          xij   Pi x   Y j 
     ij   i   Z j           y    y  Z 
     1  0001                ij   Pi   j 
                1                             1 
                                                 
    xij  Pi   xij
             x4       ~   x  X j 
                             P  
            y4 
                    ~    i y  Yj 
    yij  Pi   yij   Pi   Z 
               
                                    j
 how to find the origin? or for that matter a 3D reference point?
 affine projection preserves center of gravity
       xij ij  ij
       ~ x  x                  yij ij  ij
                                 ~ y  y
                                               i
                      i
Orthographic factorization
                                                  (Tomasi Kanade’92)
The ortographic projection equations are
         mij  Pi M j , i  1,...,m, j  1,...,n
where                                    X j 
                ~ 
               xij         Pi          
                               x
        mij   ~  , Pi   y  , M j  Y j 
               yij        Pi         Z j 
                                          
 All equations can be collected for all i and j
                  m  PM
 where  m11     m12     m1n          P1 
        m                              
                 m 22    m2n  , P   P2 , M  M , M ,..., M 
    m   21
                                            1   2       n

                                      
         m m1   mm 2    m mn         Pm 
                                        
Note that P and M are resp. 2mx3 and 3xn matrices and
therefore the rank of m is at most 3
   Orthographic factorization
                                             (Tomasi Kanade’92)
Factorize m through singular value decomposition
     m  UV T
An affine reconstruction is obtained as follows
     ~        ~
     P  U, M  V T
Closest rank-3 approximation yields MLE!

       m11    m12     m1n   P1 
       m 21                       
  min         m 22    m 2 n    P2 M , M ,..., M 
                          1 2                 n
      m               m mn   P 
       m1     mm2              m
   Orthographic factorization
                                          (Tomasi Kanade’92)
Factorize m through singular value decomposition
     m  UV T
An affine reconstruction is obtained as follows
     ~        ~
     P  U, M  V T
A metric reconstruction is obtained as follows
           ~ 1            ~
     P  PA , M  AM
Where A is computed from
     ~xxP x~ x1 ~ x T 3 linear equations per view on
            T
           1 T T
    Pii CPi A Pi  1
    P A  i         1
                         symmetric matrix C (6DOF)
     ~yyP y~ y 1 ~ y T
            T
           1 T T
    Pii CPi A Pi  1
    P A  i         1
                         A can be obtained from C
     ~xxP y~ y 0 ~ y T through Cholesky factorisation
            T T
           1
    Pii CPi A Pi  0
                 T
    P A  i         0
                        and inversion
  Examples




 Tomasi Kanade’92,
Poelman & Kanade’94
  Examples




 Tomasi Kanade’92,
Poelman & Kanade’94
  Examples




 Tomasi Kanade’92,
Poelman & Kanade’94
  Examples




 Tomasi Kanade’92,
Poelman & Kanade’94
    Perspective factorization
The camera equations

   λ ij mij  Pi M j , i  1,..., m, j  1,..., m
for a fixed image i can be written in matrix
  form as
                mi  i  Pi M
where
 m i  mi1 , mi 2 ,..., mim  , M  M1 , M 2 ,..., M m 
               i  diag λ i1 , λ i 2 ,..., λ im 
     Perspective factorization
All equations can be collected for all i as
                  m  PM
where
             m11      P1 
            m         P 
         m  2 2 , P   2 
             ...       ... 
                        
            m n  n    Pm 
 In these formulas m are known, but i,P and M
 are unknown
 Observe that PM is a product of a 3mx4 matrix
 and a 4xn matrix, i.e. it is a rank-4 matrix
     Perspective factorization
            algorithm
Assume that i are known, then PM is known.

Use the singular value decomposition
                     PM=U VT

In the noise-free case
              =diag(s1,s2,s3,s4,0, … ,0)
and a reconstruction can be obtained by setting:

           P=the first four columns of U.
             M=the first four rows of V.
        Iterative perspective
            factorization
When i are unknown the following algorithm can be
used:
1. Set lij=1 (affine approximation).
2. Factorize PM and obtain an estimate of P and M.
If s5 is sufficiently small then STOP.
3. Use m, P and M to estimate i from the camera
equations (linearly) mi i=PiM
4. Goto 2.
In general the algorithm minimizes the proximity
measure          P(,P,M)=s5
   Note that structure and motion recovered
   up to an arbitrary projective transformation
  Further Factorization work

Factorization with uncertainty
                    (Irani & Anandan, IJCV’02)


Factorization for dynamic scenes
                 (Costeira and Kanade ‘94)

                 (Bregler et al. 2000,
                  Brand 2001)
  practical structure and motion
      recovery from images

• Obtain reliable matches using matching or
  tracking and 2/3-view relations
• Compute initial structure and motion
• Refine structure and motion
• Auto-calibrate
• Refine metric structure and motion
                Sequential Structure and
                  Motion Computation


  Initialize Motion                       Initialize Structure
  (P1,P2 compatibel with F)               (minimize reprojection error)




Extend motion                       Extend structure
(compute pose through matches       (Initialize new structure,
seen in 2 or more previous views)    refine existing structure)
          Computation of initial
          structure and motion

according to Hartley and Zisserman
“this area is still to some extend a black-art”
 All features not visible in all images
  No direct method (factorization not applicable)
  Build partial reconstructions and assemble
    (more views is more stable, but less corresp.)



  1) Sequential structure and motion recovery

  2) Hierarchical structure and motion recovery
    Sequential structure and
       motion recovery
• Initialize structure and motion from two
  views
• For each additional view
   • Determine pose
   • Refine and extend structure

• Determine correspondences robustly by
  jointly estimating matches and epipolar
  geometry
Initial structure and motion
Epipolar geometry  Projective calibration

  m Fm 1  0
    T                      P1  I 0
                                 e F  ea           
    2
                                                T
                           P2         x             e
                                compatible with F

Yields correct projective camera setup
                                  (Faugeras´92,Hartley´92)

Obtain structure through triangulation
   Use reprojection error for minimization
   Avoid measurements in projective space
Determine pose towards existing structure


                                   M

                           2D-3D         2D-3D



                                                  mi+1
                          mi
                                       2D-2D   new view




x i  Pi X(x 1 ,..., x i 1 )
Compute Pi+1 using robust approach (6-point RANSAC)
Extend and refine reconstruction
 Compute P with 6-point RANSAC

• Generate hypothesis using 6 points



• Count inliers
   • Projection error d Pi Xx1 ,..., x i 1 , x i   t ?
                                  
   • 3D error d Pi x i , X  t3D ?
                          -1


                                               
   • Back-projection error d Fij x i , x j  t ?, j  i
   • Re-projection error d Pi Xx1 ,..., x i 1 , x i , x i   t
   • Projection error with covariance d  Pi Xx1 ,..., x i 1 , x i   t
• Expensive testing? Abort early if not promising
   • Verify at random, abort if e.g. P(wrong)>0.95
                                           (Chum and Matas, BMVC’02)
 Dealing with dominant planar scenes
                                       (Pollefeys et al., ECCV‘02)
• USaM fails when common features are all in a plane

• Solution: part 1 Model selection to detect problem
 Dealing with dominant planar scenes
                                        (Pollefeys et al., ECCV‘02)
• USaM fails when common features are all in a plane
• Solution: part 2 Delay ambiguous computations
                        until after self-calibration
                       (couple self-calibration over all 3D parts)
Non-sequential image
     collections
                                Problem:
                                Features are lost



   3792 points
                                and reinitialized as
                                new features

                                Solution:
                                Match with other
                                close views




                 4.8im/pt
                    64 images
         Relating to more views
For every view i
    Extract features
    Compute two view geometry i-1/i and matches
    Compute pose using robust algorithm
    For all existing structure
    Refine close views k
         Compute two view geometry k/i and matches
    Initialize new structure
        Infer new 2D-3D matches and add to list
    Refine pose using all 2D-3D matches
    Refine existing structure
    Initialize new structure




Problem:
find close views in projective frame
Determining close views

• If viewpoints are close then most image changes
  can be modelled through a planar homography

• Qualitative distance measure is obtained by looking
  at the residual error on the best possible planar
  homography

        Distance =   min median DHm, m´ 
Non-sequential image
   collections (2)




                                2170 points
   3792 points
                                              9.8im/pt

                                                64 images




                 4.8im/pt

                    64 images
Hierarchical structure and motion
             recovery

•    Compute 2-view
•    Compute 3-view
•    Stitch 3-view reconstructions
•    Merge and refine reconstruction

 F
 T
 H

PM
  Stitching 3-view reconstructions




Different possibilities
                                                        
1. Align (P2,P3) with (P’1,P’2) arg min d A P2 , P'1 H -1  d A P3 , P' 2 H -1        
                                    arg min  d X , HX' 
                                       H
2. Align X,X’ (and C’C’)                                 A       j            j
                                        H

                                    arg min  d PH X' , x 
                                                 j
                                                                 -1
3. Minimize reproj. error                                                 j       j
                                       H         j

                                             d P' HX , x'             j       j


                                   arg min  d PX , x 
                                                     j
4. MLE (merge)                                               j        j
                                      P,X    j
Refining structure and motion

• Minimize reprojection error

                                  
                   m     n
         min   D mki , Pk Mi
                         ˆ ˆ       2

         ˆ ˆ
         Pk ,M i
                   k 1 i 1



  • Maximum Likelyhood Estimation
           (if error zero-mean Gaussian noise)
  • Huge problem but can be solved efficiently
                   (Bundle adjustment)
Non-linear least-squares

X  f (P)        argmin X  f (P)
                     P
• Newton iteration
• Levenberg-Marquardt
• Sparse Levenberg-Marquardt
            Newton iteration
Taylor approximation                          Jacobian
                                                        X
     f (P0  )  f (P0 )  J                       J
                                                        P
 X  f (P )
         1

     X  f (P )  X  f (P0 )  J  e0  J
             1

    J J  J e0    J J J e0
        T       T
                                  T        -1 T



   Pi 1  Pi                 
                         J J J e0
                                T       -1 T
                                                    normal eq.

                                       
                                        -1
                         J T -1J J T -1e0
      Levenberg-Marquardt
Normal equations

 J J  N  J e0
  T                   T


Augmented normal equations

 N'   J T e0            N'  J T J  λdiag(J T J)
 λ 0  10 3
    success: λ i 1  λ i / 10   accept
     failure : λ i  10 λ i      solve again

  l small ~ Newton (quadratic convergence)
  l large ~ descent (guaranteed decrease)
      Levenberg-Marquardt

Requirements for minimization
• Function to compute f
• Start value P0
• Optionally, function to compute J
 (but numerical ok, too)
 Sparse Levenberg-Marquardt

• N 3 complexity for solving   N' -1 J T e0
   • prohibitive for large problems
     (100 views 10,000 points ~30,000 unknowns)


• Partition parameters
   • partition A
   • partition B (only dependent on A and itself)
    Sparse bundle adjustment

residuals:
normal equations:

with




note: tie points should be in partition A
    Sparse bundle adjustment

normal equations:



modified normal equations:



solve in two parts:
                      Sparse bundle adjustment

               Jacobian of                            has sparse block structure
                                 m    n
                                                              2
                                                     ˆ ˆ
                                            D m ki , Pk M i
                                k 1 i 1


          P1    P2     P3       M                                     U1
im.pts.
 view 1
                                                                             U2           W
   J                                           N  JT J 
                                                                                   U3


                                                                             WT           V
               12xm             3xn
                            (in general                           Needed for non-linear minimization
                            much larger)
                  Sparse bundle adjustment
            • Eliminate dependence of camera/motion
              parameters on structure parameters
               Note in general 3n >> 11m



                                              U-WV-1WT
                            I  WV   N 
                                       1

                           0
                                 I 
                                    

Allows much more efficient
computations                                     WT      V

e.g. 100 views,10000 points,
     solve 1000x1000, not 30000x30000        11xm      3xn

Often still band diagonal
use sparse linear algebra algorithms
    Sparse bundle adjustment

normal equations:



modified normal equations:



solve in two parts:
  Sparse bundle adjustment

• Covariance estimation
     a  U  WV W -1   

                    1
    b  Y a Y  V
            T
                              Y  WV-1

     ab  -  a Y
Next class: self-calibration


                   *
        *

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:8/25/2011
language:English
pages:49