Document Sample

Structure from motion Class 12 Read Chapter 5 Assignment 2 Chris MS regions Nathan … Brian M&S LoG features Li SIFT features Chad MS regions Seon Joo SIFT features Jason SIFT features Sudipta T&VG elliptic features Sriram … Christine … Assignment 3 • Collect potential matches from all algorithms for all pairs • Matlab ASCII format, exchange data naming convention: firstname_ij.dat chris_56.dat • Implement RANSAC that uses combined match dataset [F,inliers]=FRANSAC([chris_56; brian_56; …]) • Compute consistent set of matches and epipolar geometry • Report thresholds used, match sets used, number of consistent matches obtained, epipolar geometry, show matches and epipolar geometry (plot some epipolar lines). Due next Tuesday, Nov. 2 Papers • Each should present a paper during 20-25 minutes followed by discussion. Partially outside of class schedule to make up for missed classes. (When?) • List of proposed papers will come on-line by Thursday, feel free to propose your own (suggestion: something related to your project). • Make choice by Thursday, assignments will be made in class. • Everybody should have read papers that are being discussed. 3D photography course schedule Introduction Aug 24, 26 (no course) (no course) Aug.31,Sep.2 (no course) (no course) Sep. 7, 9 (no course) (no course) Sep. 14, 16 Projective Geometry Camera Model and Calibration (assignment 1) Feb. 21, 23 Camera Calib. and SVM Feature matching (assignment 2) Feb. 28, 30 Feature tracking Epipolar geometry (assignment 3) Oct. 5, 7 Computing F Triangulation and MVG Oct. 12, 14 (university day) (fall break) Oct. 19, 21 Stereo Active ranging Oct. 26, 28 Structure from motion Self-calibration Nov. 2, 4 Shape-from-silhouettes Space carving Nov. 9, 11 3D modeling Appearance Modeling Nov.12 papers (2-3pm SN115) Nov. 16, 18 (VMV’04) (VMV’04) Nov. 23, 25 papers & discussion (Thanksgiving) Nov.30,Dec.2 papers & discussion papers and discussion Dec.3 papers (2-3pm SN115) Dec. 7? Project presentations Ideas for a project? Chris Wide-area display reconstruction Nathan ? Brian ? Li Visual-hulls with occlusions Chad Laser scanner for 3D environments Seon Joo Collaborative 3D tracking Jason SfM for long sequences Combining exact silhouettes and Sudipta photoconsistency Sriram Panoramic cameras self-calibration Christine desktop lamp scanner Today’s class • Structure from motion • factorization • sequential • bundle adjustment Factorization • Factorise observations in structure of the scene and motion/calibration of the camera • Use all points in all images at the same time Affine factorisation Projective factorisation Affine camera The affine projection equations are X j X j xij Pi x y P y Yj xij Pi x Y j ij i Z j y y Z 1 0001 ij Pi j 1 1 xij Pi xij x4 ~ x X j P y4 ~ i y Yj yij Pi yij Pi Z j how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity xij ij ij ~ x x yij ij ij ~ y y i i Orthographic factorization (Tomasi Kanade’92) The ortographic projection equations are mij Pi M j , i 1,...,m, j 1,...,n where X j ~ xij Pi x mij ~ , Pi y , M j Y j yij Pi Z j All equations can be collected for all i and j m PM where m11 m12 m1n P1 m m 22 m2n , P P2 , M M , M ,..., M m 21 1 2 n m m1 mm 2 m mn Pm Note that P and M are resp. 2mx3 and 3xn matrices and therefore the rank of m is at most 3 Orthographic factorization (Tomasi Kanade’92) Factorize m through singular value decomposition m UV T An affine reconstruction is obtained as follows ~ ~ P U, M V T Closest rank-3 approximation yields MLE! m11 m12 m1n P1 m 21 min m 22 m 2 n P2 M , M ,..., M 1 2 n m m mn P m1 mm2 m Orthographic factorization (Tomasi Kanade’92) Factorize m through singular value decomposition m UV T An affine reconstruction is obtained as follows ~ ~ P U, M V T A metric reconstruction is obtained as follows ~ 1 ~ P PA , M AM Where A is computed from ~xxP x~ x1 ~ x T 3 linear equations per view on T 1 T T Pii CPi A Pi 1 P A i 1 symmetric matrix C (6DOF) ~yyP y~ y 1 ~ y T T 1 T T Pii CPi A Pi 1 P A i 1 A can be obtained from C ~xxP y~ y 0 ~ y T through Cholesky factorisation T T 1 Pii CPi A Pi 0 T P A i 0 and inversion Examples Tomasi Kanade’92, Poelman & Kanade’94 Examples Tomasi Kanade’92, Poelman & Kanade’94 Examples Tomasi Kanade’92, Poelman & Kanade’94 Examples Tomasi Kanade’92, Poelman & Kanade’94 Perspective factorization The camera equations λ ij mij Pi M j , i 1,..., m, j 1,..., m for a fixed image i can be written in matrix form as mi i Pi M where m i mi1 , mi 2 ,..., mim , M M1 , M 2 ,..., M m i diag λ i1 , λ i 2 ,..., λ im Perspective factorization All equations can be collected for all i as m PM where m11 P1 m P m 2 2 , P 2 ... ... m n n Pm In these formulas m are known, but i,P and M are unknown Observe that PM is a product of a 3mx4 matrix and a 4xn matrix, i.e. it is a rank-4 matrix Perspective factorization algorithm Assume that i are known, then PM is known. Use the singular value decomposition PM=U VT In the noise-free case =diag(s1,s2,s3,s4,0, … ,0) and a reconstruction can be obtained by setting: P=the first four columns of U. M=the first four rows of V. Iterative perspective factorization When i are unknown the following algorithm can be used: 1. Set lij=1 (affine approximation). 2. Factorize PM and obtain an estimate of P and M. If s5 is sufficiently small then STOP. 3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM 4. Goto 2. In general the algorithm minimizes the proximity measure P(,P,M)=s5 Note that structure and motion recovered up to an arbitrary projective transformation Further Factorization work Factorization with uncertainty (Irani & Anandan, IJCV’02) Factorization for dynamic scenes (Costeira and Kanade ‘94) (Bregler et al. 2000, Brand 2001) practical structure and motion recovery from images • Obtain reliable matches using matching or tracking and 2/3-view relations • Compute initial structure and motion • Refine structure and motion • Auto-calibrate • Refine metric structure and motion Sequential Structure and Motion Computation Initialize Motion Initialize Structure (P1,P2 compatibel with F) (minimize reprojection error) Extend motion Extend structure (compute pose through matches (Initialize new structure, seen in 2 or more previous views) refine existing structure) Computation of initial structure and motion according to Hartley and Zisserman “this area is still to some extend a black-art” All features not visible in all images No direct method (factorization not applicable) Build partial reconstructions and assemble (more views is more stable, but less corresp.) 1) Sequential structure and motion recovery 2) Hierarchical structure and motion recovery Sequential structure and motion recovery • Initialize structure and motion from two views • For each additional view • Determine pose • Refine and extend structure • Determine correspondences robustly by jointly estimating matches and epipolar geometry Initial structure and motion Epipolar geometry Projective calibration m Fm 1 0 T P1 I 0 e F ea 2 T P2 x e compatible with F Yields correct projective camera setup (Faugeras´92,Hartley´92) Obtain structure through triangulation Use reprojection error for minimization Avoid measurements in projective space Determine pose towards existing structure M 2D-3D 2D-3D mi+1 mi 2D-2D new view x i Pi X(x 1 ,..., x i 1 ) Compute Pi+1 using robust approach (6-point RANSAC) Extend and refine reconstruction Compute P with 6-point RANSAC • Generate hypothesis using 6 points • Count inliers • Projection error d Pi Xx1 ,..., x i 1 , x i t ? • 3D error d Pi x i , X t3D ? -1 • Back-projection error d Fij x i , x j t ?, j i • Re-projection error d Pi Xx1 ,..., x i 1 , x i , x i t • Projection error with covariance d Pi Xx1 ,..., x i 1 , x i t • Expensive testing? Abort early if not promising • Verify at random, abort if e.g. P(wrong)>0.95 (Chum and Matas, BMVC’02) Dealing with dominant planar scenes (Pollefeys et al., ECCV‘02) • USaM fails when common features are all in a plane • Solution: part 1 Model selection to detect problem Dealing with dominant planar scenes (Pollefeys et al., ECCV‘02) • USaM fails when common features are all in a plane • Solution: part 2 Delay ambiguous computations until after self-calibration (couple self-calibration over all 3D parts) Non-sequential image collections Problem: Features are lost 3792 points and reinitialized as new features Solution: Match with other close views 4.8im/pt 64 images Relating to more views For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm For all existing structure Refine close views k Compute two view geometry k/i and matches Initialize new structure Infer new 2D-3D matches and add to list Refine pose using all 2D-3D matches Refine existing structure Initialize new structure Problem: find close views in projective frame Determining close views • If viewpoints are close then most image changes can be modelled through a planar homography • Qualitative distance measure is obtained by looking at the residual error on the best possible planar homography Distance = min median DHm, m´ Non-sequential image collections (2) 2170 points 3792 points 9.8im/pt 64 images 4.8im/pt 64 images Hierarchical structure and motion recovery • Compute 2-view • Compute 3-view • Stitch 3-view reconstructions • Merge and refine reconstruction F T H PM Stitching 3-view reconstructions Different possibilities 1. Align (P2,P3) with (P’1,P’2) arg min d A P2 , P'1 H -1 d A P3 , P' 2 H -1 arg min d X , HX' H 2. Align X,X’ (and C’C’) A j j H arg min d PH X' , x j -1 3. Minimize reproj. error j j H j d P' HX , x' j j arg min d PX , x j 4. MLE (merge) j j P,X j Refining structure and motion • Minimize reprojection error m n min D mki , Pk Mi ˆ ˆ 2 ˆ ˆ Pk ,M i k 1 i 1 • Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) • Huge problem but can be solved efficiently (Bundle adjustment) Non-linear least-squares X f (P) argmin X f (P) P • Newton iteration • Levenberg-Marquardt • Sparse Levenberg-Marquardt Newton iteration Taylor approximation Jacobian X f (P0 ) f (P0 ) J J P X f (P ) 1 X f (P ) X f (P0 ) J e0 J 1 J J J e0 J J J e0 T T T -1 T Pi 1 Pi J J J e0 T -1 T normal eq. -1 J T -1J J T -1e0 Levenberg-Marquardt Normal equations J J N J e0 T T Augmented normal equations N' J T e0 N' J T J λdiag(J T J) λ 0 10 3 success: λ i 1 λ i / 10 accept failure : λ i 10 λ i solve again l small ~ Newton (quadratic convergence) l large ~ descent (guaranteed decrease) Levenberg-Marquardt Requirements for minimization • Function to compute f • Start value P0 • Optionally, function to compute J (but numerical ok, too) Sparse Levenberg-Marquardt • N 3 complexity for solving N' -1 J T e0 • prohibitive for large problems (100 views 10,000 points ~30,000 unknowns) • Partition parameters • partition A • partition B (only dependent on A and itself) Sparse bundle adjustment residuals: normal equations: with note: tie points should be in partition A Sparse bundle adjustment normal equations: modified normal equations: solve in two parts: Sparse bundle adjustment Jacobian of has sparse block structure m n 2 ˆ ˆ D m ki , Pk M i k 1 i 1 P1 P2 P3 M U1 im.pts. view 1 U2 W J N JT J U3 WT V 12xm 3xn (in general Needed for non-linear minimization much larger) Sparse bundle adjustment • Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m U-WV-1WT I WV N 1 0 I Allows much more efficient computations WT V e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 11xm 3xn Often still band diagonal use sparse linear algebra algorithms Sparse bundle adjustment normal equations: modified normal equations: solve in two parts: Sparse bundle adjustment • Covariance estimation a U WV W -1 1 b Y a Y V T Y WV-1 ab - a Y Next class: self-calibration * *

DOCUMENT INFO

Shared By:

Categories:

Tags:

Stats:

views: | 3 |

posted: | 8/25/2011 |

language: | English |

pages: | 49 |

OTHER DOCS BY yaofenjin

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.