Document Sample

New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation Jin Zhou June 16th, 2009 Dissertation Defense Outline Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic Sequences Rapid 3D Modeling from Single Images Robot Vision Conclusions and Future Work The Geometry of 3D to 2D Images are 2D projections of the 3D world 3D Vision – The Problem How do we extract 3D information from 2D images? ? 3D of the objects ? 3D of the cameras ? ? ? 3D Vision – Applications Augmented Reality Scene Modeling Virtual Touring 3D Imaging Robots 3D Vision – A Human Perspective Size Linear Perspective Object Connections Stereo Motion Shading Texture 3D Vision – Computational Approaches Different approaches use different cues Different problems requires different approaches. Structure from Motion (SfM) Rely on point correspondences Single View Based Modeling (SVBM) Rely on knowledge of the scene Camera calibration All approaches requires the images are calibarated first (either manual or automatic) 3D Vision – Practical Challenges Camera information is unavailable Point correspondences is not reliable and time-consuming Image resolution is limited Degeneracy 3D Vision – Limitations of Current Approaches Distortion Degeneracy Dueto high degree of freedom of geometric models Lack of geometric meaning Most approaches are purely based on algebraic derivations or imaginery objects. Not accurate or not convinient (SVBM). 3D Vision – Our Contributions Novel image rectification schemes are proposed based on sequential virtual rotation Novel approaches are proposed for the following problems Image Rectification for Stereoscopic visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic sequences Rapid Cones and Cylinders Modeling Monocular Vision Guided Mobile Robot Navigation 3D Vision – Results of Our Approaches No affine/projective distortion Can handle degeneracy Intuitive geometric meanings Lead to insights of particular problems Accurate and fast Publications What is Image Rectification? Image rectification is a process to transform the original images to new images which have desired properties. General Image Transformations H? x' H x Image transformation can be defined by a 3x3 matrix H, which is called Homography. Image Rectification based on Virtual Rotation Homography of camera rotation/zooming P KR[ I | C ], P ' K ' R '[ I | C ] x ' P ' X K ' R '[ I | C ] X K ' R '( KR) 1 KR[ I | C ] X ( K ' R ')( KR) 1 x H ( K ' R ')( KR)1 If we normalize the coordinates x K 1x ˆ x' K '1 x' ˆ Assume R = I ˆ H R ' Camera orientation is determined at the same time Advantages of the New Rectification Schemes Intuitive geometric meaning Robust Rotation parameters can be computed by various basic image features, such as points, lines and circles. Can be used for camera calibration. Can be used for 3D information extraction. Lead to non-distorted results Reason: Rotation do not introduce affine/projective distortion Rotations are Decomposed as Euler Angles 1 0 0 Rx ( ) 0 cos( ) sin( ) 0 sin( ) cos( ) cos( ) 0 sin( ) Ry ( ) 0 1 0 sin( ) 0 cos( ) cos( ) sin( ) 0 Rz ( ) sin( ) cos( ) 0 0 0 1 Rotation Parameters Can Be Estimated by Basic Image Features Each rotation has only one degree of freedom and thus only needs one constraint. Example: transforming a point on to y axis Rz p (0, d , c)T ˆ / 2 [ ] / 2 a cos( ) b sin( ) 0 otherwize Normalize arc tan(a, b) [arc tan(a, b)] Ambiguity! Image Rectification for Stereoscopic Visualization The Principle of Stereoscopic (3D) Visualization Motivation Stereo content is scarce Stereo cameras/camcorders are expensive Common users seldom use stereo cameras/camcorders We want to generate stereo content from images/videos taken by common cameras The Problem Given two arbitrary images, rectify them so that the results look like a stereo pair. Our Approach – Rectification based on Virtual Rotation We can “rotate” camera to standard stereo setup. H i KR( Ki Ri )1 Calibrated Case Constraints of the stereo camera pair: 1. The two cameras have the same intrinsic parameters (K) and orientation (R) 2. The camera’s optical axis is perpendicular to the baseline (C1 – C2) i.e. the camera’s x axis has the same direction with the baseline UnKnown R (r , s, t )T r (C1 C2 ) / || C1 C2 || 1 H i KR( Ki Ri ) t (r p) / || r p || s r t K1 Known Any vector Uncalibrated Case For the uncalibrated case, all K, R and C are unknown. We can only start from the fundamental matrix and point correspondences. Estimate H2 (homography for the second image) H i KR( Ki Ri )1 ? f 0 px px w / 2 R2 I K K2 0 f p y with p y h / 2 0 0 1 f ( w h) / 2 Determine R based on Sequential Virtual Rotation Constraints: Re2 (1, 0, 0)T ˆ e2 K 2 1e2 ˆ First rotate around z axis so that the point is transformed to x axis (i.e. y = 0) Rz (a, b, c)T (d , 0, c)T z [arctan(b, a)] Rotate around y axis so that the point is transformed to infinity. Ry (d ,0, c)T (e,0,0)T (1,0,0)T y [arctan(c, d )] R Ry Rz Estimate H1 H1 KR( K1 R1 )1 H 2 K2 R2 ( K1 R1 )1 H 2 M M ( I e2 vT ) M 0 F [e2 ] M H1 H 2 M H 2 ( I e2 vT ) M 0 H 2 ( M 0 e2 vT M 0 ) H 2 M 0 H 2 e2 vT H 2 1 H 2 M 0 H 2 e2 KRK 2 1e2 K [r , s, t ]T r ( I H 2 e2 v1T ) H 2 M 0 (v1T =vT H 2 1 ) K (1, 0, 0)T (k , 0, 0)T H1 ( I H 2 e2 vT ) H 2 M 0 H A H 2 M 0 a b c H A 0 1 0 0 0 1 Estimate H1 Determine a, b and c Property of standard stereo setup: For two points with the same depth, their projection on different images should have the same distance (Points with the same depth should have the same disparity). Approach Group points by similar disparities Then compute a, b by minimizing 2 ( a ( xi xi ) b( yi y j ) ( xi ' x j ')) ˆ ˆ ˆ ˆ ˆ ˆ 2 (|| H1 x i H1 x j || | H 2 x i ' H 2 x j ' ||) p i , j Ap p i , j Ap ax0 by0 c x0 ' ˆ ˆ ˆ Results Shear distortion Original Pair Hartley’s Method Our Method Results Shrink horizontally Original Pair Hartley’s Method Our Method Camera Auto-Calibration from the Fundamental Matrix Traditional Approaches: Kruppa Equations [e2 ] w2 [e2 ] Fw1 F T dual image of the absolute conic wi K i K iT Huang-Faugeras constraints 1 trace 2 ( EE T ) trace( EE T ) 2 0 2 E K 2 FK1 T Cons: Complex and hard to understand! Derivation for degenerate cases are purely algebraic. Our Approach We transform the original pair to a standard stereo pair through sequential virtual rotation and zooming 7 DOFs F K 2 T R2 z R2 y F S Rx R1 y R1z K11 T T 7 Parameters ( f1 , 1z , 1 y , x , f 2 , 2 z , 2 y ) Rz (1z )e1 (d1 , 0, c1 )T 0 0 0 F F S 0 0 1 0 1 0 Decomposition Illustration Fz R2 z FR1z1 K2 T R2 y F S Rx R1y K11 T c2 0 0 a b a c1 0 0 Fz 0 1 0 c d c 0 1 0 0 0 d 2 a b a 0 0 d1 f1d 2 a f1 f 2 sin( x ), c cos( x ), cos( 2 y ) f 2 d1 d1d 2 b cos( x ), d sin( x ) cos(1 y ) cos(1 y ) cos( 2 y ) Degeneracy Analysis Degeneracy Analysis Results of Monte Carlo Simulation Stereoscopic View Synthesis From Monocular Endoscopic Videos 3D imaging helps to enable faster and safer surgical operations Two view image rectification can not be applied to the new problem Challenges: 1. Image quality is poor 2. Degeneracy The Framework We proved: 1. Affine 3D reconstruction is sufficient. 2. Linear interpolation in normalized disparity field is equal to linear interpolation in 3D space. Strategy for Solving Degeneracy We assume the initial two frames have same orientation (i.e. they are rectified) The assumption makes the DOF of the fundamental matrix from 8 to 2! No Assumption Degeneracy! Assume the two frame are rectified Interpolation a) Shows the disparities based on the SfM results b) We do Delaunay triangulation and interpolate each triangle c) We pick a set of grid points from b) and do bilinear interpolation d) We fill holes using Laplacian interpolation and do smoothing. Results of Synthetic Data Disparity image after triangulation Final disparity image Ground truth Stereo images Results of Real Data Rapid Cylinders and Cones Modeling from A Single Image Overview of Our Approach Goal: Rapid + Accurate Camera Calibration (Orientation Estimation) Vertical lines Vanishing line of horizontal plane A Cone Modeling from Image Cones (two points / four points) Cylinders (two points / four points) The Coordinate Systems Observation: Most objects are standing on the ground (X,Y,Z,O) -- World Coordinate System Y is perpendicular to the ground (x,y,z,O) -- Camera Coordinate System Camera center is at the origin Orientation Estimation from Vertical Lines R (a, b, c)T (0,1,0)T First rotate around z axis so that the point is transformed to y axis (i.e. x = 0) Rotate around x axis so that the point is transformed to infinity. Orientation Estimation from a Cone RzRxRz R Edges are symmetric to y axis Rx Cross section is a circle Rx (π/2) Illustration Metric Rectification of the Ground Plane 1 0 0 H g Rx ( / 2) 0 0 1 0 1 0 Original Rectified Original Rectified Modeling Cones Cone Parameters: Μ0 (c (0, yc ,1)T , r , yv ) Μ ( R, d , M0 ) 2D Control Points for Cones Modeling: Standard view Cones on Ground General Cones Cones Modeling from the Standard View Rectify the standard view to the ground plane view by Rx(π/2) The cross section is rectified to a circle The edge line is tangent to the circle The center and radius can be determined for any point on the edge line Modeling Cones Standing on the Ground Ry Four points Standard Ground view 3D Mesh view Modeling General Cones R Five points Standard 3D Mesh View Modeling Cylinders Cylinder Parameters: L0 (c (0, yb ,1)T , r , yt ) L ( R, d , L0 ) 2D Control Points for Cones Modeling: Standard View Cylinders on Ground General Cylinders Modeling Cylinders Standing on the Ground Ry two points Standard Ground view 3D Mesh view Modeling General Cylinders R Four points Standard 3D Mesh View Screenshots of Real Data Experiments Exploiting Vertical Lines for Monocular Vision based Mobile Robot Navigation In man-made environments, vertical lines are omni-present: buildings, boxes, bookshelves, cubicle walls, door frames Many vision based systems assume the image plane is perpenticular to the ground plane We proposed methods to rectify an image plane with general pose to be vertical based on vertical lines. Rectified Images for Ground Plane Detection The normalized homography of the ground plane has a special form in the rectified images: ˆ H R ( I CnT / d ) n0 (0,1, 0)T C ( x0 , 0, z0 )T cos( ) x0 / d sin( ) H 0 ˆ 1 0 4 DOFs sin( ) z0 / d cos( ) Results Special Form 4 DOFs General Form 8 DOFs Ground Rectification for Mosaic based SLAM From vertical image plane, it’s easy to get the ground plane image After rectification, the relationship between any two views directly indicates their relative locations and orientations. Results Vision Based Control Occupied Given an rectified image Nearest left Nearest right Find the object to track 7 8 9 Identify the obstacle or ground plane 5 6 10 11 Output a turn angle. … … Adjust the camera to make object always stays at the center of the image. Movie 1 Vanishing line Movie 2 Images Obstacles Turn angle Rectified Image Occupied Other Potential Applications Surveillance/Activity Recognition/ Path Planning Different object’s size Same object’s size Ground is rectified Right angles are recovered Speed is reflected on the image Conclusions and Future Works Novel image rectification schemes are proposed in the context of exploring several practical 3D vision problems For each of the problem, we designed novel algorithms and nice results are achieved. Moreover, we gain new insights to the problems. In the future, we should combine different approaches and exploiting more visual cues for 3D vision problems. Questions? Thanks!

DOCUMENT INFO

Shared By:

Categories:

Tags:

Stats:

views: | 8 |

posted: | 8/15/2012 |

language: | Latin |

pages: | 66 |

OTHER DOCS BY ert554898

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.