Mixing Catadioptric and
Perspective Cameras
Peter Sturm
INRIA Rhône-Alpes
France
Introduction
Introduction
Introduction
• Existing results
- epipolar geometry between omnidirectional
cameras
- motion estimation
- (self-) calibration
- ...
Our Goals
• Study the geometry of hybrid stereo systems
(omnidirectional and perspective cameras)
- epipolar geometry
- trifocal tensors
- plane homographies
• Applications
- motion estimation
- calibration
- self-calibration, calibration transfer
- 3D reconstruction
- ...
Plan
Camera models
• Epipolar geometry of hybrid systems
• Derivation of matching tensors
• Applications
• Conclusions
Camera Models
Perspective and affine cameras
Camera Models
Central catadioptric cameras
• mirror (surface of revolution of a conic)
• camera
• virtual optical center
Camera Models
Central catadioptric cameras
• mirror (surface of revolution of a conic)
• camera
• virtual optical center
• calibration
Camera Models
Types of central catadioptric cameras
• hyperbola + perspective camera
• parabola + affine camera
• ...
Plan
• Camera models
Epipolar geometry of hybrid systems
• Derivation of matching tensors
• Applications
• Conclusions
Epipolar Geometry
epipolar plane
T
d3 ~ F3 x3 q'
3
epipolar line
d' ~ F q
3 3 x3 3
epipolar line
d
q q’ d’
epipole
Epipolar Geometry
epipolar conic
Epipolar Geometry
epipolar line
epipole
epipolar conic
epipoles
epipole
Epipolar Geometry
Example
Epipolar Geometry
There exists F6 x 3 with Interpretation:
“ conic ~ F q’ “ 2c1 c c
T
4 5
concretely: c6 ~ F6 x3 q' q c4
2c c q 0
2 6
3
c5 c 2c
6 3
q' Fq 0
T
T
d ~ F q'
3 3 x3 3
d' ~ F q
3 3 x3 3
epipolar
line d q’ d’
q
epipole
epipolar conic
epipoles
purely perspective case
Epipolar Geometry
There exists F6 x 3 with Interpretation:
“ conic ~ F q’ “ c c 2c1
T
4 5
concretely: c6 ~ F6 x3 q' 2c c q 0
q c4
2 6
3
c 2c c5 6 3
2 2 2
q c q c q c q q c q q c q q c 0
1 1 2 2 3 3 1 2 4 1 3 5 2 3 6
c
1
c 2
q q q q q q q q q 0
2 2 2
c 3
1 2 3 c
1 2 1 3 2 3
4
epipolar c 5
line “lifted coordinates”
c
epipole ˆ
q
6
q Fq' 0
epipolar conic
ˆ
T
epipoles
Epipolar Geometry
Example
Epipolar Geometry
BUT...
Until now, linear epipolar relation only found for:
• any combination of perspective, affine or
para-catadioptric cameras (parabolic mirrors)
Not yet for:
• other omnidirectional cameras than para-catadioptric
ones (e.g. based on hyperbolic mirrors)
Epipolar Geometry
Special case:
• combination of perspective and para-catadioptric
cameras
• epipolar conics are circles
• F is of dimension 4x3
• the « lifted coordinates » are:
2 2
q1 q 2
2
q3
q1 q3
q q
2 3
Epipolar Geometry
Epipoles:
• F
4 x3 is of rank 2
• The epipole of the perspective camera is the right
null-vector of F
• F has a one-dimensional left null-space
the two epipoles of the catadioptric camera are the
left null-vectors that are valid lifted coordinates
(quadratic constraint):
2 2
q1 q 2
2
q ~ q3
ˆ ˆˆ ˆ ˆ
q q q q
1 2
2
3
2
4
0
q1 q3
q q
2 3
Plan
• Camera models
• Epipolar geometry of hybrid systems
Derivation of matching tensors
• Applications
• Conclusions
Matching Tensors
• Multi-linear relations between coordinates of correponding
image points
• Purely perspective case: derivation based on linear
equations representing projections (3D 2D)
• Here: equations for back-projection (2D 3D directions)
- perspective cameras
Q
q K R I
Q
P P P
- tP
P
1
Q t DP P P
q P
DP R P KP q
T -1
with P
t P
Matching Tensors
• Linear equations representing back-projections
- perspective cameras
t P P R P KP q t
T -1
Q P
C
Q
- para-catadioptric cameras
Q ˆ
tC C RC BC q
T
C
q C
• BC is of dimension 3x4
• it depends
- on the mirror’s intrinsic parameters
- on the affine camera’s intrinsic parameters
Matching Tensors
• Putting the equations together
Q t R K q t R K q Q 0
T -1 T -1
P P P P
P P P P
P P
Q
T
ˆ
tC C RC BC q
C
tC C RC BC q Q 0
T
ˆ C
1
t P R T K -P1q P
P 0 I3x3 P 0
ˆ I3x3 C 0
T
t C 0 R C BCqC
6 x6 6
Q
6
Matching Tensors
• Putting the equations together
1
t P R T K -P1q P
P 0 I3x3 P 0
ˆ I3x3 C 0
T
t C 0 R C BCqC
6 x6 6
Q
6
This matrix has a kernel
its rank is lower than 6
its determinant is zero
bilinear equation on the coefficients of q and ˆ
q
q Fq 0
P C
ˆ
T
C P
Matching Tensors
• Straightforward extension to more than 2 views ...
1
t P R T K -P1q P
P 0 0 I3x3 P 0
T -1 'P
0
t'P 0 R'P K'P q'P 0 I3x3
tC R C BCqC I3x3
T ˆ C 0
0 0 9 x7 9
Q
7
Plan
• Camera models
• Epipolar geometry of hybrid systems
• Derivation of matching tensors
Applications
- Self-calibration of omnidirectional cameras from
fundamental matrices
- Calibration transfer from an omnidirectional to a
perspective camera
- Self-calibration of omnidirectional cameras from a
plane homography
• Conclusions
Applications
Self-calibration of omnidirectional cameras from
fundamental matrices:
• para-catadioptric camera has 3 intrinsic parameters
• representation as a 4-vector of homogeneous coordinates
• this vector is in the left null-space of the fundamental
matrix of this camera, defined with respect to any other
camera (perspective, affine, catadioptric)
self-calibration is possible from two or more
fundamental matrices
Applications
Calibration transfer from an omnidirectional to a
perspective camera:
• input:
- calibration of a para-catadioptric camera
- fundamental matrix with a perspective camera
closed-form solution for the focal length of the
perspective camera
Applications
Self-calibration of omnidirectional cameras from
a plane homography:
• input:
- plane homography H with a perspective camera
recovery of intrinsic parameters of para-catadioptric
camera (given by null-vector of H)
H 3x 4
Plan
• Camera models
• Epipolar geometry of hybrid systems
• Derivation of matching tensors
• Applications
Conclusions
Conclusions
• Multi-linear matching relations between
perspective, affine and para-catadioptric cameras
• Applications in calibration, self-calibration,
motion estimation, 3D reconstruction, ...
Open questions
• Fundamental matrix etc. for hyper-catadioptric cameras ?
• Plane homographies « for the inverse direction » ?
Perspectives
• Hybrid trifocal tensors for line images
• Multi-view 3D reconstruction for hybrid systems
Mixing Catadioptric and
Perspective Cameras
Peter Sturm
INRIA Rhône-Alpes
France