# Computer Vision Stereo

Document Sample

```					Chapter 7 of the textbook

Stereo vision
and
two-view geometry

• The goal of a stereo system is to get 3D information
• A stereo system consists at least of two ‘converging’
cameras rigidly attached

1
One real example:

Three topics:

• stereo geometry: epipolar geometry
geometric relation between two images
• correspondence
pixels (pts) in different images from the same pt
• reconstruction (triangulation)
3d coordinate of the pt
2
Intuitive epipolar geomegtry
Entirely characterized by the so-called epipolar geometry

Geometric concepts:
• epipole: the image of the other camera center
• epipolar plane: plane defined by the two camera centers and the space pt
• epipolar lines: intersection of the epipolar plane and image plane
• pencil of epipolar lines and planes
• baseline: distance between two camera centers

3
epipolar plane

epipolar line

u
u’

O
O
epipole

4
One real example of epipolar lines:

5
Algebraic characterisation of the
epipolar geometry: the
fundamental matrix
Given a correspondence pair u and u’,
0  uF Tu
where F is 3 by 3
Proof sketch: follow the geometric construction.
• compute the epipole in the second image
• compute the pt at infinity (or ray direction), reproject it onto the second
• define the epipolar line by these two pts
• use anti-symmetric matrix for cross-product

Assume the camera projection matrices are P=(I 0) and P’=(A a),
it can be shown that F = [a] A.
The procedure is the same even if P is of general form.                         6
Properties of the fundamental matrix F:

• rank of F
• ker(F)
• how many d.o.f.
• l’=F u
• l=…
0  uF Tu
•

7
calibrated approach
two (or more) cameras rigidly attached
=
stereo rig
=
stereo system

• calibration of each camera w.r.t. the same object: P and P’
• (optional) rectification
Correspondence using F
• disparity map using F
(computed from P and P’)
• 3D reconstruction                                          8
9

uF  u1 AB]e[   u  e  l
 u1 A 
u1 AB    

xP   u       
,               x

 0 
 a1 A  
oP  e       
,         
1    o
         
b     33   B  P     , a   33   A  P
Obtain F from the given P and P’:
• Correspondence (discussed later)
• 3D reconstruction : trianglulation
 
u i  (ui , vi )  Xi  ( xi , yi , zi )  u  (ui , vi )
i

     c x  c12 yi      c13 zi    c14                   
c11 xi  c12 yi      
 c13 zi      
 c14
ui  11 i                                   
ui 

    c31 xi  c32 yi    c33 zi    c34                   
c31 xi  c32 yi      
 c33 zi      
 c34
                                                         
c21 xi  c22 yi      
 c23 zi      
 c24
c x  c22 yi       c23 zi    c24   vi 

vi  21 i

    c31 xi  c32 yi    c33 zi    c34   
                
c31 xi  c32 yi      
 c33 zi      
 c34

Same equation as the calibration, but unknowns are now
xi, yi, zi instead of cij
1  it  44 
         
 i       
z  
40  
 iy  
 i       
 x  
10
Triangulation:

u    u’

O
O’

11
‘Modern’ uncalibrated approach:
Epipolar geometry by point
correspondences – two-view geometry
0  uF Tu
Because of

NB: it is more powerful,
‘calibration’ needs 3d info, point-correspondence does not, but not 3d
reconstruction

12
Given    ui  u
i         compute F

• 8 pts algorithm
• 7 pts algorithm (minimal data)
• (optimal and robust sol.)

13
14
9   0   9   f9n A
rewrite
                                                    
                                                     
9   0  9f           1, iv , iu ,iv , iviv , iuiv ,iu , iviu , iuiu 
                                                    
9n                                                        
for N points
0        9      ,
f)1 v ,u ,v ,vv ,uv ,u ,vu ,uu(
rewrite
0    33   f     fv 
23      13   fu    32   fv     f vv 
22            12   fuv     fu 
31        f vu 
21          fuu
11
expand
0  uF Tu
T                                       T
)1,v ,u(  u                        )1,v ,u(  u
8-point algorithm (unstable)
15
   f         21   f    1 1f 
 31                         
 32 f        22   f    12 f        33 F
                  f         
 33 f        23        13 f 
 0     0     1 
T
                     esop
V 0     2                  ce d 
0 U  m oD V S
       0        
3            0 
   0        0     1 
                           esop
F  T V    0       2                 
0 U  m oce r
            0        
0  3             0 
 f      21f     11f    
 31                    
 32 f   22 f    12 f        F  ) 33 f , 23 f ,13 f , 32 f , 22 f ,12 f , 31f , 21f ,11f (  9f
33
           f       f   
 33 f   23      13     
• F’, rank enforcement afterwards by svd!
• linear sol by svd with ||f||=1: f=v9
16
0  d  zc  2 zb  3 za
x
 z
y
0  3 yd  2y xc  y 2 xb  3 xa  0  )F (ted
*   *       1
y 1c  x 11c 
                       
*   *          *       F
    *          *       
*                      
9   vy  8 vx      9   f
• from the vanishing determinant, get a cubic equation
• one parameter solution by svd
7-point algorithm
Warning: unnormalised 8 pt algorithm is unstable!!!

Normalisation 8 pt algorithm
To make the average point as close as possible to (1,1,1)!

• normalisation by transformations
uT  u , uT  u
ˆ         ˆ
• linear solution for
u  u
ˆ    ˆ
0  uF Tu
ˆˆ ˆ

• rank enforcement
F  F
ˆ   ˆ
• denormalisation
TF TT  F
ˆ
17
18
1                1
iv         v ,  iu       u
n                n
2                    1
2  2 )) v      iv( s (    )) u  iu( s (  n
2                                      2 n
) v  iv(  2 ) u  iu(  i d ,            s
id
 us      0       s
                   
 vs      s       0  T
          0        
 1                0
ˆ
uT  u
Data normalisation: each image data is normlised independently!
Summary (or a unified view) of all methods of
computation of the fundamental matrix

19
Stereo correspondence

20
The epipolar geometry gives only a constraint, but not yet a unique
solution to the question: where is the corresponding point in the second
image of a given point in the first image?

• disparity: difference in image position of the same space pt
• disparity map: dense pixel-to-pixel corrrespondences
• stereo rectification: make the epipolar lines horizontal
an option to speed up the computation of disparity map

21
Rectification of a stereo pair of images:
two images are transformed (by a projective transformation
in image plane or by a camera rotation around the center).

, uT  u
ˆ       uT  u
ˆ

22
New rectified image plane

• equivalent to a plane parallel to the base line
• T and T’ can be computed from F, but many possibilities
• only an option, simplify the computation!                 23
Matching by correlation:
 nim  N
nd 2 ))n  p  p(I  )n  p(I(
or
 xam  N
nd)n  p  p(I)n  p(I

Very often ZNCC (Zero Normalized Cross Correlation),
on normalised images instead of I and I’,
| n || n |
   )1n 2 ()1n 2 ()I  I(  n , )1n 2 ()1n 2 () I  I(  n
n  n                                                              
 

Convert all (2n+1)(2n+1)
elements from a matrix into a
)1 n 2 ()1 n 2 (                  vector of dim (2n+1)(2n+1)
) I  I(
n
                            24
Two points u and u’ are in correspondence if
• ZNCC(u,u’) is big enoug (close to 1)
• dist(u’, Fu) is small enough (a few pixels)
25
Correspondence by correlation:

• For each point u, compute all correlations in a neighborhood u+d with
a window size s
• Take the pixel having the highest correlation score as the
correspondence
• Cross-validate the correspondence in the opposite direction from the
second to the first image
neighborhood

Correlation window

Cross-validate

26
When applied to ‘interest points’, sparse correspondence
When applied to every pixel, dense disparity map

27
Using more cameras to remove match ambiguity: a system of 3 cameras

1

3

2

28
What can we do more with F?
• Without calibration, what can we get?
• When calibrated, essential matrix, its
decomposition

29
30
Calibrated E
KF TK  E ,0  xE Tx
0  x) KF TK( Tx
0  )xK(F T)xK(
x K  u ,xK  u
0  uF Tu
From uncalibrated F
Essential matrix:
fundamental matrix for the calibrated points

Relationship between E and F

E = [t] R       From F = [a] A

Decomposition of the essential matrix into R and t

The extra algebraic constraint: the equal singular values
(more complicated) for E
31
Decomposition of E
E  USV T ,             E  [t]R  SR

 0       1    0            0     1    0
                                        
S0     1      0    0    R0     1     0     0
 0            0            0           1
         0                       0      

Two factorisation       S  US 0 U T
R  UR 0
(T)
VT   Twisted pair

Two translation     t  ker(S)  u 3

32
33
Summary of modern two-view approach:

• Given internal calibration K and K’
(more advanced studies allow us to remove this step by
self-calibration that we will not handle)
•   Compute F from point correspondences
•   Compute E
•   Decompose E to obtain R and t
•   Obtain P and P’
•   Triangulation

34
Never forget the coplanar case!

x

u
u’

O
O
When space points are planar, a homography relating u and u’
It is therefore a ‘collineation’ for COPLANAR points!        35
From P=(I 0) , P’=(A a) and a known plane p^Tx=0,
to get H
H  A  an ,      T
p  (n,1)
So that     u  Hu
P  x  (u,  )  u  Au  a , pT x  nT u    0

Or, the homography can be computed from at least 4 corresponding
Points, do it!

The homography uniquely determines point correspondences
Unlike the fundamental matrix! But only for coplanar points.

36
This homography leads to one important application:

Panoramic image or
image mosaicing
• the 3D scene is planar
• the camera is rotating around the center (similar to rectification)

Example at HK airport (virtual tour),
QuicktimeVR
Realviz, stitcher, step-by-step
http://iris.usc.edu/home/iris/elkang/iris-04/reports/2/techreport2.html

37
The images are related by a homography if the 3d scene is planar:

38
Rotating the camera around the center is equivalent to a homographical
Transformation of the image plane:

x

A pencil of lines cut by two lines

A star of lines cut by two planes
39
Compositing algorithm or mosacing algorithm:

• Compute point correspondences
• Compute the projective transformation
between the two views
• Warp the first image onto the second
• Color-blend the overlapping areas

40
Quick-time VR

Outward-looking large-scale environnement
Inward-looking small object

Example of the virtual tour of HK airport:
http://www.hkairport.com/eng/index.jsp                      41
Automatic computation of the
fundamental matrix

Chicken-egg problem: we need corresponding points to compute F,
we need F to establish correspondences …

Simultaneous automatic computation of correspondences and F

42
Illustrative example of fitting a line to a set of 2D points

• the least squares solution (orthogonal regression)
is optimal when no outliers
• but it is becoming very fragile to outliers

43
Robust line fitting
Fit a line to 2D data containing ‘bad points’---outliers

Solving two pbs: 1. A line fit to the data;
2. A classification of the data into ‘inliers’
and ‘outliers’
‘inliers’: valid or good data satisfying the ‘line’ model
‘outliers’: bad data not satisfying the model
44
How to find the best line?

repeating

• randomly draw 2 data points
• compute a line Li from these 2 points
• compute the distance to the line Li for each data point
• determine inliers/outliers by a threshold t
• compute the number of inliers Si

then
• select the Li having the largest Si
• re-estimate the final line using all inliers

45
Fischler and Bolles 1981    RANSAC (random sample consensus)

repeating
• randomly draw a sample of s data
• initiate the model Mi
• compute the distance to the model for each data pt
• determine inliers/outliers by the threshold t
• compute the size of inliers Si
then
• select the Mi having the largest Si
• re-estimate using all inliers

46
The complete algorithm of automatic computation of F:

• detect points of interest in each image
• compute the correspondences using correlation based method
• RANSAC using 7-pt algo.
• (non-linear optimal estimation on the final inliers:
this is unnecessary in many cases, so just an option)

47
RANSAC using 7-pt algo to compute F

repeating
• randomly draw a sample of 7 corresponding points
• compute Fi
• compute the distance to Fi for each corresponding pt
• determine inliers/outliers by the threshold t
• compute the size of inliers Si
then
• select the Fi having the largest Si
• re-estimate the final F using all inliers

48
How many times to repeat?

N  log( 1  p) / log( 1  (1   ) )s

p   Probability of success (only inliers)
   Outlier proportion
s    Sample size
Ex: 99.9% success rate, 50% outliers
s=2, N=17
s=4, N=72
s=6, N=293
s=7, N=588
49
s=8, N=1177
Robust statistics

From least-squares method to robust statistics
(Ransac,least median of squares (LMS))

Handle ‘big errors’---outliers!

This is useful not only for computing F, but also for automatically
computing a mosaicing of images!
50
What tells us the F, the epipolar
geometry?

51
Summary of ‘semi-modern’ two-view approach:

• Given internal calibration K and K’
(more advanced studies allow us to remove this step by
self-calibration that we will not handle)
•   Compute F from point correspondences
•   Compute E
•   Decompose E to obtain R and t
•   Obtain P and P’
•   Triangulation           What do we obtain?
52
What is a projective reconstruction?

A 3D reconstruction up to a 3D projective transformation.

What is a Euclidean (similarity) reconstruction?

A 3D reconstruction up to a 3D Euclidean transformation.

53
Projective reconstruction without
calibration
F  P  (I,0), P  ([e] F, e)

1. This is not unique, as for any v and lambda, we have
P  ([e] F  ev T , e)

2. Or the algebraic approach by epipolar geometry
(cf. Faugeras’92 ECCV,
What can be seen from an uncalibrated stereo rig?)

3. The bottome line of numerical schema is the true ‘optimal’ of F
54
Self-calibration
Projective                       metric reconstruction
(uncalibrated)
from only point correspondences using geometric self-
consistency constraints

The original idea of Maybank&Faugeras91
key components: absolute conic, fundamental matrix and Kruppa equation

Generally at least 3 images for self-calibration
with constraints such as K=K’=K’’
55

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 0 posted: 10/12/2011 language: English pages: 55
How are you planning on using Docstoc?