# Image processing and computer vision

Document Sample

```					  Image processing
and computer vision

Stereo vision

Stereo v.1b8                   1
Stereo

   Objectives:
   Basic idea of stereo vision
   Stereo reconstruction by epipolar geometry
   Stereo camera pair calibration (find Fundamental
matrix F)
   Construct the 3D (graphic) model from 2 images
Inside a computer

Graphic
Stereo v.1b8
model              2
Basic idea of
stereo vision

Stereo v.1b8            3
A simple stereo system
Assume cameras are aligned horizontally
Left     Right                         Object
(No vertical disparity)
Camera      Camera Px(x,y,z)
A point in 3D (Px)   Principle   Principle
axis        axis


z
Left image               Right image
plane                    plane
Left                                  Right
Image      xL               xR        Image
plane                                 plane
Focal
Left                   Right                                                   Length
Stereo                                                                 f
optical Baseline (B)   optical
Center                 Center                        B (Baseline)
O(left)                O(right)            Left camera center
Horizontal
(reference                                 (reference point)
Stereo v.1b8
Disparity=xL-xR   4
point)
Example
Assume cameras are aligned horizontally
(No vertical disparity)
Left image                Right image

                                                              White crosses are
overlaid on images
to show the
positions
of features.
They are not
in the original
pictures.
A corner feature is found
in a 10x10 window (w)       Find the correspondence at (xR,yR)
centered at the             Horizontal disparity=XL-XR
left image (xL,yL)
(overlay a cross)

The stereo sequence video : http://www.youtube.com/watch?v=Ursyjokuils
Stereo v.1b8                             5
Triangular calculation

Left        Right     Object
Camera      Camera Px(x,y,z)
Principle   Principle                     By similar triangle,
axis        axis
                                                  w.r.t left camera lens center
x xl'
z               = ,
z f
( x - b) x'r
=
Left                               Right               z     f
Image    X’l             X’r       Image
f b
plane                              plane
elimate x  z =
Focal                         ( xl' - x'r )
Length
f
b (Baseline)                       By similar triangle,
Left camera center
Horizontal              w.r.t right camera lens center
(reference point)
Disparity=xL-xR
Stereo v.1b8                                6

One major problem is to locate x’l and x’r The correspondence problem
Stereo vision example:
step1:feature extraction:
for the left and right image find features and locate
their windows



Left image                         right image
Features are shown by overlaid markers on images
Stereo v.1b8                 7
Stereo vision example:
step2: Correspondence problem example:
Find correspondence of f1 in the right image and
determine which is the match



Left image                                      Right image
f1:a small window               f2               f3

r1,2=Cross correlate                   r1,3=Cross correlate
f1 with f2             Stereo v.1b8
f1 with f3                 8
2D-2D Correspondence method using cross-
correlation
   Cross correlation coefficient (rf,f’) for 2 windows f
and f’ in image frame t and frame t’, respectively.
f and f’ have the same size s. It is a measure of
similarity(from 1 to -1): 1 = very similar, -1 very
un-similar.
f
xys
x, y         
 f f x, y  f       
rf , f                                                         ;
                                               
                                        
                    2                        2 
      f x, y  f                  fx, y  f  
 xys
                        xys                   

Where f , f ' are the mean values of matrix f and f ' resp.
Stereo v.1b8                   9
Which one is larger, rf,f' or rf,f''? (0.979,0.9480 resp.)
Is f' or f' ' more similar to f? Try also to calulate rf,f.

f
xys
x, y         
 f f x, y  f       
rf , f                                                          ;
                                                
                                         
                      2                        2
      f x, y  f                  f x, y  f  
 xys
                          xys                  

Where f , f' are the mean values of matrix f and f ' resp.

 1 3 5       1 3 5                         1 3 5
f  2 6 2, f   2 6 1, rf , f   ? ; f ' '  2 5 1, rf , f ''  ?
                                                
 3 7 1
             2 8 1
                              2 9 1
     
Stereo v.1b8                           10
f          x, y          
 f f x' , y  f '       
 1 3 5       1 3 5                         1 3 5                    rf , f '        x , ys

f  2 6 2, f   2 6 1, rf , f   ? ; f ' '  2 5 1, rf , f ''  ?
                                                                                 f
x , ys
x, y    f        f
2

x , ys
'
x, y    f'   
2

 3 7 1
             2 8 1
                              2 9 1
                         s  all _ range,
f  mean ( f ), f '  mean ( f ' )
    Calculation steps
f 
f'
f x, y  f 
f 'x, y  f ' 

f
x , ys
x, y                  
 f f ' x, y  f ' 

f                      2
x, y   f           
x , ys

 f  f  
2
'          '
x, y
x , ys

Stereo v.1b8                                                                           11
Exercise at home

1 4 6        2 4 6          1 5 5      %Answer by
matab

f  3 2 3, f '   3 7 3, f "  2 4 2
   F1=[
   146

                                 
   323
   1 1 2]

1 1 2
             6 1 2
              2 2 3
         

F2=[
246
   373
6 1 2]
compare                                       

   F3=[
   155
correlate( f , f ' )                          

242
2 2 3]

and                                              >> corr2(F1,F2)

   ans =

correlate( f , f " )                               0.3597

   >> corr2(F1,F3)

   ans =

     0.7513

Stereo v.1b8                              12
For stereo vision with
horizontal camera motion only.
   Matching in 2D space becomes 1D
A B        C in 3D space

A        B     C                       A’       B’     C’
Left image scan line                   Right image scan line

   For A, find rA,A’ rA,B’ rA,C’ and see which is the
biggest and determine the correspondence
Stereo v.1b8                           13
Stereo reconstruction by
epipolar geometry
   Stereo camera pair calibration : Find Fundamental
matrix (F)
   Find 2-D to 2-D feature correspondences by Cross-
correlation
   From correspondences find F
   3D model reconstruction from stereo cameras after
stereo calibration
   Step 1: Rectification of stereo images
   Step 2: Find 2-D feature correspondences (by F, and
rectification)
   Step3: 3-D Model reconstruction

Stereo v.1b8                   14
Stereo camera pair
calibration

Find Essential matrix E and
Fundamental matrix F

Stereo v.1b8       15
The stereo setup
 r11 r12 r13 
R  r21 r22 r23 
              
         r31 r32 r33 
              
cos( ) cos(  ) cos( ) sin(  ) sin(  )  sin( ) cos( )      cos( ) sin(  )  sin( ) sin(  ) 
  sin( ) cos(  ) sin( ) sin(  ) sin(  )  cos( ) cos( ) sin( ) sin(  ) cos( )  cos( ) sin(  )
                                                                                                         
  sin(  )
                               cos(  ) sin(  )                           cos(  ) cos( )              

T  t1 t2       t3 
T                                                                              Y’
y
z                                            Z’

α (yaw)                               β pitch)

x                                                                              X’
Reference (world center)                                                      γ (roll)

Right camera (R,T)
//A yaw is a counterclockwise rotation (α ) about the z-axis.
//A pitch is a counterclockwise rotation (β) about the y-axis.
//A roll is a counterclockwise rotation (γ ) about the x-axis.
Stereo v.1b8                                                                  16
Yaw pitch roll


Stereo v.1b8   17
http://en.wikipedia.org/wiki/Yaw,_pitch,_and_roll
Epipolar Geometry
X


Left side is the reference
Plane-3 3
Perpendicular to        (x2,y2) Right epipolar line
TX2 or TX1
Left epipolar line
(x1,y1)
left                                                              right Frame
Frame                                                             Plane-2 2
Plane-1 1
O2
O         e1                                             e2
1
R,T
Focal                               Base line=||T||              Focal
Stereo v.1b8                            18

length=f1                                                        length=f2
Essential matrix E (a 3x3 matrix) P.110[2] X1 is 3-D X in left
camera (reference) system X2 is 3-D X in right camera system

Exercise1:
Draw vectors TX2 or TX1 in the diagram

 2  R1  T , multiply both sides by T 
T    T  R      (i )
 2        1

Since [TA ]x TB  TA  TB and
T   skew symmetric matrix and T  T  T  T  0
T and  2 are on the same plane,  2 is perpendicu lar to (T   2 )
so X 2 (T   2 )  0, same as X 2 (T   2 )  0
T                            T

from (i),  2 (T  R1 )  0                              t1     0     t3   t2 
T
        
[T ]x   t 2     t3
               0      t1 

hence  2 * E * 1  0
T

 t 3  
    x  t 2
              0  
where E  T  R
t1
[TA ]x TB  TA  TB

Stereo v.1b8                                        19
Essential Matrix E
X 2T * E * X1  0
 X 2 Y2    Z 2  * E *  X 1 Y1         Z1   0
T

f 2 f1
no _ harm _ to _ prefix _ some _ cons tan t _ terms ,
Z 2 Z1
f2                            f1
 X 2 Y2       Z2    * E *  X 1 Y1            Z1   0
T

Z2                            Z1
f 2  * E *  x1             f1   0
T
 x2
      y2                        y1
X        Y
since  X          Z
T
Y               X, and x  f , y  f
Z        Z
   Right_image_pointT*E*left_image_point=0

Stereo v.1b8              20
Fundamental matrix F (a 3x3 matrix) from E
to prove: F= (Mint_2-T*E*Mint_1-1)

    x2T*E*x1=0
   If both camera intrinsic parameters are known,
u=Mint*x, so x=(Mint)-1*u
   *[u1,v1,1]T, [u1,v1,1]T are correspondences points,
on left, right images respectively.
   Note: (a b)T=bT aT
   u2T*(Mint_2-T*E*Mint_1-1)*u1=0
   Pixel coordinate u=Mint*x, u=[ux ,uy ,1]T
   u2T*F*u1=0, for F= (Mint_2-T*E*Mint_1-1)
   Advantage of F: No need to know Mints for both
cameras at all if you work on pixel coordinates
Stereo v.1b8                    21
Epipolar lines
   u2T*(F*u1)=0, set L2=F*u1
   So u2T*(L2)=0,
   Meaning: A point u1 in the left image
 Has an epipolar line L2 in the right image (point line
relation)
 [a b c]T=(F*u1)=L2 is 3x1 vector

 The epipolar line equation in the right image is
au+bv+c=0
 The Line L2 passes through [u2, v2 ,1] in the right



Stereo v.1b8                   22
Epipolar lines

Stereo v.1b8   23
Left-point governs right epipolar
T
[a b c] =(F*u1)=L2 is 3x1 vector
line                  The epipolar line equation in
the right image is au+bv+c=0
X              The Line L2 passes through
                                             [u2, v2 ,1] in the right image
Left side is the reference
Right epipolar line L2
(u1,v1)

Left                                                          (u2,v2) right Frame
Frame                                                                 plane1 2
plane1 1
O2
Ol
Epipole e2
Focal         Epipole e1      F   Stereo v.1b8                         Focal       24

length=f1                                                              length=f2
Similarly: Right-point governs left epipolar line
[a’ b’ c’]T=(FT*u2)=L1 is 3x1 vector
The epipolar line equation in
the right image is a’u+b’v+c’=0
The Line L1 passes through
[u1, v1 ,1] in the left image      X


Left side is the reference
(u1,v1)
Left epipolar line L1
Left                                                         (u2,v2) Right Frame
Frame                                                                plane1 2
plane1 1
O2
Ol            e1                                           e2
Epipole e2
Focal            Epipole e1                                           Focal
FT Stereo v.1b8                            25

length=f1                                                             length=f2
Epipoles

Stereo v.1b8   26
Left-point governs right epipolar
T
[a b c] =(F*u1)=L2 is 3x1 vector
line                  The epipolar line equation in
the right image is au+bv+c=0
X              The Line L2 passes through
                                             [u2, v2 ,1] in the right image
Left side is the reference
Right epipolar line L2
(u1,v1)

Left                                                          (u2,v2) right Frame
Frame                                                                 plane1 2
plane1 1
O2
Ol
Epipole e2
Focal         Epipole e1      F   Stereo v.1b8                         Focal       27

length=f1                                                              length=f2
Example
stereo images with epipolar lines
The epipolar lines of each image will join at the epipole e (it
may be outside the image frame)



Left                                                Right

Stereo v.1b8           28
http://homepages.inf.ed.ac.uk/rbf/CVDICT/CVFIG3/img85.png
Epipoles
   For simplicity, use the notation x=(s*u,s*v,s)T as an image point
   For any point x1 in image1 (left image) , x2T*F*x1=0 (proved earlier)
   On the right: image2 ,
   Epipolar line equation L2=F*x1----(i)
   line L2 contains x2, so
   x2T*{L2}=0, from (i), so x2T*{F*x1}=0
   Or {x2T*F}*x1=0 for all x1, since not all x1 are 0
   So x2T*F=0, take transpose on both sides, note (ab)T=aTbT
   FT*x2=0, since F is fixed and not 0, there is only one solution (call it e2) : x2=e2
for all x1 , which is a fixed point in the right image.
   The point e2 is called the epipole, and FT*e2=0
   A point that all epipolar lines in the right image meet.

   Similarly if you use the right image as the reference you will get
   F*x1=0, (FT become F because the direction is reversed)
   The epipole on the right image is e1, and F*e1=0

Stereo v.1b8                                   29
Epipoles Matlab or scilab program for
testing the epioles

   To solve F*e1=0 use SVD, see appendix
   [uu1,ss1,vv1]= svd(F)
   e1 is the last column of vv1
   //epipoles e1,e2 //scilab test program
    [uu1,ss1,vv1]= svd(F)
    FT=F'
   [uu2,ss2,vv2]= svd(FT)
   e1=vv1(:,3) // epipole on the left image ---------------
   e1_u=e1(1)/e1(3)
   e1_v=e1(2)/e1(3)
   e2=vv2(:,3) // epipole on the right image ---------------
   e2_u=e2(1)/e2(3)
   e2_v=e2(2)/e2(3)
   //by defination e2’*F*e1=0 or e1’*FT*e2=0 ,
   //because e1, e2 are on Image 1, image 2 resp.
   (e2')*F*e1
   (e1')*FT*e2

Stereo v.1b8   30
A summary
   x2T*E*x1=0 (essential matrix, metric coordinates in meters)
   F= (Mint_2-T*E*Mint_1-1), cameras 1, 2 can be different
   u2T*(F*u1)=0( Fundamental matrix in pixel coordinates)
   set L2=F*u1 , so u2T*(L2)=0, point line relation
   //epipoles e1,e2
    [uu1,ss1,vv1]= svd(F)
    FT=F'
   [uu2,ss2,vv2]= svd(FT)
   e1=vv1(:,3) // epipole on the left image ---------------
   e1_u=e1(1)/e1(3)
   e1_v=e1(2)/e1(3)
   e2=vv2(:,3) // epipole on the right image ---------------
   e2_u=e2(1)/e2(3)
   e2_v=e2(2)/e2(3)
   //by definition e2’*F*e1=0 or e1’*FT*e2=0 ,
   //because e1, e2 are on Image 1, image 2 resp.
   (e2')*F*e1
   (e1')*FT*e2

Stereo v.1b8    31
Test using scilab
   //rotation angles, [0,5,1]
   //pixel width 5umx5um
   //focal length 6mm/5um=1200
   //Translation 10cm , 10*1000*1000 /5um= 10*1000*1000/5
   //a 3d point is at point is [0.01, 0.02 1]meters
   //Translation
   //t=[10*1000*1000/5 0 0]'
   t=[-20*1000*1000/5 0 0]'
   //a point in 3D space is
   //
   //xx=[0.01 0.02 1 ]*1000*1000*1000/5 //in ipxel
   xx=[0 0 1 ]*1000*1000*1000/5 //in ipixel

   x=[xx 1]'

   //angs=([0,15,1]*3.1416/180)
Stereo v.1b8              32
   angs=([0,0,0]*3.1416/180)
Matlab /scilab code: stereo_test.sce
(http://www.scilab.org/)
    //rotation angles, [0,5,1]                      cosAsinB = cosA .* sinB;
    //pixel width 5umx5um                           sinAsinB = sinA .* sinB;
    //focal length 6mm/5um=1200                 
    //Translation 10cm , 10*1000*1000 /5um=         R = [ cosA.*cosB cosAsinB.*sinC-
10*1000*1000/5                                   sinA.*cosC cosAsinB.*cosC+sinA.*sinC ;
    //a 3d point is at point is [0.01, 0.02             sinA.*cosB
1]meters
sinAsinB.*sinC+cosA.*cosC
    //Translation                                    sinAsinB.*cosC-cosA.*sinC ;
    //t=[10*1000*1000/5 0 0]'                            -sinB       cosB.*sinC
    t=[-20*1000*1000/5 0 0]'                         cosB.*cosC      ];
    //a point in 3D space is                         R
    //                                                                                        Result
     R'*R                                    -7->x1(1)/x1(3)
    //xx=[0.01 0.02 1 ]*1000*1000*1000/5 //in        mint =[1200        0       512
ipxel                                                                                     512.
        0     1200        384
    xx=[0 0 1 ]*1000*1000*1000/5 //in ipixel                                                   -7->x1(2)/x1(3)
         0       0        1]
    x1= mint*[1 0 0 0; 0 1 0 0; 0 0 1 1]*x   384.
    x=[xx 1]'                                       x1(1)/x1(3)                                -7-
    x1(2)/x1(3)                              >x2(1)/x2(3)
    //angs=([0,15,1]*3.1416/180)                                                               ans =
    angs=([0,0,0]*3.1416/180)                       mext=R*[[1 0 0; 0 1 0 ; 0 0 1] -t]           536.
    x2= mint*mext*x                           -7->x2(2)/x2(3)
    cosA = cos (angs(3));                           x2(1)/x2(3)
    sinA = sin (angs(3));                           x2(2)/x2(3)                              ans = 384.
    cosB = cos (angs(2));
    sinB = sin (angs(2));
Stereo v.1b8                                               33
    cosC = cos (angs(1));
    sinC = sin (angs(1));
Further test of E,F
   skew_t=[0 -t(3) t(2)
         t(3) 0 -t(1)
         -t(2) t(1) 0]
   E=skew_t*R
   F=inv(mint)*E*mint
   L2_1=F*[100 200 1]'
   [u s v]=svd(F)
   //epipoles e1,e2
    [uu1,ss1,vv1]= svd(F)
    FT=F'
   [uu2,ss2,vv2]= svd(FT)
   e1=vv1(:,3) // epipole on the left image ---------------
   e1_u=e1(1)/e1(3)
   e1_v=e1(2)/e1(3)
   e2=vv2(:,3) // epipole on the right image ---------------
   e2_u=e2(1)/e2(3)
   e2_v=e2(2)/e2(3)
   //by definition e2’*F*e1=0 or e1’*FT*e2=0 ,
   //because e1, e2 are on Image 1, image 2 resp.
   (e2')*F*e1
   (e1')*FT*e2

Stereo v.1b8   34
Result            skew_t ..
   skew_t =                                    vv1 =
      0. 0.       0.                             - 0.0000512 - 0.9273836 0.3741117
         0.9524304 0.1139676 0.2826440
      0. 0.       4000000.
         0.3047561 - 0.3563299 - 0.8832626
      0. - 4000000. 0.                          ss1 =
   E =                                             5.712D+09 0.        0.
      0.       0.     0.                           0.      1070927.1 0.
    - 1035278.5 0.       3863702.7                 0.      0.      0.0000001
      0.     - 4000000. 0.                      uu1 =
         0.3764731 - 0.1107978 0.9197782
   F =
         0.2823314 0.9593169 - 3.636D-14
      0.       2.048D+09 6.554D+08               - 0.8823588 0.2596823 0.3924387
    - 1035278.5 1.536D+09 4.911D+08       

      0.     - 4.800D+09 - 1.536D+09
   L2_1 =
     1.0D+11 *

     4.1025536
     3.0758755
   - 9.61536


Stereo v.1b8                                     35
Result e1,e2                                  

e1 =
0.3741117
         0.2826440
       - 0.8832626
   FT =                                            e1_u =
      0.    - 1035278.5 0.                          - 0.4235566
      2.048D+09 1.536D+09 - 4.800D+09              e1_v =
      6.554D+08 4.911D+08 - 1.536D+09               - 0.32
   vv2 =                                           e2 =
    - 0.3764731 0.1107978 0.9197782
         0.9197782
    - 0.2823314 - 0.9593169 - 3.282D-13
      0.8823588 - 0.2596823 0.3924387               - 3.282D-13
   ss2 =                                              0.3924387
      5.712D+09 0.        0.                       e2_u =
      0.      1070927.1 0.                            2.34375
      0.      0.      0.0000001                    e2_v =
   uu2 =                                            - 8.362D-13
      0.0000512 0.9273836 - 0.3741117
      ans =
    - 0.9524304 - 0.1139676 - 0.2826440
    - 0.3047561 0.3563299 0.8832626                   2.181D-08
      ans =
       - 1.956D-20

      Execution done.

Stereo v.1b8                36
Eight point algorithm for E or F (p279[1A], p263[1B])
left image=[x y 1]T, right image=[x’ y’ 1]T

        f11         f12     f13                                           For F, since Mint’s are not
F   f 21     f 22       f 23                                          required, for simplicity
                          
 f 31
          f 32       f 33 
                                         we set both focal
x'    y ' 1 * F * x y 1  0
T
lengths to 1
x ' xf11  x ' yf12  x ' f13  y ' xf 21 
y ' yf 22  y ' f 23  xf 31  yf 32  f 33  0
n  8
 x1 ' x1        x1 ' y1       x1 '      y1 ' x1        y1 ' y1     y1 '     x1     y1 1
AF '   :                 :          :            :             :          :     :        : :F '  0
                                                                                       
 xn ' x n
                xn ' y n     xn '      y n ' xn       yn ' yn      y n ' xn        yn 1

F '   f11                                                                       f 33 
T
f12        f13       f 21      f 22    f 23       f 31     f 32

Stereo v.1b8                                   37
Rank and Degree of freedom of
the Fundamental matrix F
   “F is a rank 2 homogeneous matrix with 7
degrees of freedom.” p.226 [1]
   “The degrees of freedom of the fundamental
matrix is 7, it can be counted as follows: 2 for
e, 2 for e’, and 3 for the epipolar line
homography which maps a line through e to a
line through e’.” p.227 section 8.2.5 [1]

Stereo v.1b8               38
Eight (or more) Point algorithm for Fundamental matrix
F , (p91[1A], 265[1B])

   Find Correspondences n= (8 or more) left-right
image feature points
   Rank of A is 8, at least 8 points are needed.
   From A (size nx9) from left side=[x y 1], right side=[x’
y’ 1] image features , use the method of SVD to find
f(9x1).
   AF’=0, and [U,S,V]=SVD(A)
   Estimated F’=last column of V, From F’ find F
   More accurate methods can be found in [1], see
appendix of the refined SVD method.

Stereo v.1b8                  39
How to make Fundamental
Matrix found more accurate?
   Data needed to be normalized to [-1,1] to
make the solution more stable.
   "RANdom SAmple Consensus" RANSAC to
find a better F.
http://en.wikipedia.org/wiki/RANSAC

Stereo v.1b8              40
3D model reconstruction from
stereo cameras after stereo
calibration
   Step 1: Rectification of stereo images
   Step 2: Find 2-D feature correspondences (by F,
and rectification)
   Step3: 3-D Model reconstruction

Stereo v.1b8                   41
Step 1 of 3-D Model reconstruction:
Rectification
   The two images can be rectified (transformed)
so that epipolar lines become horizontal.
   So that corresponding points can be found
more easily (1D search along horizontal
epipolar lines).
   More accurate corresponding points can be
found for dense field reconstruction

Stereo v.1b8            42
Application of rectification
 A point has the
correspondence
from a tiled                     Original
epipolar line                    pair
(difficult)

A point now has
a horizontal                       Rectified
epipolar line to                   pair
search for
correspondence
(easier)            Stereo v.1b8       43
Step 2 of 3-D Model reconstruction:
Find 2-D feature correspondences

by cross correlation

Stereo v.1b8         44
Search for correspondences
   Take left image as reference (I)
   Find all (N) features in the left and rigth
images.
   For each feature in the reference image (I)
   Find the corresponding epipolar line in the right
image (I’)
   Along the epipolar line (in image I’) search for the
correct correspondence by cross-correlation.
   Till all correspondences are found
Stereo v.1b8                      45
Recall                                               [a b c]T=(F*u1)=L2 is 3x1 vector
:each point in the reference (left)
The epipolar line equation in
image has an epipolar line in the
the right image is au+bv+c=0
right image.
The Line L2 passes through
X              [u2, v2 ,1] in the right

Left side is the reference
Right epipolar line
(u1,v1)                                                   L2

Left                                                                 (u2,v2) right Frame
Frame                                                                        plane1 2
plane1 1
O2
Ol
Epipole e2
Focal          Epipole e1           F    Stereo v.1b8                       Focal          46

length=f1                                                                   length=f2
Search correspondences
   From (x,y) in the left image
   Search (x’,y’)i in the right image along the
epipolar line (with tolerance)
   Use cross-correlation to measure the
likelihood (see next slide for one example)
(x,y)1                               L’=E*[x1,y1,f1]T       The gray
(x,y)2                                         area is
(x’,y’)i          the search
(x,y)i
region
Epipolar line L’i   With certain
of (x,y)i           tolerant level
(x,y)N

Left image (I)     Stereo v.1b8
Right image (I’)              47
Example: To find one pair of
correspondence (enlarged)
                                                             W’j=3
Search region
Above L’1

Epipolar line L’1

W’j=1              W’j=2
Search region
Wi                     below L’1
A 5x5 window                          3 5x5 windows. each is a
containing                            candidate window containing a feature.
Feature                 Cj=1=correlation (Wi,W’j=1)
In the reference        Cj=2=correlation (Wi,W’j=2)
(left) image I          Cj=3=correlation (Wi,W’j=3)
Select the biggest Cj, the index j is j’,
Wi is corresponding to window Wj’.
Stereo v.1b8                            48
Step 3 of 3-D Model reconstruction:
reconstruction

By triangulation

Stereo v.1b8       49
Procedures for Stereo
reconstruction of 3D model X
   So far
   The stereo camera pair is calibrated: Essential-matrix-E
(Fundamental-matrix-F) is found.
   All good 2-D correspondences are found
   Will discuss here
   Find Translation (T) and Rotation (R) between 2 Cameras
from E (or F) .
   From R,T calculate projection matrices P1(left) and P2 (right)
   3D reconstruction: from P1 and P2 find 3D points X by
triangulation

Stereo v.1b8                        50
From F or E find Rotation (R), Translation (T) between
2 Cameras
(p.258[1]hartley-ed2,p 239[1B])

   If camera intrinsic                               0  1 0          0 1 0
parameters are known,                       W  1 0 0, Z   1 0 0,
                      
E=(Mint2)T*F*(Mint1)                              0 0 1 
                 0 0 0
      
[U , S , V ]  SVD( E )
   E= [T]x *R,
   [ ]x=skew symmetric matrix                  [T ]x  UZU T
R  UWV T
   Decompose E using
SVD, using the fact that
  t1    0        t3   t2 
R is an orthogonal                                          
[T ]x   t 2     t3
                 0      t1 

matrix                                               t 3     t 2
   x             t1     0  

Stereo v.1b8                                                51
From R,T find projection matrices P1 (left) and P2
(right)(p.244[1A],p 224[1B])

    From F or E find R,T (previous slide)
    P1 =projection of the reference camera (left)

1 0 0 0
P( 3x 4 )
1           M int1 R1 | T1   M int1 I | 0  M int1 0 1 0 0
        
0 0 1 0 
        

P2( 3 x 4 )  M int 2 R2 | T2 ,
Stereo v.1b8                   52
Triangulation to find X

X is the point at a minimum
                            distance between two vectors
{O1,(x1,y1)} and {O2,(x2,y2)}

(x1,y1)
Left epipolar line
Left                                            (x2,y2)
Frame
plane1 1
O2
Ol          e1                               e2
Epipole e2
Focal          Epipole e1                               Focal
F v.1b8
length=f1                   Stereo
length=f2
53
3D reconstruction: from P1 and P2 find 3D
points X by triangulation (p.312[1A],p297[1B])
 p11   p12    p13    p14   P1T 
     x1  P , x2  P2  for 2 cameras (P ,P2 )
1                             1                 P(3 x 4 )     p21   p22    p23
      
p24    P 2T 
                         
x1  x1  x1  P   0                                               p31
       p32    p33    p44   P 3T 
         
1
where P iT  i th row of P
after_some_manipulations
i.e. P1T   p11       p12    p13    p14 ,
x ( p  )  ( p  )  0      (i )
3T          1T
 p11         p21          p31 
p                         
y ( p 3T  )  ( p 2T  )  0      (ii)          p 
1   12 , p 2   p22 , p 3   p32 
 p13         p23          p33 
                          
x ( p 2T  )  y ( p1T  )  0      (iii)             p14         p24          p34 

p iT  i th row of P, use (i) and (ii) for 2 cameras (P1,P2 )
 x1 p13T  p1T 
1

                
y1 p13T  p12T 
A                      0, solve by SVD [appendix 1]
 x2 p2  p2 
3T     1T

             2 
 y2 p2 T  p2 T 
3
Stereo v.1b8                                                    54
Summary of reconstruction

   From 2 images and correspondences, find
Fundamental matrix F
   From F find R,T
   From R,T find P2. (P1 on the left is known because
it is the reference R=I3, T=[0 0 0]T)
   For every point correspondence pair (x1,y1)i
(x2,x2)I, i=1,,,N
   Using P1, P2 and (x1,y1)i (x2,x2)i we can find the 3D point
Xi in the 3D space (reference of the camera center on left).
   The result is all N 3D points are found

Stereo v.1b8                       55
Overall Summary
   Learned
   Stereo computer vision and epipolar geometry
   Fundamental matrix, epipoles
   How to reconstruct 3D model points from 2 stereo
views

Stereo v.1b8                56
References
   [1A] Hartley and Zisserman, Multiple geometry in computer vision,
Cambridge, University Press. 2004 (2nd edition, paperback).
   [1B] Hartley and Zisserman, Multiple geometry in computer vision,
Cambridge, University Press. 1999 (1st edition, hardback).
   [2] CVonline on stereo
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/EPSRC_
SSAZ/node22.html
   [3] On Skew Symmetric matrix
http://mathworld.wolfram.com/SkewSymmetricMatrix.html
   [4] On Skew Symmetric matrix
http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/special.html#Skew_Symmetr
ic
   [5]On RQ decomposition
http://wwwcsif.cs.ucdavis.edu/~wangjj/gsvd/codes/rq5.m
   [6] Camera calibration matlab toolbox
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples
Stereo v.1b8                               57
Useful software tools
   [7] Blloyd , Fundamental Matrix computation
( http://www.cs.unc.edu/~blloyd/comp290-
089/fmatrix/ )
   [8] P.H.S. Torr, “A Structure and Motion Toolkit in
Matlab: Interactive adventures in S and M”
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_
COPIES/TORR1/index.html
   [9]
http://ai.stanford.edu/~mitul/cs223b/draw_epipolar.m

Stereo v.1b8                 58
Appendix

Stereo v.1b8   59
Appendix 1
Refined SVD method for Af=0
   In MATLAB, [U, S, V] = svd(A);
   S is a diagonal matrix with descending values down
the diagonal.
   f= the last column of V;
   [u,s,v]=svd(f)
   Set the smallest singular value in s=0, so s becomes
s
   Approximate f=usv’.
   More accurate iterative methods are in [1]

Stereo v.1b8                 60
Scilab stereo_test.sce
   //rotation angles, [0,5,1]
   //pixel width 5umx5um
   //focal length 6mm/5um=1200
   //Translation 10cm , 10*1000*1000 /5um= 10*1000*1000/5
   //a 3d point is at point is [0.01, 0.02 1]meters
   //Translation
   //t=[10*1000*1000/5 0 0]'
   t=[-20*1000*1000/5 0 0]'
   //a point in 3D spcae is
   //
   //xx=[0.01 0.02 1 ]*1000*1000*1000/5 //in ipxel
   xx=[0 0 1 ]*1000*1000*1000/5 //in ipixel

   x=[xx 1]'

   //angs=([0,15,1]*3.1416/180)
   angs=([0,15,0]*3.1416/180)

   cosA = cos (angs(3));
   sinA = sin (angs(3));
   cosB = cos (angs(2));
   sinB = sin (angs(2));
   cosC = cos (angs(1));
   sinC = sin (angs(1));

   cosAsinB = cosA .* sinB;
   sinAsinB = sinA .* sinB;

   R = [ cosA.*cosB cosAsinB.*sinC-sinA.*cosC cosAsinB.*cosC+sinA.*sinC ;
       sinA.*cosB sinAsinB.*sinC+cosA.*cosC sinAsinB.*cosC-cosA.*sinC ;
         -sinB        cosB.*sinC          cosB.*cosC    ];
    R
    R'*R
    mint =[1200        0       512
       0       1200      384
        0        0       1]
   x1= mint*[1 0 0 0; 0 1 0 0; 0 0 1 1]*x
   x1(1)/x1(3)
   x1(2)/x1(3)

   mext=R*[[1 0 0; 0 1 0 ; 0 0 1] -t]
   x2= mint*mext*x
   x2(1)/x2(3)
   x2(2)/x2(3)
   //pause

   skew_t=[0 -t(3) t(2)
          t(3) 0 -t(1)
          -t(2) t(1) 0]
   E=skew_t*R
   F=inv(mint)*E*mint
    L2_1=F*[100 200 1]'
    [u s v]=svd(F)
    //epipole
    //epipoles e1,e2
    [uu1,ss1,vv1]= svd(F)
    FT=F'
   [uu2,ss2,vv2]= svd(FT)
   e1=vv1(:,3) // epipole on the left image ---------------
   e1_u=e1(1)/e1(3)
   e1_v=e1(2)/e1(3)
   e2=vv2(:,3) // epipole on the right image ---------------
   e2_u=e2(1)/e2(3)
   e2_v=e2(2)/e2(3)
   //by defination e2’*F*e1=0 or e1’*FT*e2=0 ,
   //because e1, e2 are on Image 1, image 2 resp.
   (e2')*F*e1
   (e1')*FT*e2


Stereo v.1b8   61
Appendix 2
Matlab plot epipolar lines
   function draw_epipolar(left_x, left_y)
   %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   %
   % This function takes as argument the (x,y) coordinate of the LEFT
   % image and plots its corresponding epipolar line on the RIGHT and
   % the left image
   %
   %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

   % Hard coding the fundamental matrix FM

   FM=[ 0.0001 -0.0437 13.3897
      0.0429 0.0033 -21.4157
     -12.1830 22.0779 -999.3629];

   close all;

   [m n] = size(left_image);

   str1 = sprintf('Left Image point: %d %d',left_x,left_y);
   str2 = sprintf('Corresponding Epipolar Line on Right Image');

   figure,imagesc(left_image); colormap(gray); title(str1); axis image;
   figure,imagesc(right_image); colormap(gray); title(str2); axis image;

   % Start plotting:

   figure(1);
   hold on;
   plot(left_x, left_y, 'r*');

   % Getting the epipolar line on the RIGHT image:

   left_P = [left_x; left_y; 1];
   right_P = FM*left_P
   disp('t1')
   pause
   right_epipolar_x = 1:2*m; %//set all values of x in the range

   % Using the eqn of line: ax+by+c=0; y = (-c-ax)/b
   %c= a=right_p(1) a=right_p(2) a=right_p(3)
   right_epipolar_y = (-right_P(3)-right_P(1)*right_epipolar_x)/right_P(2);
   disp('t2')
   pause
   figure(2); hold on; plot(right_epipolar_x, right_epipolar_y, 'r');

   % Now finding the other epipolar line on the left image itself:

   % We know that left epipole is the 3rd column of V.
   % We get V from svd of F. F=UDV'
   [FU, FD, FV] = svd(FM);
   left_epipole = FV(:,3);
   left_epipole = left_epipole/left_epipole(3);

   % Hence using the left epipole and the given input point on left
   % image we plot the epipolar line on the left image

   %here plot left left_epipole
   left_epipolar_x = 1:2*m
   size(left_epipolar_x )
   disp('test2')
   m
   pause
   left_epipolar_y = left_y + (left_epipolar_x-left_x)*(left_epipole(2)-left_y)/(left_epipole(1)-left_x);

   figure(1); hold on; plot(left_epipolar_x, left_epipolar_y, 'r');

Stereo v.1b8   62
Appendix 2
Scilab
   http://www.scilab.org/
   A poor man’s MATLAB

Stereo v.1b8   63
Appendix 4:See how to factorize E
into [T]x R
   Given E=[T]xR, E is rank2
   Make up 2 matrices Z,W                 USVT=SVD(E), S=diag(1,1,0)
because E is rank2 (by
   Z=[0 1 0                                Definition)
      -1 0 0                              =USVT=U| 1 0 0 |VT
      0 0 0]                                         |010|
   W=[0 -1 0                                         |000|
       10 0                               We known S=ZW=[ 1 0 0
       0 0 1]                                                 010
   >> ZW=                                                     0 0 0]
      1 0 0                               E= U{S}VT=U{ZW}VT
      0 1 0                               ={UZUT}{UWVT}
      0 0 0
   = [T]x      R
   So both [T]x R satisfy their
own characteristics
http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/special.html
Stereo v.1b8                            64
Appendix 5
Solve Ax=0
   To solve Ax=0, Homogeneous systems
   One solution is x=0, but it is trivial and no use.
   We need another method, SVD (Singular value
decomposition)

Stereo v.1b8                    65
What is SVD?
If R is Orthogonal
   A is mxn, decompose it
into 3 matrces: U, S, V       R is a square matrix with real entries
columns and rows are orthogonal unit vectors
   U is mxm is an
orthogonal matrix              RT R  RR T  I and
   S is mxn (diagonal             RT  R 1
matrix)                        http : //en.wikip edia.org/w iki/Orthog onal_matri x
   V is nxn is an
A  U mm SmnVnTn
orthogonal matrix
 1                0
      2             
   1, 2, n, are singular                                            
values                                    S                .       
   Columns of vectors of                                               
                 .   
U=left singular vectors
0
                   n

   Columns of vectors of
V=right singular vectors   Stereo v.1b8
 1   2  ... n  0           66
SVD (singular value
decomposition)
Singular values
Left singular vectors                                  Right singular vectors
   SVD             …
 U1,1 . . . U1,m   1
T
0  V1,1     . . . V1,n 
 .      . . .        .   2               .       . . . . 
                                                              
 .      . . .        .        .           .       . . . .   svd ( Amn )
                                           .                   
   .    . . .        .            .              . . . . 
U m ,1 . . . U m ,m   0
                                      n  Vn ,1
         . . . Vn ,n 

 1   2  ...   n
Relation with eigen values
( AT A) x  x
eigen values of ( AT A) are 1  2  ..  n
 j   j j  1,,n

Stereo v.1b8                                     67
More properties
Meaning of  i is
Eigen values of AT A   12 ,  2 ,.., n .
2       2

Define
ui  Columns of vectors of U  left singular v ectors
U  u1 .. ui .. vm 
vi  Columns of vectors of V  right singular v ectors
V  v1 .. vi .. vn 
( AT A)ui   i2ui
( AAT )vi   i2vi
Stereo v.1b8                68
SVD for Homogeneous
systems
   To solve Ax=0,
   (   Homogeneous systems)
     One solution is x=0, but it             p  norm and 2  norm
is trivial and no usage.                                         1
n
p
p
     If we set ||x||2=1, the                 x          xi  , hence
 i   
p
solution will make sense.
     So we ask a different                              n
x2        
2
question: find min(||Ax||)                              xi
i
and subject to ||x||2=1.
     Note:||x||2 is 2-norm of x

Stereo v.1b8                                69
Minimize ||Ax|| subject to ||x||2=1
   (’)=transpose
   To minimize the 2-norm of ||Ax||2, since
A=USV’                                                   ** To show
   ||Ax||2=||USV’x||2= (USV’x)’ (USV’x) by                  ||x||2=||V’x||2
definition 2-norm: ||Y||2=Y’Y                            Proof:
   So ||Ax||2= (x’VS’U’)(USV’x) because
(ABC)’=C’B’A’                                            ||V’x||2=(V’x)’ (V’x)
   so||Ax||2=(x’VS’SV’x), since U is orthogonal             =x’V(V’x)
and U’U=1
   Since x’VS’= (SV’x)’ put back to above. So
   =x’x since VV’=I , (V is orthogonal)
||Ax||2=(SV’x)’(SV’x) =||SV’x||2                         =||x||2, done
   To minimize ||Ax||2 subject to ||x||=1                   To be continued
   Or minimize =||SV’x||2 subject to ||V’x||2
   =1 (see ** on the right)                                    If R is Orthogonal
   Set y=V’x
R T R  RR T  I and
   We now minimize ||Sy||2 subject to ||y||2=1
   Note: S is diagonal with descending entries                 R T  R 1
   The solution is ys=[0, 0, ..,0, 1]’ (reason:
||ys||2=1, and just ||Sys||=||S *[0 0 ..0 1]’ || is      1                   0
the smallest, the last element of s is the                  2                  
smallest n                                                                     
   Since V’x=y, so x=(V’)-1 y . V is orthogonal,         S                .       
(V’)-1=V                                                                        
                    .   
                                           V
Xsolution=V[0 0 0.. 1]’=last column of Stereo v.1b8     0                     n      70
                        

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 28 posted: 12/10/2011 language: English pages: 70