Docstoc

Image processing and computer vision

Document Sample
Image processing and computer vision Powered By Docstoc
					  Image processing
and computer vision

                        Stereo vision




         Stereo v.1b8                   1
Stereo

   Objectives:
       Basic idea of stereo vision
       Stereo reconstruction by epipolar geometry
           Stereo camera pair calibration (find Fundamental
            matrix F)
           Construct the 3D (graphic) model from 2 images
                                                  Inside a computer




                                                   Graphic
                                Stereo v.1b8
                                                   model              2
Basic idea of
           stereo vision




   Stereo v.1b8            3
 A simple stereo system
 Assume cameras are aligned horizontally
                             Left     Right                         Object
 (No vertical disparity)
                                              Camera      Camera Px(x,y,z)
                         A point in 3D (Px)   Principle   Principle
                                              axis        axis

 
                                                                             z
Left image               Right image
plane                    plane
                                       Left                                  Right
                                       Image      xL               xR        Image
                                       plane                                 plane
                                                                                 Focal
  Left                   Right                                                   Length
          Stereo                                                                 f
  optical Baseline (B)   optical
  Center                 Center                        B (Baseline)
  O(left)                O(right)            Left camera center
                                                                  Horizontal
  (reference                                 (reference point)
                                       Stereo v.1b8
                                                                  Disparity=xL-xR   4
  point)
Example
 Assume cameras are aligned horizontally
(No vertical disparity)
           Left image                Right image


                                                              White crosses are
                                                               overlaid on images
                                                               to show the
                                                               positions
                                                               of features.
                                                               They are not
                                                               in the original
                                                               pictures.
    A corner feature is found
    in a 10x10 window (w)       Find the correspondence at (xR,yR)
    centered at the             Horizontal disparity=XL-XR
    left image (xL,yL)
    (overlay a cross)

The stereo sequence video : http://www.youtube.com/watch?v=Ursyjokuils
                                    Stereo v.1b8                             5
    Triangular calculation

         Left        Right     Object
         Camera      Camera Px(x,y,z)
         Principle   Principle                     By similar triangle,
         axis        axis
                                                  w.r.t left camera lens center
                                                       x xl'
                                        z               = ,
                                                       z f
                                                       ( x - b) x'r
                                                               =
    Left                               Right               z     f
    Image    X’l             X’r       Image
                                                                            f b
    plane                              plane
                                                       elimate x  z =
                                           Focal                         ( xl' - x'r )
                                           Length
                                           f
                b (Baseline)                       By similar triangle,
      Left camera center
                           Horizontal              w.r.t right camera lens center
      (reference point)
                           Disparity=xL-xR
                                    Stereo v.1b8                                6

One major problem is to locate x’l and x’r The correspondence problem
Stereo vision example:
step1:feature extraction:
for the left and right image find features and locate
their windows






        Left image                         right image
    Features are shown by overlaid markers on images
                            Stereo v.1b8                 7
Stereo vision example:
step2: Correspondence problem example:
Find correspondence of f1 in the right image and
determine which is the match






            Left image                                      Right image
          f1:a small window               f2               f3



        r1,2=Cross correlate                   r1,3=Cross correlate
        f1 with f2             Stereo v.1b8
                                               f1 with f3                 8
2D-2D Correspondence method using cross-
correlation
    Cross correlation coefficient (rf,f’) for 2 windows f
     and f’ in image frame t and frame t’, respectively.
     f and f’ have the same size s. It is a measure of
     similarity(from 1 to -1): 1 = very similar, -1 very
     un-similar.
                  f
                  xys
                         x, y         
                                 f f x, y  f       
rf , f                                                         ;
                                                            
                                                      
                                 2                        2 
                   f x, y  f                  fx, y  f  
              xys
                                     xys                   
                                                             
Where f , f ' are the mean values of matrix f and f ' resp.
                                      Stereo v.1b8                   9
 Which one is larger, rf,f' or rf,f''? (0.979,0.9480 resp.)
 Is f' or f' ' more similar to f? Try also to calulate rf,f.

                  f
                   xys
                           x, y         
                                   f f x, y  f       
rf , f                                                          ;
                                                             
                                                        
                                   2                        2
                   f x, y  f                  f x, y  f  
              xys
                                       xys                  
                                                              
Where f , f' are the mean values of matrix f and f ' resp.

          1 3 5       1 3 5                         1 3 5
     f  2 6 2, f   2 6 1, rf , f   ? ; f ' '  2 5 1, rf , f ''  ?
                                                         
          3 7 1
                      2 8 1
                                                      2 9 1
                                                             
                                          Stereo v.1b8                           10
                                                                                               f          x, y          
                                                                                                                    f f x' , y  f '       
     1 3 5       1 3 5                         1 3 5                    rf , f '        x , ys


f  2 6 2, f   2 6 1, rf , f   ? ; f ' '  2 5 1, rf , f ''  ?
                                                                                     f
                                                                                           x , ys
                                                                                                     x, y    f        f
                                                                                                                      2

                                                                                                                          x , ys
                                                                                                                                     '
                                                                                                                                    x, y    f'   
                                                                                                                                                  2



     3 7 1
                 2 8 1
                                                 2 9 1
                                                                            s  all _ range,
                                                                              f  mean ( f ), f '  mean ( f ' )
       Calculation steps
           f 
           f'
           f x, y  f 
           f 'x, y  f ' 

           f
           x , ys
                     x, y                  
                             f f ' x, y  f ' 

           f                      2
                     x, y   f           
           x , ys


            f  f  
                                     2
                      '          '
                     x, y
           x , ys




                                                               Stereo v.1b8                                                                           11
 Exercise at home

    1 4 6        2 4 6          1 5 5      %Answer by
                                                  matab


f  3 2 3, f '   3 7 3, f "  2 4 2
                                                 F1=[
                                                 146

                                     
                                                 323
                                                 1 1 2]

    1 1 2
                 6 1 2
                                 2 2 3
                                            
                                              
                                                  F2=[
                                                  246
                                                 373
                                                  6 1 2]
compare                                       


                                                 F3=[
                                                 155
correlate( f , f ' )                          
                                              
                                                  242
                                                  2 2 3]


and                                              >> corr2(F1,F2)

                                                 ans =

correlate( f , f " )                               0.3597

                                                 >> corr2(F1,F3)

                                                 ans =

                                                   0.7513


                          Stereo v.1b8                              12
For stereo vision with
horizontal camera motion only.
   Matching in 2D space becomes 1D
                             A B        C in 3D space




      A        B     C                       A’       B’     C’
      Left image scan line                   Right image scan line

   For A, find rA,A’ rA,B’ rA,C’ and see which is the
    biggest and determine the correspondence
                              Stereo v.1b8                           13
Stereo reconstruction by
epipolar geometry
   Stereo camera pair calibration : Find Fundamental
    matrix (F)
       Find 2-D to 2-D feature correspondences by Cross-
        correlation
       From correspondences find F
   3D model reconstruction from stereo cameras after
    stereo calibration
       Step 1: Rectification of stereo images
       Step 2: Find 2-D feature correspondences (by F, and
        rectification)
       Step3: 3-D Model reconstruction

                               Stereo v.1b8                   14
Stereo camera pair
        calibration


Find Essential matrix E and
     Fundamental matrix F




           Stereo v.1b8       15
The stereo setup
          r11 r12 r13 
    R  r21 r22 r23 
                       
         r31 r32 r33 
                       
      cos( ) cos(  ) cos( ) sin(  ) sin(  )  sin( ) cos( )      cos( ) sin(  )  sin( ) sin(  ) 
      sin( ) cos(  ) sin( ) sin(  ) sin(  )  cos( ) cos( ) sin( ) sin(  ) cos( )  cos( ) sin(  )
                                                                                                               
        sin(  )
                                     cos(  ) sin(  )                           cos(  ) cos( )              
                                                                                                                
    T  t1 t2       t3 
                        T                                                                              Y’
                 y
                            z                                            Z’

                                                                   α (yaw)                               β pitch)


x                                                                              X’
              Reference (world center)                                                      γ (roll)

                                                   Right camera (R,T)
      //A yaw is a counterclockwise rotation (α ) about the z-axis.
      //A pitch is a counterclockwise rotation (β) about the y-axis.
      //A roll is a counterclockwise rotation (γ ) about the x-axis.
                                      Stereo v.1b8                                                                  16
   Yaw pitch roll
   




                                        Stereo v.1b8   17
http://en.wikipedia.org/wiki/Yaw,_pitch,_and_roll
   Epipolar Geometry
                                    X
   

 Left side is the reference
                              Plane-3 3
                              Perpendicular to        (x2,y2) Right epipolar line
                              TX2 or TX1
  Left epipolar line
                        (x1,y1)
left                                                              right Frame
Frame                                                             Plane-2 2
Plane-1 1
                                                                   O2
   O         e1                                             e2
   1
                                  R,T
Focal                               Base line=||T||              Focal
                                     Stereo v.1b8                            18

length=f1                                                        length=f2
Essential matrix E (a 3x3 matrix) P.110[2] X1 is 3-D X in left
camera (reference) system X2 is 3-D X in right camera system

 Exercise1:
 Draw vectors TX2 or TX1 in the diagram

   2  R1  T , multiply both sides by T 
T    T  R      (i )
      2        1

  Since [TA ]x TB  TA  TB and
  T   skew symmetric matrix and T  T  T  T  0
  T and  2 are on the same plane,  2 is perpendicu lar to (T   2 )
  so X 2 (T   2 )  0, same as X 2 (T   2 )  0
        T                            T


  from (i),  2 (T  R1 )  0                              t1     0     t3   t2 
                 T
                                                                    
                                                   [T ]x   t 2     t3
                                                                             0      t1 
                                                                                           
  hence  2 * E * 1  0
            T

                                                             t 3  
                                                                x  t 2
                                                                                      0  
  where E  T  R
                                                                                t1
                                                   [TA ]x TB  TA  TB

                                    Stereo v.1b8                                        19
Essential Matrix E
     X 2T * E * X1  0
      X 2 Y2    Z 2  * E *  X 1 Y1         Z1   0
                                                   T


                                                        f 2 f1
     no _ harm _ to _ prefix _ some _ cons tan t _ terms ,
                                                        Z 2 Z1
     f2                            f1
         X 2 Y2       Z2    * E *  X 1 Y1            Z1   0
                                                          T

     Z2                            Z1
                 f 2  * E *  x1             f1   0
                                                   T
      x2
           y2                        y1
                                                X        Y
     since  X          Z
                              T
                   Y               X, and x  f , y  f
                                                Z        Z
   Right_image_pointT*E*left_image_point=0

                                         Stereo v.1b8              20
Fundamental matrix F (a 3x3 matrix) from E
to prove: F= (Mint_2-T*E*Mint_1-1)

    x2T*E*x1=0
   If both camera intrinsic parameters are known,
    u=Mint*x, so x=(Mint)-1*u
   *[u1,v1,1]T, [u1,v1,1]T are correspondences points,
    on left, right images respectively.
   Note: (a b)T=bT aT
   u2T*(Mint_2-T*E*Mint_1-1)*u1=0
   Pixel coordinate u=Mint*x, u=[ux ,uy ,1]T
   u2T*F*u1=0, for F= (Mint_2-T*E*Mint_1-1)
   Advantage of F: No need to know Mints for both
    cameras at all if you work on pixel coordinates
                          Stereo v.1b8                    21
Epipolar lines
   u2T*(F*u1)=0, set L2=F*u1
   So u2T*(L2)=0,
   Meaning: A point u1 in the left image
     Has an epipolar line L2 in the right image (point line
       relation)
     [a b c]T=(F*u1)=L2 is 3x1 vector

     The epipolar line equation in the right image is
       au+bv+c=0
     The Line L2 passes through [u2, v2 ,1] in the right



                            Stereo v.1b8                   22
Epipolar lines




    Stereo v.1b8   23
   Left-point governs right epipolar
                                                      T
                         [a b c] =(F*u1)=L2 is 3x1 vector
   line                  The epipolar line equation in
                                                 the right image is au+bv+c=0
                                  X              The Line L2 passes through
                                                [u2, v2 ,1] in the right image
 Left side is the reference
                                                              Right epipolar line L2
            (u1,v1)


Left                                                          (u2,v2) right Frame
Frame                                                                 plane1 2
plane1 1
                                                                        O2
   Ol
                                                          Epipole e2
Focal         Epipole e1      F   Stereo v.1b8                         Focal       24

length=f1                                                              length=f2
   Similarly: Right-point governs left epipolar line
        [a’ b’ c’]T=(FT*u2)=L1 is 3x1 vector
        The epipolar line equation in
        the right image is a’u+b’v+c’=0
        The Line L1 passes through
        [u1, v1 ,1] in the left image      X
   

 Left side is the reference
          (u1,v1)
  Left epipolar line L1
Left                                                         (u2,v2) Right Frame
Frame                                                                plane1 2
plane1 1
                                                                       O2
   Ol            e1                                           e2
                                                         Epipole e2
Focal            Epipole e1                                           Focal
                                       FT Stereo v.1b8                            25

length=f1                                                             length=f2
    Epipoles




Stereo v.1b8   26
   Left-point governs right epipolar
                                                      T
                         [a b c] =(F*u1)=L2 is 3x1 vector
   line                  The epipolar line equation in
                                                 the right image is au+bv+c=0
                                  X              The Line L2 passes through
                                                [u2, v2 ,1] in the right image
 Left side is the reference
                                                              Right epipolar line L2
            (u1,v1)


Left                                                          (u2,v2) right Frame
Frame                                                                 plane1 2
plane1 1
                                                                        O2
   Ol
                                                          Epipole e2
Focal         Epipole e1      F   Stereo v.1b8                         Focal       27

length=f1                                                              length=f2
Example
stereo images with epipolar lines
The epipolar lines of each image will join at the epipole e (it
may be outside the image frame)







                        Left                                                Right


                                                             Stereo v.1b8           28
 http://homepages.inf.ed.ac.uk/rbf/CVDICT/CVFIG3/img85.png
Epipoles
   For simplicity, use the notation x=(s*u,s*v,s)T as an image point
   For any point x1 in image1 (left image) , x2T*F*x1=0 (proved earlier)
   On the right: image2 ,
   Epipolar line equation L2=F*x1----(i)
   line L2 contains x2, so
   x2T*{L2}=0, from (i), so x2T*{F*x1}=0
   Or {x2T*F}*x1=0 for all x1, since not all x1 are 0
   So x2T*F=0, take transpose on both sides, note (ab)T=aTbT
   FT*x2=0, since F is fixed and not 0, there is only one solution (call it e2) : x2=e2
    for all x1 , which is a fixed point in the right image.
   The point e2 is called the epipole, and FT*e2=0
   A point that all epipolar lines in the right image meet.

   Similarly if you use the right image as the reference you will get
   F*x1=0, (FT become F because the direction is reversed)
   The epipole on the right image is e1, and F*e1=0


                                        Stereo v.1b8                                   29
Epipoles Matlab or scilab program for
testing the epioles

   To solve F*e1=0 use SVD, see appendix
       [uu1,ss1,vv1]= svd(F)
       e1 is the last column of vv1
   //epipoles e1,e2 //scilab test program
        [uu1,ss1,vv1]= svd(F)
        FT=F'
       [uu2,ss2,vv2]= svd(FT)
       e1=vv1(:,3) // epipole on the left image ---------------
       e1_u=e1(1)/e1(3)
       e1_v=e1(2)/e1(3)
       e2=vv2(:,3) // epipole on the right image ---------------
       e2_u=e2(1)/e2(3)
       e2_v=e2(2)/e2(3)
       //by defination e2’*F*e1=0 or e1’*FT*e2=0 ,
       //because e1, e2 are on Image 1, image 2 resp.
       (e2')*F*e1
       (e1')*FT*e2




                                                        Stereo v.1b8   30
A summary
   x2T*E*x1=0 (essential matrix, metric coordinates in meters)
   F= (Mint_2-T*E*Mint_1-1), cameras 1, 2 can be different
   u2T*(F*u1)=0( Fundamental matrix in pixel coordinates)
   set L2=F*u1 , so u2T*(L2)=0, point line relation
   //epipoles e1,e2
        [uu1,ss1,vv1]= svd(F)
        FT=F'
       [uu2,ss2,vv2]= svd(FT)
       e1=vv1(:,3) // epipole on the left image ---------------
       e1_u=e1(1)/e1(3)
       e1_v=e1(2)/e1(3)
       e2=vv2(:,3) // epipole on the right image ---------------
       e2_u=e2(1)/e2(3)
       e2_v=e2(2)/e2(3)
       //by definition e2’*F*e1=0 or e1’*FT*e2=0 ,
       //because e1, e2 are on Image 1, image 2 resp.
       (e2')*F*e1
       (e1')*FT*e2



                                                    Stereo v.1b8    31
Test using scilab
   //rotation angles, [0,5,1]
   //pixel width 5umx5um
   //focal length 6mm/5um=1200
   //Translation 10cm , 10*1000*1000 /5um= 10*1000*1000/5
   //a 3d point is at point is [0.01, 0.02 1]meters
   //Translation
   //t=[10*1000*1000/5 0 0]'
   t=[-20*1000*1000/5 0 0]'
   //a point in 3D space is
   //
   //xx=[0.01 0.02 1 ]*1000*1000*1000/5 //in ipxel
   xx=[0 0 1 ]*1000*1000*1000/5 //in ipixel

   x=[xx 1]'

   //angs=([0,15,1]*3.1416/180)
                                   Stereo v.1b8              32
   angs=([0,0,0]*3.1416/180)
    Matlab /scilab code: stereo_test.sce
    (http://www.scilab.org/)
    //rotation angles, [0,5,1]                      cosAsinB = cosA .* sinB;
    //pixel width 5umx5um                           sinAsinB = sinA .* sinB;
    //focal length 6mm/5um=1200                 
    //Translation 10cm , 10*1000*1000 /5um=         R = [ cosA.*cosB cosAsinB.*sinC-
     10*1000*1000/5                                   sinA.*cosC cosAsinB.*cosC+sinA.*sinC ;
    //a 3d point is at point is [0.01, 0.02             sinA.*cosB
     1]meters
                                                      sinAsinB.*sinC+cosA.*cosC
    //Translation                                    sinAsinB.*cosC-cosA.*sinC ;
    //t=[10*1000*1000/5 0 0]'                            -sinB       cosB.*sinC
    t=[-20*1000*1000/5 0 0]'                         cosB.*cosC      ];
    //a point in 3D space is                         R
    //                                                                                        Result
                                                      R'*R                                    -7->x1(1)/x1(3)
    //xx=[0.01 0.02 1 ]*1000*1000*1000/5 //in        mint =[1200        0       512
     ipxel                                                                                     512.
                                                         0     1200        384
    xx=[0 0 1 ]*1000*1000*1000/5 //in ipixel                                                   -7->x1(2)/x1(3)
                                                          0       0        1]
                                                     x1= mint*[1 0 0 0; 0 1 0 0; 0 0 1 1]*x   384.
    x=[xx 1]'                                       x1(1)/x1(3)                                -7-
                                                     x1(2)/x1(3)                              >x2(1)/x2(3)
    //angs=([0,15,1]*3.1416/180)                                                               ans =
    angs=([0,0,0]*3.1416/180)                       mext=R*[[1 0 0; 0 1 0 ; 0 0 1] -t]           536.
                                                     x2= mint*mext*x                           -7->x2(2)/x2(3)
    cosA = cos (angs(3));                           x2(1)/x2(3)
    sinA = sin (angs(3));                           x2(2)/x2(3)                              ans = 384.
    cosB = cos (angs(2));
    sinB = sin (angs(2));
                                                 Stereo v.1b8                                               33
    cosC = cos (angs(1));
    sinC = sin (angs(1));
Further test of E,F
   skew_t=[0 -t(3) t(2)
         t(3) 0 -t(1)
         -t(2) t(1) 0]
   E=skew_t*R
   F=inv(mint)*E*mint
   L2_1=F*[100 200 1]'
   [u s v]=svd(F)
   //epipoles e1,e2
        [uu1,ss1,vv1]= svd(F)
        FT=F'
       [uu2,ss2,vv2]= svd(FT)
       e1=vv1(:,3) // epipole on the left image ---------------
       e1_u=e1(1)/e1(3)
       e1_v=e1(2)/e1(3)
       e2=vv2(:,3) // epipole on the right image ---------------
       e2_u=e2(1)/e2(3)
       e2_v=e2(2)/e2(3)
       //by definition e2’*F*e1=0 or e1’*FT*e2=0 ,
       //because e1, e2 are on Image 1, image 2 resp.
       (e2')*F*e1
       (e1')*FT*e2



                                                             Stereo v.1b8   34
Result            skew_t ..
   skew_t =                                    vv1 =
      0. 0.       0.                             - 0.0000512 - 0.9273836 0.3741117
                                                    0.9524304 0.1139676 0.2826440
      0. 0.       4000000.
                                                    0.3047561 - 0.3563299 - 0.8832626
      0. - 4000000. 0.                          ss1 =
   E =                                             5.712D+09 0.        0.
      0.       0.     0.                           0.      1070927.1 0.
    - 1035278.5 0.       3863702.7                 0.      0.      0.0000001
      0.     - 4000000. 0.                      uu1 =
                                                    0.3764731 - 0.1107978 0.9197782
   F =
                                                    0.2823314 0.9593169 - 3.636D-14
      0.       2.048D+09 6.554D+08               - 0.8823588 0.2596823 0.3924387
    - 1035278.5 1.536D+09 4.911D+08       


      0.     - 4.800D+09 - 1.536D+09
   L2_1 =
     1.0D+11 *

     4.1025536
     3.0758755
   - 9.61536





                                        Stereo v.1b8                                     35
Result e1,e2                                  
                                              
                                                     e1 =
                                                        0.3741117
                                                       0.2826440
                                                     - 0.8832626
   FT =                                            e1_u =
      0.    - 1035278.5 0.                          - 0.4235566
      2.048D+09 1.536D+09 - 4.800D+09              e1_v =
      6.554D+08 4.911D+08 - 1.536D+09               - 0.32
   vv2 =                                           e2 =
    - 0.3764731 0.1107978 0.9197782
                                                       0.9197782
    - 0.2823314 - 0.9593169 - 3.282D-13
      0.8823588 - 0.2596823 0.3924387               - 3.282D-13
   ss2 =                                              0.3924387
      5.712D+09 0.        0.                       e2_u =
      0.      1070927.1 0.                            2.34375
      0.      0.      0.0000001                    e2_v =
   uu2 =                                            - 8.362D-13
      0.0000512 0.9273836 - 0.3741117
                                                    ans =
    - 0.9524304 - 0.1139676 - 0.2826440
    - 0.3047561 0.3563299 0.8832626                   2.181D-08
                                                    ans =
                                                     - 1.956D-20
                                              
                                                    Execution done.


                                           Stereo v.1b8                36
Eight point algorithm for E or F (p279[1A], p263[1B])
left image=[x y 1]T, right image=[x’ y’ 1]T

        f11         f12     f13                                           For F, since Mint’s are not
    F   f 21     f 22       f 23                                          required, for simplicity
                                  
         f 31
                  f 32       f 33 
                                                                            we set both focal
    x'    y ' 1 * F * x y 1  0
                  T
                                                                             lengths to 1
    x ' xf11  x ' yf12  x ' f13  y ' xf 21 
    y ' yf 22  y ' f 23  xf 31  yf 32  f 33  0
    n  8
            x1 ' x1        x1 ' y1       x1 '      y1 ' x1        y1 ' y1     y1 '     x1     y1 1
    AF '   :                 :          :            :             :          :     :        : :F '  0
                                                                                                  
            xn ' x n
                           xn ' y n     xn '      y n ' xn       yn ' yn      y n ' xn        yn 1
                                                                                                   
    F '   f11                                                                       f 33 
                                                                                           T
                  f12        f13       f 21      f 22    f 23       f 31     f 32

                                                              Stereo v.1b8                                   37
Rank and Degree of freedom of
the Fundamental matrix F
   “F is a rank 2 homogeneous matrix with 7
    degrees of freedom.” p.226 [1]
   “The degrees of freedom of the fundamental
    matrix is 7, it can be counted as follows: 2 for
    e, 2 for e’, and 3 for the epipolar line
    homography which maps a line through e to a
    line through e’.” p.227 section 8.2.5 [1]


                        Stereo v.1b8               38
Eight (or more) Point algorithm for Fundamental matrix
F , (p91[1A], 265[1B])

   Find Correspondences n= (8 or more) left-right
    image feature points
   Rank of A is 8, at least 8 points are needed.
   From A (size nx9) from left side=[x y 1], right side=[x’
    y’ 1] image features , use the method of SVD to find
    f(9x1).
   AF’=0, and [U,S,V]=SVD(A)
   Estimated F’=last column of V, From F’ find F
   More accurate methods can be found in [1], see
    appendix of the refined SVD method.


                            Stereo v.1b8                  39
How to make Fundamental
Matrix found more accurate?
   Data needed to be normalized to [-1,1] to
    make the solution more stable.
   "RANdom SAmple Consensus" RANSAC to
    find a better F.
    http://en.wikipedia.org/wiki/RANSAC




                      Stereo v.1b8              40
3D model reconstruction from
stereo cameras after stereo
calibration
    Step 1: Rectification of stereo images
    Step 2: Find 2-D feature correspondences (by F,
     and rectification)
    Step3: 3-D Model reconstruction




                        Stereo v.1b8                   41
Step 1 of 3-D Model reconstruction:
Rectification
   The two images can be rectified (transformed)
    so that epipolar lines become horizontal.
   So that corresponding points can be found
    more easily (1D search along horizontal
    epipolar lines).
   More accurate corresponding points can be
    found for dense field reconstruction



                       Stereo v.1b8            42
Application of rectification
 A point has the
  correspondence
  from a tiled                     Original
  epipolar line                    pair
  (difficult)

A point now has
a horizontal                       Rectified
epipolar line to                   pair
search for
correspondence
(easier)            Stereo v.1b8       43
Step 2 of 3-D Model reconstruction:
Find 2-D feature correspondences


             by cross correlation




                 Stereo v.1b8         44
Search for correspondences
   Take left image as reference (I)
   Find all (N) features in the left and rigth
    images.
   For each feature in the reference image (I)
       Find the corresponding epipolar line in the right
        image (I’)
       Along the epipolar line (in image I’) search for the
        correct correspondence by cross-correlation.
       Till all correspondences are found
                             Stereo v.1b8                      45
   Recall                                               [a b c]T=(F*u1)=L2 is 3x1 vector
   :each point in the reference (left)
                                                        The epipolar line equation in
   image has an epipolar line in the
                                                        the right image is au+bv+c=0
   right image.
                                                        The Line L2 passes through
                                         X              [u2, v2 ,1] in the right
   
 Left side is the reference
                                                                      Right epipolar line
            (u1,v1)                                                   L2


Left                                                                 (u2,v2) right Frame
Frame                                                                        plane1 2
plane1 1
                                                                               O2
   Ol
                                                               Epipole e2
Focal          Epipole e1           F    Stereo v.1b8                       Focal          46

length=f1                                                                   length=f2
Search correspondences
   From (x,y) in the left image
   Search (x’,y’)i in the right image along the
    epipolar line (with tolerance)
   Use cross-correlation to measure the
    likelihood (see next slide for one example)
              (x,y)1                               L’=E*[x1,y1,f1]T       The gray
                           (x,y)2                                         area is
                                                        (x’,y’)i          the search
                  (x,y)i
                                                                          region
                                                      Epipolar line L’i   With certain
                                                      of (x,y)i           tolerant level
                           (x,y)N

                 Left image (I)     Stereo v.1b8
                                                      Right image (I’)              47
Example: To find one pair of
correspondence (enlarged)
                                                             W’j=3
                          Search region
                          Above L’1

                       Epipolar line L’1


                                                     W’j=1              W’j=2
                           Search region
    Wi                     below L’1
    A 5x5 window                          3 5x5 windows. each is a
    containing                            candidate window containing a feature.
    Feature                 Cj=1=correlation (Wi,W’j=1)
    In the reference        Cj=2=correlation (Wi,W’j=2)
    (left) image I          Cj=3=correlation (Wi,W’j=3)
                            Select the biggest Cj, the index j is j’,
                            Wi is corresponding to window Wj’.
                                           Stereo v.1b8                            48
Step 3 of 3-D Model reconstruction:
reconstruction

                   By triangulation




                   Stereo v.1b8       49
Procedures for Stereo
reconstruction of 3D model X
   So far
       The stereo camera pair is calibrated: Essential-matrix-E
        (Fundamental-matrix-F) is found.
       All good 2-D correspondences are found
   Will discuss here
       Find Translation (T) and Rotation (R) between 2 Cameras
        from E (or F) .
       From R,T calculate projection matrices P1(left) and P2 (right)
       3D reconstruction: from P1 and P2 find 3D points X by
        triangulation



                                Stereo v.1b8                        50
From F or E find Rotation (R), Translation (T) between
2 Cameras
(p.258[1]hartley-ed2,p 239[1B])

   If camera intrinsic                               0  1 0          0 1 0
    parameters are known,                       W  1 0 0, Z   1 0 0,
                                                                            
    E=(Mint2)T*F*(Mint1)                              0 0 1 
                                                                       0 0 0
                                                                              
                                                [U , S , V ]  SVD( E )
   E= [T]x *R,
   [ ]x=skew symmetric matrix                  [T ]x  UZU T
                                                R  UWV T
   Decompose E using
    SVD, using the fact that
                                                          t1    0        t3   t2 
    R is an orthogonal                                          
                                                [T ]x   t 2     t3
                                                                           0      t1 
                                                                                         
    matrix                                               t 3     t 2
                                                           x             t1     0  


                                 Stereo v.1b8                                                51
From R,T find projection matrices P1 (left) and P2
(right)(p.244[1A],p 224[1B])

    From F or E find R,T (previous slide)
    P1 =projection of the reference camera (left)

                                                              1 0 0 0
    P( 3x 4 )
     1           M int1 R1 | T1   M int1 I | 0  M int1 0 1 0 0
                                                                      
                                                              0 0 1 0 
                                                                      

    P2( 3 x 4 )  M int 2 R2 | T2 ,
                                        Stereo v.1b8                   52
 Triangulation to find X


                                X is the point at a minimum
                               distance between two vectors
                                {O1,(x1,y1)} and {O2,(x2,y2)}

            (x1,y1)
  Left epipolar line
Left                                            (x2,y2)
Frame
plane1 1
                                                          O2
   Ol          e1                               e2
                                           Epipole e2
Focal          Epipole e1                               Focal
                            F v.1b8
length=f1                   Stereo
                                                        length=f2
                                                                53
3D reconstruction: from P1 and P2 find 3D
points X by triangulation (p.312[1A],p297[1B])
                                                                           p11   p12    p13    p14   P1T 
     x1  P , x2  P2  for 2 cameras (P ,P2 )
            1                             1                 P(3 x 4 )     p21   p22    p23
                                                                                                              
                                                                                                p24    P 2T 
                                                                                                   
      x1  x1  x1  P   0                                               p31
                                                                                 p32    p33    p44   P 3T 
                                                                                                             
                      1
                                                            where P iT  i th row of P
      after_some_manipulations
                                                            i.e. P1T   p11       p12    p13    p14 ,
      x ( p  )  ( p  )  0      (i )
          3T          1T
                                                                 p11         p21          p31 
                                                                p                         
      y ( p 3T  )  ( p 2T  )  0      (ii)          p 
                                                             1   12 , p 2   p22 , p 3   p32 
                                                                 p13         p23          p33 
                                                                                          
      x ( p 2T  )  y ( p1T  )  0      (iii)             p14         p24          p34 

      p iT  i th row of P, use (i) and (ii) for 2 cameras (P1,P2 )
            x1 p13T  p1T 
                         1

                           
             y1 p13T  p12T 
      A                      0, solve by SVD [appendix 1]
            x2 p2  p2 
                  3T     1T

                        2 
            y2 p2 T  p2 T 
                  3
                                    Stereo v.1b8                                                    54
Summary of reconstruction

   From 2 images and correspondences, find
    Fundamental matrix F
   From F find R,T
   From R,T find P2. (P1 on the left is known because
    it is the reference R=I3, T=[0 0 0]T)
   For every point correspondence pair (x1,y1)i
    (x2,x2)I, i=1,,,N
       Using P1, P2 and (x1,y1)i (x2,x2)i we can find the 3D point
        Xi in the 3D space (reference of the camera center on left).
       The result is all N 3D points are found

                                Stereo v.1b8                       55
Overall Summary
   Learned
       Stereo computer vision and epipolar geometry
       Fundamental matrix, epipoles
       How to reconstruct 3D model points from 2 stereo
        views




                           Stereo v.1b8                56
References
    [1A] Hartley and Zisserman, Multiple geometry in computer vision,
     Cambridge, University Press. 2004 (2nd edition, paperback).
    [1B] Hartley and Zisserman, Multiple geometry in computer vision,
     Cambridge, University Press. 1999 (1st edition, hardback).
    [2] CVonline on stereo
     http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/EPSRC_
     SSAZ/node22.html
    [3] On Skew Symmetric matrix
     http://mathworld.wolfram.com/SkewSymmetricMatrix.html
    [4] On Skew Symmetric matrix
     http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/special.html#Skew_Symmetr
     ic
    [5]On RQ decomposition
     http://wwwcsif.cs.ucdavis.edu/~wangjj/gsvd/codes/rq5.m
    [6] Camera calibration matlab toolbox
     http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples
                                 Stereo v.1b8                               57
Useful software tools
   [7] Blloyd , Fundamental Matrix computation
    ( http://www.cs.unc.edu/~blloyd/comp290-
    089/fmatrix/ )
   [8] P.H.S. Torr, “A Structure and Motion Toolkit in
    Matlab: Interactive adventures in S and M”
    http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_
    COPIES/TORR1/index.html
   [9]
    http://ai.stanford.edu/~mitul/cs223b/draw_epipolar.m


                          Stereo v.1b8                 58
Appendix




Stereo v.1b8   59
Appendix 1
Refined SVD method for Af=0
   In MATLAB, [U, S, V] = svd(A);
   S is a diagonal matrix with descending values down
    the diagonal.
   f= the last column of V;
   [u,s,v]=svd(f)
   Set the smallest singular value in s=0, so s becomes
    s
   Approximate f=usv’.
   More accurate iterative methods are in [1]


                          Stereo v.1b8                 60
Scilab stereo_test.sce
   //rotation angles, [0,5,1]
   //pixel width 5umx5um
   //focal length 6mm/5um=1200
   //Translation 10cm , 10*1000*1000 /5um= 10*1000*1000/5
   //a 3d point is at point is [0.01, 0.02 1]meters
   //Translation
   //t=[10*1000*1000/5 0 0]'
   t=[-20*1000*1000/5 0 0]'
   //a point in 3D spcae is
   //
   //xx=[0.01 0.02 1 ]*1000*1000*1000/5 //in ipxel
   xx=[0 0 1 ]*1000*1000*1000/5 //in ipixel

   x=[xx 1]'

   //angs=([0,15,1]*3.1416/180)
   angs=([0,15,0]*3.1416/180)

   cosA = cos (angs(3));
   sinA = sin (angs(3));
   cosB = cos (angs(2));
   sinB = sin (angs(2));
   cosC = cos (angs(1));
   sinC = sin (angs(1));

   cosAsinB = cosA .* sinB;
   sinAsinB = sinA .* sinB;

   R = [ cosA.*cosB cosAsinB.*sinC-sinA.*cosC cosAsinB.*cosC+sinA.*sinC ;
       sinA.*cosB sinAsinB.*sinC+cosA.*cosC sinAsinB.*cosC-cosA.*sinC ;
         -sinB        cosB.*sinC          cosB.*cosC    ];
    R
    R'*R
    mint =[1200        0       512
       0       1200      384
        0        0       1]
   x1= mint*[1 0 0 0; 0 1 0 0; 0 0 1 1]*x
   x1(1)/x1(3)
   x1(2)/x1(3)

   mext=R*[[1 0 0; 0 1 0 ; 0 0 1] -t]
   x2= mint*mext*x
   x2(1)/x2(3)
   x2(2)/x2(3)
   //pause



   skew_t=[0 -t(3) t(2)
          t(3) 0 -t(1)
          -t(2) t(1) 0]
   E=skew_t*R
   F=inv(mint)*E*mint
    L2_1=F*[100 200 1]'
    [u s v]=svd(F)
    //epipole
    //epipoles e1,e2
    [uu1,ss1,vv1]= svd(F)
    FT=F'
   [uu2,ss2,vv2]= svd(FT)
   e1=vv1(:,3) // epipole on the left image ---------------
   e1_u=e1(1)/e1(3)
   e1_v=e1(2)/e1(3)
   e2=vv2(:,3) // epipole on the right image ---------------
   e2_u=e2(1)/e2(3)
   e2_v=e2(2)/e2(3)
   //by defination e2’*F*e1=0 or e1’*FT*e2=0 ,
   //because e1, e2 are on Image 1, image 2 resp.
   (e2')*F*e1
   (e1')*FT*e2





                                                                             Stereo v.1b8   61
Appendix 2
Matlab plot epipolar lines
   function draw_epipolar(left_x, left_y)
   %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   %
   % This function takes as argument the (x,y) coordinate of the LEFT
   % image and plots its corresponding epipolar line on the RIGHT and
   % the left image
   %
   %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

   % Hard coding the fundamental matrix FM

   FM=[ 0.0001 -0.0437 13.3897
      0.0429 0.0033 -21.4157
     -12.1830 22.0779 -999.3629];

   close all;
   left_image = double(rgb2gray(imread('rodin0020.jpg')));
   right_image = double(rgb2gray(imread('rodin0021.jpg')));

   [m n] = size(left_image);

   str1 = sprintf('Left Image point: %d %d',left_x,left_y);
   str2 = sprintf('Corresponding Epipolar Line on Right Image');

   figure,imagesc(left_image); colormap(gray); title(str1); axis image;
   figure,imagesc(right_image); colormap(gray); title(str2); axis image;

   % Start plotting:

   figure(1);
   hold on;
   plot(left_x, left_y, 'r*');

   % Getting the epipolar line on the RIGHT image:

   left_P = [left_x; left_y; 1];
   right_P = FM*left_P
   disp('t1')
   pause
   right_epipolar_x = 1:2*m; %//set all values of x in the range

   % Using the eqn of line: ax+by+c=0; y = (-c-ax)/b
   %c= a=right_p(1) a=right_p(2) a=right_p(3)
   right_epipolar_y = (-right_P(3)-right_P(1)*right_epipolar_x)/right_P(2);
   disp('t2')
   pause
   figure(2); hold on; plot(right_epipolar_x, right_epipolar_y, 'r');


   % Now finding the other epipolar line on the left image itself:

   % We know that left epipole is the 3rd column of V.
   % We get V from svd of F. F=UDV'
   [FU, FD, FV] = svd(FM);
   left_epipole = FV(:,3);
   left_epipole = left_epipole/left_epipole(3);

   % Hence using the left epipole and the given input point on left
   % image we plot the epipolar line on the left image

   %here plot left left_epipole
   left_epipolar_x = 1:2*m
   size(left_epipolar_x )
   disp('test2')
   m
   pause
   left_epipolar_y = left_y + (left_epipolar_x-left_x)*(left_epipole(2)-left_y)/(left_epipole(1)-left_x);

   figure(1); hold on; plot(left_epipolar_x, left_epipolar_y, 'r');




                                                                                                             Stereo v.1b8   62
Appendix 2
Scilab
   http://www.scilab.org/
   A poor man’s MATLAB




                      Stereo v.1b8   63
Appendix 4:See how to factorize E
into [T]x R
                                           Given E=[T]xR, E is rank2
   Make up 2 matrices Z,W                 USVT=SVD(E), S=diag(1,1,0)
                                            because E is rank2 (by
   Z=[0 1 0                                Definition)
      -1 0 0                              =USVT=U| 1 0 0 |VT
      0 0 0]                                         |010|
   W=[0 -1 0                                         |000|
       10 0                               We known S=ZW=[ 1 0 0
       0 0 1]                                                 010
   >> ZW=                                                     0 0 0]
      1 0 0                               E= U{S}VT=U{ZW}VT
      0 1 0                               ={UZUT}{UWVT}
      0 0 0
                                           = [T]x      R
                                           So both [T]x R satisfy their
                                            own characteristics
          http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/special.html
                                   Stereo v.1b8                            64
Appendix 5
Solve Ax=0
   To solve Ax=0, Homogeneous systems
       One solution is x=0, but it is trivial and no use.
       We need another method, SVD (Singular value
        decomposition)




                             Stereo v.1b8                    65
    What is SVD?
                                   If R is Orthogonal
   A is mxn, decompose it
    into 3 matrces: U, S, V       R is a square matrix with real entries
                                  columns and rows are orthogonal unit vectors
   U is mxm is an
    orthogonal matrix              RT R  RR T  I and
   S is mxn (diagonal             RT  R 1
    matrix)                        http : //en.wikip edia.org/w iki/Orthog onal_matri x
   V is nxn is an
                                              A  U mm SmnVnTn
    orthogonal matrix
                                                    1                0
                                                         2             
   1, 2, n, are singular                                            
    values                                    S                .       
   Columns of vectors of                                               
                                                                    .   
    U=left singular vectors
                                                   0
                                                                      n
                                                                         
   Columns of vectors of
    V=right singular vectors   Stereo v.1b8
                                               1   2  ... n  0           66
SVD (singular value
decomposition)
                                             Singular values
            Left singular vectors                                  Right singular vectors
   SVD             …
       U1,1 . . . U1,m   1
                                                                        T
                                               0  V1,1     . . . V1,n 
       .      . . .        .   2               .       . . . . 
                                                                    
       .      . . .        .        .           .       . . . .   svd ( Amn )
                                                 .                   
         .    . . .        .            .              . . . . 
      U m ,1 . . . U m ,m   0
                                            n  Vn ,1
                                                           . . . Vn ,n 
                                                                         
       1   2  ...   n
      Relation with eigen values
      ( AT A) x  x
      eigen values of ( AT A) are 1  2  ..  n
       j   j j  1,,n


                                       Stereo v.1b8                                     67
More properties
  Meaning of  i is
  Eigen values of AT A   12 ,  2 ,.., n .
                                  2       2


  Define
  ui  Columns of vectors of U  left singular v ectors
  U  u1 .. ui .. vm 
  vi  Columns of vectors of V  right singular v ectors
  V  v1 .. vi .. vn 
  ( AT A)ui   i2ui
  ( AAT )vi   i2vi
                               Stereo v.1b8                68
SVD for Homogeneous
systems
   To solve Ax=0,
   (   Homogeneous systems)
         One solution is x=0, but it             p  norm and 2  norm
          is trivial and no usage.                                         1
                                                            n
                                                                  p
                                                                     p
         If we set ||x||2=1, the                 x          xi  , hence
                                                             i   
                                                      p
          solution will make sense.
         So we ask a different                              n
                                                  x2        
                                                                       2
          question: find min(||Ax||)                              xi
                                                             i
          and subject to ||x||2=1.
         Note:||x||2 is 2-norm of x


                                   Stereo v.1b8                                69
Minimize ||Ax|| subject to ||x||2=1
   (’)=transpose
   To minimize the 2-norm of ||Ax||2, since
    A=USV’                                                   ** To show
   ||Ax||2=||USV’x||2= (USV’x)’ (USV’x) by                  ||x||2=||V’x||2
    definition 2-norm: ||Y||2=Y’Y                            Proof:
   So ||Ax||2= (x’VS’U’)(USV’x) because
    (ABC)’=C’B’A’                                            ||V’x||2=(V’x)’ (V’x)
   so||Ax||2=(x’VS’SV’x), since U is orthogonal             =x’V(V’x)
    and U’U=1
   Since x’VS’= (SV’x)’ put back to above. So
                                                             =x’x since VV’=I , (V is orthogonal)
    ||Ax||2=(SV’x)’(SV’x) =||SV’x||2                         =||x||2, done
   To minimize ||Ax||2 subject to ||x||=1                   To be continued
   Or minimize =||SV’x||2 subject to ||V’x||2
   =1 (see ** on the right)                                    If R is Orthogonal
   Set y=V’x
                                                                R T R  RR T  I and
   We now minimize ||Sy||2 subject to ||y||2=1
   Note: S is diagonal with descending entries                 R T  R 1
   The solution is ys=[0, 0, ..,0, 1]’ (reason:
    ||ys||2=1, and just ||Sys||=||S *[0 0 ..0 1]’ || is      1                   0
    the smallest, the last element of s is the                  2                  
    smallest n                                                                     
   Since V’x=y, so x=(V’)-1 y . V is orthogonal,         S                .       
    (V’)-1=V                                                                        
                                                                                .   
                                           V
    Xsolution=V[0 0 0.. 1]’=last column of Stereo v.1b8     0                     n      70
                                                                                    

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:28
posted:12/10/2011
language:English
pages:70