Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

3D reconstruction

VIEWS: 0 PAGES: 50

									Fast Stereo Vision



      2004. 4. 9

      Young Ki Baik


               Computer Vision Lab
                Contents

   Stereo Vision
   Fast Stereo Vision
   Recent RTS system
   RTS system
   Fast Cooperative Stereo Algorithm
   Conclusion




                         Computer Vision Lab
                               Stereo Vision
     Key idea
            Stereo Vision
                     Scene object point


                                Epipolar plane


      Optical axes


    Epipolar lines




 Image
 plane




Left Camera            Left Camera               Computer Vision Lab
                Stereo Vision
 Key idea
   Stereo vision
     • The depth at various scene points can be recovered
       by determining the disparities of corresponding
       image points.
                                    Scene object point
                                          (x, y, z)
                     M             N
                                           P

                                                                   bf
                                                           z
                                                                d d
                                                                 l    r
                                   dr
                     dl
                          Pl            Pr
                 L             R                 Focal length
                                             f
                Cl                 Cr
                          b
                                        Computer Vision Lab
               Stereo Vision
 Key idea
   Matching points
     • Match points can be obtained through the window
       correlation approach along the epipolar line.




                                Computer Vision Lab
                 Stereo Vision
 Key idea
   3D disparity region
      • 3D disparity region is constructed and disparity map
        can be obtained using best correlation coefficient.




                                  Computer Vision Lab
                Stereo Vision
 Key idea
   3D reconstruction from Disparity map
     • 3D shape can be reconstructed using disparity map
       difference of match point.




                                Computer Vision Lab
                   Stereo Vision
 Problems and proposed solutions
   Is result accurate? ( Accuracy! )
      • Adaptive window method
      • Graph cut
      • Cooperative algorithm
      • Segmentation approach
      • Non-parametric local transform (Rank, Census
        transform)
      • …

   How long is the process time? ( Real time! )
      •   Window size invariant method
      •   Disparity range invariant method
      •   Parallel processing technique
      •   …

                                    Computer Vision Lab
            Fast Stereo Vision
 Window size invariant method
   Box filtering method
      • “Box-filtering techniques”, M.J.McDonnell (CGIP-81’)
      • “Real time correlation-based stereo : algorithm,
        implementations and applications”, Olivier Faugeras ,
        Zhengyou Zhang , … (Tech.Rep.RR-2013, INRIA,1993)
      • …




                                  Computer Vision Lab
             Fast Stereo Vision
 Window size invariant method
   Box filtering method
      • Column–wise SAD
          SAD( i, j+1, n ) = SAD( i ,j,n ) – ci,j,n(- L / 2 ) + ci,j+1,n( L / 2 – 1 )
      • Row–wise SAD
          SAD( i+1, j, n ) = SAD( i ,j,n ) – ri,j,n(- K / 2 ) + ci,j+1,n( K / 2 – 1 )



                     x1
                                    -                              +

                               x

                                         winx
                               x
                                          x1  +  -


                                                   Computer Vision Lab
            Fast Stereo Vision
 Disparity range invariant method
    Rectangular subregioning method
      • “Rectangular Subregioning and 3-D Maximum-
        Surface Techniques for Fast Stereo Matching”,
        Changming Sun (CVPR-2001)
      • “Fast Stereo Matching Using Rectangular
        subregioning and 3D Maximum-Surface Techniques”,
        Changming Sun (IJCV-2002)




                                Computer Vision Lab
                         Fast Stereo Vision
           Disparity range invariant method
              Rectangular subregioning method




Pyramid




                  Initial
                 Disparity




                                        Computer Vision Lab
           Recent RTS System
 Disparity range invariant method
    Small Vision System (Vide redesign)
      • MMX
      • P-III 1GHz, 32 Disparity range : 42 Frame


    Bumblebee Stereo Vision System (Point Grey)
      • P-IV 2.4GHz 32 Disparity range : 91 frame




                                   Computer Vision Lab
             Stereo Vision System
 System Overflow

 Camera    Input Images    Preprocessing      Stereo
  (Left)    (Left/Right)
                                             Matching
                                                          Postprocessing
                                              [SAD]
                                              [SSD]
                                              [ NC ]
 Camera
 (Right)                                      (7x7,...
                                              13x13)      Disparity Map



                            Calibration     Calibration
                             Process        Parameters




                                           Computer Vision Lab
              Stereo Vision System
 System modules

                     Module                             Function

                Camera Calibration   Camera parameter extraction for rectification
  Calibration
                   Rectification     Line fitting for stereo matching

                                     Image noise extraction by equalization and
                  Preprocessing
                                     LOG filter

    Stereo
                Stereo processing    Image correlation (SAD)
   Matching

                                     Disparity error extraction using L/R Filter and
                 Post processing
                                     Uniqueness test




                                              Computer Vision Lab
        Stereo Vision System
 Camera calibration
   Camera parameter
     • Image points are projected 3D real points
     • Projection matrix P :
         – Relation between 3D points and Image points



                                                           x : image point
                              x  Q[R | t]X  PX
                                                           X : world point
                                                          Q : intrisic parameter
                                                      X i  R : rotation matrix
                                           ui       Y  t : translation matrix
                                           
                             x i  PX i    vi   P  
                                                        i

                                          w         Zi 
                                           i        
                                                     1

                                   Computer Vision Lab
           Stereo Vision System
 Camera calibration
   Camera parameter acquisition
     • Usage of the stereo rig
     • Obtain the Camera parameter P, P’ and use rectification




      World center




                                   Computer Vision Lab
          Stereo Vision System
 Rectification
    Epipole and epipolar line
      • Baseline : Intersection line each image plane at the epipoles
      • Epipole : The point of intersection of the line joining the
        camera centers with the image plane
      • Epipolar line : The intersection of an epipolar plane with the
        image plane




                                       Computer Vision Lab
         Stereo Vision System
 Rectification
    Fundamental matrix F

      • F is the point and line relation in two view

               l  (P 'C)  (P ' P  x)  [e]x (P ' P  )x  Fx

      • F acquisition using point and line relation in two
        view
                        x T l  x T Fx  0
      • Epipole acquisition using fundamental matrix F

                             Fe  F T e  0


                                           Computer Vision Lab
         Stereo Vision System
 Rectification
    Horizontal line adjustment
      • To reduce processing time (stereo matching)
      • Rectification using epipoles in two view
      • Translation epipoles into point at infinity




                                 Computer Vision Lab
         Stereo Vision System
 Rectification
    Result Image




                    Computer Vision Lab
           Stereo Vision System
 Rectification
    Line fitting result (error)
        • Vertical line error test using cost function

                                   N
                              1
                      error 
                              N
                                  i
                                    d ( yi , yi) 2

        • Source image
            Error = 13.127293
            Min error : 1.500000 pixel2, Max error : 5.300000 pixel2


        • Fitting image
            Error = 0.012501
            Min error : 0.004366 pixel2, Max error : 0.220401 pixel2


                                       Computer Vision Lab
         Stereo Vision System
 Preprocessing
   Image noise extraction (LOG Filter)
      • To reduce noise and intensity difference error in two view




                                     Computer Vision Lab
            Stereo Vision System
 Stereo processing
   SAD(Sum of Absolute difference)
         • Disparity map building using disparity with minimum cost



     f         ( x, y , d )    I ( x  i, y  j )  I ( x  i  d , y  j )
         SAD                           r                l
                             i W j W




                                              Computer Vision Lab
         Stereo Vision System
 Post processing (Occlusion detection)
    LR Check
      • Disparity difference extraction between left and
        right image based disparity map




                          Disparity              L/R Check
         Input
        image              image                   result



                                      Computer Vision Lab
           Stereo Vision System
 Post processing (Occlusion detection)
    Uniqueness Test
        • Threshold = N * V1
        • V3 – Threshold > 0




         V3            V2
   Th                          Th
               V1                   V1 V2 V3
              Valid                 Occlusion

                               Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Ratification
     • Usage of LUT

                      Look-up
                       table




                                Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Stereo Correlation
     • The complexity of algorithm


              Method              Complexity
            Gray image                ND
            Color image               3ND
          Moving window
                                       D
            technique

         N : window size D: disparity search range



                                Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Moving window technique using near memory access
      •
              x

          y




                   (a)         (b)

                                                     Addition
                                                  Addition and
                                                  Subtraction
                                            Total sum of 2D
                                             window value
                                              Initial value

                  (c)          (d)


                               Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Moving window technique ( 2D )
          x

     y




                                                    Addition
                                                 Addition and
                                                 Subtraction
                                           Total sum of 2D
                                            window value
                                             Initial value




                                 Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Moving window technique ( 2D )
          x

     y




                                                    Addition
                                                 Addition and
                                                 Subtraction
                                           Total sum of 2D
                                            window value
                                             Initial value




                                 Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Moving window technique ( 2D )
          x

     y




                                                    Addition
                                                 Addition and
                                                 Subtraction
                                           Total sum of 2D
                                            window value
                                             Initial value




                                 Computer Vision Lab
     Fast Stereo Vision System
 Fast Algorithm
   Moving window technique ( 2D )
          x

     y




                                                    Addition
                                                 Addition and
                                                 Subtraction
                                           Total sum of 2D
                                            window value
                                             Initial value




                                 Computer Vision Lab
      Fast Stereo Vision System
 SIMD (Single Instruction Multiple Data)
    SSE2 (Streaming SIMD Extensions2)
       • 128-bit SIMD integer arithmetic operations.
       • 128-bit SIMD double precision floating point operations
       • Cache and memory management operations

                                                   2 x D oubl
                                                            e     XM M 7
                                                                  XM M 6
                                               16 x B Y TE
                                                                  XM M 5

                                                   8 x W O RD
                                                                  XM M 4
                                                                  XM M 3
                                                   4 x D W O RD   XM M 2
                                                                  XM M 1
                                                   2 x Q W RO D
                                                                  XM M 0

                      Data type and XMM Register

                                        Computer Vision Lab
    Fast Stereo Vision System

 Environment
   System : Pentium IV 1.4Ghz
   Cache Memory : 512Kbyte


 Condition
   Image size: 320x240 pixel
   Disparity searching range : 16
   Window size : 5x5 ~ 15x15



                            Computer Vision Lab
    Fast Stereo Vision System
 Result
   Frame rate


              Conv. SAD   row-SAD   mwSAD    mwSAD+SSE2
       5x5       2.79       10.53    18.12      31.85
       7x7       1.97        9.76    17.92      31.75
      11x11      1.53        7.96     17.7      31.65
      13x13      1.11        7.41    17.61      31.45
      15x15      0.92        6.57    17.48      31.45




                               Computer Vision Lab
              Fast Stereo Vision System
 Result

                               Conv. SAD                row-SAD
                               row+col SAD              mwSAD+SSE2
                  1200

                  1000
    Time (msec)




                  800

                  600

                  400

                  200

                    0
                         5x5      7x7        11x11      13x13     15x15
                                        W indow s ize


                                                     Computer Vision Lab
     Fast Stereo Vision System
 Result
    Environment
       • P-IV 2.4GHz
       • Stereo algorithm (except SSE2)
           Disparity range   Time          Frame rate
                 64          80 ms          12 frame
                 32          43 ms          23 frame
                 16          25 ms          50 frame


    Bumblebee SVS (Point Grey)
     P-IV 2.4GHz 32 Disparity range : 91 frame


                                     Computer Vision Lab
    Fast Stereo Vision System
 Result




                  Computer Vision Lab
  Cooperative Stereo Algorithm

 C.S Algorithm
   Idea (Marr & Poggio, 1976)
     • Uniqueness
     • Continuity


   Embodiment (Zitnick and Kanade, 1999)
     • Local support Area
     • Inhibition Area




                            Computer Vision Lab
     Cooperative Stereo algorithm
 Illustration of 3D disparity space




                                                    Current Element
 Disparity                                         Inhibition Element
                                                   Local support Area


         Column

                  Left Camera   Right Camera




                                       Computer Vision Lab
   Cooperative Stereo algorithm
 Other application
    Cooperative Stereo Algorithm




   Conventional
   Stereo (SAD)




    Cooperative
    Stereo (SAD)




                                    Computer Vision Lab
                Fast CS Algorithm

 Moving Window Technique (3D)
            d

    y   x




                                         Addition

                                       Addition and
                                        subtraction

                                       Total sum of
                                      3D window value
                                       Total sum of
                                      2D window value



                          Computer Vision Lab
                Fast CS Algorithm

 Moving Window Technique (3D)
            d

    y   x




                                         Addition

                                       Addition and
                                        subtraction

                                       Total sum of
                                      3D window value
                                       Total sum of
                                      2D window value



                          Computer Vision Lab
   Parallel processing Technique

 d direction redundancy
   Multiple calculation
     • 4 operation at the same time
       (floating point operations)




                                                   d

                                          Operation
                                         Source and
                                         Result data




                               Computer Vision Lab
           Fast CS Algorithm

 Environment
   System : Pentium IV 1.4Ghz
   Cache Memory : 256Kbyte


 Condition
   Image : 384x288 Tsukuba
   Disparity search range : 16
   Initial window size : 3x3
   Iteration : 1


                            Computer Vision Lab
                              Result

 Local support window : 3x3x3
                         Original     Moving Window    Parallel
                       CS Algorithm     technique     Processing
    Support region      2.100 (sec)    0.392 (sec)    0.063 (sec)

   Inhibition region    0.641 (sec)    0.133 (sec)    0.052 (sec)
     And update
         Sum            2.741 (sec)    0.525 (sec)    0.115 (sec)


 Local support window : 7x7x3
                         Original     Moving Window    Parallel
                       CS Algorithm     technique     Processing
    Support region     10.623 (sec)    0.395 (sec)    0.064 (sec)

   Inhibition region    0.643 (sec)    0.134 (sec)    0.052 (sec)
     And update
         Sum           11.266 (sec)    0.529 (sec)    0.116 (sec)


                                       Computer Vision Lab
      Fast Stereo Vision System
 Other application
    Fast Cooperative Stereo Algorithm (Result)
                   12

                   10

                   8
       Time(Sec)




                   6

                   4

                   2

                   0
                        3x3x3                 5x5x3                    7x7x3
                                      Local support area size

                        Original CS     Moving Window           Parallel Processing



                                                        Computer Vision Lab
      Fast Stereo Vision System
 Other application
    Cooperative Stereo Algorithm




                                    Computer Vision Lab
    Conclusion & Future works

 Conclusion
   Fast Stereo and Cooperative Stereo
    algorithm
     • 2D and 3D Moving Window Technique
     • Parallel Processing Technique


 Future works
   Remove redundancy
     • Disparity searching range, Iteration
   Program Optimization
   Application to other stereo algorithm


                               Computer Vision Lab

								
To top