Concurrent Stereo Matching _CSM_ by ewghwehws

VIEWS: 6 PAGES: 65

									Stereovision
Image Noise



John Morris

Department of Computer Science,
Tamaki Campus
The University of Auckland



                                  Iolanthe at 14 knots on Cockburn Sound, Perth
Stereo Image Noise Sources
 • Signal noise
      •    Electromagnetic interference eg cross-talk
      •    Quantum behaviour of electronic devices
           eg resistor shot-noise
      •    Quantization: digitization of real-valued signals
 • Geometric sources
      • Discrete pixel sensors with finite area
      • Occlusions
      • Perspective distortion
 •   Opto-Electronic sources  Single camera sources
                                           Stereo
      • Sensitivity variations between cameras (2-camera) sources
      •    Different ‘dark noise’ levels
      •    Real lenses
      •    Depth-of-focus
 • Optical Surface properties 
          Note that we use the term ‘noise’ for all problems!
                                                                    2
Stereo Image Noise Sources

Optical Surface Properties
    Lambertian scatterers
       A “perfect” scatterer scatters light uniformly in all directions
    Most correspondence algorithms assume perfect
     (Lambertian) scatterers
       This means that surface patches will appear with the same intensity –
        independent of viewing angle ..
       Simple matching
          • Intensities should be the same (perturbed by random noise!)
    Reflectors




                                                                            3
Stereo Image Noise Sources

Ordering Constraint
   Imaged points appear in the
    same order in both images
   Often used to simplify
    matching algorithms
      Particularly dynamic
       programming
   Violated by ‘poles’ – narrow
    objects in front of planes



                                   Scene points and image points in same order
                                                                                 4
Stereo Image Noise Sources              Scene: A B C
                                                   D

Ordering Constraint
   Imaged points appear in
    the same order in both
    images
   Violated by ‘poles’ –
    narrow objects in front of
    planes




                          Left: a b d        Right: d b c
                                                            5
Electronic Noise

Antennae (Receivers)
   Wires act as antennae for EM waves
   ‘Wire’ includes discrete wires
    but also
   Tracks on circuit boards
   Interconnects on chips
Transmitters
   Any wire with a changing current emits EM waves




                                                      6
 Electronic Noise

 Digital circuits  very rapid transitions (switching
  events)
        High frequency signals
              Ideal digital signal                                    Real digital signal



              ‘Instaneous’ rise or fall   ≡ infinite frequency         ‘Fast’ rise or fall
                                           perfect radiator          high frequency
                                                                      very good radiator
 Crosstalk
         One wire is influenced by neighbouring wires

Signal driven into purple wire
                                                   EM coupling
                                                                 Signal picked up on green wire
                                                                                                  7
Electronic noise

 Quantum effects                                                       e-
    Resistor ‘shot’ noise
        Resistive element is composed of discrete atoms
        Always in motion for all T > 0oK (absolute zero)
        Moving atoms ‘collide’ with electrons moving to form the
         current
       Random fluctuations in current
            or
        Noise as effective resistance changes
        Similar effects in all current carrying or producing devices        e-
            • Transistors
            • Capacitors
            • Inductors, etc




                                                                                  8
Electronic noise

 Digitization noise
    Analogue signal
        Taking all possible values
            • At least at a macroscopic level!
    Digital signal
        Represented by a range of integers
            • 0 .. 255 (8 bit signal)
            • 0 .. 4095 (12 bit signal)
            • -2048 .. 2047 (12 bit signed signal)
    A to D converter
        Decides to which integer value to map a real value
    Discretization
        Values which differed (in real domain) become the same (in integer domain)



                                                                                      9
Opto-electronic noise                     Stereo Problem

Cameras have different gain settings
    Amplifiers are not ‘matched’ perfectly

                                                 Different slopes
                                                  Gains differ
     Current




                                               Different offsets
                                              Dark currents differ
                 Light Intensity

Sensors have different ‘dark current’ characteristics
    All sensors produce some electrons (current) with no light
    Quantum ‘tunneling’ out of the sensor device
                                                                     10
Electronic Noise

 Summary




                   11
Geometric noise

‘Pixelisation’ of images
    Sensor is divided into discrete regions – pixels
    ‘Edges’ in images don’t conveniently fall onto pixel
     boundaries
Blue
object                                                      Real image
                                                            has blurred
                                                               edges


           Red object




                                                                      12
Geometric noise          Stereo Problem                          MR

                                              ML
Occlusions
                                     B
    Points visible from one                        B
     camera only
   Points which it is impossible
     to match
Perspective distortion
    Field of view in one camera
     differs from other
    Left and right images contain
     different numbers of pixels
                                         B Binocularly visible
   Impossible to match all pixels       ML Monocular Left
     correctly                           MR Monocular Right

                                                                      13
Perspective problem
Case 1 – ‘Fronto-planar’ object
Object with face
   •  to image planes
   •  to optic axes
    • extent, a
In image plane,
       AL = AR = az / f

Case 2 – Angled object
        AL > AR
Effect exacerbated by
    • Increased baseline
    • Increased angle of object
    •Verged axes,   
                                  14
 Optical effects                                                            Real lens

Real lenses                                        Image
                                                    plane
   Finite depth of field
       Image is ‘in focus’ over finite region only                   a
          • Due to deviations of real (thick) lens from
            ideal shape                                           f
       Depth of field increased by decreasing
        aperture
          • Always work with a small aperture!                        Mechanical
          • f-stop
                                                                       aperture
               • Ratio of focal length to aperture
               • ‘f-stop’ = f / a
               • ‘f’ = 1.4 – aperture wide open
               • ‘f’ = 22 – aperture closed down
                  For accurate work (objects in focus over a wide range)
                       Use more light to permit narrower aperture
                                                                                    15
 Optical effects                                            Image
                                                                                   Real lens

                                                            plane
Depth of focus
   Finite depth of field                                              a
       Image is ‘in focus’ over finite region only
          • Due to deviations of real (thick) lens from
            ideal shape
       Depth of field increased by decreasing                               Mechanical
        aperture                                                              aperture
          • Always work with a small aperture!              Depth of field
          • f-stop
               • Ratio of focal length to aperture                      f=22
               • ‘f-stop’ = f / a
               • ‘f’ = 1.4 – aperture wide open
               • f’ = 22 – aperture closed down                                 f=1.4

                 For accurate work (objects in focus over a wide range)
                      Use more light to permit narrower aperture
                                                                                               16
Lambertian Scatterers

‘Perfect scattering’
    Intensity of scattered light is the
     same in all directions
   Observers at all angles see the
     same intensity
Characteristics of a scattering
 surface
      Optically ‘rough’
      Surface roughness ≈ 
       = wavelength of incident light
      700-400 nm for visible light

                                           17
Reflectors

Perfect reflector
    100% of incident intensity reflected
    Angle of incidence i = angle of
     reflection r                          r i
   All reflected intensity seen at one
     angle
Properties of a reflecting surface
    Optically smooth
    Surface roughness  



                                                    18
Real Surfaces

 Reflector                                  Specular
                                             reflection
    Most of incident intensity reflected
    Bright ‘spot’ seen over a small angle
 Properties of a reflecting surface                      r i
    Optically smooth
    Surface roughness  
                                                   Smooth surface
 Specular reflections are the result of
  partly reflecting surfaces
    Bright spots in images
   from
    Water surfaces
    Polished glass, plastic, …


                                                                    19
  Real Surfaces
 Scattering surface
     Reasonably uniform distribution of intensity
      with angle
     Intensity of the same patch varies with angle
     Intensity mis-match between L and R images
     Systematic mis-match between views of the
      same patch
          One is always less intense than the other
     Shape of reflectance curve varies with
      surfaces
     Random mis-match between different
      patches
     Reflectance curves can be -dependent
      also
          Different views of the same patch appear to
           have different colours



                                                         20
 Real Surfaces

Grating effects
    In extreme cases
       Regular surface variations with
        dimensions 
   Grating effects
       Spectrum generation
       Small variations in view angle 
        large colour variations
       Holograms
   Systematic mis-match between
    different patches


                                           21
Ordering constraint

 If M appears to the left
  of N in the left image,
  it should appear to the
  left of N in the right
  image
    Violated by ‘poles’




                             22
Stereo Correspondence is an ill-posed problem!


                     or


       There are always multiple solutions!



                                                 23
                    Corresponding signals and possible surface profiles

View direction
                                                    right - scanline signal



                                                                    •Single surface
left scan-line signal




                                                                    reconstruction
                                                                    •Extreme disjoint
                                                                    variant



                                                                          corresponding
                                                                          Areas
                                                                          (possible matches)

                                                                                         24
Effect of Noise
but …
What happens if we use ‘noise-free’ images?




    L Image - ‘corridor’ set           Precise ‘ground truth’
    Synthetic (ray traced)                 is available

                                                                 25
Noise-free Image Matching
                                          Mismatch is the matching error
            Examine one scan line – line 152
                                     R image pixels shifted by known d
                                     (from ground truth)
                                      Zero difference with L pixel?
                                      Even ‘noise-free’ images have
   Intensity                         matching errors!


Mismatch
IL(x)-IR(x-dx)


    Disparity
 (from ground truth)


                                                                           26
Real Image Matching

      Tsukuba – line 173


   Intensity

                                   ?

Mismatch
IL(x)-IR(x-dx)
                       Occlusion
    Disparity
 (from ground truth)


                                       27
Noise-free Image Matching
            Examine one scan line – line 152



   Intensity


Mismatch
IL(x)-IR(x-dx)


                       Edge effects
    Disparity          No change in d,
 (from ground truth)
                        no occlusion!

                                               28
       Pixel-wise correspondences – ‘Tsukuba’ pair (line 173)


Distribution
of signal
differences




Grey-coded
signal
differences

                                                                29
   Dealing with the noise
   Window-based correlation



Epipolar line




                Window in             Possible windows in
                reference image       other image
     Algorithm
     For each pixel in the reference image
             For each possible disparity, d
                      Calculate signal difference between windows
                      centred on (x,y)L and (x-d,y)R
             Choose best matching window
             Output disparity
                                                                    30
Matching in the presence of noise
Correlation algorithms
    Match over a window
    Correlation functions:
       SSD (sum squared differences)  (IL(x+j.y+k)-IR(x+i-d,y+k))2
       SAD (sum absolute differences)  |IL(x+j.y+k)-IR(x+i-d,y+k)|
       Average matching errors
       Handles random intensity fluctuations
          • Electronic, quantization, … noise
       Doesn’t handle
          • Occlusions
          • Pixelisation
              • Intensity anomalies in one window compared to the other
          • Gain and dark noise variations
          • Perspective
              • Tries to assign all pixels in a window to the same disparity
          • Non-Lambertian surfaces & specular effects
                                                                               31
Improving correlation
Normalized correlation functions
    Normalize pixel intensities to mean within a window
      Some tolerance for gain and dark noise changes
       Often not worth the additional computational effort!
       Better: Correct the images at source!
          • Align amplifiers, calibrate electronics, …
Other variations of cost function
    Several hundred papers in the literature!




                                                               32
Correlation algorithms

Generally poor matching
   Mismatches (small windows) or
   Blurred edges due to large windows
Simple (almost trivial!) code
Adaptive windows improve performance 




                                          33
 Correlation-Based Methods
 Matching Performance
    Success of correlation methods depends on whether the image window in
     one image exhibits a distinctive structure that occurs rarely in the search
     region of the other image.

    How to choose the size of the window, wh?
         Too small a window
            • may not capture enough image structure and
            • may be too noise sensitive
             many false matches
         Too large a window
            • makes matching less sensitive to noise (desired) but also
            • decreases precision
              (blurs disparity map)
    An adaptive searching window has been proposed

                                                                                   34
Correlation Methods
                      Input – Ground truth



3x3 window                         7x7 window
Too noisy!                         Sharp edges
                                   are blurred!




                       Adaptive window
                       Sharp edges and
                       less noise
                                                  35
Correlation algorithms

Generally poor matching
    Mismatches (small windows) or
    Blurred edges due to large windows
Simple (almost trivial!) code
   for each pixel position,w
      for each disparity, d
         Cwd = (ILw+j-IRw+j-d)2
          Choose min(Cwd) over d  d for position w
    Fast enough?




                                                      36
Correlation algorithms
 Not fast enough for small images
    For nn image, d disparity values and hw window – O (n2dhw)
    For small (300x300) images and low depth accuracy (5%  d = 20 )
    If n = 3x102, d = 20, h=w=9, then t(n) = c  81  2 x 10  9 x 104 ≈ 1.5
     x 107
    For a 3GHz processor, cycles / pixel ≈ 3  109 / 1.5 x 107 ≈ 200
    ??? Enough to
          Compute indices,
          Fetch two pixels,
          Subtract
          Square or absolute difference
          Find minimum from d costs
          Repeat for adaptive window?
   but ……

                                                                            37
Correlation algorithms
 Moderate resolution images
    For moderate (1Mpixel) images and reasonable depth accuracy (1%  d
     = 100 )
    If n = 103, d = 100, h=w=9, then t(n) = c  81  102  106 ≈ 1010
    For a 3GHz processor, cycles / pixel ≈ 3  109 / 1010 ≈ 0.3
    Inherently parallel Cwd = (ILw+j-IRw+j-d)2
    Hardware acceleration needed!!
    MMX (Intel Graphics Processing Extensions) helps
        Optimized for simple arithmetic on pixels
       but
        MMX pipeline needs to be filled efficiently
        Max 16 operations in parallel
        Excellent for 4  4 windows 
        Usually 9  9 needed!



                                                                       38
Correlation algorithms

Easy to implement in hardware
   Match for each disparity at the same time
   Add minimum circuit
   Simple circuit ..
      But O (hw) computations for each disparity
     Relatively large circuitry




                                                    39
    SAD
    Algorithm
    Block
    Diagram
•   Note:
    High ‘fan-in’ for
    SAD calculator
    block
    Strains FPGA
    routing resources
    Large buffers
    needed for
    scanlines
    Consequence of
    window size




                        40
Correlation methods - summary

 Simplest code
 Poorest performance
    Adaptive windows don’t help much!
 Medium speed
    Window size is an important factor
 Simple hardware realization
   but
    Expensive in resource use
 Handle random noise only
    Window is just treated as vector of pixels
    No spatial information used
    Occlusions ignored

                                                  41
Dynamic Programming Stereo
 Attempts to find the ‘best path’ (sequence of disparity values)
    Can recognize occlusions!
    Averages noise over a scanline
    Essentially local
        Always moves ‘forward’ in a scanline
        Solution generated by backtracking through predecessor array
            • Doesn’t adjust values in backtrack
    Uses the ordering constraint
    Readily adapted to
        allow for gain and offset changes
        Perspective distortion
    ‘Stubborn’
        Incorporates a penalty for occlusions
        Tends to make ‘streaks’ in disparity images ..\resources\DP_Gonzalez.pdf
    Can be improved by using neighbouring scan lines
        Requires fewer scan line buffers than correlation window


                                                                                    42
Dynamic Programming Result




 Note the horizontal streaks!
 It’s like a bureaucrat: once a DP algorithm ‘decides’
 to adopt a disparity value – it doesn’t want to change its mind!
 Adding inter-scanline constraints (using a neighbouring scanline)
 generally improves this!                                         43
Dynamic Programming Performance

Better matching than correlation methods
    ‘Global’ along scan lines
    Recognizes occlusions
    Uses ordering constraint
         Uses spatial information
Time complexity          O(n2d)
      Faster than correlation methods in software
      Uses memory (typical of DP algorithms)
      For n=103, d=102, t = c 108
      c = ~30 cycles / pixel on 3GHz machine
      Not enough for real time in software!

                                                     44
Dynamic Programming

Basic idea
      Given a very large problem
      Solve small problems first,
      Save solutions to the small problems,
      Use them to solve larger problem
      Continue until large problem is solved
Dynamic programming uses storage space (for
 solutions to the small problems) to gain speed




                                                  45
Dynamic programming stereo

Small problems are the costs of a ‘path’ to a certain
 pixel
Costs are based on
    Difference between pixel intensities
    Cost for an occlusion
    Sometimes - Discontinuity cost
       Assume that disparity changes (depth changes) are relatively rare
       Alternatively – assume most surfaces are smooth




                                                                            46
 Dynamic programming stereo

                                                         Monocular Right Pixels
                                      Left scan line

                                                 Start
                     Right scan line


                                                                              This path would indicate all
              Monocular Left pixels                                           matches at zero disparity
Optimum path marks matching pixels
Diagonal path – smooth surface – no
change in disparity
Jumps – changes in disparity


                                                                                            End
                                                                     47
                                                                                                    47
  Dynamic programming stereo

                                                          Monocular Left Pixels
                                       Left scan line

                                                  Start
                        Right scan line
                                                                                  Maximum disparity


            Monocular Right pixels
DP algorithm calculates cumulative                                                    Compute costs
costs for all possible paths – using                                                  for matches in
costs for previous pixel to estimate                                                  this region
costs for the current pixel


                                                                                               End
                                                                     48
                                                                                                       48
  Dynamic programming stereo

                                                        Monocular Left Pixels
                                     Left scan line

                                                Start
                      Right scan line
As cost for each pixel at each disparity                                        Maximum disparity
is determined, save the predecessor
Use the predecessors to back track
Need w  Monocular–Right pixels
              predecessors
Biggest resource cost!
                                                                                    Compute costs
Predecessor array stores disparity of
                                                                                    for matches in
best previous point
                                                                                    this region



                                                                                             End
                                                                   49
                                                                                                     49
50
 Symmetric Dynamic Programming Stereo

Compute costs for
the Cyclopean image
centred between OL
and OR
Use visibility
constraints -
If disparity changes by
n pixels, must be n MR
or ML points between
the two B points
Only need to save
state in predecessor
array .. not disparity!
                                        51
Symmetric Dynamic Programming Stereo

Conventional Stereo
   Match pixel in L image with pixels to R of it in R image
SDPS
   For pixel position, x, in Cyclopaean image,
    match pixels at xL+d/2 with xR-d/2
   Even x positions have even disparity,
    odd x position have odd disparity
   Calculate in two phases: odd and even
   Resultant image is 2w pixels wide
   Backtrack fills in extra pixels in B regions


                                                               52
SDPS

Cost equations – see paper
Pixel mismatch is absolute difference - |gL-gR|
    Any dissimilarity measure could be used
Occlusion cost is fixed constant
    Results are not very sensitive to the actual value
Predecessor array only stores previous state
    ML, B or MR – 2 bits only




                                                          53
SDPS transitions


                                        This pixel, at
                                        disparity d0,
                                        can only have 3
                                        states – ML, B, MR
                            ML
                                        Arrows indicate
                             MR         costs used to
                                        evaluate cost to
                                        reach each state.
                                        Visibility constraints
                                        allow only these
                                        transitions.
    Note half integral x positions – SDPS
    generates 2w points/line
                                                           54
System Architecture

L                     FPGA
Camera   Interface        Line Buffers
           LVDS/       Distortion Removal
R        CameraLink     Image Alignment
Camera
                                                 PC

                                     Corrected        Host        Control
                         Stereo                                   Signals
                                      Images      Higher order
                         Matching
                                                 Interpretation
                       Disparity
                         Depth
                                       Depth
                                       Map




                                                                  55
SDPS hardware – Gidel FPGA card




                                  56
SDPS hardware – Disparity calculator block




                            Small and compact –
                            easily replicated to
                            provide large disparity
                            range


                                                  57
SDPS hardware – proposed system




                                  58
FPGA hardware

Field Programmable Gate Arrays
   Soft hardware
   ‘Programmed’ in a high level language – VHDL
   ‘Compiled’ programs soft loaded onto FPGA
      No cost to change the circuit!
Tradeoffs
   Flexible
      Pipeline or inherent parallelism easily achieved
         • Compare GPU
   Slower than state of the art CPUs – n00 MHz vs n GHz
   Memory may be a problem

                                                           59
FPGA Hardware

 FPGA = Field Programmable
  Gate Array
    ‘Circuit’ is stored in static RAM
     cells
    Changed as easily as reloading
     a new program




                                         60
FPGA Hardware
 Why is programmability important?
  or
 Why not design a custom ASIC?
    Optical systems don’t have the flexibility of a human eye
        Lenses fabricated from rigid materials
    Not possible to make a ‘one system fits all’ system
    Optical configurations must be designed for each application
          Field of view
          Resolution required
          Physical constraints
          …
    Processing hardware has to be adapted to the optical configuration
 If we design an ASIC, it will only work for one application!!



                                                                          61
Other correspondence algorithms

Belief propagation
    DP Stereo extended in all directions
Labels each pixel with a state – its disparity
Minimizes an energy function
    Two terms
       Data term – pixel difference
       Smoothness term – penalizes changes (in disparity)
Iterates until convergence (or cutoff)
    Exchanging messages with neighbouring pixels



                                                             62
Belief propagation

Best general performance
Simple BP being overtaken by segmentation-based
 algorithms
Memory intensive
    Needs to store costs for each label for each pixel in a region
       Compare last two pixels in scan line for SDPS
Iterates – so slow




                                                                  63
Algorithm Comparison




Symmetric DP stereo    Graph cut   Symmetric BP




         SCSM            CSM        Ground Truth




                                                   64
Middlebury Benchmark (MB)
                         Tsukuba                                   Sawtooth
  Algorithms
                         all          untex        disc            all          untex            disc
  MB-SBPO                0.97 1       0.28 3       5.45 3          0.19 1       0.00 1           2.09 3
  CSM                    1.15 3       0.80 12      1.86 2          0.98 13      0.62 25          1.69 2
  SCSM                   0.97 1       0.74 11      1.80 1          0.96 12      0.60 24          1.57 1


                          Venus                                    Map
  Algorithms
                          all          untex        disc           all          disc
                                                                                                 rank among
  MB-SBPO                 0.16   3     0.02    3    2.77   1       0.16   1     2.20   1
                                                                                                      40
                                                                                                  algorithms
  CSM                     1.18 12      1.04 10      1.48 3         3.08 34      7.34 18
  SCSM                    1.15 11      0.91 9       1.38 2         3.05 33      7.03 17
  * MB-SBPO    – symmetric belief propagation algorithm (best-performing Middlebury benchmark)

                                                                                                          65

								
To top