Optical flow and Tracking

Document Sample
Optical flow and Tracking Powered By Docstoc
					Optical flow and Tracking

        CISC 649/849
         Spring 2009
    University of Delaware
• Fusionflow
• Joint Lucas Kanade Tracking
• Some practical issues in tracking
What smoothing to choose?
Stereo Matching results…
       Difficulties in optical flow
• Cannot directly apply belief propagation or
  graph cut
  – Number of labels too high
• Brightness variation higher than stereo
Can we combine different flows?

Formulation as a labeling problem
• Given flows x0 and x1, find a labeling y
• Combine the flows to get a new flow xf
Graph Cut formulation
Graph cut
           Proposal Solutions
• Horn and Shunck with different smoothing

• Lucas Kanade with different window sizes

• Shifted versions of above
         Discrete Optimization
• Choose one of the proposals randomly as
  initial flow field

• Visit other proposals in random order and
  update labeling

• Combine the proposals according to the
  labeling to give fused estimate
       Continuous Optimization
• Some areas may have same solution in all
• Use conjugate gradient method on the energy
  function to decrease the energy further
• Use bicubic interpolation to calculate gradient
                  Lucas Kanade                                            Horn Schunck
             (sparse feature tracking)                                  (dense optic flow)

• assumes unknown displacement u of a pixel is           • regularizes the unconstrained optic flow equation
 constant within some neighborhood                         by imposing a global smoothness term
• i.e., finds displacement of a small window             • computes global displacement functions u(x, y)
 centered around a pixel by minimizing:                    v(x, y) by minimizing:

    •       denotes convolution with an integration        • λ: regularization parameter, Ω: image domain
      window of size ρ                                     • minimum of the functional is found by solving the
    • differentiating with respect to u and v, setting       corresponding Euler-Lagrange equations,
     the derivatives to zero leads to a linear system:       leading to:
 Limitations of Lucas-Kanade Tracking
• Tracks only those features whose minimum
  eigenvalue is greater than a fixed threshold
• Do edges satisfy this condition?
• Are edges bad for tracking?
• How can this be corrected?
Ambiguity on edges

Joint Lucas Kanade Tracking
Matrix Formulation
Iterative Solution
Joint Lucas Kanade Tracking
For each feature i,
1. Initialize ui ← (0, 0)T
2. Initialize i
For pyramid level n − 1 to 0 step −1,
1. For each feature i, compute Zi
2. Repeat until convergence:
   (a) For each feature i,
       i. Determine
       ii. Compute the difference It between the first image and
the shifted second image: It (x, y) = I1(x, y) − I2(x + ui , y + vi)
       iii. Compute ei
       iv. Solve Zi u′i = ei for incremental motion u’i
       v. Add incremental motion to overall estimate:
             ui ← ui + u′i
3. Expand to the next level: ui ← aui, where a
   is the pyramid scale factor
       How to find mean flow?
• Average of neighboring features?
  – Too much variation in the flow vectors even if the
    motion is rigid
• Calculate an affine motion model with
  neighboring features weighted according to
  their distance from tracked feature
       What features to track?
Given the Eigen values of a window are emax and
• Standard Lucas Kanade chooses windows with
  emin > Threshold
• This restricts the features to corners
• Joint Lucas Kanade chooses windows with
  max(emin,K emax ) > Threshold where K<1.


• JLK performs better on edges and untextured
• Aperture problem is overcome on edges

• Future improvements
  – Does not handle occlusions
  – Does not account for motion discontinuities
        Some issues in tracking
• Appearance change
• Sub pixel accuracy
• Lost Features/Occlusion
              Further reading
• Joint Tracking of Features and Edges. Stanley T.
  Birchfield and Shrinivas J. Pundlik. CVPR 2008
• FusionFlow: Discrete-Continuous Optimization
  for Optical Flow Estimation. V. Lempitsky, S.
  Roth, C. Rother. CVPR 2008
• The template update problem, Matthews, L.;
  Ishikawa, T.; Baker, S. PAMI 2004

Shared By: