Optical flow and Tracking
University of Delaware
• Joint Lucas Kanade Tracking
• Some practical issues in tracking
What smoothing to choose?
Stereo Matching results…
Difficulties in optical flow
• Cannot directly apply belief propagation or
– Number of labels too high
• Brightness variation higher than stereo
Can we combine different flows?
Formulation as a labeling problem
• Given flows x0 and x1, find a labeling y
• Combine the flows to get a new flow xf
Graph Cut formulation
• Horn and Shunck with different smoothing
• Lucas Kanade with different window sizes
• Shifted versions of above
• Choose one of the proposals randomly as
initial flow field
• Visit other proposals in random order and
• Combine the proposals according to the
labeling to give fused estimate
• Some areas may have same solution in all
• Use conjugate gradient method on the energy
function to decrease the energy further
• Use bicubic interpolation to calculate gradient
Lucas Kanade Horn Schunck
(sparse feature tracking) (dense optic flow)
• assumes unknown displacement u of a pixel is • regularizes the unconstrained optic flow equation
constant within some neighborhood by imposing a global smoothness term
• i.e., finds displacement of a small window • computes global displacement functions u(x, y)
centered around a pixel by minimizing: v(x, y) by minimizing:
• denotes convolution with an integration • λ: regularization parameter, Ω: image domain
window of size ρ • minimum of the functional is found by solving the
• differentiating with respect to u and v, setting corresponding Euler-Lagrange equations,
the derivatives to zero leads to a linear system: leading to:
Limitations of Lucas-Kanade Tracking
• Tracks only those features whose minimum
eigenvalue is greater than a fixed threshold
• Do edges satisfy this condition?
• Are edges bad for tracking?
• How can this be corrected?
Ambiguity on edges
Joint Lucas Kanade Tracking
Joint Lucas Kanade Tracking
For each feature i,
1. Initialize ui ← (0, 0)T
2. Initialize i
For pyramid level n − 1 to 0 step −1,
1. For each feature i, compute Zi
2. Repeat until convergence:
(a) For each feature i,
ii. Compute the difference It between the first image and
the shifted second image: It (x, y) = I1(x, y) − I2(x + ui , y + vi)
iii. Compute ei
iv. Solve Zi u′i = ei for incremental motion u’i
v. Add incremental motion to overall estimate:
ui ← ui + u′i
3. Expand to the next level: ui ← aui, where a
is the pyramid scale factor
How to find mean flow?
• Average of neighboring features?
– Too much variation in the flow vectors even if the
motion is rigid
• Calculate an affine motion model with
neighboring features weighted according to
their distance from tracked feature
What features to track?
Given the Eigen values of a window are emax and
• Standard Lucas Kanade chooses windows with
emin > Threshold
• This restricts the features to corners
• Joint Lucas Kanade chooses windows with
max(emin,K emax ) > Threshold where K<1.
• JLK performs better on edges and untextured
• Aperture problem is overcome on edges
• Future improvements
– Does not handle occlusions
– Does not account for motion discontinuities
Some issues in tracking
• Appearance change
• Sub pixel accuracy
• Lost Features/Occlusion
• Joint Tracking of Features and Edges. Stanley T.
Birchfield and Shrinivas J. Pundlik. CVPR 2008
• FusionFlow: Discrete-Continuous Optimization
for Optical Flow Estimation. V. Lempitsky, S.
Roth, C. Rother. CVPR 2008
• The template update problem, Matthews, L.;
Ishikawa, T.; Baker, S. PAMI 2004