Local Grouping for Optical Flow



Introduction

Optical flow requires spatial integration, which essentially poses a grouping question: what points belong to the same motion and what do not. Classical local approaches to optical flow, such as Lucas-Kanade, use isotropic neighborhoods and have considerable difficulty near motion boundaries.

While motion itself is a strong grouping cue, there exist many other grouping cues such as brightness and texture similarities. In this work we show that such single-image grouping cues can help avoid erroneous motion integration near boundaries and greatly improve optical flow accuracy.

Soft Grouping via Intervening Contour

Local grouping cues in a single image can be well captured in a soft boundary map such as that from the Probability-of-Boundary detector [2]. Intervening contour [3] defines a pairwise affinity between two points by measuring the "resistance" or contour energy along a straight line path. This affinity function defines the (soft) spatial neighborhood for local motion integration.

One needs to be extra careful with affinities from/to boundaries, as they provide strong motion cues but are also potential motion discontinuities, in which case the boundary motion is only consistent with the "figure" side. we use an asymmetric intervening contour scheme as follows

(a) (b) (c)
Asymmetric intervening contour: (a) pairwise affinity defined by contour energies along straight line paths; (b) A is strongly connected to B and weakly to C and D (as C and D may belong to the other side and should not influence A); (c) C is strongly connected to all three to overcome the aperture problem.

Semi-Local optical flow using Pairwise Affinity

We proceed as in Lucas-Kanade, using the differential optical flow constraint and local motion integration. We use the affinity values as weighting in a robust least square estimation (please see the paper for details). Instead of a fixed square or gaussian, the affinity-based spatial neighborhood is soft and adaptive both in anisotropicity and scale.

In a near-uniform region, affinity values remain high on long distances, resulting in large spatial neighborhoods. To make the motion integration feasible, we sample the image first, at both boundaries and corners. These sample points both retain most motion cues and are sufficient to represent motion discontinuities. Our results on the Middlebury Flow dataset are promising (note we do not use color or explicitly model smoothness of flow)

Dimentrodon Hydrangea RubberWhale Venus Marble
AAE 3.34 2.77 5.32 3.93 4.88
Army Mequon Schefflera Wooden Grove Urban Yosemite Teddy
AAE 8.00 11.1 12.6 5.84 4.68 9.29 2.11 5.75

(a) (b) (c) (d)
Visualization: (a) image (b) color-coded groundtruth (c) our results (d) angular error

References

  1. Local Grouping for Optical Flow.   [abstract] [pdf]
      Xiaofeng Ren, in CVPR '08, Anchorage 2008.

  2. Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cues.
      D. Martin, C. Fowlkes and J. Malik, PAMI 2004.

  3. Contour continuity in region based image segmentation.
      T. Leung and J. Malik, ECCV 1998.