FATE: Dense Flow from Adaptive Correspondence (NeurIPS 2026)

One scene. From image to motion.

Match groups. Recover motion at every pixel.

Image 1 of the Sintel moving-blade scene used in the method figure — 01Input images

Learned pixel groups following the curved blade and surrounding background — 02Learned pixel groups

FATE's predicted dense optical flow preserving the curved blade against the background — 03Motion at every pixel

Sintel · moving-blade example from the method figure. The detail view shares one crop across all three panels; groups and motion refer to Image 1. White outlines show each pixel’s strongest group membership. A pixel can belong partly to several groups and combine their motions.

See the difference

Precise motion where matching gets difficult

Compare FATE with WAFT, the method it builds on. Both are tested on new datasets without further training (zero-shot). Drag the divider to compare their motion estimates.

Zero-shot examples

WAFT estimates motion at four times as many matching locations

WAFT FATE (ours)

Drag to compare WAFT and FATE.

Target visibility — Target occluded or out of frame

How to read these images

Color shows motion direction, and color intensity shows speed; white means no motion. Ground truth is the known correct motion. This reference and both predictions use the same color scale for each example. In the visibility map, blue marks pixels whose matching point is hidden or outside the second image, gray the other valid pixels, and black excluded pixels. Black in the ground truth marks missing labels; KITTI's sparse labels are enlarged for display. White outlines in the pixel groups view show each pixel’s strongest membership; each pixel can combine motion from several groups.

More examples: fine structures, occlusion, and changing light

Fine structures: Image 1, full frame — Image 1 (full)

Evidence

Lowest zero-shot error on Sintel Final and KITTI

Error versus compute

Zero-shot evaluation after training on FlyingChairs and FlyingThings3D. EPE is average motion error in pixels; Fl is the percentage of pixels with large errors. Lower is better; moving left on the plot means less computation.

Show the numbers

Method

Use the same grouping to match and recover motion

Matching needs enough visual context, while motion must be estimated for every pixel. FATE learns pixel groups that adapt to the image. A token summarizes the visual information in one group. The same grouping gathers information for matching and combines token motions into a detailed motion map.

Read the full abstract

Optical flow requires reliable matching from sufficient visual context and precise motion at every pixel. Existing methods such as WAFT match and estimate flow on the same coarse grids, then recover dense flow through learned upsampling, without explicitly linking visual support for matching to dense flow reconstruction. Our key insight is that dense pixel-level flow does not require equally dense matching. We decouple correspondence scale from flow resolution by adapting the size and shape of spatial supports to image structure for reliable matching, while reusing these supports for precise dense flow reconstruction. We realize this in Flow from Adaptive Token Estimation (FATE), whose token supports expand within coherent regions and contract near motion boundaries. Shared pixel-to-token associations pool features for matching and reconstruct pixel flow from token motions. Flow supervision jointly trains pixel grouping, token matching, and flow estimation without grouping labels, allowing FATE to anticipate which pixels are likely to move together from appearance alone. Trained on FlyingChairs/FlyingThings3D, FATE achieves the highest zero-shot accuracy on Sintel Final and KITTI, a critical setting given scarce real-world flow labels. For visible pixels, FATE improves accuracy within coherent regions, and more near motion boundaries. Its largest gains occur where direct matches are unavailable: Occluded or out-of-frame pixels account for roughly 10% of valid pixels but 50% of WAFT's total error. Experiments show that adaptive correspondence through pixel-to-token grouping enables reliable matching and precise pixel-level flow, improving generalization to unseen real scenes.

FATE pipeline: feature extraction, soft pixel-to-token assignments, adaptive pooling and local attention, token matching and refinement, and adaptive broadcasting back to dense flow, supervised by ground-truth flow

A moving blade from Sintel with two examined pixels. Near the motion boundary, the pixel puts almost all its weight on the blade token below, whose support follows the boundary. In the coherent region, the pixel shares weight across several tokens, and the selected token's support is broad. — Groups narrow near edges where motion changes and spread out within regions moving similarly. Each pixel combines motion from several groups, so pixels in the same group can still have different motion estimates.

Try it: how one pixel shares its weight across nearby groups

A pixel spreads its weight over the nine nearby token anchors with one Gaussian.

\[ a_{xq} = -\tfrac12\big(\delta_{xq}-\mu_x\big)^{\!\top} G_x\,\big(\delta_{xq}-\mu_x\big), \]

\[ A_{xq}=\frac{\exp a_{xq}}{\sum_{r\in\mathcal N(x)}\exp a_{xr}}. \]

Pooling uses the normalized columns, \(P_q(A)=D_A^{-1}A^{\top}\); broadcasting uses the rows, \(P_x(A)=A\). Here \(D_A=\operatorname{diag}(A^{\top}\mathbf{1})\) contains each token’s total assignment weight; the rows of \(A\) already sum to one.

Drag the orange point \(\mu_x\) to move the Gaussian.

Spread Elongation Orientation

How spread out the weights are

one tokenshared by many

How training teaches grouping, matching, and motion

Pixel space and token space for both images. Pooling forms tokens in each image; matching gives token flow, which broadcasting returns to dense flow. The same assignments pool ground-truth flow into token targets, and three losses connect predictions and ground truth in both spaces. — The same grouping gathers image features into tokens for matching and combines token motions into motion at every pixel.

\(\mathcal L_{\text{token}}(\mathcal F_q\mid P_q)\)Predict each token's motion

\(\mathcal L_{\text{assign}}(P_x\mid\mathcal F_q)\)Assign pixels to compatible tokens

\(\mathcal L_{\text{recon}}(P_q,P_x\mid F_x^{\text{gt}})\)Preserve motion through grouping

Inspect learned groups and the pixels they draw from

Image 1 with the enlarged region boxed — Image 1

All learned groups in the enlarged region — Image 1

Analysis

Learned pixel groups beat fixed grids

First, we look at where FATE improves on WAFT. Then we compare learned groups with fixed grid cells using the same number of tokens and the same matching framework.

FATE's gain is largest where a match is missing

Gain is the reduction in motion error compared with WAFT; positive values mean FATE is better. The largest gains occur where the matching point is hidden or outside the image.

Show the numbers

Adaptive versus fixed-grid tokens within a shared matching framework

WAFTreleased fine-grid model: one token per 2×2 pixels, four times as many tokens

Fixed grid, same token countone token per 4×4 pixels+ local attention

FATEtokens on the same lattice, with learned support+ local attention

Local attentioneach token's pooled feature attends to the nearby dense features in a window

All models share the DINOv3 backbone and its dense features (one per 2×2 pixels), WAFT's iterative matching and refinement, and the training data and recipe.

FATE has the lowest error on all six metrics

WAFT, four times as many tokensFixed grid, same token countFATElower is better

With the same number of tokens, learned pixel groups give lower error than fixed grid cells.

Show the numbers

Conclusion

Detailed motion without matching every pixel.

FATE performs matching on a coarse grid of tokens while estimating flow at every pixel. Each token gathers image features from a learned spatial support whose size and shape adapt to the image. Shared pixel-to-token associations pool features for matching and blend token motions into dense flow. A pixel can draw on several tokens, allowing pixels within one support to have different motion estimates. These associations are learned from ground-truth flow, without grouping labels, and predicted from image appearance alone at inference.

Citation

@inproceedings{hsieh2026fate,
  title     = {Dense Flow from Adaptive Correspondence},
  author    = {Hsieh, Cheng-Lin and Yu, Stella X.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Acknowledgements

This project was supported in part by NSF grant 2313151, with additional computing resources provided through NAIRR Pilot CIS240421 and CIS240431. The authors thank SpixelFCN and WAFT for sharing their code.