Up until last week, every production video ML system hit one of two hard walls.

For generation: you could run sparse attention to get acceptable speed, but under multi-GPU sequence parallelism you would get 30-40% idle time on every step from straggler ranks. No one talked about this publicly, but every team running Wan 2.2 or OpenSora was eating this cost.

For understanding: any video longer than 2 minutes would either get garbage answers, or cost as much to process as running full inference on every frame individually.

Five papers dropped on arXiv between July 18 and 20 that fix both problems. None of them got the hype they deserve. This is the point where video ML stops being a demo toy and becomes something you can actually run at scale.

The state of play

All five works address the exact failure modes that were not showing up in academic benchmarks, only in production.

Paper	Core Task	Key Technical Fix	Measured Improvement	Open Source
FVAttn	Video DiT inference	Runtime load balancing for sparse attention	2.1x end-to-end DiT speedup	Not yet
TANGO	Autoregressive video generation	Test time noise distribution validation	28.3% lower FVD on 15s video	Yes
VideoTreeSearch	Grounded long video QA	Hierarchical tree search with backtracking	+12.5 mIoU on CG-Bench	Yes
MoD-VLLM	Multi-event long video understanding	Closed loop granularity scheduling	+18.7 accuracy on MEventBench	Not yet
AV-Flamingo	Audio-visual long video reasoning	3 stage curriculum training	+7.1 on Video-MME	Yes

FVAttn: The straggler problem no one talked about

Self attention is 78% of total inference time for modern video diffusion transformers at 1080p. Everyone switched to sparse top-p attention over the last 6 months. It cuts attention cost by 70% with almost no quality loss.

It also broke distributed execution completely.

Under sequence parallelism, each GPU gets assigned a fixed set of attention heads. Top-p routing produces very uneven work per head. One rank might get 1200 blocks to process. The next one gets 180. All ranks wait for the slowest one.

Before FVAttn every implementation just accepted this. Average load imbalance across 8 GPU runs was 1.34. That means 34% of all compute cycles were wasted waiting. No amount of kernel optimization would fix this. It is a scheduling problem, not a kernel problem.

FVAttn does three things. First it adds a hard top-k floor per head to eliminate extreme outliers. Then the moment the sparse mask is materialized, it scans head workloads across all ranks. It migrates exactly enough heavy heads over P2P to balance the critical path. Finally it fills any remaining slack on fast ranks with extra high value attention blocks that would otherwise have been dropped.

Overhead for this entire process is hidden behind the existing matmul operations. No extra wall time is added for scheduling.

End to end DiT speedup is 2.02 to 2.11x. No retraining required. Measured human rating quality loss was measured at 0.2%.

This is not an incremental improvement. This cuts the cost of generating 1080p video in half. Right now.

Runtime load balancing is the single most impactful inference optimization shipped in the last 12 months. Every team running video DiTs will have this implemented within 30 days.

TANGO: Stopping autoregressive video drift

Autoregressive diffusion solved the problem of generating arbitrarily long video. It also introduced a new one.

After about 12 seconds, every generated video falls apart. Frames stay individually good. The sequence stops making sense. Objects disappear. Motion drifts.

Prior work tried to fix this by anchoring individual frames to the real manifold. That did not work. The problem was never individual frames. The problem was terminal points.

There exist perfectly valid looking frames from which the model cannot continue. Paths that look correct, but for which no valid next step exists. Once the model lands on one, it will degrade over the next 4 frames.

TANGO uses one extremely simple observation. For any good trajectory, the noise predicted by the denoiser will be isotropic gaussian. For terminal points, it will not.

At every generation step, TANGO runs one extra forward pass. It checks the distribution of the predicted noise. If it deviates more than 3 sigma from expected, it backtracks one step and resamples.

That is it.

This change gives 3.1% absolute improvement on VBench. It reduces FVD by 28.3% average across 15 second videos. No retraining. Works on every existing autoregressive video model.

You will not see another autoregressive video demo that does not use this trick.

Why every long video QA agent was broken

All prior agentic long video QA implementations used exactly one action: crop_video(start, end).

Agents would start with the full video, crop down to smaller and smaller intervals.

If they cropped past the correct answer segment, they could never go back. There was no backtracking primitive. 37% of all failures on standard benchmarks were unrecoverable early mistakes.

No one noticed this for 18 months. Everyone was just measuring average accuracy and ignoring failure modes.

VideoTreeSearch

VideoTreeSearch fixes this by modelling the video as a tree, not a linear timeline.

The tree is built automatically on scene boundaries. The agent has four operations: zoom in to a child node, zoom out to parent, shift left/right to sibling, or submit answer.

This is not a minor interface change. This changes the entire search problem.

Agents trained on this structure can recover from mistakes. If they zoom into the wrong segment, they can back out and try another.

On CG-Bench this delivers +12.5 mIoU over the prior best agent. That is not a marginal gain. That is the difference between a system that works and one that does not.

MoD-VLLM: Dynamic granularity done right

All existing long video LLMs make one terrible tradeoff up front. They pick a fixed number of frames to extract from the video.

Pick too many, you blow your token budget. Pick too few, you miss the event you are looking for.

MoD-VLLM does not make this choice once. It runs a closed loop.

First it runs coarse grained encoding on the full video. It scores every segment for relevance to the query. It then runs fine grained high frame rate encoding only on segments that scored above threshold. It repeats this process until all relevant regions have been inspected at sufficient resolution.

This is obvious once you see it. No one had built it properly before.

On the new MEventBench which tests for multiple separate events across 10 minute videos, MoD-VLLM outperforms all prior baselines by 18.7 accuracy points.

AV-Flamingo: The first open long AV LLM

Every prior audio visual LLM was trained on 10 second clips. They would completely fail on anything longer than one minute.

AV-Flamingo is trained with a three stage curriculum. First short clips, then single events, then multi event long videos.

It is the first open model that can reliably answer questions about 10 minute videos that require connecting events that happened 5 minutes apart. It beats all other open models on every long video benchmark. It also beats GPT-4o on 3 of 7 tested benchmarks.

This is the baseline you should be using for long video understanding starting today.

What this changes for production roadmaps

Stop waiting for better base models. The bottlenecks were never the base models. The bottlenecks were the systems around them.

If you are building video generation:

1. Implement FVAttn this week
2. Add TANGO test time adaptation next week

You will get 2x faster generation, 2x longer coherent clips, for zero additional training cost.

If you are building video understanding:

1. Replace your agent navigation with VideoTreeSearch
2. Use AV-Flamingo as the base model
3. Add MoD-VLLM granularity scheduling

None of these require you to train anything. All can be dropped into existing pipelines this month.

Remaining open problems

None of these works solve consistent identity across 60+ second generation.

No one has yet built a good general purpose evaluation metric for long video reasoning.

Distributed inference for 4k video still has another 2x headroom left.

Agent memory across hour long videos is still an unsolved problem.

But for the first time, the obvious blocking problems are gone. You can build production systems that work. Not demos. Systems that actual users will pay for.

That is the part that matters.