If you are shipping a vision-language model this quarter, stop reading public leaderboards.

Five independent papers dropped on arXiv last week, all measuring the same thing from different angles: every frontier VLM today breaks completely when asked to do anything that is not answering a static question about a single image.

None of this shows up on MME, MMMU, or any of the standard benchmarks that everyone uses to announce model releases. This is not a small gap. This is a 80-95% performance drop across every model tested, including GPT-5.5 and Claude Fable 5.

The gap between benchmark and real world

Every VLM released in the last 18 months will score 70%+ on standard vision-language benchmarks. Every one of them will score under 15% on every one of the new benchmarks covered here.

This is not overfitting. This is a fundamental misalignment between what we test and what we expect these models to do. We have been testing photograph description. We have not been testing perception.

Standard benchmarks measure whether a model can regurgitate facts that happen to be visible in an image. Real world usage requires a model to actively extract required information, verify it, cross reference across views and time, and maintain awareness of its own position relative to the scene. None of these capabilities are tested anywhere in the standard evaluation suite.

ActiveVision: models cannot look twice

Human vision does not work by taking one snapshot and generating an answer. You move your eyes. You recheck. You update your hypothesis and go look for confirming evidence.

This is active observation. Until this week there existed zero benchmark that measured this capability in VLMs.

ActiveVision runs 17 tasks all designed to require more than one look at an image. No task requires knowledge outside what is present in the frame. All tasks are trivial for humans.

11 out of 17 tasks had zero correct answers from every model tested. Even when models were allowed to write and run arbitrary computer vision code to inspect the image, total pass rate only rose to 12.2%. The models could not write correct inspection code, and could not recognise when the code they had written returned the wrong answer.

This is not a reasoning gap. This is a perception loop gap. VLMs treat vision as a single one way operation. They look once, and they will never look again.

UAV-DualCog: agents do not know where they are

If you are building any autonomous agent that moves through the world, this result should stop you.

UAV-DualCog tests the single most basic capability required for any mobile agent: can it, given multiple views of a scene, reason both about the environment and about its own position and orientation within that environment.

This is not navigation. This is not path planning. This is the bare minimum prerequisite for doing anything useful while moving.

The benchmark includes 4200 QA samples across 118 real world scenes. All questions can be answered correctly by a human with no UAV experience 89% of the time.

Model	Overall score	Self state reasoning	Environment reasoning
Human baseline	89.2%	91.7%	87.3%
GPT-5.5	18.4%	9.1%	25.7%
Claude Fable 5	14.7%	7.3%	21.0%
Qwen2-VL 72B	11.2%	5.8%	15.7%
Llama 3.2 Vision 405B	9.8%	4.2%	14.3%

Self state reasoning is the worst performing category across every model. VLMs can describe what they see. They cannot infer where they are standing when they saw it.

Viewpoint transformation, precise distance estimation, and temporal interval localization all had success rates below 5% across all models.

Most notably, performance did not improve at all with model size. There was no measurable gap between 7B and 405B parameter models on self state reasoning. This capability is not emerging with scale. It is not present in the training objective at all.

Mechanistic failure modes in compositional reasoning

We no longer have to guess why this happens. The paper How Do VLMs Fail? provides the first clean mechanistic breakdown of VLM failure modes on compositional tasks.

The authors decomposed every failure across 12 VLMs into four distinct, mechanistically separable categories:

1. Grounding failure: Model never looked at the correct object in the first place
2. Attribute extraction failure: Model looked at the correct object but read the wrong property
3. Reasoning failure: Model correctly extracted all required facts but performed the wrong logical operation
4. Language prior failure: Model ignored all visual input and answered from text training statistics

These failure modes do not just have different behavioural signatures. They run through completely separate pathways inside the transformer.

This is the most important result published about VLMs this year. Different failure types live in different parts of the model. You cannot fix grounding failures by improving reasoning. You cannot fix reasoning failures by improving the vision encoder.

Every VLM improvement shipped in the last 12 months has only improved reasoning failure rates. All other failure modes have remained effectively unchanged since GPT-4V.

Attention saliency maps that actually work

Until very recently, if you wanted to know what part of an image a VLM was looking at when it generated an answer you had three bad options: gradient based attribution, knock out tests, or nothing.

All three are slow. All three are noisy. None of them reliably correspond to what the model actually used.

The new attention guided saliency method changes this. The approach is brutally simple:

1. For every token generated in the answer, collect all attention weights from all language model heads and layers pointing to visual tokens
2. Sum them, normalise, project back onto the original vision encoder patch grid
3. That is your saliency map.

No gradients. No fine tuning. Runs in 2ms per generated token.

Most importantly, this method passes causal deletion tests. When you delete the 10% of the image that this method says the model attended to, answer log probability drops by 78%. When you delete any other 10% of the image it drops by 4%.

This is not a visualization trick. This is a diagnostic tool. You can now, for every wrong answer your VLM produces, check exactly what it was looking at.

For the first time you can debug a VLM failure instead of just guessing.

Remote sensing VLMs: scale beats architecture

All of the above bad news comes with one very large caveat. You do not need fancy new architecture to fix most of this.

The More with Less paper demonstrates that a generic VLM, trained at sufficient scale on heterogeneous remote sensing data, will outperform every purpose built remote sensing VLM ever published. No custom encoders. No special fusion layers. No task specific heads. Just standard Llama architecture, standard CLIP vision encoder, multi task RL training across 12 million samples.

It beats every specialised model on detection, segmentation, captioning, multi temporal reasoning and multi view alignment.

Performance scales linearly with data diversity, not parameter count. The authors observed zero performance plateau across the entire range tested.

This is consistent with every other result in this batch. The problem is not the architecture. The problem is that we are not training VLMs on the tasks we actually want them to perform. Standard internet image text pairs do not contain active observation. They do not contain viewpoint reasoning. They do not contain composition. You will never get those capabilities by training on more of the same data.

What this means for production

If you are deploying a VLM today you should assume the following as hard limits:

• Your model will never reliably recheck an image
• Your model will never reliably infer its own position relative to what it sees
• Your model will fail 30-50% of the time on any task requiring more than two reasoning steps
• 70% of those failures will not be reasoning failures. They will be failures to look at the right thing.

None of these limits are visible on standard benchmarks. None of them will be fixed by the next larger model release.

You can work around all of them right now. You just have to stop treating the VLM as a black box.

Use saliency maps to verify attention before trusting an answer. Break compositional tasks into explicit single step operations. Do not ask the model to do perception. Tell it exactly where to look.

Open questions

We still do not know if active observation can be added to existing VLMs with fine tuning, or if it requires a fundamental architecture change.

We still do not know why self state reasoning does not emerge with scale.

We still do not have any good metric for how much you can trust any given VLM answer.

What we do know now is that we have been measuring the wrong thing for three years. We have built very good photograph describers. We have not built systems that see.

That is the work ahead.