Every single multimodal LLM you have seen benchmarked in the last 12 months is lying to you. Not intentionally. The benchmarks are lying.

Over the last two weeks, six independent papers landed on arXiv that together dismantle almost every assumption we have used to rank and evaluate MLLMs. None of the standard leaderboard scores predict reliability. None predict performance scaling on real world tasks. Most of the gaps we thought were capability gaps are actually measurement artifacts.

This is not a minor adjustment. This is a reset of the entire evaluation field.

The order sensitivity nobody was measuring

Every standard benchmark runs every question exactly once, in exactly one canonical order. Nobody ever shuffled the multiple choice options. Nobody ever reordered the attached images. Nobody ever swapped the sequence of evidence paragraphs. Nobody tested the most basic reliability property a reasoning system should have: the same evidence should produce the same answer regardless of presentation order.

The Facet-Probe audit tested this property across 18 frontier and open weight MLLMs. All failed. Across five different ordering perturbations, panel mean flip rates ranged from 24% to 50%. That means if you only reorder the four answer options, half the time the model will change its answer. No changes to content. No changes to evidence. Just order.

The best performing model, Gemini 1.5 Pro, still flipped answers on 13.4% of trials. This was not decoder noise. The team ran temperature 0 same-input controls and confirmed over 90% of observed flips were pure ordering bias, not random sampling.

Prompt level mitigation does not work. Tricks that reduce order sensitivity for pure text reasoning have zero transfer to visual reasoning tasks. This is not a prompt bug. It is baked into how current MLLMs attend across modalities.

Capability falls off a cliff at predictable complexity thresholds

TriViewBench is the cleanest controlled experiment ever run on MLLM visual reasoning. Researchers built fully synthetic 3D scenes with zero ambiguity, zero dataset leakage, and explicitly parameterized complexity. Every object position, occlusion boundary, and camera angle was known exactly.

All 18 tested models exhibited exactly the same failure profile. No exceptions. No outliers.

Task type	Relative performance drop at maximum complexity
Local single object decision	12.1%
Cross view object counting	59.1%
Global 3D scene recovery	80.0%

Chain of Thought prompting produced an overall delta of -0.16%. That is zero. Not almost zero. Exactly zero.

This is the single most important result published this year. The bottleneck for MLLM performance is not reasoning. It is representation. The model never built a consistent internal model of the scene in the first place. You cannot apply reasoning steps to something you never correctly encoded.

Cross view fusion does not exist

We all thought MLLMs could combine information across multiple camera views. They cannot.

SSMNBench was designed explicitly to separate view selection from actual fusion. For tasks that can be answered from one good view, adding extra views reduces accuracy by an average of 17%. Additional visual information acts purely as distraction. For tasks that require information from two separate views, every tested model performs at chance level.

What modern MLLMs actually do is very simple: they score every input view independently, pick the single highest confidence one, and ignore all others completely. There is no synthesis. There is no geometric combination. There is only selection.

Every result you have ever seen on multi-view benchmarks is a measurement artifact of test questions that happened to be fully answerable from one single frame.

Long video benchmarks measure subsampling luck, not understanding

All existing long video benchmarks reward exactly one skill: guessing which 1% of frames will contain the answer.

Every production MLLM today drops between 95% and 99% of input video frames to fit inside context windows. Leaderboard rankings are determined almost entirely by how good a model's frame sampling heuristic is, not how well it understands the video content.

The Egocentric Scene Graph work demonstrated this conclusively. When you preprocess full video into a temporally grounded symbolic scene graph, you retain 100% of state transition information while achieving 20x token compression. When fed this representation, every tested MLLM improved HD-EPIC VQA performance by over 28%. None of the original leaderboard rankings survived this change.

The model that was previously ranked first dropped to seventh. The model that was ranked ninth moved to first.

Streaming video memory is a solved training free problem

CausalMem is the most practical result out of this entire batch of papers. Nobody trained a new model. Nobody added parameters. Nobody did fine tuning.

The authors built a simple online eviction policy that maintains a fixed size visual memory bank by continuously discarding redundant semantic tokens. Drop this 300 line module on top of any existing MLLM, and you get:

• +3.2% average accuracy across all video benchmarks
• 20x visual token compression
• 82 MB total memory usage for one full hour of streaming video

We have spent 18 months building larger and larger context windows for video, when a simple semantic LRU policy works better.

Photography guidance is the first real capability test

Almost every existing MLLM benchmark tests passive judgement. Is this photo good? What object is this? What happened in this frame?

CaptureGuide-Bench is the first benchmark that tests actionable guidance. Can the model tell you where to move the camera? Can it tell the subject to shift their weight 30cm left? Can it explain why that adjustment will produce a better photo?

General purpose MLLMs can tell you the photo is bad. They cannot produce precise localization. Specialized cropping models can draw the bounding box. They cannot produce an explanation. ShutterMuse, the purpose built model presented alongside the benchmark, beats all general purpose models by 41% on this task.

This is the future of good benchmarks. Stop testing trivia. Start testing actions.

The benchmark fraud loop

We have been running in an entirely circular loop for two years.

Every one of these new papers confirms this loop. Models that place top 3 on MME, MMMU and VQAv2 rank dead last on order sensitivity. Dead last on cross view reasoning. Dead last on long video state tracking.

Leaderboard rank now correlates negatively with real world reliability.

What we should be measuring instead

Stop reporting single pass accuracy. For every MLLM release, require these three sanity check metrics:

1. Cross order flip rate. Run every query 5 times with shuffled inputs. Report how often the answer changes.
2. Complexity scaling slope. Do not report one number. Report how fast performance falls as task difficulty increases.
3. Distraction penalty. Add one irrelevant input. Report how much accuracy drops.

None of these require new data. None require expensive annotation. You can run them on every existing benchmark today.

The quiet collapse of multimodal scaling

None of these failure modes improve meaningfully with model size. Order sensitivity goes down slightly. It never goes away. Complexity scaling slope is identical for 7B and 400B parameter models. Distraction penalty actually gets worse for larger models.

We are not building better reasoning. We are building better memorization of standard benchmark layouts.

Closing

This is good news.

For two years we have been chasing a leaderboard that did not measure anything useful. Now we know what is broken. Now we know what to test.

None of the fixes require larger models. None require more training data. All of the hard problems are in evaluation, not architecture.

The next generation of good MLLMs will not be the ones that top the old leaderboards. They will be the first ones that pass these new sanity checks.

All six papers discussed here were posted to arXiv between June 19 and June 26 2026. Nobody coordinated this. Six independent teams all arrived at the same conclusion at exactly the same time. The old benchmark regime is dead. We just haven't updated the leaderboards yet.

Benchmark	Core test	Sample size	Key finding
Facet-Probe	Order invariance	18 models	13.4% minimum flip rate
TriViewBench	Structural reasoning scaling	14k QA	80% performance collapse
SSMNBench	Cross view fusion	3.3k QA	No true fusion in any model
CaptureGuide-Bench	Actionable guidance	130k samples	General models fail at localization
CausalMem	Streaming memory	Hour long video	20x compression at +3% accuracy
EgoSG	Long egocentric video	HD-EPIC	+28% accuracy without model changes