Every single multimodal LLM you have used is cheating. It does not look at the image. It guesses the answer from language priors 9 times out of 10.

This is not hot take. This is the central unspoken failure mode confirmed across nine connected papers that dropped on arXiv June 12 2026. None of these works introduce larger models. None use more training data. Every one attacks a broken foundational assumption that has been copied unchanged through three years of multimodal development.

This is the most important single week of progress in multimodal systems since LLaVA 1.0 launched.

The unspoken failure mode of self-evolving LMMs

For the last six months the field has been racing to build self-evolving multimodal models. These systems run self-play, generate their own training data, and beat fine tuned models on standard benchmarks without human labels.

All of them were broken.

Existing self-evolution schemes optimize only for answer consistency. If the model gives the same answer three times in a row, it gets a reward. There is no check that the answer had any relation to the actual pixels in the image. The VISE authors ran a control test where they fed solid grey noise images into state of the art self-evolved LMMs. 62% of the time the model output a detailed, grammatically perfect, entirely plausible caption. No hesitation. No indication it was looking at nothing.

This failure mode is called visual under-conditioning. It is present to some degree in every production multimodal model today. It is the root cause of object hallucination. It is why you ask a model how many fingers are in a photo and it will confidently tell you five even when there are seven.

VISE: Fixing visual under-conditioning without labels

VISE solves this problem with two extremely simple reward signals. No external reward model. No specialist model roles. No human annotations. No text labels of any kind.

First: take an image. Rotate it. Flip it. Crop it. Ask the model the same question. If the answer changes, apply a penalty. The model should see the same content regardless of trivial spatial transforms.

Second: take the original image. Black out a small random patch. Ask the same question. If the model gives exactly the same answer, apply a penalty. If the answer did not change when you removed evidence, the model was not using that evidence in the first place.

That is the entire framework. Applied to Qwen3-VL 2B this produces a gain of +16.85 CIDEr on COCO and +19.66 CIDEr on TextCaps. For reference this is larger than the full generation gap between Qwen2-VL and Qwen3-VL. It reduces object hallucination by 5 full Chair-I points. It works unchanged across four different model families.

DnA: Softmax attention was always noisy

Softmax attention was designed for text. It has a known flaw no one has properly fixed for seven years: it always spreads a small amount of attention weight across every token. For images this means every attention head is always looking a little bit at every pixel, diluting the signal for actual relevant features.

DnA fixes this with one simple change. Instead of running one query per attention head, run two. One positive query looks for features that match the target. One negative query looks for features that are definitely not the target. The two results are projected into orthogonal subspaces before attention calculation.

This is a drop in replacement for standard multi head attention. It adds no parameters. It requires no changes to training schedules. On ViT-B it delivers an absolute 0.8% top-1 gain on ImageNet-1K. On video transformers it delivers 1.8% improvement. On video LLMs it delivers 0.5% improvement across all benchmarks.

TOPS: You can throw away 78% of visual tokens and get better results

Visual tokens are the single largest cost in multimodal inference. A standard 1024x1024 image produces 1024 visual tokens. That is more tokens than most user prompts. For three years everyone assumed all these tokens were required.

They are not.

TOPS proves that 7 out of every 8 visual tokens are pure noise that the model actively ignores. Worse, leaving these tokens in the sequence increases hallucination, because the model will occasionally sample noise during generation.

TOPS is a training free, model agnostic pruning module. It runs once per query before visual tokens are passed to the language decoder. On LLaVA-NeXT 13B, removing 77.8% of visual tokens does not just preserve performance. It improves performance by 0.6%.

This is not a trick. This is a measurement of how much useless overhead has been carried by every multimodal model built to date. Every inference engineer should be testing this module this week.

Positional encoding stops being flat

Rotary Positional Encoding (RoPE) was designed for text. Everyone copied it directly for images and video. No one stopped to check if it was actually appropriate for spatial data.

It was not.

RayPE replaces the standard (u,v,t) grid position with 6D Plucker ray coordinates. When injected into queries and keys, the standard attention dot product automatically calculates the geometric relation between two camera rays. This adds less than 0.1% parameters to an existing video DiT. It is zero initialized so it does not break pretrained weights. It eliminates the floating object warping that plagues every current video generation model.

RoPEMover goes one step further. It demonstrates that RoPE is not just a static encoding. It is an editable coordinate system. You can directly modify the RoPE values for an object token to move that object anywhere in the image. The diffusion model will automatically correct occlusions, shadows, reflections and illumination without additional training.

Multimodal does not stop at vision and language

Every multimodal model built to date integrates at most three modalities: vision, text, audio. No one had seriously attempted olfaction.

See & Sniff changes this. The authors built a cross modal visuo-olfactory dataset by pairing existing smell samples with semantically aligned web images. They trained a joint representation model that can predict the smell of an object from a photograph, and generate spatial saliency maps showing where in the image the smell originates.

This is not a gimmick. This is the first step. There are at least a dozen human sensory modalities that will be integrated into general models over the next 24 months.

RLHF works on raw NeRF density fields

RLHF for 3D generation was always done on extracted meshes. Everyone assumed you had to convert a NeRF into a surface representation before you could apply human preference training.

This assumption was wrong.

The Sculpting NeRF paper demonstrates you can run RLHF directly on the raw sigma density values inside a neural radiance field. No mesh extraction. No multi view rendering. No shape priors. One human annotator, 2000 pairwise comparisons, and the fine tuned model produces face geometries preferred by users in 74.4% of tests.

This is how all 3D generative models will be fine tuned 6 months from now.

Trustworthy reasoning for medical MLLMs

All existing medical MLLM benchmarks are useless. They only grade the final answer. They do not check if the model actually observed the finding it claims to have seen.

CORTEX fixes this. It breaks every diagnostic answer into four explicit stages mirroring actual radiologist workflow: task understanding, visual observation, diagnostic reasoning, answer synthesis. Each stage is graded separately. You can see exactly where the model made a mistake, or where it guessed the answer without looking at the scan.

This benchmark is the first thing that will allow medical multimodal models to move from research papers into clinical use.

Calibration is not an afterthought

Medical multimodal models are catastrophically overconfident. They will state 99% confidence in an answer that is wrong 60% of the time. All existing calibration methods were designed for text only LLMs and do not work for multimodal inputs.

The calibration paper introduces a four term composite loss that reduces calibration error by 60% or more across all tested medical VQA benchmarks. It delivers this improvement with zero loss in predictive accuracy.

Every medical MLLM deployed from this point forward should have this calibration applied.

Measured gains across all work

All results presented in these papers are reproduced across multiple base models and multiple independent benchmarks.

Paper	Base Model	Task	Absolute Gain
VISE	Qwen3-VL 2B	COCO Caption	+16.85 CIDEr
VISE	Qwen3-VL 2B	TextCaps	+19.66 CIDEr
DnA	ViT-B	ImageNet-1K	+0.8% top-1
DnA	Video LLM	Zero-shot video QA	+0.5%
TOPS	LLaVA-NeXT 13B	MME	+0.6% at 77.8% pruning
Uncertainty Calibration	MedGemma 4B	Medical VQA	-60% calibration error
NeRF RLHF	EG3D	Human preference	74.4% win rate

What this means for the next 12 months

For three years multimodal progress followed exactly the same path as text LLMs: make the model bigger, train it on more data. This week of papers breaks that pattern.

None of these improvements require larger models. None require more labelled data. All of them fix fundamental broken design choices that everyone copied without question from the original 2017 transformer paper.

The next generation of multimodal models will not be 10x bigger. They will be 10x more efficient. They will hallucinate less. And they will actually look at the image.

Open questions

No one has yet tested combining these improvements. We do not know what happens when you run VISE fine tuning on a model with DnA attention, then run TOPS pruning at inference time. There is a reasonable chance the combined gain is larger than the sum of individual results.

We also do not know how far this pattern extends. How many other foundational components of the transformer architecture are simply bad defaults that no one ever bothered to challenge?