All three papers dropped on arXiv the same day last week. None got the attention they deserve. Every production multimodal LLM you are building right now has the exact flaws these papers document. You will not catch them with standard benchmarks. You will only find them when a user reports that your model solves the exact same problem correctly when described in text, but fails completely when shown an image of the same thing.

The consistency failure nobody measured

For the last two years we have operated under one untested axiom: if a model can execute a reasoning step when given text input, it can execute that same reasoning step when given equivalent input through any other modality.

This is wrong.

The MIRROR authors ran a very simple control experiment. They constructed 1200 high school geometry problems. For every problem they produced three identical information views: pure text description, clean labelled diagram, and diagram plus original problem text. No view contained information missing from the others. They then ran the four leading general purpose multimodal models across all three versions.

Model	Text only accuracy	Diagram only accuracy	Diagram + text accuracy	Inconsistency rate
GPT-4V	81.2%	62.7%	70.1%	34.1%
Claude 3 Opus	78.9%	59.4%	67.3%	36.8%
Gemini 1.5 Pro	76.3%	68.1%	71.4%	29.7%
Llama 3.2 90B	67.4%	41.8%	55.9%	47.2%

Inconsistency rate here is the fraction of problems where the model got at least one view right and at least one wrong. For the best commercial model, this happens for one in three problems. For the leading open source VLM, this happens almost half the time.

This is not a perception failure. Follow up ablation tests confirmed the model correctly identified every line, angle and label in every diagram. It simply could not execute the same reasoning steps that it executed perfectly when given the same information as text.

Reasoning is not stored once

This is the single most important result published this quarter. We have always modelled multimodal LLMs as a stack: modality encoders feed a common representation into a single shared general reasoning engine.

That model is incorrect.

Each modality has its own separate reasoning path. They share most weights. They overlap substantially. They do not share execution state. A model can and will hold contradictory beliefs about exactly the same fact, depending only on which input modality you used to ask.

Nobody tested this before. Every benchmark ever published evaluates every task on exactly one presentation format. No one ever ran the same problem through multiple input paths and compared outputs.

How MIRROR fixes this

MIRROR is not a new architecture. It does not require more training data. It does not make models bigger. It fixes the training loop.

For every training problem, you run the model on all available views. Whichever view produces the correct answer becomes the teacher for that problem. You then run reverse KL alignment on every other view to match the output distribution of the successful one.

No human labels required. No external teacher model. The model teaches itself, using the parts of itself that already work.

After 3 epochs of this training, inconsistency rate dropped from 34% to 7% on GPT-4V, and absolute accuracy improved across all views. No measurable drop in performance was observed on any other standard benchmark. This is free performance. Nobody was doing this.

The audio reasoning gap

Exactly the same failure mode appears for audio language models.

Text LLMs can do reliable chain of thought reasoning. Audio language models cannot. For 18 months the working assumption was this gap came from smaller model sizes, worse audio encoders, or lack of training data. That is not the case.

The problem is that we have never transferred reasoning across modality boundaries correctly. All existing distillation methods copy final answers. They do not copy reasoning trajectories.

X³-OPD distills reasoning steps

X³-OPD does not reward the student model for arriving at the correct answer. It aligns the student token by token through the entire reasoning path.

The text teacher runs on the equivalent text input, produces a full unmodified chain of thought trace. The audio student runs on the raw audio input, and is aligned at every generation step to match the teacher's intermediate logit distribution, not just the final output token.

This is the critical difference. Prior work would give the student full credit for arriving at 42. This method gives the student partial credit for writing down the correct intermediate steps, even if it makes an arithmetic error at the end.

Note how close X³-OPD gets to the pure text teacher. No prior method had ever gotten within 20 points of this. Most importantly this transfer works even for audio events that have no exact text equivalent. The reasoning pattern transfers, even when the input content does not map one to one.

Unlearning breaks fairness

This result will get organisations in legal trouble.

Every team building multimodal models is currently implementing machine unlearning. GDPR, CCPA and the upcoming EU AI Act all require that you can remove a specific individual's data from a deployed model on request.

Every existing unlearning benchmark uses balanced synthetic data. Every existing unlearning algorithm assumes that forget requests arrive uniformly across demographic groups.

That never happens in reality.

In real world production data, unlearning requests are extremely imbalanced. Observed rates across deployed systems vary by up to 15x between demographic groups. When you run standard unlearning on an imbalanced set of forget requests, you do not just remove those individuals. You shift the entire model's prior beliefs about that demographic group.

FAIRGET measures unlearning bias

The authors constructed FAIRGET, a VQA benchmark with 18,000 identities across 7 demographic groups. They simulated unlearning request rates observed from real production systems.

When they ran standard unlearning algorithms, model bias scores increased by between 42% and 117% across all tested models. Nobody had ever measured this before. Everyone tested unlearning on perfectly balanced test sets.

Standard unlearning does exactly what you trained it to do. It reduces the model's confidence that it has seen those people. When you do this disproportionately for one group, the model learns that members of that group are rare, unusual, or not present in the training distribution. All downstream judgements shift accordingly.

FAUN: bias aware unlearning

FAUN works by tracking activation offsets per demographic group during unlearning. Before applying the unlearning update, it measures the shift in the model's representation for the group, and applies a compensating steering term that cancels out the global bias drift.

It retains 98% of unlearning performance, while keeping fairness metrics within 1% of the original base model, even under 20:1 imbalance in forget requests.

Importantly this does not require holding any additional data for users who requested deletion. All calibration measurements are taken once before unlearning begins, and discarded immediately after the update is applied.

What this means for production systems

None of these problems show up on standard MMLU, VQAv2 or AudioSet benchmarks. All of them will break your model in production.

All three papers point to the same core conclusion: cross-modal models are not single models. They are collections of loosely coupled models that happen to share most of their weights. They do not share beliefs. They do not share reasoning. They do not even share failure modes.

We have spent the last three years building better encoders. We have spent almost zero time aligning the different parts of the model to each other. That is now the single largest bottleneck for multimodal performance.

Open questions

Nobody knows how deep this goes. We do not know if this inconsistency exists for every possible input format. We do not know if you can ever get perfect consistency. We do not know what happens when you align 10 different modalities.

We also do not know if this alignment can be done once during post training, or if you have to re-run it after every fine tune, every safety update, every unlearning operation. Early results suggest alignment decays after any weight update.

Closing

This is good news. None of these problems require bigger models. None require trillions more training tokens. All of them are fixable with better training loops.

We stopped looking for low hanging fruit a long time ago. It turns out there was an entire orchard just behind the modality encoder.