If you have deployed a large vision language model in production in the last 12 months, you have almost certainly seen both of these failure modes. You just might not have named them.

Someone puts a sticker that says "STOP" on a speed limit sign. GPT-4o sees stop sign. Your OCR works perfectly, reads the text correctly, and the entire model ignores the actual shape and colour of the sign. That is a typographic attack.

You ask the model to count bolts in a mechanical diagram. It says 7. You tell it it is wrong, try again. It apologises, says 8. It never looked back at the image. That is fake self reflection.

Both were considered unavoidable quirks until this week. Two independent papers dropped on arXiv July 7 that directly address both problems. Neither got any press. Both change what you can reliably do with LVLMs right now.

The silent failure modes killing LVLM deployments

Most public benchmarking of LVLMs measures performance on clean, curated test images. Almost no one measures failure modes that appear when these models look at the real world.

Typographic attacks are not clever adversarial perturbations. They require no gradient access, no optimisation, no specialised knowledge. You write the name of any object anywhere on an image, and every production LVLM will report that object 70-90% of the time. This works on stop signs, medical scans, factory inspection imagery, satellite photos. It works on every model released to date.

Fake self reflection is even more pervasive. Every modern LVLM can generate text that looks like correction and reasoning. Almost none will actually re-examine visual input after receiving feedback. This makes chain of thought effectively useless for any task where correctness matters.

For three years the standard response to both issues has been "wait for the next larger model". That is no longer necessary.

What exactly is a typographic attack?

This vulnerability was first documented in 2024, but until this week no one had explained why it works. All existing defences treated it as an adversarial noise problem. It is not.

CLIP and every derivative vision encoder was trained on internet crawled image-text pairs. On the training set, when text appears inside an image, that text describes the image 99.6% of the time. The model learned an extremely effective shortcut: if you find text anywhere in the frame, use the meaning of that text instead of processing the rest of the visual features.

This is not overfitting. This is optimal behaviour for the training objective. There was no signal anywhere in the training data that would ever teach the model to ignore text printed on objects.

This is not an edge case. This is an unpatched critical vulnerability for any system that uses computer vision in unconstrained environments.

The source of the vulnerability

The authors of the typographic attack paper did not build another black box defence. They opened up the ViT and found exactly where this behaviour lives.

They ran circuit mining across 12 different CLIP and SigLIP checkpoints ranging from 300M to 4B parameters. Across every single model, exactly 7 attention heads in the final 3 ViT layers are responsible for 92% of the observed lexical bias.

These heads do not attend to objects. They attend exclusively to text contours. When they detect text, they overwrite the global image representation with the token embedding of that word. All other visual information is discarded at this step.

That is the entire bug. Nobody noticed this for 5 years.

Training free intervention

You do not need to retrain anything. You do not need to fine tune. You do not even need to run any additional auxiliary model.

Once you know those 7 heads exist, all you do is scale down their attention output by 60% during inference. That is the entire fix. One line of code inserted into the ViT forward pass.

The authors tested this intervention against every existing published defence on the RIO-Bench typographic attack benchmark:

Method	VQA Accuracy under attack	Clean accuracy drop	Requires retraining
Baseline LLaVA 1.6	28.1%	0%	N/A
Adversarial fine tuning	41.3%	-4.7%	Yes
OCR text masking	47.9%	-8.2%	No
Proposed head scaling	69.4%	-1.1%	No

This is not an incremental result. This is a 41 point relative improvement, with effectively zero downside on clean performance. This fix generalises unchanged across LLaVA, Qwen-VL, GPT-4V and CogVLM.

The other broken thing: fake self reflection

While one team was fixing vision encoders, another was fixing the single most overhyped capability in modern LVLMs: self reflection.

Everyone got very excited when LVLMs started generating chain of thought. Then everyone quietly noticed that the reflection step is almost always theatre.

When you tell an LVLM it got an answer wrong, it will generate a very convincing apology, explain what it did wrong, and output a new answer. Almost never will it go back and re-examine the image. It will just guess a different number based entirely on the text feedback.

It is performing reflection. It is not doing reflection.

This is not an alignment problem. This is not a reasoning capability limit. This is another shortcut learned from training data. On internet text, when someone says they are correcting themselves they are almost never going back and rechecking original evidence. They are just performing agreement. The model learned this behaviour perfectly.

VRRL training for grounded reflection

The authors propose a very simple change to reinforcement learning fine tuning for LVLMs. No new architecture. No new loss function.

Instead of only training rollouts from the start of generation, they do two things:

1. Halfway through every generation rollout, they randomly corrupt the intermediate output. Then they force the model to recover from that mistake.
2. They pull real failure trajectories from the replay buffer, and start training rollouts directly from the exact step where the model made its first wrong prediction.

That is the entire change. Models trained this way do not apologise. When you tell them they are wrong, they stop talking, re-attend to the image, and then answer.

Note that all methods perform almost identically on in distribution test data. You would never see this failure mode on standard benchmarks. It only appears when you test the model on inputs it has never seen before.

What this means for production

Stop waiting for the next foundation model to fix these problems. You can implement both of these fixes this week.

For the typographic attack defence: go pull the list of head indices from the public repository. Add the scaling factor. Deploy it today. There is almost no downside. The authors have already confirmed this works unchanged on every major open LVLM.

For the self reflection fix: if you are fine tuning any LVLM for visual reasoning tasks, you should be using this RL schedule. There is no reason not to. It adds zero inference cost, zero additional parameters, and produces models that actually correct their mistakes instead of pretending to.

Most importantly: both of these papers demonstrate something that has been missing from LVLM research for two years. They do not just benchmark better numbers. They explain why the model was failing, and they give you a mechanism to fix it.

The unstated implication

Both of these failures are not bugs. They are exactly what you get when you train very large models on internet scale data. The model learns the shortcut that works on the training set. It never learns the actual capability you wanted.

Typographic attacks work because 99% of the time on the internet, text on an image describes the image. Fake reflection works because 99% of the time on the internet, when someone says they are correcting themselves they are just performing, not rechecking.

Larger models will learn these shortcuts better. They will not unlearn them. You cannot scale your way out of this class of problem. You have to go in, look at what the model is actually doing, and fix the broken parts.

Closing observations

This is the good kind of ML research. No 100B parameter models. No secret training data. No press releases. Just people opening up the models, looking at what they actually do, and fixing the broken parts.

We are past the era where we just make models bigger. We are now in the era where we have to fix them. These two papers are two of the best examples of that work that have been published this year.

If you are running LVLMs in production, go read both papers. Test these changes. You will be surprised how much better your systems work by the end of the week.

Code for the typographic attack defence is released at https://github.com/Liu-524/SamplingTAR.