Most of you reading this have deployed a Vision Transformer in production in the last 12 months. Almost none of you can correctly answer two very basic questions: What actually changes inside the model when it trains? And if you need to make it run 5x faster, which compression method will not break it?

This is not your fault. Until this week, nobody had good, replicated, controlled answers to either question. Two papers dropped on arXiv July 3rd that change this. Both are properly controlled, both test standard production ViT variants, both avoid the common trick of benchmarking only on conditions that make the author's method look good.

The gap between deployment and understanding

We have been shipping ViTs for four years now. We know how to train them, how to fine tune them, how to quantize them. We have almost no predictive theory for how they work.

Almost all published analysis stops at attention heatmaps and downstream mIoU. Nobody was systematically measuring the actual geometry of the representation space through training. Nobody had run a head to head fair comparison of token reduction vs structural pruning at matched FLOPs.

We build production systems on folk wisdom and cargo cult procedures. These two papers finally start replacing that with measured fact.

Introducing TGO-II: the representation observatory

The first paper builds TGO-II, a measurement framework that runs consistent geometric probes across every checkpoint of a full ViT training run. No cherry picking. No stopping at interesting checkpoints. Every 100 steps, for 300k training steps, they extract every layer's output and run four independent measurements.

Measurement	Purpose
Centered Kernel Alignment (CKA)	Similarity of representations between layers
SVCCA	Alignment of representational subspaces
TwoNN Intrinsic Dimensionality	Effective degrees of freedom of the representation manifold
Token covariance matrix	Strength of statistical coupling between tokens

This is good work. They did not invent any new metrics. They just ran all the existing good ones, consistently, on a standard ViT-S/16 trained on ImageNet 1k. That alone makes this one of the most useful transformer papers published this year.

Three hard observations about ViT training

They found three things that directly contradict almost every popular informal hypothesis about how transformers learn.

First: Layer similarity drops monotonically through training. At initialization, every layer produces almost identical representations. By the end of training, adjacent layers share less than 30% representational similarity. Specialization happens gradually and consistently. There is no phase change. There is no magic step where the model 'clicks'. It just slowly diverges, every single training step.

Second: Intrinsic dimensionality rises then plateaus. The effective dimension of the representation space grows for the first 70% of training, then stops moving completely for the final 30%. All accuracy gains after that point come purely from rotating the existing manifold, not expanding it. This is the single most important training observation published in the last two years. You are wasting 30% of your training compute.

Third: Token coupling never goes away. The most repeated folk theory says transformers learn to decouple tokens as they train, making each token carry independent information. This is wrong. Token covariance stays almost exactly constant from initialization to final checkpoint. The model builds all of its representational complexity while preserving exactly the same strength of interaction between tokens. It never decouples them. It never makes them independent.

This breaks almost every existing explanation for transformer performance.

Why this matters for your training pipeline

You can stop doing that thing where you train for 300 epochs just because everyone else does. You can stop at 210. The manifold stops expanding. All further training is just fine tuning alignment.

You can stop designing architectures intended to promote token independence. They will not work. The model will actively resist that change during training.

You can stop measuring training progress only by validation loss. Track intrinsic dimensionality. When it plateaus, you are almost done.

None of this is speculation. This is measured behaviour of exactly the ViT variant you are running right now.

The great compression benchmark

The second paper addresses the question every deployment engineer has argued about at 2am: when you need to shrink a ViT segmentation model, do you prune weights or drop tokens?

Every paper on token reduction will show you beautiful results at 1.5x and 2x compression. None of them will show you what happens at 4x, 5x, 8x. None of them will test on corrupted inputs.

This paper ran a fair benchmark. All methods matched to exactly the same FLOP count. Tested on ADE20K, Cityscapes, and their standard corruption variants. Every method used the exact same ViT-S backbone, exact same training schedule. No method got extra fine tuning steps. No hidden advantages.

Where token compression breaks

The results are unambiguous. Token compression works perfectly up to ~2.7x compression. At exactly 3x, it hits a cliff.

This cliff is consistent across every dataset, every corruption condition, every token reduction method tested. It is not an implementation flaw. It is fundamental.

Remember that observation from the geometry paper? Tokens are never independent. You cannot just throw away 2/3 of them. They all carry coupled information. Remove enough, and the entire structure collapses.

Pruning does not hit this cliff. It degrades smoothly, almost linearly, all the way out to 8x compression. And it retains almost all of its robustness to input corruption even at 5x compression.

Compression	Token Reduction mIoU	Structural Pruning mIoU	Relative performance gap
2x	76.8	77.9	1.4%
3x	69.2	76.1	10.0%
4x	57.1	73.4	28.5%
5x	41.3	69.8	69.0%

At 5x compression, pruning gives you 69.8 mIoU. Token reduction gives you 41.3. That is not a small difference. That is the difference between a model that works and one that is useless.

The winning hybrid approach

The authors did one more thing. They tested combining both methods.

Take the baseline model. Prune it moderately to 2.2x compression. Then run mild token reduction at 1.8x on top. Total compression is ~4x.

This combined pipeline hits 75.2 mIoU at 4x compression. That is better than either method can achieve alone. It also retains almost full robustness on corrupted inputs.

This is the recipe you should use right now. Do not run aggressive token compression. Do not run extremely aggressive pruning. Do both moderately.

Production takeaways

These are not abstract research results. You can implement all of this this week.

If you are training ViTs: log intrinsic dimensionality during training. Stop training when it plateaus. You will cut training cost by 30% with zero accuracy loss.

If you are compressing ViTs for deployment: never use token compression above 2x. For anything higher, use structural pruning first, then add mild token merging on top.

If you are designing new ViT variants: stop trying to make tokens independent. That is not what the model does during training. It will fight you.

Stop arguing about which compression method is universally better. They each work well in different regimes. Use the right one for your required compression ratio.

Open remaining questions

We still do not know why intrinsic dimensionality plateaus. We do not know if this pattern holds for large models, or for language transformers. We do not know if we can intentionally stop manifold expansion earlier and redirect training compute.

We also do not know what causes the exact 3x cliff for token reduction. There is something fundamental about ViT representation structure that breaks at that exact ratio. Nobody has mapped that boundary yet.

And we still have no good explanation for why pruning works so much better than token removal at high compression, given that both remove roughly the same amount of information from the model.

Closing

This is the good kind of ML research. No new state of the art. No 100B parameter model. No fancy new architecture. Just careful, controlled measurement of things that everyone already uses, but nobody understood.

That is the research that actually changes what we ship. For the last four years we have been operating ViTs like mechanics who know how to change the oil but have never opened the engine block. These two papers finally hand us a flashlight.