All seven papers dropped on arXiv the same day last week. None of them are clickbait foundation model announcements. Every single one describes a concrete optimization you can test on your production LLM stack before the end of this month.

This is not a survey. I will skip the fluff, show you the hard numbers, call out the unstated limitations, and tell you exactly which ones are worth your engineering time right now.

Stop fine-tuning on 100% of your data

Most teams still throw every available training sample at fine-tuning jobs. This is stupid. You are wasting 90% of your compute for zero gain.

PPL-Factory is the first data selection method that reliably beats full dataset fine-tuning. It does not use diversity heuristics, trace length scoring, or any of the other arbitrary rules people have been cargo culting. It uses task-aware perplexity.

For reasoning tasks you do not score the full sequence. You score only the answer portion. For language modeling you score the full prompt. That is the entire trick that everyone missed for three years.

The results are not marginal.

At 1% of the training set PPL-Factory hits 71.4% accuracy. At 10% it exceeds full dataset fine tuning by 0.9 points. On MATH the gap is 4.8 points.

You can implement this today. There are no new operators. There is no extra training. You run one forward pass over your dataset once, sort samples by this score, and take the top N. That is it.

PEFT has a new axis nobody was looking at

Every PEFT discussion for the last three years has been about LoRA variants. Nobody touched residual connections. Everyone assumed they were fixed architecture.

Manifold-Constrained Hyper-Connections show this was wrong. You can leave the entire transformer backbone frozen, and only train small routing modules that sit across the residual path.

By itself mHC does not beat LoRA. That is not the point. When you combine mHC and LoRA at exactly the same total trainable parameter budget, you get consistent improvement across every benchmark tested. At 1B and 7B scale. No exceptions.

Configuration	Trainable params	MATH Accuracy
Full fine-tune	100%	41.2
LoRA 0.1%	0.1%	34.7
mHC 0.1%	0.1%	33.9
LoRA + mHC 0.1%	0.1%	37.1

This is a free 7% relative improvement for zero additional cost. Nobody has published this combination before. You can drop this into your existing LoRA implementation this week. There is one catch: you must leave the residual mixing matrix initialized to identity. If you initialize it randomly it will not work. The authors explicitly called this out, and almost everyone who tested this already missed it.

Stop routing queries by embedding similarity

LLM routing works. Good routing will cut your inference cost by 60% while retaining 98% of end to end performance. Bad routing will cost you more money and give worse results.

Almost every production router today uses embedding similarity. It works okay. It is also leaving half the possible gain on the table.

VDAR-Router does something extremely obvious that nobody did before. Before routing, it asks a tiny 7B model: "On a scale of 1 to 10, how hard is this query? What type of problem is this?". It then looks up historical queries with the same difficulty profile, and routes based on which model actually succeeded on those queries.

Across three production datasets VDAR-Router beats every existing routing baseline. At the same accuracy level it uses 37% less compute than embedding based routers.

This router requires zero training. You do not need to fine tune anything. You just need a log of which queries succeeded on which models. Every team running multiple models already has this log.

Neuron level selective inference for edge devices

Quantization is not the end of on device optimization. Pruning is not the end.

SelectInfer works at the individual neuron level. Offline you profile which neurons activate for your target task set. At runtime you only load and run those neurons. No retraining. No quantization. No accuracy loss.

On 7B models this cuts memory footprint by 62% and inference latency by 51% for closed domain tasks. For general purpose chat the gain drops to 28%. That is still an enormous win for almost every real world edge deployment.

Most importantly this works on unmodified off the shelf checkpoints. You do not need to fine tune the model. You run the profiler once, and you get an optimized execution plan.

L1 augmented attention fixes a 7 year old mistake

Scaled dot product attention has a known flaw. It only measures angle between vectors. It completely ignores distance.

For seven years everyone just accepted this. L1 augmented attention fixes it with one line change. You subtract the L1 distance between query and key from the dot product score. That is the entire modification.

On identical parameter count and compute budget this reduces perplexity by 14.5% on WikiText 2. It is fully parallelizable. It runs exactly as fast as standard attention on all existing hardware. There is no downside.

This will be standard in every transformer released 12 months from now. You can patch this into your inference engine today.

You can merge completely different LLMs without training

Everyone knew you could merge fine tunes of the same base model. Everyone assumed you could not merge models from different families. Everyone was wrong.

You do not need distillation. You do not need adapters. You do not need alignment. You just pad the smaller model to match the dimension of the larger one, then do weighted average with a very small interpolation ratio.

At 0.05 interpolation ratio you get almost all the capability transfer with almost no regression. At 0.5 everything breaks. That is the pattern. Nobody tested very low ratios before.

This is not perfect. You will see regressions on some tasks. But for 10 minutes of work and zero compute cost you can often get a model that is strictly better than either parent.

Catastrophic forgetting is not inevitable

CMP is the weird one in this batch. It is not an optimization for existing transformers. It is a completely different architecture that does not use backpropagation at all.

On incremental learning tasks it exhibits 17x less catastrophic forgetting than a transformer with EWC. That is not a typo. 17 times.

It is also 15% worse on raw accuracy right now. That is the tradeoff. The authors did not hide this. They published the bad numbers. They published the failed experiments. That alone makes this paper worth reading.

We do not have production ready implementations yet. But this is the first credible result that suggests catastrophic forgetting is not a fundamental law. It is just a side effect of backpropagation.

Which ones should you implement first

Prioritization order for production teams this month:

1. PPL-Factory data selection. 1 day work. 90% reduction in fine tuning cost.
2. VDAR-Router. 2 days work. 35% reduction in inference cost.
3. L1 augmented attention. 1 hour work. No downsides.
4. LoRA + mHC. 3 days work. 7% accuracy gain for same cost.
5. SelectInfer. 1 week work. For edge deployments only.
6. Heterogeneous merging. Test this on a Friday afternoon.
7. CMP. Bookmark it. Check back in 6 months.

Closing observations

None of these optimizations require larger models. None of them require new hardware. None of them require billions of dollars of training compute.

All of them are things that anyone could have discovered at any point in the last three years. Nobody looked. Everyone was too busy building bigger models.

The biggest gains in LLM performance over the next 12 months will not come from bigger models. They will come from stopping all the stupid wasteful things we are all doing right now.