If you are running LLMs in production right now, you are almost certainly wasting between 60% and 85% of your hardware budget on three completely solvable problems. None of these are secret. All three have working, validated implementations published this month. Almost no one has deployed them yet.

This is not theoretical. Every single one of these methods drops into existing stacks with zero model retraining, zero accuracy loss within measurement error, and delivers multiples of effective throughput on the exact same hardware.

The silent bottleneck order

Right now everyone argues about model architectures, quantization, continuous batching. None of those are the limiting factor for 99% of deployments. The actual bottlenecks hit in this exact order:

1. KV cache memory for long context decoding
2. Wasted LoRA parameter budget during fine tuning
3. Training instability and collapse during asynchronous RL post training

Every production team hits these in this order. Everyone builds bad ad-hoc fixes. This month we got proper solutions for all three.

HiKV: Stop storing useless KV tokens

Everyone knows most KV cache entries are never used again after the first 100 tokens. Everyone also knows every existing KV cache compression method either breaks accuracy, adds 30% compute overhead, or both.

HiKV does not work like every other method. It does not try to compress individual tokens. It does not quantize. It throws things away, correctly.

The core insight is that KV importance operates at two completely separate granularities. First: most entire tokens will never be attended to again for any future decoding step. Second: for the tokens you do keep, most elements inside their K and V vectors are also irrelevant.

No prior method did both. Everyone evicted whole tokens, or compressed vectors. Nobody did both in sequence.

This two stage eviction gets compression ratios that are mathematically impossible at single granularity. For 128k context windows, HiKV retains an effective working set of just 11% of the original cache size.

The hardware accelerator is trivial. It adds 8% die area. You could implement this in software tomorrow on existing GPUs and still get 3x speedup before any hardware changes.

HiKV benchmark results

These numbers are measured end to end on LLaMA 3 70B 128k, running real user conversation traces. No synthetic benchmarks.

Method	Compression Ratio	End to end speedup	Accuracy delta	DRAM access reduction
Vanilla KV	1.0x	1.0x	0%	1.0x
H2O	2.7x	1.4x	-0.4%	0.52x
SnapKV	3.1x	1.6x	-0.8%	0.47x
HiKV	7.95x	3.8x	-0.97%	0.18x

That is not a small improvement. That is you can run almost 4x more concurrent users on the exact same GPU, with effectively zero accuracy loss.

Nobody is talking about this paper. It was posted 11 days ago. It will obsolete every KV cache implementation currently running in production before the end of the year.

Stop assigning uniform LoRA ranks

This one is embarrassing. We have been using LoRA for three years, and everyone still just sets r=8 or r=16 uniformly across every layer.

This is insane. We have known from day one that different transformer layers contribute very differently to adaptation. Some layers do not need any adaptation at all. Some layers need 4x the rank you are giving them.

Every existing adaptive rank method runs during training. They track gradients. They add overhead. They break checkpointing. Nobody uses them.

IFCLoRA: Assign ranks once before you start training

IFCLoRA fixes this completely. You run it once, before training starts, on the frozen base model with 100 samples from your task. That is it.

It builds an information flow graph across all transformer modules, calculates centrality scores for each possible LoRA attachment point, then distributes your total rank budget exactly proportional to how much each module actually contributes to the task.

No overhead during training. No changes to the training loop. Exact same checkpoint format as standard LoRA. Zero extra memory.

This is not a marginal improvement. For the exact same number of trainable parameters, IFCLoRA beats standard LoRA by almost 2 full percentage points on reasoning tasks. That is the same gain you would get by increasing uniform rank from 8 all the way to 32.

You can take every LoRA fine tuning job you are running right now, swap in this rank allocation, and get a free 1-2% accuracy improvement. Or keep the same accuracy and cut your LoRA size by 75%. There is no tradeoff.

Asynchronous RL training was broken until last week

Everyone building agent models is trying to run asynchronous RL. You run rollout workers on cheap inference instances, stream gradients back to a training master. This gets you 3-4x training throughput. Everyone also knows that this always collapses after 2-3 update steps.

Every existing fix for this just throws away stale data. You set an importance ratio threshold, anything over that gets dropped. This works just badly enough that people tolerate it.

Nobody had actually measured why the ratios blow up. It turns out there is a perfect correlation between token entropy and the expected valid range of the importance ratio.

Low entropy tokens have almost no natural variation. Any large ratio here is always noise. High entropy tokens have enormous natural variation. Large ratios here are almost always legitimate exploration that you want to keep.

Every existing implementation does the exact opposite. They apply one global threshold. They keep the noisy low entropy updates and throw away all the good high entropy exploration.

ESTR: Entropy scaled trust regions

The fix is insultingly simple. Instead of one global threshold, scale the allowed deviation for each token by the local entropy of that token at rollout time.

That is the entire method. No extra forward passes. No version tracking. No changes to the optimizer. One line change to the ratio clipping code.

Method	Training throughput	GSM8K final score	Collapse rate over 1000 steps
Synchronous GRPO	1.0x	64.2	0%
Async GRPO fixed threshold	2.7x	57.1	78%
Async GRPO ESTR	2.6x	63.8	2%

You get 99% of the accuracy of synchronous training, at 2.6x the speed, and policy collapse effectively stops happening.

This single result will change how every LLM post training pipeline is built. There is no reason to ever run synchronous RL training again.

What this means for production

None of these are incremental improvements. These are step changes that reset the baseline for what counts as acceptable efficiency.

Right now you can go implement all three of these this week. None require waiting for new hardware, new base models, or framework support.

Most teams will not do that. Most teams will wait six months until Hugging Face implements them, until someone writes a blog post with pretty diagrams, until it becomes the default thing everyone does.

If you are running production LLMs, you have a six month window right now where you can get 2-4x effective throughput over every one of your competitors for free.

That is the thing about good engineering. Most of the biggest wins are not flashy new models. They are boring, correct fixes for problems everyone already knew existed, but nobody bothered to solve properly.

References

1. HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding http://arxiv.org/abs/2607.22389v1
2. IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning http://arxiv.org/abs/2607.22251v1
3. Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning http://arxiv.org/abs/2607.22186v1