Last month I audited three production LLM deployments. All three were running on state of the art models. All three were wasting between 70% and 85% of their GPU compute on work that did not need to happen even once, let alone ten thousand times per day.

No one had bad model choices. No one had bad prompt engineering. Every single problem was inference.

This is the quiet dirty secret of the LLM industry right now. We have spent three years arguing about training benchmarks, context window marketing numbers, and which model won which leaderboard this week. Almost no one is talking about the layer where 90% of all real world cost and latency lives.

No one is optimizing the right thing

Training is a one time fixed cost. Inference is a recurring cost that scales linearly with every request, forever. For every deployed system that has existed for more than 3 months, inference accounts for 80-95% of total LLM total cost of ownership.

Almost all public research and almost all vendor marketing ignores this. Optimizations that cut inference cost by half will always deliver more real world value than a 5% improvement on MMLU. No one holds press conferences for them.

This post covers the actual useful work that landed in the last two weeks, the broken tradeoffs everyone accepts by default, and the silent bug that is currently wasting VRAM on half of all local LLM deployments.

InferScale: Stop prefilling the same memory every time

Every single RAG and memory system in production today works the exact same way. On every user request:

1. Retrieve 10-50 relevant memory chunks
2. Paste them all into the prompt
3. Hand the full prompt to the serving engine
4. Wait while the engine re-prefills every single one of those tokens from scratch

This is insane. 90% of those chunks were already prefilled for a different request 12 minutes earlier. We just throw the work away and do it all again.

InferScale fixes this. It is a GPU native memory system that replaces repeated prompt prefilling with reusable KV state.

Memory facts are encoded once, when they are written. Their final KV representation is stored permanently on the GPU. At request time, retrieved KV entries are injected directly into the paged cache. No prefill. No redundant compute.

Benchmark results from the paper are unambiguous:

Retrieved Chunks	Standard Prefill TTFT (ms)	InferScale TTFT (ms)	Improvement
10	182	114	1.6x
25	417	131	3.2x
50	792	187	4.2x
100	1561	229	6.8x

At 50 retrieved chunks InferScale delivers 72-79% lower time to first token, 3.7-4.5x higher throughput, and only 3% accuracy drop relative to standard prompt injection. Quality is effectively indistinguishable for end users.

How KV injection actually works

This was not possible before for one very specific reason: rotary position embeddings lock KV values to an exact sequence position. You cannot precompute a KV entry once and drop it into a different place in the sequence. All prior attempts at KV reuse broke completely here.

InferScale solves this with Chunked RoPE. Keys are stored before rotation is applied. The correct position offset is applied only at injection time. This is a tiny, obvious change that nobody implemented correctly for three years.

The second critical fix is Context Window Encoding. Encoding memory facts in isolation drops quality because cross attention between adjacent facts is lost. InferScale encodes each fact together with a small window of preceding context, then only caches the KV for the target fact itself. This recovers almost all of the quality loss from independent encoding.

Best of all: this implementation uses vLLM's public KV connector interface. No engine forks. No model fine tuning. You can run this today.

The silent context window scam

Unlimited context is not a feature. It is an escape hatch for people who refuse to build proper retrieval.

Model vendors will happily sell you 2 million token windows. They will not tell you that prefill latency scales linearly with input tokens. They will not tell you that attention probability falls off sharply away from the edges of the context. They will not tell you that you pay for every token every time, even if the model never looks at it.

This is not theoretical. Independent testing from the dev.to comment thread measured actual attention rate across context length:

A note placed dead centre of a 20k context window has the same effective chance of being used as a note placed at the very start of a 2k window. All the extra tokens in between are just expensive padding.

You do not have a 128k context window. You have two 8k good windows at the start and end, connected by 112k of very expensive dead space.

You cannot measure grounding by reading outputs

Most teams verify grounding by checking if the answer cites the injected chunk. That test tells you absolutely nothing.

Models will happily parrot citation markers for chunks they never read, at a 94% rate in controlled testing. Presence of the source in the answer is not evidence that the source was used. It is only evidence that the model knows it is supposed to include citations.

The only valid test is counterfactual removal. Delete the chunk. Run the exact same query. If the answer does not change, that chunk was upholstery. It was never doing any work. You were paying for tokens for no reason.

When teams run this test for the first time they typically find that between 60% and 80% of all injected context has no measurable effect on the output at all.

An even stronger test is inversion. Flip the meaning of the note. If the answer flips too, it was actually being read. If not, it was just taking up space.

This testing produced one of the most useful rules to come out of any LLM discussion: a judgement note earns its budget exactly where it contradicts the model's prior, and nowhere else. Most rule corpora are just lists of good practice that the base model already knows. You are wasting tokens telling an LLM not to store plaintext passwords.

DIRECT: Decoding optimization no one talks about

For sequence labeling tasks, almost everyone is running full generation. They feed the full prompt, wait for the model to write out complete sentences, then parse the labels back out.

DIRECT fixes this. It is a framework that makes two very obvious changes that no one had bothered to combine properly:

1. Train the model to output only raw label tokens, no surrounding natural language
2. Precompute and cache the entire prompt template once, and only decode the single label token per input

This delivers 2-6x higher throughput on all sequence labeling tasks, with higher accuracy than full generation. No model changes are required beyond a small fine tuning run.

No one talks about this. Every production NER and classification pipeline is still doing it the slow way.

The llama.cpp MTP silent VRAM leak

This is the kind of silent bug that no one notices for six months.

As of llama.cpp PR #25980, the loader will load all MTP draft tensors by default on any model that includes them, even if you have speculative decoding completely disabled. Almost every recent MoE GGUF includes these tensors.

You are paying approximately one full extra MoE layer of VRAM, for nothing. Right now. On every load.

This behaviour was not announced. There was no release note. It just showed up in a nightly build and started eating VRAM. As of this writing it is still the default behaviour. You can work around it with --no-mtp but almost no one knows that flag exists.

Production inference checklist

These are things you can do this week that will deliver more real improvement than switching to the latest model:

1. Run the counterfactual removal test on 10 random notes from your retrieval corpus. Delete the ones that do not change the output.
2. If you are running vLLM, go look at the KV connector interface. You do not have to prefill every chunk every time.
3. If you run llama.cpp, add --no-mtp to every launch command until this default is fixed.
4. For any structured output task, stop generating full responses. Cache the prompt template and only decode the tokens that actually change.
5. Never use more than 32k context for production traffic unless you have already run the position attention curve for your model and task.

What comes next

We are at the end of the era where model capability is the bottleneck. For almost every real world deployment the bottleneck is now, and will remain, inference engineering.

The next big wins will not come from bigger models. They will come from stopping doing all the stupid redundant work we have been doing for the last three years just because it was easy to demo.