The silent production LLM crisis

Nobody presents this at keynote stages. If you ask 10 senior ML engineers running production LLM systems what their biggest problem is right now, zero will say model quality.

Seven will say reliability. Six will say cost. Five will say they have no idea what their system is actually doing when it runs.

This month we got six separate, independent releases that attack exactly these problems. None of them are new foundation models. None of them have fancy demo videos. All of them are things you will be running in production before the end of the year.

Failover doesn't work if you lose context

Everyone running multi-provider LLM routing has hit this bug. You build nice failover logic. You test it. You get 99.99% uptime on your status page.

And every time failover actually triggers, half your users report that the bot forgot everything they were talking about.

Nobody measured this until now. ContinuityBench formalized the failure mode and ran 750 controlled failover events against standard production routing proxies.

Architecture	Continuity Preservation Rate	95% CI	Continuity Latency Overhead
Standard stateless proxy	0.13%	0.02%, 0.41%	127ms
Stateful history-forwarding proxy	99.20%	98.27%, 99.63%	214ms

Stateless failover works exactly once. It will keep your API returning 200 OK. It will also silently destroy every active conversation. This is not an edge case. This is how every popular LLM gateway works today.

The fix is not complicated. You forward the full conversation history on failover. Almost nobody does this.

The ContinuityBench authors also confirmed the thing everyone suspected but never measured: naive retries without jitter will cause cascading failure of your fallback provider 37% of the time under real load. Exponential backoff with jitter is not optional.

Compilation is finally not a vendor black box

For two years LLM compilation has been a walled garden. Every hardware vendor had their own closed compiler. You handed them your model. You waited. You got a binary back. You had no visibility into what it changed.

That changed with the MLIR LLM compilation paper published this month. The authors split the compilation pipeline into two clean, standard MLIR dialects:

1. TopOp: hardware and framework independent graph representation, preserves full model semantics
2. TpuOp: target specific dialect carrying quantization, layout and scheduling decisions

Every transformer layer is compiled into three separate specialized binaries: one for prefill, one for prefill with existing KV cache, one for decode. No more running the same generic kernel for workloads that have completely different memory and compute characteristics.

This implementation is not theoretical. It is shipped in the open source TPU-MLIR compiler. It works today for Qwen, Llama, InternVL and MiniCPM-V.

You are overpaying for inference hardware by 8x

Everyone looks at model parameter counts and picks instance sizes. Everyone does this wrong.

Bill Gallo got Gemma 4 26B MoE running on the smallest Inferentia 2 instance AWS sells. That is an inf2.xlarge. $0.76 an hour. The reference deployment everyone copies runs on an inf2.24xlarge for $6.49 an hour.

He did not use fp4. He did not invent new quantization. He did not change the model weights.

He just looked at memory residency. The 3GB over budget that everyone blamed on the large expert layers was not in the experts at all. It was in the lm_head that every single person had left unquantized and fully replicated across every rank.

This is the most important lesson for LLM deployment published this year. Do not eyeball the architecture. Profile actual memory usage. The thing wasting your budget is never the thing you expect.

Compile fit and deploy fit are completely separate budgets. This model required 180 GB of host RAM to compile. It runs on 11 GB. Almost no one separates these two steps. Almost everyone is renting 10x more hardware than they actually need to serve requests.

The 542 tok/s ceiling on consumer hardware

This week someone hit 542 sustained tokens per second for Qwen 3.6 35B on a single RTX 5090.

That is not batch throughput. That is single request. That is generating an entire 65,000 token completion faster than most people can read.

NInfer is not a general inference engine. It does one thing very well. It throws every possible optimization at exactly two model checkpoints. Custom weight layout. Hand written CUDA kernels. Full kernel fusion. Dedicated LM head speculative decoding.

Workload	Decode speed (tok/s)	MTP acceptance rate
Structured output	661.2	87.2%
Code	576.5	71.0%
Translation	559.3	66.6%
Story	395.9	37.7%

General purpose inference engines will never hit these numbers. They cannot. They carry overhead to support every model, every quantization, every hardware variant. For production systems that run one fixed model 99% of the time, this tradeoff is backwards.

We have hit the point where specialized, model-specific inference engines are delivering 2-4x performance over general purpose runtimes. This is not a temporary gap. This is the new normal.

Heterogeneous inference stopped being a research toy

KTransformers hit production this month. It does one thing that every vendor said was impossible: it runs large MoE models efficiently by splitting experts across GPU, CPU and disk.

Hot experts live on GPU. Warm experts live on system RAM. Cold experts live on disk. It schedules execution across all three transparently. For real world workloads where 5% of experts handle 95% of requests, this delivers almost the full performance of running the entire model on GPU, for 1/4 the hardware cost.

This is not a demo. This is production ready. You can run DeepSeek R1 on a single RTX 4090 today. You can fine tune it on four 4090s.

The most important detail almost no one noticed: KTransformers delivers 6-12x faster SFT for MoE models than ZeRO offload. Fine tuning 70B+ class MoE models no longer requires a datacenter cluster.

Observability for LLM systems is broken, and people are fixing it

Logs lie. Metrics lie. Traces almost tell the truth.

Every engineer building LLM agents has had this experience. A run takes 12 seconds. It spends 40 cents. You have a log line that says it completed successfully. And you have absolutely no idea what it actually did.

The existing observability tools for LLMs have exactly two modes:

1. Expensive hosted services that require you send all your user prompts and outputs to a third party
2. Enterprise self hosted stacks that require four separate stateful services just to store 100 traces a day

This month two separate teams released working alternatives. GitIntel demonstrated that you can instrument a full LLM pipeline end to end with standard OpenTelemetry and SigNoz in about 60 lines of code. You get per repository token counts, per step latency, rate limit tracking and full distributed tracing.

Otterscope went one step further. It is one single binary. One SQLite file. No other dependencies. It accepts standard OpenTelemetry traces. It shows you exactly what your agent did, what it sent to the model, what it got back, and what it cost.

The most important design decision in Otterscope: it stores the full raw unmodified trace payload. When the tracing standards change again next month, you can reprocess all your old data. You do not have to get normalization perfect on day one. You never do.

Stop building for scale you will never have

Almost every production LLM deployment is overbuilt by an order of magnitude. Teams implement continuous batching when they get 3 requests a minute. They deploy ClickHouse clusters when they generate 200 traces a day. They rent 24xlarge instances when an xlarge would be fine.

All of the good work released this month came from people who ignored the standard advice. They did not build for 1000 RPS. They built for the load they actually had. They optimized the thing that was actually wasting their money. They threw away all the complexity that did not solve their actual problem.

That is the real revolution happening right now. We are finally done demoing models. We have started building systems that actually run.

What comes next

Over the next 12 months almost every production LLM deployment will be rewritten. You will stop using general purpose inference runtimes. You will separate compile and deploy steps. You will run stateful failover proxies. You will stop paying for hosted observability. You will run 26B models on $0.76/hr instances.

None of this requires new research. None of this requires bigger models. All of this just requires stopping and looking at what your system is actually doing.

Most of the hard problems in LLMs are not hard. They are just things no one had bothered to measure yet.