Local LLMs crossed the production usability threshold this quarter.

You do not need a cloud bill. You do not need a $10,000 server. You do not need a team of ML engineers. Most public advice is 12 months out of date, and almost every engineer building their first local deployment is making the same three avoidable mistakes.

This is not hype. This is what people are actually running, right now, for real work.

Stop chasing advertised context windows

Every model spec sheet prints a maximum context window number. Almost none print the usable context window number.

Think of the context window as a desk. Everything the model needs to reason about has to fit on that desk. When you put too much on it, items do not fall off the edge. They get buried. The model will never tell you it can no longer see something. It will just answer with the same confident tone, using only whatever happened to end up on top of the pile.

Some models will even admit they are overwhelmed. One user reported Claude Opus repeatedly asking to pause the conversation and resume tomorrow, mid long running session. That is the good case. Almost every other model will just silently degrade.

Advertised window sizes are marketing numbers. For reliable recall you should never use more than 50% of the published value. For agent work or coding you cap it at 35%.

This is the single most unspoken rule of production LLM operation. You will test your deployment with 1000 token prompts, it will work perfectly. You will roll it out. Two weeks later an engineer will paste a 90,000 token log dump. The model will not error. It will not throw a warning. It will output extremely plausible, completely wrong answers. You will not notice for a month.

You are almost certainly overquantizing

Every tutorial tells you to use the largest quant that fits in VRAM. Everyone defaults to Q4_K_M. That was good advice 12 months ago. It is terrible advice today.

Last week an engineer posted benchmark results for Qwen 3.6 running as a coding agent. The quality jump between Q4 and Q6 quantization was not incremental. It was binary. At Q4 the agent failed 62% of multi step coding tasks. At Q6 it succeeded 71% of the time.

That gap is larger than the difference between a 7B and 14B model. For a 33% increase in memory usage you get an effective doubling of useful capability.

Nobody tells you this. Quantization quality does not degrade linearly. There are sharp cliffs. For general chat you will never notice the difference between Q4 and Q6. For tool calls, structured output, coding, and any task requiring precise recall, you will notice every single time.

If you are running anything other than casual chat, start at Q6. Go lower only after you have proven you can accept the quality tradeoff.

Pick the smallest model that will do the job

Almost every engineer evaluating local LLMs immediately reaches for the largest model they can run. This is exactly backwards.

One DevOps engineer recently replaced 80% of his team's cloud LLM usage with Gemma 4 4B. Not the 26B MoE variant. The tiny 4 billion parameter one. It runs on a stock laptop with no GPU. It never leaves the network. It parses logs, reviews Terraform configs, explains stack traces. It does all of this better for this specific set of tasks than GPT-3.5.

His team was paying $847 per month for cloud API calls to do work that a 4B model can do perfectly well.

80% of all production LLM usage is boring, routine work. None of it requires general purpose superhuman reasoning. It just requires consistency, and it requires that your internal data does not get sent to a third party.

For this work, a well tuned small model will beat a large model every single time. It will be faster. It will be cheaper. It will be more predictable.

Hardware is not your bottleneck

Last week someone posted a photo of their local LLM server. It has three used Tesla V100 cards. The fans are plugged directly into a wall socket, controlled with a manual knob. The system RAM is laptop SODIMMs jammed into desktop adapters.

This janky setup runs production workloads faster than 90% of the fancy new 4090 builds people show off.

You do not need new hardware. You can buy used V100s for $150 each on eBay. You can run a production grade 8B agent on a 10 year old Xeon CPU. LiquidAI just released LFM2.5-8B which will run acceptably fast on literally any x86 machine made after 2015.

The biggest bottleneck for local inference right now is not hardware. It is bad default configurations. Stock llama.cpp settings leave 60-70% of possible performance on the table. Almost no one changes them.

The quiet victory of boring local systems

This week Hugging Face demonstrated fully local, end to end voice conversation running on a Reachy Mini robot. No cloud calls. No API keys. Everything runs on board.

No one is calling this a revolution. No one wrote a press release. It just works.

That is the pattern right now. All the interesting progress is happening below the hype cycle. People are quietly replacing cloud APIs with local models one routine task at a time. They are not trying to beat GPT-4o. They are just trying to stop paying $1 per stack trace explanation.

This is not a future promise. This is what works today. You can stand up a local LLM server this afternoon for less than $500 one time cost. It will handle 90% of internal engineering workloads. It will have zero recurring cost. No data will ever leave your network.

It will not be perfect. It will make mistakes. It will not solve hard open ended research problems.

It will be predictable. It will be under your control. And for most teams, for most work, that is more than good enough.

---

References

1. Why does AI forget what you said (and how to fix it) - https://dev.to/aws/why-does-ai-forget-what-you-said-and-how-to-fix-it-52f6
2. I Ditched Cloud LLMs for Gemma 4 4B - https://dev.to/asamaes/i-ditched-cloud-llms-for-gemma-4-4b-a-devops-engineers-48-hour-reality-check-a7d
3. Local Reachy Mini Conversation - https://huggingface.co/blog/local-reachy-mini-conversation
4. LFM2.5-8B-A1B - https://huggingface.co/LiquidAI/LFM2.5-8B-A1B
5. Qwen3.6 Q4 vs Q6 results - https://www.reddit.com/r/LocalLLaMA/comments/1tpebhw/qwen36_huge_quality_gain_from_q4_to_q6_for_coding/