This is not a roundup of model launches. This is a roundup of engineering.

Over the last two weeks, nobody released a model that beat every benchmark. Nobody got a $100m funding round. Nobody wrote a blog post claiming AGI is 6 months away.

Instead, hundreds of engineers quietly fixed things. They found traps in optimizations everyone was copying. They doubled generation speed on 5 year old graphics cards. They built tools to let people participate in conversations. They translated 600 year old books on a $0.50 Colab instance.

This is the actual state of local LLM work right now. This is what matters.

Batching will bankrupt you if you do it wrong

Every single LLM advice thread will tell you the same thing: batch your prompts. Fewer API calls means lower cost. This is presented as an universal law.

It is a trap.

One engineer ran this experiment on a document translation pipeline. 160 text segments, zh-TW to English, GPT-4.1 nano.

Without batching: 160 API calls, $0.0024 total cost, 30.4 second runtime. Perfectly predictable, perfectly boring.

First batching implementation grouped 20 segments per request. API calls dropped 33% to 107. Everyone would have called this a win if they only looked at request count.

Cost went up 37% to $0.0033. Runtime increased 4x to 136.2 seconds. 71% of all requests were fallbacks.

100 out of those 107 API calls were retries. When one single ID went missing from the batch JSON output, the code threw away 19 perfectly good translations and retried all 20 segments from scratch. This failure mode is present in almost every batching implementation posted online. Nobody talks about it.

The fix was not removing batching. It was three small changes:

1. Replace generic JSON mode with strict schema enforcement that explicitly lists every required key
2. Keep partial successful translations, only retry missing items
3. Split truncated batches in half instead of falling back to per-item requests

After the fix: 7 total API calls. $0.0017 total cost. 22.1 second runtime. Zero fallbacks. Exactly what everyone promised batching would do, but only after you fix the part everyone skips.

Stop optimizing for API call count. Optimize for cost per completed work item.

KV cache is no longer the bottleneck

Last week someone dropped a patch that changes everything for consumer hardware. Almost nobody noticed.

Qwen 3.6 27B Q4_K_M running on a single RTX 3090. Before this patch: 19 tok/s, 21GB total VRAM usage. After: 38.6 tok/s, 17.5GB total. Full native 256K context. Resident KV cache size: 72 MiB. That is not a typo. 94% of the cache is discarded.

Needle recall stays 88-100% across the full context window. HumanEval, GSM, MATH and agent test suites return identical pass rates to full cache. Outputs are not byte identical, but correctness is exactly the same.

For five years everyone just assumed you had to store every single key and value token for the entire context. It turns out you do not.

This will be standard in every inference engine by the end of the quarter. Nobody will announce it. Nobody will write a press release. It will just work, and everyone will get double the speed for free.

The 120B model class is dead

Nobody is releasing new 100-120B models. All new releases are either 25-35B dense or 200B+ MoE. There has not been a major model launched in this bracket in three months.

This was predictable. 120B MoE was the sweet spot for 2 x H100. Then 35B dense models got good enough to run on one H100, delivering 90% of the quality at 3x the speed. Then 200B MoE stabilized to run reliably on 4 x H100 and beat everything else.

There is no longer any price/performance point where a 120B model makes sense. It is the new 70B. Everyone will keep running the existing ones for another year, but there will not be another major release in this size class.

If you are picking a model to standardize on right now, do not pick 120B.

You can fine tune a domain expert on a T4

You do not need a cluster. You do not need 8 A100s. You do not even need a good GPU.

One engineer took Gemma 4, 120 aligned lines of 15th century Korean novel, standard LoRA, 5 epochs on a 16GB T4. That is the entire project.

Before fine tuning: base Gemma 4 scored 4.85% accuracy translating classical Joseon Korean to modern Korean. It had no idea what the obsolete letters or missing word spacing meant. It produced gibberish.

After 90 minutes of training: 85.71% accuracy. Indistinguishable from human translation for all but the most ambiguous passages.

No fancy tricks. No custom architecture. Just a good small dataset, correct label masking on the data collator so the model did not waste capacity learning the prompt, and reasonable hyperparameters.

This is the most underrated capability of modern small models. You can turn any 12B+ model into a world class expert on any extremely narrow domain in 90 minutes on consumer hardware. Almost nobody is doing this. Everyone is still trying to get GPT-7 to do it.

The best AI tools do not generate answers

ThinkMode is a chrome extension that does not call any LLM. It does not write code. It does not answer questions.

It just asks you what you are trying to do before you open ChatGPT. It suggests one of 5 thinking modes. It keeps track of how often you outsource your judgment. When you have done too much, it blocks the page for 5 minutes.

That is it. And it is the most interesting AI tool released this month.

Every single AI tool is built to go faster. This one is built to go slower. Every other tool tries to reduce the amount you think. This one tries to increase it.

We are just now starting to build tools that acknowledge what AI actually does to people, not just what it does to code.

EAGLE 3 lands in llama.cpp

Speculative decoding stopped being a research project this week. EAGLE 3 support merged into mainline llama.cpp.

For end users this means +30-70% generation speed on almost every model, zero measurable quality loss, zero extra VRAM required. You will just update llama.cpp one day and everything will be much faster. You will not even have to change any launch arguments.

This is how progress actually happens in this space. Not with big announcements. Just a merge commit on a Tuesday. Three months later everyone is using it and nobody remembers what it was like before.

AI is for joining the conversation

One of the best posts this month was not about architecture or benchmarks. It was from a Korean engineer who kept getting called a bot on Reddit.

He was not using AI to generate comments. He was using AI to translate his comments from Korean so he could participate. And every single time people would tell him his writing sounded AI generated.

So he built a small translator that turns Korean into natural sounding Reddit English. Not better English. Not perfect English. Just English that sounds like a human wrote it.

This is the actual use case for AI that nobody talks about. Not replacing workers. Not writing essays. Removing the friction that stops people from talking to each other.

All of the arguments about AI detection completely miss this. There are millions of people right now using AI not to cheat, but just to have a seat at the table.

What comes next

We have hit an inflection point for local LLMs.

For three years the entire field was chasing benchmarks. Every month there was a new bigger model, a new higher MMLU score, a new claim that this one was finally as good as GPT-4.

That era is over.

Right now the important work is not making models better. It is making them useful. It is fixing the broken optimizations. It is building tools that solve actual human problems. It is running things reliably on hardware normal people actually own.

Nobody will write press releases about this work. Nobody will get on stage at conferences. But this is the work that will actually stick.

This is the good part.