If you run open LLMs in production or locally, you can stop checking twitter for 2 weeks. Everything that mattered landed this week.

The new baseline MoE: Ling-3.0-flash

Almost everyone missed this release. It dropped directly between the Kimi K3 leak, DeepSeek V4 Flash launch and Qwen 3.8 hype cycle, and got zero press. That is a mistake.

Ling-3.0-flash is a 127.5B total parameter MoE with 5.1B active parameters per token. It was released ungated, MIT licensed, with official BF16 and FP8 checkpoints. The official FP8 quant comes in at exactly 128GB on disk. This is the first >100B open model that will run unmodified on a single 128GB unified memory server. No multi-GPU required. No custom quant hacks.

This crosses an extremely important deployment threshold. Until this week every production grade open MoE required at least two A100s. Now you can run one on a cheap bare metal dedicated server for $120/month.

MoE expert granularity nobody is discussing

Ling uses an architecture choice that no other released model has adopted at this scale: 512 total experts, 8 active per token.

Every other production MoE right now runs 16-128 total experts with 2-4 active. Finer grained routing dramatically reduces expert specialization drift, cuts routing conflict, and most importantly lets you stream experts on demand instead of loading the entire model into memory. You do not keep 127B of parameters resident. You keep routing tables and the last 12 accessed experts hot. That is how this fits in 128GB.

This is not an incremental improvement. This is a different operating model for large MoEs.

Llama.cpp got the biggest speedup in 6 months

PR #25532 landed this week, and almost nobody noticed. For years llama.cpp ran sampling logic on CPU even when running 100% of model computation on GPU. For users running speculative decoding this added a full round trip across PCIe every single decode step.

This PR moves sampling entirely to the GPU. Acceptance ratios are identical. There is zero quality loss. This is pure, free speed for every single llama.cpp user.

Measured gains across standard benchmarks on a Tesla P40:

Workload	CPU sampling tok/s	GPU sampling tok/s	Delta
code_python	73.1	76.2	+4.2%
code_cpp	75.9	79.4	+4.6%
explain_concept	62.4	64.6	+3.5%
summarize	59.6	61.6	+3.4%
qa_factual	62.4	64.6	+3.5%
translation	67.0	69.6	+3.9%
creative_short	50.7	52.1	+2.8%
stepwise_math	73.6	76.6	+4.1%
long_code_review	63.8	65.7	+3.0%

On RTX 5090 this delivers a consistent 8% throughput increase. This is the largest single performance gain posted to llama.cpp in the last 6 months. It sat as an uncommented open PR for 11 days before anyone ran benchmarks.

Running 284B DeepSeek V4 Flash on one 32GB 5090

This is the most important deployment trick posted all year. Two weeks ago everyone agreed you needed at least 4x A100 to run DeepSeek V4 Flash. This week someone got it running usable on a consumer desktop.

The trick is extremely simple. You do not load all MoE experts onto the GPU. You keep only the first two full MoE layers on the GPU. Every other expert lives in system DDR5. You stream them over PCIe on demand during routing.

This setup delivers full native 1M context, 12-18 tok/s decode, and works perfectly for agentic coding workloads. It is not a toy. This is production usable performance on hardware that costs less than one month of A100 reserved cloud time.

Speculative decoding breaks during reasoning

This is the unspoken problem every single production deployment is hitting right now. Everyone turned on speculative decoding last month. Everyone noticed it gets slow when the model is thinking. Nobody published hard numbers until this week.

Speculative draft acceptance rate drops from ~90% during normal output to 30-40% during internal reasoning blocks. Throughput tracks this ratio almost perfectly. There is no magic fix. The model is doing actual work. You cannot reliably predict what it will output next.

The optimization is trivial. You just drop speculative depth from 2 tokens down to 1 token while the model is emitting thinking tags. That delivers an immediate 30% throughput increase during reasoning. No other changes required.

As of today vLLM does not do this. SGLang does not do this. No public inference runtime implements this one if statement.

LM Studio abandoned local inference

This is not drama. This is an industry signal. The most popular local LLM GUI has decided there is no money in running local models. There is only money in agent orchestration and cloud upsells.

They did not accidentally hide the download link. They are actively migrating their user base. The original LM Studio application now receives only maintenance patches required to integrate with the new Bionic agent product. All new development work has moved.

If you use LM Studio today, start planning your migration. It will not be maintained as a first class product 6 months from now.

What everyone is migrating to

Right now the exit path from LM Studio splits three clean ways:

1. llama.cpp + any lightweight web frontend. This is the default for 90% of users leaving. It works, it is fast, it supports every model released.
2. Open WebUI + vLLM backend. For users that want multi user support and OpenAI compatible API endpoints.
3. SGLang. For production deployments that care about maximum batch throughput.

Nobody is moving to another closed source GUI. Everyone is moving to composable open components. That is the quiet shift this week.

Upcoming models that will reset this again

This baseline will not last. Qwen 3.8 27B open weights land next week. Early benchmarks show it matches or beats every 70B model ever released. Qwen 3.8 Max is already performing even with DeepSeek V4 and Kimi K3 across all benchmark categories, and is 30% faster at inference.

Gemma 4 runs usable inference on 500MB of memory. We will have production grade general purpose models that run on smart watches before the end of this year.

The thing nobody is saying

All of this progress is not coming from OpenAI, Anthropic or Google. It is not coming from VC funded startups. Every single one of these breakthroughs was posted by random people on reddit.

This week we had a random engineer release a 124B open MoE. A random contributor submitted an 8% speedup to the most widely used LLM runtime on earth. A random guy running Linux Mint figured out how to run a 284B model on consumer hardware.

None of this was announced at a conference. None of this had a press release. This is just the normal pace of open LLM development right now. If you are still waiting for official vendor roadmaps you are already 6 months behind.

Closing notes

If you are deploying open LLMs this month:

• Test Ling-3.0-flash. It is the new price performance king for single node deployments.
• Apply the llama.cpp sampling patch. It is free speed with no downsides.
• Add the speculative depth switch for reasoning. That is another 30% free speed.
• Migrate away from LM Studio before they disable the old client.

This is not the peak. This is just the baseline now. Every month this gets better by 20-30%. There is no sign this is slowing down.