The quiet stack upgrade

If you only follow headline announcements you missed it. There was no keynote, no press release, no viral X thread. Over six weeks, seven separate releases landed across the Hugging Face ecosystem that together upgrade almost every part of the production ML workflow.

This isn't incremental polish. This is the point where the open ML stack stopped just catching up to closed tools, and started building things that closed vendors will never ship.

Every tool covered here is usable today. Every one solves a real pain point that every ML engineer has complained about in the last year.

Papers Without Code: fixing SOTA leaderboards

Niels Rogge from Hugging Face quietly relaunched PapersWithCode last week. Almost no one noticed. That is a tragedy.

For the last three years PwC was effectively abandoned. Leaderboards were stale, entries were manually curated, closed models were entirely excluded, and half the benchmarks had not been updated since 2023.

The new version fixes all of that. It parses every new paper on arXiv and Hugging Face automatically, extracts evaluation scores, and builds leaderboards for every benchmark. Closed models are included, tagged explicitly, and can be toggled off with one click.

You can now see exactly where open models stand without scrolling past GPT-5.5 and Mythos 5 taking every top spot. You can also view every benchmark as a scatter plot of performance vs parameter count, which is the only view that actually matters for practical model selection.

This is not perfect. Automatic parsing will miss edge cases. There is no verification that scores were run correctly. It is still the best public SOTA reference that exists. There is no alternative.

olmo-eval: evaluation for people actually building models

Most evaluation tools are built for marketing. They run once after a model is finished, produce a single number you put in a blog post, and then you never touch them again.

No one building a model uses those tools. When you are training you run evaluations 10 times a day, against every checkpoint. You don't care about the final score. You care whether the change you just made actually improved anything, or if you are just looking at noise.

That is the problem olmo-eval solves. It was built by the Allen Institute for AI for their internal Olmo model training pipeline, and they just open sourced the whole thing.

olmo-eval does three things no other tool does:

1. It reports standard error and minimum detectable effect for every score. You will never again waste three days chasing a 1.2% improvement that is just random variation.
2. It does pairwise per-question comparison between checkpoints. You can line up two runs and see exactly which questions the model started getting right, and which it broke.
3. It separates benchmark definition from runtime. You can run the exact same benchmark as a raw completion, with tool access, or inside a sandbox without rewriting a single line of the evaluation.

There is one catch. This is not a turnkey benchmark suite. This is a workbench. You will have to write code. If you are just running standard public benchmarks once, use LM Evaluation Harness. If you are training a model, stop everything and switch to olmo-eval today.

PyTorch profiling: stop guessing about kernel fusion

Hugging Face published the second part of their PyTorch profiling guide. This is the single most useful piece of technical writing about PyTorch performance published in the last two years.

Most engineers treat torch.compile as a magic button. You add the flag, things get faster, and you don't ask questions. When it doesn't work you shrug and move on.

This guide walks you through exactly what happens when you compile a model, kernel by kernel. It explains why nn.Linear already uses fused bias add so compile does nothing for single layers. It explains what the suffixes in CUTLASS kernel names mean, and how you can read them to verify that you are actually running the code you think you are.

It also explains one fact that almost no one knows: almost all of the performance gain from torch.compile does not come from fusing GPU kernels. It comes from removing CPU dispatch overhead. For small batch sizes, 80% of the improvement is just eliminating metadata operations that never touched the GPU at all.

Go read this guide. Twice. Keep it open while you profile. You will learn more from it than from every other PyTorch performance tutorial combined.

North Mini Code: what a good agent coding model actually looks like

Cohere released North Mini Code this month. It is the first open coding model that is actually good enough to use for real agent work.

This is a 30B MoE with 3B active parameters, released under Apache 2.0. It beats every open model under 120B parameters on agent coding benchmarks. More importantly, it works across every agent harness.

Most coding models are overfit to exactly one benchmark harness. They get great numbers on SWE-Agent, and completely fall apart if you change even one line of the prompt template. North Mini Code was explicitly trained on multiple harnesses during fine tuning. It loses less than 5% performance when moved between SWE-Agent, OpenCode and mini-SWE-agent.

The release notes are also a masterclass in honest technical reporting. They explain exactly how their RL training pipeline works, why they use CISPO instead of GRPO, how they avoided benchmark leakage, and exactly what tradeoffs they made at every step.

This is now the default baseline for any open coding agent. Every other model in this size class is obsolete.

Run your GitHub CI on Hugging Face Jobs

GitHub Actions is terrible for ML projects. CPU runners are slow. GPU runners are prohibitively expensive. For most open source ML libraries, running proper GPU tests is effectively impossible.

Hugging Face released a bridge that lets you run GitHub Actions jobs directly on Hugging Face Jobs. The entire change to your workflow is replacing runs-on: ubuntu-latest with runs-on: hf-jobs-t4-small.

That is it. No other changes. Logs stream back normally. All existing actions work.

For Trackio, the team that built this bridge, CPU CI runs 30% faster than GitHub hosted runners. A full GPU test suite runs on a T4 for 45 seconds and costs less than one cent.

This is not just cheaper. This is the first time small open source ML projects can run real GPU CI on every PR without maintaining self hosted runners.

There are limitations. You cannot run Windows or macOS jobs. Startup latency is around 10 seconds. For 95% of ML projects this is strictly better than GitHub hosted runners.

ms-swift: the fine-tuning framework no one talks about but everyone uses

ms-swift hit GitHub trending this month. If you have not heard of it, that is normal. Almost no one in the west talks about it. Almost everyone who actually fine tunes models for production uses it.

This is a full pipeline fine tuning framework from Alibaba's ModelScope team. It supports 1000+ models, every fine tuning method, every RLHF algorithm, every parallelism strategy, and every inference backend.

It does everything that TRL, Axolotl and LLaMA Factory do, and it does most of them better. It has native support for every variant of GRPO, full Megatron parallelism for MoE models, multimodal training, and one of the only working implementations of Ulysses sequence parallelism.

There is one catch. Most of the documentation is still only in Chinese. The English docs are incomplete. But if you can work past that, this is currently the best general purpose fine tuning framework available open source.

It is also the only framework that reliably works out of the box for new models. When Qwen 3.5 dropped, ms-swift had full support within 12 hours. Hugging Face TRL took 9 days.

mlx-lm: Apple silicon stopped being a toy

mlx-lm 0.20 landed last week. This is the point where running LLMs locally on Apple silicon stopped being a gimmick and became a serious development tool.

You can now pull any model directly from Hugging Face, quantize it, fine tune it, and run inference with one command. Performance on M3 Ultra matches A10G inference throughput for most models up to 70B parameters.

Prompt caching works correctly. Distributed fine tuning across multiple Apple silicon machines works. Streaming works. Every common sampling option is supported.

This is now the fastest way to iterate on small fine tuning runs. You can test a LoRA training run on your laptop in 10 minutes instead of waiting 2 hours for a cloud GPU to spin up.

There are still rough edges. Large models over 128B parameters are still slow. Support for multimodal models is incomplete. For anything under 70B parameters this is now the best local development experience available.

What this all adds up to

None of these releases got a big launch. None had marketing budgets. None went viral.

Together they form a complete end to end stack that you can use today to build, train, evaluate, test and deploy state of the art ML models, entirely on open source tooling, without any lock in.

Six months ago you needed at least three closed commercial tools to do this. Today you can do all of it with tools released in the last six weeks.

This is the part of the cycle where open source pulls ahead. Closed vendors are focused on launching bigger models. The open ecosystem is focused on building the tools that let normal engineers actually use them.

Closing notes

None of these tools are perfect. All of them have bugs, missing documentation and rough edges. All of them are already better than the alternatives.

If you are still using the same stack you were using at the start of this year, you are working with obsolete tools. Go try one of these this week.