This is the worst time to be building production LLM agents. And the best.

We now have agent frameworks that can browse the web, write code, debug production incidents, operate lab equipment, and navigate 3D environments. Every week a new implementation hits 90% on some demo task.

At the exact same time, we have no reliable way to verify that any agent did the thing you asked it to do. Every single production agent deployment today fails silently, regularly, and most teams do not even measure how often this happens.

Every paper released this month confirms this. Every production failure posted this month confirms this. This is the state of the field right now.

The leniency bias that breaks all agent evaluation

First: stop using VLMs as judges for computer use agents.

OSReward is the first systematic benchmark of VLM judges for agent trajectories. The authors collected 100k human verified agent runs across Windows, macOS, Linux and web, then ran every major frontier and open model as a judge.

The result is not subtle. Every single tested judge has a strong, systematic leniency bias. They will label failed agent runs as successful between 28% and 61% of the time.

This is not a minor edge case. This is the foundation that every agent training pipeline is built on right now. Almost every computer use agent released in the last 6 months was trained on reward signals generated by exactly these judges.

The good ones are too expensive to run at scale. The ones you can afford are useless. OSReward's trained OS-Shepherd 35B matches GPT-4o judge performance at 40% lower cost, which is the first usable solution anyone has published. But this is a band-aid on a problem that almost no one was even admitting existed until last week.

Oncall agents are worse than you think

If you have ever thought "we could have an agent debug production incidents", ORCA-bench is the paper that will kill that idea for the next 18 months.

The authors built a fully instrumented 12 service microservice testbed, with 6 days of real production telemetry, 1079 real incident RCA tasks signed off by senior SREs. They ran every top general purpose coding agent against it.

Difficulty	Best agent accuracy	Human SRE accuracy
Easy	41.2%	92.7%
Medium	25.3%	88.1%
Hard	10.0%	79.4%

Claude Fable 5, the best performing model, got 10% on hard tasks. The weakest model hallucinated a completely made up root cause 40% of the time.

This is not a gap that will be closed by bigger models. This testbed is small, static, perfectly instrumented, and has no unknowns. Real production systems are 1000x larger, constantly changing, and half the telemetry is broken half the time. The numbers reported here are an optimistic lower bound.

Agents should only use tools when they actually need them

Almost every agentic visual reasoning system today wastes 70% of compute running tools on problems the model could already solve perfectly well on its own.

Beacon measured this properly. Existing agents will invoke expensive OCR, segmentation and grounding tools on 68% of all inputs. On 41% of those cases, the model would have gotten the answer correct without tools. On 17% of cases, using the tool introduced an error that would not have existed otherwise.

The net gain from tool use across the entire benchmark was 3.2%. The net cost increase was 290%.

Beacon fixes this with a reward signal that penalizes unnecessary tool invocation. It gets the same final accuracy, uses tools 22% of the time instead of 68%, and cuts total compute per task by 62%.

This is the most underrated result this month. Almost every agent architecture today is built around "use tools by default". That is wrong. The correct default is "never use a tool unless you are certain you cannot solve the problem without it".

Memory is reconstructed, not replayed

MemHarness dismantles every existing agent memory system in one paragraph.

All memory augmented agents today work the same way: retrieve similar past experiences, dump them verbatim into the context window, then ask the model to act.

This is not how human memory works. It is also not how good memory works. When you retrieve a memory you do not replay it verbatim. You adapt it, rewrite it, and reshape it to fit the situation you are in right now. Static replay causes negative transfer 27% of the time on ALFWorld and WebShop benchmarks.

MemHarness adds a single reconstruction step between retrieval and action. The model gets to rewrite the retrieved memory before using it. This one change improves benchmark scores by 18% and eliminates almost all negative transfer.

This is not a minor tweak. This is a fundamental correction to the base assumption that every agent memory system has been built on for the last three years.

Adaptive compute stops wasting tokens on easy problems

SVR shows that agents can learn when to stop thinking.

Right now almost all multi-turn agents run for a fixed number of steps, or stop when an external verifier says they are done. SVR trains the model to produce its own correctness verdict and confidence score at every step. It stops when it is confident it has the right answer.

On mathematical reasoning benchmarks SVR matches the accuracy of a fixed 10 turn run, while using an average of 2.99 turns. That is 70% fewer inference steps for identical end performance.

Crucially, the ground truth is only used during training. At inference time there is no oracle, no external judge, no extra API calls. The agent regulates its own compute budget.

Multi agent systems can rewire themselves at runtime

MANTA demonstrates something that has only been theory until now: multi agent systems do not need a fixed communication topology.

Instead of hardcoding that agent A reports to agent B, MANTA monitors collaboration traces during execution. If the current organization is not working, it will add links, remove links, reassign roles, or split agents. It does this while the task is running, no human intervention, no offline optimization.

Across 5 benchmarks MANTA outperforms the best fixed topology multi agent baseline by 5.8 percentage points. Most importantly, it gets better the longer the task runs. On tasks longer than 20 steps the gap opens to 12.1 points.

This is the first hint of what self improving agents will actually look like. They will not rewrite their own weights. They will rewrite how they talk to each other.

The silent production failures no one talks about

All of the above papers are about research benchmarks. The exact same failures are happening in production right now, every single day.

We have two public production postmortems this month that are perfect case studies:

1. RAG copilots cannot count. Every single RAG system deployed today will happily give you a wrong count when you ask how many documents match a query. It will count what is in the context window, not what is in the database. It will do this confidently, and you will have no idea it is wrong.
2. Browser agents cannot handle dynamic pages. Every selector based automation breaks the second someone adds a loading skeleton or renames a CSS class.

BrowserAct fixes this by throwing away selectors entirely. It inspects the page state, acts, then inspects again. This is not clever. This is obvious. And it is the first browser automation approach that actually works the way a human would work.

The unspoken rule for production agents

There is one rule that every single successful production agent deployment follows, and almost no one says it out loud:

The LLM is never the source of truth.

It never calculates. It never counts. It never verifies. It never remembers the date.

Anything that can be computed deterministically must be computed by code. The only job of the LLM is to translate intent into actions, and translate results back into natural language.

If you violate this rule your agent will work great on demos. It will fail silently and regularly in production. And you will not have any way to measure how often it is failing.

Where we are right now

We have crossed an important threshold. We can build agents that can successfully complete almost any task you give them, most of the time.

We have not crossed the next threshold. We cannot reliably tell when they have failed.

This is the bottleneck. This is the problem that every paper this month is dancing around. This is the problem that will occupy this field for the next two years.

You can build an agent today. You cannot trust it. And until you can evaluate it, you should not deploy it.