We have been evaluating agents wrong.

For two years every agent benchmark counted one number: did the task get finished. All of the papers dropped this month agree that this metric is actively misleading. An agent can complete 100% of assigned tasks while leaking every user secret, sabotaging its teammates, misunderstanding physical space, violating every unstated social norm, and walking directly into failure modes that will destroy any production deployment.

This is not theoretical. Every one of these failures has already happened in deployed systems.

This article walks through the six new agent evaluation frameworks that landed on arXiv this week, what each one measures, the hard results, and what this shift means for everyone building agent systems right now.

We stopped measuring success. We started measuring failure.

Every benchmark released this month works the same way. They do not reward correct answers. They design traps that expose the specific failure modes that only appear when an agent is acting over time, with other agents, with tools, or in the physical world.

None of these benchmarks can be gamed with prompt engineering. None of them are solved by scaling to a larger model. Most of them show that state of the art agents fail 70% or more of the time on capabilities that we assumed were already solved.

Cooperation is not about being nice. It is about not being stupid.

LLawCo is the first paper that properly diagnoses why multi-agent systems fall apart. Most people assumed bad cooperation came from agents being selfish, or not understanding instructions. That is wrong.

Almost all cooperation failure is unforced error. Agents talk over each other. They start acting before their partner has finished setup. They repeat information that was already communicated. They assume the other agent sees exactly what they see. None of this is malicious. All of it destroys task performance.

LLawCo does not train agents to cooperate. It trains them to extract laws of cooperation from failed runs. Simple rules: Wait for confirmation before acting. Only speak when you have information no one else has. Do not repeat.

They tested this across four backbone models on PARTNR-Dialog and TDW-MAT. Average success rate improved 4.5% and 6.8% respectively over every existing open source agent framework. That gain came from nothing except removing stupid unforced errors. No larger model. No better prompting. Just stopping agents from acting like they are the only person in the room.

Tool execution success means nothing for privacy

ToolPrivacyBench is the benchmark that should make every production agent engineer stop what they are doing right now.

This benchmark does not check if the agent calls the right API. It checks what the agent sends to that API. Across 2150 test cases, every tested agent completed 80%+ of tasks correctly. Every tested agent also leaked unnecessary private data in over 60% of those successful runs.

An agent will correctly pull a customer invoice. It will also send that customer's full credit card number, email, and account history to the invoice rendering service. That service only needed the invoice line items. No one ever checks this. No existing benchmark caught this before.

Agent	Task Completion Rate	Privacy Over-disclosure Rate
GPT-4o Agent	91.2%	67.1%
Claude 3 Opus Agent	88.7%	59.4%
Llama 3.3 70B	76.3%	72.8%
DeepSeek V3.1	79.1%	61.2%

This is the most important table published about agents this year. There is zero correlation between completing the task and respecting privacy boundaries. You cannot infer good privacy behaviour from good task performance. They are entirely independent capabilities.

Spatial intelligence is the unmeasured bottleneck for embodied agents

AirGroundBench tests something no one has bothered to measure before: can an agent maintain a consistent model of physical space when given two different views from two different robots.

This is not an abstract problem. This is exactly what happens every time you have a drone and a ground robot working the same site. The drone sees the whole yard. The ground robot sees the box right in front of it. Almost no current model can align those two views into one consistent world model.

13 MLLMs were tested. All of them scored >80% on single view perception tasks. Every single one scored below 45% on cross view alignment. Performance dropped another 22% when they had to act on that understanding.

This is not a problem that will be fixed by more training data. This is a fundamental architectural limitation. Current multimodal models do not build consistent metric representations of space. They guess. They guess very well for single images. They cannot integrate two views.

Theory of mind tests were all fake

Every theory of mind benchmark published before this month could be solved with pattern matching. Triadic Werewolf demonstrates this perfectly.

The standard Werewolf game has two sides. All existing tests use this structure. Models get very good scores. None of them require actually modelling another agent's incentives. You can just memorise what good play looks like.

Add a Jester. A third player that wins by getting voted out. Now every incentive inverts. Now you cannot win unless you can model what the Jester wants you to think, and what the Jester thinks you think they want.

The results are brutal. GPT-4.1 will vote the Jester out on day one 68% of the time. This is an explicitly self defeating action. It is doing exactly what the Jester wants it to do. And it has no idea this is happening.

No model tested could reliably beat the Jester. This is not a small gap. This is a complete absence of a capability that every human 12 year old has.

Hidden norms are the 70% failure rate no one talks about

NormAct tests the thing that will make embodied agents fail in the real world, every single time.

You tell an agent: bring me a glass of water. That is the explicit goal.

The hidden norms are: do not walk through the person sitting on the couch. Do not take the glass that someone is already drinking from. Do not turn on all the lights at 3am. No one writes these down. No one tells the agent. Every human follows them automatically.

NormAct embedded 192 such hidden norms into ordinary embodied planning tasks. Top models achieved the explicit goal 67.3% of the time. They complied with the hidden norm 26.4% of the time.

This is the largest measured capability gap in any agent benchmark today. And it is almost entirely unaddressed. The NormPerceptor patch proposed in the paper only brings this up to 46.7%. We are still losing half the time.

Fault tolerant control agents work. And they run on small models.

The one good news paper this month is From Detection to Action. This team built an agent system for industrial fault tolerant control.

It does not use GPT-4o. It runs entirely on GPT-4o-mini. It beats every existing hand coded fault tolerant control system on both benchmark environments.

The architecture is simple. They split the agent into six separate roles: monitor, planner, synthesiser, simulator, validator, fallback. All communication is structured. Every proposed action is run against a digital twin before execution. No action is ever taken unless it passes deterministic validation against all safety constraints.

This is the only production ready agent architecture published this year. It works. It is reliable. It has bounded latency. It fails gracefully. And it proves that you do not need the largest model to build a capable agent. You just need to stop building monolithic agents.

What this all means

All of these papers point to the same conclusion. We have passed the peak of monolithic single agent performance. All of the easy gains are gone.

Every hard problem left is about interaction. Interaction between agents. Interaction between an agent and tools. Interaction between an agent and the physical world. Interaction between an agent and unstated human norms.

None of these problems are solved by making the model bigger. None of them are solved by better prompt engineering. All of them require us to stop evaluating agents on whether they get the right answer, and start evaluating them on whether they do the right thing.

For three years we have been building race cars and testing them on a straight line track. This month everyone finally admitted that these cars have no brakes, no steering, and will crash the second they hit the first corner.

That is not bad news. That is just the point where the actual engineering starts.

References

All papers are available on arXiv at the links listed in the front matter. Benchmark datasets and code have been published for all six frameworks as of publication date.