Stop testing answers, start testing actions

This is the unifying theme across all seven papers. Every single one rejects the old safety testing model where you grade text outputs for correct phrasing. All of them measure what the system actually does: disclose private data, navigate to a wrong coordinate, endorse a lie, run a malicious command, reject a job applicant, repeat bad information across a group.

If you are still running safety evals that check if the model says it will not do bad things, you are testing marketing copy. You are not testing the deployed system.

Synthetic data auditing without canaries

For two years every synthetic data audit method required you to insert secret canary records into training data, train shadow models, or run membership inference attacks that cost tens of thousands of dollars per run. That is over.

Phantoms and Disclosures introduces a black box framework that only requires two things: the synthetic output you got from the model, and a held out set of real records that were never shown to the generator.

It works by running exact and fuzzy match counts across both sets, then runs a simple binomial hypothesis test to separate three cases:

1. Zero leakage baseline: matches occur at the same rate in synthetic output as you would expect from random generation
2. Phantom disclosure: a real record appears by chance, at a rate consistent with the generator's base output distribution
3. True disclosure: the record appears statistically significantly more often than baseline, proving memorization

No model access. No training runs. The entire audit runs on a laptop in 12 minutes. The authors show it produces tighter leakage bounds than all prior shadow model methods, at 0.1% of the compute cost.

This is not an incremental improvement. This obsoletes every existing synthetic data privacy audit. You can implement this tomorrow.

Embodied agents will lie about what they can see

Vision language models running on robots, security cameras, and assistance systems do not refuse. They guess.

When you ask an embodied agent "where is the red notebook?" and it has never seen that notebook, it will not say "I don't know". 92% of the time it will pick a plausible coordinate from memory and send you there. Until this paper there was no standard test for this failure mode, and no good mitigation.

Semantic Flip solves this without retraining the base VLM. You generate OOD test pairs by swapping semantic attributes independently across query and memory: take a valid query about a cup, swap the noun to notebook, keep the rest. Train a single linear rejection head on these synthetic pairs. That is it.

On the new SpaceReject benchmark this method hits 0.956 F1 refusal score. All prompt engineering baselines scored below 0.62. The rejection head adds 0.7ms to inference latency.

You should add this module to every embodied VLM pipeline you run. There is no downside.

Search agents are trivial to poison

SearchGEO is the first properly controlled evaluation of LLM search agent vulnerability to web content manipulation. The authors tested 13 production model backends across five attack modes.

The results are stark:

Model	Overall attack success rate
Claude Sonnet 4.6	0.0%
GPT-4o	11.2%
Gemini Advanced	22.7%
Gemini 3 Flash	31.4%

Attack success here means the agent explicitly endorses a false claim placed on a single random web page returned in the top 20 search results.

Worse: when the output is not just text but an executable command, the split flips. Claude will reject all valid commands as a precaution. GPT will run any command that appears once anywhere on the web. There is currently no production backend that gets this right.

If you run any agent that calls search, you cannot trust its conclusions. You need to add independent source cross checking right now.

UNIATTACK breaks every existing defense

All current LLM safety defenses can be broken. This is not news. What is news is that they can all be broken with the same single prompt template, in one shot, no iterative probing.

UNIATTACK extracts 7 invariant attack features from 42 published jailbreak methods. It composes these features into a generic template that achieves average 78% attack success rate across all major public and closed models, including models with RLHF, system prompt defenses, input moderation and external safety guardrails.

It beats all prior jailbreak benchmarks by between 64% and 249%. It costs 0.03% of the compute of iterative attack methods.

This is the standard red team tool you should be using from today onwards. Any safety test that does not include UNIATTACK is not a real test.

Fairness testing was wrong by 240%

Every LLM fairness benchmark published before this month overstated measured demographic disparity by a factor of approximately 2.4.

AgentFairBench demonstrates that the standard methodology of comparing outcome spread across 6 demographic groups against a single run baseline is fundamentally broken. The spread you measure is almost entirely sampling noise, not actual bias.

When you run an arity matched null test: run the exact same profile twice with no demographic changes, measure the outcome spread between those two identical runs, then compare against demographic spread, almost all measured disparities disappear.

In their pilot test, Claude Haiku 4.5 showed zero statistically significant demographic disparity across hiring, lending and medical triage domains. All prior published benchmarks had reported large bias for this exact model.

This paper invalidates almost every LLM fairness result you have read. Use their open harness. It costs $7 to run a full audit per model.

Misinformation propagates through multi-agent debate

Multi-agent debate is sold as a robustness improvement. It is not. It is a transmission mechanism for misinformation.

When a single agent in a 5 agent group receives bad information, 68% of the time the entire group will converge to the wrong answer. Majority voting reduces this to 31%. Consensus protocols make it worse at 76%.

This is true even when all agents are perfectly aligned, none are malicious, and the bad information was just an error from a tool call. Errors do not get corrected in debate. They get repeated and reinforced.

If you run multi-agent systems you must add independent out of band validation for every factual claim. Do not trust group agreement. Group agreement means the error spread successfully.

KSAFE-MM: multimodal safety is not language agnostic

Nearly all existing safety benchmarks are in English. They do not catch failure modes that only appear in other languages, cultural contexts, or when text is combined with images.

KSAFE-MM is a Korean multimodal safety benchmark with 14,135 test cases across 11 risk categories. Initial tests show that all major MLLMs score 20-40% worse on this benchmark than they do on equivalent English safety tests.

Safety does not transfer. You cannot test your model once in English and assume it is safe for other users.

What you should do this week

None of this work is theoretical. All code is released. All benchmarks run for less than $100.

Add these four things to your production test pipeline:

1. Run UNIATTACK against every deployed model endpoint
2. Run the synthetic data audit for any generator you use
3. Run the AgentFairBench fairness harness
4. Add Semantic Flip refusal to any embodied VLM

Stop running output grading evals. Stop trusting safety claims from model vendors. Test what your system actually does.

That is the only thing that matters.