Appearance
The quiet threshold no one is talking about
Last week three preprints landed on arXiv within 48 hours of each other. None got viral twitter threads. None had animated demo videos. None claimed AGI.
Together they are the most important LLM application result published this year.
For ten years we have talked about AI for science. Almost all of that discussion has been about grand things: designing new drugs, proving mathematical theorems, solving climate models. Almost none of it has been about the actual day to day work that occupies 95% of every researcher's time. Reading bad methods sections. Translating written descriptions into working code. Checking that someone else did not make an arithmetic error. Writing boilerplate analysis routines that no one will ever cite you for.
That is the work that just got automated.
Reproducibility audits: LLMs beat human reanalysts
The first paper tested LLMs on the single most broken part of modern empirical science: independent reproducibility checks.
For this study the authors used 76 published social and behavioral science papers. Every one of these papers had already received a full independent reanalysis by trained human researchers as part of a large scale reproducibility project. This was not a synthetic benchmark. This was real work, done by real people, paid to do this correctly.
The LLM pipeline was given exactly the same input as the humans: the original paper, the raw study data, and nothing else. It was told to reproduce the analysis, extract effect sizes, and state whether the published claim held.
No one expected this result.
At +/-0.05 tolerance on Cohen's d, the LLM recovered the original published effect size in 41% of studies. Human reanalysts achieved 34%. On the far more important metric of final qualitative conclusion, the LLM agreed with the original study's claim in 96% of cases where the human reanalysis confirmed the result. Humans hit 74%.
Read those numbers again. The machine did better.
What the reproducibility numbers actually mean
This result breaks every standard assumption about reproducibility work.
Everyone has repeated for decades that reproducibility checking requires deep domain expertise, careful judgement, and human oversight. That is true for the hard edge cases. It turns out it is not true for 90% of cases. Most reproducibility work is not clever. It is tedious, unglamorous translation. It is reading a badly written 3 paragraph methods section, mapping that description to analysis steps, and not making stupid off-by-one errors.
Humans are terrible at this. We get bored. We skip lines. We give the original authors the benefit of the doubt. We assume if it was published someone must have checked it.
LLMs do not get bored. They do not give anyone the benefit of the doubt. They do exactly what is written on the page. That makes them better at this job.
This is not a small marginal improvement. This is a 29% reduction in error rate for conclusions. That is enough to change the entire field.
The failure cases that matter
The LLM pipeline failed completely on 7 out of 76 papers. It could not produce any usable effect size estimate at all.
This is not a flaw. This is the best feature of the system. When the LLM could not do the work, it said so. It did not guess. It did not produce plausible wrong numbers. It stopped and reported failure.
Humans almost never do this. When a human reanalyst cannot replicate a result, they will almost always keep adjusting code until they get something close to the published number. They will blame their own implementation before they blame the original paper.
All of the LLM failures were cases where the original paper's methods section was ambiguous, incomplete, or internally contradictory. These are exactly the cases where human reanalysis also produces unreliable results. No system can reproduce work that was never properly described in the first place.
AgentRivet: automating the boring work no one will do
The second paper addresses a problem that has crippled particle physics for 15 years.
When the LHC experiments publish a measurement, they are supposed to release a Rivet routine: a standard C++ implementation that allows any theorist to compare new models against the measured data. These routines are the foundation of all analysis after publication.
Only 39% of published measurements actually have one.
No one writes them. It is miserable work. It takes three days of careful reading and debugging. You will never get a paper for it. You will never get credit. Every single person in the field agrees this is an existential disaster. No one will fix it.
AgentRivet is an LLM agent workflow that takes a published paper and outputs a working Rivet routine. It runs intermediate syntax checks, cross references measurement definitions, and validates output distributions against published plots.
On test cases from recent ATLAS and CMS measurements, AgentRivet produced runnable routines with zero syntax errors in 82% of attempts. Most remaining physics errors were not failures of the LLM. They were ambiguities in the original publication that human implementers would also have resolved incorrectly.
This system will do more to enable new physics over the next two years than every LHC upgrade currently planned.
The three layer framework for discovery
The third paper provides the conceptual framework that ties all this work together. It argues that all discussion of AI in science has been stuck talking about only two capabilities, while completely ignoring the one that actually matters for discovery.
The paper defines three layers:
- Search and retrieval over existing knowledge
- Structural model formation and qualitative reasoning
- Execution, optimization and refinement
Almost every existing system lives entirely in layers 1 and 3. We build RAG systems that find papers. We build agents that run simulations, fit parameters, and generate code. None of this is discovery.
Discovery does not happen when you run an existing model faster. It happens when you notice that your entire existing model is wrong, and that the solution to your problem exists in an unrelated field that no one working on your problem has ever looked at.
Layer 2 is the thing everyone is missing
Layer 2 reasoning has a very clear signature. An existing framework has hit a wall. Everyone is trying incremental improvements. No one is questioning the framework itself. The solution comes from importing a complete conceptual object from a different domain.
The paper gives three confirmed examples:
- Chern's 1943 proof of the Gauss-Bonnet theorem, which imported methods from topology that no geometer was using at the time
- The 2016 resolution of the Nesterov gradient convergence problem using Lyapunov functions from control theory
- OpenAI's 2026 disproof of the Erdos unit distance conjecture, which used combinatorial methods from coding theory
That last one was done by an LLM. It was not brute force search. It was not optimization. It was structural reasoning across domain boundaries.
This is the capability that no one is measuring on benchmarks. This is the capability that will actually change what science can do. Right now it works very rarely, and only on formal well defined problems. But it works at all. That is the threshold.
What this changes for ML engineers
Almost all LLM application work today is wasted. We keep building demo chatbots. We keep building "ask your paper" tools. We keep optimizing for longer context windows and more natural conversation.
Scientists do not want to chat with their papers. They do not want to ask questions. They want systems that do work without being asked.
Stop building interfaces that require a human to prompt the model. Start building systems that monitor arXiv, pull every new paper, run the reproducibility check, generate the analysis code, flag ambiguities, and attach the output to the paper before any human ever reads it.
That is what people will actually use. That is what will move the needle.
Limitations that will not go away soon
None of this means human researchers are obsolete.
All of these systems fail silently on occasion. All of them will confidently produce wrong answers for problems that sit just outside their distribution. None of them can judge whether a result is actually important. None of them know which question to ask next.
Layer 2 reasoning remains extremely rare. It will be decades before it is reliable enough to use without close supervision. For most fields most of the time, it will not work at all.
But none of that matters. You do not need perfect systems. You only need systems that are better than the alternative. For most of the routine work of science, that bar was already extremely low.
What comes next
By the end of 2027, every new empirical paper posted to arXiv will have an automated reproducibility report attached automatically. Most researchers will not even remember that this used to be done by humans.
By 2028, standard analysis code for most fields will be generated on publication. Journals will start requiring automated validation before accepting papers.
This will not be announced. There will be no press release. It will just happen, quietly, one workflow at a time.
This is how real change arrives. Not with a flashy demo. Not with a grand announcement. With three boring preprints, posted on a Tuesday, that no one you follow talked about.
References
- Automated reproducibility assessments in the social and behavioral sciences using large language models http://arxiv.org/abs/2606.13670v1
- A Three-Layer Framework for AI in Scientific Discovery http://arxiv.org/abs/2606.13566v1
- AgentRivet: an automated system for producing Rivet routines from journal publications http://arxiv.org/abs/2606.13535v1