Nobody talks about this at conference keynotes. 90% of the work that goes into a good production LLM is not the architecture, not the parallelism trick, not the learning rate schedule. It is selecting which 3% of the raw crawled data you actually train on, and after training, proving that none of that 3% is going to blow up your model in production.

For the last three years everyone working on this has been using terrible makeshift tools. This week two papers dropped that change this. We will look at both, break down exactly what they do, where they fail, and what this means for anyone training models right now.

No one was actually using influence functions

Influence functions are the correct mathematical answer to data attribution. Given a trained model and an output, you can compute exactly which training samples contributed most to that output.

In theory.

In practice, for a 7B parameter model, running a single influence query took approximately 12 hours on an A100 last year. For a 70B model it was not feasible at all. Every published paper on this ran on 124M parameter toy models. No production team used them. Everyone instead did brute force leave-one-out ablation if they absolutely had to, or more commonly just guessed.

This is why every LLM safety team is effectively flying blind. When your model outputs copyrighted text, or toxic garbage, or a dangerous instruction, right now you cannot reliably answer the question: which exact line in the training set caused this.

How Influcoder works

Influcoder does not improve influence function calculation. It abandons running them at query time entirely.

The authors observed that gradient influence rankings are extremely consistent across decoder checkpoints. Once you have run influence calculations once for a small subset of samples, you can distill that ranking signal into a tiny encoder that will reproduce the rankings for every other sample in the dataset.

This is a gloriously obvious hack that nobody thought to try for five years.

You run full exact influence functions for 0.1% of your training corpus once, after pretraining finishes. You then train a 120M parameter BERT encoder to predict the influence rank of any sample given only its final layer embedding. That is it. There are no other tricks.

The resulting encoder runs 11,000x faster than exact influence calculation. It reproduces the top 100 most influential samples with 92% recall. For production use cases that is more than good enough. You can run attribution queries for every single generation your model serves, in real time, for negligible overhead.

Influcoder limitations and tradeoffs

This is not perfect. It will miss edge cases. The encoder will not correctly attribute samples that are semantically unique and did not appear in the 0.1% calibration set. It will also undercount influence for samples that appear multiple times near duplicates in the training corpus.

That is an acceptable tradeoff. Before this you had two options: 100% accurate results that take three days per query, or zero results at all. Now you have 92% accurate results that take 2ms. This is the difference between a mathematical curiosity and a tool you run on every production model.

The authors also note this does not work for fine tuning attribution yet. The consistency of influence rankings breaks down during parameter efficient fine tuning runs. That is the next obvious open problem.

Speech translation training data is much worse than you think

Turn to the second paper. Everyone building end to end speech to speech models already knew this. Mined parallel speech corpora are garbage.

Typical open datasets like SpeechMatrix have between 18% and 32% bad pairs. This includes completely misaligned audio, swapped languages, silent tracks, audio from completely different utterances, and semantic drift so bad the two sentences have nothing in common.

Until now all filtering worked by first running ASR on both sides, then running text based semantic similarity. This approach discards all acoustic information, introduces error from the ASR model, and still misses half the bad pairs. Everyone accepted this as unavoidable.

The rank to distill filtering pipeline

This paper introduces an approach that works entirely in the audio domain, no transcription required.

First you build a weak cheap ranker. This uses the old ASR + text similarity method. Run it over the entire corpus, generate keep / drop pseudo labels. You know these labels are noisy. That is fine.

Then you train a 300M parameter audio LLM to predict the keep / drop label directly from the raw paired audio waveforms. You do not give it text at all. You do not correct the noisy labels.

During training the audio LLM ignores the label noise. It learns the actual underlying signal that the weak ranker was approximating. It learns to detect clipped audio, background noise, misalignment, timing drift and semantic consistency all directly from the waveform.

This is not intuitive. It works extremely well.

Benchmark results

On CVSS-C German to English, training on unfiltered data gave 17.2 ASR-BLEU. Training on data filtered with the old text based method gave 17.8. Training on data filtered with this audio LLM gave 18.6.

That is +1.4 BLEU. For reference that is roughly the gain you get from doubling the size of your model, or doubling the amount of training data. You get this for free, just by throwing away the bad data correctly.

Even better: this filter generalizes across language pairs. You train the filter once, you can run it on any language pair you have not seen before. No additional labels required.

What this means for data curation

For ten years the received wisdom was that more training data is always better. That was only true because nobody could filter data well enough. We now know that for almost all model training runs, you will get better performance if you throw away the worst 30% of your dataset.

This effect is consistent across text, speech, vision and multimodal models. It is almost never talked about publicly because no vendor wants to admit that 70% of the trillion token corpus they are bragging about is actively harming model performance.

Attribution is not just for safety

Most discussion around data attribution stops at safety and copyright. That is the boring use case.

The real value is closing the training loop. Once you can reliably attribute every model output back to training samples you can do things that were impossible before:

• Automatically remove training samples that cause bad outputs, then retrain
• Rank every sample in your corpus by how much it actually improves downstream performance
• Stop wasting compute training on useless data
• Prove exactly which training data contributed to any benchmark result

This is the feedback loop that will make the next generation of models actually improve reliably instead of just getting bigger.

The quiet shift in LLM research

These two papers are representative of a very welcome shift happening right now. After three years of papers announcing larger models, new attention variants and useless benchmark records, people are finally starting to solve the actual boring hard problems that stop anyone from building reliable production models.

Nobody will win a best paper award for Influcoder. It is not flashy. It does not have a cool demo. It just solves a problem that every single LLM engineering team on the planet has been dealing with every single week for the last four years.

That is good research.

Open problems remaining

We are still very far from done. Neither paper solves the core problem of attribution across fine tuning stages. We still have no good way to measure how much a single training sample contributes across the entire model lifetime. We still cannot reliably attribute emergent behaviour. We still cannot tell the difference between a sample that was memorized and a sample that generalized.

But this is real progress. For the first time, we have tools that actually work at production scale. If you are training models this week, you should go implement both of these methods right now.

---

References

1. Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution http://arxiv.org/abs/2606.13668v1
2. Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data http://arxiv.org/abs/2606.13507v1