If you have ever built an LLM information extraction pipeline for internal use, you have almost certainly done this: write a good prompt, run 20 test documents, get 90% correct on the test set, roll it out, then watch it silently fail on 40% of real world input two weeks later.

This is the default outcome. Almost everyone stops there.

Over the last two weeks four separate teams published full papers on production LLM IE systems running at the German Central Bank, EU peacekeeping command, and computational political science research groups. None of them look like the demo pipelines you see on Twitter. All of them solved exactly the same problems.

This is not theoretical research. Every system described here is processing tens of thousands of documents for operational use.

The common production pipeline

Every single one of these teams arrived at almost exactly the same architecture. No exceptions.

You will notice there is no step labelled "prompt GPT-4o".

The LLM is only used for the two red stages. All other steps are traditional deterministic or embedding based code. This is the single most important pattern that every production implementation adopted.

None of these teams passed full documents to an LLM and asked for JSON output. That approach fails at scale. It fails silently. It cannot be audited.

Cross project comparison

All four projects operated under completely different constraints, domains and accuracy requirements. Yet they made almost identical tradeoffs.

Property	Political Network Mapping	Bundesbank Collateral Check	Peacekeeping Threat Assessment	ReaORE Open Relation
Total documents processed	11.2M	187,000	42,000	2.1M
Languages	7	2	4	13
Base model	Mistral 8x7B v0.3	Llama 3 70B	Llama 3 8B	Qwen 2 72B
Used proprietary APIs	No	No	No	No
Guided decoding enabled	Yes	Yes	Yes	Yes
Entity resolution stage	3 step cascade	2 step	1 step	2 step
Reported strict precision	68.2%	91.0%	79.4%	76.1%
Operational false positive rate	<2%	<1.2%	<4.7%	❤️.9%

Every team used open weight models. Not one single production deployment used OpenAI, Anthropic or any closed API. This is not an ideological choice. Closed APIs do not support the constrained decoding required for reliable extraction. They also cannot be run at the throughput required.

Evaluation is the broken part no one talks about

Every one of these papers explicitly rejected standard span based F1 scores as useless for operational IE.

Span based metrics score an extraction wrong if the character offset is off by one word. They will mark a correct extraction that uses a synonym as a total failure. For production use this is backwards. You do not care where the information was found. You only care if the extracted fact is true.

All four teams switched to LLM-as-judge evaluation for ground truth comparison. The Bundesbank team went one step further and built their evaluation entirely around operational error cost. For collateral verification, accepting an ineligible security is 17x worse than rejecting an eligible one. All model tuning was done to minimize this weighted loss, not F1.

This is the largest unspoken gap between demo and production IE. No one who runs these systems cares about benchmark F1. They care about the cost of each type of error.

Political elite network mapping

The largest and most technically sophisticated system described is the joint entity relation pipeline for mapping European political networks.

This system ingests 11 million news articles across 7 languages, builds a signed temporal knowledge graph of 1.2 million people and organisations, and extracts directed relationships including patronage, conflict, employment, ownership and court conviction.

Most notably this pipeline does not extract relations from co-occurrence. It explicitly extracts the sign and strength of every tie. This allowed the research team to correctly map the full 17 year fracture and collapse of the Austrian Freedom Party, including tracking individual members moving between factions, into government and eventually into criminal trials.

Strict accuracy on relation extraction was 68.2%. Lenient accuracy, counting any factually correct relation regardless of exact wording, was 93.7%.

This is the first automated system that can reliably reconstruct informal political networks at continental scale.

Bundesbank collateral eligibility extraction

Every week the German Central Bank verifies roughly 1200 new securities presented as collateral for monetary policy operations. Before this system every one of these was checked manually by an analyst reading 50-200 page prospectuses.

Manual verification took on average 112 minutes per security. The LLM pipeline completes the same check in 14 seconds. It achieves 91% document level accuracy, with a false acceptance rate of 1.1%.

This system is not replacing analysts. It is triaging. 72% of securities are now automatically accepted or rejected without human review. Analysts only review the remaining 28% of ambiguous cases.

Total analyst workload was reduced by 61%. No eligible security has ever been incorrectly rejected in 11 months of live operation.

Peacekeeping threat assessment

The EU Monitoring Mission in Georgia processes 150-200 open source media reports every day. Prior to this system analysts read every report manually to flag events relevant to mission security.

The LLM pipeline extracts structured threat events including location, actor, time, type and estimated severity. It achieves 79.4% agreement with senior analysts on threat rating.

Most importantly this system does not make final decisions. It ranks incoming reports by relevance. Analysts now see the 10 highest risk reports first, instead of reading every report in arrival order. Average response time for high severity threats dropped from 137 minutes to 21 minutes.

ReaORE: fixing open relation extraction

All the systems described above operate against a fixed predefined ontology of relations. Open relation extraction, where the system must discover and name new relation types that were not present during training, has remained mostly unusable until very recently.

ReaORE solves this with a two stage reasoning process. First it generates an initial candidate set of 8-12 possible relations. Then it runs explicit comparative reasoning between every pair of candidates to select the correct one.

This approach beats all existing open relation extraction baselines by 19-27% on standard datasets. Most importantly it almost eliminates the most common failure mode for LLM open IE: incorrectly merging distinct but similar relation types.

The four unwritten rules for production LLM IE

Every team followed exactly these rules. None of them are documented in most LLM guides.

1. Never ask the LLM to output the final structure directly. Always extract spans first, resolve entities, then extract relations. Any pipeline that does end to end extraction will have silent undetectable failure modes.
2. Always use guided decoding. Do not rely on prompts to enforce output schema. Force the model to only output valid tokens at every step. This single change reduces hallucination rates by 80-90% for extraction tasks.
3. Do not evaluate on span offsets. Evaluate on factual correctness of the output. Span F1 does not correlate with operational performance.
4. The LLM should never make the final decision. The correct role for an LLM IE system is triage, filtering and ranking. Humans should always handle edge cases.

What everyone is still getting wrong

None of these systems have good recall. All of them are tuned explicitly for precision. Every team accepted that they will miss things, in exchange for almost never outputting something that is wrong.

This is the correct tradeoff for almost every operational use case. No analyst will use a system that regularly outputs false facts. They will happily use a system that finds 70% of the correct facts with no garbage.

Almost every demo IE pipeline you see online is tuned the other way around. They optimize for high recall on test sets, and produce unusable garbage in production.

Closing observations

LLM information extraction stopped being a research problem sometime last year. It is now a solved engineering problem.

All the hard parts are no longer about the LLM itself. They are about entity resolution, evaluation, error cost modelling, and system integration. The best results do not come from better prompts or bigger models. They come from knowing where not to use an LLM.

If you are building one of these systems, you do not need to invent anything. Just copy the architecture that every one of these production teams already converged on.