Appearance
Every ML engineer building self hosted RAG right now is making a choice they don't even know they are making.
When you add a memory layer to your LLM agent, you are not picking between better and worse retrieval. You are picking one of two fundamentally incompatible architectures. No benchmark aggregate will tell you this. Almost no vendor will tell you this.
One architecture calls an LLM on every write. The other never calls an LLM at all. That single decision explains every performance number, every cost, every failure mode, and every privacy guarantee.
The two memory architectures
This is the core difference. No one has stated this plainly until now.
That is it. That is the entire fork in the road. Everything else flows from this one branch. There are no other meaningful architectural differences between production memory layers today. Every other feature, every tuning knob, every optimization is noise next to this choice.
Benchmark results: the good part for distilled memory
Let's start with the numbers that every vendor will show you. All results here are run on the public BEAM 1M token benchmark, same harness, same judge, no hidden tuning, no cherry picking.
| Metric | RE-call | Mem0 |
|---|---|---|
| BEAM temporal reasoning score | 0.408 | 0.567 |
| Overall BEAM aggregate | 0.591 | 0.641 |
Distilled memory wins. It wins cleanly. It wins on exactly the category everyone uses to demo memory systems: questions that ask about dates, deadlines, updates and sequence of events.
This is not measurement error. This is not marketing. When you have multiple raw turns saying the same deadline was set, revised, mentioned, discussed, cosine similarity has no way to know which one is the currently active value. The LLM run at write time collapses that ambiguity. It throws away all the context about when that value was asserted, and just keeps the final answer. For this exact class of question, that is exactly what you want.
No amount of reranking, no better embedding model, no retrieval trick will fully replicate this property. That is not an opinion, that is a structural limit of storing raw events.
What that accuracy costs you
This is the table you will never see on a vendor landing page. Same benchmark, same workload, measured end to end:
| Operation | RE-call | Mem0 | Ratio |
|---|---|---|---|
| LLM calls during ingest | 0 | 272 | ∞ |
| Total tokens processed | 0 | 2,612,914 | - |
| Ingest cost for this benchmark | $0.00 | $7.29 | - |
| Total ingest wall time | 67s | 288s | 4.3x |
| Data leaves your infrastructure | Never | Every write | - |
This is marginal cost. That $7.29 is not a monthly fee. That is the cost to ingest one single 1M token conversation history. Every additional user, every additional chat, every additional message you ever store adds this cost proportional to input length.
This cost does not go down with scale. It does not get cheaper with optimized inference. It is a hard tax imposed by the architecture. Every write will always require a full LLM forward pass.
Most importantly: there is no way around sending every single thing your user ever writes to an LLM provider. That is not a policy choice. That is not bad security. That is required for the architecture to work.
The failure mode no benchmark measures
BEAM and every standard memory benchmark only tests questions that have an answer. None of them test what the system does when there is no answer.
This is not an oversight. This is the single largest unmeasured axis in the entire RAG field right now.
When tested on 70 unanswerable questions:
| Behaviour | Mem0 count | Mem0 mean score |
|---|---|---|
| Correctly abstained | 38 | 0.974 |
| Invented a plausible answer | 32 | 0.016 |
46% of the time, when there is no information present, the system will confidently invent a completely fabricated fact. Not garbage. Not obviously wrong. A plausible, well formatted, internally consistent answer that looks exactly correct.
One documented example: when asked about user feedback that was never recorded, the system returned:
User testing showed a positive response: the dynamic language switching feature achieved a 90% satisfaction rate.
This was not hallucination from the generation LLM. This was retrieved from memory. The corpus did contain the phrase "achieving a 90% satisfaction rate is a strong start", written as a hypothetical by the assistant. The distillation step at write time had converted that hypothetical into a stated fact.
This failure mode is invisible on every standard benchmark. It will not show up in any aggregate score. It will only show up in production, when one of your users gets a completely invented statistic, and you have no audit trail to prove it never happened.
Paired benchmark results
When tested on the LOCOMO benchmark under strict paired conditions, the aggregate lead disappears entirely. Results below are McNemar tested on 1540 matched questions, same input, same judge, only the memory layer differs:
| Generator | Judge | RE-call score | Mem0 score | Paired p value |
|---|---|---|---|---|
| gpt-4o-mini | gpt-4o-mini | 0.416 | 0.378 | 0.0059 |
| gpt-4o-mini | gpt-4o | 0.466 | 0.412 | 0.00018 |
| gpt-4o | gpt-4o | 0.484 | 0.444 | 0.0065 |
All results here are statistically significant. Note that the margin shrinks as the generator model gets stronger, and reverses entirely on Claude Sonnet. There is no universal best system. Performance is a property of the entire stack, not just the memory layer.
There is no best system
This is the part almost every writeup gets wrong. There is no winner here. There is only a tradeoff.
Use distilled write time memory if:
- Your workload is heavy on temporal and multi hop reasoning
- You have a strong generation model
- Marginal cost per memory is not a constraint
- Sending all user content to a third party LLM is acceptable for your use case
Use raw turn memory if:
- Marginal cost must be zero at any scale
- User data can never leave your infrastructure
- You need offline or air gapped operation
- Inventing a plausible wrong answer is a worse failure than returning nothing
This is not a spectrum. You cannot have both. You cannot have zero cost ingest and automatic fact resolution. You cannot have perfect temporal reasoning and never send data off box. Every system sits exactly somewhere on this line.
What the benchmarks are hiding
BEAM is the best public memory benchmark we have right now. It is also structurally biased against abstention.
The benchmark uses a 9:1 ratio of answerable to unanswerable questions. Any policy that abstains more will gain points on 30 questions and lose points on 270. No matter how good your abstention logic is, it will always reduce your aggregate score.
This is not a flaw in the benchmark. This is a flaw in how we use benchmarks. Aggregate scores are not a universal quality metric. They are a measurement of performance for one specific risk profile: one where a wrong answer is exactly as bad as no answer.
Almost no real world deployment operates that way. For most production systems, one fabricated answer is 100x worse than 100 correct abstentions. No public benchmark currently prices that tradeoff.
The state of the open model ecosystem
This debate is happening right as the open model space is undergoing its fastest period of change ever.
We now have usable small base models like Inkling-Small that can run entire retrieval stacks entirely locally for less than 100ms per query. We have independent benchmarking like open-weights-breakout that is finally measuring real world performance instead of marketing numbers. And outside the west, the release cadence has accelerated to the point that new capable models are landing every 7 days.
None of this progress matters if every memory layer on top of them still requires calling OpenAI on every write. The entire point of running local models is defeated if you still have to send every user message to a third party to build your memory.
What comes next
Right now almost everyone is building the wrong thing. We are all racing to make retrieval more accurate, while almost no one is measuring failure modes, cost, or privacy.
The next generation of memory systems will not be the ones that score 5% higher on BEAM. They will be the ones that give you a complete ledger of every tradeoff. They will tell you exactly what you gain, exactly what you lose, and let you pick which side of the line you want to be on.
Until then: stop looking at aggregate benchmark scores. Look at the architecture. Look at the cost. Look at what happens when it fails. That is the choice you are actually making.
All raw numbers, run artifacts and source code for RE-call are published under MIT license at https://github.com/gde03/recall-rag. Full benchmark logs are available unredacted in the repository results directory.