Straight answer
● Winner on average Memorilla, 49.68 against 41.92 for xRAG and 38.02 for Mem0.
● Biggest gap PMv2 short answer: 44.60% against 30.90% for xRAG.
● It loses three NarrativeQA and PV4 to MemoRAG, PubMedQA to LLMLingua-2.
● The real headline It beats reading the whole document, on a hundredth of the tokens.
Memorilla is not competing with retrieval on recall. It is competing with full-context prompting - putting the entire document in the prompt and letting the model read it - and beating that while reading roughly one percent of the tokens is the result that matters. Here is where each approach actually wins.
The benchmark table
| Benchmark | Memorilla | Next best | Delta | Note |
|---|
| FactKG (accuracy) | 84.33% | 80.43% full context | +3.90 | 108x fewer tokens |
| TriviaQA (exact match) | 57.00 | 56.60 MemoRAG | +0.40 | 52.60 full context |
| PMv2 (short answer) | 44.60% | 30.90% xRAG | +13.70 | widest margin |
| LaMP-7 (ROUGE-L) | 48.37 | 46.84 xRAG | +1.53 | |
| LaMP-4 (ROUGE-L) | 18.15 | 14.21 full context | +3.94 | 474x fewer tokens |
Where Memorilla loses
Three benchmarks, and the pattern in them is useful. MemoRAG leads on NarrativeQA (21.80 against 19.68 ROUGE-1) and PV4 (67.88% against 66.65%). LLMLingua-2 leads PubMedQA by a wide margin, 69.00% against 59.00%.
Narrative comprehension and clinical question answering are both cases where the exact wording carries the meaning. Compressing a document into 16 soft tokens is lossy by design, and what it loses first is phrasing. If your task turns on how something was said rather than what was stated, retrieval that returns the original passage still wins.
What each approach actually is
Full context puts everything in the prompt. Perfect fidelity, linear cost, and it hits a window limit. MemoRAG and classic retrieval fetch passages and insert them - the model reads real text, so wording survives, but recall depends on the retriever. xRAG and LLMLingua-2 compress the context before the model sees it. Mem0 maintains a memory store across turns. Memorilla is different in kind: it encodes documents once with a frozen embedding model, then a trained module emits 16 soft tokens directly in the decoder's embedding space. Nothing the model reads is text.
That is why only the memory module trains - about 194 million parameters - while the decoder, a frozen Qwen3-8B, is untouched. You are not fine-tuning a model, you are giving it a different kind of input.
The continual-learning result
On TextWorld, with the memory module trained by GRPO and only 8 memory tokens, average success rate rose from 0.186 to 0.242 and pass@3 from 0.250 to 0.375, against full-history in-context learning. That is the more interesting half of the paper: an agent that accumulates experience without its context growing.
Which to use
By task
● Factual recall over many documents Memorilla. Widest margins, lowest token cost.
● Narrative or clinical text MemoRAG or plain retrieval. Wording survives.
● One document, fits the window Full context. Nothing beats reading it.
● Agent that learns across episodes Memorilla with GRPO. The TextWorld result.
The practical read: if your cost is dominated by stuffing the same documents into every prompt, this is the first compression method that beats not compressing at all on most of what was tested. If your answers depend on exact phrasing, it is the wrong tool and retrieval still has the job. Our launch coverage has the release detail.
FAQ
Is Memorilla better than RAG?
On factual recall benchmarks, yes - it averages 49.68 against 41.92 for xRAG and 38.02 for Mem0. On narrative comprehension MemoRAG still leads NarrativeQA and PV4, and LLMLingua-2 leads PubMedQA, so it depends on whether exact wording matters to your task.
Does Memorilla beat putting the whole document in the prompt?
On most benchmarks tested. FactKG 84.33% against 80.43% using 108 times fewer document tokens, and LaMP-4 ahead by 3.94 ROUGE-L using 474 times fewer.
How many memory tokens does Memorilla use?
16 in the main experiments, and 8 in the TextWorld continual-learning setting.
Do you need to retrain the language model?
No. The decoder stays frozen - the paper uses Qwen3-8B - and only the memory module of roughly 194 million parameters is trained.
When should you not use it?
When the answer depends on exact phrasing. Compressing into soft tokens is lossy by design and loses wording first, which is why retrieval still leads on narrative and clinical text.