The short version
● What it is A memory module for frozen LLMs, open-sourced by Stanford.
● The trick Compresses a document store into 16 soft memory tokens.
● Result 84.33% on FactKG against 80.43% for full context.
● Efficiency 108x fewer document tokens on that benchmark.
Stanford researchers open-sourced Memorilla, a latent memory module that lets a frozen language model recall large document collections from 16 memory tokens. A frozen embedding model encodes each document once into a persistent store; a trainable module then attends over that store and emits a small fixed set of soft tokens in the model's own input embedding space.
Only the memory module is trained - about 194 million parameters - while the decoder, a frozen Qwen3-8B, is left alone. On FactKG it scores 84.33 percent against 80.43 for full context, using 108x fewer document tokens. On LaMP-4 it beats full context by 3.94 ROUGE-L using 474x fewer. Across the suite it averages 49.68 against 41.92 for xRAG and 38.02 for Mem0.
It does not win everywhere, which is worth saying: MemoRAG leads on NarrativeQA and PV4, and LLMLingua-2 leads on PubMedQA by a wide margin. But beating full-context prompting while reading a hundredth of the tokens changes the economics of giving a model memory, and the code is public.
FAQ
What is Memorilla?
An open-source latent memory module from Stanford that compresses a document store into 16 soft memory tokens for a frozen LLM, so the model can recall large collections without reading them into context.
Does it beat full-context prompting?
On most benchmarks tested. FactKG 84.33% against 80.43%, LaMP-4 ahead by 3.94 ROUGE-L, and a suite average of 49.68 against 41.92 for the strongest compression baseline. MemoRAG and LLMLingua-2 still lead on three benchmarks.
How much cheaper is it?
108x fewer document tokens on FactKG and 474x fewer on LaMP-4, against full context.
Do you have to retrain the model?
No. The decoder stays frozen - the paper uses Qwen3-8B - and only the roughly 194-million parameter memory module is trained.