MON, NOVEMBER 16, 2026
Independent · In‑Depth · Practitioner‑Tested
✎ AI News

Stanford Squeezed a Document Library Into 16 Tokens

Memorilla compresses an entire document store into 16 soft memory tokens for a frozen LLM, beating full-context prompting on FactKG while reading 108 times fewer document tokens.

By AIToolsRecap November 16, 2026 6 min read 113 views
Home › Articles › AI News › Stanford's Memorilla: Document Memory in 16 Tok...
Stanford Squeezed a Document Library Into 16 Tokens
The short version
● What it is A memory module for frozen LLMs, open-sourced by Stanford.
● The trick Compresses a document store into 16 soft memory tokens.
● Result 84.33% on FactKG against 80.43% for full context.
● Efficiency 108x fewer document tokens on that benchmark.

Stanford researchers open-sourced Memorilla, a latent memory module that lets a frozen language model recall large document collections from 16 memory tokens. A frozen embedding model encodes each document once into a persistent store; a trainable module then attends over that store and emits a small fixed set of soft tokens in the model's own input embedding space.

Only the memory module is trained - about 194 million parameters - while the decoder, a frozen Qwen3-8B, is left alone. On FactKG it scores 84.33 percent against 80.43 for full context, using 108x fewer document tokens. On LaMP-4 it beats full context by 3.94 ROUGE-L using 474x fewer. Across the suite it averages 49.68 against 41.92 for xRAG and 38.02 for Mem0.

It does not win everywhere, which is worth saying: MemoRAG leads on NarrativeQA and PV4, and LLMLingua-2 leads on PubMedQA by a wide margin. But beating full-context prompting while reading a hundredth of the tokens changes the economics of giving a model memory, and the code is public.

FAQ

What is Memorilla?
An open-source latent memory module from Stanford that compresses a document store into 16 soft memory tokens for a frozen LLM, so the model can recall large collections without reading them into context.
Does it beat full-context prompting?
On most benchmarks tested. FactKG 84.33% against 80.43%, LaMP-4 ahead by 3.94 ROUGE-L, and a suite average of 49.68 against 41.92 for the strongest compression baseline. MemoRAG and LLMLingua-2 still lead on three benchmarks.
How much cheaper is it?
108x fewer document tokens on FactKG and 474x fewer on LaMP-4, against full context.
Do you have to retrain the model?
No. The decoder stays frozen - the paper uses Qwen3-8B - and only the roughly 194-million parameter memory module is trained.
Tags
AI NewsGenerative AICoding AI2026

Spot an inaccuracy?

We verify facts before publishing and correct errors promptly. If something in this article is wrong or outdated, let us know.

Report an error →