+207
Daily
1,106
stars
75
forks
6
watchers
11
prs
3
issues
0
releases

README

 Memorilla: Latent Semantic Memory for LLMs



  
  
  
  


This repository provides code for training and evaluating Memorilla, a memory system that gathers information from an
entire document collection into a fixed number of query-dependent memory tokens for a frozen LLM. The code is based on
our paper Memorilla: Latent Semantic Memory for LLMs.

A frozen LLM can only use what fits in its context window, so Memorilla reads an entire document collection, such as a
user's past chats or a long book, and hands the LLM just 16 memory tokens that carry what each question needs. With this
small budget, it achieves the best average performance against methods such as Mem0 and MemoRAG, and on FactKG it is
even more accurate than giving the LLM every document (84.33% vs. 80.43%) while using 108× fewer tokens. Furthermore, Memorilla can also help an agent keep learning from its own history, raising its average
success rate on TextWorld games from 18.6% to 24.2%.


  


 Contents

- Installation
- Getting started
- Efficiency
- Training and Evaluation
- Checkpoints
- Using your own data
- PersonalizationV4
- Reinforcement learning on TextWorld
- Documentation
- Development
- Security notes
- Citation
- License

 Installation

Memorilla needs Linux with CUDA GPUs and Python 3.11; the recipes were run on NVIDIA H100 80GB GPUs. Install it with
uv:

bash
$ git clone https://github.com/snap-stanford/memorilla.git && cd memorilla
$ uv venv --python 3.11 && source .venv/bin/activate
$ uv pip install -e .
$ uv pip install --no-build-isolation flash-attn==2.7.4.post1



FlashAttention-2 and optional extras

The last command installs FlashAttention-2, which training uses by default. Evaluation does not need it, and training
can use PyTorch's attention instead:

bash
$ scripts/train.sh configs/stage1_enwiki.yaml --attn_implementation sdpa


Optional extras:

bash
$ uv pip install -e ".[dev]"                  tests, linters and pre-commit
$ uv pip install -e ".[wandb]"                Weights & Biases logging
$ uv pip install -e ".[personalizationv4]"    OpenAI client for the PersonalizationV4 pipeline
$ uv pip install -e ".[rl]"                   TextWorld, for the RL data tools




 Getting started

Here is an example from the PersonaMem-v2 test set. The user has had 107 earlier conversations with an assistant,
about 32,000 tokens in total, and now asks a new question. Memorilla reads all of the conversations and hands them to
a frozen Qwen3-8B as just 16 memory tokens, about 2,000 times fewer. You need one 80 GB GPU, and the first run
downloads about 25 GB of models and the 37 MB test set.

python
from memorilla.benchmarks import PERSONA_PROMPT
from memorilla.data import ensure_data, load_split
from memorilla.vllm import MemoryVLLM

row = load_split("pmv2", "test", ensure_data("pmv2", ["test"]))[73]   one user's question and past conversations

model = MemoryVLLM.from_pretrained("memorilla/Memorilla-Qwen3-8B", "pmv2")   Qwen3-8B, the memory module and the enc

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON