Memory benchmark
The memory is measured.
“It remembers you” is a claim every companion app makes and none of them publishes. Here is ours on a public 500-question benchmark, run through the same retrieval code that answers you in the app.
The benchmark
What LongMemEval-S asks.
LongMemEval (Wu et al., ICLR 2025) is a public benchmark for long-term memory in chat assistants. Its S variant gives each of 500 questions its own haystack of roughly fifty prior chat sessions — about 115,000 tokens — of which only one to six actually contain the answer. The retrieval task is: given the question, rank those sessions.
That is precisely what Anjo’s memory does on every turn, so the harness drives the production read path rather than a reimplementation of it. Memories are written with the same function the app writes with and queried with the same function the app queries with.
The result
0.821 recall@5.
| Retriever | Recall@5 | nDCG@5 | Recall@10 |
|---|---|---|---|
| Anjo (production read path) | 0.821 | 0.774 | 0.896 |
| Stella v5 1.5B | 0.720 | 0.594 | — |
| Contriever | 0.723 | 0.634 | 0.823 |
| BM25 | 0.634 | 0.516 | 0.710 |
Baselines are the LongMemEval paper’s own Table 9, session-level indexing — the same setup this harness uses. Anjo’s mean reciprocal rank across the 500 questions is 0.826. The embedder doing the work is all-MiniLM-L6-v2, 22 million parameters, running locally.
anjo/evals/external/longmemeval_s.py · anjo/evals/baselines/Controls
Why the number is not an artefact.
Random ranking scores 0.087
The haystack is genuinely hard. Shuffling the candidates collapses recall@5 by an order of magnitude, so the corpus is not one an arbitrary ordering could pass.
Recency is not doing the work
Flattening the recency weighting to a constant moves recall@5 to 0.829 — slightly up. The lead does not come from a “show the newest thing” prior.
That second control is the one worth dwelling on. A memory system can look excellent on a benchmark purely by favouring recent sessions; removing the recency term here does not degrade the score, which means the ranking is being carried by the semantic and emotional lanes rather than by the clock.
By question type
Where it is strong, and where it is not.
| Question type | n | Recall@5 |
|---|---|---|
| Single-session, assistant | 56 | 0.964 |
| Single-session, preference | 30 | 0.900 |
| Multi-session | 133 | 0.868 |
| Knowledge update | 78 | 0.756 |
Knowledge update — a fact about you that changed, where the old version and the new version both sit in the haystack — is the weakest column, and it is published here for the same reason it is tracked internally: it is the case a longitudinal product most needs to get right, and pretending otherwise would make the rest of this page worth less.
What this does and does not show
The honest scope.
It measures retrieval
This is the retrieval sub-task — ranking the right sessions — not end-to-end question answering, and not a claim about the quality of the reply that follows.
It is not a competitor comparison
The table sets Anjo beside the benchmark’s published retrievers. No other companion product publishes a number here, so none appears in it.
The harness is a standing gate, not a launch artefact: any change to embeddings, ranking or retrieval has to re-run the offline evaluation and clear the stored baselines before it ships. A hybrid lexical lane and a cross-encoder reranker were both evaluated this way and removed, because the vector-only path scored equal or better.
FAQ
Quick answers
How good is Anjo's memory, measured?
On LongMemEval-S — a public long-term-memory benchmark from Wu et al., ICLR 2025 — Anjo's production retrieval path scores 0.821 recall@5 and 0.826 mean reciprocal rank across all 500 questions. The benchmark paper's own session-level baselines on the same setup are 0.634 for BM25, 0.723 for Contriever and 0.720 for Stella v5 1.5B.
Is that number from the real product or a special test setup?
The real product. The harness writes memories with the same function the app writes with and queries them with the same function the app queries with, so it drives the production read path rather than a reimplementation. The embedding model is the one that runs in production: all-MiniLM-L6-v2, 22 million parameters, running locally.
Does Anjo just surface the most recent conversations?
No, and this is measured directly. Flattening the recency weighting to a constant moves recall@5 to 0.829 — slightly higher, not lower — so the ranking is not being carried by a recency prior. For scale, randomly shuffling the candidates scores 0.087.
What is Anjo's weakest result on the benchmark?
Knowledge update, at 0.756 recall@5 across 78 questions — cases where a fact about you changed and both the old and new versions sit in the history. It is the lowest column in the breakdown and it is published because it is the case a longitudinal product most needs to get right.
Does this benchmark prove Anjo is better than other AI companions?
No. It compares Anjo to the retrieval baselines published in the benchmark paper, not to other companion products — no other companion product publishes a number on this benchmark. It also measures the retrieval sub-task, ranking the right past sessions, rather than end-to-end answer quality.