What 0.821 recall@5 means, and what it doesn't

"It remembers you" is a claim every companion app makes. None of them publishes a number. We do, and this note explains what the number is and where it stops.

The task

LongMemEval (Wu et al., ICLR 2025) is a public benchmark for long-term memory in chat assistants. Its S variant gives each of 500 questions a haystack of roughly fifty prior chat sessions, about 115,000 tokens. Only one to six of those sessions hold the answer. The retrieval task is to rank the sessions so the right ones come first.

That is what Anjo's memory does on every turn. So the harness, anjo/evals/external/longmemeval_s.py, drives the production read path rather than a copy of it. Memories are written with the same function the app writes with. They are queried with the same function the app queries with. The embedder is the one in production: all-MiniLM-L6-v2, 22 million parameters, running locally.

The result

Across all 500 questions, Anjo scores 0.821 recall@5, 0.774 nDCG@5, and 0.896 recall@10. Mean reciprocal rank is 0.826.

The paper's own session-level baselines on the same setup are 0.634 recall@5 for BM25, 0.723 for Contriever, and 0.720 for Stella v5 1.5B. Those come from Table 9 of the paper. The stored result lives in anjo/evals/baselines/longmemeval_s_retrieval_500.json, dated 2026-07-09.

Two controls

A retrieval number can be an artefact. We checked two ways.

First, random ranking. Shuffling the candidates scores 0.087 recall@5. The haystack is hard, and an arbitrary order does not pass it.

Second, recency. A memory system can look excellent by favouring recent sessions, because benchmark questions often ask about recent things. We flattened the recency weighting to a constant and re-ran. Recall@5 moved to 0.829, slightly up. The lead is not coming from the clock. It comes from the semantic and emotional lanes.

That second control is the one we care about. It is the difference between a memory and a "show the newest thing" prior.

By question type

The breakdown is uneven, and we publish the weak column for the same reason we track it.

• Single-session, assistant: 0.964 (n=56)
• Single-session, preference: 0.900 (n=30)
• Multi-session: 0.868 (n=133)
• Knowledge update: 0.756 (n=78)

Knowledge update is the case where a fact about you changed, and both the old version and the new version sit in the history. It is the lowest column. It is also the case a longitudinal product most needs to get right. Pretending otherwise would make the rest of this note worth less.

What the number does not show

It measures retrieval, not answers. Ranking the right sessions is the sub-task. It says nothing about the quality of the reply Anjo writes after retrieving them. End-to-end question answering on this benchmark is a different measurement, and we have not published one.

It is not a comparison with other companion products. The table sets Anjo beside the retrievers the benchmark paper published. Replika, Nomi, and Character.AI do not publish a number here, so none of them appears. The honest sentence is "no other companion product publishes one," not "Anjo beats them."

It does not mean Anjo never forgets. Recall@5 of 0.821 means the right session is missing from the top five roughly one time in six. In the app, that shows up as Anjo not bringing something up. The grounding guard keeps that failure from becoming an invented memory, but it does not make retrieval perfect.

The harness as a gate

The evaluation is not a launch artefact. Any change to embeddings, ranking, or retrieval must re-run the offline evaluation and clear the stored baseline before it ships. A hybrid lexical lane and a cross-encoder reranker were both evaluated this way and removed, because the vector-only path scored equal or better.

The website is tied to the same file. tests/test_seo_surface.py fails if the published figure drifts from the baseline. If you read 0.821 on anjo.love, it is the number in the JSON, not a number someone remembered.

Limitation

The benchmark is in English, and the embedder is small. We have not measured retrieval quality for users writing in Chinese, Japanese, or Korean, which the app supports. That is the next measurement we owe, and we will publish it whether it is good or not.