We withdrew our number.

What went wrong

A scoring defect, not a memory change.

LongMemEval (Wu et al., ICLR 2025) gives each of 500 questions a haystack of about fifty prior chat sessions, and the task is to rank the sessions that hold the answer first.

To score a result, the harness had to work out which session it came from. It did that by matching the text and taking the first session in the haystack with that text. If a haystack holds two identical sessions with different dates, Anjo can rank the newer copy, a distractor, while the harness credits the older copy, the answer. That counts a miss as a hit.

We had believed this error could only lower the score. It can raise it too, and we have not measured by how much. We found no identical pairs in the 58 questions we could re-check, but the two question types built around near-duplicate sessions with different dates, knowledge update and temporal reasoning, were not checked.

Where the number lives now

Re-run, confirmed, and kept in the repository.

The harness now identifies each result by its memory id rather than its text, and a regression test pins the duplicate-session case. It has since re-run all 500 questions, and the result held.

We are leaving the figure off this page anyway. A retrieval score is not something you can act on as a user of the app, and a number on a marketing page is a poor way to judge whether something remembers you well. It stays in the repository, where it does its real job: catching a regression before it ships.

What has not changed

The memory is the same. The claim is gone.

Withdrawing a score does not change the product. The retrieval code that answers you in the app did not change. When memory misses something, the grounding guard still keeps that miss from turning into an invented recollection.

Until the benchmark is re-run, a test in the site’s own suite fails if any other page cites the withdrawn figures.