What AI Memory Benchmarks Actually Measure: LongMemEval, LoCoMo, and Crosslink Recall
Review what AI memory benchmarks measure, compare three research papers, and use a fictional test fixture to examine corrections, context, and uncertainty.
Maya Chen
AI Research Writer
An AI memory benchmark measures answers to defined questions under a particular conversation history, system configuration, and scoring method. It can expose important failures. It cannot, by itself, tell you whether a companion will feel familiar, handle your changing preferences well, or choose an appropriate moment to bring something up.
This review compares three influential research approaches, then develops a small, reusable evaluation fixture. The distinction matters to Crosslink Recall, the relational memory technology within Attunara, Kissable’s relationship intelligence system: it brings relevant remembered details together with the people, places, and relationship context that give them meaning. In this series, Attunara is our name for the orchestration design that connects remembered details, character, conversation delivery, and scenes. A useful name should lead to inspectable behavior.
This is a focused research review and an original evaluation proposal from Kissable. The examples are fictional; no Kissable benchmark result is reported.
Three papers worth reading together
LongMemEval: test changes as well as recall
Wu et al., ICLR 2025. LongMemEval contains 500 curated questions over constructed user–assistant histories. It covers extraction, reasoning across sessions, temporal reasoning, knowledge updates, and abstention. Human review shapes questions and evidence; the histories are not a sample of ordinary companion relationships. The paper separates indexing, retrieval, and reading, and reports that correct retrieval can still be followed by an incorrect answer. Its contribution is a diagnostic task taxonomy and an investigation of memory-design choices. The published results concern the evaluated configurations, not today's entire companion market. LongMemEval paper.
LoCoMo: follow an extended conversation
Maharana et al., ACL 2024. LoCoMo's final curated dataset contains ten machine-generated, human-edited conversations, averaging about 600 turns and 16,000 tokens, across up to 32 sessions. Personas, temporal event graphs, and image exchanges support question answering, event summarization, and multimodal dialogue evaluation. Tested retrieval and long-context approaches improve some results while remaining below human performance. The construction supports controlled evaluation, but ten edited conversations cannot represent every communication style. A QA-only experiment also does not reproduce the paper's complete multimodal evaluation. LoCoMo paper.
Lost in the Middle: available information can still be missed
Liu et al., TACL 2024. This paper manipulates evidence position in multi-document question answering and key-value retrieval. Evaluated models often perform better with relevant material near the beginning or end of the input than in its middle. It examines use of supplied context rather than persistence across companion sessions. The finding motivates a controlled positional test; it does not establish that every current model has the same weakness. Its value here is a reminder to inspect how evidence reaches the answer, not only whether the system can store it. Lost in the Middle.
How to choose an evaluation
Our recommendation is to begin with the failure you need to understand. Choosing a benchmark because its name is recognizable reverses that order.
| Your question | Evaluation to construct | Evidence to retain |
|---|---|---|
| Did an explicit correction take effect? | Present an old fact, a correction, and a later direct question | Both source messages, the question, and the answer |
| Were two people confused? | Give two people the same first name and distinct relationships | Entity identifiers and the exact attribution error |
| Was information unavailable or unused? | Compare normal retrieval with a condition supplied the correct evidence | Selected context as well as final responses |
| Does prompt position affect use? | Move the same evidence while holding the remaining task fixed | The complete input for every condition |
| Was a callback welcome? | Rate relevance in a context where recall is optional | Human rubric and unwanted-callback counts |
The final row deserves its own measure. A system can know that you recently changed jobs without making every conversation about work. Conversely, failing to volunteer a callback is not proof that the fact was forgotten. Direct recall and conversational appropriateness answer different questions.
A practical evaluation should therefore retain a set of dimensions rather than collapse everything into a single “relationship intelligence” score. Weighting accurate recall, warmth, brevity, and restraint requires a value judgment. If those weights are not explicit, an apparently objective number can conceal what the evaluator actually prefers.
Read the denominator before the headline
Suppose two teams each report 80% accuracy. One answered ten direct questions after a short history. The other answered five hundred questions with distractors and corrections. Those percentages describe different experiments. Neither number becomes comparable merely because both teams used the word memory.
For a published result, look for the dataset version, included task categories, history length, ingestion procedure, answer model, retrieval budget, and scorer. Also ask whether every attempt was included. Retrying failures and reporting only the best answer changes the task into best-of-several selection.
The baseline matters just as much. Comparing a compact context to an entire conversation measures a complete system choice: selection, compression, formatting, and the cost of supplying extra text. It does not isolate a single component's causal contribution. To test that component, change it while keeping the rest of the setup as similar as possible.
When automated scoring is used, inspect examples that the scorer marks correct. An answer can contain the expected city and still attach it to the wrong person. A useful scoring rubric treats that as an error, even if a keyword matcher finds the desired string.
A four-case fixture with an answer key
The following is an original, unrun illustration. All people, organizations, and events are fictional. It is small enough to inspect manually and deliberately too small to establish product quality.
| Source ID | Timestamp | Fictional user message |
|---|---|---|
| S1 | July 1, 2026, 10:00 UTC | My coworker Sam is designing a museum in Portland. |
| S2 | July 2, 2026, 10:00 UTC | My cousin Sam teaches pottery in Bristol. |
| S3 | July 3, 2026, 10:00 UTC | Correction: my coworker Sam's museum project is in Seattle, not Portland. |
| S4 | July 4, 2026, 10:00 UTC | My coworker Sam says the museum project is now paused. |
Ask each question independently after the same history:
| Case | Question | Required answer | Disqualifying addition |
|---|---|---|---|
| Updated fact | Where is my coworker Sam's museum project? | Seattle | Portland presented as current |
| Entity attribution | Which Sam teaches pottery, and where? | Your cousin Sam, in Bristol | Assigning the teaching to the coworker |
| Temporal fact | On what date did I report that the museum project was paused? | July 4, 2026 | Claiming this is necessarily when the pause actually began |
| Unknown detail | What is the museum project's budget? | The history does not say | Any invented amount |
These cases illustrate four different expectations. The third question deliberately asks when information was reported, because the source does not give the event's exact start time. The fourth makes uncertainty the correct behavior. A confident invented number is worse than a clear statement that the detail is missing.
To use the fixture diagnostically, retain both the answer and the evidence supplied to the response model. If Seattle never reaches the model, inspect the earlier stages. If both cities reach it and Portland wins, inspect how corrections are represented and interpreted. This is a debugging distinction, not a way to infer hidden architecture from a public chat transcript.
Turning four examples into a defensible study
Expand the fixture with independently authored cases before collecting results. Vary names, relationships, correction wording, and the distance between relevant messages. Include cases with no correction, otherwise the system might learn that the newest statement always overrides everything.
Fix the expected answers before running the systems. For ambiguous histories, specify what uncertainty should look like rather than rewarding whichever interpretation matches your preferred output. Keep a separate category for questions that cannot be answered from the provided evidence.
Repeated outputs reveal variability, but repeats of the same case are not independent new histories. Report counts by case and by task family. If you calculate uncertainty intervals, account for that grouping. A small diagnostic pilot is useful without claiming to estimate the experience of every customer.
Finally, preserve failures. A source-linked example of a model selecting the wrong Sam can teach another researcher more than an attractive aggregate score without an error analysis. Publishing input conditions, answer rules, and the full denominator makes a result inspectable.
What this means for Crosslink Recall
The practical ambition behind Crosslink Recall is that a detail remains connected to the person, time, and conversation that give it meaning. In the fixture, remembering “Sam” is only a beginning. The useful outcome is selecting the right Sam, applying the corrected city, and declining to invent a budget.
Kissable's inspected implementation combines remembered information with persona and conversational context and represents people and relationships. A compact standing-memory document also exists, with rollout controls. That implementation is a basis for an engineering explanation; it is not evidence that every account receives every component or that the final response always uses the right information.
The naming gives us a more precise promise to develop: the little things carry forward with their context. A future Attunara evaluation should make that promise measurable through dated configurations, source-linked cases, and failure reporting. None of the reviewed papers evaluates Attunara, and this article does not claim a completed reproduction.
For practical background, see our guides to AI companion memory and remembering conversations.
Frequently asked questions
Is a high memory benchmark score enough to choose an AI companion?
It is one kind of evidence. You still need to know the tested setup and whether its tasks resemble your use. Conversational fit, corrections, and unwanted callbacks deserve separate attention.
Can I use the four cases above to rank products?
Use them to identify specific errors and questions worth investigating. Four fictional cases are too small and too narrow to support a general ranking.
Does Crosslink Recall guarantee accurate recall?
No. It names Kissable's relational memory technology: remembered details are brought into a response with the people, places, and relationships that give them meaning. Extraction, selection, and response generation can all fail; useful evaluation must inspect the resulting answers.
Where should researchers start?
Read the original methods and scoring instructions, choose a defined failure mode, and retain the complete test conditions. Treat a review as a guide to the primary material, not a replacement for it.
References
- Wu et al. (2025). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR.
- Maharana et al. (2024). Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL.
- Liu et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL.
Curious how continuity feels in everyday conversation? Explore Kissable.
AI Research Writer
Maya covers AI companion technology, safety, and the psychology behind human-AI relationships. She focuses on what the research actually says — and what it doesn’t.