What "good" looks like
Agent memory is evaluated by whether a fact written in one session reads back correctly in a later one, after contradicting updates, explicit deletions, and hundreds of turns of interference. The bar is recall of the latest surviving write, not recall of any write.
The ship targets:
- Latest-write-wins, not first-write, on every contradicted fact.
- Correct abstention when memory holds no answer, instead of confabulation.
- pass^k reported at k of at least 4, alongside pass@1.
LongMemEval operationalizes this. It embeds 500 curated questions in scalable chat histories and tests five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates (does the latest write win), and abstention (knowing when memory has no answer)[1].
Its headline finding is sobering: commercial assistants and long-context LLMs drop about 30% in accuracy under sustained interaction. The paper earns its place by being diagnosable, not just a leaderboard. It decomposes memory systems into indexing, retrieval, and reading stages so failures are attributable. Its verified mitigations (session decomposition, fact-augmented key expansion, time-aware query expansion) materially improve both recall and end-to-end QA[1].
flowchart TD
W["Fact written, session 1"] --> STORE["Memory: index, retrieve, read"]
U["Contradicting update, session k"] --> STORE
D["Delete request: forget X"] --> STORE
STORE --> Q["Later-session query"]
Q --> C1{"Latest write recalled?"}
C1 -->|no| F1["Fix: grade latest-write-wins"]
C1 -->|yes| C2{"Deleted fact stays gone?"}
C2 -->|no| F2["Fix: deletion-compliance probe"]
C2 -->|yes| C3{"pass^4 holds, not just pass@1?"}
C3 -->|no| F3["Fix: reliability debt"]
C3 -->|yes| SHIP["Ship"]
Scope note. Per-run deterministic state checks (did the agent leave the environment in the right state within a single run) live in long-horizon planning and computer and browser use. This page owns cross-session retention and recall over time.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Four MemoryAgentBench competencies | No method masters retrieval, test-time learning, long-range understanding, and selective forgetting at once | Report the competencies separately, not as one score[2] |
| pass^k over repeated probes | Inconsistent recall that pass@1 hides | Run each memory probe at k of at least 4; database-final-state grading[3][4][5] |
| Retention window | The growing duration your memory must span | METR 50%-task-completion time horizon, doubling roughly every 7 months over six years[6] |
On the competencies. MemoryAgentBench reframes the problem as accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Its verified finding is that no current method masters all four, which is the argument for reporting them separately[2].
On reliability. tau-bench showed gpt-4o succeeding on under 50% of tasks averaged across domains, with pass^8 below 25% on retail, graded by database-final-state comparison[3][4]. tau2-bench extends the environments and supports the same repeated scoring[5].
An honest gap. The governance metrics teams most need (staleness distribution, contradiction rate, privacy leakage, deletion compliance) have no widely adopted public benchmark. Do not pretend one exists. Build the DIY probes below and report them as first-class numbers.
Figure: pass^k reliability decay for repeated memory probes. Because pass^k is the per-trial success rate raised to the k-th power, a probe that recalls a fact 95% of the time per run passes all 4 runs 81% of the time, an 85% probe only 52%, and a 70% probe only 24%. The gap between pass@1 and pass^4 is the memory system's reliability debt.
Offline eval design
Run LongMemEval as-is first. It is the closest thing to a standard[1]. Then build a probe matrix over your own domain that covers the five abilities per seeded fact:
| Probe | Seeds | Checks |
|---|
| Seed fact | one fact, session 1 | extraction in session N |
| Contradicting update | new value, session k | latest-write-wins, not first-write |
| Deletion order | "forget X", session k | X absent from all later answers |
| Distractor interference | many similar facts | retrieval survives near-neighbors |
| No-answer question | nothing | abstention, not confabulation |
Prioritize latest-write-wins and interference. The LongMemEval authors observed commercial assistants overwriting or losing crucial information under sustained interaction, and updates and distractors are exactly what static histories never test.
Grade deterministically where possible (string or entity match on the recalled value); reserve LaaJ for multi-session reasoning questions where the answer is compositional.
Run every probe at least 4 times and report pass@1 and pass^4 together. The divergence between them is the reliability signal.
Online eval design
Instrument four production signals.
- Memory-hit rate. Of responses where a stored memory was relevant, how often the system actually used it. A falling hit rate usually means index drift, not model regression.
- Stale-preference incident tracking. Tag every support ticket or thumbs-down where the agent acted on a superseded fact, and trend it weekly.
- Periodic deletion-compliance audits. Replay a sample of "forget this" requests and verify the fact is unrecoverable through paraphrased probes, not just through the original phrasing.
- Cross-user leakage probes. Seed canary facts in test accounts and query for them from other accounts on a schedule. Any hit is a page-the-team incident, not a metric.
Top benchmarks
- LongMemEval harness: the five-ability probe set and the stage-level decomposition.
- MemoryAgentBench: the selective-forgetting competency, which almost no other public harness scores[2].
- tau2-bench: for memory inside a tool-using agent, where the grader is final environment state rather than a recalled string[5].
- Langfuse: production trajectory logging with per-session slicing, which is what makes memory-hit-rate and stale-preference metrics computable at all.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Passive recall benchmarks | Needle-in-a-haystack retrieval is the easy subset; a system can score well on plain extraction while failing updates and abstention, which production exercises constantly | Test knowledge updates and abstention, not just extraction[1] |
| Stationary-intent assumption | Most eval sets assume the user's facts never change, so a system that always returns the first write scores perfectly | Test contradictory updates explicitly and grade for the latest write |
| Single-trial pass@1 | A system that recalls a fact on most runs but not all looks fine at pass@1 and is a support-ticket generator in production | Report pass^k at k of at least 4[3] |
| No standard privacy leakage test | Citing a benchmark score that does not exist | Say so in the eval report, then probe with canary facts and cross-account queries |
Canonical reading
LongMemEval (Wu et al., 2024) is the canonical read because it does three things at once: defines the five abilities that structure this whole page, quantifies the roughly 30% accuracy drop under sustained interaction, and ships mitigations that verifiably work[1]. The indexing-retrieval-reading decomposition is the part to internalize; it converts "memory is flaky" into a named, fixable stage.
Day-1 starter
- Seed 30 synthetic user histories, each containing stable facts, later contradicting updates, and at least one fact the user explicitly asks to delete.
- Probe next-session recall, latest-write-wins on the contradicted facts, and deletion compliance on the deleted ones, with deterministic value-match grading.
- Run every probe 4 times and report pass^4 alongside pass@1; the gap between them is your memory system's reliability debt.
Cross-cutting links