An open benchmark and reference for evaluating AI agent memory systems
Every image below is generated by the release CI into plots/ — a guarded directory that pull requests cannot touch — and regenerates on every benchmark run, each figure carrying its own run date. The full k-sweep lives in RESULTS.md, guarded the same way. LongMemEval's record, including its per-run numbers, is archived in RESULTS.archived.md.
uv sync
uv run amb datasets list # locomo | longmemeval (+ variants)
uv run amb datasets pull locomo # download into .data/
uv run amb run --system naive --dataset locomo # direct mode, no LLM needed
uv run amb run --system naive --dataset locomo \
--model openai:gpt-5-mini --judge # + answer generation & grading
uv run amb report # markdown table across all runsThe naive baseline is BM25 only — no API key, no Docker, finishes in minutes. It is the fastest way to see the whole pipeline run end to end.
A neutral, reproducible benchmark for systematically evaluating AI agent memory systems. The repo provides:
- A curated registry of agent memory projects with links, descriptions, and metadata
- Standardized evaluation tasks across long-term recall, temporal reasoning, multi-session continuity, and contradiction handling
- A test harness that runs the same tasks against different memory systems
- Published results with reproducibility instructions
- An open methodology — contributions welcome for new projects, new tasks, and new metrics
This is not a marketing tool for any one system.
The AI agent memory space has grown to dozens of distinct projects in the last two years, each making different design choices: vector-based, graph-based, hierarchical, hybrid, summary-based. The trade-offs between them are real and consequential — but there's no neutral place where they're made visible.
Today, evaluation in this space typically takes one of three shapes:
- Vendor-led: Each project's docs benchmark it against carefully-selected baselines designed to make it look favorable.
- Paper-led: Academic benchmarks use specific datasets that don't always translate to production agent behavior.
- Ad-hoc: Users informally compare two or three systems on their own use cases, with no shared methodology.
None of these give a builder a reliable answer to "which agent memory system fits my use case?" — let alone "by how much." This benchmark exists to fix that gap. The goal isn't to pick a winner. It's to make the relevant trade-offs (accuracy vs. latency, recall vs. precision, setup complexity vs. memory footprint) measurable and transparent so practitioners can make informed choices.
Projects must be permissively licensed (MIT, Apache 2.0, BSD) or use a credible open-source-adjacent license. Hosted SaaS without an OSS counterpart is out of scope. Open-core projects are included via their OSS edition only — commercial/cloud features aren't benchmarked.
Running: LoCoMo · ConvoMem (planned)
LongMemEval — paused, not dropped.
The oracle variant hands every system a small candidate set, so precision is nearly free and the field compresses into 0.856–0.993 F1. A full run per system for a nearly flat table is the wrong use of the budget while the roster is still growing. It rejoins when a battery has room for a third dataset.
Existing results stay published in RESULTS.archived.md, with the tag that produced them.
To be explicit, since I maintain one of the systems here: LongMemEval is the dataset where fraise places last — 0.856 against a leader at 0.993. That is not why it's paused, and the numbers aren't moving. Losing on it is how fraise got better: the LongMemEval result is what identified recall as the binding constraint, and the retrieval work since came directly out of it. I expect a different number when it comes back. If not, that will be published too.
Pre-alpha. Methodology is being defined and the test harness is in development. First results land here on the first tagged release and regenerate on every one after (see RESULTS.md for the full detail). Expected timeline:
- Methodology v0.1 finalized: TBD
- Test harness v0.1 (first 3 systems running end-to-end): TBD
- First public results report: TBD
| System | Repository | License | Memory model | Results |
|---|---|---|---|---|
| Mem0 | mem0ai/mem0 | Apache-2.0 | Vector + extraction | RESULTS.md |
| Letta | letta-ai/letta | Apache-2.0 | Hierarchical (OS-style) | RESULTS.md |
| Graphiti | getzep/graphiti | Apache-2.0 | Temporal knowledge graph | RESULTS.md |
| Fraise | RonsenbergVI/fraise | MIT | Temporal knowledge graph + vector / text hybrid search | RESULTS.md |
| EverOS (EverMind) | EverMind-AI/EverOS | Apache-2.0 | Local-first, Markdown-native | RESULTS.md |
| Hindsight | vectorize-io/hindsight | MIT | Memory units + hybrid search | RESULTS.md |
| agentmemory | rohitg00/agentmemory | Apache-2.0 | Observation memory | RESULTS.md |
| Cognee | topoteretes/cognee | Apache-2.0 | Embedded knowledge graph | RESULTS.md |
Each candidate is vetted against the inclusion criteria in its proposal issue; verdicts and integration progress land there.
| Project | Repository | Proposal |
|---|---|---|
| ReMe | modelscope/ReMe | #72 |
| Redis Agent Memory Server | redis/agent-memory-server | #73 |
| LangMem | langchain-ai/langmem | #76 |
| MemOS | MemTensor/MemOS | #77 |
| Memobase | memodb-io/memobase | #78 |
| memary | kingjulio8238/Memary | #80 |
| supermemory | supermemoryai/supermemory | #81 |
| MemPalace | MemPalace/mempalace | #82 |
The full methodology — design principles, evaluation axes, harness design, datasets, model selection, run scoping, and cost — lives in METHODOLOGY.md.
Every benchmark is the same amb command, with or without Docker. Locally, install the integration's package and point it at a database you run yourself (adapters read connection settings from env vars with localhost defaults):
uv run --package graphiti amb run --system graphiti --dataset locomoOr let the integration's Docker environment provide the database — the container's entrypoint is uv run amb, so everything after benchmark is the identical CLI:
docker compose -f benchmarks/graphiti/docker-compose.yaml run --build --rm benchmark \
run --system graphiti --dataset locomo# smoke run: naive baseline, 2 LoCoMo conversations, retrieval scoring only
# (BM25 only, no LLM calls — finishes in seconds; use this to check plumbing)
uv run amb run --system naive --dataset locomo --limit 2 --questions 30
# agentic run: the model drives ingestion and search through the system's
# own tools; --model is mandatory — an agentic run never falls back
uv run amb run --system naive --dataset locomo --mode agentic \
--model openai:gpt-5-mini
# mem0 in docker, smoke test: 2 sessions of one conversation, 3 questions
# (needs OPENAI_API_KEY for mem0's extraction; compose reads the repo-root .env)
docker compose -f benchmarks/mem0/docker-compose.yaml run --build --rm benchmark \
run --system mem0 --dataset locomo --limit 1 --turns 40 --questions 3
# trade-off plot: one dot per system, any metric on each axis
uv run amb plot scatter --x search_latency.p50_s --y retrieval_recall --dataset locomo
# draw every chart the generated reports link, into plots/<dataset>[/<variant>]
# (the release CI runs this twice per tag: light, then --dark for the
# theme-aware images the README embeds)
uv run amb plot all
# regenerate RESULTS.md (written whole); the release CI runs this on every
# tag — README is hand-written and embeds the generated images by path
uv run amb report --latest --output RESULTS.md --tag v0.1.0See CONTRIBUTING.md for the contribution process. Roughly:
- Bug reports and improvements to existing benchmarks: open an issue.
- New adapters for systems already in scope: open a PR adding
benchmarks/<system>/. - New systems to add: open an issue first to discuss inclusion criteria.
- Methodology critique or improvements: open a discussion in GitHub Issues. This benchmark only works if the methodology is trusted. We take methodology critique seriously and document responses to substantive criticism publicly.
Running this costs real money: every official number comes from a CI run on
tagged code, and the API spend is accounted per-run in each summary.json.
If the results are useful to you or your team, you can fund the run pool through GitHub Sponsors. API credits count and are disclosed exactly like cash.
Sponsorship buys nothing about results. No influence on methodology, configuration, datasets, run timing, presentation, or which systems are included — and there is no human step in the pipeline where influence could apply. The full policy, written before any sponsor existed, is in SPONSORS.md.
If you find this work useful, please use the citation:
@software{corneille2026agentmemorybenchmarks,
author = {Corneille, Ren{\'e}-Jean},
title = {agent-memory-benchmarks: an open benchmark for {AI} agent
memory systems},
year = {2026},
url = {https://github.com/RonsenbergVI/agent-memory-benchmarks},
version = {0.0.1},
note = {Results generated by CI from tagged code; all run artifacts public}
}