Language: English | 简体中文
Correctness lab for stale memory, extreme-point selection, and skillization.
MemScope is an inspectable research platform for agent-memory correctness. It started as a small benchmark for stale recall after a user fact evolves and now also provides an EPMS reference system for evidence-linked memory selection, consolidation, verification, skillization, and forgetting.
It does not try to be another chatbot with a vector database. Instead, it makes the memory lifecycle and the error boundary visible:
fact evolution -> append-only events -> retrieval policy -> answer
When a fact changes, does a retrieval policy use the new fact, or does it bring back a semantically related but stale one? MemScope compares that behavior in two layers:
- Oracle memory: the generator writes the ground-truth memory events. This isolates retrieval and conflict handling.
- End-to-end: a small deterministic extractor reads the conversation. The oracle-to-end-to-end gap is the contribution of write-time extraction noise.
The benchmark includes paired conflict/no-conflict cases, explicit, entailed-implicit, and suggestive update classes, multiple fact slots, and reproducible seeds. Each pair shares its slot, initial value, session gap, distractors, and query; only the update evidence changes.
Suggestive cases are not treated as factual updates. They test whether a memory system can abstain instead of prematurely superseding a valid memory.
Scenario conditions are explicit: update, reaffirmation, and ambiguous.
Ambiguous evidence has expected operation NO_OP; it is not silently folded
into the no-conflict control group.
all+lexical: all candidates plus lexical ranking; exposes stale candidates and reports ties instead of hiding the tie-break.all+latest: all candidates plus recency ranking.active+lexical: active candidates plus lexical ranking.active+latest: active candidates plus recency ranking.
The old names lexical, latest, and slot-aware remain compatibility
aliases. Candidate filtering and ranking are intentionally separate so a
result can distinguish “stale memory remained eligible” from “the ranker chose
badly.”
The baseline is intentionally simple. The point is not a leaderboard; it is to make a failure mode easy to reproduce and inspect.
Memory is never mutated in place. A JSONL log records operations such as:
{"op":"ADD","t":0,"memory_id":"m0","slot":"residence","value":"London"}
{"op":"ADD","t":3,"memory_id":"m1","slot":"residence","value":"Berlin"}
{"op":"SUPERSEDE","t":3,"target":"m0","by":"m1","reason":"same slot: residence"}The current state is a fold of the log. That gives time-travel queries, provenance, replay, and selective forgetting without deleting history.
Lifecycle semantics are time-aware: SUPERSEDE changes the current value while
preserving earlier as-of states; EXPIRE is effective at its event time;
RETRACT can reinstate the prior value in an update chain; FORGET is logical
forgetting; and ERASE removes the payload while preserving a tombstone.
No model, database, or API key is required:
python -m pip install -e .
python -m memscope.cli --cases 60 --seed 7 --output results/report.json
python -m pytest -qThe installed memscope command and python -m memscope are equivalent entry
points for the regression harness.
The same project can now be used as a small regression harness:
# Run registered correctness suites; exits non-zero on a failed suite.
python -m memscope.cli test --suite all --output results/ci-report.json
# Render the machine-readable CI result as a standalone HTML report.
python -m memscope.cli report \
--input results/ci-report.json \
--output results/ci-report.html
# Export generated scenarios for inspection or another adapter.
python -m memscope.cli benchmark \
--cases 60 --scenarios results/scenarios.json \
--summary-csv results/summary.csv \
--manifest results/manifest.json \
--seed-sweep
# Gate a custom scenario file in CI.
python -m memscope.cli test \
--scenario-file results/scenarios.json \
--policy active+latest \
--max-stale-rate 0 \
--min-accuracy 1 \
--output results/custom-ci.json
# Start the dependency-free reference HTTP memory service.
python -m memscope serve --host 127.0.0.1 --port 8765
# In another shell, evaluate it without passing the oracle slot.
python -m memscope test \
--adapter http \
--endpoint http://127.0.0.1:8765 \
--scenario-file results/scenarios.json \
--output results/http-ci.jsonThe report contains per-case rows and grouped accuracy and stale_error_rate
for every layer, policy, and conflict condition. Each row also contains a
stage-level diagnosis: extraction_or_consolidation, retrieval, and
utilization status. Summary rows additionally report tie_rate; the JSON
artifact contains update-class breakdowns, paired conflict/control deltas,
writer event metrics, and optional five-seed bootstrap summaries. The manifest
records package version, source commit, scenario hash, seed, command, and time.
White-box policy evaluation and black-box system evaluation are separate:
- slot-revealed isolates retrieval with a query that names the slot.
- natural-query tests slot inference with queries such as “Where do I live now?”.
- HTTP black-box sends only messages, query text, and timestamps; oracle slot and internal memory events are withheld.
The external process contract is /reset, /observe, /query, and optional
/trace. See examples/external_system/ for a runnable reference.
With 60 generated cases and seed 7, the oracle layer reports:
| Candidate/ranker | Conflict accuracy | Stale error | Tie rate |
|---|---|---|---|
all+lexical |
0.0 |
1.0 |
1.0 |
all+latest |
1.0 |
0.0 |
0.0 |
active+lexical |
1.0 |
0.0 |
0.0 |
active+latest |
1.0 |
0.0 |
0.0 |
The negative result is intentional: all+lexical ties on the query/value
representation and selects the oldest candidate. MemScope reports this as a
baseline limitation rather than presenting it as a fair semantic retriever.
The end-to-end layer is lower because the deterministic extractor is narrow;
that gap is kept separate from the oracle retrieval result.
The current suite implements stale recall. The larger correctness lab is organized around five bug families:
- Temporal: stale recall, premature update, historical amnesia, as-of leakage.
- Identity: subject confusion, cross-user leakage, multi-entity mixups.
- Epistemic: treating planned, uncertain, hypothetical, or negated claims as facts.
- Lifecycle: duplicate, wrong merge, failed retraction, forgetting, evidence loss.
- Retrieval/utilization: evidence was missed, or retrieved evidence was misused.
This taxonomy is a roadmap, not a claim that v1.0 solves all five.
MemScope also includes an auditable Extreme-Point Memory and Skillization (EPMS) reference system. It stores raw traces and evidence in SQLite, consolidates candidate memories, verifies them, crystallizes safe declarative skills, and preserves erase tombstones. Skills are data-only DSL records and cannot execute arbitrary Python.
python -m memscope epms serve --database results/epms.sqlite
python -m memscope research run --suite all --seeds 1,2,3,4,5 --streams 200 \
--output results/research-report.jsonThe research command reports rate-distortion, representation-learning,
schema/ACT-R, and EPMS integration results separately. Each hypothesis has a
status, effect size, bootstrap interval, failures, and a claim boundary.
Deterministic suites are suitable for CI; local Qwen runs are explicit jobs
and report waiting_for_gpu instead of silently falling back to CPU.
The H2 oral upgrade keeps the original static extremeness selector as a
negative control and adds extreme-coverage, a greedy budgeted marginal-
coverage controller. Selection uses the training stream, while utility is
computed only on a held-out stream and compared with the strongest baseline
for each seed and budget.
The research foundations, hypothesis registry, attachment-derived ideas, and
claim boundaries are documented in
docs/agent-memory-research.md and the
Chinese version. The Krein–Milman
analogy is retained only as inspiration; it is not used as a theorem about
agent memory or human cognition.
The completed full run on 2026-08-20 used five deterministic seeds, three local
Qwen models, seven cached datasets, and expanded H3-H7 interventions. Its
real-model matrix contains 192 rows (189 completed, 3 environment-blocked).
The pooled results support H1, H2, H5, H8, and H10; H3 and H4 are
not_supported; H6, H7, and H9 remain inconclusive under their stated claim
boundaries. See the full-run artifacts under
results/experiments/full-run-20260820/final/.
The next research object is operation selection under drift, not a universally
best memory representation. lifecycle-control compares KEEP/DEMOTE/
REACTIVATE accessibility control with retention, recency, extremeness,
threshold, binary-forgetting, and retrospective-oracle baselines. It also
reports ablation-derived memory-value calibration and verifier calibration.
See docs/adaptive-memory-lifecycle.md
and the Chinese protocol.
The 5-seed controlled run supports LC1-LC4 under its stated evidence boundary; it does not upgrade v1.2's real-model H7 result or claim real-model online adaptation.
The complete long-term route is now implemented in
docs/unified-lifecycle-controller.md
and its Chinese version. It
unifies value calibration, verified consolidation, conditional skillization,
and accessibility control into six operations: RETAIN, CONSOLIDATE,
PROMOTE_SKILL, DORMANT, REACTIVATE, and INVALIDATE.
The older three-model text-scored Qwen queue is recorded at
results/experiments/unified-lifecycle-v1.6/gpu-queue.json. The dedicated
online phase-drift queue is
results/experiments/unified-lifecycle-v1.6/phase-drift-gpu-queue.json; it
reached completed and produced model-stratified task-execution artifacts.
That run is a controlled synthetic task, not a broad benchmark transfer claim.
The legacy MemoryEvent stream remains intentionally small. New integrations
can promote events to TraceEvent, which adds event_id, subject,
observed_at, valid_from, valid_to, epistemic status, confidence, source,
and reason fields. This leaves room for bitemporal tests where the system learns
at observed_at=7 that a fact became valid at valid_from=5.
from memscope.trace import TraceEvent
TraceEvent(
event_id="e17", op="ADD", observed_at=7,
memory_id="m7", slot="residence", value="Berlin", valid_from=5,
)| Version | Focus | Status |
|---|---|---|
v0.1 |
stale memory, oracle/E2E, implicit updates, counterfactuals | implemented |
v0.2 |
retraction, forgetting, expiration, as-of queries | implemented |
v0.3 |
four-stage traces, failure attribution, custom regression gates | implemented |
v0.4 |
experimental validity, paired controls, fair baseline matrix | implemented |
v0.5 |
HTTP black-box adapter and reference server | implemented |
v1.0 |
stable memory regression protocol and cross-system comparison | implemented |
v1.1 |
evaluation-integrity metadata, writer diagnostics, temporal lifecycle expansion, and reproducibility tooling | implemented |
v1.2 |
EPMS store, evidence-linked memories, verifier, safe skill DSL, four research suites, and HTTP v2 | implemented |
v1.3 |
adaptive lifecycle controller, ACTIVE/DORMANT/INVALID states, oracle drift protocol, value and verifier calibration | implemented |
v1.4 |
intervention-derived memory-value calibration | implemented |
v1.5 |
calibrated verifier-gated consolidation | implemented |
v1.6 |
conditional skillization and unified lifecycle operation controller | implemented |
v1.7 |
feedback-driven closed-loop policies, executable sandbox evaluation, and persistent policy transitions | implemented |
v1.7 closes the remaining lifecycle gap: feedback is persisted as a transition
outcome and changes subsequent operation policy state. Supervised value,
contextual bandit, and offline RL policies share one auditable interface. The
local execution path uses subprocess plus rlimit; it is a constrained local
sandbox, not a claim of container-grade isolation. HumanEval and MBPP executable
results are kept separate from prior text-F1 evidence.
The complete cached HumanEval/MBPP model matrix is queued at
results/experiments/closed-loop-v1.7/gpu-queue.json. The queue is resumable:
python -m memscope queue status \
--queue results/experiments/closed-loop-v1.7/gpu-queue.json
python -m memscope queue resume \
--queue results/experiments/closed-loop-v1.7/gpu-queue.jsonRun the controlled suite with:
python -m memscope research run --suite closed-loop --seeds 1,2,3,4,5 \
--streams 200 --output results/closed-loop-report.jsonThe implicit-update extractor is intentionally narrow. It is a controlled proxy for the end-to-end layer, not evidence that a general LLM memory writer works. Future work should add paraphrases, negation, retractions, and an independent LLM extractor while preserving the oracle layer.
MemScope is not a new state-of-the-art memory architecture. It is a small, inspectable testbed for understanding and comparing memory-writing, supersession, forgetting, and retrieval policies.
It does not store production memory, orchestrate agents, or provide another vector database. Its role is to test whether a memory system behaves correctly.
Integrations implement the small MemoryAdapter protocol in
memscope/adapters.py. The repository includes an OracleAdapter as a
reference implementation; framework-specific adapters remain intentionally
outside the core package so the correctness contract stays inspectable.
The reference black-box contract is:
POST /reset
POST /observe
POST /query
GET /trace
/query receives only query text and timestamp in the black-box track. The
oracle slot is withheld from the remote service.
The bundled reference service is intentionally a narrow baseline, not a claim
of general extraction quality. On a 12-case smoke run it returned 5/12
correct answers; the remaining failures are preserved as black-box evidence
rather than hidden by the oracle evaluator.
Open-source governance and protocol references live in CONTRIBUTING.md,
PROTOCOL.md, REPRODUCIBILITY.md, BENCHMARK_POLICY.md, and rfcs/.