Follow up #2 on Context-memory management for Agent Kaizen

[LevyBytes]

IMPROVE CONTEXT MEMORY

While my agents are working on long assignments to implement improvements for my harness, during human downtime I brainstorming and research and figure I would keep a record in my Github DIscussion forum under ideas so I can pull them in later and decide what to modify, implement, iterate on next. The following is a follow up on my previous #2 discussion thread.

Context-memory management for Agent Kaizen — a governed-lifecycle hypothesis

Status: hypothesis. Nothing below is implemented. This is machine-drafted analysis, grounded in Agent Kaizen's documented operation surface and the earlier memory-palace discussion (#2), posted for review before any code lands — ideas are cheapest to break before they ship.

Hypothesis

The quality of an agent's long-lived context memory is determined less by where memories are stored than by the management layer that governs their lifecycle — storage, sorting, retrieval, and maintenance — and Agent Kaizen can implement that layer almost entirely from primitives it already ships, applying a deterministic mechanism first at every function and spending model judgment only on the few decisions determinism cannot make.

The recurring failures of agent memory systems are lifecycle failures, not capacity failures. Observations never get stored because storing them mid-task costs too much. Stores fragment because nothing decides where a fact belongs, so every observation becomes its own disconnected record. Memory goes unconsulted because nothing advertises what it contains — an agent cannot query a store it does not know is populated. And archives only grow, so superseded facts sit beside current ones with equal authority until recall itself becomes unreliable. Each failure belongs to one lifecycle function, each has a deterministic countermeasure, and each countermeasure lands on a surface Kaizen already operates.

The hypothesis is testable. Each prediction below names its instrument, and all instruments already exist:

• P1 — storage: non-blocking, near-zero-cost capture gets used in real sessions; capture that gates completion or demands ceremony does not. Instrument: T6 note-event counts per session.
• P2 — discovery: advisory memory is essentially never retrieved unless a bounded digest of its contents reaches the agent at session start; with the digest, note-origin retrievals appear. Instrument: note-origin hit rate in trace_events.
• P3 — staleness: append-only advisory memory accumulates superseded facts, eventually surfacing as contradictory recalls. Instrument: a small recall eval set scored through eval_scores, including staleness probes.
• P4 — integrity: cross-referenced records rot monotonically without a deterministic integrity gate — dangling references and orphaned topics grow until linked recall stops working. Instrument: lint counts over time.
• P5 — judgment floor: the judgment-dependent functions (which topic owns a fact, what supersedes what) track the capability tier of the model making the call; deterministic guardrails bound the damage below that floor but do not create sorting quality above it. Instrument: the same recall eval set, scored per configured backend.

If P1 and P2 hold but P3 and P4 never materialize, the layer correctly stops at storage plus retrieval and the maintenance tier is never built. A hypothesis that can conclude "stop early" is the point of writing one.

Scope, and the relationship to the memory-palace hypothesis

Discussion #2 proposed the narrative lane: what to store (notes), where (the T6 run ledger and the evidence plane), and the non-negotiable rule — memory advises, Kaizen records decide. This paper generalizes one level up: it treats #2's moves as the storage function of a four-function management layer and asks what disciplined sorting, retrieval, and maintenance look like when built the same native way. Everything here inherits #2's constraints unchanged: no external dependencies, no hosted services, no background processes, R0 untouched, and advisory memory never rewrites authoritative state.

Two points from #2 are refined rather than repeated — the shape of the notes convention (sorting) and the handling of superseded facts (maintenance). Both refinements are flagged inline where they occur.

The four functions

flowchart TD
    S["Storage<br/>T6 notes · session notes<br/>immutable, non-blocking"] --> O["Sorting<br/>topic keys · bidirectional refs<br/>bounded custodian judgment"]
    O --> R["Retrieval<br/>digest · search · navigation"]
    R --> M["Maintenance<br/>integrity lint · supersession<br/>explicitly invoked consolidation"]
    M -->|"regenerates indexes and derived views"| R
    M -.->|"never edits the stored originals"| S

Loading

1. Storage — cheap, non-blocking, immutable

What is not stored mid-task dies at compaction, reset, or handoff, so the storage function has one requirement above all others: capturing an observation must cost almost nothing and must never gate completion. The mechanism is exactly #2's Phase 1 — a note point event on the T6 agent-event ledger (append-only, redaction-checked, content-hashed, idempotent on replay, inert for completion gating), plus session-boundary notes files ingested through the evidence plane:

python kaizen.py T6 --agent-run-id AR_ID --payload-json \
  "{\"event_kind\":\"note\",\"marker\":\"point\",\"summary\":\"Chunker choice rationale unverified\",\"body\":\"Recursive chunker beat fixed-window on the eval set; suspicion: fixture bias. Re-run before promoting.\"}" --json

python kaizen.py E1 --file AI/work/notes/chunker-comparison.md --task-id TASK_ID --summary "Session notes: chunker comparison + open questions" --json
python kaizen.py E3 --id DOC_ID --json

One design decision made here carries the rest of the paper: stored memory is write-once evidence. Sorting and maintenance never edit the originals; mutability is confined to derived artifacts (function 4). This separation is what makes the advisory layer structurally incapable of becoming a second, fuzzier database competing with the real one.

Storage choice also sets retrieval economics. Write-once artifacts are the ideal case for the embedding pipeline — ingested and chunked exactly once, never re-embedded, so E4 search over them stays cheap and current. Mutable summaries are the worst case for that same pipeline, because stale chunks linger at equal rank with current ones. The rule that falls out: store immutable, search it; store mutable, navigate it. Kaizen's evidence plane gets the tier it is best at.

2. Sorting — a fact lands where its topic lives

The characteristic sorting failure is fragmentation: every observation becomes its own disconnected record, the store grows while its answerability shrinks, and relations that would make memory navigable are never recorded. The countermeasures are mostly deterministic conventions, not intelligence:

• Topic-keyed notes, not date-keyed. A date records when a thing was noticed, not what it concerns; date-named files are fragmentation on a schedule. Notes files carry a topic or task slug (AI/work/notes/chunker-comparison.md), with dates inside the document. (Refinement of Native narrative memory for Agent Kaizen — a memory palace hypothesis #2, whose worked example used a date-named session file.)
• References to durable identifiers. Note bodies cite the records they concern — task ids, GOTCHA ids, evidence document ids — in a greppable [[ref]] form, so notes and authoritative records form one navigable web rather than parallel silos.
• Bidirectional linking. When a genuinely new topic is created, the topics that relate to it are patched to reference it in the same operation, so nothing enters the store unreachable.
• Generated artifacts are excluded from integrity accounting. Indexes and digests reference everything by construction; counting them as link sources makes orphan detection vacuously green. They are outputs of sorting, never evidence of it.

What remains for model judgment is small and bounded: which existing topic owns an incoming fact, and when a fact genuinely warrants a new topic. That judgment is delegated to a custodian role — whichever agent lane is executing sorting or maintenance under the rules above — so working agents never carry organizational instructions in their context. The load-bearing point of the custodian split is that spec adherence is enforced by the deterministic write path (schema validation, redaction, hashing at write), not by prompt discipline; the custodian supplies decisions, not compliance. Below the judgment floor (P5), the deterministic rules still hold — notes attach to the task they were written under and references are validated — so sorting degrades to coarse, never to corrupt.

3. Retrieval — three access patterns, three costs

Retrieval is three different problems wearing one name, and each pattern has its own mechanism:

• Push orientation. An agent cannot query memory it does not know exists — absent a session-start signal, stored knowledge sits unconsulted while the model answers from its own head (P2). Kaizen already solved this at the authoritative tier: R0 is the bounded, curated digest every session loads first. R0 stays exactly as it is; advisory memory earns its own gated channel — the R12 recall report proposed in Native narrative memory for Agent Kaizen — a memory palace hypothesis #2, stamped advisory_only, returning note-origin hits, never injected into R0. The digest describes what topics are about, not what files are named, because queries collide with concepts before they collide with filenames.
• Search. Targeted recall over the immutable tier is E4 semantic/hybrid query — already shipped, and the reason the storage function feeds the evidence plane at all.
• Navigation. From any hit, follow [[refs]] to neighboring records and read them. This spends the model's reasoning at refinement time — deciding which thread to pull next — instead of trusting a single similarity hop to land on the right chunk. It requires zero new code: the references from the sorting function plus existing read operations, documented as a workflow.

Two retrieval details round out the function. Recent notes are disproportionately valuable and should not require search: R12's default return includes a bounded most-recent-notes window alongside query hits. And because every retrieval already leaves trace_events, P2 and P3 are measurable without new instrumentation.

4. Maintenance — memory that only grows is noise with provenance

An unmaintained store fails slowly and then suddenly: superseded facts accumulate, references rot, orphans multiply, and one day recall returns confident contradictions. Maintenance is where most memory systems have nothing, and it decomposes into four mechanisms:

• Supersession. At the immutable tier nothing is ever deleted — evidence is history. But on any curated surface, a replaced fact must not survive its replacement: the write that records the new fact removes the old one everywhere on that surface, rewriting the artifact if that is what it takes. (Refinement of Native narrative memory for Agent Kaizen — a memory palace hypothesis #2, which rejected a contradiction-detector subsystem. That rejection stands — no scanning detector is proposed. Supersession here is a write-time rule applied only to derived artifacts, not a subsystem that hunts for conflicts.)
• Referential integrity lint. A deterministic verifier walks every [[ref]] and flags dangling references and orphaned topics (records nothing references, with generated indexes excluded as sources). It is a Q-lane check like any other — runs in the test suite, exposes counts, and its counts over time are the P4 instrument. Nothing about it is specific to notes; it generalizes to every cross-referenced record type in the data plane.
• Consolidation. An explicitly invoked improvement-lab (O-lane) operation: lint, re-attach orphans, regenerate indexes and digests, then score before/after on the recall eval set. Never a background process — every run is user-initiated and leaves ledger events. Because runs are scored, sorting policies become improvable through the same eval loop Kaizen applies to everything else: the management layer improves itself with the project's own machinery.
• Growth control. Filters or quotas on advisory documents are added only after pollution of normal evidence queries is recorded as a GOTCHA — Native narrative memory for Agent Kaizen — a memory palace hypothesis #2's trigger, unchanged.

The derived tier — the one genuinely new store, and it is disposable

The largest open design decision is whether a mutable current-state tier should exist at all: per-topic summaries that answer "what is true about X right now" in one read, instead of leaving the reader to synthesize current state from a pile of historical hits. The position taken here: if it is ever built, it must be derived and regenerable — compiled from immutable notes and authoritative records by consolidation runs, rebuildable from scratch at any time, advisory forever. Deleting it must lose nothing but convenience. Under that constraint the two-sources-of-truth failure cannot arise structurally, because the tier is a cache of the record, never a competitor to it. Its build trigger is P3 materializing as recorded evidence — stale or contradictory recalls in eval_scores — not the observation that it sounds useful.

Function	Failure mode prevented	Deterministic mechanism	Bounded judgment	Kaizen surface	Phase
Storage	continuity dies at compaction/reset/handoff	non-blocking append, redaction, hashing, idempotent replay	what is worth noting	T6 note, E1→E3	1
Sorting	fragmentation; unreachable records	topic keys, [[ref]] validation, generated-index exclusion	which topic owns a fact	notes conventions + Q-lane lint	1
Retrieval	discovery failure; staleness-by-similarity	bounded digest, advisory_only stamp, traces	navigation refinement	R0 (untouched) · E4 · R12 (gated)	2
Maintenance	contradiction accumulation; rot; unbounded growth	integrity lint, write-time supersession, explicit consolidation	re-attachment, summarization	Q-lane · O-lane · eval_scores	3

SWOT

Subject of the analysis: adopting the governed management layer as specified above, natively in Agent Kaizen.

Strengths

• It is governance over shipped substrate, not new infrastructure. The append-only redacted ledger, evidence pipeline, semantic/hybrid search, verifier lane, traces, evals, and bench harness all exist; the layer adds conventions, one lint, one gated report, and one explicit operation.
• Deterministic-first design degrades gracefully. Below the judgment floor, the guardrails still hold shape — valid references, task-attached notes, clean indexes — so the layer remains safe on minimum local hardware rather than being a frontier-model-only feature. That is local-first resilience by construction.
• Subordination of memory to records is structural, not aspirational. Immutable storage plus derived-only curation makes the classic failure — a second, fuzzier database silently competing with the real one — unrepresentable rather than merely discouraged.
• The discovery mechanism is already proven in production. R0 demonstrates that a bounded session-start digest gets memory consulted; the advisory tier extends a working pattern instead of inventing one.
• Every claim is measurable from day one. Each prediction names an existing instrument, and every expansion has a written trigger before any code lands — the layer can be evaluated, and abandoned, on evidence.
• Cross-surface sharing is inherited, not built. Any agent host that drives the CLI shares the same layer; there is no per-host memory silo to reconcile later.

Weaknesses

• The judgment-floor asymmetry is real. Guardrails bound damage, but sorting quality tracks the capability of the model making topic and supersession calls; on minimum hardware, organization stays coarse, and stronger backends will curate visibly better. The layer inherits a quality gradient the project has to own honestly.
• Write amplification at the derived tier. Attach-search, patch, and back-reference per fact is many model calls where an append is one; on local hardware that pushes curation to batch consolidation time and rules out heavy per-fact curation during live work.
• A mutable tier imports concurrency semantics the append-only plane never needed. Concurrent agents patching the same derived artifact need lock-or-merge discipline before shared use is safe; the source-lock lane would have to extend to cover it.
• Most of the layer is ahead of its evidence. Only the storage-function predictions have data paths that could fire today; sorting, retrieval, and maintenance mechanisms are designed against predicted failures. The written gates exist precisely because of this, but it remains design-on-spec.
• It adds standing verification surface. Integrity lint, recall evals, and consolidation correctness are new permanent tenants in the test and maintenance budget of a small project.

Opportunities

• R12's return shape can be current-state-first. When its trigger fires, the report can lead with derived summaries and attach note-origin evidence beneath — retrieval precision layered onto already-gated plumbing without changing when the report is justified.
• The integrity lint generalizes to the whole data plane. Dangling-reference detection is valuable for every cross-referenced record type, so memory work strengthens the integrity story everywhere, not just for notes.
• Consolidation runs are scored experiments. Because each run is measured on the recall eval set, sorting policy becomes an improvement-lab subject in its own right — the management layer is improvable by the same loop it serves.
• The retrieval walk is already traced. A graph-and-trace view in the extension would make memory behavior visible at a glance; easy to visualize is easy to debug, and the trace substrate is shipped.
• A thin MCP facade would widen the shared layer to any MCP-capable host, with the session digest riding the protocol's instructions field — the same proven discovery mechanism over a new transport.
• It is a distinctive public capability. Local-first, verification-gated, measured context-memory management is a data-plane feature few agent frameworks can claim, and it is publishable as such.

Threats

• Regression by convenience. The moment a derived artifact is hand-edited into de-facto authority, the second-database failure returns through the side door. Derivedness has to be enforced — regeneration overwrites hand edits by design — or it will erode one shortcut at a time.
• Scope gravity. Custodian roles pull toward daemons, consolidation pulls toward schedulers, recency windows pull toward persistent caches. Every one of those crossings ends explicit-invocation-only operation; the line holds or the local-trust story quietly dies.
• Anchoring on the derived tier. Agents that trust a stale current-state summary over fresh verification turn curation errors into propagated conclusions. advisory_only stamping mitigates; staleness probes in the recall evals are the necessary backstop.
• Pollution lag. Advisory volume can degrade normal evidence-query quality before the growth-control trigger formally fires; the E4 hit-quality signal needs watching from the first ingested note, not from the first GOTCHA.
• Eval hardness at the curation tier. Scoring "is this current-state summary correct and complete" requires judgment-grade ground truth, which is expensive to produce; the affordable eval set bounds how much curation quality can actually be validated.
• Opportunity cost. The layer competes for attention with active milestones; the phasing exists so the cheap phase can ship without betting the roadmap on the expensive one.

Conclusions

The management layer adopts in three phases, each gated on the previous phase's measurements:

• Phase 1 — storage and sorting discipline. Native narrative memory for Agent Kaizen — a memory palace hypothesis #2's moves unchanged (T6 note registry lines; session notes through E1→E3→E4), plus topic-keyed note conventions, [[ref]] citations to durable ids, and the referential-integrity lint as a Q-lane verifier. Roughly two registry lines, documentation, and one deterministic check; before/after bench runs prove no startup or write-path regression.
• Phase 2 — retrieval. R12 is built only when its written trigger fires (agents re-asking already-answered questions, recorded as a recurring GOTCHA), returning the bounded recency window plus query hits under an advisory_only stamp, with the reference-following navigation workflow documented alongside it.
• Phase 3 — maintenance and the derived tier. Write-time supersession, explicit consolidation runs, and per-topic current-state artifacts — built only on recorded P3 evidence of stale or contradictory recalls, and only as regenerable derived views.

The non-negotiables hold at every phase: R0 untouched; advisory memory never rewrites authoritative state; no background processes; no hosted services; no new dependencies; every expansion has a written trigger and a written stop condition. And the falsification path is explicit: if Phase 1 alone closes the continuity gap — no re-asking GOTCHAs, no staleness evidence — the layer stops there. A management layer that concludes it should stay small is a success of the method, not an abandonment of the idea.

Open questions for this discussion:

• Is topic-keyed organization worth the naming judgment it demands at write time, or should notes stay task-keyed and let consolidation do all topical grouping later, where it can be scored?
• Should the recency window live only inside R12, or is a separate bounded opt-in "recent advisory" report acceptable without eroding the R0 boundary?
• Where is the derived tier's regeneration boundary — full rebuild per consolidation run, or incremental patches with periodic full rebuilds?
• Below the judgment floor, should the layer refuse curation outright, or allow coarse curation and let the eval scores document the difference?

References

• Discussion #2 — Native narrative memory for Agent Kaizen — the predecessor hypothesis; its constraints and triggers are inherited here.
• Agent Kaizen — operation families cited: T6, E1/E3/E4, R0/R12 (proposed), Q-lane verifiers, O-lane improvement lab, trace_events, eval_scores.