[Feature] Archive directory GC + INDEX.jsonl rotation

[KbWen]

Summary

Bound the unbounded growth of .agentcortex/context/archive/ and the hash-chained audit ledger INDEX.jsonl with a rotation / garbage-collection pass that compacts old entries into summary indexes.

Motivation

Archived work logs and ship-history accumulate with no upper bound, and INDEX.jsonl grows monotonically. Over a long-lived project both surfaces silently inflate read cost and clutter ls/grep. Ship-history is rotated manually today; this generalizes it into a repeatable pass.

Proposed Solution

1. When archive file count exceeds a configurable threshold (e.g. 50), compress the oldest yearly batch into a single archive/<year>-summary.md (+ optional gzipped bodies).
2. Rotate INDEX.jsonl yearly, preserving the chain head/anchor so tamper-evidence still validates across the rotation boundary.
3. Add an archive-count check to validate.sh / validate.ps1 (WARN above threshold).

Files Affected

• .agentcortex/context/archive/
• .agentcortex/context/INDEX.jsonl
• .agentcortex/bin/validate.sh / validate.ps1
• new tool: .agentcortex/tools/rotate_archive.py

Scope

• Deploy / tooling change
• Guardrail update (.agent/rules/)

---

Backlog: docs/specs/_product-backlog.md #3 (depends on #1).

[KbWen]

Closing after re-audit. Verified there is no present pain point in this repo (e.g. archive ~13 files vs ~50 threshold; Spec Index ~9 entries already manually compacted; Domain Docs are small with no L2 bloat; no misroute/cache-staleness evidence). Premature — the backlog row remains as a future direction; reopen when a concrete signal appears.

[KbWen]

Reopening after a deeper, multi-pass verification (cross-checked against the actual code paths). Confirmed this is a real gap backed by an existing consumer/enforcement hook in this repo — not premature. Promoted to the 1-2 month optimization roadmap; see docs/specs/_product-backlog.md.

[KbWen]

Triage Analysis — #141 Archive directory GC + INDEX.jsonl rotation

Classification: feature (new tool + cross-platform validator change + guardrail update + a witness-compatibility design decision). Leaving a structured analysis rather than a direct fix per triage policy (complex, cross-module, blocked by #140).

1. Priority Assessment

The growth trajectory that made this speculative at filing has now crossed the proposed threshold, so this is a present pain point rather than a forecast:

Snapshot	archive/ files	INDEX.jsonl	size
At filing (2026-06-02)	~13	small	—
Backlog evidence (2026-07-10)	106	99 lines	1.3 MB
Today (2026-07-20)	125 .md + INDEX	120 lines / 60 KB	1.7 MB

The proposed WARN threshold is 50 — the archive is now ~2.5× over it. Every session that touches the archive (chain validation, decision-disposition scan, Phase-Summary audit) pays a linear read cost that keeps climbing. Priority is justified now, not hypothetically.

2. Root Cause Analysis

• Unbounded append surfaces. .agentcortex/context/archive/ accumulates one file per ship/handoff with no rotation; .agentcortex/context/archive/INDEX.jsonl is a hash-chained append-only ledger that grows monotonically (120 lines today).
• Path correction: the issue body names .agentcortex/context/INDEX.jsonl. The actual ledger is at .agentcortex/context/archive/INDEX.jsonl (confirmed: validate.sh:49, validate.ps1:241). Any implementation must target that path.
• Existing consumer / enforcement hook (the reopen justification, now verified): both validators already hash-chain-validate this ledger — validate.sh:378 "audit chain integrity" and validate.ps1:429-431. So the ledger is load-bearing for tamper-evidence; rotation cannot naively truncate it.
• No count/GC control exists today. grep archive across both validators shows path refs and chain checks only — there is no archive-count or size WARN. The rotate_archive.py tool named in the issue does not exist (NOT FOUND).

3. Proposed Approach

Three separable increments; ship #3 first for immediate value while #140 is pending.

1. Witness-compatible INDEX rotation. Backlog chore: add open-source scaffolding #3's own evidence records the binding constraint: "compaction prohibited by the append-only witness — rotation design must be witness-compatible." So do not rewrite old lines. Instead: seal the current chain at a yearly boundary into archive/INDEX-<year>.jsonl, write a genesis anchor line into the fresh INDEX.jsonl that embeds the prior chain's head hash, and teach the chain verifier to walk across the rotation boundary via that anchor. Tamper-evidence must validate end-to-end across the seam. (.agentcortex/tools/rotate_archive.py + verifier update.)
2. Archive body compaction. Above a configurable threshold (default 50), fold the oldest yearly batch of *.md into a single archive/<year>-summary.md index (optionally gzip the raw bodies). Keep this decoupled from the ledger so a compaction bug can never break tamper-evidence.
3. Validator WARN (cheapest, unblocked, do first). Add an archive-count check to validate.sh / validate.ps1 that WARNs above threshold. This is a genuine quick-win on its own and would already be firing today. Cross-platform parity is mandatory — sh + ps1 must match.

4. Risk & Trade-offs

• Highest risk: breaking the hash chain. A rotation that loses the anchor turns the tamper-evidence check from PASS into a hard FAIL for every downstream session. Mitigation: land the verifier's cross-boundary walk and a regression test before the first real rotation; dry-run mode default.
• Threshold churn / false alarms. 50 may be too low for a mature repo (we cleared it organically). Make it config-driven (.agent/config.yaml) and default the new check to WARN, never FAIL, so it never blocks a ship.
• Lifecycle-engine collision. [Feature] Tiered document lifecycle engine (4-tier state machine + config) #140 proposes a general tiered-lifecycle engine; building bespoke rotation here first risks rework. Mitigation: ship increment 3 (WARN only, zero coupling) now; gate increments 1–2 on [Feature] Tiered document lifecycle engine (4-tier state machine + config) #140's state-machine shape.
• Rollback: increment 3 is a pure additive check (revert = delete the block). Increments 1–2 must ship behind a dry-run flag with the sealed INDEX-<year>.jsonl retained, so rollback = restore the pre-rotation INDEX.jsonl from git + drop the summary file.

5. Estimated Scope

• Increment 3 (WARN check): 2 files (validate.sh, validate.ps1) + 1 test → quick-win, independently shippable now.
• Increments 1–2 (rotation + compaction): 4–5 files (new rotate_archive.py, chain verifier, both validators, config, tests) + a witness-compatibility design decision → feature (recommend an ADR for the rotation-anchor scheme given it touches tamper-evidence).
• Overall issue classification: feature.

6. Blocked By

• [Feature] Tiered document lifecycle engine (4-tier state machine + config) #140 (Tiered document lifecycle engine) — backlog dependency #1; increments 1–2 should inherit its state-machine/config shape rather than pre-empt it.
• Not blocked: increment 3 (archive-count WARN) can be split into its own quick-win PR today with no dependency on [Feature] Tiered document lifecycle engine (4-tier state machine + config) #140. Recommend doing exactly that if immediate signal is wanted while [Feature] Tiered document lifecycle engine (4-tier state machine + config) #140 is designed.