[Feature] Tiered document lifecycle engine (4-tier state machine + config)

[KbWen]

Summary

A 4-tier document lifecycle state machine (active → warm → cold → archived) with config-driven transitions, governing how governance artifacts age and compact over time. This is the umbrella engine that the archive-GC, spec-filtering, and warm→cold summarization issues build on.

Motivation

Several data surfaces (archive/, Global Lessons, Spec Index, INDEX.jsonl) grow unbounded and rely only on advisory, per-surface compaction. A single lifecycle engine gives every surface a consistent, configurable aging policy instead of ad-hoc rules scattered across workflows.

Proposed Solution

1. Define 4 tiers + transition triggers (age, count, status) in .agent/config.yaml.
2. A driver that, at /ship or a maintenance pass, moves artifacts across tiers and invokes the matching compaction (GC / status-filter / summarization).
3. Serve as the foundation for the archive-GC, spec-index-filtering, and warm→cold summarization work.

Files Affected

• .agent/config.yaml
• .agent/workflows/ship.md
• new tool under .agentcortex/tools/

Scope

• Guardrail update (.agent/rules/)
• Deploy / tooling change

---

Backlog: docs/specs/_product-backlog.md #1. Related: existing issue #14 (Work Log auto-compaction) is one consumer of this engine.

[KbWen]

Triage Analysis — #140 Tiered Document Lifecycle Engine

Classification: architecture-change (umbrella engine, 3+ files, new tooling, dependency for #141/#143/#144). Leaving an analysis comment rather than a direct fix per triage policy.

1. Priority Assessment

This is backlog entry #1 and the foundation for the lifecycle cluster: #141 (archive GC), #143 (spec-index filtering), and #144 (warm→cold summarization) all declare #1 as a dependency in docs/specs/_product-backlog.md. Shipping the engine config + driver unblocks three downstream features, so it has leverage despite its P1 (not P0) priority. The spec already exists and is well-developed: docs/specs/tiered-doc-lifecycle.md (status: draft).

2. Root Cause Analysis — what's already done vs. missing

The spec was partially implemented as "independent quick-wins" (per spec §Backlog Mapping, items #2/#4), but the umbrella engine itself is not built:

Already shipped:

• .agent/config.yaml:94-108 — a subset document_lifecycle: block exists with global_lessons_max_entries, spec_index_max_entries, raw_intake_cleanup only.
• Global Lessons cap enforced in .agent/workflows/retro.md:27.
• Spec Index cap enforced in .agent/workflows/ship.md:195.

Still missing (the actual engine):

• Config keys from spec §AC.1 are absent: warm_ttl_days, cold_ttl_days, max_warm_entries, max_active_worklogs, archive_index_max_lines, pinned_docs (verified: grep -E "warm_ttl|cold_ttl|max_warm|max_active|pinned_docs" .agent/config.yaml → no matches).
• Tier model in INDEX.jsonl: existing records (.agentcortex/context/archive/INDEX.jsonl) carry no tier field; there is no cold/ subdir under .agentcortex/context/archive/.
• Driver tool: no lifecycle driver exists under .agentcortex/tools/ (the issue proposes one). The /ship post-archival tier-transition + GC step (spec §AC.4/§AC.5) is unwired.

3. Proposed Approach (deliver the engine; defer GC/summarization to #141/#144)

Scope #140 to the config + tier-tagging + driver skeleton so the consumers have a stable contract:

1. Extend .agent/config.yaml document_lifecycle: with the missing engine tunables (spec §AC.1). Additive only — existing caps stay.
2. Add tier to the INDEX.jsonl schema (default warm on /ship archive) and document the four tier values. Backfill existing rows to tier: warm via a one-time migration in the driver.
3. New driver .agentcortex/tools/lifecycle_sweep.py — deterministic, event-driven (called from /ship), pure tier-transition bookkeeping (age/count → tier field) with no LLM summarization (that is [Feature] Warm-to-Cold LLM summarization pass in /ship #144, and spec §Non-goals keeps generation out of validate). Honor pinned_docs exemption.
4. Wire one call site in .agent/workflows/ship.md after the existing archival step.
5. validate.sh / validate.ps1 (both, for cross-platform parity): add a WARN-level warm-tier count check (spec §AC.10). Keep WARN→FAIL upgrades (§AC.9/§AC.12) out of this issue to avoid breaking existing logs.
6. Flip spec frontmatter status: draft → active and update the backlog row cleanup: v1.0 root hygiene #1 status on ship.

4. Risk & Trade-offs

• Scope creep is the main risk — the spec bundles 13 backlog items. Recommend [Feature] Tiered document lifecycle engine (4-tier state machine + config) #140 ship only config + tier field + driver skeleton + WARN check; leave actual deletion/GC to [Feature] Archive directory GC + INDEX.jsonl rotation #141 and summarization to [Feature] Warm-to-Cold LLM summarization pass in /ship #144. Document the cut explicitly so coverage isn't silently dropped.
• Data safety: tier transitions must be working-copy-only (spec §Constraints) — git history preserved. The skeleton only tags tiers; it must not delete files (deletion lands in [Feature] Archive directory GC + INDEX.jsonl rotation #141 under test).
• Cross-platform parity: any validate check must land in both .sh and .ps1 (repo guardrail).
• Rollback: additive config + a new tool + one guarded /ship call site → revert is a clean branch revert; no migration of governance state beyond the additive tier field (which downstream readers can ignore).

5. Estimated Scope

~4–5 files: .agent/config.yaml, .agent/workflows/ship.md, new .agentcortex/tools/lifecycle_sweep.py, .agentcortex/bin/validate.sh + .ps1, plus spec/backlog status. Classification: architecture-change (new cross-cutting engine + INDEX.jsonl schema change). Needs /plan with a spec update before implementation — not a same-run direct fix.

6. Blocked By

Nothing blocks #140; it is the root of the cluster. Conversely #141, #143, #144 should wait on the config + tier contract landed here. Suggest sequencing: #140 (engine) → #141 (GC) → #144 (summarization), with #143 (spec-index filtering) shippable in parallel since its cap already partially exists.