Add Stage E streaming design brief

[WilfordGrimley]

Description

Docs-only design brief for Stage E ("streaming assembly + resume contract + kill-test," #153): docs/proposals/stage-e-streaming.md.

Covers, each grounded in a cited doc/PR/live PilotRunLedger read (2026-07-24):

• Measured envelope (Stage C/D throughput across regimes, consensus_recompute cost) — includes an explicit correction where the task brief's own "Stage D ~4-5ms/card" and "~190MB RSS" figures could not be found in the written record or the live ledger and are flagged rather than restated.
• Proven primitives already built (crash-drill kill-test on open PR Graduate crash-drill script, fix test snapshot-sequence pinning #405, PRs Fix local_lands_identify write-path vote collision and dry-run yield display #411/Command-lifecycle hardening for Stage C/D pilot commands #373/Read stored ImageEvidence first in the lands identifier #391/Persist scryfall_cache volume and fail loud on a missing cache #412).
• Six design decisions: trigger, work granularity, backpressure, consensus-recompute placement, the human-backed gate in a continuous world (proposes a standing owner-ratified authorization envelope replacing per-batch polling), observability.
• Five efficiency candidates, each checked against docs/theory.md's soundness bound (decode-or-abstain + 0 resolution false-accept) — one variant explicitly marked REJECTED where it would trade soundness for speed.
• Hardware envelope & federated scalability: this box's measured specs/co-tenancy, a scale-down requirement (single-core floor mode, bounded memory, graceful pause/resume), disk/DB footprint from a live read-only query, the federation-export seam, and a forward-looking (not designed) sibling-compute-sharing subsection with an honest "not yet" recommendation.
• Owner-ratified sequencing: issue Bug-A re-scan tail: remaining sources after wave-1 measure #418's Bug-A tail is streaming's shakedown cohort (no further batch pass); AI-art detector: rescan-on-evidence-change tool (no-marker-hit negatives are permanently gated) #278 is the second consumer.
• A kill-test acceptance section extending the existing drill to the streaming dispatcher.

Refs #153, #418, #278.

Checklist

• Docs-only change; no code/migrations touched.
• Every cited number verified against a committed doc, a merged/open PR, or a live read-only query (Postgres PilotRunLedger/table sizes via docker exec, GitHub issue/PR bodies via gh api) — nothing taken on faith from the task brief.
• Interconnection: new file linked from docs/README.md's Plans & proposals table, docs/pipeline-fidelity-gate.md's "What remains open" §Bug-A item, and docs/features/catalog-completion-plan.md's Stage E resume-contract paragraph. python3 .github/scripts/docs_lint.py --strict clean.
• No migration freeze conflict (docs-only; also confirmed no deploy-freeze-active label active).
• PROTECTED CORE untouched (brief only discusses/cites local_phash.py/local_lands_identify.py, doesn't modify them).

Open items (also listed in the brief's §9)

1. §3 decision (5)'s standing authorization envelope needs explicit owner-ruled numeric bounds before implementation.
2. CardScanLog retention/pruning policy under continuous evidence-change re-entry — flagged, not decided.
3. §4 item 1's per-tier attribution counter should land before any escalation-tier reordering.
4. §6 item 1's Bug-A tail count (6,535 per Bug-A re-scan tail: remaining sources after wave-1 measure #418's snapshot vs. 17,531-10,437=7,094 per the gate doc's later, larger pool) should be re-queried fresh at cutover, not hardcoded.
5. PR Graduate crash-drill script, fix test snapshot-sequence pinning #405 (crash-drill script) is still open — §7's kill-test extension should sequence after it merges.

🤖 Generated with Claude Code