From 76719fb370eb3ae71e721ef4e0a00a15cac77ca7 Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Sat, 25 Jul 2026 01:22:29 +0000 Subject: [PATCH 1/2] Add pipeline diagram set (FIG-1..FIG-4) as committed mermaid --- .github/wiki-publish-map.json | 5 ++ docs/documentation-process.md | 15 +++++ docs/features/stage-e-operations.md | 91 +++++++++++++++++++++++++++++ docs/identification-pipeline.md | 74 +++++++++++++++++++++++ docs/pipeline-fidelity-gate.md | 50 ++++++++++++++++ docs/theory.md | 5 ++ 6 files changed, 240 insertions(+) diff --git a/.github/wiki-publish-map.json b/.github/wiki-publish-map.json index 1afc99193..dfadfd263 100644 --- a/.github/wiki-publish-map.json +++ b/.github/wiki-publish-map.json @@ -21,6 +21,11 @@ "wiki": "Identification-Pipeline", "targets": ["wiki"] }, + { + "source": "docs/pipeline-fidelity-gate.md", + "wiki": "Pipeline-Fidelity-Gate", + "targets": ["wiki"] + }, { "source": "docs/federation-v1.md", "wiki": "Federation-v1", diff --git a/docs/documentation-process.md b/docs/documentation-process.md index 4ece84a69..ae4c6f358 100644 --- a/docs/documentation-process.md +++ b/docs/documentation-process.md @@ -97,6 +97,21 @@ entry). A `Nav.Link` to `/guide` in `frontend/src/features/ui/Navbar.tsx` ships alongside this widening (ungated, same as `Download` — `/guide` has no backend dependency), closing proposal-i's "not yet built" item 3. +**Known gap, deliberately deferred (2026-07-25): mermaid diagrams are +wiki-only for now.** `docs/identification-pipeline.md` (FIG-1) and +`docs/pipeline-fidelity-gate.md` (FIG-2) each carry a committed +` ```mermaid ` fence. GitHub and the wiki both render it natively; the +site's guide page does not — `frontend/src/pages/guide/[[...slug]].tsx` +renders markdown via `marked` v18, which has no mermaid support, so a +fence would show as raw fenced source text on proxyprints.ca. Neither +page is in this file's `"site"` target list for exactly that reason. +Three ways to close the gap, owner sign-off needed before any of them: +(a) add a mermaid renderer to the guide page (new frontend dependency); +(b) commit an SVG beside each fence and have the site prefer it (no new +dependency, but a second artifact that can drift from the mermaid +source); (c) leave it wiki-only indefinitely. Nothing about this gap +blocks the `docs/`/wiki commit itself. + **Link-rewrite parity fixtures**: since one function now serves two output shapes (wiki mode vs. site mode) rather than two separate implementations, [`.github/scripts/tests/test_publish_wiki_link_rewrite.py`](../.github/scripts/tests/test_publish_wiki_link_rewrite.py) diff --git a/docs/features/stage-e-operations.md b/docs/features/stage-e-operations.md index 2fc84eabc..684a16c96 100644 --- a/docs/features/stage-e-operations.md +++ b/docs/features/stage-e-operations.md @@ -74,6 +74,42 @@ dispatched is allowed to drain (matching the existing kill-test/resume- contract's own "in-flight work drains, nothing new starts" discipline), but no NEW batch goes out while a trip is open. +### FIG-4a — Trip lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Clear + Clear --> Clear : sample under every bar
dispatch proceeds + Clear --> Tripped : any one bar crossed
EnvelopeTrip row written + Tripped --> Tripped : every further dispatch refuses
status halted-open-trip + Tripped --> Acknowledged : resolve_envelope_trip
--acknowledge-trip ID
mandatory human note + Acknowledged --> Clear : dispatch resumes + + note right of Tripped + There is no automatic edge out of Tripped. + No code path in the dispatch loop can + acknowledge a trip. This is the design. + end note +``` + +### FIG-4b — Gate anatomy + +A table, not a graph — five gates vary along five fixed attributes, which +a table renders directly and a graph would only obscure. + +| Gate | What it measures | Threshold | On fire | Resume | Cost when it fires | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Host load** | 1-minute load average, sampled fresh before every dispatch | `> 7.0` — the box has 8 OCPU with 1 pinned to networking, so 7.0 _is_ the 7 usable cores | Writes an `EnvelopeTrip`, returns `halted-new-trip`; all dispatch stops | Human `--acknowledge-trip` with a mandatory note | Total pipeline stop until a human returns. Both live trips were this bar | +| **RSS per worker** | Resident memory of the dispatching worker process | `> 512 MB` | Same — trip, halt | Same | Total stop. Never fired live | +| **Fetch-failure rate** | Failures over a rolling 500-outcome window, per worker process | `> 1%`, and never on an empty window | Same — trip, halt | Same | Total stop. Never fired live | +| **Google lockout** | Any `GoogleFetchLockoutError` | None — any single occurrence trips instantly | Stage C stops mid-batch; in-flight work drains, nothing new starts | Same | Total stop. Never fired live | +| **Concurrency cap** | Dispatches running at once, across every django-q2 worker process, via a Postgres advisory lock held on a **dedicated** connection (2026-07-25 fix, PR #453 — never Django's own shared `django.db.connection`, which a same-process `django_q` broker enqueue call could silently close mid-dispatch, auto-releasing the lock) | `2` slots, floored at 1 if misconfigured | Returns `throttled-concurrency-cap` and does nothing — **no trip, no ledger row**, but does advance the `StageEThrottleCounter` observability counter | **Automatic** — the next sweep picks the card up | One deferred micro-batch. Did not bind on either live shakedown run (see FIG-3a below) — root-caused and fixed 2026-07-25 (PR #453); see "Concurrency cap" below for the full incident writeup | + +The cap is the only one of the five that is proactive rather than +reactive, and the only one that recovers without a human. The other four +are the same mechanism with different sensors: measure, cross, stop, wait +for a person. + ## Resume requires a fresh owner action — always **There is no self-resume, in any case, for any bar.** This is a hard @@ -134,6 +170,61 @@ not an oversight: the admin is a monitoring surface for finding a `trip_id` to hand to the command, not a second, less-visible resume path that could bypass the mandatory `--note` and the CLI's own audit trail. +### FIG-3a — Shakedown timeline (dated snapshot, read 2026-07-25T01:18:57Z) + +A worked example of the runbook above, against the pipeline's first two +live PASSIVE-mode shakedown runs. This is a **dated snapshot, not a live +readout** — written as a two-incident post-mortem so it stays true after +the second trip below is eventually cleared, not as a "current state" +claim that goes stale the moment someone runs `resolve_envelope_trip`. + +```mermaid +timeline + title Stage E shakedown · host load against a 7.0 ceiling on 7 usable cores + section Run 1 — 2026-07-24 + 21:44:47Z : Backstop sweep opens the first dispatch + 21:44:50Z : 9 dispatches now running concurrently on 7 cores + 21:44:50-56Z : 7 of 9 crash · IntegrityError on a duplicate vote + 21:46:16Z : TRIP envtrip-be6e5db9 · load 11.85 vs ceiling 7.0 · ALL DISPATCH HALTS + section Between the runs + Vote collision fixed : skip-if-exists guard plus ignore_conflicts + Concurrency cap added : 2 slots via Postgres advisory locks + section Run 2 — 2026-07-25 + 00:22:13Z : Owner acknowledges trip 1 · dispatch resumes + 00:23:23-37Z : 9 dispatches · ALL 9 COMPLETE · the collision fix held + 00:23-25Z : Cap does NOT bind · 8 unlock warnings · 0 throttles + 00:25:04Z : TRIP envtrip-73e1eb6d · load 11.40 vs ceiling 7.0 · STILL OPEN +``` + +**Reading it:** one bug was fixed and one was revealed. The vote collision +is gone — run 2 wrote nine clean ledger rows where run 1 lost seven. The +concurrency cap that was supposed to prevent the load spike never engaged: +Postgres reported the advisory lock was already released by the time each +dispatch tried to unlock it, so every dispatch believed it had a free slot +and zero were ever throttled. Load reached 11.40 against a 7.0 ceiling for +the second time, and the envelope caught it for the second time. + +**State as of the stamp above:** trip `envtrip-20260724T214616-be6e5db9` +(run 1) is **acknowledged**. Trip `envtrip-20260725T002504-73e1eb6d` +(run 2) remains **open** — its root cause (the advisory lock riding +Django's shared, same-thread connection, which a same-process `django_q` +enqueue call could silently close mid-dispatch) has since been found and +fixed (PR #453, see FIG-4b and "Concurrency cap" below for the mechanism), +but a fix does not self-clear a trip: per "Resume requires a fresh owner +action — always" above, only an explicit `resolve_envelope_trip --acknowledge-trip` against this specific trip ID resumes dispatch. This +line will go stale the moment that happens; the timeline above will not. + +**Deliberately not committed here: the quantitative load-versus-ceiling +chart.** Mermaid has no primitive for a threshold line, a shaded +exceedance band, or a halted interval, so a faithful version of that chart +is hand-authored SVG, not mermaid — and isn't checked into `docs/` in this +pass (no asset-maintenance owner assigned yet, and the real +`STAGE_E_MICRO_BATCH_SIZE`/cap-tuning numbers this chart would anchor to +are still pending the phase-3 shakedown per "Micro-batch sizing" above). +The event timeline above carries the same facts in prose-adjacent form; +the chart can follow once there's real tuning data to plot against the +ceiling. + ## Phase 2 — the streaming dispatch loop Built 2026-07-24, per the owner-approved Phase 2 implementation task for diff --git a/docs/identification-pipeline.md b/docs/identification-pipeline.md index c07cadd07..acc5bf267 100644 --- a/docs/identification-pipeline.md +++ b/docs/identification-pipeline.md @@ -17,6 +17,80 @@ gate's current status; this file describes the mechanics, not the gate. **Reviewed and approved by the owner, 2026-07-21.** Written for the pre-197k review. +## FIG-1 — Pipeline flow, where a card can stop + +```mermaid +flowchart TD + IN["Card named eligible
card-create · evidence-change · cron backstop sweep"] + + IN --> G1{"streaming enabled?"} + G1 -- "no" --> X1{{"OFF
STAGE_E_STREAMING_ENABLED = False
no work, no ledger row"}} + G1 -- "yes" --> G2{"envelope trip already open?"} + + G2 -- "yes" --> X2{{"HALTED · OPEN TRIP
every dispatch refuses
no self-resume, ever"}} + G2 -- "no" --> G3{"fresh signal sample
breaches a bar?"} + + G3 -- "yes" --> X3{{"HALTED · NEW TRIP
EnvelopeTrip row persisted
all dispatch stops"}} + G3 -- "no" --> SEL["Select micro-batch
size 25, seed first, backlog fill"] + + SEL --> X4{{"ALREADY PROCESSED
excluded by the anti-join:
evidence carries every manifest key"}} + SEL --> G4{"anything eligible?"} + G4 -- "no" --> X5{{"EMPTY SELECTION
nothing left to do"}} + G4 -- "yes" --> G5{"concurrency slot free?"} + + G5 -- "no" --> X6{{"THROTTLED · CAP
all 2 slots held
deferred to the next sweep
NO ledger row is written"}} + G5 -- "yes" --> LED["Open ledger row
PilotRunLedger, status running"] + + LED --> SC["STAGE C · fetch image, extract OCR evidence
one card at a time"] + SC -- "fetch fails" --> X7{{"FETCH FAILURE
card skipped, counted,
fed to the rolling 500-card window"}} + SC -- "crash" --> X8{{"FAILED DISPATCH
ledger row marked failed
with a triage-able reason"}} + SC --> SD["STAGE D · join-key → fallback → slow-path
casts machine votes at weight 0.5"] + + SD --> CONS["Weighted consensus over every vote on the card"] + CONS --> G6{"weight ≥ 2
AND share ≥ 0.6
AND at least one human-backed vote"} + + G6 -- "no" --> X9{{"CONSENSUS FLOOR
218,351 cards park here
machine votes alone can never clear it"}} + G6 -- "yes" --> RES(["RESOLVED PRINTING
3 cards"]) + + X9 --> WTC["WTC question feed
serves ≥51% of questions from cards
one more human vote would resolve"] + WTC --> HV["Human vote cast, weight 1.0"] + HV --> CONS + + classDef halt fill:#f7768e,stroke:#8c3d4e,stroke-width:2px,color:#1a1b26 + classDef defer fill:#e0af68,stroke:#8a6a3d,stroke-width:2px,color:#1a1b26 + classDef excl fill:#7dcfff,stroke:#3f7f9c,stroke-width:2px,color:#1a1b26 + classDef work fill:#24283b,stroke:#565f89,stroke-width:1px,color:#c0caf5 + classDef gate fill:#2f3549,stroke:#ff9e64,stroke-width:2px,color:#c0caf5 + classDef done fill:#9ece6a,stroke:#5c7c3d,stroke-width:2px,color:#1a1b26 + + class X2,X3,X8,X9 halt + class X6,X7 defer + class X1,X4,X5 excl + class IN,SEL,LED,SC,SD,CONS,WTC,HV work + class G1,G2,G3,G4,G5,G6 gate + class RES done +``` + +Shape carries the meaning first: a hexagon is an interception (the card +stops here); a diamond is a gate being evaluated; a plain rectangle is +work actually happening; a stadium is a terminal outcome. Colour then +grades _why_ a hexagon stopped — red is a **hard halt** (a human must +act), amber is a **soft defer** (the system retries by itself on its +own), blue is a **correct exclusion** (nothing is wrong, the card just +doesn't need this pass). Every `classDef` sets both `fill:` and `color:` +so the diagram reads correctly under either a light or a dark GitHub/wiki +theme. + +**Reading it:** eight of the nine interceptions are cheap and local. The +ninth — CONSENSUS FLOOR — holds 218,351 of 218,355 cards, and it is the +only one that is not a fault. It is the soundness property: no volume of +machine votes resolves a printing, so the pipeline's throughput is +bounded by human attention on purpose. The loop back through the WTC +question feed is the actual design claim — machines narrow the candidate +set, humans close it. See +[`pipeline-fidelity-gate.md`](pipeline-fidelity-gate.md) for how many +cards are currently sitting at each stage. + ## What exists before anything runs - A **Card row**: name, source drive, and a content phash of the image. diff --git a/docs/pipeline-fidelity-gate.md b/docs/pipeline-fidelity-gate.md index 7a45a9079..a95ca1a0d 100644 --- a/docs/pipeline-fidelity-gate.md +++ b/docs/pipeline-fidelity-gate.md @@ -332,6 +332,56 @@ CLAUDE.md's "Governing premise: we index, we do not store images"). ## 6. Verified data snapshot (2026-07-22, live) +### FIG-2 — Catalog funnel (shape only, no counts) + +```mermaid +flowchart TD + A["CATALOG"] --> B["PERCEPTUAL HASH
extracted"] + A -.-> A1{{"no phash yet"}} + + B --> C["IMAGE EVIDENCE EXTRACTED
near-complete"] + B -.-> B1{{"phash but no evidence yet"}} + + C --> D["CARRIES ≥1 PRINTING VOTE
machine votes abundant"] + C -.-> C1{{"evidence but no vote yet
Stage D has not reached these"}} + + D --> E{"human-backed consensus gate
weight ≥ 2 · share ≥ 0.6 · a human said so"} + + E -- "cleared" --> F(["RESOLVED PRINTING
rare — by design"]) + E -- "held" --> G{{"UNRESOLVED
machine votes alone can never clear this gate"}} + E -- "ruled out" --> H(["NO MATCH"]) + + G --> I["REVIEW QUEUE
awaiting a human vote"] + + classDef stage fill:#24283b,stroke:#565f89,stroke-width:1px,color:#c0caf5 + classDef gate fill:#2f3549,stroke:#ff9e64,stroke-width:2px,color:#c0caf5 + classDef halt fill:#f7768e,stroke:#8c3d4e,stroke-width:2px,color:#1a1b26 + classDef excl fill:#7dcfff,stroke:#3f7f9c,stroke-width:2px,color:#1a1b26 + classDef done fill:#9ece6a,stroke:#5c7c3d,stroke-width:2px,color:#1a1b26 + + class A,B,C,D,I stage + class E gate + class G halt + class A1,B1,C1 excl + class F,H done +``` + +**Deliberately shape-only — no absolute counts in this diagram** (owner +ruling, 2026-07-25): every number behind each row above is real, live, +and already published in the tables directly below this figure on this +same page — repeating them inside the diagram itself would fork the +numbers this page's own rule forbids ("don't restate gate status/ +decisions elsewhere, link here") and would go stale the moment the next +pass runs. What the shape says instead: extraction (phash → image +evidence) is functionally complete; Stage D coverage (the "carries a +vote" row) is the one place more machine capacity still helps; the +human-backed consensus gate is a **hard** requirement, not a formality — +machine votes alone can never clear it, so "held" is the gate working as +designed, not a fault, and the review queue is where held cards wait for +exactly the one thing that clears them: a human vote. Absolute counts +return to this diagram once real user confirmations start accumulating +in volume; until then, read them from the tables immediately below. + **Catalog**: 218,285 cards; 218,270 with a current `ImageEvidence` row. **Vote pool** (`CardPrintingTag` / `CardTagVote` / `CardArtistVote`, diff --git a/docs/theory.md b/docs/theory.md index e69b260e7..e8d678c26 100644 --- a/docs/theory.md +++ b/docs/theory.md @@ -880,6 +880,11 @@ otherwise. ## 10. Streaming and continuous operation +See [`identification-pipeline.md`'s FIG-1](identification-pipeline.md#fig-1--pipeline-flow-where-a-card-can-stop) +for a diagram of every interception point a card can hit while moving +through the mechanism this section describes — repo-internal jargon this +file deliberately avoids, so the diagram lives there, not here. + Stage E (`docs/proposals/stage-e-streaming.md`, GitHub issue #153) moves this pipeline from discrete batch runs to continuous, event-driven dispatch. The question this section answers: does anything above change From 642518c049e7aaddf651c37d83e5713b15f85f49 Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Sat, 25 Jul 2026 01:24:45 +0000 Subject: [PATCH 2/2] Strip absolute counts from FIG-1 per shape-only ruling --- docs/identification-pipeline.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/docs/identification-pipeline.md b/docs/identification-pipeline.md index acc5bf267..ec6d3a71a 100644 --- a/docs/identification-pipeline.md +++ b/docs/identification-pipeline.md @@ -49,8 +49,8 @@ flowchart TD SD --> CONS["Weighted consensus over every vote on the card"] CONS --> G6{"weight ≥ 2
AND share ≥ 0.6
AND at least one human-backed vote"} - G6 -- "no" --> X9{{"CONSENSUS FLOOR
218,351 cards park here
machine votes alone can never clear it"}} - G6 -- "yes" --> RES(["RESOLVED PRINTING
3 cards"]) + G6 -- "no" --> X9{{"CONSENSUS FLOOR
nearly every card parks here
machine votes alone can never clear it"}} + G6 -- "yes" --> RES(["RESOLVED PRINTING
a human vote cleared it"]) X9 --> WTC["WTC question feed
serves ≥51% of questions from cards
one more human vote would resolve"] WTC --> HV["Human vote cast, weight 1.0"] @@ -82,14 +82,19 @@ so the diagram reads correctly under either a light or a dark GitHub/wiki theme. **Reading it:** eight of the nine interceptions are cheap and local. The -ninth — CONSENSUS FLOOR — holds 218,351 of 218,355 cards, and it is the -only one that is not a fault. It is the soundness property: no volume of -machine votes resolves a printing, so the pipeline's throughput is -bounded by human attention on purpose. The loop back through the WTC -question feed is the actual design claim — machines narrow the candidate -set, humans close it. See -[`pipeline-fidelity-gate.md`](pipeline-fidelity-gate.md) for how many -cards are currently sitting at each stage. +ninth — CONSENSUS FLOOR — is where the overwhelming majority of the +catalog sits by volume, and it is the only one that is not a fault. It is +the soundness property: no volume of machine votes resolves a printing, +so the pipeline's throughput is bounded by human attention on purpose. +The loop back through the WTC question feed is the actual design claim — +machines narrow the candidate set, humans close it. **Deliberately no +absolute counts in this diagram** (owner ruling, 2026-07-25, same +"shape for now, numbers once traffic starts the confirmations" posture as +FIG-2 below) — see +[`pipeline-fidelity-gate.md`](pipeline-fidelity-gate.md), the single +source of truth for gate status, for how many cards are currently sitting +at each stage. Absolute counts return to this diagram once real user +confirmations start accumulating in volume. ## What exists before anything runs