Land pipeline diagram set (FIG-1..FIG-4) as committed mermaid

.github/wiki-publish-map.json

@@ -21,6 +21,11 @@
           "wiki": "Identification-Pipeline",
           "targets": ["wiki"]
         },
+        {
+          "source": "docs/pipeline-fidelity-gate.md",
+          "wiki": "Pipeline-Fidelity-Gate",
+          "targets": ["wiki"]
+        },
         {
           "source": "docs/federation-v1.md",
           "wiki": "Federation-v1",

docs/documentation-process.md

@@ -97,6 +97,21 @@ entry). A `Nav.Link` to `/guide` in `frontend/src/features/ui/Navbar.tsx`
 ships alongside this widening (ungated, same as `Download` — `/guide` has
 no backend dependency), closing proposal-i's "not yet built" item 3.
 
+**Known gap, deliberately deferred (2026-07-25): mermaid diagrams are
+wiki-only for now.** `docs/identification-pipeline.md` (FIG-1) and
+`docs/pipeline-fidelity-gate.md` (FIG-2) each carry a committed
+` ```mermaid ` fence. GitHub and the wiki both render it natively; the
+site's guide page does not — `frontend/src/pages/guide/[[...slug]].tsx`
+renders markdown via `marked` v18, which has no mermaid support, so a
+fence would show as raw fenced source text on proxyprints.ca. Neither
+page is in this file's `"site"` target list for exactly that reason.
+Three ways to close the gap, owner sign-off needed before any of them:
+(a) add a mermaid renderer to the guide page (new frontend dependency);
+(b) commit an SVG beside each fence and have the site prefer it (no new
+dependency, but a second artifact that can drift from the mermaid
+source); (c) leave it wiki-only indefinitely. Nothing about this gap
+blocks the `docs/`/wiki commit itself.
+
 **Link-rewrite parity fixtures**: since one function now serves two output
 shapes (wiki mode vs. site mode) rather than two separate implementations,
 [`.github/scripts/tests/test_publish_wiki_link_rewrite.py`](../.github/scripts/tests/test_publish_wiki_link_rewrite.py)

docs/features/stage-e-operations.md

@@ -74,6 +74,42 @@ dispatched is allowed to drain (matching the existing kill-test/resume-
 contract's own "in-flight work drains, nothing new starts" discipline), but
 no NEW batch goes out while a trip is open.
 
+### FIG-4a — Trip lifecycle
+
+```mermaid
+stateDiagram-v2
+    [*] --> Clear
+    Clear --> Clear : sample under every bar<br/>dispatch proceeds
+    Clear --> Tripped : any one bar crossed<br/>EnvelopeTrip row written
+    Tripped --> Tripped : every further dispatch refuses<br/>status halted-open-trip
+    Tripped --> Acknowledged : resolve_envelope_trip<br/>--acknowledge-trip ID<br/>mandatory human note
+    Acknowledged --> Clear : dispatch resumes
+
+    note right of Tripped
+        There is no automatic edge out of Tripped.
+        No code path in the dispatch loop can
+        acknowledge a trip. This is the design.
+    end note
+```
+
+### FIG-4b — Gate anatomy
+
+A table, not a graph — five gates vary along five fixed attributes, which
+a table renders directly and a graph would only obscure.
+
+| Gate                   | What it measures                                                                                                                                                                                                                                                                                                                | Threshold                                                                                | On fire                                                                                                                                               | Resume                                           | Cost when it fires                                                                                                                                                                             |
+| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Host load**          | 1-minute load average, sampled fresh before every dispatch                                                                                                                                                                                                                                                                      | `> 7.0` — the box has 8 OCPU with 1 pinned to networking, so 7.0 _is_ the 7 usable cores | Writes an `EnvelopeTrip`, returns `halted-new-trip`; all dispatch stops                                                                               | Human `--acknowledge-trip` with a mandatory note | Total pipeline stop until a human returns. Both live trips were this bar                                                                                                                       |
+| **RSS per worker**     | Resident memory of the dispatching worker process                                                                                                                                                                                                                                                                               | `> 512 MB`                                                                               | Same — trip, halt                                                                                                                                     | Same                                             | Total stop. Never fired live                                                                                                                                                                   |
+| **Fetch-failure rate** | Failures over a rolling 500-outcome window, per worker process                                                                                                                                                                                                                                                                  | `> 1%`, and never on an empty window                                                     | Same — trip, halt                                                                                                                                     | Same                                             | Total stop. Never fired live                                                                                                                                                                   |
+| **Google lockout**     | Any `GoogleFetchLockoutError`                                                                                                                                                                                                                                                                                                   | None — any single occurrence trips instantly                                             | Stage C stops mid-batch; in-flight work drains, nothing new starts                                                                                    | Same                                             | Total stop. Never fired live                                                                                                                                                                   |
+| **Concurrency cap**    | Dispatches running at once, across every django-q2 worker process, via a Postgres advisory lock held on a **dedicated** connection (2026-07-25 fix, PR #453 — never Django's own shared `django.db.connection`, which a same-process `django_q` broker enqueue call could silently close mid-dispatch, auto-releasing the lock) | `2` slots, floored at 1 if misconfigured                                                 | Returns `throttled-concurrency-cap` and does nothing — **no trip, no ledger row**, but does advance the `StageEThrottleCounter` observability counter | **Automatic** — the next sweep picks the card up | One deferred micro-batch. Did not bind on either live shakedown run (see FIG-3a below) — root-caused and fixed 2026-07-25 (PR #453); see "Concurrency cap" below for the full incident writeup |
+
+The cap is the only one of the five that is proactive rather than
+reactive, and the only one that recovers without a human. The other four
+are the same mechanism with different sensors: measure, cross, stop, wait
+for a person.
+
 ## Resume requires a fresh owner action — always
 
 **There is no self-resume, in any case, for any bar.** This is a hard
@@ -134,6 +170,61 @@ not an oversight: the admin is a monitoring surface for finding a `trip_id`
 to hand to the command, not a second, less-visible resume path that could
 bypass the mandatory `--note` and the CLI's own audit trail.
 
+### FIG-3a — Shakedown timeline (dated snapshot, read 2026-07-25T01:18:57Z)
+
+A worked example of the runbook above, against the pipeline's first two
+live PASSIVE-mode shakedown runs. This is a **dated snapshot, not a live
+readout** — written as a two-incident post-mortem so it stays true after
+the second trip below is eventually cleared, not as a "current state"
+claim that goes stale the moment someone runs `resolve_envelope_trip`.
+
+```mermaid
+timeline
+    title Stage E shakedown · host load against a 7.0 ceiling on 7 usable cores
+    section Run 1 — 2026-07-24
+        21:44:47Z : Backstop sweep opens the first dispatch
+        21:44:50Z : 9 dispatches now running concurrently on 7 cores
+        21:44:50-56Z : 7 of 9 crash · IntegrityError on a duplicate vote
+        21:46:16Z : TRIP envtrip-be6e5db9 · load 11.85 vs ceiling 7.0 · ALL DISPATCH HALTS
+    section Between the runs
+        Vote collision fixed : skip-if-exists guard plus ignore_conflicts
+        Concurrency cap added : 2 slots via Postgres advisory locks
+    section Run 2 — 2026-07-25
+        00:22:13Z : Owner acknowledges trip 1 · dispatch resumes
+        00:23:23-37Z : 9 dispatches · ALL 9 COMPLETE · the collision fix held
+        00:23-25Z : Cap does NOT bind · 8 unlock warnings · 0 throttles
+        00:25:04Z : TRIP envtrip-73e1eb6d · load 11.40 vs ceiling 7.0 · STILL OPEN
+```
+
+**Reading it:** one bug was fixed and one was revealed. The vote collision
+is gone — run 2 wrote nine clean ledger rows where run 1 lost seven. The
+concurrency cap that was supposed to prevent the load spike never engaged:
+Postgres reported the advisory lock was already released by the time each
+dispatch tried to unlock it, so every dispatch believed it had a free slot
+and zero were ever throttled. Load reached 11.40 against a 7.0 ceiling for
+the second time, and the envelope caught it for the second time.
+
+**State as of the stamp above:** trip `envtrip-20260724T214616-be6e5db9`
+(run 1) is **acknowledged**. Trip `envtrip-20260725T002504-73e1eb6d`
+(run 2) remains **open** — its root cause (the advisory lock riding
+Django's shared, same-thread connection, which a same-process `django_q`
+enqueue call could silently close mid-dispatch) has since been found and
+fixed (PR #453, see FIG-4b and "Concurrency cap" below for the mechanism),
+but a fix does not self-clear a trip: per "Resume requires a fresh owner
+action — always" above, only an explicit `resolve_envelope_trip --acknowledge-trip` against this specific trip ID resumes dispatch. This
+line will go stale the moment that happens; the timeline above will not.
+
+**Deliberately not committed here: the quantitative load-versus-ceiling
+chart.** Mermaid has no primitive for a threshold line, a shaded
+exceedance band, or a halted interval, so a faithful version of that chart
+is hand-authored SVG, not mermaid — and isn't checked into `docs/` in this
+pass (no asset-maintenance owner assigned yet, and the real
+`STAGE_E_MICRO_BATCH_SIZE`/cap-tuning numbers this chart would anchor to
+are still pending the phase-3 shakedown per "Micro-batch sizing" above).
+The event timeline above carries the same facts in prose-adjacent form;
+the chart can follow once there's real tuning data to plot against the
+ceiling.
+
 ## Phase 2 — the streaming dispatch loop
 
 Built 2026-07-24, per the owner-approved Phase 2 implementation task for

docs/identification-pipeline.md

@@ -17,6 +17,85 @@ gate's current status; this file describes the mechanics, not the gate.
 **Reviewed and approved by the owner, 2026-07-21.** Written for the
 pre-197k review.
 
+## FIG-1 — Pipeline flow, where a card can stop
+
+```mermaid
+flowchart TD
+    IN["Card named eligible<br/>card-create · evidence-change · cron backstop sweep"]
+
+    IN --> G1{"streaming enabled?"}
+    G1 -- "no" --> X1{{"OFF<br/>STAGE_E_STREAMING_ENABLED = False<br/>no work, no ledger row"}}
+    G1 -- "yes" --> G2{"envelope trip already open?"}
+
+    G2 -- "yes" --> X2{{"HALTED · OPEN TRIP<br/>every dispatch refuses<br/>no self-resume, ever"}}
+    G2 -- "no" --> G3{"fresh signal sample<br/>breaches a bar?"}
+
+    G3 -- "yes" --> X3{{"HALTED · NEW TRIP<br/>EnvelopeTrip row persisted<br/>all dispatch stops"}}
+    G3 -- "no" --> SEL["Select micro-batch<br/>size 25, seed first, backlog fill"]
+
+    SEL --> X4{{"ALREADY PROCESSED<br/>excluded by the anti-join:<br/>evidence carries every manifest key"}}
+    SEL --> G4{"anything eligible?"}
+    G4 -- "no" --> X5{{"EMPTY SELECTION<br/>nothing left to do"}}
+    G4 -- "yes" --> G5{"concurrency slot free?"}
+
+    G5 -- "no" --> X6{{"THROTTLED · CAP<br/>all 2 slots held<br/>deferred to the next sweep<br/>NO ledger row is written"}}
+    G5 -- "yes" --> LED["Open ledger row<br/>PilotRunLedger, status running"]
+
+    LED --> SC["STAGE C · fetch image, extract OCR evidence<br/>one card at a time"]
+    SC -- "fetch fails" --> X7{{"FETCH FAILURE<br/>card skipped, counted,<br/>fed to the rolling 500-card window"}}
+    SC -- "crash" --> X8{{"FAILED DISPATCH<br/>ledger row marked failed<br/>with a triage-able reason"}}
+    SC --> SD["STAGE D · join-key → fallback → slow-path<br/>casts machine votes at weight 0.5"]
+
+    SD --> CONS["Weighted consensus over every vote on the card"]
+    CONS --> G6{"weight ≥ 2<br/>AND share ≥ 0.6<br/>AND at least one human-backed vote"}
+
+    G6 -- "no" --> X9{{"CONSENSUS FLOOR<br/>nearly every card parks here<br/>machine votes alone can never clear it"}}
+    G6 -- "yes" --> RES(["RESOLVED PRINTING<br/>a human vote cleared it"])
+
+    X9 --> WTC["WTC question feed<br/>serves ≥51% of questions from cards<br/>one more human vote would resolve"]
+    WTC --> HV["Human vote cast, weight 1.0"]
+    HV --> CONS
+
+    classDef halt fill:#f7768e,stroke:#8c3d4e,stroke-width:2px,color:#1a1b26
+    classDef defer fill:#e0af68,stroke:#8a6a3d,stroke-width:2px,color:#1a1b26
+    classDef excl fill:#7dcfff,stroke:#3f7f9c,stroke-width:2px,color:#1a1b26
+    classDef work fill:#24283b,stroke:#565f89,stroke-width:1px,color:#c0caf5
+    classDef gate fill:#2f3549,stroke:#ff9e64,stroke-width:2px,color:#c0caf5
+    classDef done fill:#9ece6a,stroke:#5c7c3d,stroke-width:2px,color:#1a1b26
+
+    class X2,X3,X8,X9 halt
+    class X6,X7 defer
+    class X1,X4,X5 excl
+    class IN,SEL,LED,SC,SD,CONS,WTC,HV work
+    class G1,G2,G3,G4,G5,G6 gate
+    class RES done
+```
+
+Shape carries the meaning first: a hexagon is an interception (the card
+stops here); a diamond is a gate being evaluated; a plain rectangle is
+work actually happening; a stadium is a terminal outcome. Colour then
+grades _why_ a hexagon stopped — red is a **hard halt** (a human must
+act), amber is a **soft defer** (the system retries by itself on its
+own), blue is a **correct exclusion** (nothing is wrong, the card just
+doesn't need this pass). Every `classDef` sets both `fill:` and `color:`
+so the diagram reads correctly under either a light or a dark GitHub/wiki
+theme.
+
+**Reading it:** eight of the nine interceptions are cheap and local. The
+ninth — CONSENSUS FLOOR — is where the overwhelming majority of the
+catalog sits by volume, and it is the only one that is not a fault. It is
+the soundness property: no volume of machine votes resolves a printing,
+so the pipeline's throughput is bounded by human attention on purpose.
+The loop back through the WTC question feed is the actual design claim —
+machines narrow the candidate set, humans close it. **Deliberately no
+absolute counts in this diagram** (owner ruling, 2026-07-25, same
+"shape for now, numbers once traffic starts the confirmations" posture as
+FIG-2 below) — see
+[`pipeline-fidelity-gate.md`](pipeline-fidelity-gate.md), the single
+source of truth for gate status, for how many cards are currently sitting
+at each stage. Absolute counts return to this diagram once real user
+confirmations start accumulating in volume.
+
 ## What exists before anything runs
 
 - A **Card row**: name, source drive, and a content phash of the image.

docs/pipeline-fidelity-gate.md

@@ -332,6 +332,56 @@ CLAUDE.md's "Governing premise: we index, we do not store images").
 
 ## 6. Verified data snapshot (2026-07-22, live)
 
+### FIG-2 — Catalog funnel (shape only, no counts)
+
+```mermaid
+flowchart TD
+    A["CATALOG"] --> B["PERCEPTUAL HASH<br/>extracted"]
+    A -.-> A1{{"no phash yet"}}
+
+    B --> C["IMAGE EVIDENCE EXTRACTED<br/>near-complete"]
+    B -.-> B1{{"phash but no evidence yet"}}
+
+    C --> D["CARRIES ≥1 PRINTING VOTE<br/>machine votes abundant"]
+    C -.-> C1{{"evidence but no vote yet<br/>Stage D has not reached these"}}
+
+    D --> E{"human-backed consensus gate<br/>weight ≥ 2 · share ≥ 0.6 · a human said so"}
+
+    E -- "cleared" --> F(["RESOLVED PRINTING<br/>rare — by design"])
+    E -- "held" --> G{{"UNRESOLVED<br/>machine votes alone can never clear this gate"}}
+    E -- "ruled out" --> H(["NO MATCH"])
+
+    G --> I["REVIEW QUEUE<br/>awaiting a human vote"]
+
+    classDef stage fill:#24283b,stroke:#565f89,stroke-width:1px,color:#c0caf5
+    classDef gate fill:#2f3549,stroke:#ff9e64,stroke-width:2px,color:#c0caf5
+    classDef halt fill:#f7768e,stroke:#8c3d4e,stroke-width:2px,color:#1a1b26
+    classDef excl fill:#7dcfff,stroke:#3f7f9c,stroke-width:2px,color:#1a1b26
+    classDef done fill:#9ece6a,stroke:#5c7c3d,stroke-width:2px,color:#1a1b26
+
+    class A,B,C,D,I stage
+    class E gate
+    class G halt
+    class A1,B1,C1 excl
+    class F,H done
+```
+
+**Deliberately shape-only — no absolute counts in this diagram** (owner
+ruling, 2026-07-25): every number behind each row above is real, live,
+and already published in the tables directly below this figure on this
+same page — repeating them inside the diagram itself would fork the
+numbers this page's own rule forbids ("don't restate gate status/
+decisions elsewhere, link here") and would go stale the moment the next
+pass runs. What the shape says instead: extraction (phash → image
+evidence) is functionally complete; Stage D coverage (the "carries a
+vote" row) is the one place more machine capacity still helps; the
+human-backed consensus gate is a **hard** requirement, not a formality —
+machine votes alone can never clear it, so "held" is the gate working as
+designed, not a fault, and the review queue is where held cards wait for
+exactly the one thing that clears them: a human vote. Absolute counts
+return to this diagram once real user confirmations start accumulating
+in volume; until then, read them from the tables immediately below.
+
 **Catalog**: 218,285 cards; 218,270 with a current `ImageEvidence` row.
 
 **Vote pool** (`CardPrintingTag` / `CardTagVote` / `CardArtistVote`,