Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions docs/features/stage-e-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -1014,6 +1014,117 @@ of §3-§5 as a whole):
the resulting ledger data (issue #463) is still an owner-polled action,
gated on the §8 Tron pass.

## `local_calculate_verdicts` observability additions (PR #494)

Two observability additions landed in `local_calculate_verdicts.py`, motivated
by run `20260726T165343-3e8301db` where `counters={}` appeared in the ledger
row because stdout was severed before the terminal summary printed.

### `--diff-report PATH`

Writes one JSONL line per card the run **would act on** (skipped cards
excluded) to the given path. Each line:

```json
{"card_id": "...", "calculator": "join_key|fallback|slow_path", "would_cast": <pk or null>, "existing_votes": [...]}
```

- `existing_votes`: every `CardPrintingTag` vote for the card across all
`anonymous_id` values — the complete prior-vote picture at the time of the
run.
- Stream-written: appended and flushed after each calculator completes, never
buffered in memory.
- The path is **truncated on open** (not appended to); an unwritable path
errors **before** the ledger row is created.
- Passing `--diff-report` lifts the default audit sample cap from 20 to
`sys.maxsize` — use it when you need complete per-card coverage of a batch.

### Per-calculator counters in the ledger row

`PilotRunLedger.counters` now carries a sub-dict per calculator:
`counters["join_key"]`, `counters["fallback"]`, `counters["slow_path"]`. Each
has the same shape:

```json
{
"considered": 0,
"would_cast": 0,
"votes_written": 0,
"already_voted": 0,
"skip_counts": {}
}
```

For `slow_path`, `would_cast`/`votes_written` count its **routed**
`CardScanLog` rows (the slow-path calculator casts no `CardPrintingTag` votes);
`skip_counts` is the `reason_counts` breakdown from that calculator.

Counters are persisted to the ledger row **before** the terminal summary is
printed (counters-before-output discipline), so a severed stdout no longer
produces an empty `counters={}` record.

## `rejudge_fallback_channel` — fallback-channel compare-and-retract (PR #495)

`MPCAutofill/cardpicker/management/commands/rejudge_fallback_channel.py` re-evaluates the
`stage-d-fallback-v1` channel's existing conclusions against **current stored**
`ImageEvidence` and retracts the rows where the conclusion changed, making
those cards eligible for a fresh `local_calculate_verdicts` pass.

### What it does

For each card in the fallback channel, the command re-derives the fallback
calculator's conclusion from the card's current `ImageEvidence`
(`layout_class`, `artist_ocr_name`, `symbol_phash`) — zero image fetches, zero
re-parse. If the conclusion differs from the stored vote or skip, it
**retracts** (deletes) the `CardPrintingTag` vote and any `CardScanLog` skip
rows for that card.

The command **never** touches `stage-d-join-key-v1` rows. Fallback eligibility
is gated on a join-key no-hit; retraction removes only the fallback-channel
layer, leaving the join-key layer untouched.

### Safety gate

A card is **never** retracted if `resolve_printing(card) is not None`. This
covers both a resolved printing and a resolved NO_MATCH consensus. Gated cards
are counted and their PKs are recorded in `counters["gate_refused_card_ids"]`
for human review before any manual intervention.

### Flags

| Flag | Notes |
| -------------------------- | ------------------------------------------------- |
| `--selector all-channel` | Operate on the entire fallback-channel population |
| `--card-ids-file PATH` | Operate on a newline-delimited list of card IDs |
| `--write` | Required to retract (dry-run by default) |
| `--skip-dryrun-check` | Skip the dry-run-window guard |
| `--dry-run-window-hours N` | Override the dry-run-window length |

`--selector` and `--card-ids-file` are mutually exclusive; exactly one is
required.

### Counters and ledger row

`PilotRunLedger.votes_written` is repurposed to record the **retracted** count
(same convention as `reparse_collector_evidence`). Full counter set:

```
considered, no_evidence, no_prior_fallback_state, unchanged,
changed, retracted, gate_refused, transitions
```

`transitions` is a string-keyed breakdown of conclusion changes, e.g.
`"skip:ambiguous -> vote:12345": 42`.

### Runbook

Two-step sequence to bring retracted cards back into the active population:

1. `python manage.py rejudge_fallback_channel --selector all-channel` (dry-run
→ review counters and `gate_refused_card_ids`) → rerun with `--write`
2. `python manage.py local_calculate_verdicts` (unchanged) to fill the
retracted cards through the normal calculator chain.

## See also

- [`docs/proposals/stage-e-streaming.md`](../proposals/stage-e-streaming.md)
Expand Down
12 changes: 12 additions & 0 deletions docs/infrastructure.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,18 @@ directly.
`coverage-ack: <test id or glob> — <reason>` line (same tether discipline
as `docs_lint.py`'s in-file `ALLOWLIST`). New tests and un-skipping are
always fine.
- `web-ci.yml` per-surface change gating (PR #466): a `changes` job runs
first on every push and diffs `HEAD` against the push's pre-image SHA using
plain `git diff --name-only` (same pattern as other workflows in this repo,
not a paths-filter action). It sets two boolean outputs — `backend`
(`MPCAutofill/**` prefix) and `frontend` (`frontend/**` prefix) — and every
downstream job gates itself on `needs.changes.outputs.<surface> == 'true'`.
`cloudflare-static-site/**` is not wired to any surface. Fallback: on
`workflow_dispatch`, a new branch push, or a force-push where no before SHA
is available, both outputs are set to `true` (run everything). A
`test-pre-commit` job that had been duplicated in `web-ci.yml` was removed
in this PR rather than scoped — `test-pre-commit.yml` already runs it
unconditionally.

## Push policy

Expand Down
36 changes: 20 additions & 16 deletions docs/pipeline-fidelity-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,24 +435,28 @@ one flattened figure:
| `20260724T001154-d3986cfc` (post-fire calculator re-pass write) | **1** | **§14 "Five further passes" write** — `local_calculate_verdicts` re-run over the lexicon-gate-retracted pool; 52,348 `unknown-set-code` + 16 `no-evidence` skips, 117,442 `to-review` — see §14 |

**Stage C run history** (`ImageEvidence.run_id`, current last-writer row
count per run — canaries first, then the main leg):

| run_id | date | rows |
| ------------------------------------------ | ------------- | ------: |
| `stagec-canary-20260720T1659Z` | 2026-07-20 | 223 |
| `stagec-canary-decoupled-20260720T235127Z` | 2026-07-20 | 206 |
| `ntx-0721` | 2026-07-20/21 | 9,675 |
| `stagec-20k-20260721T0227Z` | 2026-07-20/21 | 10,696 |
| `stagec-remainder-0721` | 2026-07-21/22 | 197,470 |

Full chain: the two decoupling canaries (Jul 20) validated the
fetch/compute-decoupled architecture at small scale, then `ntx-0721` +
`stagec-20k-20260721T0227Z` (Jul 20–21) ran a combined ~20k-card
extraction pass, then `stagec-remainder-0721` (Jul 21–22, 197,470-row
main leg) extracted the bulk of the remaining catalog. See
count per run — updated 2026-07-27 after the full-catalog re-extraction):

| run_id | date | rows |
| --------------------------------------- | ------------- | ------: |
| `pass-pilot-20260725` | 2026-07-25 | 100 |
| `stage-e-stream-20260725T233633221123Z` | 2026-07-25 | 22 |
| `stage-e-stream-20260725T233633687349Z` | 2026-07-25 | 23 |
| `pass-full-20260725` | 2026-07-25/26 | 194,831 |
| `pass-full-20260725-r2` | 2026-07-25/26 | 23,132 |
| **Total** | | 218,108 |

The 2026-07-25 full-catalog re-extraction (`pass-full-20260725` +
`pass-full-20260725-r2`) superseded all prior Stage C last-writer rows —
the original canaries (`stagec-canary-20260720T1659Z`,
`stagec-canary-decoupled-20260720T235127Z`), `ntx-0721`,
`stagec-20k-20260721T0227Z`, and `stagec-remainder-0721` no longer appear
as last-writer for any `ImageEvidence` row. See
[`reports/2026-07-20-decoupled-canary-confirm.md`](reports/2026-07-20-decoupled-canary-confirm.md)
and [`reports/2026-07-21-stagec-20k-extraction.md`](reports/2026-07-21-stagec-20k-extraction.md)
for the canary/20k narrative detail. None of these Stage C runs have a
for the original canary/20k narrative detail, and
[`reports/2026-07-26-stagec-full-catalog-completion.md`](reports/2026-07-26-stagec-full-catalog-completion.md)
for the full-catalog completion record. None of the Stage C runs have a
`PilotRunLedger` row of their own — see §11 for what that does and
doesn't affect.

Expand Down
92 changes: 92 additions & 0 deletions docs/reports/2026-07-26-stagec-full-catalog-completion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Stage C full-catalog extraction — completion record (2026-07-26)

Full re-extraction of the entire card catalog under the decoupled
fetch/compute architecture, superseding all prior Stage C runs (canaries,
20k cohort, and the Jul 21–22 remainder leg). Preceded by
[`2026-07-20-decoupled-canary-confirm.md`](2026-07-20-decoupled-canary-confirm.md)
and
[`2026-07-21-stagec-20k-extraction.md`](2026-07-21-stagec-20k-extraction.md).

This session did not execute the extraction — it records the result against
the live DB (`mpcautofill_django`) as of 2026-07-27. All figures below come
from `SELECT`/`count()` queries; no writes were made.

## Run parameters

| run_id | last write | rows (last-writer) |
| --------------------------------------- | ---------- | -----------------: |
| `pass-pilot-20260725` | 2026-07-25 | 100 |
| `stage-e-stream-20260725T233633221123Z` | 2026-07-25 | 22 |
| `stage-e-stream-20260725T233633687349Z` | 2026-07-25 | 23 |
| `pass-full-20260725` | 2026-07-26 | 194,831 |
| `pass-full-20260725-r2` | 2026-07-26 | 23,132 |
| **Total** | | **218,108** |

A 100-card pilot (`pass-pilot-20260725`) preceded the main legs. Two
`stage-e-stream-*` entries represent Stage E streaming activity on 45 cards
total. The two main legs (`pass-full-20260725` + `-r2`) together cover
217,963 cards.

The `pass-full-20260725` pair is a **full re-extraction**: every prior
last-writer row (`stagec-remainder-0721`, `stagec-20k-20260721T0227Z`,
`ntx-0721`, and the two Jul-20 canaries) has been replaced — none of those
run_ids appear as last-writer for any `ImageEvidence` row in the current
DB. The Stage C run history table in
[`../pipeline-fidelity-gate.md`](../pipeline-fidelity-gate.md) §6 reflects
this updated state.

## Coverage

- Total cards in DB: **218,516**
- Cards with any `ImageEvidence` row: **218,108** (99.8%)
- Cards with no `ImageEvidence` row: **408** (0.2%) — all `GOOGLE_DRIVE`-sourced

Of the 218,108 cards with evidence:

- `fetch_ok=True`: 216,494 (99.3% of those with evidence — successfully extracted)
- `fetch_ok=False`: 1,614 (0.7% — fetch attempted, image unavailable, evidence row
written with failure recorded)

The 408 cards with no evidence at all were never successfully started during any
pass; all are `GOOGLE_DRIVE`-sourced.

## Fetch-failure rate vs. prior runs

| run | cards | fetch_ok=False | rate |
| ------------------------------------------------------------- | ----------: | -------------: | --------: |
| Bundled canary (`stagec-canary-20260720T1659Z`) | 400 | 0 | 0.0% |
| Decoupled canary (`stagec-canary-decoupled-20260720T235127Z`) | 400 | 6 | 1.5% |
| 20k cohort (`stagec-20k-20260721T0227Z`) | 20,000 | 105 | 0.53% |
| **Full catalog (`pass-full-20260725` + r2)** | **218,108** | **1,614** | **0.74%** |

The full-catalog failure rate (0.74%) is higher than the 20k cohort's 0.525%.
This is consistent with broader catalog coverage: the 20k cohort used
edhrec_rank-ascending ordering (popular cards first, cold tail deferred), while
the full pass covered the entire cold tail — where per-file breakage across
independent Drive folders is more prevalent.

## No-pixels invariant

Not re-checked in this session — the schema inspection in
[`2026-07-21-stagec-20k-extraction.md`](2026-07-21-stagec-20k-extraction.md)
applies unchanged. `ImageEvidence` holds hashes, measurements, classification
strings, OCR text, and pixel coordinates; no field is capable of holding image
bytes.

## Stage completion

Stage C extraction is at **99.8% catalog coverage** (218,108 / 218,516).
The original compute profile
([`2026-07-20-pipeline-compute-profile.md`](2026-07-20-pipeline-compute-profile.md))
projected 116.2h single-threaded against a 6.2h reference budget (BLOCKING).
The decoupled fetch/compute architecture resolved that gap; this record
confirms the full catalog has now been extracted under that architecture in
production.

## Open items

1. The 408 cards with no `ImageEvidence` are unreachable by Stage D unless a
targeted re-extraction pass succeeds for them. No such pass is scheduled;
they are expected to remain unresolved unless their Drive sources are
repaired or re-indexed. All are Google Drive — no other source type
contributes to the gap.
5 changes: 5 additions & 0 deletions docs/reports/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,11 @@ a `detail` object read only when the summary signals it's needed.
`stage-d-join-key-v1` totals reconciled to 11,905, gate re-derived at
0/12,684 across both the printing- and tag-consensus engines this arc
touched.
- `2026-07-26-stagec-full-catalog-completion.md` — Stage C full-catalog
extraction completion: 218,108 / 218,516 cards (99.8%), run_ids
`pass-full-20260725` + `-r2` superseding all prior Stage C last-writer rows,
1,614 fetch_ok=False rows + 408 cards with no evidence (all Google Drive),
full-catalog fetch-failure rate 0.74%.
- `2026-07-22-knowledge-inventory.md` — pipeline-fidelity gate artifact 2
(knowledge-inventory sweep): a constant-by-constant table of every
pilot-era value's current home, 3 confirmed MISSING items, 3 open
Expand Down
Loading