What's That: data-driven question-selection policy (expected information gain per question)

[WilfordGrimley]

Part of #704.

Purpose

The owner has approved a pragmatic-first authored ranking for question selection in the near term, but wants the DATA-DRIVEN spec recorded on the board rather than lost if it is deferred. This issue is that record, not an implementation request.

The policy

Select the question with the highest expected information gain per unit of user effort, given:

1. what is already derivable from stored evidence,
2. what the machine abstained on, and
3. what this specific voter has already answered.

Concrete inputs on the board today

• Frame-style and bleed-edge chips have zero rows: 145,419 votes are derivable from stored evidence with no image fetch #619 — 145,419 votes are derivable from stored ImageEvidence with NO image fetch required: Modern Border (133,627, via collector_line_collector_number non-empty), Old Border (9,006, via illus_anchor_fired true plus empty collector number), appropriate-bleed (2,786, via bleed_class="trimmed"). Cross-check: 142,469 stored vs 142,633 derived against artbox_frame_class — the 164-row gap is artbox_phash fetch failures. Blocked ONLY on an owner write-poll (145,419 production rows), not on engineering.
• Only 1 of 21 vote channels is reachable from the pooled runner, and it casts no votes: the full run's Stage D route is NOT ESTABLISHED #618 — only 1 of 21 vote channels is reachable from the pooled cohort runner (run_image_evidence_cohort is Stage-C-only and invokes no calculator). Its standing ruling: "a zero-row channel is evidence we never ran it, not evidence it is unnecessary." Absent votes must NOT be read as absent signal — this directly bears on how "what the machine abstained on" (input 2 above) should be computed: a channel with zero votes because it was never run is a different case from a channel that ran and produced no signal.
• ImageEvidence field-level census: 'gap 0' counts extractor keys, not fields written (bleed_diff_mm 2.1%, artbox group 64.6%, artist_ocr_name 6.3%) #637 — ImageEvidence field-level census (informs which fields are actually populated and therefore which questions are already derivable).
• A zero-weight vote still buys the outcome: the question feed suggests the frozen control cohort's answer on 8,207 cards #593 — a zero-weight vote still buys the outcome; an existing feed-selection bug in the same tier-ordering/selection logic this policy would replace. Any implementation of this policy needs to be aware of A zero-weight vote still buys the outcome: the question feed suggests the frozen control cohort's answer on 8,207 cards #593's findings about question_feed.py's tier-1 "lowest pk" .first() selection and unweighted DEDUCTION/OCR vote handling.
• What's That Card / question feed: alignment pass for deduction vote sources (artist + illustration) #503 — the remaining deduction-coverage scope (question selection not considering deduction coverage) is a concrete instance of input 1 above (what is already derivable from stored evidence).
• local_illustration.py abstains whenever a card has more than one illustration. Those cards are exactly where human input is worth the most, and today they are indistinguishable in the feed from high-confidence resolved cards — a data-driven policy would surface them preferentially; today's feed does not.

Owner framing (restated for this issue)

Resolution need not be a printing — confirming machine-derived data on other high-value fields counts equally as a resolution. Difficulty may be user-selected or ramped per session; the working default (per #704's 2026-08-05 owner-decisions section) is session-ramp, pending a final decision.

Referenced issues

#619, #618, #637, #593, #503 — referenced for context above; none of them are modified by this issue.

Scope

Record only. No implementation, estimate, or design commitment is made here — this exists so the data-driven spec is not lost if the pragmatic-first authored ranking ships instead in the near term.

Corrections and owner decisions (2026-08-05)

1. CONSENSUS GATE IS 60%, NOT 51%. settings.PRINTING_TAG_MIN_SHARE = 0.6 plus settings.PRINTING_TAG_MIN_VOTES = 2 (MPCAutofill/settings.py:65-66), enforced in vote_consensus.resolve_weighted_consensus (vote_consensus.py:407-536, the share check at :530-532). The "51%" belongs to a separate SERVING policy: settings.QUESTION_FEED_LIKELY_RESOLVE_MIX_RATIO (default 0.51, question_feed.py:29-30, checked at question_feed.py:670) governs what fraction of a session's served questions come from the "one more human vote would resolve" pool — what to show a user, never whether consensus resolves.

2. MACHINE VOTES CANNOT RESOLVE ALONE, confirmed in code. _MACHINE_DERIVED_SOURCES = {DEDUCTION, OCR, FEDERATED, IMPLICIT} (vote_consensus.py:52); is_human_backed_source (:55-56) is the inverse; resolve_weighted_consensus requires winner["has_human_backed"] unconditionally (:532) — a USER or ADMIN vote in the winning group, gated independently of the weight/share math. Weights (vote_consensus.py:15-29, defaults in MPCAutofill/settings.py): ADMIN settings.PRINTING_TAG_ADMIN_WEIGHT = 5.0 (:67), DEDUCTION/OCR (machine) settings.PRINTING_TAG_MACHINE_WEIGHT = 0.5 (:68), FEDERATED settings.VOTE_FEDERATED_WEIGHT = 1.0 (:76), IMPLICIT settings.PRINTING_TAG_IMPLICIT_WEIGHT = 0.25 (:81), capped in aggregate per outcome-group at settings.PRINTING_TAG_IMPLICIT_CAP = 1.0 (:87).

3. DIFFICULTY LANES DERIVE FROM DATA STATE, not user profiling. Per the owner framing this issue already restates: easy = confirm an uncontested machine vote (_confirm_suggestion_item, the 45,154-of-46,310 single-candidate case per this issue's own 2026-08-05 line 20 data); medium = break a tie between competing votes; hard = answer cold (no printing-tag votes at all — the population is_likely_resolve_printing returns False for, per its own docstring at question_feed.py:355-357). Ramps per session by default, with a user override, matching this issue's own "difficulty may be user-selected or ramped per session" line.

4. NO NEW USER-ACTION TELEMETRY (owner ruling) — retires docs/proposals/mockups/wtc-rebuild/SPEC-wtc-rebuild.md's "ANNEX A — served-mix logging seam (backend)" (SPEC-wtc-rebuild.md:315) as a build item, recorded in PR spec(wtc-rebuild): fold 2026-08-05 owner decisions #718. The difficulty-lane model in item 3 above was chosen partly because it needs none — it reads existing vote/evidence state, not served-question history.

5. THE LANE CLASSIFIER MUST READ EVIDENCE COLUMNS, NOT VOTE PRESENCE. This follows the magnitude/exception split (see the Only 1 of 21 vote channels is reachable from the pooled runner, and it casts no votes: the full run's Stage D route is NOT ESTABLISHED #618 zero-row ruling's Corollary 2, now in docs/identification-pipeline.md, PR docs: rehome the #618 zero-row ruling, with corrected corollaries #720) — it is not a defect in any channel. For a channel that surfaces only the exception (bleed-edge: only a confidently 'trimmed' reading votes), the quantitative answer lives in the evidence column, not the vote. A card with no appropriate-bleed vote is NOT unknown: ImageEvidence.bleed_diff_mm holds the measurement directly. Classifying a card as "cold" purely from absent votes would wrongly bucket a knowable card. bleed_diff_mm coverage (compute_bleed_diff_mm, image_evidence.py:1053) is the live variable here: issue ImageEvidence field-level census: 'gap 0' counts extractor keys, not fields written (bleed_diff_mm 2.1%, artbox group 64.6%, artist_ocr_name 6.3%) #637 measured 2.1% populated; read-only query today (2026-08-06) against the in-progress full-catalog pass: 230,320 of 230,373 ImageEvidence rows populated — 99.98%, up sharply as the running pass fills it in.

6. QUESTION SUPPLY as measured today (read-only production query, 2026-08-06, CardTagVote grouped by anonymous_id and by tag__name, for the five channels this issue's referenced issues (Only 1 of 21 vote channels is reachable from the pooled runner, and it casts no votes: the full run's Stage D route is NOT ESTABLISHED #618/Frame-style and bleed-edge chips have zero rows: 145,419 votes are derivable from stored evidence with no image fetch #619) discuss):

   By identity:

   Identity	Rows
   frame-style-cast-v1	151,161
   bleed-edge-cast-v1	2,786
   layout-class-cast-v1	228,914
   ai-art-detector-v1	1,183
   local-fallback-v1	0

   By tag (all sources, including any human/other votes on the same tag):

   Tag	Rows
   Modern Border	141,808
   Old Border	9,362
   Black Border	143,662
   Borderless	77,370
   White Border	7,479
   Silver Border	417
   appropriate-bleed	2,786
   AI-Generated	1,185

   local-fallback-v1 casts nothing today (superseded — see Frame-style and bleed-edge chips have zero rows: 145,419 votes are derivable from stored evidence with no image fetch #619, already closed as shipped). Every one of the identity-scoped totals above is a lower bound on true question supply for its tag, since a small number of non-machine votes also exist on several of these tags (the by-tag vs. by-identity deltas above).

Feed structure: page one should carry the best questions

Owner, verbatim (walkthrough of /whatsthat, 2026-08-06): "Our first question page asks redudant questions and filters users down to a couple of other pages when i would propably prefer we ask the best questions on the first page (art matches, but... is a great lead, probably better than no diff printing."

This is the STRUCTURAL half of that complaint — distinct from this issue's SELECTION-ranking scope (which question is worth asking, ranked by expected information gain). This section records what the current page-one structure IS, in code, and what "redundant" concretely refers to.

Stage map (frontend/src/features/questionFeed/QuestionFeed.tsx)

One fetched QuestionFeedItem carries a type of confirm_suggestion, identify_printing, artist, or tag (MPCAutofill/cardpicker/schema_types.py:464-472). Only the first two (isCandidateType, QuestionFeed.tsx:1121-1122) go through the internal CandidateStage funnel below; artist/tag items bypass it entirely (:1735 else branch) and render as a single flat question regardless of stage.

• Level 1 — type CandidateStage = "level1" | "level2" | "level3" (:99); entered via initialStage() (:648-651) ONLY when the item is confirm_suggestion AND carries a suggestedPrinting — i.e. the machine already has one strong candidate. Renders at :1284 (stage === "level1" && item.suggestedPrinting != null): single suggested printing, "Is it this one?" (:1325), four actions — Yes (:1330-1340, casts the vote via selectCandidate), Not sure (:1342-1349, onClick={() => setStage("level2")} — explicit transition), No/different printing (:1350-1357, onClick={rejectSuggestion}), Skip (:1358-1365, advance() — fetches an unrelated new item, not a stage transition).
• Transition Level1→Level2: rejectSuggestion() (:1060-1076) always ends in setStage("level2") (:1062 guard branch, :1069 main branch), after recording the rejected candidate id client-side.
• Level 2 — the else branch at :1434-1734 (i.e. stage !== "level1"-with-suggestion and stage !== "level3"). Renders for every identify_printing item (no suggestion, initialStage returns "level2" at :651) AND for any confirm_suggestion item after Level 1's "Not sure"/"No". Shows a full candidate grid (allCandidates, :1123) — critically, the SAME suggested candidate that was just shown alone at Level 1 reappears here as one grid tile, marked "highlighted" (:1467-1471: item.type === "confirm_suggestion" && item.suggestedPrinting?.identifier === candidate.identifier), under the prompt "Which of these is it? … was suggested" (:1566-1571). Bottom-row fallback actions when nothing is picked: "None of these" (:1690-1698, selectCandidate(undefined, true)), "🎨 Art matches, not an official printing" (:1699-1710, classifyAsCustomArt), Skip (:1711-1718).
• Transition Level(1|2)→Level3: inside selectCandidate's success handler (:872-909), after a printing vote is cast, IF the picked candidate leaves an exclusion group open (getOpenExclusionGroups(candidate).length > 0, :893-894) it sets level3ChipStates and setStage("level3") (:902); otherwise it advances straight to the next item (:903-906).
• Level 3 — renders at :1377-1433 (stage === "level3"): tri-state attribute chips (border colour / frame style groups) for whatever the just-picked candidate left ambiguous. Reachable ONLY downstream of a Level 1 or Level 2 candidate selection — never a page-one destination.

What "redundant" concretely means here

The code supports two specific readings, not a paraphrase:

1. The same candidate is asked about twice. Level 1 asks "is it this one?" about exactly one candidate, alone. If the user answers "Not sure" or "No", Level 2 re-presents that identical candidate — unchanged, merely "highlighted" — inside a grid of others (:1467-1471), under a prompt that again asks the user to judge it ("Which of these is it? … was suggested", :1566-1571). The user is asked to evaluate the same printing candidate on two separate screens rather than once.
2. A high-value resolution is buried downstream instead of offered on page one. The owner's own example — "art matches, but…is a great lead" — maps to the "🎨 Art matches, not an official printing" button (:1699-1710). This is only reachable after a confirm_suggestion item runs through Level 1 (is-it-this-one, gated on having a suggestedPrinting) and arrives at Level 2's fallback row — it is never offered as a Level 1 option, even though it is a valid, distinct, immediately-answerable resolution for exactly the case Level 1 is built around (a suggested candidate the user is looking at). "no diff printing" in the owner's quote maps to Level 1's "No, different printing" (:1350-1357) / Level 2's "None of these" (:1690-1698) — options that are offered ahead of "art matches" in the funnel today, despite the owner's stated preference that "art matches" is the better lead.

No other structural reading is supported by the code — e.g. there is no candidate-list overlap between distinct QuestionFeedItem fetches (each fetch is a fresh item; the redundancy above is entirely internal to one item's Level1→Level2 flow), and artist/tag items never enter this stage machine at all, so they are a separate axis of "which page comes first," not additional evidence of internal redundancy.

---

This is the STRUCTURAL complaint (page-one flow shape and what's re-asked/deferred), separate from this issue's SELECTION-ranking scope (which single question has the highest expected information gain). Recorded per owner walkthrough, 2026-08-06.