Context
PARTIAL_MATCH — the class that carries essentially the entire headline win (see the head-to-head rerun issue) — is generated from a tiny hard-coded lookup: 5 beneficiary variant pairs + 5 counterparty variant pairs. The 32 benchmark PARTIAL_MATCH items are drawn from just 9 distinct string pairs (e.g. CHASUS33XXX→CHASUS33), and the same pairs appear in the 134 PARTIAL_MATCH training examples. The champion may have learned "these 9 strings mean PARTIAL_MATCH" — a label convention lookup — rather than the concept "same entity, abbreviated/truncated form". As a benchmark class, it currently cannot distinguish the two.
Evidence
forge/src/reconforge_forge/generator.py:103-133 — BENEFICIARY_VARIANTS (5 pairs), COUNTERPARTY_VARIANTS (5 pairs).- Re-derive: extract the variant pair used by each of the 32 seed-777 PARTIAL_MATCH items and each of the 134 training items; confirm the overlap is total (it is — 9 of the 10 pairs appear in eval, all present in training).
- Champion PARTIAL_MATCH: 26/32 caught; DeepSeek: 5/32 (most DeepSeek "errors" are defensible alternative labels — see the rerun issue).
Task
- Extend the variant tables substantially (dozens of pairs across formats: SWIFT suffix stripping, legal-form abbreviations, diacritics, word-order, truncation) and partition variant pairs disjointly between train and eval pools — the generator gets a
split-aware variant source. This makes PARTIAL_MATCH a generalization test instead of a memorization test.
- Regenerate training data and the benchmark PARTIAL_MATCH slice with held-out variants; re-evaluate the existing champion adapter on the new slice (cheap, local) to measure how much of the 26/32 was memorization vs concept.
- Publish the result either way: "champion holds at ~0.8 on unseen variants" is a strong claim upgrade; "champion drops to DeepSeek's level" is exactly the honest finding the repo's brand is built on.
- Version the dataset (v0.2.0) since task content changes; note in the ReconEval CHANGELOG.
Acceptance criteria
- Train/eval variant pairs provably disjoint (a test asserts it).
- Champion re-scored on held-out-variant PARTIAL_MATCH; number published with interval.
- Dataset version bumped with changelog entry.
Dependencies
- Interprets the head-to-head rerun result (theme/headline-integrity, T2): rubric fixes the label-definition confound; this issue fixes the memorization confound. Both are needed before the PARTIAL_MATCH win means what the blog says it means.
Cost tier
T1-cheap — generator work + local MLX re-eval of one slice (~32–100 tasks); no API cost unless DeepSeek is re-scored on the new slice (fold that into the T2 rerun).
Context
PARTIAL_MATCH — the class that carries essentially the entire headline win (see the head-to-head rerun issue) — is generated from a tiny hard-coded lookup: 5 beneficiary variant pairs + 5 counterparty variant pairs. The 32 benchmark PARTIAL_MATCH items are drawn from just 9 distinct string pairs (e.g.
CHASUS33XXX→CHASUS33), and the same pairs appear in the 134 PARTIAL_MATCH training examples. The champion may have learned "these 9 strings mean PARTIAL_MATCH" — a label convention lookup — rather than the concept "same entity, abbreviated/truncated form". As a benchmark class, it currently cannot distinguish the two.Evidence
forge/src/reconforge_forge/generator.py:103-133—BENEFICIARY_VARIANTS(5 pairs),COUNTERPARTY_VARIANTS(5 pairs).Task
split-aware variant source. This makes PARTIAL_MATCH a generalization test instead of a memorization test.Acceptance criteria
Dependencies
Cost tier
T1-cheap — generator work + local MLX re-eval of one slice (~32–100 tasks); no API cost unless DeepSeek is re-scored on the new slice (fold that into the T2 rerun).