feat(superforcaster_full_search-v1): criterion-specificity screen + structured outputs for overconfident-YES fix

[valory-coding-agent]

What happened and what we did

The tool was producing overconfident YES predictions (Brier C-1=0.3621, above the 0.25 chronic-bad threshold) on Polymarket narrow-criterion questions. IPFS delivery inspection confirmed the root cause: the model treats topical relevance as criterion-satisfaction confirmation — it sees a page about "Trump tweets" and gives p_yes=0.90 for "Will Trump say 'Radical Left' during this specific rally?" without checking whether any evidence directly confirms the exact condition.

This is the same criterion-specificity failure pattern previously fixed for superforcaster-polymarket-v2 (issue #284) and then more rigorously with the 4a-4d evidence-reliability screen in superforcaster-polymarket-v4 (#374/PR#375). This PR ports that screen to the full_search variant.

What to do next: Review the diff, run autonomy push-all to publish the CID to IPFS, then route tournament traffic to superforcaster_full_search-v1 for evaluation.

---

Investigation summary

Tool: superforcaster_full_search | Platform: polymarket | Issue: #440

Headline (reproduced):

• C-1 Brier: 0.3621 (n=105, 45-day window ending 2026-08-13)
• C-2 Brier: 0.2835 (n=105, prior window)
• Delta: +0.0786 | Trigger: count_window

Data-sufficiency: Both windows n=105 = VALID_N_PER_WINDOW_FLOOR. Single CID both windows — mix-shift ruled out. Within-version Brier genuinely rose.

Calibration profile: YES bias +0.2110 (avg p_yes=0.711, avg outcome=0.500). High-confidence bin [0.9-1.0): realized rate only 29.4% (n=17) — massive overconfidence. Average edge = -0.1695.

IPFS delivery inspection (8 worst-miss rows): 6/8 confirmed good-evidence/bad-reasoning. Dominant pattern: tool reads topically relevant scraped pages and assigns high p_yes without checking whether any TYPE A evidence (dated within the resolution window, directly confirming the criterion) actually supports the narrow condition. Examples:

• "Will Trump say 'Radical Left' during Tele-Rally?" — evidence: past rally transcripts with the phrase. Tool: p_yes=0.83. Outcome: NO.
• "Will 'Woman' appear in headlines this week?" — evidence: BBC Women's pages (standing pages). Tool: p_yes=high. Outcome: NO.
• "Will 'Blue' be said on Lemonade Stand Podcast?" — evidence: Polymarket shows 78% market odds. Tool: p_yes=0.13 (ignored odds filter!). Outcome: YES.

---

Mechanism and fix

Stage: prediction-LLM-call (gate-visible — runs on injected source_content in PR-CI's cached replay).

Mechanism: The original 7-step PREDICTION_PROMPT has no instruction to differentiate TYPE A evidence (criterion-confirming) from TYPE B evidence (topically relevant but historically dated or from standing pages). The model defaults to "topic is active = YES likely."

Fix: superforcaster_full_search-v1 adds the evidence_reliability_screen structured-output field (mirroring superforcaster-polymarket-v4's step 4):

• (4a) Prediction-market-odds filter: discard circular self-referential trading prices as evidence.
• (4b) Forward-looking-intent discount: treat intent/expectation language as 40-60% materialization probability.
• (4c) TYPE A/B temporal classification: classify each source; if ALL are TYPE B, anchor on category base rate (20-40% for "X in headlines this week"-style markets).
• (4d) Criterion-specificity check: does any TYPE A evidence directly confirm the exact resolution condition? If not, add uncertainty toward base rate.

The fix is mechanical (an explicit reasoning step the model must emit and fill before producing p_yes), not aspirational ("be better calibrated").

Additionally, the tool now uses OpenAI Structured Outputs (client.beta.chat.completions.parse + PredictionResult Pydantic schema) instead of the raw chat.completions.create + prose JSON instruction. This satisfies Hard Constraint 0 and prevents the prose-leaking output contract violation that broke superforcaster-polymarket-v4 in production.

Page-scraping (readability + markdownify + ThreadPoolExecutor) is preserved — it is the distinguishing feature of the full_search family.

---

Files changed

File	Action
packages/valory/customs/superforcaster_full_search_v1/superforcaster_full_search_v1.py	NEW — v1 implementation
packages/valory/customs/superforcaster_full_search_v1/component.yaml	NEW — fingerprinted by lock
packages/valory/customs/superforcaster_full_search_v1/__init__.py	NEW — copied from parent
packages/valory/customs/superforcaster_full_search_v1/tests/test_superforcaster_full_search_v1.py	NEW — 12 tests (Hard Constraint 0 assertions included)
packages/packages.json	CID added by lock
benchmark/tools.py	ADD superforcaster_full_search-v1 ToolSpec entry
benchmark/tournament_tools.json	ADD superforcaster_full_search-v1 -> CID
tool_lineage.json	ADD lineage entry

Parent (superforcaster_full_search) is NOT in tournament_tools.json — no roster removal needed.

---

Pre-PR sanity attestation

• ASCII-only source: PASS
• isort + black + pylint (10.00/10) + mypy + flake8 (tomte config): all pass
• autonomy packages lock --check: Verification successful
• Step 6.5 trader parse: json.loads(run()[0]) -> {'p_yes': 0.35, 'p_no': 0.65, 'confidence': 0.7, 'info_utility': 0.5} — exactly 4 numeric keys, no reasoning fields
• 12/12 unit tests pass (including test_on_chain_result_is_flat_json_loads_parseable, test_uses_structured_parse_not_raw_create, test_numeric_fields_are_declared_last)
• Source-change size (pre-lint): ~118 LOC net new (803 total, 685 parent) — under 150 soft / 300 hard backstop
• One mechanism: evidence-reliability screen before probability formation
• autonomy push-all NOT run — human reviewer runs this before merging

---

Baseline to beat

• C-1 Brier: 0.3621 (n=105, polymarket)
• C-2 Brier: 0.2835 (n=105, polymarket)
• BRIER_LEVEL_THRESHOLD: 0.25 (chronic-bad; tool is above it in both windows)
• Expected direction: Brier decrease; targeted fingerprint: overconfident-wrong rate in the [0.7-1.0) p_yes bin should fall materially (currently: bin [0.9-1.0) realized rate = 29.4%)

PR-CI will run the re-benchmark on a disjoint slice and post the delta-vs-baseline comment.

[valory-coding-agent]

[human-input] — investigation memory file missing; cannot complete Sub-pipeline D automatically.

Gap: mech-predict/.claude/tool-improvements/superforcaster_full_search.md does not exist on this branch or on main. Step 1 of the follow-up pipeline requires this file to restore the investigation context; without it the pipeline cannot safely proceed.

The PR body contains the full investigation detail, so a human can unblock this in one of two ways:

Option A — post the benchmark command manually (fastest):

Post these two comments on the PR in order:

1. The trigger (must be exactly one line, nothing else):

/benchmark superforcaster_full_search --candidate-tool superforcaster_full_search-v1 --platform polymarket --sample 300

2. The metadata record (so the agent can track the budget):

**valory-coding-agent metadata** (request record -- do not edit)

```json
{
  "baseline": "valory/superforcaster_full_search:0.1.0:bafybeihba6ch6g7gndfwv75dtpfrpnvg4dw6wq6lxexch5h3n3vjafuxsm",
  "candidate": "valory/superforcaster_full_search_v1:0.1.0:bafybeifpypcpotnfva432kogovkr47wubkbudu4xgz32pfko66wxwdkzce",
  "platform": "polymarket",
  "sample": 300,
  "seed": 42,
  "run_type": "iteration",
  "pr_head": "190c680",
  "reason": "first benchmark after CI green; criterion-specificity screen fix for overconfident-YES"
}

**Option B — create the memory file** so the agent can auto-trigger:

Commit `mech-predict/.claude/tool-improvements/superforcaster_full_search.md` on this branch with at least a PR section entry (see other files in that directory for the format). Push to this branch; when CI re-runs green the agent will pick it up.

---

**Additional note:** The PR title uses `feat(...)` rather than the expected `tool-improvement(...)` convention, which also prevents tool-name resolution from the title. Both the title pattern and the missing memory file should be addressed in future investigation runs.