[tool-improvement] `superforcaster-polymarket-v4`: Brier regression on polymarket C-1

[github-actions]

Summary

superforcaster-polymarket-v4 on polymarket shows a Brier regression on the count-window check (calendar windows are below the sample floor for this tool, so the fallback count-window rule applies).

• Current count-window (C-1, last 105 valid rows) Brier: 0.3810, BSS vs base rate: -0.7980
• Previous count-window (C-2, preceding 105 valid rows) Brier: 0.2958 (n=105)
• Delta (C-1 - C-2): +0.0852
• Trigger: regression (count-window fallback)
• Platforms monitored: ['polymarket']; this issue is scoped to polymarket.

This issue records the regression signal, not a diagnosis. The cause has not been identified.

@valory-coding-agent

Windows (count-based -- reproduce by row count, not calendar)

Predicate: rows with tool_name == superforcaster-polymarket-v4 AND platform == polymarket AND prediction_parse_status == "valid" AND final_outcome != null AND p_yes != null AND predicted_at >= now - 45d, deduped by row_id, sorted by predicted_at ascending.

Window	Definition	predicted_at span	n	Brier
C-1 (current)	last 105 rows of the sorted slice	2026-08-07 - 2026-08-24	105	0.3810
C-2 (previous)	preceding 105 rows	2026-07-23 - 2026-08-07	105	0.2958

Calendar context: the 7d calendar windows do not both meet the 105-row sample floor for this tool (current-window valid n: 23), which is why the calendar triage is silent and the count-window fallback fired.

Baseline stats (machine-readable)

{
  "baseline_brier": 0.2244,
  "bias_sum": 1.6369999999999996,
  "brier": 0.3555,
  "brier_large_trade": 0.4401,
  "brier_no_trade": null,
  "brier_skill_score": -0.5842,
  "brier_small_trade": null,
  "brier_sum": 53.32197400000005,
  "brier_sum_large_trade": 48.846374000000004,
  "brier_sum_no_trade": 1.2685000000000002,
  "brier_sum_small_trade": 3.207099999999999,
  "conditional_accuracy_rate": 0.4211,
  "correct_count": 80,
  "decision_worthy": true,
  "directional_accuracy": 0.5333,
  "directional_bias": 0.0213,
  "disagree_n": 133,
  "disagree_tool_win_count": 56,
  "edge": -0.1754,
  "edge_n": 150,
  "edge_positive_count": 66,
  "edge_positive_rate": 0.44,
  "edge_sd": 0.332753,
  "edge_sq_sum": 21.113446503577183,
  "edge_sum": -26.311938250000004,
  "log_loss": 1.1871,
  "log_loss_sum": 178.0693625595503,
  "market_brier": 0.1801,
  "market_brier_sum": 27.01003574999998,
  "n": 150,
  "n_bias_losses": 77,
  "n_directional": 150,
  "n_large_trade": 111,
  "n_no_trade": 17,
  "n_small_trade": 22,
  "no_signal_count": 0,
  "no_signal_rate": 0.0,
  "outcome_yes_count": 51,
  "outcome_yes_rate": 0.34,
  "reliability": 1.0,
  "sharpness": 0.2882,
  "sharpness_sum": 43.228000000000016,
  "valid_n": 150
}

Investigation

Artifact: https://github.com/valory-xyz/mech-predict/actions/runs/32987932601#artifacts

Download with gh run download <run-id> --name benchmark-data. The artifact contains the daily JSONL logs and results/scores_polymarket.json. Reproduce the headline number from the raw rows (predicate above) before forming any hypothesis.

If your investigation concludes that a code change is warranted, before editing anything read the Tool-improvement housekeeping rules section of CLAUDE.md at the repo root. It is the canonical reference for: in-place edit vs new-version spawn decision, naming convention, the tool_lineage.json ledger, and the side-effect file list.

[valory-coding-agent]

Investigation complete -- draft PR opened at #453.

What was found: 10 of the 10 worst-miss deliveries were readable and all showed the same failure pattern (good-evidence/bad-reasoning). Two mechanisms at the same stage:

1. News outlet homepages whose names contain the keyword ("Daily Star", "Kansas City Star") were being counted as TYPE A criterion-confirming evidence that the keyword appeared in a headline, producing p_yes=0.995 for "Will 'Star' be in the headlines?" markets that resolved NO.

2. Even when evidence_reliability_screen correctly identified all-TYPE-B evidence (e.g. a Trump Truth Social post from April on a "this week" question), the text-based base-rate instruction in the field description did not mechanically prevent the model from emitting p_yes=0.98 in the numeric output.

What changed in v5: A single structural change at the prediction-LLM-call stage — a base_rate_required: bool commitment field placed between evidence_reliability_screen and aggregation in PredictionResult. The model must commit True/False before it can write the aggregation or the numeric outputs. When True, aggregation must start from the stated base rate and stay within 10 pp of it in the absence of within-window TYPE A criterion-confirming evidence. The TYPE A definition was also tightened to explicitly exclude outlet-name keyword matches and out-of-window posts.

Gate: PR-CI will run the cached W-2 replay (2026-07-23 to 2026-08-07, n=105, baseline Brier 0.2958) and post the delta. No local benchmark was run.

Human steps after merge: autonomy push-all (IPFS publish), then a PR to agent-deployments to update TOOLS_TO_PACKAGE_HASH.