fix: port the issue-455 free-text contract fix to 9 sibling tools (lockstep)

[jmoreira-valory]

Summary

Ports the merged issue #455 fix (PRs #459 + #461, already live in superforcaster_polymarket_v4 and superforcaster_market_aware) into the 9 sibling tools that still have the exact same bug shape: a trader-template-only extract_question regex whose miss-path sends the raw prompt verbatim as the web-search query, so every non-trader caller (Pearl Connect free text) gets degraded or empty retrieval — silently.

Minimal, mechanical, lockstep port per tool — no behavior change on the trader-template path (byte-identical question + query), no template/dispatch/model/cost changes:

1. parse_prompt(prompt) -> ParsedPrompt(question, query, tier) — the frozen parse block, inserted byte-identical in every tool (the mirror now spans 11 files; verified programmatically, 8313 chars each).
2. _flagged_null_result — organic AND peopleAlsoAsk both empty → p_yes/p_no 0.5/0.5, confidence 0, with empty_retrieval / null_reason / parse_tier / scan_truncated in used_params (off-chain observability; on-chain JSON shape unchanged).
3. Unsearchable-query short-circuit — a derived query with no alphanumeric characters returns the flagged null with zero network calls.
4. Observability — scan_truncated (template tier exempt) + the single-log tier chain, same semantics as fix: bound the candidate scan + reject unsearchable queries (v4 + market-aware, lockstep) #461.

In scope (code-verified inventory, every package re-read on this branch)

Tool	Production	Notes
superforcaster	🔴 8 envs (5 Gnosis predict + clone + nvm_2469 + base_mech)	widest deployment; had NO empty-retrieval guard
superforcaster_calibrated_full_search	🔴 5 Gnosis envs	guard added before the page-scrape layer
superforcaster_full_search	🔴 3 Polygon envs (21/25/44)	guard added before the page-scrape layer
superforcaster-polymarket-v1, -v2	🔴 3 Polygon envs each	textbook pre-fix v4 shape
superforcaster-polymarket-v3	repo-only	ported with the batch
finetuned_prediction (predict-base, predict-fine-tuned, predict-fine-tuned-calibrated)	🟡 tournament	tournament pins re-pinned to the new CID
prediction_request_rag_v1 (+claude variant)	repo-only	raw prompt was a DIRECT search query on every run (not just fallback); LLM-brainstormed queries untouched
prediction_request_reasoning_v1 (+claude variant)	repo-only	same as rag; parsed.query only at the direct-leak sites

Deliberately out of scope (different fix shape; follow-up batch): factual_research family (LLM-reframe pipeline; leakage only on the zero-sub-questions fallback), prediction_request_v1, prediction_request_sme, prediction_url_cot_v1, resolve_market, resolve_market_reasoning, prediction_langchain. Tools without web retrieval (corcel, dalle, gemini, prepare_tx, propose_question, resolve_market_jury) need nothing.

Invariants held (verified per tool, then re-verified centrally)

• Frozen mirror: the parse block is byte-identical across all 11 fixed tools (programmatic == against the canonical block).
• Trader parity: on the template path, the question fed to the LLM and the Serper query are byte-identical to the old extract_question output (pinned by a test in every tool).
• LLM-input parity: on free text the LLM still receives the WHOLE prompt; only the search query is derived — the worst case of a mis-derived query is weaker sources, never a wrong forecast target.
• Each port went through an independent second-pass review (block identity, wiring order — short-circuit and guard strictly before the first network call, leftover-symbol sweep, minimal-diff check, ASCII).

Verification

• 945 customs tests pass (≈315 added/extended across the 9 tools: template parity, free-text clause derivation, degenerate-prompt short-circuit with assert_not_called on the search mock, both-empty flagged null live + cached-replay, malformed-Serper typed errors, scan-window semantics).
• isort / black / flake8 / mypy / pylint / darglint all green; autonomy packages lock + lock --check clean; zero third-party pin drift.
• benchmark/tournament_tools.json: the 3 predict-* wires re-pinned to the new finetuned_prediction CID; factual_research-v2, superforcaster-polymarket-v4, superforcaster-market-aware unchanged.

Deployment follow-up (not this PR)

The 5 production tools span BOTH platforms (Gnosis predict fleet + Polygon 21/25/44). After merge + release: agent-deployments PR re-pinning TOOLS_TO_PACKAGE_HASH for the affected envs — metadata republish per mech (per-mech url preserved via make update-metadata), as in the v0.21.29 rollout.

---

🤖 Generated with Claude Code