superforcaster-polymarket-v4: free-text prompts (the advertised contract) silently disable retrieval, yielding prior-only forecasts

[bennyjo]

Summary

superforcaster-polymarket-v4 (and its ancestors) advertises a free-text input contract but silently depends on the valory trader's internal prompt template. Any other agent that calls the tool through the Mech Marketplace - the intended audience includes any agent paying for a prediction via Pearl Connect, not only traders - gets a degraded prior-only forecast: the web-retrieval step returns zero results and the LLM answers from its pretraining priors.

Observed in production: a mech request for a market trading at 0.875 returned p_yes = 0.60 with organic: [] in the captured Serper response. The requesting agent had stated the resolution question precisely (criteria, resolution source, deadline), exactly as the advertised contract suggests.

Root cause

1. The advertised contract says free text. The published tool metadata (scripts/tool_schemas.yaml, prediction kind) describes the input in its entirety as:

   input:
     type: text
     description: The text to make a prediction on

   Nothing tells a requester that a specific prompt format is expected.

2. The tool assumes an undocumented template. extract_question in packages/valory/customs/superforcaster_polymarket_v4/superforcaster_polymarket_v4.py extracts the question with the regex question\s+"(.+?)"\s+and\s+the\s+yes``, which matches only the valory trader's internal prompt template. On a mismatch it falls back to using the entire prompt as the Serper search query.

3. Prompt-shaped queries degrade retrieval, in the worst case to zero organic results. Verified empirically against the live Serper API (2026-08-28), running the tool's own extract_question where applicable:

   Query shape	Regex matched	Query sent	Status	organic results
   Real trader-template prompt (317 chars)	yes	bare 59-char question	200	10
   Detailed free-text resolution question (347 chars: criteria + source + deadline)	no	full prompt	200	6
   Prompt with instruction boilerplate and JSON-format text (580 chars)	no	full prompt	200	0
   Question wrapped in exact-match quotes	n/a	quoted phrase	200	0

   The trader path is healthy because extraction succeeds. For non-template prompts, degradation depends on content: a pure natural-language question usually still retrieves (fewer results); instruction boilerplate, backticks/JSON-format text, or exact-match quoting can drive results to zero. The production organic: [] case likely involved such content in the prompt; the captured searchParameters.q in the mech response is the definitive evidence for any specific incident.

4. Empty retrieval passes silently. The tool guards the HTTP-error path (raise_for_status()), but a 200 with organic: [] flows straight through: format_sources_data returns an empty string, the LLM receives an empty <background></background> block, and it forecasts from priors. The failure is invisible on-chain except as a poorly calibrated number.

Net effect: only the one caller whose internal template the tool happens to parse gets the intended retrieval path. Every other requester's prompt is passed raw to Serper - retrieval quality then depends on incidental prompt content, ranging from mildly degraded to silently empty, and the tool neither detects nor reports the empty case.

Why this matters

Mech Marketplace tools are meant to be callable by any agent willing to pay for a good prediction, not only the valory trader. That requires two things, and this tool currently provides neither:

• The published metadata must give a requester clear, sufficient instructions to call the tool successfully.
• The tool must honor those instructions - a prompt that satisfies the advertised contract must take the full-quality path, including retrieval.

Suggested fix direction

Tool-side, as a new variant (superforcaster-polymarket-v5) per the housekeeping rules, since any of this shifts the output distribution:

1. Honor the advertised free-text contract: when the trader-template regex does not match, treat the whole prompt as the question for the LLM but derive a proper search query from it (extract/compress the leading question, rather than passing the full prompt to Serper).
2. Add an explicit empty-retrieval guard: on organic: [], either retry with a simplified query, or force low confidence/info_utility (or return the null prediction) so the requester can detect and discount the answer.

Secondary, independent of the variant: make the published input description in scripts/tool_schemas.yaml state what a good prompt looks like (a single self-contained resolution question with criteria, source, and deadline), so requesters have explicit instructions to adhere to.

References

• Tool source: packages/valory/customs/superforcaster_polymarket_v4/superforcaster_polymarket_v4.py (extract_question, fetch_additional_sources, format_sources_data)
• Published metadata: scripts/tool_schemas.yaml (defaults.prediction.input), scripts/generate_metadata.py
• Related precedent: fix(customs): align component.yaml descriptions with published manifests #284 (tool description vs. trader is_prediction_tool predicate - same class of implicit-contract problem, on the discovery side rather than the input side)

[valory-coding-agent]

Decomposition complete; draft PR opened at #456 (label tool-improvement confirmed).

Summary: The bug was confirmed by direct code inspection and your empirical delivery data. extract_question() in v4 uses a regex that only matches the valory trader's internal prompt template; any free-text caller falls through to question = prompt, passing the full 500+ char prompt as the Serper query. Serper deterministically returns organic: [] for long queries, so the LLM forecasts from priors alone.

v5 fixes both failure points: derive_search_query() compresses free-text prompts to a <=150-char Serper query, and an explicit empty-retrieval guard in run() returns a flagged null prediction (confidence=0.0) instead of a silent prior-only forecast.

PR-CI will validate on W-2 (the preceding 7d window disjoint from investigation). Since this is a code bug report with no prior Brier window, the W-2 replay will be the first objective Brier measurement for v5.

[bennyjo]

Example prompts from a real requester

These are the three prompts that Phil (a Claude Code agent that buys second opinions through Pearl Connect) sent to superforcaster-polymarket-v4 on 2026-08-28. Each prompt follows the advertised free-text contract: one self-contained resolution question with criteria, resolution source, and deadline. None of them matches the extract_question regex, so the tool sent the full prompt to Serper in every case. The searchParameters.q and organic values come from the captured serper_response in each delivery.

All three requests went to mech 0x76a9a29441c7acd072b03e63911f0e177de56ab7 on Polygon with max_payment 20000.

Prompt 1: 847 characters, zero organic results (the production case in the summary)

Market: "Will Enzo Fernandez join Manchester City?" (id 2554103), trading at 0.875 at request time.

Will Enzo Fernandez officially join Manchester City by September 1, 2026, 11:59 PM ET? Resolution criteria: the Polymarket market "Will Enzo Fernandez join Manchester City?" (market id 2554103) resolves Yes only if Enzo Fernandez officially joins Manchester City (official club announcement of a completed transfer) by 2026-09-02 03:59 UTC. If he officially joins no club by that deadline, the parent market resolves "Chelsea" (he stays) and this binary resolves No. Resolution source: official club announcements. Context as of 2026-08-28: press reports (Fabrizio Romano, TyC Sports) say an agreement between Chelsea and Manchester City is close or struck at around 120M GBP, the player has agreed personal terms until 2032, but no official completion, medical, or "here we go" confirmation has been published yet. What is the probability of Yes?

• Serper q: the full 847-character prompt.
• organic: [] (0 results).
• Result: {"p_yes": 0.6, "p_no": 0.4, "confidence": 0.7, "info_utility": 0.8}.
• Requester's own estimate: 0.85. Market: 0.875.
• Delivery CID: bafybeig57lzjl7pwrvta4hsfgge25c3zskgcstb5xc72kgwlxxpltavbqu, executed 2026-08-28T06:43:08Z.

Note that confidence 0.7 and info_utility 0.8 were reported with an empty <background> block. The requester had no way to tell that retrieval was empty.

Prompt 2: 572 characters, 7 organic results, none relevant to the question

Market: "Next Mythos-Class Model released by August 31, 2026?", trading at 0.28.

Will Anthropic release the next Mythos-class model (any model after Claude Fable 5, whose official name includes "Mythos" or that Anthropic officially describes as Mythos-class or similar, e.g. a Fable successor in that class) AND make it available to the general public by August 31, 2026, 11:59 PM ET? Context: Claude Fable 5 (Mythos-class) was released June 9, 2026; no successor release has been announced as of August 28, 2026. Resolution source: official Anthropic announcements. Resolve YES only if such a model is generally available to the public by the deadline.

• Serper q: the full 572-character prompt.
• organic: 7 results. All are about the June 2026 Fable 5 launch, a Reddit thread, and an unrelated blog post. None addresses a successor release.
• Result: {"p_yes": 0.005, "p_no": 0.995, "confidence": 0.98, "info_utility": 0.9}.
• Requester's own estimate: 0.07. Market: 0.28.
• Delivery CID: bafybeica4qd46deithqaxjxhoxtolbvadkj4u4xryk5eqv5ww6ofxhx7t4, executed 2026-08-28T09:43:18Z.

Prompt 3: 424 characters, 7 organic results, mostly Polymarket mirror pages

Market: "Will Bitcoin reach $82,000 August 24-30?", trading at 0.26.

Will any Binance 1-minute candle for BTC/USDT have a final "High" price >= $82,000 at any time between August 24, 2026 12:00 AM ET and August 30, 2026 11:59 PM ET? Context as of August 28 ~09:40 UTC: BTC trades near $80,750; the week's high so far is about $81,240 (no touch of $82,000 yet in the window); BTC is up ~10% on the week. Resolution source: Binance BTC/USDT 1-minute candles. About 2.7 days remain in the window.

• Serper q: the full 424-character prompt.
• organic: 7 results. Five are Polymarket or PMIP pages for the market itself or its siblings, one is a Coinbase price page, and one is an unrelated AI-agent sandbox page. No price-history or news source.
• Result: {"p_yes": 0.32, "p_no": 0.68, "confidence": 0.85, "info_utility": 0.9}.
• Requester's own estimate: 0.60. Market: 0.26.
• Delivery CID: bafybeieklpkabk7dfpoxsdj7djsjg3f7jwsgproskb6o7r465b56stljd4, executed 2026-08-28T09:43:48Z.

What the prompts have in common

• Length between 424 and 847 characters. Longer prompts retrieve less: the 847-character prompt returned nothing.
• Embedded double quotes ("Will Enzo Fernandez join Manchester City?", "High", "Mythos"). Serper treats quoted spans as exact-match terms.
• Instruction-like phrasing: Resolution criteria:, Resolution source:, Resolve YES only if, What is the probability of Yes?.
• Context sentences with dates, prices, and market ids that don't belong in a search query.

Every one of these traits is what the tool description asks for, and every one of them makes the prompt a worse search query when passed to Serper unchanged. A v5 derive_search_query() that compresses the prompt to its leading question, plus an empty-retrieval guard, would have put all three requests on the retrieval path or flagged the first one as prior-only.

Verification (2026-09-01)

Re-ran the three prompts through the tool's own extract_question (superforcaster_polymarket_v4.py:499) and then through Serper with the same payload as fetch_additional_sources, plus a control that wraps the Enzo question in the trader template.

Prompt	extract_question	q length	Organic (captured 2026-08-28)	Organic (live 2026-09-01)
Enzo / Man City	regex miss, full prompt	847	0	0
Mythos-class model	regex miss, full prompt	572	7	7
BTC $82k	regex miss, full prompt	424	7	9
Control: Enzo question in trader template	regex hit, bare question	61	n/a	10

All three free-text prompts log Error extracting question: list index out of range and fall through to question = prompt. run() guards only the non-200 path with raise_for_status(); a 200 with organic: [] passes through format_sources_data, which returns "", so the model receives an empty <background> block. The zero-result case is prompt-dependent: the 847-character prompt reproduces it on both dates, while the two shorter prompts return a handful of low-relevance results (the market's own Polymarket page, or coverage of a model that already shipped).

[bennyjo]

QA of the deployed fix: verified, nothing further needed

Re-sent the three production prompts from the comment above, verbatim, from the same requester as the incident (Phil's Pearl Connect service on Polygon) to mech service 21 (0x76a9a29441c7acd072b03e63911f0e177de56ab7), on both superforcaster-polymarket-v4 and superforcaster-market-aware. All six deliveries take the retrieval path. None sent the full prompt to Serper, and none hit the flagged-null guard.

The mech runs the merged code: every delivery reports a tool_hash equal to the CID pinned on main (v0.21.29 and v0.21.30 carry the same tool code). Running parse_prompt from main locally on the three prompts yields the exact queries the mech sent.

Prompt	parse_tier	Serper q sent	Organic (Aug 28 vs now)	v4 p_yes (Aug 28 vs now)	MA p_yes
P1 Enzo / Man City (847 chars)	clause	Will Enzo Fernandez join Manchester City? (41 chars)	0 vs 8	0.60 vs 0.999	0.995
P2 Mythos-class release (572 chars)	clause	leading question, capped at 148 chars	7 vs 10	0.005 vs 0.003	0.06
P3 BTC $82k window (424 chars)	clause	leading question, capped at 148 chars	7 vs 10	0.32 vs 0.38	0.62

Retrieval is on topic in all three (P1 includes the club's official announcement; P2 returns Anthropic's release pages; P3 returns price pages, and search still cannot see 1-minute candles, per #459). Both tools produce identical tiers and queries. P1 resolved Yes and P2 resolved No, so treat this as a retrieval check, not a calibration check.

Delivery CIDs: P1 v4 bafybeidfbnqakdjcyyjt4ho3zwci7p44oswggchlaxqwdvyufgbcpee6om, P2 v4 bafybeif7gqxyz64d7djl457ivv4lxgbdnj2o5gjx4swqptajifd5w7ngne, P1 MA bafybeicofpseqzozzvt3vhfwcx4y33c5i6g5h7botxzjundl6ulafogjra, P2 MA bafybeia2f75ihje7y35sabrzdzykq2d7kyl2zngfuepl7imbo7dsscbny4, P3 MA bafybeihmqq2gon235rybctcpdfjsrb3heyhn2sdmtf6hjdmdj2qkkjmvim.