[BFCL] Clarify default and ablation behavior of synthetic request failures in BFCL V4 Web Search

[tim2603]

Describe the issue

The BFCL V4 Agentic Web Search blog appears to be ambiguous about whether synthetic request failures are part of the default leaderboard evaluation or only used in the additional robustness experiment described in Section 7.2.

The released implementation indicates that synthetic request failures are currently disabled. However, Section 3 of the blog describes probabilistic request failures as part of the web-search environment, while Sections 7 and 7.2 present varying fetch-success rates as an additional modified evaluation setup.

ID datapoint

1. Datapoint / Model Handler permalink: N/A — this issue concerns the general Agentic Web Search evaluation setup rather than a specific datapoint or model handler.
2. Issue: Ambiguity and missing reproduction details for synthetic request failures.
3. Gorilla repo commit #: f7cf735

What is the issue

In the released implementation, the random generator used for synthetic request failures is described as belonging to a feature that is currently not used. The corresponding invocation is also commented out:

# Note: Un-comment this when we want to simulate a random error
# Flip a coin to simulate a random error
# if self._random.random() < 0.95:
#     return {"error": self._fake_requests_get_error_msg(url)}

Therefore, synthetic request failures appear to be disabled in the default BFCL V4 Agentic Web Search evaluation.

However, the blog contains two descriptions that may be read differently:

• Section 3 states that probabilistic request failures were intentionally introduced to simulate real-world browsing conditions.
• Sections 7 and 7.2 describe fetch URL errors with 100%, 50%, and 0% fetch-success rates as an additional modified evaluation setup.

This raises the following reproducibility questions:

1. Were synthetic request failures disabled for the standard BFCL V4 Agentic Web Search leaderboard runs?
2. Were they enabled only for the additional experiment described in Section 7.2?
3. How were the 100%, 50%, and 0% fetch-success conditions configured?
4. Was the probability changed manually in the source code?
5. Which random seed or failure schedule was used?

There is also a terminology difference worth clarifying. Section 7.2 refers to “URL blockers,” but the available synthetic error generator does not only generate 403 responses. It samples from several HTTP and network failure types, including:

• 503 Service Unavailable
• 429 Too Many Requests
• 403 Forbidden
• ConnectTimeout
• ReadTimeout
• ConnectionError

Thus, “synthetic request failures” may describe the implemented mechanism more precisely than “URL blockers.”

Proposed Changes

If synthetic request failures were disabled for the standard leaderboard evaluation and used only in the Section 7.2 experiment:

• Clarify this distinction in Section 3 of the blog.
• State explicitly that the default leaderboard evaluation does not inject synthetic request failures.
• Document how the 100%, 50%, and 0% fetch-success conditions from Section 7.2 were implemented.
• Document the random seed or failure schedule used for the experiment.
• Consider exposing the failure probability through an evaluation configuration option rather than requiring a source-code modification.
• Consider using the term “synthetic request failures” consistently if all six HTTP and network error types are sampled.

{
    "previous_datapoint": [],
    "updated_datapoint": []
}

Additional context

The official leaderboard page states that models were evaluated using commit f7cf735 and recommends checking out this commit or installing bfcl-eval==2025.12.17 to reproduce the results.

In that checkpoint, the synthetic error invocation is commented out. A user following the published reproduction instructions would therefore run the standard Agentic Web Search evaluation without synthetic request failures.

I also inspected a sample of the publicly released trajectories and did not observe the synthetic error messages. This is consistent with the released implementation, although the code in the referenced checkpoint is the primary basis for this issue.

[tim2603]

One additional reproducibility/naming detail I noticed while reproducing the Web Search setup:
The BFCL V4 Web Search blog refers to the search tool as duckduckgo_search (including the function definition and example trajectories), whereas the released benchmark/tool specification uses search_engine_query.
Could you clarify whether duckduckgo_search was the actual tool name exposed to the models in the reported leaderboard runs, or whether this is only an outdated/illustrative name in the blog and search_engine_query is the canonical name for reproduction?
If search_engine_query is canonical, it may be worth updating the blog to avoid ambiguity for reproductions.