From PR #346 review (2026-07-02).
1. Malformed 200 body silently degrades accuracy (bfcl_v4_multi_turn_runner.py:236).
A structurally malformed 200 body (proxy error page, {"error": ...} payload, empty choices) returns (None, None, None), which process_response treats as "no tool calls" (bfcl_v4_execution.py:192-199) — it appends a fabricated empty assistant message and advances the turn, scoring the entry as a normal completion with degraded accuracy. This is inconsistent with the transport-error path (_send_request returning None force-terminates). A malformed body is the same failure class as an HTTP 500 and should force-terminate (or be surfaced distinctly); otherwise a flaky gateway silently degrades reported accuracy.
2. cleanup() does not free bfcl-eval's cached class instances (bfcl_v4_execution.py:316).
The simulated class instances bfcl-eval creates are cached in its module globals() keyed f"{model_name}_{test_entry_id}_{class_name}_instance" and never freed. Consequences: (a) re-running an entry in one process resumes from the first run's mutated state instead of initial_config (wrong results, no error); (b) a full run accumulates thousands of instances for the process lifetime. Since _MODEL_NAME_FOR_EXECUTION is fixed, the cache key is predictable — cleanup() can delete the matching globals() entries.
Deferred out of PR #346.
From PR #346 review (2026-07-02).
1. Malformed 200 body silently degrades accuracy (
bfcl_v4_multi_turn_runner.py:236).A structurally malformed 200 body (proxy error page,
{"error": ...}payload, emptychoices) returns(None, None, None), whichprocess_responsetreats as "no tool calls" (bfcl_v4_execution.py:192-199) — it appends a fabricated empty assistant message and advances the turn, scoring the entry as a normal completion with degraded accuracy. This is inconsistent with the transport-error path (_send_requestreturningNoneforce-terminates). A malformed body is the same failure class as an HTTP 500 and should force-terminate (or be surfaced distinctly); otherwise a flaky gateway silently degrades reported accuracy.2.
cleanup()does not free bfcl-eval's cached class instances (bfcl_v4_execution.py:316).The simulated class instances bfcl-eval creates are cached in its module
globals()keyedf"{model_name}_{test_entry_id}_{class_name}_instance"and never freed. Consequences: (a) re-running an entry in one process resumes from the first run's mutated state instead ofinitial_config(wrong results, no error); (b) a full run accumulates thousands of instances for the process lifetime. Since_MODEL_NAME_FOR_EXECUTIONis fixed, the cache key is predictable —cleanup()can delete the matchingglobals()entries.Deferred out of PR #346.