[BFCL] simple_python_363: release-commit answer key expects `find_closest`, but the item only presents `restaurant_search.find_closest`

[dipak-bhujbal]

Describe the issue

At the BFCL V4 release commit, the answer key for simple_python_363 expects the function name find_closest, but that item's tool list only ever presents restaurant_search.find_closest. Since scoring compares function names by exact string match, no prediction can satisfy both the prompt and the key — a model that calls the tool by the only name it was offered is marked incorrect.

This already appears to be fixed on main. Filing it because anyone who pinned the benchmark at the V4 release commit is still scoring this item against a name it does not offer, and the fix isn't obvious from the release tag.

ID datapoint

1. Datapoint / Model Handler permalink: possible_answer/BFCL_v4_simple_python.json @ 58f57e91 — item simple_python_363
2. Issue: answer key expects a function name the item never presents to the model
3. Gorilla repo commit #: 58f57e9124ea981403792dd51e00a6577e621fae (BFCL V4 Release, BFCL V4 Release #1019)

What is the issue

The question file presents exactly one tool:

BFCL_v4_simple_python.json    simple_python_363
  user:     "Find the closest sushi restaurant with a patio in Boston."
  function: [ { "name": "restaurant_search.find_closest", ... } ]

The release-commit answer key expects a different name:

possible_answer/BFCL_v4_simple_python.json    simple_python_363   @ 58f57e91
  ground_truth: [ { "find_closest": {...} } ]

The file this is reported against is sha256 38b1bc7469d1de73a812ffce9e2b10a1d8812425fd090ed314066ccec76d0ceb, Git blob 455e9c4df114782ca187ded9665819e7ef912846, 400 rows.

At commit 9d8416a96d1d69975493f1b6d60ff07d12a1726a ("Add model Nanbeige and Fix some data bugs", #1257 — 32 commits ahead of the release), the same item expects restaurant_search.find_closest, matching the presented tool. The two rows are byte-identical apart from that key:

- {"ground_truth": [{"find_closest": {...}}], "id": "simple_python_363"}
+ {"ground_truth": [{"restaurant_search.find_closest": {...}}], "id": "simple_python_363"}

This looks like an isolated inconsistency, not a naming convention:

• Checking every item's key names against that item's own tool list at 9d8416a9, the key name is among the presented names in every row — 0 exceptions across 858 rows (400 simple_python, 200 multiple, 258 live_simple).
• Module-qualified names are the norm rather than the exception: 42% of simple_python keys, 62% of multiple, 30% of live_simple.
• At the release commit, simple_python_363 is the only row of the 400 in simple_python where the key name is absent from the item's tool list.

Note on one possible fix: stripping the module prefix before comparison would not be safe in general. 29 of the 200 multiple items offer two or more tools that differ only by module prefix (e.g. triangle_properties.get vs circle_properties.get, EuclideanDistance.calculate vs angleToXAxis.calculate). Tail-matching would make those items ambiguous, and distinguishing near-identical tool names is exactly what the category tests.

Proposed Changes

No patch needed if 9d8416a9 is the intended value — this is a request to confirm that reading. For completeness, the change already present on main:

{
 'previous_datapoint':[{"id": "simple_python_363", "ground_truth": [{"find_closest": {"amenities": [["Patio"]], "cuisine": ["Sushi", "sushi"], "location": ["Boston", "Boston, MA"]}}]}],
 'updated_datapoint':[{"id": "simple_python_363", "ground_truth": [{"restaurant_search.find_closest": {"amenities": [["Patio"]], "cuisine": ["Sushi", "sushi"], "location": ["Boston", "Boston, MA"]}}]}]
}

Additional context

The ask is narrow: confirm restaurant_search.find_closest is the intended ground truth, and if so consider noting it somewhere release-pinned consumers would see, since the V4 release commit is immutable and still carries the earlier value.

Context for why this got checked carefully: we froze BFCL v4 at a pinned commit for a study and hit the discrepancy when two pinned revisions of the same file disagreed on this one row. We resolved it by treating "every name the key expects must be among the tools that item presented" as an executable precondition rather than an assumption, which is how the 858-row figure above was produced. Happy to share that check if it would be useful upstream — it runs on the data files alone, no model required.