[BFCL] Use underscore_to_dot config flag for tool name compilation - #1346
Open
lukeinglis wants to merge 1 commit into
Open
[BFCL] Use underscore_to_dot config flag for tool name compilation#1346lukeinglis wants to merge 1 commit into
lukeinglis wants to merge 1 commit into
Conversation
The `underscore_to_dot` flag in ModelConfig controls whether dots in function names are replaced with underscores. Previously, this flag was only used by the evaluation checker (ast_checker.py) but not during tool compilation in `convert_to_tool()`. Instead, `convert_to_tool()` used a hardcoded list of ModelStyle values to decide which models needed the replacement. This caused a mismatch: tool definitions sent to the model had underscored names, but the evaluator compared against the original dotted ground truth names, leading to false failures on test cases with dotted function names (e.g., `math.factorial`, `country_info.capital`). This change: - Adds an `underscore_to_dot` parameter to `convert_to_tool()` that controls dot replacement, replacing the hardcoded ModelStyle list - Passes the ModelConfig `underscore_to_dot` flag through `build_handler()` to all handler instances - Updates all `_compile_tools` callers to forward the flag Models with `underscore_to_dot=True` in their ModelConfig will continue to have dots replaced. Models with `underscore_to_dot=False` (or new models that don't need the replacement) will preserve dotted names. Fixes ShishirPatil#1091 Signed-off-by: Luke Inglis <linglis@redhat.com> Signed-off-by: Luke Inglis <lukeinglis21@yahoo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
underscore_to_dotflag to control dot-to-underscore replacement during tool compilation, not just during evaluationModelStylelist inconvert_to_tool()with the existingModelConfig.underscore_to_dotflagbuild_handler()through to all handler_compile_toolsmethodsProblem
The
underscore_to_dotflag inModelConfigwas only used by the evaluation checker (ast_checker.py), but the actual dot-to-underscore replacement inconvert_to_tool()used a hardcoded list ofModelStylevalues. This caused a mismatch: models withunderscore_to_dot=Falsestill had their function names mangled during tool compilation, leading to evaluation failures on test cases with dotted function names (e.g.,math.factorialbecamemath_factorial).This particularly affects models evaluated via the
/v1/chat/completionsFC path (e.g., through proxies or gateways), where the model sees underscored tool names and returns underscored calls, but the evaluator compares against the original dotted ground truth.We observed ~50 point drops on affected categories (e.g.,
multiple: 94.5% to 37%) until we worked around this in a custom handler.Changes
convert_to_tool(): replaced hardcodedModelStylelist withunderscore_to_dotparameter (defaultFalsefor backward compatibility)build_handler(): passesconfig.underscore_to_dotto handler constructor_compile_toolsimplementations: forward the flag toconvert_to_tool()Backward Compatibility
Models with
underscore_to_dot=Truein theirModelConfig(e.g., OpenAI, Mistral, Anthropic) continue to have dots replaced, same as before. The behavior change only affects models withunderscore_to_dot=False(or new/custom models), which will now correctly preserve dotted function names.Fixes #1091