Skip to content

[BFCL] Use underscore_to_dot config flag for tool name compilation - #1346

Open
lukeinglis wants to merge 1 commit into
ShishirPatil:mainfrom
lukeinglis:fix/bfcl-underscore-to-dot-tool-compilation
Open

[BFCL] Use underscore_to_dot config flag for tool name compilation#1346
lukeinglis wants to merge 1 commit into
ShishirPatil:mainfrom
lukeinglis:fix/bfcl-underscore-to-dot-tool-compilation

Conversation

@lukeinglis

Copy link
Copy Markdown

Summary

  • Fixes the underscore_to_dot flag to control dot-to-underscore replacement during tool compilation, not just during evaluation
  • Replaces the hardcoded ModelStyle list in convert_to_tool() with the existing ModelConfig.underscore_to_dot flag
  • Passes the flag from build_handler() through to all handler _compile_tools methods

Problem

The underscore_to_dot flag in ModelConfig was only used by the evaluation checker (ast_checker.py), but the actual dot-to-underscore replacement in convert_to_tool() used a hardcoded list of ModelStyle values. This caused a mismatch: models with underscore_to_dot=False still had their function names mangled during tool compilation, leading to evaluation failures on test cases with dotted function names (e.g., math.factorial became math_factorial).

This particularly affects models evaluated via the /v1/chat/completions FC path (e.g., through proxies or gateways), where the model sees underscored tool names and returns underscored calls, but the evaluator compares against the original dotted ground truth.

We observed ~50 point drops on affected categories (e.g., multiple: 94.5% to 37%) until we worked around this in a custom handler.

Changes

  • convert_to_tool(): replaced hardcoded ModelStyle list with underscore_to_dot parameter (default False for backward compatibility)
  • build_handler(): passes config.underscore_to_dot to handler constructor
  • All _compile_tools implementations: forward the flag to convert_to_tool()

Backward Compatibility

Models with underscore_to_dot=True in their ModelConfig (e.g., OpenAI, Mistral, Anthropic) continue to have dots replaced, same as before. The behavior change only affects models with underscore_to_dot=False (or new/custom models), which will now correctly preserve dotted function names.

Fixes #1091

The `underscore_to_dot` flag in ModelConfig controls whether dots in
function names are replaced with underscores. Previously, this flag was
only used by the evaluation checker (ast_checker.py) but not during tool
compilation in `convert_to_tool()`. Instead, `convert_to_tool()` used a
hardcoded list of ModelStyle values to decide which models needed the
replacement.

This caused a mismatch: tool definitions sent to the model had
underscored names, but the evaluator compared against the original
dotted ground truth names, leading to false failures on test cases with
dotted function names (e.g., `math.factorial`, `country_info.capital`).

This change:
- Adds an `underscore_to_dot` parameter to `convert_to_tool()` that
  controls dot replacement, replacing the hardcoded ModelStyle list
- Passes the ModelConfig `underscore_to_dot` flag through
  `build_handler()` to all handler instances
- Updates all `_compile_tools` callers to forward the flag

Models with `underscore_to_dot=True` in their ModelConfig will continue
to have dots replaced. Models with `underscore_to_dot=False` (or new
models that don't need the replacement) will preserve dotted names.

Fixes ShishirPatil#1091

Signed-off-by: Luke Inglis <linglis@redhat.com>
Signed-off-by: Luke Inglis <lukeinglis21@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BFCL] underscore_to_dot naming issue + not properly used

1 participant