Skip to content

Add fix-tool-choice-enforcement mod - #380

Closed
tea24864 wants to merge 1 commit into
eugr:mainfrom
tea24864:mod/fix-tool-choice-enforcement
Closed

Add fix-tool-choice-enforcement mod#380
tea24864 wants to merge 1 commit into
eugr:mainfrom
tea24864:mod/fix-tool-choice-enforcement

Conversation

@tea24864

@tea24864 tea24864 commented Sep 7, 2026

Copy link
Copy Markdown

What this fixes

On affected builds, tool_choice="required" and named tool choice on
/v1/chat/completions decode free-form. The model answers in prose and no tool
call is produced.

A required request is also malformed on the way out: finish_reason is
"tool_calls" while tool_calls is null, because the serving layer forces
that value and then finds nothing to report. Any OpenAI-compatible client that
branches on finish_reason breaks on it.

Reproduced on vllm-node-b12x builds 0.1.dev20489+ga50ebee1d.d20260904 and
0.1.dev20596+g2a979314d.d20260907, serving GLM-5.3-Flash with
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice.

Cause

Enforcement is an xgrammar structural tag built in Parser.adjust_request. Two
independent breaks keep it from reaching the sampler.

  1. Nothing on the chat-completions path calls adjust_request. The only
    call site in vllm/entrypoints/ is the harmony branch of the Responses API,
    next to a TODO about unifying it with the non-harmony branch.
  2. A collapsed parser engine never builds the tag.
    DelegatingParser.adjust_request is the only implementation that calls
    _apply_structural_tag. When ParserManager.get_parser collapses a
    reasoning and tool adapter that share one engine, which glm45 + glm47
    do, it returns the engine class itself, and that class has no structural-tag
    code at all.

Fixing either half alone changes nothing on a collapsing build. Part 1 was
applied on its own and both probes returned byte-identical output.

The second break is specific to the builds this mod targets. Stock
vllm-project/vllm has no _get_parser_engine_cls branch, so get_parser
always returns a DelegatingParser subclass and part 1 alone is enough there.
The mod README documents how to tell the two apart.

What the mod does

  • Part 1, model-agnostic. Calls parser.adjust_request(request) in
    _create_chat_completion, after the parser is built and before the request is
    rendered, matching where the Responses API makes the call and where the tag
    has to land to reach to_sampling_params. Skipped when the installed vLLM
    already calls it.
  • Part 2, GLM only. Gives Glm47MoeParser an adjust_request that keeps
    the engine's behaviour and then applies the tag, mirroring
    DelegatingParser._apply_structural_tag rather than inventing a second
    policy, including the VLLM_ENFORCE_STRICT_TOOL_CALLING check that
    AbstractToolParser.get_structural_tag applies. Skipped when
    vllm/parser/glm47_moe.py is not installed.

Blast radius

The tag builder returns None for tool_choice: "none", and for "auto"
unless a tool declares strict, so ordinary auto traffic decodes as before.
Confirmed live: an auto request still calls the tool when the question needs
one, still answers directly when it does not, and the reasoning block is intact
either way.

VLLM_ENFORCE_STRICT_TOOL_CALLING=0 disables the whole path, as upstream.

The mod is opt-in. No shipped recipe declares it.

Validation

4x DGX Spark TP4 cluster, tool-eval-bench v2.6.1 hardmode, 88 scenarios,
GLM-5.3-Flash EXL3 6bpw, 0.1.dev20489:

points / 176 score TC-45
Before 159 90 fail
After, trial 1 161 91 pass
After, trial 2 160 91 pass
After, trial 3 161 91 pass

TC-45 is tool_choice=required Compliance. Diffed scenario by scenario, ten
scenarios moved and the nine besides TC-45 cancel to exactly zero, so the net
gain is entirely this fix. EXL3 4bpw moved 154 to 158 points with the same flip.

Across 17 scored hardmode runs, TC-45 tracks the engine build rather than the
quantization or checkpoint, including a 0.1.dev20596 NVFP4-Spark run that
fails the same way and rules out the EXL3 checkpoints as the cause.

Tests

Adds tests/test_tool_choice_enforcement_mod.sh, following the existing mod
test convention. It covers call placement inside the parser_cls guard and
ahead of rendering, no leakage into the streaming path, a self-contained
appended block, idempotency, fail-closed on source drift in either file,
non-GLM images, an upstream that already applies the call, and a missing vLLM
install.

./tests/test_tool_choice_enforcement_mod.sh
./tests/test_recipes.sh -v                      # 67 passed
./tests/test_launch_cluster_image_sync.sh
./tests/test_launch_cluster_vllm_pr.sh
./tests/test_instanttensor_zero_copy_mod.sh     # regression
./tests/test_radixark_dspark_mod.sh             # regression

All top-level recipes still dry-run clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A5YuMY6DErgC5h2Uztz93b

Restores tool_choice enforcement on /v1/chat/completions. On affected
builds, tool_choice="required" and named tool choice decode free-form: the
model answers in prose and no tool call is produced. A "required" request is
also malformed on the way out, reporting finish_reason "tool_calls" with
tool_calls null, which breaks any client that branches on finish_reason.

Enforcement is an xgrammar structural tag built in Parser.adjust_request.
Two independent breaks keep it from reaching the sampler:

1. Nothing on the chat-completions path calls adjust_request. The only call
   site is the harmony branch of the Responses API.
2. On builds whose ParserManager.get_parser collapses a reasoning and tool
   adapter that share one engine, glm45 + glm47 among them, the returned
   engine class has no structural-tag code at all.

Part 1 adds the call and is model-agnostic. Part 2 gives the collapsed GLM
engine the tag its delegating counterpart would have applied, mirroring
DelegatingParser._apply_structural_tag rather than inventing a second
policy. Both parts are skipped when the installed vLLM does not need them.

Measured on a 4x DGX Spark TP4 cluster with tool-eval-bench hardmode,
GLM-5.3-Flash EXL3 6bpw: 159 to 161 points of 176, with the
tool_choice=required compliance scenario flipping from fail to pass across
three trials. Diffed scenario by scenario, the net gain is entirely this fix.

Adds tests/test_tool_choice_enforcement_mod.sh, covering placement, source
drift in either file, idempotency, non-GLM images, and an upstream that
already applies the call.
@tea24864

tea24864 commented Sep 8, 2026

Copy link
Copy Markdown
Author

Closing this: I opened it against the wrong repository. This was meant for my own fork, not upstream. Sorry for the noise.

@tea24864 tea24864 closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant