Skip to content

fix(hf): handle ModelOption.THINKING chat templates - #1588

Open
planetf1 wants to merge 7 commits into
generative-computing:mainfrom
planetf1:issue-1212
Open

fix(hf): handle ModelOption.THINKING chat templates#1588
planetf1 wants to merge 7 commits into
generative-computing:mainfrom
planetf1:issue-1212

Conversation

@planetf1

@planetf1 planetf1 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Pull Request

Issue

Fixes #1212

Description

LocalHFBackend previously removed ModelOption.THINKING before applying the
tokenizer chat template. This change detects the template's supported thinking
variable (think, thinking, or enable_thinking) and forwards boolean
values under that name. It also canonicalises native aliases during option
resolution, so per-call options correctly override backend defaults.

Following review, to_mellea_model_opts_map now maps think/thinking/
enable_thinking to ModelOption.THINKING (it previously had no entries for
these, so a raw alias never folded into the sentinel the way it does for
Ollama). _filter_for_chat_template now lets the resolved sentinel value win
outright over any leftover alias key instead of deferring to whichever
happened to already be present, matching Ollama's precedence for the same
input. The ModelOption.THINKING docstring now notes that non-boolean values
are silently ignored on HF.

This PR deliberately does not add Granite 4.2 model identifiers or change
defaults; #1587 owns that work and its GPU span coverage. The tokenizer-only
integration test uses the public Granite 4.2 3B ID directly to prove the
generic mapping works with the template that first exposes a thinking control.
Its cache-presence check now also tolerates a partial local cache (cached
config.json without the tokenizer's own files) by treating an OSError from
AutoTokenizer.from_pretrained(local_files_only=True) as "not cached" rather
than letting the test error.

During validation, the parallel tool-call regression tests were found to create
a live Ollama backend despite exercising only formatter behaviour. This PR
switches them to the existing mocked backend fixture so the local suite does
not require Ollama for those tests.

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)
    • uv run pytest test/backends/test_huggingface_filter_options.py -q: 34 passed, 3 skipped; Granite 4.2 tokenizer regression executed
    • uv run pytest test/backends/test_acall_tools_parallel_execution.py -q: 4 passed without Ollama
    • uv run ruff format --check . && uv run ruff check .: passed
    • uv run mypy .: passed
    • uv run pytest test/ -m "not qualitative": 4,146 passed, 19 skipped, 128 deselected, 4 xpassed

Attribution

  • AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

Assisted-by: Codex
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Assisted-by: Codex
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
@github-actions github-actions Bot added the bug Something isn't working label Aug 26, 2026
Assisted-by: Codex
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Assisted-by: Codex
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
@planetf1
planetf1 marked this pull request as ready for review August 26, 2026 09:48
@planetf1
planetf1 requested a review from a team as a code owner August 26, 2026 09:48
Assisted-by: Codex
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

@jakelorocco jakelorocco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree with the approach; I think there's a few small concerns

Comment thread mellea/backends/huggingface.py Outdated
Comment thread mellea/backends/huggingface.py Outdated

@AngeloDanducci AngeloDanducci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like _make_backend hardcodes to_mellea_model_opts_map = {}, so test_filter_for_chat_template_uses_per_call_thinking_alias_over_default and _preserves_native_thinking_value pass only because the real map is absent. They don't exercise real behavior, which is why the ordering bug isn't caught.

See suggested code block - though I think the tests will need to be updated if that change is accepted.

Comment thread mellea/backends/huggingface.py
to_mellea_model_opts_map had no entries for think/thinking/enable_thinking,
so a raw alias key never folded into ModelOption.THINKING the way it does
for Ollama. This let a pre-existing alias key in backend_opts silently
block the resolved THINKING value in _filter_for_chat_template, giving HF
different precedence than the other backends for the same input.

Add the alias entries and let the resolved sentinel value win outright
over any alias already present, matching Ollama's behavior. Also documents
that non-boolean THINKING values are ignored on HF, and updates/adds tests
covering the alias fold and the precedence fix.

Assisted-by: Claude Code
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
_try_load_granite_tokenizer only checked for a cached config.json before
calling AutoTokenizer.from_pretrained(local_files_only=True). A cached
config.json does not guarantee the tokenizer's own files are cached too
(e.g. after a partial/interrupted download), so from_pretrained could
raise OSError instead of the test skipping as intended. Catch it and
treat it the same as an absent cache.

Assisted-by: Claude Code
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
@planetf1

Copy link
Copy Markdown
Contributor Author

Heads up: this PR and #1587 both independently touch `test/backends/test_acall_tools_parallel_execution.py`'s `backend` fixture (both switch it from a live `OllamaModelBackend()` to `mock_ollama_backend()`) — but the diffs aren't identical: this PR also drops `pytestmark = [pytest.mark.integration]` and rewords the fixture's docstring, #1587 keeps both as-is. Whichever merges second will hit a small, easily-resolved conflict on that one fixture — worth resolving deliberately (keep #1588's `pytestmark` removal + docstring wording, or not) rather than letting the merge tool guess.

No other overlap: #1587 doesn't touch `LocalHFBackend`/HF THINKING handling (that's this PR's/#1212's scope), and this PR doesn't touch Granite 4.2 defaults or CI/`quality.yml`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enhancement: huggingface should handle ModelOption.THINKING

3 participants