Smollm3-3b chat completion fails: chat_template.jinja uses unsupported .replace() / .rstrip() methods

[ambs19]

Description

smollm3-3b-generic-gpu:1 fails during chat completion before inference starts. The failure appears to happen
while Foundry Local / ONNX Runtime GenAI renders the model's chat_template.jinja.

The bundled chat template uses Python/Jinja-style string methods such as .replace(), .rstrip(), and
.lstrip(), but the runtime template parser reports these methods as unsupported.

This means the model cannot currently be evaluated or used through normal complete_chat(...) calls with system/
user messages.

Environment

• OS: macOS 14.6.1, Apple Silicon / arm64
• Python: 3.13.14
• Package: foundry-local-sdk==1.2.1
• Model: smollm3-3b-generic-gpu:1
• Model cache path observed locally:
  • .foundry_local/models/Microsoft/smollm3-3b-generic-gpu-1/v1/

Reproduction Steps

from pathlib import Path

from foundry_local_sdk import Configuration, FoundryLocalManager

workspace = Path.cwd()
app_data = workspace / ".foundry_local" / "app_data"
model_cache = workspace / ".foundry_local" / "models"
logs = workspace / ".foundry_local" / "logs"

app_data.mkdir(parents=True, exist_ok=True)
model_cache.mkdir(parents=True, exist_ok=True)
logs.mkdir(parents=True, exist_ok=True)

config = Configuration(
    app_name="smollm3_template_repro",
    app_data_dir=str(app_data),
    model_cache_dir=str(model_cache),
    logs_dir=str(logs),
)

FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

model = manager.catalog.get_model("smollm3-3b")
model.download()
model.load()

client = model.get_chat_client()

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the golden ratio?"},
]

response = client.complete_chat(messages)
print(response.choices[0].message.content)

model.unload()

## Actual Result

The request fails before a model response is generated.

Error excerpt:

FoundryLocalException: Error during chat completion: Error:
Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException:
Unknown method: replace at row 23, column 48:

{%- set custom_instructions = system_message.replace("/no_think", "").replace("/think", "").rstrip() -%}
                                               ^

The same failure occurs consistently for every chat completion request using this model.

## Expected Result

smollm3-3b-generic-gpu:1 should successfully render its chat template and produce a chat completion response.

Alternatively, if the current runtime intentionally supports only a restricted subset of Jinja/template syntax,
the catalog version of this model should use a compatible chat template.

## Suspected Cause

The model package includes:

.foundry_local/models/Microsoft/smollm3-3b-generic-gpu-1/v1/chat_template.jinja

The template contains multiple string method calls:

{%- set custom_instructions = system_message.replace("/no_think", "").replace("/think", "").rstrip() -%}
{{- custom_instructions.replace("/system_override", "").rstrip() -}}
{{ "<|im_start|>assistant\n" + content.lstrip("\n") + "<|im_end|>\n" }}
{{ "<|im_start|>assistant\n" + "<think>\n\n</think>\n" + content.lstrip("\n") + "<|im_end|>\n" }}

These appear to be unsupported by the runtime template parser used by Foundry Local / ONNX Runtime GenAI.

## Impact

This blocks use of smollm3-3b through the normal Foundry Local Python SDK chat completion flow.

In my evaluation run, the model had:

success_rate = 0 / 25

All failures had the same template parsing error, so this should be treated as a runtime/model-template
compatibility failure rather than a model quality issue.

## Possible Fix

One possible fix is to update the catalog model's chat_template.jinja so it avoids unsupported string methods
such as .replace(), .rstrip(), and .lstrip().

For example, a minimal compatibility-oriented template could avoid dynamically stripping /think, /no_think, and /
system_override markers, or use only syntax supported by the runtime template parser.

Another possible fix is to extend the runtime template parser to support these string methods if this syntax is
expected to be valid for catalog model templates.

## Additional Notes

The model's inference_model.json reports:

{
  "Name": "smollm3-3b-generic-gpu:1",
  "PromptTemplate": {
    "system": "<|im_start|>system\n{Content}<|im_end|>",
    "user": "<|im_start|>user\n{Content}<|im_end|>",
    "assistant": "<|im_start|>assistant\n{Content}<|im_end|>",
    "prompt": "<|im_start|>user\n{Content}<|im_end|>\n<|im_start|>assistant"
  }
}

However, the runtime error references chat_template.jinja, so it appears that the bundled Jinja template is being
used during chat completion.

[ambs19]

I would be happy to work on a fix for this if the maintainers agree with the proposed direction.

My preferred approach would be one of the following:

1. Update the smollm3-3b-generic-gpu:1 catalog artifact so its chat_template.jinja only uses template syntax
   supported by the Foundry Local / ONNX Runtime GenAI runtime.

2. Add a compatibility/regression test that loads the catalog model template and verifies that a basic system + user chat completion request can be rendered successfully.

3. If the intended behavior is that .replace(), .rstrip(), and .lstrip() should be supported in chat
   templates, I can help investigate where support for these string methods should be added.

Could you please confirm which direction would be most appropriate for a contribution, and whether the smollm3- 3b-generic-gpu:1 template is maintained in this repository or in a separate model catalog/artifact pipeline?

Once I know the right location for the fix, I can prepare a PR.

[samuel100]

Thanks for the detailed repro — this is exactly the kind of report that's easy to act on.

You're right that this is a runtime/template-parser gap rather than a catalog problem. A quick trace of where the rendering happens:

• Foundry Local delegates chat completion to ONNX Runtime GenAI.
• ORT GenAI's Tokenizer::ApplyChatTemplate calls OrtxApplyChatTemplate from microsoft/onnxruntime-extensions (src/models/model.cpp).
• onnxruntime-extensions renders the template via a vendored copy of the minja Jinja-subset engine at shared/api/minja.hpp.

That vendored minja.hpp is an older snapshot. Comparing it to current upstream google/minja:

String method	Upstream minja	Vendored copy in ort-extensions
.strip / .rstrip / .lstrip	✅	✅
.split	✅	✅
.replace	✅	❌ missing — hits the Unknown method throw
.upper / .lower / .title	✅	❌ missing

So smollm3-3b's template fails on the first .replace(...) call, which matches the error in your report exactly.

Preferred fix: patch this in onnxruntime-extensions (re-sync shared/api/minja.hpp with upstream minja, or at minimum add the missing string-method handlers). We do not want to rewrite the model's chat_template.jinja in the catalog — the chat template is part of the model contract authored by the model owner, and patching it per-model:

• risks subtle divergence from training-time formatting,
• doesn't scale (every new HF model using .replace / .upper / etc. would hit the same wall),
• pushes a runtime bug down into model packaging.

I've opened a tracking issue in onnxruntime-extensions: microsoft/onnxruntime-extensions#1081. Once the missing methods are merged there and pulled into the next ORT GenAI / Foundry Local release, smollm3-3b (and other HF models using these standard Jinja string methods) should work without any catalog change.

On your offer to contribute — that's much appreciated. The most useful contribution would be in onnxruntime-extensions rather than this repo: see the linked issue for scope. A regression test that renders the chat templates of a handful of popular HF models against the vendored minja would also be very welcome there, to catch future drift.

If you're blocked today and need a short-term workaround, dropping a patched chat_template.jinja into your local model cache (avoiding .replace/.upper/etc.) will unblock you while the runtime fix lands — but we'd treat that as a band-aid, not the fix.

[samuel100]

Quick update — fix has been submitted upstream as a PR against onnxruntime-extensions:

microsoft/onnxruntime-extensions#1082 — Add missing .replace, .upper, .lower string methods to vendored minja

The PR adds the three missing string-method handlers to the dispatch in shared/api/minja.hpp (matching upstream google/minja and Python str semantics), plus regression tests in test/pp_api_test/test_tokenizer_chat.cc including the exact chained .replace(...).replace(...).rstrip() pattern from your smollm3-3b repro.

Verified locally with a standalone harness that compiles minja.hpp directly and renders the failing line from smollm3-3b's chat_template.jinja:

before: ERROR: Unknown method: replace at row 1, column 6
after:  OK: "You are helpful ."

Once #1082 is merged in onnxruntime-extensions and rolled into the next ORT GenAI / Foundry Local release, this issue should resolve without any catalog-side change to smollm3-3b.

Separately, to prevent this class of issue from reaching customers in future, I've opened an internal work item for a pre-publication gate that renders every catalog model's chat template through the shipping ORT GenAI + onnxruntime-extensions binaries before publish. That's tracked outside this repo.