[Wu Shuwen (dajiaohuang)]

Description

When parsing templates like {{func key='hello world'}}, the CodeTokenizer incorrectly included spaces inside quoted named-arg values, causing NamedArgBlockSyntaxError.

Root cause

The VALUE state in the tokenizer's main loop appends every character to current_token_content without checking whether a space is a token separator. When a space inside a quoted value is followed by an alphanumeric character (the continuation of the quoted text), the space is incorrectly treated as a token boundary.

Two changes fix this:

1. Space handling in VALUE mode: A space now ends the quoted value only when the following character is not a valid quoted-value continuation (alphanumeric, _, -, $, quote, VAR_PREFIX).

2. Closing delimiter check: The closing quote is recognized as a block boundary only when the character after it is not alphanumeric. This correctly handles quoted values adjacent to punctuation like }}.

Reproduction

from semantic_kernel.template_engine.code_tokenizer import CodeTokenizer
CodeTokenizer().tokenize("{{func key='hello world'}}")
# Previously: NamedArgBlockSyntaxError
# After fix: correctly splits into FunctionIdBlock, NamedArgBlock, and trailing FunctionIdBlock

Fixes #14460.

[Copilot]

Copilot review overview

🟡 Changes recommended

The tokenizer does not enter named-argument value handling, so the reported case remains unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Named-argument quoted values are not tokenized or preserved · New

What changed in this PR

Fixes CodeTokenizer handling of spaces in quoted named-argument values.

Changes:

• Adds quoted-value space handling.
• Adds closing-quote lookahead logic.

File	Summary
python/​semantic_kernel/​template_engine/​code_tokenizer.py	Adjusts tokenizer state and quoted-value parsing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Eduard van Valkenburg (eavanvalkenburg)]

Review finding (blocking): Quoted named-argument values with spaces are still split

The advertised {{func key='hello world'}} case still starts in the FUNCTION_ID state, and the tokenizer never enters the new named-argument value branch while scanning the equals sign and quote. The space after hello therefore still takes the generic separator path and raises NamedArgBlockSyntaxError. Please enter a quoted named-argument state when = is followed by a quote, and add a regression test for this exact template.

[Wu Shuwen (dajiaohuang)]

Fixed: Completed the fix for spaces in quoted named-arg values

The original PR had the right goal but the implementation had two issues that caused the 39 test failures:

Root cause of the failures

1. The added space-handling branch (lines 91-101) incorrectly terminated quoted values on spaces, causing all tokens after a quote to be consumed into the ValBlock (e.g. "direct" arg1=$arg1 arg2="arg2" was parsed as a single ValBlock).

2. The closing-delimiter check used text[index+2] (off-by-one: it looked two characters ahead instead of one), which prevented the legitimate closing quote in escape sequences like call 'f\\'xy' from being detected, so the test test_it_throws_when_separators_are_missing no longer raised.

3. The real bug was not addressed: when parsing key='hello world', the opening quote appears after = (mid-FUNCTION_ID token), not after a space separator. The code never transitioned out of FUNCTION_ID mode into a quoted-value mode, so the space inside "'hello world' incorrectly terminated the FUNCTION_ID token at key='hello. The PR fix tried to patch this from within the VALUE/NAMED_ARG branch, but that branch was never entered in the first place.

What this fix does

• Removes the broken space-handling and the incorrect index+2 closing-delimiter check.
• Adds the missing state transition: when in FUNCTION_ID mode, if the current char is a quote (' or ") and = is already present in the token content (i.e. we just saw the opening quote of a named-arg value like key='), switch to BlockTypes.NAMED_ARG mode and set text_value_delimiter. In this mode, spaces are just appended (not treated as token separators), and the matching closing quote ends the block as a NamedArgBlock (not ValBlock).
• Handles NAMED_ARG in the end-of-loop final-token capture.
• Adds a regression test test_named_args_with_spaces_in_quoted_value covering the issue Python: CodeTokenizer splits quoted named-arg values on spaces (NamedArgBlockSyntaxError) #14460 scenario for both single and double quotes, plus multiple spaced named args.

Test results

• All 451 tests/unit/template_engine/ tests pass (was 39 failed).
• All 738 tests across template_engine, functions, and kernel pass.

The fix correctly tokenizes:

• func key='hello world' → [FunctionIdBlock("func"), NamedArgBlock("key='hello world'")] ✓
• plugin.function "direct" arg1=$arg1 arg2="arg2" → 4 blocks ✓
• Escape sequences func 'f\'oo' still work ✓
• Missing separators call 'f\\\\'xy' still raise CodeBlockSyntaxError ✓