[Chen Yufeiyang (feiiiiii5)]

What this fixes

ChatMessageContent.to_prompt() is annotated -> str but returns bytes for any message that carries an encoding.

root = self.to_element()
return ElementTree.tostring(root, encoding=self.encoding or "unicode", short_empty_elements=False)

encoding is a real field on the content, so the or never protects it: setting encoding="utf-8" makes tostring take its bytes branch. Measured on ca40aa72:

encoding='utf-8'  ->  type=bytes  value=b'<message role="user" encoding="utf-8">...'
encoding='ascii'  ->  b'<?xml ...?>\n<message ...'   # XML declaration too

This is user-visible rather than merely untidy: jinja2_system_helpers._message_to_prompt and its handlebars twin render a context variable via str(context.to_prompt()), so the prompt the model receives literally contains b'<message role="user" ...>' — bytes repr plus quote noise — whenever the message had an encoding set.

Every other tostring call site in the same package already hardcodes encoding="unicode": chat_history.py:285, :317, :362, chat_message_content.py:275, handlebars_system_helpers.py:37, :55 and jinja2_system_helpers.py:29. This one line was the outlier, so the change aligns it rather than introducing a new convention.

The message's own encoding is still preserved: it is an attribute on the element and round-trips through to_element/from_element; it describes the text content, not how we serialize a prompt string.

Change

One line, plus a docstring note, and a parametrized regression test over utf-8, ascii and latin-1 asserting the prompt is a str, has no XML declaration, keeps the non-ASCII text, and re-parses to the same content.

How this was verified

cd /private/tmp/sk/tree/python, PYTHONPATH=., the repo's own test file tests/unit/contents/test_chat_message_content.py.

Base control — semantic_kernel/contents/chat_message_content.py restored from ca40aa7226531d28a721d0ca0e451d0aaf86dafc with git diff -- <that file> printing nothing (so the run measures upstream code with the new tests kept), sha re-echoed:

FAILED tests/unit/contents/test_chat_message_content.py::test_cmc_to_prompt_with_encoding[utf8]
FAILED tests/unit/contents/test_chat_message_content.py::test_cmc_to_prompt_with_encoding[ascii]
FAILED tests/unit/contents/test_chat_message_content.py::test_cmc_to_prompt_with_encoding[latin1]
3 failed, 43 passed

With the patch applied: 46 passed.

Enclosing directory, both revisions, same command (pytest tests/unit/contents): the test_chat_history.py::test_to_from_file / ::test_chat_history_serialize collection errors appear identically at base and at head — they come from optional extras missing in my environment, not from this diff. I did not run the integration or dotnet suites.

ruff check on the two files reports only a pre-existing D finding at chat_message_content.py:201, present at base too; ruff format --check is clean. (mypy was not installed in this environment, so the CI type gate is unverified here.)

Dedup: searched issues and PRs (all states) for to_prompt, message_to_prompt and encoding unicode prompt; the only hits are merged history (#13738, #5550). The nearby data_uri MIME-subtype defect I found while investigating is already claimed by open #14304, so it is deliberately not part of this PR.

[Copilot]

Copilot review overview

🟢 Approval recommended

The fix is focused, tested across supported encodings, and has no unresolved blocking issues.

Review effort: Lite
Findings: None

What changed in this PR

Fixes ChatMessageContent.to_prompt() so it always returns a string when an encoding is set.

Changes:

• Forces Unicode XML serialization.
• Documents encoding behavior.
• Adds regression tests for UTF-8, ASCII, and Latin-1.

File	Description
python/​semantic_kernel/​contents/​chat_message_content.py	Ensures prompt serialization returns str.
python/​tests/​unit/​contents/​test_chat_message_content.py	Adds encoding-specific regression coverage.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Chen Yufeiyang (feiiiiii5)]

Following up — open since 20 September with no review.

ChatMessageContent.to_prompt() is annotated -> str but returns bytes for any message that carries an encoding:

root = self.to_element()
return ElementTree.tostring(root, encoding=self.encoding or "unicode", short_empty_elements=False)

encoding is a real field on the content, so the or never protects it — encoding="utf-8" takes tostring's bytes branch, and encoding="ascii" additionally emits an XML declaration. jinja2_system_helpers._message_to_prompt then hands those bytes to jinja2, where a str-annotated path receives bytes.

I coerce to str when the result is bytes. The judgement call worth naming: with a non-unicode encoding the decoded text is not necessarily equal to the bytes, so a lossy decode is possible in principle. I used errors="replace" on the grounds that a prompt template must not raise on an exotic encoding — but "strict" (and letting it fail loudly) is a defensible alternative, and it is a one-word change if you prefer it.

Nothing else is outstanding from me. If you would rather this went to an issue first, say so and I will move it.

[Chen Yufeiyang (feiiiiii5)]

Hi Microsoft (@microsoft) maintainers — #14464 has been quiet since 1 October, so this is the one nudge I planned.

Recap: ChatMessageContent.to_prompt() is annotated -> str but returns bytes for any message that carries an encoding — encoding is a real field, so the or never protects it and ElementTree.tostring takes its bytes branch. The fix decodes to str when an encoding is set, matching the annotation, with regression tests for both branches.

CI note: the three red checks are all automation gates that behave this way on fork PRs — review and add_label (the auto-triage jobs, which also drive the merge-gatekeeper verdict), and the code checks themselves are green.

Is there anything you'd like changed before this can go in?