fix(pipelines-audio): preserve multi-code-unit grapheme clusters in TTS chunking - #2414
Conversation
⏳ Approval required for deploying to Cloudflare Workers (Preview) for stage-web.
Hey, maintainers, kindly take some time to review and approve this deployment when you are available. Thank you! 🙏 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e61368bcfb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // punctuation markers below, so treat it as ordinary text. Dropping it | ||
| // here corrupted space-less scripts: the combining marks were discarded, | ||
| // leaving bare consonants that reassemble into different words (#2366). | ||
| buffer += value |
There was a problem hiding this comment.
For chat auto-TTS, Stage.vue calls createSpeechPipeline, whose default segmenter comes from packages/pipelines-audio/src/processors/tts-chunker.ts. That implementation still drops every cluster with value.length > 1 at lines 72-75. This line repairs only the stage-ui duplicate, which has no in-repo production import. The reported Thai text therefore remains corrupted. Apply the fix and regression test to the pipeline-owned chunker.
AGENTS.md reference: AGENTS.md:L253-L254
Useful? React with 👍 / 👎.
e61368b to
81b1270
Compare
Jorge-Polanco-Roque
changed the title
|
Good catch from the automated review — thank you. You're right that |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81b12703c7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // punctuation markers below, so treat it as ordinary text. Dropping it | ||
| // here corrupted space-less scripts: the combining marks were discarded, | ||
| // leaving bare consonants that reassemble into different words (#2366). | ||
| buffer += value |
There was a problem hiding this comment.
When input contains Windows-style \r\n, Unicode grapheme segmentation returns the pair as one cluster with value.length === 2. This branch now appends the pair and skips the hardPunctuations check, although that set defines both characters as hard boundaries. As a result, CRLF text does not flush at line breaks and sends embedded newlines inside a larger TTS request. Handle CRLF before this length guard, or classify the cluster by its contents.
Useful? React with 👍 / 👎.
…TS chunking `chunkTtsInput` skipped any grapheme cluster whose string spanned more than one UTF-16 code unit, dropping it from the text forwarded to the speech provider. For space-less scripts this is catastrophic: in Thai/Lao/Khmer/Devanagari a syllable is a base consonant plus combining vowel/tone marks, i.e. a multi-code-unit cluster, so those syllables were discarded and the bare consonants left behind reassembled into different, valid-but-wrong words — matching the corruption reported (chat shows the correct text, TTS speaks gibberish). This is the chunker behind `createSpeechPipeline`, i.e. the one chat auto-TTS actually uses. A cluster longer than one code unit is never one of the single-char punctuation markers, so it is ordinary text: append it to the buffer instead of dropping it. This also preserves emoji ZWJ sequences. Adds regression tests (Thai, Devanagari, ASCII) asserting no characters are dropped; the Thai case fails without the fix. Fixes moeru-ai#2366
|
Good catch on the CRLF case. |
81b1270 to
3d4b92e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d4b92e6d9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (value !== '\r\n') { | ||
| buffer += value |
There was a problem hiding this comment.
When punctuation carries a variation selector or combining mark, such as !\uFE0F, 。\uFE00, or ?\u20E3, clustr returns one multi-unit grapheme. This branch buffers it as ordinary text, so the hard punctuation does not flush and repeated sentences can bypass maximumWords until EOF. Classify the cluster from its base code point while preserving the full cluster in the emitted text.
Useful? React with 👍 / 👎.
|
Gentle ping — open ~a week. It preserves multi-code-unit grapheme clusters when chunking text for TTS, so emoji and combining marks aren't split mid-cluster. Happy to tweak on request. Thanks! |
1 participant
Problem
Fixes #2366. When a chat reply in Thai (or other space-less / combining scripts) is spoken automatically, the text forwarded to the TTS provider is corrupted — the chat UI shows the correct sentence, but the speech uses different, valid-but-wrong characters:
Root cause
Not the space-based splitting the report suspected — that path already segments with
Intl.Segmenter/ grapheme clusters. The real cause is inchunkTtsInput(pipelines-audio/src/processors/tts-chunker.ts), the chunker behindcreateSpeechPipelinethat chat auto-TTS uses:valueis a grapheme cluster. The guard skips any cluster whose string is longer than one UTF-16 code unit. In Thai/Lao/Khmer/Devanagari a syllable is a base consonant + combining vowel/tone marks, i.e. a multi-code-unit cluster — so those syllables were silently discarded, and the bare single-unit consonants left behind reassembled into different words. That is exactly the "entire character sequences are different" corruption in the report.Reproduced as a unit test: feeding
ดึกป่านนี้แล้วยังจะหาเรื่องกินอีกนะคะเนี่ยthroughchunkTtsInputyieldsกานแวงจะหาเองนกนะคะเยonmain.Fix
A cluster longer than one code unit is never one of the single-char punctuation markers checked below, so it is ordinary text — append it to the buffer instead of dropping it:
One line of behaviour change. This also preserves emoji ZWJ sequences (which were being dropped too).
Tests
Adds regression cases (Thai, Devanagari, ASCII) to
tts-chunker.test.tsasserting no characters are dropped. The Thai case fails without the fix and passes with it:Scope
Only the TTS chunking path is touched; the chat display is unaffected (it was already correct). The fix is language-agnostic — any base+combining script benefits, not just Thai.
Fixes #2366