[Franco Ceballos Rastello (xThreeh)]

Motivation and Context

Fixes #14566

split_plaintext_paragraph and split_markdown_paragraph can return a paragraph above max_tokens. When the last paragraph is short, _split_text_paragraph merges it into the previous one, but the merge check counts words (split(" ")) instead of using token_counter. A word is usually more than one token, so the merged paragraph can go over the limit. With tiktoken as the counter I saw chunks of up to 314 tokens for max_tokens=256.

Description

• Build the merged paragraph once and merge only if token_counter(merged) <= max_tokens. The merged text is the same as before; only the condition changes.
• Add test_split_text_paragraph_evenly_respects_max_tokens. It fails on main (last chunk 16 tokens, limit 15) and passes with this change.
• Update eight existing tests whose expected last chunk was above max_token_per_line (16 for 15, 18 for 17). Everything before the last chunk is unchanged; the last chunk is now split in two.

pytest tests/unit/text tests/unit/core_plugins passes, mypy is clean on text_chunker.py, and the repo's pre-commit hooks pass on the changed files.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Franco Ceballos Rastello (xThreeh)]

Closing: #14238 already fixes the same merge check in _split_text_paragraph and was opened first. Sorry for the duplicate.