Skip to content

[BUG] TokenManager.truncate 中文截断偶现乱码 #3

Description

@CryptoSentinelAI

复现

from ai_llm import TokenManager

tm = TokenManager("gpt-4o")
text = "这是一段包含中英文mixed content的很长文本" * 30
result = tm.truncate(text, max_tokens=100)
print(result[-10:])  # 偶尔出现截断在汉字中间

预期

截断应在完整字符边界,不会出现半个汉字。

环境

  • Python 3.11
  • ai-llm 0.3.1
  • tiktoken 0.7.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions