FIX: truncation leading to invalid strings when on UTF boundaries - #7
Conversation
SamSaffron
commented
Dec 10, 2025
SamSaffron
commented
- Generally gems should not check in gemfile lock, dependencies should be handled via gemspec
- FIX: tokenizer not handling UTF-8 truncation gracefully
SamSaffron
changed the title
| def decode(token_ids) | ||
| tokenizer.decode(token_ids) | ||
| rescue Tiktoken::UnicodeError => e | ||
| rescue Tiktoken::UnicodeError | ||
| # Handle invalid token IDs gracefully by returning empty string | ||
| "" | ||
| end | ||
|
|
||
| def safe_decode(token_ids) | ||
| tokenizer.decode(token_ids) | ||
| rescue Tiktoken::UnicodeError | ||
| # Token slice created incomplete UTF-8 sequence | ||
| # Decode individual tokens and join, skipping any that fail | ||
| token_ids | ||
| .map do |id| | ||
| begin | ||
| tokenizer.decode([id]) | ||
| rescue StandardError | ||
| "" | ||
| end | ||
| end | ||
| .join | ||
| end |
There was a problem hiding this comment.
I could be mistaken.
I am not sure who might be calling open_ai_tokenizer#decode. would it make more sense to just move this safe_decode implementation to the decode function itself?
There was a problem hiding this comment.
you know what you are probably right, it will probably give us more consistent behavior, will go ahead and do so, at least it will part decode stuff that is messy
| rescue StandardError | ||
| "" | ||
| end |
There was a problem hiding this comment.
It could be worth it to do a rails warn here, to get some signal when our tokenizers are erroring due to some unknown reason.
There was a problem hiding this comment.
warnings here will kill us, some truncation is just not possible cause the token is on a utf boundary
| rescue StandardError | ||
| rescue Tiktoken::UnicodeError |
There was a problem hiding this comment.
I think StandardError here is still good? Your choice
2 participants
