feat: Switch tokenizer based on embedding model - #53
Open
cableman wants to merge 1 commit into
Open
Conversation
ghbm-itk
suggested changes
Jul 6, 2026
|
This project could really use a linter... |
cableman
force-pushed
the
feature/embed-tokenizer
branch
from
July 8, 2026 08:55
1c256de to
8acd529
Compare
Author
yes, but that is way out-of-scope |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UPDATE: Open-webui has made the same fix in open-webui@bb6b2db
When the RAG token splitter is enabled (RAG_TEXT_SPLITTER=token), Open WebUI measures CHUNK_SIZE / CHUNK_OVERLAP with tiktoken (cl100k_base, OpenAI's GPT tokenizer) - regardless of which embedding model is actually configured.
We embed with multilingual locally hosted embedding, which use a completely different (XLM-RoBERTa / SentencePiece) tokenizer. On multilingual (Danish) text the two tokenizers disagree substantially on how many tokens a piece of text is, so the chunk sizes we compute don't match what the embedding endpoint actually sees.
Effect:
Chunks can exceed the model's max sequence length (eg. 512 tokens) -> the embedding server thorws out-of-index error
Fix:
New config RAG_TOKENIZER_MODEL names the HuggingFace tokenizer repo to use. Empty that falls back to RAG_EMBEDDING_MODEL, then to tiktoken.