feat: add --lang flag for translating non-English text queries - #308
Open
SergeyKarleev wants to merge 1 commit into
Open
feat: add --lang flag for translating non-English text queries#308SergeyKarleev wants to merge 1 commit into
SergeyKarleev wants to merge 1 commit into
Conversation
rclip's CLIP model only understands English, so non-English text queries used to search poorly. Add an optional "translate" extra (argostranslate) and a --lang [CODE] flag: pass a language code (or omit it to use the system locale's language) to force-translate the query to English, downloading that language's small NMT package on first use. Without --lang, a query only auto-translates if that language's package is already installed from a previous --lang run -- there's no implicit language detection and no surprise network calls or downloads on a plain search. --lang forces translation even for ASCII-only input, since many languages (Spanish, German, Italian, ...) are frequently written without any non-ASCII characters. An unknown language code gets "did you mean" suggestions from the package index instead of a bare failure.
Owner
There was a problem hiding this comment.
Hi! Thank you for proposing this! I was planning to add support for other languages in rclip, but I am not sure if this is the best direction we can take.
To understand this, I'll first need to benchmark this solution (both quality and speed) against:
sentence-transformers/clip-ViT-B-32-multilingual-v1siglip-base-patch16-256-i18n
These are both CLIP models, which handle multiple languages natively and are in the same "weight class" as the model rclip uses currently, making them good candidates to solve this problem.
If you can benchmark them, I'd appreciate it and might merge something sooner. Otherwise, I'll decide on the way forward once I've benchmarked them myself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
How does this PR impact the user?
rclip's CLIP model only understands English, so non-English text queries used to search poorly with no way to fix it. This adds an optional
translateextra and a--langflag:--lang CODE(ISO 639-1, e.g.es) force-translates the query to English, downloading that language's small NMT package (via argostranslate) on first use.--lang(no code) uses the system locale's language.--lang.--lang, nothing changes for existing users: no implicit language detection, no surprise network calls or downloads on a plain search.no translation package found for language "esp"; did you mean: es, pt, eo?.Description
rclip/translate.py(new) — language-package install/cache (ensure_language_installed,_has_installed_package) and the actual translate call (translate_to_english), plus the--langvalue resolution (resolve_forced_lang).rclip/model.py—Modeltakes an optionalforced_lang; phrase queries are run throughtranslate_to_englishbefore hitting the (English-only) CLIP text encoder. File-path and URL queries are never translated.rclip/main.py/rclip/utils/helpers.py— new--lang [CODE]CLI flag; the language package is installed eagerly at startup (clear failure up front, not mid-search).pyproject.toml— new optionaltranslateextra (argostranslate).Manually verified translation quality across 9 languages/scripts (Latin, Cyrillic, CJK, Arabic) before opening this PR:
Limitations
translateextra is not auto-installed if missing (only the language model is); doing so safely isn't possible across all of rclip's install methods (brew/snap/AppImage bundles have no writable,pip-able environment).--langfails fast with the install command instead.argostranslate's heavy dependency (ctranslate2) ships wheels for macOS (arm64/x86_64), Linux (manylinux x86_64/aarch64), and Windows (win_amd64) — verified on PyPI, not separately smoke-tested on Linux/Windows in this PR.Checklist