m2m-100 Model custom first token

[frederik-uni]

A solution would be to remove the tokenizer from the Translator and add it as an attribute to translate_batch & translate_batch_with_target_prefix.

Something like this would be usable. wrapping from in Arc<Mutex<?>> would be possible, but is not optimal in case it runs at the same time

impl Tokenizer for MyTokenizer {
    fn encode(&self, input: &str) -> anyhow::Result<Vec<String>> {
        let mut encoded = self.tokenizer.encode(input)?;
        encoded.insert(0, self.from.clone());
        Ok(encoded)
    }

    fn decode(&self, tokens: Vec<String>) -> anyhow::Result<String> {
        self.tokenizer.decode(tokens)
    }
}

import ctranslate2
import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.load("spm.128k.model")

source = ["__en__"] + sp.encode("Hello world!", out_type=str)
target_prefix = ["__de__"]

translator = ctranslate2.Translator("m2m_100_418m_ct2")
result = translator.translate_batch([source], target_prefix=[target_prefix])

output = sp.decode(result[0].hypotheses[0][1:])
print(output)

[jkawamoto]

Could you explain what m2m-100 model needs exactly? Do you mean that it needs to reset the tokenizer's state every translation?

Btw, you can use sys.Translator to implement custom flows.

[frederik-uni]

ctranslate usually takes tokens as input. m2m-100 or nllb use a custom first token to set the src lang
source = ["__en__"] + sp.encode("Hello world!", out_type=str). either a source_prefix would be great or a give translator as a arg.

[jkawamoto]

Thank you for the clarification. I understand the issue now. I’ll consider adding source_prefix argument.