Token ids generated instead of translation

[ahmedoumar]

Hey there, I hope you're doing fine.
when running the command: turj.translate
it returns the token ids instead of the actual translation?
(see the output below)
2022-07-07 10:41:43 | INFO | turjuman.translate | Using beam search
tensor([[ 0, 6538, 2, 76, 6380, 1]])

[elmadany]

Hi Ahmed,
could you please provide us with more details such as your input sentence and screenshot?
Thanks

[ahmedoumar]

as you can see the turj.translate returns output ids instead of translation, i have solved this by using the tokenizer and then decode the ids back to tokens:
tokenizer.decode(target, skip_special_tokens=True, clean_up_tokenization_spaces=True)

[elmadany]

To integrate Turjuman with your python code, take a look at this notebook.
https://colab.research.google.com/github/UBC-NLP/turjuman/blob/main/examples/Integrate_turjuman_with_your_code.ipynb
Thanks

[ahmedoumar]

when you run that notebook, you get only the target ids, as shown in the screenshot.

[elmadany]

Thanks Ahmed, we will check this soon

[kabapy]

quick fix
result = torj.tokenizer.batch_decode(target, skip_special_tokens=True)