Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cohere extension experiment to the paper : One Tokenizer to Rule Them All

you can support my work here

training run for the universal tokenizer finetune :

image
image image image
image image image image image image image image image image

alt text

alt text

alt text

alt text

alt text

About

biasing the universal tokenizer and an attempt to optimize compression rates in multilingual compression

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages