Skip to content

FEATURE: Add DeepSeek tokenizer - #11

Open
xfalcox wants to merge 1 commit into
mainfrom
add-deepseek-tokenizer
Open

FEATURE: Add DeepSeek tokenizer#11
xfalcox wants to merge 1 commit into
mainfrom
add-deepseek-tokenizer

Conversation

@xfalcox

@xfalcox xfalcox commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

  • add DeepSeekTokenizer backed by the DeepSeek-V4-Flash-0731 tokenizer
  • vendor the official Hugging Face tokenizer definition
  • register DeepSeek as an available LLM tokenizer and add coverage
  • bump the gem version to 1.0.0

Testing

  • bundle exec rspec
  • bundle exec rubocop
  • DeepSeek Unicode encode/decode round-trip check

Grubba27
Grubba27 approved these changes Aug 3, 2026
View reviewed changes

@Grubba27 Grubba27 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZogStriP ZogStriP ZogStriP approved these changes
@Grubba27 Grubba27 Grubba27 approved these changes

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xfalcox @ZogStriP @Grubba27