vocabulary-pruning

Here are 3 public repositories matching this topic...

Kshao09 / Pruned-BPE-Tokenizer

Python/Cython implementations and experimental data for the Pruned BPE and DH-BPE publications.

tokenizer tokenization bpe byte-pair-encoding subword-tokenization pre-tokenization pruned-bpe vocabulary-pruning dh-bpe minimum-token-dp-encoding

Updated Sep 10, 2026
Python

osamabinIaggin / laya-micro

Star

Shrink Convai's Laya typed-decision model to run on single-board computers: BPE-merge-closed vocabulary pruning plus block-wise int8, evaluated on human-labelled robot commands with majority-class baselines.

raspberry-pi robotics quantization int8 model-compression laya onnx edge-ai vocabulary-pruning system-one

Updated Sep 22, 2026
Python

chamsechan / llm-vocab-pruner

Star

Extensible output-vocabulary pruning for decoder-only LLMs while preserving multilingual input embeddings. Tested on Qwen2/Qwen2.5 and Gemma 3.

machine-translation transformers pytorch extensible model-compression inference-optimization multilingual-nlp llm decoder-only qwen2 qwen2-5 gemma3 vocabulary-pruning

Updated Aug 12, 2026
Python

Add this topic to your repo

To associate your repository with the vocabulary-pruning topic, visit your repo's landing page and select "manage topics."

Learn more