About

A simple BPE tokeniser to learn the process of tokenisation and Byte-Pair-Encoding, which recursively learns possible tokens