According to the ReadMe to train bpe tokenizer, an error happens:
vocab_size = 2043, target 2048
vocab_size = 2044, target 2048
vocab_size = 2045, target 2048
vocab_size = 2046, target 2048
vocab_size = 2047, target 2048
end traning
vocab size: 2048
starting encoding train text to token ids
start persisting train tokens ids
starting encoding valid text to token ids
start persisting valid tokens ids
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/lkhm/Desktop/ai/llm-from-scratch/llm/bpe_tokenizer.py", line 234, in
inspect_data()
^^^^^^^^^^^^
NameError: name 'inspect_data' is not defined
According to the ReadMe to train bpe tokenizer, an error happens:
vocab_size = 2043, target 2048
vocab_size = 2044, target 2048
vocab_size = 2045, target 2048
vocab_size = 2046, target 2048
vocab_size = 2047, target 2048
end traning
vocab size: 2048
starting encoding train text to token ids
start persisting train tokens ids
starting encoding valid text to token ids
start persisting valid tokens ids
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/lkhm/Desktop/ai/llm-from-scratch/llm/bpe_tokenizer.py", line 234, in
inspect_data()
^^^^^^^^^^^^
NameError: name 'inspect_data' is not defined