Support H-Net

[AndreasHolm]

Goal

Support for H-Net (https://github.com/goombalab/hnet)

Design decisions

TBD

Future Extensions

• Solve batching
• (Distilled models from Token-based to Byte-based)

Benefits

• Tokenizer-free modelling
• Adds a foundation for future byte-level models

Definition of Done

• Model train on test dataset
• Generation load model from checkpoint and generate from prompt
• Example small runnable generation example

Why Merge This?

• Immediate value unlocks byte-level training and generation