Add boundary-aware byte BPE milestone

[almondsun]

Summary

• add lossless UTF-8 byte BPE with whitespace-boundary-aware merges and exact Unicode-aligned BPC accounting
• harden tokenizer/checkpoint validation and add controlled ByteBPE320/512 configs
• record milestone 021 results, generation diagnostics, and theory documentation

Results

• ByteBPE320 best BPC: 2.028567 at step 3250
• ByteBPE512 best BPC: 2.008269 at step 1750
• corrected character control: 2.075981
• corrected BPE128 control: 2.097552

ByteBPE512 is the strongest corrected result but overfits to final BPC 2.244955.

Validation

• uv run --frozen --extra dev make check (107 tests, 91.00% coverage)
• uv run --frozen --extra dev make audit
• both full 5000-step experiments and full-validation summaries
• best/final greedy and seeded generation diagnostics
• focused post-change correctness and security audits; all findings resolved

[Copilot]

Pull request overview

Adds a new boundary-aware, lossless UTF-8 byte-level BPE tokenizer and threads exact Unicode-aligned character accounting into evaluation/training so BPC remains comparable across tokenizers. This fits the repo’s tokenizer-experiment “milestone” workflow by adding both implementation and the accompanying configs/tests/docs/results for milestone 021.

Changes:

• Introduces ByteBPETokenizer (whitespace-boundary-aware merges, byte fallback, per-token Unicode character completion counts).
• Updates training/evaluation to optionally consume precomputed per-token character counts for exact BPC accounting.
• Adds new configs + tests + documentation/reporting for the ByteBPE320/512 milestone.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/test_training_eval.py	Adds regression coverage for Unicode-aligned character counting in evaluate_tokens.
tests/test_training_artifacts.py	Adds checkpoint validation tests for byte_bpe tokenizer state.
tests/test_tokenizer.py	Adds ByteBPE round-trip, boundary, fallback, and character-count contract tests.
tests/test_config.py	Extends config validation tests for byte-BPE vocab/min-frequency constraints.
src/smallm/training/trainer.py	Adds optional character_counts support in evaluation and wires encoding-with-counts into training.
src/smallm/training/checkpoints.py	Adds tokenizer-state validation support for byte_bpe checkpoints.
src/smallm/data/tokenizer.py	Extends tokenizer factory + artifact loading to support byte_bpe; adds counts API to CharTokenizer.
src/smallm/data/byte_bpe_tokenizer.py	New ByteBPE tokenizer implementation (training, encode/decode, state, save, character counts).
src/smallm/data/bpe_tokenizer.py	Tightens min_frequency validation and adds encode_with_character_counts API.
src/smallm/data/init.py	Exposes ByteBPETokenizer from the smallm.data package.
src/smallm/config.py	Extends DataConfig validation for byte_bpe and hardens BPE limits/min-frequency constraints.
scripts/prepare_data.py	Updates CLI messaging to label the selected tokenizer type including byte_bpe.
README.md	Updates project overview + milestone index to include boundary-aware ByteBPE and milestone 021.
notes/02-data-and-tokenization.md	Documents boundary-aware byte BPE and exact Unicode-aligned BPC accounting approach.
experiments/021-boundary-aware-byte-bpe.md	New milestone 021 report with results, commands, and limitations.
docs/training.md	Updates “current status” narrative to include milestone 021 results and interpretation.
docs/experiments.md	Adds milestone 021 to the experiments index and tokenization shortcut list.
docs/codex/experiments.md	Updates agent-facing experiment summary with milestone 021 results and next question.
docs/codex/architecture.md	Updates architecture overview to include byte BPE tokenizer module.
docs/architecture.md	Updates pipeline diagram and evaluation explanation for byte-BPE character-aligned accounting.
configs/gptiny_bytebpe512_5k_lr1e-3_ctx37.yaml	Adds controlled ByteBPE512 experiment config.
configs/gptiny_bytebpe320_5k_lr1e-3_ctx46.yaml	Adds controlled ByteBPE320 experiment config.
AGENTS.md	Updates “Current Technical Status” to milestone 021 as latest evidence.

Comments suppressed due to low confidence (1)

src/smallm/training/trainer.py:110

• When character_counts is provided, evaluate_tokens assumes it is aligned 1:1 with tokens. If a caller accidentally passes a tensor with the wrong length, BPC will be silently wrong. Add a defensive length check early to fail fast.

    character_counts: torch.Tensor | None = None,
) -> EvaluationResult | None:
    if tokens.numel() < 2:
        return None
    was_training = model.training
    model.eval()
    starts = _validation_starts(tokens.numel(), block_size, max_batches)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.