Add sealed chronological test evaluation

AGENTS.md

@@ -90,11 +90,10 @@ If a relevant check cannot be run, state why and what remains unverified.
 
 ## Current Technical Status
 
-Milestone 025 is the latest modeling evidence. In a balanced 2-tokenizer × 2-corpus × 3-seed
-matrix, ByteBPE512 beats character in all six paired comparisons. Its mean advantage is `0.0619`
-BPC on Alice and `0.0252` on Peter Pan; the `+0.0367` corpus interaction means effect magnitude is
-not distribution-invariant. The next contract change should add a sealed chronological test segment
-before further model selection.
+Milestone 026 is the latest modeling evidence. Frozen 80/10/10 runs evaluated once on sealed
+terminal segments confirm ByteBPE512 over character by `0.0614` BPC on Alice and `0.0258` on Peter
+Pan. Test BPC is worse than validation for all four models. These test segments are consumed and
+must not guide further tuning; future modeling needs a new untouched evaluation distribution.
 
 ## Safety And Security
 

README.md

@@ -17,8 +17,8 @@ For a fast technical review, inspect:
 1. [`docs/architecture.md`](docs/architecture.md) for boundaries and the
    end-to-end pipeline.
 2. [`docs/experiments.md`](docs/experiments.md) for the milestone index.
-3. [`experiments/025-corpus-by-seed-matrix.md`](experiments/025-corpus-by-seed-matrix.md)
-   for the latest balanced robustness evidence and its limitations.
+3. [`experiments/026-sealed-test-evaluation.md`](experiments/026-sealed-test-evaluation.md)
+   for the latest confirmatory held-out evidence and its limitations.
 4. [`src/smallm/model/`](src/smallm/model/) for the from-scratch GPTiny model.
 5. [`src/smallm/training/`](src/smallm/training/) for training and run artifacts.
 6. [`src/smallm/data/`](src/smallm/data/) for corpus and tokenizer contracts.
@@ -50,6 +50,7 @@ chronological 90/10 split unless noted otherwise.
 | [023](experiments/023-multi-seed-robustness.md) | ByteBPE512 across three seeds | Best BPC mean ± population SD | `2.0225 ± 0.0124` | All tested seeds beat the character control; stopping varies from 2500–3000 steps. |
 | [024](experiments/024-cross-corpus-robustness.md) | Peter Pan character vs ByteBPE512 | Full-validation best bits/character | `2.1721` vs `2.1539` | ByteBPE512 replicates the direction on a second book, but only by 0.83%. |
 | [025](experiments/025-corpus-by-seed-matrix.md) | 2 tokenizers × 2 corpora × 3 seeds | Paired ByteBPE512 minus character BPC | `-0.0619` Alice; `-0.0252` Peter Pan | ByteBPE512 wins all six pairs; effect magnitude is corpus-dependent. |
+| [026](experiments/026-sealed-test-evaluation.md) | Frozen decision on terminal 10% test segments | ByteBPE512 minus character test BPC | `-0.0614` Alice; `-0.0258` Peter Pan | The tokenizer decision survives one-shot full-coverage tests on both books. |
 
 Token-level loss and perplexity are not directly comparable between character
 and BPE tokenizers because they predict different units. The tokenizer
@@ -84,7 +85,7 @@ shapes link directly to the implementation and its tests.
 | Corpus preparation | Normalized corpus output, stats, checksums, source metadata, and manifest files. |
 | Tokenization | Character, educational character-BPE, and lossless boundary-aware UTF-8 byte-BPE tokenizers. |
 | Model | Decoder-only GPT-style Transformer with causal self-attention. |
-| Evaluation | Uniform, unigram, and add-one smoothed bigram baselines. |
+| Evaluation | Uniform, unigram, and add-one bigram baselines; optional sealed chronological test evaluation. |
 | Training | Config-driven training with validation loss, optional early stopping, progress logging, checkpoints, metrics, summaries, and samples. |
 | Run records | Preserved run directories with copied dataset manifests and selected provenance fields in `summary.json`. |
 | Generation | `max_new_tokens`, `temperature`, `top_k`, `seed`, and greedy decoding. |

configs/gptiny_alice_bytebpe512_sealed.yaml

@@ -0,0 +1,39 @@
+data:
+  input_path: data/raw/input.txt
+  prepared_path: data/processed/corpus.txt
+  manifest_path: data/processed/corpus_sealed_manifest.json
+  tokenizer_path: data/processed/tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_alice_bytebpe512_sealed
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: Once
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337

configs/gptiny_alice_char_sealed.yaml

@@ -0,0 +1,37 @@
+data:
+  input_path: data/raw/input.txt
+  prepared_path: data/processed/corpus.txt
+  manifest_path: data/processed/corpus_sealed_manifest.json
+  tokenizer_path: data/processed/tokenizer_char_sealed.json
+  tokenizer_type: char
+  block_size: 64
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 256
+  block_size: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_alice_char_sealed
+  runs_dir: runs
+  batch_size: 16
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: Once
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337

configs/gptiny_peterpan_bytebpe512_sealed.yaml

@@ -0,0 +1,39 @@
+data:
+  input_path: data/raw/peter_pan_body.txt
+  prepared_path: data/processed/peter_pan_corpus.txt
+  manifest_path: data/processed/peter_pan_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/peter_pan_tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_peterpan_bytebpe512_sealed
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: Once
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337

configs/gptiny_peterpan_char_sealed.yaml

@@ -0,0 +1,37 @@
+data:
+  input_path: data/raw/peter_pan_body.txt
+  prepared_path: data/processed/peter_pan_corpus.txt
+  manifest_path: data/processed/peter_pan_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/peter_pan_tokenizer_char_sealed.json
+  tokenizer_type: char
+  block_size: 64
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 256
+  block_size: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_peterpan_char_sealed
+  runs_dir: runs
+  batch_size: 16
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: Once
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337

docs/architecture.md

@@ -21,6 +21,9 @@ flowchart TD
     H --> I[checkpoint.pt]
     H --> J[best_checkpoint.pt]
     H --> K[metrics.jsonl + summary.json]
+    J --> O[One-shot sealed test evaluator]
+    B -. terminal split .-> O
+    O --> P[test_evaluation_best.json]
     I --> L[Generation + diagnostics]
     J --> L
     K --> M[Experiment reports]
@@ -35,8 +38,8 @@ block construction.
 `model/` owns the GPT network: embeddings, causal self-attention, Transformer
 blocks, final normalization, and language-model head.
 
-`evaluation/` owns simple character-level reference models used to interpret
-validation loss.
+`evaluation/` owns simple reference models, bounded run observations, and shared evaluation
+contracts used to interpret validation and sealed-test loss.
 
 `training/` owns orchestration: dataloaders, optimization, evaluation intervals,
 progress logging, checkpoints, run directories, metrics, summaries, and copied
@@ -82,6 +85,11 @@ blocks and weights NLL by target count; summaries disclose full or sampled cover
 a complete UTF-8 byte fallback, forbids merges across whitespace boundaries, and preserves aligned
 Unicode-character completion counts for exact BPC.
 
+An optional explicit validation fraction activates a chronological train/validation/test split.
+Training does not encode the terminal test text. After checkpoint selection, the sealed evaluator
+verifies corpus and checkpoint identity, evaluates full coverage once, and refuses to overwrite its
+result artifact.
+
 ## Run Artifacts
 
 Each training run writes:
@@ -95,6 +103,7 @@ Each training run writes:
 | `best_checkpoint.pt` | Same checkpoint payload captured at the best validation step, when validation is available. |
 | `sample.txt` | Text generated at the end of training. |
 | `dataset_manifest.json` | Copied corpus manifest for run-local provenance. |
+| `test_evaluation_best.json` | Optional post-training sealed-test result with corpus and checkpoint hashes. |
 
 Run utilities resolve explicit run paths and `latest` by run name.
 

docs/codex/architecture.md

@@ -35,7 +35,7 @@ Core logic is deterministic or tensor-oriented package code:
   inspectable JSON state.
 - `smallm.data.byte_bpe_tokenizer`: lossless UTF-8 byte fallback, boundary-aware merges, and
   character-aligned evaluation counts.
-- `smallm.data.dataset`: train/validation split and shifted token blocks.
+- `smallm.data.dataset`: backward-compatible train/validation/test split and shifted token blocks.
 - `smallm.model`: causal attention, Transformer blocks, GPT forward/loss.
 - `smallm.evaluation`: baseline losses and perplexity.
 - `smallm.generation`: decoding controls over model logits.
@@ -59,6 +59,8 @@ policy explicit in config and summary fields.
 Scripts under `scripts/` are CLI adapters. They should parse arguments, load
 config/checkpoints, call package code, and print results. Do not put reusable
 domain logic in scripts when it belongs under `src/smallm/`.
+`scripts/evaluate_test.py` is the post-selection edge: it verifies run-local provenance and writes
+the one-shot sealed-test artifact without exposing test data to training orchestration.
 
 ## Module Boundaries
 
@@ -81,6 +83,7 @@ domain logic in scripts when it belongs under `src/smallm/`.
 
 - baseline probability models
 - validation loss and perplexity calculations
+- exact full-coverage sealed-test evaluation results
 
 `training/` owns:
 

docs/codex/build-and-test.md

@@ -50,7 +50,7 @@ The individual commands remain canonical and are listed below.
 python -m pytest
 ```
 
-Current expected result after milestone 025+: at least 155 tests passing with at least 90% coverage.
+Current expected result after milestone 026+: at least 163 tests passing with at least 90% coverage.
 
 ### Compile Check
 
@@ -136,6 +136,16 @@ python scripts/generate.py --run latest --run-name gptiny --prompt "Once" --gree
 python scripts/generate.py --run latest --run-name gptiny --prompt "Once" --temperature 0.8 --top-k 10 --seed 1337 --max-new-tokens 100
 ```
 
+### Sealed Test Evaluation
+
+For an explicitly configured three-way split, evaluate only after the checkpoint decision is frozen:
+
+```bash
+python scripts/evaluate_test.py --run runs/<name>/<id> --checkpoint-kind best
+```
+
+Do not rerun, delete, or use the resulting test artifact for model selection.
+
 ## Validation Matrix
 
 | Change type | Minimum validation |
@@ -147,6 +157,7 @@ python scripts/generate.py --run latest --run-name gptiny --prompt "Once" --temp
 | Model code | `python -m pytest`, compileall. |
 | Baselines | `python -m pytest`, `evaluate_baselines.py`, compileall. |
 | Training/artifacts/runs | smoke or tiny training path, `show_run.py`, `python -m pytest`, compileall. |
+| Split or sealed-test contract | Legacy smoke plus three-way fixture, one-shot evaluator, `python -m pytest`, compileall. |
 | Generation | greedy and seeded top-k generation commands, `python -m pytest`, compileall. |
 | Experiment report | Commands claimed in the report, plus tests and compileall when code changed. |
 
@@ -165,6 +176,8 @@ current task or explicitly quoted from a prior report.
 
 ## Current Milestone Status
 
-Milestone 019 is the current validation contract: frozen dependencies, Ruff, strict mypy, at
-least 90% coverage, compile and link checks, a smoke pipeline, and corrected full-validation
-character/BPE evidence.
+Milestone 026 is the current evaluation contract: legacy two-way splits remain compatible, while
+explicit validation fractions reserve a chronological test region that training leaves unencoded
+and unscored.
+Best-checkpoint test evaluation verifies corpus/checkpoint identity, requires full coverage, and
+refuses artifact overwrite.

docs/codex/experiments.md

@@ -170,3 +170,8 @@ Milestone 025 completes the balanced corpus-by-seed matrix. ByteBPE512 beats cha
 same-seed pairs. Mean candidate-minus-reference BPC is `-0.0619` on Alice and `-0.0252` on Peter
 Pan; the corpus interaction is `+0.0367`, so report a robust direction but corpus-dependent size.
 The next contract change should introduce a sealed test segment before further model selection.
+
+Milestone 026 introduces an optional `validation_split`; explicit 80/10/10 configs keep the final
+region unavailable to tokenizer fitting, training, early stopping, and checkpoint selection.
+One-shot best-checkpoint test evaluation confirms ByteBPE512 by `0.0614` BPC on Alice and `0.0258`
+on Peter Pan. Those test segments are consumed and must not guide further tuning.

docs/experiments.md

@@ -32,6 +32,7 @@ not a replacement for the original reports.
 | [023 Multi-Seed Robustness](../experiments/023-multi-seed-robustness.md) | Robustness | Three preregistered seeds average best BPC `2.0225 ± 0.0124`; every seed beats the corrected character control. |
 | [024 Cross-Corpus Robustness](../experiments/024-cross-corpus-robustness.md) | External validity | On near-size-matched Peter Pan, ByteBPE512 narrowly beats character at `2.1539` versus `2.1721` BPC. |
 | [025 Corpus-by-Seed Matrix](../experiments/025-corpus-by-seed-matrix.md) | Factorial robustness | ByteBPE512 wins all six paired comparisons; mean advantage is `0.0619` BPC on Alice and `0.0252` on Peter Pan. |
+| [026 Sealed Test Evaluation](../experiments/026-sealed-test-evaluation.md) | Confirmatory evaluation | On untouched terminal segments, ByteBPE512 beats character by `0.0614` BPC on Alice and `0.0258` on Peter Pan. |
 
 ## Topic Shortcuts
 
@@ -58,7 +59,8 @@ not a replacement for the original reports.
   [020](../experiments/020-bpe-context-and-learning-rate.md),
   [021](../experiments/021-boundary-aware-byte-bpe.md),
   [024](../experiments/024-cross-corpus-robustness.md),
-  [025](../experiments/025-corpus-by-seed-matrix.md).
+  [025](../experiments/025-corpus-by-seed-matrix.md),
+  [026](../experiments/026-sealed-test-evaluation.md).
 
 ## Current Status
 
@@ -98,3 +100,8 @@ advantage; a corpus-by-seed matrix is the next stronger test.
 Milestone 025 completes that balanced matrix. ByteBPE512 beats character for seeds 1337, 2027, and
 4242 on both Alice and Peter Pan. Its paired mean advantage is `0.0619` BPC on Alice and `0.0252`
 on Peter Pan; the `+0.0367` BPC interaction shows that effect magnitude remains corpus-dependent.
+
+Milestone 026 freezes that decision and evaluates new 80/10/10 runs once on terminal test segments.
+ByteBPE512 reaches test BPC `2.1178` versus `2.1792` on Alice and `2.2484` versus `2.2742` on Peter
+Pan. The direction and approximate margins survive, while every model's test BPC is worse than its
+validation BPC.