Skip to content

Add sealed chronological test evaluation#13

Merged
almondsun merged 1 commit into
mainfrom
agent/sealed-test-evaluation
Jul 13, 2026
Merged

Add sealed chronological test evaluation#13
almondsun merged 1 commit into
mainfrom
agent/sealed-test-evaluation

Conversation

@almondsun

Copy link
Copy Markdown
Owner

What changed

  • add backward-compatible chronological train/validation/test splitting
  • keep terminal test text out of tokenizer fitting, scoring, early stopping, and checkpoint selection
  • add a one-shot, full-coverage evaluator with manifest, corpus, summary, and checkpoint verification
  • add four frozen Alice/Peter Pan character/ByteBPE512 configurations
  • record milestone 026 theory, commands, limitations, and confirmatory results
  • harden copied dataset-manifest reads with bounded JSON-object validation

Why

Milestone 025 selected ByteBPE512 using validation evidence. This change creates a clearer information boundary and evaluates that frozen decision once on terminal held-out segments, without breaking existing two-way configurations.

Result

ByteBPE512 beats character test BPC by 0.0614 on Alice and 0.0258 on Peter Pan. All four test BPC values are worse than validation, so the report treats the terminal segments as consumed evidence and does not use them for further tuning.

Validation

  • legacy corpus/tokenizer/baseline/train/show/generate smoke pipeline
  • four frozen training runs and one-shot best-checkpoint evaluations
  • make PYTHON=.venv/bin/python check — 163 tests, 91.00% coverage
  • make PYTHON=.venv/bin/python audit — no known vulnerabilities
  • git diff --check

@almondsun
almondsun marked this pull request as ready for review July 13, 2026 06:46
Copilot AI review requested due to automatic review settings July 13, 2026 06:46
@almondsun
almondsun merged commit 3fafb54 into main Jul 13, 2026
5 checks passed
@almondsun
almondsun deleted the agent/sealed-test-evaluation branch July 13, 2026 06:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants