Skip to content

Add corpus-by-seed robustness matrix#12

Merged
almondsun merged 1 commit into
mainfrom
agent/corpus-by-seed-matrix
Jul 13, 2026
Merged

Add corpus-by-seed robustness matrix#12
almondsun merged 1 commit into
mainfrom
agent/corpus-by-seed-matrix

Conversation

@almondsun

Copy link
Copy Markdown
Owner

Summary

  • complete a preregistered 2-tokenizer × 2-corpus × 3-seed matrix with seven new fixed configs
  • add a balance-checked matrix analyzer with paired BPC contrasts and corpus interactions
  • move bounded run-artifact loading into the evaluation package and harden it against oversized or changing files
  • document milestone 025, factorial interpretation, validation-selection bias, and the sealed-test next step

Result

ByteBPE512 beats character in all six same-seed comparisons. Mean candidate-minus-reference BPC is -0.061863 on Alice and -0.025203 on Peter Pan. The +0.036660 corpus interaction shows robust direction but corpus-dependent magnitude.

Validation

  • exact 12-run matrix replay passed artifact, fingerprint, checksum, tokenizer, and seed-set validation
  • make PYTHON=.venv/bin/python check: 155 passed, 91.26% coverage
  • Ruff, strict mypy, compileall, Markdown links, and git diff --check passed
  • make PYTHON=.venv/bin/python audit: no known third-party vulnerabilities
  • direct CLI and oversized-artifact regression tests pass

@almondsun
almondsun marked this pull request as ready for review July 13, 2026 05:56
Copilot AI review requested due to automatic review settings July 13, 2026 05:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@almondsun
almondsun merged commit a79f4ee into main Jul 13, 2026
5 checks passed
@almondsun
almondsun deleted the agent/corpus-by-seed-matrix branch July 13, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants