Skip to content

Add preregistered external corpus panel#16

Merged
almondsun merged 5 commits into
mainfrom
agent/external-corpus-panel
Jul 13, 2026
Merged

Add preregistered external corpus panel#16
almondsun merged 5 commits into
mainfrom
agent/external-corpus-panel

Conversation

@almondsun

Copy link
Copy Markdown
Owner

Summary

  • preregister a 2-corpus × 2-tokenizer × 3-seed sealed-test panel before corpus access
  • record verified Project Gutenberg provenance, a transparent Lincoln metadata amendment, all validation/test results, paired effects, interactions, checkpoint identities, and limitations
  • add a bounded sealed-matrix validator/CLI with fail-closed completeness, comparability, identity, and coverage checks
  • correct likelihood-comparison fingerprints so downstream generation-only controls do not create false training mismatches
  • update reviewer, architecture, training, experiment, Codex, and theory documentation

Result

ByteBPE512 beats the character control in all six preregistered same-seed comparisons. Mean candidate-minus-reference test BPC is -0.113426 ± 0.004068 on Art of War and -0.154315 ± 0.013002 on Lincoln. The mean Lincoln-minus-Art-of-War interaction is -0.040889 ± 0.017069, so the direction is robust while magnitude remains corpus-dependent.

Validation

  • uv run --frozen --extra dev make check
    • Ruff format/lint passed
    • mypy passed over 44 source files
    • 166 tests passed with 90.74% coverage
    • compileall passed
    • Markdown links resolve
  • exact 12-cell sealed-matrix aggregation passed
  • uv run --frozen --extra dev make audit: no known vulnerabilities; local smallm package skipped because it is not on PyPI
  • post-change security review: no blocking findings

Notes

The local runtime corpora, tokenizers, checkpoints, and run artifacts remain ignored. All accessed terminal test regions are consumed and must not guide later model selection.

@almondsun
almondsun marked this pull request as ready for review July 13, 2026 09:28
Copilot AI review requested due to automatic review settings July 13, 2026 09:28
@almondsun
almondsun merged commit 7f96e84 into main Jul 13, 2026
5 checks passed
@almondsun
almondsun deleted the agent/external-corpus-panel branch July 13, 2026 09:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants