Skip to content

Add multi-seed robustness study#10

Merged
almondsun merged 1 commit into
mainfrom
agent/multi-seed-robustness
Jul 13, 2026
Merged

Add multi-seed robustness study#10
almondsun merged 1 commit into
mainfrom
agent/multi-seed-robustness

Conversation

@almondsun

Copy link
Copy Markdown
Owner

Summary

  • run the fixed ByteBPE512 early-stopping protocol across preregistered seeds 1337, 2027, and 4242
  • add a bounded aggregation CLI that verifies a canonical experiment fingerprint before comparing runs
  • report descriptive mean, population standard deviation, range, stop-step variation, and controlled generation diagnostics
  • harden run-summary/config parsing and training-seed validation

Results

  • best BPC: 2.022544 ± 0.012358 population SD
  • range: 2.008269–2.038414; all seeds beat the corrected character control 2.075981
  • best step range: 1750–2250; stop step range: 2500–3000
  • seed spread is about 49× milestone 022’s apparent weight-decay gain, confirming that delta was not decision-grade

Validation

  • uv run --frozen --extra dev make check (141 tests, 91.41% coverage)
  • uv run --frozen --extra dev make audit
  • two additional controlled training runs and best/final generation diagnostics
  • verified experiment fingerprint ed9650e463830d0d85dbbc32a40164b6699d7033017386a2cb22f2ab035431b0
  • correctness and security audits completed with all findings resolved

@almondsun
almondsun marked this pull request as ready for review July 13, 2026 04:06
Copilot AI review requested due to automatic review settings July 13, 2026 04:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@almondsun
almondsun merged commit fce19ef into main Jul 13, 2026
5 checks passed
@almondsun
almondsun deleted the agent/multi-seed-robustness branch July 13, 2026 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants