eval: compare neural dynamics on frozen holdout - #12
Merged
Conversation
lipluscodex
commented
Aug 1, 2026
lipluscodex
left a comment
Member
Author
There was a problem hiding this comment.
自己レビュー: PASS(ブロッカーなし)
受入確認:
- protocol freeze commit
4f240ddに manifest、13 variants、development/holdout 分離、選択規則、停止規則、D1 holdout/gold/provenance、実行系とテストが入り、result JSON は未存在だったことを確認。 - freeze から head まで manifest、holdout fixture、gold、provenance は不変。結果コミットの
limit参照修正は凍結値 12 と9-node fixtureのため観測値に影響せず、head からの再生成で development/holdout JSON がともにバイト一致。 - development では5候補が gate 通過し、固定 tie-break で
budget-025を選択。全13 variants と失敗/tradeoff を保存。 - holdout は current と選択候補のみを一度評価。relation MRR 0.3611 -> 0.3254 の退行により固定 stop rule で不採用。
EngineConfig.activation_strategyの既定はcurrent_positive_additiveのまま。path 3/3、feedback isolation、D1 zero-write provenance を確認。- ローカル 34 tests PASS、GitHub CI SUCCESS、worktree clean。
判定区分:
- 今回: マージ可。実験の陰性結果を含め、Issue #11 の検証目的を満たす。
- 残存リスク: fixture は固定 Li+ wiki subset と feature-hashing encoder に限定され、一般 corpus や別 graph topology への一般化は未確認。
- 受容したトレードオフ: development winner は未見 holdout に一般化しなかったため、既定実装を変えず、失敗結果を再利用可能な artifact として残す。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11
結果観測前に固定した development / holdout と13 variantsを再現可能に比較し、全候補、diagnostics、Pareto選択を保存します。
holdoutを一度だけ開封した結果、選択候補はrelation退行で不採用とし、既定strategyを変更せず全tradeoffを記録します。