eval: benchmark graph retrieval on real Li+ corpus - #10
Merged
Conversation
lipluscodex
commented
Aug 1, 2026
lipluscodex
left a comment
Member
Author
There was a problem hiding this comment.
AI self-review
PR #10 head 5fbbcc8 を Issue #9 の acceptance criteria、commit順序、PR diff、固定result、production provenance、CIに対して確認し、PASSと判定した。
Acceptance criteria
- 実D1から12 wiki node / 26 mention edgeの弱連結fixtureをread-only取得: PASS
- direct_lookup / relation / negative_control各4件、計12件のgoldとsource URLを固定: PASS
- baseline / graphを同一corpus・encoder・queryで比較し、全体/cohort別MRR、Hit@3、rank deltaを出力: PASS
- one-hop 2件 / two-hop 2件のendpoint・edge type完全照合: PASS
- feedback前後のcredited / uncredited edgeと全case rankを比較: PASS
- 観測結果を支持 / 不支持 / 判定不能として固定し、不支持を隠さない: PASS
- benchmark結果をCI閾値へ合わせず、固定inputからの再生成一致として検証: PASS
Freeze / provenance audit
- commit befe245でfixture、gold、判定規則をresult追加前に固定
- commit 7b24977で初回resultを追加
- fixture SHA-256 b3b305aabb57803c2782c3998215e1cbcf9b5e6cdef0f641abc98520d4400cf9
- gold SHA-256 a10af7a1a4ec5f0ef66a228b05e14fe0160e4ff5f9b3209073f38fdb90028c71
- 再取得後もfixture / gold / resultは不変
- wiki-only captureと無関係だった#178のstale gapを5fbbcc8で除去し、known_gaps=[]へ修正
Observed result
- H1 supported: relation MRR 0.2583 -> 0.3833、3改善 / 0悪化
- H2 unsupported: negative control 2件悪化、MRR 1.0000 -> 0.7083
- H3 supported: explanation path 4/4一致
- H4 supported: credited edge 1本のみ1.0 -> 1.14、uncredited edge / non-target rank変更なし
- overall MRRは0.7528 -> 0.6972。graphが常にbaselineより良いという結論は支持しない
Verification
- Local: 25/25 tests PASS
- benchmark再生成: committed resultと一致
- GitHub CI run 30701752410: SUCCESS
- Merge state: CLEAN
- Diff check / worktree: clean
Scope deviation: なし。
Review partition:
- now: blockerなし
- later: H2不支持の一般的な改善検討をIssue #11として記録済み。固定goldへ過適合させず別versionで評価する
- accepted: D1は損失ありsnapshot、既定encoderはlearned embeddingではなく、本結果は固定Li+ subsetと現行MVPに限定する
Next step: squash mergeし、Closes #9でissueを自動closeする。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9
実 D1 から read-only で取得した12 node / 26 edgeのconnected fixtureと、初回結果を見る前に固定した12件のgold queryを追加します。
同一入力でbaseline / graph、説明経路、feedback isolationを再生成し、H1/H3/H4支持・H2不支持という観測結果を閾値調整せず保存します。