Skip to content

eval: benchmark graph retrieval on real Li+ corpus - #10

Merged
lipluscodex merged 3 commits into
mainfrom
9-real-corpus-benchmark
Aug 1, 2026
Merged

eval: benchmark graph retrieval on real Li+ corpus#10
lipluscodex merged 3 commits into
mainfrom
9-real-corpus-benchmark

Conversation

@lipluscodex

Copy link
Copy Markdown
Member

Closes #9

実 D1 から read-only で取得した12 node / 26 edgeのconnected fixtureと、初回結果を見る前に固定した12件のgold queryを追加します。
同一入力でbaseline / graph、説明経路、feedback isolationを再生成し、H1/H3/H4支持・H2不支持という観測結果を閾値調整せず保存します。

@lipluscodex lipluscodex linked an issue Aug 1, 2026 that may be closed by this pull request
@lipluscodex lipluscodex self-assigned this Aug 1, 2026

@lipluscodex lipluscodex left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI self-review

PR #10 head 5fbbcc8 を Issue #9 の acceptance criteria、commit順序、PR diff、固定result、production provenance、CIに対して確認し、PASSと判定した。

Acceptance criteria

  • 実D1から12 wiki node / 26 mention edgeの弱連結fixtureをread-only取得: PASS
  • direct_lookup / relation / negative_control各4件、計12件のgoldとsource URLを固定: PASS
  • baseline / graphを同一corpus・encoder・queryで比較し、全体/cohort別MRR、Hit@3、rank deltaを出力: PASS
  • one-hop 2件 / two-hop 2件のendpoint・edge type完全照合: PASS
  • feedback前後のcredited / uncredited edgeと全case rankを比較: PASS
  • 観測結果を支持 / 不支持 / 判定不能として固定し、不支持を隠さない: PASS
  • benchmark結果をCI閾値へ合わせず、固定inputからの再生成一致として検証: PASS

Freeze / provenance audit

  • commit befe245でfixture、gold、判定規則をresult追加前に固定
  • commit 7b24977で初回resultを追加
  • fixture SHA-256 b3b305aabb57803c2782c3998215e1cbcf9b5e6cdef0f641abc98520d4400cf9
  • gold SHA-256 a10af7a1a4ec5f0ef66a228b05e14fe0160e4ff5f9b3209073f38fdb90028c71
  • 再取得後もfixture / gold / resultは不変
  • wiki-only captureと無関係だった#178のstale gapを5fbbcc8で除去し、known_gaps=[]へ修正

Observed result

  • H1 supported: relation MRR 0.2583 -> 0.3833、3改善 / 0悪化
  • H2 unsupported: negative control 2件悪化、MRR 1.0000 -> 0.7083
  • H3 supported: explanation path 4/4一致
  • H4 supported: credited edge 1本のみ1.0 -> 1.14、uncredited edge / non-target rank変更なし
  • overall MRRは0.7528 -> 0.6972。graphが常にbaselineより良いという結論は支持しない

Verification

  • Local: 25/25 tests PASS
  • benchmark再生成: committed resultと一致
  • GitHub CI run 30701752410: SUCCESS
  • Merge state: CLEAN
  • Diff check / worktree: clean

Scope deviation: なし。

Review partition:

  • now: blockerなし
  • later: H2不支持の一般的な改善検討をIssue #11として記録済み。固定goldへ過適合させず別versionで評価する
  • accepted: D1は損失ありsnapshot、既定encoderはlearned embeddingではなく、本結果は固定Li+ subsetと現行MVPに限定する

Next step: squash mergeし、Closes #9でissueを自動closeする。

@lipluscodex
lipluscodex merged commit 038ad2e into main Aug 1, 2026
1 check passed
@lipluscodex
lipluscodex deleted the 9-real-corpus-benchmark branch August 1, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eval: benchmark graph retrieval on real Li+ corpus

1 participant