Repository navigation
feat: index GitHub Wiki content into RAG (W4, patch) - #129
Merged
liplus-lin-lay merged 1 commit intoApr 25, 2026
Merged
Conversation
Adds a new ingestion surface for GitHub Wiki pages so the unified `search`
tool can return wiki content alongside repository docs / issues / PRs / etc.
Driven by the liplus-language move toward wiki-only judgment records — without
this, migrated `docs/[a-z].-*.md` decision-log entries would disappear from
RAG search after migration.
Design choices (locked in advance with Master):
- Coverage: every text-based wiki extension supported by GitHub (md /
markdown / mediawiki / org / rst / rest / textile / pod / asciidoc / creole)
- Type representation: new `wiki_doc` type (parallel to existing `doc`,
not a sub-flavour of it; same-name page in both surfaces returns two hits)
- Repo selection: every repo in POLL_REPOS automatically; wiki existence
detected per-repo by probing `{repo}.wiki.git/info/refs?service=git-upload-pack`
(200 = ingest, 404 = skip)
- Cron: independent slot at `:45 * * * *` (siblings: light :00 / comments
:15 / diffs :30) so the new surface gets its own per-Worker subrequest budget
- Conflict policy: docs/ and wiki are both indexed and surfaced; the row's
`type` field discriminates
- Page enumeration: HTML-scrape `/{repo}/wiki/_pages` (no public REST API
for wiki content); per-page extension probed by URL fallback chain and
cached on the WikiDocRecord for subsequent polls
- Change detection: SHA-256 of raw content (no git blob SHA without the wiki
git smart-HTTP protocol)
Worker side:
- src/types.ts — new WikiDocRecord, type union extension, wiki_path /
wiki_extension VectorMetadata fields
- src/store.ts — wiki_docs SQLite table + idempotent CREATE in
initSchema + upsert / get / list / recent / delete
accessors + HTTP handlers (/upsert-wiki-doc /
wiki-doc / wiki-docs / recent-wiki-docs)
- src/pipeline.ts — wikiDocVectorId (prefix "w") + processAndUpsertWikiDoc
+ sha256Hex helper exported for poller reuse
- src/poller.ts — wikiExists / listWikiPages / fetchWikiContent /
pollWiki + WIKI_CRON dispatch + runWikiSurface
- src/mcp.ts — type filter enum extended to include "wiki_doc";
scan-mode wiki block; search-mode metadata mapping
(wiki_path / wiki_extension); URL builder branch
producing https://github.com/{repo}/wiki/{page};
title enrichment for wiki rows; inlineDocContent
extended to fetch wiki raw markup (parallel to docs)
- wrangler.toml — crons append "45 * * * *"
Schema impact:
- D1 search_docs reuses existing columns (no migration); rows carry
`type='wiki_doc'` and use `doc_path` to store the wiki page slug.
- DO IssueStore adds wiki_docs table on next access (CREATE IF NOT EXISTS,
no migration needed; existing data preserved).
- Vectorize accepts the new metadata fields without schema change.
Docs:
- docs/0-requirements{.md,.ja.md} — Cron Poller section bumped to four
triggers, dedicated wiki-poller subsection, type enum expanded,
Structured State Store inventory updated
- README{.md,.ja.md}, mcp-server/README.md, mcp-server/manifest.json —
surface coverage + tool description updated
Tests:
- tsc --noEmit on src/ passes (exit 0)
- mcp-server `node --check server/index.js` passes
- No new automated tests added — wiki ingestion is integration-bound (live
HTML scrape + raw content fetch) and exercised in production after deploy
v0.x patch under "Anything may change" — additive new surface, existing
ingestion paths untouched.
Closes #128
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
github-rag-mcp | 5b05bd0 | Apr 25 2026, 05:33 PM |
liplus-lin-lay
commented
Apr 25, 2026
liplus-lin-lay
left a comment
liplus-lin-lay
left a comment
Member
Author
There was a problem hiding this comment.
AI 自己レビュー (auto, patch)
仕様適合
- #128 の design choice 10 軸すべて反映確認 (拡張子全種 / 新 type / 全 repo 自動 / 独立 cron / 両 index / raw fetch / wiki URL / smart-HTTP probe / patch / W4 後別 PR)
- target_files 全 9 ファイルすべて touched
- 既存 doc / issue / PR / release / diff / comment ingestion 経路は完全に無修正、追加のみ
内容整合
- D1 schema は migration なしで
type='wiki_doc'row を受け入れる前提を確認 (既存search_docsテーブルは type 列に CHECK 制約なし) - DO
wiki_docsテーブルはCREATE TABLE IF NOT EXISTS、既存 instance の next access で created、既存データ無破壊 tokenizerKindForType既定nat経路に乗るため新 type も問題なし- Vectorize metadata は keys 自由なので新
wiki_path/wiki_extension受け入れ可 - vector ID prefix は
w:で既存d:/r:/c:/ic:/pv:/pc:と衝突なし
TypeScript / build
tsc --noEmit -p tsconfig.jsonexit 0- mcp-server
node --check server/index.jspass - Cloudflare Workers Builds CI も pass
自己リスク申告
- HTML scrape の脆弱性:
/{repo}/wiki/_pagesの HTML 構造は GitHub UI 変更に依存。link href shape が変わると enumeration が静かに 0 件に落ちる。tolerant regex で吸収狙いだが、long-term には isomorphic-git or 公式 API への移行が望ましい (現時点では未実装) - 拡張子 probe コスト: 初回 page は 10 拡張子試行 (404 確認まで)、bulk import 時に GitHub raw API への fan-out が大きい。次回以降は cached extension を使うので 1 fetch に収まる
- change detection の精度: SHA-256 content hash だが whitespace / trailing newline 揺れには敏感、git のように tree 整合チェックではない。実用上問題ないが、過剰再 embed の可能性あり
- 新規 unit test 不足: integration-bound と判断したが、
listWikiPagesの HTML parse ロジックやwikiExistsの status 検出は単体テスト可。後続で追加候補 - Worker 未 deploy: Cloudflare Workers Builds CI は pass したが、本番 deploy 後に `pollWiki` が初回 cron で正常動作するかは production 観測待ち
判定
auto patch self-review 承認、gh pr merge --squash で進める。本番 deploy 後の最初の :45 cron で wiki 取り込みが動くかを観測する。失敗時は webhook event か log で検知できる構造。
liplus-lin-lay
deleted the
128-feat-index-github-wiki-content-into-rag-w4
branch
April 25, 2026 17:35
This was referenced Apr 26, 2026
Merged
This was referenced May 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
GitHub Wiki コンテンツを github-rag-mcp の RAG index に取り込む新規 ingestion 経路を追加。`mcp__github-rag-mcp__search` で wiki ページが他 surface と同列に検索結果に出る。Liplus-Project/liplus-language で進行中の判断記録 wiki 寄せ運用の前提条件。
設計判断（事前 lock 済み）
変更内容
Worker side:
Schema impact:
Docs:
テスト
既存ingestion経路への影響
ゼロ。すべて additive:
バージョン
patch (v0.8.4 milestone)。v0.x 初期開発期で additive なので minor bump せず patch 据え置き。
関連