installation

github-rag-mcp Installation Guide

Language: English | Japanese

Prerequisites

• Cloudflare account
• GitHub account with access to the target repositories
• Node.js 18+
• npm
• Wrangler CLI

Install Wrangler if needed:

npm install -g wrangler

1. Clone and install dependencies

git clone https://github.com/Liplus-Project/github-rag-mcp.git
cd github-rag-mcp
npm install

2. Log in to Cloudflare

wrangler login

3. Create Cloudflare resources

3.1 Vectorize index

wrangler vectorize create github-rag-issues --dimensions 1024 --metric cosine

3.2 Metadata indexes

Create metadata indexes before relying on structured filters.

wrangler vectorize create-metadata-index github-rag-issues --type string --property-name repo
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name type
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name state
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name milestone
# Expanded label/assignee fields (stored for future Vectorize OR-filter support)
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name label_0
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name label_1
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name label_2
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name label_3
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name assignee_0
wrangler vectorize create-metadata-index github-rag-issues --type string --property-name assignee_1

3.3 KV namespace

wrangler kv namespace create OAUTH_KV

Update wrangler.toml with the returned namespace ID.

3.4 D1 database (FTS5 sparse side of hybrid retrieval)

Create a D1 database for the BM25 / FTS5 sparse index.

wrangler d1 create github-rag-fts

Update wrangler.toml with the returned database_id:

[[d1_databases]]
binding = "DB_FTS"
database_name = "github-rag-fts"
database_id = "<paste-the-id-here>"
migrations_dir = "migrations"

Apply the initial migration (creates search_docs + the two FTS5 virtual tables):

wrangler d1 migrations apply github-rag-fts

For the first deploy against a brand-new D1 database, also run the same command with --remote:

wrangler d1 migrations apply github-rag-fts --remote

4. Create the GitHub App

Create a GitHub App for OAuth and repository access.

Recommended settings:

Field	Value
Homepage URL	https://<your-worker>.workers.dev
Callback URL	https://<your-worker>.workers.dev/oauth/callback
Webhook URL	https://<your-worker>.workers.dev/webhooks/github
Webhook active	enabled

Recommended repository permissions:

• Issues: read
• Pull requests: read
• Checks: read
• Commit statuses: read
• Contents: read
• Metadata: read

Subscribe to these events:

• Issues
• Pull requests
• Push
• Release

Install the app on the repositories you want to track.

5. Configure secrets

Set these secrets in Cloudflare:

• GITHUB_CLIENT_ID
• GITHUB_CLIENT_SECRET
• GITHUB_TOKEN
• GITHUB_WEBHOOK_SECRET

Example:

echo "<client-id>" | wrangler secret put GITHUB_CLIENT_ID
echo "<client-secret>" | wrangler secret put GITHUB_CLIENT_SECRET
echo "<github-token>" | wrangler secret put GITHUB_TOKEN
echo "<webhook-secret>" | wrangler secret put GITHUB_WEBHOOK_SECRET

6. Configure variables

Set POLL_REPOS to a comma-separated list of repositories.

Example:

[vars]
POLL_REPOS = "owner/repo1,owner/repo2"

7. Deploy the worker

wrangler deploy

8. Verify the deployment

Check the following:

• OAuth callback works
• webhook deliveries return success
• cron runs appear in Cloudflare logs
• the MCP endpoint is reachable

Recommended verification flow:

1. Open the worker URL and complete OAuth.
2. Edit an issue in a tracked repository.
3. Confirm a webhook delivery reaches the worker.
4. Confirm the record appears in search results.

9. Re-index if metadata filtering was added later

If metadata indexes were created after vectors already existed, reset stored hashes so the next cron run re-embeds everything.

Admin endpoint:

POST /admin/reset-hashes?repo=owner/repo

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

10. Replay a commit-diff period (gap backfill)

If commit diffs are missing for a period — an older poller version dropped them, or a commit keeps failing and holds the watermark — rewind the diff poller watermark. The next :30 cron re-covers the period from since, walking it oldest-first without skipping.

Admin endpoint:

POST /admin/diff-watermark?repo=owner/repo&since=2026-07-06T00:00:00Z

Parameters:

• repo — owner/repo, must be listed in POLL_REPOS
• since — any parseable timestamp; stored as the watermark the next run resumes from
• phase — forward (default) rewinds diffs:{repo}; backfill moves diffs_backfill:{repo} (used to release a stalled historical walk)

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

Operational notes:

• catch-up rate is 5 commits per repo per cron run (~120 commits/day at the hourly :30 trigger), so a multi-week gap takes several days to drain
• new commits are unaffected while a replay is in flight: the webhook path indexes them in real time
• upserts are idempotent on (repo, commit_sha, file_path), so re-covering an already-indexed period is safe
• verify progress with the {repo} diffs: forward [...] line in the worker logs, or by searching type: "diff" for the period

11. Re-segment the full-text index after applying migration 0006

Migration 0006_fts5_segmented_nat_index.sql adds the content_fts column and seeds it with a copy of the raw content. The actual word segmentation only exists in JavaScript (Intl.Segmenter), so rows indexed before the migration stay unsegmented until this endpoint walks them. Until it finishes, Japanese phrase queries keep returning zero sparse candidates for those rows.

Run it once after applying the migration. Rows indexed after the deploy are already segmented by the ingest path.

Order the upgrade as apply migration → deploy the worker → run this backfill. Between the migration and the deploy, the still-running previous version writes no content_fts, so rows it indexes in that window land in the v3 index as empty text; the backfill rewrites them from the raw content, so the window heals itself. The reverse order does not work: a worker deployed before the migration hits no such column: content_fts on every upsert.

Admin endpoint:

POST /admin/backfill-fts-segments

Parameters:

• repo — optional owner/repo filter; omit to walk every repository
• cursor — rowid to resume from (default 0); pass back the nextCursor of the previous response
• limit — rows per call, 1..200 (default 50)

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

Response:

{ "repo": null, "cursor": 0, "limit": 50, "scanned": 50, "updated": 41, "nextCursor": 812, "done": false }

Operational notes:

• call it repeatedly, feeding nextCursor back in, until done is true
• safe to restart from cursor=0 at any point: a row whose segmentation already matches is skipped without a write, so a re-run of a finished backfill reports updated: 0
• each call issues at most two D1 operations, so an interrupted run never leaves a half-written batch
• verify with a Japanese phrase query in fusion: "sparse_only" mode — sparse_candidates should be non-zero

12. Index a wiki without waiting for the cron

The :45 wiki cron walks a bounded number of pages per run and resumes from a stored cursor, so a deep wiki reaches full coverage in ceil(pages / 20) hours. Use this endpoint when that is too slow — a freshly connected repository, a bulk wiki import, or a coverage repair after a poller fix.

Admin endpoint:

POST /admin/backfill-wiki?repo=owner/repo

Parameters:

• repo — owner/repo, must be listed in POLL_REPOS
• limit — raw-content fetch attempts per call, 1..40 (default 20)
• cursor — page slug to resume after; omit to continue from the stored cursor, or pass empty (cursor=) to restart from the head of the enumeration (which restarts the lap there too)

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

Response:

{ "repo": "owner/repo", "pages": 77, "fetches": 20, "visited": 20, "embedded": 18,
  "skipped": 2, "failed": 0, "removed": 3, "orphansDeferred": 5, "orphansWithheld": 0,
  "startCursor": "", "nextCursor": "current-architecture-as-concession",
  "lapAnchor": "", "wrapped": false, "enumerated": true, "done": false }

Operational notes:

• call it repeatedly until done is true; it shares the cron's cursor, so the two advance one another rather than fighting
• done (= wrapped) means the cursor completed a lap of the enumeration, not that one call covered every page. A wiki with more pages than limit cannot finish in one call; the lap closes after ceil(pages / limit) calls — 4 for a 77-page wiki at the default limit=20 (issue #188)
• lapAnchor is the slug the current lap started after ("" = the head of the enumeration). The lap runs from the page after the anchor around to the anchor itself, so lapAnchor plus nextCursor shows how far the lap has come
• enumerated: false means the /wiki/_pages scrape failed — nothing was indexed and, deliberately, nothing was reaped; retry rather than treating it as an empty wiki
• orphansDeferred counts reap candidates the call never reached, because it hit either the per-run delete cap or the per-run probe cap; keep calling until it reaches 0. A candidate that was reached and withheld is reported by orphansWithheld, not here — the two budgets are separate so a withheld candidate cannot spend a delete slot and stall the real deletions ordering behind it (issue #197)
• fetches may exceed limit by up to 3 on a call whose first page needs more candidates than the budget allows. Probing a page is up to 4 attempts (two filename candidates x md / markdown), and a probe cut short mid-list is not an observed result, so the walk would otherwise re-probe that page on every call without ever moving the cursor. The first page of each call is therefore allowed to finish its candidate list; every page after it stops at the budget (issue #192)
• orphansWithheld counts reap candidates whose content still served (or whose existence probe could not conclude), so the delete was withheld. Non-zero means the _pages scrape came back short of the live wiki — the pages themselves are intact and were protected, but the enumeration is what to investigate; the worker log names each withheld page (issue #187)
• verify coverage by comparing search_docs rows (type = 'wiki_doc') against the page list at https://github.com/{repo}/wiki/_pages

13. Purge pre-migration doc vectors (one-off)

Vector IDs moved from plain text ({repo}#doc-{path}) to a hashed scheme in April 2026. Every delete path computes the current ID, so doc vectors written before that migration can never be named again: they stay in Vectorize, answer dense queries with their pre-migration content, and take a candidate slot away from the live row for the same file. Deleting the file does not help — the reap removes the current-generation row and leaves the legacy one (issue #204).

Run this once per repository indexed before the migration. It deletes only IDs it rebuilds in the legacy format, and re-embeds nothing.

Admin endpoint:

POST /admin/purge-legacy-vectors?repo=owner/repo

Parameters:

• repo — owner/repo
• dry_run — true reports the counts and calls Vectorize not at all
• surface — doc (default), the only value accepted. The migration changed every surface's ID, but doc is the only surface where orphans have actually been measured
• limit — IDs per call, 1..2000 (default 500)
• cursor — offset to resume from; pass back the nextCursor of the previous response

Body (optional):

{ "paths": [".claude/CLAUDE.md", ".claude/rules/model/absolute.md"] }

Paths already deleted from the repository. Their legacy IDs cannot be enumerated from anything the worker still holds, so they have to be named explicitly. Candidates from paths are covered before the tree, so a capped run reaches them first.

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

Response:

{ "repo": "owner/repo", "surface": "doc", "dryRun": false, "candidates": 512,
  "skippedOversize": 3, "treeTruncated": false, "cursor": 0, "limit": 500,
  "targeted": 500, "deleted": 500, "remaining": 12, "nextCursor": 500, "done": false }

Operational notes:

• call it repeatedly, feeding nextCursor back in, until done is true. remaining is the count the per-run cap left behind. Send the same paths body on every call of a walk: the cursor indexes an ordered list that starts with them, so dropping them shifts every later position
• safe to repeat: deleting an absent ID is a no-op, so a re-run of a finished purge reports the same candidates and changes nothing. If a call fails mid-walk, resume from the same cursor
• skippedOversize counts legacy IDs longer than Vectorize's 64-byte ID cap. Those were rejected at upsert time — that overflow is what forced the migration — so no vector is keyed to them and they are not sent
• treeTruncated: true means the Git Trees API cut its listing short, so the tree half of the candidate set is partial. The explicit paths half is unaffected
• current-generation vectors are never at risk: the two ID formats are disjoint ({repo}#doc-… vs d:…) and only rebuilt legacy IDs are passed to the delete call
• the residue is closed, not growing — the legacy scheme stopped writing at the migration — so this endpoint is a one-off per repository, not a recurring job
• verify by searching for a file that was double-indexed: the pre-migration copy (old content, dense-only) should stop appearing

14. Close indexed rows that stayed open after the item was closed

A row indexed while its issue or PR was open carries state: "open". The state change is mirrored onto the index by the metadata-only path, and that path used to advance its own diff baseline before the mirror writes it guards — so a mirror write that failed was never retried, and the row kept answering searches as a live item (issue #209). The ordering is fixed at the source; this endpoint repairs the rows the old ordering left behind.

Nothing is re-embedded: the true state comes from one state=open listing per repo, the sparse side is an UPDATE, and the dense side re-upserts the existing vector values with only state replaced. /admin/reset-hashes is the wrong tool here — it triggers a full re-embedding of the repository.

Admin endpoint:

POST /admin/backfill-issue-state?repo=owner/repo

Parameters:

• repo — owner/repo
• dry_run — true reports the counts and writes to neither D1 nor Vectorize
• limit — rows examined per call, 1..1000 (default 200)
• cursor — issue number to resume after; pass back the nextCursor of the previous response

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

Response:

{ "repo": "owner/repo", "dryRun": false, "openOnGitHub": 33, "cursor": 0, "limit": 200,
  "scanned": 165, "stale": 132, "ftsUpdated": 132, "vectorsUpdated": 130,
  "vectorsMissing": 2, "nextCursor": null, "done": true }

Operational notes:

• call it repeatedly, feeding nextCursor back in, until done is true
• safe to repeat: a row GitHub still lists as open is skipped without a write, so a re-run of a finished repair reports stale: 0
• the direction is one-way (open → closed). That is the direction the defect produced and the one that harms retrieval — a closed decision resurfacing as a live one. It also keeps the scan proportional to the open set rather than to the whole index
• the run aborts with an error rather than closing anything if the open-item listing would exceed 50 pages. Absence from that listing is what marks a row closed, so a partial listing must never be used
• vectorsMissing counts stale rows with no vector to refresh. Those are the missing-index-entry surface (issue #210) and are out of scope here; their sparse half still exists and is still repaired
• if the dense write fails, the whole call fails before D1 is touched, leaving the window unrepaired rather than half-repaired. Retry with the same cursor
• verify with a state: "closed" search for an item you know was closed, or by counting search_docs rows: SELECT COUNT(*) FROM search_docs WHERE repo = ? AND type IN ('issue','pull_request') AND state = 'open' should match the repository's real open count

15. Index the issues and pull requests that never reached the index

The poller embeds at most 50 items per run but fetches up to 200, and it used to move its watermark past the whole batch regardless. Everything the embedding budget deferred was therefore marked for a retry that no later since window would fetch, and roughly 55% of the issue and pull request history of the indexed repositories never reached the index (issue #210). The watermark is fixed at the source, but that only stops the leak: a stranded item is re-fetched only when its updated_at moves, and closed history never moves again. This endpoint walks the gap directly.

Run it after deploying the watermark fix. In the other order, items indexed by the backfill are joined by new ones falling into the same hole.

Admin endpoint:

POST /admin/backfill-issue-index?repo=owner/repo

Parameters:

• repo — owner/repo
• dry_run — true measures the gap over the scan range and fetches nothing from GitHub
• limit — candidate numbers attempted per call, 1..20 (default 15). Ignored on a dry run
• cursor — issue number to resume after; pass back the nextCursor of the previous response

Authentication:

• send the same GITHUB_TOKEN value in the GITHUB_TOKEN header

Response:

{ "repo": "owner/repo", "dryRun": false, "cursor": 0, "limit": 15, "maxNumber": 1690,
  "scannedTo": 603, "candidates": 15, "attempted": 15, "indexed": 14, "absent": 1,
  "failed": 0, "nextCursor": 603, "done": false }

Operational notes:

• start with dry_run=true to size the job. It spends no embedding budget and reports candidates over the range it scanned (up to 5000 numbers per call)
• call it repeatedly, feeding nextCursor back in, until done is true. At the default limit a repository missing 900 items takes 60 calls
• this endpoint embeds, unlike the two repairs above, and the ingest fan-out per candidate is expensive: measured at roughly 40 of the invocation's 1000 subrequests (issue #216). The defaults are calibrated to it — 15 is the largest value production completed with failed: 0, 25 lost its last candidate on every call, and 50 failed the whole call with Too many subrequests by single Worker invocation. Anything above 20 is refused
• the limit does not have to be exact. The cursor is held one below the first candidate the call failed to ingest, so the next call reopens on that number instead of skipping it — an over-generous limit costs a wasted call, not a missing item
• safe to repeat: a number that already carries a search_docs row is never fetched, so a re-run of a finished sweep reports candidates: 0. A call that fails mid-sweep is resumed from the same cursor
• absent counts numbers GitHub answers 404 for — deleted issues, and numbers whose item was transferred out. They stay candidates on every future sweep, which is why a finished repository still reports a small non-zero candidates. They do not hold the cursor: nothing will ever ingest them, so holding there would stall the sweep rather than bound a retry
• failed counts candidates whose embed did not land. The next call retries them first, because the cursor was held below the earliest one
• a candidate that fails on every attempt therefore stops the sweep, and says so: nextCursor comes back equal to the cursor you passed in, with failed at 1 or more. Step over it by hand with cursor = nextCursor + 1 — the blocking number is nextCursor + 1, and the run log names it as held before #N
• the ingest is forced past the body-hash check, so an item whose vector exists but whose FTS5 row is missing is repaired too. This is why the endpoint is not equivalent to waiting for the next poll
• verify by counting distinct numbers: SELECT COUNT(DISTINCT number) FROM search_docs WHERE repo = ? AND type IN ('issue','pull_request') should approach the repository's real issue + PR count

Troubleshooting

GITHUB_TOKEN not configured

The worker secret is missing or misconfigured.

POLL_REPOS not configured

The plain-text variable is missing.

GitHub API 401 or 403

The token is expired or missing required scopes.

OAuth callback fails

The GitHub App callback URL does not exactly match the worker callback URL.

Webhook verification fails

GITHUB_WEBHOOK_SECRET does not match the value configured in the GitHub App.