From d8ae39f15650b4a8497801cae004fedb5427c382 Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Sun, 2 Aug 2026 21:13:14 +0900 Subject: [PATCH 1/2] fix(docs): reap graph edges and cap doc deletions per run [poller, docs, tests] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repository-docs reap tore down three surfaces (Vectorize, D1 FTS5, store row) while the wiki reap tore down four, and it walked the whole deleted set with no per-run budget. Both gaps are closed by adopting the wiki reap's shape. - deleteEdgesForVector is now called for every reaped doc, and the four surfaces are torn down independently so a Vectorize failure no longer strands the D1 rows users retrieve. - MAX_DOC_DELETIONS_PER_REPO_PER_RUN (5) caps the reap. At ~4 subrequests per deletion an unbounded loop over a mass deletion could exhaust the light cron's invocation budget alone; a single PR removing 66 .md files puts all of them on one run. - The tree ETag is held back while deletions are outstanding. Without this the next run answers 304 and returns before it looks at the leftover, so the drain would stall until the tree changed again. Same hold the fetch cap already used; the two branches are now one write. pollDocs is exported for the new test file. edge 削除は溜まったものを流すためではなく、wiki 側との対称性のために入れて いる。doc_edges の端点は現状どちらも wiki vector ID なので doc vector ID は 1 行も一致せず、この DELETE は no-op になる。issue 本文の「edge table に溜ま り続ける」という記述はこの点で実測と食い違っており、PR 本文に訂正を書いた。 効果は将来 doc 側の edge writer が入ったときに「vector を削除する」の意味が 2 経路で同じであり続けること。 Refs #203 --- docs/0-requirements.ja.md | 6 +- docs/0-requirements.md | 6 +- src/poller-docs.test.ts | 288 ++++++++++++++++++++++++++++++++++++++ src/poller.ts | 136 ++++++++++++------ 4 files changed, 390 insertions(+), 46 deletions(-) create mode 100644 src/poller-docs.test.ts diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index e9a0bf6..bb6e61c 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -149,6 +149,10 @@ commit diff poller は 2-phase 構成: トレードオフは liveness で、毎回失敗し続ける commit はその phase の watermark を止める。run log に境界 commit の SHA が出るほか、`POST /admin/diff-watermark`（installation guide 参照）で watermark を手動で移動できる。旧版 poller が取りこぼした期間を再走査させる経路も同じ endpoint。 +docs poller は `If-None-Match` 付きの条件付きリクエストで repository tree を読み、保存済みの doc record と差分を取る。blob SHA が動いた entry は re-embed し、store にあって tree に無い entry は削除する。**削除は 4 面を teardown する** — Vectorize / D1 FTS5 / graph edge table / structured store — それぞれ独立に実行するので、Vectorize の失敗が実際に retrieval される D1 行を取り残すことはない。edge の teardown は溜まったものを流すためではなく wiki 側との対称性のためで、現状 `doc_edges` の端点は両側とも wiki vector ID なので doc vector ID は 1 行も一致しない。将来 repository doc が edge の端点になったときに、「vector を削除する」の意味が 2 経路で同じであり続けることが効果（issue #203）。 + +**削除の枠.** 削除は 1 repo 1 run あたり `MAX_DOC_DELETIONS_PER_REPO_PER_RUN`（既定 5）で cap する。wiki の削除と同じ guard で、1 件あたり約 4 subrequest かかるため、大量削除に対して上限なく回すと light cron の invocation 予算を単独で食い潰し、後ろに並ぶ repo を飢えさせうる — 1 つの PR が `.md` を 66 件削除すれば、その全件が 1 run に集中する。削除済み doc の store 行は消えるので残りの集合は縮む一方であり、drain は単調。したがってこの surface には per-run cap が 2 本あり、**どちらが効いても tree ETag は据え置く**: fetch 枠は未処理の変更 doc を残し（issue #149）、削除枠は未削除の doc を残す（issue #203）。どちらの場合も ETag を進めてしまうと次 run が 304 で返り、残りを見ないまま終わる。`lastPolledAt` は進めるので run 自体は観測できる。 + wiki poller は `:45` cron 専属で、GitHub Wiki content の唯一の取り込み経路。Wiki は別 git repo (`{repo}.wiki.git`) に存在し、REST API も webhook event も持たないため、poller が repo ごとに 3 段の HTTP 呼び出しで処理する: 1. `https://github.com/{repo}.wiki.git/info/refs?service=git-upload-pack` を打って wiki 存在検出（200 = 存在、404 = 無効化済 or 未設置 = skip）。 @@ -295,7 +299,7 @@ Durable Object + SQLite は次の structured record を保持する。 - **エッジ抽出**: wiki ページ index 時（`processAndUpsertWikiDoc`）、同 repo の既知 wiki slug が本文に出現したら A→B の "mention" エッジを生成（`src/graph.ts` の `indexWikiEdges`）。**決定的 slug-match（LLM 不要・ロスなし）**。dst は計算で求まるので未 index でも記録可（dangling 可）。typed（supersede/depend/conflict）は将来スコープ。 - **traversal**: `queryNeighbors` が `WITH RECURSIVE`（標準 SQLite、拡張不要）で seed の 1–2 hop neighbor を無向に辿る。 - **retrieval 統合**: `search` の `graph_expand`（既定 false）/ `graph_hops`（既定 1）。true の時のみ、RRF 後の最終結果を seed に neighbor を辿り、関連 wiki ページを `graph_hop` / `graph_from` 付きで末尾に append。**false の時は既存挙動と完全同一（回帰なし）**。 -- **delete fan-out**: wiki ページ削除時に `deleteEdgesForVector`（当該 vector を端点に持つエッジを除去）。 +- **delete fan-out**: `deleteEdgesForVector` が当該 vector を端点に持つエッジを除去する。cron の削除は両方これを呼ぶ — 実際に行が存在する wiki 側と、doc vector ID が 1 行も一致しない repository docs 側（後者は将来 doc 側 edge writer が入ったときのための対称性、issue #203）。 - **backfill**: `POST /admin/backfill-edges?repo=owner/repo`（GITHUB_TOKEN ヘッダ）。既存 index 済み wiki の content から一括抽出（GitHub 再取得不要）。 - **評価**: 本番 ship 後の実運用観測（judgment-learning が関連判断を拾えるか）。offline eval harness は作らない。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 7a96156..069bdd9 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -150,6 +150,10 @@ Both phases obey one watermark invariant: **a watermark never moves past a commi The tradeoff is liveness: a commit that fails on every attempt blocks its phase's watermark. The run log names the boundary commit, and `POST /admin/diff-watermark` (see the installation guide) moves the watermark manually — the same endpoint used to replay a period whose commits were lost by an earlier version of the poller. +The docs poller reads the repository tree with a conditional `If-None-Match` request and diffs it against the stored doc records: entries whose blob SHA moved are re-embedded, entries present in the store but absent from the tree are reaped. **The reap tears down four surfaces** — Vectorize, D1 FTS5, the graph edge table, and the structured store — each independently, so a Vectorize failure cannot strand the D1 rows users actually retrieve. The edge teardown is symmetry with the wiki reap rather than a backlog drain: `doc_edges` endpoints are wiki vector IDs on both sides today, so a doc vector ID matches no row. It is what makes "delete a vector" mean the same thing on both paths if repository docs ever become edge endpoints (issue #203). + +**Reap budget.** The reap is capped at `MAX_DOC_DELETIONS_PER_REPO_PER_RUN` (default 5) per repo per run, the same guard the wiki reap carries: at ~4 subrequests per deletion an unbounded loop over a mass deletion could exhaust the light cron's invocation budget on its own and starve every repo behind it — a single PR removing 66 `.md` files puts all of them on one run. The drain is monotonic, since a reaped doc's store row is gone and the leftover set only shrinks. Two per-run caps therefore exist on this surface, and **either one holds the tree ETag back**: the fetch cap leaves changed docs unprocessed (issue #149) and the delete cap leaves deletions unreaped (issue #203), and in both cases advancing the ETag would make the next run answer 304 and return before it looked at the leftover. `lastPolledAt` still advances so the run stays observable. + The wiki poller runs in the `:45` cron and is the only ingestion path for GitHub Wiki content. Wiki pages live in a separate git repo (`{repo}.wiki.git`) that GitHub does not expose through the REST API or webhook events; the poller therefore performs three lightweight HTTP calls per repo: 1. Probe `https://github.com/{repo}.wiki.git/info/refs?service=git-upload-pack` to detect whether the wiki exists (200 = present, 404 = disabled or absent — skip the rest). @@ -297,7 +301,7 @@ An additive graph layer that indexes relationships between Decision-Structure wi - **Edge extraction**: at wiki index time (`processAndUpsertWikiDoc`), when another known wiki slug in the same repo appears in the content, an A→B "mention" edge is written (`indexWikiEdges` in `src/graph.ts`). **Deterministic slug-match (no LLM, no lossy extraction)**; the dst ID is computed, so dangling edges to not-yet-indexed pages are allowed. Typed edges (supersede/depend/conflict) are future scope. - **Traversal**: `queryNeighbors` walks 1–2 hop undirected neighbors of the seeds via `WITH RECURSIVE` (standard SQLite, no extension). - **Retrieval integration**: `search` gains `graph_expand` (default false) / `graph_hops` (default 1). Only when true, the final result set seeds a traversal and related wiki pages are appended (tagged `graph_hop` / `graph_from`). **When false the behavior is byte-identical to before (no regression).** -- **Delete fan-out**: on wiki page deletion, `deleteEdgesForVector` removes edges touching that vector. +- **Delete fan-out**: `deleteEdgesForVector` removes edges touching a vector. Both cron reaps call it — the wiki one, which is where rows actually exist, and the repository-docs one, where a doc vector ID matches nothing today and the call stands as symmetry against a future doc-edge writer (issue #203). - **Backfill**: `POST /admin/backfill-edges?repo=owner/repo` (GITHUB_TOKEN header) re-extracts edges from the stored content of already-indexed wiki pages (no GitHub refetch). - **Evaluation**: real-use observation after ship (does judgment-learning surface related decisions), not an offline eval harness. diff --git a/src/poller-docs.test.ts b/src/poller-docs.test.ts new file mode 100644 index 0000000..f0402f2 --- /dev/null +++ b/src/poller-docs.test.ts @@ -0,0 +1,288 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import type { Env, DocRecord } from "./types.js"; + +// `pollDocs` fans out to the embed pipeline (Contents API + Workers AI + +// Vectorize + D1 + Store DO) and to the two D1 teardown helpers. What is under +// test here is the *reap* — the delete fan-out and its per-run budget — so the +// embed entry point and both teardown helpers are replaced with controllable +// fakes and only the Git Trees API call reaches the stubbed global fetch. +// `docVectorId` and the rest of `./pipeline.js` stay real. +const { + processAndUpsertDocMock, + deleteFtsRowMock, + deleteEdgesForVectorMock, +} = vi.hoisted(() => ({ + processAndUpsertDocMock: vi.fn(), + deleteFtsRowMock: vi.fn(), + deleteEdgesForVectorMock: vi.fn(), +})); + +vi.mock("./pipeline.js", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, processAndUpsertDoc: processAndUpsertDocMock }; +}); + +vi.mock("./fts.js", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, deleteFtsRow: deleteFtsRowMock }; +}); + +vi.mock("./graph.js", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, deleteEdgesForVector: deleteEdgesForVectorMock }; +}); + +const { pollDocs } = await import("./poller.js"); +const { docVectorId } = await import("./pipeline.js"); + +const REPO = "acme/widgets"; +const WATERMARK_KEY = `docs:${REPO}`; +const TREE_URL = `https://api.github.com/repos/${REPO}/git/trees/HEAD?recursive=1`; + +/** Constant mirrored from `src/poller.ts`. */ +const DELETE_BUDGET = 5; + +/** + * Stub the global fetch with a fake Git Trees API returning `paths` as blobs. + * + * `etag` is what the response carries; `expectNotModified` makes the stub assert + * the conditional-request contract by returning 304 when the poller sends back + * the matching `If-None-Match`. + */ +function stubTree(paths: string[], etag = 'W/"tree-1"') { + const conditionalHits: string[] = []; + + const fetchMock = vi.fn(async (input: string | URL, init?: RequestInit) => { + const url = String(input); + if (url !== TREE_URL) { + throw new Error(`unexpected fetch in docs stub: ${url}`); + } + const headers = (init?.headers ?? {}) as Record; + const sent = headers["If-None-Match"]; + if (sent) conditionalHits.push(sent); + if (sent === etag) { + return new Response(null, { status: 304 }); + } + return new Response( + JSON.stringify({ + sha: "treesha", + truncated: false, + tree: paths.map((path) => ({ path, type: "blob", sha: `blob-${path}` })), + }), + { status: 200, headers: { "Content-Type": "application/json", ETag: etag } }, + ); + }); + + vi.stubGlobal("fetch", fetchMock); + return { fetchMock, conditionalHits }; +} + +/** + * In-memory IssueStore stand-in covering the docs surface: the record list the + * poller diffs against, the docs watermark row holding the tree ETag, and the + * per-path DELETE the reap issues. + */ +function makeDocStore(seed: DocRecord[] = [], etag?: string) { + const records = new Map(seed.map((d) => [d.path, d])); + const watermarks = new Map(); + if (etag !== undefined) { + watermarks.set(WATERMARK_KEY, { lastPolledAt: "2026-08-01T00:00:00Z", etag }); + } + const deletes: string[] = []; + + const stub = { + async fetch(request: Request): Promise { + const url = new URL(request.url); + const path = url.pathname; + + if (request.method === "GET" && path === "/docs") { + return Response.json([...records.values()]); + } + if (request.method === "GET" && path === "/watermark") { + const key = url.searchParams.get("repo") ?? ""; + const wm = watermarks.get(key); + if (!wm) return new Response("not found", { status: 404 }); + return Response.json({ repo: key, ...wm }); + } + if (request.method === "POST" && path === "/watermark") { + const body = (await request.json()) as { + repo: string; + lastPolledAt: string; + etag?: string; + }; + watermarks.set(body.repo, { lastPolledAt: body.lastPolledAt, etag: body.etag }); + return new Response("ok"); + } + if (request.method === "DELETE" && path === "/doc") { + const docPath = url.searchParams.get("path") ?? ""; + deletes.push(docPath); + records.delete(docPath); + return new Response("ok"); + } + return new Response("ok"); + }, + }; + + return { + stub: stub as unknown as DurableObjectStub, + records, + deletes, + etag: () => watermarks.get(WATERMARK_KEY)?.etag, + }; +} + +function makeDocEnv() { + const vectorDeletes: string[] = []; + const deleteByIds = vi.fn(async (ids: string[]) => { + vectorDeletes.push(...ids); + }); + const env = { + GITHUB_TOKEN: "test-token", + VECTORIZE: { deleteByIds }, + DB_FTS: {} as unknown, + } as unknown as Env; + return { env, vectorDeletes }; +} + +/** A stored doc whose blob SHA matches what `stubTree` reports, so it is + * "unchanged" and never enters the embed path. */ +const stored = (path: string): DocRecord => ({ + repo: REPO, + path, + blobSha: `blob-${path}`, + updatedAt: "2026-08-01T00:00:00Z", +}); + +beforeEach(() => { + processAndUpsertDocMock.mockReset(); + processAndUpsertDocMock.mockResolvedValue({ embedded: true, failed: false }); + deleteFtsRowMock.mockReset().mockResolvedValue(undefined); + deleteEdgesForVectorMock.mockReset().mockResolvedValue(undefined); +}); + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +describe("poller: pollDocs delete fan-out", () => { + it("tears down graph edges alongside the vector, the FTS5 row and the store record", async () => { + // One doc still in the store, gone from the tree. Pre-fix the reap hit three + // surfaces and left the edge table untouched, so a doc vector ID could stay + // an edge endpoint with nothing to remove it (issue #203). + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env, vectorDeletes } = makeDocEnv(); + + await pollDocs(REPO, env, store.stub); + + const goneId = await docVectorId(REPO, "docs/gone.md"); + + expect(store.deletes).toEqual(["docs/gone.md"]); + expect(vectorDeletes).toEqual([goneId]); + expect(deleteFtsRowMock).toHaveBeenCalledTimes(1); + expect(deleteFtsRowMock.mock.calls[0][1]).toBe(goneId); + expect(deleteEdgesForVectorMock).toHaveBeenCalledTimes(1); + expect(deleteEdgesForVectorMock.mock.calls[0][1]).toBe(goneId); + }); + + it("keeps tearing down the later surfaces when Vectorize fails", async () => { + // The D1 rows are the ones users actually retrieve; a dense-side failure + // must not strand them. Same independence the wiki reap already has. + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env } = makeDocEnv(); + (env.VECTORIZE.deleteByIds as ReturnType).mockRejectedValue( + new Error("vectorize down"), + ); + + await pollDocs(REPO, env, store.stub); + + expect(deleteFtsRowMock).toHaveBeenCalledTimes(1); + expect(deleteEdgesForVectorMock).toHaveBeenCalledTimes(1); + expect(store.deletes).toEqual(["docs/gone.md"]); + }); + + it("leaves the reap alone when nothing was deleted", async () => { + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md")]); + const { env, vectorDeletes } = makeDocEnv(); + + await pollDocs(REPO, env, store.stub); + + expect(store.deletes).toEqual([]); + expect(vectorDeletes).toEqual([]); + expect(deleteFtsRowMock).not.toHaveBeenCalled(); + expect(deleteEdgesForVectorMock).not.toHaveBeenCalled(); + }); +}); + +describe("poller: pollDocs delete budget", () => { + it("caps deletions per run and drains the rest on later runs", async () => { + // The case that raised the issue: a single PR removed 66 `.md` files, all of + // which land on the next run. Unbounded, that is ~4 subrequests x 66 in one + // LIGHT_CRON invocation shared with pollRepo and pollReleases (issue #203). + const deletedPaths = Array.from( + { length: DELETE_BUDGET * 2 + 1 }, + (_, i) => `docs/gone-${String(i).padStart(2, "0")}.md`, + ); + const store = makeDocStore([stored("docs/keep.md"), ...deletedPaths.map(stored)]); + const { env } = makeDocEnv(); + + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(DELETE_BUDGET); + + // Monotonic drain: a reaped doc's store row is gone, so the leftover set + // only shrinks. No path is reaped twice and none is skipped. + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(DELETE_BUDGET * 2); + + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(deletedPaths.length); + + expect([...store.deletes].sort()).toEqual([...deletedPaths].sort()); + expect(new Set(store.deletes).size).toBe(deletedPaths.length); + expect(store.records.has("docs/keep.md")).toBe(true); + expect(store.records.size).toBe(1); + }); + + it("holds the tree ETag back while deletions are outstanding", async () => { + // Advancing the ETag with a backlog left would make the next run answer 304 + // and return before it ever looks at `deletedDocs` — the drain would stall + // until the tree happened to change again (issue #203). + const deletedPaths = Array.from( + { length: DELETE_BUDGET + 1 }, + (_, i) => `docs/gone-${i}.md`, + ); + const store = makeDocStore([stored("docs/keep.md"), ...deletedPaths.map(stored)]); + const { env } = makeDocEnv(); + + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(DELETE_BUDGET); + expect(store.etag()).toBeUndefined(); + + // Next run: the poller sends no If-None-Match, so it sees the tree again and + // reaps the leftover. + const { conditionalHits } = stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(conditionalHits).toEqual([]); + expect(store.deletes).toHaveLength(deletedPaths.length); + + // Backlog cleared — now the ETag is allowed to advance. + expect(store.etag()).toBe('W/"tree-1"'); + }); + + it("advances the tree ETag when the reap finished inside its budget", async () => { + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env } = makeDocEnv(); + + await pollDocs(REPO, env, store.stub); + + expect(store.deletes).toEqual(["docs/gone.md"]); + expect(store.etag()).toBe('W/"tree-1"'); + }); +}); diff --git a/src/poller.ts b/src/poller.ts index ce6efde..9e65789 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -107,6 +107,26 @@ const MAX_COMMENT_FETCHES_PER_REPO_PER_RUN = 10; * unchanged in the store until the doc is successfully upserted). */ const MAX_DOC_FETCHES_PER_REPO_PER_RUN = 10; +/** Maximum docs reaped (Vectorize + FTS5 + graph edges + store row) per repo per + * cron run. Mirrors `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`: each reap fans out + * to 4 subrequests, so an unbounded loop over a mass deletion could exhaust the + * LIGHT_CRON invocation budget on its own and starve every repo behind it — + * docs share that invocation with `pollRepo` and `pollReleases` (issue #203). + * A real case: github-webhook-mcp deleted 66 `.md` files in one PR, all of + * which land on the next single run. + * + * Numeric design: 5 deletes x 4 subrequests x 5 repos = 100 subrequests for the + * reap, on top of the ~250 the docs fetch cap already allows. Together with + * pollRepo and pollReleases the LIGHT_CRON worst case stays around 850, inside + * the 1000 ceiling. + * + * Remaining deletions are reaped next run: a reaped doc's store row is gone, so + * the leftover set only shrinks and the drain is monotonic. The tree ETag is + * deliberately *not* advanced while deletions are outstanding — the next run + * would otherwise short-circuit on 304 and never see them (same hold as the + * fetch cap, see the watermark write below). */ +const MAX_DOC_DELETIONS_PER_REPO_PER_RUN = 5; + /** Maximum number of release records the releases poller upserts per repo per * cron run. The GitHub Releases endpoint returns all recent releases in a * single API call, so the GH-side fetch cost is fixed at 1, but each release @@ -788,8 +808,15 @@ async function fetchFileContent( /** * Poll a single repository for documentation file updates. * Uses Git Trees API for change detection and Contents API for fetching changed files. + * + * Files present in the store but absent from the current tree are reaped from + * Vectorize, D1 FTS5, the graph edge table, and the structured store — capped + * per repo per run, with the tree ETag held back while a backlog remains so the + * next run can still see it (issue #203). + * + * Exported for tests; production callers reach it through `handleScheduled`. */ -async function pollDocs( +export async function pollDocs( repo: string, env: Env, storeStub: DurableObjectStub, @@ -917,61 +944,82 @@ async function pollDocs( } } - // Handle deleted files: remove from Vectorize, D1 FTS5, and the structured store. + // Handle deleted files: remove from Vectorize, D1 FTS5, the graph edge table, + // and the structured store — the same four surfaces the wiki reap tears down. + // + // The edge teardown is a symmetry guard, not a backlog drain. Today every + // `doc_edges` endpoint is a wiki vector ID on both sides (`indexWikiEdges` is + // the only writer and `knownWikiSlugs` filters `type = 'wiki_doc'`), so a doc + // vector ID matches no row and the DELETE is a no-op. What it buys is that + // "delete a vector" means the same thing on both paths: if repository docs + // ever become edge endpoints, the reap already tears them down instead of + // leaking rows that retrieval silently skips as dangling (issue #203). + let removedDocs = 0; + let deleteBudgetExhausted = false; for (const doc of deletedDocs) { - try { - const dvid = await docVectorId(repo, doc.path); - await env.VECTORIZE.deleteByIds([dvid]); + if (removedDocs >= MAX_DOC_DELETIONS_PER_REPO_PER_RUN) { + deleteBudgetExhausted = true; + console.warn( + `pollDocs: delete budget reached for ${repo} ` + + `(${MAX_DOC_DELETIONS_PER_REPO_PER_RUN} deletions). ` + + `${deletedDocs.length - removedDocs} remaining docs will be reaped next cron run.`, + ); + break; + } + + const dvid = await docVectorId(repo, doc.path); + // Each surface is torn down independently: a Vectorize failure must not + // strand the D1 rows, which are the ones users actually retrieve. + for (const [surface, run] of [ + ["vector", () => env.VECTORIZE.deleteByIds([dvid])], + ["FTS5 row", () => deleteFtsRow(env.DB_FTS, dvid)], + ["graph edges", () => deleteEdgesForVector(env.DB_FTS, dvid)], + [ + "store record", + () => + storeStub.fetch( + new Request( + `http://store/doc?repo=${encodeURIComponent(repo)}&path=${encodeURIComponent(doc.path)}`, + { method: "DELETE" }, + ), + ), + ], + ] as Array<[string, () => Promise]>) { try { - await deleteFtsRow(env.DB_FTS, dvid); - } catch (ftsErr) { + await run(); + } catch (err) { console.error( - `Failed to delete FTS5 row for doc ${repo}/${doc.path}:`, - ftsErr instanceof Error ? ftsErr.message : String(ftsErr), + `Failed to delete ${surface} for doc ${repo}/${doc.path}:`, + err instanceof Error ? err.message : String(err), ); } - await storeStub.fetch( - new Request( - `http://store/doc?repo=${encodeURIComponent(repo)}&path=${encodeURIComponent(doc.path)}`, - { method: "DELETE" }, - ), - ); - } catch (err) { - console.error( - `Failed to delete doc vector ${repo}/${doc.path}:`, - err instanceof Error ? err.message : String(err), - ); } + removedDocs++; } // Update watermark with ETag — but only when the run completed without - // hitting the per-run fetch cap. If we did hit the cap, leftover changed - // docs still need to be processed; persisting the new ETag would cause the - // next cron to short-circuit on 304 and never see them. Skipping the ETag - // update forces a fresh tree fetch next run so `changedEntries` repopulates - // (issue #149). - if (!fetchBudgetExhausted) { - await storeStub.fetch( - new Request("http://store/watermark", { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ repo: watermarkKey, lastPolledAt: now, etag: responseEtag }), - }), - ); - } else { - // Still bump lastPolledAt so observability can see the run happened, but - // keep the prior ETag so the next cron re-fetches the tree. - await storeStub.fetch( - new Request("http://store/watermark", { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ repo: watermarkKey, lastPolledAt: now, etag: storedEtag }), + // hitting a per-run cap. If either cap was hit, leftover work remains: + // changed docs still to embed (issue #149) or deleted docs still to reap + // (issue #203). Persisting the new ETag would cause the next cron to + // short-circuit on 304 and never see either. Holding the prior ETag forces a + // fresh tree fetch next run so `changedEntries` and `deletedDocs` both + // repopulate. lastPolledAt is still bumped so observability sees the run. + const holdEtag = fetchBudgetExhausted || deleteBudgetExhausted; + await storeStub.fetch( + new Request("http://store/watermark", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + repo: watermarkKey, + lastPolledAt: now, + etag: holdEtag ? storedEtag : responseEtag, }), - ); - } + }), + ); console.log( - `${repo} docs: ${docEntries.length} found, ${embedded} embedded, ${skipped} unchanged, ${failed} failed, ${deletedDocs.length} deleted, ` + + `${repo} docs: ${docEntries.length} found, ${embedded} embedded, ${skipped} unchanged, ${failed} failed, ` + + `${removedDocs}/${deletedDocs.length} deleted, ` + `fetches_issued=${fetchesIssued}/${MAX_DOC_FETCHES_PER_REPO_PER_RUN}`, ); } From 7533ae817d07fffea60d8ac987d58cb16f99be30 Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Sun, 2 Aug 2026 21:21:15 +0900 Subject: [PATCH 2/2] fix(docs): drop the no-op edge teardown from the doc reap [poller, docs, tests] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit deleteEdgesForVector on a doc vector ID matches zero rows: indexWikiEdges is the only doc_edges writer, its src id is the page's wvid and the dst id it computes is a wikiDocVectorId too, and the backfill in index.ts filters WHERE type = 'wiki_doc'. A doc vector id cannot be an endpoint. The call is removed. Justifying it as symmetry against a future doc-edge writer was a push-surplus argument for a statement with no behavior behind it. With the premise gone the asymmetry is not a defect either: wiki pages have edges and repository docs do not, so the two reaps differing is the data differing, not the handling. The knowledge is kept as a comment on the delete loop, naming the invariant and what to do if it changes. Per-deletion fan-out is 3 subrequests, not 4; the budget arithmetic in the constant's doc comment is corrected to match. Unchanged from the prior commit, all three load-bearing on real defects: per-surface independent teardown, MAX_DOC_DELETIONS_PER_REPO_PER_RUN, and the ETag hold that keeps the drain moving once the cap exists. Tests drop the edge assertions and pin those three instead, plus a new case fixing that an FTS5 failure still reaches the store DELETE. 親側の裁定による。subtractive-structural-beauty の (A)(B) 適用で、0 行にしか マッチしない DELETE は behavior に load-bearing でないため削除。知識だけ コメントとして残す形に寄せた。 Refs #203 --- docs/0-requirements.ja.md | 6 ++--- docs/0-requirements.md | 6 ++--- src/poller-docs.test.ts | 49 +++++++++++++++++++-------------------- src/poller.ts | 33 +++++++++++--------------- 4 files changed, 44 insertions(+), 50 deletions(-) diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index bb6e61c..8df2363 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -149,9 +149,9 @@ commit diff poller は 2-phase 構成: トレードオフは liveness で、毎回失敗し続ける commit はその phase の watermark を止める。run log に境界 commit の SHA が出るほか、`POST /admin/diff-watermark`（installation guide 参照）で watermark を手動で移動できる。旧版 poller が取りこぼした期間を再走査させる経路も同じ endpoint。 -docs poller は `If-None-Match` 付きの条件付きリクエストで repository tree を読み、保存済みの doc record と差分を取る。blob SHA が動いた entry は re-embed し、store にあって tree に無い entry は削除する。**削除は 4 面を teardown する** — Vectorize / D1 FTS5 / graph edge table / structured store — それぞれ独立に実行するので、Vectorize の失敗が実際に retrieval される D1 行を取り残すことはない。edge の teardown は溜まったものを流すためではなく wiki 側との対称性のためで、現状 `doc_edges` の端点は両側とも wiki vector ID なので doc vector ID は 1 行も一致しない。将来 repository doc が edge の端点になったときに、「vector を削除する」の意味が 2 経路で同じであり続けることが効果（issue #203）。 +docs poller は `If-None-Match` 付きの条件付きリクエストで repository tree を読み、保存済みの doc record と差分を取る。blob SHA が動いた entry は re-embed し、store にあって tree に無い entry は削除する。**削除は 3 面を teardown する** — Vectorize / D1 FTS5 / structured store — それぞれ独立に実行するので、Vectorize の失敗が実際に retrieval される D1 行を取り残すことはない。wiki 側の 4 面に対してここが 3 面なのは、doc vector ID が `doc_edges` の端点になりえないため。`indexWikiEdges` が唯一の writer であり、src 側も算出される dst 側も wiki vector ID になる。この不変条件が変わったら、ここに edge の teardown を足すこと（issue #203）。 -**削除の枠.** 削除は 1 repo 1 run あたり `MAX_DOC_DELETIONS_PER_REPO_PER_RUN`（既定 5）で cap する。wiki の削除と同じ guard で、1 件あたり約 4 subrequest かかるため、大量削除に対して上限なく回すと light cron の invocation 予算を単独で食い潰し、後ろに並ぶ repo を飢えさせうる — 1 つの PR が `.md` を 66 件削除すれば、その全件が 1 run に集中する。削除済み doc の store 行は消えるので残りの集合は縮む一方であり、drain は単調。したがってこの surface には per-run cap が 2 本あり、**どちらが効いても tree ETag は据え置く**: fetch 枠は未処理の変更 doc を残し（issue #149）、削除枠は未削除の doc を残す（issue #203）。どちらの場合も ETag を進めてしまうと次 run が 304 で返り、残りを見ないまま終わる。`lastPolledAt` は進めるので run 自体は観測できる。 +**削除の枠.** 削除は 1 repo 1 run あたり `MAX_DOC_DELETIONS_PER_REPO_PER_RUN`（既定 5）で cap する。wiki の削除と同じ guard で、1 件あたり 3 subrequest かかるため、大量削除に対して上限なく回すと light cron の invocation 予算を単独で食い潰し、後ろに並ぶ repo を飢えさせうる — 1 つの PR が `.md` を 66 件削除すれば、その全件が 1 run に集中する。削除済み doc の store 行は消えるので残りの集合は縮む一方であり、drain は単調。したがってこの surface には per-run cap が 2 本あり、**どちらが効いても tree ETag は据え置く**: fetch 枠は未処理の変更 doc を残し（issue #149）、削除枠は未削除の doc を残す（issue #203）。どちらの場合も ETag を進めてしまうと次 run が 304 で返り、残りを見ないまま終わる。`lastPolledAt` は進めるので run 自体は観測できる。 wiki poller は `:45` cron 専属で、GitHub Wiki content の唯一の取り込み経路。Wiki は別 git repo (`{repo}.wiki.git`) に存在し、REST API も webhook event も持たないため、poller が repo ごとに 3 段の HTTP 呼び出しで処理する: @@ -299,7 +299,7 @@ Durable Object + SQLite は次の structured record を保持する。 - **エッジ抽出**: wiki ページ index 時（`processAndUpsertWikiDoc`）、同 repo の既知 wiki slug が本文に出現したら A→B の "mention" エッジを生成（`src/graph.ts` の `indexWikiEdges`）。**決定的 slug-match（LLM 不要・ロスなし）**。dst は計算で求まるので未 index でも記録可（dangling 可）。typed（supersede/depend/conflict）は将来スコープ。 - **traversal**: `queryNeighbors` が `WITH RECURSIVE`（標準 SQLite、拡張不要）で seed の 1–2 hop neighbor を無向に辿る。 - **retrieval 統合**: `search` の `graph_expand`（既定 false）/ `graph_hops`（既定 1）。true の時のみ、RRF 後の最終結果を seed に neighbor を辿り、関連 wiki ページを `graph_hop` / `graph_from` 付きで末尾に append。**false の時は既存挙動と完全同一（回帰なし）**。 -- **delete fan-out**: `deleteEdgesForVector` が当該 vector を端点に持つエッジを除去する。cron の削除は両方これを呼ぶ — 実際に行が存在する wiki 側と、doc vector ID が 1 行も一致しない repository docs 側（後者は将来 doc 側 edge writer が入ったときのための対称性、issue #203）。 +- **delete fan-out**: wiki ページ削除時に `deleteEdgesForVector`（当該 vector を端点に持つエッジを除去）。repository docs 側の削除はこれを呼ばない — doc vector ID はここの端点になりえないため（issue #203）。 - **backfill**: `POST /admin/backfill-edges?repo=owner/repo`（GITHUB_TOKEN ヘッダ）。既存 index 済み wiki の content から一括抽出（GitHub 再取得不要）。 - **評価**: 本番 ship 後の実運用観測（judgment-learning が関連判断を拾えるか）。offline eval harness は作らない。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 069bdd9..1304ffc 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -150,9 +150,9 @@ Both phases obey one watermark invariant: **a watermark never moves past a commi The tradeoff is liveness: a commit that fails on every attempt blocks its phase's watermark. The run log names the boundary commit, and `POST /admin/diff-watermark` (see the installation guide) moves the watermark manually — the same endpoint used to replay a period whose commits were lost by an earlier version of the poller. -The docs poller reads the repository tree with a conditional `If-None-Match` request and diffs it against the stored doc records: entries whose blob SHA moved are re-embedded, entries present in the store but absent from the tree are reaped. **The reap tears down four surfaces** — Vectorize, D1 FTS5, the graph edge table, and the structured store — each independently, so a Vectorize failure cannot strand the D1 rows users actually retrieve. The edge teardown is symmetry with the wiki reap rather than a backlog drain: `doc_edges` endpoints are wiki vector IDs on both sides today, so a doc vector ID matches no row. It is what makes "delete a vector" mean the same thing on both paths if repository docs ever become edge endpoints (issue #203). +The docs poller reads the repository tree with a conditional `If-None-Match` request and diffs it against the stored doc records: entries whose blob SHA moved are re-embedded, entries present in the store but absent from the tree are reaped. **The reap tears down three surfaces** — Vectorize, D1 FTS5, and the structured store — each independently, so a Vectorize failure cannot strand the D1 rows users actually retrieve. It is three and not the wiki reap's four because a doc vector ID is never a `doc_edges` endpoint: `indexWikiEdges` is the only writer and both the source and the computed destination IDs are wiki vector IDs. Add the edge teardown here if that invariant changes (issue #203). -**Reap budget.** The reap is capped at `MAX_DOC_DELETIONS_PER_REPO_PER_RUN` (default 5) per repo per run, the same guard the wiki reap carries: at ~4 subrequests per deletion an unbounded loop over a mass deletion could exhaust the light cron's invocation budget on its own and starve every repo behind it — a single PR removing 66 `.md` files puts all of them on one run. The drain is monotonic, since a reaped doc's store row is gone and the leftover set only shrinks. Two per-run caps therefore exist on this surface, and **either one holds the tree ETag back**: the fetch cap leaves changed docs unprocessed (issue #149) and the delete cap leaves deletions unreaped (issue #203), and in both cases advancing the ETag would make the next run answer 304 and return before it looked at the leftover. `lastPolledAt` still advances so the run stays observable. +**Reap budget.** The reap is capped at `MAX_DOC_DELETIONS_PER_REPO_PER_RUN` (default 5) per repo per run, the same guard the wiki reap carries: at 3 subrequests per deletion an unbounded loop over a mass deletion could exhaust the light cron's invocation budget on its own and starve every repo behind it — a single PR removing 66 `.md` files puts all of them on one run. The drain is monotonic, since a reaped doc's store row is gone and the leftover set only shrinks. Two per-run caps therefore exist on this surface, and **either one holds the tree ETag back**: the fetch cap leaves changed docs unprocessed (issue #149) and the delete cap leaves deletions unreaped (issue #203), and in both cases advancing the ETag would make the next run answer 304 and return before it looked at the leftover. `lastPolledAt` still advances so the run stays observable. The wiki poller runs in the `:45` cron and is the only ingestion path for GitHub Wiki content. Wiki pages live in a separate git repo (`{repo}.wiki.git`) that GitHub does not expose through the REST API or webhook events; the poller therefore performs three lightweight HTTP calls per repo: @@ -301,7 +301,7 @@ An additive graph layer that indexes relationships between Decision-Structure wi - **Edge extraction**: at wiki index time (`processAndUpsertWikiDoc`), when another known wiki slug in the same repo appears in the content, an A→B "mention" edge is written (`indexWikiEdges` in `src/graph.ts`). **Deterministic slug-match (no LLM, no lossy extraction)**; the dst ID is computed, so dangling edges to not-yet-indexed pages are allowed. Typed edges (supersede/depend/conflict) are future scope. - **Traversal**: `queryNeighbors` walks 1–2 hop undirected neighbors of the seeds via `WITH RECURSIVE` (standard SQLite, no extension). - **Retrieval integration**: `search` gains `graph_expand` (default false) / `graph_hops` (default 1). Only when true, the final result set seeds a traversal and related wiki pages are appended (tagged `graph_hop` / `graph_from`). **When false the behavior is byte-identical to before (no regression).** -- **Delete fan-out**: `deleteEdgesForVector` removes edges touching a vector. Both cron reaps call it — the wiki one, which is where rows actually exist, and the repository-docs one, where a doc vector ID matches nothing today and the call stands as symmetry against a future doc-edge writer (issue #203). +- **Delete fan-out**: on wiki page deletion, `deleteEdgesForVector` removes edges touching that vector. The repository-docs reap does not call it, because a doc vector ID is never an endpoint here (issue #203). - **Backfill**: `POST /admin/backfill-edges?repo=owner/repo` (GITHUB_TOKEN header) re-extracts edges from the stored content of already-indexed wiki pages (no GitHub refetch). - **Evaluation**: real-use observation after ship (does judgment-learning surface related decisions), not an offline eval harness. diff --git a/src/poller-docs.test.ts b/src/poller-docs.test.ts index f0402f2..a534bb3 100644 --- a/src/poller-docs.test.ts +++ b/src/poller-docs.test.ts @@ -2,19 +2,14 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import type { Env, DocRecord } from "./types.js"; // `pollDocs` fans out to the embed pipeline (Contents API + Workers AI + -// Vectorize + D1 + Store DO) and to the two D1 teardown helpers. What is under +// Vectorize + D1 + Store DO) and to the D1 FTS teardown helper. What is under // test here is the *reap* — the delete fan-out and its per-run budget — so the -// embed entry point and both teardown helpers are replaced with controllable +// embed entry point and the teardown helper are replaced with controllable // fakes and only the Git Trees API call reaches the stubbed global fetch. // `docVectorId` and the rest of `./pipeline.js` stay real. -const { - processAndUpsertDocMock, - deleteFtsRowMock, - deleteEdgesForVectorMock, -} = vi.hoisted(() => ({ +const { processAndUpsertDocMock, deleteFtsRowMock } = vi.hoisted(() => ({ processAndUpsertDocMock: vi.fn(), deleteFtsRowMock: vi.fn(), - deleteEdgesForVectorMock: vi.fn(), })); vi.mock("./pipeline.js", async (importOriginal) => { @@ -27,11 +22,6 @@ vi.mock("./fts.js", async (importOriginal) => { return { ...actual, deleteFtsRow: deleteFtsRowMock }; }); -vi.mock("./graph.js", async (importOriginal) => { - const actual = await importOriginal(); - return { ...actual, deleteEdgesForVector: deleteEdgesForVectorMock }; -}); - const { pollDocs } = await import("./poller.js"); const { docVectorId } = await import("./pipeline.js"); @@ -157,7 +147,6 @@ beforeEach(() => { processAndUpsertDocMock.mockReset(); processAndUpsertDocMock.mockResolvedValue({ embedded: true, failed: false }); deleteFtsRowMock.mockReset().mockResolvedValue(undefined); - deleteEdgesForVectorMock.mockReset().mockResolvedValue(undefined); }); afterEach(() => { @@ -165,10 +154,9 @@ afterEach(() => { }); describe("poller: pollDocs delete fan-out", () => { - it("tears down graph edges alongside the vector, the FTS5 row and the store record", async () => { - // One doc still in the store, gone from the tree. Pre-fix the reap hit three - // surfaces and left the edge table untouched, so a doc vector ID could stay - // an edge endpoint with nothing to remove it (issue #203). + it("reaps only the doc absent from the tree, on all three surfaces", async () => { + // One doc still in the store, gone from the tree. All three surfaces are + // keyed to the same `docVectorId`, and the surviving doc is left alone. stubTree(["docs/keep.md"]); const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); const { env, vectorDeletes } = makeDocEnv(); @@ -181,13 +169,14 @@ describe("poller: pollDocs delete fan-out", () => { expect(vectorDeletes).toEqual([goneId]); expect(deleteFtsRowMock).toHaveBeenCalledTimes(1); expect(deleteFtsRowMock.mock.calls[0][1]).toBe(goneId); - expect(deleteEdgesForVectorMock).toHaveBeenCalledTimes(1); - expect(deleteEdgesForVectorMock.mock.calls[0][1]).toBe(goneId); + expect(store.records.has("docs/keep.md")).toBe(true); }); it("keeps tearing down the later surfaces when Vectorize fails", async () => { - // The D1 rows are the ones users actually retrieve; a dense-side failure - // must not strand them. Same independence the wiki reap already has. + // The surfaces are torn down independently. Pre-fix one outer try wrapped + // the whole item, so a Vectorize failure skipped the store DELETE and left + // the row behind — while the FTS5 row, guarded by its own inner try, was + // already gone. The D1 rows are the ones users actually retrieve. stubTree(["docs/keep.md"]); const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); const { env } = makeDocEnv(); @@ -198,7 +187,18 @@ describe("poller: pollDocs delete fan-out", () => { await pollDocs(REPO, env, store.stub); expect(deleteFtsRowMock).toHaveBeenCalledTimes(1); - expect(deleteEdgesForVectorMock).toHaveBeenCalledTimes(1); + expect(store.deletes).toEqual(["docs/gone.md"]); + }); + + it("keeps tearing down the store record when the FTS5 delete fails", async () => { + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env, vectorDeletes } = makeDocEnv(); + deleteFtsRowMock.mockRejectedValue(new Error("d1 down")); + + await pollDocs(REPO, env, store.stub); + + expect(vectorDeletes).toHaveLength(1); expect(store.deletes).toEqual(["docs/gone.md"]); }); @@ -212,14 +212,13 @@ describe("poller: pollDocs delete fan-out", () => { expect(store.deletes).toEqual([]); expect(vectorDeletes).toEqual([]); expect(deleteFtsRowMock).not.toHaveBeenCalled(); - expect(deleteEdgesForVectorMock).not.toHaveBeenCalled(); }); }); describe("poller: pollDocs delete budget", () => { it("caps deletions per run and drains the rest on later runs", async () => { // The case that raised the issue: a single PR removed 66 `.md` files, all of - // which land on the next run. Unbounded, that is ~4 subrequests x 66 in one + // which land on the next run. Unbounded, that is 3 subrequests x 66 in one // LIGHT_CRON invocation shared with pollRepo and pollReleases (issue #203). const deletedPaths = Array.from( { length: DELETE_BUDGET * 2 + 1 }, diff --git a/src/poller.ts b/src/poller.ts index 9e65789..6f4cbb2 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -107,17 +107,17 @@ const MAX_COMMENT_FETCHES_PER_REPO_PER_RUN = 10; * unchanged in the store until the doc is successfully upserted). */ const MAX_DOC_FETCHES_PER_REPO_PER_RUN = 10; -/** Maximum docs reaped (Vectorize + FTS5 + graph edges + store row) per repo per - * cron run. Mirrors `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`: each reap fans out - * to 4 subrequests, so an unbounded loop over a mass deletion could exhaust the +/** Maximum docs reaped (Vectorize + FTS5 + store row) per repo per cron run. + * Mirrors `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`: each reap fans out to 3 + * subrequests, so an unbounded loop over a mass deletion could exhaust the * LIGHT_CRON invocation budget on its own and starve every repo behind it — * docs share that invocation with `pollRepo` and `pollReleases` (issue #203). * A real case: github-webhook-mcp deleted 66 `.md` files in one PR, all of * which land on the next single run. * - * Numeric design: 5 deletes x 4 subrequests x 5 repos = 100 subrequests for the + * Numeric design: 5 deletes x 3 subrequests x 5 repos = 75 subrequests for the * reap, on top of the ~250 the docs fetch cap already allows. Together with - * pollRepo and pollReleases the LIGHT_CRON worst case stays around 850, inside + * pollRepo and pollReleases the LIGHT_CRON worst case stays around 825, inside * the 1000 ceiling. * * Remaining deletions are reaped next run: a reaped doc's store row is gone, so @@ -810,9 +810,9 @@ async function fetchFileContent( * Uses Git Trees API for change detection and Contents API for fetching changed files. * * Files present in the store but absent from the current tree are reaped from - * Vectorize, D1 FTS5, the graph edge table, and the structured store — capped - * per repo per run, with the tree ETag held back while a backlog remains so the - * next run can still see it (issue #203). + * Vectorize, D1 FTS5, and the structured store — each surface independently, + * capped per repo per run, with the tree ETag held back while a backlog remains + * so the next run can still see it (issue #203). * * Exported for tests; production callers reach it through `handleScheduled`. */ @@ -944,16 +944,12 @@ export async function pollDocs( } } - // Handle deleted files: remove from Vectorize, D1 FTS5, the graph edge table, - // and the structured store — the same four surfaces the wiki reap tears down. - // - // The edge teardown is a symmetry guard, not a backlog drain. Today every - // `doc_edges` endpoint is a wiki vector ID on both sides (`indexWikiEdges` is - // the only writer and `knownWikiSlugs` filters `type = 'wiki_doc'`), so a doc - // vector ID matches no row and the DELETE is a no-op. What it buys is that - // "delete a vector" means the same thing on both paths: if repository docs - // ever become edge endpoints, the reap already tears them down instead of - // leaking rows that retrieval silently skips as dangling (issue #203). + // Handle deleted files: remove from Vectorize, D1 FTS5, and the structured + // store. No graph-edge teardown here, unlike the wiki reap: both endpoints of + // every `doc_edges` row are wiki vector IDs (`indexWikiEdges` is the only + // writer, and the dst ID it computes is a `wikiDocVectorId` too), so a doc + // vector ID is never an edge endpoint. Add the teardown here if that + // invariant changes (issue #203). let removedDocs = 0; let deleteBudgetExhausted = false; for (const doc of deletedDocs) { @@ -973,7 +969,6 @@ export async function pollDocs( for (const [surface, run] of [ ["vector", () => env.VECTORIZE.deleteByIds([dvid])], ["FTS5 row", () => deleteFtsRow(env.DB_FTS, dvid)], - ["graph edges", () => deleteEdgesForVector(env.DB_FTS, dvid)], [ "store record", () =>