fix(docs): tear down doc reap surfaces independently and cap it per run [poller, docs, tests]

docs/0-requirements.ja.md

@@ -149,6 +149,10 @@ commit diff poller は 2-phase 構成:
 
 トレードオフは liveness で、毎回失敗し続ける commit はその phase の watermark を止める。run log に境界 commit の SHA が出るほか、`POST /admin/diff-watermark`（installation guide 参照）で watermark を手動で移動できる。旧版 poller が取りこぼした期間を再走査させる経路も同じ endpoint。
 
+docs poller は `If-None-Match` 付きの条件付きリクエストで repository tree を読み、保存済みの doc record と差分を取る。blob SHA が動いた entry は re-embed し、store にあって tree に無い entry は削除する。**削除は 3 面を teardown する** — Vectorize / D1 FTS5 / structured store — それぞれ独立に実行するので、Vectorize の失敗が実際に retrieval される D1 行を取り残すことはない。wiki 側の 4 面に対してここが 3 面なのは、doc vector ID が `doc_edges` の端点になりえないため。`indexWikiEdges` が唯一の writer であり、src 側も算出される dst 側も wiki vector ID になる。この不変条件が変わったら、ここに edge の teardown を足すこと（issue #203）。
+
+**削除の枠.** 削除は 1 repo 1 run あたり `MAX_DOC_DELETIONS_PER_REPO_PER_RUN`（既定 5）で cap する。wiki の削除と同じ guard で、1 件あたり 3 subrequest かかるため、大量削除に対して上限なく回すと light cron の invocation 予算を単独で食い潰し、後ろに並ぶ repo を飢えさせうる — 1 つの PR が `.md` を 66 件削除すれば、その全件が 1 run に集中する。削除済み doc の store 行は消えるので残りの集合は縮む一方であり、drain は単調。したがってこの surface には per-run cap が 2 本あり、**どちらが効いても tree ETag は据え置く**: fetch 枠は未処理の変更 doc を残し（issue #149）、削除枠は未削除の doc を残す（issue #203）。どちらの場合も ETag を進めてしまうと次 run が 304 で返り、残りを見ないまま終わる。`lastPolledAt` は進めるので run 自体は観測できる。
+
 wiki poller は `:45` cron 専属で、GitHub Wiki content の唯一の取り込み経路。Wiki は別 git repo (`{repo}.wiki.git`) に存在し、REST API も webhook event も持たないため、poller が repo ごとに 3 段の HTTP 呼び出しで処理する:
 
 1. `https://github.com/{repo}.wiki.git/info/refs?service=git-upload-pack` を打って wiki 存在検出（200 = 存在、404 = 無効化済 or 未設置 = skip）。
@@ -295,7 +299,7 @@ Durable Object + SQLite は次の structured record を保持する。
 - **エッジ抽出**: wiki ページ index 時（`processAndUpsertWikiDoc`）、同 repo の既知 wiki slug が本文に出現したら A→B の "mention" エッジを生成（`src/graph.ts` の `indexWikiEdges`）。**決定的 slug-match（LLM 不要・ロスなし）**。dst は計算で求まるので未 index でも記録可（dangling 可）。typed（supersede/depend/conflict）は将来スコープ。
 - **traversal**: `queryNeighbors` が `WITH RECURSIVE`（標準 SQLite、拡張不要）で seed の 1–2 hop neighbor を無向に辿る。
 - **retrieval 統合**: `search` の `graph_expand`（既定 false）/ `graph_hops`（既定 1）。true の時のみ、RRF 後の最終結果を seed に neighbor を辿り、関連 wiki ページを `graph_hop` / `graph_from` 付きで末尾に append。**false の時は既存挙動と完全同一（回帰なし）**。
-- **delete fan-out**: wiki ページ削除時に `deleteEdgesForVector`（当該 vector を端点に持つエッジを除去）。
+- **delete fan-out**: wiki ページ削除時に `deleteEdgesForVector`（当該 vector を端点に持つエッジを除去）。repository docs 側の削除はこれを呼ばない — doc vector ID はここの端点になりえないため（issue #203）。
 - **backfill**: `POST /admin/backfill-edges?repo=owner/repo`（GITHUB_TOKEN ヘッダ）。既存 index 済み wiki の content から一括抽出（GitHub 再取得不要）。
 - **評価**: 本番 ship 後の実運用観測（judgment-learning が関連判断を拾えるか）。offline eval harness は作らない。
 

docs/0-requirements.md

@@ -150,6 +150,10 @@ Both phases obey one watermark invariant: **a watermark never moves past a commi
 
 The tradeoff is liveness: a commit that fails on every attempt blocks its phase's watermark. The run log names the boundary commit, and `POST /admin/diff-watermark` (see the installation guide) moves the watermark manually — the same endpoint used to replay a period whose commits were lost by an earlier version of the poller.
 
+The docs poller reads the repository tree with a conditional `If-None-Match` request and diffs it against the stored doc records: entries whose blob SHA moved are re-embedded, entries present in the store but absent from the tree are reaped. **The reap tears down three surfaces** — Vectorize, D1 FTS5, and the structured store — each independently, so a Vectorize failure cannot strand the D1 rows users actually retrieve. It is three and not the wiki reap's four because a doc vector ID is never a `doc_edges` endpoint: `indexWikiEdges` is the only writer and both the source and the computed destination IDs are wiki vector IDs. Add the edge teardown here if that invariant changes (issue #203).
+
+**Reap budget.** The reap is capped at `MAX_DOC_DELETIONS_PER_REPO_PER_RUN` (default 5) per repo per run, the same guard the wiki reap carries: at 3 subrequests per deletion an unbounded loop over a mass deletion could exhaust the light cron's invocation budget on its own and starve every repo behind it — a single PR removing 66 `.md` files puts all of them on one run. The drain is monotonic, since a reaped doc's store row is gone and the leftover set only shrinks. Two per-run caps therefore exist on this surface, and **either one holds the tree ETag back**: the fetch cap leaves changed docs unprocessed (issue #149) and the delete cap leaves deletions unreaped (issue #203), and in both cases advancing the ETag would make the next run answer 304 and return before it looked at the leftover. `lastPolledAt` still advances so the run stays observable.
+
 The wiki poller runs in the `:45` cron and is the only ingestion path for GitHub Wiki content. Wiki pages live in a separate git repo (`{repo}.wiki.git`) that GitHub does not expose through the REST API or webhook events; the poller therefore performs three lightweight HTTP calls per repo:
 
 1. Probe `https://github.com/{repo}.wiki.git/info/refs?service=git-upload-pack` to detect whether the wiki exists (200 = present, 404 = disabled or absent — skip the rest).
@@ -297,7 +301,7 @@ An additive graph layer that indexes relationships between Decision-Structure wi
 - **Edge extraction**: at wiki index time (`processAndUpsertWikiDoc`), when another known wiki slug in the same repo appears in the content, an A→B "mention" edge is written (`indexWikiEdges` in `src/graph.ts`). **Deterministic slug-match (no LLM, no lossy extraction)**; the dst ID is computed, so dangling edges to not-yet-indexed pages are allowed. Typed edges (supersede/depend/conflict) are future scope.
 - **Traversal**: `queryNeighbors` walks 1–2 hop undirected neighbors of the seeds via `WITH RECURSIVE` (standard SQLite, no extension).
 - **Retrieval integration**: `search` gains `graph_expand` (default false) / `graph_hops` (default 1). Only when true, the final result set seeds a traversal and related wiki pages are appended (tagged `graph_hop` / `graph_from`). **When false the behavior is byte-identical to before (no regression).**
-- **Delete fan-out**: on wiki page deletion, `deleteEdgesForVector` removes edges touching that vector.
+- **Delete fan-out**: on wiki page deletion, `deleteEdgesForVector` removes edges touching that vector. The repository-docs reap does not call it, because a doc vector ID is never an endpoint here (issue #203).
 - **Backfill**: `POST /admin/backfill-edges?repo=owner/repo` (GITHUB_TOKEN header) re-extracts edges from the stored content of already-indexed wiki pages (no GitHub refetch).
 - **Evaluation**: real-use observation after ship (does judgment-learning surface related decisions), not an offline eval harness.
 

src/poller-docs.test.ts

@@ -0,0 +1,287 @@
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import type { Env, DocRecord } from "./types.js";
+
+// `pollDocs` fans out to the embed pipeline (Contents API + Workers AI +
+// Vectorize + D1 + Store DO) and to the D1 FTS teardown helper. What is under
+// test here is the *reap* — the delete fan-out and its per-run budget — so the
+// embed entry point and the teardown helper are replaced with controllable
+// fakes and only the Git Trees API call reaches the stubbed global fetch.
+// `docVectorId` and the rest of `./pipeline.js` stay real.
+const { processAndUpsertDocMock, deleteFtsRowMock } = vi.hoisted(() => ({
+  processAndUpsertDocMock: vi.fn(),
+  deleteFtsRowMock: vi.fn(),
+}));
+
+vi.mock("./pipeline.js", async (importOriginal) => {
+  const actual = await importOriginal<typeof import("./pipeline.js")>();
+  return { ...actual, processAndUpsertDoc: processAndUpsertDocMock };
+});
+
+vi.mock("./fts.js", async (importOriginal) => {
+  const actual = await importOriginal<typeof import("./fts.js")>();
+  return { ...actual, deleteFtsRow: deleteFtsRowMock };
+});
+
+const { pollDocs } = await import("./poller.js");
+const { docVectorId } = await import("./pipeline.js");
+
+const REPO = "acme/widgets";
+const WATERMARK_KEY = `docs:${REPO}`;
+const TREE_URL = `https://api.github.com/repos/${REPO}/git/trees/HEAD?recursive=1`;
+
+/** Constant mirrored from `src/poller.ts`. */
+const DELETE_BUDGET = 5;
+
+/**
+ * Stub the global fetch with a fake Git Trees API returning `paths` as blobs.
+ *
+ * `etag` is what the response carries; `expectNotModified` makes the stub assert
+ * the conditional-request contract by returning 304 when the poller sends back
+ * the matching `If-None-Match`.
+ */
+function stubTree(paths: string[], etag = 'W/"tree-1"') {
+  const conditionalHits: string[] = [];
+
+  const fetchMock = vi.fn(async (input: string | URL, init?: RequestInit) => {
+    const url = String(input);
+    if (url !== TREE_URL) {
+      throw new Error(`unexpected fetch in docs stub: ${url}`);
+    }
+    const headers = (init?.headers ?? {}) as Record<string, string>;
+    const sent = headers["If-None-Match"];
+    if (sent) conditionalHits.push(sent);
+    if (sent === etag) {
+      return new Response(null, { status: 304 });
+    }
+    return new Response(
+      JSON.stringify({
+        sha: "treesha",
+        truncated: false,
+        tree: paths.map((path) => ({ path, type: "blob", sha: `blob-${path}` })),
+      }),
+      { status: 200, headers: { "Content-Type": "application/json", ETag: etag } },
+    );
+  });
+
+  vi.stubGlobal("fetch", fetchMock);
+  return { fetchMock, conditionalHits };
+}
+
+/**
+ * In-memory IssueStore stand-in covering the docs surface: the record list the
+ * poller diffs against, the docs watermark row holding the tree ETag, and the
+ * per-path DELETE the reap issues.
+ */
+function makeDocStore(seed: DocRecord[] = [], etag?: string) {
+  const records = new Map<string, DocRecord>(seed.map((d) => [d.path, d]));
+  const watermarks = new Map<string, { lastPolledAt: string; etag?: string }>();
+  if (etag !== undefined) {
+    watermarks.set(WATERMARK_KEY, { lastPolledAt: "2026-08-01T00:00:00Z", etag });
+  }
+  const deletes: string[] = [];
+
+  const stub = {
+    async fetch(request: Request): Promise<Response> {
+      const url = new URL(request.url);
+      const path = url.pathname;
+
+      if (request.method === "GET" && path === "/docs") {
+        return Response.json([...records.values()]);
+      }
+      if (request.method === "GET" && path === "/watermark") {
+        const key = url.searchParams.get("repo") ?? "";
+        const wm = watermarks.get(key);
+        if (!wm) return new Response("not found", { status: 404 });
+        return Response.json({ repo: key, ...wm });
+      }
+      if (request.method === "POST" && path === "/watermark") {
+        const body = (await request.json()) as {
+          repo: string;
+          lastPolledAt: string;
+          etag?: string;
+        };
+        watermarks.set(body.repo, { lastPolledAt: body.lastPolledAt, etag: body.etag });
+        return new Response("ok");
+      }
+      if (request.method === "DELETE" && path === "/doc") {
+        const docPath = url.searchParams.get("path") ?? "";
+        deletes.push(docPath);
+        records.delete(docPath);
+        return new Response("ok");
+      }
+      return new Response("ok");
+    },
+  };
+
+  return {
+    stub: stub as unknown as DurableObjectStub,
+    records,
+    deletes,
+    etag: () => watermarks.get(WATERMARK_KEY)?.etag,
+  };
+}
+
+function makeDocEnv() {
+  const vectorDeletes: string[] = [];
+  const deleteByIds = vi.fn(async (ids: string[]) => {
+    vectorDeletes.push(...ids);
+  });
+  const env = {
+    GITHUB_TOKEN: "test-token",
+    VECTORIZE: { deleteByIds },
+    DB_FTS: {} as unknown,
+  } as unknown as Env;
+  return { env, vectorDeletes };
+}
+
+/** A stored doc whose blob SHA matches what `stubTree` reports, so it is
+ *  "unchanged" and never enters the embed path. */
+const stored = (path: string): DocRecord => ({
+  repo: REPO,
+  path,
+  blobSha: `blob-${path}`,
+  updatedAt: "2026-08-01T00:00:00Z",
+});
+
+beforeEach(() => {
+  processAndUpsertDocMock.mockReset();
+  processAndUpsertDocMock.mockResolvedValue({ embedded: true, failed: false });
+  deleteFtsRowMock.mockReset().mockResolvedValue(undefined);
+});
+
+afterEach(() => {
+  vi.unstubAllGlobals();
+});
+
+describe("poller: pollDocs delete fan-out", () => {
+  it("reaps only the doc absent from the tree, on all three surfaces", async () => {
+    // One doc still in the store, gone from the tree. All three surfaces are
+    // keyed to the same `docVectorId`, and the surviving doc is left alone.
+    stubTree(["docs/keep.md"]);
+    const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]);
+    const { env, vectorDeletes } = makeDocEnv();
+
+    await pollDocs(REPO, env, store.stub);
+
+    const goneId = await docVectorId(REPO, "docs/gone.md");
+
+    expect(store.deletes).toEqual(["docs/gone.md"]);
+    expect(vectorDeletes).toEqual([goneId]);
+    expect(deleteFtsRowMock).toHaveBeenCalledTimes(1);
+    expect(deleteFtsRowMock.mock.calls[0][1]).toBe(goneId);
+    expect(store.records.has("docs/keep.md")).toBe(true);
+  });
+
+  it("keeps tearing down the later surfaces when Vectorize fails", async () => {
+    // The surfaces are torn down independently. Pre-fix one outer try wrapped
+    // the whole item, so a Vectorize failure skipped the store DELETE and left
+    // the row behind — while the FTS5 row, guarded by its own inner try, was
+    // already gone. The D1 rows are the ones users actually retrieve.
+    stubTree(["docs/keep.md"]);
+    const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]);
+    const { env } = makeDocEnv();
+    (env.VECTORIZE.deleteByIds as ReturnType<typeof vi.fn>).mockRejectedValue(
+      new Error("vectorize down"),
+    );
+
+    await pollDocs(REPO, env, store.stub);
+
+    expect(deleteFtsRowMock).toHaveBeenCalledTimes(1);
+    expect(store.deletes).toEqual(["docs/gone.md"]);
+  });
+
+  it("keeps tearing down the store record when the FTS5 delete fails", async () => {
+    stubTree(["docs/keep.md"]);
+    const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]);
+    const { env, vectorDeletes } = makeDocEnv();
+    deleteFtsRowMock.mockRejectedValue(new Error("d1 down"));
+
+    await pollDocs(REPO, env, store.stub);
+
+    expect(vectorDeletes).toHaveLength(1);
+    expect(store.deletes).toEqual(["docs/gone.md"]);
+  });
+
+  it("leaves the reap alone when nothing was deleted", async () => {
+    stubTree(["docs/keep.md"]);
+    const store = makeDocStore([stored("docs/keep.md")]);
+    const { env, vectorDeletes } = makeDocEnv();
+
+    await pollDocs(REPO, env, store.stub);
+
+    expect(store.deletes).toEqual([]);
+    expect(vectorDeletes).toEqual([]);
+    expect(deleteFtsRowMock).not.toHaveBeenCalled();
+  });
+});
+
+describe("poller: pollDocs delete budget", () => {
+  it("caps deletions per run and drains the rest on later runs", async () => {
+    // The case that raised the issue: a single PR removed 66 `.md` files, all of
+    // which land on the next run. Unbounded, that is 3 subrequests x 66 in one
+    // LIGHT_CRON invocation shared with pollRepo and pollReleases (issue #203).
+    const deletedPaths = Array.from(
+      { length: DELETE_BUDGET * 2 + 1 },
+      (_, i) => `docs/gone-${String(i).padStart(2, "0")}.md`,
+    );
+    const store = makeDocStore([stored("docs/keep.md"), ...deletedPaths.map(stored)]);
+    const { env } = makeDocEnv();
+
+    stubTree(["docs/keep.md"]);
+    await pollDocs(REPO, env, store.stub);
+    expect(store.deletes).toHaveLength(DELETE_BUDGET);
+
+    // Monotonic drain: a reaped doc's store row is gone, so the leftover set
+    // only shrinks. No path is reaped twice and none is skipped.
+    stubTree(["docs/keep.md"]);
+    await pollDocs(REPO, env, store.stub);
+    expect(store.deletes).toHaveLength(DELETE_BUDGET * 2);
+
+    stubTree(["docs/keep.md"]);
+    await pollDocs(REPO, env, store.stub);
+    expect(store.deletes).toHaveLength(deletedPaths.length);
+
+    expect([...store.deletes].sort()).toEqual([...deletedPaths].sort());
+    expect(new Set(store.deletes).size).toBe(deletedPaths.length);
+    expect(store.records.has("docs/keep.md")).toBe(true);
+    expect(store.records.size).toBe(1);
+  });
+
+  it("holds the tree ETag back while deletions are outstanding", async () => {
+    // Advancing the ETag with a backlog left would make the next run answer 304
+    // and return before it ever looks at `deletedDocs` — the drain would stall
+    // until the tree happened to change again (issue #203).
+    const deletedPaths = Array.from(
+      { length: DELETE_BUDGET + 1 },
+      (_, i) => `docs/gone-${i}.md`,
+    );
+    const store = makeDocStore([stored("docs/keep.md"), ...deletedPaths.map(stored)]);
+    const { env } = makeDocEnv();
+
+    stubTree(["docs/keep.md"]);
+    await pollDocs(REPO, env, store.stub);
+    expect(store.deletes).toHaveLength(DELETE_BUDGET);
+    expect(store.etag()).toBeUndefined();
+
+    // Next run: the poller sends no If-None-Match, so it sees the tree again and
+    // reaps the leftover.
+    const { conditionalHits } = stubTree(["docs/keep.md"]);
+    await pollDocs(REPO, env, store.stub);
+    expect(conditionalHits).toEqual([]);
+    expect(store.deletes).toHaveLength(deletedPaths.length);
+
+    // Backlog cleared — now the ETag is allowed to advance.
+    expect(store.etag()).toBe('W/"tree-1"');
+  });
+
+  it("advances the tree ETag when the reap finished inside its budget", async () => {
+    stubTree(["docs/keep.md"]);
+    const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]);
+    const { env } = makeDocEnv();
+
+    await pollDocs(REPO, env, store.stub);
+
+    expect(store.deletes).toEqual(["docs/gone.md"]);
+    expect(store.etag()).toBe('W/"tree-1"');
+  });
+});