diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index e9a0bf6..8df2363 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -149,6 +149,10 @@ commit diff poller は 2-phase 構成: トレードオフは liveness で、毎回失敗し続ける commit はその phase の watermark を止める。run log に境界 commit の SHA が出るほか、`POST /admin/diff-watermark`(installation guide 参照)で watermark を手動で移動できる。旧版 poller が取りこぼした期間を再走査させる経路も同じ endpoint。 +docs poller は `If-None-Match` 付きの条件付きリクエストで repository tree を読み、保存済みの doc record と差分を取る。blob SHA が動いた entry は re-embed し、store にあって tree に無い entry は削除する。**削除は 3 面を teardown する** — Vectorize / D1 FTS5 / structured store — それぞれ独立に実行するので、Vectorize の失敗が実際に retrieval される D1 行を取り残すことはない。wiki 側の 4 面に対してここが 3 面なのは、doc vector ID が `doc_edges` の端点になりえないため。`indexWikiEdges` が唯一の writer であり、src 側も算出される dst 側も wiki vector ID になる。この不変条件が変わったら、ここに edge の teardown を足すこと(issue #203)。 + +**削除の枠.** 削除は 1 repo 1 run あたり `MAX_DOC_DELETIONS_PER_REPO_PER_RUN`(既定 5)で cap する。wiki の削除と同じ guard で、1 件あたり 3 subrequest かかるため、大量削除に対して上限なく回すと light cron の invocation 予算を単独で食い潰し、後ろに並ぶ repo を飢えさせうる — 1 つの PR が `.md` を 66 件削除すれば、その全件が 1 run に集中する。削除済み doc の store 行は消えるので残りの集合は縮む一方であり、drain は単調。したがってこの surface には per-run cap が 2 本あり、**どちらが効いても tree ETag は据え置く**: fetch 枠は未処理の変更 doc を残し(issue #149)、削除枠は未削除の doc を残す(issue #203)。どちらの場合も ETag を進めてしまうと次 run が 304 で返り、残りを見ないまま終わる。`lastPolledAt` は進めるので run 自体は観測できる。 + wiki poller は `:45` cron 専属で、GitHub Wiki content の唯一の取り込み経路。Wiki は別 git repo (`{repo}.wiki.git`) に存在し、REST API も webhook event も持たないため、poller が repo ごとに 3 段の HTTP 呼び出しで処理する: 1. `https://github.com/{repo}.wiki.git/info/refs?service=git-upload-pack` を打って wiki 存在検出(200 = 存在、404 = 無効化済 or 未設置 = skip)。 @@ -295,7 +299,7 @@ Durable Object + SQLite は次の structured record を保持する。 - **エッジ抽出**: wiki ページ index 時(`processAndUpsertWikiDoc`)、同 repo の既知 wiki slug が本文に出現したら A→B の "mention" エッジを生成(`src/graph.ts` の `indexWikiEdges`)。**決定的 slug-match(LLM 不要・ロスなし)**。dst は計算で求まるので未 index でも記録可(dangling 可)。typed(supersede/depend/conflict)は将来スコープ。 - **traversal**: `queryNeighbors` が `WITH RECURSIVE`(標準 SQLite、拡張不要)で seed の 1–2 hop neighbor を無向に辿る。 - **retrieval 統合**: `search` の `graph_expand`(既定 false)/ `graph_hops`(既定 1)。true の時のみ、RRF 後の最終結果を seed に neighbor を辿り、関連 wiki ページを `graph_hop` / `graph_from` 付きで末尾に append。**false の時は既存挙動と完全同一(回帰なし)**。 -- **delete fan-out**: wiki ページ削除時に `deleteEdgesForVector`(当該 vector を端点に持つエッジを除去)。 +- **delete fan-out**: wiki ページ削除時に `deleteEdgesForVector`(当該 vector を端点に持つエッジを除去)。repository docs 側の削除はこれを呼ばない — doc vector ID はここの端点になりえないため(issue #203)。 - **backfill**: `POST /admin/backfill-edges?repo=owner/repo`(GITHUB_TOKEN ヘッダ)。既存 index 済み wiki の content から一括抽出(GitHub 再取得不要)。 - **評価**: 本番 ship 後の実運用観測(judgment-learning が関連判断を拾えるか)。offline eval harness は作らない。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 7a96156..1304ffc 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -150,6 +150,10 @@ Both phases obey one watermark invariant: **a watermark never moves past a commi The tradeoff is liveness: a commit that fails on every attempt blocks its phase's watermark. The run log names the boundary commit, and `POST /admin/diff-watermark` (see the installation guide) moves the watermark manually — the same endpoint used to replay a period whose commits were lost by an earlier version of the poller. +The docs poller reads the repository tree with a conditional `If-None-Match` request and diffs it against the stored doc records: entries whose blob SHA moved are re-embedded, entries present in the store but absent from the tree are reaped. **The reap tears down three surfaces** — Vectorize, D1 FTS5, and the structured store — each independently, so a Vectorize failure cannot strand the D1 rows users actually retrieve. It is three and not the wiki reap's four because a doc vector ID is never a `doc_edges` endpoint: `indexWikiEdges` is the only writer and both the source and the computed destination IDs are wiki vector IDs. Add the edge teardown here if that invariant changes (issue #203). + +**Reap budget.** The reap is capped at `MAX_DOC_DELETIONS_PER_REPO_PER_RUN` (default 5) per repo per run, the same guard the wiki reap carries: at 3 subrequests per deletion an unbounded loop over a mass deletion could exhaust the light cron's invocation budget on its own and starve every repo behind it — a single PR removing 66 `.md` files puts all of them on one run. The drain is monotonic, since a reaped doc's store row is gone and the leftover set only shrinks. Two per-run caps therefore exist on this surface, and **either one holds the tree ETag back**: the fetch cap leaves changed docs unprocessed (issue #149) and the delete cap leaves deletions unreaped (issue #203), and in both cases advancing the ETag would make the next run answer 304 and return before it looked at the leftover. `lastPolledAt` still advances so the run stays observable. + The wiki poller runs in the `:45` cron and is the only ingestion path for GitHub Wiki content. Wiki pages live in a separate git repo (`{repo}.wiki.git`) that GitHub does not expose through the REST API or webhook events; the poller therefore performs three lightweight HTTP calls per repo: 1. Probe `https://github.com/{repo}.wiki.git/info/refs?service=git-upload-pack` to detect whether the wiki exists (200 = present, 404 = disabled or absent — skip the rest). @@ -297,7 +301,7 @@ An additive graph layer that indexes relationships between Decision-Structure wi - **Edge extraction**: at wiki index time (`processAndUpsertWikiDoc`), when another known wiki slug in the same repo appears in the content, an A→B "mention" edge is written (`indexWikiEdges` in `src/graph.ts`). **Deterministic slug-match (no LLM, no lossy extraction)**; the dst ID is computed, so dangling edges to not-yet-indexed pages are allowed. Typed edges (supersede/depend/conflict) are future scope. - **Traversal**: `queryNeighbors` walks 1–2 hop undirected neighbors of the seeds via `WITH RECURSIVE` (standard SQLite, no extension). - **Retrieval integration**: `search` gains `graph_expand` (default false) / `graph_hops` (default 1). Only when true, the final result set seeds a traversal and related wiki pages are appended (tagged `graph_hop` / `graph_from`). **When false the behavior is byte-identical to before (no regression).** -- **Delete fan-out**: on wiki page deletion, `deleteEdgesForVector` removes edges touching that vector. +- **Delete fan-out**: on wiki page deletion, `deleteEdgesForVector` removes edges touching that vector. The repository-docs reap does not call it, because a doc vector ID is never an endpoint here (issue #203). - **Backfill**: `POST /admin/backfill-edges?repo=owner/repo` (GITHUB_TOKEN header) re-extracts edges from the stored content of already-indexed wiki pages (no GitHub refetch). - **Evaluation**: real-use observation after ship (does judgment-learning surface related decisions), not an offline eval harness. diff --git a/src/poller-docs.test.ts b/src/poller-docs.test.ts new file mode 100644 index 0000000..a534bb3 --- /dev/null +++ b/src/poller-docs.test.ts @@ -0,0 +1,287 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import type { Env, DocRecord } from "./types.js"; + +// `pollDocs` fans out to the embed pipeline (Contents API + Workers AI + +// Vectorize + D1 + Store DO) and to the D1 FTS teardown helper. What is under +// test here is the *reap* — the delete fan-out and its per-run budget — so the +// embed entry point and the teardown helper are replaced with controllable +// fakes and only the Git Trees API call reaches the stubbed global fetch. +// `docVectorId` and the rest of `./pipeline.js` stay real. +const { processAndUpsertDocMock, deleteFtsRowMock } = vi.hoisted(() => ({ + processAndUpsertDocMock: vi.fn(), + deleteFtsRowMock: vi.fn(), +})); + +vi.mock("./pipeline.js", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, processAndUpsertDoc: processAndUpsertDocMock }; +}); + +vi.mock("./fts.js", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, deleteFtsRow: deleteFtsRowMock }; +}); + +const { pollDocs } = await import("./poller.js"); +const { docVectorId } = await import("./pipeline.js"); + +const REPO = "acme/widgets"; +const WATERMARK_KEY = `docs:${REPO}`; +const TREE_URL = `https://api.github.com/repos/${REPO}/git/trees/HEAD?recursive=1`; + +/** Constant mirrored from `src/poller.ts`. */ +const DELETE_BUDGET = 5; + +/** + * Stub the global fetch with a fake Git Trees API returning `paths` as blobs. + * + * `etag` is what the response carries; `expectNotModified` makes the stub assert + * the conditional-request contract by returning 304 when the poller sends back + * the matching `If-None-Match`. + */ +function stubTree(paths: string[], etag = 'W/"tree-1"') { + const conditionalHits: string[] = []; + + const fetchMock = vi.fn(async (input: string | URL, init?: RequestInit) => { + const url = String(input); + if (url !== TREE_URL) { + throw new Error(`unexpected fetch in docs stub: ${url}`); + } + const headers = (init?.headers ?? {}) as Record; + const sent = headers["If-None-Match"]; + if (sent) conditionalHits.push(sent); + if (sent === etag) { + return new Response(null, { status: 304 }); + } + return new Response( + JSON.stringify({ + sha: "treesha", + truncated: false, + tree: paths.map((path) => ({ path, type: "blob", sha: `blob-${path}` })), + }), + { status: 200, headers: { "Content-Type": "application/json", ETag: etag } }, + ); + }); + + vi.stubGlobal("fetch", fetchMock); + return { fetchMock, conditionalHits }; +} + +/** + * In-memory IssueStore stand-in covering the docs surface: the record list the + * poller diffs against, the docs watermark row holding the tree ETag, and the + * per-path DELETE the reap issues. + */ +function makeDocStore(seed: DocRecord[] = [], etag?: string) { + const records = new Map(seed.map((d) => [d.path, d])); + const watermarks = new Map(); + if (etag !== undefined) { + watermarks.set(WATERMARK_KEY, { lastPolledAt: "2026-08-01T00:00:00Z", etag }); + } + const deletes: string[] = []; + + const stub = { + async fetch(request: Request): Promise { + const url = new URL(request.url); + const path = url.pathname; + + if (request.method === "GET" && path === "/docs") { + return Response.json([...records.values()]); + } + if (request.method === "GET" && path === "/watermark") { + const key = url.searchParams.get("repo") ?? ""; + const wm = watermarks.get(key); + if (!wm) return new Response("not found", { status: 404 }); + return Response.json({ repo: key, ...wm }); + } + if (request.method === "POST" && path === "/watermark") { + const body = (await request.json()) as { + repo: string; + lastPolledAt: string; + etag?: string; + }; + watermarks.set(body.repo, { lastPolledAt: body.lastPolledAt, etag: body.etag }); + return new Response("ok"); + } + if (request.method === "DELETE" && path === "/doc") { + const docPath = url.searchParams.get("path") ?? ""; + deletes.push(docPath); + records.delete(docPath); + return new Response("ok"); + } + return new Response("ok"); + }, + }; + + return { + stub: stub as unknown as DurableObjectStub, + records, + deletes, + etag: () => watermarks.get(WATERMARK_KEY)?.etag, + }; +} + +function makeDocEnv() { + const vectorDeletes: string[] = []; + const deleteByIds = vi.fn(async (ids: string[]) => { + vectorDeletes.push(...ids); + }); + const env = { + GITHUB_TOKEN: "test-token", + VECTORIZE: { deleteByIds }, + DB_FTS: {} as unknown, + } as unknown as Env; + return { env, vectorDeletes }; +} + +/** A stored doc whose blob SHA matches what `stubTree` reports, so it is + * "unchanged" and never enters the embed path. */ +const stored = (path: string): DocRecord => ({ + repo: REPO, + path, + blobSha: `blob-${path}`, + updatedAt: "2026-08-01T00:00:00Z", +}); + +beforeEach(() => { + processAndUpsertDocMock.mockReset(); + processAndUpsertDocMock.mockResolvedValue({ embedded: true, failed: false }); + deleteFtsRowMock.mockReset().mockResolvedValue(undefined); +}); + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +describe("poller: pollDocs delete fan-out", () => { + it("reaps only the doc absent from the tree, on all three surfaces", async () => { + // One doc still in the store, gone from the tree. All three surfaces are + // keyed to the same `docVectorId`, and the surviving doc is left alone. + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env, vectorDeletes } = makeDocEnv(); + + await pollDocs(REPO, env, store.stub); + + const goneId = await docVectorId(REPO, "docs/gone.md"); + + expect(store.deletes).toEqual(["docs/gone.md"]); + expect(vectorDeletes).toEqual([goneId]); + expect(deleteFtsRowMock).toHaveBeenCalledTimes(1); + expect(deleteFtsRowMock.mock.calls[0][1]).toBe(goneId); + expect(store.records.has("docs/keep.md")).toBe(true); + }); + + it("keeps tearing down the later surfaces when Vectorize fails", async () => { + // The surfaces are torn down independently. Pre-fix one outer try wrapped + // the whole item, so a Vectorize failure skipped the store DELETE and left + // the row behind — while the FTS5 row, guarded by its own inner try, was + // already gone. The D1 rows are the ones users actually retrieve. + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env } = makeDocEnv(); + (env.VECTORIZE.deleteByIds as ReturnType).mockRejectedValue( + new Error("vectorize down"), + ); + + await pollDocs(REPO, env, store.stub); + + expect(deleteFtsRowMock).toHaveBeenCalledTimes(1); + expect(store.deletes).toEqual(["docs/gone.md"]); + }); + + it("keeps tearing down the store record when the FTS5 delete fails", async () => { + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env, vectorDeletes } = makeDocEnv(); + deleteFtsRowMock.mockRejectedValue(new Error("d1 down")); + + await pollDocs(REPO, env, store.stub); + + expect(vectorDeletes).toHaveLength(1); + expect(store.deletes).toEqual(["docs/gone.md"]); + }); + + it("leaves the reap alone when nothing was deleted", async () => { + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md")]); + const { env, vectorDeletes } = makeDocEnv(); + + await pollDocs(REPO, env, store.stub); + + expect(store.deletes).toEqual([]); + expect(vectorDeletes).toEqual([]); + expect(deleteFtsRowMock).not.toHaveBeenCalled(); + }); +}); + +describe("poller: pollDocs delete budget", () => { + it("caps deletions per run and drains the rest on later runs", async () => { + // The case that raised the issue: a single PR removed 66 `.md` files, all of + // which land on the next run. Unbounded, that is 3 subrequests x 66 in one + // LIGHT_CRON invocation shared with pollRepo and pollReleases (issue #203). + const deletedPaths = Array.from( + { length: DELETE_BUDGET * 2 + 1 }, + (_, i) => `docs/gone-${String(i).padStart(2, "0")}.md`, + ); + const store = makeDocStore([stored("docs/keep.md"), ...deletedPaths.map(stored)]); + const { env } = makeDocEnv(); + + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(DELETE_BUDGET); + + // Monotonic drain: a reaped doc's store row is gone, so the leftover set + // only shrinks. No path is reaped twice and none is skipped. + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(DELETE_BUDGET * 2); + + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(deletedPaths.length); + + expect([...store.deletes].sort()).toEqual([...deletedPaths].sort()); + expect(new Set(store.deletes).size).toBe(deletedPaths.length); + expect(store.records.has("docs/keep.md")).toBe(true); + expect(store.records.size).toBe(1); + }); + + it("holds the tree ETag back while deletions are outstanding", async () => { + // Advancing the ETag with a backlog left would make the next run answer 304 + // and return before it ever looks at `deletedDocs` — the drain would stall + // until the tree happened to change again (issue #203). + const deletedPaths = Array.from( + { length: DELETE_BUDGET + 1 }, + (_, i) => `docs/gone-${i}.md`, + ); + const store = makeDocStore([stored("docs/keep.md"), ...deletedPaths.map(stored)]); + const { env } = makeDocEnv(); + + stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(store.deletes).toHaveLength(DELETE_BUDGET); + expect(store.etag()).toBeUndefined(); + + // Next run: the poller sends no If-None-Match, so it sees the tree again and + // reaps the leftover. + const { conditionalHits } = stubTree(["docs/keep.md"]); + await pollDocs(REPO, env, store.stub); + expect(conditionalHits).toEqual([]); + expect(store.deletes).toHaveLength(deletedPaths.length); + + // Backlog cleared — now the ETag is allowed to advance. + expect(store.etag()).toBe('W/"tree-1"'); + }); + + it("advances the tree ETag when the reap finished inside its budget", async () => { + stubTree(["docs/keep.md"]); + const store = makeDocStore([stored("docs/keep.md"), stored("docs/gone.md")]); + const { env } = makeDocEnv(); + + await pollDocs(REPO, env, store.stub); + + expect(store.deletes).toEqual(["docs/gone.md"]); + expect(store.etag()).toBe('W/"tree-1"'); + }); +}); diff --git a/src/poller.ts b/src/poller.ts index ce6efde..6f4cbb2 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -107,6 +107,26 @@ const MAX_COMMENT_FETCHES_PER_REPO_PER_RUN = 10; * unchanged in the store until the doc is successfully upserted). */ const MAX_DOC_FETCHES_PER_REPO_PER_RUN = 10; +/** Maximum docs reaped (Vectorize + FTS5 + store row) per repo per cron run. + * Mirrors `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`: each reap fans out to 3 + * subrequests, so an unbounded loop over a mass deletion could exhaust the + * LIGHT_CRON invocation budget on its own and starve every repo behind it — + * docs share that invocation with `pollRepo` and `pollReleases` (issue #203). + * A real case: github-webhook-mcp deleted 66 `.md` files in one PR, all of + * which land on the next single run. + * + * Numeric design: 5 deletes x 3 subrequests x 5 repos = 75 subrequests for the + * reap, on top of the ~250 the docs fetch cap already allows. Together with + * pollRepo and pollReleases the LIGHT_CRON worst case stays around 825, inside + * the 1000 ceiling. + * + * Remaining deletions are reaped next run: a reaped doc's store row is gone, so + * the leftover set only shrinks and the drain is monotonic. The tree ETag is + * deliberately *not* advanced while deletions are outstanding — the next run + * would otherwise short-circuit on 304 and never see them (same hold as the + * fetch cap, see the watermark write below). */ +const MAX_DOC_DELETIONS_PER_REPO_PER_RUN = 5; + /** Maximum number of release records the releases poller upserts per repo per * cron run. The GitHub Releases endpoint returns all recent releases in a * single API call, so the GH-side fetch cost is fixed at 1, but each release @@ -788,8 +808,15 @@ async function fetchFileContent( /** * Poll a single repository for documentation file updates. * Uses Git Trees API for change detection and Contents API for fetching changed files. + * + * Files present in the store but absent from the current tree are reaped from + * Vectorize, D1 FTS5, and the structured store — each surface independently, + * capped per repo per run, with the tree ETag held back while a backlog remains + * so the next run can still see it (issue #203). + * + * Exported for tests; production callers reach it through `handleScheduled`. */ -async function pollDocs( +export async function pollDocs( repo: string, env: Env, storeStub: DurableObjectStub, @@ -917,61 +944,77 @@ async function pollDocs( } } - // Handle deleted files: remove from Vectorize, D1 FTS5, and the structured store. + // Handle deleted files: remove from Vectorize, D1 FTS5, and the structured + // store. No graph-edge teardown here, unlike the wiki reap: both endpoints of + // every `doc_edges` row are wiki vector IDs (`indexWikiEdges` is the only + // writer, and the dst ID it computes is a `wikiDocVectorId` too), so a doc + // vector ID is never an edge endpoint. Add the teardown here if that + // invariant changes (issue #203). + let removedDocs = 0; + let deleteBudgetExhausted = false; for (const doc of deletedDocs) { - try { - const dvid = await docVectorId(repo, doc.path); - await env.VECTORIZE.deleteByIds([dvid]); + if (removedDocs >= MAX_DOC_DELETIONS_PER_REPO_PER_RUN) { + deleteBudgetExhausted = true; + console.warn( + `pollDocs: delete budget reached for ${repo} ` + + `(${MAX_DOC_DELETIONS_PER_REPO_PER_RUN} deletions). ` + + `${deletedDocs.length - removedDocs} remaining docs will be reaped next cron run.`, + ); + break; + } + + const dvid = await docVectorId(repo, doc.path); + // Each surface is torn down independently: a Vectorize failure must not + // strand the D1 rows, which are the ones users actually retrieve. + for (const [surface, run] of [ + ["vector", () => env.VECTORIZE.deleteByIds([dvid])], + ["FTS5 row", () => deleteFtsRow(env.DB_FTS, dvid)], + [ + "store record", + () => + storeStub.fetch( + new Request( + `http://store/doc?repo=${encodeURIComponent(repo)}&path=${encodeURIComponent(doc.path)}`, + { method: "DELETE" }, + ), + ), + ], + ] as Array<[string, () => Promise]>) { try { - await deleteFtsRow(env.DB_FTS, dvid); - } catch (ftsErr) { + await run(); + } catch (err) { console.error( - `Failed to delete FTS5 row for doc ${repo}/${doc.path}:`, - ftsErr instanceof Error ? ftsErr.message : String(ftsErr), + `Failed to delete ${surface} for doc ${repo}/${doc.path}:`, + err instanceof Error ? err.message : String(err), ); } - await storeStub.fetch( - new Request( - `http://store/doc?repo=${encodeURIComponent(repo)}&path=${encodeURIComponent(doc.path)}`, - { method: "DELETE" }, - ), - ); - } catch (err) { - console.error( - `Failed to delete doc vector ${repo}/${doc.path}:`, - err instanceof Error ? err.message : String(err), - ); } + removedDocs++; } // Update watermark with ETag — but only when the run completed without - // hitting the per-run fetch cap. If we did hit the cap, leftover changed - // docs still need to be processed; persisting the new ETag would cause the - // next cron to short-circuit on 304 and never see them. Skipping the ETag - // update forces a fresh tree fetch next run so `changedEntries` repopulates - // (issue #149). - if (!fetchBudgetExhausted) { - await storeStub.fetch( - new Request("http://store/watermark", { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ repo: watermarkKey, lastPolledAt: now, etag: responseEtag }), - }), - ); - } else { - // Still bump lastPolledAt so observability can see the run happened, but - // keep the prior ETag so the next cron re-fetches the tree. - await storeStub.fetch( - new Request("http://store/watermark", { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ repo: watermarkKey, lastPolledAt: now, etag: storedEtag }), + // hitting a per-run cap. If either cap was hit, leftover work remains: + // changed docs still to embed (issue #149) or deleted docs still to reap + // (issue #203). Persisting the new ETag would cause the next cron to + // short-circuit on 304 and never see either. Holding the prior ETag forces a + // fresh tree fetch next run so `changedEntries` and `deletedDocs` both + // repopulate. lastPolledAt is still bumped so observability sees the run. + const holdEtag = fetchBudgetExhausted || deleteBudgetExhausted; + await storeStub.fetch( + new Request("http://store/watermark", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + repo: watermarkKey, + lastPolledAt: now, + etag: holdEtag ? storedEtag : responseEtag, }), - ); - } + }), + ); console.log( - `${repo} docs: ${docEntries.length} found, ${embedded} embedded, ${skipped} unchanged, ${failed} failed, ${deletedDocs.length} deleted, ` + + `${repo} docs: ${docEntries.length} found, ${embedded} embedded, ${skipped} unchanged, ${failed} failed, ` + + `${removedDocs}/${deletedDocs.length} deleted, ` + `fetches_issued=${fetchesIssued}/${MAX_DOC_FETCHES_PER_REPO_PER_RUN}`, ); }