From c34e0f4f11978add1726ce8e0a1fbd965df27d0b Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Sun, 2 Aug 2026 16:16:10 +0900 Subject: [PATCH] fix(wiki): stop a withheld reap candidate from spending a delete slot [poller, docs, tests] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split the reap loop's per-run budget in two: deletes stay on MAX_WIKI_DELETIONS_PER_REPO_PER_RUN (5), probes move to the new MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN (15). The loop now walks the whole orphan candidate list and breaks on whichever budget runs out first, and orphansDeferred reports the candidates it never reached instead of the overflow past the delete cap. 孤児リストは安定ソートされているため、列挙が短いままだと同じ保留候補が毎 run 削除枠を埋め、その後ろに並ぶ「本当に削除された page」が列挙の回復まで到達すら されなかった。枠を分けたことで保留は probe 枠だけを消費する。probe 上限 15 = 最大 30 subrequest で、walk の fetch 予算とは別枠のまま。 Closes #197 --- docs/0-requirements.ja.md | 4 +-- docs/0-requirements.md | 4 +-- docs/installation.ja.md | 2 +- docs/installation.md | 2 +- src/poller-wiki.test.ts | 56 ++++++++++++++++++++++++++++++++++ src/poller.ts | 63 +++++++++++++++++++++++++++++++-------- 6 files changed, 112 insertions(+), 19 deletions(-) diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index 8f7142d..2cbd1b9 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -161,9 +161,9 @@ wiki poller は `:45` cron 専属で、GitHub Wiki content の唯一の取り込 **周回の完了.** 2 本目の watermark 行 `wiki-lap:{repo}` が *lap anchor* — 現在の周回がどの slug の次から始まったか — を保持する。pass は anchor の直前の page に到達した時点、つまり cursor が一周して戻ってきた時点で `wrapped: true` を返し、anchor はその page へ移動して次の周回がその次から始まる。これが admin endpoint の `done` を到達可能にしている。「この 1 回の pass で全 page を踏破した」という意味では、1 pass の fetch 予算より page 数が多い wiki で真になりようがなく、「`done` まで呼び続けろ」という手順に停止条件が無かった（issue #188）。anchor は保存した index ではなく slug 順で解決するので、周回の途中で page が増減しても desync しない。`cursor=` を明示指定した場合はその地点から新しい周回を開始する。 -**孤児の削除.** 現在の `_pages` index に無い page は Vectorize / D1 FTS5 / graph edge table / structured store から削除する。候補集合は structured store **と** 実際の `search_docs` 行の和集合。store から消えているのに index には残っている page は store だけを見る差分からは見えず、これが改名済み 8 page を数ヶ月間 search から引ける状態で残した原因だった（issue #184）。影響範囲は 4 つの guard で抑える — `_pages` index が読めなかった run では削除を完全に skip（空の slug 集合は「全部消えた」ではなく「こちらが盲目」の意）、残った候補は削除前に 1 件ずつ実在を probe する（下記）、1 repo 1 run あたり `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`（既定 5）で cap、各 surface は独立に teardown して Vectorize の失敗が実際に retrieval される D1 行を取り残さないようにする。 +**孤児の削除.** 現在の `_pages` index に無い page は Vectorize / D1 FTS5 / graph edge table / structured store から削除する。候補集合は structured store **と** 実際の `search_docs` 行の和集合。store から消えているのに index には残っている page は store だけを見る差分からは見えず、これが改名済み 8 page を数ヶ月間 search から引ける状態で残した原因だった（issue #184）。影響範囲は 4 つの guard で抑える — `_pages` index が読めなかった run では削除を完全に skip（空の slug 集合は「全部消えた」ではなく「こちらが盲目」の意）、残った候補は削除前に 1 件ずつ実在を probe する（下記）、1 repo 1 run あたり削除は `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`（既定 5）・probe は `MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN`（既定 15）と枠を分けて cap、各 surface は独立に teardown して Vectorize の失敗が実際に retrieval される D1 行を取り残さないようにする。 -**削除前の実在確認.** index が読めなかった場合の guard が守るのは列挙の**全面**失敗。**部分**失敗 — `_pages` が実際より少ない page を返す — は集合のレベルでは本物の削除と区別がつかず、実在する page が削除候補に混入する。1 run 5 件の cap は速度を落とすだけで、cron が繰り返す以上は止まらない。そこで確認を集合の外へ出し、page 本体に問い合わせる。候補ごとに raw content を fetch し、候補となる全 extension で 404 を実際に観測した場合にのみ削除を許可する（issue #187）。200・ネットワークエラー・その他の status はいずれも削除を見送り、`orphansWithheld` として報告する。見送りが deadlock を作ることはない — 本当に削除された page は 404 を返し続けるので、後続の run で消える。判定は page 単位なので、正当な大量削除は従来どおりの速度で進む。probe の宛先は `raw.githubusercontent` であって、描画側の `github.com/{repo}/wiki/{slug}` ではない。2026-08-01 の実測では、存在しない page は wiki の root へ 302 redirect して 200 に着地するため、不在を報告できない。コストは候補 1 件あたり extension 数（`md` / `markdown`）以下の subrequest で、walk の fetch 予算とは別枠。guard 導入前から変わらない残余の穴: title 由来のファイル名で保存された page（`E.-Li-language` → `E.-Li+language.md`）は自分の slug に対して 404 を返すが、そのファイル名を運ぶ link text は、当の page が欠けている列挙の中にしか存在しない。別軸として、候補数が索引済み page 数の半分に達した run では警告ログのみを出す — 比率では大量整理と列挙の欠損を区別できないので何も決定させず、run の形を読める状態にするだけに留める。 +**削除前の実在確認.** index が読めなかった場合の guard が守るのは列挙の**全面**失敗。**部分**失敗 — `_pages` が実際より少ない page を返す — は集合のレベルでは本物の削除と区別がつかず、実在する page が削除候補に混入する。1 run 5 件の cap は速度を落とすだけで、cron が繰り返す以上は止まらない。そこで確認を集合の外へ出し、page 本体に問い合わせる。候補ごとに raw content を fetch し、候補となる全 extension で 404 を実際に観測した場合にのみ削除を許可する（issue #187）。200・ネットワークエラー・その他の status はいずれも削除を見送り、`orphansWithheld` として報告する。見送りが deadlock を作ることはない — 本当に削除された page は 404 を返し続けるので、後続の run で消える。判定は page 単位なので、正当な大量削除は従来どおりの速度で進む。probe の宛先は `raw.githubusercontent` であって、描画側の `github.com/{repo}/wiki/{slug}` ではない。2026-08-01 の実測では、存在しない page は wiki の root へ 302 redirect して 200 に着地するため、不在を報告できない。コストは候補 1 件あたり extension 数（`md` / `markdown`）以下の subrequest で、walk の fetch 予算とは別枠、probe 枠によって 1 repo 1 run あたり 30 で頭が止まる。**見送りが消費するのは probe 枠であって削除枠ではない。** 削除枠に付けても索引の正しさは保たれるが、drain が止まる — 候補リストは安定ソートされているので、列挙が短いままの間は同じ見送り候補が毎 run 削除枠を埋め、その後ろに並ぶ「本当に削除された page」は列挙が回復するまで到達すらされない（issue #197）。そこで loop は候補リスト全体を走査し、どちらかの枠が尽きた時点で打ち切る。`orphansDeferred` は「削除枠から溢れた数」ではなく「この run で到達しなかった候補数」を表す。guard 導入前から変わらない残余の穴: title 由来のファイル名で保存された page（`E.-Li-language` → `E.-Li+language.md`）は自分の slug に対して 404 を返すが、そのファイル名を運ぶ link text は、当の page が欠けている列挙の中にしか存在しない。別軸として、候補数が索引済み page 数の半分に達した run では警告ログのみを出す — 比率では大量整理と列挙の欠損を区別できないので何も決定させず、run の形を読める状態にするだけに留める。 **即時復旧.** `POST /admin/backfill-wiki?repo=owner/repo[&limit=N][&cursor=SLUG]` が同じ pass を明示予算（1..40、既定 20）で即時実行する。cron と cursor を共有するので互いに前進させ合う。response の `done: true` まで繰り返し呼ぶ — 1 周に必要な呼び出し回数は 1 回ではなく ceil(pages / limit) 回。`cursor=`（空）を渡すと列挙の先頭から walk（と周回）をやり直す。各 call は独立した Worker invocation なので、それぞれ独自の subrequest 予算を持つ。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index fea31cd..1008880 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -162,9 +162,9 @@ Change detection uses a content SHA-256 hash (no git blob SHA is available witho **Lap completion.** A second watermark row, `wiki-lap:{repo}`, holds the *lap anchor* — the slug the current sweep started after. The pass reports `wrapped: true` when it reaches the page immediately before the anchor, i.e. when the cursor has come all the way back around, and the anchor then moves to that page so the next lap starts after it. This is what makes the admin endpoint's `done` reachable: reporting "this one pass saw every page" can never be true for a wiki with more pages than one pass may fetch, so the documented "call until `done`" loop had no terminating condition (issue #188). The anchor is resolved by slug order rather than by a stored index, so a page added or deleted mid-lap cannot desync it. An explicit `cursor=` override opens a fresh lap at that point. -**Orphan reap.** Pages absent from the current `_pages` index are removed from Vectorize, D1 FTS5, the graph edge table, and the structured state store. The candidate set is the union of the structured store *and* the live `search_docs` rows: a page missing from the store but still present in the index is invisible to a store-only diff, which is how eight renamed-away pages stayed resolvable in search for months (issue #184). Four guards bound the blast radius — reaping is skipped entirely when the `_pages` index could not be read (an empty slug set means "we are blind", not "everything was deleted"), every remaining candidate is probed for existence before it is deleted (below), the reap is capped at `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN` (default 5) per repo per run, and each surface is torn down independently so a Vectorize failure cannot strand the D1 rows users actually retrieve. +**Orphan reap.** Pages absent from the current `_pages` index are removed from Vectorize, D1 FTS5, the graph edge table, and the structured state store. The candidate set is the union of the structured store *and* the live `search_docs` rows: a page missing from the store but still present in the index is invisible to a store-only diff, which is how eight renamed-away pages stayed resolvable in search for months (issue #184). Four guards bound the blast radius — reaping is skipped entirely when the `_pages` index could not be read (an empty slug set means "we are blind", not "everything was deleted"), every remaining candidate is probed for existence before it is deleted (below), the pass is capped per repo per run on two separate budgets — `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN` (default 5) for deletes and `MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN` (default 15) for probes — and each surface is torn down independently so a Vectorize failure cannot strand the D1 rows users actually retrieve. -**Pre-reap existence probe.** The index-unreadable guard covers a *total* enumeration failure. A **partial** one — `_pages` returning fewer pages than the wiki holds — is indistinguishable from a real deletion at the set level, and pushes live pages into the reap set; the per-run cap slows that but does not stop it, because the cron repeats. So the check leaves the set and addresses the page: each candidate's raw content is fetched, and only an observed 404 on every candidate extension authorizes the delete (issue #187). A 200, a network error, or any other status withholds it and is reported as `orphansWithheld`; withholding cannot deadlock, since a page that really was deleted keeps answering 404 and drains on a later run. The verdict is per page, so a legitimate bulk deletion still drains at the normal rate. The probe addresses `raw.githubusercontent`, not the rendered `github.com/{repo}/wiki/{slug}` URL: measured 2026-08-01, a nonexistent page there 302-redirects to the wiki root and lands on 200, so it can never report absence. Cost is at most one subrequest per candidate per extension (`md`, `markdown`), spent outside the walk's fetch budget. Residual gap, unchanged from before the guard: a page stored under a title-derived filename (`E.-Li-language` → `E.-Li+language.md`) answers 404 to its own slug, and the link text that carries the filename only exists in the enumeration the page is missing from. A separate warn-only log fires when the candidate set reaches half the indexed page set — a ratio cannot separate a bulk cleanup from a short enumeration, so it decides nothing and only leaves the shape of the run readable. +**Pre-reap existence probe.** The index-unreadable guard covers a *total* enumeration failure. A **partial** one — `_pages` returning fewer pages than the wiki holds — is indistinguishable from a real deletion at the set level, and pushes live pages into the reap set; the per-run cap slows that but does not stop it, because the cron repeats. So the check leaves the set and addresses the page: each candidate's raw content is fetched, and only an observed 404 on every candidate extension authorizes the delete (issue #187). A 200, a network error, or any other status withholds it and is reported as `orphansWithheld`; withholding cannot deadlock, since a page that really was deleted keeps answering 404 and drains on a later run. The verdict is per page, so a legitimate bulk deletion still drains at the normal rate. The probe addresses `raw.githubusercontent`, not the rendered `github.com/{repo}/wiki/{slug}` URL: measured 2026-08-01, a nonexistent page there 302-redirects to the wiki root and lands on 200, so it can never report absence. Cost is at most one subrequest per candidate per extension (`md`, `markdown`), spent outside the walk's fetch budget and bounded by the probe budget at 30 per repo per run. **Withholding spends a probe, not a delete slot.** Charging it to the delete budget was correct for the index but wrong for the drain: the candidate list is stably sorted, so while an enumeration stays short the same withheld heads fill the delete budget on every run and a page that really was deleted, ordering behind them, is never reached until the enumeration recovers (issue #197). So the loop walks the whole candidate list and stops on whichever budget runs out first, and `orphansDeferred` reports what it never reached rather than what fell past the delete cap. Residual gap, unchanged from before the guard: a page stored under a title-derived filename (`E.-Li-language` → `E.-Li+language.md`) answers 404 to its own slug, and the link text that carries the filename only exists in the enumeration the page is missing from. A separate warn-only log fires when the candidate set reaches half the indexed page set — a ratio cannot separate a bulk cleanup from a short enumeration, so it decides nothing and only leaves the shape of the run readable. **Immediate recovery.** `POST /admin/backfill-wiki?repo=owner/repo[&limit=N][&cursor=SLUG]` runs the same pass on demand with an explicit fetch budget (1..40, default 20), sharing the cron's cursor so the two advance one another. Call it repeatedly until the response reports `done: true` — one lap takes ceil(pages / limit) calls, not one; pass an empty `cursor=` to restart the walk (and the lap) from the head. Each call is its own Worker invocation and therefore gets its own subrequest budget. diff --git a/docs/installation.ja.md b/docs/installation.ja.md index 009984f..06be740 100644 --- a/docs/installation.ja.md +++ b/docs/installation.ja.md @@ -289,7 +289,7 @@ POST /admin/backfill-wiki?repo=owner/repo - `done`（= `wrapped`）は「cursor が列挙を一周した」という意味で、1 回の呼び出しで全 page を踏破したという意味ではない。`pages` が `limit` を超える wiki では 1 回で `true` にはならず、`ceil(pages / limit)` 回で成立する。77 page を既定 `limit=20` で回すなら 4 回（issue #188） - `lapAnchor` は現在の周回の起点 slug（`""` は列挙の先頭）。周回は `lapAnchor` の次の page から始まり、`lapAnchor` に戻ってきた時点で閉じる。`nextCursor` と併せて見れば途中経過が分かる - `enumerated: false` は `/wiki/_pages` の scrape が失敗したという意味。何も索引せず、意図的に何も削除していない。空の wiki と解釈せず再試行すること -- `orphansDeferred` は per-run cap を超えて削除待ちの page 数。0 になるまで呼び続ける +- `orphansDeferred` は、その run で**到達しなかった**削除候補の数。削除枠と probe 枠のどちらかが尽きて打ち切った分にあたる。0 になるまで呼び続ける。到達した上で見送った候補は `orphansWithheld` の側に数えられる — 枠が分かれているので、見送りが削除枠を消費して後ろに並ぶ実削除を止めることはない（issue #197） - `orphansWithheld` は、削除候補に挙がったが content がまだ配信されていた（あるいは実在確認が結論を出せなかった）ため削除を見送った page 数。0 でない場合、`_pages` の scrape が**実際の wiki より少なく返っている**という意味。page 自体は無傷で守られており、調べるべきは列挙のほう。見送った page 名は worker のログに出る（issue #187） - カバレッジの確認は `search_docs` の `type = 'wiki_doc'` 行と `https://github.com/{repo}/wiki/_pages` の page 一覧を突き合わせる diff --git a/docs/installation.md b/docs/installation.md index 457a61b..3a917d1 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -289,7 +289,7 @@ Operational notes: - `done` (= `wrapped`) means the cursor completed a lap of the enumeration, not that one call covered every page. A wiki with more `pages` than `limit` cannot finish in one call; the lap closes after `ceil(pages / limit)` calls — 4 for a 77-page wiki at the default `limit=20` (issue #188) - `lapAnchor` is the slug the current lap started after (`""` = the head of the enumeration). The lap runs from the page after the anchor around to the anchor itself, so `lapAnchor` plus `nextCursor` shows how far the lap has come - `enumerated: false` means the `/wiki/_pages` scrape failed — nothing was indexed and, deliberately, nothing was reaped; retry rather than treating it as an empty wiki -- `orphansDeferred` counts pages still to be reaped past the per-run cap; keep calling until it reaches 0 +- `orphansDeferred` counts reap candidates the call never reached, because it hit either the per-run delete cap or the per-run probe cap; keep calling until it reaches 0. A candidate that was reached and withheld is reported by `orphansWithheld`, not here — the two budgets are separate so a withheld candidate cannot spend a delete slot and stall the real deletions ordering behind it (issue #197) - `orphansWithheld` counts reap candidates whose content still served (or whose existence probe could not conclude), so the delete was withheld. Non-zero means the `_pages` scrape came back **short of the live wiki** — the pages themselves are intact and were protected, but the enumeration is what to investigate; the worker log names each withheld page (issue #187) - verify coverage by comparing `search_docs` rows (`type = 'wiki_doc'`) against the page list at `https://github.com/{repo}/wiki/_pages` diff --git a/src/poller-wiki.test.ts b/src/poller-wiki.test.ts index 5118765..c2568db 100644 --- a/src/poller-wiki.test.ts +++ b/src/poller-wiki.test.ts @@ -657,6 +657,62 @@ describe("poller: pollWiki orphan reap", () => { expect(rawRequests.length).toBeLessThanOrEqual(4 + 5 * 2); }); + it("reaps a deleted page sitting behind a run of withheld candidates", async () => { + // Issue #197: a withheld candidate used to spend a delete slot. The orphan + // list is stably sorted, so five withheld heads filled the whole budget on + // every run and the page that really was deleted, ordering after them, was + // never even looked at until the enumeration recovered. + const live = Array.from({ length: 5 }, (_, i) => `a-live-${i}`); + const wiki: FakeWiki = { + listed: [{ slug: "current" }], + files: { + current: "c", + Home: "h", + // Present in the wiki but missing from `_pages`: the short enumeration. + ...Object.fromEntries(live.map((p) => [p, "still here"])), + }, + }; + const store = makeWikiStore(); + const { env } = makeWikiEnv(["current", ...live, "z-really-deleted"]); + stubWiki(wiki); + + const summary = await pollWiki(REPO, env, store.stub); + + expect(summary.orphansWithheld).toBe(5); + expect(summary.removed).toBe(1); + expect(store.deletes).toEqual(["z-really-deleted"]); + // All six candidates were reached, so nothing was deferred. + expect(summary.orphansDeferred).toBe(0); + }); + + it("caps the probes per run and defers the candidates it never reached", async () => { + // The probe budget is what keeps "walk the whole orphan list" bounded: 20 + // live candidates, none of them deletable, must not cost 20 probes. + const live = Array.from({ length: 20 }, (_, i) => `live-${String(i).padStart(2, "0")}`); + const wiki: FakeWiki = { + listed: [{ slug: "current" }], + files: { + current: "c", + Home: "h", + ...Object.fromEntries(live.map((p) => [p, "still here"])), + }, + }; + const store = makeWikiStore(); + const { env } = makeWikiEnv(["current", ...live]); + const { requestedNames } = stubWiki(wiki); + + const summary = await pollWiki(REPO, env, store.stub); + + // MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN = 15. Each live candidate answers + // 200 on its first extension, so one probe is one raw request here. + const probed = requestedNames().filter((n) => n.startsWith("live-")); + expect(probed).toHaveLength(15); + expect(summary.orphansWithheld).toBe(15); + expect(summary.removed).toBe(0); + // Deferred = candidates this run never reached, not "past the delete cap". + expect(summary.orphansDeferred).toBe(5); + }); + it("reaps nothing when the page index could not be read", async () => { // An unreadable `_pages` yields an empty slug set. Treating that as "every // page was deleted" would wipe the repo's entire wiki index. diff --git a/src/poller.ts b/src/poller.ts index 13af6cf..9f163f0 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -1026,6 +1026,23 @@ const MAX_WIKI_FETCHES_PER_REPO_PER_RUN = 20; * set only shrinks, so the drain is monotonic (issue #184). */ const MAX_WIKI_DELETIONS_PER_REPO_PER_RUN = 5; +/** Maximum pre-reap existence probes issued per repo per cron run. + * + * Separate from MAX_WIKI_DELETIONS_PER_REPO_PER_RUN because a withheld + * candidate must not spend a delete slot. The orphan list is stably sorted, so + * while an enumeration stays short the same withheld candidates hold the head + * of it on every run, and a page that really was deleted sitting behind them + * is never reached — the index stays correct, but the delete stalls until the + * enumeration recovers (issue #197). With the budgets split, the reap loop + * walks the whole orphan list and stops on whichever budget runs out first: + * withholding costs a probe, deleting costs a probe *and* a delete slot. + * + * Sized at 3x the delete budget. A probe costs at most `WIKI_EXTENSIONS.length` + * = 2 subrequests, so this ceiling is 30, alongside the ~4-per-delete fan-out + * of at most 5 deletes. Both are spent outside the walk's fetch budget and stay + * well inside the Worker's 1000-subrequest invocation ceiling (issue #130). */ +const MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN = 15; + /** Fraction of the indexed page set which, once the reap candidate set reaches * it, is logged as an anomaly. Warn-only on purpose: a ratio cannot separate a * legitimate bulk cleanup (wiki tidy-up, mass rename) from an enumeration that @@ -1435,9 +1452,9 @@ type WikiReapProbe = "alive" | "gone" | "inconclusive"; * * Cost: at most `WIKI_EXTENSIONS.length` subrequests per candidate, and one * when the stored extension serves the verdict on the first attempt. The reap - * loop is already capped at `MAX_WIKI_DELETIONS_PER_REPO_PER_RUN`, so the - * guard's per-run ceiling is that product. It is separate from the walk's fetch - * budget and never consumes it. + * loop caps the number of calls at `MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN`, so + * the guard's per-run ceiling is that product. It is separate from the walk's + * fetch budget and never consumes it. */ async function probeWikiPageAlive( repo: string, @@ -1498,7 +1515,9 @@ export interface WikiPollSummary { skipped: number; failed: number; removed: number; - /** Orphans detected but deferred past the delete budget. */ + /** Orphan candidates the reap loop never reached, because it stopped on the + * delete or the probe budget. A withheld candidate *was* reached, so it is + * reported by `orphansWithheld` and not counted here (issue #197). */ orphansDeferred: number; /** Reap candidates whose content still served (or whose probe could not * complete), so the delete was withheld. Non-zero means the `_pages` @@ -1542,7 +1561,10 @@ export interface WikiPollSummary { * table, and the structured store. Reaping is skipped entirely when the * enumeration failed, capped per run, and — because a *partially* enumerated * index is indistinguishable from a real deletion at the set level — withheld - * for any candidate whose raw content still serves (issue #187). + * for any candidate whose raw content still serves (issue #187). The per-run + * cap is two budgets, not one: a withheld candidate spends a probe but no + * delete slot, so it cannot hold the head of the sorted candidate list and + * starve the real deletions behind it (issue #197). */ export async function pollWiki( repo: string, @@ -1553,6 +1575,7 @@ export async function pollWiki( const fetchBudget = opts.fetchBudget ?? MAX_WIKI_FETCHES_PER_REPO_PER_RUN; const embedBudget = opts.embedBudget ?? MAX_WIKI_EMBEDDINGS_PER_RUN; const deleteBudget = opts.deleteBudget ?? MAX_WIKI_DELETIONS_PER_REPO_PER_RUN; + const probeBudget = MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN; const persistCursor = opts.persistCursor ?? true; const empty = (startCursor: string): WikiPollSummary => ({ @@ -1725,13 +1748,6 @@ export async function pollWiki( if (!currentSlugs.has(pageName)) orphanSet.add(pageName); } const orphans = Array.from(orphanSet).sort(compareSlugs); - const orphansDeferred = Math.max(0, orphans.length - deleteBudget); - if (orphansDeferred > 0) { - console.warn( - `${repo} wiki: ${orphans.length} orphans found, reaping ${deleteBudget} this run ` + - `(${orphansDeferred} deferred to the next run).`, - ); - } // Warn-only anomaly signal. A reap set this large against what is indexed is // either a legitimate bulk cleanup or an enumeration that came back short; @@ -1749,10 +1765,19 @@ export async function pollWiki( ); } - for (const pageName of orphans.slice(0, deleteBudget)) { + // The loop walks the *whole* candidate list and stops on whichever budget + // runs out. Slicing to the delete budget instead let a withheld candidate + // spend a delete slot, and since the list is stably sorted the same withheld + // heads would repeat every run while the real deletions behind them waited + // for the enumeration to recover (issue #197). + let probes = 0; + for (const pageName of orphans) { + if (removed >= deleteBudget || probes >= probeBudget) break; + // Existence check before the delete. The candidate is only "orphaned" as // far as the enumeration knows, and the enumeration is exactly what may // have come back short (issue #187). + probes++; const probe = await probeWikiPageAlive( repo, pageName, @@ -1800,6 +1825,18 @@ export async function pollWiki( removed++; } + // Every candidate the loop reached cost exactly one probe, so the probe count + // is also the reached count: what is left over is what this run never looked + // at. Withheld candidates were looked at and are reported separately. + const orphansDeferred = Math.max(0, orphans.length - probes); + if (orphansDeferred > 0) { + console.warn( + `${repo} wiki: ${orphans.length} orphans found, ${removed} reaped and ` + + `${orphansWithheld} withheld this run ` + + `(${orphansDeferred} not reached, deferred to the next run).`, + ); + } + // A completed lap re-anchors on the page that closed it, so the next lap // starts at the page after it and the walk keeps moving forward. const nextLapAnchor = wrapped ? lapFinalSlug : lapAnchor;