[arberx]

Two defects in the crawl budget layer. Both let a crawl exceed a stated limit and then report the wrong reason for stopping.

A soft stop hid the hard stop behind it

CrawlBudget.stop() was first-write-wins across a single field, so once a soft reason latched (max-query-variants, max-depth, max-links-per-page) a later hard reason could not replace it.

The fetch loop continues while the reason is not a hard stop:

while (frontier.length > 0 && (!isHardFetchStop(budget.terminationReason) || hasQueuedRoot()))

So a latched soft reason kept the crawler issuing real requests at the audited origin and discarding every response until the frontier drained. The run then reported the soft reason, sending anyone diagnosing it to the wrong budget.

Hard and soft now latch separately and the getter prefers hard. First-write-wins still applies within a class, never across them.

Measured against a fixture origin with a 1,000,000-byte budget:

	reported reason	bytes read	requests to origin
before	max-query-variants	4,197,578 (4.2x over budget)	85
after	max-bytes	1,234,598	25

The fetch budgets had no relationship to the page budget

maxPages and maxBytes each resolved against an independent flat default, so the smaller silently won. Asking for 1,000 pages of a site serving ~745 KB each returned 140, because the 100 MB byte default ran out first, with no error, no warning, and a partial flag naming the wrong budget.

maxFetches, maxDurationMs and maxBytes now derive from maxPages when the caller has not set them. This raises unset ceilings only:

• a stated budget is honoured exactly, never scaled
• omitting maxPages still gets the documented flat defaults

Also exports resolveSiteCrawlLimits so a caller can ask what budgets a crawl will run under, rather than recomputing the derivation and drifting from it.

Tests

test/crawl-budget-coherence.test.ts covers both defects against a real fixture HTTP server, plus unit coverage of the derivation.

Worth noting: two earlier versions of this test file passed without the fix. The fixture linked paths with no query strings, so maxQueryVariants never latched and the bug was never triggered. The current fixture uses small pages so many are crawled and links enqueued before the byte cap trips. Verified by stashing src/crawl.ts and re-running: the two latch tests fail without the fix, and the derivation assertions compare directly against DEFAULT_SITE_CRAWL_LIMITS so they cannot pass at the flat values.

The scaling only binds above ~67 pages (bytes) or ~300 (duration), so it is not behaviourally testable at reasonable cost. That is why the derivation is exported and unit-tested directly instead of shipped unverified.

624 tests pass across 53 files on Node 20, 22 and 24 (reproduced locally in containers matching both CI legs). Typecheck and lint clean.

Also bumps CRAWL_ENGINE_VERSION 2.0.0 to 2.1.0. The report shape is unchanged so CRAWL_SCHEMA_VERSION stays at 2.0, but that constant exists so persisted checkpoints can detect a change in crawl semantics, and the same options can now produce a larger crawl than 6.0.0 did.

Minor version bump for the new exported helper.

🤖 Generated with Claude Code

---

Review follow-ups

Six findings survived an adversarial pass and a round of review on 0cdb2ff. One (no floor under the derivation) was already fixed by the third commit. The rest are addressed here.

The default path was the one case the fix did not reach. The derivation keyed off whether maxPages was present, not off the resolved page count. So runSiteCrawl(url) resolved to the documented 1,000 pages but kept the flat 100MB and still truncated at ~140 pages, while runSiteCrawl(url, {maxPages: 1000}) got 1.5GB. Identical work, budgets 14x apart, decided by whether a key was written down. A test asserted that discontinuity as if it were the guarantee.

A derived budget could overflow. maxPages: Number.MAX_VALUE produced Infinity for all three fetch-side budgets. JSON.stringify writes that as null, so a report could state a crawl ran under no byte budget at all; and a duration past 2^31-1 ms makes setTimeout fire immediately, turning the longest possible budget into none. Every resolved limit is clamped to a safe integer, duration to the setTimeout ceiling.

Scaling ignored retries and pacing, though both are the caller's own knobs. A retry is a real request against maxFetches, so one recoverable 429 per page exhausted a 5,000-page budget at ~3,700 pages. requestDelayMs is time the crawl is required to spend not fetching, so 5,000 paced pages were budgeted 2,000s for work that cannot finish in under 5,000s: the politeness knob was causing the truncation.

The wiring was unguarded. The derivation tests all called the pure function directly; the whole suite passed with runSiteCrawl reverted to flat defaults. There is now an end-to-end test that crawls past the flat byte ceiling and fails if that wiring is severed. It gzips its fixture, so it crosses a 100MB budget over ~1.5KB of loopback traffic.

Why this is 7.0.0 and not 6.1.0

A default crawl now returns more pages, reads more bytes, and takes longer than 6.0.0 did, so stored crawls are not comparable across the boundary. Under this repo's own rule that is a major. The README badge, which ships in the npm tarball, was stale at 6.0.0.