Repository navigation
Give the derived duration budget room to be a ceiling - #76
Merged
Merged
Conversation
SCALED_PAGES_PER_SECOND was 2.5, the arithmetic rate at the default concurrency of 5 against a two-second origin. That is a plausible average and a bad ceiling. Measured on a ~8,700-page production site: 7,351 pages in 58 minutes, or 2.11 pages per second including parse, redirects and retries. The derived clock expired at 84% of a crawl that was making steady progress, and the run was reported max-duration even though the page budget had never bound. That is the same silent truncation the budget scaling exists to remove, moved from bytes to the clock. The constant is now 1 page per second, for the reason SCALED_BYTES_PER_PAGE is already generous: the budget is a ceiling, not an allocation, so over-provisioning costs nothing while under-provisioning truncates a healthy crawl. An origin can now be twice as slow as that measurement and the crawl still ends on its page budget. A stated maxDurationMs is honoured exactly, requestDelayMs still adds on top, and the flat default remains the floor for small crawls. The test pins the headroom relationship against the observed rate rather than the constant, so it keeps meaning something if the constant moves again. Engine 2.2.0 to 2.3.0: shape unchanged, semantics changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SCALED_PAGES_PER_SECONDwas 2.5, the arithmetic rate at the default concurrency of 5 against a two-second origin. That is a plausible average and a bad ceiling.Measured on a ~8,700-page production site: 7,351 pages in 58 minutes, or 2.11 pages per second including parse, redirects and retries. The derived clock expired at 84% of a crawl that was making steady progress, and the run was reported
max-durationeven though the page budget had never bound. That is the same silent truncation the budget scaling exists to remove, moved from bytes to the clock.The constant is now 1 page per second, for the reason
SCALED_BYTES_PER_PAGEis already generous, in its own words: the budget is a ceiling, not an allocation, so over-provisioning costs nothing while under-provisioning truncates a healthy crawl.A stated
maxDurationMsis still honoured exactly,requestDelayMsstill adds on top, and the flat default remains the floor for small crawls.The test pins the headroom relationship against the observed rate rather than the constant itself, so it keeps meaning something if the constant moves again. Verified it fails with the old 2.5 restored. 626 tests pass; typecheck, lint, e2e clean.
Minor bump 7.1.0 to 7.2.0; crawl engine 2.2.0 to 2.3.0 (shape unchanged, semantics changed).
🤖 Generated with Claude Code