[arberx]

What

deriveDeadLinks accepted two different things as a dead link:

target.statusCode >= 400                          -> dead link  the site said it is broken
target.state === 'fetch-error' && statusCode null -> dead link  WE failed to ask

Only the first is evidence. A finding now requires a real status code (statusCode: number, reason: 'http-error'). A target that never answered goes to a new deadLinks.unverified array with the transport error that stopped it, under its own key prefix so a consumer storing findings by key cannot merge the buckets back together.

Page fetches also get maxFetchRetries extra attempts (default 2, 500ms doubling) before anything classifies the observation.

Why

The root cause is one level deeper than the classifier: the traversal files both an HTTP 4xx/5xx response and a failed transport under state: 'fetch-error', so only the status code separates them — and the classifier keyed off the state. There was also no retry anywhere in the crawl, so one refused connection permanently decided a URL's fate, at concurrency 5 with zero delay by default.

On one 228-page site this produced 15 "dead links" across 6 URLs, every one statusCode: null with no error string. All 6 served a 200 in under a second on a manual check moments later. That output was client-facing: an agency hands a client a site-health report naming six live pages as broken, the client clicks them, and the report loses credibility.

Re-crawling the same site with this build: 226 pages, 0 dead links, 0 unverified, 0 fetch errors.

Retry scope

Only TIMEOUT and UNREACHABLE are retried. A blocked host, a redirect limit, an oversized body and an exhausted budget all reproduce on a retry, so retrying them would only spend budget. Retries consume the fetch budget and are paced like any other request; the backoff is abortable and clipped to the remaining duration budget. maxFetchRetries: 0 restores single-attempt behavior.

summary.fetchRetries: { attempted, recovered } is new and optional — a nonzero recovered is direct evidence the crawler rather than the site was the flaky party, which is the question you have to answer before believing any crawl-derived link finding.

Tests

test/crawl-dead-link-classification.test.ts, 9 cases. Written and then verified falsifiable: against the pre-fix logic, 5 of the 9 fail. They cover the invariant (no null-status observation can reach findings, across three transport failure modes), both directions of the split (a real 404 and a real 503 are still findings), retry recovery, the retry bound, and that a permanent failure is not retried.

Full suite: 608 passing, typecheck and lint clean.

Breaking

Crawl schema 1.2 -> 1.3, crawl engine 1.2.0 -> 1.3.0.

CrawlDeadLinkResult gains a required unverified array and CrawlDeadLinkFinding narrows to statusCode: number / reason: 'http-error'. Reading a finding still compiles (both are narrowings); constructing a result does not. unverified is deliberately required rather than optional — a consumer skipping it is the exact failure this fixes.

Dead-link counts drop against 5.x on any site where the crawler ever failed to connect, so a stored crawl cannot be compared across this boundary.

Note the bug is present in published 5.0.0, so a consumer on 5.x is affected today.

🤖 Generated with Claude Code

[arberx]

Six of the seven addressed; the seventh is deliberately deferred with reasoning below. Each was reproduced before being fixed.

#	Finding	Status
1	Retry violates maxDurationMs	Fixed
2	HTTP 429 labeled broken	Fixed
3	Breaking schema versioned 1.3	Fixed — now 2.0
4	unverified includes targets that answered	Fixed
5	Redirect-terminal response discarded	Deferred, see below
6	Retry accounting + lost observation	Fixed
7	README badge 5.0.0	Fixed

1 — reproduced at 10716ms on a 3000ms budget. The backoff was clipped to the deadline, but the retry it released then ran on the fixed 10s socket timeout, and no budget check can interrupt that: they run before a request starts and after it ends, never during. Every request is now bounded by min(10s, remaining maxDurationMs). Worth noting this overrun predates retries — the FIRST attempt had it too — so the fix covers both.

2 — confirmed: a 429 target produced {statusCode: 429, reason: 'http-error'} with no retry. 429 is the one error status that describes our own request rate rather than the resource, so calling it broken blames a site for how hard we crawled it. Now retried first, honouring Retry-After (bounded to 10s and clipped by the deadline); still-throttled targets report unverified / reason: 'throttled'. 5xx deliberately stays a dead link — that is conventional broken-link semantics and not a rate signal.

4 — confirmed: a redirect loop produced reason: 'fetch-error' with error: "Too many redirects (>5)." — a target that answered, repeatedly, described as silent. reason is now a typed union: unreachable / throttled / redirect-limit / body-too-large / unknown, and rows carry statusCode when there was one. This required CrawlPageObservation.errorCode — the typed code behind the human error string — since branching on the message text would be a string match on prose.

6 — confirmed, and the second half was the worse one. With the fetch budget exhausted, the target had no page row and no unverified entry at all: the URL vanished from the crawl with nothing to explain the gap, because the budget error propagated in place of the original transient one. The counter now increments only once a retry can actually be dispatched, and a deadline expiring during backoff re-raises the ORIGINAL failure so the observation survives.

5 — deferred, deliberately. Real, and reproduced. But that early return predates this work (#65), and fixing it properly means caching bodies across the frontier so a later re-fetch can reuse a response we already hold — a traversal change with dedup, ordering and checkpoint implications. Bundling it into a classification fix on an already-breaking PR trades a contained change for a risky one. This branch does improve its consequence: such a link is now reported unverified rather than as a fabricated dead link. Happy to open it as a follow-up issue.

3 — agreed, and the reasoning matters: 1.1 → 1.2 was explicitly documented as additive, so shipping a required field and two narrowed unions as a minor would tell a consumer the opposite of the truth.

Tests: 613 passing, typecheck and lint clean. New cases cover the 429 path both ways (still-throttled, and retried-then-answers), redirect-limit naming, the deadline bound (asserts under 6s where the bug measured 10.7s), and that a deadline expiring during backoff still records the page with attempted: 0.

🤖 Generated with Claude Code