Make crawler's frontier durable - #47
Merged
Merged
Conversation
One crawler held one order open for its process lifetime, keeping the frontier on the heap. A crash lost it, and a run of millions of URLs serialized through one process. The frontier becomes a JetStream work queue, `YACY_CRAWL_FRONTIER`, carrying one message per discovered URL. Delivery leases the URL, `ReturnAfter` carries the defer and the retry delay, and `Ack` removes it. A worker publishes a discovered URL before anything durable, so a death loses no URL, and claims the URL it is delivered in KV `YACY_VISIT_CLAIMS`, which makes the visit exclusive. `Create` failing on a redelivery is the worker's own earlier claim, so a restart re-visits without spending that URL's deferrals or attempts. KV `YACY_ACCEPTED_ORDERS` holds the order every worker rebuilds the predicate from; `YACY_CRAWL_ORDERS` is unchanged. `profileadmission` stops recording and becomes the predicate `Admits`. `MaxPagesPerHost` becomes a CAS counter spent at claim time, and now counts visited pages rather than queued URLs, disposing as `host-pages-spent`. The run page budget and the frontier cap go with the heap they bounded, and with them `YACYCRAWLER_RUN_PAGE_BUDGET`, `YACYCRAWLER_FRONTIER_CAP`, and `yacycrawler_budget_exhaustions_total`. An order is now acknowledged once the order and its seeds are durable, so per-order completion stops being observable. `pullintake.PendingMessage` gains `Deliveries`, which the claim rule reads to tell a redelivery from a duplicate. Arch-lint: the `serviceruntime` vendor allowance widens with `poisonhalt` and `pullintake`, which both new intakes use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An order intake accepts an order and makes its seeds durable; it never completes the order, and it never redelivers a message. Rename the observer to `OrderAccepted` and `OrderReturned`, with the metrics and the dashboard legends that carry them, so `Return` is the one word for that fact from `PendingMessage` to Prometheus. Split `visitor.go` into one level per function. `Visit` names the recrawl decision, the fetch, and the outcome of the fetch; `outcomeOfFetch` holds only the status switch; `outcomeOfPageContent` is the only function that works. `absorb` and `absorptionOf` go, and `VisitOutcome` takes its three constructors, one per conclusion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A visit read as six ways of saying "outcome" over two result vocabularies, and the decisions a page's content forces sat in one function that no name fit. Each decision now belongs to the unit that owns it. The robots grammar lives once, in pagerobots. A fetched page carries the directives it was served instead of two parsed booleans, so the crawler reads what the headers and the markup state through one call. A visit records itself and reports progress in the switch that decides it, so the fetched-page path is one scenario a reader can follow. Names that promised a value while writing state now carry verbs, and completed, deferred, and retryable outcomes are named for the value they build. Which refusals an order ignores travels as one value, applied once as a mask, so a page that refuses indexing reads the same wherever it is read. A visitor comes from VisitorFor, the derivation that binds it. Disposal reasons are split by the unit that reports them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The disposal reasons lived in two files split by which unit reported them, a fact that exists nowhere else, is not checkable, and is none of the disposal package's business. They are one closed enum and now live in one file named for the type it declares, in visit-pipeline order. One word carried two refusals: the fetch-level cease and the markup noindex were both Refused. The cease keeps the word pagefetch and refusal already use for it, so the demand reads as cease from the wire to the metric label. NotAPage covered two unrelated fetch endings: a client-error answer from the target, and a redirect landing on a URL no canonical form accepts. No single word covers both, so the fetch status splits in two and each half carries its own reason. The three settlement reasons all end an allowance, and now all say so. Spending a host page stays the verb; exhausted is the state it reaches, matching deferrals and retries. The one rule about a Reason - whether it disposes at all - moved onto the type from the outcome that held it, so NotDisposed stops being knowledge every caller keeps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An order naming a pattern the crawler cannot read used to reach the frontier, and every visit under it then halted the whole crawl intake: the profile was compiled per visit, and the compile failure was reported as an undecodable message. The message settled nothing, so it came back and halted intake again on every restart. The order is now read once, at each point it enters the crawler. An AcceptedOrder carries what the operator sent together with the rule that decides which URLs it admits, so an order that cannot be read cannot be built, and no later step has to remember to check. Order intake reads the order it received; the order store reads back what it kept. Visit intake asks the order what it admits and no longer knows that a profile is compiled at all. An order the crawler cannot read returns for redelivery like every other order it cannot finish. Dropping what an operator sent is a policy decision that does not belong in a consumer branch. Returns now hold the message back before it comes again, so a failure that persists no longer naks in a tight loop. visitcrawl refuses an unreadable pattern where the operator types it. Arch rules dropped: acceptedorder is a new component, depending on profileadmission; orderintake, visitintake and acceptedordersjetstream may depend on it. visitintake no longer depends on profileadmission. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A claim recorded nothing about who took it, so visit intake guessed from the delivery count: a message that found the URL claimed carried on when the broker had delivered it before, on the assumption that the claim was its own. The frontier admits the same URL under two messages, so the assumption was wrong whenever one of them returned before it ever claimed, and the crawler fetched the page twice. The claim now carries its holder, and the ledger answers what it found: the claim was taken, the holder resumed its own, or another holder has it. The holder is the frontier message, which stays the same across its own redeliveries. Visit intake reads the answer, and the rule that only a fresh claim spends a host page needs no delivery count to hold. Nothing needed to know that a message came back again, so pending messages no longer report it. A returned visit logged a hand-written reason beside the error that already carried one, and for a spent host page the reason was invented: a write the ledger refused was reported as a claim it could not read. Every step wraps its own error, so the reason is gone and the error stands alone. The store that keeps accepted orders keeps them: accepting is what building an AcceptedOrder does. What visit intake reports on is the frontier, not a settlement. Arch rules dropped: visitclaim is a new component; visitintake and visitclaimsjetstream may depend on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six test files each carried their own broker: a jetstream.Msg fake, an iterator over a slice, and a source that hands it out. They had drifted apart on how a settlement is recorded and what a message knows about where it sits, so a test read differently depending on which service it belonged to. pullintaketest now holds the one message and the one source. A message records what the intake settled it as and how long it was held back, and it names the stream position it sits at, which the crawler's visit intake needs and the others leave at zero. A source hands out the messages it was given, refuses to open, or holds its first message until a test releases it. The three scrape request intakes waited on a settlement channel to learn what the intake decided. A run returns only once its workers are done, so they read the settlement off the message instead, and the tests that asked for a message only to await its channel now name the message they are asserting on. Arch rules dropped: pullintaketest is a new component using nats. The pullintake component no longer covers its subdirectories. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Config structs restated each consumer's collaborator list a second time and called it configuration, though only the retry bounds and the concurrency configure anything. A missing field left an interface nil until the first delivered message. Both constructors now take their collaborators as parameters; the visit consumer needs eight, so it drops revive's argument-limit on that one declaration. FrontierProgress reported no frontier fact: both of its methods say what became of a pending visit, so it is PendingVisitProgress, a sibling of OrderProgress. The field that publishes onto the frontier is now named for that place, so it agrees with putDiscoveredURLsOnFrontier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The visit claims took every failed write as contention, so a broken broker read as another worker winning the key, and the host page a failed spend never took was skipped for good once the redelivery resumed the claim. jetstreamrecord now owns the read-modify-write: it retries only on ErrKeyExists, backs off between attempts, and hands an absent key to the rule as the zero record, so a host allowance is checked on the first page as well. A visit claim records the host page it spent, and the consumer asks for that page on every delivery. Arch-lint gains the component jetstreamrecord, which may use nats, and the edge visitclaimsjetstream -> jetstreamrecord. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pagemarkup promised markup and parsed HTML alone: its media gate admits text/html and application/xhtml+xml, its sentinel is ErrNotHTML, and it yields *html.Node, so both readers import x/net/html themselves. It is pagehtml now, and its value is an ElementTree, because the walk yields element nodes only and no caller can reach a title or the body text through it. When a second format arrives, pagehtml sits beside its sibling and pagevisit keeps the choice between them. pagerobots said robots in its name and refusals in everything it exports. Refusal is the fact, stated by a meta tag or by a header the package already takes, so the package is pagerefusals and robots names the protocol at the one argument that carries it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The wiring held ten constants while the service config held the rest of the same category, so a tuning change first had to pick a file. Every crawl tunable now reaches the wiring through ServiceConfig, following the bucket spec methods already there. The ops server limits stay in the wiring, where the sibling services declare them too. The fetch retry limit and MaxAttemptsPerURL were two names for one fact; the field name is now the only one. The pending visit ack wait was derived in the wiring and is now derived where the value lives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The refusal package held two constants and no rule: nothing computed a Demand, both call sites wrote a literal, and the only consumer was a Prometheus label. Cease and Defer were a second spelling of two fetch statuses, invented so one metric could carry a label, and the label words sat in a component four packages depended on. The demand is a method now, VisitProgress declares the three refusals a page states and PendingVisitProgress the one the intake honors, and the label spellings live beside labelDemand where the rest of the transport words are. refusals_honored_total counted two of the four refusals it promised: a honored noindex reached the dashboard as a disposal reason and a honored nofollow reached nothing. The visitor reports both now, so the metric counts every refusal it honors. FetchCeased is 401, 403 and 451, which is the target refusing access to one URL, not a demand to stop crawling. It is FetchAccessRefused, the disposal reason is AccessRefused, and the demand label is access-refusal, so one word carries the fact from the status line to the panel. pages_disposed_total drops the reason crawl-ceased for access-refused. Arch-lint drops the component refusal and its four edges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The name travelled from the visitcrawl environment through the order contract to every crawler, and no rule ever consulted it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A readability pass over yacycrawler against the project rules. Storage: jetstreamrecord gains BucketSpec and KeyOf, so every bucket owner opens and addresses its records one way. Host page allowances leave the visit claims bucket for YACY_HOST_PAGE_ALLOWANCES, which retires the key discriminator that only existed because two vocabularies shared one bucket. In-flight counters do not migrate; both buckets expire in seven days. Rules: visitallowance spends what a pending visit is allowed and names the allowance that ran out, so the visiting consumer no longer holds the retry delay. pagehtmlreading reads a fetched page as HTML, which leaves pagevisit at one level of abstraction. Refusals the order ignores move to pagerefusals, where the refusal vocabulary lives. pagehtml yields named elements and an Element that reads its own attributes, so its callers no longer touch net/html. Configuration: fixed tunables live beside the environment-driven ones in ServiceConfig, and the crawler asks it whether recrawl suppression is on rather than reading the grace twice. Removed: the unread PageReference contract type, the clock package no caller slept on, ElementTree.Elements, and VisitOutcome.Disposed. Arch-lint allows more: new components pagehtmlreading, visitallowance and hostpageallowancesjetstream, new edges onto jetstreamrecord from acceptedordersjetstream and recrawlrulesdueaftergrace. The nethtml vendor edge drops from pagerefusals and linkdiscovery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The visiting consumer called its port allowance, the same word as the allowance every method on it returns, so one line read the value and the collaborator with one name. The port is the ledger; an allowance is what it answers with. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012RM6rTFEkzTquJAq5n9gM2
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012RM6rTFEkzTquJAq5n9gM2
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012RM6rTFEkzTquJAq5n9gM2
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012RM6rTFEkzTquJAq5n9gM2
fetchtiming owned no state, no endpoint, and no value type — only a stage of the visit, so the visitor now times the fetch it already makes. It reports the duration for every attempt, failures included, so yacycrawler_fetch_duration_seconds means what its help text says. VisitProgress.FetchTook names the duration observed, not the event that carries it. Arch-lint drops the fetchtiming component and the progress observer's edge to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012RM6rTFEkzTquJAq5n9gM2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #44
Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling.