Give the possibility to create queue groups to improve politeness control

[saraaubry]

In Heritrix, queues are used to group URLs and to manage both the crawl budget and the communication with a web server. Queue assignment is configured through a QueueAssignmentPolicy, for example:

<bean id="queueAssignmentPolicy"
      class="org.archive.crawler.frontier.HostnameQueueAssignmentPolicy">
    <property name="forceQueueAssignment" value="" />
    <property name="deferToPrevious" value="true" />
    <property name="parallelQueues" value="1" />
</bean>

Several implementations already exist, including HostnameQueueAssignmentPolicy, DomainnameQueueAssignmentPolicy, SurtAuthorityQueueAssignmentPolicy, etc.
This policy is called from the FrontierPreparer during the CANDIDATE processing chain.
At the BnF, we also use a custom DomainnameQueueAssignmentPolicy implementation provided by NetarchiveSuite.

Problem

Web servers increasingly protect themselves against web crawlers. During our crawls, we have identified two common situations.

1. Multiple host queues targeting the same server

With the default HostnameQueueAssignmentPolicy, a large website may be split into several host queues. Although these are separate queues, they often resolve to the same physical web server. As a result, Heritrix opens multiple simultaneous connections to the same server, whereas a human user would normally use only one. This may trigger anti-bot protections, leading to permanent blocking (HTTP 403 or Heritrix status -2) or temporary rate limiting (HTTP 426 or HTTP 429). The European Union websites (europa.eu) are a typical example.

2. Site-specific politeness requirements

Some websites require significantly longer delays between requests than the default politeness configuration. For example, africultures.com and afrik.com requested a minimum interval of one minute between consecutive requests. Although Heritrix already supports site-specific politeness settings through Sheets, this mechanism only changes delay parameters and does not allow multiple queues to share the same crawl schedule.

We propose to introduce the concept of Queue Groups.

A Queue Group is a logical collection of existing queues. All queues may share the same schedule and politeness constraints but individual queues may have different budget (in terms of nb of URLs or size). Instead of scheduling each queue independently, Heritrix would schedule the entire group as a single communication channel toward a web server. Within a Queue Group, Heritrix should perform round-robin scheduling between the member queues.

For example:

• one URL from european-union.europa.eu
• then one URL from commission.europa.eu
• then one URL from data.europa.eu
  ...

This approach provides several benefits:

• only one active connection toward the target server;
• fair distribution of requests across all hosts;
• avoids spending too much time on a single queue while others remain idle;
• naturally skips temporarily unavailable hosts without blocking the entire crawl.

Example 1: European Union websites

Create a Queue Group named europa_eu containing:
european-union.europa.eu
commission.europa.eu
data.europa.eu
op.europa.eu
youth.europa.eu
...

The crawler should treat these queues as a single scheduling unit while continuing to alternate between the individual queues.

**Example 2: Change politeness on the EU group **

Associate the europa_eu group to a Sheet Queue Group with a Sheet redefining longer politeness delays:
<property name="minDelayMs" value="10000" />

Heritrix already provides the Sheet mechanism to override configuration values for specific SURTs, domains, hosts or URL paths. Sheets should also apply to queue groups.

Expected benefits of the combination of groups+sheets are:

• fewer HTTP 403, 426 and 429 responses
• hopefully a reduced risk of temporary or permanent blocking
• and a simpler configuration for large multi-host websites.

[ato]

I think this is an important problem to solve. It's very common to have many websites hosted on same server or service, potentially with completely different domain names and for the crawler to trigger limits or even overwhelm the server. Using IP-based queues sort of solves the politeness problem, but has lots of side-effects like loss of limit granularity and balance.

Option 1. Hierarchical queue groups

If I'm understanding correctly the proposal is to have the frontier select the next URI with two levels of round robin like this:

---
config:
  layout: elk
---
graph TD
    Frontier[Frontier]

    RR1[round-robin between groups]
    RR2[round-robin within group]
    RR3[round-robin within group]

    QG1[Queue Group: europa.eu]
    QG2[Queue Group: example.com]

    Q1[Queue: european-union.europa.eu]
    Q2[Queue: commission.europa.eu]
    Q3[Queue: data.europa.eu]
    Q4[Queue: www.example.com]

    Frontier --> RR1

    RR1 --> QG1
    RR1 --> QG2

    QG1 --> RR2
    RR2 --> Q1
    RR2 --> Q2
    RR2 --> Q3

    QG2 --> RR3
    RR3 --> Q4

    classDef frontier stroke:#818cf8,fill:#eef2ff
    classDef queueGroup stroke:#2dd4bf,fill:#f0fdfa
    classDef queue stroke:#a78bfa,fill:#f5f3ff
    classDef routing fill:none,stroke:none,color:#64748b

    class Frontier frontier
    class QG1,QG2 queueGroup
    class Q1,Q2,Q3,Q4 queue
    class RR1,RR2,RR3 routing

Loading

If the delay is enforced at the Queue Group level, this solves the politeness problem, but it also makes significant change to scheduling balance. Assuming both groups are continuously ready, the scheduling opportunities for each queue would be approximately:

• 17% european-union.europa.eu
• 17% commission.europa.eu
• 17% data.europa.eu
• 50% example.com

I can imagine this is potentially desirable in some cases, like giving similar weight to sites that just happen to use a lot of subdomains but not in others like unrelated sites that use the same hosting service.

I think this would require substantial changes to the frontier and opens up a lot of questions like:

• is there precedence and budgets at both levels?
• how do snoozing and exhaustion work?

Option 2. Flat queues with shared politeness groups

I can imagine also solving this in a different way, where we keep the current scheduler doing a flat round robin of queues but gate queue eligibility with a separate set of politeness groups:

---
config:
  layout: elk
---
graph TD
    Frontier[Frontier]
    RR[flat round-robin between queues]

    Q1[Queue: european-union.europa.eu]
    Q2[Queue: commission.europa.eu]
    Q3[Queue: data.europa.eu]
    Q4[Queue: www.example.com]

    PG1[Politeness Group: europa.eu]
    PG2[Politeness Group: example.com]

    G1[Shared concurrency and delay gate]
    G2[Shared concurrency and delay gate]

    Frontier --> RR

    RR --> Q1
    RR --> Q2
    RR --> Q3
    RR --> Q4

    Q1 -. politeness key .-> PG1
    Q2 -. politeness key .-> PG1
    Q3 -. politeness key .-> PG1
    Q4 -. politeness key .-> PG2

    PG1 --> G1
    PG2 --> G2

    classDef frontier stroke:#818cf8,fill:#eef2ff
    classDef queue stroke:#a78bfa,fill:#f5f3ff
    classDef politenessGroup stroke:#2dd4bf,fill:#f0fdfa
    classDef gate stroke:#f59e0b,fill:#fffbeb
    classDef routing fill:none,stroke:none,color:#64748b

    class Frontier frontier
    class Q1,Q2,Q3,Q4 queue
    class PG1,PG2 politenessGroup
    class G1,G2 gate
    class RR routing

Loading

Before dispatching a URI, the frontier would check the queue's politeness group:

• If its gate is available, the URI is dispatched and the resulting crawl delay applied as a shared cooldown.
• If its gate is busy or cooling down, that queue is parked until the gate becomes eligible, and selection continues with another queue.

With this design the crawl scheduling opportunity of each queue remains evenly distributed:

• 25% european-union.europa.eu
• 25% commission.europa.eu
• 25% data.europa.eu
• 25% example.com

(Obviously the actual fetch percentage will depend on delays, response times, queue exhaustion, precedence, retries etc. But let's assume this is a large crawl with many queues where the effect of these factors on scheduling is minor.)

I suspect politeness groups would be easier to add to the current frontier implementation than queue groups as they would just affect queue readiness with most of the rest of structure remaining the same.

Edit: It occurred to me that some organizations use lots of small per-site crawl jobs rather than a single large crawl job containing all the sites. For these users it could be very desirable to have server politeness shared between multiple jobs or even multiple instances of Heritrix. That also seems more feasible with the politeness groups approach.

[haja91150]

Our proposition for the approach is based on your option 2. We want to introduce "Queue Groups" as an additional scheduling layer on top of the existing queues.
Each host keeps its own queue and its own classKey, budget, precedence, snooze state, and checkpoint behavior. The group only controls how those queues are scheduled together.
In practice, a Queue Group provides two shared constraints:

• a shared concurrency limit, through maxParallelInGroup;
• a shared minimum delay between fetches, through groupMinDelayMs.

With maxParallelInGroup = 1, all queues in the group behave as a single crawl channel from the server’s point of view. This means that, for example, several europa.eu subdomains can be crawled as one polite unit instead of opening multiple simultaneous connections to the same infrastructure.
If one member is empty, snoozed, already in process, or temporarily unavailable, it is skipped and does not block the rest of the group.
This keeps the behavior fair across the grouped hosts while avoiding the situation where one host is drained before the others get a chance to run.

We will implemented this as an opt-in and additive feature. If no Queue Group configuration is present, the frontier behaves exactly as before. Ungrouped queues continue to follow the original scheduling path without any change.
The approach is therefore:

• keep the existing queue assignment policies unchanged;
• keep one queue per host/classKey;
• add grouping only at the frontier scheduling level;
• apply shared politeness and concurrency only to queues that belong to a configured group;
• preserve existing per-queue budgets and precedence.

The implementation adds three main pieces:

• QueueGroup: represents the configuration and runtime state of one group. It stores the group name, the member matchers, the maximum allowed parallelism, and the shared minimum delay. It also manages the group-level scheduling state, including the shared gate and the round-robin rotation between discovered member queues.
• QueueGroupManager: acts as the optional registry for all configured groups. It resolves which group, if any, a given queue belongs to, caches those lookups, and records the concrete queues discovered at runtime. It also allows groups or members to be added or removed dynamically, which can be useful from the scripting console.
• WorkQueueFrontier integration: the frontier remains responsible for selecting the next eligible queue, but now checks whether a queue belongs to a group before emitting its next URI. If the queue is not grouped, the original behavior is preserved. If it is grouped, the frontier applies the group gate and the group round-robin rules before allowing the URI to be fetched.

A key part of the design is the group gate.
The group gate is the shared politeness mechanism for all queues in the same group. It decides whether the group is currently allowed to emit another URI. The gate can be closed for two reasons:

• Shared concurrency is already full
  If the number of active queues in the group has reached queueGroup.getMaxParallelInGroup(), no other member of the group may start a fetch until one of the active fetches finishes.
• The group is still in shared cooldown
  After a fetch finishes, the group arms a shared cooldown. The next fetch from any member of the group must wait until this cooldown expires. The effective delay is based on the normal per-URI politeness delay, but can be raised by groupMinDelayMs. effective delay : max(curi.getPolitenessDelay(), groupMinDelayMs)

When the gate is closed, the queue is not emitted. Instead, it is put aside until the group becomes eligible again, allowing the frontier to continue serving other queues outside the group, or other groups that are ready.
When the gate is open, the frontier still checks the group round-robin order. This ensures that the next URI comes from the member whose turn it is, rather than always selecting the first ready queue. Once a member is allowed through, the group acquires a shared slot. When the fetch completes, that slot is released and the group cooldown is updated.
In other words, the group gate makes the group behave like a single polite communication channel toward the target server, while the round-robin logic keeps distribution fair between the individual host queues inside that channel.

This also supports sites that require stricter politeness delays, such as africultures.com and afrik.com, by placing them in a group with a higher groupMinDelayMs.

We think about adding observability in the frontier summary report, so that configured groups expose their current state: configured members, discovered queues, active in-process count, and remaining cooldown time.

We would like to add queueGroup annotation in CrawlUri to display it in Crawl Report.