Summary
download_delay sleeps before every dequeue in the global dispatch loop, not per-domain. With concurrent_requests=10 and delay=1.0s, effective throughput is capped at 1 req/s globally, making the concurrency setting nearly irrelevant.
Category
Bug / Design
Severity
Medium
Location
- File:
src/spiders/engine.rs
- Line: ~190 (main dispatch loop)
Details
The robots.txt path correctly applies Crawl-delay per-domain in process_request. However, the user-facing download_delay sleeps unconditionally in the global dequeue loop. For multi-domain crawls where the intent is "1s between requests to the same host", the current implementation over-throttles by a factor of concurrent_requests.
Suggested Fix
Track last-request-time per domain and apply the delay only when the interval has not elapsed for that specific domain, rather than sleeping before every global dequeue.
Effort Estimate
30 min
Automated finding by repo-health-agent v1.0
Summary
download_delaysleeps before every dequeue in the global dispatch loop, not per-domain. Withconcurrent_requests=10anddelay=1.0s, effective throughput is capped at 1 req/s globally, making the concurrency setting nearly irrelevant.Category
Bug / Design
Severity
Medium
Location
src/spiders/engine.rsDetails
The robots.txt path correctly applies
Crawl-delayper-domain inprocess_request. However, the user-facingdownload_delaysleeps unconditionally in the global dequeue loop. For multi-domain crawls where the intent is "1s between requests to the same host", the current implementation over-throttles by a factor ofconcurrent_requests.Suggested Fix
Track last-request-time per domain and apply the delay only when the interval has not elapsed for that specific domain, rather than sleeping before every global dequeue.
Effort Estimate
30 min
Automated finding by repo-health-agent v1.0