feat: add per-domain rate limiting to prevent abuse

[ChefControl]

Context

The project vision states the tool should have "rate limiting" by default and be "hard to misuse for DoS or abuse."

Currently, 8 feeder replicas can hammer a single domain with concurrent requests — there is no delay between requests and no per-domain concurrency limit.

Proposed Solution

• Add a configurable per-domain request rate (default: 1 request/second/domain)
• Track last request timestamp per domain (either in Neo4j or in-memory per feeder)
• Add per-domain concurrency limit (max N feeders crawling the same domain simultaneously)
• Make rate limit configurable via environment variable

Files to Modify

• feeder/src/job.rs — add rate limiting check before fetching URLs
• feeder/src/config.rs — add rate limit configuration
• shared/src/neo4j_client.rs or new module — domain rate tracking if using Neo4j

Acceptance Criteria

• Per-domain rate limiting enforced (default 1 req/sec)
• Rate limit configurable via environment variable
• Per-domain concurrency limit prevents multiple feeders hitting same domain
• Rate limiting works correctly across multiple feeder replicas

[ChefControl]

A workspace is ready for this issue.

_{Powered by Worktree · Install}