feat: implement robots.txt support for respectful crawling

[ChefControl]

Context

The project vision (docs/project-vision.md) states: "Rate limiting, robots.txt awareness, and polite user-agent strings by default."

Currently, the crawler fetches any URL without checking the target site's robots.txt. This directly contradicts the project's core "Respect others" principle.

Proposed Solution

• Fetch and parse /robots.txt before crawling each new domain
• Use the Rust robotstxt crate (Google's robots.txt parser)
• Cache the parsed robots.txt per domain per crawl to avoid repeated fetches
• Skip URLs that are disallowed by robots.txt, marking them as a new status (e.g., ROBOTS_BLOCKED)
• Support the Crawl-delay directive

Files to Modify

• shared/src/crawler.rs — add robots.txt fetch and check logic
• feeder/src/job.rs — integrate robots.txt check before processing URLs

Acceptance Criteria

• Crawler fetches and parses robots.txt for each new domain encountered
• Disallowed URLs are skipped and marked appropriately
• Crawl-delay directive is respected
• robots.txt results are cached per domain per crawl
• Unit tests for robots.txt parsing and URL filtering

[ChefControl]

A workspace is ready for this issue.

_{Powered by Worktree · Install}