feat: add configurable maximum URL cap per crawl

[ChefControl]

Context

A depth-5 crawl on a large site can generate millions of URL nodes with no limit. This can overwhelm Neo4j and effectively constitute a denial of service against the target.

Proposed Solution

• Add a configurable max URL limit per crawl (default: 10,000)
• Check the current URL count before creating new child nodes in the feeder
• Stop discovering new URLs once the cap is reached (existing PENDING URLs can still be processed)
• Expose the limit as a parameter in the crawl creation API
• Display the limit in the frontend crawl form

Files to Modify

• manager/src/routes/crawl.rs — accept max_urls parameter
• manager/src/models/ — add field to request/response types
• feeder/src/job.rs — check URL count before creating children
• frontend/src/pages/NewCrawl.tsx — add max URLs input

Acceptance Criteria

• Configurable max URL cap with sensible default
• Feeder stops creating new child nodes when cap is reached
• API accepts max_urls parameter
• Frontend exposes the setting

[ChefControl]

A workspace is ready for this issue.

_{Powered by Worktree · Install}