Scheduled Crawling & Monitoring System

[stokry]

Add support for scheduled/recurring web crawling jobs with change detection, monitoring, and notification capabilities. This feature would enable users to automatically track changes on websites over time, making Deepcrawl suitable for production monitoring use cases.

Currently, Deepcrawl only supports one-time URL processing. Users who need to:
• Monitor price changes on e-commerce sites
• Track content updates on competitor websites
• Monitor product availability
• Aggregate news from multiple sources
• Track SEO changes over time

…must manually trigger crawls repeatedly, which is not scalable for production workflows.

Maybe I can help with this.

[lumpinif]

Hey, stokry, thanks for suggesting this.

Currently deepcrawl is under a high volume of free usage, and scheduled crawling would be great to have, as long as we can take advantage of cloudflare infra to make this happen, such as queue and cron jobs, and pr is more than welcome if you'd like to do this.

In the meantime, I am seeking an opportunity to work on transitioning deepcrawl into "one cli command to deploy deepcrawl" just like how we use shadcn to add a component into our codebase. I believe having one cli to add a saas is feasible and something great to try.

Thank you for suggesting this again.

[stokry]

Wow, that sounds even better.

The shadcn-style “one CLI command to deploy Deepcrawl” idea is really compelling. I’d love to know what stage this is currently in and how solid the direction already is.