Description
The cancel operation (DELETE /api/v1/crawls/:id) sets all PENDING/IN-PROGRESS URLs to CANCELLED atomically, but feeders that are already mid-processing overwrite the CANCELLED status back to COMPLETED and continue creating new PENDING children. This effectively undoes the cancel.
Steps to Reproduce
- Start a deep crawl (e.g.,
https://wikipedia.org at depth 5)
- Wait until feeders are actively processing (multiple IN-PROGRESS jobs)
- Cancel the crawl via
DELETE /api/v1/crawls/:id
- Observe: the crawl continues growing,
cancelled count stays at 0
Root Cause
Two issues in feeder/src/job.rs:
1. update_job_status unconditionally overwrites status (line 82)
MATCH (n:URL {name: $name, http_type: $http_type, current_depth: $current_depth, crawl_id: $crawl_id})
SET n.job_status = $status, n.attempts = $attemptsWhen a feeder finishes processing a job, it calls update_job_status(graph, job, "COMPLETED", ...) which overwrites the CANCELLED status set by the cancel operation.
Fix: Add a guard clause:
MATCH (n:URL {name: $name, http_type: $http_type, current_depth: $current_depth, crawl_id: $crawl_id})
WHERE n.job_status <> 'CANCELLED'
SET n.job_status = $status, n.attempts = $attempts2. batch_create_children doesn't check for cancellation (line 167)
After a feeder completes fetching HTML and resolving DNS, it creates new PENDING children without checking if the crawl was cancelled in the meantime. These new children get picked up by other feeders, perpetuating the crawl.
Fix: Check cancellation before creating children in feeding(), or have batch_create_children skip creation if the crawl is cancelled.
3. is_cancelled check is a one-shot (line 269)
The cancellation check at the top of feeding() only runs once. If cancel fires after this check but before update_job_status, the feeder won't notice.
Expected Behavior
After cancel, the crawl should stop within seconds. No new children should be created, and no CANCELLED statuses should be overwritten.
Affected Files
feeder/src/job.rs — update_job_status, batch_create_children, feeding
Description
The cancel operation (
DELETE /api/v1/crawls/:id) sets all PENDING/IN-PROGRESS URLs to CANCELLED atomically, but feeders that are already mid-processing overwrite the CANCELLED status back to COMPLETED and continue creating new PENDING children. This effectively undoes the cancel.Steps to Reproduce
https://wikipedia.orgat depth 5)DELETE /api/v1/crawls/:idcancelledcount stays at 0Root Cause
Two issues in
feeder/src/job.rs:1.
update_job_statusunconditionally overwrites status (line 82)When a feeder finishes processing a job, it calls
update_job_status(graph, job, "COMPLETED", ...)which overwrites the CANCELLED status set by the cancel operation.Fix: Add a guard clause:
2.
batch_create_childrendoesn't check for cancellation (line 167)After a feeder completes fetching HTML and resolving DNS, it creates new PENDING children without checking if the crawl was cancelled in the meantime. These new children get picked up by other feeders, perpetuating the crawl.
Fix: Check cancellation before creating children in
feeding(), or havebatch_create_childrenskip creation if the crawl is cancelled.3.
is_cancelledcheck is a one-shot (line 269)The cancellation check at the top of
feeding()only runs once. If cancel fires after this check but beforeupdate_job_status, the feeder won't notice.Expected Behavior
After cancel, the crawl should stop within seconds. No new children should be created, and no CANCELLED statuses should be overwritten.
Affected Files
feeder/src/job.rs—update_job_status,batch_create_children,feeding