Skip corrupt jobs in list and include corrupt_jobs_count in response - #519
Open
jmadler wants to merge 3 commits into
Open
Skip corrupt jobs in list and include corrupt_jobs_count in response#519jmadler wants to merge 3 commits into
jmadler wants to merge 3 commits into
Conversation
- get_queue_registry_jobs_count: safe fetch per job, skip NoSuchJobError/ DeserializationError, return (total_items, jobs, corrupt_count) - list_jobs: include corrupt_jobs_count in JSON response - README: document list resilience and corrupt_jobs_count
Collaborator
|
@jmadler we did not have any response back, we will wait for one more week and then close this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When listing jobs (e.g. /data/jobs///...), a single missing or corrupt job (e.g. NoSuchJobError or DeserializationError from queue.fetch_job()) can make the whole list request fail with a 500. That makes the jobs UI brittle when Redis has stale or corrupted job IDs in a registry.
This change fixes with two improvements:
get_queue_registry_jobs_count
Fetches each job in a loop and catches NoSuchJobError and DeserializationError. Corrupt/missing jobs are skipped and counted. The function still returns total registry size and the list of successfully fetched jobs, and now also returns a third value: the number of skipped (corrupt) jobs.
list_jobs Uses the new 3-tuple and includes corrupt_jobs_count in the JSON response so the UI or API consumers can show how many jobs were skipped.
Type of change
Checklist: