fix(eda): worker subscribed to a dead topic — async extractions never processed - #30
Merged
Conversation
… processed The async worker's PostgresEventBus subscribed to `flydocs.jobs` (the `pyfly.eda.destinations` default), but the submit handler publishes `extraction.submitted` to `jobs_topic` = `flydocs.extractions`. The `worker` container therefore consumed from a topic nothing publishes to, so every queued async extraction sat unprocessed until the reaper revived it (~10 min later). In practice queue-backed submissions never completed. Align the `destinations` default to `flydocs.extractions` (matching `IDPSettings.jobs_topic`). Still env-overridable via `FLYDOCS_EDA_DESTINATIONS`; `bbox-worker` keeps its own `flydocs.extractions.post_processing` override. Also bump the locked pyfly dependency to 26.6.99 and release 26.6.4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Queue-backed (
POST /api/v1/extractions) extractions never completed. A submitted extraction sat inqueuedindefinitely; only the reaper revived it (~10 min later, afterqueued_orphan_threshold_s).Root cause
The async worker's
PostgresEventBussubscribes topyfly.eda.destinations, whose default inpyfly.yamlwasflydocs.jobs. But the submit handler publishes theextraction.submittedevent tojobs_topic=flydocs.extractions(IDPSettings.jobs_topic,config.py). So theworkercontainer subscribed to a topic nothing publishes to:while the outbox row was published to
flydocs.extractions. The consumer offset never advanced.Fix
Align the
destinationsdefault toflydocs.extractions(matchingjobs_topic). Behaviour is unchanged for anyone who already setsFLYDOCS_EDA_DESTINATIONS, and thebbox-workerkeeps its ownflydocs.extractions.post_processingoverride +flydocs-bbox-workersgroup.Also
pyflyframework dependency to26.6.99.26.6.4(version + CHANGELOG).Verification
Reproduced and confirmed against a live stack (api + worker + bbox-worker on
postgresEDA): before, a submitted extraction stayedqueuedwith the worker idle; after, the worker immediately claims it (destinations=['flydocs.extractions']), runs the pipeline (load → classify → plan_tasks → extract → …) and delivers the result webhook.