Skip to content

fix(eda): worker subscribed to a dead topic — async extractions never processed - #30

Merged
ancongui merged 1 commit into
mainfrom
fix/worker-eda-topic
Jun 12, 2026
Merged

fix(eda): worker subscribed to a dead topic — async extractions never processed#30
ancongui merged 1 commit into
mainfrom
fix/worker-eda-topic

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Problem

Queue-backed (POST /api/v1/extractions) extractions never completed. A submitted extraction sat in queued indefinitely; only the reaper revived it (~10 min later, after queued_orphan_threshold_s).

Root cause

The async worker's PostgresEventBus subscribes to pyfly.eda.destinations, whose default in pyfly.yaml was flydocs.jobs. But the submit handler publishes the extraction.submitted event to jobs_topic = flydocs.extractions (IDPSettings.jobs_topic, config.py). So the worker container subscribed to a topic nothing publishes to:

PostgresEventBus started: channel=flydocs_eda destinations=['flydocs.jobs'] group=flydocs-workers

while the outbox row was published to flydocs.extractions. The consumer offset never advanced.

Fix

Align the destinations default to flydocs.extractions (matching jobs_topic). Behaviour is unchanged for anyone who already sets FLYDOCS_EDA_DESTINATIONS, and the bbox-worker keeps its own flydocs.extractions.post_processing override + flydocs-bbox-workers group.

Also

  • Bump the locked pyfly framework dependency to 26.6.99.
  • Release 26.6.4 (version + CHANGELOG).

Verification

Reproduced and confirmed against a live stack (api + worker + bbox-worker on postgres EDA): before, a submitted extraction stayed queued with the worker idle; after, the worker immediately claims it (destinations=['flydocs.extractions']), runs the pipeline (load → classify → plan_tasks → extract → …) and delivers the result webhook.

… processed

The async worker's PostgresEventBus subscribed to `flydocs.jobs` (the
`pyfly.eda.destinations` default), but the submit handler publishes
`extraction.submitted` to `jobs_topic` = `flydocs.extractions`. The `worker`
container therefore consumed from a topic nothing publishes to, so every
queued async extraction sat unprocessed until the reaper revived it (~10 min
later). In practice queue-backed submissions never completed.

Align the `destinations` default to `flydocs.extractions` (matching
`IDPSettings.jobs_topic`). Still env-overridable via `FLYDOCS_EDA_DESTINATIONS`;
`bbox-worker` keeps its own `flydocs.extractions.post_processing` override.

Also bump the locked pyfly dependency to 26.6.99 and release 26.6.4.
@ancongui
ancongui merged commit 92fd7aa into main Jun 12, 2026
8 checks passed
@ancongui
ancongui deleted the fix/worker-eda-topic branch June 12, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant