Skip to content

Expand background worker supervision and DLQ coverage#164

Merged
aidankhogg merged 2 commits into
dev/alphafrom
codex/expand-background-worker-coverage
Jun 29, 2026
Merged

Expand background worker supervision and DLQ coverage#164
aidankhogg merged 2 commits into
dev/alphafrom
codex/expand-background-worker-coverage

Conversation

@aidankhogg

Copy link
Copy Markdown
Contributor

Motivation

  • Provide robust supervision and operator visibility for long-running background consumers (PKI rotation, org admission, monitoring, DLQ replay, WHOIS/DNS consumers) so crashes, restarts, and disabled dependency states are observable.
  • Prevent unsafe DLQ replay by requiring an explicit resolved marker before moving messages back to primary queues.
  • Surface expected worker status in operator surfaces so netengine health and doctor commands can show running/failed/disabled workers.

Description

  • Replace the simple task map with a structured ConsumerSupervisor that tracks WorkerStatus (states: registered, running, failed, disabled, stopped, completed), restart counts, timestamps, and provides get_structured_status() for operator use.
  • Add register_disabled(name, reason) to record workers that are intentionally disabled (e.g., when PGMQ is unavailable) and avoid starting them.
  • Register workers with stable, operator-visible names across phases: pki.cert_rotation, ands.org_admission, services.org_admission, monitoring.world_health, dlq.services_admissions, and normalize registry consumer names (e.g., whois_server, dns_updates).
  • Add DLQReplayWorker in netengine/workers/dlq_worker.py that watches DLQs and only replays messages when payload.dlq_resolved is true to prevent replaying unresolved poison messages.
  • Inject PGMQ client into MonitoringService so monitored publishing can use the orchestrator's client and register monitoring under monitoring.world_health; add netengine/workers/registry.py helpers and expose expected worker statuses in the operator /api/v1/health response and netengine doctor output.

Testing

  • Ran static/packaging checks: python -m compileall -q netengine tests/test_background_workers.py and ruff linting with no remaining errors. (passed)
  • Unit tests for the new behaviors in tests/test_background_workers.py were added and executed, asserting crash visibility, graceful shutdown, disabled-worker registration, and DLQ guarded replay; these tests passed.
  • Ran the full relevant test matrix with pytest (including tests/test_pki_features.py, tests/test_event_queue_dlq_mapping.py, tests/integration/test_phase5_registries.py, tests/integration/test_m8_operator_api.py, tests/test_cli.py) and observed all tests passing (145 passed).

Codex Task

@aidankhogg aidankhogg self-assigned this Jun 29, 2026
@aidankhogg aidankhogg added the enhancement New feature or request label Jun 29, 2026
@aidankhogg aidankhogg merged commit ef220af into dev/alpha Jun 29, 2026
@aidankhogg aidankhogg deleted the codex/expand-background-worker-coverage branch June 29, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant