Expand background worker supervision and DLQ coverage#164
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
netenginehealth anddoctorcommands can show running/failed/disabled workers.Description
ConsumerSupervisorthat tracksWorkerStatus(states:registered,running,failed,disabled,stopped,completed), restart counts, timestamps, and providesget_structured_status()for operator use.register_disabled(name, reason)to record workers that are intentionally disabled (e.g., when PGMQ is unavailable) and avoid starting them.pki.cert_rotation,ands.org_admission,services.org_admission,monitoring.world_health,dlq.services_admissions, and normalize registry consumer names (e.g.,whois_server,dns_updates).DLQReplayWorkerinnetengine/workers/dlq_worker.pythat watches DLQs and only replays messages whenpayload.dlq_resolvedis true to prevent replaying unresolved poison messages.MonitoringServiceso monitored publishing can use the orchestrator's client and register monitoring undermonitoring.world_health; addnetengine/workers/registry.pyhelpers and expose expected worker statuses in the operator/api/v1/healthresponse andnetengine doctoroutput.Testing
python -m compileall -q netengine tests/test_background_workers.pyandrufflinting with no remaining errors. (passed)tests/test_background_workers.pywere added and executed, asserting crash visibility, graceful shutdown, disabled-worker registration, and DLQ guarded replay; these tests passed.pytest(includingtests/test_pki_features.py,tests/test_event_queue_dlq_mapping.py,tests/integration/test_phase5_registries.py,tests/integration/test_m8_operator_api.py,tests/test_cli.py) and observed all tests passing (145 passed).Codex Task