Worker health server: /actuator/health over HTTP in worker + bbox-worker (26.6.3) - #29
Merged
Merged
Conversation
flydocs worker and flydocs bbox-worker now run a Starlette + uvicorn
health server (assembled from pyfly's actuator) as a sibling asyncio
task, so Kubernetes probes the worker pods with httpGet against
/actuator/health/{liveness,readiness} instead of exec shims. Binds
0.0.0.0; port = FLYDOCS_WORKER_HEALTH_PORT, falling back to
FLYDOCS_PORT; 0 disables it (and task dev:worker sets 0 so the
documented serve+worker same-host flow keeps working).
Lifecycle: the health task joins the worker/reaper
asyncio.wait(FIRST_COMPLETED) set; when any task finishes, the stop
flags flip, siblings drain through their own exit paths within a grace
period, and stragglers are cancelled and awaited before the pyfly
shutdown — so a bind failure (uvicorn's SystemExit, converted to
RuntimeError by serve_health) or a dead worker takes the whole process
down cleanly. SIGTERM now stops worker, reaper, and health server
gracefully (uvicorn's own signal capture is disabled; the CLI owns
signals). Access log off; only health + info exposed by default
(loggers/metrics 404 per pyfly's secure-by-default web exposure).
Indicator discovery uses pyfly >= 26.6.98's public
pyfly.actuator.install_health_indicators; database_health and
eda_health participate in both probes, matching the API process
(documented in docs/deployment.md). The pyfly dependency adds the web
extra so starlette and uvicorn are declared instead of transitive.
Also: compose healthchecks for worker/bbox-worker, env_template entry
(plus realigning two stale defaults), deployment docs, and the webhook
User-Agent / version endpoint examples bumped per the release ritual.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the platform work order (2026-06-12):
flydocs workerandflydocs bbox-workermust serve/actuator/health/*over HTTP so thecore-flydocs0.1.5 chart's httpGet probes can replace the interimexec: kill -0 1probes.src/flydocs/worker_health.py: Starlette app over pyfly's actuator routes (build_actuator_routes+ the new publicpyfly.actuator.install_health_indicatorsfrom pyfly 26.6.98), served by uvicorn as a sibling asyncio task. Binds0.0.0.0, access log off, lifespan off, no FastAPI.FLYDOCS_WORKER_HEALTH_PORT, falling back toFLYDOCS_PORT(default 8400);0= disabled, gated before uvicorn ever sees it (uvicorn treats port 0 as "ephemeral").task dev:workersets0so the documented two-terminal dev flow keeps working.asyncio.wait(FIRST_COMPLETED)set. On first exit the stop flags flip, siblings drain through their own exit paths (grace 5s), stragglers are cancelled and awaited, then pyfly shuts down. Bind failure = uvicornSystemExitconverted toRuntimeErrorso it flows the normal dead-task path and kills the process.SIG_DFLrestored and kill the process before cleanup, verified against uvicorn 0.46); the CLI owns SIGTERM and stops worker + reaper + health server gracefully. SIGINT keeps KeyboardInterrupt for Ctrl-C.health+infoonly;/actuator/loggersand/actuator/metrics404 unless opted in viapyfly.management.endpoints.web.exposure.include.database_health/eda_healthparticipate in both probes (parity with the API process), documented indocs/deployment.mdwith the Spring-strict alternative.worker/bbox-worker,env_templateentry, deployment docs, CHANGELOG 26.6.3, version bump everywhere the release ritual touches (incl. webhook User-Agent and version endpoint examples),pyfly[...,web]>=26.6.98so starlette/uvicorn are declared deps.Testing
tests/unit/test_worker_health.py: probe routes/status codes, readiness-only DOWN keeps liveness 200, exposure 404s + opt-in, container scan, port resolution, server factory config, real uvicorn round-trip (serve → 200 → graceful stop), bind-failure → RuntimeError, SIGTERM handler, drain-then-cancel semantics. Hardened against ambientFLYDOCS_WORKER_HEALTH_PORT/.env.ruff check+ruff format --checkclean.docker compose configvalid.Deploy notes (platform)
Strict order per the work order: image first (this release → tag
v26.6.3), then bumpcore-flydocschartVersion 0.1.4 → 0.1.5. With the current liveness semantics a sustained DB outage flips liveness and restarts worker pods (same as the API today) — see "Worker health" in docs/deployment.md.