Skip to content

Worker health server: /actuator/health over HTTP in worker + bbox-worker (26.6.3) - #29

Merged
ancongui merged 1 commit into
mainfrom
feat/worker-health-server
Jun 12, 2026
Merged

Worker health server: /actuator/health over HTTP in worker + bbox-worker (26.6.3)#29
ancongui merged 1 commit into
mainfrom
feat/worker-health-server

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Summary

Implements the platform work order (2026-06-12): flydocs worker and flydocs bbox-worker must serve /actuator/health/* over HTTP so the core-flydocs 0.1.5 chart's httpGet probes can replace the interim exec: kill -0 1 probes.

  • New src/flydocs/worker_health.py: Starlette app over pyfly's actuator routes (build_actuator_routes + the new public pyfly.actuator.install_health_indicators from pyfly 26.6.98), served by uvicorn as a sibling asyncio task. Binds 0.0.0.0, access log off, lifespan off, no FastAPI.
  • Port: FLYDOCS_WORKER_HEALTH_PORT, falling back to FLYDOCS_PORT (default 8400); 0 = disabled, gated before uvicorn ever sees it (uvicorn treats port 0 as "ephemeral"). task dev:worker sets 0 so the documented two-terminal dev flow keeps working.
  • Lifecycle: health task joins the existing asyncio.wait(FIRST_COMPLETED) set. On first exit the stop flags flip, siblings drain through their own exit paths (grace 5s), stragglers are cancelled and awaited, then pyfly shuts down. Bind failure = uvicorn SystemExit converted to RuntimeError so it flows the normal dead-task path and kills the process.
  • Signals: uvicorn's signal capture is disabled (it would re-raise SIGTERM with SIG_DFL restored and kill the process before cleanup, verified against uvicorn 0.46); the CLI owns SIGTERM and stops worker + reaper + health server gracefully. SIGINT keeps KeyboardInterrupt for Ctrl-C.
  • Exposure: secure-by-default — health + info only; /actuator/loggers and /actuator/metrics 404 unless opted in via pyfly.management.endpoints.web.exposure.include.
  • Probe semantics: database_health / eda_health participate in both probes (parity with the API process), documented in docs/deployment.md with the Spring-strict alternative.
  • compose healthchecks for worker/bbox-worker, env_template entry, deployment docs, CHANGELOG 26.6.3, version bump everywhere the release ritual touches (incl. webhook User-Agent and version endpoint examples), pyfly[...,web]>=26.6.98 so starlette/uvicorn are declared deps.

Testing

  • 18 new tests in tests/unit/test_worker_health.py: probe routes/status codes, readiness-only DOWN keeps liveness 200, exposure 404s + opt-in, container scan, port resolution, server factory config, real uvicorn round-trip (serve → 200 → graceful stop), bind-failure → RuntimeError, SIGTERM handler, drain-then-cancel semantics. Hardened against ambient FLYDOCS_WORKER_HEALTH_PORT/.env.
  • Full unit suite: 273 passed. ruff check + ruff format --check clean. docker compose config valid.

Deploy notes (platform)

Strict order per the work order: image first (this release → tag v26.6.3), then bump core-flydocs chartVersion 0.1.4 → 0.1.5. With the current liveness semantics a sustained DB outage flips liveness and restarts worker pods (same as the API today) — see "Worker health" in docs/deployment.md.

flydocs worker and flydocs bbox-worker now run a Starlette + uvicorn
health server (assembled from pyfly's actuator) as a sibling asyncio
task, so Kubernetes probes the worker pods with httpGet against
/actuator/health/{liveness,readiness} instead of exec shims. Binds
0.0.0.0; port = FLYDOCS_WORKER_HEALTH_PORT, falling back to
FLYDOCS_PORT; 0 disables it (and task dev:worker sets 0 so the
documented serve+worker same-host flow keeps working).

Lifecycle: the health task joins the worker/reaper
asyncio.wait(FIRST_COMPLETED) set; when any task finishes, the stop
flags flip, siblings drain through their own exit paths within a grace
period, and stragglers are cancelled and awaited before the pyfly
shutdown — so a bind failure (uvicorn's SystemExit, converted to
RuntimeError by serve_health) or a dead worker takes the whole process
down cleanly. SIGTERM now stops worker, reaper, and health server
gracefully (uvicorn's own signal capture is disabled; the CLI owns
signals). Access log off; only health + info exposed by default
(loggers/metrics 404 per pyfly's secure-by-default web exposure).

Indicator discovery uses pyfly >= 26.6.98's public
pyfly.actuator.install_health_indicators; database_health and
eda_health participate in both probes, matching the API process
(documented in docs/deployment.md). The pyfly dependency adds the web
extra so starlette and uvicorn are declared instead of transitive.

Also: compose healthchecks for worker/bbox-worker, env_template entry
(plus realigning two stale defaults), deployment docs, and the webhook
User-Agent / version endpoint examples bumped per the release ritual.
@ancongui
ancongui merged commit d3582a3 into main Jun 12, 2026
8 checks passed
@ancongui
ancongui deleted the feat/worker-health-server branch June 12, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant