Worker health server: /actuator/health over HTTP in worker + bbox-worker (26.6.3)

CHANGELOG.md

@@ -6,6 +6,45 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project uses **CalVer `YY.M.PP`** (PEP 440 may normalise patch numbers
 for the Python wheel — e.g. `26.06.00` → `26.6.0`).
 
+## [26.6.3] - 2026-06-12
+
+### Added
+
+- **Worker health server.** `flydocs worker` and `flydocs bbox-worker` now
+  serve `GET /actuator/health`, `/actuator/health/liveness`, and
+  `/actuator/health/readiness` over HTTP (Starlette + uvicorn, assembled
+  from pyfly's actuator in `src/flydocs/worker_health.py`), so Kubernetes
+  probes the worker pods with httpGet instead of `exec` shims. The server
+  binds `0.0.0.0`, runs as a sibling asyncio task of the worker and reaper
+  (any task dying — including a failed bind — takes the whole process down
+  for a clean pod restart), keeps its access log off, and honours pyfly's
+  secure-by-default endpoint exposure (`/actuator/loggers`,
+  `/actuator/metrics` → 404 unless opted in). Indicator discovery uses
+  pyfly ≥ 26.6.98's public `pyfly.actuator.install_health_indicators`;
+  `database_health` and `eda_health` participate in both probes, matching
+  the API process. See the "Worker health" section in `docs/deployment.md`.
+- New setting `worker_health_port` (`FLYDOCS_WORKER_HEALTH_PORT`): unset
+  reuses `FLYDOCS_PORT`; `0` disables the worker health server.
+- `docker compose` healthchecks for the `worker` and `bbox-worker`
+  services against `/actuator/health/readiness`.
+- The worker modes now shut down gracefully on SIGTERM: worker, reaper,
+  and health server stop, and the pyfly shutdown runs before the process
+  exits.
+
+### Changed
+
+- `pyfly` dependency floor raised to 26.6.98 and the `web` extra added, so
+  `starlette` and `uvicorn` are declared (previously they only arrived
+  transitively).
+
+### Documentation
+
+- `env_template`: realigned `FLYDOCS_JOBS_TOPIC` and
+  `FLYDOCS_ASYNC_TIMEOUT_S` with the defaults in `config.py`.
+- `docs/deployment.md`: metrics endpoints require exposure opt-in via
+  `pyfly.management.endpoints.web.exposure.include`; the secure default
+  exposes only `health,info`.
+
 ## [26.6.2] - 2026-05-31
 
 ### Changed

Taskfile.yml

@@ -105,7 +105,14 @@ tasks:
       - uv run uvicorn flydocs.main:app --reload --host 0.0.0.0 --port {{.PORT}}
 
   dev:worker:
-    desc: Run the EDA worker that consumes the job queue.
+    desc: |
+      Run the EDA worker that consumes the job queue. The worker health
+      server is disabled here because it would otherwise reuse FLYDOCS_PORT
+      and collide with the API started by ``task dev:serve`` on the same
+      host; export FLYDOCS_WORKER_HEALTH_PORT (host env wins over this
+      default) to probe it locally.
+    env:
+      FLYDOCS_WORKER_HEALTH_PORT: "0"
     cmds:
       - uv run flydocs worker
 

docker-compose.yml

@@ -125,12 +125,18 @@ services:
         condition: service_healthy
       gotenberg:
         condition: service_healthy
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "http://localhost:8400/actuator/health/readiness"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
 
-  # Second-stage EDA consumer: subscribes to ``flydocs.bbox.refine``
-  # events that ``worker`` publishes when an extraction finishes with
+  # Second-stage EDA consumer: subscribes to the
+  # ``extraction.post_processing.requested`` events that ``worker``
+  # publishes when an extraction succeeds with
   # ``options.stages.bbox_refine == true``. Grounds bboxes against the
   # PDF text layer / configured OCR engine, persists the refined result,
-  # transitions the job ``PARTIAL_SUCCEEDED -> SUCCEEDED``, and fires the
+  # updates ``post_processing.bbox_refinement.status``, and fires the
   # final webhook. Same image as the API + worker; only the entrypoint
   # differs.
   bbox-worker:
@@ -162,6 +168,11 @@ services:
         condition: service_healthy
       gotenberg:
         condition: service_healthy
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "http://localhost:8400/actuator/health/readiness"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
 
 volumes:
   postgres_data:

docs/deployment.md

@@ -246,6 +246,38 @@ probe by registering another `pyfly.actuator.health.HealthIndicator`
 bean (from `fireflyframework-pyfly`) in `core/configuration.py` — the
 lifespan rescan picks it up automatically.
 
+### Worker health
+
+`flydocs worker` and `flydocs bbox-worker` serve the same
+`/actuator/health/*` endpoints from a lightweight Starlette + uvicorn
+server (`src/flydocs/worker_health.py`) that runs as one more asyncio
+task inside the worker process, so all three workloads take the probes
+above. Specifics:
+
+- **Port.** `FLYDOCS_WORKER_HEALTH_PORT` when set, otherwise
+  `FLYDOCS_PORT` (default 8400). `0` disables the server — useful when
+  `serve` and `worker` share a host in dev. The server binds `0.0.0.0`
+  because the kubelet probes the pod IP, never loopback.
+- **Probe semantics.** Indicators discovered from the DI container
+  (`database_health`, `eda_health`) carry no probe group, so — exactly
+  like the API process — they participate in **both** liveness and
+  readiness: a sustained DB or broker outage flips liveness to 503 and
+  the kubelet restarts the worker pod until the dependency returns.
+  For Spring-strict liveness (process-alive only), register the
+  indicators with `groups={ProbeGroup.READINESS}`.
+- **Exposure.** Only `health` and `info` are mounted by default;
+  `/actuator/loggers`, `/actuator/metrics`, etc. return 404 unless
+  opted in via `pyfly.management.endpoints.web.exposure.include` —
+  note that key applies to the API **and** the worker health port,
+  since both build their routes from the same pyfly config.
+- **Lifecycle.** The health server joins the worker's
+  `asyncio.wait(FIRST_COMPLETED)` task set: if any task dies —
+  including a failed bind — the whole process exits and the pod
+  restarts cleanly. SIGTERM stops worker, reaper, and health server
+  gracefully and runs the pyfly shutdown before the process exits.
+- **No access log.** Probes fire every few seconds; the health
+  server's access log is disabled.
+
 > **W3C trace context** is propagated by `fireflyframework-pyfly`'s
 > default `CorrelationFilter`: every response echoes back `X-Correlation-Id`,
 > `X-Request-Id`, `traceparent`, `tracestate`, and `X-Tenant-Id` when
@@ -257,7 +289,7 @@ lifespan rescan picks it up automatically.
 
 | Telemetry      | Surface                                                                                              |
 | -------------- | ---------------------------------------------------------------------------------------------------- |
-| **Metrics**    | Prometheus at `GET /actuator/metrics` — CQRS handler latency, HTTP histograms, runtime metrics.       |
+| **Metrics**    | `GET /actuator/metrics` + Prometheus scrape at `GET /actuator/prometheus` — CQRS handler latency, HTTP histograms, runtime metrics. 404 until exposed via `pyfly.management.endpoints.web.exposure.include` (the secure default exposes only `health,info`). |
 | **Traces**     | OpenTelemetry. Configure via standard env vars (`OTEL_EXPORTER_OTLP_ENDPOINT`, …). One span per pipeline node. |
 | **Logs**       | structlog JSON. Every line carries `request_id`; correlation across API + worker is just a grep.       |
 | **Health**     | `/actuator/health`, `/actuator/health/liveness`, `/actuator/health/readiness`, `/actuator/info`.        |

env_template

@@ -9,6 +9,11 @@
 # ----------------------------------------------------------------------------
 FLYDOCS_LOG_LEVEL=INFO
 FLYDOCS_PORT=8400
+# Port for the HTTP health server the worker modes (``flydocs worker`` /
+# ``flydocs bbox-worker``) run for Kubernetes httpGet probes against
+# /actuator/health/{liveness,readiness}. Unset reuses FLYDOCS_PORT;
+# 0 disables it (dev setups running serve + worker on the same host).
+#FLYDOCS_WORKER_HEALTH_PORT=8400
 
 # ----------------------------------------------------------------------------
 # Persistence
@@ -32,7 +37,7 @@ FLYDOCS_EDA_ADAPTER=postgres
 FLYDOCS_REDIS_URL=redis://localhost:6379/0
 
 # Topic / stream name for job-submitted events.
-FLYDOCS_JOBS_TOPIC=flydocs.jobs
+FLYDOCS_JOBS_TOPIC=flydocs.extractions
 
 # ----------------------------------------------------------------------------
 # Multimodal extraction
@@ -54,7 +59,7 @@ FLYDOCS_MAX_BYTES=33554432
 # Per-call timeouts (seconds). Sync requests get the shorter one; async jobs
 # retry up to ``JOB_MAX_ATTEMPTS`` times on timeout.
 FLYDOCS_SYNC_TIMEOUT_S=60
-FLYDOCS_ASYNC_TIMEOUT_S=300
+FLYDOCS_ASYNC_TIMEOUT_S=1200
 FLYDOCS_JOB_MAX_ATTEMPTS=3
 
 # ----------------------------------------------------------------------------

pyproject.toml

@@ -2,7 +2,7 @@
 name = "flydocs"
 # CalVer YY.MM.PP -- bumped per release. Note that PEP 440 normalises
 # ``26.05.01`` -> ``26.5.1`` in the built wheel filename.
-version = "26.6.2"
+version = "26.6.3"
 description = "Pure-multimodal Intelligent Document Processing service: structured fields + bounding boxes, validation, authenticity checks, LLM judge, and a business-rule engine. Sync + queue-backed async APIs over fireflyframework-pyfly and -agentic. Part of Firefly OperationOS, platform-agnostic by design."
 readme = "README.md"
 requires-python = ">=3.13"
@@ -16,8 +16,10 @@ authors = [
 dependencies = [
     # Firefly Framework runtime (DI, CQRS, EDA, web, observability, resilience,
     # actuator, data-relational, security). Pulled in with the extras we use
-    # so a fresh ``uv sync`` is enough to boot the full stack.
-    "pyfly[fastapi,observability,security,data-relational,postgresql,eda,redis,client,scheduling,cli]>=26.5.4",
+    # so a fresh ``uv sync`` is enough to boot the full stack. The ``web``
+    # extra declares starlette + uvicorn, which the worker health server
+    # imports directly; the floor carries ``pyfly.actuator.install_health_indicators``.
+    "pyfly[fastapi,web,observability,security,data-relational,postgresql,eda,redis,client,scheduling,cli]>=26.6.98",
 
     # GenAI metaframework -- FireflyAgent with multimodal content (BinaryContent/ImageUrl)
     # over pydantic-ai. Pulls in the OpenAI / Anthropic / Bedrock providers via pydantic-ai-slim.

src/flydocs/__init__.py

@@ -24,4 +24,4 @@
 `PromptRegistry`).
 """
 
-__version__ = "26.6.2"
+__version__ = "26.6.3"

src/flydocs/app.py

@@ -29,7 +29,7 @@
 @enable_core_stack
 @pyfly_application(
     name="flydocs",
-    version="26.6.2",
+    version="26.6.3",
     description=(
         "flydocs -- pure-multimodal document extraction with bounding "
         "boxes. Part of Firefly OperationOS, platform-agnostic."