feat(api): report renderer health to Kubernetes - #797
Open
vaernion wants to merge 2 commits into
Open
Conversation
github-actions
Bot
added
documentation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enhances apps/api Kubernetes probe behavior by modeling renderer (Chromium/Puppeteer) progress across the 10 generation slots, adding dedicated startup/readiness/liveness health endpoints, and enforcing a bounded end-to-end generation deadline to avoid pods staying ready while rendering is effectively stalled.
Changes:
- Introduces a renderer health model (
RendererHealth) based on per-job progress stages and monotonic time, with tests validating stall/readiness/liveness transitions. - Adds cached startup warm-up rendering plus new
/health/startup,/health/readiness, and/health/livenessendpoints while preserving/health. - Bounds full generation (including retries/recycle) to 50s and configures Fastify handler timeout at 55s; wires new env vars and NAIS probe configuration.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| apps/api/server.ts | Wires scheduler health, registers renderer health metrics, and registers new health routes plus startup warm-up render. |
| apps/api/README.md | Documents new timing/env settings and the health/probe behavior. |
| apps/api/lib/rendererHealth.ts | Implements renderer health state machine based on active-job progress and monotonic time. |
| apps/api/lib/rendererHealth.test.ts | Adds unit tests for health transitions, stall detection, and snapshot reporting. |
| apps/api/lib/otel.ts | Adds renderer health-related metrics and registers observable gauges/counters. |
| apps/api/lib/healthRoutes.ts | Adds startup/readiness/liveness endpoints and preserves /health. |
| apps/api/lib/healthRoutes.test.ts | Tests cached startup warm-up behavior and readiness vs liveness semantics. |
| apps/api/lib/handler.ts | Passes scheduler-provided progress reporter and timeout budget into generation pipeline. |
| apps/api/lib/generationScheduler.ts | Adds max job duration support, tracks pending jobs, and integrates renderer health + transition logging/metrics. |
| apps/api/lib/generationScheduler.test.ts | Extends environment parsing tests to cover new timing variables and job duration constraint. |
| apps/api/lib/generateDocument.ts | Plumbs progress + timeout budget into mdToPdf for both template and custom paths. |
| apps/api/lib/documentGenerationRoute.ts | Uses request.signal for cancellation propagation and removes manual abort-signal wiring. |
| apps/api/lib/core/md-to-pdf.ts | Plumbs progress reporting down into output generation. |
| apps/api/lib/core/index.ts | Adds progress + total timeout budget support to mdToPdf via useBrowserWithRetry. |
| apps/api/lib/core/helpers.ts | Introduces DOCUMENT_GENERATION_TIMEOUT_MS and HTTP_HANDLER_TIMEOUT_MS constants. |
| apps/api/lib/core/get-browser.ts | Enforces total generation deadline across acquire/render/retry/recycle and reports renderer progress stages. |
| apps/api/lib/core/get-browser.test.ts | Adds deadline/retry behavior tests with progress stage expectations. |
| apps/api/lib/core/generate-output.ts | Emits progress stage updates across page creation, content loading, and output production. |
| apps/api/lib/app.test.ts | Verifies Fastify handler timeout configuration. |
| apps/api/app.ts | Configures Fastify handlerTimeout from the new constant. |
| .nais/api.yaml | Configures new env vars and switches probes to startup/readiness/liveness endpoints with updated timings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Track generation-slot progress and expose separate startup, readiness, and liveness checks so stalled renderer capacity can be removed from service and eventually restarted. Bound generation below the ingress timeout, preserve browser recycle/retry within that budget, and publish health diagnostics through metrics and transition logs.\n\nCloses #779\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vaernion
force-pushed
the
agents/implement-issue-779
branch
from
August 7, 2026 11:49
ab3fad0 to
7a1a774
Compare
Brevgen2 Unit Test Results 5 files 29 suites 1s ⏱️ Results for commit 4f6a1b8. ♻️ This comment has been updated with latest results. |
Integration Test Results30 tests 30 ✅ 1m 13s ⏱️ Results for commit 4f6a1b8. ♻️ This comment has been updated with latest results. |
Address code review on the renderer health work: - measure the generation deadline from the moment a request enters the queue, so queue wait plus render stays below the Fastify handler timeout instead of stacking on top of it - validate that queue wait and the renderer stall timings fit inside the job duration, so a misconfiguration fails at startup instead of silently disabling the probes - lower the default stall threshold and recovery grace to 20s each so stalled and unhealthy are reachable within a job's budget - stop treating deadline exhaustion as browser instability, which made a busy pod recycle Chromium out from under queued requests - resolve the remaining budget before invoking the render function so a passed deadline cannot orphan the render promise - make getSnapshot pure and drive transitions from job lifecycle events plus a monitor interval, instead of from whoever happens to poll - report renderer health gauges from a single batched observation and emit 0/1 for every health state - surface per-job stall counts so partial slot degradation is visible - pass the scheduler clock through to renderer health - relax the readiness probe to failureThreshold 3 with a 5s timeout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The existing
/healthendpoint only confirmed that Fastify could answer HTTP. A pod could therefore remain ready while all document-generation slots were stalled and requests eventually received HAProxy 504 responses.Closes #779.
What changed
/healthgetSnapshot()pure and drive health transitions from job lifecycle events plus a 5-second monitor interval, rather than from whoever happens to pollRenderer progress uses six operational stages: browser acquisition, page creation, content loading, output production, browser recycling, and retrying.
Timings
Validation
pnpm test --filter api— 64 tests passedpnpm typecheck --filter apipnpm lint --filter apipnpm build --filter api