Skip to content

Commit 613a7d6

Browse files
committed
feat: add health endpoints for monitoring
Add /health/live, /health/ready and /health/detail, with checks for the database, the RabbitMQ messenger transport and detection result freshness. Liveness touches no dependencies. Readiness reports the aggregated status only, never which dependency failed. The detailed endpoint discloses internals and is protected by the ITKBasicAuth Traefik middleware on its own router. ^/health is excluded from the Symfony firewalls on purpose: both user providers are Doctrine entity providers, so an application-level firewall could not authenticate while the database is unavailable. Results are cached in a dedicated filesystem-backed cache.health pool so that polling cannot amplify into load on the dependencies being checked.
1 parent f6ea60a commit 613a7d6

18 files changed

Lines changed: 689 additions & 2 deletions

.env

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,12 @@ APP_KEEP_RESULTS=5
6868
APP_ECONOMICS_URI=https://economics.itkdev.dk
6969
APP_ECONOMICS_API_KEY=changeme
7070
###< economics ###
71+
72+
###> health ###
73+
# Seconds to cache the health check results, so that monitoring polling
74+
# /health/ready cannot amplify into load on the database and the broker.
75+
HEALTH_CACHE_TTL=15
76+
# Seconds since the last detection result before ingest is reported degraded.
77+
# The harvester currently reports several hundred times an hour.
78+
HEALTH_INGEST_MAX_AGE=1800
79+
###< health ###

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
- [#91](https://github.com/itk-dev/devops_itksites/pull/91) Health endpoints
11+
- Add `/health/live`, `/health/ready` and `/health/detail` endpoints
12+
- Add health checks for database, RabbitMQ transport and detection result freshness
13+
- Cache check results in a dedicated `cache.health` pool
14+
- Exclude `^/health` from the firewalls and protect `/health/detail` with `ITKBasicAuth`
1015
- [#90](https://github.com/itk-dev/devops_itksites/pull/90)
1116
- Fixed user API key migration failing on databases with more than one user
1217
- Generated an API key for existing users, as users created since already get

README.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,43 @@ Use the API key to make an authenticated request, e.g.
6363
curl --header 'accept: application/json' --header 'authorization: Apikey <the API key>' https://itksites.local.itkdev.dk/api/sites
6464
```
6565

66+
## Health checks
67+
68+
Three endpoints report on the application, in increasing order of detail:
69+
70+
| Endpoint | Access | Checks |
71+
| --- | --- | --- |
72+
| `/health/live` | Public | Nothing – only that the app responds |
73+
| `/health/ready` | Public | All checks, aggregated status only |
74+
| `/health/detail` | `ITKBasicAuth` in Traefik | Per-check results and timings |
75+
76+
`/health/ready` answers `200` when everything is well and `503` when it is not.
77+
It deliberately does not say *what* is wrong – point monitoring at this one and
78+
read `/health/detail` when it goes red:
79+
80+
``` shell
81+
curl --silent https://itksites.local.itkdev.dk/health/detail | jq
82+
```
83+
84+
The checks cover the database, the RabbitMQ messenger transport and the
85+
freshness of the most recent detection result. The last one catches an ingest
86+
pipeline that has stopped while the application itself is still serving
87+
requests.
88+
89+
`HEALTH_INGEST_MAX_AGE` sets how old the most recent detection result may be
90+
before ingest is reported as degraded.
91+
92+
Results are cached for `HEALTH_CACHE_TTL` seconds so that polling does not turn
93+
into load on the dependencies. The cache is the dedicated, filesystem-backed
94+
`cache.health` pool in `config/packages/cache.yaml` – it has to keep working
95+
while the database and the broker are down, and the adapter can be swapped
96+
there without touching code.
97+
98+
`^/health` is excluded from the Symfony firewalls: both user providers are
99+
Doctrine entity providers, so an authenticated endpoint would fail to
100+
authenticate during a database outage and answer `500` rather than reporting
101+
that the database is down.
102+
66103
## Development
67104

68105
```sh

config/packages/cache.yaml

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,5 +15,15 @@ framework:
1515
#app: cache.adapter.apcu
1616

1717
# Namespaced pools use the above "app" backend by default
18-
#pools:
19-
#my.dedicated.cache: null
18+
pools:
19+
# Health check results.
20+
#
21+
# Filesystem-backed on purpose: this pool has to keep working while
22+
# the database and the message broker are down, which is exactly
23+
# when the health endpoints matter.
24+
#
25+
# It is a dedicated pool so the adapter can be swapped without
26+
# touching code: cache.adapter.apcu is faster and shared between
27+
# FPM workers, at the cost of being cleared on every FPM restart.
28+
cache.health:
29+
adapter: cache.adapter.filesystem

config/packages/security.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,14 @@ security:
2121
dev:
2222
pattern: ^/(_(profiler|wdt)|css|images|js)/
2323
security: false
24+
# The health endpoints must answer while the database is down, so they
25+
# cannot go through a firewall: both user providers above are Doctrine
26+
# entity providers and authentication would itself fail. /health/detail
27+
# is protected by the ITKBasicAuth middleware in Traefik instead.
28+
health:
29+
pattern: ^/health
30+
security: false
31+
2432
api:
2533
pattern: ^/api
2634
custom_authenticators:

config/services.yaml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,23 @@ services:
2727
App\Handler\DetectionResultHandlerInterface:
2828
tags: [app.handler.detection_result_handler]
2929

30+
App\Health\HealthCheckInterface:
31+
tags: [app.health.check]
32+
33+
App\Health\HealthChecker:
34+
arguments:
35+
$checks: !tagged_iterator app.health.check
36+
$cache: '@cache.health'
37+
$cacheTtl: '%env(int:HEALTH_CACHE_TTL)%'
38+
39+
App\Health\Check\RabbitMqHealthCheck:
40+
arguments:
41+
$transport: '@messenger.transport.async'
42+
43+
App\Health\Check\IngestFreshnessHealthCheck:
44+
arguments:
45+
$maxAgeSeconds: '%env(int:HEALTH_INGEST_MAX_AGE)%'
46+
3047
App\EventListener\RemovedRelationsListener:
3148
tags:
3249
- name: 'doctrine.event_listener'

docker-compose.server.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,8 @@ services:
5050
# Cron-metrics protection.
5151
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-metrics.rule=Host(`${COMPOSE_SERVER_DOMAIN:?}`) && PathPrefix(`/cron-metrics`) "
5252
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-metrics.middlewares=ITKMetricsAuth@file"
53+
# Detailed health check protection. /health/live and /health/ready stay
54+
# public; only /health/detail discloses internals.
55+
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-health.rule=Host(`${COMPOSE_SERVER_DOMAIN:?}`) && PathPrefix(`/health/detail`)"
56+
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-health.entrypoints=websecure"
57+
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-health.middlewares=ITKBasicAuth@file"

docker-compose.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,10 @@ services:
7575
# Cron-metrics protection.
7676
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-metrics.rule=Host(`${COMPOSE_DOMAIN:?}`) && PathPrefix(`/cron-metrics`) "
7777
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-metrics.middlewares=ITKMetricsAuth@file"
78+
# Detailed health check protection. /health/live and /health/ready stay
79+
# public; only /health/detail discloses internals.
80+
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-health.rule=Host(`${COMPOSE_DOMAIN:?}`) && PathPrefix(`/health/detail`)"
81+
- "traefik.http.routers.${COMPOSE_PROJECT_NAME:?}-health.middlewares=ITKBasicAuth@file"
7882

7983
mail:
8084
image: axllent/mailpit
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace App\Controller;
6+
7+
use App\Health\HealthChecker;
8+
use App\Health\HealthStatus;
9+
use Symfony\Component\HttpFoundation\JsonResponse;
10+
use Symfony\Component\HttpFoundation\Response;
11+
use Symfony\Component\Routing\Attribute\Route;
12+
13+
/**
14+
* Health endpoints, in three tiers.
15+
*
16+
* /health/live Public. No dependencies at all. Says only that PHP-FPM is up
17+
* and routing works. It must never touch the database: a
18+
* liveness probe that fails during a database outage makes an
19+
* orchestrator restart a container that is not the problem.
20+
*
21+
* /health/ready Public, but opaque. Runs every check and answers 200 or 503
22+
* with the aggregated status only. The status code is the
23+
* payload; which dependency failed is not disclosed. This is
24+
* the endpoint monitoring should watch.
25+
*
26+
* /health/detail Per-check results, timings, queue depth. Discloses internals
27+
* and MUST be protected at the edge — see the ITKBasicAuth
28+
* middleware on the nginx service in docker-compose.server.yml.
29+
*
30+
* Authentication deliberately happens in Traefik rather than in Symfony. Both
31+
* user providers in config/packages/security.yaml are Doctrine entity
32+
* providers, so an application-level firewall on these routes would fail to
33+
* authenticate during a database outage and answer 500 — precisely when the
34+
* endpoint needs to answer "the database is down". ^/health is therefore
35+
* excluded from the Symfony firewalls entirely.
36+
*/
37+
readonly class HealthController
38+
{
39+
public function __construct(
40+
private HealthChecker $healthChecker,
41+
) {
42+
}
43+
44+
#[Route('/health/live', name: 'app_health_live', methods: ['GET'])]
45+
public function live(): JsonResponse
46+
{
47+
return $this->respond(['status' => HealthStatus::Ok->value], true);
48+
}
49+
50+
#[Route('/health/ready', name: 'app_health_ready', methods: ['GET'])]
51+
public function ready(): JsonResponse
52+
{
53+
$healthy = $this->healthChecker->isHealthy($this->healthChecker->run());
54+
55+
return $this->respond(
56+
['status' => $healthy ? HealthStatus::Ok->value : HealthStatus::Degraded->value],
57+
$healthy
58+
);
59+
}
60+
61+
#[Route('/health/detail', name: 'app_health_detail', methods: ['GET'])]
62+
public function detail(): JsonResponse
63+
{
64+
$results = $this->healthChecker->run();
65+
$healthy = $this->healthChecker->isHealthy($results);
66+
67+
$checks = [];
68+
foreach ($results as $result) {
69+
$checks[$result->name] = array_filter([
70+
'status' => $result->status->value,
71+
'message' => $result->message,
72+
'details' => $result->details,
73+
], static fn (mixed $value): bool => null !== $value && [] !== $value);
74+
}
75+
76+
return $this->respond([
77+
'status' => $healthy ? HealthStatus::Ok->value : HealthStatus::Degraded->value,
78+
'checks' => $checks,
79+
], $healthy);
80+
}
81+
82+
/**
83+
* @param array<string, mixed> $payload
84+
*/
85+
private function respond(array $payload, bool $healthy): JsonResponse
86+
{
87+
$response = new JsonResponse(
88+
$payload,
89+
$healthy ? Response::HTTP_OK : Response::HTTP_SERVICE_UNAVAILABLE
90+
);
91+
92+
// Health responses are cached inside HealthChecker, never by the client
93+
// or an intermediary.
94+
$response->headers->set('Cache-Control', 'no-store, private');
95+
96+
return $response;
97+
}
98+
}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace App\Health\Check;
6+
7+
use App\Health\HealthCheckInterface;
8+
use App\Health\HealthCheckResult;
9+
use Doctrine\DBAL\Connection;
10+
use Psr\Log\LoggerInterface;
11+
12+
/**
13+
* Verifies that the database accepts connections and answers queries.
14+
*/
15+
readonly class DatabaseHealthCheck implements HealthCheckInterface
16+
{
17+
public function __construct(
18+
private Connection $connection,
19+
private LoggerInterface $logger,
20+
) {
21+
}
22+
23+
public function getName(): string
24+
{
25+
return 'database';
26+
}
27+
28+
public function check(): HealthCheckResult
29+
{
30+
$start = microtime(true);
31+
32+
try {
33+
$this->connection->executeQuery('SELECT 1');
34+
} catch (\Throwable $throwable) {
35+
$this->logger->error('Health check "database" failed: {message}', [
36+
'message' => $throwable->getMessage(),
37+
'exception' => $throwable,
38+
]);
39+
40+
// The caller gets no detail; the reason stays in the logs.
41+
return HealthCheckResult::degraded($this->getName(), 'Unable to query the database.');
42+
}
43+
44+
return HealthCheckResult::ok($this->getName(), [
45+
'response_time_ms' => round((microtime(true) - $start) * 1000, 1),
46+
]);
47+
}
48+
}

0 commit comments

Comments
 (0)