Skip to content

Commit 1963c2e

Browse files
tillotillo
committed
monitoring + bootstrap: track user-facing availability, expose Garage via ingress
monitoring-rules: replace kube_deployment_status_replicas_unavailable > 0 with (spec_replicas - status_replicas_available) > 0 across all *Unavailable alerts (mail, identity, gitlab, files, notes, websites, dns, home, ntp, monitor-the-monitoring). The old metric counts surge pods from a stuck rolling update against the deployment, firing major_outage on the statuspage even when desired replicas are all ready. The new expression compares ready replicas against spec — fires only when actual user-facing capacity is degraded. bootstrap: new Ingress garage-bootstrap.mdapi.ch -> garage:3900 (no ModSec, unlimited body size, 600s timeouts). GitLab container registry will be repointed to this endpoint; presigned blob URLs sign against a publicly resolvable hostname so containerd's host-netns DNS can resolve them. Previously presigned URLs referenced garage.bootstrap.svc.cluster.local and every fresh layer pull failed with 'no such host' on the node.
1 parent 66add7b commit 1963c2e

11 files changed

Lines changed: 166 additions & 47 deletions

File tree

‎bootstrap/garage-ing.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## Public Ingress for the bootstrap-namespace Garage S3 endpoint.
2+
##
3+
## Reason for existence: GitLab's container registry (registry.mdapi.ch)
4+
## stores layer blobs in this Garage instance and 307-redirects pulls to
5+
## presigned S3 URLs. Previously the registry's `regionendpoint` was the
6+
## in-cluster `http://garage.bootstrap.svc.cluster.local:3900`, so the
7+
## presigned URL's hostname only resolves via CoreDNS — but containerd's
8+
## image pull runs in the node's network namespace using the host's
9+
## resolv.conf (BPI-R4 at 192.168.1.1), which doesn't know cluster-internal
10+
## SVC names. Every fresh image pull (incl. all `:latest` repulls) failed
11+
## with `no such host` until the cached layer was reused.
12+
##
13+
## Switching the registry's `regionendpoint` to `https://garage-bootstrap.mdapi.ch`
14+
## (this Ingress) makes the presigned URL hostname publicly resolvable; nodes
15+
## reach Garage through the standard ingress-nginx → garage:3900 path.
16+
##
17+
## Notes:
18+
## - ModSec is DISABLED: S3 PUT bodies are arbitrary binary (layer blobs);
19+
## CRS rules would block every push.
20+
## - proxy-body-size is unlimited — image layers can be multi-GB.
21+
## - proxy timeouts are 600s for slow node-side pulls.
22+
## - SigV4 signs the Host header (`garage-bootstrap.mdapi.ch`); nginx
23+
## preserves the client Host upstream so Garage validates the signature
24+
## against the same value.
25+
## - Garage uses path-style addressing only (no `root_domain` under
26+
## `[s3_api]` in garage-cm.yml), so it doesn't care which Host header
27+
## reaches it.
28+
## - The bucket has no public-read policy; all access requires SigV4 over
29+
## valid credentials, so exposing this endpoint publicly is safe.
30+
31+
apiVersion: networking.k8s.io/v1
32+
kind: Ingress
33+
metadata:
34+
name: garage
35+
namespace: bootstrap
36+
annotations:
37+
cert-manager.io/issuer: letsencrypt-prod
38+
nginx.ingress.kubernetes.io/enable-modsecurity: "false"
39+
nginx.ingress.kubernetes.io/proxy-body-size: "0"
40+
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
41+
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
42+
spec:
43+
ingressClassName: nginx
44+
rules:
45+
- host: garage-bootstrap.mdapi.ch
46+
http:
47+
paths:
48+
- path: /
49+
pathType: Prefix
50+
backend:
51+
service:
52+
name: garage
53+
port:
54+
number: 3900
55+
tls:
56+
- hosts:
57+
- garage-bootstrap.mdapi.ch
58+
secretName: garage-bootstrap-cert

‎monitoring-rules/01-mail.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,9 @@ spec:
1010
- alert: MailDeploymentUnavailable
1111
expr: |
1212
max by (namespace, deployment) (
13-
kube_deployment_status_replicas_unavailable{namespace="mail"} > 0
13+
( kube_deployment_spec_replicas{namespace="mail"}
14+
- kube_deployment_status_replicas_available{namespace="mail"}
15+
) > 0
1416
)
1517
for: 5m
1618
labels:

‎monitoring-rules/02-identity.yml‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,13 @@ spec:
1010
- alert: IdentityKeycloakUnavailable
1111
expr: |
1212
max by (namespace, deployment) (
13-
kube_deployment_status_replicas_unavailable{
14-
namespace=~"keycloak|envuassu",
15-
deployment="keycloak"
16-
} > 0
13+
( kube_deployment_spec_replicas{
14+
namespace=~"keycloak|envuassu", deployment="keycloak"
15+
}
16+
- kube_deployment_status_replicas_available{
17+
namespace=~"keycloak|envuassu", deployment="keycloak"
18+
}
19+
) > 0
1720
)
1821
for: 3m
1922
labels:
@@ -47,9 +50,13 @@ spec:
4750
## users but new logins and group lookups break.
4851
expr: |
4952
max by (namespace, deployment) (
50-
kube_deployment_status_replicas_unavailable{
51-
namespace="openldap", deployment=~"openldap|lam"
52-
} > 0
53+
( kube_deployment_spec_replicas{
54+
namespace="openldap", deployment=~"openldap|lam"
55+
}
56+
- kube_deployment_status_replicas_available{
57+
namespace="openldap", deployment=~"openldap|lam"
58+
}
59+
) > 0
5360
)
5461
for: 5m
5562
labels:

‎monitoring-rules/03-gitlab.yml‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,13 @@ spec:
99
rules:
1010
- alert: GitLabWebserviceUnavailable
1111
expr: |
12-
kube_deployment_status_replicas_unavailable{
13-
namespace="bootstrap", deployment=~"gitlab-webservice-.*"
14-
} > 0
12+
( kube_deployment_spec_replicas{
13+
namespace="bootstrap", deployment=~"gitlab-webservice-.*"
14+
}
15+
- kube_deployment_status_replicas_available{
16+
namespace="bootstrap", deployment=~"gitlab-webservice-.*"
17+
}
18+
) > 0
1519
for: 5m
1620
labels:
1721
severity: critical
@@ -39,9 +43,13 @@ spec:
3943

4044
- alert: GitLabSidekiqDown
4145
expr: |
42-
kube_deployment_status_replicas_unavailable{
43-
namespace="bootstrap", deployment=~"gitlab-sidekiq-.*"
44-
} > 0
46+
( kube_deployment_spec_replicas{
47+
namespace="bootstrap", deployment=~"gitlab-sidekiq-.*"
48+
}
49+
- kube_deployment_status_replicas_available{
50+
namespace="bootstrap", deployment=~"gitlab-sidekiq-.*"
51+
}
52+
) > 0
4553
for: 10m
4654
labels:
4755
severity: warning

‎monitoring-rules/04-files.yml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,15 @@ spec:
99
rules:
1010
- alert: FilesDeploymentUnavailable
1111
expr: |
12-
kube_deployment_status_replicas_unavailable{
13-
namespace=~"owncloud|spider3|tv",
14-
deployment=~"owncloud|spider3|sftpgo"
15-
} > 0
12+
( kube_deployment_spec_replicas{
13+
namespace=~"owncloud|spider3|tv",
14+
deployment=~"owncloud|spider3|sftpgo"
15+
}
16+
- kube_deployment_status_replicas_available{
17+
namespace=~"owncloud|spider3|tv",
18+
deployment=~"owncloud|spider3|sftpgo"
19+
}
20+
) > 0
1621
for: 5m
1722
labels:
1823
severity: critical

‎monitoring-rules/05-notes.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@ spec:
99
rules:
1010
- alert: JoplinUnavailable
1111
expr: |
12-
kube_deployment_status_replicas_unavailable{namespace="joplin", deployment="joplin"} > 0
12+
( kube_deployment_spec_replicas{namespace="joplin", deployment="joplin"}
13+
- kube_deployment_status_replicas_available{namespace="joplin", deployment="joplin"}
14+
) > 0
1315
for: 5m
1416
labels:
1517
severity: critical

‎monitoring-rules/06-websites.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,13 @@ spec:
1111
# Aggregate by namespace so a single alert per affected site.
1212
- alert: WebsiteUnavailable
1313
expr: |
14-
kube_deployment_status_replicas_unavailable{
15-
namespace=~"spider3|coiffuredreams|dellambrogio|ivodellambrogio|mdapiorg|textopolis|tillo|flame"
16-
} > 0
14+
( kube_deployment_spec_replicas{
15+
namespace=~"spider3|coiffuredreams|dellambrogio|ivodellambrogio|mdapiorg|textopolis|tillo|flame"
16+
}
17+
- kube_deployment_status_replicas_available{
18+
namespace=~"spider3|coiffuredreams|dellambrogio|ivodellambrogio|mdapiorg|textopolis|tillo|flame"
19+
}
20+
) > 0
1721
for: 5m
1822
labels:
1923
severity: critical

‎monitoring-rules/07-dns.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,13 @@ spec:
99
rules:
1010
- alert: TechnitiumUnavailable
1111
expr: |
12-
kube_deployment_status_replicas_unavailable{
13-
namespace="technitium", deployment="technitium"
14-
} > 0
12+
( kube_deployment_spec_replicas{
13+
namespace="technitium", deployment="technitium"
14+
}
15+
- kube_deployment_status_replicas_available{
16+
namespace="technitium", deployment="technitium"
17+
}
18+
) > 0
1519
for: 3m
1620
labels:
1721
severity: critical

‎monitoring-rules/08-home.yml‎

Lines changed: 35 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,13 @@ spec:
1717
- alert: HomeAssistantUnavailable
1818
expr: |
1919
max by (namespace, deployment) (
20-
kube_deployment_status_replicas_unavailable{
21-
namespace="home-assistant", deployment="home-assistant"
22-
} > 0
20+
( kube_deployment_spec_replicas{
21+
namespace="home-assistant", deployment="home-assistant"
22+
}
23+
- kube_deployment_status_replicas_available{
24+
namespace="home-assistant", deployment="home-assistant"
25+
}
26+
) > 0
2327
)
2428
for: 5m
2529
labels:
@@ -51,9 +55,13 @@ spec:
5155
- alert: MqttBrokerUnavailable
5256
expr: |
5357
max by (namespace) (
54-
kube_deployment_status_replicas_unavailable{
55-
namespace="mqtt", deployment="eclipse-mosquitto"
56-
} > 0
58+
( kube_deployment_spec_replicas{
59+
namespace="mqtt", deployment="eclipse-mosquitto"
60+
}
61+
- kube_deployment_status_replicas_available{
62+
namespace="mqtt", deployment="eclipse-mosquitto"
63+
}
64+
) > 0
5765
)
5866
for: 3m
5967
labels:
@@ -69,9 +77,13 @@ spec:
6977
- alert: FrigateUnavailable
7078
expr: |
7179
max by (namespace) (
72-
kube_deployment_status_replicas_unavailable{
73-
namespace="frigate", deployment="frigate"
74-
} > 0
80+
( kube_deployment_spec_replicas{
81+
namespace="frigate", deployment="frigate"
82+
}
83+
- kube_deployment_status_replicas_available{
84+
namespace="frigate", deployment="frigate"
85+
}
86+
) > 0
7587
)
7688
for: 5m
7789
labels:
@@ -85,9 +97,13 @@ spec:
8597
- alert: AppDaemonUnavailable
8698
expr: |
8799
max by (namespace) (
88-
kube_deployment_status_replicas_unavailable{
89-
namespace="appdaemon", deployment="appdaemon"
90-
} > 0
100+
( kube_deployment_spec_replicas{
101+
namespace="appdaemon", deployment="appdaemon"
102+
}
103+
- kube_deployment_status_replicas_available{
104+
namespace="appdaemon", deployment="appdaemon"
105+
}
106+
) > 0
91107
)
92108
for: 10m
93109
labels:
@@ -101,9 +117,13 @@ spec:
101117
- alert: EsphomeUnavailable
102118
expr: |
103119
max by (namespace) (
104-
kube_deployment_status_replicas_unavailable{
105-
namespace="esphome", deployment="esphome"
106-
} > 0
120+
( kube_deployment_spec_replicas{
121+
namespace="esphome", deployment="esphome"
122+
}
123+
- kube_deployment_status_replicas_available{
124+
namespace="esphome", deployment="esphome"
125+
}
126+
) > 0
107127
)
108128
for: 15m
109129
labels:

‎monitoring-rules/09-ntp.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,13 @@ spec:
2424
- alert: NtpChronyUnavailable
2525
expr: |
2626
max by (namespace) (
27-
kube_deployment_status_replicas_unavailable{
28-
namespace="ntppool", deployment="chrony"
29-
} > 0
27+
( kube_deployment_spec_replicas{
28+
namespace="ntppool", deployment="chrony"
29+
}
30+
- kube_deployment_status_replicas_available{
31+
namespace="ntppool", deployment="chrony"
32+
}
33+
) > 0
3034
)
3135
for: 5m
3236
labels:

0 commit comments

Comments
 (0)