Etappe 54: „down" ist ein Status, keine Diagnose

Direkt aus dem Ausfall vom 16.–18.08. MatrixCtrl hat vier Komponenten als down
gemeldet — korrekt, binnen Sekunden, 37 Stunden lang — und nie gesagt, warum. Der
Scheduler hat den Grund die ganze Zeit in einem FailedScheduling-Event
veröffentlicht.

„Ist es kaputt" beantwortet kubectl get pods schon. Wofür ein Admin-Werkzeug da
ist, ist der Satz „es kann nicht platziert werden, weil es 8500m auf einem
6000m-Node anfordert". Zwischen diesen beiden Sätzen liegen die 37 Stunden.

Der Teil, der mehr ist als eine Event-Meldung auszugeben, ist die Rechnung — und
die hat zwei Fallen, die dieses Projekt auf die harte Tour kennengelernt hat. Der
Request eines Pods ist nicht die Summe seiner Container, sondern
max(Summe(Container), Maximum(Init-Container)): Synapses render-config und db-wait
hatten je 4000m geerbt, während der synapse-Container 1000m anforderte — Synapse
reservierte also 4000m, während es nur auf die Datenbank wartete. Und ein
resources-Block kann mehrere Container abdecken, weshalb aus 4000m bei postgres
8000m wurden.

Zwei bewusste Grenzen: „größer als jeder Node" wird von „Cluster gerade voll"
getrennt, weil nur Letzteres sich von selbst lösen kann. Und es wird kein Wert
vorgeschlagen — was eine Komponente anfordern sollte, hängt davon ab, was sonst
noch laufen soll.

Nicht enthalten ist die vorbeugende Hälfte (Warnung beim Speichern der Config).
Dafür müsste die Chart gerendert werden, weil die Container-Topologie eine
Eigenschaft der Chart ist und nicht der bearbeiteten YAML. Als P1-16b vermerkt
statt geraten.

Ende-zu-Ende gegen einen absichtlich nicht planbaren Pod geprüft, dessen 40000m in
einem Init-Container hinter einem 100m-Container steckten.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: BxnnyG

CHANGELOG.md

@@ -15,6 +15,32 @@ matching image, so a version identifies one exact pair
 
 ## [Unreleased]
 
+## [0.1.52] — 2026-08-30
+
+### Added
+
+- **A component that is `down` now says why, when the reason is the scheduler.** The
+  panel showed four components down for 37 hours during the outage of 2026-08-16…18
+  without once mentioning that postgres was asking for more CPU than the node had —
+  the scheduler had been saying so in a `FailedScheduling` event the whole time. The
+  dashboard now carries the scheduler's own words plus the arithmetic: the pod's
+  effective request against the largest node's allocatable.
+- Pods that ask for more than any single node can provide are called out separately.
+  A full cluster may place them later; these never will, and telling someone to wait
+  is worse than saying nothing.
+
+### Notes
+
+- The effective request is `max(sum(containers), max(initContainers))`, not the sum of
+  the containers. Synapse's init containers had inherited 4000m each while its own
+  container asked for 1000m, so it reserved 4000m while merely waiting for the
+  database — a naive sum would have reported 1000m and made the diagnosis look wrong.
+- No suggested value. What a component *should* request depends on what else is meant
+  to run on the node; naming the arithmetic is the panel's job, choosing the number is
+  the operator's.
+- Verified against a live unschedulable pod whose 40000m request sat in an init
+  container behind a 100m app container, not by unit test alone.
+
 ## [0.1.51] — 2026-08-17
 
 ### Fixed

deploy/helm/matrixctrl/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: matrixctrl
 description: Admin layer for self-hosted Matrix / Element Server Suite (ESS)
 type: application
-version: 0.1.51
-appVersion: "0.1.51"
+version: 0.1.52
+appVersion: "0.1.52"
 keywords:
   - matrix
   - element

docs/BACKLOG.md

@@ -1063,6 +1063,16 @@ implemented, OIDC state consumed atomically via `DELETE … RETURNING` (CSRF-saf
   that last case, where it costs one login.
 
 - **P1-16 · A component can be `down` for 37 hours without the panel saying why (S4).**
+  ✅ **First half done 2026-08-30 (E54, [DESIGN.md §4.54](DESIGN.md)).** A Pending
+  component now says why, with the scheduler's own words and the effective request
+  against the node's allocatable — `max(sum(containers), max(initContainers))`, which
+  is what hid 4000m of Synapse's reservation inside an init container. Verified against
+  a live unschedulable probe pod, not only by unit test.
+  *Still open as **P1-16b**: the preventive half* — warning at config-save time. It
+  needs the chart's container topology (which containers share a `resources` block,
+  which init containers inherit it), so it means rendering the chart with the Helm SDK
+  rather than reading the values file. Recorded rather than guessed.
+  *Original entry:*
   From the outage of 2026-08-16…18 ([DESIGN.md §4.53](DESIGN.md)): postgres was
   unschedulable after the node shrank from 32 cores to 6, and MatrixCtrl reported
   `down` — correctly and immediately — with no cause attached. The reason was in a

docs/DESIGN.md

@@ -1955,3 +1955,47 @@ placed* is a different fault from one that is crashing, and the arithmetic that
 it (requests vs allocatable) is available locally.
 
 Recorded as **P1-16**. Affects S4, S14.
+
+### §4.54 — "down" is a status, not a diagnosis (2026-08-30, agent, etappe 54)
+
+Built directly from §4.53. During that outage the panel reported four components
+`down` — correctly, within seconds, for 37 hours — and never said *why*. The scheduler
+had been publishing the reason in a `FailedScheduling` event the entire time.
+
+`kubectl get pods` already answers "is it broken". The reason an admin tool exists is
+to answer **"it cannot be placed, because it asks for 8500m on a 6000m node"**. The gap
+between those two sentences is where 37 hours went.
+
+What makes this more than printing an event message is the arithmetic, and the
+arithmetic has two traps that this project met the hard way:
+
+**A pod's request is not the sum of its containers.** It is
+`max(sum(containers), max(initContainers))`, because init containers run alone and
+first. Synapse's `render-config` and `db-wait` had each inherited 4000m while the
+`synapse` container asked for 1000m — so Synapse reserved 4000m the whole time it was
+*merely waiting for the database*. A naive sum reports 1000m and makes the diagnosis
+look wrong.
+
+**One `resources` block can cover several containers.** `postgres.resources` applied to
+postgres *and* postgres-ess-updater, so 4000m written once reserved 8000m. A
+per-container number understates the pod by half.
+
+Verified end to end rather than by unit test alone: a deliberately unschedulable probe
+pod, whose 40000m request sat in an *init* container behind a 100m app container, was
+diagnosed live at 40000m against 6000m allocatable — the max-of-init path exercised on
+real cluster data, then the probe deleted.
+
+Two deliberate limits. `ExceedsNode` separates "larger than any node" from "the cluster
+is full right now", because only the second one can resolve itself — telling an
+operator to wait for a pod that can never fit is worse than saying nothing. And the
+panel does not propose a number: what postgres *should* request depends on what else
+the operator intends to run, so naming the arithmetic is the job and choosing the value
+is theirs.
+
+The preventive half — warning at config-save time, before the value becomes an outage
+at the next reboot — is **not** here, and the reason is worth recording: knowing what a
+config *would* request means knowing the chart's container topology (which containers
+share a block, which init containers inherit it), which is a property of the chart and
+not of the YAML being edited. MatrixCtrl has the Helm SDK and can render it; inferring
+it from the values file would be exactly the guessing this project refuses. P1-16b.
+Affects S4.

docs/ROADMAP.md

@@ -59,6 +59,7 @@ Etappes 1–10 are **reconstructed from `git log`** (39 commits, 2026-05-27 →
 | 19 | Calling — the ports that must be forwarded, and an explicit "this half is not checkable from here" | ✅ 2026-08-01 · `v0.1.19` · [plan](plans/etappe-19-calling-reachability.md) |
 | 32 | Release Notes auf der Upgrade-Seite + Version aus der Liste übernommen — die andere Hälfte der Pin-Warnung | ✅ 2026-08-05 · `v0.1.33` · [plan](plans/etappe-32-release-notes.md) |
 | 33 | OIDC-Init wiederholen statt einmalig aufgeben — ein Neustart vor MAS sperrte den Operator 11 h aus dem eigenen Panel aus | ✅ 2026-08-06 · `v0.1.34` · [plan](plans/etappe-33-oidc-retry.md) |
+| 54 | „down" ist ein Status, keine Diagnose — warum ein Pod nicht eingeplant werden kann, mit der Rechnung dahinter | 🔄 gebaut 2026-08-30 · `v0.1.52` · [plan](plans/etappe-54-why-it-cannot-be-placed.md) |
 | 53 | Das lauteste Element der Seite war doppelt falsch — „postgres in Restart-Schleife", obwohl postgres nie neu gestartet ist | ✅ 2026-08-17 · `v0.1.51` · [plan](plans/etappe-53-restart-banner.md) |
 | 52 | Einmal verbinden, und da landen wo man war — der Rücksprung hing am State, der Auto-Reconnect an seiner Schleifenbremse | ✅ 2026-08-17 · `v0.1.50` · [plan](plans/etappe-52-silent-reconnect.md) |
 | 51 | Der UDP-Puffer, den die SFU anfordert und nicht bekommt — und der Zähler, der aus dem falschen Namespace gelesen worden wäre | ✅ 2026-08-17 · `v0.1.49` · [plan](plans/etappe-51-udp-buffer-preflight.md) |

docs/plans/etappe-54-why-it-cannot-be-placed.md

@@ -0,0 +1,59 @@
+# Etappe 54 — `down` is not a diagnosis
+
+P1-16, straight out of the outage of 2026-08-16…18 ([DESIGN.md §4.53](../DESIGN.md)).
+
+MatrixCtrl reported postgres, Synapse, MAS and haproxy as `down`. Correctly, and
+within seconds. For **37 hours**. What it never said was *why* — and the reason was in
+a `FailedScheduling` event the whole time, one API call from data the panel was already
+fetching.
+
+`kubectl get pods` already says "it is broken". An admin tool exists to say
+**"it cannot be placed, because it asks for 8500m on a 6000m node"**. That sentence is
+the etappe.
+
+## What makes this more than reading an event message
+
+Kubernetes' own message is `0/1 nodes are available: 1 Insufficient cpu.` — true, and
+not enough to act on. It does not say how much was asked for, or how much there is.
+Both are knowable here, and the arithmetic is where the two traps live:
+
+**A pod's request is not the sum of its containers.** It is
+`max(sum(containers), max(initContainers))`. Synapse's `render-config` and `db-wait`
+had inherited 4000m each, so Synapse reserved 4000m the entire time it was *only
+waiting for the database* — while its own `synapse` container asked for 1000m. Summing
+containers would have reported 1000m and made the diagnosis look wrong.
+
+**One `resources` block can cover several containers.** `postgres.resources` applies to
+postgres *and* postgres-ess-updater, so 4000m written once reserved 8000m. Reporting
+the per-container number would understate it by half.
+
+So the panel computes the *effective* request the scheduler actually uses, and shows it
+against the node's allocatable. That is the number that explains the failure.
+
+## Scope
+
+**Ships:** for a component whose pods are Pending, why — the scheduler's own reason,
+and for the `Insufficient cpu|memory` case the effective request against allocatable.
+Rendered where the component already says `down`.
+
+**Does not ship: warning at config-save time.** The valuable half, and the honest
+reason it is not here: to know what a config *would* request, the chart has to be
+rendered — the topology (which containers share a `resources` block, which init
+containers inherit it) is a property of the chart, not of the YAML being edited.
+MatrixCtrl has the Helm SDK and could dry-run it, but that is a second etappe with its
+own failure modes, and guessing the topology from the values file is exactly the kind
+of inference this project refuses. Stays P1-16b.
+
+**Does not ship: suggesting a fix.** "Reduce postgres to 750m" depends on what else the
+operator intends to run. Naming the arithmetic is the job; choosing the number is
+theirs.
+
+## Definition of done
+
+- A Pending component says why, with the scheduler's reason
+- For insufficient cpu/memory it shows effective request vs allocatable
+- The effective request is `max(sum(containers), max(initContainers))` — asserted by a
+  test built on the real shape that caused the outage
+- A component that is `down` for any other reason is unaffected
+- Nothing is claimed when there is no event to read: absent is not "fine"
+- `make check` green

internal/k8s/health.go

@@ -29,6 +29,10 @@ type ComponentHealth struct {
 	// LastRestart is when the most recent restart happened, absent when nothing has
 	// restarted. It is what turns a bare count into something an operator can judge.
 	LastRestart *time.Time `json:"last_restart,omitempty"`
+	// Unschedulable says *why* a down component is down, when the answer is "the
+	// scheduler refused it". Nil for every other kind of failure — a component that is
+	// crashing must not be described as unplaceable (etappe 54).
+	Unschedulable *Unschedulable `json:"unschedulable,omitempty"`
 }
 
 // ComponentHealth reports every ESS workload. StatefulSets matter as much as
@@ -47,16 +51,21 @@ func (c *Client) ComponentHealth(ctx context.Context, namespace string) ([]Compo
 			desired = *d.Spec.Replicas
 		}
 		restarts, by, looping, lastRestart := c.podRestarts(ctx, namespace, d.Spec.Selector.MatchLabels)
+		var why *Unschedulable
+		if workloadStatus(d.Status.ReadyReplicas, desired) == "down" {
+			why = c.WhyUnschedulable(ctx, namespace, d.Spec.Selector.MatchLabels)
+		}
 		result = append(result, ComponentHealth{
-			Name:        d.Name,
-			Kind:        "Deployment",
-			Status:      workloadStatus(d.Status.ReadyReplicas, desired),
-			Ready:       d.Status.ReadyReplicas,
-			Desired:     desired,
-			Restarts:    restarts,
-			RestartsBy:  by,
-			Looping:     looping,
-			LastRestart: lastRestart,
+			Name:          d.Name,
+			Kind:          "Deployment",
+			Status:        workloadStatus(d.Status.ReadyReplicas, desired),
+			Ready:         d.Status.ReadyReplicas,
+			Desired:       desired,
+			Restarts:      restarts,
+			RestartsBy:    by,
+			Looping:       looping,
+			LastRestart:   lastRestart,
+			Unschedulable: why,
 		})
 	}
 
@@ -70,16 +79,21 @@ func (c *Client) ComponentHealth(ctx context.Context, namespace string) ([]Compo
 			desired = *s.Spec.Replicas
 		}
 		restarts, by, looping, lastRestart := c.podRestarts(ctx, namespace, s.Spec.Selector.MatchLabels)
+		var why *Unschedulable
+		if workloadStatus(s.Status.ReadyReplicas, desired) == "down" {
+			why = c.WhyUnschedulable(ctx, namespace, s.Spec.Selector.MatchLabels)
+		}
 		result = append(result, ComponentHealth{
-			Name:        s.Name,
-			Kind:        "StatefulSet",
-			Status:      workloadStatus(s.Status.ReadyReplicas, desired),
-			Ready:       s.Status.ReadyReplicas,
-			Desired:     desired,
-			Restarts:    restarts,
-			RestartsBy:  by,
-			Looping:     looping,
-			LastRestart: lastRestart,
+			Name:          s.Name,
+			Kind:          "StatefulSet",
+			Status:        workloadStatus(s.Status.ReadyReplicas, desired),
+			Ready:         s.Status.ReadyReplicas,
+			Desired:       desired,
+			Restarts:      restarts,
+			RestartsBy:    by,
+			Looping:       looping,
+			LastRestart:   lastRestart,
+			Unschedulable: why,
 		})
 	}