Etappe 44: die Calls-Zahlen sterben mit dem Pod, den unser eigener Hook löscht

P2-29 verlangte eine Inventur vor dem Bauen. Sie ergab einen Befund, der die
Form aller drei Teile bestimmt hat: jeder LiveKit-Zähler ist prozesslebenslang.
Zehn Stunden nach dem 26.8.0-Upgrade stand auf einem Server, der seit Monaten
läuft, überall 0 — nicht weil nie telefoniert wurde, sondern weil der
Post-Upgrade-Hook den SFU-Pod bei jedem ESS-Upgrade löscht, um hostNetwork
wiederherzustellen. Mehrmals pro Woche.

Eine Statistikseite direkt auf diesen Zahlen hätte "seit dem letzten Upgrade"
gesagt und wie "jemals" geklungen — §4.24 mit anderem Gegenstand.

Also aufgezeichnet statt gelesen (§4.42). Das Muster stand schon im Paket:
internal/rtc/watcher.go misst DNS auf einem Timer, weil eine aus Seitenaufrufen
gebaute Historie genau dort Lücken hat, wo niemand hingesehen hat.

- Live: Räume und Teilnehmer, beide die ganze Zeit auf dem Metrics-Port und von
  nichts gelesen — dieselbe Lücke wie P1-10.
- Verlauf: Calls, Gesprächszeit, SFU-Neustarts über 24 h und pro Tag.
- Ein Zähler, der niedriger zurückkommt, ist ein Neustart und wird als solcher
  gespeichert, nie als negatives Delta. Alle drei Zähler werden geprüft, weil der
  Hauptzähler auf dieser Instanz meist auf beiden Seiten null ist.
- Summen sind exakt trotz grobem Intervall: die Zähler sind kumulativ, ein Call
  zwischen zwei Messungen zählt mit. Nur der Zeitpunkt ist aufs Intervall genau,
  und die Seite sagt das.
- Ein fehlgeschlagener Read schreibt nichts. Eine Null wäre eine erfundene Ruhe
  und beim nächsten Read ein erfundener Neustart.

Teilnehmer-Identitäten werden bewusst nicht gelesen: die RoomService-API würde
"wer telefoniert mit wem" liefern, und keine Frage dieser Seite braucht das.

Live geprüft: alle sechs Metriken auf der echten SFU vorhanden, Sampler schreibt
im 60-Sekunden-Takt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: BxnnyG

CHANGELOG.md

@@ -15,6 +15,39 @@ matching image, so a version identifies one exact pair
 
 ## [Unreleased]
 
+## [0.1.44] — 2026-08-16
+
+### Added
+
+- **The calls page says whether anyone is actually on a call.** Rooms and
+  participants, live from the SFU. Both numbers were on the metrics port from the
+  beginning and nothing read them — which is the same gap P1-10 was about: every
+  check on this page was green while the feature was completely dead, because none
+  of them asked whether anyone had ever used it.
+
+- **A call history that survives the SFU.** Calls, talk time and SFU restarts over
+  24 hours and per day.
+
+  This had to be recorded rather than read. Every LiveKit counter is
+  process-lifetime, and the post-upgrade hook deletes the SFU pod on every ESS
+  upgrade to restore `hostNetwork` — so a statistics page built directly on those
+  numbers would silently mean "since the last upgrade" while reading as "ever".
+  Measured ten hours after the 26.8.0 upgrade, every counter on the server was `0`.
+
+  The totals are exact regardless of the sampling interval: the underlying counters
+  are cumulative, so a call that starts and ends entirely between two samples is
+  still counted. Only its *timing* is bounded by the interval, and the page says so
+  rather than leaving it to be discovered.
+
+  A counter that comes back lower means the SFU restarted. That is recorded as a
+  restart — never as a negative delta — and shown, because it explains a
+  discontinuity in every other series on the page.
+
+- **No participant identities are read or stored.** LiveKit's RoomService API would
+  give room names and participants; it is deliberately not used. "Three people are
+  in a call" and "who is in a call with whom" are different classes of data, and
+  none of the questions this page answers needs the second.
+
 ## [0.1.43] — 2026-08-16
 
 Three etappes in one release: 0.1.42 was written but never built — the image build

cmd/matrixctrl/main.go

@@ -266,7 +266,14 @@ func main() {
 	// Observe the announced RTC address on a timer. Doing it only on page view
 	// would leave the history with gaps exactly where nobody was looking, and the
 	// thing being measured is *when* the address changed.
-	go rtc.NewWatcher(rtc.NewStore(pool), rtcHandler.AnnouncedHost, 0).Start(context.Background())
+	rtcStore := rtc.NewStore(pool)
+	go rtc.NewWatcher(rtcStore, rtcHandler.AnnouncedHost, 0).Start(context.Background())
+
+	// Sample the SFU's counters on a timer, for a stronger version of the same
+	// reason: LiveKit's counters are process-lifetime, and the post-upgrade hook
+	// deletes the SFU pod on every ESS upgrade. Unrecorded, the call history does
+	// not merely have gaps — it is destroyed several times a week (E44).
+	go rtc.NewSampler(rtcStore, rtcHandler.MetricsReader(), 0).Start(context.Background())
 
 	srv := server.New(addr, router)
 

deploy/helm/matrixctrl/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: matrixctrl
 description: Admin layer for self-hosted Matrix / Element Server Suite (ESS)
 type: application
-version: 0.1.43
-appVersion: "0.1.43"
+version: 0.1.44
+appVersion: "0.1.44"
 keywords:
   - matrix
   - element

docs/BACKLOG.md

@@ -474,7 +474,17 @@ implemented, OIDC state consumed atomically via `DELETE … RETURNING` (CSRF-saf
   *Why it matters:* k3s on ARM boards is a realistic home-server case, and the
   README does not currently say the image is amd64-only.
 
-- **P2-29 · Calls show no audit, no connections and no statistics (S14).** Reported
+- **P2-29 · Calls show no audit, no connections and no statistics (S14).**
+  **Two of four done 2026-08-16 (E44, [DESIGN.md §4.42](DESIGN.md)):** live rooms and
+  participants, and a recorded history of calls, talk time and SFU restarts that
+  survives the pod. The inventory this entry demanded found the reason none of it
+  could simply be read: every LiveKit counter is process-lifetime, and the
+  post-upgrade hook deletes the SFU pod on every ESS upgrade.
+  *Still open:* per-call audit entries with more than a count — that needs either the
+  RoomService API (participant identities, deliberately not read) or Synapse-side
+  call events, and neither is a small addition. And "Statistik ausweiten auf das
+  ganze", which is a design question rather than a missing counter.
+  Original entry: Reported
   by the operator 2026-08-16: "calls soll auch wenn möglich audit zeigen
   verbindungen zeigen maby auch statistik (statistik maby ausweiten auf das ganze) /
   logs länge".

docs/DESIGN.md

@@ -1404,3 +1404,61 @@ waits in one blocking call and the boundary is not observable. `internal/rollout
 keeps its no-client-go rule, so the assembly is a pure function with unit tests, and
 a live test proves the two reads it depends on are permitted under E40's namespaced
 Role and that Generation is actually populated. Affects S11.
+
+### §4.42 — A counter that resets is not a history (2026-08-16, operator + agent, etappe 44)
+
+The operator asked the calls page for audit, connections and statistics. The
+inventory that P2-29 demanded before building any of it produced one fact that
+decided the shape of all three.
+
+Every number LiveKit publishes is **process-lifetime**. Read ten hours after the
+26.8.0 upgrade, on a server that has been running for months:
+
+    livekit_room_total                   0
+    livekit_participant_total            0
+    livekit_room_duration_seconds_count  0
+    livekit_quality_score_count          0
+
+None of that means "this deployment has never carried a call". It means "this
+process has not", and the process is young because MatrixCtrl's own post-upgrade
+hook deletes the SFU pod on every ESS upgrade to restore `hostNetwork` — several
+times a week here. A statistics panel reading those counters directly would have
+said "since the last upgrade" in the voice of "ever", which is §4.24 again with a
+different subject: reporting confidently on the part that happens to be readable.
+
+**So the history is recorded, not read.** The precedent was already in the package:
+`internal/rtc/watcher.go` samples DNS on a timer because "a history built from page
+views has gaps exactly where nobody was looking, which is most of the time." Here
+the argument is stronger — unrecorded, this history is not merely gappy, it is
+destroyed on a schedule.
+
+Three things make the recording trustworthy:
+
+- **A reset is a first-class event, not an anomaly to smooth over.** When a counter
+  comes back lower, the delta is the *new value* — everything the old process
+  counted was already recorded by samples taken while it ran — and the restart is
+  stored, because it explains a discontinuity in every other series. All three
+  counters are checked, not one: the headline counter is zero on both sides of most
+  restarts on this instance, so watching only it would miss them.
+- **Exact totals from an inexact interval.** Both underlying counters are
+  cumulative, so calls and minutes between two samples are right even for a call
+  that began and ended between them. Only the *timing* is coarse, and the page says
+  which is which instead of implying both are precise.
+- **A failed read records nothing.** An unreachable metrics endpoint is not an SFU
+  with zero rooms. Writing a zero would fabricate a quiet period, and would then
+  read as a counter reset on the next successful sample — inventing a restart that
+  never happened.
+
+**What was deliberately not built:** LiveKit's RoomService API would give room names
+and participant identities. "Three people are in a call" and "who is in a call with
+whom" are different classes of data, and no question this page answers needs the
+second. The gauges answer "connections" without reading a secret, minting an admin
+token, or storing anyone's identity.
+
+**Also not built:** "Statistik ausweiten auf das ganze". A panel-wide statistics
+story is a design question about what an operator wants to see over time, not a
+matter of finding more counters. The plan says so rather than half-doing it — the
+general shape is better argued from one working example than in advance.
+**Consequences:** migration 011 keeps the raw counter values beside the resolved
+deltas, so a bug in the delta logic can be corrected against the original
+observations rather than having destroyed them. Affects S14.

docs/ROADMAP.md

@@ -51,6 +51,7 @@ Etappes 1–10 are **reconstructed from `git log`** (39 commits, 2026-05-27 →
 | 19 | Calling — the ports that must be forwarded, and an explicit "this half is not checkable from here" | ✅ 2026-08-01 · `v0.1.19` · [plan](plans/etappe-19-calling-reachability.md) |
 | 32 | Release Notes auf der Upgrade-Seite + Version aus der Liste übernommen — die andere Hälfte der Pin-Warnung | ✅ 2026-08-05 · `v0.1.33` · [plan](plans/etappe-32-release-notes.md) |
 | 33 | OIDC-Init wiederholen statt einmalig aufgeben — ein Neustart vor MAS sperrte den Operator 11 h aus dem eigenen Panel aus | ✅ 2026-08-06 · `v0.1.34` · [plan](plans/etappe-33-oidc-retry.md) |
+| 44 | Calls: wer gerade telefoniert, und ein Verlauf — die SFU-Zähler sterben mit dem Pod, den der Post-Upgrade-Hook jedes Mal löscht | 🔄 gebaut 2026-08-16 · `v0.1.44` · [plan](plans/etappe-44-call-history.md) |
 | 43 | Das Upgrade-Fenster zeigte eine Uhr statt Fortschritt — dazu Versionen mit Datum und Notes, und der Typecheck, der nie etwas geprüft hat | ✅ 2026-08-16 · `v0.1.43` · [plan](plans/etappe-43-upgrade-progress.md) |
 | 42 | Räume verbinden ging und funktionierte dann nicht — der Scope, den E36 bewusst weggelassen hatte; dazu Historie über Neustarts hinweg schnell | ✅ 2026-08-16 · `v0.1.43` · [plan](plans/etappe-42-rooms-connect-and-latency.md) |
 | 41 | Räume, zweite Hälfte — Detail, Mitglieder, und die eine Aktion, die man zurücknehmen kann | ✅ 2026-08-16 · `v0.1.43` · [plan](plans/etappe-41-room-detail.md) |

docs/plans/etappe-44-call-history.md

@@ -0,0 +1,95 @@
+# Etappe 44 — The calls page reports on a counter that resets
+
+P2-29, reported by the operator 2026-08-16:
+
+> "calls soll auch wenn möglich audit zeigen verbindungen zeigen maby auch
+> statistik (statistik maby ausweiten auf das ganze) / logs länge"
+
+Four asks: per-call audit, live connections, RTC statistics, and a wider statistics
+story. The backlog entry said this needed an inventory pass before any of it, because
+§4.24 is the standing warning here — the calls page once reported confidently on the
+SFU while the failing calls were legacy 1:1, which it had no way to see.
+
+## Inventory
+
+**What LiveKit exposes on this deployment**, read from the SFU's own metrics port:
+
+| metric | kind | what it answers |
+|---|---|---|
+| `livekit_room_total` | gauge | rooms open **right now** |
+| `livekit_participant_total` | gauge | participants **right now** |
+| `livekit_room_duration_seconds_count` | counter | rooms that have completed |
+| `livekit_room_duration_seconds_sum` | counter | seconds of room time |
+| `livekit_quality_score_{count,sum}` | histogram | call-quality samples |
+| `livekit_forward_latency_ns_count` | counter | forwarded-media samples |
+| `livekit_node_packet_total` | counter | packets, rises without any call |
+
+Three of these are already parsed — `internal/rtc/media.go` reads the counters that
+prove media flowed (E23). The two **gauges** are not read by anything, and they are
+exactly "Verbindungen zeigen".
+
+**The fact that shapes everything else:** all of these are *process-lifetime*. Read
+live, ten hours after the 26.8.0 upgrade, every single one is `0`:
+
+    livekit_room_total                   0
+    livekit_participant_total            0
+    livekit_room_duration_seconds_count  0
+    livekit_quality_score_count          0
+
+That is not "this server has never carried a call". It is "this **process** has not",
+and the process is younger than the counters suggest, because the post-upgrade hook
+deletes the SFU pod on every ESS upgrade to restore `hostNetwork`. On this instance
+that is several times a week.
+
+So a statistics page built directly on these numbers would silently mean "since the
+last upgrade" while looking like "ever" — the §4.24 mistake with a different subject.
+**History has to be recorded by MatrixCtrl or it does not exist.**
+
+**The precedent is already in the package.** `internal/rtc/watcher.go` + `store.go`
+exist for exactly this reason, for a different quantity: DNS observations are sampled
+on a timer and persisted, because "a history built from page views has gaps exactly
+where nobody was looking, which is most of the time." The same sentence applies here.
+
+**What is deliberately not built:** LiveKit's RoomService API (port 7880) would give
+room names and participant identities. It needs the SFU's API secret and a minted
+admin token, and what it returns is *who is in a call with whom*. That is a different
+class of data from "three people are in a call", and it is not needed to answer any
+of the four asks. The gauges answer "connections" without it.
+
+## What this etappe does
+
+**Sampling.** A `Sampler` alongside the existing `Watcher`: reads the metrics port on
+a timer, writes one row per observation. The endpoint is already fetched by the RTC
+handler, so this is a second caller of a known-good path, not a new integration.
+
+**Counter resets are first-class.** When a counter comes back *lower* than the last
+sample, the SFU restarted. The delta is then the new value, not a negative number —
+and the restart itself is worth recording, because "the SFU restarted" explains a gap
+in every other series on the page.
+
+**Exact totals, coarse timing.** `room_duration_seconds_{count,sum}` are cumulative,
+so the number of calls and the total minutes between two samples are exact *even for
+calls that began and ended entirely between them*. What the sampling interval bounds
+is only *when* — the same trade the Watcher documents for DNS, and it gets stated on
+the page rather than left for someone to discover.
+
+**The page gains two sections:** what is happening now (rooms, participants, and how
+long the SFU has been up so the zero can be read correctly), and what has happened
+(calls per day, total minutes, quality samples, restarts). The existing call-path
+assessment stays exactly where it is — it answers a question neither of these does.
+
+**"Statistik ausweiten auf das ganze" is not in scope**, and the plan says so rather
+than half-doing it. A panel-wide statistics story is a design question about what an
+operator wants to see over time, not a matter of finding more counters. This etappe
+makes one series real; the shape of the general case should be argued from a working
+example rather than in advance.
+
+## Definition of done
+
+- A sample is written on a timer and survives an SFU restart
+- A restart is visible as a restart, never as a negative delta or a reset to zero
+- The live section distinguishes "no calls" from "the SFU restarted a minute ago"
+- Totals are exact across a restart, verified against a counter that actually moved
+- The page states the interval, so "when" is never read as more precise than it is
+- No participant identity is read or stored
+- `make check` green, and a live test against the real SFU

internal/api/handlers/rtc.go

@@ -23,13 +23,13 @@ type RTCHandler struct {
 	configStore *config.Store
 	essNS       string
 	essRelease  string
-	addresses   *rtc.Store
+	store       *rtc.Store
 }
 
 func NewRTCHandler(k8sClient *k8s.Client, cfgStore *config.Store, essNS, essRelease string, db *pgxpool.Pool) *RTCHandler {
 	return &RTCHandler{
 		k8s: k8sClient, configStore: cfgStore, essNS: essNS, essRelease: essRelease,
-		addresses: rtc.NewStore(db),
+		store: rtc.NewStore(db),
 	}
 }
 
@@ -83,7 +83,7 @@ func (h *RTCHandler) Status(w http.ResponseWriter, r *http.Request) {
 	if len(ips) > 0 {
 		resolved = ips[0]
 	}
-	if err := h.addresses.Record(ctx, host, resolved); err != nil {
+	if err := h.store.Record(ctx, host, resolved); err != nil {
 		log.Printf("rtc: could not record address observation for %q: %v", host, err)
 	}
 
@@ -99,7 +99,12 @@ func (h *RTCHandler) Status(w http.ResponseWriter, r *http.Request) {
 		"ports":          ports,
 		"freshness":      freshness,
 		"media":          media,
-		"call_paths":     paths,
+		// The uptime was already computed for the findings and thrown away. The page
+		// needs it in its own right: every counter here is process-lifetime, so
+		// "0 Räume" a minute after a restart and "0 Räume" all day are the same
+		// number meaning different things (E44).
+		"sfu_uptime": uptime,
+		"call_paths": paths,
 		"findings": append(
 			rtc.AssessWithFreshness(ports, host, ips, resolveErr, freshness, freshnessDetail),
 			rtc.AssessMedia(media, mediaOK, uptime),
@@ -138,7 +143,7 @@ func (h *RTCHandler) freshness(ctx context.Context, host string) (rtc.Freshness,
 		return rtc.FreshnessUnknown, "Für matrixRTC ist kein Hostname konfiguriert."
 	}
 
-	obs, err := h.addresses.Newest(ctx, host)
+	obs, err := h.store.Newest(ctx, host)
 	if err != nil {
 		return rtc.FreshnessUnknown, "Die Adress-Historie konnte nicht gelesen werden."
 	}
@@ -332,6 +337,47 @@ func (h *RTCHandler) mediaEvidence(ctx context.Context) (rtc.MediaEvidence, bool
 	return ev, ok, uptime
 }
 
+// History answers "what has actually happened on this SFU" — the question the
+// metrics port cannot answer about anything before the current process (E44).
+func (h *RTCHandler) History(w http.ResponseWriter, r *http.Request) {
+	ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
+	defer cancel()
+
+	day, err := h.store.SamplesSince(ctx, time.Now().Add(-24*time.Hour), 0)
+	if err != nil {
+		Error(w, http.StatusInternalServerError, "could not read samples: "+err.Error())
+		return
+	}
+	daily, err := h.store.Daily(ctx, 30)
+	if err != nil {
+		Error(w, http.StatusInternalServerError, "could not aggregate samples: "+err.Error())
+		return
+	}
+
+	JSON(w, http.StatusOK, map[string]any{
+		"last_24h": rtc.Sum(day),
+		"daily":    daily,
+		// Reported so the page can say how precise "when" is. The totals do not
+		// depend on it — both underlying counters are cumulative, so a call that
+		// starts and ends between two samples is still counted — but the timing
+		// within an interval is lost, and a reader should be told which is which.
+		"interval_seconds": int(rtc.SamplerInterval.Seconds()),
+	})
+}
+
+// MetricsReader exposes the SFU read for the sampler.
+//
+// The uptime is dropped: it comes from the pod's start time rather than the metrics
+// body, and a sample is about what the counters said, not about how the process
+// that said it was doing. The page still shows uptime, because there the point is
+// to let the reader tell "no calls" from "restarted a minute ago".
+func (h *RTCHandler) MetricsReader() func(context.Context) (rtc.MediaEvidence, bool) {
+	return func(ctx context.Context) (rtc.MediaEvidence, bool) {
+		ev, ok, _ := h.mediaEvidence(ctx)
+		return ev, ok
+	}
+}
+
 func formatUptime(d time.Duration) string {
 	switch {
 	case d < time.Hour:

internal/api/router.go

@@ -90,6 +90,7 @@ func NewRouter(deps Deps) http.Handler {
 		}
 		if deps.RTC != nil {
 			r.Get("/api/v1/rtc/status", deps.RTC.Status)
+			r.Get("/api/v1/rtc/history", deps.RTC.History)
 			r.Get("/api/v1/users", deps.Users.List)
 		}