Feature request: More system metrics for debuggability (Activation SLI, crash accounting, and capacity signals)

[JeffLuoo]

Activation SLI, crash accounting, and capacity signals

Related: #433, #198, #27, #174, #547, #166

Summary

This proposal covers the four metrics that complement the initial set in #433.

All four follow the #433 cardinality rule — labels limited to template, worker pool, sandbox class, and closed outcome/reason enums; never actor, atespace, or session — and reuse the internal/ateattr label conventions.

Naming follows OTel semantic conventions (revised per review): dedicated instruments instead of a state attribute; self-descriptive instrument names (eligible_workers); all new attribute keys namespaced under ate.*; and error.type restricted to standardized values (gRPC status codes at RPC boundaries), with substrate-specific taxonomies in namespaced attributes such as ate.failure.reason.

Each metric is either a label extension on an existing instrument or a single counter/gauge at a code choke point where the data is already in hand, so implementation cost is low.

Relationship to the north stars (docs/architecture.md)

North star	Target	Status today	Relationship to these metrics
Activation latency	100ms @ p95	Unmeasurable — atenet.router.route.duration mixes warm routes (ms) and cold starts (up to 20s+) in one distribution	Measured directly by metric 1: p95 of resume=triggered
Throughput	1000 wakeups/sec	Not isolatable from warm lookups	Measured directly by metric 1: rate of resume=triggered
Scale	1B actors/cluster	Not measured by anything in this proposal — under the cardinality rule these metrics count operations, not actors, and freely double-count	Not measured here. The population measurement is #433's ate.actors (total actors tracked in the store). Metrics 2–4 provide what operating at that scale requires: actor-loss rate (2) and saturation visibility (3, 4)

At a glance

#	Metric	Measures	Golden signal	North star / driver
1	atenet.router.route.duration — new labels	Edge routing latency, split warm vs. cold, failures classified	Latency, Traffic	Activation latency SLO; throughput; #198 pressure signal; #27 baseline
2	ate.actor.crashes	Terminal actor failures, by cause	Errors	Reliability at 1B-actor scale (actor-loss rate)
3	ate.scheduler.eligible_workers	Eligible free workers per scheduling decision	Saturation	Activation latency (no-capacity prevention); #198 warm-buffer input
4	ate.workerpool.desired_workers / ate.workerpool.ready_workers	Worker pods requested vs. actually ready	Saturation	Scale-to-zero / #198 autoscaling control loop

---

1. Extend atenet.router.route.duration with resume and richer outcome labels

Golden signals: Latency + Traffic. North star: makes the 100ms p95 activation target and the 1000 wakeups/sec throughput target measurable.

• Measures: time from Envoy handing request headers to ext_proc until the target worker endpoint is resolved (Host parse + ResumeActor + authority rewrite), excluding actor compute and the response.
• Change: labels only, on the existing histogram.
  • New label ate.router.resume ∈ {none, triggered, joined}:
    • none — actor was already RUNNING; the call was a pure location lookup. This series is the steady-state routing SLI every warm request pays.
    • triggered — this request won the per-actor singleflight and executed the resume. Its latency is the activation latency measured at the edge — the series for the 100ms p95 north star. Its rate is wakeups/sec — the throughput north star.
    • joined — a resume was already in flight; this request parked on the shared call. Kept separate because it is a biased sample of activation latency: one 20s cold start with 50 queued callers must count as 1 slow activation in SLO math, not 51.
  • Split the catch-all outcome=error into no_capacity (FailedPrecondition "no free workers available"), timeout, lock_conflict (Aborted), resume_error. "Fleet is full" (capacity/ops problem) and "routing is broken" (bug) are currently the same data point.
  • Attribute-key migration: the histogram's existing bare keys (actor_template_namespace, actor_template_name, outcome) predate the namespacing convention — migrate them to ate.template.namespace, ate.template.name, ate.router.outcome in the same change, with a coordinated dashboard update, rather than leaving two conventions side by side.
• Emitted by: atenet-router. Code anchors: cmd/atenet/internal/router/extproc.go (classifyOutcome), resumer.go (singleflight).
• Why P0: without the resume split, the platform's most important promise has no measurement at the user boundary. outcome=no_capacity rate is the capacity-pressure signal feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198 specifies at the request edge, and the baseline for how many requests Router needs to park requests and wait for capacity #27's parking would save. The joined:triggered ratio quantifies wake storms.
• CUJ: "An agent user's request hung or got a 503 — was the actor cold, was the fleet full, or is routing broken?"

2. ate.actor.crashes (counter)

Golden signal: Errors. North star: reliability accounting required to operate at 1B-actor scale — today the platform cannot state its actor-loss rate.

Naming note (from review discussion): the name deliberately matches the state machine's terminal STATUS_CRASHED state — the metric counts entries into that state, not process-level crashes. Not all entries are data loss in fact (some are conservative terminal-izations of control-plane inconsistencies); the ate.failure.reason label keeps those populations separable.

• Measures: count of actors entering the terminal CRASHED state, attributed to the operation that failed and the underlying cause.
• Labels: operation.name ∈ {resume, suspend, pause}; ate.failure.reason — the ateerrors.Reason enum (TERMINAL_FILE_SYSTEM_ERROR, INVALID_SANDBOX_ASSET, INVALID_CHECKPOINT_RESULT, FAILED_SAVE_SNAPSHOT, INVALID_OBJECT_URL, FAILED_GET_EXTERNAL_OBJECT, INVALID_CONTAINER_CONFIG) plus control-plane causes (corrupted_assignment, worker_reassigned, worker_pod_gone); ate.template.name/namespace, ate.workerpool.name, ate.sandbox.class.
• Emitted by: ateapi — one increment inside crashActor, the single choke point every terminal failure flows through. Code anchors: cmd/ateapi/internal/controlapi/crash.go; internal/ateerrors/ateerrors.go.
• Why P0: CRASHED is terminal unavailability — with possible loss of un-persisted state — and has no alertable signal of any kind today — discoverable only by listing actors or grepping logs. The cause label is essentially free: atelet already tags failures with a typed Reason as AIP-193 ErrorInfo and ateapi already parses it back. This is the definitional gap in the Errors golden signal.
• CUJ: "Actors are ending up CRASHED — how many, and why?" Alert: crash rate > 0. A spike of TERMINAL_FILE_SYSTEM_ERROR = node disks full; FAILED_GET_EXTERNAL_OBJECT = storage backend unhealthy; per-template attribution catches one bad template crashing repeatedly.

3. ate.scheduler.eligible_workers (histogram, {worker})

Golden signal: Saturation. North star: protects the activation-latency target under load; direct input to #198's warm-buffer controller.

• Measures: the number of eligible free workers remaining after all constraint filters, sampled at every scheduling decision — how close to empty the schedulable pool was.
• Labels: ate.workerpool.name (when determinable), ate.sandbox.class, ate.scheduling.constraint ∈ {none, required_nodes, selector}.
• Emitted by: ateapi, one record of len(candidates) inside Scheduler.Schedule. Code anchor: cmd/ateapi/internal/scheduling/scheduling.go.
• Why P0: ate.workerpool.workers (utilization) and Initial set of platform metrics around lifecycle, scheduling, and fleet-saturation #433's assignment-duration (outcome) bracket the decision but miss how close to empty it was and why. ErrNoCapacity fires when the constraint intersection is empty — which can happen while the pool is half idle (e.g., paused actors pinned via RequiredNodes to a cordoned node). Without candidate counts, both states produce identical 503s with opposite remediations (scale up vs. fix selectors/drain policy). The candidate-count distribution trending toward zero is the leading saturation indicator feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198 needs — it warns before the first rejection, whereas the 503 rate only reports after users are failing.
• CUJ: "Resumes are failing with 503 but ate.workerpool.workers shows idle workers — why?" Alert: p50 candidates < N as a pre-saturation warning.

4. ate.workerpool.desired_workers / ate.workerpool.ready_workers (observable UpDownCounters, {worker})

Golden signal: Saturation (supply side). North star: makes scale-to-zero and #198 autoscaling operable — closes the control loop between demand signals and delivered capacity.

• Measures: per pool, how many worker pods were requested (spec.replicas) versus how many are actually Ready — whether commanded capacity was delivered.
• Labels: ate.workerpool.name on both instruments. Dedicated instruments rather than one metric with a state attribute, matching K8s semconv (k8s.deployment.desired_pods / k8s.deployment.available_pods): desired + ready is not a meaningful sum, unlike idle + assigned on the existing ate.workerpool.workers.
• Emitted by: atecontroller, from the WorkerPool status sync, which already reads both numbers every reconcile. Code anchor: cmd/atecontroller/internal/controllers/workerpool_controller.go:101-121.
• Why P0: the other saturation signals say when to scale; nothing says whether scaling worked. With the scale subresource + pod selector merged (Add pod selector to the WorkerPool scale subresource #547) and feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198 about to write spec.replicas programmatically, an autoscaler without a delivery signal is an open control loop: desired rises, nodes are exhausted, ready flatlines, and every substrate-side metric insists the mitigation was applied while requests keep getting 503s. desired − ready is also the anti-windup input feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198's warm-buffer refill computes against (capacity already in flight). Failures in this gap (quota, nodepool exhaustion, stuck pods) belong to the cluster-admin persona — this metric converts "the substrate seems full" into "the cluster owes us 20 pods."
• CUJ: "I scaled the pool (or the HPA did) — did the capacity actually arrive?" Alert: desired − ready > 0 sustained beyond a few minutes.

---

Prerequisite

Metric 4 requires OTel wiring in atecontroller (internal/serverboot init), which currently has none — only the unscraped controller-runtime registry on :8080. This wiring is a one-time cost also needed by planned P1/P2 atecontroller metrics (golden-snapshot pipeline), so doing it here pays it down early.

What's next

I propose metrics based on north star and SRE golden signals. More metrics proposal will come in separate issue, this issue focus most important metrics in my opinion. Open for discussion and feedbacks.

[JeffLuoo]

Krisztian F (@krisztianfekete) Julian Gutierrez Oschmann (@juli4n) Max Smythe (@maxsmythe) fyi, please take a look

[maxsmythe]

Thanks for this!

One question about naming and one pedantic nit about north star targets.

Metric name ate.actor.crashes -- I wonder about the most appropriate naming for this. If I read your intent right, this is for any operation where non-persisted data is potentially lost? Do all of these necessarily signify data loss? Does a resume where the old workers is not available mean data loss if the snapshot is pulled from GCS or similar? Some of these seem like scheduling errors or potentially control plane corruption rather than a problem with the actor?

I get the analogy, but "crashes" to me implies something sticky with a specific actor (process crashed, OOMKill, etc.). Should those signals be mixed? If not is there a better name?

Looking at the table of proposed metrics... you say the "scale" measurement is resolved by adding these metrics. I don't think this is true. Given the cardinality decision of not including actor name in metrics, I don't think these metrics speak to scale since they will freely double-count actors.

I think probably the only way to measure "scale" as contemplated by the docs/architecture.md is to run a counter of the total number of actors tracked in valkey.

I think all of these proposed metrics are valuable, just pushing back on the notion that they contribute to measuring that specific north star target.

[JeffLuoo]

Thank you for your feedback!!

Naming: The metric counts transitions into the existing terminal STATUS_CRASHED state, not process crashes, with reasons. And you're right that not all of them are data loss: when a worker dies under a RUNNING actor, the syncer already resets it to SUSPENDED (snapshot in GCS is still authoritative) — no crash recorded. crashActor only fires when the platform can't prove the snapshot is authoritative: suspend interrupted before upload (definite loss), or mid-resume inconsistencies like worker_reassigned (uncertain). The reason label separates workload causes from control-plane causes, so the two populations stay distinguishable — but I agree crashes in name may overclaim.

The proposal is to rename it to ate.actor.terminal_failures; if CRASHED is misleading as as a state name too, that's a separate issue.

For the "scale" measure, I will amend the table. Some metrics will help measure the signals we want to achieve in our north star like the latency. I remove the term "resolve by" from the description.

[maxsmythe]

Did not realize there was also a state name. That changes things. IMO naming alignment is primary. Not sure what thoughts went into the state names.

100% agree that these are helpful metrics for achieving north star goals. Merely pedantry :)

[JeffLuoo]

Angela (@Angelawork) will help work on it.

[krisztianfekete]

Thanks for fleshing these out!

Couple of comments to follow OTel semantic conventions w/r/t metric names and attributes:

• We should probably go with dedicated gauges for desired and ready replica metrics instead of ate.workerpool.replicas to match K8s semconv (you also reference kube-state-metrics)
• Re. ate.scheduler.candidates: what do you think about _eligible_workers, or similar?
• We should use namespaced attributes
• We should not roll our own error.type value for the error classes across metrics