feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools

[omeryahud]

Summary

WorkerPool replicas are static today: a human sets spec.replicas and the controller applies it verbatim. There is no mechanism that grows or shrinks a pool in response to actor demand. This issue opens a discussion on an autoscaling story for WorkerPools — what it should optimize for, why "just attach an HPA" is not sufficient for substrate's workload shape, and a concrete approach we've been exploring (a warm-buffer model with a fast request-edge scale-up trigger and a slow, metric-based scale-down). The goal here is to align on direction and on the in-tree vs. out-of-tree boundary.

Autoscaling is already named as future work in docs/roadmap.md ("Worker horizontal autoscaling: Ability to rapidly scale up nodes and warm Pods to meet actor demand") and docs/architecture.md ("Autoscaling: We will need to be able to automatically scale the number of workers up and down based on demand…"), but there's no design or tracking issue for it yet.

Current state

• WorkerPoolSpec carries only replicas + ateomImage. No min/max bounds, no target utilization, no metric, no demand input.
• The scale subresource is exposed: the CRD declares
  +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,
  but HPA integration is incomplete; no pool-occupancy/capacity metrics exist
• A full pool fails fast: when no idle worker exists, AssignWorkerStep returns FailedPrecondition "no free workers available", which the router maps to HTTP 503. There is no queue/wait (tracked separately in Router needs to park requests and wait for capacity #27) and no capacity creation in response.

Net: the substrate partially provides the lever (the scale subresource) but no control loop and partial signals to drive one.

Why autoscaling matters

1. The density premise requires elasticity. Substrate's whole value is multiplexing many actors onto few warm workers. Without autoscaling, operators must statically over-provision for peak (wasting money on idle workers) or under-provision (and serve 504 timeouts under load). Elastic pools are what let density translate into actual cost savings.
2. Cost — scale-to-zero / scale-to-floor for idle pools. Idle actors are just snapshots in object storage; a pool serving no active actors is pure waste. Autoscaling (down to a small floor, or to zero for cold-start-tolerant pools) directly reduces spend.
3. Activation-latency SLO under bursts. Agent workloads tend to wake in correlated storms (sub-agent tasks/CI fan-out). Maintaining warm headroom and refilling it reactively is what helps the storm pass.
4. Eliminating avoidable 503s. Today an empty pool returns 503 with no recourse. Pairing capacity creation with the request-parking work in Router needs to park requests and wait for capacity #27 turns "no capacity → error" into "no capacity → brief wait → served."
5. Operability. Operators shouldn't hand-tune replicas per pool per time-of-day. A control loop (with sane bounds) mandatory for running this in production.

The core constraint - why a naive metric→HPA isn't enough

The binding latency is pod start (schedule + image pull + ateom boot + readiness — seconds to minutes; plus node provisioning if the cluster is full). Two consequences shape any design:

• No autoscaler makes the triggering request fast. The request that finds an empty pool will not be served by the pod that its miss provisions. Bursts are served by capacity that already exists — a warm buffer of idle workers, and preemption of idle/low-priority actors (via SuspendActor, will be tracked in a different issue). Autoscaling's real job is to keep the buffer stocked and track the trend, not to serve the burst in front of it.
• Pure metric-loop reactivity is bounded by detection lag. Prometheus scrape (~15–30s) + KEDA/HPA poll & stabilization (tens of seconds) is perfectly fine for smooth/diurnal demand, but too slow as the sole up-path for spiky wake-storms with a lean buffer — the buffer empties and stays empty through the lag window.

So "reactive enough" should be read as "fast enough to refill the buffer before the next comparable burst," not "fast enough to serve this burst."

Proposed approach (for discussion)

1. A warm-buffer invariant, replenished on consumption

Each pool targets a small number of idle (warm) workers. When a resume consumes a slot and the buffer dips below target, emit a refill — sized net of in-flight provisioning (anti-windup), so the loop doesn't pile on scale-ups while pods are still booting:

deficit = max(0, target - (free + in_flight)) // buffer refill
scale_to = clamp(current + deficit, reservation_floor, max_replicas).  // rate-limited per step

When preemption becomes available, preemptible actors will be deducted from the deficit.

2. Up/down asymmetry — and a single writer

• Scale-up is latency-critical → fast, event-driven, triggered at the request edge. The router/control plane already sees the miss (the FailedPrecondition) and the buffer level at t=0; that's the earliest possible signal. The trigger raises a replica floor immediately; the kube write stays off the request hot path (event-driven, delegated to a controller).
• Scale-down is safety-critical, not latency-critical → slow, hysteretic, metric-based. Premature shrink throws away a warm worker you'll want in 30s, and draining one means suspending a live actor. This is exactly what a metric loop (HPA/KEDA-style, with stabilization windows) is good at; it must never drop below per-pool reservations/floor.
• One writer to spec.replicas. A fast trigger and a metric loop must not both write the scale subresource, or they fight (trigger scales up, lagging metric scales back down → thrash). A single component should own the field and take both inputs (fast floor + slow shrink).

flowchart TD
  REQ["Cold resume, no free worker"] --> EDGE["Request edge senses miss / buffer drain (t=0)"]
  EDGE --> SIGNAL["Emit capacity-pressure signal (net of in-flight)"]
  SIGNAL -->|event-driven| WRITER["Single autoscaler: raise replica floor"]
  WRITER --> SCALE["Patch WorkerPool /scale"]
  SCALE --> PODS["New warm workers (slow) join the buffer"]
  METRIC["Sustained low occupancy (metric loop)"] -->|hysteretic| WRITER
  WRITER --> DOWN["Shrink, never below reservation floor"]

Loading

3. Where should this live?

Substrate is intentionally low-opinion, so the policy could reasonably live outside the core. Much of the above is buildable today as an external controller — it can drive the existing scale subresource and read pool occupancy via ListWorkers — provided substrate exposes the signals.

4. Scale-to-zero and node headroom

• Scale-to-zero is attractive for idle/cold-start-tolerant pools, but the first post-zero request pays full cold start; keep a per-pool floor for latency-sensitive pools.
• Pod scale-up only helps if a node has room. A pod request that lands Pending behind cluster-autoscaler/Karpenter is not fast. Warm node headroom is a complementary lever worth calling out.

Signals required

• A capacity-pressure signal from the router when it rejects a request due to not finding a free worker.
• Domain metrics that don't exist yet, that will help the down-path story: per-pool free/idle workers, occupancy, and "no free workers" rate.

Open questions

1. Down-path: is a stock HPA/KEDA on the scale subresource an acceptable default for scale-down?
2. Buffer model: should "warm headroom" be a first-class field on WorkerPoolSpec (e.g., minReady/targetBuffer), or purely an external controller's concern?
3. Any other thoughts and suggestions are welcome!

[omeryahud]

Update: dropping the stock-HPA down-path (re: open question 1)

When I opened this I assumed scale-down would be a stock metric loop (HPA/KEDA on the scale subresource), with a separate fast trigger only for scale-up. Building it out, I dropped HPA and folded both directions into a single in-tree autoscaler. Why:

1. Single-writer integrity — the decisive one. The original issue specifies spec.replicas has exactly one writer, for the fast up-path and the slow down-path. Creating a component that synchronizes pressure events from ateapi, and scale down events from an extern autoscaler is awkward (e.g. the trigger bumps the HPA's minReplicas instead of /scale) and still splits one control law across two systems. One component owning both directions makes single-writer structural instead of a coordination problem.

2. Fewer dependencies, no detection lag. No HPA/KEDA/Prometheus to install or operate. The loop reads occupancy straight from ateapi (ListWorkers) — no scrape lag.
   What it is now: a single WorkerPoolAutoscaler in ate-controller, the sole writer of spec.replicas. Up-path = event-driven capacity-pressure from ateapi (new WatchCapacityPressure stream, emitted where the resume misses). Down-path = the same loop's hysteretic poll on occupancy (which doubles as the safety net). minReady / targetBuffer / maxReplicas are first-class WorkerPoolSpec fields (re: open question 2).

flowchart TD
  subgraph dp["Data plane · ateapi"]
    MISS["Resume finds no free worker<br/>(AssignWorkerStep — t=0)"]
    OCC["Per-pool occupancy<br/>free / in-flight (ListWorkers)"]
  end
  MISS -->|"capacity-pressure event<br/>WatchCapacityPressure · fast"| AS
  OCC -->|"reconcile poll ~10s<br/>hysteretic · safety net"| AS
  subgraph ctl["ate-controller · single writer of spec.replicas"]
    AS["WorkerPoolAutoscaler 
UP immediate
 DOWN after stabilization"]
  end
  AS -->|"patch /scale"| WP["WorkerPool.spec.replicas"]
  WP --> WPR["WorkerPoolReconciler<br/>materializes the Deployment"]
  WPR --> PODS["Warm worker pods<br/>join / drain the buffer"]
  PODS -.->|"register / release"| OCC

Loading