Dynamic headroom: derive placeholder count and size from recent worker spawns#894
Merged
Conversation
…r spawns Replace the hardcoded constant node-headroom (46 vCPU / 360Gi placeholders, DUCKGRES_K8S_HEADROOM_NODES) and the legacy demand-percent mode with a fully dynamic controller, with no count/size config at all: - Every worker pod spawn is logged best-effort to a new config-store table (duckgres_worker_spawn_log, goose migration 000016) with its shape. - Slot COUNT = clamp(peak spawns in any 5-minute bucket of the last hour, 1, max(4, ceil(25% of live worker pods))). Serial spawns reuse one warm slot, so only burst concurrency needs parallel headroom; the fleet-relative cap grows with the deployment and bounds spawn-storm cost without anyone bumping a constant. Fleet size is only the ceiling, never the target, so an idle-worker leak cannot inflate the placeholder pool. - Slot SIZE = componentwise max worker shape spawned in the last 7 days, falling back to the live fleet's max shape, then the configured/default worker request. Client-sized (GUC) workers can exceed any configured maximum, so observed spawns are the only honest sizing source. - Scale-up/refill is immediate; scale-down is lazy (one slot per 10 minutes) and shape drift >20% replaces placeholders in place, so flapping never thrashes pods. - Enable switch is the existing DUCKGRES_K8S_PLACEHOLDER_PRIORITY_CLASS env (a placeholder without a priority class below workers would never be preempted, so it was already a hard prerequisite). HEADROOM_NODES and HEADROOM_PERCENT envs are removed; the k8s/headroom-rbac.yaml node-read ClusterRole is deleted (nothing reads node allocatable anymore). - New gauges: duckgres_headroom_slots_desired/_slots_cap/_peak_spawn_burst/ _slot_cpu_millicores/_slot_memory_bytes. At current fleet shape (15 vCPU / 120Gi workers) this cuts steady-state headroom from 138 vCPU / 1080Gi (3 node-sized placeholders, ~216% of live worker demand) to one worker-sized slot when idle, scaling with real burst demand when busy. Headroom placeholders stay disabled in the e2e job (no placeholder PriorityClass there): real placeholders would consume shared mw-dev capacity and outlive the per-PR CP; covered by fake-clientset unit tests and a real postgres round-trip for the spawn-log SQL (see tests/e2e-mw-dev/README.md).
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: needs review Warnings
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The headroom placeholder pool is hardcoded to an r6gd node shape (46 vCPU / 360Gi per placeholder,
controlplane/headroom.goconstants) with a fixed count fromDUCKGRES_K8S_HEADROOM_NODES. That shape dates from the exclusive worker-per-node era; with today's ~15 vCPU / 120Gi workers, 3 placeholders reserve 138 vCPU / 1080Gi — ~216% of live worker demand — and preempting one 46-vCPU placeholder to schedule one 15-vCPU worker wastes most of the reserve granularity. The legacyDUCKGRES_K8S_HEADROOM_PERCENTmode is dead config in prod (nodes>0 wins) and was retired for amplifying idle-worker leaks.Change
Placeholder count and size are now derived dynamically from real spawns — no count/size knobs at all:
duckgres_worker_spawn_log(goose migration 000016; cross-CP by construction, pruned to the sizing window by the leader reconcile). A log failure never fails the spawn.clamp(peak spawn burst, 1, cap)— the peak number of spawns in any 5-minute bucket of the last hour. Serial spawns reuse one warm slot; only burst concurrency needs parallel headroom. Floor 1 (never zero while enabled).cap = max(4, ceil(25% × live workers))— fleet-relative, so it grows with the deployment without anyone bumping a constant, and bounds a spawn-storm bug's cost. Fleet size is only the ceiling, never the target, so an idle-worker leak cannot inflate the pool (the failure that killed percent mode).DUCKGRES_K8S_PLACEHOLDER_PRIORITY_CLASS(already a hard prerequisite — an unprioritized placeholder would never be preempted).DUCKGRES_K8S_HEADROOM_NODES/DUCKGRES_K8S_HEADROOM_PERCENTare removed.k8s/headroom-rbac.yaml(node-read ClusterRole) deleted — nothing reads node allocatable anymore.duckgres_headroom_slots_desired/_slots_cap/_peak_spawn_burst/_slot_cpu_millicores/_slot_memory_bytes. Desired pinned at cap = demand wants more than the fleet ratio allows.At current fleet shape this cuts steady-state headroom from 138 vCPU / 1080Gi to one worker-sized slot when idle, scaling with real burst demand when busy.
Rollout
The prod values overlay already sets
placeholderPriorityClassName, so dynamic mode enables itself on deploy; the old env vars become inert. Follow-up in the charts repo: dropheadroomNodes/headroomPercentvalues, their env plumbing, and theduckgres-headroom-nodesClusterRole gate. Existing node-sized placeholders are >20% shape-drifted from the new plan, so the first reconcile replaces them with worker-sized ones.Testing
controlplane/headroom_test.go: burst→count, spawn-log→size with live-fleet and default fallbacks, fleet-relative cap under a spawn storm, ceiling-not-target (idle-leak regression), cap floor, lazy scale-down one-per-delay, shape-drift replacement (incl. legacy node-sized pods), disabled-converges-to-zero, stats-error skips tick, best-effort spawn recording.tests/configstore/spawn_log_postgres_test.go: real-postgres round-trip of the burst-bucket / componentwise-max SQL and pruning; migration asserts inmigrations_postgres_test.go.tests/e2e-mw-dev/README.md"Deliberately not covered here"; spawn recording itself runs in-Job on every worker spawn.🤖 Generated with Claude Code