Skip to content

Raise prod CPU limit to 2 and request to 500m to stop CFS throttling#131

Merged
elffjs merged 1 commit into
mainfrom
raise-cpu-limit-for-throttling
Jun 10, 2026
Merged

Raise prod CPU limit to 2 and request to 500m to stop CFS throttling#131
elffjs merged 1 commit into
mainfrom
raise-cpu-limit-for-throttling

Conversation

@elffjs

@elffjs elffjs commented Jun 10, 2026

Copy link
Copy Markdown
Member

Why

Prod pods are being CPU-throttled in 5–35% of enforcement periods, continuously, on every pod (CPUThrottlingHigh alerts) — despite averaging only 0.2–0.33 cores against the 1-core limit.

The cause is a burst/limit mismatch, not capacity: Go 1.25's container-aware GOMAXPROCS has a documented floor of 2 (confirmed live: go_sched_gomaxprocs_threads 2), so the process can consume CPU at 2× the 1-core quota during bursts, exhausting a 100ms enforcement period's allowance in 50ms and then stalling for the remainder. Each stall lands on in-flight work — webhook HTTP sends and Kafka batches — adding tail latency that feeds both the consumer lag and the spurious webhook failure counts we've been fixing.

What

  • CPU limit 1 → 2: matches GOMAXPROCS so bursts can no longer outrun the quota. Limits don't affect scheduling/bin-packing, so this reserves nothing on the node.
  • CPU request 250m → 500m: reflects actual usage (~0.3 cores average) for honest scheduling and CFS share weight.

Verification

After deploy, the throttle ratio should drop to ~0:

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="prod",container="vehicle-triggers-api"}[5m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="prod",container="vehicle-triggers-api"}[5m]))

🤖 Generated with Claude Code

…ottling

Pods average 0.2-0.33 cores but are throttled in 5-35% of CFS periods
all day: Go 1.25 floors GOMAXPROCS at 2, so with a 1-CPU quota the
process can exhaust a 100ms period's allowance in 50ms and stall for
the rest, adding tail latency to webhook sends and Kafka consumption.
A limit of 2 matches GOMAXPROCS so bursts can't outrun the quota.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@elffjs
elffjs merged commit 8a1ca29 into main Jun 10, 2026
4 checks passed
@elffjs
elffjs deleted the raise-cpu-limit-for-throttling branch June 10, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant