HTTP client: num_workers must be a multiple of the endpoint count for even load — auto-scale workers for multiple endpoints

[viraatc]

Observed behavior (reported from disagg/agg deployments)

With multiple frontend/agg servers behind the client (e.g. a 9×DEP8 aggregation setup), num_workers for the HTTP client must be a multiple of the number of endpoints (9, 18, …) or load is skewed:

• num_workers not a multiple of len(endpoints) → some servers receive more requests than others (endpoints with ⌈w/n⌉ workers get proportionally more traffic than those with ⌊w/n⌋).
• num_workers < len(endpoints) → some servers receive no traffic at all and sit idle.

Mechanism

Each worker process is statically pinned to exactly one endpoint at startup:

# worker.py:150
endpoint_url = endpoint_urls[worker_id % len(endpoint_urls)]

so the per-endpoint request share is proportional to how many workers landed on it — there is no request-level balancing across endpoints.

The auto default makes this easy to hit: num_workers=-1 resolves from NUMA topology (min(max(8, numa_cpu_count), 24)), which is not endpoint-aware — e.g. auto→24 workers with 9 endpoints gives 6 endpoints 3 workers and 3 endpoints 2 workers (a 50% traffic difference between servers), and auto→8 with 9 endpoints leaves one server idle.

Suggested fixes (roughly increasing scope)

1. Validate/warn at config time — in HTTPClientConfig._resolve_defaults, warn when num_workers % distinct_endpoints != 0, and warn loudly (or error) when num_workers < distinct_endpoints (silent idle servers invalidate multi-server perf runs).
2. Make the auto default endpoint-aware — when num_workers=-1 and multiple distinct endpoints are configured, round the NUMA-derived count to a multiple of the endpoint count (e.g. nearest multiple, floored at distinct_endpoints).
3. Longer-term: request-level balancing — decouple workers from single endpoints (per-worker pools to all endpoints with least-loaded/round-robin dispatch). Larger change to the worker hot path; only worth it if the static pinning becomes limiting beyond the divisibility issue.

Related

• The max_connections port budget multiplies by distinct endpoints while workers each serve one endpoint — if endpoints outnumber workers, unreachable endpoints inflate the budget (noted in Connection pool: treat EADDRNOTAVAIL as backpressure so the full-budget auto ceiling is safe under max_throughput bursts #418).
• Interim guidance for disagg/agg users: set num_workers explicitly to a multiple of the frontend count (e.g. 9 or 18 for 9×DEP8).