[serve] Columnar zero-copy autoscaling-metrics ingest

[johntaylor-cell]

Why are these changes needed?

The ServeController aggregates autoscaling metrics in a single-threaded control loop.
In aggregate mode with metrics collected on handles, each handle report covers every
replica it routes to, so at high fan-in the controller spends a large share of the loop
just cloudpickle-decoding those reports before it can aggregate them.

This PR serializes the reports columnarly (flat float64 arrays) and ingests them
natively with numpy, replacing the cloudpickle path for wide handle reports. numpy +
zlib release the GIL, so the decode overlaps the rest of the loop instead of serializing
in front of it.

Performance

Controller-scaling benchmark (handle-collection, HAProxy-on), cloudpickle vs columnar:

replicas / handle	ingest ms/s (cloudpickle → columnar)	effective per-loop cost	speedup
64	29 → 4	114 → 41	2.8×
256	507 → 19	663 → 49	13.5×
1024	752 → 128	944 → 128	7.4×
4096	773 → 48	1228 → 212	5.8×

Cloudpickle ingest cost explodes with fan-in (it's serial in front of the loop); columnar
keeps it small and GIL-free, so the effective per-loop cost is max(loop, ingest)
instead of loop + ingest. The crossover is ~64 replicas/handle, which is exactly what
the width gate targets.

(At a pinned steady-state fleet the loop isn't ingest-bound, so columnar alone is
~neutral on loop there and contributes mainly via lower deployment-state cost; the win
above is the high-fan-in ingest regime it targets.)

What changed

• Width gate (RAY_SERVE_COLUMNAR_METRICS_MIN_REPLICAS, default 64): producers pick
  columnar only for handle reports covering ≥ the gate's replica count — the measured
  decode/merge crossover. Thin handle reports and all replica reports stay on the
  cloudpickle/object path, so small deployments never regress. Format is chosen by report
  type + width, not a separate on/off flag.
• Wire-detected format: the controller routes on the SCR1 magic (framed outside
  zlib, O(1) detect), independent of how a report was produced — so a fleet mid-rollout
  (mixed columnar/cloudpickle senders) aggregates correctly.
• Columnar payload: carries running_requests on the hot path plus custom
  autoscaling metrics in a side array store, exact-equivalent to the object path. numpy
  imports lazily, so serve-minimal installs (which never hit the columnar path) are
  unaffected.

An earlier revision also gated this behind an on/off RAY_SERVE_COLUMNAR_METRICS flag
(default off); that flag has been removed now that the path is proven — columnar is
selected purely by report type + the width gate. The width threshold stays env-tunable.

Correctness (mixed rollout)

• Dedup-at-write: a source lives in exactly one store; switching wire format clears
  the other, so a mixed fleet never double-counts.
• Unified cross-format staleness gate: one per-source last-accepted timestamp gates
  both the object and columnar ingest paths (per replica and per handle), so a delayed
  report in either format can't overwrite fresher data the other wrote.
• Running-replica gate: the columnar aggregate keys off whether a running replica
  reported (not merely a non-empty array store), so a stale stopped-replica array left
  before on_replica_stopped can't suppress handle-collected running metrics — matching
  the object path exactly.

Test Plan

Unit suites: columnar wire-format + width gate, columnar-vs-object equivalence (every
aggregation function), controller ingest dispatch (aggregate vs simple mode), custom
metrics, mixed-rollout aggregation, cross-format staleness, and the stale-replica-array
regression.

An attached html document lists all of the optimizations that were implemented as part of a larger Serve controller scaling effort. This PR is for optimization A5.

[gemini-code-assist]

Code Review

This pull request introduces a columnar serialization format and array-native reference merge operations for Ray Serve's autoscaling metrics, replacing cloudpickle-based serialization with zero-copy numpy/array-based encoding to reduce CPU and latency overhead on the controller's event loop. It also implements reconciler optimizations, including a 'dirty-set' replica tracking mechanism to avoid container churn and an autoscale aggregation cache. The review feedback suggests optimizing the compression level in _frame from level=9 to a lower level to reduce CPU overhead, adding defensive length checks in the decode function to prevent crashes on corrupted payloads, and addressing the unused ready_set parameter and _ref_ready helper in deployment_state.py to improve maintainability.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[johntaylor-cell]

OPTIMIZATIONS_SUMMARY.html
ray_serve_controller_scaling_perf.html

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit c21fa24. Configure here.}