Skip to content

[serve] Read the multiplex marker statically so probing cannot initialize handles - #65064

Merged
abrarsheikh merged 3 commits into
ray-project:masterfrom
johntaylor-cell:serve-multiplex-probe-static
Jul 28, 2026
Merged

[serve] Read the multiplex marker statically so probing cannot initialize handles#65064
abrarsheikh merged 3 commits into
ray-project:masterfrom
johntaylor-cell:serve-multiplex-probe-static

Conversation

@johntaylor-cell

@johntaylor-cell johntaylor-cell commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Follow on to PR #64045

Why

_callable_uses_multiplexing() probed instance attributes with getattr(), which on a
DeploymentHandle triggers __getattr__options() → eager Router initialization.
Under direct ingress this scan runs at startup on every ingress replica, so each one
holding a handle built a Router that would otherwise stay lazy — each adding a
controller long-poll subscription and metric-report stream. Cost scales with replicas.

What

Both probes in the scan are now side-effect free:

  • The marker is read with inspect.getattr_static(), which never invokes __getattr__.
  • The instance __dict__ is read with object.__getattribute__() rather than
    getattr(obj, "__dict__", {}), which falls back to __getattr__ on a __slots__
    class. getattr_static is not usable here — for __dict__ it returns the type's
    getset_descriptor rather than the instance mapping.

Detection is unchanged. Adds a regression test asserting the scan never invokes
__getattr__, covering both a handle-holding callable and a __slots__ callable; the
existing test only covers false positives, so it cannot catch either.

Test

4096-replica controller benchmark, HAProxy on, vs the immediate parent commit (n=1 per
arm): handle metric report delay 7275ms → 827ms, deployment_state_update
0.242s → 0.151s, controller asyncio tasks 12182 → 10004.

@johntaylor-cell
johntaylor-cell requested a review from a team as a code owner July 28, 2026 02:43
@johntaylor-cell johntaylor-cell self-assigned this Jul 28, 2026
@johntaylor-cell johntaylor-cell added go add ONLY when ready to merge, run all tests serve Ray Serve Related Issue performance labels Jul 28, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces the use of getattr with inspect.getattr_static when checking for the multiplexed function marker in _callable_uses_multiplexing. This prevents unintended side effects, such as eagerly initializing DeploymentHandle Routers during probing. A unit test has also been added to verify this behavior. The reviewer suggests a further improvement: applying inspect.getattr_static to retrieve __dict__ later in the same function to ensure the entire probing process is completely side-effect-free.

Comment thread python/ray/serve/_private/utils.py
…lize handles

_callable_uses_multiplexing() scans a replica instance's attributes to detect
multiplexing wired up dynamically in a constructor. It read the marker with
getattr(), which is not side-effect free: DeploymentHandle.__getattr__ returns
options(method_name=...), and options() eagerly initializes the handle's Router.

So probing a handle constructed one. Under direct ingress the scan runs at startup
on every ingress replica (replica.py _raise_if_multiplexing_with_direct_ingress),
so every ingress replica holding a handle built a Router that would otherwise stay
lazy -- each carrying a controller long-poll subscription and a periodic metric
report stream. The cost scales with replica count.

On the 4096-replica controller benchmark (HAProxy on), comparing against the
immediate parent commit, this raised handle metric report ingest delay from 383ms
to 7275ms, deployment_state_update from 0.117s to 0.242s, and controller asyncio
tasks from 9302 to 12182. Reading the marker statically restores it to 827ms /
0.151s / 10004.

inspect.getattr_static never invokes __getattr__ or __getattribute__, and reads the
marker the decorator sets directly on the wrapper, so detection is unchanged. The
existing 'is True' identity check is kept.

Adds a regression test asserting the scan never invokes __getattr__ at all. The
existing test only covers false positives, whose fake handle has no side effect and
so cannot catch this.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: john.taylor <john.taylor@anyscale.com>
@johntaylor-cell
johntaylor-cell force-pushed the serve-multiplex-probe-static branch from 82faa99 to 93c3873 Compare July 28, 2026 02:48
johntaylor-cell and others added 2 commits July 28, 2026 02:57
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: john.taylor <john.taylor@anyscale.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: john.taylor <john.taylor@anyscale.com>
Comment thread python/ray/serve/_private/utils.py
Comment thread python/ray/serve/_private/utils.py

@abrarsheikh abrarsheikh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does getattr cause options to be called?

@johntaylor-cell

johntaylor-cell commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

why does getattr cause options to be called?

utils.py:76        getattr(obj, MULTIPLEXED_FUNCTION_MARKER_ATTR, False)
                   └─ obj is a DeploymentHandle (the replica's self.hello_world)
                   └─ normal lookup FAILS: not in instance __dict__, not in type MRO
                        ↓  Python falls back to __getattr__
handle.py:304      DeploymentHandle.__getattr__(self, name)
handle.py:305          return self.options(method_name=name)
                        ↓
handle.py:1141     options(...) → return self._options(...)
                        ↓
handle.py:223-224  _options():  if not self.is_initialized:
                                    self._init() <-------------------

Accept added delay and close PR? Or merge PR and fix benchmark to send requests to the replicas?

@abrarsheikh
abrarsheikh merged commit 3149ca9 into ray-project:master Jul 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests performance serve Ray Serve Related Issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants