Skip to content

[Fix][Core/Dashboard] component_cpu_percentage metric always zero for dashboard subprocess modules - #65055

Open
MortalHappiness wants to merge 2 commits into
ray-project:masterfrom
MortalHappiness:fix/dashboard-component-cpu-percentage-always-zero
Open

[Fix][Core/Dashboard] component_cpu_percentage metric always zero for dashboard subprocess modules#65055
MortalHappiness wants to merge 2 commits into
ray-project:masterfrom
MortalHappiness:fix/dashboard-component-cpu-percentage-always-zero

Conversation

@MortalHappiness

@MortalHappiness MortalHappiness commented Jul 27, 2026

Copy link
Copy Markdown
Member

Description

The component_cpu_percentage metric is currently always zero for dashboard subprocess modules. This PR fixes this.

Related issues

Similar symptom with #29848

Additional information

Before

image

After

image

… dashboard subprocess modules

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
@MortalHappiness
MortalHappiness force-pushed the fix/dashboard-component-cpu-percentage-always-zero branch 2 times, most recently from afe3086 to 7a81cd7 Compare July 27, 2026 23:17
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
@MortalHappiness
MortalHappiness force-pushed the fix/dashboard-component-cpu-percentage-always-zero branch from 7a81cd7 to 4553db5 Compare July 27, 2026 23:24
@MortalHappiness
MortalHappiness marked this pull request as ready for review July 27, 2026 23:40
@MortalHappiness
MortalHappiness requested a review from a team as a code owner July 27, 2026 23:40

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces caching of psutil.Process handles for dashboard subprocess modules to ensure accurate CPU usage reporting, as psutil.Process.cpu_percent() requires a persistent handle to measure relative CPU usage. It also adds comprehensive unit tests to verify this behavior. The reviewer's feedback recommends handling potential exceptions like psutil.NoSuchProcess or psutil.AccessDenied when instantiating psutil.Process for a PID, and checking for None before recording metrics to prevent the metric recording loop from crashing.

Comment on lines 384 to 387
self._record_cpu_mem_metrics_for_proc(
proc, subprocess_module_handle.module_cls.__name__
self._get_subprocess_module_proc(pid),
subprocess_module_handle.module_cls.__name__,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Since _get_subprocess_module_proc can return None if the process is missing or inaccessible, we should check if the returned process is not None before attempting to record its metrics.

Suggested change
self._record_cpu_mem_metrics_for_proc(
proc, subprocess_module_handle.module_cls.__name__
self._get_subprocess_module_proc(pid),
subprocess_module_handle.module_cls.__name__,
)
proc = self._get_subprocess_module_proc(pid)
if proc is not None:
self._record_cpu_mem_metrics_for_proc(
proc,
subprocess_module_handle.module_cls.__name__,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a valid suggestion.

Comment on lines +405 to +418
def _get_subprocess_module_proc(self, pid: int) -> psutil.Process:
"""Return the cached psutil.Process for pid, creating it if needed.

Reusing the handle is what gives cpu_percent() a baseline to measure
against. See the comment on self._subprocess_module_procs.
"""
proc = self._subprocess_module_procs.get(pid)
# is_running() is False once the process is gone, and also once the pid
# has been reused by an unrelated process, in which case the cached
# handle's baseline no longer describes the process being measured.
if proc is None or not proc.is_running():
proc = psutil.Process(pid)
self._subprocess_module_procs[pid] = proc
return proc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If a subprocess module exits or restarts, or if there are permission restrictions, psutil.Process(pid) can raise psutil.NoSuchProcess or psutil.AccessDenied. Wrapping the instantiation in a try-except block and returning None prevents these exceptions from crashing the periodic metric recording loop.

    def _get_subprocess_module_proc(self, pid: int) -> Optional[psutil.Process]:
        """Return the cached psutil.Process for pid, creating it if needed.

        Reusing the handle is what gives cpu_percent() a baseline to measure
        against. See the comment on self._subprocess_module_procs.
        """
        proc = self._subprocess_module_procs.get(pid)
        # is_running() is False once the process is gone, and also once the pid
        # has been reused by an unrelated process, in which case the cached
        # handle's baseline no longer describes the process being measured.
        if proc is None or not proc.is_running():
            try:
                proc = psutil.Process(pid)
                self._subprocess_module_procs[pid] = proc
            except (psutil.NoSuchProcess, psutil.AccessDenied):
                return None
        return proc

@ray-gardener ray-gardener Bot added the core Issues that should be addressed in Ray Core label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Issues that should be addressed in Ray Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants