[cuda] fix: hide nvml-only unstartable gpu listings

AGENTS.md

@@ -188,7 +188,7 @@ This file defines how coding agents should work in this repository.
 - Treat public `gpu_ids` as visible device ordinals after user-supplied CUDA or ROCm visibility filtering. Do not rewrite visibility masks inside KeepGPU command paths; reject explicit CUDA/ROCm ordinals outside the current visible device count before starting keep workers.
 - Keep CUDA telemetry aligned with visible CUDA ordinals: `get_gpu_utilization(index)` receives the visible rank used by `CudaGPUController`, and `gpu_monitor.py` resolves `CUDA_VISIBLE_DEVICES` numeric/UUID tokens to the correct NVML handle. If that mapping is malformed, duplicate/equivalent, ambiguous, or unsupported, return `None` without partial NVML handle queries so eco-safe backoff applies instead of falling back to a possibly wrong physical index.
 - Keep ROCm telemetry aligned with visible ROCm ordinals: resolve `ROCR_VISIBLE_DEVICES` as the base mask and one matching `HIP_VISIBLE_DEVICES`/`CUDA_VISIBLE_DEVICES` overlay before querying ROCm SMI. If the mapping is malformed, conflicting, unsupported, or out of range, return unavailable utilization rather than querying a guessed SMI index.
-- Keep GPU listing IDs aligned with start APIs: `list_gpus`/`/api/gpus` must expose `id` as the visible ordinal users can pass as `gpu_ids`; physical/vendor identifiers belong in explicit metadata fields such as `physical_id` and must not be accepted implicitly as selection IDs.
+- Keep GPU listing IDs aligned with start APIs: `list_gpus`/`/api/gpus` must expose `id` as the visible ordinal users can pass as `gpu_ids`; physical/vendor identifiers belong in explicit metadata fields such as `physical_id` and must not be accepted implicitly as selection IDs. CUDA NVML records must only be returned when Torch CUDA can address the same visible ordinal set; do not advertise NVML-only devices that controller startup cannot use.
 - Keep GPU listing platform precedence aligned with controller platform
   detection: HIP/ROCm torch builds must prefer ROCm listing over NVML CUDA
   listing, and must not fall back to NVML CUDA records when torch's active

README.md

@@ -93,7 +93,9 @@ Flags that matter:
   - `keep-gpu service-stop`: stop the ownership-verified auto-started local daemon.
   - `keep-gpu list-gpus`: fetch telemetry from local service. Each listed
     `id` is the visible ordinal accepted by `--gpu-ids`; optional
-    `physical_id`/`uuid` fields are metadata only.
+    `physical_id`/`uuid` fields are metadata only. CUDA devices are listed only
+    when Torch CUDA can start the same visible ordinal set; NVML-only inventory
+    is hidden instead of producing unusable `gpu_ids`.
   - `status`, `stop`, and `list-gpus` print structured JSON objects, including
     `{"error": "..."}` for service/runtime errors after CLI parsing succeeds,
     that tools such as `jq` can parse directly. Malformed JSON-RPC service
@@ -174,7 +176,7 @@ to zero devices.
   curl http://127.0.0.1:8765/health
   curl http://127.0.0.1:8765/api/sessions
   ```
-- Methods: `start_keep`, `stop_keep` (optional `job_id`, default stops all), `status` (optional `job_id`), `list_gpus` (basic info). REST session creation accepts a JSON object body, not arrays or scalar values. Omitting `gpu_ids` uses all visible GPUs, and omitting `busy_threshold` uses the eco-safe default `25`; explicit values must be unique visible ordinals in the service process environment. `list_gpus` returns those same start-compatible ordinals as `id`/`visible_id`; `physical_id` and `uuid` are informational metadata, not valid substitutes for `gpu_ids`. Empty, duplicate, or out-of-range lists are invalid and startup fails if no GPUs resolve. Public numeric session inputs must stay finite and bounded: interval values are positive seconds, including fractional seconds, capped by the runtime wait limit, and VRAM byte-equivalent values are capped at 1 PiB. Custom `job_id` values must be unique across active and starting sessions, and only `null`/omitted means generated or all-sessions; custom IDs must be non-empty strings containing only letters, digits, `.`, `_`, `-`, or `~`. Status responses include reserved jobs as `state="starting"` while controller startup is still in progress.
+- Methods: `start_keep`, `stop_keep` (optional `job_id`, default stops all), `status` (optional `job_id`), `list_gpus` (basic info). REST session creation accepts a JSON object body, not arrays or scalar values. Omitting `gpu_ids` uses all visible GPUs, and omitting `busy_threshold` uses the eco-safe default `25`; explicit values must be unique visible ordinals in the service process environment. `list_gpus` returns those same start-compatible ordinals as `id`/`visible_id`; `physical_id` and `uuid` are informational metadata, not valid substitutes for `gpu_ids`. On CUDA, NVML records are returned only when Torch CUDA can address the same visible ordinal set, so NVML-only devices are not advertised as startable. Empty, duplicate, or out-of-range lists are invalid and startup fails if no GPUs resolve. Public numeric session inputs must stay finite and bounded: interval values are positive seconds, including fractional seconds, capped by the runtime wait limit, and VRAM byte-equivalent values are capped at 1 PiB. Custom `job_id` values must be unique across active and starting sessions, and only `null`/omitted means generated or all-sessions; custom IDs must be non-empty strings containing only letters, digits, `.`, `_`, `-`, or `~`. Status responses include reserved jobs as `state="starting"` while controller startup is still in progress.
 - Supported REST route/method failures remain machine-readable: validation
   errors use JSON `400` responses, unknown API routes use JSON `404`, and
   unexpected service/runtime failures use JSON `500` instead of closing the

docs/concepts/architecture.md

@@ -124,3 +124,8 @@ GPU listing follows the same precedence. On HIP/ROCm torch builds, `list_gpus`
 prefers ROCm records and ROCm SMI metadata instead of returning NVML CUDA records
 first on mixed hosts. If ROCm SMI is unavailable, listing falls back to
 torch's HIP-backed device records rather than NVML CUDA records.
+On non-HIP CUDA builds, NVML records are returned only when Torch CUDA reports a
+matching positive visible-device count and each visible ordinal can be selected
+with `torch.cuda.set_device()`. If that NVML/Torch trust check fails, listing
+falls back to Torch CUDA records and does not probe ROCm SMI on non-HIP builds,
+so CUDA devices are not misreported as ROCm inventory.

docs/guides/cli.md

@@ -81,7 +81,9 @@ keep-gpu list-gpus
 ```
 
 The `id` field in this output is the visible ordinal to pass to `--gpu-ids`.
-`status`, `stop`, and `list-gpus` print JSON objects, including
+On CUDA, NVML telemetry is listed only when Torch CUDA can start the same
+visible ordinal set; NVML-only devices are hidden rather than advertised as
+usable selections. `status`, `stop`, and `list-gpus` print JSON objects, including
 `{"error": "..."}` for service/runtime errors after CLI parsing succeeds, that
 can be parsed directly with `jq` or a single `json.loads()` call.
 Malformed JSON-RPC service envelopes are reported as JSON error objects instead

docs/guides/mcp.md

@@ -167,6 +167,9 @@ curl http://127.0.0.1:8765/api/sessions
 `/api/gpus` returns start-compatible visible ordinals as `id`/`visible_id`.
 Optional `physical_id` or `uuid` fields describe the underlying vendor device
 only; clients should not send those metadata values as `gpu_ids`.
+On CUDA, NVML records are exposed only when Torch CUDA can start the same
+visible ordinal set, so NVML-only devices are omitted instead of advertised as
+usable session targets.
 On ROCm, `physical_id` is included only when KeepGPU can safely resolve
 `ROCR_VISIBLE_DEVICES` and one matching HIP/CUDA overlay to a ROCm SMI index;
 otherwise utilization is reported as unavailable rather than guessed.

docs/guides/python.md

@@ -3,6 +3,10 @@
 Embed KeepGPU directly inside orchestration scripts so GPUs stay warm only for
 the stages you choose.
 
+For global sessions and telemetry helpers, public CUDA IDs are Torch-startable
+visible ordinals. NVML may add utilization and vendor metadata, but NVML-only
+devices are not exposed as `gpu_ids` targets when Torch CUDA cannot start them.
+
 ## Keep a single GPU while you do CPU work
 
 ```python

docs/plans/cuda-nvml-startable-listing.md

@@ -0,0 +1,87 @@
+# CUDA NVML Startable Listing Plan
+
+## Background
+
+`get_gpu_info()` uses NVML before Torch on CUDA systems so list responses can
+include utilization and vendor metadata. `GlobalGPUController`, however, starts
+CUDA sessions with visible Torch ordinals from `torch.cuda.device_count()`.
+
+When NVML can see CUDA devices but Torch CUDA is unavailable, reports zero
+devices, or reports a visible count that does not match a CUDA visibility mask,
+`list_gpus` can advertise IDs that users cannot successfully pass as
+`gpu_ids`.
+
+## Goal
+
+Keep public GPU listings start-compatible: CUDA NVML records should be returned
+only when Torch CUDA can address the same visible ordinal set. If NVML cannot be
+trusted for the startable set, fall back to Torch listing when possible; if Torch
+cannot start CUDA devices, list no CUDA devices.
+
+## Solution
+
+- Add RED coverage for NVML-visible but Torch-unstartable CUDA environments.
+- Require a trustworthy Torch CUDA visible count before returning NVML CUDA
+  records.
+- Require the resolved CUDA visible-token count to match Torch's visible count.
+- Preserve HIP/ROCm precedence so HIP Torch builds never fall back to NVML CUDA
+  records.
+- Update agent and user docs to state that listed CUDA IDs are start-compatible
+  Torch visible ordinals, not NVML-only inventory.
+
+## Tasks
+
+- [x] Add RED tests for Torch CUDA unavailable, zero count, device-count failure,
+      and CUDA visibility/Torch count mismatch.
+- [x] Implement the minimal NVML guard and Torch fallback behavior.
+- [x] Update `AGENTS.md`, README, CLI reference, MCP guide, and Python/API docs.
+- [x] Run focused GPU info tests.
+- [x] Run targeted GPU/MCP and global-controller-adjacent tests.
+- [x] Run full tests, docs build, hooks, and whitespace checks.
+- [x] Request local subagent code review and resolve findings before PR.
+
+## Verification Log
+
+- RED:
+  `PYTHONPATH=$PWD/src pytest tests/utilities/test_gpu_info.py -q` failed with
+  four focused failures:
+  Torch CUDA unavailable, Torch count zero, Torch count failure, and CUDA
+  visible-token/Torch count mismatch still returned NVML records.
+- GREEN focused:
+  `PYTHONPATH=$PWD/src pytest tests/utilities/test_gpu_info.py -q` passed with
+  24 tests and 1 skipped.
+- MCP/GPU targeted:
+  `PYTHONPATH=$PWD/src pytest tests/mcp tests/utilities/test_gpu_info.py -q`
+  passed with 193 tests and 1 skipped.
+- Global-controller targeted:
+  `PYTHONPATH=$PWD/src pytest tests/global_controller tests/utilities/test_gpu_info.py -q`
+  passed with 76 tests and 2 skipped.
+- Local review follow-up:
+  added RED tests for two reviewer findings:
+  matching NVML/Torch counts still listed a CUDA device when
+  `torch.cuda.set_device(0)` failed, and non-HIP fallback could label a CUDA
+  Torch fallback as ROCm when `rocm_smi` was importable. Both focused
+  regressions now pass.
+- Final review follow-up:
+  added a RED test proving non-HIP listing must not fall through to ROCm SMI
+  when both NVML and Torch reject an unstartable CUDA ordinal. Removed the
+  non-HIP ROCm SMI fallback and updated architecture/docstring wording.
+- Review follow-up targeted:
+  `PYTHONPATH=$PWD/src pytest tests/utilities/test_gpu_info.py -q` passed with
+  27 tests and 1 skipped after the final review follow-up.
+- Review follow-up MCP/GPU targeted:
+  `PYTHONPATH=$PWD/src pytest tests/mcp tests/utilities/test_gpu_info.py -q`
+  passed with 196 tests and 1 skipped after the final review follow-up.
+- Review follow-up global-controller targeted:
+  `PYTHONPATH=$PWD/src pytest tests/global_controller tests/utilities/test_gpu_info.py -q`
+  passed with 79 tests and 2 skipped after the final review follow-up.
+- Full suite:
+  `PYTHONPATH=$PWD/src pytest tests -q` passed with 568 tests and 11 skipped
+  after the local review follow-up.
+- Docs build:
+  `PYTHONPATH=$PWD/src mkdocs build` passed. It emitted the existing Material
+  for MkDocs warning and unnav'd docs notices, including this plan page.
+- Hooks:
+  `pre-commit run --all-files` passed.
+- Whitespace:
+  `git diff --check` passed.

docs/reference/api.md

@@ -9,7 +9,9 @@ visible device ordinals after CUDA or ROCm visibility filtering. Passing an
 empty, duplicate, or out-of-range list is invalid, and startup raises
 `ValueError` if discovery resolves to zero devices. Telemetry may expose
 metadata such as `physical_id`, but those fields are not accepted as selection
-IDs.
+IDs. On CUDA, NVML telemetry records are returned only when Torch CUDA can start
+the same visible ordinal set, so NVML-only devices are not exposed as public
+`gpu_ids`.
 
 `GlobalGPUController` validates local constructor inputs (`gpu_ids`, `interval`,
 `busy_threshold`, and `vram_to_keep`) before platform or hardware probing.

docs/reference/cli.md

@@ -97,8 +97,10 @@ envelopes are reported as JSON error objects instead of empty success results.
 Returns GPU telemetry from service. `id` and `visible_id` are the visible
 ordinals accepted by `--gpu-ids` and service `gpu_ids`; optional `physical_id`
 or `uuid` fields are metadata only. The output is a directly parseable JSON
-object, including `{"error": "..."}` for service/runtime errors after CLI
-parsing succeeds. Malformed JSON-RPC service envelopes are reported as JSON
+object. On CUDA, NVML records are listed only when Torch CUDA can start the same
+visible ordinal set; NVML-only devices are hidden rather than advertised as
+usable `gpu_ids`. Service/runtime errors after CLI parsing succeeds are reported
+as `{"error": "..."}`. Malformed JSON-RPC service envelopes are reported as JSON
 error objects instead of empty success results. Invalid endpoint values are
 reported as JSON errors before RPC.
 

src/keep_gpu/utilities/gpu_info.py

@@ -34,6 +34,48 @@ def _cuda_visible_tokens() -> Optional[List[str]]:
     return tokens
 
 
+def _torch_cuda_visible_count() -> Optional[int]:
+    cuda = getattr(torch, "cuda", None)
+    if cuda is None:
+        return None
+    try:
+        if not cuda.is_available():
+            return 0
+        count = int(cuda.device_count())
+    except Exception as exc:
+        logger.debug("Torch CUDA visible count failed: %s", exc)
+        return None
+    if count < 0:
+        return None
+    return count
+
+
+def _torch_cuda_visible_ordinals_startable(count: int) -> bool:
+    cuda = getattr(torch, "cuda", None)
+    if cuda is None:
+        return False
+
+    current_device = None
+    try:
+        try:
+            current_device = int(cuda.current_device())
+        except Exception as exc:
+            logger.debug("Torch CUDA current device query failed: %s", exc)
+
+        for idx in range(count):
+            cuda.set_device(idx)
+        return True
+    except Exception as exc:
+        logger.debug("Torch CUDA visible ordinal probe failed: %s", exc)
+        return False
+    finally:
+        if current_device is not None:
+            try:
+                cuda.set_device(current_device)
+            except Exception as exc:
+                logger.debug("Torch CUDA device restore failed: %s", exc)
+
+
 def _lookup_nvml_uuid_handle(pynvml, token: str):
     uuid_lookup = getattr(pynvml, "nvmlDeviceGetHandleByUUID", None)
     if uuid_lookup is None:
@@ -107,6 +149,14 @@ def _query_nvml() -> List[Dict[str, Any]]:
         if visible_tokens is None:
             visible_tokens = [str(idx) for idx in range(count)]
 
+        torch_visible_count = _torch_cuda_visible_count()
+        if torch_visible_count is None or torch_visible_count <= 0:
+            return []
+        if len(visible_tokens) != torch_visible_count:
+            return []
+        if not _torch_cuda_visible_ordinals_startable(torch_visible_count):
+            return []
+
         for token in visible_tokens:
             if token.isdigit() and int(token) >= count:
                 return []
@@ -307,7 +357,7 @@ def get_gpu_info() -> List[Dict[str, Any]]:
     when the underlying vendor identity is known.
 
     Tries ROCm first for HIP torch builds, otherwise NVML first (CUDA), then
-    ROCm SMI, then torch.cuda, then MPS data.
+    torch.cuda, then MPS data.
     """
     if getattr(getattr(torch, "version", None), "hip", None):
         try:
@@ -331,13 +381,6 @@ def get_gpu_info() -> List[Dict[str, Any]]:
     except Exception as exc:
         logger.debug("NVML info failed: %s", exc)
 
-    try:
-        infos = _query_rocm()
-        if infos:
-            return infos
-    except Exception as exc:
-        logger.debug("ROCm info failed: %s", exc)
-
     try:
         infos = _query_torch()
         if infos: