Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ This file defines how coding agents should work in this repository.
- Treat custom `job_id` values as reserved from the moment startup begins; duplicate starts must fail before another controller can begin keep-alive work.
- Stop requests must not miss starting sessions; wait for startup to settle before returning `not found` or taking a stop-all snapshot.
- Stop-all may release independent sessions concurrently, but must not duplicate release work for `stopping` sessions and must keep deterministic additive result fields.
- Keep utilization backoff eco-safe: when telemetry is unavailable and `busy_threshold >= 0`, controllers should sleep instead of running keepalive compute. Only `busy_threshold=-1` is the explicit unconditional mode.
- Keep utilization backoff eco-safe: valid `busy_threshold` values are `-1` or `0..100`; when telemetry is unavailable and `busy_threshold >= 0`, controllers should sleep instead of running keepalive compute. Only `busy_threshold=-1` is the explicit unconditional mode.
- Avoid scattering platform-specific branching across unrelated modules; prefer one clear decision path then platform-specific controller classes.
- Preserve simple controller flow: global controller orchestrates per-GPU controllers; single-GPU controllers handle device-level keep/release loops.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Flags that matter:
- Blocking mode knobs:
- `--vram` (`1GiB`, `750MB`, or bare bytes like `1073741824`): how much memory to pin.
- `--interval` (positive seconds): sleep between keep-alive bursts.
- `--busy-threshold`: skip work when telemetry reports higher utilization or cannot report utilization; `-1` disables utilization backoff.
- `--busy-threshold`: `0..100` skips work when telemetry reports higher utilization or cannot report utilization; `-1` disables utilization backoff.
- `--gpu-ids`: target a non-negative subset; otherwise all visible GPUs are guarded.
- Service mode commands:
- `keep-gpu serve`: run local service (HTTP + dashboard).
Expand Down Expand Up @@ -106,7 +106,7 @@ with GlobalGPUController(gpu_ids=[0, 1], vram_to_keep="750MB", interval=90, busy
## What you get

- Battle-tested keep-alive loop built on PyTorch.
- NVML-based utilization monitoring (by way of `nvidia-ml-py`) to avoid hogging busy GPUs; optional ROCm SMI support by way of `pip install keep-gpu[rocm]`. If utilization is unavailable and `busy_threshold` is non-negative, KeepGPU sleeps for that cycle instead of running compute.
- NVML-based utilization monitoring (by way of `nvidia-ml-py`) to avoid hogging busy GPUs; optional ROCm SMI support by way of `pip install keep-gpu[rocm]`. Valid `busy_threshold` values are `-1` or `0..100`; if utilization is unavailable and the threshold is non-negative, KeepGPU sleeps for that cycle instead of running compute.
- CLI + API parity: same controllers power both code paths.
- Continuous docs + CI: mkdocs + mkdocstrings build in CI to keep guidance up to date.

Expand Down
4 changes: 2 additions & 2 deletions docs/concepts/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ CLI args ──▶ GlobalGPUController ──▶ [CudaGPUController rank=0]
- Calls `_monitor_utilization` (by way of NVML) to detect real activity.
4. If utilization exceeds `busy_threshold`, or if utilization is unavailable
while `busy_threshold` is non-negative, the worker just sleeps for one more
`interval`. Otherwise it runs a new batch of ops. `busy_threshold=-1` is the
explicit unconditional mode.
`interval`. Otherwise it runs a new batch of ops. Valid thresholds are `-1`
or `0..100`; `busy_threshold=-1` is the explicit unconditional mode.
5. When you call `release()` (or exit the context), every worker sets a stop
event, joins the thread, and clears the device cache. Release attempts every
worker and then raises a summary if any worker failed to stop.
Expand Down
4 changes: 2 additions & 2 deletions docs/guides/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ The dashboard provides:
| `--gpu-ids` | Comma-separated non-negative GPU IDs. Omit to use all visible devices. | all |
| `--vram` | Per-GPU memory target (`512MB`, `1GiB`, or bare bytes). | `1GiB` |
| `--interval` | Positive seconds between keep-alive cycles. | `300` |
| `--busy-threshold` / `--util-threshold` | Back off when utilization exceeds this value or telemetry is unavailable; `-1` disables utilization backoff. | `-1` |
| `--busy-threshold` / `--util-threshold` | `0..100` backs off when utilization exceeds this value or telemetry is unavailable; `-1` disables utilization backoff. | `-1` |

## Remote sessions

Expand All @@ -106,4 +106,4 @@ of `keep-gpu status`.
- **Start cannot reach service**: run `keep-gpu serve --host 127.0.0.1 --port 8765`.
- **Need to close background service**: run `keep-gpu stop --all` first, then `keep-gpu service-stop`. Use `keep-gpu service-stop --force` only for an unresponsive auto-started daemon; it still refuses to signal a PID that KeepGPU cannot verify as its own.
- **OOM during keep**: reduce `--vram` or free GPU memory before starting.
- **No utilization data**: on CUDA, ensure `nvidia-ml-py` works and `nvidia-smi` is available; on ROCm, check the optional `rocm-smi` extra. With non-negative `busy_threshold`, KeepGPU sleeps when utilization is unavailable. On Mac M series, utilization is expected to be `null`, so use `--busy-threshold -1` only when you intentionally want unconditional keepalive compute.
- **No utilization data**: on CUDA, ensure `nvidia-ml-py` works and `nvidia-smi` is available; on ROCm, check the optional `rocm-smi` extra. Valid `busy_threshold` values are `-1` or `0..100`. With non-negative `busy_threshold`, KeepGPU sleeps when utilization is unavailable. On Mac M series, utilization is expected to be `null`, so use `--busy-threshold -1` only when you intentionally want unconditional keepalive compute.
7 changes: 4 additions & 3 deletions docs/guides/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,9 +93,10 @@ The dashboard provides live telemetry, tracked session state, and start/stop con
CUDA and ROCm devices include memory and utilization when the platform APIs are
available. Mac M series devices report best-effort MPS memory counters and use
`null` for unsupported fields such as utilization.
When utilization is unavailable and `busy_threshold` is non-negative, controllers
sleep instead of running keepalive compute; `busy_threshold=-1` is the explicit
unconditional mode.
Valid `busy_threshold` values are `-1` or `0..100`. When utilization is
unavailable and `busy_threshold` is non-negative, controllers sleep instead of
running keepalive compute; `busy_threshold=-1` is the explicit unconditional
mode.
Stop controls show timed-out or failed releases instead of claiming success when
the backend keeps a session visible for follow-up cleanup. Retained session cards
show `Releasing` or `Release failed` with the backend error detail when present.
Expand Down
3 changes: 2 additions & 1 deletion docs/guides/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ with GlobalGPUController(
```

- Each `CudaGPUController` runs in its own thread.
- `busy_threshold` throttles the keep-alive loop when utilization spikes. When
- `busy_threshold` accepts `-1` or a percentage in `0..100`. Non-negative
thresholds throttle the keep-alive loop when utilization spikes. When
utilization telemetry is unavailable, non-negative thresholds sleep instead of
running compute; use `busy_threshold=-1` only for explicit unconditional
keepalive work.
Expand Down
40 changes: 40 additions & 0 deletions docs/plans/busy-threshold-range.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Busy-Threshold Range Plan

## Background

`busy_threshold` is documented as a utilization percentage, with `-1` reserved
as the explicit unconditional keepalive mode. The current validator rejects
values below `-1`, but accepts values above `100`. A value such as `101` makes a
fully utilized GPU eligible for keepalive compute without using the explicit
`-1` escape hatch.

## Goal

Keep utilization backoff eco-safe by accepting only `-1` or percentages in the
inclusive `0..100` range across the CLI, Python controllers, MCP/JSON-RPC, REST,
and dashboard inputs.

## Design

- Keep `-1` as the only unconditional keepalive sentinel.
- Accept integer thresholds from `0` through `100`, inclusive.
- Reject booleans, non-integers, values below `-1`, and values above `100`
through the centralized `validate_busy_threshold()` helper.
- Let existing CLI, controller, MCP, REST, and dashboard paths inherit the same
central rule.
- Update user docs and agent guidance to state the allowed range.

## Todo

- [x] Add failing tests for `101` rejection and `100` acceptance.
- [x] Implement the centralized validator change.
- [x] Add interface-level coverage where cheap to prove CLI/MCP inherit the
central rule.
- [x] Close local-review parity gap by updating dashboard validation, rebuilding
static assets, and adding REST rejection coverage.
- [x] Close local-review coercion gap by rejecting blank/boolean dashboard
threshold values and documenting the CLI help range.
- [x] Update `AGENTS.md`, README, and CLI/MCP docs with the allowed range.
- [x] Run targeted threshold tests, full tests, docs build, and pre-commit.
- [ ] Open a GitHub PR, run local subagent review, resolve all comments, then
squash merge.
4 changes: 2 additions & 2 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ These options apply when you run `keep-gpu` without subcommands.
| `--interval INTEGER` | seconds | Sleep duration between utilization checks and keep-alive batches. |
| `--gpu-ids TEXT` | comma-separated non-negative ints | Subset of GPUs to guard (for example, `0,2`). Omit to use all visible GPUs. |
| `--vram TEXT` | human size or bare bytes | Amount of memory each GPU controller allocates (`512MB`, `1GiB`, `1073741824`). |
| `--busy-threshold INTEGER` / `--util-threshold INTEGER` | percent | Back off when utilization is above this value or unavailable; `-1` disables utilization backoff. |
| `--busy-threshold INTEGER` / `--util-threshold INTEGER` | percent | `0..100` backs off when utilization is above this value or unavailable; `-1` disables utilization backoff. |
| `--threshold TEXT` | deprecated | Legacy alias: numeric values map to busy-threshold, size strings map to vram. |

## Service mode
Expand All @@ -46,7 +46,7 @@ Starts a keep session and returns immediately with `job_id`.
| `--gpu-ids` | all | Comma-separated GPU IDs. |
| `--vram` | `1GiB` | Per-GPU keep memory target. |
| `--interval` | `300` | Keep cycle interval in seconds. |
| `--busy-threshold` / `--util-threshold` | `-1` | Back off when utilization is above this value or telemetry is unavailable; `-1` disables utilization backoff. |
| `--busy-threshold` / `--util-threshold` | `-1` | `0..100` backs off when utilization is above this value or telemetry is unavailable; `-1` disables utilization backoff. |
| `--job-id` | auto | Optional custom id. Must be unique across active and starting sessions. |
| `--host` | `127.0.0.1` | Service host to contact. |
| `--port` | `8765` | Service port to contact. |
Expand Down
20 changes: 15 additions & 5 deletions src/keep_gpu/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from rich.console import Console

from keep_gpu.utilities.logger import setup_logger
from keep_gpu.utilities.session_config import validate_interval
from keep_gpu.utilities.session_config import validate_busy_threshold, validate_interval

DEFAULT_SERVICE_HOST = "127.0.0.1"
DEFAULT_SERVICE_PORT = 8765
Expand Down Expand Up @@ -236,6 +236,13 @@ def _validate_cli_interval(interval: int) -> int:
raise typer.BadParameter(str(exc)) from exc


def _validate_cli_busy_threshold(busy_threshold: int) -> int:
try:
return validate_busy_threshold(busy_threshold)
except ValueError as exc:
raise typer.BadParameter(str(exc)) from exc


def _service_base_url(host: str, port: int) -> str:
return f"http://{host}:{port}"

Expand Down Expand Up @@ -480,6 +487,7 @@ def _run_blocking(
console.print(
"[yellow]`--threshold` for utilization is deprecated; use `--busy-threshold`.[/yellow]"
)
busy_threshold = _validate_cli_busy_threshold(busy_threshold)

gpu_id_list = _parse_gpu_ids(gpu_ids)
if gpu_id_list is not None:
Expand Down Expand Up @@ -538,8 +546,9 @@ def main(
"--busy-threshold",
"--util-threshold",
help=(
"Back off when utilization is above this percent or telemetry is "
"unavailable; -1 disables utilization backoff (blocking mode)."
"Back off when utilization is above this 0..100 percent threshold "
"or telemetry is unavailable; -1 disables utilization backoff "
"(blocking mode)."
),
),
):
Expand Down Expand Up @@ -585,8 +594,8 @@ def start(
"--busy-threshold",
"--util-threshold",
help=(
"Back off when utilization is above this percent or telemetry is "
"unavailable; -1 disables utilization backoff."
"Back off when utilization is above this 0..100 percent threshold "
"or telemetry is unavailable; -1 disables utilization backoff."
),
),
job_id: Optional[str] = typer.Option(
Expand Down Expand Up @@ -616,6 +625,7 @@ def start(
"""
try:
interval = _validate_cli_interval(interval)
busy_threshold = _validate_cli_busy_threshold(busy_threshold)
parsed_gpu_ids = _parse_gpu_ids(gpu_ids)
auto_started = _ensure_service_running(host, port, auto_start=auto_start)
result = _rpc_call(
Expand Down
Loading
Loading