[controllers] fix: defer keep allocation while busy

[Wangmerlyn]

Summary

• defer CUDA, ROCm, and MPS keep tensor allocation when non-negative busy-threshold backoff says the device is busy or telemetry is unavailable
• preserve busy_threshold=-1 as the explicit unconditional allocation/compute mode
• update AGENTS.md, architecture, CLI/API/MCP/Python docs, and add the implementation plan
• add no-GPU regression coverage for busy deferral, busy-to-idle allocation, unconditional mode, and ROCm retry accounting

Verification

• PYTHONPATH=$PWD/src pytest tests/rocm_controller/test_rocm_backoff.py -q -> 6 passed
• PYTHONPATH=$PWD/src pytest tests/cuda_controller/test_throttle.py tests/rocm_controller/test_rocm_backoff.py tests/macm_controller/test_macm_backoff.py -q -> 17 passed, 1 skipped
• PYTHONPATH=$PWD/src pytest tests/cuda_controller/test_throttle.py tests/rocm_controller/test_rocm_backoff.py tests/macm_controller/test_macm_backoff.py tests/global_controller/global_keep_test.py tests/single_gpu_controller/test_release_contract.py -q -> 27 passed, 2 skipped
• PYTHONPATH=$PWD/src pytest tests -q -> 263 passed, 11 skipped
• PYTHONPATH=$PWD/src mkdocs build -> passed with existing Material/nav warnings
• pre-commit run --all-files -> passed
• git diff --check && git diff --cached --check -> clean

Local review

• Local subagent review resolved the CLI docs parity finding and ROCm retry regression suggestion. Final review reported no blockers and ready to commit/open PR.

Summary by CodeRabbit

• Bug Fixes

  • Keep-alive work now waits before allocating or running when GPU utilization is unavailable or above the configured threshold.
  • Busy thresholds now behave more consistently across supported platforms, including the default and -1 unconditional mode.

• Documentation

  • Clarified busy_threshold behavior in architecture, CLI, API, and user guide docs.
  • Added a new planning note for deferred allocation while busy.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e01e0c60-7159-4486-9c8f-ae3e36d141d3

📥 Commits

Reviewing files that changed from the base of the PR and between 8d726fa and d201648.

📒 Files selected for processing (14)

• AGENTS.md
• docs/concepts/architecture.md
• docs/guides/cli.md
• docs/guides/mcp.md
• docs/guides/python.md
• docs/plans/defer-allocation-while-busy.md
• docs/reference/api.md
• docs/reference/cli.md
• src/keep_gpu/single_gpu_controller/cuda_gpu_controller.py
• src/keep_gpu/single_gpu_controller/macm_gpu_controller.py
• src/keep_gpu/single_gpu_controller/rocm_gpu_controller.py
• tests/cuda_controller/test_throttle.py
• tests/macm_controller/test_macm_backoff.py
• tests/rocm_controller/test_rocm_backoff.py

---

📝 Walkthrough

Walkthrough

All three GPU controllers (CUDA, ROCm, MPS) now query utilization and run _should_run_batch before allocating the keep tensor, sleeping for interval when busy or telemetry is unavailable. Documentation across guides, references, and architecture docs is updated accordingly, and a new plan document is added.

Changes

Defer Keep-Tensor Allocation While GPU Is Busy

Layer / File(s)	Summary
Pre-allocation utilization gating in all three controllers src/keep_gpu/single_gpu_controller/cuda_gpu_controller.py, src/keep_gpu/single_gpu_controller/rocm_gpu_controller.py, src/keep_gpu/single_gpu_controller/macm_gpu_controller.py	Each controller's _keep_loop now checks _should_run_batch before allocating the keep tensor; when the GPU is busy or telemetry is unavailable, the loop sleeps for interval and retries instead of allocating immediately. busy_threshold=-1 bypasses this check.
Behavioral tests across CUDA, ROCm, and MPS tests/cuda_controller/test_throttle.py, tests/rocm_controller/test_rocm_backoff.py, tests/macm_controller/test_macm_backoff.py	_StopAfterOneWait / _StopAfterWaits helper classes enable deterministic loop termination. New tests cover: busy-only deferral, busy-then-idle allocation, retry-budget preservation under repeated deferrals, and negative-threshold unconditional allocation.
Documentation and guidelines AGENTS.md, docs/plans/defer-allocation-while-busy.md, docs/concepts/architecture.md, docs/reference/api.md, docs/reference/cli.md, docs/guides/cli.md, docs/guides/mcp.md, docs/guides/python.md	All busy_threshold descriptions updated to state that deferral occurs before keep-tensor allocation (not just before compute). New plan document added with background, solution checklist, and verification commands.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• Wangmerlyn/KeepGPU#56: Introduced RocmGPUController._keep_loop with busy_threshold-based sleep behavior, the same function this PR modifies to defer tensor allocation.
• Wangmerlyn/KeepGPU#70: Introduced MacMGPUController and its MPS keep-loop, which this PR extends with pre-allocation _should_run_batch gating.
• Wangmerlyn/KeepGPU#75: Implemented telemetry-unknown throttling via _should_run_batch across controllers, the same predicate this PR moves earlier in the keep-loop to gate allocation.

Poem

🐇 Hop hop, the GPU's too busy today,
So I'll nap in my burrow and keep tensors at bay.
No allocate-first when the utilization's high—
We wait for the idle before reaching the sky.
busy_threshold=-1? Then charge right through!
Otherwise, little rabbit sleeps the interval due. 🌙

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 2.78% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: deferring controller keep allocation while busy.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/defer-allocation-while-busy

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request implements deferred keep-tensor allocation across CUDA, ROCm, and MPS (MacM) controllers, ensuring that VRAM is not allocated when the GPU is busy or telemetry is unavailable. The changes update the keep loops in each controller to check utilization backoff before allocation, add comprehensive unit tests to verify this behavior, and update all relevant documentation. As there are no review comments, I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.