[controllers] fix: surface keep worker startup failures - #101
Conversation
|
Warning Review limit reached
More reviews will be available in 53 minutes and 52 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds a synchronous startup handshake to ChangesSynchronous keep() startup handshake
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a startup handshake mechanism for both CUDA and ROCm single-GPU controllers to ensure that keep() does not report success if backend startup setup fails. It introduces synchronous propagation of fatal startup errors (such as device-selection errors) to prevent the service layer from registering false active sessions. The changes include updates to documentation, implementation of the handshake using threading events in the controllers, and comprehensive unit and integration tests to verify the new behavior. There are no review comments, so I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/keep_gpu/single_gpu_controller/cuda_gpu_controller.py`:
- Line 204: The startup exception handler in CUDA_GPUController is intentionally
broad, but Ruff now flags it with BLE001. Add a targeted suppression comment on
the except Exception as exc block in the startup path, and include a short
rationale that this catch is deliberate for startup fallback/error handling.
Keep the suppression local to the specific handler so pre-commit stops reporting
the warning without masking other broad exceptions.
In `@src/keep_gpu/single_gpu_controller/rocm_gpu_controller.py`:
- Line 113: Add local Ruff suppressions for the intentional broad exception
handlers in the ROCm GPU controller: the best-effort cleanup catch in the
cleanup path and the backend startup failure propagation catch should each be
marked with the appropriate BLE001 noqa comment. Update the affected `except
Exception as exc` blocks in `RocmGpuController` so the intent is explicit and
Ruff no longer flags these deliberate broad catches.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 26dca3f2-d398-4faf-b787-5ea6d266c324
📒 Files selected for processing (10)
AGENTS.mddocs/concepts/architecture.mddocs/guides/python.mddocs/plans/controller-startup-failures.mddocs/reference/api.mdsrc/keep_gpu/single_gpu_controller/cuda_gpu_controller.pysrc/keep_gpu/single_gpu_controller/rocm_gpu_controller.pytests/cuda_controller/test_keep_and_release.pytests/mcp/test_server.pytests/rocm_controller/test_rocm_backoff.py
1 participant
Summary
keep()only reports success after fatal backend setup succeeds.Review loop
Verification
keep()did not raise and service returned a job id.3 passed3 passedPYTHONPATH=$PWD/src pytest tests/cuda_controller tests/rocm_controller tests/mcp/test_server.py -q-> 100 passed, 5 skippedPYTHONPATH=$PWD/src pytest tests -q-> 312 passed, 11 skippedPYTHONPATH=$PWD/src mkdocs build-> exit 0, known MkDocs/Material and unnav'd plan noticespre-commit run --all-files-> passedSummary by CodeRabbit
Bug Fixes
Documentation