agent: fix cgroups_notifier deadlock when cgroup is destroyed#350
Draft
snir911 wants to merge 1 commit into
Draft
agent: fix cgroups_notifier deadlock when cgroup is destroyed#350snir911 wants to merge 1 commit into
snir911 wants to merge 1 commit into
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
Member
|
Hi Snir ! Please fix this upstream and we will cherry-pick. |
When a cgroup is destroyed while kata-agent is processing container deletion (e.g., systemd deactivates the scope before signal_process completes), two issues occur that cause the deletion to hang: 1. **cgroups_notifier infinite loop**: The inotify event loop attempts to read from non-existent cgroup files (memory.events, cgroup.events) repeatedly, causing thousands of failed read attempts, log buffer overflow (1000+ dropped messages), CPU spinning, and overwhelming the tokio async runtime. 2. **signal_process RPC hang**: The get_pids() call fails with an error when trying to read from the destroyed cgroup, causing the RPC to return an error instead of completing successfully. This leaves CRI-O waiting indefinitely for the response. **Fixes:** 1. **notifier.rs**: Check if cgroup paths exist BEFORE attempting to read from them in the event loop. This allows clean exit when the cgroup is destroyed. 2. **rpc.rs**: Make get_pids() failure non-fatal in signal_process. If the cgroup is already destroyed, there are no processes left to signal anyway, so we can safely continue and return success. Together, these fixes allow container deletion to complete cleanly even when systemd races to clean up the cgroup before kata-agent finishes processing. Fixes: Container deletion hanging indefinitely Fixes: Node corruption due to leaked vxlan interfaces Signed-off-by: Snir Schreiber <ssheribe@redhat.com>
snir911
force-pushed
the
osc-release_hang_fix
branch
from
eaa56a7 to
89e9ebd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a cgroup is destroyed while the notifier event loop is processing events, the loop attempts to read from non-existent cgroup files (memory.events, cgroup.events) repeatedly. This causes:
The issue is triggered when systemd deactivates the cgroup scope before kata-agent finishes processing the signal_process RPC, which can happen due to timing races in systemd service dependencies.
Fix by checking if cgroup paths exist BEFORE attempting to read from them, rather than after. This allows the event loop to exit cleanly when the cgroup is destroyed, preventing the infinite loop.
Fixes: Container deletion hanging indefinitely
Fixes: Node corruption due to leaked vxlan interfaces
Co-Authored-By:: Claude Code noreply@anthropic.com