elasticmanager: disable the cpu threshold alert, it cannot work - #64
Open
ehb54 wants to merge 1 commit into
Open
elasticmanager: disable the cpu threshold alert, it cannot work#64ehb54 wants to merge 1 commit into
ehb54 wants to merge 1 commit into
Conversation
The first real between frame gap produced a false positive. Slot 2 was warned about at 18:08 and recovered on its own at 19:39, having been perfectly healthy the whole time. The gap with no waxsis container ran 1h41m; the alert fired 25 minutes in and stood for 1h31m. The threshold was sized expecting gaps around ten minutes. This one was ten times that. More importantly it is not a tuning problem. During the gap slot 2 read 2%. Genuinely idle slot 1 reads 2% on every sample in the file. A healthy job between frames and an abandoned instance are numerically identical from the instance, because while the job does its local work in the saxsafold container the instance really is doing nothing, which is also what a stranded instance is doing. There is no signal there to threshold, and raising consecutive past this gap only trades a false positive for a longer blind window, with no safe value since gap length is set by work happening elsewhere. Removes probe:min_pct so nothing is raised. Sampling continues and em_probe.log keeps accumulating, which is the second time that data has corrected a guess. probe_check() stays, inert without min_pct, since its counter and one-alert-per-episode handling are reusable once the trigger is a signal that actually separates the two cases: the acquiring pid, carried in the tag, which is a fact rather than an inference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first real gap produced a false positive
The job was healthy throughout. It resumed on its own and climbed back to 83%.
I sized the threshold expecting gaps around ten minutes. The first one observed was ten times that.
Why this is not a tuning problem
During the gap slot 2 read 2%. Genuinely idle slot 1 reads 2% on every single sample in the file. A healthy job between frames and an abandoned instance are not merely similar at the instance level, they are numerically identical.
That is not a threshold that needs raising. There is no signal there to threshold. While the job does its local work in the saxsafold container, the instance is doing nothing, and nothing is exactly what a stranded instance is also doing.
Raising
consecutivepast this gap would not fix it either: the gap length is set by whatever the job is doing elsewhere, so the next one could be longer, and every increase trades a false positive for a longer blind window. There is no value that is safe.Change
Remove
probe:min_pctso no alert is raised. Sampling continues unchanged, andem_probe.logkeeps accumulating — the data is doing its job, this is the second time it has corrected a guess of mine.probe_check()stays inem_openstack.php. It is inert withoutmin_pct, and its counter and one-alert-per-episode machinery are reusable once the trigger is a signal that actually distinguishes the two cases.What would work
Liveness has to come from the client, not the instance.
bin/em.phpin saxsafold already putsgethostname()in the tag; addinggetmypid()makes "is this slot stranded" into "does that pid still exist in that container", which is a fact rather than an inference, and is true regardless of whether the job happens to be using the instance at that moment.Load then becomes useful corroboration rather than the detector: a dead pid and a floor reading is unambiguous.
Restart impact
em_config.jsonisconfigscope.🤖 Generated with Claude Code