Skip to content

elasticmanager: disable the cpu threshold alert, it cannot work - #64

Open
ehb54 wants to merge 1 commit into
php7designerfrom
em-disable-cpu-alert
Open

elasticmanager: disable the cpu threshold alert, it cannot work#64
ehb54 wants to merge 1 commit into
php7designerfrom
em-disable-cpu-alert

Conversation

@ehb54

@ehb54 ehb54 commented Jul 27, 2026

Copy link
Copy Markdown
Owner

The first real gap produced a false positive

2026-07-27 18:08:31 - WARNING: slot 2 looks idle: 24% cpu, under 50% for 3 consecutive probes ...
2026-07-27 19:39:28 - probe: slot 2 back above 50% cpu (52%)

The job was healthy throughout. It resumed on its own and climbed back to 83%.

gap with no waxsis container 1h 41m (17:43 to 19:24)
alert fired 25 min into the gap
time spent alerting on a healthy job 1h 31m
lowest reading during sustained work 72%
reading during the gap decayed to 2%, held there for 5 samples

I sized the threshold expecting gaps around ten minutes. The first one observed was ten times that.

Why this is not a tuning problem

During the gap slot 2 read 2%. Genuinely idle slot 1 reads 2% on every single sample in the file. A healthy job between frames and an abandoned instance are not merely similar at the instance level, they are numerically identical.

That is not a threshold that needs raising. There is no signal there to threshold. While the job does its local work in the saxsafold container, the instance is doing nothing, and nothing is exactly what a stranded instance is also doing.

Raising consecutive past this gap would not fix it either: the gap length is set by whatever the job is doing elsewhere, so the next one could be longer, and every increase trades a false positive for a longer blind window. There is no value that is safe.

Change

Remove probe:min_pct so no alert is raised. Sampling continues unchanged, and em_probe.log keeps accumulating — the data is doing its job, this is the second time it has corrected a guess of mine.

probe_check() stays in em_openstack.php. It is inert without min_pct, and its counter and one-alert-per-episode machinery are reusable once the trigger is a signal that actually distinguishes the two cases.

What would work

Liveness has to come from the client, not the instance. bin/em.php in saxsafold already puts gethostname() in the tag; adding getmypid() makes "is this slot stranded" into "does that pid still exist in that container", which is a fact rather than an inference, and is true regardless of whether the job happens to be using the instance at that moment.

Load then becomes useful corroboration rather than the detector: a dead pid and a floor reading is unambiguous.

Restart impact

em_config.json is config scope.

php em_install.php --fetch --install --restart

🤖 Generated with Claude Code

The first real between frame gap produced a false positive. Slot 2 was warned
about at 18:08 and recovered on its own at 19:39, having been perfectly healthy
the whole time. The gap with no waxsis container ran 1h41m; the alert fired 25
minutes in and stood for 1h31m.

The threshold was sized expecting gaps around ten minutes. This one was ten
times that.

More importantly it is not a tuning problem. During the gap slot 2 read 2%.
Genuinely idle slot 1 reads 2% on every sample in the file. A healthy job
between frames and an abandoned instance are numerically identical from the
instance, because while the job does its local work in the saxsafold container
the instance really is doing nothing, which is also what a stranded instance is
doing. There is no signal there to threshold, and raising consecutive past this
gap only trades a false positive for a longer blind window, with no safe value
since gap length is set by work happening elsewhere.

Removes probe:min_pct so nothing is raised. Sampling continues and em_probe.log
keeps accumulating, which is the second time that data has corrected a guess.
probe_check() stays, inert without min_pct, since its counter and
one-alert-per-episode handling are reusable once the trigger is a signal that
actually separates the two cases: the acquiring pid, carried in the tag, which
is a fact rather than an inference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant