[test_check_hw_mgmt_sysfs] Skip sensors checking in old version - #15
Open
JibinBao wants to merge 3 commits into
Open
[test_check_hw_mgmt_sysfs] Skip sensors checking in old version #15JibinBao wants to merge 3 commits into
JibinBao wants to merge 3 commits into
Conversation
JibinBao
pushed a commit
that referenced
this pull request
Jul 30, 2026
…et#25887) <!-- Please make sure you've read and understood our contributing guidelines; https://github.com/sonic-net/SONiC/blob/gh-pages/CONTRIBUTING.md Please provide following information to help code review process a bit easier: --> ### Description of PR <!-- - Please include a summary of the change and which issue is fixed. - Please also include relevant motivation and context. Where should reviewer start? background context? - List any dependencies that are required for this change. --> Summary: `test_ssh_stress` opens many SSH sessions to stress the DUT, then checks that CPU and memory settle back down once the load stops. That final recovery check (`run_post_test_system_check`) is flaky on the VPP KVM baseline -- it reports a failure even when the DUT is healthy: ``` Failed: CPU usage increased by more than 20 points during test and did not reduce from raised value Initial Value: 0.25252525252525254, Max value: 0.47, Post-Test Value: 0.47 ``` Why it happens -- two small issues that combine: 1. **The measurement is misleading.** `get_system_stats()` calls plain `vmstat`, which reports CPU usage *averaged since boot*. During a short test that average hardly moves, so the peak recorded during the test (`max_cpu`) and the reading taken afterwards (`post_cpu`) come out almost equal. 2. **The comparison is unreliable.** The check asserted `max_cpu - post_cpu > 0.1` (i.e. "did CPU drop a bit from its peak?"). Because those two numbers are nearly equal, `post_cpu` often lands *at or above* `max_cpu` -- note `Max == Post == 0.47` above -- so the difference is `<= 0` and the test fails, even though CPU never actually stayed high. Fixes the failures tracked in ADO PBI 38650838. ### Type of change <!-- - Fill x for your type of change. - e.g. - [x] Bug fix --> - [x] Bug fix - [ ] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [x] Test case improvement ### Back port request <!-- Only check a release or feature branch when the PR links a GitHub issue or ADO work item above. The linked tracker should explain the failure in detail, including whether it is a day-one issue or a regression, the affected branch/image/platform/test, and why this branch needs the fix. Backport or cherry-pick requests without a linked issue/work item may not be favored. --> - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 - [ ] 202512 - [ ] 202605 Tracking issue/work item for backport/cherry-pick request: ADO 38650838 Failure type: day-one issue ### Approach #### What is the motivation for this PR? The check is meant to catch a real problem -- CPU or memory staying high after the test -- but as written it also fires on healthy runs, so the test is unreliable on the VPP baseline. #### How did you do it? Two changes, both about how usage is measured and interpreted (the stress test itself is unchanged): 1. **Measure the current load, not the since-boot average.** `get_system_stats()` now runs `vmstat 1 2` and reads the 1-second sample, so `max_cpu` and the post-test readings reflect the load right now. 2. **Compare against the starting baseline, not the peak.** After the test, take a few readings while usage settles, keep the lowest one, and require it to be within 20 points of the pre-test baseline. A process that stays busy still fails the check; a normal spike that settles (even with a noisy first reading) passes. The 20-point sensitivity is unchanged. #### How did you verify/test it? Elastictest, `ssh/test_ssh_stress.py` repeated 20x on `vms-kvm-vpp-t1-lag` (t1-lag-vpp, VPP): - **Before (master -- reproduces the bug):** https://elastictest.org/scheduler/testplan/6a47b0475f6e97252deeb62a -- run #15 fails in the test body with `CPU usage increased by more than 20 points ... did not reduce` (`Initial 0.2525, Max 0.47, Post 0.47`). - **After (this PR -- fixed):** https://elastictest.org/scheduler/testplan/6a47f6fda577b9de602ff366 -- 20/20 with no CPU-recovery failure. (The 4 non-blocking errors there are an unrelated `memory_utilization` teardown alarm on frr_bgp/zebra memory that also occurs on master; it is not this test's check and is out of scope for this PR.) Also: - Unit-checked the two functions with a mock DUT: `vmstat 1 2` parsing reads the current sample; a normal spike -> recover passes; a process that stays busy still fails. - `flake8 --max-line-length=120` clean; `py_compile` clean. #### Any platform specific information? On `master`, `test_ssh_stress` runs only on `asic_type == 'vpp'`, so this primarily affects the VPP KVM baseline. #### Supported testbed topology if it's a new test case? N/A (existing test, not a new test case). ### Documentation <!-- (If it's a new feature, new test case) Did you update documentation/Wiki relevant to your implementation? Link to the wiki page? --> No documentation changes. Signed-off-by: Augustine Lee <augustinelee@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
In mellanox device and the versions of 201911and 202012, skip sensors checking in test of test_check_hw_mgmt_sysfs.
Summary:
Fixes # (issue)
Type of change
Back port request
Approach
What is the motivation for this PR?
Skip sensor checking for the versions of 201911 and 202012
How did you do it?
When version is 201911, 202012, skip sensors checking
How did you verify/test it?
Run test of test_check_hw_mgmt_sysfs
Any platform specific information?
Any
Supported testbed topology if it's a new test case?
Any
Documentation