[action] [PR:26199] fix(ansible): serialize module resolution - #1355
Merged
Conversation
### Description of PR
Summary:
Serialize pytest-ansible module discovery to prevent concurrent SONiC neighbor initialization from temporarily reporting existing custom modules such as `sonic_basic_facts` as missing.
Fixes # (issue)
ADO: 38183684
### Type of change
- [x] Bug fix
- [x] Testbed and Framework(new/improvement)
- [ ] New Test case
- [ ] Skipped for non-supported platforms
- [ ] Test case improvement
### Back port request
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [x] 202605
Tracking issue/work item for backport/cherry-pick request: 38183684
Failure type: regression
### Approach
#### What is the motivation for this PR?
PR KVM runs intermittently fail while constructing SONiC neighbor hosts in parallel. `AnsibleHostBase.__getattr__` asks pytest-ansible whether `sonic_basic_facts` exists, but pytest-ansible resolves modules through mutable plugin-loader caches shared by all threads in the current Python process. A concurrent cache update can return a false unresolved result, causing an `AttributeError` before the test body runs. The issue is active on master and 202605, and most affected plans pass on retry after the loader and facts caches are warm.
#### How did you do it?
Added a process-local re-entrant lock around the two Ansible module-resolution operations in `AnsibleHostBase`: checking whether the dynamic module exists and resolving the executable module wrapper. Registered the lock with `os.register_at_fork` so a forked child reinitializes its copy instead of inheriting a locked guard from a vanished parent thread. Remote Ansible execution remains outside the lock, preserving parallel multi-host operations.
#### How did you verify/test it?
Reviewed a representative Elastictest failure where one of four SONiC neighbors reported `sonic_basic_facts` missing while the other three resolved it, followed by a successful module retry. The failing `nbrhosts` path uses `SafeThreadPoolExecutor`, which is backed by `multiprocessing.pool.ThreadPool`; xdist is not the concurrency boundary for the four neighbor constructors. Validated fork safety by holding the lock in a parent thread, forking a child, and confirming the child could acquire the reinitialized lock.
#### Any platform specific information?
The failure was observed on KVM VS `t0-64-32` runs using SONiC neighbors. The fix is in the shared Ansible host wrapper and is not ASIC-specific.
#### Supported testbed topology if it's a new test case?
Not a new test case.
### Documentation
Not applicable.
Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
Collaborator
Author
|
Original PR: sonic-net/sonic-mgmt#26199 |
Collaborator
Author
|
/azp run |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. |
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
Serialize pytest-ansible module discovery to prevent concurrent SONiC neighbor initialization from temporarily reporting existing custom modules such as
sonic_basic_factsas missing.Fixes # (issue)
ADO: 38183684
Type of change
Back port request
Tracking issue/work item for backport/cherry-pick request: 38183684
Failure type: regression
Approach
What is the motivation for this PR?
PR KVM runs intermittently fail while constructing SONiC neighbor hosts in parallel.
AnsibleHostBase.__getattr__asks pytest-ansible whethersonic_basic_factsexists, but pytest-ansible resolves modules through mutable plugin-loader caches shared by all threads in the current Python process. A concurrent cache update can return a false unresolved result, causing anAttributeErrorbefore the test body runs. The issue is active on master and 202605, and most affected plans pass on retry after the loader and facts caches are warm.How did you do it?
Added a process-local re-entrant lock around the two Ansible module-resolution operations in
AnsibleHostBase: checking whether the dynamic module exists and resolving the executable module wrapper. Registered the lock withos.register_at_forkso a forked child reinitializes its copy instead of inheriting a locked guard from a vanished parent thread. Remote Ansible execution remains outside the lock, preserving parallel multi-host operations.How did you verify/test it?
Reviewed a representative Elastictest failure where one of four SONiC neighbors reported
sonic_basic_factsmissing while the other three resolved it, followed by a successful module retry. The failingnbrhostspath usesSafeThreadPoolExecutor, which is backed bymultiprocessing.pool.ThreadPool; xdist is not the concurrency boundary for the four neighbor constructors. Validated fork safety by holding the lock in a parent thread, forking a child, and confirming the child could acquire the reinitialized lock.Any platform specific information?
The failure was observed on KVM VS
t0-64-32runs using SONiC neighbors. The fix is in the shared Ansible host wrapper and is not ASIC-specific.Supported testbed topology if it's a new test case?
Not a new test case.
Documentation
Not applicable.
Signed-off-by: Sonic Build Admin sonicbld@microsoft.com