[module_base]: Add 'recovery' transition type to _TRANSITION_TIMEOUT_DEFAULTS - #709
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…DEFAULTS
chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock via set_module_state_transition(name, 'recovery'). Since 'recovery' was absent from _TRANSITION_TIMEOUT_DEFAULTS, the transition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped, blocking DPU recovery.
Add "recovery": 600 (10 min) so the recovery lock is accepted and chassisd can power-cycle an unresponsive DPU.
Verified on-device: with the fix, a control-plane-down DPU was power-cycled (reboot-cause "Switch rebooted DPU") and recovered to ready_status=true.
Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@vvolam LGTM Consider adding a focused unit test that verifies the new transition type is accepted: assert module.set_module_state_transition("DPU0", "recovery") is True |
…accepted Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
@rameshraghupathy Added a new testcase for a happy path too. Please review again |
|
@vvolam Thanks, the new happy-path test covers my earlier comment. LGTM |
vvolam
added
Request for 202605 Branch
Tested for 202605 branch
|
Please add results or the description of what kind of tests are run from 202605 |
@vaibhavhd Tests ran on both master and 202605 image on a smartswitch. Updated the PR description with the same. |
|
Cherry-pick PR to 202605: #726 |
mssonicbld
added
Included in 202605 Branch
and removed
Created PR to 202605 Branch
labels
5 participants
Description
Add a
"recovery": 600(10 min) entry toModuleBase._TRANSITION_TIMEOUT_DEFAULTS.set_module_state_transition()validates the requestedtransition_typeagainst the keys of_TRANSITION_TIMEOUT_DEFAULTS. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock viaset_module_state_transition(name, 'recovery'); because'recovery'was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery.Motivation and Context
The
'recovery'transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in_TRANSITION_TIMEOUT_DEFAULTS, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829.recoveryhas noplatform.jsonoverride (onlystartup/shutdown/reboot/halt_servicesare overridable in_load_transition_timeouts()), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared infinally; 600 s is ample headroom.How Has This Been Tested?
tests/module_base_test.pyto expect the new key.python3 -m pytest tests/module_base_test.py→ 115 passed.ready_status=true.Additional Information (Optional)
Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the
'recovery'transition this default enables.