[action] [PR:829] [chassisd]: Enhance DPU auto-recovery robustness and race condition handling

[mssonicbld]

Description

Enhance the DPU auto-recovery state machine in chassisd to fix race conditions between concurrent operations (shutdown/reboot vs power-cycle), improve correctness of the default behavior, and refactor for maintainability.

Key changes:

• Read dpu_auto_recovery from DEVICE_METADATA instead of FEATURE table
• Default to disabled when the dpu_auto_recovery field is missing (safer default)
• Direct transition to ManualIntervention when auto-recovery is disabled (skip WaitForSelfRecovery)
• PCIe detach/reattach around power-cycle for clean hardware reset
• Check reset_limit before issuing power-cycle to avoid wasted resets
• Acquire state transition lock during power-cycle to prevent races with gnoi shutdown/reboot
• Re-check transition_in_progress after lock acquisition to close TOCTOU window
• Abort power-cycle if lock cannot be acquired (another operation in progress)
• Unify field name: use transition_in_progress (drop legacy state_transition_in_progress)
• Ignore transition_type == 'recovery' so chassisd does not suppress itself
• Refactor: extract _process_single_dpu_recovery to reduce update_dpu_recovery_state size
• Refactor: extract _enter_booting/_enter_ready/_enter_wait_for_self_recovery transition helpers
• Refactor: extract _should_handle_hw_offline for offline-state guard clarity
• Refactor: extract _handle_boot_timeout_expired shared by Booting and PowerCycle states

Depends on sonic-net/sonic-platform-common PR for get_module_state_transition/set_module_state_transition/clear_module_state_transition platform API.

Motivation and Context

Addresses review comments from PR #815. The DPU auto-recovery feature had several race conditions:

1. Power-cycle recovery vs planned shutdown: If gnoi initiated a shutdown while chassisd was about to power-cycle, both operations would conflict, potentially bricking the DPU.
2. TOCTOU between check and lock: A gap between checking if a transition was in progress and acquiring the lock allowed concurrent operations to slip through.
3. Default-enabled risk: Missing dpu_auto_recovery field defaulted to enabled, which could cause unexpected power-cycles on systems without explicit configuration.

How Has This Been Tested?

1. Unit tests: Comprehensive unit tests added in tests/test_dpu_auto_recovery.py covering all state transitions, race conditions, lock acquisition failures, and edge cases.

2. On-device testing on a 202605/master image (SmartSwitch), via the sonic-mgmt smartswitch/platform_tests suite, which exercises the DPU failure-mode / auto-recovery scenarios:

   • Control-plane-down → auto-recovery power-cycle → DPU returns to ready_status=true
   • Auto-recovery disabled → direct ManualIntervention (no power-cycle)
   • Power-cycle with lock contention (gnoi shutdown holding the transition lock)
   • TOCTOU re-check after lock acquisition
   • Self-suppression (transition_type == 'recovery' exclusion)
   • Unrecoverable state after reset_limit exceeded; admin-down/up clears it
   • NPU crash → bulk power-cycle of all admin-up DPUs

   Final results (202605/master): the end-to-end auto-recovery path was confirmed — chassisd power-cycled the failed DPU and it returned to ready_status=true. After the accompanying fixes (device DEVICE_METADATA.dpu_auto_recovery YANG leaf, the module_base recovery transition-timeout default, and a recovery-timeout bump), test_dpu_failure_modes finished 5 passed / 2 failed / 1 skipped (0 errored). Both remaining failures were environmental/test-harness — a databasedpu container inactive at test start, and a config-reload case that ran on pre-fix test code — not auto-recovery regressions.

3. Verified: No regressions on remaining DPUs (remained in Ready state throughout DPU0 destructive testing)

Signed-off-by: Sonic Build Admin sonicbld@microsoft.com

Additional Information (Optional)

[mssonicbld]

Original PR: #829

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).