Skip to content

[Mellanox] Implement get_midplane_down_reason for DPU module - #28227

Open
chartsai-nvidia wants to merge 2 commits into
sonic-net:masterfrom
chartsai-nvidia:chartsai/pmon-hld-update
Open

[Mellanox] Implement get_midplane_down_reason for DPU module#28227
chartsai-nvidia wants to merge 2 commits into
sonic-net:masterfrom
chartsai-nvidia:chartsai/pmon-hld-update

Conversation

@chartsai-nvidia

@chartsai-nvidia chartsai-nvidia commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Why I did it

Mellanox DpuModule did not implement the pmon ModuleBase.get_midplane_down_reason()
interface, so it fell back to the generic base behavior instead of reporting the actual DPU
hardware reset cause. As a result, when a DPU's midplane went down there was no way for pmon to
surface whether it was caused by a power loss, a thermal shutdown, or a reset from the main board.

Work item tracking
  • Microsoft ADO (number only): N/A

How I did it

  • Implemented DpuModule.get_midplane_down_reason() in
    platform/mellanox/mlnx-platform-api/sonic_platform/module.py.
  • The implementation scans the existing reboot_cause_map (the per-DPU
    /var/run/hw-management/<dpu>/system/reset_* sysfs files) and returns the
    (ChassisBase.REBOOT_CAUSE_*, description) tuple of the first file that reads 1.
  • Falls back to (ChassisBase.REBOOT_CAUSE_NON_HARDWARE, '') when no reset-cause file is
    asserted, matching the existing get_reboot_cause() contract.
  • Logs a notice with the resolved reason so the cause is recoverable from syslog.
  • Reuses the existing ChassisBase.REBOOT_CAUSE_* constants rather than introducing new
    midplane-specific constants.

How to verify it

Run the Mellanox platform API unit tests:

cd platform/mellanox/mlnx-platform-api
python3 -m pytest tests/test_module.py::TestModule::test_dpu_module -v

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305
  • 202311
  • 202405
  • 202411

Description for the changelog

Implement get_midplane_down_reason for Mellanox DPU module

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

dgsudharsan
dgsudharsan previously approved these changes Jul 7, 2026
gpunathilell
gpunathilell previously approved these changes Jul 9, 2026
@liat-grozovik

Copy link
Copy Markdown
Collaborator

/azpw run Azure.sonic-buildimage

@mssonicbld

Copy link
Copy Markdown
Collaborator

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Build not found. Please close and reopen the PR or rebase your branch to trigger a new build.

liat-grozovik
liat-grozovik previously approved these changes Jul 15, 2026
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@chartsai-nvidia

Copy link
Copy Markdown
Contributor Author

/azpw retry

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1165748:

✅Stage Build:

  • Job mellanox: retried.

@chartsai-nvidia

Copy link
Copy Markdown
Contributor Author

/azpw retry

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1165748:

✅Stage Build:

  • Job mellanox: retried.

@chartsai-nvidia

Copy link
Copy Markdown
Contributor Author

/azpw retry

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1165748:

✅Stage Build:

  • Job mellanox: retried.

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request label(s) for branch(es): 202605, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202605: <test result>) in the Test result section as well in your PR description.

---Powered by SONiC BuildBot

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

add tests

Signed-off-by: Charles Tsai <chartsai@nvidia.com>
Signed-off-by: Charles Tsai <chartsai@nvidia.com>
@chartsai-nvidia
chartsai-nvidia force-pushed the chartsai/pmon-hld-update branch from b879f2c to 421be44 Compare July 24, 2026 23:30
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@chartsai-nvidia

Copy link
Copy Markdown
Contributor Author

/azpw ms_conflict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants