Skip to content

[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow… #2385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chartsai-nvidia wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow… #2385

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d37f07e
[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow…
chartsai-nvidia Jun 11, 2026
761566a
Update smartswitch-pmon.md
chartsai-nvidia Jul 16, 2026
f37e8cc
fix R618, R655, R657, R1098
chartsai-nvidia Jul 25, 2026
dbe1b48
fix R630 R659 R722
chartsai-nvidia Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions doc/smart-switch/pmon/smartswitch-pmon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
| 0.4 | 06/06/2024 | Ramesh Raghupathy | Added schema for DPU health-info and called out phase:1 and phase:2 activities for DPU health-info. Added key suffix to module reboot-cause to avoid key conflicts |
| 0.5 | 04/30/2025 | Gagan Punathil Ellath | Added Post Startup and Pre shutdown sections for DPU |
| 0.6 | 06/12/2025 | Gagan Punathil Ellath | DPU State Management Implementation |
| 0.7 | 06/11/2026 | Charles Tsai | Updated DPU reboot-cause and midplane-down reason |

@rameshraghupathy rameshraghupathy Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia What is the rationale for this change?

@chartsai-nvidia chartsai-nvidia Jul 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change updates the HLD to reflect the addition of the new reboot cause mechanism and get_midplane_down_reason() API. The version number is updated accordingly.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia The question was why are we changing the existing behavior of capturing and reporting the reboot-cause of DPUs even when the DPUs are down, which was done intentionally in the existing implementation to help debug DPU failure.

@gpunathilell gpunathilell Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rameshraghupathy This was already discussed in the platform meeting, we need to make sure that the behavior is aligned to the switch side, we do not want 2 reboot causes for each DPU reboots (->offline->Online) . That is why the get_midplane_down_reason() is being added newly so that the platform can still provide additional info to help debug DPU failure. Do you want this to be explicitly recorded?


## Definitions / Abbreviations

Expand Down Expand Up @@ -614,9 +615,10 @@ The smartswitch needs to know the reboot cause for the NPU and the DPUs.
```
#### DPU Reboot Cause
* The smartswitch needs to know the reboot cause for all the DPUs.
* The NPU hardware should be capable of providing the DPU reboot-cause even when the DPUs are dead.
* The reboot-cause can be captured by the platforms at the appropriate time. The reboot-cause should be reported only when the midplane of DPU transitions to online by calling the `get_reboot_cause` API.
* Each DPU SONiC publishes a `boot_id` (a fresh UUID generated per boot from `/proc/sys/kernel/random/boot_id`) into its `DPU_STATE` entry in CHASSIS_STATE_DB. The NPU chassisd compares the reported `boot_id` against the last `boot_id` it persisted; when they differ, the NPU chassisd calls `get_reboot_cause()`, writes a history record containing the cause and the `boot_id`, and then refreshes the `REBOOT_CAUSE` entries in CHASSIS_STATE_DB from those records. A changed `boot_id` can only become visible on the NPU once the DPU is reachable and able to write into the NPU's CHASSIS_STATE_DB, so the reported reboot-cause is naturally aligned with the midplane coming online.
* The get_reboot_cause will return the current reboot-cause of the module.
* For persistent storage of the DPU reboot-cause and reboot-cause-history files use the existing mechanism and host storage path under "/host/reboot-cause/module/dpux".
* Persistent DPU reboot-cause records are stored under `/host/reboot-cause/module/dpux/history/`, and each record includes the `boot_id`. The `previous-reboot-cause.json` symlink identifies the latest record and provides the persisted `boot_id` baseline. Unlike the NPU workflow, DPU capture does not use `module/dpux/reboot-cause.txt`.
* The storage and retrieval of the reboot-cause of the Switch and PDUs are shown in the sequence diagram

<p align="center"><img src="./images/dpu-reboot-seq.svg"></p>
Expand All @@ -625,12 +627,9 @@ The smartswitch needs to know the reboot cause for the NPU and the DPUs.
* The switch boots up. Determines the NPU reboot cause.
* Processes the previously stored NPU and DPU reboot-cause files and history files.
* A maximum of ten reboot-cause history entries per dpu will be persisted just like the npu.
* Updates the NPU reboot-cause into the StateDB and the DPU reboot-cause into the ChassisStateDB.
* The reboot-cause service (`process-reboot-cause`) updates the NPU reboot-cause in StateDB and rebuilds the DPU `REBOOT_CAUSE|DPUx|<time>` entries in ChassisStateDB from the per-DPU history files.
* The above process is a one-shot event on boot up.
* The module_db_update function in the NPU-PMON chassisd is an existing function constantly updating the operational status of the DPUs.
* This function looks for DPU operational status change events and when the DPUs come out of "offline" state, issues "get_reboot_cause" API to the platform.
* The dpu operational state transition from 'offline' to 'online' guarantees the reboot of a DPU.
* The platform code will extract the DPU reboot cause from the NPU hardware itself even when the DPU is not reachable.
* The DPU reboot cause will be mapped to one of the following existing reboot causes and returned back.
* REBOOT_CAUSE_POWER_LOSS = "Power Loss"
* REBOOT_CAUSE_THERMAL_OVERLOAD_CPU = "Thermal Overload: CPU"
Expand All @@ -651,12 +650,27 @@ return REBOOT_CAUSE_NON_HARDWARE + ', ' + 'kernel panic'
* The switch reboot use case will follow the same sequence.
* The pmon container restart should not affect this sequence as the states are persisted either in the DB or in the file system.

#### Runtime DPU reboot-cause capture in chassisd (`boot_id` trigger)

* A DPU reboot is detected when the NPU chassisd is notified of a change to the DPU's `boot_id` in CHASSIS_STATE_DB via a DPU_STATE subscription. The `boot_id` itself is generated by the DPU kernel at boot, before the midplane is necessarily reachable, but it can only become visible to the NPU once the midplane is up.

**DPU side (publisher).** The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE). The DPU publishes the value when its chassisd starts, and thereafter whenever the local `boot_id` is missing from `CHASSIS_STATE_DB` or differs from the value currently published there.

**NPU side (consumer / capture).** The NPU `chassisd` subscribes to `DPU_STATE|DPUx` in `CHASSIS_STATE_DB` and reacts whenever `boot_id` appears or changes. When such an event happens, chassisd checks whether the `boot_id` in the DB differs from the persisted one. If it differs, chassisd persists the `boot_id` along with the reboot cause. When no persisted `boot_id` exists — including the first upgrade to a version carrying this feature — or the persisted file is missing or cannot be parsed, it records a reboot cause and also adds a note to the record's `comment` field stating that no previous `boot_id` was known.

A DPU that has not published a `boot_id` produces no event, so nothing is recorded; likewise, restarting NPU-side chassisd or pmon while the DPU keeps running finds an unchanged `boot_id` and records nothing.

The history records under `/host/reboot-cause/module/dpux/history/` are the source of truth; the `REBOOT_CAUSE` entries in `CHASSIS_STATE_DB` are a projection rebuilt from them on every capture and by `process-reboot-cause` at NPU boot. A capture writes the record before refreshing the DB, and the persisted `boot_id` is a field of the record itself, so the baseline advances only after the record is in place. Records are identified by `boot_id`, so a re-attempted capture does not add a second record for the same boot.

Each history JSON file is updated atomically by writing the complete JSON to a temporary file in the same directory and renaming it into place. The `previous-reboot-cause.json` symlink is also replaced atomically, rather than removed and recreated.

#### Schema for REBOOT_CAUSE of DPUs on switch ChassisStateDB
```
Key: "REBOOT_CAUSE|DPU0|2024_06_06_09_31_18"

"REBOOT_CAUSE|DPU0|2024_06_06_09_31_18": {
"value": {
"boot_id": <boot-id>,
"cause": "Software causes (Reboot)",
"comment": "User issued 'reboot' command [User: admin, Time: Thu Jun 6 09:46:43 AM UTC 2024]",
"device": "DPU0",
Expand Down Expand Up @@ -692,6 +706,7 @@ dpu_data_plane_state: up refers to configuration downloaded, the pipeline stage
key: DPU_STATE|DPU0

"id": "1",
"boot_id": <boot-id>,
Comment thread
chartsai-nvidia marked this conversation as resolved.
”dpu_midplane_link_state”: "up"
“dpu_midplane_link_time": "timestamp",
"dpu_midplane_link_reason": "up_down_related string",
Expand All @@ -710,6 +725,12 @@ The DPU state management is implemented through a combination of classes that ha
* The switch updates the midplane state after querying the `is_midplane_reachable` platform API for the corresponding DPU. This is an universal implementation for all platforms
* The midplane state is updated at a specific frequency by the chassisd running on the switch (The frequency is once every 10 seconds - as per `CHASSIS_INFO_UPDATE_PERIOD_SECS` in chassisd)
* If the midplane state is down (which means that the DPU is no longer accessible through the midplane) This means that the state information which is present is no longer valid at the current instant, so the state information for control plane and data plane is set to 'down' but the data plane and the control plane reasons and the timestamps are retained as is for further debugging.
* On an `up -> down` midplane transition, chassisd records a reason string in `dpu_midplane_link_reason`:
- **Planned** — a DPU lifecycle operation is active, so the link loss is expected. Recorded as `Planned: '<transition_type>'`.
- **Unplanned** — otherwise, chassisd queries the platform through `get_midplane_down_reason()` and records `Unplanned: '<reason>'`, or `Unplanned: 'Unknown'` when the platform provides no reason.
* A DPU lifecycle operation counts as active only when all three hold: `transition_in_progress` is `"True"`, `transition_type` is one of the valid types, and `transition_start_time` has not exceeded the timeout for that `transition_type`. A stale value left behind by a completed, timed-out or failed operation therefore cannot classify a later failure as Planned.
* The fields are cleared when the operation completes, and are also cleared automatically once `transition_start_time` is older than the timeout for its `transition_type`.
* The resolved reason is persisted to `/host/reboot-cause/module/<dpu>/midplane-down-reason.txt` and reused while the midplane stays down, so the first observed reason survives a chassisd restart. It is removed when the midplane returns to `up`.

2. **DpuStateManagerTask Class - present in Chassisd on DPU - Dependent on chassisd being enabled on DPU**
* If there is a platform specific implementation of `get_dataplane_state` and `get_controlplane_state` implementation, Then these functions are called in polling mode by the chassisd running on DPU to update the relevant state information
Expand Down Expand Up @@ -790,6 +811,15 @@ def get_reboot_cause(self):
A tuple (string, string) where the first element is a string containing the cause of the previous reboot. This string must be one of the predefined strings in this class. If the first string is "REBOOT_CAUSE_HARDWARE_OTHER", the second string can be used to pass a description of the reboot cause.
```

#### Get midplane down reason

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Why do we need a new API? The get_reboot_cause itself should be sufficient. Why midplane state alone needs an API?

def get_midplane_down_reason(self):
```
Retrieves the cause of the midplane down reason

Returns:
A tuple (string, string) where the first element is a string containing the cause of midplane down reason. This string must be one of the REBOOT_CAUSE_* strings predefined in ChassisBase, the same list used by get_reboot_cause. If the first string is "REBOOT_CAUSE_HARDWARE_OTHER", the second string can be used to pass a description of the midplane down reason.
```

#### DPU_STATE Use Case
* This API can be be used by High Availability, Load Balancing, Debug, error recovery (reset, power cycle) and fault management logics. The exact HA use case will be updated in the HA document.

Expand Down Expand Up @@ -1045,7 +1075,7 @@ System status summary
* The "show system-health ..." is extended to include the DPU_STATE detail.
show system-health DPU \<dpu-index\> <font>**`Executed on the switch. This CLI is not available on the DPU.`**</font>
```
When the idex is "all" shows the detailed state of all DPUs
When the index is "all" shows the detailed state of all DPUs

Oper-Status definition:
Online : All states are up
Expand All @@ -1065,12 +1095,19 @@ DPU1 Online dpu_midplane_link_state up W
dpu_control_plane_state up Wed 20 Oct 2023 06:52:28 PM UTC
dpu_data_plane_state up Wed 20 Oct 2023 06:52:28 PM UTC

root@sonic:~#show system-health DPU 0
root@sonic:~#show system-health DPU DPU0

Name Oper-Status State-Detail State-Value Time Reason
DPU0 Offline dpu_midplane_link_state down Wed 20 Oct 2023 06:52:28 PM UTC PCIe link is down
DPU0 Offline dpu_midplane_link_state down Wed 20 Oct 2023 06:52:28 PM UTC Planned: 'shutdown'
dpu_control_plane_state down Wed 20 Oct 2023 06:52:28 PM UTC
dpu_data_plane_state down Wed 20 Oct 2023 06:52:28 PM UTC

root@sonic:~#show system-health DPU DPU0

Name Oper-Status State-Detail State-Value Time Reason
DPU0 Offline dpu_midplane_link_state down Wed Jun 10 06:39:57 PM UTC 2026 Unplanned: 'Thermal - Overload'
dpu_control_plane_state down Thu Jun 11 05:25:49 PM UTC 2026
dpu_data_plane_state down Thu Jun 11 05:25:49 PM UTC 2026
```
#### System health cli extended further as shown
* Detailed output from the switch can be obtained with the following CLI
Expand Down