Skip to content

Address racey SAI PHY Serdes object causing bad syncd setup #1945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
justin-wong-ce wants to merge 11 commits into
base: master
Choose a base branch
from
+419 −22
Open

Address racey SAI PHY Serdes object causing bad syncd setup #1945

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e4b680b
Address racey SAI PHY Serdes object causing bad syncd setup
justin-wong-ce Jun 18, 2026
d67380d
retriving -> retrieving
justin-wong-ce Jun 18, 2026
0d31fcf
Change to more deterministic handling of SAI_STATUS_OBJECT_IN_USE
justin-wong-ce Jun 18, 2026
117aa72
Cast revised handling to other code path
justin-wong-ce Jun 18, 2026
b2eac07
Further casts
justin-wong-ce Jun 18, 2026
220a847
Fix whiteline and typo
justin-wong-ce Jun 18, 2026
2713748
Switch back to retries but with 5 retries
justin-wong-ce Jul 4, 2026
faf8c21
bump tries count, make code more easy to understand
justin-wong-ce Jul 7, 2026
27efe15
Add unit test for retry logic
justin-wong-ce Jul 14, 2026
a33c0e8
added comments for new test cases
justin-wong-ce Jul 14, 2026
8aa4ae9
fix typo
justin-wong-ce Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 68 additions & 22 deletions syncd/FlexCounter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
#include <vector>
#include <string>
#include <mutex>
#include <thread>

#include "FlexCounter.h"
#include "VidManager.h"
Expand Down Expand Up @@ -2229,16 +2230,30 @@ class PortPhySerdesAttrContext : public AttrContext<sai_port_serdes_attr_t, Port

sai_attribute_t attr;
attr.id = SAI_PORT_SERDES_ATTR_PORT_ID;
sai_status_t status = Base::m_vendorSai->get(Base::m_objectType, port_serdes_rid, 1, &attr);

if (status == SAI_STATUS_SUCCESS)
for (uint32_t tries = 1; tries <= 5; tries++)
{
port_rid = attr.value.oid;
return true;
}
sai_status_t status = Base::m_vendorSai->get(Base::m_objectType, port_serdes_rid, 1, &attr);
Comment on lines +2234 to +2236

@prgeor prgeor Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justin-wong-ce ?

@justin-wong-ce justin-wong-ce Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to update the description after applying changes from #1945 (comment)


SWSS_LOG_ERROR("PORT_PHY_SERDES_ATTR: Failed to get port RID for port serdes RID:0x%" PRIx64 ", status:%d",
port_serdes_rid, status);
if (status == SAI_STATUS_SUCCESS)
{
port_rid = attr.value.oid;
return true;
}
else if (status == SAI_STATUS_OBJECT_IN_USE and tries < 5)
{
// SAI object is busy - retry in 10ms
SWSS_LOG_WARN("PORT_PHY_SERDES_ATTR: SAI object in use, retry getting port RID for port serdes RID:0x%" PRIx64 "...",
port_serdes_rid);
std::this_thread::sleep_for(chrono::milliseconds(10));

@prgeor prgeor Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justin-wong-ce this is risky to sleep assuming we have say 256 ports which will result in 256 * 10 * 5 = 12.8 secs.

@justin-wong-ce justin-wong-ce Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2.5% chance is the probability of it happening to 1 port on a config-reload or reboot. Of course yes, there is the chance of it happening to all ports at the same time and I agree this is not an optimal solution.

}
else
{
SWSS_LOG_ERROR("PORT_PHY_SERDES_ATTR: Failed to get port RID for port serdes RID:0x%" PRIx64 ", status:%d",
port_serdes_rid, status);
break;
}
}
return false;
}

Expand Down Expand Up @@ -2311,13 +2326,28 @@ class PortPhySerdesAttrContext : public AttrContext<sai_port_serdes_attr_t, Port
attr.id = SAI_PORT_ATTR_HW_LANE_LIST;
attr.value.u32list.count = 0; // Query with count=0 to get the actual lane count
attr.value.u32list.list = nullptr;
sai_status_t status = Base::m_vendorSai->get(SAI_OBJECT_TYPE_PORT, port_rid, 1, &attr);

if (status != SAI_STATUS_BUFFER_OVERFLOW)
for (uint32_t tries = 1; tries <= 5; tries++)
{
SWSS_LOG_ERROR("PORT_PHY_SERDES_ATTR: Failed to get hardware lane count for port RID:0x%" PRIx64 ", status:%d",
port_rid, status);
return;
sai_status_t status = Base::m_vendorSai->get(SAI_OBJECT_TYPE_PORT, port_rid, 1, &attr);

Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
// The SAI status expected is SAI_STATUS_BUFFER_OVERFLOW since we pass in a nullptr
// This is the agreed method with Broadcom for retrieving the actual lane count
if (status == SAI_STATUS_BUFFER_OVERFLOW)
break;
else if (status == SAI_STATUS_OBJECT_IN_USE && tries < 5)
{
// SAI object is busy - retry in 10ms
SWSS_LOG_WARN("PORT_PHY_SERDES_ATTR: SAI object in use, retry getting hardware lane count for port RID:0x%" PRIx64 "...",
port_rid);
std::this_thread::sleep_for(chrono::milliseconds(10));
}
else
{
SWSS_LOG_ERROR("PORT_PHY_SERDES_ATTR: Failed to get hardware lane count for port RID:0x%" PRIx64 ", status:%d",
port_rid, status);
return;
}
}

laneCount = attr.value.u32list.count;
Expand Down Expand Up @@ -2345,18 +2375,34 @@ class PortPhySerdesAttrContext : public AttrContext<sai_port_serdes_attr_t, Port
sai_attribute_t attr;
attr.id = attrId;

sai_status_t status = Base::m_vendorSai->get(
Base::m_objectType,
port_serdes_rid,
1,
&attr);

if (status != SAI_STATUS_SUCCESS)
bool failed = false;
for (uint32_t tries = 1; tries <= 5; tries++)
{
SWSS_LOG_ERROR("PORT_PHY_SERDES_ATTR: Failed to get port serdes count attr %s for port_serdes RID:0x%" PRIx64 ", status:%d",
sai_serialize_port_serdes_attr(attrId).c_str(), port_serdes_rid, status);
continue;
sai_status_t status = Base::m_vendorSai->get(
Base::m_objectType,
port_serdes_rid,
1,
&attr);

if (status == SAI_STATUS_SUCCESS)
break;
else if (status == SAI_STATUS_OBJECT_IN_USE && tries < 5)

@prgeor prgeor Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justin-wong-ce I think we need to understand who else is modifying the serdes object to keep it busy in BRCM SDK. Adding a busy retry only mitigates the problem to some extent but there could be other race condition in future where these retries may not be sufficient. Do you know how else is modifying the serdes object?

@justin-wong-ce justin-wong-ce Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To my knowledge, there is just not enough time between creating the object and reading from it.

@justin-wong-ce justin-wong-ce Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Port phy serdes objects are first created in intialization.

Then some time later, a reconfiguration can happen. This can be during intialization or during polling.
To apply the new config, the object has to be recreated.

There is a chance a poll will happen too quickly after the object is created.

@justin-wong-ce justin-wong-ce Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, there is no mechanism to partially delay the polling only on the Port PHY Serdes objects, or on object creation. However, I am also not familiar with the orchagent code.

If there is a way to:

  1. stop polling on certain objects when destroy object
  2. when an object isc reated, check the object is ready for polling before turning on polling

^then we should use this approach, this will be the ideal fix.

{
// SAI object is busy - retry in 10ms
SWSS_LOG_WARN("PORT_PHY_SERDES_ATTR: SAI object in use, retry getting port serdes count attr %s for port_serdes RID:0x%" PRIx64 "...",
sai_serialize_port_serdes_attr(attrId).c_str(), port_serdes_rid);
std::this_thread::sleep_for(chrono::milliseconds(10));
}
else
{
SWSS_LOG_ERROR("PORT_PHY_SERDES_ATTR: Failed to get port serdes count attr %s for port_serdes RID:0x%" PRIx64 ", status:%d",
sai_serialize_port_serdes_attr(attrId).c_str(), port_serdes_rid, status);
failed = true;
break;
}
}
if (failed)
continue;

count = attr.value.u32;

Expand Down
Loading
Loading