Address racey SAI PHY Serdes object causing bad syncd setup by justin-wong-ce · Pull Request #1945 · sonic-net/sonic-sairedis

justin-wong-ce · 2026-06-18T01:10:08Z

Description of PR

Summary:
Mitigate a race condition in Port PHY Serdes initialization (syncd) where the SAI is still busy with the object from its creation when syncd's intialization attempts to read attributes from the object for syncd's own setup.

This race condition will cause a failing SAI call that is needed for initializing syncd, causing polls to the Port PHY Serdes object to fail continuously later on. The chances of this happening is around ~2.5% per config reload / reboot.

Do this by allowing 5 retries with 10ms wait intervals in between tries. The retry will only trigger when the failure reason is SAI_STATUS_OBJECT_IN_USE.

Fixes #1946

Type of change

Approach

What is the motivation for this PR?

Fix syslog ERRs that logs continuously when this scenario occurs.

Work item tracking

Microsoft ADO (number only):

How did you do it?

Added multiple attempts for interacing with Port PHY Serdes object over SAI during setup.

How did you verify/test it?

Continuous syslog ERRs no longer occur. The retry logic can be seen working in the syslogs as well. The most retry attempts observed is 1.

Any platform specific information?

Broadcom.

Documentation

Mitigate a race condition in Port PHY Serdes initialization (syncd) where the SAI is still busy with the object from its creation when syncd's intialization attempts to read attributes from the object for syncd's own setup. Do this by allowing 3 retries with 10ms wait intervals in between tries. The retry will only trigger when the failure reason is SAI_STATUS_OBJECT_IN_USE. Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T01:10:16Z

/azp run

azure-pipelines · 2026-06-18T01:10:27Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T02:00:49Z

/azp run

azure-pipelines · 2026-06-18T02:01:01Z

Azure Pipelines successfully started running 1 pipeline(s).

r12f · 2026-06-18T05:39:02Z

-            port_rid = attr.value.oid;
-            return true;
-        }
+            sai_status_t status = Base::m_vendorSai->get(Base::m_objectType, port_serdes_rid, 1, &attr);


hey Justin - the fix is a bit of "by-chance".

Also supposedly,

flex counter should not fail syncd init

flex counter should work after the objects gets created

so I wonder if we have looked deeper into the failure/race and see what might be happening?

As mentioned above, the race condition is from the SAI not being able to handle the Port PHY Serdes object creation and accessing of its attribute in extremely quick succession.

Either the SAI needs to be able to delay a request and not return SAI_STATUS_OBJECT_IN_USE, or Sonic (syncd initialization for this case) needs to be able to handle SAI_STATUS_OBJECT_IN_USE properly.

the fix is a bit of "by-chance"

I can change the fix such that it will retry until SAI does not return the SAI_STATUS_OBJECT_IN_USE, rather than just 3 times - this will remove the chance element.

I think it is reasonable to retry when the SAI returns SAI_STATUS_OBJECT_IN_USE since the object is busy and not available for read.

Please let me know what you think of the latest changes

what if we just simply skip the failed round and retry in the next pulling interval?

retrying 3 times is not going to guarantee the fix anyway.

This error is occuring during the initialization stage and this code that is being edited is for the initialization of Port PHY Serdes.
This will print the following ERR syslog once - during syncd's intialization:

30979 2026 Jun 7 20:56:03.499522 qspf201 ERR syncd#syncd: :- initAttrData: PORT_PHY_SERDES_ATTR: port_serdes_rid:0xb005700000000 has no serdes count attribute information 30980 2026 Jun 7 20:56:03.499522 qspf201 WARNING syncd#syncd: :- collectData: PORT_PHY_SERDES_ATTR: Failed to initialize attribute SAI_PORT_SERDES_ATTR_TX_FIR_TAPS_LIST for RID:0xb005700000000, skipping object

If SAI_STATUS_OBJECT_IN_USE is returned here and we don't retry, this will result missing info (maps) that is used to do other initalization steps later on, as it is treated as a failure.

If the info is missing, it will cause polling to fail all the time.
The following ERR syslog will print at every single poll interval:

26929 2026 Jun 7 20:55:59.007758 qspf201 ERR syncd#syncd: [none] SAI_API_PORT:brcm_sai_get_port_serdes_attribute_cmn:18248 Port 11 hardware lane count get failed with error -17. 26930 2026 Jun 7 20:55:59.007758 qspf201 ERR syncd#syncd: :- updatePortSerdesTapsCountMap: PORT_PHY_SERDES_ATTR: Failed to get port serdes count attr SAI_PORT_SERDES_ATTR_TX_FIR_COUNT for port_serdes RID:0xb005700000000, status:-17

@justin-wong-ce i see you have removed the retry loop, I dont think this is a right approach.
We should not retry without an upper bound. If we believe a retry of 3 times with a 10ms is not enough we can increase this to 5 retries but we should not leave it unbounded.
@r12f fyi.

@r12f please advise whether if we can go forward with 5 retries

Just to check, is it that because flexcounter is running in its own thread and starts as part of syncd initialization, it could be that flexcounter is starting to get port stats before SAI initialization (triggered from the sync main thread) completes? If so, should the flexcounter thread start instead be delayed until after SAI initialization?

Yes and no - the situaiton with the Port PHY Serdes objects are a bit special.

The Port PHY Serdes objects are first initialized like any other counter during syncd's initialization. There are no problems up to this part.

After a port configuration update, the configuration for these objects also change.

Since these objects only support CREATE, the objects have to be re-created to be reconfigured. What happens in the code is it will re-create these objects, then proceed to read from the objects from SAI so it can instantiate some variables in FlexCounter.cpp for polling - these reads usually happens on the next poll on these objects (which may or may not be the 1st poll).

The time between is not sufficient in some cases since the reconfiguration happens late into initialization or sometimes when polling has already started.
Althought its a rare occurence, it still needs to be accounted for.

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T18:29:47Z

/azp run

azure-pipelines · 2026-06-18T18:29:59Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T18:38:02Z

/azp run

azure-pipelines · 2026-06-18T18:38:15Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T18:43:58Z

/azp run

azure-pipelines · 2026-06-18T18:44:12Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T19:05:28Z

/azp run

azure-pipelines · 2026-06-18T19:05:43Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T21:04:02Z

/azp run

azure-pipelines · 2026-06-18T21:04:14Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-19T11:21:18Z

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

mssonicbld · 2026-07-14T00:15:09Z

/azp run

azure-pipelines · 2026-07-14T00:15:20Z

Azure Pipelines successfully started running 1 pipeline(s).

justin-wong-ce · 2026-07-14T00:45:49Z

Note: I am aware of the cherry-pick conflict to 202511. I have a local branch that addresses this conflict ready and will make a direct backport when this PR is approved.

    flexCounter->removeCounter(testPortSerdesOid);
 }

+TEST_F(TestPortPhySerdesAttr, RetryOnObjectInUseThenSucceed)


Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-07-14T01:17:39Z

/azp run

azure-pipelines · 2026-07-14T01:17:50Z

Azure Pipelines successfully started running 1 pipeline(s).

tjchadaga · 2026-07-14T08:10:08Z

@justin-wong-ce - are these failure logs only seen on a specific platform?

justin-wong-ce · 2026-07-14T16:40:34Z

@justin-wong-ce - are these failure logs only seen on a specific platform?

I do not have access to other platforms - I can only test on Broadcom.

I cannot say for sure if it happens on other platforms so I only noted Broadcom in the PR description.

mssonicbld · 2026-07-14T16:47:24Z

/azp run

azure-pipelines · 2026-07-14T16:47:36Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-07-14T17:11:53Z

/azp run

azure-pipelines · 2026-07-14T17:12:06Z

Azure Pipelines: Successfully started running 1 pipeline(s).

justin-wong-ce · 2026-07-14T19:59:03Z

Restarting CI due to failure from:

The process '/usr/bin/python' failed with exit code 1
BuildTrixie • amd64 • [Auto] Generate build coverage properties

ENOENT: no such file or directory, open '/__w/1/s/.diff-coverage/codecoverageProperties.json'
BuildTrixie • amd64 • [Auto] Update build coverage properties
Bash exited with code '100'.
Test • vstest • Install dependencies

justin-wong-ce · 2026-07-14T19:59:08Z

/azpw retry

mssonicbld · 2026-07-14T19:59:10Z

Retrying failed(or canceled) jobs...

mssonicbld · 2026-07-14T19:59:12Z

Retrying failed(or canceled) stages in build 1164893:

✅Stage Test:

Job vstest: retried.

r12f · 2026-07-15T22:24:48Z

hi @tjchadaga - do you mind to help check this PR?

rookie-who · 2026-07-26T17:01:24Z

Hi — this PR is flagged for cherry-pick to the 202605 branch (Request for 202605 Branch). To move forward with the cherry-pick, could you please:

Test this change on the 202605 branch and add the test results to the PR description
Add the label Tested for 202605 branch once testing is complete

This helps us track readiness and ensures we don't cherry-pick untested changes. Thanks!

Comment by @vaibhavhd via automated tooling

mssonicbld · 2026-07-26T18:19:51Z

This PR has backport request label(s) for branch(es): 202511,202605, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202511: <test result>) in the Test result section as well in your PR description.

_{---Powered by SONiC BuildBot}

Copilot

Pull request overview

Mitigates a race during syncd Port PHY Serdes flex-counter initialization where immediate SAI get calls can fail with SAI_STATUS_OBJECT_IN_USE, leaving syncd in a degraded state that later causes continuous polling errors.

Changes:

Add retry-with-delay logic (10ms backoff) for key SAI get operations used to initialize Port PHY Serdes collection (port RID lookup, hardware lane count query, and tap/count attribute reads).
Add unit tests that validate both “eventual success after transient OBJECT_IN_USE” and “failure after retry exhaustion” behaviors.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`syncd/FlexCounter.cpp`	Adds OBJECT_IN_USE retry handling during Port PHY Serdes attribute initialization to avoid permanent bad state caused by transient SAI busy responses.
`unittest/syncd/TestPortPhySerdesAttr.cpp`	Adds tests covering retry success and exhaustion scenarios for the Port PHY Serdes initialization path.

+ * and results in a good syncd state such that RX_VGA entries appears in
+ * PORT_PHY_ATTR_TABLE after retries.


prgeor · 2026-08-03T17:34:55Z

+        for (uint32_t tries = 1; tries <= 5; tries++)
        {
-            port_rid = attr.value.oid;
-            return true;
-        }
+            sai_status_t status = Base::m_vendorSai->get(Base::m_objectType, port_serdes_rid, 1, &attr);


@justin-wong-ce ?

Forgot to update the description after applying changes from #1945 (comment)

prgeor · 2026-08-03T18:05:24Z

+                // SAI object is busy - retry in 10ms
+                SWSS_LOG_WARN("PORT_PHY_SERDES_ATTR: SAI object in use, retry getting port RID for port serdes RID:0x%" PRIx64 "...",
+                               port_serdes_rid);
+                std::this_thread::sleep_for(chrono::milliseconds(10));


@justin-wong-ce this is risky to sleep assuming we have say 256 ports which will result in 256 * 10 * 5 = 12.8 secs.

The 2.5% chance is the probability of it happening to 1 port on a config-reload or reboot. Of course yes, there is the chance of it happening to all ports at the same time and I agree this is not an optimal solution.

prgeor · 2026-08-03T20:29:28Z

+
+                if (status == SAI_STATUS_SUCCESS)
+                    break;
+                else if (status == SAI_STATUS_OBJECT_IN_USE && tries < 5)


@justin-wong-ce I think we need to understand who else is modifying the serdes object to keep it busy in BRCM SDK. Adding a busy retry only mitigates the problem to some extent but there could be other race condition in future where these retries may not be sufficient. Do you know how else is modifying the serdes object?

To my knowledge, there is just not enough time between creating the object and reading from it.

Port phy serdes objects are first created in intialization.

Then some time later, a reconfiguration can happen. This can be during intialization or during polling.
To apply the new config, the object has to be recreated.

There is a chance a poll will happen too quickly after the object is created.

AFAIK, there is no mechanism to partially delay the polling only on the Port PHY Serdes objects, or on object creation. However, I am also not familiar with the orchagent code.

If there is a way to:

stop polling on certain objects when destroy object

when an object isc reated, check the object is ready for polling before turning on polling

^then we should use this approach, this will be the ideal fix.

retriving -> retrieving

d67380d

Signed-off-by: Justin Wong <jvwong@arista.com>

StormLiangMS requested a review from lolyu June 18, 2026 05:33

davidm-arista added Request for 202511 Branch Request for 202605 Branch labels Jun 18, 2026

StormLiangMS requested review from r12f and removed request for r12f June 18, 2026 05:35

r12f reviewed Jun 18, 2026

View reviewed changes

github-advanced-security AI found potential problems Jun 18, 2026

View reviewed changes

Comment thread syncd/FlexCounter.cpp Fixed

Comment thread syncd/FlexCounter.cpp Fixed

Change to more deterministic handling of SAI_STATUS_OBJECT_IN_USE

0d31fcf

Signed-off-by: Justin Wong <jvwong@arista.com>

Cast revised handling to other code path

117aa72

Signed-off-by: Justin Wong <jvwong@arista.com>

Further casts

b2eac07

Signed-off-by: Justin Wong <jvwong@arista.com>

Fix whiteline and typo

220a847

Signed-off-by: Justin Wong <jvwong@arista.com>

justin-wong-ce closed this Jun 18, 2026

justin-wong-ce reopened this Jun 18, 2026

github-advanced-security AI found potential problems Jul 14, 2026

View reviewed changes

Comment thread unittest/syncd/TestPortPhySerdesAttr.cpp

flexCounter->removeCounter(testPortSerdesOid);

}

TEST_F(TestPortPhySerdesAttr, RetryOnObjectInUseThenSucceed)

added comments for new test cases

a33c0e8

Signed-off-by: Justin Wong <jvwong@arista.com>

justin-wong-ce force-pushed the port-phy-retry branch from 3ac7f55 to a33c0e8 Compare July 14, 2026 01:06

tjchadaga reviewed Jul 14, 2026

View reviewed changes

Comment thread syncd/FlexCounter.cpp Outdated

tjchadaga requested a review from prgeor July 14, 2026 08:03

fix typo

8aa4ae9

Signed-off-by: Justin Wong <jvwong@arista.com>

justin-wong-ce force-pushed the port-phy-retry branch from 23718ac to 8aa4ae9 Compare July 14, 2026 17:10

prgeor requested a review from Copilot August 3, 2026 16:57

Copilot started reviewing on behalf of prgeor August 3, 2026 16:58 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

prgeor reviewed Aug 3, 2026

View reviewed changes

		* and results in a good syncd state such that RX_VGA entries appears in
		* PORT_PHY_ATTR_TABLE after retries.

Uh oh!

Conversation

justin-wong-ce commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Approach

What is the motivation for this PR?

Work item tracking

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r12f Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 19, 2026

Uh oh!

mssonicbld commented Jul 14, 2026

Uh oh!

azure-pipelines Bot commented Jul 14, 2026

Uh oh!

justin-wong-ce commented Jul 14, 2026

Uh oh!

mssonicbld commented Jul 14, 2026

Uh oh!

azure-pipelines Bot commented Jul 14, 2026

Uh oh!

Uh oh!

tjchadaga commented Jul 14, 2026

justin-wong-ce commented Jun 18, 2026 •

edited

Loading

r12f Jun 18, 2026 •

edited

Loading