Fixing sonic-buildimage failures due to PR#1906 by sivat6 · Pull Request #1942 · sonic-net/sonic-sairedis

sivat6 · 2026-06-16T10:40:42Z

Description of PR

Noticed the following error logs in syncd.

2026 Jun 15 06:51:58.579634 vlab-01 ERR syncd#syncd: :- run: select failed: 2

Summary:
Fixes #
processPendingDampingSync() was directly sending notifications via m_notifications->send() which might affect the fd that select() is managing resulted in inconsistent state.

Introduced a queue with mutex protection. processPendingDampingSync() enqueues the notifications and flushPendingDampingNotifications() dequeues and sends in main thread.

Type of change

Bug fix

Approach

What is the motivation for this PR?

Build issue in sonic-buildimage

Work item tracking

None

How did you do it?

The processPendingDampingSync() just enqueues the notifications. Later flushPendingDampingNotifications() dequeues and sends in main thread.

How did you verify/test it?

Unit Testing done. Without the fix, I can see the run:select_failed messages every 1 sec. After the fix, I am not seeing this error logs.

Any platform specific information?

None

Documentation

None

mssonicbld · 2026-06-16T10:40:50Z

/azp run

azure-pipelines · 2026-06-16T10:41:03Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-16T10:42:31Z

/azp run

azure-pipelines · 2026-06-16T10:42:44Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-16T11:18:15Z

/azp run

azure-pipelines · 2026-06-16T11:18:27Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-16T11:19:49Z

/azp run

azure-pipelines · 2026-06-16T11:19:59Z

Azure Pipelines successfully started running 1 pipeline(s).

lolyu

Can you add UT to your change?

mssonicbld · 2026-06-17T16:07:16Z

/azp run

azure-pipelines · 2026-06-17T16:07:50Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-17T16:08:11Z

/azp run

azure-pipelines · 2026-06-17T16:08:23Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T01:20:56Z

/azp run

azure-pipelines · 2026-06-18T01:21:07Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T01:21:58Z

/azp run

azure-pipelines · 2026-06-18T01:22:08Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T09:50:35Z

/azp run

azure-pipelines · 2026-06-18T09:50:47Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T09:51:28Z

/azp run

azure-pipelines · 2026-06-18T09:51:41Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T10:39:23Z

/azp run

Copilot

Pull request overview

This PR updates syncd link event damping handling to avoid sending port state change notifications directly from the damping-sync path, instead queuing them and flushing/sending from the main Syncd::run() loop. This aims to prevent select()-loop instability observed in sonic-buildimage builds and improves robustness of notification delivery timing.

Changes:

Introduce a mutex-protected pending-notifications queue in Syncd, with processPendingDampingSync() enqueueing and a new flushPendingDampingNotifications() sending from the main thread.
Update the main Syncd::run() loop to explicitly handle Select::TIMEOUT/ERROR/SIGNALINT and flush queued damping notifications in a controlled place.
Add/extend unit tests covering enqueueing, overflow behavior, flush behavior, and integrated flow; add required build flags for private-member inspection in syncd unit tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
`syncd/Syncd.cpp`	Implements enqueue+flush notification flow and adjusts the main select loop control flow.
`syncd/Syncd.h`	Adds pending notification queue + mutex and declares the new flush method.
`unittest/syncd/TestSyncd.cpp`	Adds unit tests for the new queue/flush behavior and fixes constexpr linkage.
`unittest/syncd/Makefile.am`	Adds `-fno-access-control` to support tests inspecting internal `Syncd` state.

lolyu

Please fix the concerns about code quality from previous PR: #1906 that is also observed in this PR.

linux-foundation-easycla · 2026-06-22T04:41:28Z

✅ login: sivat6 / name: Sivakumar Thirukkanna Thevar (1428b71, 4727ad3, 78d4b34, 7affd78, 917351f, 9d8c761, a132544, a747fd0)
✅ login: sivat6 / name: sivat6 (db011c7)
❌ - login: @Copilot / name: Copilot Autofix powered by AI. The commit (db011c7) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

mssonicbld · 2026-06-22T04:41:29Z

/azp run

azure-pipelines · 2026-06-22T04:41:41Z

Azure Pipelines successfully started running 1 pipeline(s).

linux-foundation-easycla · 2026-06-22T07:41:43Z

✅ login: sivat6 / name: Sivakumar Thirukkanna Thevar (1428b71, 4727ad3, 75d6c68, 78d4b34, 7affd78, 917351f, 9d8c761, a132544, a747fd0, beb8f98)
✅ login: sivat6 / name: sivat6 (db011c7)
❌ - login: @Copilot / name: Copilot Autofix powered by AI. The commit (db011c7) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

mssonicbld · 2026-06-22T07:41:43Z

/azp run

azure-pipelines · 2026-06-22T07:41:55Z

Azure Pipelines successfully started running 1 pipeline(s).

linux-foundation-easycla · 2026-06-22T07:48:42Z

✅ login: sivat6 / name: Sivakumar Thirukkanna Thevar (1428b71, 368ce6c, 4727ad3, 75d6c68, 78d4b34, 7affd78, 917351f, 9d8c761, a132544, a747fd0, beb8f98)
✅ login: sivat6 / name: sivat6 (db011c7)
❌ - login: @Copilot / name: Copilot Autofix powered by AI. The commit (db011c7) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

mssonicbld · 2026-06-22T07:48:43Z

/azp run

azure-pipelines · 2026-06-22T07:48:57Z

Azure Pipelines successfully started running 1 pipeline(s).

…c-net#1912) What I did Added handling for SAI_OBJECT_TYPE_METER_BUCKET_ENTRY in meta/MetaKeyHasher.cpp: operator== overload for sai_meter_bucket_entry_t comparing (switch_id, eni_id, meter_class). sai_get_hash for the same struct. case in the extension switch of both MetaKeyHasher::operator() (hash) and MetaKeyHasher::operator() (equality). Unit test MetaKeyHasher.operator_eq_meter_bucket_entry in unittest/meta/TestMetaKeyHasher.cpp. Why I did it MetaKeyHasher::operator() / operator== are the hasher/comparator used by the unordered containers keyed by sai_object_meta_key_t inside meta/Meta.cpp (e.g. m_saiObjectCollection). Any meta-key whose object type isn't enumerated in the two switch statements falls through to SWSS_LOG_THROW("not handled: ..."). SAI_OBJECT_TYPE_METER_BUCKET_ENTRY is a DASH extension entry type already supported by sai_serialize_meter_bucket_entry / sai_deserialize_meter_bucket_entry in meta/SaiSerialize.cpp, but it was never added to the hasher. As a result, unit tests Meta.quad_mcast_fdb_entry and Meta.quad_impc_entry started failing with: :- operator(): not handled: SAI_OBJECT_TYPE_METER_BUCKET_ENTRY when the meta layer iterated extension entry metadata while validating create/get/remove on mcast_fdb / ipmc entries. The companion test Meta.remove_meter_bucket_entry passes because its code path does not exercise the hasher branch. How I verified it New unit test MetaKeyHasher.operator_eq_meter_bucket_entry exercises both operator() overloads (hash + equality) with SAI_OBJECT_TYPE_METER_BUCKET_ENTRY keys. Existing meta unit tests Meta.quad_mcast_fdb_entry / Meta.quad_impc_entry should now pass on CI. Details if related Hash is computed from (eni_id, meter_class), matching the pattern used for other extension entries such as SAI_OBJECT_TYPE_ENI_TRUSTED_VNI_ENTRY. switch_id is intentionally not folded into the hash (consistent with other entry hashers) but is included in the equality check. Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

* Adding link event damping support Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Revert "Adding link event damping support" This reverts commit 0953e68. Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Adding link event damping support Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Addressing review comments Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Wrongly checked !=0 instead of ==0 Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing indentation issues Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fix build errors Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing build error Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing unit test failure Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing white space error and missing SWSS_LOG_ENTER Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing dpkg dependency problems for libswsscommon Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Removing my changes in build-template.yml file Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing spell check failures Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Removing the fix added to build-swss-template.yml Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Adding more test cases to increase coverage Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing the failed UT case Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing the typo in the last commit Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Removing few test cases Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Removing similar tests Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Removing the notification from timer thread to main thread Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Test purpose to see PR# 1925 fixes current build issue Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Remove the code of PR# 1925 Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Fixing build issue Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * SWSS_LOG_ENTER missed in new API Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Adding suppress and reuse threshold invalid config check Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Adding tests to increase coverage Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Added SWSS_LOG_ENTER() in the APIs in test file Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Addressed review comments Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Addressed code review comments Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Adding tests under unittest and removing the unnecessary tests from syncd/tests/TestSyncdLinkEventDamping.cpp Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Trigger pipeline Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Adding SWSS_LOG_ENTER() Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Debugging the test case Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Tweaking some timing value in one test Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> * Tweaking params Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net> --------- Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

…cd::run() Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

mssonicbld · 2026-06-22T07:53:07Z

/azp run

azure-pipelines · 2026-06-22T07:53:13Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

sivat6 · 2026-06-22T11:11:43Z

New PR Opened,

#1948

sivat6 force-pushed the master branch from f722849 to 5c90732 Compare June 16, 2026 10:42

sivat6 mentioned this pull request Jun 16, 2026

[submodule][master] Update submodule sonic-sairedis to the latest HEAD automatically sonic-net/sonic-buildimage#27870

Merged

sivat6 force-pushed the master branch from 765beb6 to bb9df95 Compare June 16, 2026 11:19

sivat6 changed the title ~~Fixing the Elastictest failures due to PR# 1906~~ Fixing sonic-buildimage Elastictest failures due to PR#1906 Jun 16, 2026

lolyu reviewed Jun 17, 2026

View reviewed changes

sivat6 force-pushed the master branch from 1c82633 to eb888f0 Compare June 17, 2026 16:08

sivat6 force-pushed the master branch from cda3284 to 44c9034 Compare June 18, 2026 01:21

sivat6 changed the title ~~Fixing sonic-buildimage Elastictest failures due to PR#1906~~ Fixing sonic-buildimage failures due to PR#1906 Jun 18, 2026

sivat6 force-pushed the master branch from b443148 to fb618e7 Compare June 18, 2026 09:51

prsunny requested a review from Copilot June 19, 2026 18:52

Copilot started reviewing on behalf of prsunny June 19, 2026 18:53 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

lolyu requested changes Jun 21, 2026

View reviewed changes

lolyu and others added 12 commits June 22, 2026 13:22

Fixing the Elastictest failures due to select failed message from Syn…

334f789

…cd::run() Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Added SWSS_LOG_ENTER in flushPendingDampingNotifications

6b2e88f

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Adding UT cases

5dbf6c6

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Tweaking up/down events

1036ee0

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Added test cases

50bef46

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Spell check errors

3e46c15

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Removing dead code, whiteline changes and moved the select error cases

f31b630

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Handled starvation case + spell check

d50d18f

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Addressed code review comments

d881705

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

Few alignment corrections

d66eda1

Signed-off-by: Sivakumar Thirukkanna Thevar <sthirukkanna@juniper.net>

sivat6 force-pushed the master branch from 368ce6c to d66eda1 Compare June 22, 2026 07:52

sivat6 closed this Jun 22, 2026

Uh oh!

Conversation

sivat6 commented Jun 16, 2026

Description of PR

Type of change

Approach

What is the motivation for this PR?

Work item tracking

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

Uh oh!

mssonicbld commented Jun 16, 2026

Uh oh!

azure-pipelines Bot commented Jun 16, 2026

Uh oh!

mssonicbld commented Jun 16, 2026

Uh oh!

azure-pipelines Bot commented Jun 16, 2026

Uh oh!

mssonicbld commented Jun 16, 2026

Uh oh!

azure-pipelines Bot commented Jun 16, 2026

Uh oh!

mssonicbld commented Jun 16, 2026

Uh oh!

azure-pipelines Bot commented Jun 16, 2026

Uh oh!

lolyu left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jun 17, 2026

Uh oh!

azure-pipelines Bot commented Jun 17, 2026

Uh oh!

mssonicbld commented Jun 17, 2026

Uh oh!

azure-pipelines Bot commented Jun 17, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lolyu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linux-foundation-easycla Bot commented Jun 22, 2026

Uh oh!

mssonicbld commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

lolyu left a comment •

edited

Loading