[ZmqOrch] Optimize memory by popping batch size at a time - #3910
Merged
Conversation
Signed-off-by: Vivek Reddy <vkarri@nvidia.com>
Signed-off-by: Vivek <vkarri@nvidia.com>
Signed-off-by: Vivek <vkarri@nvidia.com>
Signed-off-by: Vivek Reddy <vkarri@nvidia.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
9 tasks
oleksandrivantsiv
approved these changes
Sep 30, 2025
prsunny
approved these changes
Oct 2, 2025
Collaborator
|
@liuh-80 , please review/signoff |
mssonicbld
added a commit
to mssonicbld/sonic-buildimage-msft
that referenced
this pull request
Oct 2, 2025
<!--
Please make sure you've read and understood our contributing guidelines:
https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md
** Make sure all your commits include a signature generated with `git commit -s` **
If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx" or "resolves #xxxx"
Please provide the following information:
-->
#### Why I did it
Increase the pop batch size and Max Bulker limit to 65536 to speed up applying the high volume Dash configuration
Depends on
sonic-net/sonic-sairedis#1660
sonic-net/sonic-swss-common#1084
sonic-net/sonic-swss#3910
##### Work item tracking
- Microsoft ADO **(number only)**:
#### How I did it
#### How to verify it
```
root@sonic:/home/admin# ps -aux | grep orch
root 11118 1.5 0.4 464804 267368 pts/0 Sl 02:50 0:00 /usr/bin/orchagent -d /var/log/swss -b 65536 -z zmq_sync -k 65536 -m B0:CF:0E:20:8E:DE -q tcp://eth0-midplane
2025 Sep 30 18:48:38.911835 sonic NOTICE swss#orchagent: :- main: Setting maximum bulk size in bulk mode as 65536
```
Apply Scale config and verify
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->
#### Which release branch to backport (provide reason below if selected)
<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
#### Tested branch (Please provide the tested image version)
<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->
- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->
#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
<!--
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->
#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->
#### A picture of a cute animal (not mandatory but encouraged)
Merged
9 tasks
mssonicbld
added a commit
to Azure/sonic-buildimage-msft
that referenced
this pull request
Oct 3, 2025
…1691) <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md failure_prs.log skip_prs.log Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it Increase the pop batch size and Max Bulker limit to 65536 to speed up applying the high volume Dash configuration Depends on sonic-net/sonic-sairedis#1660 sonic-net/sonic-swss-common#1084 sonic-net/sonic-swss#3910 ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it #### How to verify it ``` root@sonic:/home/admin# ps -aux | grep orch root 11118 1.5 0.4 464804 267368 pts/0 Sl 02:50 0:00 /usr/bin/orchagent -d /var/log/swss -b 65536 -z zmq_sync -k 65536 -m B0:CF:0E:20:8E:DE -q tcp://eth0-midplane 2025 Sep 30 18:48:38.911835 sonic NOTICE swss#orchagent: :- main: Setting maximum bulk size in bulk mode as 65536 ``` Apply Scale config and verify <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 202205 - [ ] 202211 - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged)
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
prabhataravind
approved these changes
Oct 8, 2025
Contributor
Author
|
/azpw run Azure.sonic-swss |
Collaborator
|
/AzurePipelines run Azure.sonic-swss |
|
Azure Pipelines successfully started running 1 pipeline(s). |
liuh-80
approved these changes
Oct 9, 2025
Collaborator
|
Cherry-pick PR to msft-202506: Azure/sonic-swss.msft#159 |
balanokia
pushed a commit
to balanokia/sonic-swss
that referenced
this pull request
Nov 17, 2025
…3910) * [ZmqOrch] Optimize memory by popping batch size at a time What I did Used a reference instead of a unnecessary copy of a set object Optimize memory by popping batch size at a time NOTE: Please merge only after the below two PR's are merged sonic-net/sonic-swss-common#1084 sonic-net/sonic-sairedis#1660
theasianpianist
pushed a commit
to theasianpianist/sonic-swss
that referenced
this pull request
Feb 4, 2026
…3910) * [ZmqOrch] Optimize memory by popping batch size at a time What I did Used a reference instead of a unnecessary copy of a set object Optimize memory by popping batch size at a time NOTE: Please merge only after the below two PR's are merged sonic-net/sonic-swss-common#1084 sonic-net/sonic-sairedis#1660 Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
baorliu
pushed a commit
to baorliu/sonic-swss
that referenced
this pull request
Feb 23, 2026
…3910) * [ZmqOrch] Optimize memory by popping batch size at a time What I did Used a reference instead of a unnecessary copy of a set object Optimize memory by popping batch size at a time NOTE: Please merge only after the below two PR's are merged sonic-net/sonic-swss-common#1084 sonic-net/sonic-sairedis#1660 Signed-off-by: Baorong Liu <96146196+baorliu@users.noreply.github.com>
Contributor
|
Tagging @deepak-singhal0408 for viz |
Contributor
@venkit-nexthop since this PR is merged already, can you help create a new issue for the regression you are observing, while tagging this PR? |
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 15, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 15, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 15, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 15, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 16, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 17, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
venkit-nexthop
added a commit
to venkit-nexthop/sonic-swss
that referenced
this pull request
May 18, 2026
PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
deepak-singhal0408
added a commit
to deepak-singhal0408/sonic-swss
that referenced
this pull request
May 19, 2026
Restore do...while drain loop in ZmqConsumer::execute() for the non-ordered queue path. The single pops() call from PR sonic-net#3910 may leave messages unprocessed if more arrive during execution, hurting route convergence throughput under high ZMQ load. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>
prsunny
pushed a commit
that referenced
this pull request
May 27, 2026
* [zmqorch]: Restore drain-until-empty consumer for non-DPU/fabric PR #3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in #3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
Yogapriya-cisco
pushed a commit
to prhoskot/sonic-swss
that referenced
this pull request
Jul 13, 2026
…3910) * [ZmqOrch] Optimize memory by popping batch size at a time What I did Used a reference instead of a unnecessary copy of a set object Optimize memory by popping batch size at a time NOTE: Please merge only after the below two PR's are merged sonic-net/sonic-swss-common#1084 sonic-net/sonic-sairedis#1660 Signed-off-by: Yogapriya Mohankumar <ymohanku@cisco.com>
Yogapriya-cisco
pushed a commit
to prhoskot/sonic-swss
that referenced
this pull request
Jul 13, 2026
…ic-net#4578) * [zmqorch]: Restore drain-until-empty consumer for non-DPU/fabric PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai> Signed-off-by: Yogapriya Mohankumar <ymohanku@cisco.com>
leenr
pushed a commit
to leenr/sonic-swss
that referenced
this pull request
Aug 4, 2026
…3910) * [ZmqOrch] Optimize memory by popping batch size at a time What I did Used a reference instead of a unnecessary copy of a set object Optimize memory by popping batch size at a time NOTE: Please merge only after the below two PR's are merged sonic-net/sonic-swss-common#1084 sonic-net/sonic-sairedis#1660
leenr
pushed a commit
to leenr/sonic-swss
that referenced
this pull request
Aug 4, 2026
…ic-net#4578) * [zmqorch]: Restore drain-until-empty consumer for non-DPU/fabric PR sonic-net#3910 changed ZmqConsumer::execute() to perform a single batch pop per invocation, primarily to reduce peak memory in DASH (DPU) workloads. For non-DPU / non-fabric switches that receive high volumes of APPL_DB route updates from fpmsyncd, this regressed throughput: each execute() call no longer drains the consumer table, so updates take many more polls to apply. Introduce a ZmqRouteConsumer subclass that restores the prior drain-until-empty loop in execute(). ZmqOrch::addConsumer() selects ZmqRouteConsumer for non-DPU / non-fabric switch types; DPU and fabric continue to use ZmqConsumer to preserve the bounded-memory behavior added in sonic-net#3910. Also define gMySwitchType in orchagent/p4orch/tests/test_main.cpp so p4orch_tests links once zmqorch.cpp references the global. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I did
NOTE: Please merge only after the below two PR's are merged
sonic-net/sonic-swss-common#1084
sonic-net/sonic-sairedis#1660
Why I did it
To reduce peak memory usage when applying high-volume dash configuration
How I verified it
Details if related