sonic-sairedis #1774 (Smart Counter Poll) does O(ports) per-port getStats at counter init, starving orchagent and crashing it on high port-count switches

[gpunathilell]

Is it platform specific

Generic — affects any switch with a high number of ports. Root cause is in the vendor-agnostic syncd FlexCounter counter-initialization code.

Importance or Severity

High — orchagent aborts during initialization; ports are never programmed and all interfaces stay oper-down (no data plane). Reproducible on every boot / config reload.

Description of the bug

sonic-sairedis PR #1774 ("Smart Counter Poll to allow counters to work properly on Broadcom platforms") changed FlexCounter counter initialization from "assume all ports support the same counters" to per-port capability discovery:

• [master] Smart Counter Poll to allow counters to work properly on Broadcom platforms. sonic-sairedis#1774

What the change does at counter init (in syncd):

• Runs a capability check (queryStatsCapability) to get the set of counters the object type supports — this is cheap and done once.
• Then, for every port, issues individual getStats calls to empirically determine which of those counters that specific port actually supports (a getStats of the candidate counter set per port, and per-counter getStats in the fallback). Ports are then bucketed into "counter groups" by what they support.

Why it breaks on high port-count switches:

• The per-port getStats step turns counter init into an O(number-of-ports) loop of live hardware reads.
• A single getStats call costs on the order of ~136 ms, and on a 512-port switch that is 512 such calls ≈ ~70 seconds just to discover per-port capability.
• While this loop runs, syncd holds vendor SAI/SDK access serialized. orchagent's concurrent synchronous SAI calls (e.g. the sai_query_stats_capability it issues during counter setup) cannot be serviced — they starve behind the discovery loop.
• Those synchronous calls hit the 60 s getresponse timeout, which desynchronizes the request/response channel; orchagent then throws an uncaught exception and aborts (SIGABRT) in its FLEX_COUNTER_TABLE drain. Ports are never programmed.

Note this is the best case (uniform ports, one getStats per port); heterogeneous ports fall back to per-counter probing and are even more expensive.

Confirmed on a 512-port switch (instrumented syncd/orchagent): the entire per-port discovery runs inside a single syncd event. syncd processes its request queue with one thread, in order, so that one event holds the queue at the head for ~74 s (elapsed_ms=74220, 514 ports) and blocks everything behind it. orchagent's next synchronous call therefore waits past its 60 s getresponse timeout → gets 0 values → aborts:

discovery START             t=0
orchagent TIMEOUT           t=+60.06 s   (60 s sync-op timeout)
discovery DONE              t=+74.2 s
orchagent -> 0 values -> std::runtime_error -> SIGABRT (FLEX_COUNTER_TABLE drain)

The failure condition is simply ports × per-port getStats > 60 s, so it is not platform specific: invisible at low port counts, hit by any platform carrying #1774 once the count is high enough (empirically > ~256 ports).

Steps to Reproduce

1. Boot (or run sudo config reload -y) an image that includes sonic-sairedis [vs-test]: not forward routes with no-export community #1774, on a switch with a high port count.
2. Watch syncd counter init and orchagent in the logs, and show interface status.

Actual Behavior and Expected Behavior

Actual: During counter init, syncd spends tens of seconds in the per-port getStats discovery loop; orchagent's synchronous SAI call times out at 60 s, and orchagent aborts (exception in the FLEX_COUNTER_TABLE drain) and hangs. All interfaces remain oper-down.

Expected: Counter init completes within the SAI response-timeout window; orchagent stays up and programs ports; interfaces come up.

Relevant log output

# orchagent - synchronous SAI call to syncd times out during counter init, then aborts
ERR  swss#orchagent: wait: SELECT operation result: TIMEOUT on getresponse
ERR  swss#orchagent: waitForGetResponse: logic error, get response returned 0 values!
ERR  swss#orchagent: drain: Exception caught: table=FLEX_COUNTER_TABLE, error=...0 values!
     swss#orchagent: terminate called after throwing an instance of 'std::runtime_error'   # SIGABRT

# syncd - per-port counter-init discovery: one getStats per port, ~136 ms each, ~512 ports
#   bulkAdd START  group=PORT  nports=512
#   ... 512 x [ per-port getStats, ~136 ms each ] ...
#   grouping DONE  group=PORT  nports=512  elapsed_ms=~70000

Output of show version, show techsupport

Image includes sonic-sairedis #1774. show techsupport available on request / to be attached.

Attach files (if any)

(attach generate_dump techsupport with the orchagent core + syncd/swss logs)

[justin-wong-ce]

Hi there, I am looking into this and have fixes locally to address it.
The fix will be enabling the per-port discovery code path based on the HWSKU config or ASIC type.

I will have draft PRs opened to fix this on Monday EOD PST - I have 2 ways of going about it, one will be HWSKU config based, one will be ASIC type based. I will have both methods available as a draft PR to see which one fits the community more.

I have also ran additional profiling tests on 2 TH5 topos with the merged master change (sonic-net/sonic-sairedis#1774) cherry-picked to 202605:
Arista-7060X6-64PE-B-C512S2 (512 intfs) and Arista-7060X6-64PE-B-C448O16 (464 intfs + LAGs).
There is no crashing and please see the below data, this data is the averaged results from 3 runs:

Arista-7060X6-64PE-B-C512S2, 512 intfs:

Item	Value
Total addCounters + bulkAddCounters execution time	9599.063ms
Execution time of new logic only	3421.033ms
Additional size of data structures	511KB

Arista-7060X6-64PE-B-C448O16, 464 intfs (+ some LAGs):

Item	Value
Total addCounters + bulkAddCounters execution time	10131.861ms
Execution time of new logic only	2376.12ms
Additional size of data structures	472.125KB

I think the biggest difference is the cost of the getStats call between Broadcom platforms and whichever platform the issue is seen on. On Broadcom platforms the getStats call cost is essentially 0.

@gpunathilell do you mind providing more info on the platform and ASIC the issue is seen on?

[justin-wong-ce]

I have 2 methods for consideration and I am splitting the fix into 2 PRs.

The 1st PR is generic to both methods and works by itself - adding an argument to syncd:
sonic-net/sonic-sairedis#2000

I also have 2 sets of draft PRs, one for each method:
HWSKU config based:
buildimage PR: #28506
sairedis PR: sonic-net/sonic-sairedis#2002

ASIC type based:
sairedis PR: sonic-net/sonic-sairedis#2001

• This one is currently using the logic of enabling when Broadcom.
• Perhaps it should be disable when [failing platform]?

[justin-wong-ce]

After some offline discussion, we will take in both methods.

For platforms matching the Broadcom ASIC type, it will auto-use the flag.
We will also have a HWSKU config knob to enable/disable the toggle, which should override the ASIC type-based toggle (in both true and false scenarios).

As a 1st step, sonic-net/sonic-sairedis#2000 and sonic-net/sonic-sairedis#2001 should be merged first to unblock qualification first. This will add the ASIC type-based toggle.

Then, we should merge #28506 and sonic-net/sonic-sairedis#2002. These 2 changes adds HWSKU folder based conifg to toggle the feature and overrides the the ASIC-type toggle introduced in the previous 2 PRs.

FYI @lolyu @gpunathilell

[prabhataravind]

@justin-wong-ce has fixes to address this.