Is it platform specific
broadcom
Importance or Severity
High
Previous Working Version
20251110.37
Steps to Reproduce
In 20251110.38 onwards on certain HWSKUs:
- Some interfaces will wrongly report missing counters in COUNTERS_DB (elaborated below)
- Consequently, during SNMPWALK, they will return None, which will terminate SNMPWALK early and cause counters for higher SNMP interfact indexes to be ignored/skipped
More detail:
After bulk requests were re-enabled sonic-net/sonic-swss#4727. On certain HwSKUs, if a bulk request includes a counter unsupported by a particular port, the vendor SAI reports failure for that port object. syncd then skips writing all counters from that request for that port to COUNTERS_DB, including counters the port does support. This is one error that sonic-net/sonic-sairedis#1774 is tackling but does not have an active backport to 202511 at the moment. You can see this through the command:
admin@sonic:~$ docker exec -i snmp python3 - <<'PY'
from sonic_ax_impl.mibs.ietf.rfc1213 import InterfacesUpdater, DbTables
u = InterfacesUpdater()
u.reinit_connection()
u.reinit_data()
u.update_data()
counter = DbTables.SAI_PORT_STAT_IF_IN_DISCARDS
print(f'{"Index":<8} {"Interface":<20} {"ifInDiscards"}')
for sub_id in u.if_range:
index = sub_id[0]
name = u.oid_name_map.get(index) or u.oid_lag_name_map.get(index)
if name and (name.startswith("Ethernet") or name.startswith("PortChannel")):
print(f"{index:<8} {name:<20} {u.get_counter(sub_id, counter)}")
PY
Index Interface ifInDiscards
1 Ethernet0 0
5 Ethernet4 0
9 Ethernet8 0
...
257 Ethernet256 None
261 Ethernet260 None
1101 PortChannel101 5
1102 PortChannel102 7
...This output shows that Ethernet256 and Ethernet260 are missing the counter, while later interfaces such as PortChannel101 still have valid values internally.
However, an SNMP walk encounters None at ifIndex 257 and incorrectly treats that as the end of the ifInDiscards column. It therefore does not return the valid PortChannel entries at indexes 1101 and above.
This was exposed by snmp/test_snmp_interfaces.py::test_snmp_interfaces_error_discard when the test selected a RIF whose ifIndex was higher than the first interface with a missing counter.
Impact of this regression
As above:
-
Interfaces may be missing counters they support because, if one counter in a bulk request is unsupported by a port, the vendor SAI can fail the entire request for that port. syncd then writes none of the requested counters for that port.
-
SNMP walks exhibit reproducibly incorrect behavior when they encounter one of these missing counter values: instead of skipping the interface, the walk terminates the affected IF-MIB counter column and omits valid entries with higher ifIndex values.
Relevant log output
Before the change (.37)
admin@sonic:~$ docker exec -i snmp python3 - <<'PY'
> from sonic_ax_impl.mibs.ietf.rfc1213 import InterfacesUpdater, DbTables
>
> u = InterfacesUpdater()
> u.reinit_connection()
> u.reinit_data()
> u.update_data()
>
> counter = DbTables.SAI_PORT_STAT_IF_IN_DISCARDS
>
> print(f'{"Index":<8} {"Interface":<20} {"ifInDiscards"}')
> for sub_id in u.if_range:
> index = sub_id[0]
> name = u.oid_name_map.get(index) or u.oid_lag_name_map.get(index)
> if name and (name.startswith("Ethernet") or name.startswith("PortChannel")):
> print(f"{index:<8} {name:<20} {u.get_counter(sub_id, counter)}")
> PY
Index Interface ifInDiscards
1 Ethernet0 0
5 Ethernet4 0
...
257 Ethernet256 0
261 Ethernet260 0
1101 PortChannel101 4
1102 PortChannel102 8
...
12000 Ethernet-Rec0 0Output of show version, show techsupport
Attach files (if any)
No response
Is it platform specific
broadcom
Importance or Severity
High
Previous Working Version
20251110.37
Steps to Reproduce
In 20251110.38 onwards on certain HWSKUs:
More detail:
After bulk requests were re-enabled sonic-net/sonic-swss#4727. On certain HwSKUs, if a bulk request includes a counter unsupported by a particular port, the vendor SAI reports failure for that port object. syncd then skips writing all counters from that request for that port to COUNTERS_DB, including counters the port does support. This is one error that sonic-net/sonic-sairedis#1774 is tackling but does not have an active backport to 202511 at the moment. You can see this through the command:
This output shows that Ethernet256 and Ethernet260 are missing the counter, while later interfaces such as PortChannel101 still have valid values internally.
However, an SNMP walk encounters None at ifIndex 257 and incorrectly treats that as the end of the ifInDiscards column. It therefore does not return the valid PortChannel entries at indexes 1101 and above.
This was exposed by
snmp/test_snmp_interfaces.py::test_snmp_interfaces_error_discardwhen the test selected a RIF whose ifIndex was higher than the first interface with a missing counter.Impact of this regression
As above:
Interfaces may be missing counters they support because, if one counter in a bulk request is unsupported by a port, the vendor SAI can fail the entire request for that port. syncd then writes none of the requested counters for that port.
SNMP walks exhibit reproducibly incorrect behavior when they encounter one of these missing counter values: instead of skipping the interface, the walk terminates the affected IF-MIB counter column and omits valid entries with higher ifIndex values.
Relevant log output
Before the change (.37)
Output of
show version,show techsupportAttach files (if any)
No response