Bug: high swss memory increase during fast link flaps

[anamehra]

Is it platform specific

generic

Importance or Severity

Critical

Description of the bug

We have observed high memory usage by swss when link flaps very fast and if continued for long duration, causes memory exhaustion leading to Kernel panic due to OOM. In some circumstances as CPU load increases, this results in swss abort due to SAI delays and recovering the memory.

There are few PRs in community towards this and need some traction:

[swss-common] Enhanced NotificationConsumer queue to avoid memory lea… by yongxin-acctoncn · Pull Request #989 · sonic-net/sonic-swss-common

[swss] Notifier drains all notification messages to avoid memory leaks in some cases by yongxin-acctoncn · Pull Request #3560 · sonic-net/sonic-swss

These two PRs help handling memory at swss but due to the high SAI notifications from libsai, causes memory buffer increase on database redis.

The following HLD brings in link dampning which can help reducing link flap notifications:

Link event damping HLD by Ashish1805 · Pull Request #1071 · sonic-net/SONiC

Steps to Reproduce

High rate link flaps

Actual Behavior and Expected Behavior

The system memory usage should not increase alarming and the system should be able to handle the scenario without melt down.

Relevant log output

Output of show version, show techsupport

Attach files (if any)

No response

[arlakshm]

Updates after chat with @prsunny.
The below options are being considered to handle this issue:

• throttling notifications overall (link flaps or mac-moves) especially for duplicated notifications
• link damping feature onboarding

There are some concerns with changes proposed in this PR mentioned in this issue. Internal discussion within MSFT is ongoing

[prabhataravind]

MSFT is working on a long-term solution for this. Assigned to Prince for now.

[rlhui]

@arlakshm , @prsunny , chassis community would like to participate in discussions on this issue. Can we please have a meeting on this (or discussed in another WG?) Thanks.

[senthil-nexthop]

@prsunny @prabhataravind @rlhui
I proposed a HLD to address the unbounded growth of swss memory in the presence of a storm that uses a "opt-in" approach to address the risk of making a change to a base class. The HLD was discussed in the foundation WG last Tuesday. I'd be happy to chat with you on how we can combine our efforts. Here's the link to the HLD: NotificationQueuePolicy

[prabhataravind]

@anamehra are you able to reproduce this in 202605? We have had some improvements from nexthop that went into 202605 that should help address this issue. @senthil-nexthop for viz.

[anamehra]

@anamehra are you able to reproduce this in 202605? We have had some improvements from nexthop that went into 202605 that should help address this issue. @senthil-nexthop for viz.

I will try out this next week. Thanks