Bug: [HA] Packet drops during SWO at 1M CPS

[vivekrnv]

Is it platform specific

generic

Importance or Severity

Critical

Description of the bug

During a planned HA switchover, hamgrd declares the switchover complete and repoints the VIP route to the incoming active DPU before that DPU has finished programming its switching_to_active role. Traffic arrives at a DPU whose dataplane still holds its standby-era forwarding rules and traffic is lost until the programming catches up.

Happens in 10% of the cases

{
  'opackets': 119979998,
  'ipackets': 119949100,
  'obytes_gb': 155.974,
  'ibytes_gb': 157.373,
  'avg_tx_pkt': 1300.0,
  'avg_rx_pkt': 1312.0,
  'packet_loss': 30898,
  'packet_loss_percentage': 0.026
}

State transitions for SWO (old -> new)

Component	Old state	New state
hamgrd scope, incoming active (smartswitch-202)	HA_STATE_STANDBY	HA_STATE_SWITCHING_TO_ACTIVE -> HA_STATE_ACTIVE
DPU SAI role, incoming active	SAI_DASH_HA_ROLE_STANDBY	SWITCHING_TO_ACTIVE -> ACTIVE
hamgrd scope, outgoing active (smartswitch-203)	HA_STATE_ACTIVE	HA_STATE_SWITCHING_TO_STANDBY -> HA_STATE_STANDBY
DPU SAI role, outgoing active	SAI_DASH_HA_ROLE_ACTIVE	STANDBY
VIP 40.202.203.1/32 primary on NPU-202	tunnel to 203.1.0.1	20.0.202.1 (local DPU)
VIP 40.202.203.1/32 primary on NPU-203	20.0.203.1 (local DPU)	tunnel to 202.1.0.1

Ordering

Time (23:09:26.xxxxxx UTC)	Event
.781298	incoming active: SAI set SWITCHING_TO_ACTIVE dispatched
.806030	outgoing active: SAI set STANDBY returns (18.1 ms), acks standby
.810397	hamgrd incoming: Pending next state: HA_STATE_ACTIVE, reason: peer acked standby role
.810460	hamgrd incoming: switchover state=completed
.813018	VNET_ROUTE_TUNNEL_TABLE primary moved to the local DPU — traffic switched
.815500	loss window opens
.835727	loss window closes (20.227 ms, 30,904 packets)
.875643	incoming active: SAI set SWITCHING_TO_ACTIVE finally returns
.878762	incoming active: SAI set ACTIVE dispatched

There is a window possible where both DPU' are Stanby and the packets are lost during that interval

Steps to Reproduce

1. SmartSwitch pair (two NPUs, one DPU HA pair). Steady state: DPU-B active, DPU-A standby, VIP
   route pointing at DPU-B.
2. Start TRex outbound TCP CPS traffic through the VIP: SYN + RST (flow create/delete),
   2 Mpps, ~1M CPS, sport/dport 2000-60000, 60 s.
3. Trigger a planned switchover from the standby side (set desired_ha_state=Active on DPU-A's
   scope, then approve the pending switchover operation).
4. Compare TRex opackets against ipackets.

Reproduces roughly 10% of the time. The failure correlates with how long the incoming active
DPU's role programming takes — the longer that SAI call, the changes of packet loss

Actual Behavior and Expected Behavior

Actual: 30,904 packets lost in a single 20.227 ms window at the switchover. TRex independently
reported 30,898 lost (opackets 119,979,998 vs ipackets, 0.03%).

Expected: 0 packets lost. HaMgrD should wait on asic_acked state before proceeding with transition

Relevant log output

# ---- hamgrd, incoming active (smartswitch-202): switchover starts ----
23:09:26.778827  Drive NPU HA state machine based on SwitchoverApproved
23:09:26.778884  Pending next state: HA_STATE_SWITCHING_TO_ACTIVE, reason: switchover approved, transitioning to active
23:09:26.778943  Sent SwitchoverRequest to peer Vdpu-203-dpu0:haset0_0
23:09:26.779411  Sending message: ... {"ha_role": "switching_to_active"} -> DASH_HA_SCOPE_TABLE

# ---- incoming active DPU: the role programming that is never waited on ----
sairedis.rec:
2026-07-31.23:09:26.781298|s|SAI_OBJECT_TYPE_HA_SCOPE:oid:0x120080000000aa|SAI_HA_SCOPE_ATTR_DASH_HA_ROLE=SAI_DASH_HA_ROLE_SWITCHING_TO_ACTIVE
syslog:
23:09:26.783053 syncd#SAI: set_dispatch_attrib_handler: Set DASH_HA_ROLE, val:SWITCHING_TO_ACTIVE
23:09:26.875643 orchagent: setHaScopeHaRole: Set HA Scope role for haset0_0 to HA_ROLE_SWITCHING_TO_ACTIVE   <-- call returned

# ---- hamgrd, outgoing active (smartswitch-203): steps down and acks ----
23:09:26.782375  Pending next state: HA_STATE_SWITCHING_TO_STANDBY, reason: peer requested switchover
sairedis.rec:
2026-07-31.23:09:26.784360|s|SAI_OBJECT_TYPE_HA_SCOPE:oid:0x120080000000aa|SAI_HA_SCOPE_ATTR_DASH_HA_ROLE=SAI_DASH_HA_ROLE_STANDBY
syslog:
23:09:26.806030 orchagent: setHaScopeHaRole: Set HA Scope role for haset0_0 to HA_ROLE_STANDBY   <-- 18.1 ms
23:09:26.807678 hamgrd: Drive NPU HA state machine based on DpuStateChanged   <-- records acked_asic_ha_state=standby

# ---- hamgrd, incoming active: goes Active on the PEER's ack and moves traffic ----
23:09:26.810314  Drive NPU HA state machine based on PeerStateChanged
23:09:26.810397  Pending next state: HA_STATE_ACTIVE, reason: peer acked standby role
23:09:26.810460  Switchover state updated: id=7a6105d2-afa3-41b8-8c11-378de660a0fc, state=completed
23:09:26.810583  HA scope transitioned: peer acked standby role scope=Vdpu-202-dpu0:haset0_0 state=HA_STATE_ACTIVE
23:09:26.811681  [ha-set] Received HaScopeStateUpdate with owner=2 state=HA_STATE_ACTIVE, updating VNET ROUTE table.

# ---- NPU-202: traffic switch reaches the dataplane ----
swss.rec:
2026-07-31.23:09:26.813018|VNET_ROUTE_TUNNEL_TABLE:Vnet_55:40.202.203.1|SET|primary:20.0.202.1|monitoring:custom_bfd|endpoint:20.0.202.1,203.1.0.1|endpoint_monitor:20.0.202.1,20.0.203.1|check_directly_connected:true
sairedis.rec:
2026-07-31.23:09:26.814691|s|SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"40.202.203.1/32",...}|SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID=oid:0x40000000005c5
   (oid 0x...5c5 = SAI_NEXT_HOP_TYPE_IP, SAI_NEXT_HOP_ATTR_IP=20.0.202.1 -> local DPU)

# ---- loss window, from the capture on the outgoing active DPU's port ----
23:09:26.815500  first leaked VXLAN packet, outer dst = outgoing active DPU PA, UDP 4789, VNI 101, inner TCP SYN/RST
23:09:26.815725  host kernel replies ICMP type 3 code 3 (port unreachable)
23:09:26.835727  last leaked packet total 30,904 packets over 20.227 ms


# ---- incoming active DPU: programming finally completes, long after traffic moved ----
23:09:26.875643 orchagent: HA_ROLE_SWITCHING_TO_ACTIVE  (returned)
2026-07-31.23:09:26.878762|s|...SAI_HA_SCOPE_ATTR_DASH_HA_ROLE=SAI_DASH_HA_ROLE_ACTIVE
23:09:26.883231 orchagent: Set HA Scope role for haset0_0 to HA_ROLE_ACTIVE

Output of show version, show techsupport

SONiC Software Version: SONiC.master

Attach files (if any)

[vivekrnv]

Potential fix PR's

sonic-net/sonic-swss#4820
sonic-net/sonic-dash-ha#212