[vpp] VXLAN L3 dataplane: inner-aware encap hash, underlay ECMP, source-independent decap (patches 0015-0017)

[aaronber0614]

Description of PR

Summary:
Enable correct L3 VXLAN forwarding on the sonic-vpp platform by adding the VPP
dataplane patch series that the saivpp adaptation and tests depend on.

Part of sonic-net/sonic-buildimage#25777

This is the dataplane layer (layer 1) of a cross-repo feature and lands first:

1. this PR - VPP patches (first)
2. [vpp] saivpp VXLAN L3 decap: source-independent decap, inner source MAC, BD teardown fix sonic-sairedis#1996 - saivpp adaptation
3. sonic-buildimage submodule bump (auto via mssonicbld)
4. [vpp] VXLAN L3 dataplane test enablement sonic-mgmt#26220 - test enablement (last)

Changes (vppbld/patches):

• 0015-vxlan-encap-inner-aware-flow-hash.patch - hash on the inner (overlay)
  flow for VXLAN encap so IPinIP/VXLAN transit traffic spreads across ECMP
  next-hops and LAG members instead of polarizing onto a single path.
• 0017-sonic-vxlan-vnet-source-independent-decap.patch - source-independent
  (decap-any) VNET decap plus review hardening (checked wildcard bihash return codes and an O(1) L3-promiscuous BVI receive bitmap).
• vppbld/patches/series - register the two patches.

Type of change

• New feature

Approach

What is the motivation for this PR?

On sonic-vpp, L3 VXLAN transit traffic polarized onto a single next-hop / bond
member (no inner-aware hashing) and source-independent VNET decap was missing.

These fixes require VPP dataplane behavior that cannot be handled in the SAI
adaptation layer (saivpp / vslib/vpp):

• Inner flows collapsed onto a single underlay path / bond member because the
  VPP encap node derived the outer UDP source port from a mis-parsed inner
  frame on L3-routed tunnels (patch 0015).
• Source-independent (secondary-VTEP / RIOT) decap needs VPP to accept any
  outer source for a given local dst + VNI; VPP keys decap on the exact
  (src, dst, vni) tuple, so a new wildcard match path was required (patch
  0017). The matching saivpp inner-source-MAC and decap wiring lands in
  [vpp] saivpp VXLAN L3 decap: source-independent decap, inner source MAC, BD teardown fix sonic-sairedis#1996.

Note on underlay ECMP fairness: an earlier revision of this series also carried
a patch (0016) that tightened VPP's global FIB multipath load-balance tolerance
(multipath_next_hop_error_tolerance 0.1 -> 0.01) so non-power-of-2 equal-cost
groups allocate more buckets. That constant is shared by every ASIC target, not
just the VPP VXLAN underlay, so it has been dropped to avoid changing shared
forwarding behavior. The resulting bucket skew for non-power-of-2 groups is a
property of VPP's default multipath allocation and is handled in the sonic-mgmt
VXLAN ECMP test tolerance instead (sonic-net/sonic-mgmt#26220).

How did you do it?

Added patches 0015 and 0017 to the vppbld patch series (after master's sflow
patches 0012-0014).

How did you verify/test it?

Built a sonic-vpp image from this branch and ran the hashing/VXLAN regression
suite on a t1-lag-vpp testbed: fib/test_fib.py hashing pass; the VXLAN ECMP and
entropy suites (Test_VxLAN_ecmp_random_hash, Test_VxLAN_entropy) pass across all
four encap types with the default multipath tolerance and the test-side
tolerance in #26220; VNET decap tracks green. CI build+boot proof via the
validation image PR sonic-net/sonic-buildimage#28437.

Any platform specific information?

sonic-vpp platform only.

Documentation

No SONiC doc/HLD changes in this PR.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[mssonicbld]

/azp run

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR advances SONiC-VPP’s VXLAN L3 dataplane support by adding a new VPP patch trio (0015–0017) to improve underlay load-balancing behavior, correctly hash on inner (overlay) flows for VXLAN encap, and enable source-independent (“decap-any”) VNET decap. It also bumps VPP_VERSION to ensure downstream builds rebuild VPP with these patches applied.

Changes:

• Register VPP patches 0015–0017 in the build patch series.
• Add VXLAN encap inner-aware flow hashing and underlay ECMP tolerance tightening.
• Add VXLAN VNET “decap-any” (source-independent) decap support plus a BVI L3-promiscuous fast path for decap-and-route.
• Bump VPP_VERSION to force a rebuild and avoid stale cached artifacts.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vppbld/patches/series	Adds patches 0015–0017 to the VPP patch application order.
vppbld/patches/0015-vxlan-encap-inner-aware-flow-hash.patch	Updates VXLAN encap behavior to hash on inner IP flow (overlay) for better ECMP/LAG spreading.
vppbld/patches/0016-sonic-vxlan-underlay-ecmp-loadbalance-tolerance.patch	Tightens global multipath normalization tolerance to retain equal-cost paths.
vppbld/patches/0017-sonic-vxlan-vnet-source-independent-decap.patch	Implements decap-any VXLAN VNET decap behavior and adds BVI routing bypass for decap-and-route.
rules/vpp.mk	Bumps VPP_VERSION to force rebuild with the new patch set.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

@yejianquan heads-up on the BuildVpp amd64-trixie red on this PR (and thanks for merging master in) — it is not VXLAN-related; it's a pipeline-wide trixie regression in the VPP external-deps build.

Evidence:

• The same patches (0015-0017) passed amd64-trixie on 2026-07-21 (ADO build 1170922); after the master merge they fail on 2026-07-27 (build 1175766). Only trixie fails — amd64-bookworm and DCO are green.
• The identical failure reproduces on an unrelated platform-vpp PR ([vpp] Add sonic_ext RIF loopback packet action nodes and binary API #259, build 1177419), so it isn't this branch/series.

Root cause (from #259's fuller log) — the vpp-ext-deps package build fails at debug-symbol stripping, not at compile:

dh_strip: error: objcopy --only-keep-debug --compress-debug-sections \
  debian/vpp-ext-deps/opt/vpp/external/x86_64/bin/dpdk-test-cmdline \
  .../usr/lib/debug/.build-id/62/....debug returned exit code 1
dh_strip: error: Aborting due to earlier error
make[4]: *** [debian/rules:20: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make[3]: *** [Makefile:68: vpp-ext-deps_26.10-0_amd64.deb] Error 2
...
cp: cannot stat '*_amd64.deb': No such file or directory
make: *** [Makefile:72: ci-build] Error 1

i.e. trixie's objcopy/dh_strip (Debian 13 binutils/debhelper) can't process the DPDK dpdk-test-cmdline binary's debug sections, so no .deb is produced. Independent of any patch series.

Separately, I've addressed the two review comments on this PR: fixed the stale 0016 patch reference (0012 -> 0015), and replied on the l3-promiscuous BVI bypass (it wraps only the l2_to_bvi_dmac_check() return, i.e. the MAC check; TO_BVI_ERR_ETHERTYPE is a separate untouched path, so non-IP frames are still dropped).

Happy to open a separate tracking issue for the trixie ext-deps break if it isn't already being tracked.

Co-authored by CoPilot

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

vppbld/patches/0017-sonic-vxlan-vnet-source-independent-decap.patch:262

• Same issue as the IPv4 wildcard maintenance: the decap-any wildcard key for IPv6 is built from local dst IP + (UDP dst port, FIB, VNI), but the survivor selection also requires src_port to match. Since the decap lookup key does not include UDP source port, this can incorrectly delete the wildcard entry while another matching decap-any tunnel remains.

+	      if (ip6_address_is_equal (&st->src.ip6, &t->src.ip6) &&
+		  st->vni == t->vni &&
+		  st->encap_fib_index == t->encap_fib_index &&
+		  st->src_port == t->src_port)
+		{

vppbld/patches/0017-sonic-vxlan-vnet-source-independent-decap.patch:216

• In the decap-any delete path, the wildcard entry is keyed by (local dst IP, UDP dst port, FIB, VNI). The survivor selection additionally requires src_port to match, but the decap lookup key does not include UDP source port (see e.g. vppbld/plugins/tunterm_acl/tunterm_acl_decap.c:61-65, which keys on udp->dst_port, fib_index, and vni_reserved). This can cause the wildcard entry to be removed even though another decap-any tunnel with the same wildcard key still exists, breaking source-independent decap for that remaining tunnel.

This issue also appears on line 258 of the same file.

+	      if (st->src.ip4.as_u32 == t->src.ip4.as_u32 &&
+		  st->vni == t->vni &&
+		  st->encap_fib_index == t->encap_fib_index &&
+		  st->src_port == t->src_port)
+		{

[yejianquan]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[yejianquan]

Thanks for putting this together — I went through all three PRs (#262, sonic-sairedis#1996, sonic-mgmt#26220). Overall the series looks solid: the rollback ladders and refcounting in the saivpp side are careful, and the scope note you wrote into 0016 is exactly the kind of transparency that makes a downstream VPP patch reviewable. A few comments, none of them blocking except the CI note at the end.

0015 — inner-aware VXLAN encap hash

The diagnosis is convincing and worth calling out explicitly: vnet_l2_compute_flow_hash() reads the ethertype at current_data + l2.l2_len, and for an L3-routed tunnel l2_len is 0/unset, so the returned hash carries no inner entropy and every inner flow to a given endpoint collapses onto one outer UDP source port. That does invalidate the assumption patch 0011 documented when it put VXLAN out of scope, and fixing the entropy carrier in the encap node is the right layer to fix it at.

One suggestion: 0011 was deliberately strict about being opt-in ("Existing hash-eth-l34 and IP_FLOW_HASH_DEFAULT are byte-for-byte unchanged"). 0015 replaces vnet_l2_compute_flow_hash() unconditionally at all three call sites, so L2/EVPN tunnels take the new path too. In practice tagged and non-IP frames fall back to the generic helper so the result should be equivalent, but gating the inner-aware path on the tunnel being L3 would keep the L2 path byte-for-byte identical and stay consistent with 0011's philosophy. Worth considering if it's cheap at these call sites.

Minor: vxlan_encap_inner_flow_hash() dereferences (ip4_header_t *)(eh + 1) without a length check. The pre-existing helper isn't much stricter and the encap path generally assumes well-formed frames, so this is a nit rather than a defect — just flagging it.

0016 — global multipath load-balance tolerance

Two things worth separating here.

First, in its defence: this patch is not a nice-to-have, it's load-bearing. At the default 0.1 tolerance a 3-way ECMP group quantizes to 16 buckets, which is roughly a 6% inherent distribution skew before any hashing behaviour enters the picture. That alone would make the distribution assertions in sonic-mgmt#26220 unpassable at any tolerance in that range, so I don't think the series works without it.

Second, the reservation: as you note, multipath_next_hop_error_tolerance is a global FIB constant, so this changes bucket allocation for every ECMP group in the system, not just the VXLAN underlay, and it's the kind of change upstream is unlikely to take as a hard-coded constant. Would it be worth exposing it as a startup-config knob instead (in the spirit of 0010's no-class-e-drop option) and having sonic-vpp set it to 0.01 in startup.conf? That keeps the behaviour you need, leaves the door open to upstreaming, and lets any other consumer of this VPP build opt out. Happy to defer if that's more churn than it's worth right now, but a second opinion from someone with VPP FIB background on the bucket-memory/rebuild cost at scale would be reassuring either way.

0017 — source-independent decap

No objections. The blast radius is well contained: decap_any only takes effect for explicitly marked tunnels, the l2_to_bvi L3-promiscuous bypass is gated per ingress sw_if_index, and the mark is cleared on tunnel delete so a recycled index can't inherit it. Re-pointing the wildcard entry at a surviving sibling when tunnels share (src, vni) is a nice detail. I read your reply on the l2_to_bvi return-value question and agree the scoping makes it acceptable.

CI

The current red on 738d1df looks like infrastructure, not your patches. The ADO timeline for build 1177441 shows the amd64-trixie leg failing under Free disk space on / is lower than 5%; Currently used: 95.05%, while amd64-bookworm on the same run succeeded. A re-run should clear it — worth doing before this goes for approval, since the PR currently reads as failing.

[aaronber0614]

Thanks @yejianquan, appreciate the thorough pass. Responses per patch:

0015 — L3 gating suggestion

I dug into this and I don't think a gate is needed, because the new vxlan_encap_inner_flow_hash() is already byte-for-byte identical to vnet_l2_compute_flow_hash() for every standard L2 frame. Working it through against the pinned VPP (3f9e978), where ethernet_header_t is packed {dst[6], src[6], type} (sizeof 14, type at offset 12):

• Untagged inner IP (l2_len == 14):

  • old: l3h = eh + l2_len = eh+14; ethertype = *(u16*)(l3h-2) = *(eh+12) = eh->type; IP → ip4/6_compute_flow_hash(eh+14)
  • new: ethertype = eh->type (eh+12); l3h = eh+1 = eh+14; IP → ip4/6_compute_flow_hash(eh+14)

  Same ethertype offset, same L3 pointer, same hash.

• Tagged (VLAN): new reads eh->type == 0x8100, not IP4/IP6, so it falls back to vnet_l2_compute_flow_hash → identical.

• Non-IP (ARP/MPLS): new falls back → identical.

So L2/EVPN tunnels already take an identical path; the only frames whose hash actually changes are the L3-routed ones this patch is meant to fix. I also considered implementing the gate via vnet_buffer(b)->l2.l2_len == 0, but that field is a union opaque that the ip4/ip6-rewrite path overwrites, so it isn't reliably zero on the L3 side — gating on it would risk sending L3 frames back to the miscomputing helper and reintroducing the collapse. Given the L2 path is provably unchanged, I've left 0015 as-is; happy to add a comment recording this equivalence if you think it's worth it.

The unchecked (ip4_header_t *)(eh + 1) deref is a fair nit — as you said, it matches the pre-existing helper's assumption of well-formed frames at these encap call sites, so I've left it, but it's noted.

0016 — startup-config knob

Agreed this is the better long-term shape, and I'd like to do it — but as a follow-up rather than in this PR. Converting the const f64 into a runtime knob is real plumbing (startup-config parser + test) and, more importantly, it changes FIB bucket allocation, so it wants the VPP-FIB second opinion you mentioned plus a dev-VM re-qualification, and its own VPP_VERSION bump. Proposed design, mirroring 0010's no-class-e-drop:

load-balance {
    multipath-tolerance 0.01
}

— default preserving the upstream 0.1, with sonic-vpp's startup.conf setting 0.01. That keeps 0016's behaviour, leaves the door open to upstreaming, and lets other consumers opt out. Does splitting it into a follow-up (tracked against the same feature) work for you? For this series I've kept the hard-coded constant, since as you noted the sonic-mgmt distribution assertions can't pass without it.

CI

The trixie red has actually moved past the earlier ext-deps dh_strip toolchain break — build 1178230 builds all the VPP .debs successfully and now fails only on the agent running out of disk (/ at 95%), the infra issue you spotted. I'll re-run to land it on a clean agent before this goes up for approval.

Co-authored by CoPilot

[aaronber0614]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[yejianquan]

Thanks for the detailed responses — all three land well.

0015 — withdrawing the gating suggestion

Your equivalence argument is right and I'm happy to drop this. I checked vnet_l2_compute_flow_hash() against upstream to confirm the offsets rather than take it on trust:

u8 *l3h = (u8 *) eh + vnet_buffer (b)->l2.l2_len;
u16 ethertype = clib_net_to_host_u16 (*(u16 *) (l3h - 2));

With l2_len == 14 that reads the ethertype at eh+12 (i.e. eh->type) and sets l3h = eh+14, which is exactly what the new helper does via eh->type and eh + 1. Tagged frames hit 0x8100 and fall through to the original helper, and MPLS does too — so the special MPLS branch in the original is still reached. The L2 path really is unchanged, and the point about l2.l2_len being overwritten on the rewrite path is a good reason not to gate on it either. No change needed; a short comment recording the equivalence would be a nice-to-have for the next reader, but I don't think it's required.

The unchecked deref is fine to leave as-is given it matches the pre-existing assumption at these call sites.

0016 — follow-up split works for me

Agreed, and your reasoning for splitting is better than doing it here: the knob changes bucket allocation, so it wants its own qualification and version bump rather than riding along with the VXLAN work. The proposed shape is exactly what I had in mind:

load-balance {
    multipath-tolerance 0.01
}

defaulting to upstream 0.1 with sonic-vpp opting in via startup.conf. Keeping the hard-coded constant in this PR is the right call given the sonic-mgmt assertions depend on it. Worth linking the follow-up to the same feature issue so it doesn't get lost once this series merges.

CI

Confirmed green now (4/4). For the record, the earlier red never got as far as compiling VPP — all 17 patches applied cleanly at 12:26:30 and the failure was in vpp-ext-deps packaging (dh_strip/objcopy on dpdk-test-cmdline), with amd64-bookworm succeeding on the same commit. So the patches were never implicated.

Nothing outstanding from me on this one.

[yue-fred-gao]

smaller tolerance means more bucket with same weight distribution. It can increase memory usage with more buckets and reduce throughput (inline bucket is 4).
The thing I don't agree is treating vpp as only a test vehicle and make vpp behaves the same way as sonic-mgmt desires. We have hardware platform (cisco C0 product) using vpp as dataplane, which cares memory usage and throughput. If we look at sonic-mgmt, different platforms have different capabilities and test cases can adapt to it. vpp is no different. Is it possible to change the test case for vpp to adapt to the capability? It doesn't lose code coverage.

[aaronber0614]

@yue-fred-gao thanks for reviewing! I removed patch 0016 and we keep tolerance of 0.1. I updated the sonic-mgmt PR so that random hash check uses 0.2 for VPP only, while others stay at 0.3. I revalidated on the t1-lag-vpp testbed with 0016 dropped: Test_VxLAN_ecmp_random_hash and Test_VxLAN_entropy pass 20/20 across all four encap types, no cores, and BGP fully converged.

[yue-fred-gao]

have you considered or started upstreaming the changes to fdio/vpp? I am concerned with the growing number of patches. Each patch adds potential task of merging with upstream changes. So we should try to make the change "upstreamable". For example, don't change existing behavior and new feature can be turned on in someway.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

have you considered or started upstreaming the changes to fdio/vpp? I am concerned with the growing number of patches. Each patch adds potential task of merging with upstream changes. So we should try to make the change "upstreamable". For example, don't change existing behavior and new feature can be turned on in someway.

@yue-fred-gao yes I have considered upstreaming but you're right that it's better to make the changes "upsteramable". I will this task to my backlog items to work at.

For 0015 (inner aware VXLAN encap hash) - I think this is the best upstream candidate. vxlan_encap_inner_flow_hash() is byte for byte identical to vnet_l2_compute_flow_hash() for every existing frame type, untagged L2, tagged, and non IP all take the same path, so it does not change existing behavior. It only adds inner entropy for the L3 routed tunnel frames that today collapse onto a single outer UDP source port, which is a straight bug fix. Once it merges upstream I drop the local patch and bump vpp_version, so the series gets smaller.

For 0017 (source independent decap) - this is already opt in in the shape you want. The decap_any behavior only takes effect for tunnels explicitly marked for it, the default decap path is unchanged, and the mark is cleared on tunnel delete. It touches more API surface, so I expect it to need more discussion with the VPP maintainers, but maybe I can upstream this as an opt in feature and carry it downstream only until it lands.

For 0016 - dropped the patch

Thanks

Co-authored with CoPilot.

[yue-fred-gao]

have you considered or started upstreaming the changes to fdio/vpp? I am concerned with the growing number of patches. Each patch adds potential task of merging with upstream changes. So we should try to make the change "upstreamable". For example, don't change existing behavior and new feature can be turned on in someway.

@yue-fred-gao yes I have considered upstreaming but you're right that it's better to make the changes "upsteramable". I will this task to my backlog items to work at.

For 0015 (inner aware VXLAN encap hash) - I think this is the best upstream candidate. vxlan_encap_inner_flow_hash() is byte for byte identical to vnet_l2_compute_flow_hash() for every existing frame type, untagged L2, tagged, and non IP all take the same path, so it does not change existing behavior. It only adds inner entropy for the L3 routed tunnel frames that today collapse onto a single outer UDP source port, which is a straight bug fix. Once it merges upstream I drop the local patch and bump vpp_version, so the series gets smaller.

For 0017 (source independent decap) - this is already opt in in the shape you want. The decap_any behavior only takes effect for tunnels explicitly marked for it, the default decap path is unchanged, and the mark is cleared on tunnel delete. It touches more API surface, so I expect it to need more discussion with the VPP maintainers, but maybe I can upstream this as an opt in feature and carry it downstream only until it lands.

For 0016 - dropped the patch

Thanks

Co-authored with CoPilot.

Thanks. I will review it the next week.