[vpp] VXLAN L3 dataplane test enablement

[aaronber0614]

Description of PR

Summary:
Enable the L3 VXLAN test cases on the sonic-vpp platform and make the
hash-distribution checks statistically robust on the vpp dataplane.

Part of sonic-net/sonic-buildimage#25777

This is the test layer (layer 4) of a cross-repo feature and MUST merge after
the dataplane and SAI layers are in a built sonic-vpp image, otherwise the
newly-enabled tests fail in CI. Landing order:

1. [vpp] VXLAN L3 dataplane: inner-aware encap hash, underlay ECMP, source-independent decap (patches 0015-0017) sonic-platform-vpp#262 - VPP patches (first)
2. [vpp] saivpp VXLAN L3 decap: source-independent decap, inner source MAC, BD teardown fix sonic-sairedis#1996 - saivpp adaptation
3. sonic-buildimage submodule bump (auto via mssonicbld)
4. this PR - tests (last)

Note for reviewers: the t1-lag-vpp Elastictest leg on this PR is
currently expected to fail. The PR CI builds a sonic-vpp KVM image from
master buildimage, whose platform/vpp and src/sonic-sairedis
submodules do not yet contain the dataplane (#262) and saivpp (#1996)
changes these tests exercise. Once #262 and #1996 merge and mssonicbld
advances the submodule pointers, the rebuilt image carries the VNET-decap
dataplane and this leg passes (validated locally: test_vnet_decap 4/4,
test_vxlan_multiple_tunnels 16/16 on a full image built from the paired
PRs).

Changes:

• conditional_mark (tests_mark_conditions.yaml and
  tests_mark_conditions_sonic_vpp.yaml) plus per-test skip markers in
  test_vnet_decap.py and test_vxlan_bfd_tsa.py: remove the asic_type vpp skip
  so test_vxlan_ecmp (random_hash and entropy), test_vnet_decap, and
  test_vxlan_bfd_tsa run on sonic-vpp. Originally from
  debug vxlan vpp tests #25480 by Chenyang Wang; authorship preserved.
• test_vxlan_ecmp.py: statistically-justified tolerance and packet-count for the
  random_hash and entropy distribution checks, paired with the VPP-side
  load-balance tolerance change so the 2-endpoint splits pass reliably rather
  than flaking at a too-tight bound.

Supersedes debug PR #25480.

Type of change

• Test case improvement
• Testbed and Framework(new/improvement)

Back port request

• 202311
• 202405
• 202411
• 202505
• 202511
• 202512
• 202605

Approach

What is the motivation for this PR?

The L3 VXLAN test cases were skipped on the vpp platform. With the dataplane
(patches 0015-0017) and saivpp decap support in place, enable them and make the
hash-distribution checks statistically robust so they do not flake.

How did you do it?

Removed the asic_type vpp skip conditions for the VXLAN tests and raised the
packet-count/tolerance on the random_hash and entropy distribution checks in
test_vxlan_ecmp.py.

How did you verify/test it?

Validated on a t1-lag-vpp testbed against a sonic-vpp image built from the
paired PRs: fib/test_fib.py 16/16 hashing pass; vxlan/test_vxlan_ecmp.py
random_hash and entropy pass; test_vnet_decap and test_vxlan_bfd_tsa green.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[github-actions]

@StormLiangMS @wangxin @yxieca A user wants to merge changes to the conditional mark files into master. Please review.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Enables L3 VXLAN dataplane test coverage on the SONiC-VPP platform and reduces flakiness in VXLAN ECMP distribution assertions by increasing sample size and loosening tolerance bounds.

Changes:

• Enable VXLAN tests on asic_type=vpp by relaxing conditional-mark skips and platform guards.
• Make VXLAN ECMP random-hash / entropy distribution checks statistically more robust (higher packet count, higher tolerance).
• Add newly-enabled VXLAN tests to the t1-lag-vpp PR pipeline leg.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/vxlan/test_vxlan_ecmp.py	Increase packet_count and tolerance for random-hash and entropy checks to reduce distribution-test flakiness.
tests/vxlan/test_vxlan_bfd_tsa.py	Allow VPP asic type to use the standard tolerance path (enables test execution on VPP).
tests/vxlan/test_vnet_decap.py	Allow VPP asic type past the ASIC gating so VNET decap tests can run on VPP.
tests/common/plugins/conditional_mark/tests_mark_conditions.yaml	Adjust skip condition so test_vxlan_multiple_tunnels can run on VPP even when platform isn’t in the allowlist.
tests/common/plugins/conditional_mark/tests_mark_conditions_sonic_vpp.yaml	Remove the VPP-specific conditional-mark skips for VXLAN ECMP/entropy/underlay-ECMP tests (now enabled on VPP).
.azure-pipelines/pr_test_scripts.yaml	Add VXLAN multiple-tunnels and BFD TSA tests to the t1-lag-vpp PR test list.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

tests/vxlan/test_vxlan_ecmp.py:1505

• The new statistical rationale comment has an incorrect std/mean formula for per-nexthop counts in an N-way ECMP split. For a multinomial with p=1/N, std/mean = sqrt((N-1)/packet_count) (N cancels), not sqrt((N-1)/(N*packet_count)). This also makes the stated sigma margins incorrect and could mislead future tuning of packet_count/tolerance.

        # Overlay ECMP distribution over N nexthops is multinomial: each
        # nexthop's received count has std/mean = sqrt((N-1)/(N*packet_count)).
        # With N=3 and packet_count=1000 that is ~2.6%, so the default 3%
        # tolerance is only ~1.1 sigma and this check flakes on a perfectly
        # healthy dataplane. Send more packets (better resolution) and use a

.azure-pipelines/pr_test_scripts.yaml:734

• The t1-lag-vpp test list contains duplicate entries (vnet_bgp_route_precedence and vxlan_route_advertisement), which can cause the same tests to run twice and unnecessarily extend CI runtime. Since this list is being edited, please deduplicate it.

  - vxlan/test_vnet_bgp_route_precedence.py
  - vxlan/test_vxlan_multi_tunnel.py
  - vxlan/test_vxlan_route_advertisement.py
  - vxlan/test_vxlan_multiple_tunnels.py
  - vxlan/test_vxlan_bfd_tsa.py

[yejianquan]

Reviewed together with sonic-platform-vpp#262 and sonic-sairedis#1996. The enablement side looks good, and un-skipping Test_VxLAN_ecmp_random_hash, Test_VxLAN_entropy and Test_VxLAN_underlay_ecmp is a real coverage gain for the t1-lag-vpp topology. One scoping suggestion on the tolerance change.

The statistics check out

I worked through the justification in the test_vxlan_random_hash comment independently and it holds up. packet_count is per-nexthop (the PTF sender loops over test_nhs sending packet_count each, and the assertion compares each nexthop against packet_count), so with N nexthops the per-bin relative standard deviation really is sqrt((N-1)/(N*packet_count)). Reproducing your numbers: N=3, packet_count=1000 gives 2.58% (your ~2.6%), so the 3% default was ~1.16 sigma (your ~1.1); at packet_count=2000 the 7% tolerance is ~3.83 sigma (your ~3.8).

A quick Monte-Carlo over the actual pass condition (test fails if any nexthop falls outside the band), assuming idealised per-packet hashing:

configuration	failure rate
packet_count=1000, tolerance=3%	~46%
packet_count=2000, tolerance=7%	~0.03%

So this isn't papering over a marginal check — the old parameters were genuinely unusable for any hash that behaves randomly, and 7% still catches a real >7% imbalance. Worth noting this interacts with patch 0016 in #262: at the default VPP multipath tolerance a 3-way group quantizes to 16 buckets (~6% inherent skew), so on VPP the old 3% assertion could not have passed regardless of packet count.

Suggestion: scope the relaxation to VPP

The tolerance and packet_count changes are unconditional, so they also apply to cisco-8000, mellanox, vs and marvell-teralynx. Those platforms have been passing at 3%, which suggests their hashing over the sequentially-incrementing tcp_dport distributes more uniformly than the idealised random model — in which case 3% was a meaningful check for them and 7% loosens it for no benefit. Doubling packet_count also doubles the runtime of these cases on every platform.

setUp() already has a per-asic hook (data['tolerance'], set alongside underlay_tolerance), so keying the looser tolerance off asic_type there — 0.07 for vpp, 0.03 elsewhere — would get the flake fix where it's needed while leaving the other platforms' sensitivity and runtime untouched. Same for the packet_count bump if that's easy to thread through.

Minor

The two Copilot comments above (the test_vnet_decap.py skip message and the conditional_mark YAML) both look already addressed in the current diff — the message does mention VPP now, and the vpp skip block is deleted rather than commented out. Probably safe to resolve them.

The failing impacted-area-kvmtest-t1-lag-vpp leg is expected until #262 and #1996 land and the submodules advance, as your description notes — flagging only so it isn't mistaken for a real regression at merge time.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

Thanks @yejianquan. Pushed a22f66407 scoping the relaxation to VPP as you suggested.

Scope to VPP. You're right that the tolerance (0.03 → 0.07) and packet_count (1000 → 2000) were unconditional and loosened the check / doubled runtime for cisco-8000, mellanox, vs and marvell-teralynx too. Added dedicated per-asic keys in setUp() alongside data['tolerance']:

if asic_type == "vpp":
    data['ecmp_hash_tolerance'] = 0.07
    data['ecmp_hash_packet_count'] = 2000
else:
    data['ecmp_hash_tolerance'] = 0.03
    data['ecmp_hash_packet_count'] = 1000

and read them at the random-hash and entropy call sites. I used dedicated keys rather than overloading data['tolerance'], since that value is shared by the other balancing checks. test_verify_entropy's intentional tolerance=0.75 is untouched. Net effect: VPP gets the flake fix, the other platforms keep the tighter 3% sensitivity and the shorter 1000-packet runtime. Thanks also for independently reproducing the sigma math — it matches what I had.

Copilot comments. Resolved both — the test_vnet_decap.py skip message now mentions VPP, and the conditional_mark block is deleted rather than commented, as you noted.

t1-lag-vpp leg. Confirmed — expected red until #262 and sonic-sairedis#1996 land and the submodules advance; not a real regression.

Co-authored by CoPilot

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[yejianquan]

Thanks — I checked a22f664075 and the scoping is exactly what I was after:

if asic_type == "vpp":
    data['ecmp_hash_tolerance'] = 0.07
    data['ecmp_hash_packet_count'] = 2000
else:
    data['ecmp_hash_tolerance'] = 0.03
    data['ecmp_hash_packet_count'] = 1000

Introducing dedicated ecmp_hash_* keys rather than overloading the existing tolerance is cleaner than what I suggested — the underlay tolerances stay independent, and cisco-8000 / mellanox / vs / marvell-teralynx keep both their 3% sensitivity and their original runtime. The call sites reading self.vxlan_test_setup['ecmp_hash_tolerance'] / ['ecmp_hash_packet_count'] all look consistent.

No further comments from me on this PR. The remaining impacted-area-kvmtest-t1-lag-vpp red is the expected image dependency on sonic-platform-vpp#262 and sonic-sairedis#1996, as your description says.

One cross-PR note so it isn't lost in the merge sequencing: I've flagged what I believe is a real gap in sonic-sairedis#1996 (the map-entry-granularity early return in install_l3_vxlan_decap_terms() skipping decap terms for tunnels that appear after the map entry). If that analysis holds it would surface precisely in test_vxlan_multiple_tunnels.py, which this PR adds to the t1-lag-vpp set — so it's worth watching that case specifically on the first full-image run after #1996 lands.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).