[vpp] saivpp VXLAN L3 decap: source-independent decap, inner source MAC, BD teardown fix

[aaronber0614]

Description of PR

Summary:
Enable L3 VXLAN decap on the sonic-vpp platform (saivpp / vslib/vpp) and fix
several pre-existing decap setup/teardown defects found during review.

Part of sonic-net/sonic-buildimage#25777

Depends on the VPP patch series in sonic-net/sonic-platform-vpp#262 being in the
built image first. All changes are confined to vslib/vpp.

Changes:

• Source-independent (secondary-VTEP) VNET tunnel decap (decap_any) on both the
  primary and secondary L3 VNET paths; L2 EVPN tunnels intentionally keep exact
  outer-source validation.
• Set the L3 VXLAN tunnel inner source MAC to the router MAC via a new
  sw_interface_set_mac_by_index binary-API helper, so the inner Ethernet header
  matches what HW ASICs and the VNET decap tests expect (default was VPP's
  auto-generated 02:fe:.. MAC).
• Review hardening of the secondary-VTEP decap setup/teardown: refcount the
  shared VRF0 VTEP local-receive route so tearing down one VNI does not break
  its siblings; check every VPP API return in create_vxlan_decap_term and roll
  back the BD/BVI on failure instead of returning success half-programmed; and
  install the L3 decap terms from the tunnel-create path too, so a tunnel
  created after its map entry still gets programmed.
• Detach the VXLAN tunnel interface from its bridge-domain before deleting the
  BD in remove_vpp_vxlan_decap: the tunnel was still a BD member at delete time
  (it is removed later by remove_vpp_vxlan_encap), so
  vpp_bridge_domain_add_del(is_add=0) failed with -120 (BD in use). Benign (BD
  reclaimed later) but it tripped loganalyzer during teardown.

Files: vslib/vpp/{TunnelManager.cpp, TunnelManager.h, SwitchVpp.cpp,
vppxlate/SaiVppXlate.c, vppxlate/SaiVppXlate.h}.

Type of change

• Bug fix
• New feature

Approach

What is the motivation for this PR?

Enable correct L3 VXLAN decap on sonic-vpp and remove decap setup/teardown
defects: source-dependent decap, wrong inner source MAC, and the BD-in-use
teardown error.

How did you do it?

Changes in vslib/vpp (TunnelManager, SwitchVpp, SaiVppXlate) to install
source-independent decap terms, set the tunnel inner source MAC, and fix the BD
member-detach ordering on teardown.

How did you verify/test it?

Rebuilt syncd-vpp, hot-swapped onto a t1-lag-vpp testbed, and ran the VXLAN
regression suite: test_vnet_decap 4/4, test_vxlan_multiple_tunnels 16/16, and
vxlan/test_vxlan_ecmp.py with zero -120 bridge-domain-delete errors and zero
leaked bridge-domains after teardown.

Any platform specific information?

sonic-vpp (vslib/vpp) only; no other platform is affected.

Documentation

No doc/HLD changes.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[yejianquan]

Reviewed alongside sonic-platform-vpp#262 and sonic-mgmt#26220. The new decap-term code reads well — the rollback ladders in create_vxlan_decap_term() and the refcounted vxlan_secondary_vtep_local_receive() (so sibling VNIs sharing a secondary VTEP don't tear each other's local-receive down) both look right, and the BD-detach fix in remove_vpp_vxlan_decap() for the -120 "BD in use" failure is a good catch.

One thing worth fixing before merge, plus a couple of nits.

Stale patch numbers in comments

Two comments reference "patch 0014" as the VPP patch that implements the source-independent decap:

• vslib/vpp/TunnelManager.cpp — "decap matches the source-independent (src, vni) wildcard that patch 0014 registers on tunnel add"
• vslib/vpp/vppxlate/SaiVppXlate.h — "Must match VXLAN_DECAP_ANY_FLAG in the VPP 0014 patch (src/plugins/vxlan/vxlan.h)"

In the series as it stands in sonic-platform-vpp#262 that patch is 0017 (0017-sonic-vxlan-vnet-source-independent-decap.patch); 0014-sflow-per-port-api.patch is an unrelated sFlow change. This looks like the same numbering drift you already fixed on the 0016 description in 738d1df. Cosmetic, but since these two comments are precisely the pointers a future reader follows to find the other half of the contract, they're worth correcting.

For what it's worth I did check the part that actually matters: VPP_VXLAN_DECAP_ANY_FLAG here and VXLAN_DECAP_ANY_FLAG in the VPP patch are both (1u << 31), so the two sides agree.

Encoding the flag in decap_next_index

Smuggling decap_any in the high bit of the wire decap_next_index avoids an API CRC bump, which I understand as the motivation, and decap_next_index values are small enough that bit 31 is safe in practice. The cost is a silent version coupling: saivpp built with this change and run against a VPP without patch 0017 would send decap_next_index = 0x80000001 and get undefined behaviour rather than a clean failure. The VPP_VERSION bump in #262 plus in-image version locking covers this in practice — a sentence in the header comment stating that dependency explicitly (rather than only "must match") would make the coupling obvious to anyone tempted to build the two halves separately.

Nits

install_l3_vxlan_decap_terms() returns SAI_STATUS_SUCCESS on several paths that have just logged SWSS_LOG_ERROR (missing VRF, decap create failure). If that's a deliberate best-effort policy — plausible, given it's also used from the late-tunnel hook where partial state is expected — a short comment saying so would help; otherwise those look like they should propagate.

In vxlan_decap_term_set_dst(), the IPv6 branch relies on s6_addr[8] ^= 0x01 to keep ip46_address_is_ip4(dst) false. There is a corner case where that flips the wrong way: a source of the form ::0100:0000:xxxx:xxxx (bytes 0-7 zero, byte 8 == 0x01, bytes 9-11 zero) produces an all-zero top 12 bytes, i.e. exactly the misclassification the helper exists to prevent. Not a realistic VTEP address, but forcing a fixed non-zero byte and asserting the result would be airtight.

create_vxlan_decap_term() defines the rollback_bd_bvi lambda and then open-codes the same four steps in the three later failure paths. They're consistent today, which is the main thing, but reusing the lambda would keep them that way.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

Thanks for the careful read @yejianquan. Pushed ff8519fe addressing all of it.

Stale patch numbers (0014 → 0017). Fixed both pointers — the vxlan_decap_term_set_dst header in TunnelManager.cpp and the VPP_VXLAN_DECAP_ANY_FLAG comment in SaiVppXlate.h. Both said "0014" because that was the patch number when I wrote them on Jul 14; the series was later renumbered to 0015–0017 when the sflow patches (0012–0014) landed on master. Confirmed the two flag values are both (1u << 31) as you noted.

FYI, while there I noticed a third reference at SaiVppXlate.h:253 on the VPP_BOND_API_LB_ALGO_L34_INNER comment — "sonic-platform-vpp patch 0010-sonic-inner-aware-flow-hash", where that patch is actually 0011 in the current series. But that line is pre-existing (your commit 06e38d8a from the inner-aware LAG hash work), not part of this PR's diff, so I left it untouched to keep the PR scoped. Happy to fix it in a separate change if you'd like.

Version coupling of the high-bit flag. Added an explicit sentence to the SaiVppXlate.h header comment: saivpp built with this flag requires a VPP with patch 0017; against an older VPP the decap_next_index = 0x80000001 is undefined behaviour rather than a clean failure, so the two must be version-locked (which the VPP_VERSION bump + in-image version lock enforces in practice).

install_l3_vxlan_decap_terms() SUCCESS returns. Confirmed deliberate — it's called from both the map-entry hook and the late-tunnel hook, so a legitimately-incomplete intermediate state (mapper present but no tunnel yet, or vice-versa) has to return SUCCESS or it would fail an otherwise valid create. Added a comment at the top of the function stating that best-effort / idempotent policy; the logged-ERROR paths (no VRF, per-tunnel decap create failure) are skipped and re-attempted on the next hook.

IPv6 s6_addr[8] corner case. Good catch. Made it correct-by-construction: keep the s6_addr[8] ^= 0x01 flip for dst != src, and additionally force s6_addr[0] = 0x20 (fixed non-zero, non-multicast) so the top 12 bytes can never collapse to zero regardless of src — the ::0100:0:xxxx:xxxx case you described is now impossible. I didn't add an assert() because it isn't used elsewhere in this file and would be compiled out under NDEBUG; forcing the byte makes the invariant hold unconditionally, which is strictly stronger than asserting after the fact.

rollback_bd_bvi lambda. Reused it in the three post-tunnel failure paths (create_vpp_vxlan_encap, bridge, local-receive) instead of open-coding the same four steps.

[Copilot]

Pull request overview

This PR enhances the sonic-vpp (vslib/vpp) VXLAN implementation to support correct L3 VNET decapsulation behavior (including source-independent decap) and hardens VXLAN decap setup/teardown to avoid known failure modes during tunnel lifecycle operations.

Changes:

• Add support for source-independent VXLAN decap terms (“decap_any”) by encoding a flag into decap_next_index for patched VPP VXLAN behavior.
• Add a VPP binary-API helper to set a tunnel interface MAC by sw_if_index, and use it so L3 VXLAN inner source MAC matches the router MAC.
• Add L3 secondary-VTEP decap term install/teardown logic and a late-tunnel hook to handle out-of-order tunnel vs. map-entry creation; adjust BD teardown ordering to avoid BD-in-use errors.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vslib/vpp/vppxlate/SaiVppXlate.h	Adds decap-any flag encoding definitions, extends VXLAN tunnel struct, and declares sw_interface_set_mac_by_index.
vslib/vpp/vppxlate/SaiVppXlate.c	Implements sw_interface_set_mac_by_index and encodes decap_any into the wire decap_next_index.
vslib/vpp/TunnelManager.h	Introduces APIs and state for L3 secondary-VTEP decap term install/teardown and refcounting.
vslib/vpp/TunnelManager.cpp	Programs decap-any for L3 VNET, sets tunnel MAC to router MAC, adds L3 decap term install/remove logic, and fixes BD teardown ordering.
vslib/vpp/SwitchVpp.cpp	Invokes the late-tunnel hook after VXLAN tunnel creation to backfill missing L3 decap terms.

[yejianquan]

Thanks — the updates in ff8519fe all look right to me.

The IPv6 fix is better than what I suggested: forcing s6_addr[0] = 0x20 makes the invariant hold unconditionally instead of only detecting a violation after the fact, and you're right that an assert() would have been compiled out under NDEBUG anyway. The best-effort policy comment on install_l3_vxlan_decap_terms() and the explicit version-coupling sentence on the flag both answer what I was asking for.

On the third stale reference at SaiVppXlate.h:253 — good catch, and thanks for leaving it out of scope. That one is mine (06e38d8a); I'll put up a separate one-line change for it rather than widen this PR.

One thing I'd like to flag before this merges

I think the Copilot comment on the early return in install_l3_vxlan_decap_terms() is a genuine gap, not a false positive, and it happens to sit on the exact scenario this series enables. The guard is keyed on the map-entry OID:

sai_object_id_t term_oid;
sai_deserialize_object_id(map_entry_serialized_oid, term_oid);
if (m_vxlan_decap_term_map.find(term_oid) != m_vxlan_decap_term_map.end()) {
    return SAI_STATUS_SUCCESS;   // already installed
}

but the work it guards is per-tunnel — the function goes on to loop over every tunnel referencing the mapper and create a decap term for each, storing them as a vector under that one key. So:

1. Map entry E is created while tunnel A already references mapper M. A term is installed for A and m_vxlan_decap_term_map[E] becomes non-empty.
2. Tunnel B is created later against the same mapper M. handle_l3_vxlan_tunnel_create(B) walks M's entries, reaches E, and returns at the guard above.
3. B's secondary-VTEP decap term is never installed, and nothing retries it.

The pure late-tunnel case you designed the hook for is safe, because when no tunnel exists yet created stays empty and the if (!created.empty()) means nothing is recorded, so the next hook re-attempts. The same is true when the only candidate is the primary VTEP and is_local_skip fires. The gap needs at least one non-local tunnel installed at map-entry time plus a further tunnel arriving afterwards — which is plausibly what a multi-VTEP ordering in test_vxlan_multiple_tunnels.py produces, and that test is being added to the t1-lag-vpp PR set in sonic-mgmt#26220.

Making the guard per (map entry, VTEP) rather than per map entry would close it — e.g. keep the vector but skip only the tunnels whose VTEP IP already has a term recorded under that entry, so a newly-appearing tunnel still gets one. That also makes the function properly idempotent in the sense the comment claims.

Worth confirming against a run of test_vxlan_multiple_tunnels.py once the image carries these changes, since that's the case that would expose it.

On the other three Copilot comments

For what it's worth, I read the one about the handle_l2_vxlan_tunnel_map_entry reference in the handle_l3_vxlan_tunnel_create() comment as a false positive: the VNI_TO_VIRTUAL_ROUTER_ID branch genuinely does live inside handle_l2_vxlan_tunnel_map_entry, so the comment is accurate — the function name is just historically misleading. Renaming it is out of scope here.

The two about unchecked return codes (sw_interface_set_mac_by_index() in create_vpp_vxlan_encap(), and the unbridge in remove_vpp_vxlan_decap()) both seem fair to me. The MAC one especially: if it fails the tunnel silently keeps the 02:fe:.. auto MAC, which is the exact condition the change exists to prevent, and the caller still sees success — at minimum worth an error log.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

Thanks @yejianquan — you're right on all three, and I've pushed the fix as fc3bb733.

Per-(map entry, VTEP) guard

Implemented exactly as you suggested. I kept the vector under the map-entry key but replaced the whole-entry early-return with per-VTEP idempotency. The function now:

• looks up the entry's existing vector (if any) instead of returning on its presence;
• in the tunnel loop, skips only VTEPs whose source IP already has a term recorded — either under this entry from a prior hook (existing) or installed earlier in the same pass (created) — using the existing saivs::sai_ip_address_equal on TunnelVPPData.src_ip;
• appends the newly-created terms to the entry's vector (vec.insert(vec.end(), ...)) instead of overwriting it.

So the step-3 scenario you described — tunnel B arriving after entry E was processed with tunnel A already termed — now installs B's decap term and appends it, while A is skipped. That also makes the function properly idempotent in the sense the comment claims. I checked the teardown path (handle_l2_vxlan_tunnel_map_entry_removal) iterates the full vector, so appending is safe on removal.

The two unchecked return codes

Both added as (non-fatal) error logs:

• sw_interface_set_mac_by_index() in create_vpp_vxlan_encap() — logs if setting the router MAC fails, so the silent 02:fe:.. auto-MAC fallback (the exact condition the change exists to prevent) is now diagnosable instead of returning success.
• the unbridge set_sw_interface_l2_bridge_by_index(..., is_add=false, ...) in remove_vpp_vxlan_decap() — logs if the detach fails, which is what precedes the -120 (BD in use) on the subsequent BD delete.

The handle_l2_vxlan_tunnel_map_entry naming comment

Agreed — false positive, left as-is. The VNI_TO_VIRTUAL_ROUTER_ID branch does live in handle_l2_vxlan_tunnel_map_entry; the name is just historically misleading.

Verification

Built these changes into a docker-syncd-vpp image and ran test_vxlan_multiple_tunnels.py on a t1-lag-vpp testbed — i.e. the multi-VTEP ordering you flagged: 16/16 passed, BGP 24/24 established, no new cores.

And thanks for taking the third stale reference at SaiVppXlate.h:253 in a separate change — left it out of scope here.

Co-authored with CoPilot

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

vslib/vpp/TunnelManager.cpp:470

• remove_vpp_vxlan_decap() returns the bridge-domain id to dynamic_bd_id_pool before attempting to delete the bridge domain. If vpp_bridge_domain_add_del(bd_id, false) fails (e.g., BD still in use), the id can be reallocated while the old bridge-domain still exists; the current code also logs a successful delete unconditionally.

This issue also appears on line 1305 of the same file.

    m_switch_db->dynamic_bd_id_pool.free(tunnel_data.bd_id);
    refresh_interfaces_list();
    //bd is create automatically when the fist interface is add to it but requires manual deletion
    vpp_bridge_domain_add_del(tunnel_data.bd_id, false);

vslib/vpp/TunnelManager.cpp:1309

• remove_vxlan_decap_term() frees the bridge-domain id back to dynamic_bd_id_pool before deleting the bridge-domain (and ignores the delete return code). If vpp_bridge_domain_add_del() fails, the id can be reused while the old bridge-domain still exists; the current code also logs successful teardown regardless of BD delete result.

    snprintf(hw_bvi_ifname, sizeof(hw_bvi_ifname), "bvi%u", tunnel_data.bd_id);
    delete_bvi_interface(hw_bvi_ifname);
    m_switch_db->dynamic_bd_id_pool.free(tunnel_data.bd_id);
    refresh_interfaces_list();
    vpp_bridge_domain_add_del(tunnel_data.bd_id, false);

vslib/vpp/TunnelManager.cpp:1297

• remove_vxlan_decap_term() deletes the decap-only VXLAN tunnel without setting req.decap_any=true. Since vpp_vxlan_tunnel_add_del encodes the source-independent decap behavior via the high-bit flag, omitting it on delete can leave the decap-any entry/tunnel state behind in VPP.

    req.instance = ~0;
    req.vni = tunnel_data.vni;
    req.decap_next_index = ~0;
    sai_ip_address_t src_nc = tunnel_data.src_ip;
    sai_ip_address_t_to_vpp_ip_addr_t(src_nc, req.src_address);

vslib/vpp/SwitchVpp.cpp:1528

• handle_l3_vxlan_tunnel_create() is invoked unconditionally even if create_l2_vxlan_tunnel() fails and SwitchVpp::create returns a failure status. That can program L3 secondary-VTEP decap state for a tunnel create that ultimately fails, potentially leaking VPP resources.

        sai_status_t status = m_tunnel_mgr.create_l2_vxlan_tunnel(object_id, sw_if_index);
        SWSS_LOG_INFO("L2 VXLAN tunnel create for %s: status=%d sw_if_index=%u",
            serializedObjectId.c_str(), status, sw_if_index);

        // Late-tunnel hook: install any L3 secondary-VTEP decap terms whose
        // TUNNEL_MAP_ENTRY was created before this tunnel existed (M3).
        m_tunnel_mgr.handle_l3_vxlan_tunnel_create(object_id);
        return status;

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

@yejianquan quick follow up on your cross note about the map entry granularity early return in install_l3_vxlan_decap_terms().

Fixed in fc3bb733. Rather than returning as soon as the map entry has any recorded term, it now skips only the VTEPs that already have a decap term under that entry and installs one for any tunnel that appears later, appending to the entry's vector instead of overwriting it. That also covers the Copilot comment on the same early return, plus the two unchecked return codes (the router MAC in create_vpp_vxlan_encap and the unbridge in remove_vpp_vxlan_decap).

Worth recording for the merge notes though. I ran test_vxlan_multiple_tunnels.py on a t1-lag-vpp testbed with a docker-syncd-vpp image carrying the fix and it passes 16 of 16, and it also passes on the pre fix base ff8519fe. So the exact ordering that would expose the gap you described isn't actually reproduced by the test as it stands today. I'm treating the change as a defensive correctness fix for the multi VTEP decap path rather than something the current test forces, and the single VTEP path behaves exactly as before.

For the record the run stayed healthy end to end, BGP 24 of 24 and no new cores. If you think it's worth locking down I can add a dedicated multi VTEP ordering case to the sonic-mgmt set as a follow up so this path gets explicit coverage.