fix(cgnat): key mappings and bypass by the session vrf on every programming path - #505
Merged
Conversation
BSpendlove
temporarily deployed
to
integration-rig
August 27, 2026 15:22 — with
GitHub Actions
Inactive
BSpendlove
force-pushed
the
fix/cgnat-inside-vrf
branch
from
August 27, 2026 15:28
e92f57b to
f9bc1b8
Compare
BSpendlove
temporarily deployed
to
integration-rig
August 27, 2026 15:28 — with
GitHub Actions
Inactive
BSpendlove
force-pushed
the
fix/cgnat-inside-vrf
branch
from
August 28, 2026 16:39
f9bc1b8 to
485ba2c
Compare
BSpendlove
force-pushed
the
fix/cgnat-inside-vrf
branch
from
August 28, 2026 20:45
485ba2c to
d39ef08
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CGNAT only works in the default VRF. Activation, release and bypass pass VRF 0 to the allocator and to VPP while
sess.VRFis in hand in the same function, so the plugin keys the mapping under fib 0 and looks it up under the subscriber's real fib: every packet from a subscriber in a non-default VRF takes aNO_MAPPINGdrop. Two subscribers with the same inside address in different VRFs, the case ADR 0006 exists for, collide in the allocator instead of coexisting. Finding B2 of the August CGNAT audit. Closes #481.Building the rig case the issue asks for (two customer VRFs sharing one inside prefix and one outside pool) found three more places where identity was the address alone, each of which stopped the second VRF before CGNAT was reached. They are fixed here as separate commits because the suite cannot pass without them; say if you want any of them split out.
Change
internal/cgnat: activation, HA-sync restore, bypass and release resolve the session's VRF name to its table id through the resolver the component already holds (typed as the existingvrfResolverinterface so tests can inject one) and pass it to the allocator and to every southbound call. A name that does not resolve is refused and logged, never keyed by table 0. The synced-mapping path keys by the local resolution rather than the peer's table id, since the identity that crosses nodes is the VRF name. Reconcile's inside-prefix path errors on an unresolvable VRF instead of programming table 0.plugins/dhcp4/local: the provider's lease map was keyed by address alone, so the second VRF's DISCOVER for an address the first VRF had leased was refused withalready leased. Leases are keyed by (pool, address); pools are VRF-scoped by config.pkg/dhcp:ResolveV4filledPoolNameonly on first allocation, so the REQUEST after DISCOVER carried an empty pool and the lease key above degraded to address-only on that packet. Later packets keep the pool the address came from.pkg/ifmgr: the address index held one interface per address, so with 100.64.0.1 on a loopback in each customer VRFHasIPv4InFIB(gateway, table)answered for one VRF only and the ARP component ignored the other VRF's gateway ARP; those subscribers never left setup. The index now holds every interface carrying an address with its table, copy-on-write so the punt-path readers stay lock-free.pkg/vrfmgr,pkg/southbound/vpp: a VRF declared IPv4-only got one VPP table, so its gateway loopback sat in IPv6 table 0 while linux-cp (which mirrors the Linux VRF, dual-family by nature) had already created IPv6 table N and the IPoE plugin bound every session interface to it. VPP refuses unnumbered unless both ends share every table, so no session in that VRF could take the loopback as its unnumbered source; the failure was only logged. Every VRF now gets both VPP tables and every member is bound for both families; the declared families keep gating what is provisioned inside the VRF. Suite 53 declares CUSTOMER-B IPv4-only to hold this.test-infra/vpp-plugins:osvbng_cgnat_plugin.soandosvbng_pppoe_plugin.soare the binaries the rig runs below used, built withmake vpp-dev(release tree, VPP v26.06) from osvbng-vpp main plus veesix-networks/osvbng-vpp#32 and veesix-networks/osvbng-vpp#33, so CI exercises the plugin side of this fix. They are dev-loop builds, stripped, hence smaller than the release-built drop around them; the nextmake bump-submodulesafter those PRs merge replaces them from a release build.internal/pppoe,internal/ipoe: PPPoE's normal teardown published Released without the address, VRF or service group, so CGNAT's handler returned before releasing the block and every PPPoE mapping leaked until restart (audit B1, #479). The event now carries the identity the session held, like the VPP-failure path already did; IPoE's Released events gain the service group for the same reason (bypass release keys on it).internal/cgnat: a block the allocator still held for a (vrf, ip), left by a session whose release never arrived, was committed for the next session without a dataplane call, so that subscriber forwarded untranslated (the second half of #479). The dataplane is programmed on every activation; the plugin's add is a no-op for an identical mapping and refreshes one whose interface moved.tests/29-radius-vrf,tests/30-l2tp-lns,tests/31-l2tp-lac:OSVBNG_NUM_INTERFACEScounted one interface too few, so the entrypoint started VPP before the last data link was wired and lost the race on this box; the count now covers every link.tests/53-cgnat-vrf: new suite. An IPoE group in CUSTOMER-A (dual-family) and a PPPoE group in CUSTOMER-B (IPv4-only) share 100.64.0.0/24 and theresidentialpool in the default VRF. It asserts the same inside address is live in both VRFs, one mapping per session keyed by its VRF id with every (outside, block) distinct, VPP holding one mapping per session, NAT streams verified in both VRFs, and after release (DHCP release and PADT) no session and no mapping left.tests/common.robot's log capture now reads containerlab 0.73's inspect JSON (keyed by lab name), which is why failed suites left no container logs behind. The suite is intests/ci-suites.txt, so the per-PR tier runs the case this PR is about.Verification
Unit:
make testgreen (the threepkg/componentstate-file tests fail only while a VPP lab is pinning cores on the same box; they pass on the idle box). New tests: five ininternal/cgnat(per-VRF keying, same address in two VRFs gets disjoint blocks, release under the VRF key, unresolvable VRF refused, bypass carries the VRF), two inplugins/dhcp4/local, one inpkg/dhcp, two inpkg/ifmgr.golangci-lint run --new-from-rev=origin/mainreports 0 issues.Rig, dataplane provenance: the two plugins committed to the drop in this PR, see above.
Rig, final sequence on this image,
tests/rf-run.sh, one run each, back to back:53-cgnat-vrf15 of 15 with CUSTOMER-B IPv4-only,24-vrf-lite19 of 19 (VRF regression, dual-family VRFs with the LCP namespace),08-cgnat-ipoe-pba14 of 14. By hand on the same image with both VRFs IPv4-only: zeroset unnumbered failedin the BNG log, andset interface unnumbered <session> use loop102from vppctl succeeds as programmed, where before the loopback had to be rebound to IPv6 table 101 first. An earlier 53 run before the VRF-table fix, with the suite as merged here, had every session logset unnumbered failed: retval=-74; a run with both VRFs dual-family did not, which is how the table mismatch was found. The PPPoE decap bound was checked by hand before and after (fib index 10 refused with -76 on the committed drop, accepted on the rebuilt plugin); details in veesix-networks/osvbng-vpp#33.One earlier sequence, on the first image built during this work, had 08 and 10 fail their health check: the BNG container never logged started and nothing in the lab answered. Not reproduced by hand with that image nor in the final sequence, and the container-log capture that would have explained it was broken at the time (fixed here).
This box has 172.20.0.0/16 taken by another Docker network, so the runs above were made on a local branch carrying #506 with
OSVBNG_LAB_MGMT_PREFIX=172.31; the suites themselves are the ones in this PR.Plugin error path, on the rig, from a small Go client using the osvbng bindings against a deployed BNG (plugins from osvbng-vpp#32 at its second commit): bypass add and delete for table 4090 answer
No such FIB / VRF (-3), the same for table 0 answer 0, inside prefix add and subscriber mapping add for table 4090 answer -3, an unknown pool still answersNo such entry (-6)first, and the plugin records nothing for a refused call (its inside-prefix handler used to append deterministic parameters and logaddedregardless; fixed in #32).Full sweep on this box, every non-skipped suite once, in order, on the final plugin drop and image, the 19 suites with a private management network run from throwaway copies rewritten to 172.31 so they could deploy here:
The six that were red in that pass, and what each one was:
create host-interface eth1: Invalid interface (-71): its topology setOSVBNG_NUM_INTERFACES: "1", and the entrypoint builds its wait list as eth0 plus eth1 to eth(N-1), so it waited for eth0 alone and started VPP before containerlab had wired eth1. Main passes it by timing (11 of 11 on a main-built image here); the branch lost twice. The count is now 2, and 29 and 31 had the same undercount (three data links, count 3) and are corrected to 4; 30 passes 11 of 11 twice after; 31 passes 11 of 11 with its count; 29 failed once more on a different window (eth2 present at the wait, gone by the time VPP created it, containerlab still wiring) and passes 11 of 11 on the next run.