Skip to content

fix(cgnat): key mappings and bypass by the session vrf on every programming path - #505

Merged
BSpendlove merged 11 commits into
mainfrom
fix/cgnat-inside-vrf
Aug 28, 2026
Merged

fix(cgnat): key mappings and bypass by the session vrf on every programming path#505
BSpendlove merged 11 commits into
mainfrom
fix/cgnat-inside-vrf

Conversation

@BSpendlove

@BSpendlove BSpendlove commented Aug 27, 2026

Copy link
Copy Markdown
Member

Problem

CGNAT only works in the default VRF. Activation, release and bypass pass VRF 0 to the allocator and to VPP while sess.VRF is in hand in the same function, so the plugin keys the mapping under fib 0 and looks it up under the subscriber's real fib: every packet from a subscriber in a non-default VRF takes a NO_MAPPING drop. Two subscribers with the same inside address in different VRFs, the case ADR 0006 exists for, collide in the allocator instead of coexisting. Finding B2 of the August CGNAT audit. Closes #481.

Building the rig case the issue asks for (two customer VRFs sharing one inside prefix and one outside pool) found three more places where identity was the address alone, each of which stopped the second VRF before CGNAT was reached. They are fixed here as separate commits because the suite cannot pass without them; say if you want any of them split out.

Change

internal/cgnat: activation, HA-sync restore, bypass and release resolve the session's VRF name to its table id through the resolver the component already holds (typed as the existing vrfResolver interface so tests can inject one) and pass it to the allocator and to every southbound call. A name that does not resolve is refused and logged, never keyed by table 0. The synced-mapping path keys by the local resolution rather than the peer's table id, since the identity that crosses nodes is the VRF name. Reconcile's inside-prefix path errors on an unresolvable VRF instead of programming table 0.

plugins/dhcp4/local: the provider's lease map was keyed by address alone, so the second VRF's DISCOVER for an address the first VRF had leased was refused with already leased. Leases are keyed by (pool, address); pools are VRF-scoped by config.

pkg/dhcp: ResolveV4 filled PoolName only on first allocation, so the REQUEST after DISCOVER carried an empty pool and the lease key above degraded to address-only on that packet. Later packets keep the pool the address came from.

pkg/ifmgr: the address index held one interface per address, so with 100.64.0.1 on a loopback in each customer VRF HasIPv4InFIB(gateway, table) answered for one VRF only and the ARP component ignored the other VRF's gateway ARP; those subscribers never left setup. The index now holds every interface carrying an address with its table, copy-on-write so the punt-path readers stay lock-free.

pkg/vrfmgr, pkg/southbound/vpp: a VRF declared IPv4-only got one VPP table, so its gateway loopback sat in IPv6 table 0 while linux-cp (which mirrors the Linux VRF, dual-family by nature) had already created IPv6 table N and the IPoE plugin bound every session interface to it. VPP refuses unnumbered unless both ends share every table, so no session in that VRF could take the loopback as its unnumbered source; the failure was only logged. Every VRF now gets both VPP tables and every member is bound for both families; the declared families keep gating what is provisioned inside the VRF. Suite 53 declares CUSTOMER-B IPv4-only to hold this.

test-infra/vpp-plugins: osvbng_cgnat_plugin.so and osvbng_pppoe_plugin.so are the binaries the rig runs below used, built with make vpp-dev (release tree, VPP v26.06) from osvbng-vpp main plus veesix-networks/osvbng-vpp#32 and veesix-networks/osvbng-vpp#33, so CI exercises the plugin side of this fix. They are dev-loop builds, stripped, hence smaller than the release-built drop around them; the next make bump-submodules after those PRs merge replaces them from a release build.

internal/pppoe, internal/ipoe: PPPoE's normal teardown published Released without the address, VRF or service group, so CGNAT's handler returned before releasing the block and every PPPoE mapping leaked until restart (audit B1, #479). The event now carries the identity the session held, like the VPP-failure path already did; IPoE's Released events gain the service group for the same reason (bypass release keys on it).

internal/cgnat: a block the allocator still held for a (vrf, ip), left by a session whose release never arrived, was committed for the next session without a dataplane call, so that subscriber forwarded untranslated (the second half of #479). The dataplane is programmed on every activation; the plugin's add is a no-op for an identical mapping and refreshes one whose interface moved.

tests/29-radius-vrf, tests/30-l2tp-lns, tests/31-l2tp-lac: OSVBNG_NUM_INTERFACES counted one interface too few, so the entrypoint started VPP before the last data link was wired and lost the race on this box; the count now covers every link.

tests/53-cgnat-vrf: new suite. An IPoE group in CUSTOMER-A (dual-family) and a PPPoE group in CUSTOMER-B (IPv4-only) share 100.64.0.0/24 and the residential pool in the default VRF. It asserts the same inside address is live in both VRFs, one mapping per session keyed by its VRF id with every (outside, block) distinct, VPP holding one mapping per session, NAT streams verified in both VRFs, and after release (DHCP release and PADT) no session and no mapping left. tests/common.robot's log capture now reads containerlab 0.73's inspect JSON (keyed by lab name), which is why failed suites left no container logs behind. The suite is in tests/ci-suites.txt, so the per-PR tier runs the case this PR is about.

Verification

Unit: make test green (the three pkg/component state-file tests fail only while a VPP lab is pinning cores on the same box; they pass on the idle box). New tests: five in internal/cgnat (per-VRF keying, same address in two VRFs gets disjoint blocks, release under the VRF key, unresolvable VRF refused, bypass carries the VRF), two in plugins/dhcp4/local, one in pkg/dhcp, two in pkg/ifmgr. golangci-lint run --new-from-rev=origin/main reports 0 issues.

Rig, dataplane provenance: the two plugins committed to the drop in this PR, see above.

Rig, final sequence on this image, tests/rf-run.sh, one run each, back to back: 53-cgnat-vrf 15 of 15 with CUSTOMER-B IPv4-only, 24-vrf-lite 19 of 19 (VRF regression, dual-family VRFs with the LCP namespace), 08-cgnat-ipoe-pba 14 of 14. By hand on the same image with both VRFs IPv4-only: zero set unnumbered failed in the BNG log, and set interface unnumbered <session> use loop102 from vppctl succeeds as programmed, where before the loopback had to be rebound to IPv6 table 101 first. An earlier 53 run before the VRF-table fix, with the suite as merged here, had every session log set unnumbered failed: retval=-74; a run with both VRFs dual-family did not, which is how the table mismatch was found. The PPPoE decap bound was checked by hand before and after (fib index 10 refused with -76 on the committed drop, accepted on the rebuilt plugin); details in veesix-networks/osvbng-vpp#33.

One earlier sequence, on the first image built during this work, had 08 and 10 fail their health check: the BNG container never logged started and nothing in the lab answered. Not reproduced by hand with that image nor in the final sequence, and the container-log capture that would have explained it was broken at the time (fixed here).

This box has 172.20.0.0/16 taken by another Docker network, so the runs above were made on a local branch carrying #506 with OSVBNG_LAB_MGMT_PREFIX=172.31; the suites themselves are the ones in this PR.

Plugin error path, on the rig, from a small Go client using the osvbng bindings against a deployed BNG (plugins from osvbng-vpp#32 at its second commit): bypass add and delete for table 4090 answer No such FIB / VRF (-3), the same for table 0 answer 0, inside prefix add and subscriber mapping add for table 4090 answer -3, an unknown pool still answers No such entry (-6) first, and the plugin records nothing for a refused call (its inside-prefix handler used to append deterministic parameters and log added regardless; fixed in #32).

Full sweep on this box, every non-skipped suite once, in order, on the final plugin drop and image, the 19 suites with a private management network run from throwaway copies rewritten to 172.31 so they could deploy here:

Suite Result
01-smoke 10 of 10
02-smoke-ha 26 of 26
03-ipoe-local 8 of 8
04-pppoe-local 9 of 9
05-ipoe-radius 11 of 11
06-pppoe-radius 9 of 9
07-dhcp-relay-proxy 0 of 11
08-cgnat-ipoe-pba 14 of 14
10-cgnat-pppoe-pba 14 of 14
12-cgnat-ha-ipoe-pba 28 of 28
13-cgnat-ha-pppoe-pba 28 of 28
14-ha-failover-ipoe 26 of 26
15-ha-failover-pppoe 26 of 26
16-ha-failover-radius-ipoe 23 of 25
17-ha-tracker-promotion-ipoe 28 of 28
20-routing-policy 21 of 21
21-cli-openapi 16 of 16
22-mss-clamp 9 of 9
23-radius-coa 16 of 16
24-vrf-lite 19 of 19
26-dhcpv6-ldra 0 of 6
27-api-pagination 0 of 7
28-northbound-api-tls 5 of 5
29-radius-vrf 11 of 11
30-l2tp-lns 0 of 11
31-l2tp-lac 11 of 11
32-ipoe-opdb-restore 1 of 3, 1 skipped
33-pppoe-opdb-restore 2 of 3, 1 skipped
34-cgnat-restart-idempotent 2 of 2
35-ipoe-family-disabled 7 of 7
36-aaa-empty-username 5 of 5
37-ha-graceful-switchover-ipoe 27 of 27
38-l2gw-static 9 of 9
39-l2gw-dynamic 13 of 13
40-l2gw-packet-trigger 9 of 9
40-l2gw-vxlan 10 of 10
42-ipoe-vxlan-pwhe 8 of 8
43-pppoe-vxlan-pwhe 8 of 8
44-lac-vxlan-pwhe 11 of 11
45-ha-pwhe-ipoe 14 of 14
46-l2gw-evpn 10 of 10
47-ipoe-evpn-pwhe 8 of 8
48-pppoe-evpn-pwhe 8 of 8
49-lac-evpn-pwhe 11 of 11
50-ha-evpn-pwhe 17 of 17
51-ipoe-hqos-svlan 13 of 13
52-pppoe-hqos-svlan 10 of 10
53-cgnat-vrf 15 of 15

The six that were red in that pass, and what each one was:

  • 07-dhcp-relay-proxy and 30-l2tp-lns: the box had never built the Kea and xl2tpd images, so their deploys failed. With the images built, 07 passes 11 of 11; 30 failed at bootstrap with create host-interface eth1: Invalid interface (-71): its topology set OSVBNG_NUM_INTERFACES: "1", and the entrypoint builds its wait list as eth0 plus eth1 to eth(N-1), so it waited for eth0 alone and started VPP before containerlab had wired eth1. Main passes it by timing (11 of 11 on a main-built image here); the branch lost twice. The count is now 2, and 29 and 31 had the same undercount (three data links, count 3) and are corrected to 4; 30 passes 11 of 11 twice after; 31 passes 11 of 11 with its count; 29 failed once more on a different window (eth2 present at the wait, gone by the time VPP created it, containerlab still wiring) and passes 11 of 11 on the next run.
  • 26-dhcpv6-ldra and 27-api-pagination: the BNG container came up before containerlab had wired its interfaces (the same restart race) under back-to-back labs. Both pass standalone on this branch (6 of 6, 7 of 7) and on a main-built image.
  • 16-ha-failover-radius-ipoe: 23 of 25 here (3 of 5 sessions synced to the standby at kill time). On a main-built image on the same box it is 4 of 25, so the branch is not what fails it; it needs its own look.
  • 32-ipoe-opdb-restore: the VPP-restart step lost the gateway ping inside its 30-second window once. It passes twice in a row on this branch afterwards and once on main; the test's own documentation notes the window.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cgnat: inside VRF is hardcoded to 0 on every programming path, so translation only works in the default VRF

1 participant