fix(docker): wait for the dataplane API socket before starting osvbngd

[BSpendlove]

Problem

Three suites failed the nightly sweep on 20 August (01-smoke, 13-cgnat-ha-pppoe-pba, 17-ha-tracker-promotion-ipoe), and two failed the same way on the 19th (13-cgnat-ha-pppoe-pba, 16-ha-failover-radius-ipoe). No product code had changed since the last green sweep and the dataplane under test was identical, so the failure is in how the container starts, not in what it runs.

The entrypoint launched VPP, slept a fixed five seconds, probed the CLI socket once, printed Dataplane API not responding yet when that failed, and continued anyway. Roughly eight seconds after launch it exec'd osvbngd, which connects to /run/osvbng/dataplane_api.sock immediately and exits when the socket is missing. That is a race against VPP's startup, and it is decided in seconds.

On a box running several labs at once VPP loses it. From the rig's captured container log:

Dataplane process running (PID 36)
connect: No such file or directory
Dataplane API not responding yet
chmod: cannot access '/run/osvbng/dataplane_api.sock': No such file or directory
Failed to connect to VPP: VPP API socket file /run/osvbng/dataplane_api.sock does not exist

VPP was alive and still in plugin init; its plugin banners appear after that line. osvbngd is PID 1, so its exit took the container with it, containerlab sets restart=always, and the restarted container came back with a fresh docker eth0 but without the veths containerlab wired into the original netns:

Error: Peer netns reference is invalid.
ERROR: Timeout waiting for interfaces after 300 seconds

From there it waited 300s, exited, and restarted forever. docker ps taken mid-sweep shows it: the wedged BNGs read "created 11 minutes ago, Up About a minute" while every healthy container in the same labs reads "Up 11 minutes".

That "not ready at five seconds" does not mean "never ready" is visible in the same sweep. Two containers, ha-tracker-promotion-ipoe-bng2 and ipoe-hqos-svlan-bng1, printed Dataplane API not responding yet at the probe and then connected fine about three seconds later, when osvbngd happened to start after the socket appeared. The margin between working and permanently wedged was a few seconds of scheduling luck.

Change

Poll for the API socket rather than sleeping past it, bounded by OSVBNG_DATAPLANE_READY_TIMEOUT (60s default), failing with the dataplane log when the socket never appears or when the process dies during startup. 60s is roughly twenty times the observed recovery margin while still failing well inside the suite timeout; it is the reviewable number here, so say if you want it elsewhere.

Dropping the fixed sleeps also makes healthy startup faster: the socket is ready about a second in, against the eight seconds the entrypoint always spent.

This does not change the fact that any osvbngd exit wedges a lab node under restart=always. It removes the cause that has actually been firing.

Verification

Ran on the rig's dataplane build (dataplane-latest, fb3580a4), the same image contents the failing sweep used, confirmed by hashing the plugin .so in the image against the one in the deb.

• The three failed suites, run serially: 01-smoke 10/10, 13-cgnat-ha-pppoe-pba 28/28, 17-ha-tracker-promotion-ipoe 28/28.
• Five suites in parallel with the rig's per-slot core map, on the fixed image: four green. 13-cgnat-ha-pppoe-pba failed two of 28 on Only 0/10 stream flows verified, NAT traffic not flowing bidirectionally, after both BNGs came up healthy. That is traffic forwarding under contention, not startup, and the suite passes 28/28 when run alone.
• The added wait logic tested directly on all three paths: socket appears late (proceeds), socket never appears (bounded, fails with the log), process dies (fails immediately).
• Live container shows Dataplane API socket ready after 1s (PID 36), restart count 0.

Not verified: the wedge itself never reproduced locally, including under the five-lane parallel run above. This box has 24 cores against the rig's 16, so five lanes leave eight cores spare where the rig's budget is saturated exactly. The mechanism is established from the rig's own logs rather than from a local reproduction, and the real test is a nightly sweep on the rig.

Related, not fixed here

The rig's core budget assumes one BNG per lab: five slots times three cores is 15, plus main=0, exactly the box's 16. HA suites run two BNGs, so both land on the same worker and cp cores, and every VPP on the box puts its main thread on core 0, up to ten of them. VPP's main thread is what creates the API socket, which is why the HA suites are the repeat offenders on both nights. Worth a separate look at lane count or per-lab core assignment.

Three evidence gaps also turned up while chasing this, all separate changes: a failing suite's own container logs are destroyed by robot's teardown before the workflow captures them, Capture Container Logs in tests/common.robot is a silent no-op (/tmp/test-logs on the rig is empty and dated 16 August), and the run-qa-tests.sh console log holding the per-test detail is never copied to /var/log/osvbng-ci.